MANILA—At the center of a gunbattle and hostage standoff between Islamic State-linked militants and government troops in the southern Philippines is a faction leader who is trying to build a caliphate in the troubled region.

Isnilon Hapilon, 51, leads a branch of the Abu Sayyaf group known for kidnapping and beheading foreign tourists. His faction, which President Rodrigo Duterte has pledged to destroy, has preoccupied the Philippine military since 2014, when he publicly declared loyalty to Islamic State.

Mr. Hapilon’s group hails from the southwestern island of Basilan, but it has brought fighters to the main southern island in Mindanao and raised his profile by striking alliances with other militant groups. Other elements have split off and conducted their own operations, some more ransom-hungry, than Mr. Hapilon’s overtly religious focus.

“I do think that Hapilon has taken on something of a symbolic importance,” said Marc Singer, director of business intelligence at risk consultancy PSA Group in Manila. “Central Mindanao is now a kind of complex chess board where the pieces are all in disarray and you have any number of disparate groups” fighting for influence.

A combined police and military operation on Tuesday sought to arrest Mr. Hapilon, sparking a gunfight in the southern city of Marawi. The arrest attempt was unsuccessful. Meanwhile, rebels from the Maute group, another organization aligned with Mr. Hapilon, attacked the city, occupied buildings and waved a black Islamic State flag.

The standoff escalated Wednesday, when a Roman Catholic bishop in Marawi said he received a call from the militants the previous evening. The bishop said they had taken several of his congregation and a priest hostage, according to the official news website of the Catholic Bishops’ Conference of the Philippines. The militants threatened to kill the hostages if their demands of a cease-fire and safe passage out of Marawi weren’t met, according to the Conference news service. There was no immediate confirmation from government officials.

Mr. Duterte responded to the attack on Marawi during a trip to Moscow by declaring martial law for the entire main southern island of Mindanao, home to around 22 million people. In a press conference in Manila on Wednesday, Mr. Duterte said sporadic skirmishes with the Maute group continued, “showing the group’s capability to sow terror.” He said a local police chief had been beheaded by the group and that he would consider extending martial law to the entire Philippines if he felt it necessary.

Mindanao’s lawless western regions have become a concern for security services internationally as Islamic State’s strongholds in the Middle East are destroyed. Dozens of fighters loyal to the group have fled to the Philippines to join local extremist gangs, seeking a way to continue their fight. Several groups are “auditioning for recognition” by Islamic State, Alan Peter Cayetano, the Philippines’ foreign secretary said during a press conference in Russia.

Abu Sayyaf started as a religious splinter group from an independence movement in the Muslim south, loosely allying itself with al Qaeda in the mid-1990s. Its tactics have varied, from kidnapping and murder, to a ferry bombing in 2004 and a bus bombing in Manila in 2005.

The Philippine armed forces, which received assistance from the U.S. after the Sept. 11, 2001 terror attacks, have periodically killed Abu Sayyaf leaders and splintered the group further. Now, it survives mostly on kidnappings for ransom and other illicit business, with Mr. Hapilon one of the few leaders with enough seniority to unite members inside and outside Abu Sayyaf itself.

Mr. Hapilon’s stature was boosted when Islamic State leadership anointed him as its “emir” in the Philippines. Philippine officials said Mr. Hapilon was wounded in a January airstrike, sparking speculation at the time that he might have been killed.

Mr. Hapilon is on the U.S. Department of Justice list of most-wanted terrorists world-wide with a reward of up to $5 million. He was personally involved in several kidnappings, including a 2001 incident when 20 people including an American who was later beheaded were taken from a high-end island resort.

The Philippines’ inability to dislodge groups such as Abu Sayyaf and Maute has alarmed its neighbors. In February, Australian Foreign Affairs Minister Julie Bishop said her country was readying itself for the possibility of Islamic State declaring a caliphate in the southern Philippines.

Yet many security experts play down the capability of Mr. Hapilon and his allies to set up a caliphate, arguing that Mindanao’s militant groups are too splintered by ethnicity and ideology to gain the critical mass necessary to gain de facto control of a large amount of territory.

The Maute group, founded by two brothers by that name, rose to notoriety after a series of attacks last year including one in August where gunmen seized control of Butig, a town near Marawi. The military eventually reclaimed the town several days later.

Mr. Hapilon and the Maute group “used the ISIS brand to build a fiercer image,” said Joseph Franco, a research fellow at the S. Rajaratnam School of International Studies in Singapore. “It’s like wearing a Che Guevara shirt,” he said, but “trying to flash the ISIS creds when the operational links are very tenuous.”