KUALA LUMPUR—Malaysia’s open-border policy has long helped lend it notoriety as a way station for conspirators to plot terror attacks abroad.

Now, many people worry that two major terror-linked plots on Malaysia’s own soil show that the Muslim-majority nation is increasingly becoming a host to other people’s deadly conflicts.

The plots were the February killing of North Korean dictator Kim Jong Un’s half brother with VX nerve agent, which police blame on a North Korean-led hit team, and an alleged Islamic State-linked attempt to attack King Salman bin Abdulaziz’s visiting delegation last week with a car bomb. They followed Islamic State’s first successful attack in Malaysia last year and dozens of terror-related arrests in 2016.

“In the past, terrorists did not target Malaysia, but today the threat has changed,” said Rohan Gunaratna, a professor of security studies at Nanyang Technical University in Singapore. “Malaysia has become a really important target.”

Khalid Abu Bakar, Malaysia’s chief of police, told The Wall Street Journal that authorities have raised the alert level since the two plots last month but that “people are free to move and do their routine.”

The Foreign Ministry website says “Malaysia condemns all acts, methods and practices of terrorism and transnational crimes.” After Australia and Britain issued travel warnings for Malaysia last year, a Foreign Ministry spokesman said there was “nothing to be alarmed of.” Malaysia later asked the countries to lower their threat levels. Britain has retained its warning, while Australia softened its language slightly earlier this year but left its “level of advice” the same.

These and other events have sparked debate within Malaysia’s government over its open-door policy. The country has maintained relaxed borders for decades to help it forge a business and travel hub between East Asia and the Middle East that could rival Singapore’s and Thailand’s. Tourism is now Malaysia’s second-largest foreign exchange earner after manufactured exports.

Malaysia is among the most welcoming nations, extending visa-free or visa-on-arrival access to 164 countries and territories, according to Canada’s Arton Capital. By comparison, the U.S. does so for 43 countries and France, 91.

When many Western nations curbed travel from some Middle Eastern states after the Sept. 11, 2001, terrorist attacks in the U.S., Malaysia eased travel privileges for the region, luring millions of new visitors even as it added new security precautions. Until recently, even North Koreans enjoyed visa-free travel to Malaysia, a privilege only a handful of countries extend it, including Haiti and Belarus.

Proposals to alter the open-border policy have been contested. Last year, Malaysia’s National Security Council, a state body, proposed restoring visa requirements for several Middle East nations, citing the need for greater controls following an Islamic State-linked attack. Prime Minister Najib Razak’s cabinet quickly rejected the idea, saying it was discriminatory, diplomatically complex and bad for the economy.

Malaysia has largely maintained its open-borders strategy over the years even it has cracked down on illegal immigration. In January, the government said it had detained 150,000 illegal immigrants in a multi-year effort. In December, it curbed student visas over concerns some were involved in terrorist activities. And last month it rescinded North Korea’s visa privileges.

For decades, Malaysia’s open borders have attracted transnational criminals political exiles, offering a transportation hub and good business infrastructure, security analysts say.​

The Sept. 11 attacks and the U.S.S. Cole bombing in the Gulf of Aden were partly planed by Al Qaeda operatives meeting in Kuala Lumpur, American officials say. The 9/11 Commission Report said militants linked to Osama bin Laden saw Malaysia as an ideal place for wounded militants to recover.

Indonesians Abu Bakar Bashir, spiritual head of a Southeast Asian affiliate of Al Qaeda, and Riduan Isamuddin—known as Hambali—once Southeast Asia’s most-wanted terrorist, both lived in Malaysia in the 1980s and 1990s. Mr. Bashir was convicted and imprisoned in Indonesia; Mr. Isamuddin is a Guantanamo Bay detainee. Leaders of an Indonesian separatist movement in the Aceh region also set up a base in Malaysia.

Another regular Malaysia visitor is Zakir Naik, an Indian Muslim preacher who has urged all Muslims to be terrorists, praised Bin Laden and called for the death penalty for homosexuality. India banned Mr. Zakir’s foundation for five years last year, citing subversive speeches and divisive ideology. He has denied that he supports terrorism.

“As long as Malaysia allows political exiles who are still active to live in Malaysia, political violence not related to Malaysia will occur on Malaysian soil,” James Chin, director of the Asia Institute at the University of Tasmania wrote in a recent article. “The Kim killing was not the first and will not be the last.”

The Ministry of Foreign Affairs didn’t respond to requests for comment.

The alleged Saudi plot illustrates Malaysia’s predicament.

Police arrested seven men from Yemen, Indonesia, Malaysia and an undisclosed East Asian nation tied to planned attacks and said one man had ties to a regional terror group that uses Malaysia as a transit point and “hideout.” They said the man entered the country in 2011 on a fake student visa and enrolled at a public university here. The four Yemeni men helped a syndicate make fake travel documents, police said. Some of the men were taking orders from a Malaysian who joined Islamic State in Syria in 2015 with the intent of launching attacks back home, authorities said.

“Malaysia is a place where local and international networks of all sorts are built and seeds are sowed slowly and steadily because of its peaceful and apathetic setting,” said Achmad Sukarsono, an analyst with Washington, DC-based Eurasia Group, a risk consultancy. “As long as they lie low, stay quiet and act normal, people with extreme ideas can connect with like-minded others.”