KUALA LUMPUR, Malaysia—Rapid-fire moves by Malaysia and North Korea to hold each other’s citizens hostage on Tuesday substantially escalated diplomatic tensions that have been simmering since the estranged half brother of North Korea’s ruler Kim Jong Un was killed in Kuala Lumpur three weeks ago.

Both nations banned each other’s citizens visiting or resident in their countries from leaving, an unusual and swift deterioration in relations that put both at risk of violating international law and increased the likelihood of further isolating North Korea by upending a rare friendly relationship with a foreign government.

Experts in international relations said the sweeping, tit-for-tat bans were likely the first of their kind since World War II. They were rooted in the slaying of Kim Jong Nam at Kuala Lumpur’s international airport on Feb. 13, which Malaysia has blamed on a North Korean spy operation using VX nerve agent, a deadly chemical. North Korea has accused Malaysia of a biased and bungled investigation.

Photos: The Kim Jong Nam Murder Case

The airport death of a dictator’s half brother becomes a murder mystery and then a diplomatic flashpoint

On March 7 North Korea raised the stakes, barring all Malaysian citizens in the country from leaving; Malaysia quickly slapped a reciprocal ban on North Korean embassy officials, later widened to cover all North Koreans. Malaysian police said three North Korean suspects still in the country were hiding in the embassy, and that they were willing to wait five years to speak with them. Here police cordoned off the embassy’s main entrance, a blockade lifted later in the day. A

Kim Jong Nam with his father, not-yet-dictator Kim Jong Il, in 1981. At one time considered the heir apparent, Kim Jong Nam is believed to have fallen out of …

After 2001, Kim Jong Nam never returned to live in North Korea, instead living in China—Beijing and then Macau—a sometime critic of the Pyongyang regime. That ended on Feb. 13, when he was killed at Kuala Lumpur International Airport in Malaysia by two women. This still image from closed-circuit TV footage shows the attack, circled in red—one of the women, in a white top, reaching around Mr. Kim to apply something to his face. FUJI

His body was taken for an autopsy to the forensic wing of Hospital Kuala Lumpur—where police said someone tried to break in.

Malaysia’s Inspector-General of Police Khalid Abu Bakar at a Feb. 22 press conference, identifying an official at North Korea’s embassy and an employee of its state airline as among seven North Korea suspects still at large—four of whom had fled the country, police said. One other North Korean was in custody, in addition to the two women—one a Vietnamese national, the other Indonesian. The North Korean government declared that the death was due to natural causes (a ‘heart stroke’) and declined even to confirm the identity of the victim, instead referring to him as ‘Kim Chol,’ the name on his diplomatic passport.

On Feb. 24, 11 days after the killing, police identified the cause of Kim Jong Nam’s death as the toxic nerve agent VX, a weapon of mass destruction banned by the United Nations. Police ordered a sweep of the airport for hazardous substances; here, a hazmat crew scanning the decontamination zone on Feb. 26.

On March 1, the two women were charged with murder, which carries a sentence of death by hanging. Here 25-year-old Indonesian Siti Aisyah, in the same red shirt in which she was arrested, arrived at court. No official pleas were entered, and the next court date was set for April 13.

The only North Korean detained in the case, Kuala Lumpur resident Ri Jong Chol, was released March 3 due to insufficient evidence and deported; here he appreciated a middle seat on a flight to Beijing. After leaving, Mr. Ri accused Malaysia of conspiring with other nations to damage North Korea’s honor, echoing an accusation by North Korea’s ambassador to Malaysia, Kang Chol.

Make that, former ambassador. As once-friendly relations frayed, Malaysia first said it would cancel visa-free entry for North Koreans, and then that it would expel Mr. Kang. He arrived in suit and a tie—and wearing his official-issue lapel pin bearing the images of Kim Jong Il and his father, North Korea founder Kim Il Sung—at Kuala Lumpur airport the afternoon of March 6. That same day North Korea likewise declared the Malaysian ambassador, whom Malaysia had recalled home on Feb. 20, persona non grata.

On March 7 North Korea raised the stakes, barring all Malaysian citizens in the country from leaving; Malaysia quickly slapped a reciprocal ban on North Korean embassy officials, later widened to cover all North Koreans. Malaysian police said three North Korean suspects still in the country were hiding in the embassy, and that they were willing to wait five years to speak with them. Here police cordoned off the embassy’s main entrance, a blockade lifted later in the day.

“North Korea is really shooting itself in the foot,” said Jeffrey Robertson, a professor who specializes in South Korean diplomacy at Yonsei University in Seoul. North Korea’s actions will give fresh momentum to South Korean attempts to persuade other states—or even the United Nations—to sever ties with Pyongyang, he said.

North Korea threw down another challenge at the international community Monday by launching four ballistic missiles that landed in Japanese waters. It also ordered the expulsion of Malaysia’s ambassador, who was at home for consultations, after the Southeast Asian country ordered Pyongyang’s ambassador to leave following disparaging remarks.

Pyongyang’s announcement that it was banning visiting Malaysians from leaving North Korea could violate international law, legal experts said. Malaysia quickly retaliated, with Prime Minister Najib Razak declaring a similar ban and calling an emergency meeting of Malaysia’s national security council. After the meeting, Mr. Najib said on his personal blog that his priority was to bring home Malaysians in North Korea.

“Malaysia has a legitimate justification for banning North Korean nationals from leaving: There has been a grave violation of Malaysian law and perhaps also international law,” said Robert Volterra, a partner at international law specialist Volterra Fietta.

North Korea, by contrast, would struggle to justify its actions, which are based on Pyongyang’s allegations that Malaysia is colluding with other foreign states to harm North Korea’s reputation, Mr. Volterra said.

Analysts estimate that several dozen Malaysian business people and tourists are trapped in North Korea. Malaysian government officials say about a thousand North Koreans are in Malaysia. Neither government has provided exact figures.

Malaysian officials have identified seven North Korean suspects still at large in the killing of Mr. Kim and say they believe at least three of them are being sheltered by the North Korean embassy in Kuala Lumpur. South Korean intelligence officials say most of the seven suspects come from North Korea’s state security and foreign ministries. Only two suspects are in custody, women from Vietnam and Indonesia who claim they were duped into applying what Malaysia says was VX nerve agent to Mr. Kim’s face.

”For national security reasons, any government can restrict travel in and out of their borders for a short term” but in the longer term this would violate rights such as the U.N. Declaration of Human Rights, said Lucy Reed, director of the Center for International Law at the National University of Singapore.

Legal experts said any dispute about the departure restrictions would have no clear resolution mechanism, but a breach of international law by North Korea could prompt discussion of sanctions tightening by the U.N. Security Council.

Until Mr. Kim was killed, the two countries kept a cordial diplomatic presence in their capitals. The relationship has been unwound with surprising speed. Malaysia has ended visa-free travel for North Koreans and refused demands by Pyongyang’s diplomats to turn over Mr. Kim’s body to them.

“That hurts the North Koreans. There are only a handful of countries that they can go to visa-free, and you would think they would like to keep as many of them as they possibly could, especially one with good infrastructure and links to other countries in the region,” said Andray Abrahamian, associate director of research with Singapore-based Choson Exchange, a nonprofit organization that teaches entrepreneurship to North Koreans.

Malaysian police briefly blockaded the North Korean embassy Tuesday. The country’s deputy home minister told reporters police would wait years if necessary to question the suspects.

A U.N. treaty bars Malaysia from entering the embassy or arresting diplomatic staff. The same treaty grants all members of a diplomatic mission freedom of movement unless for reasons of national security. That cover wouldn’t extend to the two nondiplomatic North Korean suspects Malaysian police say are in the embassy. As long as the two remain in the embassy, they can’t be arrested or questioned.

The current impasse may take time to resolve, said Mr. Volterra. “There is a long history of people taking shelter in embassies.”

—Yantoultra Ngui in Penang, Malaysia, and Ben Otto in Kuala Lumpur contributed to this article.