North Korea fired what appeared to be a new type of ballistic missile Sunday, days after the election of a new South Korean president who advocates more engagement with Pyongyang.

Experts said the missile had a greater range than any other weapon North Korea has successfully launched, and U.S. President Donald Trump said the missile landed “close to Russian soil.”

The missile rose to a height of about 1,240 miles, Japanese Defense Minister Tomomi Inada said, a much steeper trajectory than usual for a North Korean missile test. She said officials were looking into the possibility that it was a “new type of ballistic missile.”

Yoshihide Suga, Japan’s chief cabinet secretary, said the missile traveled for about 30 minutes and landed some 500 miles east of the launch site. A spokesman for South Korea’s Joint Chiefs of Staff estimated the distance at 435 miles.

Maj. Rob Shuford, a spokesman for U.S. Pacific Command in Hawaii, said the launch didn’t appear to be consistent with that of an intercontinental ballistic missile.

Independent experts said the missile, if fired at a conventional angle, could have flown 2,800 miles—far enough to reach the U.S. military base in Guam. That is a “considerably longer range than its current missiles,” said David Wright, co-director of the Global Security Program at the Union of Concerned Scientists, in an analysis of the launch.

While North Korea’s Taepodong-2 rocket has flown farther than Sunday’s missile, North Korea classifies it as a satellite launcher that isn’t designed to deliver a warhead back to earth. It is, however, banned by United Nations sanctions because similar technology could be used to make an intercontinental ballistic missile.

The latest launch comes as worries intensify in the international community about North Korea’s nuclear ambitions. Leader Kim Jong Un indicated in January that he seeks to develop an ICBM capable of delivering a nuclear warhead to the continental U.S., and North Korea has stepped up its missile tests since then. North Korea has test-launched missiles 10 times this year.

Moon Jae-in, South Korea’s newly elected president, convened an emergency meeting of the National Security Council hours after Sunday’s launch, according to a spokesman for the presidential Blue House in Seoul.

During the meeting, Mr. Moon condemned the missile launch as “a grave violation of U.N. Security Council resolutions” and expressed frustration at the North for testing a missile just days after Mr. Moon had said in his inaugural address that he would do whatever it takes to establish peace on the Korean Peninsula, according to the spokesman.

Mr. Moon called North Korea’s missile test a “reckless provocation” and promised a decisive response, although he also kept open the possibility of dialogue, calling for “a change in attitude” by Pyongyang, the spokesman said.

The South Korean leader, who took office Wednesday, had campaigned on a policy of seeking closer ties with Pyongyang, primarily through economic engagement. But he also said during the campaign that provocative actions by North Korea would make it more difficult for the country to avoid tightening sanctions sought by the U.S.

Lee Seong-hyon, a research fellow at the Sejong Institute in Seoul, said it was common for North Korea to test a new leader in South Korea, adding that it gave Mr. Moon a chance to burnish his national-security credentials and commitment to Seoul’s traditional alliance with Washington.

“Pyongyang gave an opportunity for Seoul and Washington to publicly affirm their alliance,” said Mr. Lee. “For Moon, by declaring today that there won’t be any ‘unconditional dialogue’ with North Korea, he brushed aside some skepticism that he may be ‘soft’ on North Korea.”

The White House said Mr. Trump had been briefed on the launch. “With the missile impacting so close to Russian soil—in fact, closer to Russia than Japan—the President cannot imagine that Russia is pleased,” the White House said in a statement.

The missile launch came as Chinese President Xi Jinping convened a gathering of world leaders, including Russian President Vladimir Putin, in Beijing to kick off a high-profile infrastructure plan that reflects China’s ambitions for influence across Central Asia and into Europe. China’s Foreign Ministry on Sunday noted Security Council resolutions that limit North Korea’s ballistic-missile launches and urged all parties to “exercise restraint and do nothing to further worsen regional tensions.”

Mr. Putin discussed the situation on the Korean peninsula with his Chinese counterpart during a planned meeting Sunday, and expressed concern over the latest missile test, Kremlin spokesman Dmitry Peskov told reporters, Interfax news agency reported.

Matt Pottinger, the U.S. National Security Council’s senior director for East Asia, was in Beijing for Mr. Xi’s gathering when North Korea launched its latest test. “Any time a ballistic missile lands offshore from your country, especially when it’s an illegal launch of a missile that was built under an illegal program, that would be a problem for any country,” he said.

Mr Pottinger is due to arrive in Seoul on Monday to meet with the new Moon administration, according to South Korea’s quasiofficial Yonhap News Agency.

A spokesman for the Blue House in Seoul said Lt. Gen. H.R. McMaster, the White House’s national-security adviser, had a 25-minute phone call with his South Korean counterpart, Kim Kwan-jin, after the missile test. Both sides condemned the launch and reaffirmed their commitment to working toward a denuclearized North Korea.

North Korea fired the missile about 5 a.m. Pyongyang time from Kusong, near North Korea’s border in the northwest with China. North Korea’s most recent launch from Kusong took place in February, during a summit meeting between Mr. Trump and Japanese Prime Minister Shinzo Abe.

The February launch featured a new type of missile for North Korea, one that uses a solid fuel-powered engine that would give Pyongyang more stealth and reliability in its launches. The test involved an intermediate-range ballistic missile that was modified from a missile that North Korea launched from a submarine last year. It was later paraded through the streets of Pyongyang in April for a national holiday.

The missile that North Korea fired Sunday flew further than the previous one launched from Kusong, and its high trajectory—which missile experts said appeared to be a record for the isolated nation—seemed designed to ensure that it didn’t land in Japan’s territory.

Even so, Mr. Abe said in a statement that the launch was “unacceptable,” adding that Japan would strongly protest to North Korea. In a second statement, Mr. Abe said Japan, the U.S. and South Korea should “analyze information in close coordination.”

Japan’s foreign minister, Fumio Kishida, said he spoke with his South Korean counterpart, Yun Byung-se, and the two agreed to work together to step up pressure on North Korea.

Separately, U.S. Treasury Secretary Steven Mnuchin made North Korea’s nuclear-weapons threat a priority in talks with finance ministers from the Group of Seven leading economies over the weekend in Bari, Italy.

Treasury officials said that reining in North Korea as a national security threat, along with terror financing, were discussed in both the general G-7 meetings and all the secretary’s bilateral meetings. Mr. Mnuchin “underscored the importance of working together to address the threats posed by North Korea,” the Treasury Department said after his meeting with German Minister of Finance Wolfgang Schäuble.

“We are entertaining all the tools in our arsenal,” a senior Treasury official said Saturday after the G-7 talks, but before the North’s latest ballistic missile launch.

Those potential actions include using the Treasury’s Office of Foreign Assets Control, the department responsible for sanctions, “and similar ones overseas to try to stop [North Korea],” the official said.

“We don’t comment on future sanctions, but we’re obviously going to consider every tool in our arsenal to combat any illicit activity and terror financing,” he said, emphasizing the position has been a standing policy of the Trump administration.

—Alastair Gale and Peter Landers in Tokyo, Chun Han Wong and Charles Hutzler in Beijing, James Marson in Moscow, and Ian Talley in Washington contributed to this article.

Write to Jonathan Cheng at jonathan.cheng@wsj.com