Sweden’s parliament voted to oust Prime Minister Stefan Löfven in a vote of confidence Tuesday, deepening the nation’s political uncertainty in the aftermath of an inconclusive election.

Of parliament’s 349 members, 204 voted against Mr. Löfven remaining in his position. He will lead a transitional government until a new one can be formed, but that process could take months.

The vote was a direct consequence of the Sept. 9 election, in which no single party secured a clear majority and Mr. Löfven’s Social Democrats lost support amid a surge in popularity for the nationalist Sweden Democrats.

That rise is the latest example of how the growing prominence of anti-immigration groups has shaken up the region’s political order in recent years. Parties long considered on the political fringes have joined the government in Austria, toppled the old establishment in Italy, and become the largest opposition force in Germany.

Europe’s political landscape is now more fractured than ever, with many countries having struggled to form stable governments able to make bold policy choices.

Sweden—an affluent and politically stable nation, known for its consensual politics and generous welfare state—is now entering uncharted territory.

The Sweden Democrats’ swell in support—securing 62 seats in the election—means it will play a pivotal role in the nation’s political outlook, but all of the other parties have refused to collaborate with it.

Unless an unlikely grand coalition of the left and right is forged, support from the Sweden Democrats would be needed to form a government.

The Center Party and Liberals, two of the four parties that comprise the center-right Alliance, have said they would quit the group if the Moderates and Christian Democrats soften their stances and negotiate a deal with the Sweden Democrats to garner their support.

Following Tuesday’s vote, newly elected parliamentary speaker Andreas Norlén, a lawmaker from the Moderate Party, will explore options toward forming a new government. Initial talks are due to begin Thursday.

But that could take weeks or even months, said Torbjörn Isaksson, chief analyst at Nordea Markets.

“It is still open how the new government will look,” he said. “But with Norlén as speaker the probability increases that the new PM will be Ulf Kristersson, the leader of the Moderates, as the speaker has a key role in the process electing a PM.”

Under Swedish rules, talks over forming a government can last for four rounds. Should the talks fail, a fresh election would be held.

But there is no time limit for when the speaker must propose a new prime minister so although an extra election can’t be ruled out completely, it seems unlikely, Mr. Isaksson said.