Any coalition that emerges is likely to be highly unstable.
In Saxony, the conservatives won 42 seats to the AfD's 41, while Sarah Wagenknecht's party came in third with 15 seats.
In Thuringia, Scholz's Social Democratic Party (SPD) won just six seats, while its coalition partners, the Greens and the liberal FDP, won none. The SPD also performed poorly in Saxony, coming in fifth.
This election clearly showed the unpopularity of Germany's ruling 'Traffic Light' coalition, named after the parties' respective colours: red, yellow and green.
With all three ruling parties performing poorly, they are likely to be more vocal about their stance on a national coalition.
Already, leading figures from both parties are saying they must stand up for their values. This is likely to lead to more conflict within the national government. Ministers are saying they will not break up the coalition and bring down the government, but the fact that they are saying this is a sign of how difficult it is within the coalition.
Ms Weidel said the people had “voted out” of the coalition government and urged Mr Scholz and his partners to “pack their bags and empty the chairs” because “the voters want a different government and different politics”.
The biggest issue for AfD voters on Sunday was immigration, particularly refugees and asylum seekers.
Although the AfD still cannot exercise governmental power at the regional or national level, the party exerts influence in mainstream politics.
When the AfD entered the Berlin Bundestag in 2017, critics said its fierce anti-immigrant rhetoric had muddied the debate.
Some say the political and media discourse has become more aggressive, with CDU leader Friedrich Merz accused of parroting the AfD's rhetoric.
Either way, in a bid to win back AfD voters, mainstream parties are taking a harder line on issues like immigration and pushing for measures to make it easier to deport people whose asylum applications have been rejected.
Aslihan Yesilkaya-Yurtbay, the federal chairman of the Turkish Community Association in Berlin, called the election results “shocking and scary.” She added that many young people of her generation were already planning to leave Germany.
“The future of this country for citizens with immigrant backgrounds is being called into question,” she said.
AfD also wants to stop arms supplies to Ukraine, as does Sarah Wagenknecht's BSW.
About 5 million eastern Germans were eligible to vote on Sunday.
In the third eastern state, Brandenburg, where a vote is scheduled for three weeks later, the polls show the AfD ahead, but the Social Democrats and conservatives just a few points behind.
