Exit polls show his Democratic Party of Japan overwhelmingly defeated the Liberal Democratic Party, which has governed almost unbroken since 1955.
PM Taro Aso has conceded defeat and said he would resign as LDP head.
Media forecasts give the DPJ 308 of the 480 seats in the lower house to the LDP's 119, almost an exact reversal of their previous standing.
Japan's Nikkei stock market index jumped to an 11-month high in early trading as the scale of the DPJ's victory became clear, but the rise of the yen and Chinese stock falls led to an overall fall of 0.3%.
Official results are still to be released.
Mr Hatoyama, the wealthy heir to an industrial and political dynasty, is expected to announce a transition team later in the day.
He is expected to be confirmed as prime minister when parliament meets in about two weeks.
His Cabinet is expected to be in place by then, and his party is also in coalition talks with two smaller opposition parties whose support it needs in the upper house.
"It's taken a long time, but we have at last reached the starting line," Mr Hatoyama told a news conference at his home in Tokyo on Monday.
"This is by no means the destination. At long last we are able to move politics, to create a new kind of politics that will fulfil the expectations of the people."
Mr Aso said he would step down as LDP leader - his successor is expected to be named in September.
"I have no plan to run for re-election," he said, quoted by the Associated Press. "The most important thing is rejuvenating our party."
Kotaro Tamura, another LDP lawmaker, said: "We made too many mistakes. Very crucial mistakes... we changed prime minister three times without holding an election."
'Close partnership'
Correspondents say attention will now to turn to whether he can deliver on his election promises.
He must steer the world's second biggest economy back to sustainable growth after a crushing recession, and tackle record unemployment.
Mr Hatoyama has also promised to expand the welfare state, even though Japan is already deeply in debt and the rapidly ageing population is straining social security budgets.
On foreign affairs, the DPJ says it plans to create a new diplomacy less subservient to the US and to improve relations with Japan's Asian neighbours.
The White House has already said it hopes to forge a strong ties with the incoming government.
"We are confident that the strong US-Japan alliance and the close partnership between our two countries will continue to flourish under the leadership of the next government," White House spokesman Robert Gibbs said.
Kyodo News agency put turn-out at 69%, up from 67.5% in 2005.
Officials said people turned out despite a combination of typhoon-triggered rainfall around Tokyo and a government warning that a swine flu epidemic was under way.
