The result is a huge blow to President Barack Obama, whose healthcare reform programme is now in doubt.
Democrat Martha Coakley conceded she had lost the race after early results gave Mr Brown a healthy lead.
The Republican win has robbed the Democrats of their filibuster-proof 60-seat majority in the Senate.
This will make it much harder for Mr Obama to pass a healthcare reform bill - the most important domestic policy objective of his first year as president.
The BBC's Paul Adams, in Boston, says Ms Coakley's defeat is a humiliating blow for the Democrats and their agenda, and a deeply unwelcome anniversary present for President Obama a year after his inauguration.
He adds that it is one of the biggest political upsets in years, and a devastating blow for the Democrats in a seat held for almost half a century by Edward Kennedy, a colossus of the party.
In a celebratory speech, Mr Brown said that the voters of Massachusetts had "delivered a great victory".
He said: "Tonight, the independent voice of Massachusetts has spoken. The voters of this commonwealth defied the odds and the experts."
He also criticised President Obama's healthcare reform proposals, saying they would raise taxes, destroy jobs and increase debt.
Speaking to her supporters after conceding the election in a telephone call to Mr Brown, Ms Coakley admitted that she was "heartbroken at the result".
Mr Obama had campaigned personally on behalf of Ms Coakley.
Senate Democratic leader Harry Reid said he would welcome Mr Brown to the Senate.
He added that senators "will move to seat him as soon as the proper paperwork has been received" from Massachusetts officials.
Lacklustre campaign
Analysts say the race should have been an easy win for Ms Coakley in a state which traditionally has voted for Democratic candidates for the US Senate.
