Robert Gates confirmed Gen David McKiernan would effectively be sacked less than a year after taking command.
He will be replaced by Gen Stanley McChrystal, who is seen as having a better understanding of the conflict.
The change comes as the US boosts troops numbers in Afghanistan and prepares for a change in strategy.
Gen McKiernan's time as US commander in Afghanistan has coincided with a surge in violence.
His successor currently serves as the director of US Joint Chiefs of Staff, and was previously a director of special operations forces.
Announcing the removal of Gen McKiernan from his role, Mr Gates said new military leadership was needed to go along with a new strategy and a new ambassador.
"This is the right time to make the change," he said.
"Our mission there requires new thinking and new approaches from our military leaders."
He said the decision was in the best interest of US national security and the success of the Afghanistan mission.
It was made after consulting the chairman of the Joint Chiefs of Staff, Admiral Mike Mullen, and the commander of the US Central Command, Gen David Petraeus.
The change also had the approval of President Barack Obama.
Correspondents say Gen McChrystal is a specialist in the kind of counter-insurgency strategy the Obama administration plans to implement in Afghanistan.
Strategic goals
The change comes as President Obama's administration prepares to send thousands of extra troops to Afghanistan, and amid pressure on international forces to reduce the numbers of civilians killed by coalition air strikes.
With plans announced for a phased pullout of US troops from Iraq, Afghanistan was recently confirmed as the primary focus of US military operations.
The US is sending 21,000 additional troops to the country, to join an existing force of 38,000.
However, the new strategy is expected to pair non-military methods and reconstruction with a stronger armed force on the ground.
But relations with President Hamid Karzai's Afghan government have been strained by a recent air strike which some Afghan officials say killed as many as 150 people.
On Sunday, Gen Petraeus said "tactical actions" should not undermine strategic goals.
Gen McKiernan, who will also lose his role as head of the Nato-led International Security Assistance Force (Isaf), recently described the situation in the country as a "stalemate".
