The justices on Monday reversed a lower court ruling that had allowed a lawsuit brought by Javaid Iqbal to go forward.
Mr Iqbal, from Pakistan, argued the two officials were responsible for a policy that saw him singled out for abuse on the basis of his religion and race.
The court ruled that his complaint failed to back up this claim.
Mr Iqbal spent some six months in solitary confinement in a federal prison in Brooklyn in 2002.
In his lawsuit, he said he had suffered physical and verbal abuse and had been singled out for mistreatment because of ethnic and religious discrimination.
The government argued that there was nothing to link Mr Mueller and Mr Ashcroft to the alleged abuse of Mr Iqbal.
'Secure conditions'
In a ruling by five votes to four, the Supreme Court justices overturned a ruling by a New York appeals court that had allowed Mr Iqbal's lawsuit to proceed.
"The complaint does not show or even intimate that petitioners purposefully housed detainees" in prison "due to their race, religion or national origin," wrote Justice Anthony Kennedy in the court's majority opinion.
"All it plausibly suggests is that the nation's top law enforcement officers, in the aftermath of a devastating attack, sought to keep suspected terrorists in the most secure conditions available until the suspects could be cleared of terrorist activity."
The court ruled that Mr Iqbal may be able to bring a case against others officials if "unconstitutional misconduct" could be proved.
Mr Iqbal was arrested in November 2001 on charges unrelated to terrorism. His lawsuit alleged that two months later he was moved to a prison in New York where he was held for more than 150 days in solitary confinement, the Associated Press reports.
He was cleared of any links to terrorism and deported to Pakistan in January 2003 after pleading guilty to fraud, according to AP.
Monday's ruling appeared to be narrow, limited to the facts of this particular case, although it could be cited as precedent in other lawsuits, Reuters reports.
Dozens of Muslim men were arrested in the aftermath of the 11 September attacks, and the justice department's inspector general found that many of those held at the Brooklyn detention centre suffered abuse.
In 2006, the US government agreed to pay $300,000 (£195,000) to settle with Mr Iqbal's co-plaintiff, Egyptian Ehab Elmaghraby.
