Javier Hernandez stretched United's 2-0 first-leg advantage in the sixth minute before Aras Ozbiliz made it 1-1 on the night 31 minutes later. Toby Alderweireld scored with three minutes left but Ajax could do no more than scare United, which is in the second-tier tournament after it was eliminated from the Champions League group stage.
The English powerhouse progressed 3-2 on aggregate.
United will face Atletic Bilbao in the round of 16 after the Spanish side beat Lokomotiv Moscow 1-0 to advance on away goals.
Valencia and Atletico Madrid also progressed, as did PSV Eindhoven and Udinese.
United started strongly at Old Trafford and Hernandez scored following good approach work from Park Ji-sung and Dimitar Berbatov. Ozbiliz hit a 20-meter (yard) shot past goalkeeper David de Gea for 1-1 and, after De Gea made a point-blank stop from Christian Eriksen's header, Alderweireld bundled the ball into the net from close range.
One more goal would have take the Dutch side through to the round of 16 on away goals but United held out.
Manager Alex Ferguson blamed United's problems on his selection of a defense composed of Phil Jones, Chris Smalling and the Da Silva twins. None are vastly experienced.
"Tonight I used too many young players in the back four positions," Ferguson said. "It was a big risk we took."
Trailing 2-1 from the first leg, Bilbao was reduced to 10 men when Fernando Amorebieta was sent off in the 60th for a second booking. But two minutes later, Fernando Llorente flicked a corner kick toward the far post where Iker Munian hit an unstoppable volley into the roof of goal.
Valencia beat Stoke 1-0 for the second time in eight days to advance 2-0 on aggregate. Pablo Hernandez lost his marker and cut along the goal line before slipping a pass to Jonas in the box, where the Brazilian forward tapped into an unguarded net for a 24th-minute winner.
Diego Godin scored in the 48th to wrap up a comfortable 4-1 aggregate win for Atletico Madrid over Lazio, heading in unmarked from a corner kick.
Atletico coach Diego Simeone remains unbeaten in nine games since taking charge of the Spanish club at the start of the year, while Lazio has lost three straight to further pressure manager Edy Reja.
Udinese continued Italy's interest in style with a 3-0 win at PAOK Thessaloniki that ended the Greek side's 11-match unbeaten run in Europe and clinched progress by the same score.
Danilo headed in a corner from Giovanni Pasquale in the sixth minute and Floro Flores fired a free kick past goalkeeper Dario Kresic nine minutes later. Udinese captain Maurizio Domizzi made it 3-0 with a penalty in the 51st after Stelios Malezas had pulled Flores to the ground.
"My players did a terrific job," Udinese coach Francesco Guidolin said. "They showed a lot of character and exceeded my expectations. I couldn't have played in this stadium, under this pressure."
Hannover won 1-0 at FC Brugge to advance 3-1, PSV eased past Trabzonspor 4-1 and 6-2 on aggregate, FC Twente advanced 2-0 after a 1-0 win over Steaua Bucharest, and AZ Alkmaar went through 2-0 after a 1-0 win at Anderlecht.
Sporting Lisbon beat Legia Warsaw 1-0 to advance 3-2 on aggregate, while fellow Portuguese club Braga was eliminated 2-1 with a 1-0 defeat at Besiktas.