Last season's double winners, who enjoy substantial financial backing from the energy drinks company, beat visitors Dudelange 4-3 in their second qualifying round, second leg tie but were eliminated on away goals after losing 1-0 last week.
Red Bull took over the club in 2005 with ambitious plans of making Salzburg Champions League regulars, only to see their team fail five times in a row in the qualifiers.
The only time they have taken part in the group stage was as the old Austria Salzburg in 1994/95.
Their latest failure is a huge setback for coach Roger Schmidt and sporting director Ralf Rangnick, the former Schalke 04 coach, who took over in the close season.
Salzburg's troubles in Tuesday's match began when they lost possession in midfield, allowing Thierry Steinmetz to break clear and score in the 26th minute.
Jakob Jantscher and Martin Hinteregger put Salzburg 2-1 ahead at halftime, only for Aurelien Joachim to level with his sixth goal in four matches in the qualifying tournament.
Steinmetz struck again to make it 3-2 but Dudelange then had to endure a nerve-wracking finale after forward Sofian Benzouien was sent off in the 74th minute.
Salzburg's Cristiano converted an 81st minute penalty and fellow substitute Gonzalo Zarate scored a minute later to put the hosts ahead on the night but the Austrians could not find the fifth goal they needed in the final eight minutes.
There was another surprise as Shakhtar Donetsk, who have taken part in the group stage eight times, were knocked out by Czech champions Slovan Liberec, who scored the decisive goal in the last minute of extra time.
Trailing 1-0 from the first leg, Shakhtar levelled the aggregate score with a Zhambyl Kukeyev penalty five minutes before halftime. Both sides had chances before substitute Jan Blazek scored in the 120th minute to make it 1-1 and send Liberec through 2-1 on aggregate.
Norwegian side Molde, coached by Ole Gunnar Solskjaer, went through 4-1 on aggregate after a 1-1 draw at Latvian champions Ventspils and in the third qualifying round will face Swiss champions FC Basel, who last season eliminated his former club Manchester United
Two goals from Cameroon striker Jacques Zoua and one from Chilean midfielder Marcelo Diaz, Switzerland's most expensive close season signing, gave Basel a 3-0 win over Estonia's Flora Tallinn as they completed a 5-0 aggregate victory.
Debrecen, one of only two Hungarian sides to have played in the group stage, beat Albanian champions Skenderbeu 3-0 to qualify 3-1 on aggregate and Lithuanian champions Ekranas ended the hopes of Ireland's Shamrock Rovers with a 2-1 win.
Two goals from Nemanja Tomic helped Partizan Belgrade to a 3-1 win over Malta's FC Valletta as they went through 7-2.