The hawkish Abe, who took office promising to boost Japan's global security profile, had seen his clout dwindle after a drubbing in upper house elections in July, but the announcement came as a bolt out of the blue.
"I determined today that I should resign," a weary-looking Abe told a news conference.
Senior officials said health was a factor in the decision but Abe said he was going because a new prime minister would be better able to resolve a deadlock over extending a controversial mission to support US military efforts in Afghanistan.
Abe, at 52 Japan's youngest prime minister since the end of World War Two, reshuffled his cabinet only last month to rekindle public approval, but a poll this week showed support was stuck below 30 percent.
"There are many things I reflect on," the soft-spoken grandson of another prime minister said. "It is my responsibility that my old and new cabinet could not secure the public's trust."
Japanese stocks fell and the yen dipped briefly on concerns about political uncertainty. Chief Cabinet Minister Kaoru Yosano told reporters that Abe's health was one reason for the departure, but did not specify what the health issue was.
Abe aides were not available to comment on speculation that an article was about to be published in a tabloid magazine on suspicions that Abe had evaded taxes, and that this had dictated the timing of his departure.
Abe will stay on in a caretaker role until a successor is chosen from his Liberal Democratic Party (LDP) in a party election that officials proposed be held on Sept. 19.
LDP Secretary-General Taro Aso, a close Abe ally who shares most of his hawkish views on security policy, is seen as frontrunner to become the new prime minister.
Other names floated include former finance minister Sadakazu Tanigaki and former chief cabinet secretary Yasuo Fukuda.
The LDP and its junior partner have a huge majority in parliament's lower house, which picks the prime minister.
Abe had indicated that he would step down if he failed to extend a Japanese naval mission supporting US-led operations in Afghanistan, but the timing of his move was unexpected.
"It is the worst possible timing," LDP lawmaker Gen Nakatani told Fuji Television on Tuesday.
Five facts about Taro Aso, possible successor to Abe
Taro Aso, Secretary-General of Japan's ruling coalition and a close ally of Prime Minister Shinzo Abe, is widely seen as the frontrunner to replace Abe after he announced on Wednesday he would resign. Here are five facts on Taro Aso:
* Aso, 66, served as foreign minister in Abe's first cabinet, before taking the ruling Liberal Democratic Party (LDP)'s number-two post as Secretary-General in an Aug. 27 reshuffle. He has also served as minister for economic planning and for posts and telecommunications.
* Aso came in a distant second to Abe last year, in his second bid for leadership of the LDP. Coming from a venerable political family, Aso has been open about his desire for the top job. His grandfather, then-prime minister Shigeru Yoshida, negotiated the peace treaty ending World War II. Aso's father-in-law was also a prime minister and his sister is married to a cousin of Emperor Akihito.
* Aso shares Abe's goal of a bigger global security role for Japan. Last October, after becoming foreign minister, Aso said there was nothing wrong with discussing whether Japan, the only country to suffer an atomic bombing, should possess nuclear weapons. But he has also said that he would stay away from Tokyo's Yasukuni Shrine, seen by many in Asia as a symbol of Japan's past militarism, if elected prime minister.
* A rarity among Japan's mostly staid politicians, Aso appeals to fellow fans of manga comics, and can work a crowd with amusing patter. He recently authored two books, one of which -- "Tremendous Japan," about Japan's tremendous potential -- has become a best-seller.
* His brash manner has provoked controversy. Aso was recently forced to apologize over a flippant remark about Alzheimer's disease, and he stirred anger in the two Koreas in 2003 for remarks seen as praising Japan's 1919-1945 colonization of the peninsula. Earlier this year, he criticized US policy in Iraq and said Japanese with their "yellow faces" would be more successful at Middle East diplomacy than "blond, blue-eyed Westerners" since Japan had never exploited the region.