Worshippers shouted, "Break the chains, let Hagia Sophia Mosque open," and "God is great" before kneeling in prayer as tourists looked on.
Turkey's secular laws prevent Muslims and Christians from formal worship within the 6th-century monument, the world's greatest cathedral for almost a millennium before invading Ottomans converted it into a mosque in the 15th century.
"Keeping Hagia Sophia Mosque closed is an insult to our mostly Muslim population of 75 million. It symbolises our ill-treatment by the West," Salih Turhan, head of the Anatolian Youth Association, which organised the event, told the crowd, whose male and female worshippers prayed separately according to Islamic custom.
The government has rejected requests from both Christians and Muslims to hold formal prayers at the site, historically and spiritually significant to adherents of both religions.
The rally's size and location signals more tolerance for religious expression under Prime Minister Recep Tayyip Erdoğan, whose party traces its roots to a banned Islamist movement.
His government has also allowed Christian worship at sites that were off-limits for decades, as it seeks to bring human rights in line with the European Union, which it aims to join.
Turhan told Reuters his group staged the prayers ahead of celebrations next week marking the 559th anniversary of the Ottoman Sultan Mehmet's conquest of Byzantine Constantinople.
"As the grandchildren of Mehmet the Conqueror, seeking the re-opening Hagia Sophia as a mosque is our legitimate right," Turhan said in an interview.
Worshippers refrained from entering the museum, one of Turkey's most-visited tourist destinations and whose famous dome is considered a triumph of Byzantine architecture.
Some devout Turks believe that barring worship at Hagia Sophia is an affront against Sultan Mehmet, who designated it as a mosque and who, like other Ottoman leaders, served as caliph to the Islamic world.
Under Erdoğan, many Turks have come to embrace their imperial Ottoman past and question the more austere, Western-oriented reforms that followed the last sultan's overthrow in 1923.
The shift coincides with a stalled EU bid and declining expectations Turkey will ever join the mostly Christian bloc.
The government's active diplomatic engagement in the Middle East with lands that once belonged to the Ottoman empire has also prompted Turks to reexamine the NATO member's Western tilt.
Meanwhile, some Orthodox argue Hagia Sophia should be returned to its original state as a Christian basilica.
In 2010, 200 or so Greek American Orthodox aborted plans to pray at Hagia Sophia after the Turkish government threatened to block their entry into the country on security grounds.
The Ecumenical Patriarchate, spiritual leader of the world's 250 million Orthodox, does not support efforts to revert its former dominion into a church.
"We want it to remain a museum in line with the Republic of Turkey's principles," said Father Dositheos Anagnostopulos, the patriarch's spokesman.
"If it were to become a mosque, Christians wouldn't be able to pray there, and if it became a church it would be chaos."
Only a few thousand Greek Orthodox faithful are left in Turkey, but the patriarch's seat remains in Istanbul, a vestige of the Byzantine Empire.