Canada said it had not, however, severed diplomatic relations between the countries. Operations at its embassy in Tripoli have been suspended for several weeks, an official said.
"The activities carried out in Canada by the five Libyan diplomats are considered inappropriate and inconsistent with normal diplomatic functions," the Department of Foreign Affairs said in a short statement.
A foreign affairs spokeswoman declined to elaborate on the reasons for the explusions or identify which diplomats had been asked to leave, citing Canada's policy of not commenting on state-to-state communications.
Efforts to reach officials at the Libyan Embassy for comment were not successful.
Canada has fighter jets participating in NATO air operations over Libya and has allowed them to attack ground targets in that mission. A Canadian warship is also part of the NATO effort to enforce an arms embargo.