Before the construction of Fort Saint Elme was completed in 1552, only a watchtower built in the 9th century occupied its location. Named Torre de la Guardia, it allowed protecting locals from the naval invasions of Norman and Barbary peoples. In 1538, Emperor Charles V decided to build a defensive structure around the old tower to supply and defend Collioure, Port-Vendres, and ultimately Perpignan. The architect of Fort Saint Elme, Benedetto of Ravenna, built a fortress with six branches capable of withstanding all the artillery of the time.