The actual Villa stands on the ruins of the ancient country residence of the emperor Domitian built so as to dominate from above the Tyrrhenian Sea; it was a huge complex that included the imperial palace, the servants' houses, a theater, the cryptoporticus and the hippodrome. The successors of Domitian didn't live in the Villa but indeed they demolished some parts of the building to reuse its materials in new constructions.
Around 1200 on these ruins was built the castle of the Genoese Gandolfi family, hence the current name of Castel Gandolfo and the first Pope who decided to regularly live here was Urban VIII in 1626; Carlo Maderno was commissioned by this Pope to readapt and expand the old medieval castle, in order to make it suitable to its new function of Pontifical Palace. In 1628 Taddeo Barberini bought some lands in the area and there formed the Villa Barberini, completed in 1635 (in its properties are still visible remains of the ancient Villa of Domitian).
Starting from 1870 there were long periods, in which all this property was left closed and abandoned by the Popes who used to live somewhere else, but over the years the building has also undergone several extensions and renovations. It was only with the signing of the Lateran Pacts (11 February 1929) that the Pontifical Palace of Castel Gandolfo with its attached villas, was officially recognized as property of the Holy See.
Then, there were made connections among the three villas that made up the complex (Giardino del Moro, Villa Cybo and Villa Barberini) by the overpass and the loggia; in 1934 there was also transferred the Astronomical Observatory of the Jesuit Fathers.