Two houses, interconnected in 1603, used to stand in the place where the Kolowrat Palace is now. At the beginning of the 17th century, the then owner of the house, Vilém st. Popel of Lobkovice, had the house enlarged. The next owner, Maximiliana of Götz, rebuilt the house into a two-storey structure.
Countess Marie Barbora Černínová of Chudenice, wife of the rich Heřman Jakub Černín, bought the palace in the 18th century. The new owners decide to rebuild the house with the help of architect Ignác Palliardi. They also bought the Small Fürstenberg Palace, connected it to the Kolowrat Palace and built a terraced garden. At the end of the 18th century, the palace was owned by the nephew of Countess Černínová, Count Jan Arnošt Schaffgotsch, who had both buildings separated again.
Several aristocratic families owned the palace during the 19th century. The palace was owned by Count Zdeněk Kolowrat Krakovský from 1886 and he rebuilt it, adapted its interior in the Baroque-revival style that can be seen to this day and put his picture gallery, library and collection of coins there. After his death, his son Hanuš Kolowrat Krakovský lived in the palace and added an additional horse stable to its north-west part.
After the end of the WWI, in December 1918, representatives of the new Czechoslovak Republic concluded a lease contract with the owner of the palace and used the 2nd floor as offices for the Ministry of Social Care. The state bought the whole palace in December 1920 and used it as the