Pilica was owned in the 14th century by the family of the Toporczyks, owners of the castle in Smolen. Probably they had first built the castle in Pilica. This family began shortly after the name of the village, called Pileckimi. In 1569 the local property was bought by Padniewski's coat of arms. Soon, near the turn of the 16th and 17th century they rebuilt the castle, or they raised it from the beginning. It was a residence in the style of the Italian, late Renaissance, to which the Padnewski family moved from Smolenia.

​

Soon another owner, Jerzy Zbaraski, continued to expand. The building often changed owners. So soon the gentlemen became the gentlemen here, and then the famous Stanislaw Warszycki, who made another reconstruction. The palace, with forty chambers, was surrounded by fortifications with six bastions. It was - as for these times - a modern fortress. In 1655, however, the Swedes won the castle, but Warszycko managed to recover it soon. In 1705, during the Northern War, the Swedish army again took over. In 1731 it was bought by Maria of Wesslów Sobieska, rebuilding in French style - in baroque style. The 19th century brought another renaissance, neo-renaissance style and decoration of the fortification in a fashionable then romantic style.

​

In the Polish People's Republic, the castle, taken over by the last owners, Arkuszewski, became the seat of a correctional institution for young people. At the end of the Polish People's Republic, during the slamming of the progressive renovation, remains of a medieval building, probably the original castle of the Topors. In the time of our coming, the palace, which the heirs admonished, became the subject of processes.

​

The palace, which - hopefully - will regain its former glory, along with the surrounding fortifications, is located within the park, which is an outstanding example of gardening. It lays in itself a 6-meter lime in the perimeter of the labyrinth, called Elizabeth's linden (lady at the nearby castle in Smolen, wife of Władysław Jagiełło). Also fortifications here are one of the more interesting objects of this type. Next to it is a mound dedicated to the January insurgents. The area of ​​the castle is also a place of martyrology - during the occupation, a German platoon, a motorized gendarmerie was executed under the walls of the castle of crime, shooting about 80 Poles and about 70 Jews.