1122-1313
In 1122 a certain "Thiemo de Chalwinberch" is mentioned in a document. This ennobled family can be traced back to Callenberg until 1231. The last ennobled freed-man residing in Callenberg, Ulrich III., sold the castle to the bishopric of Wurzburg in 1231. However already in the following year the castle belonged to the Earl Poppo VII of Henneberg. In 1313 the knight Hermann Hunt (Hund) von Sternberg acquired "Fortress Callenberg" from Earl Berthold VII. of Henneberg and maintained it as feudal tenure.

1592-1826
Through devolution the castle became the property of Duke Johann Casimir of Saxe-Coburg in 1592, after the death of the last von Sternberg. Until 1825 the ducal treasury and the Castle of Callenberg were property of the Dukes of Saxe- Meiningen. It is only in 1826 that the Coburg Dukes are fortunate to be masters of the Castle of Callenberg again.

1831-1945
In 1831 the reconstruction of the upper castle is concluded, after Carl Alexander of Heideloff's plans. In 1856/57, under the reign of Duke Ernst II. the lower part of the castle is altered in Neogothic style.
Until 1945 the castle was the summer residence of the Coburg Dukes.

Since 1998
After extensive restoration the former summer home of the Coburg Dukes was restored and since 1998 lodges the collection of ducal art treasures, with precious pieces of furniture, paintings, porcelain, traditional handicrafts and a selection of weapons from four centuries.