On the Heunsberg in Valkenburg is the ruin of one of the few high fortifications in the Netherlands. It was built at the end of the 11th century for the Valkenburg lord as a keep, a fortified house. A wooden palisade probably first stood around it. Due to the successes of the lords of Valkenburg, a little residential center was gradually formed at the bottom of the mountain. Later that settlement grew into a true fortified city. The building materials (marl) were plentiful.

There were not only defense works at the castle. The entire town came to lie within a strong enclosure. Later this ring wall was expanded with two strongholds. One of those defenses was on the site of the current Den Halder castle. The city wall was also equipped with city gates and a number of towers. Just like the castle, the city wall consisted entirely of marlstone. The used marl stones were probably largely from the underground quarry in Valkenburg aan de Geul, known as the Velvet Cave.

Within the fortress there was a market square, the current Berkelstraat. There were also several striking buildings, such as a tithe barn, a guesthouse for the sick, the St. Barbara and Nicholas Church and an inn that still offers visitors accommodation under the name Hotel de l’Empereur. The castle was blown up in 1672. That meant the end of the reign of the lords of Valkenburg. A century later a major fire occurred in the city. After the disaster of 1773, most houses were rebuilt, of course again in marlstone.

With the rise of tourism around 1850, hotels, cafés, restaurants, villas and country residences of marl were built. Various architects, including Pierre Cuypers, have given many buildings in the city a special appearance. Valkenburg also slowly turns ‘marl’ yellow outside the fortress.