In the middle of the fourteenth century, Heinrich van Mierlaer, provost of Utrecht's cathedral, had a fortified castle built to replace an original court that had been burnt down in 1322 by Count William III of Holland. In the early seventeenth century, after the death of chaplain Hennin de Bossu, the House stood empty for some time and fell into disrepair. In 1635, the cathedral chapter sells Huis Doorn to Canon Jonkheer Reynier van Golsteyn, who has it restored for an amount of 5,517 guilders, 6 stuivers and 8 pennies.

A drawing by Jan de Beijer from 1750 suggests that provost Frederik Willem van Diest, who becomes owner in 1701, had the castle renovated again. Among other things, the wall of the courtyard has been lowered on the east side and the south-western tower of the towers has been raised and the north-western one has been demolished.

In 1762, Herman Frederik Richard Lijnslager inherits Huis Doorn and has it adapted to the fashion of the time. The medieval appearance, however, remains largely intact. In 1792, Wendela Eleonora ten Hove becomes the new owner and she has the house radically renovated again. The House has had its current appearance since 1800; a neoclassical country house with largely plastered facades, where only the tower on the southwest corner and the moat are reminiscent of the original castle.

There is much less clarity about the interior. Most traces have disappeared during the renovations of 1762 and in particular 1796. Pieces of gilded leather have been found under the floor of the gobelin room, a decoration that is also mentioned in the deed of sale of 1762.