In the mid-14th century, Heinrich van Mierlaer, provost of the Dom church in Utrecht, had a fortified castle built to replace an original homestead, which had been burnt down in 1322 by count Willem III of Holland. In the early seventeenth century, after the death of Dom provost Hennin de Bossu, the house stood empty for some time and fell into disrepair. In 1635, the Dom chapter sold Huis Doorn to canon squire Reynier van Golsteyn, who had it restored at the amount of 5,517 guilders. 

A drawing by Jan de Beijer from 1750 suggests that provost Frederik Willem van Diest, who became the propietor in 1701, had the castle rebuilt once more. On the east side, among other things, the wall of the courtyard was lowered, and of the towers, the south-west one was raised and the one on the north-west demolished. 

In 1762, Herman Frederik Richard Lijnslager inherited Huis Doorn and had it altered in accordance with the fashion of the time. However, the medieval appearance remains largely intact. In 1792, Wendela Eleonora ten Hove became the new owner and she again had the house extensively renovated. Since 1800, the house has had its current appearance; a neo-classical country house with largely plastered facades, with only the tower on the south-west corner and the moat reminiscent of the original castle. 

There is much less clarity about the interior. Most traces have disappeared during the renovations of 1762 and especially 1796. Pieces of gold leather were found under the floor of the gobelin room, a decoration also mentioned in the 1762 deed of sale.