Originally in the possession of the Earls of Cornwall, Richard of Winchester and then his son Edmund of Almain, the castles of Berkhamsted and Hertford to the north of London (Berkhamsted not being far from Langley, notably, and also Ashridge priory) escheated back to the crown of Edward Longshanks. The late king then gave these castles as dower to his second wife (who was 20 and he was 60) when they married in 1299. Despite it being the property of the widowed queen Margaret of France, her step-son granted it to Piers Gaveston when the latter was made Earl of Cornwall, but it was returned to her upon his later exile.

It is currently the dower possession of the dowager queen Margaret of France, but it often still serves as a larger royal residence.