Like you said, the house was built by the church, In 1635 it was sold to:
- Reynier van Golsteyn, inherited in ? by his son;
- Jhr. Philip van Golsteyn, sold in 1649
- Count Caius Barthram van Broeckdorff, given to his son in 1659:
- Count Caius Laurentius van Broeckdorff, sold in 1701:
- Frederik Willem van Diest,after his death the house was inherited in ? by his daughter:
- Anna Juliana van Diest & her husband Arent Carel baron von Hamersteyn, after their death the house was inherited in 1762 by:
-
Herman Frederik Richard Lijnslager,
sold in 1792 to:
- Wendela Eleonora ten Hove d. 1814
- Jhr. Mr. Andries Munter, Lord of Sleeburg, Doorn and Den Bosch (son of Wendela) d. 1861;
- Johanna Maria Warin (wife of Jhr. Andries) d. 1874;
- Count Samuel Johan van Limburg Stirum (son-in-law of Andries and Johanna); he auctioned the house in 1874 to:
- Mrs Cornelia Henrietta Labouchere, born Jkvr. van Lennep; d. 1902, house sold to:
- Baroness Wilhelmina van Heemstra, born Jkvr. de Beaufort & great-grand mother of Audrey Hepburn. She sold the house in 1919/20 to Emperor Wilhelm II.
None of the families that lived in the house after or before Lijnbergen are called 'von Doornum'. But all owners did carry the title 'Heer van Doorn' or 'Vrouwe van Doorn' (Lord - or Lady of Doorn). Von/van Doornum is a regular name, not belonging to the Dutch nobility & certainly not a count (we only have very few of them) or patriciate AFAIK.
------
In the story as your father heard it, count Albert van Doorum was supposed to have married a daughter of an owner that came áfter Mr. Lijnslager. Assuming that the genealogical data are correct, the owners had the following daughters, none of them married a van/von Doornum:
Wendela ten Hove: 2 daughters ->
(1) Margaretha Munter, m. Jhr. Mr. Jacob Boreel van Hogelanden
(2) Cornelia Munter, m. Baron Frederik Willem van Reede
Jhr Andries Munter: 1 daughter ->
(1) Susanna, m. Count Samuel of Limburg-Stirum ; no children
Cornelia Henrietta Labouchere, born Jkvr. van Lennep: 2 daughters ->
(1) Anna Catharina Labouchere, m. Jhr. Theodore Louis Lambert Prins van Westdorpe
(2) Henriette Labouchere, m. Jhr. Hendrik Daniel Wijnand Hooft
Cornelia Henrietta van Lennep b. 14 Sep 1821 Amsterdam, NH, NL d. 10 Dec 1902 Huis te Doorn: Geneagraphie - Families all over the world
Bss Wilhelmina van Heemstra: 2 sons, no daughters.
The village is not named after a count von/van Doornum. The original name of Doorn was 'Thorhem' , which means home of Thor (Germanic god of thunder). The Vikings named the place such, as they worshipped Thor there. Over the years the name slowly transformed from Thorhem in Doorn.
The information about the owners you can find here in Dutch:
Huis doorn
------
In short: it makes a nice story but there seems to be little truth in it. I am not sure, but another possibility is that the great grandmother who lived there was a member of the staff.