NEWBURY, England -- On the day my sister-in-law and I made our pilgrimage from London to Highclere Castle, where Downton Abbey is filmed, my husband took the kids to the Harry Potter studios. So we were all in our own fantasy worlds: the children pretending to ride Quidditch brooms, and Efrat and I hoping to feel the presence of Lord Grantham and Lady Mary, fictional though they may be.

Just as you’d expect, Highclere is set amid rolling hills, grazing sheep, and beautiful trees — enough English countryside to last a Downton fan a lifetime. With its golden stone, impressive tower, and Gothic turrets, Highclere looks exactly as it does on television — but being there allows you to feel its grandeur. It has hundreds of rooms (the current resident doesn’t even know the exact count, that’s how big it is). The walls are so lovely — some are covered in leather, others in beautiful green French silk — and the ceilings so ornate, the staircases so grand, the furniture so pretty, that you practically have to remind yourself to enjoy the pastoral views outside the enormous windows.

But there was one surprise. You go expecting a sophisticated Masterpiece theater-y crowd, and instead you find groupies. We had to wait to enter the mansion because a 68-year-old woman, arms thrust in the air, was posing for a photo outside the enormous front door. She was imagining herself as Lady Cora (Elizabeth McGovern) and wishing she had a large staff behind her.

“Today’s my birthday,” she said. “This is a present from my daughter.”

A moment later I encountered another woman in her 60s (the castle was crawling with them, husbands in tow) who wished the tour included costumes. “Wouldn’t it be fun to wear one of Lady Mary’s gowns?”

Inside Highclere it was the same story. In the saloon (one of the many rooms where Maggie Smith holds court as the Dowager Countess) a docent told me she can easily distinguish the history buffs, who’ve been visiting the castle for years, from the Downton fans, who started showing up only after the drama’s first season ran, in fall 2010 in the United Kingdom and in January 2011 in the United States.

The scholarly group is curious about Sir Charles Barry, who designed both Highclere and the Houses of Parliament; and the various Earls of Carnarvon, who have lived in the castle for over 300 years. They’re interested in all of the art, not just Van Dyck’s equestrian portrait of Charles I. That painting, as the Downton crowd knows, hangs in the state dining room, the setting for many delicious zingers and painful staff slip-ups.

Those visiting Highclere because it’s a historic house — and not the set of a nighttime soap — are interested to learn that an early Anglo-Saxon charter suggests people have lived on the spot for some 1,300 years. They take the time to see the onsite Egyptian exhibit (there because the fifth Earl of Carnarvon was a discoverer of Tutankhamun’s tomb). They’re not disappointed that the gift shop doesn’t sell replicas of Lady Mary’s necklaces, or Thomas bobbleheads. When they look at the family photos on display, they’re eager to see the actual residents (the eighth Earl and Countess of Carnarvon), and not Ladies Mary, Edith, and Sybil as little girls.