The gray zone here is "full potential." A modern stereo receiver from a major brand (Yamaha, Denon, or similar) that targets home consumers will be able to drive them quite nicely. I occasionally drive mine from a Yamaha RS-201 and they sound very nice.
An argument can be made that a dedicated headphone amp will be better because it is dedicated to the task, and/or is not stepping down the speakers' watt level amp power to get down to milliwatt headphone levels.
Do I like my Valhalla 2 headphone amp with them better than the Yamaha? Yes. Is it the quality difference between, say, Jimmy Buffett singing the national anthem vs. a choir of angels? No.
However, assuming that you have a specific reason for wanting a receiver (for example, you want to run speakers and headphones from one device) , then the value added by those capabilities will far outweigh any audible difference to a new audiophile.
I'd say go with the receiver and enjoy its full capabilities. There's plenty of time to upgrade and play with headphone amps later, when they fit your budget/space/design/"I just want one" requirements. Your main challenge will be that you won't be able to look at a lot of reviews about how they sound with headphones, since receiver reviews usually focus on speaker performance.
Just be sure to test it right away, and return it if you hear anything bad that stands out (I had a vintage amp for a short time that had a clearly audible hiss from the headphone out).