As others have said, the FiiO BTR3 just isn’t a strong enough amp for the HD 6XX. While the HD 598/599/PC37X are 50 Ω resistance and high sensitivity, headphones designed for portable use (and small portable amps like those found in phones and the BTR3), the HD 6XX is a “big boy” studio and home headphone at 300 Ωs and 102 dB Sensitivity. Actually, it’s slightly less sensitive, but that’s where you get your “loud enough,” and the resistance actually varies (and increases beyond 300 Ω) with different frequencies, and that is probably where you get your congested, veiled sound.
Price isn’t the best indicator of suitability (and I’m only comfortable with US$ and EU€), but consider that the HD 650 was originally launched in 2003 for $500 as Sennheiser’s production flagship... and you’re trying to power it off of a tiny $50 device. Consider inflation on the HD 650’s original price (the HD 6XX version really is an incredible deal) and the fact that the FiiO’s cost budget is split between the cost of the Amp, DAC, and Bluetooth, then its fairly easy to see you’ve invested way more in your headphone than your amp. You could easily spend $500 or more on just an Amp for the HD 6XX and continue to hear improvements in clarity and firm impact, however I’m not particularly trying to make you spend lots of extra money.
You would hear a substantial improvement from a $100 Monoprice Liquid Spark, a Drop Cavalli Tube Hybrid, or other dedicated desktop amp. If you have a home theater receiver with a 6.5mm headphone jack, that might show off more of what the HD 6XX is capable of. If you like portable Bluetooth, and FiiO, I can tell you from experience that the Q5 and Q5S with AM2 amp module (or better) had a really addictive sound quality with the HD 650/6XX. If you see an amp you can get shipped from Drop or buy locally, send us a link in a reply, and we might be able to drop some thoughts for you.
Now, I’m not there, and I can’t test your headphone, so I don’t actually know if it’s defective, but the symptoms you describe sound exactly like my experience with not driving the headphone with an adequate amp. Some of it is sheer power (I’d guess the BTR3 only gives 10-16 mW of power to the HD 6XX, varying depending on frequency), some of it is the clarity and low distortion of the amp. With a good amp, the HD 6XX should actually sound cleaner and clearer than the HD 598, while at the same time giving you a harder bass, and sound like an obvious upgrade.