Yes, you need an amp.
Your amp is a critical component of your audio chain. Amp topology and components will account for a large share of your audio quality. A Class-A amp on clean power (using a power conditioner, noise-filtering cable or you happen to have a clean circuit in a new build home etc) and outputing through quality FETs driven by a low-noise transformer will sound vastly more dimensional, dynamic, and resolving than your sound card's trashy op-amps running off your PC's noisy PCI power rail. A pair of 300B tube monoblocks using exotic paper caps, custom-wound transformers and point-to-point silver wiring is going to sound even better.
Weight and size tend to go up in proportion with amp quality. Class-A is best, but those need fat transformers and lots of heatsinks to dissipate the waste heat that is inherent to them. Class-AB are a common trade-off but they still need good transformers, quality capacitor banks and cooling. Class-D is junk, unless you are willing to drop big dollars on a ICE, NC400 or similar. Good amps have loads of big audio-grade capacitors in order to drive extended bass notes with quality and control, and also to smooth out power delivery to all these FETs. Volume is the LEAST important concern of an amp; what's hard is making a clean, low-distortion, highly dynamic and resolving output.