We've always believed that you should get the best product that your budget allows. In some cases, it's even better to save for a bit and get the product you really want, rather than settle for a second choice. Personally, I've always recommended that you should invest in products that directly impact your senses, like vision (display), touch (mouse and keyboard), and ears (audio).
Headphones are all the rage now. Enthusiasts have been using headphones for decades, but it wasn't until recently that headphones have become a fashion statement—thanks, Beats! Then there are gaming headsets; everyone's making them now. Companies that you would have never thought would make audio products now make them. So, how to separate the wheat from the chaff?
Just like computer equipment, there are more choices out there than there are good, quality products. Headphones and gaming headsets too, can be broken down just like graphics cards or CPUs. There are entry-level products, midrange offerings, and truly high-end flagships.
On the hi-fi side of things, prices can range from $80 to well over $4,000. In the gaming space, prices are regulated to more modest levels, ranging between $30 to $300 in most cases.
What are the differences between hi-fi headphones and gaming headsets?
In a nutshell? An attached microphone. But if you look around at websites for hi-fi headphone companies and gaming companies, you'll start to see a pattern:
Here's the thing: audio is really a subjective topic. A pair of headphones that may sound good to one person might sound awful to another. Some people prefer headphones that emphasize bass frequencies. The bass may not even be quality bass and completely drowns out the rest of the frequency spectrum, but as long as it slams hard, it's enough. Some headphones will emphasize the mids and highs, the more transient details. Some people like this, and some may feel the sound is too harsh.
If you read headphone reviews—and any hi-fi related articles in general—you'll no doubt come across many colorful terms such as warmth, air, attack, decay, slam, open, bloom, transparency, sound stage, and so on. While these adjectives are fun to use, they don't actually mean much.
The question then, based on the priorities of different manufacturers, is: Do gaming headset makers not care about audio quality? We're going to find out. And we're going to use the same testing system and metrics to quantify the seemingly unquantifiable, for all types of headphones.
To properly test headphones in a quantifiable way, we're relying on Brüel & Kjær's cutting-edge head and torso simulator (HATS). Who is Brüel & Kjær? The company was founded in 1942 with the goal of developing sound- and vibration-measuring instruments.
Today, Brüel & Kjær operates all over the world and supplies measurement and testing equipment to audio, telecom, defense, automotive, and aerospace industries. Some of its customers include Boeing, NASA, SpaceX, Lockheed Martin, the US Department of Defense, Ferrari, Nissan, Bosch, and many more. The Brüel & Kjær's LDS V994 shaker was used to test NASA's Mars Curiosity rover; you don't accrue this kind of clientele by being second rate.
4128-C Head and Torso Simulator
We decided to rely on Brüel & Kjær's 4128-C HATS to test all our headphones. The Brüel & Kjær HATS is the benchmark standard in testing any audio products that interface with either your ears or your mouth. The 4128-C contains built-in mouth and ear simulators that are calibrated to IEC 60318-4/ITU-T Rec. P.57 Type 3.3 specifications (PDF), mimicking established international standards on adult human heads and torsos.
HATS Type 4128 C with simulated pinnae.
The HATS comes with two anatomically correct pinnae (ears) that attach to the side of the head. The symmetrical pinnae are soft and comply with IEC and ITU-T specifications. Each ear is stamped on the inside with an expiration date as the materials decay over time.
Yes, average adult human ears look like this; $700 per ear.
Inside the head module are precision sensory microphones attached to a simulated ear canal. They are incredibly sensitive and can even pick up minute sounds such as micro-phonics, which are sounds produced when there are vibrations in the audio cable (analog stage).
The HATS is attached to and powered by a Brüel & Kjær Nexus Conditioning Amplifier, which powers the HATS and has calibrated pre-amplifiers for the microphones. The amplifier also has a built in battery for situations where there's no available power source.
Not your average power supply.
Microsoft has a video on YouTube that shows off its anechoic chamber inside Building 87 of its Redmond campus, where a Brüel & Kjær HATS is being used for testing. It just so happens that Microsoft's anechoic chamber holds the record for being the world's quietest place, at -20.6 dBA. The human threshold sits at 0 dBA.
RME Fireface UCX
The entire HATS system is then connected to an RME Fireface UCX analog to digital interface. Brüel & Kjær prefers using the Fireface UCX for audio testing due to its locked and low latency, which is important as the latency is predictable and can be compensated for. The unit is potent, and supports sampling rates up to 192kHz at 24-bit resolution across 36 channels.
Fireface UCX front.Fireface UCX rear.
The Fireface UCX boasts excellent specifications that are suitable for duties such as measurement. Total harmonic distortion (THD) comes in at -105 dB at less than 0.0005 percent, which is quite a feat.
Ideally, you'd want to set up this entire system in an anechoic chamber to avoid environmental sound interference. However, with the software that we're using, the need for such a chamber is largely negated. The whole setup looks like this:
The full Brüel & Kjær HATS system we're using for testing.
SoundCheck by Listen Inc.
We're using an application called Soundcheck, an industrial-level analysis program used for audio related tests. If you thought Photoshop or Office were expensive, SoundCheck is on another level entirely. You'll need to cough up about $10,000, so it's definitely not designed for the average home user. However, Vince Rey at Brüel & Kjær graciously provided us with license keys necessary to install SoundCheck, and I was on my way. Well, not quite.
SoundCheck is incredibly comprehensive and powerful; you can't just go in there and run some basic tests. Listen Inc. does provide pre-written sequences that you can use, but SoundCheck allows you to design your own test sequences.
A test is broken down into two basic parts, the sequence and its steps. Steps are logical and can be scripted with some basic programming syntax to allow lows and conditions. Put them all together and you have your basic sequence. When an audio test is run, SoundCheck runs through each step in the sequence and places the results in memory. What you do with the data stored in memory is entirely up to you. SoundCheck can save results into Excel spreadsheets, or as images where appropriate, or maintain results in memory for other things like manipulation or displaying them graphically.
Listen Inc.'s incredibly powerful SoundCheck in use.
Learning to create your own sequences in SoundCheck is no easy task. Thankfully, Listen Inc. offered to come by and give me some training over two days. Honestly, I could have used more. But video tutorials were available, too, so I spent about a week watching and reading, after which I was able to create custom sequences and generate results that were useful.
Building a custom sequence in SoundCheck.
In the screenshot above is the Sequences Editor window. On the left pane are all the steps that you can add into a sequence. Primarily, a sequence is composed of at least the four following parts:
Stimulus: these steps define what kind of tones and sweeps will be played through the headphone. For the purpose of our tests, we're relying on Stweeps, which are stepped-sweeps that start at the highest frequency and progress down to the lowest frequency. Log sweeps are also used to test for things such as impulse response, and start from the lowest frequencies and move up.
Acquisition: these parts define how the stimulus is recorded.
Analysis: these steps calculate such things as frequency response, THD, impedance, etc.
Display: here you customize how you want SoundCheck to show the results gathered.
I decided to also use some steps in Post-Processing to calculate other useful information, such as left/right tracking. PP steps allow you to apply arithmetic calculations to the analysis data to generate new data.
How to read the charts
Reading output charts is relatively easy, and gives a good indication of how a pair of headphones or a gaming headset will sound. The HATS system can measure microphones too, and the same data applies.
Frequency response
Frequency response shows the response of the headphones across the entire audible frequency spectrum. As you can see, I've set the test sweep to occur between 10Hz and 20kHz, but tests are usually done between 20Hz and 20kHz.
From the example readout, we're able to deduce that the headphone in question has excellent bass extension with some trail off starting at 30Hz. The bass frequencies are stronger than the mid ranges, and climb at about 2kHz. Between 2kHz and 6kHz is essentially where the manufacturer intentionally tunes the headphones, but after about 8kHz the spikes and dips you see are caused by the actual structure of the headphone itself. At 10kHz, the headphone's response begins a curve downward.
Ideally, you'll want a pair of headphones that has a completely flat frequency response across the entire spectrum, but this is impossible. Physics, driver dynamics, and resonance will impact frequency response. The intention of the manufacturer also plays a big part. If you measure Beats, for example, you'll expect a bump in the bass frequencies.
A good pair of headphones will have bass linearity between 20Hz to 100Hz with +/- 5dB of variance. Treble frequencies between 1kHz and 10kHz should be, on average, flat, depending on the design. Upper treble is much more difficult to control, but good headphones will not exhibit crazy spikes and valleys.
When there are odd bumps in the frequency range between 100Hz and 3kHz, you've got problems. The above headphones show a smooth response in those ranges, indicating good engineering to mitigate resonance.
Total Harmonic Distortion
Total harmonic distortion (THD) measures audible distortion emitted by the drivers in the headphones. Good headphones should not exhibit THD beyond two percent at 98dB. You'd be surprised that even some flagship hi-fi headphones will have THD that reaches 10 percent, which is unacceptable.
Expect good headphones to maintain less than one percent THD in almost the entire frequency spectrum. Bass frequencies are more difficult to control. Headphones that emphasize a lot of bass will produce huge THD below 100Hz. In other words, the bass you hear may sound punchy, but it might also be bad quality bass. You want something that delivers clean and tight bass response, that doesn't cause other frequencies to be smothered.
You may have noticed that the THD test isn't THD+N (noise). This is because, well, the world is full of noise. There's no sense measuring that, too. The THD being measured is what the headphone produces, only.
Left/Right Tracking
No two drivers are exactly alike. Manufacturers do their best to make sure both the left and right drivers are matched as closely as possible, but sometimes this isn't the case. Expect left/right tracking to be severe if the headphone in question is of poor quality.
Impedance
The impedance plot allows us to see whether a voice coil is aligned to its magnet. If alignment is poor, the plot will have spikes throughout the frequency range. Magnets that are properly aligned will generate a smooth line like the one you see above. When the voice-coil and magnets are not aligned, the headphone will actually send current fluctuations back, and we can measure that with impedance changes.
This plot mainly only applies to dynamic voice-coil drivers and does not impact headphones that are planar-magnetic or electro-static because they don't use voice-coils that need alignment.
Objective and subjective tests
Aside from hard measurements, there are other important factors:
Comfort
A headphone's comfort goes a long way in its usage in everyday life. There's no point having a great pair of cans if you have to take them off in an hour due to fatigue.
Within this scope, we'll look at the headphone band as well as ear cups. Ear cups can be of several different materials. In hi-fi headphones, some sort of faux leather is frequently used, but manufacturers are now introducing lambskin and velour. Also keep in mind that ear cups directly impact the sound as well as the frequency response. Some ear pads will add or take away bass.
Right: lambskin; left: velour.
Ear cup sizes will also help with comfort. Circumaural headphones are those that go around your entire ear. These usually provide the best comfort. Superaural headphones are those that sit directly on your ears and generally don't provide very good long-wear comfort. All gaming headsets are circumaural and most high-end headphones are also circumaural. Those who are familiar with Grados will find that most of the company's headphones are superaural; worst yet, the foam is stiff.
Cable quality
The quality of the cable definitely matters. Poor cables can introduce microphonics, which introduce noise into your output. Good headphones will also have cables that are removable. Imagine spending hundreds or thousands of dollars on your dream headphones only to accidentally run over your non-removable cable with your office chair. Ouch.
Obviously, this doesn't apply to wireless headsets.
Build quality
We also examine overall build quality. Finish, fit, and materials are important. Does the finish come off easily? Do the headphones feel like they'll snap in two if you have an extra-large cranium? Of course, the more you spend on headphones, the better you expect the quality to be.
Removable boom mics are also a plus. If your mic happens to break, the headset essentially becomes useless for gaming. Thankfully, a good number of gaming headsets come with removable mics. Being able to change the pads will also give points in build quality. They wear and stink up over time due to sweat, so being able to change them out is a plus.
Sound off
We hope to establish a testing standard that can be used with all headphones and gaming headsets, and that going in-depth will allow you to see which products are standouts and which are just pure marketing. I'll be working closely with the folks at Brüel & Kjær and Listen Inc. to continue developing and fine tuning the tests.
I call this the headphone "Arena."
Aside from all the detailed measurements, how the headphones actually sound is up to personal preference. I'm tempted to leave out personal preferences entirely, but it does impact the rating some.
At the end of the day, there are just too many options on the market. The object is to weed out the good from the bad. Are the gaming headsets going to be able to compete with actual hi-fi headphones? Are there any substantial difference between one gaming headset to another? Does pricing actually make a real impact in quality? We'll find out!
