GSMArena labs: Introducing our new smartphone speaker test

If you've been following our reviews, you may have noticed that we haven't posted any audio tests recently.

The reason for that is we've been working hard behind the scenes to deliver a new smartphone speaker audio quality test that would be more useful for those looking for their next smartphone. We'll take this opportunity to detail the changes we made and our motivation behind them.

Why we are discontinuing the old loudspeaker and wired audio output tests

The two tests were designed in 2007, which was ages ago in smartphone terms. Back then, the goal of the loudspeaker test was to gauge how loud your phone will ring when you get a call or how well the phone is geared towards use in speakerphone mode. That's why we simply measured the peak loudness of the phone’s speakers while playing three different audio samples (a classic phone ringing, some pink noise, and then human speech). It was great if you wanted to know how loud this particular phone was, but it didn't give you any indication of how well it sounded.

However, with the advance of smartphones, the usage leans more and more towards multimedia consumption - people gaming, listening to music, and watching movies on their phones' speakers, so we decided that measuring the sound quality was equally important too. Also, our new test shows average loudness instead of peak loudness, making them more indicative of the experience you will get with multimedia rather than when your phone's merely ringing.

The wired headphones audio quality test, on the other hand, was rendered obsolete by a bunch of factors. Key among those is that phones have become so good that the difference between the best and the worst is measurable only with lab equipment. We updated the test to add headphones' resistance part to the equation, but still, the differences are so minimal so as not to matter. And this has been the case for several years now.

Your choice of headphones has a much bigger impact on the audio experience than the output of the phone.

Then there's the fact that fewer and fewer phones now come with a 3.5mm jack, so the importance of the external USB-C to 3.5mm adapters that you would use adds another unknown variable. The active dongles have their own DACs so the audio output barely depends on the phone itself.

On the other hand, the passive ones can sound differently when you use them on different phones, so our test of a particular phone + adapter combo probably has little relevance to your experience in real life. Makers have mostly stopped bundling adapters with their phones, so you are bound to rely on third-party products and the combinations there are endless.

Finally, Bluetooth headphones are on the rise more than ever and many people are leaving wires behind altogether.

In the end, we felt it would be much more useful to put our limited resources into a test that's infinitely more relevant to your user experience. With multimedia consumption on mobile higher than ever, we chose to focus on a test that's more informative and does not rely as much on the type of accessories you use with your phone.

How we test the loudspeaker now

Our new test now includes playing back a few select tracks through the phone's speaker(s) in an environment with controlled acoustic properties and recording that playback with an audio recorder.

The first benefit of this is that you can now listen to the recordings by all tested phones. This way you don't have to rely solely on our subjective commentary about what they sound like (we recommend using headphones for playback).

But you don't have to trust just your hearing on this. Our audio sample set also includes a specialized audio track that sweeps the whole frequency range. We analyze the recording of this track by specialized software that tells us how well the phone's speaker(s) reproduces the different frequencies.

A look on the frequency response chart will easily tell you whether one phone reproduced the bass, treble, and mid frequencies better than another phone. (Tip: The closer to the flat line of 0db, the better.)

And finally, based on the recordings we've made, we can analyze the average loudness of the speaker output, which gets us a loudness score. And since we now measure the weighted average loudness of a phone and not the peak one, this loudness score is not affected by peak bursts of high tones which are irrelevant for multimedia consumption.

LUFS stands for "Loudness Units relative to Full Scale" - it's an industry-standard unit used in audio normalization for broadcast TV as well as streaming music and video.

You should keep in mind two things when reading those. The LUFS unit represents a relative measurement - the number is derived when compared to a particular baseline. So unlike decibels, you can't compare them to the sound of a TV or the roar of an airplane. The number here will only serve the purpose of comparing one phone to the other in our particular test.

Note that higher is better and the numbers are negative, meaning that the closer the LUFS reading is to zero, the louder the sound. To make it easier for you, we’ve also assigned ratings to each phone, ranging from Poor to Excellent. These are not directly comparable to the ratings from our old loudness test since, as we already explained, they measure average rather than peak loudness.

Finally, you can use our new tool to compare any number of phones we've subjected to the new test. We've already retested a bunch of phones we had available, but going forward the new test will become a part of our regular review routine so there will be plenty to choose from.

How to use the new widget

You will find our new Speakerphone test widget, just like the one above, embedded in its corresponding section in our reviews. It offers key info of how loud the phone is (measured in LUFS plus our rating) and it shows the frequency response chart, allowing you to instantly spot phones that have weak bass or whose output in the high frequencies is too overbearing.

You can compare the frequency response graphs to get a feel for the audio. The chart goes from +30dB to -30dB, but keep in mind that -30dB means 1,000 times quieter (and +30dB is 1,000 times louder).

An ideal speaker would produce a flat line at 0, so any deviations above or below the zero line are imperfections of the output.

The widget becomes even more useful when you add one or more phones to compare (tap Add to comparison and use the search).

When comparing two phones, what you need to look at is how close the lines are to zero and also how flat they are - any peaks or valleys are audible distortions of the sound in that particular part of the frequency spectrum.

Instead of just reading the charts, you can hit the Play button and hear the difference in the sound of the compared phones yourself. We recommend that you use headphones for this unless you have good quality speakers hooked up to your PC (if you’re reading the results on your phone, definitely use headphones).

We’ve preselected the first phone in the example embedded earlier in this presentation. The Asus ROG Phone II in Outdoor mode is currently the loudest phone in our database (and we think it will keep that title for quite a while). However, add the iPhone 11 Pro to the comparison, and you’ll instantly see and hear the difference in bass and that the ROG scarifies bass frequency reproduction in the pursuit of loudness.

In fact, that was another issue with our old test – our ringtone sample leaned heavily on high-frequency sound. Some phones had speakers that produced a tinny sound that were better suited to produce these high-pitched sounds. This accounted for a higher decibel rating in our old peak loudness test but subjectively, the sound was harsh and sometimes unpleasant even. Our new test shows more nuanced results, separating the practical (hearing a notification) from the entertainment side of things.