MPEG Audio Decoder Compliance
Introduction
The purpose of this page is to share an objective analysis of various MPEG audio decoders, in particular the results of tests for decoder compliance with respect to the ISO/IEC 11172 international standard.
Decoder Compliance
Part 4 of ISO/IEC 11172 defines the compliance tests for MPEG-1 audio and video decoders. Section 2.6.3 defines the computational accuracy tests for audio decoders. It states:
To be called an ISO/IEC 11172-3 audio decoder, the decoder shall provide an output such that the rms level of the difference signal between the output of the decoder under test and the supplied reference output is less than 2−15⁄√12 for the supplied sine sweep (20Hz–10kHz) with an amplitude of −20dB relative to full scale. In addition, the difference signal shall have a maximum absolute value of at most 2−14 relative to full-scale.
Then it says:
To be called a limited accuracy ISO/IEC 11172-3 audio decoder, the decoder shall provide an output for a provided test sequence such that the rms level of the difference signal between the output of the decoder under test and the supplied reference output is less than 2−11⁄√12 for the supplied sine sweep (20Hz–10kHz) with an amplitude of −20dB relative to full scale.
Thus, for the purpose of testing computational accuracy, we shall use three categories of compliance: fully compliant, limited accuracy, and not compliant.
Designations
ISO/IEC 11172-4 designates two kinds of audio decoders. Section 2.6.3 states:
An ISO/IEC 11172-3 compliant decoder that is able to support at least one but not all combinations of the options defined in 2.4.3 such as bit rates, sampling rates and modes, will be designated as an ISO/IEC 11172-3 Layer “N” audio decoder. Decoders that support all combinations are designated as Full Layer “N” ISO/IEC 11172-3 audio decoders, where “N” indicates I, II or III.
It should also be noted that, according to section 0.2 of ISO/IEC 11172-3:
An ISO/IEC 11172-3 Audio Layer N decoder is able to decode bitstream data which has been encoded in Layer N and all layers below N.
About the Tests
The following results have been obtained in tests conducted by Underbit in accordance with Annex A of ISO/IEC 11172-4.
All measurements are carried out relative to full scale where decoder output signals are normalized to be between −1.0 and +1.0. The supplied sine sweep with an amplitude of −20dB relative to full scale has an absolute amplitude of ±0.1 and a precision of 24 bits.
As instructed, where the output of a decoder is fewer than 24 bits, the remaining least significant bits are set to zero for purposes of normalization and measurement. (The number of output bits from the decoder is noted.)
Where the compliance test data contains two channels, or in the case of Layer II where multiple compliance test streams exist, the rms level and maximum difference values are calculated separately for each, and the greatest overall value used for the result.
Calculation of RMS
The rms (root-mean-square) level of the difference signal is calculated as
where ti is the ith output sample from the decoder under test, ri is the ith sample from the reference output, and N is the number of samples.
The maximum absolute value of the difference signal is simply the greatest value of |ti − ri|.
Test Results
Decoder | Bits | Layer I | Layer II | Layer III | |||
---|---|---|---|---|---|---|---|
RMS Level | Max Diff | RMS Level | Max Diff | RMS Level | Max Diff | ||
Floating-Point Decoders | |||||||
amp 0.7.6 | 16 | failed | 2.099×10−5 | 3.421×10−5 | 1.959×10−5 | 3.362×10−5 | |
limited accuracy | limited accuracy | ||||||
Amplay 1 | 16 | failed | failed | 7.860×10−2 | 1.487×10−1 | ||
not compliant | |||||||
Apollo 37 | 16 | failed | 2.099×10−5 | 3.421×10−5 | 1.961×10−5 | 3.529×10−5 | |
limited accuracy | limited accuracy | ||||||
Audioactive MP3 Decoder 1.12 | 16 | failed | failed | 8.602×10−6 | 1.538×10−5 | ||
fully compliant | |||||||
CoolPlayer
7224 [Xaudio] |
16 | 8.728×10−6 | 1.526×10−5 | 9.336×10−5 | 3.175×10−3 | 8.602×10−6 | 1.538×10−5 |
fully compliant | limited accuracy | fully compliant | |||||
FreeAmp 2.1 beta 5 | 16 | failed | 1.408×10−1 | 2.259×10−1 | 1.410×10−1 | 2.003×10−1 | |
not compliant | not compliant | ||||||
Jet-Audio
4.7 (64-bit float mode) |
16 | 1.351×10−1 | 1.938×10−1 | 1.367×10−1 | 2.519×10−1 | 1.366×10−1 | 1.941×10−1 |
not compliant | not compliant | not compliant | |||||
LAME
3.86 beta [mpglib] |
16 | failed | failed | 8.870×10−6 | 1.836×10−5 | ||
limited accuracy | |||||||
maplay 1.2 | 16 | 1.738×10−5 | 3.052×10−5 | 1.850×10−5 | 3.052×10−5 | failed | |
limited accuracy | limited accuracy | ||||||
maplay 1.2+ for Win32 1.A | 16 | 8.728×10−6 | 1.526×10−5 | 9.996×10−2 | 2.659×10−1 | 8.602×10−6 | 1.526×10−5 |
fully compliant | not compliant | fully compliant | |||||
mp3blaster 2.0b17 | 16 | 1.738×10−5 | 3.052×10−5 | 6.654×10−1 | 1.130×100 | 1.783×10−5 | 3.088×10−5 |
limited accuracy | not compliant | limited accuracy | |||||
MP3 Decoder 1.45 | 16 | failed | 9.995×10−2 | 2.659×10−1 | 1.074×10−3 | 3.479×10−2 | |
not compliant | not compliant | ||||||
MP3Player 1.23 | 16 | 8.728×10−6 | 1.526×10−5 | 8.759×10−6 | 1.526×10−5 | 8.602×10−6 | 1.538×10−5 |
fully compliant | fully compliant | fully compliant | |||||
MP3 To Wave Converter 1.14 | 16 | 5.234×10−2 | 1.255×10−1 | 9.793×10−2 | 2.226×10−1 | 9.982×10−2 | 2.000×10−1 |
not compliant | not compliant | not compliant | |||||
MpegDJ GoWave! 1.41 | 16 | failed | 1.854×10−5 | 3.779×10−5 | 1.780×10−5 | 3.064×10−5 | |
limited accuracy | limited accuracy | ||||||
mpg123 0.59r | 16 | 8.728×10−6 | 1.526×10−5 | 8.759×10−6 | 1.526×10−5 | 8.602×10−6 | 1.538×10−5 |
fully compliant | fully compliant | fully compliant | |||||
MusicMatch Jukebox
5.10.0149 [Xaudio] |
16 | 8.728×10−6 | 1.526×10−5 | 8.759×10−6 | 1.526×10−5 | 8.602×10−6 | 1.538×10−5 |
fully compliant | fully compliant | fully compliant | |||||
NAD 0.94pb3 | 16 | 8.822×10−3 | 1.107×10−1 | 9.639×10−2 | 2.000×10−1 | 9.991×10−2 | 2.000×10−1 |
not compliant | not compliant | not compliant | |||||
QuickTime 4.1.2, x86 | 16 | failed | failed | 8.602×10−6 | 1.526×10−5 | ||
fully compliant | |||||||
RightClick-MP3
1.65e [L3dec 2.72] |
16 | failed | failed | 8.602×10−6 | 1.538×10−5 | ||
fully compliant | |||||||
Shibatch mpg123 plug-in 1.18+ for Winamp | 16 | failed | failed | 8.854×10−6 | 1.836×10−5 | ||
limited accuracy | |||||||
Sonique 1.63 | 16 | 1.394×10−1 | 2.001×10−1 | 8.759×10−6 | 1.526×10−5 | 1.410×10−1 | 2.003×10−1 |
not compliant | fully compliant | not compliant | |||||
Sonique
1.808 [audioEnlightenment 4.70] |
16 | 8.981×10−6 | 1.836×10−5 | 8.994×10−6 | 1.895×10−5 | 8.867×10−6 | 1.836×10−5 |
limited accuracy | limited accuracy | limited accuracy | |||||
Sonique
1.808 [audioEnlightenment 4.858 (Default Decoder)] |
16 | 8.981×10−6 | 1.836×10−5 | 8.994×10−6 | 1.895×10−5 | 8.867×10−6 | 1.836×10−5 |
limited accuracy | limited accuracy | limited accuracy | |||||
splay 0.8.2 | 16 | 1.738×10−5 | 3.052×10−5 | 1.863×10−5 | 4.232×10−5 | 1.783×10−5 | 3.088×10−5 |
limited accuracy | limited accuracy | limited accuracy | |||||
UltraPlayer 2.00 | 16 | 8.965×10−6 | 1.824×10−5 | 1.262×10−2 | 1.297×10−1 | 8.854×10−6 | 1.836×10−5 |
limited accuracy | not compliant | limited accuracy | |||||
Winamp
2.65 [Nullsoft MPEG audio decoder plug-in 2.33 (486)] |
16 | 8.980×10−6 | 1.836×10−5 | 9.020×10−6 | 1.907×10−5 | 9.066×10−6 | 5.877×10−5 |
limited accuracy | limited accuracy | limited accuracy | |||||
Winamp
2.65 [Nullsoft MPEG audio decoder plug-in 2.33 (Pentium)] |
16 | 8.980×10−6 | 1.836×10−5 | 9.020×10−6 | 1.907×10−5 | 9.067×10−6 | 5.877×10−5 |
limited accuracy | limited accuracy | limited accuracy | |||||
Winamp
2.65 [Nullsoft MPEG audio decoder plug-in 2.33 (MMX)] |
16 | 4.524×10−5 | 1.634×10−4 | 4.539×10−5 | 1.811×10−4 | 9.067×10−6 | 5.877×10−5 |
limited accuracy | limited accuracy | limited accuracy | |||||
Winamp
2.666 [Nullsoft MPEG Audio Decoder 2.666] |
16 | 8.727×10−6 | 1.526×10−5 | 8.759×10−6 | 1.526×10−5 | 8.602×10−6 | 1.526×10−5 |
fully compliant | fully compliant | fully compliant | |||||
Xaudio
1.1.0, x86 [decoder library 1.3.1] |
16 | 8.728×10−6 | 1.526×10−5 | 8.759×10−6 | 1.538×10−5 | 8.602×10−6 | 1.538×10−5 |
fully compliant | fully compliant | fully compliant | |||||
Integer Decoders | |||||||
Intel® IPP 1.01, StrongARM | 16 | failed | failed | 8.631×10−6 | 1.812×10−5 | ||
fully compliant | |||||||
MAD 0.11.4b, ARM | 24 | 4.667×10−8 | 2.384×10−7 | 4.906×10−8 | 2.384×10−7 | 5.338×10−8 | 2.384×10−7 |
fully compliant | fully compliant | fully compliant | |||||
MAD 0.11.4b, x86 | 24 | 6.198×10−8 | 3.576×10−7 | 6.198×10−8 | 2.384×10−7 | 9.000×10−8 | 9.537×10−7 |
fully compliant | fully compliant | fully compliant | |||||
MAD
0.11.4b, x86 (optimized for accuracy) |
24 | 4.667×10−8 | 2.384×10−7 | 4.906×10−8 | 2.384×10−7 | 5.555×10−8 | 2.384×10−7 |
fully compliant | fully compliant | fully compliant | |||||
MAD
0.11.4b, x86 (optimized for speed) |
24 | 7.131×10−6 | 2.480×10−5 | 7.250×10−6 | 2.789×10−5 | 7.227×10−6 | 2.730×10−5 |
fully compliant | fully compliant | fully compliant | |||||
mpg123 0.59r-arm32 | 16 | 6.972×10−2 | 1.000×10−1 | 7.058×10−2 | 1.300×10−1 | 7.051×10−2 | 1.001×10−1 |
not compliant | not compliant | not compliant | |||||
splay 0.8.2-fp1 | 16 | 2.180×10−5 | 6.604×10−5 | 1.961×10−5 | 4.458×10−5 | 5.618×10−2 | 1.190×10−1 |
limited accuracy | limited accuracy | not compliant | |||||
Xaudio
1.1.0, ARM [decoder library 1.3.1] |
16 | 6.945×10−2 | 9.965×10−2 | 2.003×10−5 | 3.576×10−5 | 2.065×10−5 | 6.580×10−5 |
not compliant | limited accuracy | limited accuracy |
Legend | ||
---|---|---|
fully compliant | The rms level of the difference signal is less than 2−15⁄√12 (≅ 8.810×10−6) and the maximum absolute value of the difference signal is less than or equal to 2−14 (≅ 6.104×10−5). | The decoder is an ISO/IEC 11172-3 Layer “N” audio decoder. |
limited accuracy | The rms level of the difference signal is less than 2−11⁄√12 (≅ 1.410×10−4). (The maximum absolute value of the difference signal does not matter.) | The decoder is a limited accuracy ISO/IEC 11172-3 Layer “N” audio decoder. |
not compliant | The rms level of the difference signal is greater than or equal to 2−11⁄√12 (≅ 1.410×10−4). | The decoder is not a compliant ISO/IEC 11172-3 Layer “N” audio decoder. |
failed | The decoder failed to produce output for the given test bitstream. | The decoder is not a Full Layer “N” ISO/IEC 11172-3 audio decoder. |
Notes
Full Compliance
It should be noted that a fully compliant result from these tests does not mean the decoder in question is without flaws.
These tests only verify the computational accuracy of decoders against a reference signal that spans only 20Hz–10kHz of the frequency spectrum. While this test may be enough to detect a problem with a decoder, it is not enough to certify that a given decoder will correctly reconstruct frequencies outside this range, or that the same decoder will correctly handle any valid bitstream. To obtain these kinds of assurances, further tests are necessary.
Quantization Method
It should further be noted that these test results can be affected by the method used by each decoder to quantize its PCM output. Some decoders may use dithering or noise shaping to reduce the negative effects of quantization noise and improve the perceived audio quality, while others may simply round each output sample to the nearest integral value. Unfortunately, with few exceptions, it is not known what quantization method is used by each of the tested decoders.
It is possible for an otherwise fully compliant decoder that implements dithering with 16-bit output to appear only to have limited accuracy. The decoder’s output before quantization should be the true measure of its accuracy.