Stupid guessing games

[Mark Bassett]

I're recently created and uploaded an OSX app designed to test whether users can hear a difference between two versions of a track, one CD-quality, one MP3.

One of the tracks is a 16-bit file, the other is a 320kbps MP3 (VBR).

If this was a controlled experiment you'll need to get 16 or more correct out of 20 to demonstrate that you can hear a difference.

The app is available here

http://markmckinnonbassett.wordpress.com/applications/

Or if you're Neil Young, you'll probably agree with this poorly-argued article suggesting that it's all bullshit, which IMO is the audio engineering equivalent of climate change denial.

http://tapeop.com/blog/2012/04/11/problem-bing-and-why-neil-young-right-about-sound-/

Can of worms now open...

[ChrisW]

Very interested in this. Thanks
Dropbox said more than a one hour download, so I may not download tonight.

[Mark Bassett]

Very interested in this. Thanks
Dropbox said more than a one hour download, so I may not download tonight.

It's a 45MB file, shouldn't take that long though.

[Manning]

Mark - that "A/B" comparison article you linked is spot on.

If you listened to an audio file of a Scotsman and then an American talking, you would have no problem telling them apart.

However if you listened to an audio file of (say) an Armenian and an Uzbek talking, you would probably have enormous trouble identifying any differences, despite the fact that they would be speaking entirely different languages (and are thus far more "different" than the first example).

Most A/B tests of audio files are like the second case above. Only if you sat and listened to the source material long enough to gain a deep familiarity, will it become like the first example.

As a final point - having been married to a Turkish girl for a number of years and having spent a lot of time in that country, I can now identify different regional accents of Turkish, which I'm pretty certain most people here can't do.

So when an audio expert tells me they can hear differences between an MP3 and a WAV file that I personally can't hear, I'm not so quick to disagree with them.

[Mark Bassett]

Mark - that "A/B" comparison article you linked is spot on.

Sure, aside from the logical fallacies marbled throughout his argument and the fact that author is misleading the reader to the point of straight up lying to them, then sure, spot on.

I agree with what you're saying about language but they're are not the same thing, and that article isn't talking about spending time getting to know the material, then do a A/B test and you'll be good to go.

You're essentially saying, and correct me if I'm wrong, that if I was to compare two languages, or dialects that I am unfamiliar with, I would struggle to pick up on the differences between the two, probably wouldn't hear any difference. But if I was exposed to either of the languages/dialects for a sufficient time, my ability to tell the difference between the two would improve, I agree.

The problem is that this isn't what the article is saying.

The article isn't saying that although you might not be able to hear a difference between mp3 and wav right now, all you need to do is spend enough time with say the mp3 version, and eventually you'll be able to tell the two apart in a A/B test - exactly like the scenario you pitched with language.

The article is saying that you can never do an A/B test, the only acceptable test is to "give people a music collection that's all MP3s for a month, then give them that same collection as 24bit WAVs for a month, and then ask which one's which". Any test that 'flips' between one and the other is void.

You're suggesting that A/B tests can and will work, but it all comes down to how well you know the material, I agree, most people I dare say would.

The article is saying the A/B tests are void, period, and any test that rapidly changes between stimuli misses the point completely. It suggests that the test needs to be delivered in the same manner that people listen to music... "If you can't tell the difference when you flip flip flip between two subtly different sounds, please know that I'm here cheerleading for you to slow down and be a real human being".

How you listen to a record, and how you test for discrimination aren't related.

[jkhuri44]

vocal tonalities between languages and audio quality are 2 different things??

[Mark Bassett]

vocal tonalities between languages and audio quality are 2 different things??

They are most definitely related, after all, both simply come down to can you hear a difference between 2 sounds.

But a test like the one I made isn't a test of quality, we're not talking preferences here. It's a test of discrimination.

[audioio]

vocal tonalities between languages and audio quality are 2 different things??

They are most definitely related, after all, both simply come down to can you hear a difference between 2 sounds.

Well, sort of. The WAV and the MP3 are different encodings of a recording of the exact same sound(s), whereas two people from different regions of Turkey saying the same words (no idiom or vernacular) are not the exact same sound(s).

[Mark Bassett]

vocal tonalities between languages and audio quality are 2 different things??

They are most definitely related, after all, both simply come down to can you hear a difference between 2 sounds.

Well, sort of. The WAV and the MP3 are different encodings of a recording of the exact same sound(s), whereas two people from different regions of Turkey saying the same words (no idiom or vernacular) are not the exact same sound(s).

How the difference between the two sounds came into being is irrelevant.

They are both test of auditory discrimination at heart, nothing more.

[Mark Bassett]

Forgive me folks if I'm a little edgy about this. I've spent 4 years researching in this field, and I'm at an 'interesting' place in my studies. This article just grinds my gears.

[Mark Bassett]

EDIT: Not worth it.

[gigpiglet]

I've downloaded (took about 5 or 10min)
but havnt had a chance to do it yet…
hopefully today

[Manning]

Mark - that "A/B" comparison article you linked is spot on.

The article is saying that you can never do an A/B test, the only acceptable test is to "give people a music collection that's all MP3s for a month, then give them that same collection as 24bit WAVs for a month, and then ask which one's which". Any test that 'flips' between one and the other is void.

You're suggesting that A/B tests can and will work, but it all comes down to how well you know the material, I agree, most people I dare say would.

LOL perhaps I should have read the article a bit more carefully. Anyway, in hindsight, my real main point was about respecting expertise.

I know people who insist that because *they* can't hear a given difference, then that difference doesn't exist. (I'm actually one of the people who can't hear those differences.)

But I'm not comfortable disputing the opinions of those who have spent years studying an issue.

[rick]

I like to think i can hear pretty well
Its kinda ok for me to say that i think

I do comparator tests all day .. Which is better
This or that ..?
as mark ( who worked for me for years can attest)
I am rarely if ever 2 nd guessing what i believe to be correct
An i am pretty much spot on with my gut which is why
I am still in business i revkon

But can i pass other non work related ab tests ..?
You know some days depending on how i feel
What i have eating in the previous few days
I ascert my trained ears can hear anything anybody
Elses trained ears can hear
On other days i very much doubt i can hear
Like a trained audio freak at all


Point being that the ab test might be constant
But at a extreme level even trained hearing is
Subjective to enviromental influences

Is the margin for error in judgement
Significant ..?
Statistically .. Highly likely
In the real world talking about music
Highly unlikrly


All that said i do working method a/ b tests
All day .. I never hear a chorus or a verse
Untill the end of the day when i have a " listen"
To what the scientest within has done

I like the end of the day as a life experience. Much much more
Then the testing part of the day

Most people i have taught to work the way i do
Have found it revolutionary
Some guys just couldnt grasp how it works
Its probably just not how they are wired
Not how they can descern life through sound

I wont be downloading the ab test
I am sitting in the sun eating a hamburger
Life Its very satisfying as is

[ChrisW]

The weird rural internet…
My second attempt to download only took 10 minutes.

[justmull]

So - haven't listened to the test yet, haven't fired up my mac.
But a question - did you use joint stereo or discrete stereo mp3 encoding? To me I can never hear the difference between the latter and the original at 320kbps in the majority of situations if I've converted it myself.

[Mark Bassett]

So - haven't listened to the test yet, haven't fired up my mac.
But a question - did you use joint stereo or discrete stereo mp3 encoding? To me I can never hear the difference between the latter and the original at 320kbps in the majority of situations if I've converted it myself.

Conversion was done in iTunes.

Stereo Bit Rate: 320kbps
VBR: Off
Sample Rate: (Auto - 44.1)
Channels: (Auto - 2)
Stereo Mode: Normal
Smart Encoding Adjustments: On
Filter Frequencies below 10Hz: On

The Mp3 was then converted to a .wav file. This was necessary as Max/MSP doesn't play MP3s.

What you are switching between in the App is a 24-bit .wav file and a 24-bit .wav version of a 320kbps MP3.