Well, in many cases, they are indeed learning something false (or, to be more charitable, something that is not sufficiently true). For example, a facial recognition system that has trouble detecting dark-skinned faces. Black and brown people clearly have faces, but because the systems were not trained on their facial data, and the capability to adequately adjust exposure or other aspects of the sensor stack was not built into the system, it provides an objectively wrong solution given perfectly reasonable inputs.
I agree that this process is one of resolving blind spots, but I disagree that the blind spots are simply areas devoid of light. AI/ML systems are frequently employed to augment or stand in for human perception, which is known to be necessarily incomplete with respect to reality. In other words, they can learn things that seem true to us but that are false from another perspective, or undesirable once exposed. What's exciting about them is that they provide an opportunity to interrogate the flaws in our individual perception with a systematized observation and analysis, in a much more sophisticated manner than in the past. But fulfilling that potential requires humility.
Yeah that's a good point, about having sufficient examples in the training set. I think it's not as simple as that though, several years after that made headlines people have still not been able to close the face recognition accuracy gap. Experts are debating different explanations for this.
That wasn't entirely my point, however. The training set was only a part of the problem, which in its totality was that the developers failed to consider that the functioning of their system could be affected by the biases they didn't even know they had, in this case of being more likely to see light-skinned faces as belonging to a human.
Their failure was not just in lacking diverse training sets, but diverse QA, or at least QA looking for those blindspots which eventually became evident.
So, correct, it's not as simple as having "sufficient" data.
Normally the developers wouldn't classify images themselves (their time is too valuable). In any case, I think failing to detect a face as human should be rare unless the photo has very poor lighting or the person is almost hidden.
The developers, or the project managers, are the ones setting the parameters: the expected outputs and the processes necessary for that output to be generated. That they don't classify the images themselves doesn't take away from the failure of the system to work as expected being a result of their failure to fully understand the problem, which is itself reflected in their unconcious biases.
Your expectation was the same one they had, and it was wrong, which is the crux of the issue.
There's only been one case of being unable to recognise black faces that I know of, and it was shown later to be due to the lighting conditions the guy was using leading to very low contrast imagery. The same problem was replicated with white faces: there was no racism anywhere as you would expect given that unconscious bias hasn't been shown to exist at all (the studies that claim to show it have all collapsed).
If you asked the developers of the facial recognition library, "does your software have problems with very low contrast conditions" they'd surely have answered yes. Fully conscious of the issue but, that's software. It's hard to get everything right 100% of the time.
I agree that this process is one of resolving blind spots, but I disagree that the blind spots are simply areas devoid of light. AI/ML systems are frequently employed to augment or stand in for human perception, which is known to be necessarily incomplete with respect to reality. In other words, they can learn things that seem true to us but that are false from another perspective, or undesirable once exposed. What's exciting about them is that they provide an opportunity to interrogate the flaws in our individual perception with a systematized observation and analysis, in a much more sophisticated manner than in the past. But fulfilling that potential requires humility.