Supermarket facial recognition failure: why automated systems must put the human factor first

By Mark Rickerby, University of Canterbury

The incident of a woman misidentified by facial recognition technology at a Rotorua supermarket should have come as no surprise.

When Foodstuffs North Island announced its intention to trial this technology in February, as part of a strategy to combat retail crime, technology and privacy experts immediately raised concerns.

In particular, the risk of Māori women and women of colour being discriminated against was raised, and has now been borne out by what happened in early April to Te Ani Solomon.

Speaking to media this week, Solomon said she thought ethnicity was a “huge factor” in her wrongful identification. “Unfortunately, it will be the experience of many Kiwis if we don’t have some rules and regulations around this.”

The supermarket company’s response that this was a “genuine case of human error” fails to address the deeper questions about such use of AI and automated systems.

https://twitter.com/1NewsNZ/status/1782141002645409919?t=a6pVgmQ08y-tIIpk_OUAFA&s=19

Automated decisions and human actions

Automated facial recognition is often discussed in the abstract – as pure algorithmic pattern matching, with emphasis on assessing correctness and accuracy.

These are rightfully important priorities for systems that deal with biometric data and security. But with such crucial focus on the results of automated decisions, it’s easy to overlook concerns about how these decisions are applied.

Designers use the term “context of use” to describe the everyday working conditions, tasks and goals of a product. With facial recognition technology in supermarkets, the context of use goes far beyond traditional design concerns such as ergonomics or usability.

It requires consideration of how automated trespass notifications trigger in-store responses, protocols for managing those responses, and what happens when things go wrong. These are more than just pure technology or data problems.

This perspective helps us understand and balance the impact of engineering and design interventions at different levels of a system.

Investing in improving prediction accuracy seems an obvious priority for facial recognition systems.

Read Full Story https://theconversation.com/supermarket-facial-recognition-failure-why-automated-systems-must-put-the-human-factor-first-228284