The UK’s privacy watchdog has warned against use of so-called “emotion analysis” technologies for anything more serious than kids’ party games, saying there’s a discrimination risk attached to applying “immature” biometric tech that makes pseudoscientific claims about being able to recognize people’s emotions using AI to interpret biometric data inputs.
Such AI systems ‘function’, if we can use the word, by claiming to be able to ‘read the tea leaves’ of one or more biometric signals, such as heart rate, eye movements, facial expression, skin moisture, gait tracking, vocal tone etc, and perform emotion detection or sentiment analysis to predict how the person is feeling — presumably after being trained on a bunch of visual data of faces frowning, faces smiling etc (but you can immediately see the problem with trying to assign individual facial expressions to absolute emotional states — because no two people, and often no two emotional states, are the same; hence hello pseudoscience!).
The watchdog’s deputy commissioner, Stephen Bonner, appears to agree that this high tech nonsense must be stopped — saying today there’s no evidence that such technologies do actually work as claimed (or that they will ever work).
“Developments in the biometrics and emotion AI market are immature. They may not work yet, or indeed ever,” he warned in a statement. “While there are opportunities present, the risks are currently greater. At the ICO, we are concerned that incorrect analysis of data could result in assumptions and judgements about a person that are inaccurate and lead to discrimination.
“The only sustainable biometric deployments will be those that are fully functional, accountable and backed by science. As it stands, we are yet to see any emotion AI technology develop in a way that satisfies data protection requirements, and have more general questions about proportionality, fairness and transparency in this area.”
“The ICO will continue to scrutinise the market, identifying stakeholders who are seeking to create or deploy these technologies, and explaining the importance of enhanced data privacy and compliance, whilst encouraging trust and confidence in how these systems work,” added Bonner.
The watchdog has fuller biometrics guidance coming in the spring — which it said today will highlight the need for organizations to pay proper mind to data security — so Bonner’s warning offers a taster of more comprehensive steerage coming down the pipe in the next half year or so.
“Organisations that do not act responsibly, posing risks to vulnerable people, or fail to meet ICO expectations will be investigated,” the watchdog added.
Its blog post gives some examples of potentially concerning uses of biometrics — including AI tech being used to monitoring the physical health of workers via the use of wearable screening tools; or the use of visual and behavioural methods such as body position, speech, eyes and head movements to register students for exams.
“Emotion analysis relies on collecting, storing and processing a range of personal data, including subconscious behavioural or emotional responses, and in some cases, special category data. This kind of data use is far more risky than traditional biometric technologies that are used to verify or identify a person,” it continued. “The inability of algorithms which are not sufficiently developed to detect emotional cues, means there’s a risk of systemic bias, inaccuracy and even discrimination.”
The ICO may be feeling moved to make more substantial interventions in this area because UK lawmakers aren’t being proactive when it comes to biometrics regulation.
The GDPR regulates the processing of biometrics data when it’s used for identifying individuals — and also includes a right to human review of certain substantial algorithmic decisions. So if the government is intent on ripping up the current rulebook it raises the question of how — or even whether — biometric technologies will be regulated in the UK in the future?
And that makes the ICO’s public pronouncements on the risks of pseudoscientific biometric AI systems all the more important. (It’s also noteworthy that the regulator name-checks the involvement of the Ada Lovelace Institute (which commissioned the aforementioned legal review) and the British Youth Council which it says will be involved in a process of public dialogues it plans to use to help shape its forthcoming ‘people-centric’ biometrics guidance.)
“Supporting businesses and organisations at the development stage of biometrics products and services embeds a ‘privacy by design’ approach, thus reducing the risk factors and ensuring organisations are operating safely and lawfully,” the ICO added, in what could be interpreted as rather pointed remarks on government policy priorities.
The regulator’s concern about emotional analysis tech is not an academic risk, either.
It’s not clear what the status of the underlying ‘AI lie detector’ technology is now. The Manchester company involved in the ‘proof of concept’ project — which was also linked to research at Manchester Metropolitan University — was dissolved this summer, per Companies House records. But the iBorderCtrl project was also criticized on transparency grounds, and has faced a number of freedom of information actions seeking to lift the lid on the project and the consortium behind it — with, apparently, limited success.
Its CEO Ali Parser said at the time that the emotion-scanning tech had been built and implied it would be coming to market — however the company later rowed back on the claim, saying the AI had only been used in pre-market testing and development had been deprioritized in favor of other AI-powered features.
The ICO will surely be happy that Babylon had a rethink on using AI to claim its software could perform remote emotion-scanning.
Its blog post goes on to cite other current examples where biometric tech, more broadly, is being used — including in airports for streamlining passenger journeys; financial companies using live facial recognition tech for remote ID checks; and companies using voice recognition for convenient account access, instead of having to remember passwords.
The regulator doesn’t make specific remarks on the cited use-cases but it looks likely it will be keeping a close eye on all applications of biometrics given the high potential risks to people’s privacy and rights — even as its most special attention will be directed toward uses of the tech that slip their chains and stray into the realms of science fiction.
The ICO’s blog post notes that its look into “biometrics futures” is a key part of the its “horizon-scanning function”. Which is technocrat speak for ‘scrutiny of this type of AI tech being prioritized because it’s fast coming down the pipe at us all’.
“This work identifies the critical technologies and innovation that will impact privacy in the near future — its aim is to ensure that the ICO is prepared to confront the privacy challenges transformative technology can bring and ensure responsible innovation is encouraged,” it added.
UK watchdog warns against AI for emotional analysis, dubs ‘immature’ biometrics a bias risk by Natasha Lomas originally published on TechCrunch