TECHNOLOGY & HUMAN RIGHTS

The World Bank and co. may be paving a ‘Digital Road to Hell’ with support for dangerous digital ID

Global actors, led by the World Bank, are energetically promoting biometric and other digital ID systems that are increasingly linked to large-scale human rights violations, especially in the Global South. A report by researchers at New York University warns that these systems, promoted in the name of development and inclusion, might be achieving neither. Rather than the equitable digital future envisioned by the World Bank and its Identification for Development (ID4D) Initiative, the report argues that “despite undoubted good intentions on the part of some, [these systems] may well be paving a digital road to hell.”

The report, at over 100 pages, is intended to be a “carefully researched primer as well as a call to action to all of those with an interest in safeguarding human rights to set their gaze more firmly on the multidimensional dangers associated with digital ID systems.” Governments around the world have been investing heavily in digital identification systems, often with biometric components (digital ID). The rapid proliferation of such systems is driven by a new development consensus, packaged and promoted by key global actors like the World Bank, but also by governments, foundations, vendors and consulting firms. This new ‘manufactured consensus’ holds that digital ID can contribute to inclusive and sustainable development—and is even a prerequisite for the realization of human rights.

Drawing inspiration from the Aadhaar system in India, the dangerous digital ID model that is being promoted prioritizes what the primer refers to as an ‘economic identity’.  The goal of such systems is primarily to establish ‘uniqueness’ of individuals, commonly with the help of biometric technologies. The ultimate objective of such digital ID systems is to facilitate economic transactions and private sector service delivery while also bringing new, poorer, individuals into formal economies and ‘unlocking’ their behavioral data. As the Executive Chairman of the influential ID4Africa, a platform where African governments and major companies in the digital ID market meet, put it at the start of its 2022 Annual Meeting earlier this week, digital ID is no longer about identity alone but “enables and interacts with authentication platforms, payments systems, digital signatures, data sharing, KYC systems, consent management and sectoral delivery platforms.”

Unlike ‘traditional systems’ of civil registration, such as birth registration, this new model of economic identity commonly sidesteps difficult questions about the legal status of those it registers and the rights associated with that status. The promises of inclusion and flourishing digital economies might appear attractive on paper, but digital ID systems have consistently failed to deliver on these promises in real world situations, especially for the most marginalized. In fact, evidence is emerging from many countries, most notably the mega digital ID project Aadhaar in India, of the severe and large-scale human rights violations linked to this model. These systems may in fact exacerbate pre-existing forms of exclusion and discrimination in public and private services. The use of new technologies may furthermore lead to novel forms of harm, including biometric exclusion, discrimination, and the many harms associated with “surveillance capitalism.”

Meanwhile, the benefits of digital ID remain ill-defined and poorly documented. From what evidence does exist, it seems that those who stand to benefit most may not be those “left behind”,but instead a small group of companies and governments. After all, where digital ID systems have tended to excel is in generating lucrative contracts for biometrics companies and enhancing the surveillance and migration-control capabilities of governments.

With such powerful backing, digital ID has taken on the guise of an unstoppable juggernaut and inevitable hallmark of modernity and development in the 21st century, and the dissenting voices of civil society have been written off as Luddites and barriers to progress. Nevertheless, the report calls on human rights organizations, other civil society organizations, and advocates who may have been on the sidelines of these debates to get more involved.  The actual and potential human rights violations arising from this model of digital ID can be severe and potentially irreversible. The human rights community can play an important role in ensuring that such transformational changes are not rushed and are based on serious evidence and analysis. It can also ensure that there is sufficient public debate, with full transparency and involving all relevant stakeholders, not in the least the most marginalized and most affected individuals. Where necessary to safeguard human rights, such dangerous digital ID systems should be stopped altogether.

This post was originally published as a press release on June 17, 2022.

TECHNOLOGY & HUMAN RIGHTS

Sorting in Place of Solutions for Homeless Populations: How Federal Directives Prioritize Data Over Services

National data collection and service prioritization were supposed to make homeless services more equitable and efficient. Instead, they have created more risks and bureaucratic burdens for homeless individuals and homeless service organizations.

While serving as an AmeriCorps VISTA member supporting the IT and holistic defense teams at a California public defender, much of my time was spent navigating the data bureaucracy that now weighs down social service providers across the country. In particular, I helped social workers and other staff members use tools like the Vulnerability Index – Service Prioritization Decision Assistance Tool (VI-SPDAT) and a Homeless Management Information System (HMIS). While these tools were ostensibly designed to improve care for homeless and housing insecure people, all too often they did the opposite.

An HMIS is a localized information network and database used to collect client-level data and data on the provision of housing and services to homeless or at-risk persons. In 2011, Congress passed the HEARTH Act, mandating the use of HMIS by communities in order to receive federal funding. HMIS demands coordinated entry, a process by which certain types of data are cataloged and clients are ranked according to their perceived need. One of the most common tools for coordinated entry—and the one used by the social workers I worked with—is VI-SPDAT. VI-SPDAT is effectively a questionnaire which involves a battery of highly invasive questions which seek to determine the level of need of the homeless or housing insecure individual to whom it is administered.

These tools have been touted as game-changers, but while homelessness across the country, and especially in California, continued to decrease modestly in the years immediately following the enactment of the HEARTH act, it began to increase again in 2019 and sharply increased in 2020, even before the onset of the COVID-19 pandemic. This is not to suggest a causal link; indeed, the evidence suggests that factors such as rising housing costs and a worsening methamphetamine epidemic are at the heart of rising homelessness. But there is little evidence that intrusive tools like VI-SPDAT alleviate these problems.

Indeed, these tools have themselves been creating problems for homeless persons and social workers alike. There have been harsh criticisms from scholars like Virginia Eubanks about the accuracy and usefulness of VI-SPDAT. It has been found to produce unreliable and racially biased results. Rather than decreasing bias as it purports to do, VI-SPDAT has baked bias into its algorithms, providing a veneer of scientific objectivity for government officials to hide behind.

But, even if these tools were to be made more reliable and less biased,  they would nonetheless cause harm and stigmatization. Homeless individuals and social workers alike report finding the assessment dehumanizing and distressing. For homeless individuals, it can also feel deeply risky. Those who don’t score high enough on the assessment are often denied housing and assistance altogether. Those who score too high run the risk of involuntary institutionalization.

Meanwhile, these tools place significant burdens on social workers. To receive federal funding, organizations must provide not only an intense amount of highly intimate information about homeless persons and their life histories, but also a minute accounting of every interaction between the social worker and the client. One social worker would frequently work with clients from 9-5, go home to make dinner for her children, and then work into the wee hours of the night attempting to log all of her data requirements.

I once sat through a 45-minute video call with a veteran social worker who broke down into tears worried that the grant funding her position might be taken away if her record keeping was less than perfect, but the design of the HMIS made it virtually impossible to be completely honest. The system anticipated that four-hour client interactions could easily be broken down into distinct chunks—discussed x problem from 4:15 to 4:30, y problem from 4:30 to 4:45, and so on. Of course, anyone who has ever had a conversation with another human being, let alone a human being with mental disabilities or substance use problems, knows that interactions are rarely so tidy and linear.

While this data is claimed to be kept very secure, in reality, hundreds of people in dozens of organizations typically have access to any given HMIS. There are guidelines in place to protect the data, but there is minimal monitoring to ensure that these guidelines are being followed, and many users found them very difficult to follow while working from home during the pandemic. I heard multiple stories of police or prosecutors improperly accessing information from HMIS. Clients can request to have their information removed from the system, but the process for doing so is rarely made clear to them, nor is this process clear even for the social workers processing the data.

After years of criticism, OrgCode—the group which develops VI-SPDAT—announced in 2021 that it would no longer be pushing VI-SPDAT updates, and as of 2022 it is no longer providing support for the current iteration of VI-SPDAT. While this is a commendable move from OrgCode, stakeholders in homeless services must acknowledge the larger failures of HMIS and coordinate entry more generally. Many of the other tools used to perform coordinated entry have similar problems to VI-SPDAT, in part because coordinated entry in effect requires this intrusive data collection about highly personal issues to determine needs and rank clients accordingly. The problems are baked into the data requirements of coordinated entry itself.

The answer to this problem cannot be to completely do away with any classification tools for housing insecure individuals, because understanding the scope and demographics of homelessness is important in tackling it. But clearly a drastic overhaul of these systems is needed to make sure that they are efficient, noninvasive, and accurate. Above all, it is crucial to remember that tools for sorting homeless individuals are only useful to the extent that they ultimately provide better access to the services that actually alleviate homelessness, like affordable housing, mental health treatment, and addiction support. Demanding that beleaguered social service providers prioritize data collection over services, all while using intrusive, racially biased, and dehumanizing tools, will only worsen an intensifying crisis.

May 17, 2022. Batya Kemper, J.D. program, NYU School of Law.

TECHNOLOGY & HUMAN RIGHTS

“Killing two birds with one stone?” The Cashless COVID Welfare Payments Aimed at Boosting Consumption

In launching its COVID-19 relief payments scheme, the South Korean government had two goals: providing a safety net for its citizens and boosting consumption for the economy. It therefore provided cashless payments, issuing credit card points rather than cash. However, this had serious implications for the vulnerable.

In May 2020, South Korea’s government distributed its COVID-19 emergency relief payments to all households through cashless channels. Recipients predominantly received points on credit cards rather than cash transfers. From the outset, the government stated explicitly that this universal transfer scheme had two goals: it was not only intended to mitigate the devastating impacts of the pandemic on people’s livelihoods, but also explicitly aimed at simultaneously boosting consumption in the South Korean economy. Providing cash would not necessarily boost consumption as it could be placed in savings accounts. Therefore, credit card points were offered instead to require recipients to spend the relief. But in trying to “kill two birds with one stone” by promoting consumption through the relief program, the government jeopardized the welfare aim of this program.

Once the payouts began, the government boasted that the delivery of the relief funds was timely and efficient. The relief program had been launched based on business agreements with credit card companies for “rapid and smooth” payment, and indeed, it was true that the card-based channel enabled distribution which was much faster than in other countries. Although “offline” applications for the relief program could be made in-person at banks, the scheme was designed around the submission of applications through credit-card companies’ websites or apps. The relief funds were then deposited onto recipients’ credit card or debit card in the form of points—which were separated from normal credit card points—within two days after applying. In September 2021, during the second round of universal relief payments known as the “COVID-19 Win-Win National Relief Fund,” 90% of expected recipients received their payments within 12 days.

Restricting spending to boost spending

However, paying recipients in credit card points meant restricting their access to cash. While low-income households received the relief fund in cash during the first round of COVID-19 relief, they had to apply for the payment in the second round and could only choose among cashless methods which included credit cards and debit cards. To make matters worse, the policy placed constraints on where points could be used, in the name of encouraging consumption and growing the local economy. The points could only be used in designated places, and could not be used to pay for utility bills, repay a mortgage, nor for online shopping. They could not be transferred to others’ bank accounts or withdrawn as cash. Therefore, recipients had no choice but to use their relief funds in certain local restaurants, markets, or clothing stores, etc. If the points had not been used approximately 3-4 months after disbursement, then they were returned to the national treasury. All of these conditions were the outcome of the fact that the policy specifically aimed at boosting consumption.

Jeopardizing the welfare aim

These restrictions had significant repercussions on people in poverty, in two key ways. First, the relief fund failed to fulfill the right to social protection of vulnerable people at risk. As utility bills, telecommunication fees, and even health insurance fees could not be paid with the points, many were left unable to pay for the things they needed to pay for, while much-needed funds remained effectively stranded on the card. What use is a card meant only for restaurants and shops when one is in arrears on utility bills, health insurance fees, and at risk of electricity supply and health insurance benefits being cut off? Those who needed cash immediately sometimes handed their credit cards to other people to use, and then requested payment back in cash below the value. It was also reported that a number of people bought products at stores where relief fund points could be used, and then sold the products at a lower price on the second-hand online market to obtain cash. Although the government warned that it would crack down on such “illegal transactions,” the demand for cash could not be controlled.

Second, the right to housing of vulnerable populations was not sufficiently protected through this scheme. Homeless persons, who needed the most help, were severely affected because the cashless relief funds could not function as a payment method for monthly rent. Homeless people and slice-room dwellers were the group which most strongly agreed that “the COVID-19 relief fund should be distributed in cash” in a survey. Further, given that low-income people spent a higher proportion of their income on rent than those from other social classes, the fact that the relief funds could not be used on rent also significantly affected low-income households. A number of temporary or informal workers who lost their jobs due to the pandemic were on the verge of being pushed into poorer conditions because they could not afford their rent. The relief program could not help these groups cover some of their most urgent expenditures—housing costs—at all.

Boosting consumption can be expected as an indirect effect of government relief funds, but it must not be adopted as a specific goal of such programs. Attempting to achieve this consumption-oriented goal through the relief payments resulted in the scheme’s design imposing limitations on the use of funds, thereby undermining the scheme’s ability to help those in the most extreme need. As the government set boosting consumption as one of the aims of the program and seemingly prioritized it over the welfare aim, the delivery of the payments was devised in an inappropriate way that did not take the most vulnerable into account.

Killing two birds with one stone?

The Korea Development Institute (KDI) found that only about 30% of the first emergency relief funds led to an increase in consumption, while the remaining 70% led to household debt repayment or savings. In the end, it seemed that the cashless relief stipend did not successfully increase consumption, all while it caused the weakening of its social security function.
Such schemes aimed at “killing two birds with one stone” were doomed to fail from the beginning because these two goals come into tension with one another in the program’s design. The consumption aim is likely to harm the welfare aim through pushing for cashless, controlled, and restricted use. The sole purpose of emergency relief funds in a crisis should be to provide assistance for the most vulnerable. Such schemes should be delivered in a way that will best fulfill this aim, they should be focused on providing a safety net, and should be designed from the perspective of right-holders, and not of consumers.

April 19, 2022. Bo Eun Kwon, LLM program, NYU School of Law whose interests include international human rights law, economic and social rights, and digital governance. She has worked at the National Human Rights Commission of Korea.

TECHNOLOGY & HUMAN RIGHTS

Experimental automation in the UK immigration system

The UK government is experimenting with automated immigration systems. The promised benefits of automation are inevitably attractive, but these experiments routinely expose people—including some of the most vulnerable—to unacceptable risks of harm.

In April 2019, The Guardian reported that couples accused of sham marriages were increasingly being subjected to invasive investigations by the Home Office, the UK government body responsible for immigration policy. Couples reported having their wedding ceremonies interrupted to be quizzed about their sex life, being told they were not in a genuine relationship because they were wearing pajamas in bed, and being present while their intimate photos were shared between officials.

The official tactics reported are worrying enough, but it has since come to light through the efforts of a legal charity (the Public Law Project) and investigative journalists that an automated system is largely determining who gets investigated in the first place. An algorithm, hidden from public view, is sorting couples into “pass” and “fail” categories, based on eight unknown criteria.
Couples who “fail” this covert algorithmic test are subjected to intrusive investigations. They must attend an interview and hand over extensive evidence about their relationship, a process which has been described as “insulting” and “grueling.” These investigations can also prevent couples from getting married altogether. If the Home Office decides that a couple has failed to “comply” with an investigation—even if they are in a genuine relationship—the couple is denied a marriage certificate and forced to start the process all over again. One couple was reportedly ruled non-compliant for failing to provide six months of bank statements for an account that had only been open for four months. This makes it difficult for people to plan their weddings and their lives. And the investigation can lead to other immigration enforcement actions, such as visa cancellation, detention, and deportation. In one case, a sham marriage dawn raid led to a man being detained for four months, until the Home Office finally accepted that his relationship was genuine.

We know little about how this automated system operates in practice or its effectiveness in detecting sham marriages. The Home Office refuses to disclose or otherwise explain the eight criteria at the center of the system. There is a real risk that the system is racially discriminatory, however. The criteria were derived from historical data, which may well be skewed against certain nationalities. The Home Office’s own analysis shows that some nationalities, including Bulgarian, Greek, Romanian and Albanian people, receive “fail” ratings more frequently than others.

The sham marriages algorithm is, in many respects, a typical case of the deployment of automation in the UK immigration system. It is not difficult to understand why officials are seeking to automate immigration decision-making. Administering immigration policy is a tough job. Officials are often inexperienced and under pressure to process large volumes of decisions. Each decision will have profound effects for those subjected to it. This is not helped by the dense complexity of, and frequent changes in, immigration law and policy, which can bamboozle even the most hardened administrative lawyer. All of this, of course, takes place in an environment where migration remains one of the most vexed issues on the political agenda. Automation’s promised benefits of greater efficiency, lower costs, and increased consistency are, from the government’s perspective, inevitably attractive.

But in reality, a familiar pattern of risky experimentation and failure is already emerging. It begins with the Home Office deploying a novel automated system with the goal of cheaper, quicker, and more accurate decision-making. There is often little evidence to support the system’s effectiveness in delivering those goals and scant consideration of the risks of harm. Such systems are generally intended to benefit the government or the general, non-migrant population, rather than the people subject to them. When the system goes wrong and harms individuals, the Home Office fails to take adequate steps to address those harms. The justice system—with its principles and procedures developed in response to more traditional forms of public administration—is left to muddle through in trying to provide some form of redress. That redress, even where best efforts are made, is often unsatisfactory.

This is the story we seek to tell in our new book, Experiments in Automating Immigration Systems, through an exploration of three automated immigration systems in the UK: a voice recognition system used to detect fraud in English language testing; an algorithm for identifying “risky” visa applications; and automated decision-making in the process for EU citizens to apply to remain in the UK after Brexit. It is, at its core, a story of risky bureaucratic experimentation that routinely exposes people, including some of the most vulnerable, to unacceptable risks of harm. For example, some of the students caught up in the English language testing scandal were detained and deported, while others had to abandon their studies and fight for years through the courts to prove their innocence. While we focus on the UK experience, this story will no doubt be increasingly familiar in many countries around the world.

It is important to remember, however, that this story is just beginning. While it would be naïve to think that the tensions in public administration can ever be wholly overcome, the government must strive to reap the benefits of automation for all of society, in a way that is sensitive to and mitigates the attendant risks of injustice. That work is, of course, best led by the government itself.

But the collective work of journalists, charities, NGOs, lawyers, researchers, and others will continue to play a crucial role in ensuring, as far as possible, that automated administration is just and fair.

March 14, 2022. Joe Tomlinson and Jack Maxwell.
Dr. Joe Tomlinson is a Senior Lecturer in Public Law at the University of York.
Jack Maxwell is a barrister at the Victorian Bar.