[LINK] It Was Supposed to Detect Fraud. It Wrongfully Accused Thousands Instead.

[Bernard Robertson-Dunn]

[If you think this is about RoboDebt, read on......

It Was Supposed to Detect Fraud. It Wrongfully Accused Thousands Instead.
https://www.theatlantic.com/technology/archive/2020/06/michigan-unemployment-fraud-automation/612721/
How Michigan’s attempt to automate its unemployment system went horribly
wrong
Stephanie Wykstra and Undark
8:00 AM ET

In 2014, Carmelita Colvin was living just north of Detroit and taking
classes at a local college when she received a letter from the Michigan
Unemployment Insurance Agency. The letter stated that she’d committed
unemployment fraud and that she owed more than $13,000 in repayment of
benefits and fines.

Colvin’s reaction, she recalled, was: “This has got to be impossible. I
just don’t believe it.” She’d collected unemployment benefits in 2013
after the cleaning company she worked for let her go, but she’d been
eligible. She couldn’t figure out why she was being charged with fraud.

What Colvin didn’t realize at the time was that thousands of others
across the state were experiencing the same thing. The agency had
introduced a new computer program—the Michigan Integrated Data Automated
System, or MiDAS—to not only detect fraud, but automatically charge
people with misrepresentation and demand repayment. Although the agency
still hasn’t publicly released details about the algorithm,
class-actions lawsuits allege that the system searched unemployment data
sets and used flawed assumptions to flag people for fraud, such as
deferring to an employer who said an employee had quit—and was thus
ineligible for benefits—when that person had actually been laid off.

Over a two-year period, the agency charged about 40,000 people, billing
them for about five times the original benefits, which included
repayment and fines of 400 percent plus interest. Amid later outcry, the
agency ran a partial audit and admitted that 93 percent of the changes
had been erroneous—yet the agency had already taken millions from people
and failed to repay them for years. So far, the agency has made no
public statements explaining what, exactly, went wrong. (Lynda Robinson,
an agency representative, declined Undark’s interview request by email,
writing: “We cannot comment due to pending litigation.”)

Government use of automated systems is on the rise in many domains, from
criminal justice and health care to teacher evaluation and job
recruitment. But the people who use the algorithms don’t always
understand how they work, and the functions are even murkier to the public.

“These types of tools can be used to inform human judgment, but they
should never be replacing human beings,” said Frank Pasquale, a law
professor at the University of Maryland who studies accountability in
the use of these opaque algorithms. One of the big dangers, he said, is
that the systems fail to give people due-process rights. If an algorithm
is “used by the government, there should be full transparency”—both in
how the software works and in the data it uses—“to the people who are
affected, at the very least.”

In cases like Michigan’s, flawed automated systems punish people the
agencies are supposed to help, said Michele Gilman, a University of
Baltimore law professor who directs a legal clinic that represents
clients with public-benefits cases. She pointed to other examples,
including algorithms adopted in states such as Arkansas and Idaho that
agencies used to cut Medicaid benefits, sometimes erroneously. And the
issues extend beyond the United States: A Dutch court recently found
that an algorithm used to detect welfare fraud violated human rights and
ordered the government to stop using it.

In Michigan, while the agency says it repaid $21 million, attorneys in
the class-action lawsuits argue that this doesn’t account for all the
damages. People like Colvin suffered long-term harm—many came out with
damaged credit and lost job opportunities and homes. More than 1,000
filed for bankruptcy.

Roughly half the states in the U.S. have modernized the software and
information-technology infrastructure for their unemployment-insurance
systems. In many cases, these updates are crucial to keeping the systems
running smoothly, and many help claimants more easily file for benefits.
But this is not always the case—in Florida, for example, the new system
adopted about six years ago made it much harder for people to apply.
Gilman and other researchers whom Undark spoke with are also concerned
that in the coming years, states may adopt algorithms that lead to
similar problems as in Michigan—particularly if they cut human review of
fraud charges, in violation of federal due-process requirements. Amid an
unprecedented surge in unemployment claims during the COVID-19
pandemic—more than 40 million Americans have filed since mid-March—the
problems could be amplified.

The automated system in Michigan is “a case study in all the ways an
algorithm can go wrong,” Gilman added. “The citizens shouldn’t be the
guinea pigs in testing whether the systems work.”

Michigan’s automated system popped up just after the Great Recession,
when the auto and other manufacturing industries were hit hard and
workers applied for unemployment benefits at high rates. Such benefits
are intended to help those who have lost a job through no fault of their
own, and are funded by federal and state payroll taxes. Benefits
typically last 26 weeks, though the federal government covers extended
benefits during economic downturns, including today’s COVID-19 crisis.

In March 2011, Rick Snyder, the newly elected Republican governor of
Michigan, signed a bill shortening state unemployment benefits from 26
to 20 weeks. The bill also allocated funding for software to detect
unemployment fraud.

The state’s unemployment agency hired three private companies to develop
MiDAS, as well as additional software. The new system was intended to
replace one that was 30 years old and to consolidate data and functions
that had previously been spread over several platforms, according to the
agency’s self-nomination for an award from the National Association of
State Chief Information Officers. The contract to build the system was
for more than $47 million.

At the same time as the update, the agency laid off hundreds of
employees who had previously investigated fraud claims.

While MiDAS’s exact process isn’t public, interviews with attorneys and
court documents from class-action lawsuits provide some context. Part of
the system searched records of employers and claimants in its database,
then flagged people for potential unemployment fraud. Next, MiDAS sent
questionnaires to an electronic mailbox on the benefits website that
recipients may not have had reason to monitor, gave them 10 days to
respond, and then sent a letter informing them they had been charged
with fraud. After a 30-day appeal period, the system began garnishing
wages and tax refunds. The agency later acknowledged that in the
majority of the cases from 2013 to 2015, the system ran from start to
finish without any human review.

Using automated decision-making systems to detect fraud is a problem
because the stakes are so high, said Julia Simon-Mishel, an attorney at
Philadelphia Legal Assistance who has served on the advisory committee
for Pennsylvania’s unemployment-benefit modernization for the past two
years. The automated systems don’t always gather key information or
allow for “a back-and-forth that is required by federal guidelines, in
terms of the conversations that states need to have with an employer and
a worker,” she said.

“If you’re just using analytics to automatically flag fraud without
anything else, you have a garbage in, garbage out problem,” she added.
“Who knows whether the data you’re using to build your predictive
analytics is correct?”

Some of the bad data come from simple, unintentional application errors.
Many public-benefit applications are complicated, said Gilman, who
called them “the tax code for poor people.” Because of this complexity,
the majority of overpayments are due to either the employee or the
employer making a mistake on a form, Gilman said. But, she added, “a
mistake is not intentional fraud.”

In October 2013, MiDAS began flagging people for fraud. Soon after,
lawyers across the state were deluged with calls from people who were
bewildered by their fraud charges. Those who couldn’t afford a lawyer
faced another hurdle: At the time, Michigan did not extend the right to
free legal representation to those charged with unemployment fraud. (The
state extended this right through legislation in 2017.) Some turned to
nonprofits, including the Unemployment Insurance Clinic at the
University of Michigan Law School (renamed the Workers’ Rights Clinic in
2019) and the Maurice and Jane Sugar Law Center for Economic and Social
Justice, a legal-advocacy nonprofit in Detroit.

Anthony Paris, an attorney at the Sugar Law Center, told Undark that he
and his colleagues assisted hundreds of clients who were accused of
fraud from 2013 to 2015, including Colvin. Over time, the legal team
pieced together an understanding of the source of the surge of fraud
charges: The new computer system had searched the unemployment database
going back six years and flagged people for fraud based on error-prone
rules.

One rule led the agency to incorrectly accuse people of working while
claiming unemployment benefits. Here, the system assumed, as described
in a federal class-action lawsuit filed in 2015 by the Sugar Law Center
and others, that the reported income from a part of the year had been
earned over a longer period. For example, a plaintiff named Kevin Grifka
had reported $9,407.13 in earnings in 2015 before he was laid off in
early February and claimed unemployment benefits. The agency later
claimed that he had made $723.62 each week throughout the first quarter
of the year—including the period when he said he was unemployed. As the
lawsuit describes, this was based on dividing his income before being
laid off by the 13 weeks of the whole quarter; the same erroneous
calculation was used to charge other plaintiffs.

The system also automatically accused people of intentional fraud when
their story didn’t match that of their employer. Rather than assign a
staff member to investigate discrepancies between what the employer and
the employee reported, Paris said, the system was programmed “to
automatically assume that the employer was right.”

This is what led to Colvin’s fraud charge, according to court documents
reviewed by Undark. Colvin’s employer reported that she had quit. Colvin
said that her employer had told her she was suspended from work due to
an issue reported at a house that was cleaned by her and another
employee. Colvin never heard back and assumed she was laid off. Late in
2019, when Paris represented Colvin at a court hearing, her employer was
called in to testify. The employer acknowledged that she’d assumed that
Colvin had quit based on secondhand accounts from two other workers.

Colvin’s case highlighted another problem with MiDAS: It was designed
with insufficient attention to the right to due process. It’s a common
problem in automated systems, said Jason Schultz, a law professor at NYU
and a legal and policy researcher for the school’s AI Now Institute.
“Even though the systems claim to notify people, they typically only
notify them of the end result,” he added. “They don’t usually give them
any information about how to actually understand what happened, why a
decision was made, what the evidence is against them, what the rationale
was, what the criteria were, and how to fix things if they’re wrong.”

Although MiDAS had several steps for notification, the forms were
confusing, leading some innocent people to self-incriminate, according
to Steve Gray—then the director of the University of Michigan’s
clinic—who described the problem in a co-authored letter to the U.S.
Department of Labor. Moreover, many people never saw the messages. The
first one, the questionnaire, appeared in an online account that most
people checked only when they were receiving benefits. And by the time
the second notification, by letter, came around, many people had moved
to a new address, according to a 2016 report from Michigan’s auditor
general.

Brian Russell, an electrician from Zealand, Michigan, learned about his
alleged fraud only after the state took $11,000 from his 2015 tax
refund. He visited a state unemployment-agency office and was astounded
to learn that the government would take even more: “They were charging
me $22,000, and I had no idea what was going on.”

Russell said he tried everything he could think of to figure out why he
was being charged—he visited the nearest unemployment office and called
the agency “probably hundreds of times,” but the phone system was
overloaded. He’d wait for hours on hold. (The state auditor’s report
found that of the more than 260,000 calls made to the line during
business hours in a one-month period in 2014, about 90 percent went
unanswered.) Once Russell finally got a person on the line, he said he
was told he’d missed the 30-day window to appeal.

The University of Michigan’s law clinic eventually appealed Russell’s
case to an administrative-law judge, who dismissed Russell’s charge. The
agency refunded most of the money. (Russell said he is still owed about
$1,500.) But in the meantime, he said, he had to declare bankruptcy,
which affected his credit score. And because he didn’t have the money
for basic needs, he had to forgo some of his diabetes treatments and
move into a friend’s basement.

While Michigan is a striking case of an automated system going wrong,
it’s not the only area in which governments are using
artificial-intelligence systems for decision making. Such systems are
used in criminal justice, policing, child protective services, the
allocation of health-care benefits, teacher evaluations, and more. In
many cases, governments are hiring private companies to build these
systems, Frank Pasquale said, and because the companies want to protect
their intellectual property, they sometimes hide details of how the
products work. “The trade-secret protection is a real problem, because
you can’t even understand what the problems are,” he said.

Over the past few years, policy makers, academics, and
advocates—including at organizations such as the AI Now Institute at NYU
and Georgetown Law’s Center on Privacy and Technology—have explored ways
that governments could build better, more accountable automated systems.

Agencies also need to understand the limitations of a particular system
before they implement it, said Julia Stoyanovich, an assistant professor
of computer science at NYU. “My belief is that folks in government just
think that AI is magic somehow,” she said, which can lead to big
problems when the automated systems don’t run smoothly. Jason Schultz
agreed, adding that when government employees don’t have the background
or resources to evaluate claims made by private companies, “they’ll just
get sold snake oil.”

Advocates and researchers are also pushing for governments to solicit
guidance before implementing a new system, both from experts and from
people who will be targeted by the new tools. In 2018, the AI Now
Institute published a framework for this process, which Schultz
co-authored. The idea is that governments need to think through steps
for algorithms before they deploy them, Schultz said, asking questions
like: “Does this system actually do what it says it’s going to do? Is it
going to impact society in a way that actually helps and doesn’t hurt?”
The authors also suggested that the assessment should be published and
open for public comment.

Another model for accountability is designating an oversight body to
audit algorithms. Early this year, attorneys in the Michigan
class-action lawsuits called for the creation of a task force to oversee
all algorithms and automated systems implemented by the state
government. Other states and cities have formed similar task forces.
Stoyanovich, who served on one formed in New York City, said she thought
it was an important step. But it was also widely criticized—researchers
from the AI Now Institute and other organizations, for instance,
critiqued the effort for not adequately involving the wider community.
And while oversight bodies could be important, Pasquale said, they need
to have “real teeth” in order to yield meaningful enforcement.

But one of the biggest questions is whether and when to use algorithms
to begin with—particularly for tools that could quickly erode privacy
and civil liberties, such as facial recognition. “I don’t think any
government has to adopt any of these systems,” Schultz said. “I think
spending millions of dollars on an unproven system that has no
accountability and no way of ensuring it works is a really, really bad
idea when it’s not required.”

In 2016—in the midst of a flood of negative media
attention—Representative Sander Levin of Michigan pushed the agency to
conduct an internal audit of the approximately 60,000 fraud charges in
the prior two years. The agency responded with a partial audit,
reviewing 22,000 of the MiDAS fraud charges and determining that nearly
all were wrong. (The agency also reviewed another set of fraud charges
made with some level of human review; while the error rate was still
high, at 44 percent, the automated system had more than twice that rate.)

By 2017, as part of a settlement of a class-action lawsuit, the agency
agreed that it wouldn’t use the automated system without human review
(and clarified that it had actually stopped the practice in late 2015).
The agency also agreed to review and overturn the remaining erroneous
charges that it had issued from 2013 to 2015, later reporting that it
had reversed about two-thirds of the fraud determinations and had repaid
$21 million.

Also in 2017, state legislators passed a law to prevent such a disaster
from happening again through improvements in fraud-charge notifications
and reduced fines. And in 2019, in a move that several attorneys
described to Undark as a step in a positive direction, Michigan Governor
Gretchen Whitmer appointed Gray, formerly the head of the
unemployment-insurance law clinic at the University of Michigan Law
School, as the director of the agency.

In available court documents, which include depositions of agency staff
and the three companies, none of the people involved in making or
approving the automated system admit responsibility (as plaintiffs’
lawyers confirmed). While the ongoing cases should eventually reveal
more details—including discovery involving more than 1 million documents
such as internal agency emails—the suits will take a long time to work
their way through the courts, Paris said.

Meanwhile, for some people who were falsely accused of fraud, the
struggle continues. In addition to the class-action lawsuits, in March
2020, the current director of the University of Michigan Law School’s
Workers’ Rights Clinic testified to the state Senate Oversight Committee
that she believes close to 20,000 of those charged by MiDAS are still
being actively pursued and having their wages garnished.

For Colvin, it wasn’t until early this year, nearly six years after she
had been charged, that an administrative judge finally dismissed her
case. In the middle of that stretch, in 2017, Colvin completed her
bachelor’s degree in criminal justice, adding to a previous associate’s
degree in forensic photography, and applied for a job with a county
sheriff’s office. She soon got a call letting her know that she wouldn’t
be hired, because of the fraud charge and her outstanding debt to the state.

Today, Colvin said she is working as a security guard to support herself
and her daughter. She is 32, and wonders if it was a waste of time to
get two degrees in criminal justice.

“I can’t get the job I wanted,” she said, “because they suggested that I
was a criminal.”

-- 

Regards
brd

Bernard Robertson-Dunn
Canberra Australia
email: brd at iimetro.com.au