AI is rocking the world of policing -- and the consequences are still unclear.

British police are poised to go live with a predictive artificial intelligence system that will help officers assess the risk of suspects re-offending.

It's not Minority Report(yet) but certainly sounds scary. Just like the evil AIs in the movies, this tool has an acronym: HART, which stands for Harm Assessment Risk Tool, and it's going live in Durham after a long trial.

The system, which classifies suspects at a low, medium, or high risk of committing a future offence, was tested in 2013 using data that Durham police gathered from 2008 to 2012.

Its results are mixed.

Forecasts that a suspect was low risk turned out to be accurate 98 percent of the time, while forecasts that they were high risk were accurate 88 percent of the time.

That's because the tool was designed to be very, very cautious and is likely to assign someone as medium or high risk to avoid releasing suspects who may commit a crime.

A self-learning system

According to Sheena Urwin, head of criminal justice at Durham Constabulary, during the testing HART didn't impact officers' decisions and, when live, it will "support officers' decision making" rather than define it.

Urwin also explained to the BBCthat suspects with no offending history would be less likely to be classed as high risk -- unless they were arrested for serious crimes.

Police could use HART to decide whether to keep a suspect in custody for more time, release them on bail before charge or whether to remand them in custody.

However, privacy and advocacy groups have expressed fears that the algorithm could replicate and amplify inherent biases around race, class, or gender.

"This can be hard to detect, particularly in self-learning systems, which carry greater risks," Jim Killock, Executive Director of Open Rights Group, told Mashable.

“While this process is reported to be 'advisory', there could be a tendency for officers to trust the machine on the assumption that it is neutral."

Racial bias?

Whenever the data is systematically biased, outcomes can be discriminatory because learning models bring into the foreground assumptions that have been tacitly made by humans.

The Durham system includes data such as postcode and gender which go beyond a suspect's offending history.

Even though the system is very accurate, let's say 88 percent of the time, a "subset of the population can still have a much higher chance of being misclassified," Frederike Kaltheuner, policy officer for Privacy International, told Mashable.

For instance, if minorities are more likely to be put in the wrong basket, a system that is accurate on paper can still be racially biased.

"It's important to stress that accuracy and fairness are not necessarily the same thing," Kaltheuner said.

Last year, an investigation by U.S. news site ProPublicashone a light on the alleged racial bias of an algorithm used by law enforcement to forecast the likelihood of a repeated offense.

Among other things, the algorithm was making overly negative predictions about black versus white suspects. The firm behind the system denied the allegations.

For example, ProPublicareports the cases of Brisha Borden, a black 18-year-old teenager who stole a child’s bicycle and scooter, and Vernon Prater, a white 41-year-old who was picked up for shoplifting $86.35 worth of tools.

While Prater was a seasoned criminal, having already been convicted of armed robbery and attempted armed robbery and Borden had just a handful of misdemeanours, something odd happened when they were arrested and charged.

A computer algorithm predicting the likelihood of each committing a future crime gave Borden a 8 (high risk) while Prater received a low-risk score, just 3.

Two years later, exactly the opposite happened: Prater was serving an eight-year prison term for breaking into a warehouse and stealing thousands of dollars’ worth of electronics while Borden had not received any new charges.

It's only advisory

HART authors, Durham police, and the University of Cambridge's centre for evidence-based policing, defended the system in a submission to a parliamentary inquiry.

"Simply residing in a given postcode has no direct impact on the result, but must instead be combined with all of the other predictors in thousands of different ways before a final forecasted conclusion is reached."

They also noted that the model is just "advisory."

However, advocacy groups believe there are questions that need to be assessed before the system is used to make life-changing decisions for individuals.

"The lack of transparency around this system is disturbing," Silkie Carlo, policy officer at Liberty, said.

“If the police want to maintain public trust as technology develops, they need to be up front about exactly how it works.

"With people’s rights and safety at stake, Durham Police must open up about exactly what data this AI is using.”