Like something straight out of science fiction, China is turning to artificial intelligence (AI) to help run its judicial system, putting humans at the mercy of machines.

China has been working on updating its court system since 2016 when each court was effectively siloed from all others. The government forced courts to digitalize and feed their data into a central system, giving an AI the ability to analyze and learn from 100,000 court cases a day.

Beijing’s Supreme Court now believes the AI is ready to take the next step, informing the country’s courts in an update that the AI must be consulted on every case going forward. If a judge disagrees with the AI’s assessment, the judge must submit their dissension in writing, according to South China Morning Post.

Critics say judges are deferring to the AI — even if its recommendation is based on less suitable case law or references — to avoid the hassle of challenging it.

“It is too early to sell the smart court system as a panacea,” said Sun Yubao, a judge with the People’s Court of Yangzhou Economic and Technological Development Zone in Jiangsu province.

“We need to reduce the public’s high expectation of artificial intelligence and defend the role of a judge. AI cannot do everything,” he wrote in a paper in Legality Vision.

His sentiments were echoed by Zhang Linghan, professor of law at the China University of Political Science and Law in Beijing. Warning that “humans will gradually lose free will with an increasing dependency on technology,” Zhang expressed concern about the possibility of humans being subject to machines and AIs.

Those concerns seem borne out by the fact that an AI prosecutor is already charging people with crimes in big cities based on its evaluation of evidence.

The other elephant in the room is the reliance on Big Tech to make the whole system work. China’s lawmakers have a complicated history with tech companies and their executives, yet the entire AI system puts a tremendous amount of power in the hands of those programming the AI.

Only time will tell if the system lives up to the expectation. In the meantime, it still sounds like something out of a science fiction move — and not one that ends well for humanity.