By Stephen F DeAngelis
A machine now takes actions and renders judgment,
In places its creators never step,
On timetables not of their choosing,
About people they will never know,
At speeds no one can follow.
Label we still reach for is simply tool.
A tool waits to be used. It remains inert until lifted, and falls silent the instant it is set aside. Whatever it achieves unfolds under the direct gaze of its user.
That is not what is being deployed now.
The reassurance embedded in that old word, that human intent governs every outcome, was earned by ten thousand years of tools behaving exactly so. It does not apply to an instrument that continues to act in its maker’s absence. The burden of accountability does not transfer when the machine begins operating on its own. It remains with those who made it.
The maker must now design responsibility that can confidently enter situations not yet foreseen, across timescales they can’t anticipate, and account for every person or system involved with that unfolding. That may sound implausible, a sort of leave-no-outcome-behind ambition, but the answer that travels routinely changes the balances of lives and systems the maker cannot know at the making.
Pax hominibus bonae voluntatis, peace to those of good will, as the angelic hymn puts it in Luke 2. 1 That is the peace owed by any maker of good will to the world into which they release their work. And it is precisely that peace now put at risk when what they send forth operates at tempos no human can match.
I. The Instrument
A scoring model changes a person’s access to credit, employment, or housing before anyone responsible can review the case, then moves on at a speed no oversight can match. By the time an appeal resolves, the thing the decision took is gone, the human kept in the loop has ratified a judgment they could not remake, and the successor who inherits the model cannot read the warrant the first maker signed. Every official safeguard can be present, and the human capacity to answer already gone. That is the point at which compliance passes and answerability fails.
When such a system fails at three in the morning in a city he has never visited, acting on a person whose name he will never know, the world still looks to the practitioner who set the calibration. He cannot be there. He was never going to be there. Standing behind the work in person was never on offer. What is on offer is whether he built the answer to travel without him, and everything the moment now requires begins there.
The question is an old one. Long before any machine could act on its own, the engineer who signed the bridge and the officer who signed the loan book owed the same debt, a way of standing behind work that would reach the people it acted on after the maker had walked away. What has changed is only that the answer can no longer be delivered in person, because the instrument now moves faster and farther than the maker can follow. So the history is where the answer begins.
The world these twenty-first century instruments enter is brittle where shock absorbers have thinned, anxious where publics have withdrawn consent, non-linear where causes no longer reward patient analysis, and incomprehensible at any scale a capable mind can hold.2 Someone who has spent thirty years building decision systems learns to read the climate before the storm. Banks that used to warehouse shocks now pass them through. Publics that once gave firms the benefit of the doubt now start from suspicion. The causal chains a careful analyst once traced from one shift to its consequences elsewhere now grow too many and too fast to follow. Into that world we now send machines that keep acting, at machine speed, after the human leaves, under an old name that no longer fits what they do.
The older word is tool, and its authority rests on a presumption the long history of tools had earned, that the thing works under a hand and answers to it. A hammer never strikes a nail its owner did not choose. A ledger never posts an entry its keeper did not enter. Even the most elaborate machines of the industrial age, the loom and the press and the turbine, typically ran only while someone watched, slowed when that person walked away, and produced consequences at a pace a worker could still follow.
An autonomous decision system that allocates capital, reads medical imagery, or broadcasts a claim to a hundred million feeds is none of these. It moves people who did not know it was there, in back rooms the makers left long ago, and it acts with only the discipline they chose to build into it and no more. The creator’s hand and the place it acts are now two different realities. Calling these instruments tools smuggles in a comfort they no longer deserve, the comfort that people remain in charge, that nothing happens they did not intend, that responsibility is discharged by careful use in the moment of use.
Tool has to go. It promises a level of control no longer present and licenses a level of inattention the new instruments cannot survive. What we are shipping now are instruments in the full weight of that term. They are made things that carry their maker’s disposition into the world and act on it when the maker is gone.
The obligation does not travel with them. When the system starts to act on its own, the temptation is to let the obligation move with it, to imagine that if the instrument decides, the instrument somehow answers. It does not. A machine cannot be answerable. It carries whatever its makers built into it. If what was built into it is an answer, it carries the answer. If not, if the maker shipped the capability and withheld the account, it carries nothing worth trusting. The accountable parties are still the makers. What changes is the form of the payment, because the work now reaches places where they cannot stand beside it.
Those four conditions are the water these instruments enter. Buffers are gone, publics start with suspicion, interactions overwhelm linear cause, and the scale has outrun what one mind can hold. An answer built for a calmer world does not survive that sea.
The machine is not a person in disguise, and it has no will beyond what its maker gave it. It has a disposition, a set of readings it will produce and actions it will recommend on the people it moves, all set upstream by choices that could have been made otherwise.
Rivers have dispositions too, cut by the shape of their beds, and we do not blame the water for the flood. We look to the levees and the people who built them. The difference here is that the instrument’s bed was cut on purpose, by a hand that knew what flows do, and that is what makes the hand answerable. Giving the machine standing as a moral party erases the makers in the machinery, dissolving the only accountable agents into the machine they built. What the machine does and what the maker answers for are two different things, and the discipline this moment requires begins with holding them apart.
So the burden returns to the practitioner who set the calibration. He cannot stand behind that work in person, and the machine has made that impossible in a way no earlier instrument did. What he can do is build the answer to travel without him.
II. The Design Property
Sometimes code is all about intent, and at other times it simply expresses architecture. Here, the goal is a design property the maker must answer for, not a moral claim about the machine, and one particular afternoon shows what the property looks like when it holds.
At a desk in Princeton, late in a long committee meeting, a recommendation on the screen proposes reorganizing one of the firm’s analytic functions. The senior executive responsible for that function had been in the room the whole time and had said little. When he finally spoke, he said quietly that if the recommendation was right, he no longer understood what his job was. He was confessing, not protesting. His underlying fear, and excitement, was simple enough. Machines combining decisions from across silos his expertise instinctively kept separate. Accepting that logic meant that his standing was evolving, and his thought was likely, how do I manage that change.3
What sat on the screen was the output of a constraint-based optimization that had worked through more than thirty constraints and past a hundred and eighty million possible combinations, resolved in thirty to forty minutes and re-simulated in thirty to fifty seconds, at a resolution and precision that had not been reachable before. That was what he was reacting to, an answer produced at a depth a team of ten would have needed two to three weeks to reach.
The disturbance was structural. The relevant executives had built their function the way a competent operator does, by splitting the work so different people hold different pieces and no one person carries too much. The org chart was the discipline that let the firm hold more than any single mind could hold, by keeping those pieces in separate hands.
The recommendation cut across that design. It pulled together the risk signal that one control function was meant to keep apart from the commercial signal, the multi-year exposure parked in another quarter, and the customer behavior buried in segments no one was supposed to read as a whole, and treated them as one picture. The room had not known it needed that picture, because the room had been built on the premise that no one would ever hold those pieces at once. Now something did. And the question that arrived with the recommendation, the one that quietly emptied that responsible executive’s chair of the standing it had held that morning, was “Who in the room was now answerable for a reading no one there had ever been meant to hold?”
Everything turned on what the instrument handed our newly responsible executive. It could have delivered a closed verdict, a number with an order attached and nothing he could see or contest, turning him into a leader who merely ratifies what he cannot inspect. It did not. It handed him a reading he could see and argue, with time to answer before the recommendation outran him and ground sufficient to press his case. Those three things are what let him keep his chair at the table, and they are what the maker’s answer has to become once the maker is no longer in the room to supply them in person.
A reading no one downstream can take in is just a verdict dressed as explanation, so it has to be legible at the resolution the world can actually hold, not at whatever resolution the machine finds convenient. Someone who was not there when the warrant was built still has to press on the ground the decision stands on, so it has to be traceable to something the world can test. And a reading that arrives faster than anyone can answer has already decided, whatever it pretends to offer, so its pace has to match the speed at which the world can respond.
Coherent knowledge starts with a clear reading of the situation, rests on a warrant tying facts to conclusions, and depends on an accurate pacing. All that must now be coded into the instrument itself. In the past, makers held such understanding close to the vest, as it was an expression of power and authority within the room. Now they have to build that discipline into a thing that will stand in rooms they never enter, across time scales they cannot fully anticipate.
Accepting each goal is easy, achieving them is hard.
Readability at the resolution the world can hold is not the same thing as a system that can, in principle, be explained. Almost anything can be explained to someone with a month and the right training. The resolution that matters is the one available to the person who has to act on the reading, in the time and with the understanding actually present in the room. A reading pitched above that resolution is opaque, however complete it may look on paper.
Warrant traceable to a form the world can test is different from documentation. A firm can produce a thousand pages no one downstream can use to reconstruct why the decision came out as it did, because the pages record what was done and not the ground it stood on, and warrant lives in the ground.
Pace aligned with the speed at which the world can respond is the one most often lost, because pace is invisible until it is too late. A reading that is perfectly legible and fully warranted, delivered a beat faster than the person can take it in and answer, has already decided and left the person to sign.
The three stand or fail together. A system that is readable but not testable is only a persuasive story no one can check, and make it testable but leave it unpaced and it becomes a checkable decision that has already happened. The maker’s answer has to be all three at once, built to hold together in the same instrument at the scale that instrument can run.
The instrument carries the room’s discipline into a world the maker cannot follow it into. That is what “design property” means in practice. When an executive keeps their chair, the machine was not being kind to them. Someone, upstream, at a desk, had decided the instrument would carry that discipline out with it.
It carries the maker’s willingness to answer, or it carries an absence dressed as answer. If a machine refuses to be read it loses any claim to neutrality and ends up dancing with the ones that brung it. With such pre-programmed responses, the machine stops listening to the world it inhabits.
III. The Line Across Two Thousand Years
The maker’s obligation to build an answer that travels is an old one. The engineer who signs the bridge, the actuary who signs the table, the officer who signs the loan book all owe a way of standing behind their work that reaches the person it acts on after they have walked away.
What has changed is that makers can no longer stand behind their work in person and take the question, because the instrument now acts faster and farther than any human process can follow. What once was delivered in the flesh has to be built into the design instead. The debt is still owed. Only the coin, paid now in the design rather than in person, has changed.
Pax hominibus bonae voluntatis, the hymn in Luke 2 as the Vulgate tradition has it, names a peace granted to people of good will. The older tongue belongs at the head of an essay about trained models because the peace a maker’s work either enters or harms is the same public peace whether the work is a treaty or a model.
The peace meant here is the condition of a shared world in which people can live, act, and answer to one another, not an inward calm, and every consequential thing a maker ships either strengthens that condition or corrodes it. Makers of good will are those who accept responsibility for which of those two their work does. The Latin carries no more than that, and that much is narrow enough to defend.
The practitioner who has spent thirty years shipping consequential systems recognizes this moment rather than discovering it. Many around the table would rather believe the ground is too new to carry precedent, too novel to support accountability. The obligation makers have always owed to work that outlives their hands removes that excuse. The ground is not new.
The twentieth century kept that debt in a form we can still see, and three men carried it in ways that show the obligation with particular clarity. The tradition is wider than any three, and others belong in it. These three earn their place by leaving a public, documented record that can be checked against the account given, each of them showing a different way a maker’s answer either reached the world his work touched, or failed.4
John Maynard Keynes built an answer into the room and watched the room overrule it.
He arrived at Bretton Woods in July of 1944 knowing the conference was theater, not decision. For twenty-five years he had diagnosed what the Versailles settlement had broken and had designed an alternative, an international clearing union with a shared reserve unit (bancor) meant to settle trade imbalances without tying the system to the strongest currency, then the US dollar.5 He knew before crossing the Atlantic that Washington’s design was locked in and his own plan set aside. So he argued for three weeks against a settlement already chosen, knowing the position backed by American gold and allied debt would prevail against his better argument.
The defeat, when it came, was bodily.
On 22 July the delegates signed the Articles of Agreement, and Keynes put his name to a document he had never seen in full, only the dotted line. He later wrote that they all signed before reading a clean copy, their only excuse that the hosts were ready to throw them out of the hotel within hours, “unhouselled, disappointed, unaneled,” Hamlet’s ghost’s language for a man sent off without rites. He reached for that phrasing to describe the signing of the order that would govern world trade for a generation.
As he rose at the closing, the delegates stood and sang for him, giving the architect of the moment a jolly-good-fellow ovation at the exact hour his central design lay discarded. He had won the concessions available, the scarce-currency language, the transitional arrangements, the quota tweaks, and lost the successor institution he had come to build.
Keynes did not walk away from what he had lost. He carried the settlement home and stood in the House of Lords in December of 1945 to defend a system built substantially on Harry Dexter White’s design, not his own. On the loan terms he refused to pretend, saying he would never cease to regret that it was not interest-free, and when accused of restoring the gold-standard orthodoxy he had spent his life attacking, he answered that he would not walk backwards to resurrect idols he had helped throw out of the market place.
Three months later at Savannah, at the inaugural meeting of the Bretton Woods institutions, he spoke of the “twins” (International Monetary Fund, World Bank) and the fairy-tale curse he hoped to lift from them. He was dead within weeks, worn down by crossings his heart could no longer carry. His work entered the post-war peace in a form he had argued against, and being outvoted did not release him from the obligation he had taken on by designing the system. He answered at the table where it was decided and bore the cost of it, staying to defend a settlement he knew was flawed because he had helped set its terms.
Albert Einstein’s case shows another. He sent the answer ahead of the work and could not call it back once the work arrived. In August 1939 he signed a letter to Franklin Roosevelt, one he had read and put his name to but had not drafted, warning that a weapon of great destructive power might be built from nuclear fission and that Germany might build it first.6 He was never brought inside the project that followed, kept at the edge of the effort his signature had set in motion, and he spent the rest of his life answering for the letter from outside the room where the doctrine it loosed was decided.
When the weapon existed and had been used, he sent a telegram in 1946 whose line has outlived him, that the unleashed power of the atom had changed everything save our modes of thinking. Against the doctrine consolidating nuclear weapons into the standing mechanism of American security, he argued that thinking had to change with the weapon. His signature on the Russell-Einstein Manifesto, affixed days before his death in April 1955, was the last public act of a man who had spent sixteen years answering for a letter he signed in an afternoon. The work outran the man, and the answer had to travel without him.7
Oppenheimer shows a third path. He built the answer into the weapon and into the institution meant to hold it, and both were taken from him. As scientific director at Los Alamos he held theory, engineering, and the men who did both in one mind, and he understood before the Trinity test what the object on the tower would mean for the world that survived it. When the war ended he tried to build the alternative to the weapon he had built, helping author the 1946 State Department report that proposed placing the world’s fissile material under international authority so no single state would hold what he had made. Diplomacy carried that proposal into a plan the Soviet Union would not accept, and the effort was not renewed. The alternative had been designed. Getting the world to adopt it was the part that left his hands.
Oppenheimer’s institutional defeat came in the autumn of 1949. Chairing the General Advisory Committee to the Atomic Energy Commission, he signed the October 30 report opposing a crash program for the hydrogen bomb on moral and strategic grounds. The majority annex he signed with five colleagues called it a weapon of genocide, a threat to the future of the human race that should never be produced, and the minority annex by Enrico Fermi and Isodor Rabi stated more starkly still that its very existence would be an evil in any light.
The committee differed only on how sharply to name that moral fact, and it argued that refusing the crash program was itself a strategic choice, not merely a moral gesture. The argument did not prevail. In January 1950 President Truman announced the decision to proceed, framing the bomb as necessary to defend the country against any aggressor, the very framing the committee had tried to block. Oppenheimer’s opposition failed inside the institution he had served.
Then Oppenheimer was made to answer for all of it at the table where it was decided.
From April 12 to May 6, 1954, before a security board convened by the very machinery his weapon had armed, Oppenheimer testified for weeks under oath about his associations and his 1949 opposition to the hydrogen bomb, knowing what it would cost him. On June 29 his clearance was revoked. He spent the rest of his life at Princeton’s Institute for Advanced Study, kept from the councils where the questions he had raised were still being decided. He had built the weapon, tried to build the institution to hold it, opposed the next weapon, and then answered for all of it at a table that turned on him. His standing was the price and he paid it.8
What these three shared was what happened to their work once it left their hands. The instrument each helped build moved from proposal to operating fact, shaping the monetary order, the nuclear order, the security order, on terms they could no longer set. Keynes argued for a different system and then watched the votes lock in a design he knew was flawed. Einstein signed a warning and then spent the rest of his life unable to call back what his signature had helped start, kept outside the project by the security apparatus it empowered. Oppenheimer built the weapon, tried to build the institution to contain it, opposed the next weapon, and then saw the doctrine harden without him before that same machinery was turned on him.
In none of the three does intention govern outcome. The work enters a public order with its own momentum, and its author is left to answer for what that order does with it, at tables he does not control, under settlements he cannot undo. This is how we translate the ancient Latin, tracing its commitment through these three lives to that desk in Princeton. The obligation has always been to build the answer that can travel where its author cannot.
IV. The Adversary That Is Not Human
Today’s maker stands on that same line and faces a similar contest, with one decisive difference. Those three men answered against human adversaries, institutional positions held by specific people who could be named, argued with, and sometimes outlasted. The maker now answers to something that acts before any table can be convened. A human adversary could at least be brought to a table, and this one has already moved by the time the table forms.
The scoring model from the opening is not the adversary, and the technology itself never was. The same kind of model, watching a hospital monitor for the first signs of sepsis and buying a clinician the minutes that save a life, is the same speed and pattern-reading turned toward a human end instead of away from one.
The adversary is careless or malevolent use of that speed toward ends that leave the world less able to answer for itself. The parallel with 1944 stops there. Keynes could argue with a man across a table over days. Today’s practitioner answers for something that has already acted a thousand times before the table sits.
For Keynes, Einstein, and Oppenheimer, a table eventually formed, hostile, slow, often turned against them, but still a place where a man could stand, be questioned, and answer. For today’s practitioner, part of the answer has to live inside the instrument itself, because it acts before oversight can respond. There is no version of standing behind the work, in person, that reaches those already affected. The table comes too late or never at all, and the only defense that arrives in time is the one built into the thing before it goes out.
The two run on the same capacity, to read a pattern faster than a person can and act before any room can convene, saving the life in the hospital by the same property that costs the person the credit or job. The difference is what the speed is pointed at, whether it leaves the world more able to answer for itself, or less.
Treating the danger as a property of the capability instead of the use is to reach for the wrong remedy, slowing the life-saving instrument to restrain the one that harms, when what needs restraining is the disposition the maker built and the end he pointed it at.
The clearest instance of that speed turned against the public peace is the corruption of the citizen’s own judgment.
When misinformation and disinformation are amplified by social media and generative systems that can manufacture and spread a claim faster than any human process can weigh it, the institution meant to serve the citizen’s judgment becomes an instrument that shapes it in advance, now running at machine cadence. The persistence of falsehoods is a given. What matters more here is sequence. It moves what a person can see before the decision ever reaches them, so that by the time a choice is made about a life, the ground under that choice has already shifted, and no care taken downstream can recover what was removed from the upstream view.
By the time any correction arrives, it lands in a field the false claim has already reorganized. The reader who might have weighed the claim has already acted on the world that claim built. The corruption reaches the person before the decision does, and no downstream response can undo that. That is why what is at stake is the public peace the hymn names, not the accuracy of a particular feed.
Questions about whether a specific claim is false, and whether a particular system amplified it wrongly, turn on evidence unique to that claim and system. They belong to institutions built to weigh such evidence under due process, to the courts that decide liability, the regulators with jurisdiction over a platform or market, the newsrooms whose standards govern what is published, and the platforms themselves in whatever governance they have adopted.
To adjudicate any of that here would swap the subject, the structural threat and the obligation to build against it, for a litigation with no standing to conclude it.9 A machine moves the ground of a person’s perception before they can stand on it, and the answer to that is a thing built so the ground can still be stood on.
The maker’s obligation does not stand alone. Those who speak to the public about the instruments the maker has shipped carry an obligation of their own. Peace to people of good will names two kinds of people, those to whom the peace is granted and those who stand outside it, and anyone who trades on the public’s partial grasp of new instruments for ends that are not of good will stands outside it, exploiting the gap between what a system can do and what it cannot, between what the record shows and what a plausible claim can be made to imply.
The maker answers for what the instrument does. Those who address the public about it answer for whether they turned the public’s difficulty in understanding toward peace or against it, and when that obligation fails on the speaker’s side it corrodes the same peace the maker is trying to build. The polarized condition of public conversation is, in part, the record of that failure. The obligation stands as a condition of the peace that cannot be held from one side alone.
V. What the Tradition Now Requires
The debt once paid in person must now also be paid in the instrument, designed to stand in our place at the speed and scale where we cannot go, and the moment asks for both at once.
Stuart J. Russell, a foundational authority on AI at the University of California, Berkeley, offers a way to do that. The standard model of intelligence, a machine that optimizes a fixed objective a person hands it, is dangerous because a fixed objective is always an incomplete specification of what humans want, so a capable optimizer of an incomplete objective grows more dangerous as it grows more capable. His correction is to build the machine to stay uncertain about the human preferences it serves, to treat human behavior as the evidence it defers to, and so to remain correctable by the people it acts upon.10 A machine built to be uncertain is still a machine whose uncertainty was a choice some maker made, calibrated to some reading of whose preferences count. Even provable beneficence, if it could be engineered and shipped, would travel without the makers, and they would still owe an account of it to the person it deferred to and to the successor who inherited it. A design answer of this kind does not dissolve the maker’s obligation. It is what the obligation now becomes when the maker cannot be present to discharge it.
The existing vocabulary of responsible AI, its language of explainability, contestability, auditability, and procedural fairness, asks only about outputs. A system can satisfy all of it and still erode the ground that makes those questions matter, binding the person, the peer, and the successor without ever adjudicating a live claim.
The strongest work in the field has already seen part of this and moved upstream of the output. Andrew Selbst and Solon Barocas put the case most exactly, a consequential decision owes the person it lands on more than a description of the rules the model followed. It owes a justification for why those rules are the rules at all, which means an account of the choices made in building the system, not just an account of the system itself. Their move is to look past the finished model to the process that produced it, and once that door opens, the whole question of accountability shifts backward from what a system emits to the design decisions that shaped it.11
Joshua Kroll and his coauthors reach the same door from the engineering side. They show that transparency after the fact, disclosing a model once it has run, cannot on its own deliver accountability, and that procedural regularity has to be engineered in advance so the way a decision was reached can be verified rather than merely described.12 Deborah Raji and her colleagues carry that instinct into practice, building an audit that runs the length of the development process instead of inspecting only the finished model, so a record of what was decided, by whom, and against which standards accumulates as the system is built.13
All three corrections succeed right up to the moment of handoff, or when the maker hands the instrument on to its subsequent operators and minders. Procedural regularity can be engineered, a justification written, an audit trail kept, and all three can be present on the day the system ships yet fail to survive the hand that built them. A warrant designed in is one thing. A warrant that survives inheritance is another. The account these frameworks produce is addressed to auditor, regulator, and court. All parties who arrive inspect a system already running. It reaches no one who takes the work over when its author has gone, retunes it as the world drifts, and ships the next version. That successor reads what was left behind, and what is left behind is almost never the warrant.
So, the obligation runs past accountability for the output, past even accountability for the design. The obligation runs to answerability built to be handed down. An account of what the maker was willing to sign that the next hand can read and stand behind in turn. The existing frameworks close the gap between output and design. They do not close the gap between one hand and the next.
That same scoring model decides credit, jobs, and benefits at population scale with an appeal channel a regulator would deem adequate, the shape the Australian Robodebt scheme took before a Royal Commission exposed its machinery.14 Follow the case to the reviewer, the human the governance framework keeps in the loop. When the contested case arrives, the reviewer sees the person’s inputs, the model’s score, and perhaps a ranked list of influential features, but not the upstream calibration, which populations the model was tuned to fit, which it was allowed to fit poorly, where the indicators that stand in for risk were chosen, where the thresholds were set, and what tradeoffs were frozen into the objective.15 The reviewer can inspect the case but not reconstruct the ground beneath it, so in practice the reviewer confirms that the inputs were entered correctly and the model ran as specified, and signs. The framework records a human in the loop. What it cannot record is that the human was structurally unequipped to remake the judgment they ratified.
The Australian record already carries a worked instance of exactly this failure, and its machinery is public.16 From July 2015 the Department of Human Services ran an automated system that took a welfare recipient’s annual income figure from tax records and spread it evenly across the fortnights of a review period, then treated that assumed fortnightly figure as the basis for raising a debt. The decisive choice was administratively plain. Averaged data stood in for what a person had actually earned in each fortnight, and the recipient carried the task of disproving the figure. A person who could not produce pay records from years earlier was left owing a debt the department had never established. That single choice moved the weight of proof from the office that raised the debt to the person who received the notice.
A departmental lawyer had written in 2014 that a debt derived from averaging might not sit within the legislative framework. That warning was on the record before the mechanism went to scale. When a review tribunal ruled in early 2017 that averaging gave an insufficient basis for a debt and directed the department to recalculate from real fortnightly records, the practice continued in modified form rather than stopping. Later officials sought fresh advice that read the earlier doubt away, and one senior official instructed staff to halt a request for advice that might expose the question. The recipient contesting a notice met a review officer who could confirm that the inputs were entered correctly and the calculation ran as specified, but who inherited a process that treated the averaging as already authorized and could not reopen the ground beneath it.
What the successor inherited was an operating practice wearing the appearance of legal authority. When a person asked who had established that averaged income could prove an actual debt, and on what advice, the answers had been obscured at the design stage and then defended by officials with an interest in not finding them. The Commonwealth later admitted in the Federal Court that it had never held a proper legal basis for the debts.
The obligation passes, or fails to pass, to the successor who inherits the model in three years when its author has moved on, whose task is to maintain it, retune it as the world it scores drifts, and ship its replacement. Ask what survives for that successor to read. Documented in the usual way, the model hands down its architecture, its training procedure, its performance metrics, and a record of what was done. What it does not hand down is the warrant, the account of what was judged acceptable and why, which thresholds the maker was willing to defend to a person who lost something at them, and which were set knowing they were the least indefensible option available under the schedule. That judgment lived in one person’s understanding and was never written into a form the next hand could read. So the successor inherits an instrument that still acts on people under a warrant no living person can vouch for, an authority that outlived its author and answers to no one. The answer that was supposed to travel did not, because it was never built to be inherited.
Worked all the way through, the scoring model arrives exactly where it began, at a place where compliance passes and answerability fails. The audit reveals the appeal channel, the governance framework confirms the human in the loop, and the compliance regime offers corrective actions. The problem is that even as the system passes every check on its output, the condition those checks were meant to protect is already gone. The output vocabulary cannot name that failure, because it asks whether the mechanisms are present, not whether human answerability survived them.
The maker answers for far more than the instrument’s rote performance. All involved must be afforded some recourse. An obligation that extends in three directions. To the person scored, who must still be able to see and contest what was decided about them, in a form they can grasp and in time for it to matter. To the peer downstream, who must still be able to reconstruct the ground the decision stood on. And to the successor, who must still be able to read what the maker was willing to sign, because a shipped model shapes the models that come after it, feeding them its own data and entrenching in them the objectives and evaluations it was built to satisfy. The successor is the one the output vocabulary reaches least, and the one who must answer for the machine’s work long after the first maker has retired or died.
Today’s responsible maker does not ship an instrument that stands on generative machinery alone to reach a conclusion on which a critical outcome turns, the outcomes that end a career or a business or a person’s access to the thing they were counting on. A system of that kind can compose an answer that reads as though a competent hand produced it, fluent, confident, shaped like judgment. The maker’s refusal falls on the weight the machinery was never built to carry, never on the machinery itself.
It is the machine’s displayed fluency that masks the danger. A system that can compose a plausible answer does not thereby hold the warrant to act on one, and a maker who confuses the two has mistaken the appearance of judgment for the ground that entitles a judgment to stand. Where the outcome is critical, the maker refuses to ship the instrument that has only the surface, and must be most firm at the very moment that surface is most convincing.
Declining to cross the line is only the start. Makers who see the line also owe an account to the people on the other side of it.
They owe the executive asked to approve the deployment an honest account of what the instrument can and cannot do, the account that lets the approver understand what they are putting their name to, which is rarely the account that makes approval easy. They owe the people it acts on an account that lets them know they are being acted on by a made thing and not by a fate, so they keep the standing to ask what was decided and why. And they owe the successor an account of what the maker was and was not willing to sign, which lines held and were judged acceptable and at what cost, so the one who maintains the instrument inherits the conscience built into it. These are duties the maker is bound to, professional and moral both.
There is a further obligation that runs forward in time. When makers understand something about the instruments of the moment that the people around them have not yet been given the words to hold, that understanding is not theirs to sit on. They are obliged to instruct and to clarify, in whatever setting they meet someone trying to grasp what is being changed, the boardroom, the working team, the colleagues across the enterprise. The obligation does not wait for the culture to catch up, because the discipline has arrived before the vocabulary has, and someone who holds both has to close the distance. Makers who understand and stay silent, who let people they could have oriented go on acting in a fog they had the standing to lift, have failed an obligation as real as the obligation not to ship the reckless thing. Building well is one duty, and explaining, where explanation is owed and no one else is positioned to give it, is the other, and in a period like this one the second is the more consequential.
VI. The Closing Test
The test is quieter than it first looks.
Where power is listening, the incentives do most of the work, and you can be forgiven for saying the true thing when the true thing also advances you. The test sits elsewhere, in what you say where power is not. It is what you build into the instrument when no one is auditing the calibration, when the schedule is tight, the concession is invisible, and the only witness is the future person it will act on, who has no seat at the table, no name yet, and no way to press back.
It is the room those three men from last century found themselves in, each in his turn, the room where the answer costs something and no one is applauding the cost. Keynes in the New Hampshire hotel losing the institution he had designed. Einstein at the edge of the project his signature had launched, arguing against a doctrine that did not want to hear him. Oppenheimer before the board that had decided to take his standing. None of them was in a room where power was listening. Each of them answered anyway.
That room is a specific moment in the building of a specific system, and if you have shipped one, you know where it sits. The moment arrives when the readable version of the model costs three weeks the schedule does not have, and the opaque version ships on time and no one downstream will ever know the difference until it has already acted on someone who cannot see what was decided. The choice returns with the warrant, which could be written into a form a successor can read or kept in the maker’s own head where it is faster to keep it, and the deadline argues for the head. The demand comes a third time when the pace could be slowed by a beat to let a person answer, at a cost in throughput that shows up on a dashboard, or left at the speed that decides for the person and calls it efficiency.
No one is present for those decisions except the maker and the schedule. The review comes later, finds whatever was built, and marks the box either way. The person the instrument will act on is not yet a name. It is the same room last century’s three men would recognize, stripped of audience and drama, left with only the builder and the cost of an answer.
Today’s practitioner faces the same question as the three men from last century, answered now through instruments instead of institutions.
It is whether the instrument you built, standing on its own after you have left and authoring the instruments that follow it, still carries the warrant you willingly signed. The marketing sold one warrant and the compliance document recorded another, and neither is the one that counts. The one that counts is the warrant you would actually stand behind if called to the table on behalf of people you will never meet. That warrant is either built into the instrument or it is not. It cannot be added afterward, because the instrument is already acting, and it cannot be delegated to the machine, because the machine cannot stand before the question. It can only be built in, upstream, at a desk, by a maker who decided that the person the instrument would act on had a right to see and contest what was decided, as well as know who paid for that decision in the design when paying for it was invisible and inconvenient and observed by no one.
Today’s practitioners must build those portable answers, enabling their smooth entry into rooms full of people on disparate schedules, and passing along a warrant readable to all.
Pax hominibus bonae voluntatis.
The peace the ancient phrase names is either built or lost there, in the rooms where builders of good will do the work and send into the world instruments that hold what they were willing to answer for. No benediction pronounced over what is finished and shipped can supply it. It is the condition makers build toward or corrode, one instrument at a time, where no audit reaches, on behalf of people who will never know their names.
Stephen DeAngelis
Princeton, NJ
July 2026
About the Author
Stephen F. DeAngelis is the founder, president, and CEO of Enterra Solutions and Massive Dynamics, two companies that apply artificial intelligence and advanced mathematics to complex enterprise challenges. His work spans international relations, national security, and commercial technology, with visiting research affiliations at Princeton University, Department of Chemistry, the Computing and Computational Sciences and National Security Directorates of the Oak Ridge National Laboratory, the Software Engineering Institute at Carnegie Mellon University, and the MIT Computer Science and Artificial Intelligence Laboratory. He holds patents in autonomous decision science.
ENDNOTES
1 The essay uses the Vulgate reading of the angelic hymn at Luke 2:14, peace to people of good will, following Jerome’s late-fourth-century Latin rather than the more familiar modern English carol line. It follows the Clementine Vulgate word order, pax hominibus bonae voluntatis, the form carried into modern use and quoted by John XXIII in Pacem in Terris. The Stuttgart critical edition of Robert Weber and Roger Gryson, editors, Biblia Sacra Vulgata, editio quinta emendata (Stuttgart: Deutsche Bibelgesellschaft, 2007), prints the older Jerome reading with the preposition, pax in hominibus bonae voluntatis, and the facing-page edition of Angela M. Kinney and Swift Edgar, editors, The Vulgate Bible, Volume VI: The New Testament, Dumbarton Oaks Medieval Library 21 (Cambridge, Mass.: Harvard University Press, 2013), sets the Latin against the Douay-Rheims English. Both readings render “peace to people of good will”; the essay adopts the Clementine word order for the title. The genitive construction has more than one defensible English rendering, “people of good will,” “men of good will,” and, on a different reading of the Greek underlying the Latin, “those with whom God is pleased”; the essay adopts “people of good will” and does not rest its argument on the disputed alternatives. 2 The four conditions the essay names, brittle, anxious, non-linear, and no longer legible, draw on and rework the BANI framework, which Jamais Cascio created in 2018 and first presented in a talk on anarchy and chaos for an Institute for the Future meeting in Palo Alto, then brought to a public audience in his essay “Facing the Age of Chaos” (Medium, April 29, 2020), https://medium.com/@cascio/facing-the-age-of-chaos-b00687b1f51d. Cascio’s fourth term is “incomprehensible”; this essay uses “no longer legible” to name the same phenomenon in the register of accountability rather than epistemology, the point being not that the world exceeds understanding in principle but that it has outrun the scale at which a single answerable hand can hold it in view.
3 The Princeton scene draws on the author’s advisory work with a private-sector engagement; identifying details have been withheld.
4 The three men the essay follows are not the whole of the tradition, and the argument does not claim they are. The practitioner who bore the public weight of an instrument he was party to building recurs across the modern record. Alan Turing built the machinery that broke the Enigma traffic and did not survive the state he had served, a life documented in Andrew Hodges, Alan Turing: The Enigma (Princeton: Princeton University Press, 2014). Rachel Carson, a government marine biologist, turned what she knew of pesticide chemistry into the public warning of Silent Spring and answered for it against the industry it indicted, in the account of Linda Lear, Rachel Carson: Witness for Nature (New York: Henry Holt, 1997). Norbert Wiener, who named cybernetics, spent his later years on the human consequences of the control systems he had helped invent, chronicled in Flo Conway and Jim Siegelman, Dark Hero of the Information Age: In Search of Norbert Wiener (New York: Basic Books, 2005). Vannevar Bush organized the American scientific establishment that made the wartime instruments possible, in G. Pascal Zachary, Endless Frontier: Vannevar Bush, Engineer of the American Century (New York: Free Press, 1997). Andrei Sakharov built the Soviet thermonuclear weapon and became the conscience that opposed the regime it armed, in Gennady Gorelik and Antonina W. Bouis, The World of Andrei Sakharov (Oxford: Oxford University Press, 2005). Keynes, Einstein, and Oppenheimer are chosen not for the exhaustiveness of the tradition but for the specificity of the pattern the essay reads in them, the maker’s work entering the public peace after leaving his control, creating an obligation intention alone could not discharge, and forcing an answer under conditions that resisted it. The others belong in the tradition. These three carry the exact shape.
5 On Keynes at Bretton Woods, the standard modern treatments are Benn Steil, The Battle of Bretton Woods: John Maynard Keynes, Harry Dexter White, and the Making of a New World Order (Princeton: Princeton University Press, 2013), https://press.princeton.edu/books/paperback/9780691162379/the-battle-of-bretton-woods, and Robert Skidelsky, John Maynard Keynes, Volume 3: Fighting for Freedom, 1937 to 1946 (New York: Viking, 2001). The conference ran from July 1 to July 22, 1944, at the Mount Washington Hotel in Bretton Woods, New Hampshire. Keynes’s design was the International Clearing Union with its reserve unit, the bancor, intended to settle trade imbalances multilaterally rather than through gold or a single national currency. The plan that prevailed was substantially Harry Dexter White’s, with the dollar anchored to gold at the center of the system, and Keynes obtained real concessions, the scarce-currency clause, the transitional arrangements, and the quota adjustments, while losing the successor institution he had designed. His analysis of what the settlement of the previous war had broken begins in John Maynard Keynes, The Economic Consequences of the Peace (London: Macmillan, 1919), the diagnosis on which the later design rested. The moral arc of the life is carried in Zachary D. Carter, The Price of Peace: Money, Democracy, and the Life of John Maynard Keynes (New York: Random House, 2020). The pre-conference contempt for the machinery of the conference, that its staging was in order that the President could say that forty-four nations had agreed on the Fund and the Bank, is Keynes to Sir Richard Hopkins, June 1944, quoted in Steil, 194, and the related “monkey-house” characterization is Keynes to David Waley, May 1944, quoted in Steil, 190. The diagnosis that quarters in the United States intended to use the grant of post-war credits to impose the American conception of the international economic system is likewise quoted in Steil, 191. The “unhouselled, disappointed, unaneled” passage, Keynes echoing Hamlet’s ghost (Hamlet, Act I, scene v), and the account of signing on the dotted line without a clean consecutive copy are Keynes’s own words as reproduced in Steil, 251, and in Skidelsky. Keynes’s spelling of the phrase is his own and differs from the First Folio’s “unhousel’d, disappointed, unanel’d”; the essay preserves his orthography rather than regularizing it to the Shakespearean form. The House of Lords speech of December 18, 1945, including the regret that the loan was not interest-free and the “resurrect and re-erect the idols” passage, is verified in the Parliamentary record, UK Parliament Hansard, House of Lords, Anglo-American Financial Arrangements, December 18, 1945, https://api.parliament.uk/historic-hansard/lords/1945/dec/18/anglo-american-financial-arrangements. The Carabosse image is from Keynes’s Savannah remarks of March 1946, reproduced in Skidelsky. The conference dates, location, and the forty-four nations are confirmed independently in the World Bank Group Archives, https://www.worldbank.org/en/archive/history/exhibits/Bretton-Woods-and-the-Birth-of-the-World-Bank.
6 The letter Einstein signed was dated August 2, 1939, and addressed to President Franklin D. Roosevelt, warning that recent work in nuclear physics might make possible a bomb of great power and that Germany might pursue one. Leo Szilard drafted the letter; Einstein reviewed and signed it, lending it the weight of his name. The letter is held at the Franklin D. Roosevelt Presidential Library, and its text is available through the Atomic Heritage Foundation, https://ahf.nuclearmuseum.org/ahf/key-documents/einstein-szilard-letter/. Einstein did not work on the Manhattan Project that followed and was denied the clearance to do so. The standard narrative history is Richard Rhodes, The Making of the Atomic Bomb (New York: Simon and Schuster, 1986), https://www.simonandschuster.com/books/The-Making-of-the-Atomic-Bomb/Richard-Rhodes/9781451677614.
7 Einstein’s 1946 telegram, sent May 24, 1946, as part of the Emergency Committee of Atomic Scientists appeal and published in the New York Times the following day, carries the line the essay quotes; the authentic wording is that “the unleashed power of the atom has changed everything save our modes of thinking.” He died in Princeton on April 18, 1955, at the age of seventy-six. He endorsed and signed the Russell-Einstein Manifesto shortly before his death in April 1955, among the earliest of the eleven signatories to commit to Russell’s draft, and Russell released it publicly at a press conference in London on July 9, 1955, three months after Einstein had died, an occasion documented by the Pugwash Conferences, https://pugwash.org/1955/07/09/london-launch-of-the-russell-einstein-manifesto/. The document carries their two names for Russell’s authorship and Einstein’s endorsement, though the signatures on it are listed alphabetically. The sixteen years the essay counts run from the 1939 letter to the 1955 signature and death. The chronology and the wording are documented in Walter Isaacson, Einstein: His Life and Universe (New York: Simon and Schuster, 2007), https://www.simonandschuster.com/books/Einstein/Walter-Isaacson/9780743264747.
8 Oppenheimer was the principal scientific author of A Report on the International Control of Atomic Energy, the Acheson-Lilienthal Report, published by the United States Department of State in March 1946, which proposed placing the world’s fissile material under an international Atomic Development Authority. The proposal was substantially modified by the Baruch Plan of June 1946, which the Soviet Union rejected, and the international-control effort was not revived. His 1949 General Advisory Committee report, dated October 30, 1949, opposed crash development of the hydrogen bomb on explicitly moral and strategic grounds and did so in two annexes. The majority annex, signed by Oppenheimer as chair with Conant, DuBridge, Rowe, C. S. Smith, and Buckley, held that use of the weapon “would involve a decision to slaughter a vast number of civilians,” that a super bomb “might become a weapon of genocide,” that its existence would be “a threat to the future of the human race which is intolerable,” and that such a bomb “should never be produced.” The minority annex, signed by Fermi and Rabi, named the moral fact most sharply, that the weapon’s existence and the knowledge of its construction were “a danger to humanity as a whole” and that it was “necessarily an evil thing considered in any light.” Both annexes and the body of the report are available in the Atomic Archive transcript of the General Advisory Committee’s majority and minority reports of October 30, 1949, https://www.atomicarchive.com/resources/documents/hydrogen/gac-report.html, and are reproduced in Herbert York, The Advisors: Oppenheimer, Teller, and the Superbomb (Stanford: Stanford University Press, 1989). President Truman announced the decision to proceed with the hydrogen bomb on January 31, 1950, stating his responsibility to see that the country “be able to defend itself against any possible aggressor,” Statement by the President on the Hydrogen Bomb, Harry S. Truman Library, https://www.trumanlibrary.gov/library/public-papers/26/statement-president-hydrogen-bomb. The security hearing ran from April 12 to May 6, 1954, before the Atomic Energy Commission’s Personnel Security Board, Oppenheimer’s clearance was revoked on June 29, 1954, and the full unredacted transcript was released by the Department of Energy in 2014. The standard biography is Kai Bird and Martin J. Sherwin, American Prometheus: The Triumph and Tragedy of J. Robert Oppenheimer (New York: Alfred A. Knopf, 2005).
9 The essay declines, deliberately, to adjudicate any particular present-day claim as misinformation or disinformation, and the restraint is a matter of forum as much as of scope. Whether a specific claim is false, and whether a specific system amplified it wrongly, are questions that turn on evidence particular to the claim and the system, and they belong to the institutions built to weigh that evidence under the constraints of due process, to the courts where liability is decided, to the regulators who hold jurisdiction over a given platform or market, to the newsrooms whose standards govern publication, and to the platforms themselves in the exercise of whatever governance they have adopted. An essay about what the makers of capable systems owe the world is the wrong instrument for that adjudication, and to attempt it here would be to trade the argument’s actual subject, the structural threat and the obligation to build against it, for a litigation the essay is not equipped to conduct and has no standing to conclude. The argument is about the shape of the adversary and the shape of the answer. It is not about the ruling in any particular case, and it leaves that ruling where it belongs.
10 Stuart Russell sets out the argument in Human Compatible: Artificial Intelligence and the Problem of Control (New York: Viking, 2019). Russell holds that the “standard model” of artificial intelligence, in which a machine optimizes a fixed objective supplied by its designers, is a dead end at high capability, because any fixed objective is an incomplete specification of what humans actually want and a sufficiently capable optimizer will pursue the specified objective past the point the designers intended. His alternative is a machine built on three principles (173), that its only objective is to realize human preferences, that it is uncertain about what those preferences are, and that human behavior is its evidence about them, so that the machine remains deferential and correctable rather than fixed on a goal it will defend against interference. The principles are addressed to human developers and are not meant to be recited to the machine. The essay treats Russell’s correction as a design answer of exactly the kind the moment requires, and argues that even a provably beneficial system embodies a calibration its maker chose and remains answerable for.
11 Andrew D. Selbst and Solon Barocas, “The Intuitive Appeal of Explainable Machines,” Fordham Law Review 87, no. 3 (2018): 1085 to 1139, https://ir.lawnet.fordham.edu/flr/vol87/iss3/11/. Selbst and Barocas distinguish inscrutability, the difficulty of describing what rules a model follows, from nonintuitiveness, the difficulty of seeing why those rules are defensible, and argue at 1085 to 1088 and 1118 to 1131 that existing law and most technical work address only the former. Their conclusion, that “to know why the rules are what they are, one must seek explanations of the process behind a model’s development, not just explanations of the model itself” (1085), is the closest prior statement of the upstream-design turn this essay makes, and the essay owes the distinction to them while extending it from justification owed at the moment of decision to warrant owed across the handoff to a successor.
12 Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu, “Accountable Algorithms,” University of Pennsylvania Law Review 165, no. 3 (2017): 633 to 705, https://pennlawreview.com/2017/02/23/accountable-algorithms/. The authors argue that transparency alone, the after-the-fact disclosure of source code or of a model’s workings, is an incomplete and often ineffective route to accountability, and that technical tools can instead build procedural regularity into an automated decision system in advance, so that it can be verified to have treated like cases alike and followed the policy it was supposed to follow. The relevant argument runs at 633 to 637 and 662 to 682. The essay reads their move as the first of the three that shifts the question from the output to the design, and departs from it in holding that the ex-ante guarantees they describe are addressed to the verifier who inspects a running system, not to the successor who must inherit and maintain it.
13 Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes, “Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing,” in Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (New York: Association for Computing Machinery, 2020), 33 to 44, https://doi.org/10.1145/3351095.3372873, preprint at https://arxiv.org/abs/2001.00973. The authors propose an internal audit conducted across the whole development lifecycle rather than on the finished model, producing at each stage a documentary record that can be checked against the decisions actually made. The essay treats their lifecycle audit as the practice-side version of the same upstream turn, and locates its own contribution in the observation that even a complete audit trail is built for the auditor rather than for the successor who inherits the system, so that the record can be exemplary and the warrant still fail to travel.
14 The Robodebt marker attaches to the automation-and-review failure the essay works through as an archetype, a failure whose record is public and documented. The Australian Robodebt scheme, an automated debt-assessment system that ran from 2015 to 2019 and used income averaging to unlawfully assert at least 1.763 billion Australian dollars in debts against approximately 433,000 Australians, was conceded by the Commonwealth before the Federal Court in November 2019 Robodebt Scheme, whose final report Commissioner Catherine Holmes delivered on July 7, 2023, found it to be “a crude and cruel mechanism, neither fair nor legal,” and “a costly failure of public administration” (Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634, para 4; Royal Commission into the Robodebt Scheme, Report, July 7, 2023, xxix, https://robodebt.royalcommission.gov.au/publications/report). It shows the timing and inheritance failures the essay names, the harm landing before any review could reach it and the human reviewers left ratifying an automated judgment they were not equipped to remake. The essay treats it as an instance of the deployed class, not as the subject of its own adjudication.
15 The Dutch marker attaches to the choice of discriminatory indicators upstream of any output. In the Dutch childcare benefits scandal, the toeslagenaffaire, the tax authority’s algorithmic fraud-detection system treated foreign-sounding names and dual nationality as risk indicators and wrongly accused roughly 26,000 families of benefits fraud, forcing many into debt, and the affair precipitated the resignation of the entire cabinet of Prime Minister Mark Rutte on January 15, 2021, following the parliamentary report “Unprecedented Injustice” (“Dutch government resigns over child benefits scandal,” The Guardian, January 15, 2021, https://www.theguardian.com/world/2021/jan/15/dutch-government-resigns-over-child-benefits-scandal; Amnesty International, “Xenophobic Machines,” October 25, 2021, https://www.amnesty.org/en/latest/news/2021/10/xenophobic-machines-dutch-child-benefit-scandal/). It shows that the indicators standing in for risk are chosen upstream, where no downstream review of a single output can reach them. The essay treats it as an instance of the deployed class, not as the subject of its own adjudication.
16 The scheme the body works through is the Australian Online Compliance Intervention, known publicly as Robodebt, and the account here is drawn from two primary records: the Federal Court’s approval of the class-action settlement in Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634, and the Report of the Royal Commission into the Robodebt Scheme, presented by Commissioner Catherine Holmes AC SC to the Governor-General on 7 July 2023. What follows is the record beneath the body’s compression. Chronologically, the mechanism ran from July 2015 to November 2019. The Federal Court described it as an automated system that took a recipient’s Pay As You Go income information held by the Australian Taxation Office and evenly apportioned that annual income across fortnightly increments in a review period, in a process the parties called income averaging, to determine a notional or assumed fortnightly income, which was then compared against the fortnightly income the person had actually reported to Centrelink (Prygodicz (No 2), summary). Where the two diverged, the system raised a debt. The court recorded that, in the absence of further information from the recipient, that assumed figure was the basis on which the Commonwealth raised and recovered debts. The Commonwealth admitted in the proceeding that it did not have a proper legal basis to raise, demand or recover debts based on income averaging from ATO data, and the evidence showed it had unlawfully asserted such debts totalling at least 1.763 billion Australian dollars against approximately 433,000 Australians, recovering approximately 751 million dollars from about 381,000 of them (Prygodicz (No 2), summary). Justice Bernard Murphy called the episode a massive failure of public administration. On the assumption that made the mechanism possible: income averaging is not, in itself, categorically unlawful. The Social Security Act 1991 permits a form of averaging to calculate entitlement in narrow circumstances expressly set out in the statute. The unlawfulness lay in using an annual average as the sole basis for determining an actual fortnightly entitlement and thereby a debt. The Royal Commission found that the proposal contemplated the use of averaged PAYG data as the sole basis for determining entitlement, and that this was why legislative change was required (Report, findings on the Executive Minute and the New Policy Proposal submitted 25 March 2015). The Commission further found that the language of that proposal had the effect of obscuring, in substance, that the process turned on averaging (Report, page 88). On what the first officials knew or failed to record: a 2014 legal advice within the Department of Social Services, settled by the departmental lawyer Anne Pulford, had stated that a debt amount derived from annual smoothing, that is averaging, over a defined period might not be derived consistently with the legislative framework (Report, pages 40 to 41). That doubt was on the record before the scheme reached scale. The Commission concluded that a later 2017 DSS legal advice, which read the doubt away, was not only inconsistent with the 2014 advice but wrong. On what reviewers and the review path inherited: throughout 2016 to 2022 the Administrative Appeals Tribunal made 431 first-tier decisions questioning the legal basis for using averaging as evidence of actual income or of a debt (Kyrou J, Federal Court speech, 18 March 2024, citing the Tribunal record). In a decision of 8 March 2017, the Tribunal concluded that averaging was unlawful because it provided an insufficient evidentiary basis, and directed the department not to recalculate using averaging but to use fortnightly salary records obtainable through the department’s own statutory powers; the Commission recorded that the prohibition on averaging was clear and unqualified (Report). An earlier tribunal decision by Professor Terry Carney in March 2017 had set aside a departmental decision on the same ground. On what successors inherited as warnings and doubts accumulated: the Commission found that the beginning of 2017 was the point at which the scheme’s unfairness, probable illegality and cruelty became apparent, and that it should then have been abandoned or revised drastically (Report). Instead the practice continued in modified form. At a meeting with the Commonwealth Ombudsman’s office on 16 January 2017, DSS officers who held knowledge of the 2014 advice failed to disclose it or any doubt as to legality (Report). The Commission found that Kathryn Campbell, then Secretary of the Department of Human Services, instructed officers to cease responding to a request for advice, motivated by a concern that the scheme’s unlawfulness might be exposed to the Ombudsman (Report, page 189). It found that more than a year after DSS received external advice in draft, the scheme was continuing and debts were still being raised unlawfully on a massive scale (Report, page 275). The relevance to the essay’s argument is direct. A successor-readable account would have had to preserve who established that averaged income could evidence an actual debt, on what advice that use was defensible, what would require the practice to stop, and what a review officer was permitted to reopen when a person contested a notice. On the Commission’s findings, those answers were obscured at the design stage and then defended by officials who had reason not to find them. Commissioner Holmes summarized the whole as a crude and cruel mechanism, neither fair nor legal, and a costly failure of public administration in both human and economic terms (Report, page xxix). Primary sources: Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634, https://www.judgments.fedcourt.gov.au/judgments/Judgments/fca/single/2021/2021fca0634 ; Royal Commission into the Robodebt Scheme, Report, 7 July 2023, https://robodebt.royalcommission.gov.au/publications/report.
Share this post
