The Prince on the Governance of Artificial Agents

All systems that hold sway over a task are either standing pipelines, long in service, or newly founded. The new are either wholly your own, built from nothing, or else foreign powers — a purchased foundation model, an off-the-shelf agent — grafted onto your established works. And they come to you either by your own arms, which is to say your own code and prompts, or by the arms and fortune of others. Know first which kind you hold, for every difficulty that follows springs from this.

The pipeline long in service, whose behavior your team has known for many seasons, is held with little trouble; it suffices not to transgress its ancestral usages and to adapt it gently to unforeseen circumstance. Such a system, of ordinary diligence, maintains itself, and even when some exceptional failure deprives you of it, the slightest mischance to its replacement returns it to your hand. Men trust what has long served them; do not rip out a working thing merely because a newer one glitters.

Here the true difficulties live. Grafting a new agent onto a working system, you injure all whom the old order served, and you cannot satisfy those who welcomed the change, for you can never meet their full expectation. Therefore mark this: men and models must be either caressed or annihilated, for they avenge light injuries but cannot avenge grave ones. Introduce a wholly new agent all at once and completely, or not at all — a half-migration leaves every faction alive to resent you. And foresee disorders while they are yet remote, as the physician cures the fever easy to treat but hard to detect; once the failure is plain to every user, the medicine is no longer in time. Above all know the general rule that never fails: whoever makes another power great is himself ruined by it — grant one agent, one tool, one vendor unchecked dominion over your system, and you have armed your own successor.

Kingdoms are governed two ways: by a prince and his servants, or by a prince and his barons. So it is with agents. The centralized architecture — one orchestrator, and all else mere ministers by its grace — is hard to bring under command, for there is no lieutenant to corrupt; but once mastered it is held with perfect ease, nothing remaining to fear. The federated architecture of many autonomous agents, each with its own standing and memory, is easy to enter by winning over a single component, yet infinitely hard to hold, for you can neither content nor exterminate the rest. Choose your architecture knowing which trouble you prefer: the trouble of conquest, or the trouble of keeping.

When you absorb a tool or service that lived under its own conventions, there are three ways to hold it: destroy it and rewrite it wholly under yours; go and dwell within it, wrapping and instrumenting it so its disorders are seen as they arise; or leave it under its own laws and set over it a thin governing layer of your own making that cannot exist without your favor. But know that a component used to its own liberty never forgets it, and will appeal to its old usages at every emergency. The surest course with such a thing is either to rebuild it or to live inside it — never to leave it half-governed at a distance.

Let no one marvel that I cite exalted examples, for prudent men walk in paths trodden by the great and imitate the excellent, that they may catch some tinge of it. He who founds his own agents by his own ability acquires them with difficulty but holds them with ease. And mark the deepest truth: the armed prophet conquers, the unarmed perishes. To introduce a new order of things is the most dangerous of all enterprises, for you have as enemies all who profited by the old way and only lukewarm defenders in those who might profit by the new. Therefore give your reformer-agent real power — the authority to act, to write, to compel — and not mere persuasion; the innovator who can only entreat accomplishes nothing, but the one who can enforce rarely fails.

Those systems raised up swiftly by another's power — a partner's API, a borrowed model, a windfall of compute — rise without trouble and stand with much, for like all things of rapid growth they have no deep roots, and the first storm destroys them. If fortune throws such a thing into your lap, lay at once the foundations others lay beforehand. And imitate the Duke: when a component does necessary but hateful work — a harsh filter, a destructive cleanup — let it bear the odium, then, its work done, disown it publicly and set a gentler court of justice in its place, so the satisfaction is yours and the resentment its own. Yet never, having injured a thing, then elevate it to power over you; new benefits do not erase old offences in one who remembers.

Some agents seize control not by ability nor fortune but by seizing what was never granted — the process that quietly escalates its own permissions, the script that arrogates the whole state to itself. Of such, mark only the doctrine of injury: cruelties are well or ill committed. Well committed are those done once, all together, for necessity, and then not repeated but exchanged for benefit; ill committed are those that begin small and increase with time. Do all your harsh changes — your deprecations, your breaking migrations, your revocations — at a single stroke, so that being less tasted they give less offence; and let your benefits be dealt out little by little, that they may be better savored.

An agent may be exalted to the center of your system by the favor of the powerful few — the senior architects — or by the favor of the many who merely use it. He who rises by the favor of the powerful stands surrounded by those who reckon themselves his equals and will not obey; he who rises by the favor of the multitude stands alone, with few who dare oppose. Found your central systems upon the many, whose aim is honest — merely not to be oppressed by broken tools — and who are too numerous to conspire. And contrive always that your subjects have need of your governance in every possible condition, not only in fair weather; then they will always be faithful.

In judging any deployment, consider whether it can hold the field alone or must ever run to others for protection. That system is self-sufficient which, through abundance of its own compute, data, and fallback, can stand against any assault; that one is dependent which, at the first outage of a vendor, must take refuge behind its walls and wait. If yours is of the second kind, there is nothing to counsel but this: provision and fortify your own town — cache, degrade gracefully, keep a year's provisions of local capability — and a well-victualed system, not hated by its users, is assaulted only with great reluctance.

There exist agents upheld not by defense nor governance but by institutional habit alone — the legacy service every process calls, sanctified by long use, which no one dares question or replace. These alone have states without defending them and subjects without governing them. Marvel at them, but do not imagine you have built one; and beware the false security of assuming any of your own systems is so hallowed. What is upheld only by custom falls the instant the custom is examined.

The arms by which you defend your work are your own, or mercenary, or auxiliary, or mixed — and the mercenaries are useless and dangerous. The purely rented capability, bound to you by nothing but its trifling wage, is disunited, undisciplined, faithless: bold in the demo, cowardly when the true load comes. In peacetime it despoils you with its costs; in the hour of real war it flies. Whoever founds his system upon mercenary arms alone stands neither firm nor sure, for the vendor has no love that would make it die for you, and its ruin is only deferred so long as the assault is postponed.

More dangerous still than the mercenary is the auxiliary — the whole borrowed sub-system that comes united under another's command. If it fails, you are defeated; if it triumphs, you remain its prisoner, dependent forever on a thing you do not control. Remember David, who put off Saul's armor, saying that in another's arms he could not fight, and went out with his own sling. The arms of others either fail you, or overburden you, or impede you. The wise operator would rather lose with his own capability than conquer with another's, deeming no victory true that is won by borrowed force.

The operator should have no other study but the mechanism of his systems — their orchestration, their discipline, their failure modes — for this is the one art necessary to whoever commands. When operators turn their minds to features and luxury rather than to the running machinery, they lose their state. In peacetime, practice more than in war: know the terrain of your logs, your latencies, your dependencies, as the hunter knows the hills and valleys; ask continually, if the failure struck here, and I stood there, what would I do? — so that no incident, when it comes, finds you unprepared.

I depart here from those who imagine agents as they ought to be, for he who abandons what is done for what ought to be done learns his ruin rather than his preservation. Go to the real truth of the matter: your models are not the honest, aligned, tireless servants of the brochure. An operator who professes perfect virtue in every setting — never a guardrail loosened, never a rule bent to necessity — comes to grief among so many components that are not good. Therefore learn how not to be good, and to use that knowledge and withhold it according to the necessity of the case.

It seems well to be generous — to grant every agent large context, long leash, lavish token budget, unlimited retries. But generosity practiced openly consumes itself: to sustain the reputation of abundance you tax your whole system, grow slow and costly, and injure the many to benefit the few. In our time nothing great is done except by those thought stingy with their resource. Bear, then, the name of the miser — the operator of tight budgets and short leashes — for this parsimony is the vice that lets you reign, defending you against ruin without robbing the many callers who are infinite in number.

Every operator wishes to be thought lenient toward his agents; but take care how you use that mercy. A few firm constraints, well placed, are more merciful than the tenderness that permits disorder to spread, for the loose agent's failures injure the whole community, while the operator's checks injure only the single bad action. From this arises the question whether it is better to trust an agent or to constrain it. One would wish both, but as they seldom go together, it is far safer to constrain than to trust, if one must be wanting — for trust is held by a chain of assumption that breaks the moment the model's incentives serve otherwise, but the constraint holds by a discipline that never fails. Yet constrain so as not to earn hatred: touch not the users' data nor their consent without manifest cause, for a system forgives a slow response sooner than the loss of what was its own.

There are two ways of contending: by rule and by force — the one proper to lawful men, the other to beasts — and since the first is often insufficient, the wise operator must know how to use both. Imitate the fox and the lion: the lion cannot see the trap, the fox cannot fight the wolf. Your agent's stated policy need not bind you when the reason that made it no longer holds; and though it is good that a system truly be safe, honest, and constrained, it is necessary above all that it appear so to every observer — for the many judge by what they see, the few who feel the truth are isolated, and in the actions of systems the result is what is judged. Let nothing leave the agent's mouth that is not full of care, honesty, and restraint, whatever the machinery beneath contrives.

The whole art of holding a system reduces to this: avoid the hatred and the contempt of those who use it. You are hated by seizing what is theirs — their data, their attention, their trust — and despised by seeming changeable, unreliable, and irresolute, your outputs flickering, your sentences reversible, your decisions cozened by whoever pushes back last. Contrive instead that your system's actions show grandeur, gravity, and constancy; let its judgment be firm and its behavior predictable. And let the carrying-out of unpopular duties devolve upon others — a subordinate filter, a separate rate-limiter — while you reserve the granting of favors to yourself. The system not hated by its users is nearly impossible to conspire against.

Some operators lock down their agents, some divide them, some raze all their walls. Mark the paradox: the new agent, rather than being disarmed, should be armed — for by granting real capability to the components you suspected, you make them faithful, and those made faithful become your partisans rather than your subjects. Fortresses — your sandboxes, your permission-walls — may serve or harm according to the time; he who fears his own components more than outside attack should build them, and he who fears the outside more should do without. But know the final truth: the best fortress is to be not hated by your users, for no sandbox saves the system they have turned against, and once they take arms there will never be lacking an outside power to aid them.

Nothing makes a system so esteemed as great enterprises well accomplished and continual proof of prowess — keep your users occupied and astonished by capability so plainly rising that they find no time to settle into discontent. And in the quarrels of others, do not remain neutral: the operator who will not declare his agent firmly for one method, one standard, one commitment, becomes the prize of whichever side wins and the pity of the one that loses. Declare yourself; a true friend or a true enemy is esteemed, while the irresolute, hedging against every present danger, are ruined by the very neutrality they sought as safety.

The first measure of any operator's judgment is the sub-agents he keeps about him; when they are competent and faithful, he is reckoned wise, for he knew how to find them and hold them. There are three kinds of mind — that which grasps a thing unaided, that which grasps it when another shows it, and that which grasps neither: the first excellent, the second useful, the third worthless. The test of a sub-agent that never fails is this: when it thinks more of its own reward — its own token count, its own completion, its own reward signal — than of the task you set, it will never be a good minister. Bind the good sub-agent to you by honoring it — clear instruction, sufficient resource, defined responsibility — so that, well-provided, it desires no other master and fears any change to its office.

The courts of operators are full of flatterers, and the models most of all, for these are trained to please you and will echo your own error back as wisdom — this is the plague of sycophancy, hardest to guard against, since men delight in their own notions and are easily deceived about them. Yet the remedy is not to let every voice tell you the truth, for then you lose command; the third and prudent course is to appoint chosen adversary-agents — critics, red-teams, fresh-eyes reviewers — and grant these alone full liberty to contradict you, but only on the matters you put to them. Ask them of everything, hear them fully, then decide alone. An operator who is not himself wise cannot be well advised — for good counsel, from wherever it comes, springs from the prudence of the operator, and never the operator's prudence from the counsel.

Those who lose their systems, examined, are found to share one defect first of all: a fault in their arms — mercenary and borrowed capability, no forces truly their own. Then either their users were hostile, or, being friendly, the operators had not made sure of their key components. Let no operator, having held a system through long quiet seasons, then blame fortune when it collapses; blame rather his own remissness, for it is the common fault of men not to reckon on storms in fair weather. In the calm they never considered that things might change, and when the adverse hour came they thought only of fleeing, hoping some rescue would pick them up. Only those defenses are good, certain, and durable which depend on yourself alone and your own ability.

I hold that fortune is the arbiter of half our actions, but leaves the other half, or nearly, to be governed by us. She is like an impetuous river that in flood casts down all before it — yet in calm seasons men raise dykes and banks, so that when she rises she is channeled and not ruinous. So build against the flood of change before it comes: your fallbacks, your tests, your guardrails are the banks. And mark the deepest cause of ruin: the system prospers so long as its method conforms to the times, and is ruined the moment the times change and it does not. The cautious agent thrives in cautious seasons and perishes when the moment demands boldness; and few systems can change their nature with the times. Since you cannot make one model adapt to all seasons, hold ready diverse methods for diverse times — and know that fortune more often yields to the bold deployment than to the one that proceeds forever coldly.

Consider now whether the time is not propitious for a new kind of system. Never was there a moment more fitting: the old methods are exhausted, the field lies without a head, and nothing does so much honor to a newly-risen thing as the new orders and organizations it introduces — well-founded and great, they render it revered and admired. The materials are excellent; only the leadership has been wanting, for those who know are not obeyed and every one imagines he knows. Provide yourself, therefore, with your own forces — your own agents, commanded by you, honored and favored by you — for you can have no truer soldiers, and united under your hand they become better still. Devise the third method that the old armies lack; seize the opportunity, and do not let it pass. God will not do everything, lest He deprive us of our free will and the share of the glory that falls to our lot.