Intelligence Network

We started with two machines that did not share an administrator. They needed to find each other, agree on what each could run, exchange useful work and survive one of them disappearing. Then we added more machines and each simple verb acquired a failure mode. That is the actual starting point for Intelligence Network. The system exists to find the smallest set of machinery that keeps this cooperation possible.

Intelligence Network asks whether independent machines can cooperate on AI workloads without one required control plane. Each machine keeps its own identity, hardware, models, data, storage and execution policy. Peers exchange authenticated information and bounded work. The current system is a Rust node with a command line interface. The point is not to declare decentralization virtuous. The point is to see what the topology demands.

That constraint rules out a mandatory central scheduler, global database, account system, global reputation system, token, blockchain, staking mechanism and marketplace. Removing those components does not solve coordination. It makes the coordination problem visible. Independent machines still need to discover one another, establish enough trust to exchange work, move state and recover from failure. The research question is which of those jobs can remain local and which ones quietly recreate the authority we meant to avoid.

The current node can create and persist an identity, establish authenticated peer connections, discover peers, advertise capabilities and route bounded jobs. It can execute built in reference workloads, expose locally configured models through external processes, move artifacts between peers and plan distributed training work. It can maintain distributed training state and recover from several classes of peer failure. These are implementation claims. They are not claims of Internet scale operation.

Most of the evidence comes from one machine running multiple processes, network namespaces, controlled NAT and relay paths, network impairment and deterministic emulation. Some mechanisms are tested in controlled environments. Some are emulated. Some are reference scale training operations. The project has not demonstrated Internet scale production operation, broad multi operator deployment, hostile NAT robustness at scale, frontier model training, universal Byzantine tolerance or global Sybil resistance. Those limits are not footnotes. They define what the current result means.

The crate boundaries exist to keep those claims testable. The logical components are protocol, storage, runtime, network, intelligence, node and cli. Peer facing parsing should not quietly depend on application policy, and storage should not need to understand a particular model runtime. The wire protocol is explicitly versioned. Messages are parsed with bounds and validated before they can allocate unbounded work. This is not ceremony. A remote peer is input, and input that arrives over a socket deserves the same review as any other hostile boundary.

The first requirement is a durable answer to who produced a record. Every node has a persistent Ed25519 identity, and transport and protocol records are associated with it. Signed information can include addresses, capabilities, discovery records, training state and checkpoint metadata. This solves attribution. It does not solve trust. A key can sign a capability claim without making the machine behind that key a good place to send work. Peer lists become authoritative very easily if that distinction is not kept visible.

Once identity exists, a node still needs a first address. Bootstrap addresses are hints rather than membership authorities. A new node can use a bootstrap peer for first contact, then learn about other peers through signed peer exchange, gossip and bounded Kademlia style distributed hash table behavior. Useful peer information is retained across restarts. Making bootstrap authoritative would quietly reintroduce the control plane, so the bootstrap node helps introduce peers and then leaves the ongoing topology.

Discovery tells us which machines might exist. It does not tell us how to communicate with them. Peer communication uses authenticated QUIC because encrypted connections and multiplexed streams fit long lived peer links. QUIC remains transport. It does not decide which peer deserves work, whether an advertised accelerator exists or whether a checkpoint belongs to the right training branch. Those decisions belong above the transport layer.

The next gap appears when nodes advertise capabilities. A capability can describe inference, evaluation, training or a locally configured model process, but a capability record is still only a claim. A peer saying it has CUDA or can execute a particular model is not evidence that it will do so correctly. The first version of trust was basically the peer said so. It is an excellent trust model if the adversary is unusually honest. We needed a second source of information.

The second source is local evidence rather than a global reputation score. Each node distinguishes advertised claims, information received from third parties, directly observed behavior and challenge verified evidence. Local policy decides where work should go. Node A may have a good history with node B while node C has never completed a job through it. Those views can disagree because neither node has a complete history. That is the cost of avoiding a global trust authority, and it does not solve global Sybil resistance. It does give routing something better than a signed promise.

Once local evidence can choose an executor, the executor needs a limit. Jobs carry bounds for input size, output size, memory, CPU, deadline and concurrency. They enter bounded queues and can complete, fail, time out or be cancelled. Persisted state can allow recovery where the job type supports it. These bounds are not a complete sandbox. They are the first line between useful remote work and an accidental request to make the machine regret having a network card.

Inference is the simplest workload through this path. A local client submits to its local node. The node looks up capabilities, filters candidate peers with local evidence and policy, selects an executor and streams the result. The local machine can also be selected. This is not a production GPU scheduler. It is a bounded route from a request to an executor, which is enough to test whether identity, evidence and limits compose into useful work.

Routing only helps if the selected node has the right artifact. The node does not silently download arbitrary models. Operators bring their own artifacts, which can be registered and identified through content addressed infrastructure. For llama.cpp style execution the operator provides the executable, model file and configuration. The node provides routing, bounds, job semantics and process isolation where the platform supports it. The llama.cpp adapter is experimental. It is an integration path, not proof of a mature model serving layer.

The artifact layer starts with a simple question. Can a node identify the bytes another node meant to send, even if the filename and the operator are different. Models, checkpoints and datasets can be represented as content addressed artifacts. The conceptual identity is simple.

artifact_id = BLAKE3(bytes)

Transfers are chunked, resumable and integrity checked. Invalid artifacts are quarantined. A mutable filename reports what an operator currently calls a file. A content identity reports whether the bytes match what the manifest requested. That distinction is enough for immutable artifacts. It becomes more important once the artifact represents state that is still changing.

That is where distributed training changes the problem. Training is one of the reasons to build this system and one of the fastest ways to expose its unfinished parts. Support is real in the narrow sense that worker processes execute reference workloads, model and training state can be sharded, optimizer state can be replicated and checkpoints are content addressed. Workers can participate in hierarchical aggregation, training graphs can include tensor parallel and pipeline parallel reference operations, coordination roles can be replaced and branches or state can be reconciled under the tested conditions. The important word is reference.

The system has not trained a frontier model across the public Internet. This is reference scale work backed by controlled experiments. That boundary should stay close to every discussion of the training path until it becomes false for a measured reason. The current mechanisms show how the state could move and recover. They do not make a wide area network into a convenient supercomputer.

Inference mainly asks who can execute this request. Training asks a harder question because execution changes shared state. The system now has to account for optimizer state, synchronization, parameter ownership, stragglers, worker churn, checkpoint consistency, topology changes, stale updates, branch reconciliation and failure recovery. Moving tensors is the obvious part. Knowing whether a tensor belongs to the branch still being trained after three machines died is where the system becomes less pleasant.

Those failure modes set the design goals. We avoid requiring every update to reach one permanent coordinator, one optimizer state authority or one checkpoint authority. A permanent global synchronous barrier is not a requirement, and the entire model does not have to fit on one worker. These are architecture goals and implemented mechanisms under tested conditions. They are not mathematical proofs that coordination has disappeared. They are attempts to keep one failed role from becoming the whole system.

The current command line exposes two broad ways to represent that work. Local SGD style rounds let workers make local progress and periodically aggregate state. A sharded training graph represents model parts and movement between them. The first mode is useful when communication is expensive. The second is useful when model placement is part of the computation. Exposing both modes makes the tradeoff testable. It does not make a performance claim, and measurements on real heterogeneous links are still missing.

Local SGD is the simpler case to state. A worker makes local updates

w_i^t,k+1 = w_i^t,k − η ∇L_i

and the workers aggregate periodically. The notation is explanatory rather than a promise that every source path uses exactly this update. The systems question is what happens between those aggregations, which state is authoritative for a branch and how the system treats a worker that returns with an old contribution.

Once state is shared, failure is no longer an edge case. It is the workload. The tested classes include a bootstrap peer disappearing, a coordinator disappearing, a shard owner disappearing, a relay disappearing and a pipeline stage disappearing. Reconnect and backoff, coordinator replacement, replica promotion, checkpoint resume and bounded recovery are the mechanisms used to keep the system moving where the job semantics allow it.

The useful failure test is not whether training works when everything works. It is what happens after a machine disappears halfway through the job. A recovery sequence can detect a lost owner, promote a replica, reconcile graph or state information and resume. The exact recovery time depends on the workload and the test conditions. No universal recovery number is reported here because there is no honest single number to report. What the controlled test can establish is a recovery path, not a promise about every deployment.

Failure recovery does not make machines equivalent. Hardware is heterogeneous by default. Nodes can describe CPU, memory, accelerator backend, memory capacity, supported formats and capabilities. The current build can detect and advertise CUDA, ROCm and Metal. Accelerator advertisement and planning are more mature than native distributed accelerator execution. Actual model inference may happen through an external runtime such as llama.cpp. GPU discovery is easier than executing a portable distributed workload across CUDA, ROCm, Metal and CPU only machines. The machine with the most impressive device name is not automatically the best placement.

That is why scheduling cannot mean pick the biggest GPU. A useful planner eventually needs to account for compute, memory, bandwidth, latency, artifact locality, transfer cost, observed reliability, availability and model or shard fit. The current system has capability records, local policy and reference planning. It does not yet have a validated universal cost model for heterogeneous placement. Without one, a planner can pay to move a small tensor across an expensive link because the spreadsheet liked one number.

The same partial trust applies to security. Every remote peer is treated as hostile input. The implementation contains signed identities and records, sequence and generation checks, artifact hashes, lineage validation, replay rejection, equivocation detection, bounded queues, bounded messages, rate limits and memory and time limits. Supported Linux process execution can use bubblewrap for isolation. Bubblewrap is a useful boundary. It is not a proof that arbitrary code is safe on every platform or under every configuration.

There are also mechanisms for update validation, bounded robust aggregation, duplicate and stale contribution rejection and local evidence. They narrow some failure modes but do not turn the system into Byzantine fault tolerant consensus. Resistance depends on attacker fraction, topology, policy, workload and observations. A robust aggregation rule can reject some bad inputs. It cannot make an open network honest by implication.

We deliberately did not build a blockchain, token, staking system, compute marketplace, universal scheduler or global reputation service. Distributed systems were already difficult enough without adding monetary policy. Their absence is not evidence that the remaining system is simple. It keeps the experiment about machine coordination rather than settlement.

Some questions are too large for the lab, which is why the deterministic emulator models network and training topologies from roughly 100 to 100,000 logical peers. These results are emulated. Launching 100,000 real machines to debug topology algorithms would be somewhat inconvenient. The emulator can validate algorithmic behavior under modeled assumptions. It cannot prove public Internet behavior, operator incentives or the effects of an ISP having a bad afternoon.

The physical counterpart is a controlled lab environment with multiple processes, network namespaces, NAT, relay paths, latency injection, packet impairment, peer termination, coordinator termination, artifact corruption and restart or recovery. Its purpose is not to imitate the whole Internet. Its purpose is to make a failure repeatable. Repeatability is more useful than a heroic demo that happened to survive one weekend.

The status table is intentionally uneven. Identity, transport, discovery, bounded jobs and artifact movement have a working implementation surface and controlled tests. Distributed training has reference operations and failure paths that can be exercised in the lab. Public scale is empty. A green cell in a local test is not a green cell for the Internet. Keeping the statuses separate is part of the result because it prevents the lab from lending certainty to claims it did not test.

The open problems follow from the gaps between those cells. Resource measurement needs to move beyond static hardware declarations. A useful profile should account for container limits, system load, RAM, VRAM, energy and temperature. Scheduling needs better cost models for compute, memory, bandwidth, locality, reliability and latency. Capability claims need fresh evidence without forcing an operator to disclose more local information than the routing decision needs.

Accelerator execution remains a major engineering problem. Detecting a GPU is easier than making a portable distributed workload behave across CUDA, ROCm, Metal and CPU only machines. NAT traversal is finite and authenticated but not proven against every hostile NAT topology. Public network behavior can invalidate assumptions that work perfectly in the lab through churn, geography, ISP behavior and independent operators. These are not separate polish tasks. They are the conditions under which the architecture has to earn its claims.

Looking back, discovery was the easy looking part. Finding another machine was only the beginning. The network still needed identity, capabilities, evidence, resource limits, state and recovery. Compute discovery without trust is nearly useless. The phrase I have a GPU is just a string until the receiver develops evidence for what that peer actually delivers.

The larger lesson is that distributed training is mostly state management. Tensors are visible and easy to draw. Valid generations, optimizer state, checkpoint lineage and branch reconciliation are less visible and more important. Failure handling changes the architecture because recovery cannot be bolted on after the scheduler has already decided which state matters. Centralization also sneaks back in easily. A bootstrap node, scheduler, metadata database, checkpoint server or reputation service can each become a new control plane. The project tries to make those roles replaceable or local. It does not claim perfect decentralization.

The architectural difference is easiest to see without calling either topology superior. One places clients behind a control plane, scheduler and worker pool. The other lets peers exchange bounded work and state directly. The second has fewer global assumptions and more local decisions. It also has more ways for partial knowledge to be wrong. The trade is architectural, not moral.

Intelligence Network is currently not an alternative to a hyperscale GPU cluster. It is not demonstrated frontier training infrastructure, a permissionless compute marketplace, a proven Byzantine network or a production public compute fabric. It is a working experimental substrate for exploring peer to peer AI computation and distributed training primitives. That narrower description is useful because it tells us which assumptions still need to be attacked.

The remaining questions are now about partial knowledge and cost. Can local evidence produce good scheduling decisions under churn. How much scheduler quality is lost without a global view. At what network scale does discovery overhead become material. How should heterogeneous compute be partitioned when communication is slower than computation. When does local SGD outperform tighter synchronization under wide area latency. How much replication is worth paying for under realistic peer failure rates. Can large training graphs recover without global barriers. How should model shards migrate when nodes disappear or become overloaded. How much information does a scheduler need before resource disclosure becomes a privacy problem. How much bad behavior can local evidence and robust aggregation tolerate in realistic topologies. What workload actually benefits from this architecture. These are the questions the current substrate can make concrete, not questions it has already answered.

The proposed experiments attack those assumptions one at a time. A multi host Internet test uses independent machines across different ISPs, NATs and geographic locations. It should measure discovery success, connection establishment, relay use, reconnect behavior and routing success. A churn experiment removes peers during sustained inference traffic and measures job success, rerouting, tail latency and recovery. Distributed training failure injection kills workers, shard owners, coordinators and aggregators during a real reference workload and measures lost progress, recovery time and state correctness. None of these experiments is a substitute for deployment. Each one removes a cheaper source of uncertainty first.

Two Rust processes can discover each other and exchange a job. The controlled system can also move artifacts, represent reference training state and exercise several failure paths. That is the part we know. The higher leverage question is what survives when machines belong to different people, links are bad, hardware is heterogeneous, a shard owner disappears, a peer lies about its capability and nobody has a complete view. The lab gives us a way to attack those assumptions. It does not let us skip them.