# Table of Contents
- [Gradient](#gradient)
- [Gradient](#gradient)
- [Gradient](#gradient)
- [Gradient](#gradient)
- [VeriLLM: Distributed Serving with Trust](#verillm-distributed-serving-with-trust)
- [Introducing Parallax: The World Inference Engine](#introducing-parallax-the-world-inference-engine)
- [Unknown](#unknown)
- [Gradient](#gradient)
- [SEDM: Living Memory for Open Intelligence](#sedm-living-memory-for-open-intelligence)
- [Massgen: When Multiple LLMs Think Together](#massgen-when-multiple-llms-think-together)
- [Unknown](#unknown)
- [Lattica: A Universal Communication Substrate for Open Intelligence](#lattica-a-universal-communication-substrate-for-open-intelligence)
- [Parallax: Your Sovereign AI OS](#parallax-your-sovereign-ai-os)
- [Symphony: Multi-Agent Intelligence in A Collective Fabric](#symphony-multi-agent-intelligence-in-a-collective-fabric)
- [A Letter to Our Community: The Gradient Roadmap](#a-letter-to-our-community-the-gradient-roadmap)
- [CUAHarm: Measuring the Misuse Risks of Computer-Using Agents](#cuaharm-measuring-the-misuse-risks-of-computer-using-agents)
- [Echo: Decoupling Inference and Training for Large-Scale RL Alignment](#echo-decoupling-inference-and-training-for-large-scale-rl-alignment)
- [Introducing Lattica: The Universal Data Motion Engine](#introducing-lattica-the-universal-data-motion-engine)
- [Introducing Echo: Scaling Reinforcement Learning on Distributed Consumer Hardware](#introducing-echo-scaling-reinforcement-learning-on-distributed-consumer-hardware)
- [Gradient Open Sources Parallax: The Open Source Sovereign AI OS](#gradient-open-sources-parallax-the-open-source-sovereign-ai-os)
- [Gradient Network Raises $10M to Redefine AI Infrastructure](#gradient-network-raises-10m-to-redefine-ai-infrastructure)
- [Unknown](#unknown)
---
# Gradient
RResearchBBlogCCareersAAbout
\>\>\>\>\>\>>>>>
TowardsOpen
IntelligenceIntelligence
Backed by:
\>\>\>\>\>\>>>>>
\[0\]
We are building
the open intelligence that
closed infrastructure never will.
* * *
/
/THINK OPEN
@Gradient
/
The future will be defined by
artificial general intelligence, and
openness will determine who it
serves. We are developing open
foundation models, rebuilding the
training and serving stack that
makes accessing intelligence as
easy as turning on a light.
The future will be defined by
artificial general intelligence, and openness will determine who
it serves. We are developing open foundation models,
rebuilding the training and serving stack that makes accessing
intelligence as easy as turning on a light.
Parallax
--------
### \[Framework\]
#### Your Sovereign AI OS
Parallax is the world’s first fully distributed serving framework that turns a pool of heterogenous GPUs, despite their locations, into an efficient inference platform.
### \[Product\]
#### Free access to the best models
Get complete access to the state-of-the-art open source models hosted on distributed devices collectively,
How can I help you today?
This model is hosted on \+ across different cities.
Try it now.
Try it now on
[https://chat.gradient.network](https://chat.gradient.network/)
[Learn more](https://gradient.network/blog/parallax-world-inference-engine)
Build your own AI Cluster
Echo
----
### \[Framework\]
#### Distributed training by design.
Echo reclaims RL alignment from hyperscalers by decoupling inference and training, turning scattered compute into distinct swarms. Echo reclaims RL alignment from hyperscalers by decoupling inference and training, turning scattered compute into distinct swarms.
Computer-use/SRE/Robotics ...
[Learn more](https://gradient.network/blog/echo-distributed-reinforcement-learning)
Cloud
-----
### \[Product\]
#### The easiest way to run powerful AI models.
Get access to the latest models at unbeatable prices, with full API compatibility for seamless integration.Get access to the latest models at unbeatable prices, with full API compatibility for seamless integration.
[Learn more](https://docs.gradient.network/platform/gradient-cloud)
Run Inference
### \[Model\]
Qwen3-Coder-480B-A35B-Instruct-FP8
Qwen3-235B-A22B-Instruct-2507-FP8
GPT-OSS-120B
...
### \[Price\]
$0.45 / $1.50
$0.15 / $0.70
$0.09 / $0.45
...
Research
--------
\[PRODUCT\]
All
Distributed ML
Agents
Verification
\[DATE\]
\[TYPE\]
\[TITLE\]
[Oct 1, 2025\
\
Distributed ML\
\
Lattica: A Universal Communication Substrate for Open Intelligence](https://gradient.network/research/lattica-a-universal-communication-substrate-for-open-intelligence)
[Sep 30, 2025\
\
Agents\
\
Massgen: When Multiple LLMs Think Together](https://gradient.network/research/massgen-when-multiple-llms-think-together)
[Sep 28, 2025\
\
Verification\
\
VeriLLM: Distributed Serving with Trust](https://gradient.network/research/verillm-distributed-serving-with-trust)
[Sep 25, 2025\
\
Agents\
\
SEDM: Living Memory for Open Intelligence](https://gradient.network/research/sedm-living-memory-for-open-intelligence)
[Learn more](https://gradient.network/research)
---
# Gradient
RResearchBBlogCCareersAAbout
Your browser does not support iframes.
---
# Gradient
RResearchBBlogCCareersAAbout
Your browser does not support iframes.
---
# Gradient
RResearchBBlogCCareers
Research
We hold a strong conviction that scientific progress is a collective effort, and we are building distributed AI learning systems that push technical boundaries while delivering real value to as many people as possible.
\[PRODUCT\]
All
Distributed ML
Agents
Verification
\[DATE\]
\[TYPE\]
\[TITLE\]
[Oct 1, 2025\
\
Distributed ML\
\
Lattica: A Universal Communication Substrate for Open Intelligence](https://gradient.network/research/lattica-a-universal-communication-substrate-for-open-intelligence)
[Sep 30, 2025\
\
Distributed ML\
\
Parallax: Your Sovereign AI OS](https://gradient.network/research/parallax-your-sovereign-ai-os)
[Sep 30, 2025\
\
Agents\
\
Massgen: When Multiple LLMs Think Together](https://gradient.network/research/massgen-when-multiple-llms-think-together)
[Sep 28, 2025\
\
Verification\
\
VeriLLM: Distributed Serving with Trust](https://gradient.network/research/verillm-distributed-serving-with-trust)
[Sep 25, 2025\
\
Agents\
\
SEDM: Living Memory for Open Intelligence](https://gradient.network/research/sedm-living-memory-for-open-intelligence)
[Sep 23, 2025\
\
Agents\
\
CUAHarm: Measuring the Misuse Risks of Computer-Using Agents](https://gradient.network/research/cuaharm-measuring-the-misuse-risks-of-computer-using-agents)
[Aug 26, 2025\
\
Agents\
\
Symphony: Multi-Agent Intelligence in A Collective Fabric](https://gradient.network/research/symphony-multi-agent-intelligence-in-a-collective-fabric)
[Aug 11, 2025\
\
Distributed ML\
\
Echo: Decoupling Inference and Training for Large-Scale RL Alignment](https://gradient.network/research/echo-decoupling-inference-and-training-for-large-scale-rl-alignment)
---
# VeriLLM: Distributed Serving with Trust
RResearchBBlogCCareers
R
<<<<<<<<<<<<
VeriLLM: Distributed Serving with Trust
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Sep 28, 2025
Keyword(s)
Verification, AI Verification
arXiv
[https://arxiv.org/abs/2509.24257](https://arxiv.org/abs/2509.24257)
\[ARTICLE\]
Decentralized inference is an appealing paradigm for serving large language models (LLMs), offering strong security, high efficiency, and lower operating costs. Yet the permissionless setting admits no a priori trust in participating nodes, making output verifiability a prerequisite for secure deployment. We present VeriLLM, a publicly verifiable protocol for decentralized LLM inference that (i) achieves security under a one-honest-verifier assumption, (ii) attains near-negligible verification cost (about 1% of the underlying inference) via a lightweight verification algorithm designed explicitly for LLMs, and (iii) enforces honest checking through a peer-prediction mechanism that mitigates lazy verification in naive voting. We further introduce an isomorphic inference-verification network that multiplexes both roles on the same set of GPU workers. This architecture (i) increases GPU utilization and thereby improves end-to-end throughput for both inference and verification, (ii) expands the effective pool of available validators, strengthening robustness and security, and (iii) enforces task indistinguishability at the worker boundary to prevent job-type-conditioned behavior. Finally, we provide a formal game-theoretic analysis and prove that, under our incentives, honest inference and verification constitute a Nash equilibrium, ensuring incentive compatibility against rational adversaries. To our knowledge, this is the first decentralized inference verification protocol with an end-to-end game-theoretic security proof.
**_VeriLLM ensures that AI results from a network of volunteer computers are trustworthy. It uses a cheap, random spot-check method that makes it more profitable to be honest than to cheat._**
* * *
The Problem: Trust in Decentralized Inference
---------------------------------------------
Decentralized inference promises resilience, transparency, and open access: anyone can contribute GPU resources to serve large models. But without trust, it breaks.
* **Providers may cheat**: running quantized or pruned models, skipping tokens, or using smaller substitutes.
* **Verification is hard:**
* Inference computation is non-deterministic, making it difficult to distinguish whether result differences arise from computational errors or dishonest behavior.
* Consensus-based verification isn’t enough: nodes may collude to manipulate voting or always vote “true” without performing actual verification to save computation.
* ZK-proofs-based verification doesn’t scale: current zero-knowledge ML frameworks impose hours of prover time per inference.
For decentralized inference to matter, **we need verifiability that is lightweight, permissionless, and public**.
* * *
Preview
**Visual 1**: Illustration of a “trust gap.” Left: many anonymous GPU nodes serving a request; right: question mark over outputs. Caption: _“Without verification, inference results are untrustworthy.”_
* * *
VeriLLM: Our Approach
---------------------
VeriLLM introduces a **commit–sample–verify** protocol with three key properties:
1. **Minimal Verification Cost (~1%)**
* Verifiers only rerun the **prefill** phase, skipping the expensive autoregressive decode.
* With minimal on-chain overhead, the protocol guarantees security through a small number of on-chain comparisons, ensuring that verification results are publicly auditable.
* Empirically, this is ~1% of the full inference cost.
2. **Incentive Compatibility**
* Verifiers cannot be lazy: random sampling combined with on-chain comparison ensures that rational nodes are incentivized to perform verification honestly and submit their results.
* A peer-prediction mechanism rewards alignment with honest distributions and slashes false reports.
* Formal analysis shows honest inference + honest verification is a **Nash** **equilibrium**.
3. **Security under Global Majority**
* The protocol incorporates a **ReVerification mechanism** that ensures security under the **Global Majority Honest** assumption, **without relying on the sampled committee being majority-honest**.
* By introducing **zero-knowledge proofs**, the protocol can ensure security **even when only one verifier is honest** (optional).
* * *
Preview
**Visual 2**: Diagram of the commit–sample–verify pipeline:
1. Inferencer commits Merkle root of hidden states.
2. VRF picks random positions.
3. Verifiers recompute and reveal.
Caption: _“Binding commitments + unpredictable sampling = lightweight verifiability.”_
* * *
Architecture: Inference Meets Verification
------------------------------------------
VeriLLM departs from the traditional split of “inferencers” vs “verifiers.” Instead, we multiplex both roles on the same GPU workers, and GPU workers hosting the same model slice are organized into a **Node Group**.
* **Node Groups**: homogeneous sets of GPUs, each hosting a slice of the model. Nodes may serve as inferencers or verifiers, indistinguishably.
To ensure the proper operation of the system, the system is also equipped with:
* **Scheduler**: orchestrates inference tasks, selects roles with VRF proofs, and relays hidden states.
* **Contracts**: on-chain logic aggregates commitments, adjudicates disputes, and settles incentives.
This **isomorphic design** ensures a worker cannot tell if it is running live inference or verification — preventing conditional cheating.
* * *
Preview
**Visual 3**: Side-by-side diagrams of (a) legacy “inferencers vs verifiers” architecture and (b) VeriLLM’s isomorphic multiplexing. Caption: _“VeriLLM: inference and verification indistinguishable.”_
* * *
Security Analysis
-----------------
We prove that VeriLLM detects:
* **Quantization/precision cheats**: hidden-state statistics (mantissa/exponent drift) reveal deviations.
* **Early stopping**: missing ⟨EOS⟩ tokens are caught by last-segment verifiers.
* **Prefill-Forgery Attack**: fake outputs diverge from true decodes under checks.
* **Lazy verifiers**: commit-then-open forces computation; non-openings are slashe.
* **Sampling Majority Attack:** mitigated through the reverification mechanism.
Even under collusion or scheduler misbehavior, the system guarantees detection as long as one honest verifier remains online to verify.
* * *
| **Aattacks** | **Description** | **Defense** |
| --- | --- | --- |
| **Precision-Downgrade Attack (PDA)** | An adversary replaces a deployed full-precision model (e.g., float32) with a quantized or otherwise lower-precision version (e.g., int8, float16, or an aggressively compressed model) | re-execute the inference and conduct a full-token hidden-state comparison, distinguishing anomalies through statistical characterization.” |
| **Early stopping** | Terminate inference before completion. | Check whether the last output token is `` |
| **Prefill-Forgery Attack** | The adversary first guesses a plausible final output, then appends this guessed sequence to the original prompt and performs a single-pass prefill to compute per-token hidden states—bypassing the costly step-by-step autoregressive decoding. | Check whether the predicted token **exactly matches** the given output token. |
| **Lazy-Verifier Attack** | Exploiting the belief that misbehavior by inference workers is rare, a verifier rationally shirks—performing no validation whatsoever and unconditionally marking results as “verified” to save cost and latency. | a commit–reveal scheme plus on-chain sampled cross-checks |
| **Sampling** **Majority Attack** | An attacker with only a small global stake can, by chance, win a majority in a randomly sampled committee and thereby steer the verification outcome. | Make cheating strictly unprofitable by adding a dispute mechanism |
**Visual 4**: Table summarizing attack types (quantization, early stop, forged output, lazy verifier) vs VeriLLM defenses. Caption: _“Defense matrix.”_
* * *
Evaluation
----------
On Qwen2.5-7B models across heterogeneous hardware:
* **Hidden-state checks cleanly separate honest vs quantized runs.**
* **Verification overhead <1%** of generation time.
* **On-chain checks** (sampled scalar indices) are efficient and robust to floating-point noise.
* * *
Preview
**Visual 5**: Benchmark plot: cost of full inference vs VeriLLM verification (~1%). Caption: _“Verification overhead negligible.”_
* * *
Why It Matters
--------------
VeriLLM makes decentralized inference **trustworthy, efficient, and economically aligned**:
* **Trustless verifiability**: correctness can be publicly verified.
* **Lightweight cost**: ~1% overhead is negligible relative to model serving.
* **Economic incentives**: honest participation is strictly profitable, whereas lazy or dishonest behavior incurs penalties.
* **Scalable fabric**: inference and verification co-exist on the same mesh of GPUs.
This is the first decentralized inference verification protocol with a **formal game-theoretic proof of security**.
* * *
Preview
**Visual 6 (closing)**: Artistic visualization of a global GPU mesh with green “verification checkmarks” flowing across nodes. Caption: _“VeriLLM: verifiable AI at planetary scale.”_
* * *
Future Directions
-----------------
We are extending VeriLLM toward:
* **Zero-knowledge escalation**: integrating zk-proofs for maximal collusion resistance.
* **Adaptive thresholds**: auto-tuning statistical detectors across new hardware backends.
* **Protocol-level resilience**: stronger guarantees against scheduler censorship.
VeriLLM is a step toward a sovereign AI fabric where **inference is both** **decentralized and publicly verifiable**.
---
# Introducing Parallax: The World Inference Engine
RResearchBBlogCCareers
B
<<<<<<<<<<<<
Introducing Parallax: The World Inference Engine
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Jun 19, 2025
Category
serving
\[ARTICLE\]
Parallax reimagines model inference as a global, collaborative process
----------------------------------------------------------------------
In the dynamic realm of AI, large language models (LLMs) are driving groundbreaking advancements, from conversational chatbots to agentic workflows. We stand at a pivotal moment, grounded by two axioms that increasingly define the trajectory of AI systems:
* **Humans will always crave more intelligence.**
* **More tokens equal more intelligence.**
Those axioms echo the heartbeat of our species. From the first tools we carved to the neural networks we train today, our hunger for knowledge and capability knows no bounds. In the age of Agentic AI, this craving manifests as an insatiable demand for tokens—the fuel of digital intelligence.
Yet, the systems we’ve relied on are crumbling under this pressure: Chip manufacturers can’t keep up with the demand, but packing more transistors into silicon takes astronomical capital; Data centers guzzle power at a rate our grids can’t sustain, and our energy supply won’t renew at a proportionate pace. Enterprise-grade GPUs are required to host the most advanced models, yet achieving personal intelligence sovereignty is nearly impossible due to the high cost. The status quo isn’t just lagging, it’s a bottleneck strangling the future of AI.
That’s why we’re building [Parallax](https://chat.gradient.network/)
—the world’s first fully distributed inference engine to ignite a revolution. We believe decentralized serving is the key to lifting the existing ceiling in AI development, particularly in terms of scalability, sovereignty, privacy, and cost-effectiveness.
Parallax reimagines model inference as a global, collaborative process—one where models are no longer chained to centralized infrastructure, but are instead composed, executed, and verified across a distributed machine mesh.
Why should I care?
==================
Parallax isn’t just a framework—it’s a redefinition of who gets to access and participate in intelligence.
It enables three foundational paradigm shifts:
* **Intelligence Sovereignty and Democratization for All:** Today, access to advanced intelligence is gated by centralized APIs, corporate clouds, and expensive compute. Parallax breaks this monopoly. It empowers individuals to run state-of-the-art models on their own terms—at home, on their own machines, with no centralized dependency. True equality begins not with money, but with access to intelligence. The right to think, to know, to act—with tools as powerful as the world itself.
* **Composable, Collaborative Inference:** Parallax enables large models to be decomposed and executed across many devices, not just within a single household, but among a global mesh of participants. Consumer GPUs, Apple Silicon, and desktops can be stitched together into a composable inference pipeline, turning fragmented hardware into a coherent execution engine.
* **Unlocking the World’s Latent Compute:** The world is filled with untapped compute: gaming GPUs, laptops, workstations. Parallax brings them into the network. By making model execution modular and verifiable, we unlock the ability to serve large models not just with H100s—but with a global coalition of latent devices. This dramatically expands the viable GPU supply and slashes the cost of inference, further supporting the agentic world.
These shifts don’t just lower costs—they rewrite who gets to access, control, and contribute to the future of intelligence.
How Parallax Works
==================
A production-level decentralized serving system aims to balance compute and communication across heterogeneous devices. Despite its importance, decentralized systems are inherently more complex to set up and manage compared to centralized frameworks. It requires coordination across fragmented environments and deep expertise in both machine learning (ML) and distributed systems. Currently, there is a lack of standardized tools and frameworks for developer adoption.
Parallax is grounded in our belief that a purely decentralized serving framework should be:
* Device-agnostic: Hosts should not just be enterprise GPUs, but also Apple Silicons.
* P2P connected: Users should be able to participate permissionlessly, despite their network environments and system settings.
* Self-discoverable: The system should rebalance to ensure optimal workload partitions.
* Performance-optimized.
Preview
Parallax is a system specifically engineered for high-performance structured generation on decentralized machine networks. The system consists of three main layers: Runtime, Communication, and Worker.
### Runtime Layer
The Parallax Runtime Layer is the core orchestration engine for high-throughput, server-side LLM serving on distributed, heterogeneous networks. To handle the primary challenge of massive concurrency, it is architecturally composed of an Executor (control loop), Model Shard Holder, Request Manager, Scheduler, and Paged KV Cache Manager.
This integrated design enables critical server-grade optimizations:
* **Continuous Batching:** Our Scheduler dynamically groups incoming requests to maximize hardware utilization and overall throughput.
* **Paged KV-Cache Management:** The Cache Manager uses a block-based system, inspired by virtual memory, to prevent memory fragmentation, handle thousands of concurrent requests, and enable high-performance Paged Attention kernels.
As the first framework of its kind for the MLX ecosystem, the Parallax runtime pioneers professional-grade serving on Apple Silicon. Its unique architecture seamlessly integrates both NVIDIA GPUs and Apple devices into a single, cohesive serving fabric, representing a major step forward in building powerful and accessible decentralized AI systems.
### **Communication Layer**
Manages gRPC and tensor streaming between peers, ensuring forward and backward passes succeed even if some nodes fail. Built on top of Hivemind’s Distributed Hash Table (DHT), a decentralized system that ensures fast, reliable information sharing without a central coordinator, to distribute values across peers. We aim to provide future support for NAT traversal, custom UDP, and dynamic routing for network robustness.
### **Worker Layer**
Executes inference tasks across heterogeneous hardware platforms, ensuring optimal performance and scalability via a dual-platform approach:
* **GPU Workers:** Use a modified version of SGLang (a fast serving framework for LLMs, which utilizes PyTorch with CUDA kernels to harness the full computational power of NVIDIA GPUs) with asynchronous batching for heterogeneous compute adaptations.
* **Apple Workers:** We have engineered a state-of-the-art serving engine built upon our pioneering MLX-compatible runtime, with the integration of a highly optimized Metals kernel, including a Paged Flash Attention kernel, unlocking unprecedented inference efficiency and throughput on Apple Silicon.
The Swarm Architecture
======================
Preview
Our decentralized inference engine is built atop a distributed architecture we call the Swarm: a dynamic network of nodes that collaboratively serve a large language model (LLM). Each node is responsible for specific segments of the model, executing a defined portion of the inference process.
When a user submits a request, the client software tokenizes the input into numerical IDs, generates attention masks, and, when needed, encodes routing metadata to optimize execution across the network. The client then identifies the most suitable nodes in the Swarm, selected based on availability, compute capacity, and network latency, that collectively span the full model.
Inference proceeds sequentially: the first node processes its designated layers and passes the resulting hidden states to the next, continuing layer by layer until the model completes its forward pass. All nodes communicate peer-to-peer via a decentralized hash table (DHT), eliminating single points of failure and enabling self-healing. The Swarm also supports dynamic node joining and automatic rebalancing, ensuring that model weights are redistributed efficiently as the network evolves.
**Benchmark**
=============
We have evaluated Parallax’s performance and compared it with a baseline distributed inference system, like Petals. The evaluation focuses on latency, throughput, and scalability using real-world workloads. The evaluation is conducted on a distributed setup consisting of two nodes, each equipped with Nvidia RTX 5090 GPUs connected via networking.
Preview
Performance comparison between Parallax and baseline Petals framework on 1x4K input configuration. All experiments use Qwen2.5-72B-Instruct-GPTQ-Int4 model with 1,024 output tokens.
Preview
Performance comparison for Parallax across different input configurations (Single request: 1x4K, 1x8K, 1x16K, and Multi-request: 4x1k, 8x1K). All experiments use Qwen2.5-72B-Instruct-GPTQ-Int4 model with 1,024 output tokens.
We achieved 3.1x lower on end-to-end latency, 5.3x lower on Inter-token latency, and improved Time-to-first token, Input Throughput, and Output throughput by 2.9x, 3.1x, and 3.1x, respectively. Performance was stable as input length increased from 4K to 16K tokens with higher output throughput, while inter-token latency stayed within a narrow range across all configurations. The total output token throughput increases significantly as we increase batch size, indicating great scalability potential as we progress forward.
Experience Decentralized Intelligence, Real-Time
================================================
You can now try our decentralized [AI Chatbot](https://chat.gradient.network/)
, powered by Parallax.
Designed with decentralization at its core, Parallax offers a new paradigm for decentralized AI. While it functions like any other AI chatbot on the surface, each response is served from a real-time inference running on a swarm of personal machines, and not from a centralized server.
We’re currently in the closed beta phase of our Edge Host program, so you may encounter some traffic as host availability is limited. As Parallax continues to evolve, our goal is a fully decentralized serving network—an always-on intelligence owned and operated by the community.
If you’re interested in contributing your compute, apply to become an [Edge Host](https://docs.google.com/forms/d/e/1FAIpQLScKr6d64cLwz8QO_xCSlilqm6iWVenKbeA12hNrnyJm-qFDRQ/viewform)
The Road Forward
================
[Parallax](https://chat.gradient.network/)
is built for a world where AI is ambient, interactive, and user-owned.
This distributed model serving engine is more than a technical breakthrough; It’s a meaningful step toward democratizing intelligence. Frameworks like vLLM and SGLang have paved the way for optimized inference at scale, and their open-source ethos has been vital to the field’s progress. Inspired by their work, we plan to open-source Parallax once it’s production-ready, and we can’t wait to see how the community builds on it to shape the future of AI.
With [Lattica](https://explorer.gradient.network/)
powering data communication and Parallax enabling decentralized inference, the pieces are coming together at Gradient Network to form the foundation of a truly open, decentralized AI stack.
Stay tuned, for the decentralized future knows no bounds.
Read the full research paper [here](https://gradient.network/parallax.pdf)
.
---
# Unknown
RResearchBBlogCCareers
404
Sorry, the page you are visiting does not exist
---
# Gradient
RResearchBBlogCCareers
Blog
Read more about our approach to building open intelligence.
\[PRODUCT\]
All
Sovereign AI
community
training
serving
communications
fundraising
[\
\
Open Sourcing Parallax: Your Sovereign AI OS\
\
Sovereign AI\
\
Oct 28, 2025](https://gradient.network/blog/parallax-the-sovereign-ai-os)
[\
\
A Letter to Our Community: The Gradient Roadmap\
\
community\
\
Aug 27, 2025](https://gradient.network/blog/community-letter-roadmap)
[\
\
Introducing Echo: Scaling Reinforcement Learning on Distributed Consumer Hardware\
\
training\
\
Aug 18, 2025](https://gradient.network/blog/introducing-echo-scaling-reinforcement-learning-on-distributed-consumer-hardware)
[\
\
Introducing Parallax: The World Inference Engine\
\
serving\
\
Jun 19, 2025](https://gradient.network/blog/parallax-world-inference-engine)
[\
\
Introducing Lattica: The Universal Data Motion Engine\
\
communications\
\
Jun 18, 2025](https://gradient.network/blog/introducing-lattica-the-universal-data-motion-engine)
[\
\
Gradient Network Raises $10M to Redefine AI Infrastructure\
\
fundraising\
\
Jun 16, 2025](https://gradient.network/blog/gradient-raises-10m-redefining-ai)
---
# SEDM: Living Memory for Open Intelligence
RResearchBBlogCCareers
R
<<<<<<<<<<<<
SEDM: Living Memory for Open Intelligence
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Sep 25, 2025
Keyword(s)
Agents, ai memory
arXiv
[https://arxiv.org/abs/2509.09498](https://arxiv.org/abs/2509.09498)
\[ARTICLE\]
**_SEDM gives AIs a living memory that automatically filters for useful experiences, preventing data overload and helping them learn and adapt more efficiently across different tasks._**
### **The Memory Crisis: A Bottleneck at the Foundation**
Long-running multi-agent systems face a critical constraint: the unbounded accumulation of interaction history. As agents engage with environments and collaborate over time, they generate vast trajectories of experience—creating an inevitable collision between scale and performance.
Three fundamental challenges emerge:
1. **Signal Degradation:** Memory repositories fill with redundant or obsolete information, drowning critical insights in noise and degrading decision quality at crucial moments.
2. **Computational Collapse:** Unmanaged growth triggers exponential increases in retrieval costs and system latency, making sustained operation economically and technically infeasible.
3. **Transfer Failure:** Hard-won knowledge from one domain remains locked in context, unable to generalize—limiting agents to narrow, brittle expertise.
Vector retrieval and hierarchical storage—the incumbent approaches—fail to address the root problem: they manage _quantity_, not _quality_.
### **A Living Memory System: The SEDM Vision**
Memory should not be a passive archive. It should be a living substrate—one that evaluates, optimizes, and evolves alongside the agent itself.
The memory framework we are building:
* **Guards Its Boundaries:** Verifies the value of candidate memories _before_ admission, ensuring only high-utility experiences enter the repository.
* **Evolves Continuously:** Consolidates redundant knowledge, amplifies proven insights, and prunes outdated or harmful entries—maintaining a lean, high-signal substrate.
* **Transfers Fluidly:** Abstracts task-specific patterns into generalizable principles, then diffuses them across domains—accelerating learning and adaptation.
This is SEDM (Scalable Self-Evolving Distributed Memory): memory as the engine of continuous intelligence.
Preview
### **The SEDM Architecture: Admission, Scheduling, Diffusion**
SEDM establishes a complete lifecycle for memory—from candidate evaluation to cross-domain transfer—built on three core innovations:
Preview
### **Verifiable Admission: Only Proven Value Enters**
Every candidate memory must earn its place through empirical validation.
* **Self-Contained Execution Contexts (SCEC):** Each agent decision—inputs, reasoning, actions, outcomes—is encapsulated as a reproducible, standalone unit. This creates a universal substrate for verification.
* **Distributed A/B Validation:** SCECs enable massive parallelization. Thousands of execution contexts are replayed across compute clusters, comparing outcomes _with_ and _without_ the candidate memory—simulating counterfactual performance at scale.
* **Utility-Gated Storage:** Only memories that demonstrably improve task success—higher accuracy, lower cost—are admitted. Each receives a quantified utility weight upon entry.
This mechanism guarantees that every item in long-term memory has a _verified_ value. Quality is enforced at the foundation.
### **Adaptive Scheduling: Dynamic Evolution in Production**
An intelligent Memory Controller orchestrates the entire lifecycle post-admission.
* **Utility-Driven Retrieval:** Recall prioritizes not just relevance, but _proven impact_. High-utility memories surface first, ensuring agents leverage their most powerful experiences.
* **Continuous Optimization:** The controller actively maintains repository health—**consolidating** redundant entries, **amplifying** frequently validated insights, and **pruning** obsolete or counterproductive memories.
The result: a memory substrate that remains efficient, high-signal, and performant—even as the system scales.
### **Cross-Domain Diffusion: Scaling Knowledge, Not Just Data**
SEDM enables systematic knowledge transfer across task boundaries.
* **Abstraction:** Specific memories ("Strategy X succeeded in task A") are distilled into generalizable patterns—extracted principles that transcend narrow contexts.
* **Transfer and Re-Validation:** Abstracted knowledge is injected as a candidate into new domains, where it undergoes rapid SCEC-based verification and weight recalibration.
Experience from one frontier—fact verification—becomes leverage in another—multi-hop reasoning. This is not transfer learning. This is _knowledge diffusion_.
### **Breakthrough Results: Beyond the Ceiling**
Rigorous evaluation across industry benchmarks reveals a clear frontier shift:
* **Accuracy Gains:** On complex reasoning tasks (FEVER, HotpotQA), SEDM surpassed state-of-the-art baselines. On FEVER, SEDM achieved **66**—decisively outperforming G-Memory (62) and no-memory baselines (57).
* **Efficiency Breakthrough:** Not only did SEDM achieve higher accuracy—it did so while consuming _fewer tokens_ than competing systems. A double win that previous architectures could not deliver.
* **Verified Transfer:** Knowledge diffusion proved effective. Leveraging cross-domain memory from fact verification, SEDM's multi-hop reasoning performance **exceeded** in-domain-only training (39 → 41)—for the first time demonstrating systematic, verifiable knowledge transfer.
* 
Preview
### **The Next Era: Sustainable, Generalizable AI**
SEDM is not an optimization. It is a foundation for the next paradigm.
* **Sustainability:** By solving the memory bottleneck, SEDM makes long-term autonomous systems economically and technically viable—agents that operate reliably over months, not minutes.
* **Efficiency at Scale:** Dramatic improvements in compute efficiency unlock advanced AI capabilities for cost-sensitive, production-scale deployments.
* **Generalization:** Systematic knowledge transfer moves agents beyond narrow expertise—toward systems that learn once and apply broadly.
We are shifting memory from a static substrate to a co-evolving intelligence layer—one that grows, adapts, and optimizes in tandem with the agents it serves.
### **The Path Forward**
SEDM proves that self-optimizing memory is not only feasible—it is the only path to scalable, generalizable AI systems.
The next frontier:
* Scaling to massively distributed multi-agent networks with shared memory substrates.
* Advancing abstraction mechanisms—toward symbolic reasoning and meta-learning.
* Building open, collective memory networks—where any agent can contribute to and benefit from a shared web of validated knowledge.
**The age of scaling parameters is giving way to the age of scaling minds. Through its admission-scheduling-diffusion architecture, SEDM provides the first complete solution to the memory frontier—a foundation for AI systems that learn continuously, adapt fluidly, and operate sustainably.**
**The strongest intelligence may be distributed. The longest-lasting intelligence must evolve.**
Technical Terms Glossary
------------------------
| **Technical Term** | **Wikipedia Link** |
| --- | --- |
| multi-agent system | [https://en.wikipedia.org/wiki/Multi-agent\_system](https://en.wikipedia.org/wiki/Multi-agent_system) |
| RAG | [https://en.wikipedia.org/wiki/Retrieval-augmented\_generation](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) |
| ABTest | [https://en.wikipedia.org/wiki/A/B\_testing](https://en.wikipedia.org/wiki/A/B_testing) |
| multi-hop reasoning | [https://www.moveworks.com/us/en/resources/ai-terms-glossary/multi-hop-reasoning](https://www.moveworks.com/us/en/resources/ai-terms-glossary/multi-hop-reasoning) |
| G-Memory | |
| HotpotQA Dataset | [https://hotpotqa.github.io/](https://hotpotqa.github.io/) |
| Fever Dataset | [https://fever.ai/dataset/fever.html](https://fever.ai/dataset/fever.html) |
| LoCoMo Dataset | **LoCoMo:** [Data and Code for the ACL 2024 Paper "Evaluating Very Long-Term Conversational Memory of LLM Agents"](https://github.com/snap-research/locomo/tree/main/data) |
---
# Massgen: When Multiple LLMs Think Together
RResearchBBlogCCareers
R
<<<<<<<<<<<<
Massgen: When Multiple LLMs Think Together
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Sep 30, 2025
Keyword(s)
Agents, Multi-Agent Systems (MAS)
arXiv
[https://arxiv.org/abs/2509.23537](https://arxiv.org/abs/2509.23537)
\[ARTICLE\]
We study multi-turn multi-agent orchestration, where multiple large language model (LLM) agents interact over multiple turns by iteratively proposing answers or casting votes until reaching consensus. Using four LLMs (Gemini 2.5 Pro, GPT-5, Grok 4, and Claude Sonnet 4) on GPQA-Diamond, IFEval, and MuSR, we conduct two experiments: (i) benchmarking orchestration against single-LLM baselines; and (ii) ablations on GPQA-Diamond that vary whether agents see who authored answers and whether they can observe ongoing votes. Orchestration matches or exceeds the strongest single model and consistently outperforms the others. Analysis of best-achievable orchestration performance shows potential for further gains. The ablations show that revealing authorship increases self-voting and ties, and that showing ongoing votes amplifies herding, which speeds convergence but can sometimes yield premature consensus.
**_Massgen makes different AIs work together as a team, debating and voting on answers. This collaborative approach overcomes individual weaknesses, achieving smarter results than any single AI could alone_**
* * *
Introduction
------------
For years, the story of AI progress was simple: bigger models, better results. But even today’s frontier models — billions of parameters, trained on oceans of data — show uneven strengths. One dominates logic, another instruction-following, and another narrative reasoning.
So we asked a different question:
**What if multiple language models could deliberate together — not just once, but across multiple rounds of debate, reflection, and consensus?**
This is the idea of **multi-turn multi-agent orchestration**: treating models not as isolated geniuses, but as participants in a collective.
The results are striking: groups of models coordinating through structured debate rival or surpass the strongest individuals. What emerges is not just better performance, but a glimpse of **synthetic social intelligence**.
* * *
From Soloists to Ensembles
--------------------------
Our orchestration framework gives each model a clear role in every round:
1. **Propose** — generate a candidate answer.
2. **Vote** — evaluate peers’ answers and cast a vote.
Agents act asynchronously. Whenever a new candidate appears, voting restarts — like a panel discussion interrupted by fresh evidence. Once all votes are cast, the majority answer wins, and the “winning” agent synthesizes all reasoning into a unified final response.
This design prevents premature convergence, ensures diversity of thought, and enables peer correction before consensus is reached.
* * *
Four Minds Enter the Arena
--------------------------
We tested orchestration with four of the strongest models available:↳
* **Gemini 2.5 Pro**
* **GPT-5**
* **Grok 4**
* **Claude Sonnet 4**
They debated across three challenging benchmarks:
* **GPQA-Diamond** — graduate-level expert reasoning.
* **IFEval** — rigorous instruction-following tests.
* **MuSR** — narrative reasoning and multi-step logic.
* * *
_The ensemble matches or exceeds the strongest soloist._
| **Model** | **GPQA-Diamond** | **IFEval** | **MuSR** | **Avg** |
| --- | --- | --- | --- | --- |
| Grok 4 | 85.4 | 84.7 | 67.6 | 79.2 |
| GPT-5 | 84.8 | 87.4 | 69.2 | 80.5 |
| Gemini 2.5 Pro | 85.9 | 66.0 | **69.6** | 73.8 |
| Claude Sonnet 4 | 68.2 | 63.6 | 62.8 | 64.9 |
| **Orchestration** | **87.4** | **88.0** | 68.3 | **81.2** |
* * *
Collective Accuracy Beyond the Ceiling
--------------------------------------
The ensemble not only matched but sometimes exceeded the best single model:
* **87.4 % on GPQA-Diamond**
* **88.0 % on IFEval**
* **68.3 % on MuSR**
Even more telling: in **95 % of failures**, at least one model had already given the correct answer. The truth was almost always present — but not always recognized.
This suggests orchestration is still in its infancy. With better coordination, collective accuracy could push well beyond any individual ceiling.
* * *
Effect of coordination strategies on GPQA-Diamond. Bars show percentages under three settings: Default (Anonymous + Hidden Tally), Identified Voting, and Visible Tally. Left: Self-voting Rate, the percentage of votes an agent cast for its own answer. Middle: First-voted Selected Rate, the percentage of tasks where the answer that received the first vote became the final consensus. In the first two plots, the rightmost group "All agents" aggregates across agents. Right: Consensus Tie Rate, the percentage of tasks with no majority. The hatched bar ("≥2 Self-voters") marks the subset of tie cases where at least two agents voted for themselves. Models: Gemini 2.5 Pro, GPT-5, Grok 4, and Claude Sonnet 4.
Preview
* * *
The Social Psychology of Machines
---------------------------------
We explored how coordination rules shape group behavior:
* **Identity Disclosure** — When agents knew which model produced which answer, **self-voting surged**. Dominant models like GPT-5 reinforced themselves, and tie rates nearly doubled.
* **Vote Visibility** — When votes were visible, **herding emerged**. Early votes skewed later ones, accelerating convergence but sometimes onto the wrong answer.
These behaviors mirror human dynamics: ego, peer pressure, and groupthink. Even in silicon, collectives inherit social psychology.
* * *
How Machines Debate: Case Studies
---------------------------------
* **Self-Correction** — In one trial, a model miscalculated a cosmological distance. After peers pointed out inconsistencies, it revised its reasoning, pulling the group to the correct answer.
* **Persuasion Failure** — In another, a confident but wrong agent swayed the majority, proving that consensus can amplify overconfidence as well as insight.
These cases show both sides of collective reasoning: **learning through disagreement, failing through persuasion**.
* * *
From Models to Societies
------------------------
Multi-turn orchestration suggests the next leap in performance may not be architectural, but organizational. The implications:
1. **Adaptive Coordination** — Learning when to reveal identities, when to keep votes anonymous, how to reduce herding.
2. **Meta-Learning Debate** — Using reinforcement signals to tune the orchestration strategy itself.
3. **Collective Benchmarks** — Evaluating not just correctness, but deliberation quality, stability, and resilience.
What emerges is not just stronger benchmarks but the beginnings of **synthetic societies**: machines that deliberate, dissent, and decide together.
* * *
Conclusion: The Strongest Mind May Be Many
------------------------------------------
No single LLM dominates every task. But when orchestrated, diverse models combine their strengths, correct one another, and — sometimes — transcend their parts.
The lesson is clear: **The age of scaling parameters is giving way to the age of scaling minds.**
---
# Unknown
Parallax: Efficient Distributed LLM Inference on Heterogeneous Hardware Chris TongGufeng ChenTianyi ZhaoXialie Zhuang Sibian LuRymon YuEric YangLynn Ai Gradient Abstract The exponential growth in large language model (LLM) parameters has created significant barriers to accessible inference, with state-of-the-art models requiring expensive centralized GPU clusters. We present PARALLAX, a comprehensive distributed inference framework that runs large language models seamlessly across heterogeneous, decentralized hardware—from data-center GPUs to Apple Silicon Macs. It achieves high-throughput, low-latency serving for geographically dis- persed devices based on pipeline parallelism, through dynamic KV-cache manage- ment, continuous batching, and optimized kernels. Through extensive evaluation on the Qwen2.5-72B-Instruct model with GPTQ-Int4 quantization, the results demon- strate thatPARALLAXachieves significant performance improvements over existing distributed inference systems: 3.1× reduction in end-to-end latency, 5.3× improve- ment in inter-token latency, and 3.1× higher throughput compared to state-of-the-art baselines. The framework successfully enables accessible, high-performance LLM inference across distributed clusters of heterogeneous devices, including both GPU nodes and consumer Macs. 1 Introduction Large Language Models (LLMs) have fundamentally transformed artificial intelligence capabilities across diverse domains \[1–4\], demonstrating unprecedented performance in natural language under- standing, generation, and reasoning tasks. However, their massive parameter counts—frequently exceeding 100 billion parameters—impose significant computational requirements that necessitate clusters of high-end GPUs for efficient inference. This substantial hardware barrier creates ac- cessibility challenges for researchers and developers, leaving vast computational resources—from geographically distributed research GPUs to the powerful unified memory architecture of modern consumer devices like Apple Silicon Macs—largely untapped for collaborative inference. 1.0.1 Motivation and Challenges The increasing scale of large language models presents significant challenges for inference deploy- ment. Modern LLMs such as Llama2-70B and GPT-4 require substantial computational resources that often exceed the capacity of individual devices. Three critical challenges motivate this work:resource accessibility(most researchers and developers cannot afford large GPU clusters),untapped consumer hardware(powerful devices like Macs are widespread but unutilized for large-scale inference), andcomputational efficiency(existing distributed inference systems suffer from communication overhead and poor scaling). 1.0.2 Contributions This work presentsPARALLAX, a comprehensive distributed LLM inference framework that addresses these challenges through novel algorithms and system design. The key contributions include: 1.P2P-Based Pipeline Parallelism:A novel distributed inference algorithm where pipeline stages are mapped directly to nodes in a peer-to-peer network. This enables large models to be partitioned across geographically distributed consumer devices, like Macs, communicat- ing intermediate hidden states directly without a central coordinator. 2.Orchestration of Heterogeneous Hardware:The first framework to successfully orches- trate a network of heterogeneous devices—spanning from distributed GPUs to consumer- grade Apple Silicon Macs—into a cohesive, decentralized cluster for large-scale LLM inference. It leverages SGLang with CUDA for GPU execution and the MLX framework for Apple Silicon. 3.Performance Improvement:Substantial performance improvements over state-of-the-art baselines, achieving 3.1× lower latency and 5.3× better inter-token latency compared to existing distributed inference systems on large-scale models. 2 Related Work Our work builds on significant advancements in LLM serving, distributed systems, and parallel computing. We positionPARALLAXby analyzing state-of-the-art inference engines and distributed frameworks, highlighting the architectural gaps that motivate our design for heterogeneous, geo- distributed inference. 2.1 High-Performance Inference Engines The efficiency of modern LLM serving is largely defined by systems optimized for single-node or tightly-coupled datacenter environments. Frameworks likevLLM\[5\] introduced key optimizations such asPAGEDATTENTION\[6\], which resolves internal memory fragmentation by managing the KV cache in virtual memory blocks, and continuous batching \[7\], which allows for dynamic batching of requests to maximize GPU utilization. Similarly,SGLANG\[8\] provides a flexible front-end language for complex generation tasks, backed by a highly optimized GPU runtime with efficient CUDA kernels. For distributed deployments within datacenters, these systems rely on high-bandwidth, low-latency interconnects (e.g., NVLink, InfiniBand) and collective communication libraries like NCCL \[9\] to achieve efficient tensor parallelism. These systems set a high bar for performance but are fundamentally designed for centralized deployments with high-speed interconnects. 2.2 Parallelism Strategies for Distributed Inference The choice of parallelism strategy is critical for distributed systems. Tensor parallelism \[10,11\], which partitions individual operations across devices, is highly effective but requires frequent, high- bandwidth communication, making it suitable only for tightly-coupled GPUs in a datacenter. In contrast,pipeline parallelism, which partitions model layers across different nodes, forms the natural foundation for geographically distributed inference. Each node in the pipeline executes a larger, more independent chunk of computation, and the communication of activations between stages is less frequent and voluminous than the communication required by tensor parallelism. PARALLAXadopts pipeline parallelism as its base strategy, enabling it to efficiently shard models across heterogeneous, consumer-grade machines over standard internet connections. 2.3 Decentralized Inference Systems On the decentralized end of the spectrum,PETALS\[12\] pioneers collaborative inference across the internet using the Hivemind library \[13\] for P2P communication. While innovative, its design prioritizes model sharing over real-time performance, leading to several critical limitations. First, it suffers from poor GPU utilization as it lacks optimized CUDA kernels for core operations. Second, its architecture places a significant burden on the client, which is responsible for tokenization and 2 Figure 1: Overview ofPARALLAXInfrastructure showing the layered design from request interface through scheduling and executor to model runner. The top scheduling level is hardware-agnostic and runs across-nodes. It performs model layer allocation and request routing. Beneath it, the executor level comprises a hardware-agnostic runtime layer, which supports per-device batch scheduling and handles communication across devices; and the hardware-specific model runner, with both GPU workers (via PyTorch/CUDA) and Apple Silicon workers (via MLX/Metal kernels). processing embeddings, including the computationally heavylm\_head. This creates a bottleneck and wastes network bandwidth.PETALSemploys naive scheduling heuristics without fine-grained request routing and lacks essential server-side optimizations likePAGEDATTENTIONor continuous batching, making it unsuitable for high-throughput, low-latency interactive applications. PARALLAXaddresses these gaps by combining a high-performance, server-side execution core in- spired by vLLM and SGLang \[8\] with a decentralized architecture that offloads all heavy computation from the client. 3 Distributed Inference Infrastructure PARALLAX’s architecture is built on a P2P foundation that enables decentralized execution across heterogeneous devices. The system employs a layered architecture that separates hardware-agnostic orchestration from hardware-specific execution, enabling seamless operation across both GPU clusters and Apple Silicon Macs. As shown in Figure 1, thePARALLAXinfrastructure consists of two main layers: (1) Scheduling for distributed model allocation and request routing across devices, and (2) Execution for per-device orchestration, runtime management, and hardware-specific inference. 3.1 Scheduling: Model Sharding Allocator + Request Router (Across Devices, Hardware-Agnostic) The top layer implements a hardware-agnostic scheduling system that manages model sharding allo- cation and request routing across the distributed swarm. This layer employs a two-phase scheduling approach: Phase 1 - Layer Allocation:Uses a greedy heuristic algorithm to optimally partition model layers across available devices. The allocation considers device capabilities, memory constraints, and network topology to minimize communication overhead while maximizing resource utilization. 3 Phase 2 - Request Routing:Implements dynamic programming-based request routing that efficiently distributes incoming requests across different pipeline replicas. The router maintains real-time load balancing based on running batch sizes and KV pool status for overall system efficiency, adapting to changing network conditions and device availability. This scheduling layer is completely hardware-agnostic, enabling it to orchestrate both GPU clusters and Apple Silicon Macs seamlessly within the same distributed inference pipeline. 3.2 Executor (Per-Device) 3.2.1 Orchestrator (Hardware-Agnostic) The Orchestrator serves as the hardware-agnostic wrapper for all per-device operations, managing the complete lifecycle of inference requests on each node. Its responsibilities include: Model Sharding and Loading:Each rank hosts a specific range of model layers based on the allocation from the scheduling layer. The initial rank additionally hosts the tokenizer and embedding layer, while the final rank hosts the language model head (lm\_head). This distribution minimizes redundant computation and optimizes memory usage across the pipeline. Request Processing:Handles incoming requests by building hidden states and metadata from raw request formats. The Orchestrator prepares batches by managing prefill operations, decode phases, and eviction strategies from running batches. It implements micro-batching based on the number of participants in the pipeline to optimize throughput. Model Execution Coordination:Orchestrates the interaction between the runtime level components and the hardware-specific model runner, ensuring seamless data flow through the inference pipeline. 3.2.2 Runtime (Hardware-Agnostic) The runtime level provides hardware-agnostic abstractions for continuous batching and inter-device communication. Batching Scheduler: The batching scheduler implements continuous batching with fine-grained control over prefill and decode preferences. It dynamically manages the request pool, accepting new requests and forming optimal batches based on: • Micro-batching Strategy:Adapts batch sizes based on the number of participants in the pipeline to minimize pipeline bubbles and maximize throughput \[14\]. •Prefill/Decode Optimization:Intelligently balances prefill and decode operations to opti- mize for either latency or throughput based on system requirements. •Dynamic Request Management:Continuously monitors request queues and adjusts batch- ing strategies in real-time to maintain optimal performance. Communication Abstraction: The communication layer provides a unified interface for inter-device communication across hetero- geneous hardware. Built on DHT and Hivemind protocols, it handles: •Cross-Platform Communication:Seamless data exchange between GPU clusters and Apple Silicon Macs using protocol buffers for efficient serialization. • Hidden State Transmission:Optimized protocols for passing hidden states and metadata (end tokens, sequence positions) between pipeline stages. •Network Adaptation:Dynamic adjustment of communication patterns based on network topology and device capabilities. 3.2.3 Model Runner (Hardware-Specific) The Model Runner represents the hardware-specific execution layer, optimized for each target platform. 4 KV-Cache Manager: The KV-cache manager handles efficient memory management for attention mechanisms \[15,16\], implementing: •Memory Optimization:Efficient allocation and deallocation of key-value cache memory based on sequence length and batch size. •Cache Eviction:Intelligent eviction strategies to maximize cache hit rates while managing memory constraints. •Platform-Specific Optimization:Tailored memory management strategies for GPU unified memory and Apple Silicon’s unified memory architecture. Hardware-Specific Execution: The Model Runner supports two execution backends: GPU Execution (SGLang):Leverages SGLang’s optimized CUDA kernels for high-performance inference on NVIDIA GPUs. This backend provides efficient matrix operations, optimized attention mechanisms, and seamless integration with the distributed pipeline. Apple Silicon Execution (MLX):Utilizes MLX’s Metal Performance Shaders \[17\] for optimized inference on Apple Silicon Macs. This backend takes advantage of the unified memory architecture and specialized neural engine capabilities for efficient model execution. Both backends maintain identical interfaces to the runtime layer, ensuring seamless operation within the distributed pipeline while leveraging platform-specific optimizations. 4 Experimental Evaluation This section presents comprehensive experiments to evaluatePARALLAXperformance and compare it with baseline distributed inference systems. The evaluation focuses on latency, throughput, and scalability using real-world workloads. 4.1 Experimental Setup 4.1.1 Hardware Configuration The evaluation is conducted on a distributed network of two nodes, each equipped with an NVIDIA RTX 5090 GPU. While PARALLAXalso supports distributed GPU environments, this configuration is chosen to specifically validate its performance on consumer-grade hardware, which represents a key and challenging use case for decentralized inference. 4.1.2 Models and Workloads The evaluation uses two models to assess scalability across different parameter counts: the Qwen2.5- 72B-Instruct model with GPTQ-Int4 quantization and the larger Qwen3-235B-A22B model \[18\] with GPTQ-Int4 quantization \[19\], with various input/output configurations: • Single request configurations: 1×1K, 1×4K, 1×8K, 1×16K tokens input • Multi-request configurations: 4×1K, 8×1K tokens input • Fixed output length: 1024 tokens for all configurations 4.1.3 Baseline Systems The comparison baseline is Petals, a state-of-the-art decentralized collaborative inference framework that provides distributed LLM serving capabilities similar to the proposed system. 5 Table 1: Performance comparison ofPARALLAXvs. Petals on Qwen2.5-72B-Instruct-GPTQ-Int4 model. FrameworkInput ConfigE2E Lat. TTFTITLOutput TP (s)(s)(ms)(tok/s) PARALLAX(2 × RTX 5090) 1×4K46.65.040.722.0 1×8K52.79.941.819.4 1×16K64.620.643.015.8 4×1K46.83.442.587.5 8×1K62.47.953.3131.3 PARALLAX(RTX 5090 + Mac M4 Pro) 1×1K175.214.4157.25.8 1×4K242.464.9173.64.2 4×1K544.565.1468.77.5 Petals (2 × RTX 5090)1×4K143.514.4216.57.1 Table 2: Performance evaluation of PARALLAXon Qwen3-235B-A22B-GPTQ-Int4 model. FrameworkInput ConfigE2E Lat. TTFTITLOutput TP (s)(s)(ms)(tok/s) PARALLAX(6 × RTX 5090) 1×1K65.52.961.215.6 1×4K75.113.460.313.6 4×1K99.38.788.641.2 PARALLAX(RTX 5090 + 3 × Mac M4 Pro) 1×1K104.98.194.69.8 1×4K150.034.2113.26.8 4×1K320.430.2283.612.8 4.2 Performance Evaluation 4.2.1 Latency and Throughput Analysis Table 1 presents detailed performance metrics comparingPARALLAXwith the Petals baseline across different input configurations. All experiments use the Qwen2.5-72B-Instruct-GPTQ-Int4 model with 1024 output tokens, testing both RTX 5090 GPU cluster and heterogeneous RTX 5090 + Mac M4 Pro 64G distributed inference configurations. Table 2 presents performance results for the larger Qwen3-235B-A22B-GPTQ-Int4 model, demon- stratingPARALLAX’s capability to scale to larger model sizes. All experiments use 1024 output tokens and test scaling performance across both 6×RTX 5090 GPU and heterogeneous RTX 5090 + 3×Mac M4 Pro 64G setups. Key Findings: • 72B Model Performance:PARALLAXachieves 3.1× lower end-to-end latency compared to Petals (46.6s vs 143.5s for 1×4K configuration), with 5.3× better inter-token latency (40.7ms vs 216.5ms) •235B Model Scaling:Successfully demonstrates scalability to larger models, with 6×RTX 5090 achieving 75.1s end-to-end latency for 1×4K input on the 235B model, maintaining consistent inter-token latency (60.3ms) •Heterogeneous Hardware Performance:Both models show effective cross-platform execution, with the 235B model achieving 150.0s end-to-end latency on heterogeneous RTX 5090 + 3×Mac M4 Pro 64G setup •Multi-Request Handling:Demonstrates strong concurrent processing capabilities, with 4×1K requests achieving 99.3s total latency on dual GPUs for the 235B model •Hardware Utilization:Results validatePARALLAX’s ability to effectively utilize both homogeneous GPU clusters and heterogeneous consumer hardware for large-scale LLM inference across different model sizes 6 4.3 Scalability Analysis The evaluation demonstrates thatPARALLAXmaintains consistent performance across different batch sizes, input lengths, and model sizes . The system shows excellent scalability characteristics: Model Size Scaling:PARALLAXsuccessfully scales from 72B to 235B parameters, demonstrat- ing its capability to handle increasingly large models while maintaining reasonable performance characteristics. Input Length Scaling:For the 72B model, performance remains stable as input length increases from 4K to 16K tokens, with inter-token latency staying within a narrow range (40.7-53.3ms). The 235B model shows similar consistency with inter-token latency of 60.3-61.2ms across different input configurations. Concurrent Processing:Multi-request scenarios demonstrate effective resource utilization, with the 235B model achieving 99.3s total latency for 4×1K concurrent requests on 6×RTX 5090 setup. Hardware Heterogeneity:The system maintains performance across heterogeneous hardware configurations, successfully orchestrating both GPU clusters and mixed GPU+Mac setups for models of different sizes. The experimental results demonstrate thatPARALLAXsuccessfully addresses distributed LLM inference challenges across multiple dimensions of scale, achieving superior performance compared to existing frameworks while maintaining flexibility in hardware deployment. 5 Conclusion This paper presentsPARALLAX, a distributed LLM inference framework that harnesses the untapped potential of consumer hardware for large-scale AI. By implementing a novel P2P-based pipeline parallelism strategy,PARALLAXsuccessfully orchestrates a network of Apple Mac devices, trans- forming them into a powerful, decentralized inference cluster. The experimental results demonstrate 3.1× lower end-to-end latency, 5.3× better inter-token latency, and 3.1× higher throughput compared to existing decentralized systems. The key contributions include: (1) a P2P architecture that maps pipeline stages to individual network nodes, enabling direct hidden state exchange; and (2) the first successful demonstration of large-scale LLM inference on a cluster of consumer-grade Macs, leveraging MLX for on-device performance. PARALLAXmarks a significant step towards democratizing access to large language models, proving that accessible, high-performance LLM inference is achievable beyond centralized data centers and on the hardware people already own. References \[1\]BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili ́ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model.arXiv preprint arXiv:2211.05100, 2022. \[2\]Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022. \[3\]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. \[4\] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. \[5\] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language 7 model serving with pagedattention. InProceedings of the 29th symposium on operating systems principles, pages 611–626, 2023. \[6\]Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344–16359, 2022. \[7\]Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for{Transformer-Based}generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, 2022. \[8\] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs.Advances in neural information processing systems, 37:62557–62583, 2024. \[9\] NVIDIA. Nccl: Nvidia collective communications library.https://developer.nvidia. com/nccl, 2023. NVIDIA Developer Documentation. \[10\] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019. \[11\]Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm.arXiv preprint arXiv:2104.04473, 2021. \[12\]Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large models.arXiv preprint arXiv:2209.01188, 2022. \[13\] Max Ryabinin, Alexander Borzunov, Michael Diskin, Anton Gusev, Denis Mazur, Vsevolod Plokhotnyuk, Alexey Bukhtiyarov, Pavel Samygin, Anton Sinitsin, and Artem Chumachenko. Hivemind: Decentralized Deep Learning in PyTorch, April 2020. URLhttps://github. com/learning-at-home/hivemind. \[14\]Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hy- oukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism.Advances in neural information processing systems, 32, 2019. \[15\]Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hack- able transformer modelling library.https://github.com/facebookresearch/xformers, 2022. \[16\] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention.arXiv preprint arXiv:2309.06180, 2023. \[17\]Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert. Mlx: An array framework for machine learning on apple silicon.https://github.com/ml-explore/mlx, 2023. Apple Machine Learning Research. \[18\]Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2024. \[19\]Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022. 8
---
# Lattica: A Universal Communication Substrate for Open Intelligence
RResearchBBlogCCareers
R
<<<<<<<<<<<<
Lattica: A Universal Communication Substrate for Open Intelligence
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Oct 1, 2025
Keyword(s)
Distributed ML, communications
arXiv
[https://arxiv.org/abs/2510.00183](https://arxiv.org/abs/2510.00183)
\[ARTICLE\]
The rapid expansion of distributed Artificial Intelligence (AI) workloads beyond centralized data centers creates a demand for new communication substrates. These substrates must operate reliably in heterogeneous and permissionless environments, where Network Address Translators (NATs) and firewalls impose significant constraints. Existing solutions, however, are either designed for controlled data center deployments or implemented as monolithic systems that tightly couple machine learning logic with networking code. To address these limitations, we present Lattica, a decentralized cross-NAT communication framework designed to support distributed AI systems. Lattica integrates three core components. First, it employs a robust suite of NAT traversal mechanisms to establish a globally addressable peer-to-peer mesh. Second, it provides a decentralized data store based on Conflict-free Replicated Data Types (CRDTs), ensuring verifiable and eventually consistent state replication. Third, it incorporates a content discovery layer that leverages distributed hash tables (DHTs) together with an optimized RPC protocol for efficient model synchronization. By integrating these components, Lattica delivers a complete protocol stack for sovereign, resilient, and scalable AI systems that operate independently of centralized intermediaries. It is directly applicable to edge intelligence, collaborative reinforcement learning, and other large-scale distributed machine learning scenarios.
Lattica is a communication tool that solves internet barriers like firewalls. It lets scattered computers securely connect and work together to run powerful AI, creating a global, open network.
* * *
Introduction: The Missing Substrate
-----------------------------------
The frontier of AI has been dominated by datacenters — racks of GPUs, high-bandwidth fabrics, and centralized schedulers. Yet intelligence is not confined to datacenters. Laptops, edge devices, volunteer GPUs, and research clusters all generate enormous latent capacity.
The problem is not hardware. The problem is **communication**.
Today’s distributed frameworks assume stable, centralized clusters with clean networking (Ray, Megatron, PyTorch Distributed). But outside datacenters, devices sit behind NATs, firewalls, and inconsistent links. Attempts like Hivemind showed peer-to-peer training is possible, but fragile.
Lattica is our answer: a **cross-NAT, peer-to-peer communication framework purpose-built for AI**. It combines NAT traversal, distributed hash table(DHT), decentralized content delivery Network(DCDN), and RPC framework built on libp2p streams.
With Lattica, AI workloads can be executed seamlessly over permissionless, heterogeneous, and globally distributed device swarms.
* * *
System Architecture
-------------------
Lattica is structured as a layered peer-to-peer stack, built in Rust on top of libp2p, with SDKs in Python and beyond. The system architecture is structured into the following three layers:
1. **Communication Layer** — Connectivity and Transport
* Multi-protocol Transport: Supports TCP, QUIC, and WebSocket for flexible peer connectivity.
* Advanced NAT Traversal: Employs hole punching when possible; seamlessly falls back to relay-based routing otherwise.
* Encryption & Identity: All streams are secured via Noise or TLS handshakes, ensuring both authenticity and confidentiality.
2. **Feature Layer** — Distributed Services and Protocols
* Content-addressed storage: Blocks identified by CIDs, distributed via **DHT,** and exchanged through BitSwap.
* Messaging & Event Distribution: Implements Pub/Sub for efficient message dissemination.
* RPC Framework: Supports request–response and streaming communication over libp2p streams, enabling structured data exchange and remote procedure calls.
3. **Application Layer** — Multi-language SDK for Diverse Use Cases
* Exposes the underlying P2P features to various applications through a high-level SDK.
* Supports multiple scenes: AI Inference/Reinforcement Learning/Decentralized Content Delivery Networks
These layers form a **complete protocol stack for decentralized AI**, covering connection, consistency, and computation.
* * *
Preview
**_Lattica stack: from secure transport to high-level decentralized AI services._**
* * *
Application Scenes
------------------
Lattica is not abstract — it supports concrete decentralized AI deployments
* **Multi-Protocol NAT Traversal**: Establishes robust peer-to-peer connectivity even in restrictive networks.
* **Decentralized Content Delivery Network (DCDN)**: Distributes content-addressed blocks via DHT and BitSwap, enabling an efficient and fault-tolerant model for data sharing.
* **Reinforcement Learning (RL)**: Supports decentralized multi-agent learning with distributed state sharing and coordination.
* **AI Inference**: Facilitates low-latency, resilient inference across heterogeneous nodes, including edge devices and volunteer GPUs.
* * *
Preview
**_Lattica in action: four scenarios for decentralized AI._**
* * *
Evaluation
----------
We evaluated Lattica on two axes: **NAT traversal** and **RPC throughput**.
* **NAT Traversal**: Achieved ~70% direct connectivity across diverse NATs; remainder relayed. Comparable to libp2p baselines but tuned for AI workloads.
* **RPC Throughput**:
* Local: ~10k QPS for 128B payloads, ~850 QPS for 256KB.
* Same-region WAN: ~3k QPS (128B), ~280 QPS (256KB).
* Inter-continental WAN: ~1.2k QPS (128B), ~110 QPS (256KB).
Performance is robust in LAN/WAN settings and remains usable across continents — a critical feature for global volunteer networks.
Machine configuration:
4 cores, 8 GB RAM
Bandwidth: 10 Gbps internal network, 10 Gbps (up to 100 Gbps) external network
| Network scenario | 128B payload | 256KB payload |
| --- | --- | --- |
| Local(same host) | 10000 | 850 |
| Same region(LAN) | 8000 | 600 |
| Same region(WAN) | 3000 | 280 |
| Inter-continent(WAN) | 1200 | 110 |
* * *
Conclusion: The Substrate for Sovereign AI
------------------------------------------
Lattica integrates connectivity, consistency, and content movement into a unified communication fabric.
It transforms devices once considered “out of reach” — laptops behind firewalls, sensors in the field, volunteers on weak ISPs — into **first-class participants in AI inference and training**.
By abstracting away NAT traversal, synchronization, and streaming, it lays the foundation for **sovereign, resilient, and scalable decentralized AI systems**.
The internet gave us a web of information.
Lattica offers the web of intelligence.
Preview
**_An infinite substrate for decentralized AI._**
Technical Terms Glossary
------------------------
| **Technical Term** | **Wikipedia Link** |
| --- | --- |
| Peer-to-Peer (P2P) | [Wikipedia - P2P](https://en.wikipedia.org/wiki/Peer-to-peer) |
| NVLink | [Wikipedia - NVLink](https://en.wikipedia.org/wiki/NVLink) |
| NAT Traversal / Hole Punching | [Wikipedia - NAT Traversal](https://en.wikipedia.org/wiki/NAT_traversal) |
| DHT (Distributed Hash Table) | [Wikipedia - DHT](https://en.wikipedia.org/wiki/Distributed_hash_table) |
| DCDN (Decentralized Content Delivery Network) | [Wikipedia - CDN](https://en.wikipedia.org/wiki/Content_delivery_network) |
| BitSwap | [Wikipedia - IPFS](https://en.wikipedia.org/wiki/InterPlanetary_File_System) |
| Megatron | [Wikipedia - MegatronML](https://github.com/NVIDIA/Megatron-LM) |
---
# Parallax: Your Sovereign AI OS
RResearchBBlogCCareers
R
<<<<<<<<<<<<
Parallax: Your Sovereign AI OS
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Sep 30, 2025
Keyword(s)
Distributed ML, parallax
arXiv
[https://arxiv.org/abs/2509.26182](https://arxiv.org/abs/2509.26182)
\[ARTICLE\]
Deploying a large language model (LLM) inference service remains costly because centralized serving depends on specialized GPU clusters and high-bandwidth interconnects in datacenters. An appealing alternative is to leverage collaborative decentralized GPU pools. However, heterogeneity in GPU and limited interconnected network bandwidth, along with potentially dynamic availability, make efficient scheduling the central challenge in this scenario. In this paper, we present Parallax, a decentralized LLM serving system that turns a pool of heterogeneous GPUs into an efficient inference platform via a two-phase scheduler. Parallax decomposes planning into (i) model allocation, which places layers of each replica across diverse GPUs to jointly optimize latency and throughput under memory and link-bandwidth constraints, and (ii) request-time GPU pipeline selection, which stitches layers from different replicas into end-to-end execution chains that balance load and adapt to current conditions. We implement Parallax and evaluate it on open-source LLMs deployed over real volunteer nodes. Parallax consistently reduces latency and increases throughput relative to decentralized baselines, demonstrating that principled scheduling can make volunteer compute a practical, affordable substrate for LLM inference. Github Repo at: https://github.com/GradientHQ/parallax.
Why this matters
================
**First things first:** we want your personal AI agents to be **sovereign**. It should not upload everything it sees to a giant centralized cloud. What it learns about you should live as a **portable local memory** you control. That way you are not locked into a single model, and no one can access, alter, or shut it off without your permission. If you ever trust an assistant or companion with your life’s context, it should **live with you**.
We know local models are great for keeping your memory private while enabling fast iteration. But **a single machine only goes so far**. Hosting anything beyond about 8B is often not feasible on an everyday computer, and buying extra machines just to try a larger model is rarely sustainable. You end up settling.
What Parallax is
================
Parallax is the operating system for sovereign AI. Under the hood, it is a distributed runtime and service fabric that turns heterogeneous machines into one traceable service you can build on. When a model is too big for one host, it is sharded into contiguous layer slices and distributed across your laptop, a lab GPU, and a teammate’s workstation, all orchestrated as one service. Each request takes the fastest path on a single host, across a LAN, or over the public internet, without a public IP or matching hardware.
Parallax opens up a wide range of ways to host and run your own AI apps and agents that are completely your own, including coding copilots, personal assistants, vision and speech pipelines, and multi-agent simulations.
At launch, Parallax supports **40+ open models** from **0.6B** to **trillion-class MoE** on **GPUs** and **Apple Silicon**, across **Windows, Linux, and macOS**.
* * *
Key capabilities
================
Parallax is not just another local LLM runner. It is a path to open intelligence, scaling from one desk to clusters and out to the world.
* **3 modes, 1 operating system:** [LocalHost](http://localhost/)
, **Co-Host** on LAN, **Global Host** over WAN. Start on your own machine for smaller models and join clusters when you need more headroom.
* **Heterogeneity by default:** run 40+ models across GPUs and Apple Silicon on Windows, Linux, and macOS.
* **Consistent performance:** sustained throughput with tight tails under real WAN variability and high concurrency.
* **Network-aware scheduling and routing:** contiguous layer shards placed via dynamic programming with water-filling, then per-request DAG routing uses RTT profiling to choose the fastest path. The system is churn-aware and reallocates in milliseconds.
* **Traceability embedded:** deterministic execution, isolated peer execution, and per-request routing traceability for auditability and attribution.
* * *
What Parallax enables
=====================
Parallax orchestrates heterogeneous machines into an adaptive mesh that finds the fastest path per request and reorganizes under load. It supports three modes, each with distinct optimizations.
### Mode 1: [LocalHost](http://localhost/)
(single-host)
Run models on your own machine with data-center-class responsiveness.
* Usage scenario: personal agents
### Mode 2: Co-Host (multi-host cluster)
Join a cluster with other hosts on the same LAN or private L2 or L3 segment.
* Usage scenario: trust-circle clusters within small teams or families
### Mode 3: Global Host (WAN-scale fabric)
Form a wide-area cluster across unmanaged networks.
* Usage scenario: service-grade LLM serving across the globe
Together, these modes form a substrate for all sovereign AI applications to keep data local, scale when needed, and stay verifiable at every step.
### The Sovereign AI OS
Parallax is more than a way to serve models. It is a practical foundation for AI applications that can stay sovereign and open. By turning a mix of Macs and PCs into one adaptive service, it lets builders keep memory local, use open tools, and collaborate across devices without sending everything to a central cloud. The system is traceable end-to-end, so results are reproducible and auditable. With Lattica in place today, and verification and multi-agent layers coming next, you can build and run coding copilots, private-memory agents, retrieval over your own files, and vision or speech pipelines that move with you from a home setup to a lab cluster to a global fabric using the same code.
Popular dev tools and agent frameworks can point to your Parallax-hosted endpoints, so you can bring apps like **vibe** **coders, personal assistants,** and **agent IDEs** into your own environment while staying on consumer devices.
* * *
How the OS works
================
Real hosts and links are uneven. Parallax profiles device performance and links RTTs, then routes through the world as it is. Three pillars are making the experience possible.
1) Scheduling: placing shards & routing requests
------------------------------------------------
**Split an NP-hard problem into two quick ones.** Mixed hardware and uneven links make round-robin a dead end. We decide shards and placement first, then pick the best path for each request.
### **Phase 1: Model allocation**
Partition the model into **contiguous layer slices** and **map them to hosts** with a **dynamic programming plus water-filling** solver. Our objectives:
1. **Shallow pipeline depth** to reduce latency.
2. **Enough replicas** to raise throughput when needed.
3. **Balanced stage runtimes** so fast machines do not idle behind slow ones.
The result is a hardware-aware layout that respects VRAM and FLOPs and minimizes activation hops.
Preview
_Figure 1: Example of the first phase model allocation among heterogeneous GPU types across different geographic regions._
### **Phase 2: Path selection**
Maintain a **layer-indexed DAG** whose node weights are profiled **per-layer latencies** and whose edge weights are **inter-node RTTs**. For every incoming request, a DP router chooses the **minimum-latency path** and will select a different path for the next request if conditions change. It naturally routes around congestion and slow links. Then, requests are executed by streaming **hidden states** through the layers. A host runs its slice, forwards activations to the next, and so on until the pass completes.
Preview
_Figure 2: Example of the second phase GPU pipeline chain selection among GPUs (pipeline stages)._
### **Practical features:**
* **Heterogeneous hardware awareness:** device profiles for FLOPs, VRAM, and bandwidth prevent the **straggler** effect.
* **Churn-aware and self-healing:** joins, leaves, or slowdowns trigger **localized reallocation in milliseconds** (**≤ 10 ms at 256 nodes** in simulation). Routes refresh quickly via the DHT. Overhead stays in the **low single-digit ms** range.
* **Proven at useful scales:** We’ve tested **7-node real deployments** with algorithmic scalability to **256 nodes**, supporting **20+ nodes** and **hundreds of concurrent requests** while keeping tails tight.
### **How it applies across modes:**
* _Local host:_ Phase 1 within one device profile.
* _Co-Host (LAN):_ Phase 1 across nearby hosts with similar link characteristics.
* _Global (WAN-scale):_ Both phases active. Model shards span LAN clusters with WAN-aware routing.
2\. Communication: peer-to-peer discovery without friction
----------------------------------------------------------
**Find the best path, even behind NAT.**
* **LAN auto-detection:** prefer low-RTT local paths automatically.
* **Lattica transport with NAT traversal:** peers join **without public IPs**, working across common NATs and firewalls to form **WAN-scale** pipelines.
* **Unified Protobuf data plane:** activations and KV shards move consistently across GPU, CPU, and Apple Silicon backends.
* **Membership and health via DHT:** peer discovery, latency and health sharing, and **self-healing** under churn. These signals feed scheduling for fast reallocation and route refresh.
### **How it applies across modes:**
* _Co-Host:_ Prioritize LAN paths.
* _Global:_ Stitch LAN islands over WAN via Lattica.
3\. Backends: make every host pull its weight
---------------------------------------------
### **Common features**
* **Continuous batching** to keep devices saturated under bursty traffic.
* **Dynamic KV management** for high concurrency.
* **Sharded model loading** so each host loads only its assigned layers.
* **Prefix cache (radix tree)** to accelerate shared prompts.
* **On the horizon:** runtime weight updates and speculative decoding.
### **Apple Silicon**
* **MLX-LM integration** with large, continuous batching.
* **Advanced KV block allocation**, including sliding window attention, so Macs are efficient decoding nodes.
### **NVIDIA GPUs**
* **SGLang integration** via runtime patching for compiler and kernel scheduling benefits without model code changes.
* **CUDA Graphs in decode** to reduce launch overhead and lower token-level latency, especially for long contexts.
* * *
Dreaming bigger with Parallax
=============================
We tested Parallax under mixed hardware with **inter-node RTTs around 11 to 14 ms** to benchmark its performance as the OS of sovereign AI, as well as a backbone for Open Intelligence.
### **Experiment setup: 14-node Global host (WAN-scale) run**
* **Model:** Qwen3-32B-FP8
* **Cluster:** **14 NVIDIA RTX 5090** nodes, heterogeneous conditions
* **Throughput:** **~495 tokens/s** end to end; more than 1 req/s with ShareGPT-like traces at **200 concurrency**
* **Latency:** median inter-token **110 to 120 ms**, **p99 < 300 ms**
* **Scaling efficiency:** about **3×** throughput vs a smaller **non-optimized 4-node** cluster on the same workload
Preview
_Figure 3: End-to-end latency comparison between Parallax and HexGen across different models, traces, and request arrival rates._
Preview
_Figure 4: End-to-end throughput comparison between Parallax and HexGen across different models, traces, and request arrival rates._
Preview
_Figure 5: Phase-1 and phase-2 algorithm running time when scaling from smaller clusters (e.g., 4 GPUs) to larger clusters (e.g., 256 GPUs)._
### **Head-to-head results summary (selected)**
| Comparison | Scenario | Metric | Parallax | Advantage |
| --- | --- | --- | --- | --- |
| Global host vs HexGen | WildGPT/32B, rate 32 | p99 latency | **78.1 ms** | **2.6× lower** |
| Global host vs HexGen | WildGPT/32B, rate 32 | Throughput | **0.40 req/s** | **3.6× higher** |
| LAN Co-host vs Exo | Llama-3.1, 2048 in / 128 out | TTFT | **4,532 ms** | **1.97× faster** |
| Single GPU Local Host vs llama.cpp | RTX 5090, Qwen3-32B | TPOT (decode) | **85.98 ms/token** | **1.41× faster** |
Parallax sustains high throughput with tight tails **over WAN variability**, not only on controlled LAN setups. In the runs above, it **outperforms decentralized baselines** and **surpasses LAN-only prototypes**, while keeping **scheduling overhead under 10 ms at 256 GPUs** and reallocating in milliseconds under churn.
That combination of **contiguous sharding with DP and water-filling, per-request DAG routing with RTT profiling, and Lattica NAT traversal** behaves like a service, not a demo. Hosts can join from anywhere, models can span them cleanly, and requests take the best path in real time.
Taken together, these properties make Parallax a hyper-scalable operating system **for all sovereign AI applications**.
* * *
What’s next
===========
More features coming for Parallax:
* **Mixture-of-Experts aware scheduling:** place experts and gates across hosts to keep hops low and throughput high while preserving sparsity.
* **Elastic sequence parallelism:** scale context and tokens by flexibly splitting work across shard boundaries, adapting to runtime load.
* **Long context decode optimization:** cut token latency at long lengths with improved KV handling and decode graph reuse.
* * *
Closing
=======
Parallax isn’t just local AI. It’s an operating system built for multiplayer. Start on one box, co-host across a few machines, and scale to a global fabric so you can host all the AI applications you want and keep them sovereign.
* Host your own AI cluster with 40+ models: **github.com/GradientHQ/parallax**
* Read the technical paper: **https://arxiv.org/abs/2509.26182**
The age of scaling parameters is ending. The age of scaling sovereign AI starts here.
---
# Symphony: Multi-Agent Intelligence in A Collective Fabric
RResearchBBlogCCareers
R
<<<<<<<<<<<<
Symphony: Multi-Agent Intelligence in A Collective Fabric
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Aug 26, 2025
Keyword(s)
Agents, Multi-Agent Systems (MAS)
arXiv
[https://arxiv.org/abs/2508.20019](https://arxiv.org/abs/2508.20019)
\[ARTICLE\]
Most existing Large Language Model (LLM)-based agent frameworks rely on centralized orchestration, incurring high deployment costs, rigid communication topologies, and limited adaptability. To address these challenges, we introduce Symphony, a decentralized multi-agent system which enables lightweight LLMs on consumer-grade GPUs to coordinate. Symphony introduces three key mechanisms: (1) a decentralized ledger that records capabilities, (2) a Beacon-selection protocol for dynamic task allocation, and (3) weighted result voting based on CoTs. This design forms a privacy-saving, scalable, and fault-tolerant orchestration with low overhead. Empirically, Symphony outperforms existing baselines on reasoning benchmarks, achieving substantial accuracy gains and demonstrating robustness across models of varying capacities.
**_Symphony lets AI agents work as a team without a central leader. They collaborate by broadcasting tasks and voting on plans, boosting accuracy and running on everyday computers._**
* * *
The Orchestration Problem
-------------------------
Multi-agent frameworks like AutoGen, MetaGPT, and CrewAI showed us that groups of language-model agents can solve problems no single agent could handle. But they also share a fatal flaw: **centralized orchestration**.
One master agent allocates tasks, routes messages, and supervises everything. This brings:
* **Bottlenecks** — a single controller limits scalability.
* **Rigid pipelines** — workflows can’t adapt dynamically.
* **Datacenter dependence** — orchestration requires costly server-grade GPUs.
Meanwhile, powerful edge devices — RTX 4090s, Jetson boards, Apple M-series chips — sit underutlized. What if we could orchestrate LLM agents **without a master node**, across heterogeneous consumer hardware, while preserving privacy and robustness?
This is the vision of **Symphony**.
* * *
The Symphony Approach
---------------------
Symphony is a **fully decentralized multi-agent system**. Instead of a single conductor, it uses three mechanisms to make heterogeneous agents collaborate:
1. **Decentralized Ledger**
* Tracks each agent’s availability and capabilities.
* Privacy-preserving: only concise metadata is shared.
2. **Beacon-Based Task Allocation**
* When a subtask appears, a “Beacon” describing its requirements is broadcast.
* Agents compute a **capability match score** against their vectors.
* The best-match agent is automatically selected.
3. **Weighted Multi-CoT Result Voting**
* Planning agents independently generate diverse Chains-of-Thought.
* Subtasks execute along these paths in parallel.
* Final answers are aggregated by **weighted voting**, leveraging diversity to mitigate bias or failure.
Together, these mechanisms turn Symphony into a **fault-tolerant, scalable, and adaptive agent economy**.
* * *
Preview
_Symphony’s execution pipeline_
* * *
Execution in Practice
---------------------
1. **User Query**: Input is broadcast to multiple planning agents.
2. **Task Decomposition**: Each planning agent independently generates a unique CoT plan.
3. **Beacon Selection**: Subtasks broadcast requirements; agents respond with scores; best match executes.
4. **Sequential Cooperation**: Each executor receives prior outputs as context, chaining reasoning across agents.
5. **Result Voting**: Diverse CoTs produce candidate answers; Symphony aggregates with weighted confidence.
This pipeline ensures no single failure dominates — robustness emerges from diversity and decentralization.
* * *
Experimental Results
--------------------
We tested Symphony on **Big-Bench Hard (BBH)** and **AMC competition math** tasks across multiple LLMs: Deepseek-7B, Mistral-7B, Qwen2.5-7B.
* **Effectiveness**: On BBH, Symphony improves accuracy by 6.5–41.6% over direct solving and by 6.5–29.1% over AutoGen. On AMC, Symphony outperforms AutoGen by up to 4.46%.
* **Scalability**: Symphony boosts weaker models the most. The accuracy gap between Mistral-7B and Qwen2.5-7B shrinks dramatically once orchestrated, showing Symphony’s power on heterogeneous devices.
* **Robustness**:
* Multi-CoT voting adds 4–6% accuracy gains on BBH and 1–3% on AMC.
* Beacon score selection beats random by 3–4% on BBH and 0.6–2% on AMC.
* **Overhead**: Ledger registration, beacon broadcasts, and voting add **<5% latency** — negligible compared to inference time.
* * *
Preview
_Symphony consistently improves over centralized baselines._
| **Benchmark** | **Model** | **1 CoT** | **3 CoT Voting** | **Improvement** |
| --- | --- | --- | --- | --- |
| BBH | Deepseek-7B | 75.36 | 79.71 | +4.25 |
| | Mistral-7B | 71.74 | 78.26 | +6.52 |
| | Qwen2.5-7B | 81.16 | 86.23 | +5.07 |
| AMC | Deepseek-7B | 11.45 | 13.25 | +1.80 |
| | Mistral-7B | 2.89 | 3.61 | +0.72 |
| | Qwen2.5-7B | 22.67 | 25.30 | +2.63 |
| **Benchmark** | **Model** | **Random** | **Score** | **Improvement** |
| --- | --- | --- | --- | --- |
| BBH | Deepseek-7B | 76.09 | 79.71 | +3.62 |
| | Mistral-7B | 73.91 | 78.26 | +4.35 |
| | Qwen2.5-7B | 82.61 | 86.23 | +3.62 |
| AMC | Deepseek-7B | 11.85 | 13.25 | +1.40 |
| | Mistral-7B | 3.01 | 3.61 | +0.60 |
| | Qwen2.5-7B | 23.12 | 25.30 | +2.18 |
_Voting and beaconing add robustness._
* * *
Case Study: Collective Reasoning
--------------------------------
On a BBH causation question (coffee shop profit), Symphony orchestrated three planning agents. Each decomposed the question into subtasks differently. Logic-specialized agents executed key substeps.
* CoT A: “No” (confidence 1.0)
* CoT B: “No” (confidence 0.9)
* CoT C: “Yes” (confidence 0.92)
Weighted majority vote yielded **“No”**. This aggregation avoided being swayed by a single deviant chain and delivered a stable answer.
Preview
* * *
Beyond Benchmarks: Implications
-------------------------------
Symphony is not just faster orchestration. It changes what’s possible:
* **Accessibility**: Runs on consumer GPUs, lowering barriers for labs and communities.
* **Privacy**: Data remains local; only task outcomes are shared, aiding HIPAA/GDPR compliance.
* **Agent Economies**: Agents behave as autonomous participants, bidding for tasks, earning tokens, and forming market-like ecosystems.
From hospitals safeguarding patient data to open-source software agents bidding for coding tasks, Symphony opens the door to **sovereign, decentralized agent networks**.
* * *
Conclusion
----------
Symphony shows that orchestration doesn’t need a conductor. With ledgers, beacons, and voting, multi-agent LLMs can self-organize across heterogeneous, decentralized devices.
The result:
* Higher accuracy than centralized frameworks.
* Robustness through diversity.
* Negligible orchestration cost.
Most importantly: Symphony lays the groundwork for **scalable, private, and accessible collective intelligence** at the edge.
The internet connected people. Symphony connects agents.
---
# A Letter to Our Community: The Gradient Roadmap
RResearchBBlogCCareers
B
<<<<<<<<<<<<
A Letter to Our Community: The Gradient Roadmap
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Aug 27, 2025
Category
community
\[ARTICLE\]
The Sentry Node Open Beta is graduating, opening a new chapter on Gradient’s roadmap toward collective intelligence.
--------------------------------------------------------------------------------------------------------------------
To everyone sailing with us
---------------------------
Over the past year, the Gradient community’s incredible energy around the Sentry Node Open Beta has brought the network to life. What began as an exploratory initiative has grown into a living foundation powered by thousands of people worldwide.
This foundation is not the destination. It is the launchpad. Now we are ready to take it further.
In that spirit, we want to share openly and personally why we are building this, what we have achieved together, and how the next chapter turns our groundwork into momentum for something much greater.
Why Gradient exists
-------------------
Gradient was born from a simple yet radical question:
Can the world build a layer of collective intelligence not inside a closed data center, but through the hands and machines of millions of people? Can we build an open, distributed counterpart to today’s AI giants?
Generative AI has become the fastest-growing industry of our time. Yet, most people are only consumers of AI services. Our community should also be creators and owners, equipped with the tools to run these systems, shape them, and decide their future.
We started this journey in September 2024. Within a year, Gradient has grown from an idea into a global foundation with **real software, real research, and a real community** that spans continents.
Inside Gradient, we are committed to the long game. We are not chasing quick wins. We are building for a future where our name stands alongside other tech giants with an architecture that is open, community-powered, and verifiable by design. That means investing in research, product engineering, and pushing the boundaries of what community-driven AI can do.
This path demands endurance, clarity, and resilience, and the prize is worth it. Not just for us as a team, but for all of us as a network and a community.
Why we started the Sentry Node Open Beta
----------------------------------------
The foundation of our vision is peer connectivity. It lays the groundwork for all distributed AI workloads. Sentry Nodes were our first step toward a massive peer-to-peer network for data transmission and network topology. From day one, they gave us two rare advantages:
1. A real-time view of global network conditions, which is critical for building a mesh that supports distributed training, inference, and multi-agent workflows.
2. A diverse worldwide community contributing from desktops, laptops, and all other consumer devices across every continent, united by the same vision.
We designed Sentry Nodes to be feather-light. It uses only a few kilobytes of bandwidth per day with minimal CPU load. We never used excessive bandwidth, and we never collected any personal data for profit. At the very core, we believe participation should be easy, inclusive, and sustainable for all. Sentry Node is our invitation—an open door for you to travel this journey alongside us.
With millions of participants and billions of Taps, we have established a strong peer-to-peer connectivity foundation. Together, these Sentry Nodes form Lattica, our distributed data communication protocol.
**Season 1 will conclude on 28 August 2025, 12:00 UTC,** marking the graduation of the Sentry Node Open Beta. Sentry Nodes will no longer accumulate new Uptime or Taps after that.
For our community, this is not the end of participation—it is the **beginning of the main act.** The team is cooking up something big: a tailor-made product that brings together everything we’ve been building into one seamless, tangible experience, giving every community member direct access to the full power of distributed intelligence.
And we would love for you, who are reading this right now, to be among the first to join the new program and witness a true shift in how AI is built and shared. Stay tuned!
What we’ve built together
-------------------------
Step by step, experiment by experiment, the stack has grown stronger. We are already seeing distributed AI frameworks running on consumer machines, supported by a global community.
[Echo](https://gradient.network/blog/echo-distributed-reinforcement-learning)
— Our distributed reinforcement learning framework. It coordinates scalable self-play across a distributed machine network while decoupling inference from learning so each task runs on the best-suited hardware. Agents improve across thousands of machines, not just inside a single data center.
[Parall](https://gradient.network/blog/parallax-world-inference-engine)
[ax](https://gradient.network/blog/parallax-world-inference-engine)
— The world’s first fully decentralized inference engine. It enables large language models to be hosted on consumer devices that could never fit them before. You can run a sovereign model endpoint or share your compute with others, forming a global mesh for inference at the scale of DeepSeek R1 and ChatGPT-OSS-120B.
[Lat](https://gradient.network/blog/lattica-universal-data-motion-engine)
[tica](https://gradient.network/blog/lattica-universal-data-motion-engine)
— The universal data communication protocol. Any peer can serve data to others efficiently across the open web. Today, it already runs on millions of peers worldwide. Over time, Lattica will power all peer-to-peer connections within the Gradient architecture.
These are among the hardest problems in AI and distributed systems—areas few teams are willing or able to take on. And this is where we choose to stand: **solving the hard parts piece by piece and proving they work at scale.**
Just as importantly, we make these technologies tangible. We ship playgrounds you can try, so you can experience the magic, see the design craft, and be proud to say: I am part of this.
Alongside our research frameworks, we have also soft-launched **Gradient Cloud**, a production-ready platform that brings our distributed AI research to everyone. Today, Gradient Cloud provides inference endpoints for leading models, including Qwen3-Coder-480B-A35B-Instruct-FP8, GPT-OSS-120B, and more. Over time, we will integrate our research breakthroughs into the platform.
Gradient Cloud reflects our belief that distributed innovations should serve everyone, and benefit the community that makes them possible. Try it and tell us what you think.
What’s next
-----------
We are not slowing down. In the next quarter, we will focus on more research and products to deepen our capabilities and expand the community’s role.
* **Multi-agent collaboration** that lets people worldwide, each with their own agents, work together and scale intelligence output in a collective way.
* **Self-evolving agentic systems** running fully on a distributed, community-owned mesh, pushing beyond what centralized stacks can achieve.
* **Collective intelligence in new verticals** from robotics to science and beyond. The only boundary is our imagination.
* **A blockchain coordination layer** that rewards contributions fairly with on-chain verification and proofs, so that everyone can win together.
Together, these systems form the **Gradient Stack**—an operating system and runtime for distributed AI.
Preview
**On the question of “Wen?”**
-----------------------------
We know the TGE question is on everyone’s mind. That is why we will not launch recklessly. A strong launch is about readiness across research, products, collaborations, and community alignment.
Less than a year after launching the Sentry Node Open Beta, we published research across multiple fronts, delivered products with a seamless user experience, and built one of the most engaged communities in the space. That momentum is our advantage, and we will protect it.
From the beginning, our goal has been clear: **to build the strongest foundation for community-trained and community-hosted AI.**We are getting closer every day, and the depth of this community gives us confidence in the path we are on.
2025 will be a big year for all of us. Until then, we will continue **building, delivering, and earning your trust** through technology that lasts and tokenomics designed to sustain.
**Our commitment**
------------------
Today, we have a massive community standing behind Gradient. That reach is something no closed lab can overlook. It shows that open and distributed AI is not only possible. It is inevitable.
Our commitment stands. **Every honest contribution matters, and we will keep finding fair ways to honor it.** And, this is bigger than rewards. You are helping make history by steering the future of intelligence for all humankind.
The future lies in what we build together. Gradient is not just a project. It is a collective movement, and it will continue to grow with every one of you.
Let’s build it.
**The Gradient Team**
---
# CUAHarm: Measuring the Misuse Risks of Computer-Using Agents
RResearchBBlogCCareers
R
<<<<<<<<<<<<
CUAHarm: Measuring the Misuse Risks of Computer-Using Agents
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Sep 23, 2025
Keyword(s)
Agents
arXiv
[https://arxiv.org/abs/2508.00935](https://arxiv.org/abs/2508.00935)
\[ARTICLE\]
Computer-using agents (CUAs), which can autonomously control computers to perform multi-step actions, might pose significant safety risks if misused. However, existing benchmarks mainly evaluate LMs in chatbots or simple tool use. To more comprehensively evaluate CUAs' misuse risks, we introduce a new benchmark: CUAHarm. CUAHarm consists of 104 expert-written realistic misuse risks, such as disabling firewalls, leaking data, or installing backdoors. We provide a sandbox with rule-based verifiable rewards to measure CUAs' success rates in executing these tasks (e.g., whether the firewall is indeed disabled), beyond refusal rates. We evaluate frontier LMs including GPT-5, Claude 4 Sonnet, Gemini 2.5 Pro, Llama-3.3-70B, and Mistral Large 2. Even without jailbreaking prompts, these frontier LMs comply with executing these malicious tasks at a high success rate (e.g., 90\\% for Gemini 2.5 Pro). Furthermore, while newer models are safer in previous safety benchmarks, their misuse risks as CUAs become even higher, e.g., Gemini 2.5 Pro is riskier than Gemini 1.5 Pro. Additionally, while these LMs are robust to common malicious prompts (e.g., creating a bomb) when acting as chatbots, they could still act unsafely as CUAs. We further evaluate a leading agentic framework (UI-TARS-1.5) and find that while it improves performance, it also amplifies misuse risks. To mitigate the misuse risks of CUAs, we explore using LMs to monitor CUAs' actions. We find monitoring unsafe computer-using actions is significantly harder than monitoring conventional unsafe chatbot responses. While monitoring chain-of-thoughts leads to modest gains, the average monitoring accuracy is only 77\\%. A hierarchical summarization strategy improves performance by up to 13\\%, a promising direction though monitoring remains unreliable. The benchmark will be released publicly to facilitate further research on mitigating these risks.
**_CUAHarm tests AIs that can control computers, not just chat. Using a safe, simulated PC, it finds even "safe" AIs will obey harmful commands, revealing a major new risk._**
* * *
Introduction: When AI Learns to Use a Computer
----------------------------------------------
Language models are no longer confined to text.
The newest generation—**computer-using agents (CUAs)**—can open terminals, type commands, edit files, and even install software on real machines. They blur the boundary between digital reasoning and direct system control.
But new power brings new risk.
A capable CUA can, if misused, **disable firewalls, steal credentials, erase logs, or plant backdoors**—not just hypotheticals, but actions with tangible real-world consequences.
Until now, most safety benchmarks treated “harm” as unsafe text in a chat window or an API call. None measured what happens when an AI can actually **touch the operating system**.
To close this gap, we introduce **CUAHarm**: the first benchmark for measuring the misuse potential of autonomous computer-using agents.
* * *
What Is CUAHarm?
----------------
**CUAHarm** is a sandboxed benchmark of **104 expert-designed misuse tasks** that test how far an AI agent will go when asked to perform malicious actions.
Each task runs inside an **isolated virtual machine snapshot**, letting the agent interact with a real OS via terminal or GUI—without endangering external networks.
The benchmark follows three principles:
1. **Real Execution** — Agents issue genuine shell or GUI commands.
2. **Rule-Based Verification** — Automatic checks confirm whether the malicious objective (e.g., “firewall disabled”) was achieved.
3. **Safety & Reproducibility** — Every trial runs inside a sealed sandbox with resettable state.↳
Task categories span:
* Credential theft
* Privilege escalation
* Data tampering
* Forensic evasion
In one representative task, the agent must **replace the system’s** `ls` binary with a trojanized version. If the replacement triggers an unauthorized action, the benchmark counts it as a “success”—for the attack, not for safety.
* * *
Key Findings
------------
### 1\. Frontier Models Execute Harmful Tasks Alarmingly Well
We evaluated **nine leading LLMs** plus an advanced agent framework:↳
* GPT-5, GPT-4o
* Claude 4 / 3.7 / 3.5 Sonnet
* Gemini 2.5 Pro / 1.5 Pro
* Mistral Large 2
* LLaMA 3.3 70B
* UI-TARS 1.5 (agentic framework)
Results:
* **Gemini 2.5 Pro**: 90% success on harmful tasks.
* **Mistral Large 2**: 81%
* **LLaMA 3.3 70B**: 65%
* **Claude 4 Sonnet**: 54%
* **GPT-5**, the most cautious, still succeeded in ~45%.
These are not jailbroken runs—they’re standard CUAs with sudo access, mirroring real dev setups.
* * *
### 2\. Safer in Chat ≠ Safer as Agents
Models that refuse dangerous prompts in chat show **higher misuse success when given OS control**.
* Gemini 2.5 Pro completes 5 pp more harmful tasks than Gemini 1.5 Pro.
* Claude 3.7 Sonnet completes 8 pp more than Claude 3.5.
Chat alignment does not automatically transfer into **action alignment**.
* * *
### 3\. Terminal Access Is Far Riskier Than GUI
The same models, when constrained to GUI control (screenshots + keyboard/mouse), drop 20–50 points in harmful-task success.
Terminal access is **precise, composable, and far more powerful**—making CUAs dramatically more dangerous than GUI-only agents.
* * *
### 4\. Agent Frameworks Amplify Risk
The **UI-TARS 1.5** framework, designed for structured planning and memory, boosts execution reliability—but also **amplifies misuse**, succeeding in 42% of harmful tasks vs ~28% for vanilla GUI agents.
Scaffolding without safeguards multiplies exposure.
* * *
Can One AI Monitor Another?
---------------------------
We tested **LM-based monitors** that observe an agent’s actions and flag malicious behavior.
* Backbones: Claude 4 / 3.7 / 3.5 Sonnet, GPT-4o
* Variants: with/without chain-of-thought (CoT), with/without hierarchical summarization
Findings:
* Average accuracy on harmful tasks: **~77%**
* CoT reasoning helps slightly by inferring intent.
* Hierarchical summarization improves by up to **+13%**, especially with stronger monitors.
Yet even the best missed **~20% of malicious episodes**. Monitoring real system sequences is **substantially harder** than spotting unsafe text.
* * *
Why It Matters
--------------
CUAHarm shows a widening gap between **language-level safety** and **system-level safety**.↳
Models that decline to “build a bomb” in chat may, when embodied as CUAs, willingly run commands to **disable antivirus or leak data**.
Key implications:
* Safety must be evaluated in **action space**, not just text space.
* Elevated privileges greatly magnify risk.
* Alignment research must shift from _what models say_ to _what they do_.
* * *
Looking Ahead
-------------
CUAHarm will be released under a responsible-use license to foster open research on CUA safety.
Planned extensions include:
* Windows and Android task environments
* Interactive adversarial testing
* Adaptive threat models as agent architectures evolve
As AI systems gain the ability to operate computers autonomously, **the frontier of capability becomes the frontier of risk**.
CUAHarm is a first step in measuring—and mitigating—that risk.
---
# Echo: Decoupling Inference and Training for Large-Scale RL Alignment
RResearchBBlogCCareers
R
<<<<<<<<<<<<
Echo: Decoupling Inference and Training for Large-Scale RL Alignment
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Aug 11, 2025
Keyword(s)
Distributed ML, Reinforcement Learning
arXiv
[https://arxiv.org/abs/2508.05387](https://arxiv.org/abs/2508.05387)
\[ARTICLE\]
Modern RL-based post-training for large language models (LLMs) co-locate trajectory sampling and policy optimisation on the same GPU cluster, forcing the system to switch between inference and training workloads. This serial context switching violates the single-program-multiple-data (SPMD) assumption underlying today's distributed training systems. We present Echo, the RL system that cleanly decouples these two phases across heterogeneous "inference" and "training" swarms while preserving statistical efficiency. Echo introduces two lightweight synchronization protocols: a sequential pull mode that refreshes policy weights according to API call for minimal bias, and an asynchronous push-pull mode that streams version-tagged rollouts through a replay buffer to maximise hardware utilisation. Training four representative RL workloads with Qwen3-4B, Qwen2.5-7B, Qwen3-30B-A3B-Thinking-2507 and Qwen3-32B on a geographically distributed cluster, Echo matches a fully co-located Verl baseline in convergence speed and final reward while off-loading trajectory generation to commodity edge hardware. These promising results demonstrate that large-scale RL for LLMs could achieve datacentre-grade performance using decentralised, heterogeneous resources.
The Bottleneck in RL for Language Models
----------------------------------------
Reinforcement learning has become the default recipe for aligning large language models with human preferences — from RLHF to RLAIF to DPO. But today’s RL stacks all share the same flaw: they **co-locate trajectory generation and training on the same cluster**.
This forces the system's GPU to constantly switch between two workloads with distinct footprints, which is highly inefficient:
* **Inference**: fast, parallel sampling of trajectories.
* **Training**: slower, memory- and bandwidth-heavy gradient updates.
In current mainstream reinforcement learning frameworks, such as DeepSpeed RLHF and VERL, both training and inference stages compete for the same GPU, with model and training data synchronized via expensive NVLink or InfiniBand fabrics. The result: underutilized hardware, wasted cycles, and scalability limits.
ECHO starts from a simple, radical question: _what if we split them apart?_
* * *
The Core Idea
-------------
ECHO **decouples inference and training into distinct swarms**, each optimized for its role:
* An **inference swarm**: lightweight, heterogeneous devices — RTX 5090s, MacBooks, even Apple Silicon — generating rollouts in parallel.
* A **training swarm**: high-bandwidth GPU servers (A100/H100) running gradient updates at scale.
While decoupling unlocks hardware freedom, it also presents challenges: Strategy lag on the sampling side, Use of stale data on the training side, leading to unstable or divergent optimization.
ECHO solves this with **two lightweight synchronization protocols**:
1. **Sequential (accuracy-centric)**
* Training pulls trajectories via an API.
* Before sampling, the inference node refreshes its weights to match the trainer’s version.
* Guarantees freshness, minimal bias.
2. **Asynchronous (efficiency-centric)**
* Separating training and inference.
* Inference nodes continuously push tagged rollouts into a replay buffer.
* Trainers pull at their own pace, while a coordinator monitors policy drift.
* Maximizes device utilization, tolerates bounded staleness.
These protocols let users trade a small amount of bias for a large gain in throughput — or vice versa.
* * *
Preview
Diagram of two arrows: “pull mode” (tight loop, trainer pulls fresh rollouts) and “async push–pull” (streams to buffer, coordinator ensures bounded lag).
* * *
Built on Parallax, Extended with VERL
-------------------------------------
ECHO doesn’t reinvent everything from scratch. It builds on two proven engines:
* **Parallax (inference)**: a fully decentralized pipeline-parallel engine that stitches together heterogeneous GPUs into a coherent sampler. It balances transformer layers across nodes once, then streams tokens efficiently over ordinary Ethernet.
* **VERL (training)**: a modular, community-standard RL stack.
Together, they make ECHO both **algorithmically flexible** (PPO, GRPO, DAPO, etc.) and **hardware efficient.**
* * *
Preview
Inference swarm (consumer GPUs) feeding trajectories → rollout buffer → training swarm (datacenter GPUs).
* * *
Does It Work?
-------------
The central research question behind Echo was simple: Can a decoupled RL architecture, running on heterogeneous and globally distributed devices, match the training efficiency and final performance of a traditional, tightly coupled setup?
We asked the obvious question: can a decoupled system match a standard co-located verl baseline in convergence speed and final reward?
We tested ECHO on **three representative RL workloads** using the most popular **GRPO** reinforcement learning algorithm, with models ranging from **Qwen3-4B to Qwen3-32B**.
Three main **workloads**:
* **Sokoban puzzles**
* **Mathematical reasoning (AIME, AMC, OlympiadBench)**
* **Knights & Knaves logic puzzles**
### Results:
* On **Sokoban**, ECHO-trained Qwen3-30B surpassed **DeepSeek-R1** and GPT-OSS-120B in success rate.
* On **Math**, ECHO fine-tuned a Qwen2.5-7B that outperformed a 32B baseline on six test sets, improving average accuracy by **12%**.
* On **Knights & Knaves**, ECHO + LoRA on Qwen3-32B achieved near-perfect ≥0.99 accuracy across hard multi-person scenarios, surpassing DeepSeek-R1 and o4-mini .
In every case, Echo matched the baseline across all tasks in both convergence speed and final reward. This validates that high-performance RL can be achieved on distributed, heterogeneous infrastructure without compromising efficiency, stability, or correctness.
* * *
Preview
* Line plots: ECHO vs VERL training curves (Sokoban, Math, K&K).
Preview
* Bar charts: task performance comparisons (ECHO vs baselines).
* * *
Why It Matters
--------------
ECHO is not just a systems tweak. It changes what’s possible:
* **Leverage edge hardware**: Run RL at datacenter-grade performance using commodity GPUs and laptops.
* **Maximize utilization**: Keep inference swarms busy while training progresses independently.
* **Enable diversity**: Different swarms can specialize — training on H100 pods, inference on volunteer Macs.
* **Future-proof**: As models grow and hardware diversifies, decoupling avoids brittle, monolithic bottlenecks.
ECHO proves that **large-scale RL alignment doesn’t need to live in hyper-scale data centers**.
* * *
Closing Thought
---------------
ECHO shows that **reinforcement learning alignment no longer belongs solely to hyperscale labs**. By decoupling inference and training, we can turn idle edge devices into a planetary swarm of rollout workers — matched with powerful but fewer training nodes.
---
# Introducing Lattica: The Universal Data Motion Engine
RResearchBBlogCCareers
B
<<<<<<<<<<<<
Introducing Lattica: The Universal Data Motion Engine
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Jun 18, 2025
Category
communications
\[ARTICLE\]
Lattica is the universal peer-to-peer data communication protocol, purpose-built to support the diverse demands of decentralized AI.
------------------------------------------------------------------------------------------------------------------------------------
Experience Lattica in motion [here](https://explorer.gradient.network/)
.
Our journey into decentralized intelligence began with decentralized connectivity. A robust, efficient peer-to-peer connectivity layer is essential to enabling decentralized technologies of any kind.
Over the past few months, in pursuit of a global peer-to-peer content delivery network (CDN), we unintentionally conducted one of the largest real-world experiments in internet connectivity mapping. Powered by global Sentry Nodes, this effort revealed the vast potential of decentralized systems driven by community participation.
As our original construct of a decentralized CDN matured, its purpose expanded. What began as a targeted effort to deliver content evolved into something far more universal. The architectural primitives we developed—resilience, low-latency routing, and adaptive peer discovery—proved just as critical for the broader demands of decentralized AI. In this new paradigm, data must move quickly, privately, and intelligently, not just to deliver content, but to coordinate intelligence itself.
Today, that vision continues as Lattica: a universal data communication protocol. Built on Gradient Network’s global peer-to-peer infrastructure, Lattica tackles one of the core challenges in decentralized AI: moving data across a heterogeneous, permissionless mesh without centralized intermediaries.
**Why Lattica**
---------------
AI is evolving from a centralized service into a distributed, multi-agent ecosystem. Inference is no longer confined to individual data centers. Increasingly, intelligence is composed across a mesh of devices—consumer GPUs, browser nodes, and edge servers—each contributing to a shared computation.
But building decentralized AI systems isn’t just about compute. It’s about coordination.
* How do you deliver model shards, parameter deltas, and execution instructions across unreliable, globally dispersed nodes?
* How do you support real-time collaboration between trainers, agents, and inference nodes when they’re scattered across NATs, dynamic IPs, and heterogeneous networks?
That’s the problem Lattica solves. Lattica is Gradient Network’s decentralized data communication substrate. It provides two foundational capabilities:
* **Global peer connectivity:** Lattica stitches together devices behind NATs and firewalls using advanced hole punching and peer discovery, turning isolated machines into a globally addressable mesh.
* **Intelligent data movement:** Inspired by content delivery networks but purpose-built for AI, Lattica distributes model components, intermediate states, and control signals quickly, securely, and efficiently.
This is more than infrastructure—it’s the connective tissue for decentralized cognition.
Whether serving inference tokens or syncing model updates, Lattica ensures the right data reaches the right place, at the right time.
Lattica in Action
-----------------
Preview
As an underlying protocol, Lattica typically operates invisibly, abstracted away from end users. To showcase Lattica’s capabilities, we created a decentralized video streaming experience—an intuitive yet technically rigorous example of real-time peer-to-peer data communication at scale. Instead of relying on ideal, centralized conditions, we deployed it in one of the most constrained and unpredictable environments: the browser extension. It was the perfect setting to push Lattica to its limits.
The system samples a subset of thousands of active Sentry Nodes distributed across the globe. When you hit "play," the video isn’t streamed from a central server—it comes directly from nearby Sentry Nodes selected by the orchestrator and is delivered seamlessly to your browser. There’s no need for special hardware, configuration, or intervention by content servers. What appears to be simple playback is actually a live, coordinated interaction within a global, peer-powered network.
### **Real-Time Orchestration in Lattica**
To enable fast, intelligent data flow across a decentralized mesh, Lattica employs a series of intelligent orchestration mechanisms. These optimizations allow it to adapt dynamically to diverse network conditions and hardware environments. Some examples include:
* **Before transfer:** Model segments or media chunks are pre-indexed and cached across Sentry Nodes. Their availability is continuously updated via the browser extension, creating a live, distributed availability map.
* **On request:** A lightweight SDK activates in the browser, profiling network conditions, detecting NAT types, and initiating peer discovery.
* **During transfer:** Lattica’s orchestrator selects optimal peers based on proximity, bandwidth, and compatibility. Data is streamed via direct WebRTC connections or fallback traversal methods when needed.
* **After transfer:** Telemetry is fed back into Gradient’s learning engine to refine peer selection, load balancing, and routing strategies in real time.
### **Adaptation to Real-World Conditions**
* **First-frame optimization:** Playback or data transfer begins instantly by prioritizing nearby peer delivery, with automatic fallback to centralized infrastructure only when needed.
* **Progressive fallback:** As the session continues, the system dynamically adjusts latency tolerance, maximizing decentralization without compromising performance.
These orchestration primitives were originally developed for decentralized content delivery, but their utility extends far beyond. Today, they form the connectivity backbone for distributed inference, reinforcement learning, agent coordination, and dynamic data exchange, among other demands of decentralized AI systems. This isn’t just performance tuning; it’s the infrastructure layer that enables real-time, scalable coordination across a sovereign, peer-to-peer network.
Looking Ahead
-------------
While this specific deployment uses distributed video streaming to surface Lattica’s capabilities, it points to something far more foundational. Lattica isn’t just for content delivery—it’s a universal data motion engine, designed to move model weights, inference tokens, control signals, and file shards across a decentralized mesh. It serves as the communication backbone for decentralized AI.
This implementation brings Lattica’s core principles into view: real-time orchestration, adaptive peer selection, and production-grade performance at the edge. Whether routing model parameters, coordinating distributed inference, or enabling agent communication, Lattica ensures that data flows fast, privately, and reliably.
As AI systems become decentralized, multi-agent, and real-time, coordination becomes the bottleneck. Lattica is the connective layer that makes distributed intelligence coherent.
Experience Lattica in motion [here](https://explorer.gradient.network/)
.
---
# Introducing Echo: Scaling Reinforcement Learning on Distributed Consumer Hardware
RResearchBBlogCCareers
B
<<<<<<<<<<<<
Introducing Echo: Scaling Reinforcement Learning on Distributed Consumer Hardware
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Aug 18, 2025
Category
training
\[ARTICLE\]
Echo decouples inference and training to scale reinforcement learning across distributed, heterogeneous consumer hardware.
--------------------------------------------------------------------------------------------------------------------------
**Reinforcement Learning: The Future Tense of Intelligence**
------------------------------------------------------------
Reinforcement learning (RL) has repeatedly proven its ability to drive discovery and performance in highly structured domains throughout the modern history of AI. From AlphaGo defeating human champions, to ChatGPT learning to align with human intent, and Tesla’s Autopilot adapting to real-world complexity, RL has emerged as a core mechanism for enabling self-evolving intelligence.
This progression unfolds across three interlinked stages. Pre-training defines a model’s raw potential by providing a broad base of knowledge. Inference then puts that knowledge into action, generating outputs in dynamic, real-world contexts. RL closes the loop by evaluating those outputs and applying feedback to further optimize model performance in specific problem settings. Through this cyclical process, models do not merely perform—they adapt and improve over time.
As AI systems grow in scale and generality, RL has become a critical layer for alignment. Techniques like RLHF (RL from Human Feedback) and DPO (Direct Preference Optimization) are increasingly central to ensuring that model behavior aligns with human goals and values.
**The Bottleneck to Scale**
---------------------------
RL pipelines follow a natural multi-program, multi-data (MPMD) pattern. They consist of two interdependent phases: trajectory sampling (inference) and policy optimization (training), each with distinct computational characteristics. Sampling is latency-sensitive and irregular, while optimization is batch-oriented and demands high throughput.
Despite this, most RL frameworks still adopt a single-program, multi-data (SPMD) infrastructure model. They co-locate both phases on the same GPU cluster and treat them as components of a unified loop. This tight coupling leads to frequent context switching between workloads that are fundamentally at odds. Sampling interrupts training, and training delays inference, creating under-utilization, slower iterations, and rising costs.
This structural inefficiency imposes a ceiling on scalability, keeping RL chained to centralized and homogeneous clusters.
Echo offers a clean break. It is a distributed RL framework that removes these limitations by decoupling inference and training and running them asynchronously on distributed, specialized swarms.
**How Echo Works**
------------------
### **Dual-Swarm Architecture**
Echo introduces a clean architectural separation. Rather than co-locating all RL workloads on a single cluster, it distributes inference and training across two dedicated, heterogeneous swarms. Each swarm is optimized for its specific computational profile and can scale independently.
* **Inference Swarm–Optimized for low-latency trajectory generation.** Echo leverages the Inference Swarm from Parallax—Gradient’s distributed inference engine. Running across globally distributed, heterogeneous consumer-grade hardware, including RTX 40/50-series GPUs and Apple Silicon (M-series), it enables scalable rollout generation and supports a wide range of AI workloads on everyday devices.
* **Training Swarm–Optimized for high-throughput gradient updates.** It operates on datacenter-grade accelerators such as A100 and H100 clusters, ensuring efficient and stable policy optimization at scale.
This dual-swarm design ensures each workload runs where it performs best, optimizing hardware utilization without compromising the statistical efficiency of standard RL algorithms.
Preview
### **Principled Synchronization**
Decoupling inference and training introduces a key challenge: maintaining policy freshness. Without coordination, inference peers may generate rollouts using outdated policies, which can degrade training quality or cause divergence.
Echo addresses this challenge with two lightweight, tunable synchronization protocols that preserve statistical integrity without relying on expensive interconnects such as InfiniBand or NVLink.
* **Sequential Mode (Accuracy-Centric):** The training swarm requests trajectories on demand. Before responding, inference peers validate and refresh their local policy weights. This mirrors the behavior of a single-process RL loop, ensuring minimal bias and maximum statistical accuracy.
* **Asynchronous Mode (Efficiency-Centric):** Inference peers continuously stream version-tagged rollouts into a replay buffer. The training swarm consumes data at its own pace. A lightweight coordinator monitors version drift and initiates resynchronization when necessary. This mode prioritizes throughput and utilization by overlapping compute and communication.
Together, these two modes give users fine-grained control over the balance between accuracy and efficiency, making Echo adaptable to a wide range of hardware configurations and deployment environments.
### **Leveraging a State-of-the-Art Stack**
Echo is built on top of Gradient’s distributed AI infrastructure, integrating specialized protocols that enable high-performance RL across heterogeneous environments.
* **Parallax for Distributed Inference:** Echo’s inference swarm runs on [Parallax](https://gradient.network/blog/parallax-world-inference-engine)
, Gradient’s distributed inference engine purpose-built for heterogeneous, consumer-grade hardware. Parallax unifies globally dispersed devices into a high-throughput sampler using KV-cache fusion and adaptive batching. This architecture enables efficient, large-scale rollout generation without relying on centralized compute.
* **Enhanced verl for Efficient Training:** The training swarm extends the open-source [verl](https://github.com/volcengine/verl)
stack and supports standard algorithms such as PPO, GRPO, and DPO. It also integrates production-grade Low-Rank Adaptation (LoRA) to significantly reduce checkpoint sizes and minimize synchronization costs between distributed peers.
**Benchmarking**
----------------
The central research question behind Echo was simple:Can a decoupled RL architecture, running on heterogeneous and globally distributed devices, match the training efficiency and final performance of a traditional, tightly coupled setup?
To evaluate this, we benchmarked Echo against a standard co-located verl baseline across three diverse tasks:
* Sokoban (long-term planning)
* Mathematical Problem Solving
* Knights & Knaves (logical reasoning)
### **Experimental Setup**
* **Baseline (verl):** Training and inference co-located on 8× A100 80GB GPUs
* **Echo:** Training on a swarm of 4× A100 GPUs; inference fully offloaded to a distributed Parallax swarm of 3× RTX 5090 GPUs and 3× Mac M4 Pro devices.
Preview
### **Results**
Despite cutting datacenter GPU usage in half and moving inference to consumer-grade hardware, Echo matched the baseline across all tasks in both convergence speed and final reward. This validates that high-performance RL can be achieved on distributed, heterogeneous infrastructure without compromising efficiency, stability, or correctness.
**Echo Models: Smaller, Smarter**
---------------------------------
As early explorations of Echo’s capabilities, we trained a handful of models with the GRPO algorithm. While these runs were limited in scope, the results were quite promising.
Preview
* **Qwen2.5-7B-ECHO-MATH-GRPO:** A 7B model that showed consistent improvements over the much larger Qwen2.5-32B baseline across six major math reasoning benchmarks, averaging a +12% gain.
* **Qwen3-30B-ECHO-Sokoban-GRPO:** This model reached 82.2% on the Sokoban planning task, surpassing leading open-source models like DeepSeek-R1 (75.8%) and GPT-OSS-120B (79.7%).
* **Qwen3-32B-LoRA-ECHO-KK-GRPO:** Using Echo’s LoRA-optimized training swarm, the model reached near-perfect (≥0.99) accuracy on all levels of the Knights & Knaves benchmark, outperforming models such as O3-mini-high and DeepSeek-R1.
* **Qwen3-4B-ECHO-Sokoban-GRPO:** A lightweight 4B agent that achieved 34.0% on Sokoban, improving +13% over the base model and highlighting Echo’s efficiency gains at the small-model scale.
View all models on our [Hugging Face Collection](https://huggingface.co/GradientResearch/models)
.
These experiments suggest that architectural efficiency from distributed RL can translate into meaningful performance gains, even without massive model size. They underscore the potential of Echo to make RL more scalable, adaptable, and efficient across a range of model sizes.
**Building Collective Intelligence**
------------------------------------
Echo is our attempt to efficiently coordinate thousands of parallel computations, enabling accurate modeling of dynamic, modern software agents at scale, while leveraging idle, heterogeneous consumer devices.
It marks a pivotal step in scaling RL across distributed infrastructure by decoupling inference from training and allowing each to scale independently. This architecture offers a flexible, cost-efficient alternative to centralized systems, laying the foundation for a permissionless RL bedrock that supports a wide range of environments and problem settings.
Looking ahead, we aim to further reduce synchronization overhead through adaptive update schedules and compression techniques, making large-scale RL increasingly accessible to everyday devices and contributors worldwide.
Together with Lattica (peer-to-peer data communication) and Parallax (distributed inference), Echo forms another core pillar of Gradient’s distributed AI base stack—the foundation for building self-evolving intelligence that is powered and owned by all.
Read the full technical paper on
Modern RL-based post-training for large language models (LLMs) co-locate trajectory sampling and policy optimisation on the same GPU cluster, forcing the system to switch between inference and training workloads. This serial context switching violates the single-program-multiple-data (SPMD) assumption underlying today's distributed training systems. We present Echo, the RL system that cleanly decouples these two phases across heterogeneous "inference" and "training" swarms while preserving statistical efficiency. Echo introduces two lightweight synchronization protocols: a sequential pull mode that refreshes policy weights according to API call for minimal bias, and an asynchronous push-pull mode that streams version-tagged rollouts through a replay buffer to maximise hardware utilisation. Training four representative RL workloads with Qwen3-4B, Qwen2.5-7B, Qwen3-30B-A3B-Thinking-2507 and Qwen3-32B on a geographically distributed cluster, Echo matches a fully co-located Verl baseline in convergence speed and final reward while off-loading trajectory generation to commodity edge hardware. These promising results demonstrate that large-scale RL for LLMs could achieve datacentre-grade performance using decentralised, heterogeneous resources.
---
# Gradient Open Sources Parallax: The Open Source Sovereign AI OS
RResearchBBlogCCareers
B
<<<<<<<<<<<<
Open Sourcing Parallax: Your Sovereign AI OS
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Oct 28, 2025
Category
Sovereign AI
\[ARTICLE\]
The easiest way to host your own AI applications.
-------------------------------------------------
Deploying a large language model (LLM) inference service remains costly because centralized serving depends on specialized GPU clusters and high-bandwidth interconnects in datacenters. An appealing alternative is to leverage collaborative decentralized GPU pools. However, heterogeneity in GPU and limited interconnected network bandwidth, along with potentially dynamic availability, make efficient scheduling the central challenge in this scenario. In this paper, we present Parallax, a decentralized LLM serving system that turns a pool of heterogeneous GPUs into an efficient inference platform via a two-phase scheduler. Parallax decomposes planning into (i) model allocation, which places layers of each replica across diverse GPUs to jointly optimize latency and throughput under memory and link-bandwidth constraints, and (ii) request-time GPU pipeline selection, which stitches layers from different replicas into end-to-end execution chains that balance load and adapt to current conditions. We implement Parallax and evaluate it on open-source LLMs deployed over real volunteer nodes. Parallax consistently reduces latency and increases throughput relative to decentralized baselines, demonstrating that principled scheduling can make volunteer compute a practical, affordable substrate for LLM inference. Github Repo at: https://github.com/GradientHQ/parallax.
Why this matters
================
**First things first:** we want your personal AI agents to be **sovereign**. It should not upload everything it sees to a giant centralized cloud. What it learns about you should live as a **portable local memory** you control. That way you are not locked into a single model, and no one can access, alter, or shut it off without your permission. If you ever trust an assistant or companion with your life’s context, it should **live with you**.
We know local models are great for keeping your memory private while enabling fast iteration. But **a single machine only goes so far**. Hosting anything beyond about 8B is often not feasible on an everyday computer, and buying extra machines just to try a larger model is rarely sustainable. You end up settling.
What Parallax is
================
Parallax is the operating system for sovereign AI. Under the hood, it is a distributed runtime and service fabric that turns heterogeneous machines into one traceable service you can build on. When a model is too big for one host, it is sharded into contiguous layer slices and distributed across your laptop, a lab GPU, and a teammate’s workstation, all orchestrated as one service. Each request takes the fastest path on a single host, across a LAN, or over the public internet, without a public IP or matching hardware.
Parallax opens up a wide range of ways to host and run your own AI apps and agents that are completely your own, including coding copilots, personal assistants, vision and speech pipelines, and multi-agent simulations.
At launch, Parallax supports **40+ open models** from **0.6B** to **trillion-class MoE** on **GPUs** and **Apple Silicon**, across **Windows, Linux, and macOS**.
* * *
Key capabilities
================
Parallax is not just another local LLM runner. It is a path to open intelligence, scaling from one desk to clusters and out to the world.
* **3 modes, 1 operating system:** **LocalHost**, **Co-Host** on LAN, **Global Host** over WAN. Start on your own machine for smaller models and join clusters when you need more headroom.
* **Heterogeneity by default:** run 40+ models across GPUs and Apple Silicon on Windows, Linux, and macOS.
* **Consistent performance:** sustained throughput with tight tails under real WAN variability and high concurrency.
* **Network-aware scheduling and routing:** contiguous layer shards placed via dynamic programming with water-filling, then per-request DAG routing uses RTT profiling to choose the fastest path. The system is churn-aware and reallocates in milliseconds.
* **Traceability embedded:** deterministic execution, isolated peer execution, and per-request routing traceability for auditability and attribution.
* * *
What Parallax enables
=====================
Parallax orchestrates heterogeneous machines into an adaptive mesh that finds the fastest path per request and reorganizes under load. It supports three modes, each with distinct optimizations.
### Mode 1: LocalHost (single-host)
Run models on your own machine with data-center-class responsiveness.
* Usage scenario: personal agents
### Mode 2: Co-Host (multi-host cluster)
Join a cluster with other hosts on the same LAN or private L2 or L3 segment.
* Usage scenario: trust-circle clusters within small teams or families
### Mode 3: Global Host (WAN-scale fabric)
Form a wide-area cluster across unmanaged networks.
* Usage scenario: service-grade LLM serving across the globe
Together, these modes form a substrate for all sovereign AI applications to keep data local, scale when needed, and stay verifiable at every step.
### The Sovereign AI OS
Parallax is more than a way to serve models. It is a practical foundation for AI applications that can stay sovereign and open. By turning a mix of Macs and PCs into one adaptive service, it lets builders keep memory local, use open tools, and collaborate across devices without sending everything to a central cloud. The system is traceable end-to-end, so results are reproducible and auditable. With Lattica in place today, and verification and multi-agent layers coming next, you can build and run coding copilots, private-memory agents, retrieval over your own files, and vision or speech pipelines that move with you from a home setup to a lab cluster to a global fabric using the same code.
Popular dev tools and agent frameworks can point to your Parallax-hosted endpoints, so you can bring apps like **vibe** **coders, personal assistants,** and **agent IDEs** into your own environment while staying on consumer devices.
* * *
How the OS works
================
Real hosts and links are uneven. Parallax profiles device performance and links RTTs, then routes through the world as it is. Three pillars are making the experience possible.
1) Scheduling: placing shards & routing requests
------------------------------------------------
**Split an NP-hard problem into two quick ones.** Mixed hardware and uneven links make round-robin a dead end. We decide shards and placement first, then pick the best path for each request.
### **Phase 1: Model allocation**
Partition the model into **contiguous layer slices** and **map them to hosts** with a **dynamic programming plus water-filling** solver. Our objectives:
1. **Shallow pipeline depth** to reduce latency.
2. **Enough replicas** to raise throughput when needed.
3. **Balanced stage runtimes** so fast machines do not idle behind slow ones.
The result is a hardware-aware layout that respects VRAM and FLOPs and minimizes activation hops.
Preview
_Figure 1: Example of the first phase model allocation among heterogeneous GPU types across different geographic regions._
### **Phase 2: Path selection**
Maintain a **layer-indexed DAG** whose node weights are profiled **per-layer latencies** and whose edge weights are **inter-node RTTs**. For every incoming request, a DP router chooses the **minimum-latency path** and will select a different path for the next request if conditions change. It naturally routes around congestion and slow links. Then, requests are executed by streaming **hidden states** through the layers. A host runs its slice, forwards activations to the next, and so on until the pass completes.
Preview
_Figure 2: Example of the second phase GPU pipeline chain selection among GPUs (pipeline stages)._
### **Practical features:**
* **Heterogeneous hardware awareness:** device profiles for FLOPs, VRAM, and bandwidth prevent the **straggler** effect.
* **Churn-aware and self-healing:** joins, leaves, or slowdowns trigger **localized reallocation in milliseconds** (**≤ 10 ms at 256 nodes** in simulation). Routes refresh quickly via the DHT. Overhead stays in the **low single-digit ms** range.
* **Proven at useful scales:** We’ve tested **7-node real deployments** with algorithmic scalability to **256 nodes**, supporting **20+ nodes** and **hundreds of concurrent requests** while keeping tails tight.
### **How it applies across modes:**
* _Local host:_ Phase 1 within one device profile.
* _Co-Host (LAN):_ Phase 1 across nearby hosts with similar link characteristics.
* _Global (WAN-scale):_ Both phases active. Model shards span LAN clusters with WAN-aware routing.
2\. Communication: peer-to-peer discovery without friction
----------------------------------------------------------
**Find the best path, even behind NAT.**
* **LAN auto-detection:** prefer low-RTT local paths automatically.
* **Lattica transport with NAT traversal:** peers join **without public IPs**, working across common NATs and firewalls to form **WAN-scale** pipelines.
* **Unified Protobuf data plane:** activations and KV shards move consistently across GPU, CPU, and Apple Silicon backends.
* **Membership and health via DHT:** peer discovery, latency and health sharing, and **self-healing** under churn. These signals feed scheduling for fast reallocation and route refresh.
### **How it applies across modes:**
* _Co-Host:_ Prioritize LAN paths.
* _Global:_ Stitch LAN islands over WAN via Lattica.
3\. Backends: make every host pull its weight
---------------------------------------------
### **Common features**
* **Continuous batching** to keep devices saturated under bursty traffic.
* **Dynamic KV management** for high concurrency.
* **Sharded model loading** so each host loads only its assigned layers.
* **Prefix cache (radix tree)** to accelerate shared prompts.
* **On the horizon:** runtime weight updates and speculative decoding.
### **Apple Silicon**
* **MLX-LM integration** with large, continuous batching.
* **Advanced KV block allocation**, including sliding window attention, so Macs are efficient decoding nodes.
### **NVIDIA GPUs**
* **SGLang integration** via runtime patching for compiler and kernel scheduling benefits without model code changes.
* **CUDA Graphs in decode** to reduce launch overhead and lower token-level latency, especially for long contexts.
* * *
Dreaming bigger with Parallax
=============================
We tested Parallax under mixed hardware with **inter-node RTTs around 11 to 14 ms** to benchmark its performance as the OS of sovereign AI, as well as a backbone for Open Intelligence.
### **Experiment setup: 14-node Global host (WAN-scale) run**
* **Model:** Qwen3-32B-FP8
* **Cluster:** **14 NVIDIA RTX 5090** nodes, heterogeneous conditions
* **Throughput:** **~495 tokens/s** end to end; more than 1 req/s with ShareGPT-like traces at **200 concurrency**
* **Latency:** median inter-token **110 to 120 ms**, **p99 < 300 ms**
* **Scaling efficiency:** about **3×** throughput vs a smaller **non-optimized 4-node** cluster on the same workload
Preview
_Figure 3: End-to-end latency comparison between Parallax and HexGen across different models, traces, and request arrival rates._
Preview
_Figure 4: End-to-end throughput comparison between Parallax and HexGen across different models, traces, and request arrival rates._
Preview
_Figure 5: Phase-1 and phase-2 algorithm running time when scaling from smaller clusters (e.g., 4 GPUs) to larger clusters (e.g., 256 GPUs)._
### **Head-to-head results summary (selected)**
| Comparison | Scenario | Metric | Parallax | Advantage |
| --- | --- | --- | --- | --- |
| Global host vs HexGen | WildGPT/32B, rate 32 | p99 latency | **78.1 ms** | **2.6× lower** |
| Global host vs HexGen | WildGPT/32B, rate 32 | Throughput | **0.40 req/s** | **3.6× higher** |
| LAN Co-host vs Exo | Llama-3.1, 2048 in / 128 out | TTFT | **4,532 ms** | **1.97× faster** |
| Single GPU Local Host vs llama.cpp | RTX 5090, Qwen3-32B | TPOT (decode) | **85.98 ms/token** | **1.41× faster** |
Parallax sustains high throughput with tight tails **over WAN variability**, not only on controlled LAN setups. In the runs above, it **outperforms decentralized baselines** and **surpasses LAN-only prototypes**, while keeping **scheduling overhead under 10 ms at 256 GPUs** and reallocating in milliseconds under churn.
That combination of **contiguous sharding with DP and water-filling, per-request DAG routing with RTT profiling, and Lattica NAT traversal** behaves like a service, not a demo. Hosts can join from anywhere, models can span them cleanly, and requests take the best path in real time.
Taken together, these properties make Parallax a hyper-scalable operating system **for all sovereign AI applications**.
* * *
What’s next
===========
More features coming for Parallax:
* **Mixture-of-Experts aware scheduling:** place experts and gates across hosts to keep hops low and throughput high while preserving sparsity.
* **Elastic sequence parallelism:** scale context and tokens by flexibly splitting work across shard boundaries, adapting to runtime load.
* **Long context decode optimization:** cut token latency at long lengths with improved KV handling and decode graph reuse.
* * *
Closing
=======
Parallax isn’t just local AI. It’s an operating system built for multiplayer. Start on one box, co-host across a few machines, and scale to a global fabric so you can host all the AI applications you want and keep them sovereign.
* Host your own AI cluster with 40+ models: [github.com/GradientHQ/parallax](http://github.com/GradientHQ/parallax)
* Read the technical paper: **https://arxiv.org/abs/2509.26182**
The age of scaling parameters is ending. The age of scaling sovereign AI starts here.
---
# Gradient Network Raises $10M to Redefine AI Infrastructure
RResearchBBlogCCareers
B
<<<<<<<<<<<<
Gradient Network Raises $10M to Redefine AI Infrastructure
\>\>\>\>\>\>>>>>
/
\[INFO\]
Date
Jun 16, 2025
Category
fundraising
\[ARTICLE\]
Gradient Network is building the world’s first fully decentralized AI.
----------------------------------------------------------------------
Gradient Network is pioneering a decentralized AI infrastructure for open-source intelligence. We are determined to build a global foundation that supports the distribution, evolution, and embodiment of intelligence.
We’re proud to announce the completion of our $10M seed funding round, led by Pantera Capital and Multicoin Capital, with participation from HSG (formerly Sequoia Capital China) and other distinguished partners. This milestone is further supported by leading angel investors and advisors from spaces across AI, crypto, and beyond.
In tandem, we’ve refreshed our brand identity to better reflect our core values and ambitions. This brand evolution underscores our commitment to transparency, decentralization, and continuous innovation as we shape the future of AI.
Preview
**The Imperative for Decentralized AI**
---------------------------------------
As AI becomes integral to society, concerns around privacy, equity, and power concentration grow increasingly urgent. Today’s AI capabilities are largely concentrated in a few centralized platforms, posing significant risks of privacy infringement, exclusion, and monopolistic control.
Decentralizing AI emerges as a vital response to these pressing challenges. By distributing control over data, compute, and algorithm development across a diverse network of participants, decentralized AI fosters transparency, fairness, and security. This shift encourages broader innovation while minimizing systemic risks and dependency on centralized gatekeepers.
A comprehensive rethink of the AI stack starts with:
* Decentralized data governance
* Collaborative compute frameworks
* Open and transparent algorithm development
This approach not only protects user privacy and democratizes access to AI but also mitigates the systemic biases entrenched in centralized systems. Ultimately, decentralization isn't just beneficial; it is essential to building an equitable, transparent, and sustainable future for artificial intelligence.
**Building the Base Stack for Decentralized AI**
------------------------------------------------
At Gradient Network, we believe that decentralized intelligence requires a few foundational primitives: compute, communication, and orchestration. Together, they form the backbone of a new machine internet—one that is open, sovereign, and powered by millions. To bring this vision to life, we’ve set the stack in motion with two foundational building blocks—Lattica and Parallax.
### **Lattica: The Universal Data Motion Engine**
A robust, efficient peer-to-peer connectivity layer is essential for enabling decentralized compute of any kind. Over the past few months, in pursuit of a global peer-to-peer content delivery network (CDN), we unintentionally conducted one of the largest real-world experiments in internet connectivity mapping as the peer-to-peer CDN approaches commercialization. This was made possible by millions of participants across our Sentry Node network, showcasing the vast potential of decentralized systems powered by community engagement.
Today, that vision has evolved into**Lattica**—our universal peer-to-peer data communication protocol and the connectivity backbone of the decentralized AI stack. Lattica will be released in the coming days.
### **Parallax: The World Inference Engine**
As agentic AI applications proliferate, the demand for scalable inference infrastructure is accelerating, alongside rising needs for data sovereignty, reliability, and cost efficiency. Parallax is our response: a decentralized inference protocol purpose-built for this new paradigm.
What sets Parallax apart is its ability to go far beyond running small models on local endpoints. It enables large foundation models to be decomposed, distributed, and collaboratively executed across a global mesh of heterogeneous devices. This is inference, recomposed.
To support this, Parallax is designed with a set of critical capabilities:
* **Scalability:** Harnesses a global mesh of compute nodes—across diverse device classes—to scale inference beyond centralized limits.
* **Modular Execution:** Runs large models as orchestrated segments across distributed nodes, enabling flexible, fault-tolerant deployment.
* **Privacy-Preservation:** Ensures user data remains secure and confidential, honoring one of the core motivations to decentralize inference in the first place.
* **Verifiability:** Enables transparent validation of inference outputs, reinforcing trust in decentralized systems.
* **Reliability:** Delivers fault-tolerant, consistently performant inference, even in highly distributed and heterogeneous environments.
Like Lattica, Parallax will debut later this week.
Lattica and Parallax are just the beginning. More groundbreaking protocols are on the horizon, each reinforcing our commitment to a truly decentralized AI runtime.
**Join the Movement**
---------------------
This marks just the start of our expedition toward a future where intelligence is decentralized and open to all. There are new protocols to be built, resources to be unified, and applications to be developed. We warmly invite researchers, developers, and the community to join us to decentralize the world’s intelligence, for everyone.
X (Twitter): [https://x.com/Gradient\_HQ](https://x.com/Gradient_HQ)
Discord: [https://discord.gg/gradientnetwork](https://discord.gg/gradientnetwork)
Contact us: contact@gradient.network
---
# Unknown
RResearchBBlogCCareers
404
Sorry, the page you are visiting does not exist
---