NSP: The Live Application Substrate for AI Agent Automation

NSP (Nelieo Substrate Protocol) is a runtime substrate daemon that attaches to running applications and exposes their internal state and actions as typed JSON. Instead of taking screenshots and guessing pixel coordinates, your AI agent reads live data directly from the application’s runtime and calls typed functions to act on it — the same way a human developer would if they had direct access to the app’s memory.

Why Agents Fail Without NSP

Every vision-based AI agent today operates the same way: capture a screenshot, send it to a vision model, infer where to click, hope the click lands correctly, and repeat.

Screenshot (2 MB PNG)
  → Vision model inference (~8 seconds)
  → "Click at pixel (847, 392)"
  → Miss by 3 pixels
  → Screenshot again
  → Repeat forever

At 85% per-action accuracy — an optimistic figure for state-of-the-art vision agents — a 50-step task has a 0.03% end-to-end success rate. Agents don’t fail because the underlying AI is weak. They fail because the interface is fundamentally broken: screenshots are lossy compressions of structured data that already existed in memory.

Metric	NSP	Vision-Based Agent
State read latency	~50 ms	8,000–15,000 ms
Action accuracy (JS apps)	>99%	66–82%
50-step task completion	~95%	~0.03%
Actions per minute	~120	~4–7
State verification	Always	Never
Works offline	Yes	Depends on model

How NSP Solves It

NSP attaches directly to a running application’s runtime and reads its live state as structured JSON. Your agent calls typed functions. No screenshots, no pixel coordinates, no guessing.

GET /substrate/v1/state/12847

{
  "app_name": "Gmail",
  "runtime": "v8",
  "confidence": 0.97,
  "objects": {
    "inbox.unread_count": 47,
    "inbox.emails[0].subject": "Q3 Performance Report",
    "inbox.emails[0].from": "alice@example.com",
    "inbox.emails[0].is_read": false
  },
  "actions": {
    "send_reply": {
      "signature": "fn(thread_id: string, body: string) -> bool",
      "reversibility": "irreversible_write"
    },
    "archive": {
      "signature": "fn(email_id: string) -> bool",
      "reversibility": "reversible_write"
    }
  }
}

The result is >99% accuracy on JavaScript apps, 120 actions per minute, and continuous 24/7 operation with state verification after every action.

The Four Layers

The NSP daemon (axon-daemon) runs on the host machine alongside your applications. It has four layers that work together to turn live process memory into an agent-callable API.

Process Watcher

Continuously scans all running OS processes every 500 ms, classifies each process’s runtime (V8, JVM, CLR, or Native), and manages the full probe attachment lifecycle.

Runtime Probes

Language-specific probes attach to live processes and extract semantic state — no source code, no UI automation, and no prior instrumentation required.

Semantic Engine

Converts raw runtime data extracted by each probe into structured SSF JSON with stable, human-readable dot-notation keys.

Agent API

A REST and WebSocket server on localhost:7842 that any AI agent can call to read state, subscribe to real-time changes, and execute actions.

Runtime Coverage

NSP supports four runtime families, each with a dedicated probe strategy and measured accuracy baseline.

Runtime	Applications	Probe Method	Accuracy
JavaScript / V8	Gmail, Slack, VS Code, Notion, GitHub Desktop, Salesforce, WhatsApp, Discord, Shopify	CDP remote debugging + fast-path state extraction	>99%
Java / JVM	SAP, Eclipse, IntelliJ, Jenkins, Elasticsearch, ArduPilot GCS	Native JVM instrumentation + reflection	>97%
.NET / CLR	Excel, Word, PowerPoint, Visual Studio, Mission Planner, Azure tools	CLR heap inspection + managed bootstrapper	>97%
Native C++	Bloomberg Terminal, Photoshop, AutoCAD, Chrome internals	Process memory reading + Visual-Memory Grounding	>75%

Native C++ accuracy is lower because compiled binaries have no metadata for semantic labeling. NSP uses Visual-Memory Grounding — a hybrid of memory layout analysis and a local vision model — to close the gap. Expect V8 and JVM accuracy on the applications your agents will use most.

Next Steps

Quickstart

Install the daemon, configure Chrome, and call your first state endpoint in under 5 minutes.

Core Concepts

Understand SubstrateState, SSF, action schemas, confidence scores, and reversibility tiers.

Python SDK

Install nelieo-nsp, attach to apps by name, and use typed helpers with full async support.

API Reference

Complete reference for every REST and WebSocket endpoint exposed by the daemon.

Get Started

The Daemon

Runtime Probes

Security

Guides

Why Agents Fail Without NSP

How NSP Solves It

The Four Layers

Process Watcher

Runtime Probes

Semantic Engine

Agent API

Runtime Coverage

Next Steps

Quickstart

Core Concepts

Python SDK

API Reference

​Why Agents Fail Without NSP

​How NSP Solves It

​The Four Layers

Process Watcher

Runtime Probes

Semantic Engine

Agent API

​Runtime Coverage

​Next Steps

Quickstart

Core Concepts

Python SDK

API Reference

Why Agents Fail Without NSP

How NSP Solves It

The Four Layers

Runtime Coverage

Next Steps