NSP Production Deployment, Hardening, and Monitoring

Before you route real traffic through an NSP-based agent, work through every item below. These steps protect your data, keep your agent running reliably, and make failures observable before they become incidents.

Daemon Configuration

Harden the daemon’s axon.toml before going live. The default config is tuned for local development; production needs tighter settings.

Enable API key authentication

Confirm authentication is on. Run a health check and look for "auth_enabled": true:

curl http://localhost:7842/health

If auth_enabled is false, update your config and restart the daemon:

[auth]
require_key = true
keys_file   = "C:\\ProgramData\\Nelieo\\axon_keys.json"

Generate a production key with axon-daemon.exe keygen and store it in your secrets manager — not in source code.

Bind to localhost only

The daemon must never listen on a network interface. Verify your config:

[server]
host = "127.0.0.1"   # Never 0.0.0.0 in production
port = 7842

CDP gives remote code execution access to any browser the daemon probes. Exposing the daemon port externally would give that access to anyone on the network.

Set appropriate rate limits

Tune rate limits to match your agent’s actual usage pattern. The defaults are conservative:

[rate_limit]
enabled            = true
state_per_second   = 100   # Max state reads/s per API key
action_per_second  = 10    # Max action dispatches/s per API key
snapshot_per_minute = 2    # Heap snapshots are expensive; keep this low

If your agent uses streaming (session.watch()), the /watch WebSocket endpoint is not rate-limited — you can reduce state_per_second accordingly.

Enable persistent schema registry

Without a persistent registry, the daemon relearns every app schema from scratch on each restart, causing a 30–90 second confidence warm-up per app:

[registry]
in_memory = false
db_path   = "C:\\ProgramData\\Nelieo\\axon_registry.db"

Schedule a daily backup of axon_registry.db (see the Backup section below).

Lock down CORS

If you expose a dashboard that calls the daemon directly from a browser, restrict CORS to exactly those origins:

[cors]
allow_all_origins = false
allowed_origins   = ["https://your-dashboard.internal.com"]

Never set allow_all_origins = true in production.

Set structured logging at info level

debug logging generates gigabytes of output per day on a busy daemon. Use info and structured JSON for log aggregator compatibility:

[logging]
level   = "info"   # Not "debug"
format  = "json"   # Structured for Datadog, Splunk, Loki, etc.

Agent Code Safety

Use verify=True on all irreversible actions

Any action with reversibility: irreversible_write — send_reply, send_email, delete, take_off, and others — must include a verify_expression. Without it, the daemon rejects the request:

# ❌ Dangerous — no verification on a destructive action
await session.execute("delete", parameters={"email_id": email_id})

# ✅ Safe — verify the email count dropped
before_total = session.get_int("inbox.total_count")
await session.execute(
    "delete",
    parameters={"email_id": email_id},
    verify=True,
    verify_expression=f"inbox.total_count < {before_total}",
    verify_timeout_ms=5000,
)

Gate on confidence before critical actions

Never execute actions against low-confidence state. A confidence below 0.95 means the probe hasn’t fully mapped the application yet:

MIN_CONFIDENCE = 0.95

if session.confidence < MIN_CONFIDENCE:
    logger.warning(
        "Confidence {:.2f} below threshold — skipping action",
        session.confidence,
    )
    return

await session.execute("send_reply", ...)

Handle NSPConfidenceTooLowError gracefully

Even with a pre-check, confidence can drop between the check and the execute call. Catch the error and retry with backoff:

import asyncio
import nelieo_nsp as nsp

async def safe_execute(session, action, parameters, **kwargs):
    for attempt in range(3):
        try:
            return await session.execute(action, parameters=parameters, **kwargs)
        except nsp.NSPConfidenceTooLowError:
            if attempt == 2:
                raise
            wait = 2 ** attempt
            logger.warning("Confidence too low — retrying in {}s", wait)
            await asyncio.sleep(wait)
            await session.refresh(fresh_probe=True)

Use wait_for_app instead of attach on startup

client.attach() raises NSPNotFoundError if the app isn’t ready yet. wait_for_app retries until the app is detected and confidence is sufficient:

# ❌ Fails immediately if Gmail hasn't been detected yet
session = await client.attach("Gmail")

# ✅ Waits up to 60s — safe on startup and after restarts
session = await client.wait_for_app("Gmail", timeout=60.0)

Use streaming instead of polling

Polling burns your rate limit and adds latency. Streaming is free (not rate-limited) and reacts within 50–100ms:

# ❌ Hammers the rate limit
while True:
    await session.refresh()
    process_state(session.state)
    await asyncio.sleep(0.5)

# ✅ Zero polling overhead, immediate reaction
async with session.watch(auto_refresh_on_event=True) as stream:
    async for event in stream:
        process_state(session.state)

Store API keys in environment variables

Never hardcode sk_nsp_... keys in source code. Use environment variables or a secrets manager:

import os
import nelieo_nsp as nsp

API_KEY = os.environ["NSP_API_KEY"]  # Set in deployment environment
client  = nsp.NSPClient(api_key=API_KEY)

Verify that repr() outputs and log statements never surface key values.

Testing Before Going Live

Test reversible actions first

Before enabling any irreversible_write action in production, verify your agent logic with reversible_write equivalents. For example, test with archive and mark_read before enabling send_reply or delete. Reversible actions can be undone; irreversible ones cannot.

Verify your verify_expression catches failures

Deliberately cause an action to fail (e.g., pass an invalid email_id) and confirm that result.success is False and your agent handles the error path correctly. A verify_expression that always evaluates to true provides no protection.

Load test before high-frequency deployment

Measure your daemon’s actual throughput before deploying an agent that fires frequently:

import asyncio
import time
import nelieo_nsp as nsp

async def load_test():
    async with nsp.NSPClient(api_key="sk_nsp_...") as client:
        session = await client.attach("Gmail")

        start  = time.monotonic()
        count  = 0
        errors = 0

        for _ in range(500):
            try:
                await session.refresh()
                count += 1
            except nsp.NSPRateLimitError:
                errors += 1
                await asyncio.sleep(0.01)

        elapsed = time.monotonic() - start
        print(f"Completed {count} reads in {elapsed:.1f}s ({count/elapsed:.0f} req/s)")
        print(f"Rate limit hits: {errors}")

asyncio.run(load_test())

Monitoring

Set up monitoring before your first production deployment, not after the first incident.

Poll the health endpoint

The daemon exposes a health endpoint at GET http://localhost:7842/health. Configure your monitoring system (Datadog, Prometheus, Uptime Kuma, etc.) to poll it every 30 seconds:

curl http://localhost:7842/health

{
  "status": "ok",
  "uptime_secs": 86400,
  "tracked_processes": 3,
  "auth_enabled": true,
  "version": "1.4.2"
}

Alert on status != "ok" and on tracked_processes dropping to 0 unexpectedly.

Alert on daemon restarts

A drop in uptime_secs between two consecutive health checks means the daemon restarted. Restarts trigger cold probe warm-ups for any apps not in the persisted registry.

Monitor for 429 responses

Rate limit errors (HTTP 429) from your agent mean it’s polling too fast. Instrument your agent code to count NSPRateLimitError exceptions and send that metric to your observability platform. A non-zero rate is a signal to switch to streaming.

Set up log forwarding

Forward daemon logs to a centralized aggregator. With format = "json" in axon.toml, the logs are structured and immediately queryable:

[logging]
level  = "info"
format = "json"

For log rotation on Windows, the daemon writes to stdout by default. Wrap it with a log rotation tool (e.g. nxlog, Windows Event Log forwarding, or a sidecar container) before production.

Daemon Auto-Start

Install as a Windows Service

Run this once as Administrator to register the daemon as a managed Windows Service:

axon-daemon.exe install

The service is named NSPDaemon and starts automatically at boot.

Configure automatic restart on failure

Open Services → NSPDaemon → Properties → Recovery and set:

First failure: Restart the Service
Second failure: Restart the Service
Subsequent failures: Restart the Service
Reset fail count after: 1 day

This ensures the daemon recovers from transient crashes without manual intervention.

Verify the service starts after reboot

Reboot a test machine and confirm the daemon is running before deploying to production:

Get-Service NSPDaemon
# Status: Running

Key Management and Backup

Restrict permissions on axon_keys.json

The keys file grants full access to all NSP operations. Restrict it to the service account only:

$acl  = Get-Acl "C:\ProgramData\Nelieo\axon_keys.json"
$rule = New-Object System.Security.AccessControl.FileSystemAccessRule(
    "NT SERVICE\NSPDaemon", "Read", "Allow"
)
$acl.SetAccessRule($rule)
Set-Acl "C:\ProgramData\Nelieo\axon_keys.json" $acl

Rotate keys when team members leave

Generate a new key with axon-daemon.exe keygen, update your deployment secrets, restart the agent process, and then remove the old key from axon_keys.json. The daemon reloads the keys file without a restart.

Back up axon_registry.db daily

The schema registry database stores the learned app schemas. Losing it means a 30–90 second cold-probe warm-up per app on the next daemon start. Schedule a daily copy:

# Run as a scheduled task
$src  = "C:\ProgramData\Nelieo\axon_registry.db"
$dest = "C:\Backups\NSP\axon_registry_$(Get-Date -Format yyyyMMdd).db"
Copy-Item $src $dest

Keep at least 7 days of backups. If the database becomes corrupt, restore the most recent backup and restart the daemon.

Quick Reference: Common Production Issues

Symptom	Diagnosis	Fix
`NSPConnectionError` on agent startup	Daemon not running	`Start-Service NSPDaemon`
`confidence: 0.0` after 60s	Chrome missing `--remote-debugging-port`	Relaunch Chrome with the debug flag
`probe_status: "error"` in health	Probe crashed, retrying automatically	Check daemon logs; usually transient
HTTP 429 from agent	Agent polling too fast	Switch to `session.watch()` streaming
`verification_failed` on actions	App UI slow to settle after action	Increase `verify_timeout_ms`
`schema_version` keeps changing	App updated, NSP relearning schema	Normal; confidence recovers in 30–60s
`tracked_processes: 0` in health	No apps detected	Check target apps are running and reachable

Get Started

The Daemon

Runtime Probes

Security

Guides

Daemon Configuration

Agent Code Safety

Testing Before Going Live

Monitoring

Daemon Auto-Start

Key Management and Backup

Quick Reference: Common Production Issues

​Daemon Configuration

​Agent Code Safety

​Testing Before Going Live

​Monitoring

​Daemon Auto-Start

​Key Management and Backup

​Quick Reference: Common Production Issues

Daemon Configuration

Agent Code Safety

Testing Before Going Live

Monitoring

Daemon Auto-Start

Key Management and Backup

Quick Reference: Common Production Issues