> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nelieo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# NSP Production Deployment, Hardening, and Monitoring

> A step-by-step checklist covering daemon hardening, agent code safety, monitoring, log rotation, key management, and backup for NSP production deployments.

Before you route real traffic through an NSP-based agent, work through every item below. These steps protect your data, keep your agent running reliably, and make failures observable before they become incidents.

## Daemon Configuration

Harden the daemon's `axon.toml` before going live. The default config is tuned for local development; production needs tighter settings.

<Steps>
  <Step title="Enable API key authentication">
    Confirm authentication is on. Run a health check and look for `"auth_enabled": true`:

    ```bash theme={null}
    curl http://localhost:7842/health
    ```

    If `auth_enabled` is `false`, update your config and restart the daemon:

    ```toml theme={null}
    [auth]
    require_key = true
    keys_file   = "C:\\ProgramData\\Nelieo\\axon_keys.json"
    ```

    Generate a production key with `axon-daemon.exe keygen` and store it in your secrets manager — not in source code.
  </Step>

  <Step title="Bind to localhost only">
    The daemon must never listen on a network interface. Verify your config:

    ```toml theme={null}
    [server]
    host = "127.0.0.1"   # Never 0.0.0.0 in production
    port = 7842
    ```

    CDP gives remote code execution access to any browser the daemon probes. Exposing the daemon port externally would give that access to anyone on the network.
  </Step>

  <Step title="Set appropriate rate limits">
    Tune rate limits to match your agent's actual usage pattern. The defaults are conservative:

    ```toml theme={null}
    [rate_limit]
    enabled            = true
    state_per_second   = 100   # Max state reads/s per API key
    action_per_second  = 10    # Max action dispatches/s per API key
    snapshot_per_minute = 2    # Heap snapshots are expensive; keep this low
    ```

    If your agent uses streaming (`session.watch()`), the `/watch` WebSocket endpoint is not rate-limited — you can reduce `state_per_second` accordingly.
  </Step>

  <Step title="Enable persistent schema registry">
    Without a persistent registry, the daemon relearns every app schema from scratch on each restart, causing a 30–90 second confidence warm-up per app:

    ```toml theme={null}
    [registry]
    in_memory = false
    db_path   = "C:\\ProgramData\\Nelieo\\axon_registry.db"
    ```

    Schedule a daily backup of `axon_registry.db` (see the Backup section below).
  </Step>

  <Step title="Lock down CORS">
    If you expose a dashboard that calls the daemon directly from a browser, restrict CORS to exactly those origins:

    ```toml theme={null}
    [cors]
    allow_all_origins = false
    allowed_origins   = ["https://your-dashboard.internal.com"]
    ```

    Never set `allow_all_origins = true` in production.
  </Step>

  <Step title="Set structured logging at info level">
    `debug` logging generates gigabytes of output per day on a busy daemon. Use `info` and structured JSON for log aggregator compatibility:

    ```toml theme={null}
    [logging]
    level   = "info"   # Not "debug"
    format  = "json"   # Structured for Datadog, Splunk, Loki, etc.
    ```
  </Step>
</Steps>

## Agent Code Safety

<Steps>
  <Step title="Use verify=True on all irreversible actions">
    Any action with `reversibility: irreversible_write` — `send_reply`, `send_email`, `delete`, `take_off`, and others — must include a `verify_expression`. Without it, the daemon rejects the request:

    ```python theme={null}
    # ❌ Dangerous — no verification on a destructive action
    await session.execute("delete", parameters={"email_id": email_id})

    # ✅ Safe — verify the email count dropped
    before_total = session.get_int("inbox.total_count")
    await session.execute(
        "delete",
        parameters={"email_id": email_id},
        verify=True,
        verify_expression=f"inbox.total_count < {before_total}",
        verify_timeout_ms=5000,
    )
    ```
  </Step>

  <Step title="Gate on confidence before critical actions">
    Never execute actions against low-confidence state. A confidence below `0.95` means the probe hasn't fully mapped the application yet:

    ```python theme={null}
    MIN_CONFIDENCE = 0.95

    if session.confidence < MIN_CONFIDENCE:
        logger.warning(
            "Confidence {:.2f} below threshold — skipping action",
            session.confidence,
        )
        return

    await session.execute("send_reply", ...)
    ```
  </Step>

  <Step title="Handle NSPConfidenceTooLowError gracefully">
    Even with a pre-check, confidence can drop between the check and the execute call. Catch the error and retry with backoff:

    ```python theme={null}
    import asyncio
    import nelieo_nsp as nsp

    async def safe_execute(session, action, parameters, **kwargs):
        for attempt in range(3):
            try:
                return await session.execute(action, parameters=parameters, **kwargs)
            except nsp.NSPConfidenceTooLowError:
                if attempt == 2:
                    raise
                wait = 2 ** attempt
                logger.warning("Confidence too low — retrying in {}s", wait)
                await asyncio.sleep(wait)
                await session.refresh(fresh_probe=True)
    ```
  </Step>

  <Step title="Use wait_for_app instead of attach on startup">
    `client.attach()` raises `NSPNotFoundError` if the app isn't ready yet. `wait_for_app` retries until the app is detected and confidence is sufficient:

    ```python theme={null}
    # ❌ Fails immediately if Gmail hasn't been detected yet
    session = await client.attach("Gmail")

    # ✅ Waits up to 60s — safe on startup and after restarts
    session = await client.wait_for_app("Gmail", timeout=60.0)
    ```
  </Step>

  <Step title="Use streaming instead of polling">
    Polling burns your rate limit and adds latency. Streaming is free (not rate-limited) and reacts within 50–100ms:

    ```python theme={null}
    # ❌ Hammers the rate limit
    while True:
        await session.refresh()
        process_state(session.state)
        await asyncio.sleep(0.5)

    # ✅ Zero polling overhead, immediate reaction
    async with session.watch(auto_refresh_on_event=True) as stream:
        async for event in stream:
            process_state(session.state)
    ```
  </Step>

  <Step title="Store API keys in environment variables">
    Never hardcode `sk_nsp_...` keys in source code. Use environment variables or a secrets manager:

    ```python theme={null}
    import os
    import nelieo_nsp as nsp

    API_KEY = os.environ["NSP_API_KEY"]  # Set in deployment environment
    client  = nsp.NSPClient(api_key=API_KEY)
    ```

    Verify that `repr()` outputs and log statements never surface key values.
  </Step>
</Steps>

## Testing Before Going Live

<AccordionGroup>
  <Accordion title="Test reversible actions first">
    Before enabling any `irreversible_write` action in production, verify your agent logic with `reversible_write` equivalents. For example, test with `archive` and `mark_read` before enabling `send_reply` or `delete`. Reversible actions can be undone; irreversible ones cannot.
  </Accordion>

  <Accordion title="Verify your verify_expression catches failures">
    Deliberately cause an action to fail (e.g., pass an invalid `email_id`) and confirm that `result.success` is `False` and your agent handles the error path correctly. A `verify_expression` that always evaluates to `true` provides no protection.
  </Accordion>

  <Accordion title="Load test before high-frequency deployment">
    Measure your daemon's actual throughput before deploying an agent that fires frequently:

    ```python theme={null}
    import asyncio
    import time
    import nelieo_nsp as nsp

    async def load_test():
        async with nsp.NSPClient(api_key="sk_nsp_...") as client:
            session = await client.attach("Gmail")

            start  = time.monotonic()
            count  = 0
            errors = 0

            for _ in range(500):
                try:
                    await session.refresh()
                    count += 1
                except nsp.NSPRateLimitError:
                    errors += 1
                    await asyncio.sleep(0.01)

            elapsed = time.monotonic() - start
            print(f"Completed {count} reads in {elapsed:.1f}s ({count/elapsed:.0f} req/s)")
            print(f"Rate limit hits: {errors}")

    asyncio.run(load_test())
    ```
  </Accordion>
</AccordionGroup>

## Monitoring

Set up monitoring before your first production deployment, not after the first incident.

<Steps>
  <Step title="Poll the health endpoint">
    The daemon exposes a health endpoint at `GET http://localhost:7842/health`. Configure your monitoring system (Datadog, Prometheus, Uptime Kuma, etc.) to poll it every 30 seconds:

    ```bash theme={null}
    curl http://localhost:7842/health
    ```

    ```json theme={null}
    {
      "status": "ok",
      "uptime_secs": 86400,
      "tracked_processes": 3,
      "auth_enabled": true,
      "version": "1.4.2"
    }
    ```

    Alert on `status != "ok"` and on `tracked_processes` dropping to `0` unexpectedly.
  </Step>

  <Step title="Alert on daemon restarts">
    A drop in `uptime_secs` between two consecutive health checks means the daemon restarted. Restarts trigger cold probe warm-ups for any apps not in the persisted registry.
  </Step>

  <Step title="Monitor for 429 responses">
    Rate limit errors (`HTTP 429`) from your agent mean it's polling too fast. Instrument your agent code to count `NSPRateLimitError` exceptions and send that metric to your observability platform. A non-zero rate is a signal to switch to streaming.
  </Step>

  <Step title="Set up log forwarding">
    Forward daemon logs to a centralized aggregator. With `format = "json"` in `axon.toml`, the logs are structured and immediately queryable:

    ```toml theme={null}
    [logging]
    level  = "info"
    format = "json"
    ```

    For log rotation on Windows, the daemon writes to stdout by default. Wrap it with a log rotation tool (e.g. `nxlog`, Windows Event Log forwarding, or a sidecar container) before production.
  </Step>
</Steps>

## Daemon Auto-Start

<Steps>
  <Step title="Install as a Windows Service">
    Run this once as Administrator to register the daemon as a managed Windows Service:

    ```cmd theme={null}
    axon-daemon.exe install
    ```

    The service is named `NSPDaemon` and starts automatically at boot.
  </Step>

  <Step title="Configure automatic restart on failure">
    Open **Services → NSPDaemon → Properties → Recovery** and set:

    * First failure: **Restart the Service**
    * Second failure: **Restart the Service**
    * Subsequent failures: **Restart the Service**
    * Reset fail count after: **1 day**

    This ensures the daemon recovers from transient crashes without manual intervention.
  </Step>

  <Step title="Verify the service starts after reboot">
    Reboot a test machine and confirm the daemon is running before deploying to production:

    ```powershell theme={null}
    Get-Service NSPDaemon
    # Status: Running
    ```
  </Step>
</Steps>

## Key Management and Backup

<AccordionGroup>
  <Accordion title="Restrict permissions on axon_keys.json">
    The keys file grants full access to all NSP operations. Restrict it to the service account only:

    ```powershell theme={null}
    $acl  = Get-Acl "C:\ProgramData\Nelieo\axon_keys.json"
    $rule = New-Object System.Security.AccessControl.FileSystemAccessRule(
        "NT SERVICE\NSPDaemon", "Read", "Allow"
    )
    $acl.SetAccessRule($rule)
    Set-Acl "C:\ProgramData\Nelieo\axon_keys.json" $acl
    ```
  </Accordion>

  <Accordion title="Rotate keys when team members leave">
    Generate a new key with `axon-daemon.exe keygen`, update your deployment secrets, restart the agent process, and then remove the old key from `axon_keys.json`. The daemon reloads the keys file without a restart.
  </Accordion>

  <Accordion title="Back up axon_registry.db daily">
    The schema registry database stores the learned app schemas. Losing it means a 30–90 second cold-probe warm-up per app on the next daemon start. Schedule a daily copy:

    ```powershell theme={null}
    # Run as a scheduled task
    $src  = "C:\ProgramData\Nelieo\axon_registry.db"
    $dest = "C:\Backups\NSP\axon_registry_$(Get-Date -Format yyyyMMdd).db"
    Copy-Item $src $dest
    ```

    Keep at least 7 days of backups. If the database becomes corrupt, restore the most recent backup and restart the daemon.
  </Accordion>
</AccordionGroup>

## Quick Reference: Common Production Issues

| Symptom                               | Diagnosis                                | Fix                                         |
| ------------------------------------- | ---------------------------------------- | ------------------------------------------- |
| `NSPConnectionError` on agent startup | Daemon not running                       | `Start-Service NSPDaemon`                   |
| `confidence: 0.0` after 60s           | Chrome missing `--remote-debugging-port` | Relaunch Chrome with the debug flag         |
| `probe_status: "error"` in health     | Probe crashed, retrying automatically    | Check daemon logs; usually transient        |
| HTTP 429 from agent                   | Agent polling too fast                   | Switch to `session.watch()` streaming       |
| `verification_failed` on actions      | App UI slow to settle after action       | Increase `verify_timeout_ms`                |
| `schema_version` keeps changing       | App updated, NSP relearning schema       | Normal; confidence recovers in 30–60s       |
| `tracked_processes: 0` in health      | No apps detected                         | Check target apps are running and reachable |