Learns what's "normal" for YOUR server over ~7 days, then flags deviations:
# Example: Server normally uses 20-40% CPUMean: 30% | StdDev: 5%# Suddenly CPU hits 60%Z-score = (60 - 30) / 5 = 6.0Z-score > 3.0 → Anomaly!# Even though 60% is below threshold,# it's unusual for THIS server
3. Log Pattern Detection
Real-time matching triggers immediate anomalies:
# Log line appears:"sshd: Failed password for root from 192.168.1.100"↓Matches: "Failed password"↓Security anomaly created → Alert sent
AI Diagnosis
When an anomaly is detected, Spyd gathers context and consults AI for root cause analysis:
Context Provided to AI
Anomaly Details
Type, metric, actual vs threshold, severity
System State
Current CPU, memory, disk, load snapshot
Top Processes
PID, name, CPU%, memory%, user, full command
Correlation Hints
High CPU + High Load = CPU-bound, etc.
Process Patterns
Fork bombs, single dominators, suspicious users
Recent Logs
Matched error/security events
AI Capabilities
The AI operates as a Silent Guardian — an L3+ Senior SysAdmin who:
1. Analyzes ALL active anomalies together (holistic, not per-metric)
2. Runs diagnostic commands to gather evidence and correlate signals
4. Distinguishes real threats from routine operations (apt updates, backups, cron jobs)
5. Decides whether to notify you or handle it silently
6. Provides plain-English explanations with clear actions when escalating
Silent Guardian Mode
Spyd works as a silent guardian — monitoring, analyzing, and investigating 24/7 without bothering you. Alerts are only sent when the server is genuinely at risk.
1.Observation Window — Anomalies must persist for 3+ consecutive checks (~3 min) before any action. Transient spikes are silently dropped.
2.Log Noise Filter — Common harmless patterns (TLS handshakes, connection resets, bot probes) are automatically suppressed.
3.Holistic AI Investigation — ALL active anomalies are batched into one AI analysis. The AI decides: notify_user: true/false.
4.Clean Notifications — When you do get an alert, it's one paragraph: what's wrong, why it matters, what to do. No jargon, no boilerplate.
Smart Alerting: A healthy server produces ~0-2 notifications per week, not 10+ per day. Transient spikes, routine log errors, and backup jobs are handled silently. Only security threats, sustained critical resources, and service crashes reach you.
Command Reference
Core Commands
init
spyd init
Interactive setup wizard that guides you through initial configuration including AI provider, monitoring thresholds, and notification channels.
$ spyd initWelcome to Spyd Setup!
AI Configuration
? Enable AI analysis? Yes
? AI Provider: OpenAI
? API Key: sk-...
? Model: gpt-4o-mini
Monitoring Thresholds
? CPU Threshold (%): 80
? Memory Threshold (%): 85
? Disk Threshold (%): 90
Configuration saved to ~/.config/spyd/config.yaml
start
spyd start
Starts the Spyd daemon in the background. The daemon collects system metrics every 60 seconds (configurable) and triggers AI analysis when thresholds are exceeded.
$ spyd startDaemon started (PID: 12345)
stop
spyd stop
Gracefully stops the running daemon. Completes the current metric collection cycle before shutting down.
$ spyd stopDaemon stopped
restart
spyd restart
Stops and restarts the daemon. Useful after configuration changes.
$ spyd restartDaemon stopped
Daemon started (PID: 12346)
status
spyd status
Displays current system health snapshot including CPU, memory, disk usage, load average, and the most recent AI analysis.
$ spyd status
+---------------------------------------------------------------+
| Spyd Status |
+---------------------------------------------------------------+
Daemon: Running (PID: 12345)
Uptime: 2h 15m 30s
System Metrics:
CPU: 45.2% [========----------]
Memory: 62.1% [============------]
Disk: 55.3% [===========-------]
Load: 1.2, 0.8, 0.6
AI Analysis:
Last checked: 2m ago
Status: System healthy
Provider: OpenAI (gpt-4o-mini)
Notifications:
Telegram: Enabled
Email: Enabled
start --foreground
spyd start -f
Runs daemon in foreground with verbose logging. Useful for debugging or running in Docker/systemd.
$ spyd start --foregroundSpyd daemon started (PID: 12345)
2024-01-15T10:30:00Z INF Collecting metrics...
2024-01-15T10:30:01Z INF CPU: 45.2%, Memory: 62.1%, Disk: 55.3%
2024-01-15T10:30:01Z INF All metrics within thresholds
Ask AI about a specific error, log, or topic. Get expert-level explanations for server issues.
$ spyd explain "what does OOM killer mean?"
Asking AI (openai)...
===============================================================
AI EXPLANATION
===============================================================
The OOM (Out of Memory) Killer is a Linux kernel mechanism that
activates when the system runs critically low on memory. It
selects and terminates processes to free up RAM and prevent a
complete system crash.
When triggered, you'll see messages like:
"Out of memory: Killed process 1234 (nginx)"
Common causes:
1. Memory leak in an application
2. Insufficient RAM for workload
3. Too many processes running simultaneously
To investigate:
$ dmesg | grep -i "out of memory"
$ grep -i "killed process" /var/log/syslog
# With additional context$ spyd explain "nginx error: upstream timed out" --context "high traffic spike"
Send a test alert through configured notification channels to verify setup.
# Test all enabled channels$ spyd test-alert
+---------------------------------------------------------------+
| Spyd Test Alert |
+---------------------------------------------------------------+
Sending test alert...
Severity: warning
Message: This is a test notification to verify your Spyd alert configuration.
Channel: all
Channels tested:
telegram: @your_chat_id
email: [admin@example.com]
# Test specific channel$ spyd test-alert --channel telegram# Test with critical severity$ spyd test-alert --severity critical
analyze
spyd analyze [--mode MODE]
Trigger manual AI analysis of current system state. Collects metrics, runs anomaly detection, and provides AI insights.
# Analyze current state (auto mode)$ spyd analyze# Quick analysis without running commands$ spyd analyze --mode basic# Thorough analysis with diagnostic commands$ spyd analyze --mode hybrid
Configuration Commands
configure ai
spyd configure ai [flags]
Update AI provider settings. Shows full configuration summary when run without flags.
# Show current AI configuration$ spyd configure ai# Update provider and model$ spyd configure ai --provider openai --model gpt-4o-mini# Enable/disable AI command execution$ spyd configure ai --command-enable$ spyd configure ai --command-disable
configure monitoring
spyd configure monitoring
Interactive configuration for monitoring thresholds (CPU, Memory, Disk, Load Average).
configure alerts
spyd configure alerts [--channel CHANNEL]
Set up notification channels. Use --channel flag to configure a specific channel directly.
Check for and install the latest version. Automatically stops the daemon, installs the new binary, and restarts the daemon.
$ spyd upgradeCurrent version: 0.1.0
Checking for updates...
Latest version: 0.2.0
Downloading spyd_linux_amd64...
Stopping daemon...
Installing new version...
Restarting daemon...
Successfully upgraded to v0.2.0!# Check only (don't upgrade)$ spyd upgrade --check# Force upgrade even if same version$ spyd upgrade --force
uninstall
spyd uninstall [--yes]
Complete uninstallation with zero traces. Removes binary, configuration, data, logs, and service files. Use --yes to skip confirmation prompts.
$ spyd uninstall
Spyd Complete Uninstaller
This will remove Spyd and all its data with zero traces.
Continue with complete uninstall? [y/N] y
[1/7] Stopping Spyd daemon...
Stopping daemon (PID: 12345)...
[2/7] Removing systemd services...
[3/7] Skipping launchd (not macOS)
[4/7] Removing binary...
Removed: /usr/local/bin/spyd
[5/7] Removing v1 artifacts...
[6/7] Removing configuration, data, and logs...
Remove configuration? [y/N] y
Removed: ~/.config/spyd
Remove data and database? [y/N] y
Removed: ~/.local/share/spyd
[7/7] Cleaning up PID files...
Verifying cleanup...
Spyd has been completely uninstalled.# Skip all confirmations$ spyd uninstall --yes
version
spyd version [-f|--full]
Show version information. Use --full for banner and build details.
$ spyd versionSpyd Version 0.1.0$ spyd version --full[ASCII banner]
Version: 0.1.0
Commit: abc1234
Built: 2024-01-15T10:00:00Z
Go: go1.21.5
doctor
spyd doctor
Diagnose configuration and connectivity issues. Checks AI provider, notification channels, and system requirements.
$ spyd doctor
Spyd Doctor - Diagnosing your setup...
[Configuration]
Config file: ~/.config/spyd/config.yaml
Status: Valid
[AI Provider]
Provider: OpenAI
API Key: sk-...xxxx (set)
Connection: OK
[Notifications]
Telegram: OK (message sent)
Email: OK (SMTP connection successful)
[System]
OS: linux
Architecture: amd64
Memory: 8GB
Disk: 100GB free
All checks passed!
# Server identificationserver:name:my-server# AI Configurationai:enabled:trueprovider:openai# openai, anthropic, ollama, litellm, azureapi_key:sk-...model:gpt-4o-minitimeout:30# secondsinvestigation_mode:auto# auto, always, never# Monitoring Thresholdsmonitoring:interval:60# seconds between checksthresholds:cpu_percent:80memory_percent:85disk_percent:90load_average:0# 0 = auto (based on CPU cores)baseline:enabled:truelearning_days:7process_count:20# top processes to track# Alert Settingsalerts:dedup_window:300# seconds (5 minutes)max_per_hour:10min_severity:warning# info, warning, criticalobservation_window:180# seconds — anomalies must persist this longmin_consecutive_checks:3# must appear in N consecutive checksdaily_digest:enabled:false# opt-in daily summary of silently handled eventstime:"08:00"# HH:MM local time# Storagestorage:db_path:~/.local/share/spyd/spyd.dbretention_days:30max_size_mb:500# Daemondaemon:pid_file:/tmp/spyd.pidlog_level:info# debug, info, warn, errorhealth:enabled:trueport:19999
Note: AI command execution is enabled by default. The AI can only run safe, read-only commands (ps, df, tail, etc.) to investigate issues. All commands are whitelisted and audited.
Alert Channels
Spyd supports 6 notification channels. Configure them using spyd configure alerts or by editing the config file.
📧
Email
💬
Slack
📱
Telegram
🎮
Discord
🔔
Desktop (ntfy)
🔗
Webhooks
Email
SMTP-based email notifications. Supports Gmail, Outlook, and custom SMTP servers.
Subject: [Spyd] Alert on my-server
Spyd Alert — my-server
Your server's CPU has been maxed out at 98% for over 3 minutes.
A process called "xmrig" is using all available CPU — this is
a crypto miner and should not be on your server.
What to do: Kill the miner and investigate how it got installed.
Run: kill -9 12345 && find /tmp -name "xmrig*" -delete
Root cause: Cryptocurrency mining malware (xmrig) running as
www-data from /tmp/.hidden/xmrig, likely installed via
compromised web application.
--
Spyd Monitoring
Slack
Send alerts to Slack channels using incoming webhooks.
┌──────────────────────────────────────────────────────┐
│ Spyd Alert — my-server │
├──────────────────────────────────────────────────────┤
│ Your server's disk is 93% full and still growing. │
│ The largest consumers are Docker images (12GB) and │
│ old log files (8GB). │
│ │
│ *What to do:* Clean up Docker with │
│ `docker system prune -a` and rotate logs with │
│ `journalctl --vacuum-size=500M`. │
│ │
│ *Root cause:* Accumulated Docker images and │
│ unrotated logs consuming 20GB of reclaimable space. │
├──────────────────────────────────────────────────────┤
│ Spyd • my-server │
└──────────────────────────────────────────────────────┘
Telegram
Instant notifications via Telegram bot. Create a bot with @BotFather.
notifications:telegram:enabled:truebot_token:123456789:ABC-DEF1234ghIkl-zyx57W2v1u123ew11chat_id:123456789# or -100123456789 for groups
Spyd Alert — my-server
Your server's CPU has been maxed out at 98% for over
3 minutes. A process called "xmrig" is using all
available CPU — this is a crypto miner and should not
be on your server.
What to do: Kill the miner and investigate how
it got installed.
kill -9 12345 && find /tmp -name "xmrig*" -deleteRoot cause: Cryptocurrency mining malware running
as www-data from /tmp/.hidden/xmrig.
Discord
Send rich embed notifications to Discord channels via webhooks.
Setup: In Discord, go to Server Settings → Integrations → Webhooks → New Webhook. Copy the webhook URL.
🎮 Sample Discord Alert
┌──────────────────────────────────────────────────────┐
│ Spyd Alert — my-server │
├──────────────────────────────────────────────────────┤
│ Your server's CPU has been maxed out at 98% for │
│ over 3 minutes. A process called "xmrig" is using │
│ all available CPU — this is a crypto miner and │
│ should not be on your server. │
│ │
│ **What to do:** Kill the miner and investigate how │
│ it got installed. │
│ `kill -9 12345 && find /tmp -name "xmrig*" -delete` │
│ │
│ **Root cause:** Cryptocurrency mining malware │
│ running as www-data from /tmp/.hidden/xmrig. │
├──────────────────────────────────────────────────────┤
│ Spyd • my-server │
└──────────────────────────────────────────────────────┘
Desktop (ntfy)
Cross-platform push notifications via ntfy.sh. Works on phones, desktops, and browsers.
notifications:desktop:enabled:trueserver:https://ntfy.sh# or self-hostedtopic:my-server-alerts# your unique topictoken:# optional, for private topicspriority:# optional override (urgent, high, default)
Setup:
Install the ntfy app on your phone or desktop
Subscribe to your topic (e.g., my-server-alerts)
Configure Spyd with the same topic name
That's it! Alerts will push to all subscribed devices
🔔 Sample Desktop (ntfy) Alert
Title: Spyd Alert — my-server
Priority: urgent
Your server's CPU has been maxed out at 98% for over
3 minutes. A process called "xmrig" is using all
available CPU — this is a crypto miner and should not
be on your server.
What to do: Kill the miner and investigate how it got
installed. Run: kill -9 12345
Webhooks
Generic HTTP webhooks for custom integrations, PagerDuty, Opsgenie, or any service that accepts webhooks.
notifications:webhook:enabled:trueurl:https://your-service.com/webhook/spydmethod:POST# POST or GET
🔗 Sample Webhook Payload (JSON)
{
"id": "alert-1705312200",
"title": "Spyd Alert",
"message": "CPU maxed at 98% for 3+ minutes (xmrig crypto miner)",
"severity": "critical",
"metric": "cpu",
"value": 98.2,
"threshold": 80.0,
"server": "my-server",
"timestamp": "2026-03-05T10:30:00Z",
"diagnosis": {
"summary": "Crypto miner xmrig consuming all CPU...",
"root_cause": "Cryptocurrency mining malware running as www-data",
"what_to_do": "Kill the miner and investigate how it got installed",
"confidence": 0.95
}
}
Testing Alerts: Use spyd test-alert to verify your notification setup is working correctly before relying on it for real alerts.
AI Providers
Spyd supports multiple AI providers. Choose based on your needs: cloud providers for convenience, or local models for privacy.
OpenAI
Fast, accurate analysis with GPT-4o-mini or other OpenAI models. Requires API key.
ai:enabled:trueprovider:openaiapi_key:sk-...model:gpt-4o-mini# or gpt-4o, gpt-4-turbo
Anthropic (Claude)
Claude models for detailed, nuanced analysis. Requires API key.
Privacy Note: When using cloud providers (OpenAI, Anthropic, Azure), only redacted system summaries are sent for analysis. Sensitive information (usernames, paths, IPs) is automatically removed using configurable privacy patterns.
Alert Samples
Spyd's Silent Guardian system filters out noise and only sends clean, actionable alerts. Here's what you'll see — and what you won't.
Crypto Miner Detected
A crypto miner (xmrig) has been running at 100% CPU for 5+ minutes. This is a real threat — Spyd escalates immediately.
Result: 1 clean notification
Spyd Alert — my-server
Your server's CPU has been maxed out at 98% for over 3 minutes.
A process called "xmrig" is using all available CPU — this is a
crypto miner and should not be on your server.
What to do: Kill the miner and investigate how it got installed.
Run: kill -9 12345 && find /tmp -name "xmrig*" -delete
Root cause: Cryptocurrency mining malware (xmrig) running as
www-data from /tmp/.hidden/xmrig, likely installed via
compromised web application.
Disk at 93% and Growing
Disk usage is sustained above 90% and still climbing. Spyd identifies what's consuming space and tells you exactly how to reclaim it.
Result: 1 clean notification
Spyd Alert — my-server
Your server's disk is 93% full and still growing. The largest
consumers are Docker images (12GB) and old log files (8GB).
What to do: Clean up Docker with "docker system prune -a" and
rotate logs with "journalctl --vacuum-size=500M".
Root cause: Accumulated Docker images and unrotated logs
consuming 20GB of reclaimable space.
Transient CPU Spike (apt update)
A routine apt update causes CPU to spike to 87% for ~90 seconds, then returns to normal.
Result: 0 notifications
The observer drops it — the spike disappears before reaching
the 3-minute persistence gate. No AI analysis needed.
No notification sent. Nothing in your inbox.
Previously: 2 notifications (alert + "Resolved" spam)
TLS Handshake Errors (Bot Scanners)
Bot scanners hit your server generating 50+ TLS handshake failures per hour. These are routine internet noise.
Result: 0 notifications
The noise filter catches SSL/TLS patterns automatically.
These lines never even reach the rate-limit counter.
Suppressed patterns include:
- SSL_do_handshake() failed
- TLS handshake failed
- Connection reset by peer
- Broken pipe
- Connection timed out
Previously: 1 log alert notification
Memory Spike During Nightly Backup
Your nightly rsync backup causes memory to spike to 89% for a few minutes, then returns to normal.
Result: 0 notifications
Even if the spike persists past the observer, the AI recognizes
rsync/cron as a routine backup pattern and suppresses it.
AI reasoning: "Memory spike caused by rsync backup job running
as root via cron. This is a scheduled maintenance task that will
resolve on its own. No user action needed."
Previously: 2 notifications (alert + "Resolved" spam)
Daily Digest (Optional)
If enabled, Spyd sends one summary per day showing what it handled silently. Disabled by default.
Daily Summary
Daily Summary — my-server
Your server was healthy yesterday. Spyd handled 4 event(s)
silently:
- cpu spike (peak 87.3%, apt-get update running)
- memory spike (peak 89.4%, rsync backup job)
- CPU returned to normal (peak 87.3%, now 4.2%)
- Memory returned to normal (peak 89.4%, now 42.1%)
No action needed.
# Enable daily digest in config.yamlalerts:daily_digest:enabled:truetime:"08:00"# sends at 8:00 AM local time