Crystal-Agents — Operating & Instructions Manual

Version: v0.3.0
Date: 2026-06-22

This manual covers day-to-day operation: starting the system, running workflows, managing approvals, monitoring the scheduler, generating reports, and handling the dashboard.

1. Quick Start

Starting the Full Stack

# From the project root:
python -m orchestrator.cli integrated

This launches the dashboard (port 8000), scheduler loop, and Telegram bot (if enabled) as supervised subprocesses. Access the dashboard at http://<host>:8000.

Default dashboard credentials: set via CRYSTAL_DASHBOARD_USERNAME / CRYSTAL_DASHBOARD_PASSWORD env vars, or in config/config.json under dashboard.username / dashboard.password.

Starting Individual Components

# Dashboard only
python -m orchestrator.cli dashboard --port 8000

# Scheduler loop only
python -m orchestrator.cli scheduler loop

# Telegram bot only
python -m orchestrator.cli telegram

Docker Deployment

# Build and start all services
docker compose up --build -d

# Run a one-off workflow in a new container
docker compose run --rm crystal-agents run --task content --prompt "Draft a launch post."

# Pull and restart integrated service (production update)
docker compose pull integrated
docker compose up -d --force-recreate integrated

2. Running Workflows

2.1 From the CLI

# Content creation
python -m orchestrator.cli run --task content --prompt "Write a LinkedIn post about our Q2 results."

# Code workflow (feature specification)
python -m orchestrator.cli run --task feature_spec --prompt "Add export-to-CSV to the reports page."

# Git workflow (branch → code → PR)
python -m orchestrator.cli run --task git_workflow --prompt "Fix the pagination bug in the orders list."

# SEO audit
python -m orchestrator.cli run --task seo_audit --prompt "https://example.com"

# GEO scan
python -m orchestrator.cli run --task geo_scan --prompt "crystal erp software"

# Code audit
python -m orchestrator.cli run --task code_audit --prompt "Review authentication module for security issues."

# Chat (general purpose)
python -m orchestrator.cli run --task chat --prompt "What's the status of the Q3 marketing campaign?"

Available task types: content, content_review, content_publish, feature_spec, git_workflow, bug_triage, campaign_creation, lead_followup, proposal_generation, release_preparation, customer_reporting, business_onboarding, codebase_onboarding, seo_audit, code_audit, audit_fix_loop, geo_scan, chat

2.2 From the Dashboard

1. Open http://<host>:8000/home
2. Enter your prompt in the text area
3. Select a workflow from the dropdown (or leave blank for auto-routing)
4. Optionally select a GitHub repo
5. Click Submit

The run appears in the recent runs table. Click the title to view results.

2.3 From Microsoft Teams

Send a message to the configured Teams channel:

/status                           — List all scheduler jobs
/approve <key>                    — Approve a pending action
/deny <key>                       — Deny a pending action
Write a blog post about our AI   — Any other text triggers auto-routed workflow

Teams responses arrive in two parts:

1. Immediate acknowledgment: “Working on your request…”
2. Async result: A formatted Adaptive Card with the run summary (arrives when the workflow completes)

2.4 Auto-Routing Behavior

When no workflow is specified, the CEO Router classifies your prompt. Examples of keyword-triggered routing:

If no keyword matches, the prompt is sent to the LLM for classification.

3. Managing Approvals

Some actions require human approval (configured in config/policy.json). Currently these tools require approval:

• repo_writer — Writing files to the git workspace
• git_ops — Git operations (branch, commit, push, PR)
• publisher — Publishing content externally
• email_sender — Sending emails
• Certain scheduler jobs (e.g., ai_insights_campaign)

3.1 Via Dashboard

1. Navigate to Approvals in the sidebar
2. Pending approvals show with: action key, workflow context, timestamp, and description
3. Click Approve (green) or Deny (red)
4. Approved actions execute immediately; denied actions are skipped with an error logged

3.2 Via Teams

/approve abc123    — Approve pending action with key "abc123"
/deny abc123       — Deny pending action with key "abc123"

3.3 Approval Lifecycle

Pending (24h TTL) → Approved → Executed → Logged to approvals.jsonl
                   → Denied   → Skipped  → Logged to approvals.jsonl
                   → Expired  → Auto-denied after 24 hours

4. Scheduler

4.1 Viewing Scheduled Jobs

python -m orchestrator.cli scheduler list

Output:

Scheduled Jobs:
  salesorderemail     30 21 * * *   scripts/salesorderemail_scheduler.sh
  invoiceemail        35 21 * * *   scripts/invoiceemail_scheduler.sh
  indentemail         40 21 * * *   scripts/indentemail_scheduler.sh
  paymentsemail       45 21 * * *   scripts/paymentsemail_scheduler.sh
  receiptsemail       50 21 * * *   scripts/receiptsemail_scheduler.sh
  aremail             55 21 * * 1   scripts/aremail_scheduler.sh
  aiinsights           0  2 * * *   scripts/aiinsights_scheduler.sh
  check_s3_backup_one 15 22 * * *   scripts/check_s3_backup_one.sh
  check_s3_backup_two 20 22 * * *   scripts/check_s3_backup_two.sh
  web_health           0  9,11,13,15,17 * * 0-6  scripts/web_health.sh
  qreport              0 10-18 * * 1-6         scripts/qreport_scheduler.sh
  check_diskspace     30  9 * * *   scripts/check_diskspace.sh
  ai_insights_campaign 0 10 * * 1   scripts/send_campaign.sh [APPROVAL REQUIRED]

4.2 Running Due Jobs Manually

# Run all jobs that are due right now
python -m orchestrator.cli scheduler run-due

# Force-run a specific job (skips cron check)
python -m orchestrator.cli scheduler run-due --now

4.3 Via Dashboard

• Scheduler page (/scheduler): View all jobs, their schedules, and last run status
• Click Run Now next to any job to trigger it immediately
• Pending approvals appear in the Approvals page

4.4 Cron Schedule Reference

All jobs use standard 5-field cron:

minute  hour  day-of-month  month  day-of-week
  30     21       *           *         *        ← daily at 9:30 PM
  55     21       *           *         1        ← Mondays at 9:55 PM
   0   9,11,13,15,17  *       *        0-6      ← every 2 hours, 9am–5pm, every day
   0    10-18    *           *        1-6       ← hourly 10am–6pm, Mon–Sat

5. Reports

5.1 Generating Reports from CLI

# Generate daily and weekly summaries
python -m orchestrator.cli reports

Reports are written to logs/:

• daily_summary_<date>.md
• weekly_summary_<week>.md

5.2 Via Dashboard

• Daily Summary (/reports/daily): Aggregated run activity for today
• Weekly Summary (/reports/weekly): Aggregated run activity for the current week

5.3 What Reports Contain

• Total runs executed
• Breakdown by workflow type
• Approval actions (approved/denied/expired)
• Scheduler job execution history
• Notable outputs and errors

6. Dashboard Reference

6.1 Navigation

6.2 Run Details

Click any run title in the recent runs table to view:

• Result: The workflow’s final output (rendered markdown)
• Manifest: Raw JSON metadata (run ID, task type, prompt, timestamps, status)
• Status: completed, failed, running, or pending_approval

6.3 Common Dashboard Operations

Submit a workflow:

1. Go to /home
2. Fill the prompt textarea
3. Pick a workflow from the dropdown (or leave blank for auto-routing)
4. Click Submit

Approve a pending action:

1. Go to /approvals
2. Find the pending item by key and description
3. Click Approve (green) or Deny (red)

Manually run a scheduler job:

1. Go to /scheduler
2. Find the job in the table
3. Click Run Now

View a report:

1. Click Daily Summary or Weekly Summary in the sidebar under Reports

7. Configuration Reference

7.1 Environment Variables

All secrets should be set as environment variables, not in config/config.json:

# Model / LLM
CRYSTAL_MODEL_API_KEY=sk-...          # API key for the LLM provider

# Dashboard
CRYSTAL_DASHBOARD_USERNAME=admin
CRYSTAL_DASHBOARD_PASSWORD=<strong-password>

# Teams
CRYSTAL_TEAMS_WEBHOOK_URL=https://... # Teams incoming webhook URL
CRYSTAL_TEAMS_SECURITY_TOKEN=...      # HMAC secret for Teams auth

# Telegram
CRYSTAL_TELEGRAM_BOT_TOKEN=...        # Bot token from @BotFather
CRYSTAL_TELEGRAM_CHAT_ID=...          # Default chat ID

# Backblaze B2
CRYSTAL_B2_APPLICATION_KEY_ID=...
CRYSTAL_B2_APPLICATION_KEY=...

7.2 Key Config File Settings (config/config.json)

{
  "model": {
    "provider": "openai",           // API format (openai-compatible)
    "name": "deepseek-v4-flash",    // Model name sent to API
    "temperature": 0.1,             // 0.0–1.0, lower = more deterministic
    "max_tokens": 4096,             // Max response tokens
    "context_window_tokens": 1000000,
    "api_base": "https://api.deepseek.com/v1"
  },
  "scheduler": {
    "enabled": true,                // Master switch for scheduler
    "poll_interval_seconds": 60     // How often to check for due jobs
  },
  "git": {
    "workspace_root": "D:/Esh-Projects/crystalagentsworkspace",
    "github_repo": "eshwar-crystal/crystal-agents",
    "pr_draft": true                // Create PRs as drafts
  }
}

7.3 Policy Configuration (config/policy.json)

Controls which agents can use which tools and which tools require approval:

{
  "agent_tools": {
    "developer": ["repo_reader", "repo_writer", "git_ops", "web_search"],
    "content": ["web_search", "repo_reader", "publisher"]
  },
  "tool_approval_required": {
    "repo_writer": true,
    "git_ops": true,
    "publisher": true,
    "email_sender": true
  },
  "default_tools": ["web_search"],
  "default_approval_required": false
}

8. Common Workflows — Step-by-Step Examples

Example 1: Content Creation (Blog Post)

Goal: Write and review a blog post about a new product feature.

CLI:

python -m orchestrator.cli run --task content --prompt "Write an 800-word blog post announcing our new real-time analytics dashboard. Target audience: small business owners. Tone: professional but approachable."

What happens:

1. ResearchAgent searches for context about “real-time analytics dashboard” and “small business”
2. ContentAgent drafts the post using the research
3. ReviewerAgent reviews for tone, accuracy, and completeness
4. Final output saved to runs/<run_id>/result.md
5. Teams notification sent with summary

Dashboard equivalent: Go to /home, paste the prompt, select “content” from dropdown, submit.

Example 2: Code Feature via Git Workflow

Goal: Implement a feature, commit to a branch, and create a PR.

CLI:

python -m orchestrator.cli run --task git_workflow --prompt "Add rate limiting to the API endpoints using a token bucket algorithm. Limit: 100 requests per minute per IP."

What happens:

1. DeveloperAgent reads the current codebase
2. DeveloperAgent writes the rate limiting code (approval required)
3. You approve the file write via dashboard or /approve <key> in Teams
4. ReviewerAgent reviews the changes
5. DeveloperAgent creates a branch, commits, and opens a PR (approval required for git_ops)
6. You approve the git operations

Example 3: SEO Audit

Goal: Audit a website for SEO issues.

CLI:

python -m orchestrator.cli run --task seo_audit --prompt "https://example.com"

What happens:

1. SEOAgent crawls the website (up to crawling_limit pages, default 10)
2. ReviewerAgent reviews findings
3. Report generated with: meta tag issues, heading structure, keyword density, performance notes, broken links
4. Results saved to runs/<run_id>/result.md

Example 4: Customer Report Generation

Goal: Generate a report for a client.

CLI:

python -m orchestrator.cli run --task customer_reporting --prompt "Generate the monthly performance report for client 'Acme Corp'. Include: traffic stats, conversion rates, content performance, and recommendations."

What happens:

1. AnalyticsAgent gathers data and constructs the report
2. Report saved to generated/customer_reports/
3. Available for download via dashboard

Example 5: Scheduled Email Automation

Goal: Automated daily sales order emails.

Config (already in config/config.json):

{
  "name": "salesorderemail",
  "schedule": "30 21 * * *",
  "script": "scripts/salesorderemail_scheduler.sh",
  "enabled": true,
  "requires_approval": false
}

The scheduler runs scripts/salesorderemail_scheduler.sh every day at 9:30 PM. No manual intervention needed — it runs automatically as long as scheduler.enabled is true.

Checking results: Go to /scheduler in the dashboard to see execution history (status, exit code, stdout/stderr).

9. Troubleshooting

Dashboard won’t start

• Check port is available: netstat -an | findstr 8000
• Verify config/config.json is valid JSON
• Check env vars are set correctly

Workflow stuck on “pending_approval”

• Check /approvals in dashboard for pending items
• Approvals expire after 24 hours; check for expired items
• Verify Teams webhook is reachable if using Teams for approvals

Scheduler jobs not running

• Verify scheduler.enabled: true in config/config.json
• Check cron expression is valid (use scheduler list to verify parsing)
• Ensure scripts exist and are executable
• Check logs/scheduler_history.jsonl for error details

Teams notifications not arriving

• Verify teams.enabled: true and webhook_url is set
• The webhook URL is a secret — set it via CRYSTAL_TEAMS_WEBHOOK_URL env var
• Check that the Teams channel’s incoming webhook connector is active
• Test manually: curl -X POST <webhook_url> -H "Content-Type: application/json" -d '{"text":"test"}'

LLM calls failing

• Verify API key is set via CRYSTAL_MODEL_API_KEY
• Check api_base URL is reachable from the server
• Check model.name matches what the provider expects
• Increase max_tokens if responses are truncated

“Workflow not found” error

• Use --task chat for general questions (it’s the catch-all)
• Check the task type spelling against the available list in Section 2.1
• Note: publish is NOT a valid task type — use content_publish

10. File Locations Reference

11. Security Notes

• Never commit .env or any file containing API keys to the repository
• Set all secrets as environment variables, not in config/config.json
• The dashboard uses HTTP Basic Auth — run it behind a reverse proxy with TLS in production
• Teams webhook endpoints are not protected by dashboard auth; they use separate HMAC verification
• The scheduler only runs scripts from the configured allowed_script_roots directories
• Tool approvals are the primary security boundary — review config/policy.json carefully before modifying
• Rotate the dashboard password and all API keys regularly