Crystal Agents — End-to-End Test Plan

Purpose: Verify all functionality after Docker deployment
Pre-requisite: Docker is up (docker compose ps shows all containers Running)
Test approach: Manual black-box testing via Telegram and Dashboard

Test Environment Checklist

Before starting, confirm:

• docker compose ps — all 3 containers (dashboard, telegram, scheduler) show Up
• Dashboard loads at http://<vps-ip>:8000 and login works
• Telegram bot responds to a test message
• .env has CRYSTAL_AGENTS_BOT_TOKEN, OPENAI_API_KEY (or GOOGLE_API_KEY) set

Test Area 1 — System Startup & Config

TC-01: Dashboard Login

Steps:

1. Open http://<vps-ip>:8000 in a browser
2. Enter credentials from config.json → dashboard.username / password

Expected: Dashboard home page loads, shows “Crystal Dashboard” heading and model name.

TC-02: Model Name Shown on Dashboard

Steps:

1. Look at the home page hero section
2. Note the “Model:” text under the heading

Expected: Shows the model name from config.json (e.g. deepseek-v4-flash).

TC-03: Change AI Model via Config

Steps:

1. Edit config/config.json — change model.name to a different value (e.g. deepseek-r1)
2. Restart: docker compose restart telegram dashboard
3. Open Dashboard home

Expected: The new model name appears on the home page.

Restore: Change back to original model after the test.

Test Area 2 — Telegram Interface

TC-04: Bot Responds to a Simple Message

Steps:

1. Open Telegram, send: Hello, what can you do?

Expected: Bot responds within ~30 seconds with a description of capabilities.

TC-05: /status Command

Steps:

1. Send /status in Telegram

Expected: Bot replies with a list of scheduled jobs (salesorderemail, web_health, etc.)

TC-06: Natural Language Routing — Content

Steps:

1. Send: Draft a short LinkedIn post about AI automation for small businesses

Expected:

• Bot sends “🔄 Working on your request…”
• Within 60–90 seconds, bot sends the drafted LinkedIn post
• Dashboard → Home shows a new run with workflow: content and status: succeeded

TC-07: Natural Language Routing — Chat

Steps:

1. Send: What is GEO (Generative Engine Optimization)?

Expected: Bot responds conversationally explaining GEO, without triggering a workflow run.

TC-08: Slash Command — /seo_audit

Steps:

1. Send: /seo_audit https://example.com

Expected:

• Bot acknowledges and runs the audit
• Report appears in generated/ directory
• Dashboard → Audits shows new audit entry

TC-09: Slash Command — /geo_scan

Steps:

1. Send: /geo_scan Crystal Agents

Expected:

• Bot runs the GEO scan workflow
• Returns citation scores for ChatGPT, Claude, Perplexity, Grok
• Dashboard → GEO shows the new scan entry

TC-10: Slash Command — /code_audit

Steps:

1. Send: /code_audit . (the dot scans the Crystal Agents repo itself)

Expected:

• Bot runs code audit
• Dashboard → Issues shows detected code issues
• Dashboard → Audits shows new audit entry

Test Area 3 — The Approval System

TC-11: Approval via Dashboard

Steps:

1. Send via Telegram: /dev Add a comment to the README
2. Watch for a Telegram message saying approval is needed
3. Go to Dashboard → Approvals
4. Find the pending item and click Approve

Expected:

• Before approval: run is paused, approval badge on header shows 1
• After approval: run continues, Telegram sends the result
• Dashboard → Approvals shows item as approved

TC-12: Denial via Telegram

Steps:

1. Trigger any write action (e.g. /dev Add a test file)
2. Wait for the Telegram approval notification with action_key
3. Reply: /deny <action_key>

Expected:

• Bot replies: “❌ Denied: repo_writer for run …”
• Dashboard → Approvals shows item as denied
• Run completes with status failed or partial

TC-13: Approve via Telegram

Steps:

1. Repeat TC-11 but this time, instead of using the Dashboard, reply in Telegram: /approve <action_key> (use the key from the approval notification)

Expected: Same result as TC-11 — run completes successfully.

Test Area 4 — Workflow Tests

TC-14: Content Workflow

Steps:

1. Dashboard → Home → Type: Write a short blog post about supply chain automation
2. Select workflow: Content Draft
3. Click Run Workflow
4. Confirm the dialog (verify workflow name and prompt preview appear)

Expected:

• Confirmation dialog shows correct workflow and prompt
• Button disables after click
• Run appears in recent runs table
• Telegram notification sent
• Run status: succeeded

TC-15: Run Flow Confirmation Dialog — Cancel Test

Steps:

1. Dashboard → Home → Enter any prompt
2. Click Run Workflow
3. When the confirmation dialog appears, click Cancel

Expected:

• Form does NOT submit
• Button re-enables
• No new run created

TC-16: SEO Audit Workflow

Steps:

1. Telegram: Run an SEO audit on https://example.com

Expected:

• seo_audit workflow runs
• Audit report generated in generated/
• Dashboard → Audits shows the new entry with a score

TC-17: GEO Scan Workflow

Steps:

1. Dashboard → Home → Prompt: Crystal Agents brand
2. Workflow: GEO & Citation Scan
3. Run Workflow

Expected:

• Workflow completes
• Dashboard → GEO shows new citation scan with platform breakdown (ChatGPT, Claude, Perplexity, Grok)
• History entry saved to logs/geo.jsonl

TC-18: Business Onboarding Workflow

Steps:

1. Telegram: /onboard_website https://example.com

Expected:

• Workflow scans website, identifies competitors, analyzes brand voice
• Memory files created/updated: memory/company.md, memory/product.md, memory/marketing.md, memory/pricing.md, memory/soul.md
• Run succeeds

Verify:

cat memory/company.md
# Should contain content derived from example.com

TC-19: Codebase Onboarding Workflow

Steps:

1. Telegram: /onboard_codebase . (scan the Crystal Agents repo itself)

Expected:

• Directory tree is mapped
• Languages detected (Python, Shell Script, etc.)
• memory/architecture.md and memory/coding_standards.md updated with Crystal Agents codebase details
• AGENTS.md updated

TC-20: Content Publish Workflow (with approval)

Steps:

1. Telegram: Draft and publish a post about AI agents on LinkedIn

Expected:

• content_publish workflow runs
• Content is drafted
• Approval request sent to Telegram before publishing
• After approval: post logged to logs/publishes.jsonl
• Dashboard → Publishes shows the new entry

TC-21: Audit → Fix → PR Loop

Steps:

1. Telegram: /audit_fix_loop .

Expected:

• Code audit runs first
• Highest priority issue selected
• git_workflow triggered to fix it
• Approval request sent before pushing
• After approval: branch created, changes committed

TC-22: Git Workflow — End to End

Steps:

1. Telegram: /dev Add a docstring to the ReportGenerator class in reports/summary.py

Expected:

• Plan step: agent analyses the task
• Branch step: ai/task-<run_id> branch created in git.workspace_root
• Code step: agent reads the file and prepares a change
• Push step: PAUSED — approval request sent
• After approval: branch pushed to GitHub, Draft PR created
• Telegram notification with PR URL

Test Area 5 — Dashboard Pages

TC-23: Action Feed

Steps:

1. After running any workflow, go to Dashboard → Action Feed (/actions)

Expected: Shows a timestamped list of every tool call made by agents (e.g. web_search: searched for..., repo_writer: wrote to memory/company.md)

TC-24: Issues Page

Steps:

1. After running TC-10 (code_audit), go to Dashboard → Issues (/issues)

Expected: Shows list of found issues with severity, file, line number, and status (open/resolved)

TC-25: Publishes Page

Steps:

1. After running TC-20 (content_publish), go to Dashboard → Publishes (/publishes)

Expected: Shows publish log entry with platform, title, timestamp, and status

TC-26: GEO Page

Steps:

1. After running TC-17 or TC-09, go to Dashboard → GEO (/geo)

Expected: Shows citation history with platform breakdown table and authority scores

TC-27: Approvals Page — Badge Count

Steps:

1. Trigger a workflow that requires approval (TC-11)
2. Check the Dashboard header

Expected: “Approvals” chip shows a red badge with the pending count number

TC-28: Scheduler Page

Steps:

1. Go to Dashboard → Scheduler (/scheduler)

Expected: Shows all configured jobs, their schedules, descriptions, and recent run history

Test Area 6 — Scheduler

TC-29: Manual Job Trigger via Telegram

Steps:

1. Send: Run the web_health job

Expected: Bot triggers web_health script and replies with the output/result

TC-30: Scheduled Job Runs Automatically

Steps:

1. Note the next scheduled time for check_diskspace (9:30 AM)
2. Wait until that time (or temporarily change the schedule to run in 2 minutes for testing)
3. Check Telegram

Expected: Telegram notification received with the disk space check results

To test without waiting: Edit config.json, change check_diskspace schedule to run in 2 minutes from now (e.g. "35 22 * * *" for 10:35 PM), restart scheduler, watch Telegram.

Test Area 7 — Reports

TC-31: Daily Report via Dashboard

Steps:

1. Go to Dashboard → Daily Summary (/reports/daily)

Expected: Report shows:

• Runs count for today
• Pending approval count
• ## Git Activity section
• ## Scheduler Health section (jobs run, pass/fail)
• Recent approval events

TC-32: Weekly Report via Dashboard

Steps:

1. Go to Dashboard → Weekly Summary (/reports/weekly)

Expected: Same sections as daily but covering 7 days.

TC-33: Report via CLI

Steps:

docker compose run --rm dashboard python -m orchestrator.cli reports

Expected: Daily and weekly summaries written to logs/daily_summary.md and logs/weekly_summary.md without errors.

TC-34: Git Activity in Reports (after TC-22)

Steps:

1. After completing TC-22 (git_workflow run), go to /reports/daily

Expected: ## Git Activity section shows the git_workflow run with run ID and status.

Test Area 8 — Session Memory (Multi-turn Conversation)

TC-35: Conversation Context Retained

Steps:

1. Send: Write a LinkedIn post about supply chain disruptions
2. Wait for response
3. Send: Now make it shorter and more casual

Expected: Second response is a shorter, more casual version of the first post — demonstrating that the agent remembered the conversation context.

Test Summary Table

Priority Order for Testing

Run tests in this order to build confidence progressively:

1. P0 — Core (must pass): TC-01, TC-04, TC-06, TC-14, TC-15
2. P1 — Critical paths: TC-02, TC-05, TC-08, TC-11, TC-13, TC-31
3. P2 — Workflows: TC-16, TC-17, TC-18, TC-19, TC-22
4. P3 — Dashboard pages: TC-23 through TC-28
5. P4 — Scheduler: TC-29, TC-30
6. P5 — Advanced: TC-20, TC-21, TC-34, TC-35