AI-driven Continuous Testing
System Overview
| Area |
Role |
| VS Code |
Common entry point for Testing, Tasks, debugging, and environment setup |
| GitHub Actions |
CI entry point for hosted Mock/unit tests and optional self-hosted hardware tests |
| pytest CT |
Runs TEST ID scenarios using a final mock or hil fixture mode |
| unittest |
Runs function-level framework and extension tests without TEST ID or fixture mode |
| Result pipeline |
Normalizes execution data and generates Markdown reports |
| MCP server |
Allows an external local MCP host to discover and run allowlisted Pytest/Unittest operations through stdio |
| Local LLM |
Analyzes pytest CT results only; it is not used for unittest |
| MkDocs |
Publishes pytest and unittest Markdown and execution history |
Go to VSCode Testing
MCP Server Role
test_envs.mcp_server is the local test-control boundary for an MCP-capable host such as Codex or VS Code. It runs through stdio; it does not open a network port or accept arbitrary shell commands.
| Responsibility |
MCP server behavior |
| Environment discovery |
Returns OS, Python/venv, Ollama model, Pandoc, and execution permission status before a test starts |
| Allowlisted execution |
Lists and runs only cataloged Pytest TEST IDs and defined Unittest scopes; arbitrary paths and node IDs are rejected |
| Test orchestration |
Serializes requests, runs the selected test process, then returns normalized status and artifact paths as structured MCP output |
| Report operations |
Generates or refreshes Markdown/Pandoc reports; MkDocs publication is an explicit opt-in operation |
| Safety boundary |
HIL requires PYTEST_HIL_ALLOW=true; documentation publication requires MKDOC_UPDATE=true; both are disabled by default |
| CI separation |
Local MCP is a direct, local control path. GitHub Issue Forms continue to route only to GitHub-hosted or self-hosted runners |
flowchart TD
HOST[Codex / VS Code] -->|stdio MCP| SERVER[test_envs.mcp_server]
SERVER -->|allowlisted request| TEST[Pytest CT or Unittest]
TEST --> RESULT[Normalized result and reports]
RESULT -->|structured response| HOST
For the tool list, configuration, and transport details, see MCP Environment.
Documents
| Scope |
Document |
Components |
| Python Environment |
python_environment.md |
OS, Python, venv, dependencies |
| Local LLM Environment |
local_llm_environment.md |
Ollama, model, prompt, check |
| VS Code Environment |
vscode_environment.md |
Settings, Launch, Tasks, Testing |
| MCP Environment |
mcp_environment.md |
External stdio tools, execution controls, HIL safety |
| Pytest Operation |
pytest_operation.md |
TEST ID, Fixture, Mock/HIL, Local LLM |
| Unittest Operation |
unittest_operation.md |
Function, Execution ID, Result, Markdown |
| Pytest / HIL / Mock |
pytest_framework.md |
Test cases, fixture mapping, mode selection, HIL gate |
| Unittest |
unittest_framework.md |
Function result contract |
| Pytest Results |
tests/pytest/index.md |
TEST ID, Execution history |
| Unittest Results |
tests/unittest/index.md |
Function count, Execution history |
TEST Flow
The project supports three test-entry forms. Each form runs the same allowlisted test pipeline and creates execution artifacts: normalized result JSON and log files, canonical Markdown, published test-history Markdown, and optional Pandoc HTML/DOCX files. Test source files are selected from the catalog or scope; execution creates result and report files rather than arbitrary new test source code.
| Test-entry form |
Request and execution |
Generated documentation and files |
| VS Code |
Testing explorer, Run and Debug, or Tasks runs local Pytest CT and Unittest |
Result JSON/log → Markdown → optional HTML/DOCX; MkDocs history is updated when requested |
| AI Agent MCP |
Codex calls get_test_envs, selects an allowlisted TEST ID or scope, then calls an MCP test tool |
Structured MCP result with result/log/Markdown paths; optional HTML/DOCX conversion uses the same report pipeline |
| GitHub Issue and Actions |
A GitHub Issue Form or workflow_dispatch request is normalized, labeled, and run by a GitHub-hosted or self-hosted runner |
Workflow artifacts and canonical Markdown; selected HTML/DOCX reports are generated by the same Pandoc conversion path |
flowchart TD
VSCODE[VS Code]
AGENT[Codex AI Agent / MCP]
ISSUE[GitHub Issue Form or workflow_dispatch]
TEST[Allowlisted Pytest CT or Unittest]
RESULT[result JSON and log]
MARKDOWN[Canonical Markdown and test history]
PANDOC[Optional HTML / DOCX]
VSCODE --> TEST
AGENT --> TEST
ISSUE --> TEST
TEST --> RESULT --> MARKDOWN --> PANDOC
VS Code-based flow
Go To VS Code Testing Setup
Go To VS Code Testing
flowchart TD
VSCODE[VS Code]
EXTENSION[VS Extension Testing]
TASK[VS Code Task]
ENV[TEST System Environment]
PYTEST[Pytest Operation]
UNITTEST[Unittest Operation]
PYTEST_RESULT[Pytest Result]
UNITTEST_RESULT[Unittest Result]
LLM[Local LLM]
REPORTER["**src:test_envs/tools/mkdocs_reporter**<br/>MarkdownReporter"]
MARKDOWN["Pytest / Unittest Markdown"]
MKDOCS["MkDocs (TEST Results)"]
PANDOC_REPORTER["**src:test_envs/tools/pandoc_reporter**<br/>convert"]
PANDOC[Pandoc Report - HTML / DOCX]
VSCODE --> EXTENSION
VSCODE --> TASK
TASK --> ENV
ENV -. Runtime and configuration .-> EXTENSION
EXTENSION --> PYTEST
EXTENSION --> UNITTEST
TASK --> PYTEST
PYTEST --> PYTEST_RESULT --> LLM --> REPORTER
UNITTEST --> UNITTEST_RESULT --> REPORTER
TASK --> REPORTER
REPORTER --> MARKDOWN
MARKDOWN -. Manual --docs .-> MKDOCS
MARKDOWN --> PANDOC_REPORTER --> PANDOC
TASK --> MKDOCS
TASK --> PANDOC_REPORTER
classDef testResults fill:#fff3bf,stroke:#f08c00,stroke-width:4px,color:#5f3d00,font-weight:bold
class MKDOCS testResults
| VS Code entry |
Execution scope |
| VS Extension Testing |
Discovers and runs both pytest CT and unittest through the pytest adapter |
| VS Code Task |
Prepares the environment, runs pytest CT, generates MkDocs reports, and converts the latest Markdown to Pandoc HTML/DOCX |
| Common Markdown reporter |
test_envs/tools/mkdocs_reporter renders both pytest and unittest results through MarkdownReporter |
| Pandoc reporter |
test_envs/tools/pandoc_reporter converts the latest common Markdown report to HTML or DOCX |
Automatically Generate Report
Analyze Result.log and result.json using an LLM,
then automatically update the analysis report in the MkDocs documentation.
AI Agent-based MCP flow
Codex or an Ollama-powered AI Agent can use the local ai-driven-ci-ct MCP server as a direct test-control interface. The agent first inspects the environment and allowlisted inventory, selects an approved test or scope, then receives the normalized result and generated report paths as structured tool output. Ollama supplies the local model reasoning; the MCP server performs the allowlisted test execution.
flowchart TD
AGENT[Codex or Ollama AI Agent]
ENV[get_test_envs]
INVENTORY[get_test_list_pytest / get_test_list_unittest]
SELECT[Select allowlisted TEST ID or Unittest scope]
MCP[test_envs.mcp_server]
PYTEST[run_test_pytest]
UNITTEST[run_test_unittest]
REPORT[Markdown / Pandoc reports]
RESPONSE[Structured MCP result]
AGENT --> ENV --> INVENTORY --> SELECT --> MCP
MCP --> PYTEST --> REPORT --> RESPONSE --> AGENT
MCP --> UNITTEST --> REPORT
classDef safety fill:#fff3bf,stroke:#f08c00,stroke-width:2px,color:#5f3d00
class SELECT safety
test_envs/configs/test_config.json maps an AI Agent command to an allowlisted YAML rule under test_envs/configs/test_rules/. For example, default, uart, usb, network, and all select their registered rule; selected runs listed Pytest TEST IDs and Unittest scopes, while all runs the corresponding catalog or declared scope. Run get_ollama_test_plan(command="usb") to inspect a selection, then run_ollama_test_workflow(command="usb") to execute it.
flowchart TD
CONFIG[test_config.json command map] --> PLAN[test_rules/*.yaml]
PLAN --> SELECT{1. Select test type and mode}
SELECT --> PYTEST[Pytest: all or selected TEST IDs]
SELECT --> UNITTEST[Unittest: all or selected scopes]
PYTEST --> LOG[2. Test log and normalized result]
UNITTEST --> LOG
LOG --> MARKDOWN[Markdown report]
MARKDOWN --> FORMAT{3. Pandoc output}
FORMAT --> DOCX[DOCX]
FORMAT --> HTML[HTML]
PYTEST --> OLLAMA[Ollama result analysis]
OLLAMA --> MARKDOWN
| AI Agent capability |
MCP boundary |
| Inspect readiness |
Reads OS, venv, Ollama, Pandoc, and permission status before execution |
| Automate with Ollama |
An Ollama-powered agent can inspect the catalog, choose an approved test, call the MCP test tool, and review the structured result |
| Choose a test |
Uses only cataloged Pytest TEST IDs and declared Unittest scopes |
| Run and review |
Starts an allowlisted operation and receives status, logs, and report paths without parsing terminal output |
| Use Local LLM analysis |
Pytest report generation can invoke the configured local Ollama model; unittest does not invoke Local LLM analysis |
| Keep safety controls |
HIL remains blocked until PYTEST_HIL_ALLOW=true; MkDocs publication remains blocked until MKDOC_UPDATE=true |
This is a local MCP path, separate from GitHub Issue and GitHub Actions execution. Ollama does not receive unrestricted shell access: test execution remains limited to the MCP server's cataloged Pytest TEST IDs and declared Unittest scopes.
GitHub Actions-based flow
flowchart TD
PUSH["Push to main<br/>Issue Form or Workflow files"]
PYTEST_ISSUE["pytest_request.yml<br/>Issue opened / edited / reopened"]
UNITTEST_ISSUE["unittest_request.yml<br/>Issue opened / edited / reopened"]
CHECK_ISSUE["test_check.yml<br/>Issue opened / edited / reopened"]
MANUAL[workflow_dispatch]
subgraph LABEL_JOB["labels job · ubuntu-latest"]
LABELS["Create test-request-runner<br/>and test-check-runner labels"]
end
subgraph REQUEST_JOB["request job · ubuntu-latest"]
CHECKOUT_REQUEST[Checkout]
APPLY_LABEL[Ensure and apply matching Issue label]
PARSER[test_envs.tool_github.issue_parser]
OUTPUTS["Normalize request_kind, test settings,<br/>revision, reports, and runner_labels"]
CHECKOUT_REQUEST --> APPLY_LABEL --> PARSER --> OUTPUTS
end
subgraph TEST_JOB["test job · runs-on: runner_labels"]
CHECKOUT_TEST[Checkout requested revision]
SETUP_PYTHON[Setup Python 3.12]
REQUEST_KIND{request_kind}
ENV_CHECK["TEST-CHECK<br/>Host / OS / Python / Ollama"]
TEST_TYPE{test_type}
PYTEST["Pytest CT<br/>TEST ID + marker / mock / hil"]
UNITTEST["Unittest through pytest<br/>All or CT Framework"]
PYTEST_RESULT[Pytest Result]
UNITTEST_RESULT[Unittest Result]
COVERAGE[Optional Coverage]
PIPELINE[test_envs.test_pipeline.pipeline]
LLM["Local LLM<br/>Pytest only"]
REPORTER["test_envs/tools/mkdocs_reporter<br/>Canonical Markdown"]
MKDOCS["MkDocs (TEST Results)<br/>when report_mkdocs is enabled"]
PANDOC["test_envs.tools.pandoc_reporter<br/>optional HTML / DOCX"]
GITHUB_REPORTER[test_envs.tool_github.github_reporter]
ISSUE[GitHub Issue Comment]
ARTIFACT["GitHub Artifact<br/>results / reports / coverage"]
CHECKOUT_TEST --> SETUP_PYTHON --> REQUEST_KIND
REQUEST_KIND -->|environment-check| ENV_CHECK --> GITHUB_REPORTER
REQUEST_KIND -->|test| TEST_TYPE
TEST_TYPE -->|Pytest| PYTEST --> PYTEST_RESULT
TEST_TYPE -->|Unittest| UNITTEST --> UNITTEST_RESULT
PYTEST_RESULT --> COVERAGE
UNITTEST_RESULT --> COVERAGE
PYTEST_RESULT --> PIPELINE
UNITTEST_RESULT --> PIPELINE
PIPELINE -->|Pytest analysis| LLM --> REPORTER
PIPELINE -->|Unittest · no LLM| REPORTER
REPORTER -. report_mkdocs .-> MKDOCS
REPORTER -. report_html / report_docx .-> PANDOC
PIPELINE --> GITHUB_REPORTER --> ISSUE
PYTEST_RESULT --> ARTIFACT
UNITTEST_RESULT --> ARTIFACT
COVERAGE --> ARTIFACT
REPORTER --> ARTIFACT
MKDOCS --> ARTIFACT
PANDOC --> ARTIFACT
end
PUSH --> LABELS
PYTEST_ISSUE --> CHECKOUT_REQUEST
UNITTEST_ISSUE --> CHECKOUT_REQUEST
CHECK_ISSUE --> CHECKOUT_REQUEST
MANUAL --> CHECKOUT_REQUEST
OUTPUTS -->|GitHub-hosted or Self-hosted Runner| CHECKOUT_TEST
classDef testResults fill:#fff3bf,stroke:#f08c00,stroke-width:4px,color:#5f3d00,font-weight:bold
class MKDOCS testResults
| GitHub Actions entry |
Execution scope |
pytest_request.yml |
Pytest-only form: TEST ID, Fixture mode, Test Environment, OS, revision, Coverage, and optional Pandoc |
unittest_request.yml |
Unittest-only form: scope, Test Environment, OS, revision, Coverage, and optional Pandoc; no TEST ID or Fixture mode |
test_check.yml |
Selects Test Environment and OS; the workflow detects Python and Ollama state and comments on the Issue |
continuous-test.yml |
Routes GitHub Issue requests to GitHub-hosted or GitHub Self-hosted runners |
| Pytest TEST ID catalog |
test_catalog.json is generated from Python CT markers; Setup 4 synchronizes the VS Code, Issue Form, and workflow-dispatch choices |
| Common Markdown reporter |
test_envs/tools/mkdocs_reporter always renders the canonical Markdown; MkDocs publication is enabled separately and the generated reports are uploaded as artifacts |
Github Issues and Automatically Generate Report
continuous-test.yml
| Workflow |
Primary role |
Local LLM rule |
continuous-test.yml |
Unified hosted, Windows, and self-hosted HIL test-request automation |
Pytest uses Local LLM with deterministic fallback; Unittest bypasses Local LLM |
| Item |
Current behavior |
| Workflow name |
Test Request |
| Trigger |
Request Issue opened, edited, or reopened; or manual workflow_dispatch |
| Duplicate prevention |
The workflow does not subscribe to labeled; automatic label attachment therefore does not create a second run |
| Request Job |
Detects Pytest, Unittest, or TEST-CHECK from the form title and emits normalized execution settings |
| Issue labels |
A relevant default-branch push creates both labels; the request job also creates and applies the matching label as a first-Issue fallback |
| Runner routing |
GitHub Issue requests select GitHub-hosted or Self-hosted Runner plus Ubuntu/Windows; Local MCP is not an Issue option |
| Timeout |
60 minutes |
| Permissions |
Repository contents read; issues write |
| Python |
actions/setup-python@v5, Python 3.12, pip cache |
| Environment |
Test requests create .venv and install requirements.txt; TEST-CHECK inspects the selected runner directly |
| Unittest |
Runs All Unittest or CT Framework Python through pytest without Local LLM analysis |
| Pytest CT |
GitHub-hosted Ubuntu/Windows supports Mock only; physical HIL requires the selected GitHub Self-hosted hardware runner |
| Coverage |
Optional terminal or HTML pytest-cov report |
| Report |
Issue Forms normalize Log, canonical Markdown, Pandoc DOCX, and Pandoc HTML separately |
| MkDocs publication |
report_mkdocs copies the generated Markdown into docs/tests; Markdown generation itself is represented by report_markdown |
| Issue output |
github_reporter posts the existing canonical Test Report Markdown, including its Result JSON-based test_envs table |
| Artifact |
Uploads results, MkDocs pages, .coverage, and htmlcov/ |
| Node.js |
No project Node.js setup or command; official GitHub Actions manage their own embedded runtime |
Test System
.
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── pytest_request.yml
│ │ ├── unittest_request.yml
│ │ └── test_check.yml
│ ├── workflows/
│ │ ├── continuous-test.yml
│ │ └── github_pages.yaml
│ └── pull_request_template.md
├── .vscode/
│ ├── launch.json
│ ├── mcp.json
│ ├── settings.json
│ └── tasks.json
├── docs/
│ ├── index.md
│ ├── mcp_environment.md
│ ├── vscode_environment.md
│ ├── python_environment.md
│ ├── local_llm_environment.md
│ ├── pytest_{framework,operation}.md
│ ├── unittest_{framework,operation}.md
│ └── tests/{pytest,unittest}/
├── docx/
│ ├── imgs/
│ ├── lua/
│ ├── reference/reference.docx
│ └── setup_reference.json
├── test_envs/
│ ├── configs/{config_env.json,config_ollama.json,check_envs.json,check_ollama.json,test_config.json,test_rules/}
│ ├── mcp_server/{__main__.py,server.py,runner.py}
│ ├── test_pipeline/{environment_setup.py,pipeline.py}
│ ├── tool_github/{issue_parser.py,github_reporter/}
│ ├── tools/
│ │ ├── configuration/
│ │ ├── test_catalog/
│ │ ├── result_normalizer/
│ │ ├── mkdocs_reporter/
│ │ ├── pandoc_reporter/
│ │ ├── local_llm/
│ │ └── test_result/
│ └── tests/
│ ├── pytest/{fixtures,test_cases,test_equipments,test_interfaces}/
│ └── unittest/{ct_framework,python,c_cpp,firmware,common}/
├── test_reports/
│ ├── results/{pytest/test_cases,unittest}/
│ ├── markdown/{pytest/test_cases,unittest}/
│ ├── pandocs/{pytest/test_cases,unittest}/
│ └── local_llm/
├── DESIGN.md
├── README.md
├── mkdocs.yml
├── pytest.ini
└── requirements.txt
| GitHub path |
Role |
.github/ISSUE_TEMPLATE/*.yml |
Collects Pytest, Unittest, and environment-check requests from GitHub Issues |
.github/workflows/continuous-test.yml |
Normalizes an Issue or manual request, selects a runner, executes the test pipeline, and uploads artifacts/comments |
.github/workflows/github_pages.yaml |
Builds and publishes the MkDocs site |
.github/pull_request_template.md |
Provides the pull-request review template |
| VS Code path |
Role |
.vscode/mcp.json |
Registers the local ai-driven-ci-ct stdio MCP server |
.vscode/settings.json |
Selects the project Python interpreter and editor/test settings |
.vscode/launch.json |
Provides setup, check, report, and debug launch profiles |
.vscode/tasks.json |
Provides setup, test, report, MkDocs, and Git tasks |
| Scope |
Path |
| Project configuration |
test_envs/configs/config_env.json, test_envs/configs/config_ollama.json |
| Environment checks |
test_envs/configs/check_envs.json, test_envs/configs/check_ollama.json |
| pytest |
test_programs/pytest |
| unittest |
test_programs/unittest |
| Results |
test_reports/results |
| Local LLM logs |
test_reports/local_llm |
| Markdown |
test_reports/markdown/pytest/test_cases, test_reports/markdown/unittest |
| Pandoc |
test_reports/pandocs/pytest/test_cases, test_reports/pandocs/unittest; DOCX reference: docx/reference/reference.docx |
TEST Identifier
| Identifier |
pytest |
unittest |
Format |
| TEST ID |
O |
X |
CT-<TARGET>-<NNN> |
| Execution ID |
O |
O |
YYYYMMDD_HHMMSS_ffffff |
Local LLM
The Local LLM is part of the pytest report branch only. A unittest result is converted directly to Markdown without an Ollama request, Local LLM log, analysis payload, or escalation decision.
| Runner |
Local LLM |
Processing rule |
| pytest CT |
O |
Analyze result, logs, warnings, and optional source diff before Markdown generation |
| unittest |
X |
Generate execution summary and function results directly from normalized data |
Local LLM roles
| Role |
Input |
Output / decision |
| Evidence collection |
Pytest result JSON, errors, warnings, important log lines |
Bounded evidence payload; no invented evidence |
| Prompt selection |
Non-empty CT marker test_prompt; otherwise configured ollama.default_prompt |
Effective analysis instruction |
| Result summary |
Status, duration, metrics, statistics, and logs |
Human-readable summary |
| Failure classification |
Result status and captured failure evidence |
classification and failure_analysis |
| Confidence estimation |
Available test and log evidence |
confidence from 0.0 to 1.0 |
| Warning analysis |
Captured warning lines |
Structured warnings with Critical, Important, or Low severity |
| Source review |
Optional local Git diff from --source-review |
source_review; otherwise Not requested |
| Recommendation |
Analysis findings |
Actionable recommendations |
| Escalation judgment |
LLM escalation flag, confidence, failure classification, and repeated failures |
needs_escalation plus Codex escalation reasons |
| Offline fallback |
Ollama request failure or invalid response |
Deterministic summary, classification, warnings, and recommendation |
Pytest analysis flow
Pytest Result JSON + Test Log
+
Optional Source Diff
↓
Prompt selection
├── @pytest.mark.ct(test_prompt="...")
└── ollama.default_prompt
↓
Local LLM / Deterministic Fallback
↓
Test analysis + Escalation decision
↓
Pytest Markdown + MkDocs
| Item |
Path / rule |
| Runtime |
Ollama |
| Configuration |
test_envs/configs/config_ollama.json |
| Analysis payload |
Pytest result JSON → test_analysis |
| Diagnostic log |
test_reports/local_llm/<execution-id>_local_llm.log |
| Unittest Local LLM log |
Not generated |
| Detailed setup |
local_llm_environment.md |