콘텐츠로 이동

AI-driven Continuous Testing

System Overview


Area Role
VS Code Common entry point for Testing, Tasks, debugging, and environment setup
GitHub Actions CI entry point for hosted Mock/unit tests and optional self-hosted hardware tests
pytest CT Runs TEST ID scenarios using a final mock or hil fixture mode
unittest Runs function-level framework and extension tests without TEST ID or fixture mode
Result pipeline Normalizes execution data and generates Markdown reports
MCP server Allows an external local MCP host to discover and run allowlisted Pytest/Unittest operations through stdio
Local LLM Analyzes pytest CT results only; it is not used for unittest
MkDocs Publishes pytest and unittest Markdown and execution history


Go to VSCode Testing

AI Agent workflow based on MCP

Connect Codex, Ollama, and GitHub operations through MCP
Support Local MCP execution, GitHub Issue based TEST automation, and documentation-first project flow
https://jeonghunlee.github.io/local-ai-agent-mcp/index.html


MCP Server Role


test_envs.mcp_server is the local test-control boundary for an MCP-capable host such as Codex or VS Code. It runs through stdio; it does not open a network port or accept arbitrary shell commands.

Responsibility MCP server behavior
Environment discovery Returns OS, Python/venv, Ollama model, Pandoc, and execution permission status before a test starts
Allowlisted execution Lists and runs only cataloged Pytest TEST IDs and defined Unittest scopes; arbitrary paths and node IDs are rejected
Test orchestration Serializes requests, runs the selected test process, then returns normalized status and artifact paths as structured MCP output
Report operations Generates or refreshes Markdown/Pandoc reports; MkDocs publication is an explicit opt-in operation
Safety boundary HIL requires PYTEST_HIL_ALLOW=true; documentation publication requires MKDOC_UPDATE=true; both are disabled by default
CI separation Local MCP is a direct, local control path. GitHub Issue Forms continue to route only to GitHub-hosted or self-hosted runners
flowchart TD
    HOST[Codex / VS Code] -->|stdio MCP| SERVER[test_envs.mcp_server]
    SERVER -->|allowlisted request| TEST[Pytest CT or Unittest]
    TEST --> RESULT[Normalized result and reports]
    RESULT -->|structured response| HOST

For the tool list, configuration, and transport details, see MCP Environment.


Documents


Scope Document Components
Python Environment python_environment.md OS, Python, venv, dependencies
Local LLM Environment local_llm_environment.md Ollama, model, prompt, check
VS Code Environment vscode_environment.md Settings, Launch, Tasks, Testing
MCP Environment mcp_environment.md External stdio tools, execution controls, HIL safety
Pytest Operation pytest_operation.md TEST ID, Fixture, Mock/HIL, Local LLM
Unittest Operation unittest_operation.md Function, Execution ID, Result, Markdown
Pytest / HIL / Mock pytest_framework.md Test cases, fixture mapping, mode selection, HIL gate
Unittest unittest_framework.md Function result contract
Pytest Results tests/pytest/index.md TEST ID, Execution history
Unittest Results tests/unittest/index.md Function count, Execution history


TEST Flow


The project supports three test-entry forms. Each form runs the same allowlisted test pipeline and creates execution artifacts: normalized result JSON and log files, canonical Markdown, published test-history Markdown, and optional Pandoc HTML/DOCX files. Test source files are selected from the catalog or scope; execution creates result and report files rather than arbitrary new test source code.

Test-entry form Request and execution Generated documentation and files
VS Code Testing explorer, Run and Debug, or Tasks runs local Pytest CT and Unittest Result JSON/log → Markdown → optional HTML/DOCX; MkDocs history is updated when requested
AI Agent MCP Codex calls get_test_envs, selects an allowlisted TEST ID or scope, then calls an MCP test tool Structured MCP result with result/log/Markdown paths; optional HTML/DOCX conversion uses the same report pipeline
GitHub Issue and Actions A GitHub Issue Form or workflow_dispatch request is normalized, labeled, and run by a GitHub-hosted or self-hosted runner Workflow artifacts and canonical Markdown; selected HTML/DOCX reports are generated by the same Pandoc conversion path
flowchart TD
    VSCODE[VS Code]
    AGENT[Codex AI Agent / MCP]
    ISSUE[GitHub Issue Form or workflow_dispatch]
    TEST[Allowlisted Pytest CT or Unittest]
    RESULT[result JSON and log]
    MARKDOWN[Canonical Markdown and test history]
    PANDOC[Optional HTML / DOCX]

    VSCODE --> TEST
    AGENT --> TEST
    ISSUE --> TEST
    TEST --> RESULT --> MARKDOWN --> PANDOC


VS Code-based flow


Go To VS Code Testing Setup
Go To VS Code Testing


flowchart TD
    VSCODE[VS Code]
    EXTENSION[VS Extension Testing]
    TASK[VS Code Task]
    ENV[TEST System Environment]
    PYTEST[Pytest Operation]
    UNITTEST[Unittest Operation]
    PYTEST_RESULT[Pytest Result]
    UNITTEST_RESULT[Unittest Result]
    LLM[Local LLM]
    REPORTER["**src:test_envs/tools/mkdocs_reporter**<br/>MarkdownReporter"]
    MARKDOWN["Pytest / Unittest Markdown"]
    MKDOCS["MkDocs (TEST Results)"]
    PANDOC_REPORTER["**src:test_envs/tools/pandoc_reporter**<br/>convert"]
    PANDOC[Pandoc Report - HTML / DOCX]

    VSCODE --> EXTENSION
    VSCODE --> TASK
    TASK --> ENV
    ENV -. Runtime and configuration .-> EXTENSION

    EXTENSION --> PYTEST
    EXTENSION --> UNITTEST
    TASK --> PYTEST

    PYTEST --> PYTEST_RESULT --> LLM --> REPORTER
    UNITTEST --> UNITTEST_RESULT --> REPORTER

    TASK --> REPORTER
    REPORTER --> MARKDOWN
    MARKDOWN -. Manual --docs .-> MKDOCS
    MARKDOWN --> PANDOC_REPORTER --> PANDOC
    TASK --> MKDOCS
    TASK --> PANDOC_REPORTER

    classDef testResults fill:#fff3bf,stroke:#f08c00,stroke-width:4px,color:#5f3d00,font-weight:bold
    class MKDOCS testResults


VS Code entry Execution scope
VS Extension Testing Discovers and runs both pytest CT and unittest through the pytest adapter
VS Code Task Prepares the environment, runs pytest CT, generates MkDocs reports, and converts the latest Markdown to Pandoc HTML/DOCX
Common Markdown reporter test_envs/tools/mkdocs_reporter renders both pytest and unittest results through MarkdownReporter
Pandoc reporter test_envs/tools/pandoc_reporter converts the latest common Markdown report to HTML or DOCX

Automatically Generate Report

Analyze Result.log and result.json using an LLM, then automatically update the analysis report in the MkDocs documentation.


AI Agent-based MCP flow


Codex or an Ollama-powered AI Agent can use the local ai-driven-ci-ct MCP server as a direct test-control interface. The agent first inspects the environment and allowlisted inventory, selects an approved test or scope, then receives the normalized result and generated report paths as structured tool output. Ollama supplies the local model reasoning; the MCP server performs the allowlisted test execution.

flowchart TD
    AGENT[Codex or Ollama AI Agent]
    ENV[get_test_envs]
    INVENTORY[get_test_list_pytest / get_test_list_unittest]
    SELECT[Select allowlisted TEST ID or Unittest scope]
    MCP[test_envs.mcp_server]
    PYTEST[run_test_pytest]
    UNITTEST[run_test_unittest]
    REPORT[Markdown / Pandoc reports]
    RESPONSE[Structured MCP result]

    AGENT --> ENV --> INVENTORY --> SELECT --> MCP
    MCP --> PYTEST --> REPORT --> RESPONSE --> AGENT
    MCP --> UNITTEST --> REPORT

    classDef safety fill:#fff3bf,stroke:#f08c00,stroke-width:2px,color:#5f3d00
    class SELECT safety

test_envs/configs/test_config.json maps an AI Agent command to an allowlisted YAML rule under test_envs/configs/test_rules/. For example, default, uart, usb, network, and all select their registered rule; selected runs listed Pytest TEST IDs and Unittest scopes, while all runs the corresponding catalog or declared scope. Run get_ollama_test_plan(command="usb") to inspect a selection, then run_ollama_test_workflow(command="usb") to execute it.

flowchart TD
    CONFIG[test_config.json command map] --> PLAN[test_rules/*.yaml]
    PLAN --> SELECT{1. Select test type and mode}
    SELECT --> PYTEST[Pytest: all or selected TEST IDs]
    SELECT --> UNITTEST[Unittest: all or selected scopes]
    PYTEST --> LOG[2. Test log and normalized result]
    UNITTEST --> LOG
    LOG --> MARKDOWN[Markdown report]
    MARKDOWN --> FORMAT{3. Pandoc output}
    FORMAT --> DOCX[DOCX]
    FORMAT --> HTML[HTML]
    PYTEST --> OLLAMA[Ollama result analysis]
    OLLAMA --> MARKDOWN
AI Agent capability MCP boundary
Inspect readiness Reads OS, venv, Ollama, Pandoc, and permission status before execution
Automate with Ollama An Ollama-powered agent can inspect the catalog, choose an approved test, call the MCP test tool, and review the structured result
Choose a test Uses only cataloged Pytest TEST IDs and declared Unittest scopes
Run and review Starts an allowlisted operation and receives status, logs, and report paths without parsing terminal output
Use Local LLM analysis Pytest report generation can invoke the configured local Ollama model; unittest does not invoke Local LLM analysis
Keep safety controls HIL remains blocked until PYTEST_HIL_ALLOW=true; MkDocs publication remains blocked until MKDOC_UPDATE=true

This is a local MCP path, separate from GitHub Issue and GitHub Actions execution. Ollama does not receive unrestricted shell access: test execution remains limited to the MCP server's cataloged Pytest TEST IDs and declared Unittest scopes.


GitHub Actions-based flow


flowchart TD
    PUSH["Push to main<br/>Issue Form or Workflow files"]
    PYTEST_ISSUE["pytest_request.yml<br/>Issue opened / edited / reopened"]
    UNITTEST_ISSUE["unittest_request.yml<br/>Issue opened / edited / reopened"]
    CHECK_ISSUE["test_check.yml<br/>Issue opened / edited / reopened"]
    MANUAL[workflow_dispatch]

    subgraph LABEL_JOB["labels job · ubuntu-latest"]
        LABELS["Create test-request-runner<br/>and test-check-runner labels"]
    end

    subgraph REQUEST_JOB["request job · ubuntu-latest"]
        CHECKOUT_REQUEST[Checkout]
        APPLY_LABEL[Ensure and apply matching Issue label]
        PARSER[test_envs.tool_github.issue_parser]
        OUTPUTS["Normalize request_kind, test settings,<br/>revision, reports, and runner_labels"]

        CHECKOUT_REQUEST --> APPLY_LABEL --> PARSER --> OUTPUTS
    end

    subgraph TEST_JOB["test job · runs-on: runner_labels"]
        CHECKOUT_TEST[Checkout requested revision]
        SETUP_PYTHON[Setup Python 3.12]
        REQUEST_KIND{request_kind}
        ENV_CHECK["TEST-CHECK<br/>Host / OS / Python / Ollama"]
        TEST_TYPE{test_type}
        PYTEST["Pytest CT<br/>TEST ID + marker / mock / hil"]
        UNITTEST["Unittest through pytest<br/>All or CT Framework"]
        PYTEST_RESULT[Pytest Result]
        UNITTEST_RESULT[Unittest Result]
        COVERAGE[Optional Coverage]
        PIPELINE[test_envs.test_pipeline.pipeline]
        LLM["Local LLM<br/>Pytest only"]
        REPORTER["test_envs/tools/mkdocs_reporter<br/>Canonical Markdown"]
        MKDOCS["MkDocs (TEST Results)<br/>when report_mkdocs is enabled"]
        PANDOC["test_envs.tools.pandoc_reporter<br/>optional HTML / DOCX"]
        GITHUB_REPORTER[test_envs.tool_github.github_reporter]
        ISSUE[GitHub Issue Comment]
        ARTIFACT["GitHub Artifact<br/>results / reports / coverage"]

        CHECKOUT_TEST --> SETUP_PYTHON --> REQUEST_KIND
        REQUEST_KIND -->|environment-check| ENV_CHECK --> GITHUB_REPORTER
        REQUEST_KIND -->|test| TEST_TYPE
        TEST_TYPE -->|Pytest| PYTEST --> PYTEST_RESULT
        TEST_TYPE -->|Unittest| UNITTEST --> UNITTEST_RESULT
        PYTEST_RESULT --> COVERAGE
        UNITTEST_RESULT --> COVERAGE
        PYTEST_RESULT --> PIPELINE
        UNITTEST_RESULT --> PIPELINE
        PIPELINE -->|Pytest analysis| LLM --> REPORTER
        PIPELINE -->|Unittest · no LLM| REPORTER
        REPORTER -. report_mkdocs .-> MKDOCS
        REPORTER -. report_html / report_docx .-> PANDOC
        PIPELINE --> GITHUB_REPORTER --> ISSUE
        PYTEST_RESULT --> ARTIFACT
        UNITTEST_RESULT --> ARTIFACT
        COVERAGE --> ARTIFACT
        REPORTER --> ARTIFACT
        MKDOCS --> ARTIFACT
        PANDOC --> ARTIFACT
    end

    PUSH --> LABELS
    PYTEST_ISSUE --> CHECKOUT_REQUEST
    UNITTEST_ISSUE --> CHECKOUT_REQUEST
    CHECK_ISSUE --> CHECKOUT_REQUEST
    MANUAL --> CHECKOUT_REQUEST
    OUTPUTS -->|GitHub-hosted or Self-hosted Runner| CHECKOUT_TEST

    classDef testResults fill:#fff3bf,stroke:#f08c00,stroke-width:4px,color:#5f3d00,font-weight:bold
    class MKDOCS testResults


GitHub Actions entry Execution scope
pytest_request.yml Pytest-only form: TEST ID, Fixture mode, Test Environment, OS, revision, Coverage, and optional Pandoc
unittest_request.yml Unittest-only form: scope, Test Environment, OS, revision, Coverage, and optional Pandoc; no TEST ID or Fixture mode
test_check.yml Selects Test Environment and OS; the workflow detects Python and Ollama state and comments on the Issue
continuous-test.yml Routes GitHub Issue requests to GitHub-hosted or GitHub Self-hosted runners
Pytest TEST ID catalog test_catalog.json is generated from Python CT markers; Setup 4 synchronizes the VS Code, Issue Form, and workflow-dispatch choices
Common Markdown reporter test_envs/tools/mkdocs_reporter always renders the canonical Markdown; MkDocs publication is enabled separately and the generated reports are uploaded as artifacts


Github Issues and Automatically Generate Report



continuous-test.yml


Workflow Primary role Local LLM rule
continuous-test.yml Unified hosted, Windows, and self-hosted HIL test-request automation Pytest uses Local LLM with deterministic fallback; Unittest bypasses Local LLM
Item Current behavior
Workflow name Test Request
Trigger Request Issue opened, edited, or reopened; or manual workflow_dispatch
Duplicate prevention The workflow does not subscribe to labeled; automatic label attachment therefore does not create a second run
Request Job Detects Pytest, Unittest, or TEST-CHECK from the form title and emits normalized execution settings
Issue labels A relevant default-branch push creates both labels; the request job also creates and applies the matching label as a first-Issue fallback
Runner routing GitHub Issue requests select GitHub-hosted or Self-hosted Runner plus Ubuntu/Windows; Local MCP is not an Issue option
Timeout 60 minutes
Permissions Repository contents read; issues write
Python actions/setup-python@v5, Python 3.12, pip cache
Environment Test requests create .venv and install requirements.txt; TEST-CHECK inspects the selected runner directly
Unittest Runs All Unittest or CT Framework Python through pytest without Local LLM analysis
Pytest CT GitHub-hosted Ubuntu/Windows supports Mock only; physical HIL requires the selected GitHub Self-hosted hardware runner
Coverage Optional terminal or HTML pytest-cov report
Report Issue Forms normalize Log, canonical Markdown, Pandoc DOCX, and Pandoc HTML separately
MkDocs publication report_mkdocs copies the generated Markdown into docs/tests; Markdown generation itself is represented by report_markdown
Issue output github_reporter posts the existing canonical Test Report Markdown, including its Result JSON-based test_envs table
Artifact Uploads results, MkDocs pages, .coverage, and htmlcov/
Node.js No project Node.js setup or command; official GitHub Actions manage their own embedded runtime


Test System


.
├── .github/
│   ├── ISSUE_TEMPLATE/
│   │   ├── pytest_request.yml
│   │   ├── unittest_request.yml
│   │   └── test_check.yml
│   ├── workflows/
│   │   ├── continuous-test.yml
│   │   └── github_pages.yaml
│   └── pull_request_template.md
├── .vscode/
│   ├── launch.json
│   ├── mcp.json
│   ├── settings.json
│   └── tasks.json
├── docs/
│   ├── index.md
│   ├── mcp_environment.md
│   ├── vscode_environment.md
│   ├── python_environment.md
│   ├── local_llm_environment.md
│   ├── pytest_{framework,operation}.md
│   ├── unittest_{framework,operation}.md
│   └── tests/{pytest,unittest}/
├── docx/
│   ├── imgs/
│   ├── lua/
│   ├── reference/reference.docx
│   └── setup_reference.json
├── test_envs/
│   ├── configs/{config_env.json,config_ollama.json,check_envs.json,check_ollama.json,test_config.json,test_rules/}
│   ├── mcp_server/{__main__.py,server.py,runner.py}
│   ├── test_pipeline/{environment_setup.py,pipeline.py}
│   ├── tool_github/{issue_parser.py,github_reporter/}
│   ├── tools/
│   │   ├── configuration/
│   │   ├── test_catalog/
│   │   ├── result_normalizer/
│   │   ├── mkdocs_reporter/
│   │   ├── pandoc_reporter/
│   │   ├── local_llm/
│   │   └── test_result/
│   └── tests/
│       ├── pytest/{fixtures,test_cases,test_equipments,test_interfaces}/
│       └── unittest/{ct_framework,python,c_cpp,firmware,common}/
├── test_reports/
│   ├── results/{pytest/test_cases,unittest}/
│   ├── markdown/{pytest/test_cases,unittest}/
│   ├── pandocs/{pytest/test_cases,unittest}/
│   └── local_llm/
├── DESIGN.md
├── README.md
├── mkdocs.yml
├── pytest.ini
└── requirements.txt

GitHub path Role
.github/ISSUE_TEMPLATE/*.yml Collects Pytest, Unittest, and environment-check requests from GitHub Issues
.github/workflows/continuous-test.yml Normalizes an Issue or manual request, selects a runner, executes the test pipeline, and uploads artifacts/comments
.github/workflows/github_pages.yaml Builds and publishes the MkDocs site
.github/pull_request_template.md Provides the pull-request review template


VS Code path Role
.vscode/mcp.json Registers the local ai-driven-ci-ct stdio MCP server
.vscode/settings.json Selects the project Python interpreter and editor/test settings
.vscode/launch.json Provides setup, check, report, and debug launch profiles
.vscode/tasks.json Provides setup, test, report, MkDocs, and Git tasks


Scope Path
Project configuration test_envs/configs/config_env.json, test_envs/configs/config_ollama.json
Environment checks test_envs/configs/check_envs.json, test_envs/configs/check_ollama.json
pytest test_programs/pytest
unittest test_programs/unittest
Results test_reports/results
Local LLM logs test_reports/local_llm
Markdown test_reports/markdown/pytest/test_cases, test_reports/markdown/unittest
Pandoc test_reports/pandocs/pytest/test_cases, test_reports/pandocs/unittest; DOCX reference: docx/reference/reference.docx


TEST Identifier


  • TEST
    • Pytest
    • Unittest
Identifier pytest unittest Format
TEST ID O X CT-<TARGET>-<NNN>
Execution ID O O YYYYMMDD_HHMMSS_ffffff


Local LLM


The Local LLM is part of the pytest report branch only. A unittest result is converted directly to Markdown without an Ollama request, Local LLM log, analysis payload, or escalation decision.


Runner Local LLM Processing rule
pytest CT O Analyze result, logs, warnings, and optional source diff before Markdown generation
unittest X Generate execution summary and function results directly from normalized data


Local LLM roles


Role Input Output / decision
Evidence collection Pytest result JSON, errors, warnings, important log lines Bounded evidence payload; no invented evidence
Prompt selection Non-empty CT marker test_prompt; otherwise configured ollama.default_prompt Effective analysis instruction
Result summary Status, duration, metrics, statistics, and logs Human-readable summary
Failure classification Result status and captured failure evidence classification and failure_analysis
Confidence estimation Available test and log evidence confidence from 0.0 to 1.0
Warning analysis Captured warning lines Structured warnings with Critical, Important, or Low severity
Source review Optional local Git diff from --source-review source_review; otherwise Not requested
Recommendation Analysis findings Actionable recommendations
Escalation judgment LLM escalation flag, confidence, failure classification, and repeated failures needs_escalation plus Codex escalation reasons
Offline fallback Ollama request failure or invalid response Deterministic summary, classification, warnings, and recommendation


Pytest analysis flow


Pytest Result JSON + Test Log
             +
Optional Source Diff
             ↓
Prompt selection
├── @pytest.mark.ct(test_prompt="...")
└── ollama.default_prompt
             ↓
Local LLM / Deterministic Fallback
             ↓
Test analysis + Escalation decision
             ↓
Pytest Markdown + MkDocs


Item Path / rule
Runtime Ollama
Configuration test_envs/configs/config_ollama.json
Analysis payload Pytest result JSON → test_analysis
Diagnostic log test_reports/local_llm/<execution-id>_local_llm.log
Unittest Local LLM log Not generated
Detailed setup local_llm_environment.md