AI-driven Continuous Testing

System Overview

Go to VSCode Testing

Documents

TEST Flow

VS Code-based flow

Go To VS Code Testing Setup
Go To VS Code Testing

flowchart TD
    VSCODE[VS Code]
    EXTENSION[VS Extension Testing]
    TASK[VS Code Task]
    ENV[TEST System Environment]
    PYTEST[Pytest Operation]
    UNITTEST[Unittest Operation]
    PYTEST_RESULT[Pytest Result]
    UNITTEST_RESULT[Unittest Result]
    LLM[Local LLM]
    REPORTER["**src:test_envs/tools/mkdocs_reporter**<br/>MarkdownReporter"]
    MARKDOWN["Pytest / Unittest Markdown"]
    MKDOCS["MkDocs (TEST Results)"]
    PANDOC_REPORTER["**src:test_envs/tools/pandoc_reporter**<br/>convert"]
    PANDOC[Pandoc Report - HTML / DOCX]

    VSCODE --> EXTENSION
    VSCODE --> TASK
    TASK --> ENV
    ENV -. Runtime and configuration .-> EXTENSION

    EXTENSION --> PYTEST
    EXTENSION --> UNITTEST
    TASK --> PYTEST

    PYTEST --> PYTEST_RESULT --> LLM --> REPORTER
    UNITTEST --> UNITTEST_RESULT --> REPORTER

    TASK --> REPORTER
    REPORTER --> MARKDOWN
    MARKDOWN -. Manual --docs .-> MKDOCS
    MARKDOWN --> PANDOC_REPORTER --> PANDOC
    TASK --> MKDOCS
    TASK --> PANDOC_REPORTER

    classDef testResults fill:#fff3bf,stroke:#f08c00,stroke-width:4px,color:#5f3d00,font-weight:bold
    class MKDOCS testResults

GitHub Actions-based flow

flowchart TD
    PUSH["Push to main<br/>Issue Form or Workflow files"]
    PYTEST_ISSUE["pytest_request.yml<br/>Issue opened / edited / reopened"]
    UNITTEST_ISSUE["unittest_request.yml<br/>Issue opened / edited / reopened"]
    CHECK_ISSUE["test_check.yml<br/>Issue opened / edited / reopened"]
    MANUAL[workflow_dispatch]

    subgraph LABEL_JOB["labels job · ubuntu-latest"]
        LABELS["Create test-request-runner<br/>and test-check-runner labels"]
    end

    subgraph REQUEST_JOB["request job · ubuntu-latest"]
        CHECKOUT_REQUEST[Checkout]
        APPLY_LABEL[Ensure and apply matching Issue label]
        PARSER[test_envs.tool_github.issue_parser]
        OUTPUTS["Normalize request_kind, test settings,<br/>revision, reports, and runner_labels"]

        CHECKOUT_REQUEST --> APPLY_LABEL --> PARSER --> OUTPUTS
    end

    subgraph TEST_JOB["test job · runs-on: runner_labels"]
        CHECKOUT_TEST[Checkout requested revision]
        SETUP_PYTHON[Setup Python 3.12]
        REQUEST_KIND{request_kind}
        ENV_CHECK["TEST-CHECK<br/>Host / OS / Python / Ollama"]
        TEST_TYPE{test_type}
        PYTEST["Pytest CT<br/>TEST ID + marker / mock / hil"]
        UNITTEST["Unittest through pytest<br/>All or CT Framework"]
        PYTEST_RESULT[Pytest Result]
        UNITTEST_RESULT[Unittest Result]
        COVERAGE[Optional Coverage]
        PIPELINE[test_envs.test_pipeline.pipeline]
        LLM["Local LLM<br/>Pytest only"]
        REPORTER["test_envs/tools/mkdocs_reporter<br/>Canonical Markdown"]
        MKDOCS["MkDocs (TEST Results)<br/>when report_mkdocs is enabled"]
        PANDOC["test_envs.tools.pandoc_reporter<br/>optional HTML / DOCX"]
        GITHUB_REPORTER[test_envs.tool_github.github_reporter]
        ISSUE[GitHub Issue Comment]
        ARTIFACT["GitHub Artifact<br/>results / reports / coverage"]

        CHECKOUT_TEST --> SETUP_PYTHON --> REQUEST_KIND
        REQUEST_KIND -->|environment-check| ENV_CHECK --> GITHUB_REPORTER
        REQUEST_KIND -->|test| TEST_TYPE
        TEST_TYPE -->|Pytest| PYTEST --> PYTEST_RESULT
        TEST_TYPE -->|Unittest| UNITTEST --> UNITTEST_RESULT
        PYTEST_RESULT --> COVERAGE
        UNITTEST_RESULT --> COVERAGE
        PYTEST_RESULT --> PIPELINE
        UNITTEST_RESULT --> PIPELINE
        PIPELINE -->|Pytest analysis| LLM --> REPORTER
        PIPELINE -->|Unittest · no LLM| REPORTER
        REPORTER -. report_mkdocs .-> MKDOCS
        REPORTER -. report_html / report_docx .-> PANDOC
        PIPELINE --> GITHUB_REPORTER --> ISSUE
        PYTEST_RESULT --> ARTIFACT
        UNITTEST_RESULT --> ARTIFACT
        COVERAGE --> ARTIFACT
        REPORTER --> ARTIFACT
        MKDOCS --> ARTIFACT
        PANDOC --> ARTIFACT
    end

    PUSH --> LABELS
    PYTEST_ISSUE --> CHECKOUT_REQUEST
    UNITTEST_ISSUE --> CHECKOUT_REQUEST
    CHECK_ISSUE --> CHECKOUT_REQUEST
    MANUAL --> CHECKOUT_REQUEST
    OUTPUTS --> CHECKOUT_TEST

    classDef testResults fill:#fff3bf,stroke:#f08c00,stroke-width:4px,color:#5f3d00,font-weight:bold
    class MKDOCS testResults

GitHub Actions Flow

continuous-test.yml

Workflow roles

Test System Paths

.
├── .github/
│   ├── ISSUE_TEMPLATE/
│   │   ├── pytest_request.yml
│   │   ├── unittest_request.yml
│   │   └── test_check.yml
│   └── workflows/
│       ├── continuous-test.yml
│       └── github_pages.yaml
├── .vscode/
│   ├── settings.json
│   ├── launch.json
│   └── tasks.json
├── docs/
├── site/
└── test_envs/
    ├── configs/
    │   ├── config.json
    │   ├── check.json
    │   ├── pytest/
    │   └── unittest/
    ├── tests/
    ├── reports/
    └── tools/

Identifier

Local LLM

The Local LLM is part of the pytest report branch only. A unittest result is converted directly to Markdown without an Ollama request, Local LLM log, analysis payload, or escalation decision.

Local LLM roles

Pytest analysis flow

Pytest Result JSON + Test Log
             +
Optional Source Diff
             ↓
Prompt selection
├── @pytest.mark.ct(test_prompt="...")
└── ollama.default_prompt
             ↓
Local LLM / Deterministic Fallback
             ↓
Test analysis + Escalation decision
             ↓
Pytest Markdown + MkDocs