콘텐츠로 이동

MCP External Test Execution


test_envs.mcp_server exposes the repository test pipeline to an external MCP host through the local stdio transport.


Exposed Tools


test_envs\mcp_server

MCP tool Purpose
get_test_envs Checks OS, Python/venv, Ollama/model, Pandoc, and MCP permission flags
get_test_list_pytest Reads test_catalog.json and lists Pytest TEST IDs, source paths, fixture metadata, modes, and HIL availability
get_test_list_unittest Lists Unittest scopes and discovered test files
get_test_list_all Returns the combined Pytest and Unittest inventory
run_test_pytest Runs one allowlisted Pytest CT and returns its normalized result and report paths
run_test_unittest Runs all Unittest tests, CT Framework Python, or another allowlisted extension scope
run_test_all Runs run_test_pytest and run_test_unittest sequentially
get_ollama_test_plan Reads and validates the command-selected YAML rule registered in test_config.json without executing it
run_ollama_test_workflow Runs the command-selected YAML rule's allowlisted Pytest and Unittest selections, then generates Markdown and optional DOCX/HTML reports
update_latest_result Reads the latest normalized result and log and updates Markdown or Pandoc DOCX/HTML outputs
update_mkdocs Publishes the latest generated Markdown into docs/tests


Install


.\.venv\Scripts\python.exe -m pip install -r requirements.txt


VS Code MCP


The workspace configuration is stored in .vscode/mcp.json. Open the Command Palette, run MCP: List Servers, and start ai-driven-ci-ct.


{
  "servers": {
    "ai-driven-ci-ct": {
      "type": "stdio",
      "command": "${workspaceFolder}/.venv/Scripts/python.exe",
      "args": ["-m", "test_envs.mcp_server"],
      "cwd": "${workspaceFolder}",
      "env": {
        "TEST_NAME": "local_01",
        "PYTEST_HIL_ALLOW": "false",
        "MKDOC_UPDATE": "false",
        "OLLAMA_TEST_CONFIG": "${workspaceFolder}/test_envs/configs/test_config.json"
      }
    }
  }
}



GitHub access uses the GitHub MCP already installed and authenticated in VS Code. This workspace file registers only ai-driven-ci-ct, preventing duplicate GitHub MCP tools or conflicting server definitions.


Any MCP host that supports stdio can use the same Python command and repository working directory. Standard output is reserved for MCP protocol messages; test process output is captured and returned in the tool result.


Set a unique TEST_NAME such as local_01, local_02, or hil_lab_01 for each installed test server. Results created through this server record test_request as mcp.

OLLAMA_TEST_CONFIG selects test_config.json, which maps commands such as default, all, uart, usb, and network to YAML files under test_rules/. An AI Agent passes the command to get_ollama_test_plan or run_ollama_test_workflow; the MCP server rejects unregistered commands and any YAML target outside test_rules/.

VS Code exposes the same execution path as the TEST AI Agent: Ollama MCP Workflow task. Its picker supplies a registered command to python -m test_envs.mcp_server.ollama_workflow --command <command>.

The CLI and VS Code Task print progress for each selected Pytest TEST ID and Unittest scope, including completion status and report generation. The MCP tool response remains structured and returns the same final result paths.


Connection Test


Run the protocol contract tests without starting a child process:


.\.venv\Scripts\python.exe -m pytest -p no:cacheprovider test_programs/unittest/ct_framework/python/test_mcp_runner.py -k "not stdio" -vv


Run the optional integration test that launches the server as an external stdio process:


$env:AI_CT_RUN_STDIO_TEST = "true"
.\.venv\Scripts\python.exe -m pytest -p no:cacheprovider test_programs/unittest/ct_framework/python/test_mcp_runner.py -k stdio -vv


Result Flow


flowchart TD
    HOST[External MCP Host] -->|stdio| SERVER[test_envs.mcp_server]
    SERVER --> PYTEST[Pytest CT]
    SERVER --> UNITTEST[Unittest]
    PYTEST --> RESULT[test_reports/results]
    UNITTEST --> RESULT
    RESULT --> MARKDOWN[test_reports/markdown]
    MARKDOWN -. update_mkdocs .-> DOCS[docs/tests]
    MARKDOWN -. pandoc .-> PANDOC[test_reports/pandocs]
    SERVER --> RESPONSE[Structured MCP Result]


markdown generates the canonical Markdown report. update_mkdocs separately copies that generated Markdown into docs/tests. Pandoc DOCX uses docx/reference/reference.docx.

The MCP Pytest allowlist is loaded from test_programs/pytest/test_cases/test_catalog.json. Adding a Python CT case therefore requires no MCP source-code edit; run python -m test_envs.tools.test_catalog to register it.


Local MCP remains an independent external test-control path. GitHub Issue Forms do not select Local MCP; Issue automation belongs to GitHub-hosted or GitHub Self-hosted runners.


HIL Safety


Pytest HIL execution is disabled by default. Enable it only on a machine connected to the intended equipment:


$env:PYTEST_HIL_ALLOW = "true"
.\.venv\Scripts\python.exe -m test_envs.mcp_server


update_mkdocs is also disabled by default. Set MKDOC_UPDATE=true only when the MCP client is allowed to update docs/tests.


The MCP server does not accept arbitrary pytest paths, node IDs, or shell commands. It only accepts the configured TEST IDs and Unittest scopes. Test calls are serialized to prevent report collisions.


Network Boundary


The default server is local stdio, so it is available to external MCP applications running on the same machine without opening a TCP port. A remotely hosted Streamable HTTP endpoint requires authentication, TLS, and runner authorization and is intentionally not enabled by this configuration.