Pytest CT Framework

Scope

The pytest area runs continuous tests (CT) that combine a test case,
a fixture composition, DUT interfaces, and optional measurement or control equipment.
Every pytest CT execution resolves to one of two modes: mock or hil.

Architecture

flowchart TD
    A[pytest CT test case] --> B[ct marker]
    A --> C[fixture_NNN]
    B --> D[Fixture mode selection]
    C --> E[Test interface]
    C --> F[Test equipment]
    E --> G[DUT or loopback]
    F --> G
    A --> H[ct_result recorder]
    D --> H
    H --> I[Result JSON and test log]

Repository Structure

test_envs/tests/
├── pytest/                       # CT execution mode: mock or hil
│   ├── test_cases/
│   │   ├── test_fixture_001_uart_timing.py
│   │   ├── test_fixture_002_usb_loopback.py
│   │   └── test_fixture_003_network_loopback.py
│   ├── fixtures/
│   │   ├── fixture_001_uart_saleae.py
│   │   ├── fixture_002_usb_digilent.py
│   │   ├── fixture_003_network.py
│   │   ├── fixture_004_jtag_fpga.py
│   │   └── fixture_005_full_hil.py
│   ├── test_equipments/
│   │   ├── hil_base.py
│   │   ├── fpga/{mock,hil}/
│   │   ├── saleae/{mock,hil}/
│   │   └── digilent/{mock,hil}/
│   ├── test_interfaces/
│   │   ├── base.py
│   │   ├── hil_base.py
│   │   ├── usb/{mock,hil}/
│   │   ├── uart/{mock,hil}/
│   │   ├── jtag/{mock,hil}/
│   │   └── network/{mock,hil}/
│   └── conftest.py
└── unittest/
    ├── ct_framework/
    │   └── python/
    ├── python/
    ├── c_cpp/
    ├── firmware/
    └── common/

Identifiers and Test Cases

Current executable CT mapping

FIXTURE-004 and FIXTURE-005 are registered metadata/fixture scaffolds,
but there are no matching test_fixture_004_*.py or test_fixture_005_*.py CT test cases.

CT Marker Contract

@pytest.mark.ct(
    test_id="CT-UART-001",
    category="timing",
    fixture_id="FIXTURE-001",
    fixture_mode="mock",
    test_prompt="",
)

Collection performs the following validation for every CT item:

• Required marker fields exist.
• test_id is a non-empty string and is not duplicated.
• A file named test_fixture_<NNN>_*.py declares fixture_id="FIXTURE-<NNN>".
• The test function requests fixture_<NNN>.
• The fixture ID exists in the registry.
• fixture_mode is mock or hil, and that mode is enabled in FIXTURE_META.

An invalid contract raises a pytest collection/usage error before test execution.

Fixture Registry and Metadata

fixture_registry() imports every fixture_*.py module in the fixtures package and reads its FIXTURE_META mapping.
Registry validation therefore applies even to fixture scaffolds that do not yet have test cases.

FIXTURE_META = {
    "fixture_id": "FIXTURE-001",
    "interfaces": ["UART"],
    "equipments": ["Saleae"],
    "modes": {
        "mock": {"enabled": True},
        "hil": {"enabled": True},
    },
}

Registered fixture catalog

enabled: true declares that a mode may be selected. It does not prove that the fixture has been wired to physical hardware. The current repository has no end-to-end executable HIL CT test case.

Fixture-Based Test Case Composition

A CT Test Case does not create hardware or communication objects directly. It requests one numbered fixture, and that fixture composes the required interfaces and equipment for the selected test topology.

Test Case
   ├── @pytest.mark.ct
   │   ├── TEST ID
   │   ├── Fixture ID
   │   └── Default mock or hil mode
   ├── fixture_<NNN>
   │   ├── Interface: DUT communication and data transfer
   │   ├── Equipment: measurement, capture, or target control
   │   └── connect → yield → disconnect lifecycle
   └── ct_result
       ├── metrics: interface and protocol measurements
       └── statistics: equipment observations

Current Fixture and Test Case Behavior

The numbered contract connects the files and runtime objects: test_fixture_001_*.py must declare FIXTURE-001 and request fixture_001. Collection stops before execution if the number, marker metadata, fixture argument, or registry entry does not agree.

Equipment Extension

FPGA, Saleae, and Digilent are equipment adapters. Each equipment type owns separate mock/ and hil/ implementations so a fixture can keep the same Test Case contract while changing the runtime backend.

test_equipments/
├── fpga/{mock,hil}/
├── saleae/{mock,hil}/
├── digilent/{mock,hil}/
└── wireshark/{mock,hil}/            # Planned extension

Wireshark should be added as Network capture equipment, not as a replacement for NetworkInterface. The Network interface continues to send and receive DUT traffic; the Wireshark equipment adapter starts capture, stops capture, and converts packet observations into fixture statistics.

A future Wireshark CT extension can use this structure:

test_envs/tests/pytest/
├── test_equipments/wireshark/{mock,hil}/
├── fixtures/fixture_006_network_wireshark.py
└── test_cases/test_fixture_006_network_capture.py

The extension must add Wireshark to FIXTURE_META.equipments, declare both mode flags, keep capture cleanup in finally, and record capture statistics through ct_result.statistics. HIL availability must fail or skip explicitly and must never fall back silently to mock behavior.

Mock and HIL Mode Selection

pytest CT
├── mock    # Simulation or loopback implementation
└── hil     # Physical hardware implementation

--fixture-mode=marker is the CLI default, but marker is not an execution mode.
It means “use the mock or hil value declared by the CT marker.” effective_fixture_mode() resolves the final mode and verifies that it is enabled by the fixture metadata.

none is also not a pytest execution mode.
It is used only for result fields such as equipment_mode when a fixture has no equipment.

Mock behavior

HIL adapters and gate

HILTransportInterface delegates connection and I/O to injected handlers and rejects reads/writes while disconnected. HILEquipmentController similarly delegates equipment connect/disconnect operations. The UART, USB, JTAG, Network, Saleae, Digilent, and FPGA HIL classes currently inherit these generic adapters without device-specific logic.

A completed HIL fixture must:

• Enable modes.hil.enabled only when HIL selection is intended.
• Instantiate the HIL interface/equipment with real handlers.
• Connect every configured interface and equipment before yielding.
• Disconnect resources in a finally block.
• Fail or skip explicitly if hardware or configuration is unavailable.
• Never fall back silently to a mock implementation.

full_hil is an independent gate scaffold and skips unless CICT_HIL=1; it is not currently connected to a fixture_005 CT test case.

Interface Lifecycle

All DUT transports implement this contract:

connect()
disconnect()
read(size=-1)
write(data)
execute(command)

TestInterface also supports a context manager. Its default execute() encodes a command, writes it, and returns one read. Current composed fixtures connect tools before yield and disconnect them in reverse order in finally.

Result Recording

Tests receive ct_result: CTResultRecorder and add measurement data explicitly:

ct_result.metrics.update(measurement.metrics())
ct_result.statistics.update(measurement.statistics())

After the test call, the fixture creates a normalized ResultRecord.

Status is PASS, FAIL, or SKIP from the pytest call report. The environment is github_local_runner when GITHUB_ACTIONS=true, otherwise local; runner name comes from RUNNER_NAME or defaults to local.

The generated test log contains these sections:

[TEST]
[STDOUT]
[STDERR]
[EQUIPMENT]
[INTERFACE]

Statistics are written to EQUIPMENT; metrics are written to INTERFACE.

Commands

• pytest.ini

  [pytest]
  testpaths = test_envs/tests
  python_files = test_*.py
  addopts = -ra
  markers =
      ct: continuous/integration test
      hardware: test requiring real hardware
  

  
  

• Powershell
  Python Command Examples in venv

All configured tests under pytest.ini testpaths

.\.venv\Scripts\python.exe -m pytest

pytest CT area only

.\.venv\Scripts\python.exe -m pytest test_envs/tests/pytest -m ct -s

.\.venv\Scripts\python.exe -m pytest test_envs/tests/pytest/test_cases --fixture-mode=marker

Force Mock mode

.\.venv\Scripts\python.exe -m pytest test_envs/tests/pytest/test_cases --fixture-mode=mock

Select one TEST ID
VSCode -> Task
VSCode -> Testing

.\.venv\Scripts\python.exe -m pytest test_envs/tests/pytest/test_cases --test-id CT-UART-001 --fixture-mode=mock

For the current three CT test cases,
forcing --fixture-mode=hil fails with an explicit “HIL implementation is required” message.

VS Code

Report

Pytest and unittest connect to the same Markdown-first Report pipeline. Pytest adds Local LLM analysis and conditional escalation before the shared Markdown is generated.

Pytest result JSON + test log
              ↓
test_envs.tools.test_result
              ↓
Local LLM analysis + escalation decision
              ↓
test_envs.tools.mkdocs_reporter
              ↓
Canonical Markdown
       ┌──────┴──────┐
       ↓             ↓
MkDocs Results   Pandoc Report

Generate pending Markdown and publish MkDocs result pages:

.\.venv\Scripts\python.exe -m test_envs.tools.test_result --pending --docs

Convert the latest Markdown to HTML:

.\.venv\Scripts\python.exe -m test_envs.tools.pandoc_reporter --latest --format html

Convert the latest Markdown to DOCX:

.\.venv\Scripts\python.exe -m test_envs.tools.pandoc_reporter --latest --format docx

The same commands are available from VS Code through Run and Debug-Report and Tasks-Report. Pytest result pages are published under Pytest Results.