Example pipelines (verification)#

The runnable trees under examples/ are part of how we document the framework. Their layout and commands are listed in examples/manifest.yaml. Pytest checks that the manifest matches the tree on disk and runs selected pipelines as subprocess smoke tests.

Manifest#

Each top-level entry under pipelines describes one examples/<slug>/ directory that contains pipeline.py.

Field

Meaning

slug

Directory name under examples/.

ci_tier

always — run on every default pytest/CI invocation (dev mode in the shipped configs). cluster_optional — prod-oriented demos; tests live under tests/integration/examples_cluster/ and follow the same collection rules as Real cluster integration tests. manual — not part of default CI; run locally when you opt in (see below).

requirements

Tags for humans and for cluster gating logic (yt_cluster, s3, docker_image, heavy_python, etc.).

commands

Non-empty list of argv lists (after python). Usually [["pipeline.py"]]. See examples/08_multiple_configs/README.md for a tree that lists extra configs so each entry point is exercised.

If someone adds examples/<new>/pipeline.py but forgets the manifest, pytest fails with a pointer to examples/manifest.yaml.

Default CI (no YT cell)#

From the repo root, with the usual dev install:

conda run -n yt-framework -- pytest tests/integration/example_pipelines/test_smoke.py -m examples --tb=short

The same tests are collected when you run the full suite with pytest -m "not yt_cluster" (GitHub Actions uses that filter). They execute python pipeline.py (and any extra argv from the manifest) under each example directory. Timeouts are per slug so map-heavy demos get more wall clock.

Tests prepend the repository root to PYTHONPATH for each subprocess so import yt_framework works even when the active interpreter is a minimal tool venv (for example the pre-commit hook environment). They also prepend the active interpreter’s bin directory to PATH so dev-mode job shells resolve python3 to an environment that has omegaconf and the rest of the dev extra installed.

Manual opt-in (video_gpu)#

The GPU sample pulls in torch and related packages that we do not install on the default CI image. To run it locally:

export YT_FRAMEWORK_EXAMPLE_VIDEO_GPU=1
conda run -n yt-framework -- pytest tests/integration/example_pipelines/test_smoke.py::test_example_pipeline_manual_when_enabled -q

Without the env var, the test skips immediately.

Real cluster (optional)#

Prod-only examples (06_s3_integration, 07_custom_docker) are covered in Example pipelines on a real cell.

See also#