Real cluster integration tests#

These tests call a live YTsaurus cell through YTProdClient and yt.wrapper. They live under tests/integration/yt_cluster/ and use the pytest marker yt_cluster.

When pytest collects them#

pytest loads that directory only if both YT_PROXY and YT_TOKEN are set after this lookup:

  1. YT_FRAMEWORK_CLUSTER_TEST_ENV — path to a KEY=value file (same format as configs/secrets.env).

  2. Otherwise, if yt-cluster-test.env exists at the repository root, that file is read.

  3. Values from the process environment override the file for the keys listed in the example file.

If the pair is missing, the whole package is skipped at collection time. On CI hosts, CI=true also forces that skip. GitHub Actions additionally runs pytest -m "not yt_cluster", so those tests never run in CI even if credentials were present by mistake.

Setup#

Copy the template and fill in real values (the real file is gitignored):

cp yt-cluster-test.example.env yt-cluster-test.env

Never commit yt-cluster-test.env. You can instead export YT_PROXY and YT_TOKEN in the shell or in CI secrets.

Run#

From the repo root, using the project Conda env:

conda run -n yt-framework -- pytest -m yt_cluster -xvs

Narrower:

conda run -n yt-framework -- pytest tests/integration/yt_cluster/test_yql.py -xvs

What the tests assume#

  • Cypress namespace: //tmp/yt-framework/testing/<session_id>/… for tables and files. The session fixture creates that node and deletes it recursively after the run.

  • Host scratch: /tmp/yt-framework/testing/<session_id>/… for local files used in upload_file / upload_directory. It is removed when the session ends.

  • Docker: map, map-reduce, reduce, and vanilla jobs do not pass docker_image in OperationResources, so the cell’s default job image is used. The scripts call python3 and trivial shell (true). Secure-env command wrapping also relies on python3 and bash. If a job fails with “command not found”, your default image may differ; adjust the command in the test to match the image, not the framework.

  • YQL helpers on the client call run_yql with pool name default inside the library. Map/sort operations use YT_TEST_POOL (default default) when the test passes op_resources.pool.

  • Optional knobs in the env file: YT_TEST_POOL, YT_TEST_POOL_TREE, YT_TEST_MEMORY_GB, YT_TEST_CPU_LIMIT, YT_TEST_JOB_COUNT.

  • Map extras (test_map_operations.py): mapper env propagation, a second run_map with append=True, the job= alias (no command), two Cypress file dependencies, forwarded options such as title, max_row_weight, and max_failed_jobs, and a check that sensitive env keys are not present in the operation’s plain mapper.environment while jobs still read them at runtime (via secure_vault + promotion).

See also#