Feedback Testing Harness

scverse-backends includes a small optional runner for backend feedback tests. It is not a required test framework and it does not define what “correct” means for any scientific API.

The recommended pattern is:

• A host may expose reusable CPU-vs-backend checks for its own semantics.

• A backend may run those checks in its own CI for early compatibility feedback.

• scverse-backends only provides run_conformance, a tiny runner that resolves the backend, skips unimplemented functions, and reports results.

This keeps domain-specific correctness tests close to the host library, while backend maintainers get a simple way to run the same checks before opening or updating integrations.

Host-Side Checks

Hosts can publish a small testing module with functions that compare the host CPU reference against an active backend. The checks should be regular pytest-style assertions and can use whatever fixtures or tolerances make sense for that host.

# example_host/testing/backend_feedback.py
from __future__ import annotations

import numpy as np


def _make_data(n_obs=500, n_vars=100):
    rng = np.random.default_rng(42)
    return rng.random((n_obs, n_vars), dtype=np.float32)


def _test_compute_score(backend_name: str) -> None:
    import example_host as eh

    data = _make_data()

    cpu = eh.compute_score(data.copy(), method="fast", copy=True)
    with eh.settings.use_backend(backend_name):
        backend = eh.compute_score(data.copy(), method="fast", copy=True)

    np.testing.assert_allclose(cpu, backend, atol=1e-5, rtol=1e-3)


CONFORMANCE_TESTS = {
    "compute_score": _test_compute_score,
}

The host decides whether this module is public API, testing API, or experimental helper API. scverse-backends does not require a specific location.

Backend-Side Use

Backends can import the host’s checks and run them in backend CI:

# my_backend/tests/test_example_host_feedback.py
from __future__ import annotations

from scverse_backends.testing import run_conformance
from example_host._backends import get_backend
from example_host.testing.backend_feedback import CONFORMANCE_TESTS


def test_example_host_feedback():
    results = run_conformance(
        backend_name="my_backend",
        tests=CONFORMANCE_TESTS,
        get_backend=get_backend,
    )
    for function_name, status in results.items():
        assert status == "PASSED", f"{function_name}: {status}"

This is intended as fast feedback for backend maintainers. A host may still keep its normal backend-independent unit tests, and a backend may add deeper backend-specific tests around performance, memory behavior, or hardware-specific edge cases.

Runner Behavior

run_conformance does four small things:

1. Resolves backend_name using the host’s get_backend.

2. Iterates over {function_name: test_callable} entries.

3. Skips functions the backend does not implement.

4. Records "PASSED", "SKIPPED (...)", or "FAILED: ..." and, by default, re-raises the first failure so pytest shows the useful traceback.

The runner intentionally does not import the host, NumPy, CuPy, or any backend runtime. All scientific setup and assertions remain inside host- or backend-owned tests.