Custom Benchmark Suites

Creating a Benchmark Suite

Organise benchmarks in a dedicated directory:

benchmarks/
  __init__.py
  test_numeric.py
  test_string.py
  test_data_structures.py

Reproducible Benchmarks

import pyvorin_thin.benchmark_event as be

def benchmark_numeric_sum():
    event = be.build_benchmark_event(
        license_key="PYV-XXXX",
        workload_source=source,
        benchmark_name="numeric_sum",
        cpython_runtime_ms=cpython_time,
        pyvorin_runtime_ms=pyvorin_time,
    )
    be.write_benchmark_event_to_local_json(event)

Regression Detection

Store baseline benchmark results and fail CI if speedup drops below threshold:

with open('baseline.json') as f:
    baseline = json.load(f)

assert current_speedup >= baseline['speedup'] * 0.9, "Performance regression detected"

Visualising Trends

Upload benchmark JSON files to the dashboard or parse them into Grafana / Datadog for trend visualisation.