fhelium.benchmarks

Public benchmark definitions, registry, and custom-workload interfaces.

BenchmarkCheck

class View source

BenchmarkCheck(name: str, passed: bool, oracle: str, metric: str, observed: Any, comparison: str, limit: Any | None, unit: str | None = None, details: Mapping[str, Any] = field(default_factory=dict))

One recorded correctness observation and acceptance rule.

Attributes

Name	Type	Default/value
name	str
passed	bool
oracle	str
metric	str
observed	Any
comparison	str
limit	Any | None
unit	str | None	None
details	Mapping[str, Any]	field(default_factory=dict)

to_dict

method

def to_dict() -> dict[str, Any]: ...

from_dict

method

def from_dict(payload: Mapping[str, Any]) -> BenchmarkCheck: ...

BenchmarkDefinition

class View source

BenchmarkDefinition(name: str, title: str, description: str, profiles: tuple[BenchmarkProfile, ...], runner: BenchmarkRunner, workload_id: str, category: str = 'workload', requirements: Mapping[str, Any] = field(default_factory=dict))

One discoverable benchmark workload and its execution profiles.

Attributes

Name	Type	Default/value
name	str
title	str
description	str
profiles	tuple[BenchmarkProfile, ...]
runner	BenchmarkRunner
workload_id	str
category	str	'workload'
requirements	Mapping[str, Any]	field(default_factory=dict)

profile

method

def profile(name: str | None=None) -> BenchmarkProfile: ...

BenchmarkMetric

class View source

BenchmarkMetric(name: str, value: int | float, unit: str, statistic: str, direction: MetricDirection, dimensions: Mapping[str, Any] = field(default_factory=dict), samples: tuple[int | float, ...] = ())

One normalized finite benchmark measurement.

Attributes

Name	Type	Default/value
name	str
value	int | float
unit	str
statistic	str
direction	MetricDirection
dimensions	Mapping[str, Any]	field(default_factory=dict)
samples	tuple[int | float, ...]	()

to_dict

method

def to_dict() -> dict[str, Any]: ...

from_dict

method

def from_dict(payload: Mapping[str, Any]) -> BenchmarkMetric: ...

BenchmarkProfile

class View source

BenchmarkProfile(name: str, description: str, parameters: Mapping[str, Any] = field(default_factory=dict))

Named parameter set exposed by the CLI and TUI.

parameters contains JSON-compatible defaults understood by the corresponding benchmark runner. with_overrides returns a new profile and does not mutate the registered definition.

Attributes

Name	Type	Default/value
name	str
description	str
parameters	Mapping[str, Any]	field(default_factory=dict)

with_overrides

method

def with_overrides(overrides: Mapping[str, Any]) -> BenchmarkProfile: ...

BenchmarkRegistry

class View source

BenchmarkRegistry()

Ordered process-local collection of named benchmark definitions.

register

method

def register(definition: BenchmarkDefinition, *, replace: bool=False) -> BenchmarkDefinition: ...

Register and return definition under its unique public name.

Parameters

• definition: Complete benchmark definition to retain by reference.
• replace: Replace an existing definition with the same name.

Raises

• KeyError: If the name already exists and replace is false.

get

method

def get(name: str) -> BenchmarkDefinition: ...

Return the definition named name or raise KeyError.

values

method

def values() -> tuple[BenchmarkDefinition, ...]: ...

Return definitions in registration order.

BenchmarkResult

class View source

BenchmarkResult(benchmark: str, profile: str, workload_id: str, effective_parameters: Mapping[str, Any], timed_boundary: BenchmarkTimedBoundary, metrics: Sequence[BenchmarkMetric], correctness: Sequence[BenchmarkCheck], rows: list[dict[str, Any]] = field(default_factory=list), scalars: dict[str, Any] = field(default_factory=dict), metadata: dict[str, Any] = field(default_factory=dict), notes: list[str] = field(default_factory=list), evidence: list[dict[str, Any]] = field(default_factory=list))

Common result header plus workload-specific benchmark evidence.

Normalized metrics and correctness checks are the stable comparison and publication surface. rows, scalars, metadata, and evidence remain extensible for workload-specific diagnostics.

Attributes

Name	Type	Default/value
benchmark	str
profile	str
workload_id	str
effective_parameters	Mapping[str, Any]
timed_boundary	BenchmarkTimedBoundary
metrics	Sequence[BenchmarkMetric]
correctness	Sequence[BenchmarkCheck]
rows	list[dict[str, Any]]	field(default_factory=list)
scalars	dict[str, Any]	field(default_factory=dict)
metadata	dict[str, Any]	field(default_factory=dict)
notes	list[str]	field(default_factory=list)
evidence	list[dict[str, Any]]	field(default_factory=list)

correctness_passed

property

correctness_passed: bool

Whether at least one check exists and all checks passed.

to_dict

method

def to_dict() -> dict[str, Any]: ...

from_dict

method

def from_dict(payload: Mapping[str, Any]) -> BenchmarkResult: ...

BenchmarkTimedBoundary

class View source

BenchmarkTimedBoundary(id: str, description: str, includes: tuple[str, ...], excludes: tuple[str, ...], synchronization: str)

Stable definition of what one benchmark measurement includes.

Attributes

Name	Type	Default/value
id	str
description	str
includes	tuple[str, ...]
excludes	tuple[str, ...]
synchronization	str

to_dict

method

def to_dict() -> dict[str, Any]: ...

from_dict

method

def from_dict(payload: Mapping[str, Any]) -> BenchmarkTimedBoundary: ...

load_builtin_benchmarks

function View source

def load_builtin_benchmarks() -> BenchmarkRegistry: ...

Load built-in definitions once and return the global registry.

register_benchmark

function View source

def register_benchmark(definition: BenchmarkDefinition | None=None, *, replace: bool=False) -> BenchmarkDefinition | Callable[[BenchmarkDefinition], BenchmarkDefinition]: ...

Register a definition on the public process-global registry.

The function accepts a definition directly or can be used as a decorator. Custom benchmark-file hooks should prefer their supplied registry instead of relying on this process-global object.

registry

data View source

registry = BenchmarkRegistry()