FHElium

Appearance

fhelium.benchmarks.v1 ​

FHElium Benchmark v1 model, report I/O, and fixed-run entrypoint.

BENCHMARK_VERSION ​

constant View source

python
BENCHMARK_VERSION = 'v1'

BenchmarkReport ​

class View source

python
BenchmarkReport(benchmark_version: str, manifest_sha256: str, execution: BenchmarkExecution, platform: PlatformSnapshot, started_at: str, finished_at: str | None, status: ReportStatus, cases: list[CaseRecord])

Outer report for one complete or checkpointed Benchmark v1 run.

benchmark_version versions the entire Benchmark specification. FHElium, Python, Torch, CUDA, and native-build versions remain platform provenance. No total or composite score is defined.

Attributes ​

NameTypeDefault/value
benchmark_versionstr
manifest_sha256str
executionBenchmarkExecution
platformPlatformSnapshot
started_atstr
finished_atstr | None
statusReportStatus
caseslist[CaseRecord]

requires_nonzero_exit ​

property

python
requires_nonzero_exit: bool

Whether a CLI should fail due to failure or interruption.

suggested_exit_code ​

property

python
suggested_exit_code: int

Return the decision for a CLI without exiting the process.

to_dict ​

method

python
def to_dict() -> dict[str, Any]: ...

validate_state ​

method

python
def validate_state() -> None: ...

Reject contradictory report lifecycle and outcome fields.

from_dict ​

method

python
def from_dict(payload: Mapping[str, Any]) -> BenchmarkReport: ...

BenchmarkRunner ​

class View source

python
BenchmarkRunner(*, platform_collector: Callable[..., PlatformSnapshot]=collect_platform, report_writer: ReportWriter=write_report_atomic, clock: Callable[[], str]=_utc_now)

Run the fixed Benchmark v1 specification with atomic checkpoints.

The constructor binds the five package-owned cases and definitions directly. The CLI and run_benchmark accept no registry or case/profile/parameter overrides.

run ​

method

python
def run(*, execution: BenchmarkExecution, output_path: str | PathLike[str], invocation: Sequence[str] | None=None, progress: ProgressCallback | None=None, platform_snapshot: PlatformSnapshot | None=None) -> BenchmarkReport: ...

Run Benchmark v1, checkpoint after every case, and return its report.

BenchmarkExecution ​

class View source

python
BenchmarkExecution(backend: ExecutionBackend, device: str)

Report-level backend and indexed device shared by every v1 case.

Attributes ​

NameTypeDefault/value
backendExecutionBackend
devicestr

to_dict ​

method

python
def to_dict() -> dict[str, str]: ...

from_dict ​

method

python
def from_dict(payload: Mapping[str, Any]) -> BenchmarkExecution: ...

CaseFailure ​

class View source

python
CaseFailure(stage: str, error_type: str, message: str, traceback: tuple[str, ...])

Structured exception information retained in a Benchmark v1 run.

Attributes ​

NameTypeDefault/value
stagestr
error_typestr
messagestr
tracebacktuple[str, ...]

to_dict ​

method

python
def to_dict() -> dict[str, Any]: ...

from_dict ​

method

python
def from_dict(payload: Mapping[str, Any]) -> CaseFailure: ...

CaseRecord ​

class View source

python
CaseRecord(id: str, title: str, category: str, benchmark: str, workload_id: str, profile: str, parameters: Mapping[str, Any], requirements: Mapping[str, Any], comparison: Mapping[str, Any], status: CaseStatus = CaseStatus.PENDING, started_at: str | None = None, finished_at: str | None = None, result: BenchmarkResult | None = None, unavailable: CaseUnavailable | None = None, failure: CaseFailure | None = None)

Resolved case identity, execution state, and optional leaf result.

Attributes ​

NameTypeDefault/value
idstr
titlestr
categorystr
benchmarkstr
workload_idstr
profilestr
parametersMapping[str, Any]
requirementsMapping[str, Any]
comparisonMapping[str, Any]
statusCaseStatusCaseStatus.PENDING
started_atstr | NoneNone
finished_atstr | NoneNone
resultBenchmarkResult | NoneNone
unavailableCaseUnavailable | NoneNone
failureCaseFailure | NoneNone

validate_state ​

method

python
def validate_state() -> None: ...

Reject contradictory terminal-state payloads.

to_dict ​

method

python
def to_dict() -> dict[str, Any]: ...

from_dict ​

method

python
def from_dict(payload: Mapping[str, Any]) -> CaseRecord: ...

CaseStatus ​

class View source

python
CaseStatus()

Bases: StrEnum

Lifecycle state of one Benchmark v1 case.

Attributes ​

NameTypeDefault/value
PENDING'pending'
RUNNING'running'
MEASURED'measured'
UNAVAILABLE'unavailable'
FAILED'failed'
INTERRUPTED'interrupted'

CaseUnavailable ​

class View source

python
CaseUnavailable(reason: str, details: Mapping[str, Any] = field(default_factory=dict))

Structured explanation for a case that cannot be measured.

Attributes ​

NameTypeDefault/value
reasonstr
detailsMapping[str, Any]field(default_factory=dict)

to_dict ​

method

python
def to_dict() -> dict[str, Any]: ...

from_dict ​

method

python
def from_dict(payload: Mapping[str, Any]) -> CaseUnavailable: ...

FHEliumBuildIdentity ​

class View source

python
FHEliumBuildIdentity(version: str, distribution: Mapping[str, Any], source_git: Mapping[str, Any], native: Mapping[str, Any])

FHElium distribution, source, and native-build provenance.

Attributes ​

NameTypeDefault/value
versionstr
distributionMapping[str, Any]
source_gitMapping[str, Any]
nativeMapping[str, Any]

to_dict ​

method

python
def to_dict() -> dict[str, Any]: ...

from_dict ​

method

python
def from_dict(payload: Mapping[str, Any]) -> FHEliumBuildIdentity: ...

PlatformSnapshot ​

class View source

python
PlatformSnapshot(system: Mapping[str, Any], cpu: Mapping[str, Any], memory: Mapping[str, Any], python: Mapping[str, Any], fhelium_build: FHEliumBuildIdentity, torch: Mapping[str, Any], cuda: Mapping[str, Any], environment: Mapping[str, Any], invocation: tuple[str, ...], probe_errors: tuple[ProbeError, ...] = ())

Normalized hardware, runtime, build, and invocation provenance.

Attributes ​

NameTypeDefault/value
systemMapping[str, Any]
cpuMapping[str, Any]
memoryMapping[str, Any]
pythonMapping[str, Any]
fhelium_buildFHEliumBuildIdentity
torchMapping[str, Any]
cudaMapping[str, Any]
environmentMapping[str, Any]
invocationtuple[str, ...]
probe_errorstuple[ProbeError, ...]()

to_dict ​

method

python
def to_dict() -> dict[str, Any]: ...

from_dict ​

method

python
def from_dict(payload: Mapping[str, Any]) -> PlatformSnapshot: ...

ProbeError ​

class View source

python
ProbeError(probe: str, error_type: str, message: str)

Non-fatal error from an optional platform-information probe.

Attributes ​

NameTypeDefault/value
probestr
error_typestr
messagestr

to_dict ​

method

python
def to_dict() -> dict[str, str]: ...

from_dict ​

method

python
def from_dict(payload: Mapping[str, Any]) -> ProbeError: ...

ReportStatus ​

class View source

python
ReportStatus()

Bases: StrEnum

Lifecycle state of one Benchmark v1 report.

Attributes ​

NameTypeDefault/value
RUNNING'running'
COMPLETED'completed'
FAILED'failed'
INTERRUPTED'interrupted'

collect_platform ​

function View source

python
def collect_platform(*, invocation: Sequence[str] | None=None, environ: Mapping[str, str] | None=None) -> PlatformSnapshot: ...

Collect normalized reproducibility data without requiring every probe.

Optional probe failures are represented in probe_errors. CUDA device and P2P inventory deliberately disables the disruptive bandwidth test. Environment capture is limited to a fixed CUDA/NCCL allowlist.

read_report ​

function View source

python
def read_report(path: str | os.PathLike[str]) -> BenchmarkReport: ...

Read one strict Benchmark v1 report from UTF-8 JSON.

run_benchmark ​

function View source

python
def run_benchmark(*, execution: BenchmarkExecution, output_path: str | PathLike[str], invocation: Sequence[str] | None=None, progress: ProgressCallback | None=None) -> BenchmarkReport: ...

Run the package-owned Benchmark v1 specification.

sanitize_invocation ​

function View source

python
def sanitize_invocation(arguments: Sequence[str]) -> tuple[str, ...]: ...

Remove credential-like values and host-specific home-directory paths.

write_report_atomic ​

function View source

python
def write_report_atomic(path: str | os.PathLike[str], report: BenchmarkReport) -> None: ...

Validate and atomically persist a BenchmarkReport.

Towards full-stack encrypted execution infrastructures

Released under the MIT License. Copyright VisualDust and FHElium contributors