Choose a homogeneous batch size

Homogeneous batching is a workload execution choice. It can reduce launch overhead and execute several independent messages together, but it also multiplies the active RNS/NTT working set. There is no batch size that is optimal for every preset, depth, GPU, and evaluator.

Use the runnable homogeneous batching tutorial before following this guide.

Prerequisites

Start with ciphertexts sharing configuration, key relation, value state, and placement, and a correct per-message evaluator. Prepare a distinct callable specialization for each admitted batch shape when using fc.compile; keep preparation outside warmed throughput measurements.

1. Preserve one mathematical comparison

Compare the same logical messages and the same evaluator:

batched:  evaluator(ciphertext with batch_shape=(B,))
looped:   [evaluator(member) for member in ciphertext.unbind_batch()]

Keep fixed:

• cleartext inputs and expected outputs;
• preset, depth, scale, and active prime ids;
• NTT backend and direct keyset;
• evaluator schedule, graph mode, and hoisting policy;
• warmup, timed region, and synchronization rule.

All ciphertext members must have one effective encryption-key lineage. Runtime values do not prove parameter or key compatibility; retain both relations in the benchmark setup.

Validate batched ciphertext data exactly against a stacked loop result when the paths begin from the same ciphertext members, then decrypt both against a cleartext oracle.

2. Separate three reproducible questions

Do not report a single ratio as “batch performance.” Measure:

Comparison	Question answered
Singleton batch [1, ...] vs an unbatched value	What does a singleton batch prefix cost?
One B4/B8 evaluation vs a loop over the same B members	Does batching reduce latency for the same logical work?
Complete batched evaluator vs complete loop, with peak memory	Which policy fits the deployed workload and memory budget?

Run every side of these comparisons from the same installed build and with the same evaluator. Record the source revision, software versions, and input construction so the batch-size comparison can be reproduced.

3. Estimate the active working set

A useful first proxy for one ModUp/NTT key-switch digit is:

$$W_{digit}=B\cdot |Q{P}_{active}|\cdot N\cdot 8\text{bytes}.$$

The full peak additionally includes NTT read/write traffic, key rows, two key-switch accumulators, automorphism temporaries, ciphertext outputs, and allocator behavior. The proxy still explains why the same B can be favorable at a later depth and unfavorable at depth zero.

Expanded diagram

100%

Compare the complete active set with effective cache capacity. Multiple live tensors can cross the effective capacity threshold even when one digit alone is smaller than L2.

4. Sweep the depths used by the evaluator

Depth reduces the active Q rows. For Preset.slots32768_scale40_depth34_int64 in the reference configuration:

Depth	Active QP rows	One-message extended QP digit
0	39	19.5 MiB
5	34	17.0 MiB
10	29	14.5 MiB
15	24	12.0 MiB
20	19	9.5 MiB
25	14	7.0 MiB
30	9	4.5 MiB

Do not benchmark only depth zero if production work mostly occurs near the end of the chain.

5. Measure latency and peak memory together

Run:

python examples/07_eager_batching.py \
  --preset slots32768-scale40-depth34-int64 --depth 0 --batch-sizes 1,4,8

python examples/07_eager_batching.py \
  --preset slots32768-scale40-depth34-int64 --depth 20 --batch-sizes 1,4,8

python examples/07_eager_batching.py \
  --preset slots32768-scale40-depth34-int64 --depth 30 --batch-sizes 1,4,8

Repeat enough times for stable medians. If the production evaluator uses CUDA Graph, compare batch graph with a loop captured under the same policy; graph capture can remove much of the host-launch disadvantage of the loop.

Reference measurement: RTX PRO 6000 Blackwell

The following data is a dated historical worked example for the measured platform and workload. It preserves the original FHElium 0.20-era values and configuration rather than representing the current benchmark catalog. The page does not retain the PRO 6000 run's source commit or complete software provenance, so use it as context and rerun the workload before making a deployment decision.

Field	Reference value
Date	2026-07-24
GPU	One NVIDIA RTX PRO 6000 Blackwell Max-Q
L2	128 MiB
Device	cuda:0
Presets	40-bit scale family at each recorded slot capacity
Default backend	radix2_compact_group8_smem8
Timing	CUDA-synchronized warm runs; median wall latency
Correctness	Bit-for-bit batched-vs-loop ciphertext data plus cleartext oracle

Setup, key generation, and constant preparation were outside operator and workload timing unless the row names an end-to-end public API.

Singleton batch

For [1, slots] versus [slots]:

• NTT, plaintext multiply, relinearize, rotation, and decryption were generally within 0–4%;
• later-depth rescale reached about 6% overhead;
• encode reached 12–13%, but the absolute difference was about 0.018 ms;
• complete matrix-vector (MxV) and rotate-many workloads were approximately at parity.

This establishes B1 compatibility; it does not predict B4/B8 scaling.

Operator speedup over an explicit loop

Values below are loop latency / batch latency; values below one favor the loop.

Configuration	B	NTT	Rescale	Ct multiply	Relinearize	Rotate	Decrypt	Encrypt
logN = 14, L0	4	2.33x	3.58x	3.82x	3.55x	3.57x	2.65x	3.42x
	8	3.21x	6.80x	7.26x	5.49x	5.90x	4.24x	5.57x
logN = 16, L0	4	0.86x	0.54x	0.72x	0.92x	0.92x	1.60x	1.05x
	8	0.63x	0.53x	0.67x	0.72x	0.74x	1.24x	0.82x
logN = 16, L30	4	1.93x	3.47x	3.47x	1.52x	1.57x	2.58x	2.38x
	8	2.27x	5.11x	2.29x	1.55x	1.56x	3.66x	3.37x

The mechanism is operation-specific. Encode and decrypt can benefit while a key-switch-heavy evaluator at the same depth loses.

logN = 16 crossover

Depth	B4 NTT	B8 NTT	B4 relin	B8 relin	B4 MxV-8	B8 MxV-8
0	0.85x	0.63x	0.92x	0.72x	0.84x	0.70x
5	0.90x	0.82x	0.89x	0.76x	0.84x	0.73x
10	1.01x	0.88x	0.88x	0.89x	0.87x	0.81x
15	1.12x	0.92x	0.98x	0.94x	0.98x	0.86x
20	1.35x	1.08x	1.08x	0.95x	1.09x	0.92x
25	1.54x	1.45x	1.26x	1.16x	1.26x	1.12x
30	1.96x	2.26x	1.51x	1.55x	1.76x	1.71x

On this GPU, B4 became useful earlier than B8. Key-switch and complete-workload crossovers occurred later than the raw NTT crossover because keys, accumulators, automorphism, and output tensors enlarge the working set.

Complete workload and peak memory

Workload/configuration	B	Batch	Loop	Speedup	Batch peak	Loop peak
MxV-8, logN = 14, L0	4	4.586 ms	18.137 ms	3.95x	85 MiB	26 MiB
	8	6.159 ms	40.987 ms	6.65x	162 MiB	34 MiB
MxV-8, logN = 16, L0	4	97.232 ms	81.511 ms	0.84x	1512 MiB	484 MiB
	8	233.103 ms	163.607 ms	0.70x	3024 MiB	620 MiB
MxV-8, logN = 16, L30	4	7.267 ms	12.755 ms	1.76x	216 MiB	74 MiB
	8	14.458 ms	25.435 ms	1.76x	433 MiB	84 MiB
Rotate-many-7, logN = 16, L0	4	52.581 ms	34.573 ms	0.66x	2274 MiB	1334 MiB
	8	115.457 ms	68.966 ms	0.60x	4548 MiB	2343 MiB

Hardware contrast: RTX A6000

A second historical measurement on 2026-07-25 used commit 9f4a1756bc20dd17f172b70ae88b9cf1ffd855c4 and one NVIDIA RTX A6000 (SM86, 6 MiB L2, 47.4 GiB device memory). The same-build batch and loop paths matched exactly at the ciphertext-data depth. Singleton batches remained at parity: the median loop/batch ratio was 1.001x for logN = 14 and 1.000x for both logN = 15 and logN = 16.

The one-message QP digit size identifies whether increasing B creates a new fit-to-spill transition:

logN / depth	Active QP rows	QP digit per message	Measured behavior
14 / L0	9	1.125 MiB	B4/B8 eager evaluators benefited from launch amortization.
14 / L3	6	0.750 MiB	B8 MxV-8 reached 2.55x.
15 / L0	19	4.750 MiB	B1 nearly fits L2; B2 and larger crossed the effective capacity threshold.
15 / L8	11	2.750 MiB	B2 MxV-8 was 1.08x, while B4 was 0.90x.
16 / L0	39	19.500 MiB	B1 already streamed beyond L2; radix16_compact gave modest batch gains.
16 / L25	14	7.000 MiB	B8 MxV-8 was 1.07x with radix16_compact.
16 / L30	9	4.500 MiB	B8 MxV-8 was 1.09x with radix16_compact.

The full active set remains larger than the digit proxy. Ask whether B changes the cache-residency regime, which can happen even when a single digit tensor is smaller than nominal L2 capacity.

CUDA Graph also changed the policy because it removed most of the explicit loop's host-submission disadvantage. Values remain loop / batch:

Configuration	Eager	Graph replay
logN = 14, L0 B8 MxV-8, group8	1.65x	0.91x
logN = 14, L0 B8 rotate-many-7, group8	1.56x	0.82x
logN = 15, L0 B8 MxV-8, group4	0.89x	0.85x
logN = 16, L0 B8 MxV-8, radix16	1.05x	1.04x
logN = 16, L25 B8 MxV-8, radix16	1.07x	1.03x

On this A6000, logN = 15 exposed the clearest fit-to-spill loss: one depth-zero message was close to the 6 MiB L2 capacity, while even B2 was not. At logN = 16, L0 B1 was already larger than L2, so increasing B did not introduce the same new cache transition. The genuine radix-16 backend reduced enough transform work to retain small batch gains. A dispatch rule therefore needs measured backend structure and graph policy in addition to L2 size.

6. Put the choice in application code

Use a transparent policy:

def evaluate_requests(requests, *, use_homogeneous_batch, evaluator):
    if use_homogeneous_batch:
        return evaluator(Ciphertext.stack_batch(requests))
    return [evaluator(request) for request in requests]

In a real application, prefer creating the batch before encryption so stack_batch does not add a copy. The snippet only makes the selection point visible.

A reasonable starting policy for the reference GPU was:

logN = 14:
    benchmark and normally batch B4/B8

logN = 16 depth <= 15, key-switch-heavy workload:
    normally loop

logN = 16 depth 20:
    test B4; normally loop B8

logN = 16 depth >= 25:
    benchmark and normally batch B4/B8

For the measured RTX A6000, a more useful starting point was:

logN = 14 eager evaluator:
    benchmark B4/B8

logN = 14 graph evaluator:
    compare against a graph-captured explicit loop

logN = 15 key-switch-heavy evaluator:
    begin with the explicit loop

logN = 16:
    begin with radix16_compact and treat measured batch gains as modest

Re-run the example when the GPU, backend, preset, depth distribution, graph policy, workload, or memory budget changes.

Verify the outcome

Choose the smallest batch that meets the application’s throughput target without violating latency or memory limits. Retain unbatched, B1, and larger-batch results with active depth and operation schedule; do not merge measurements across incompatible signatures.

Related documentation

• Homogeneous batching tutorial
• CKKS workload cost model
• Benchmark methodology
• Optimize a workload systematically