CUDA Graph matrix-vector multiplication
Example source: examples/18_runtime_cuda_graph.py
This example captures a fixed packed CKKS matrix-vector evaluator
Run the example
python examples/18_runtime_cuda_graph.py \
--device cuda:0 \
--preset slots8192-scale40-depth7-int64 \
--size 8 \
--capture-warmup 3 \
--benchmark-warmup 10 \
--runs 1002
3
4
5
6
7
1. Identify static and dynamic state
The workload computes y = A @ x with cyclic diagonals. Its state divides into:
| Static across replays | Dynamic for each replay |
|---|---|
| engine and CKKS context | encrypted input vector |
| encoded matrix diagonals | ciphertext payload |
| direct rotation keys | request-specific values |
| operation schedule | matching value input signature |
Encryption and decryption remain outside the graph.
2. Prepare operation-ready constants
diagonals = [
engine.prepare_plaintext_for_multiplication(
engine.encode(
cyclic_diagonal_slots(matrix, step, engine.num_slots),
depth=0,
)
)
for step in range(matrix.size(0))
]
rotation_keys = {
step: engine.rotation_key(step)
for step in range(1, matrix.size(0))
}2
3
4
5
6
7
8
9
10
11
12
13
These values are captured as callable state. Changing their object identity, storage address, depth, or shape after capture would invalidate the captured schedule.
3. Bind static state
from functools import partial
schedule = partial(
matrix_vector,
engine=engine,
diagonals=diagonals,
rotation_keys=rotation_keys,
)2
3
4
5
6
7
8
The resulting callable has one dynamic argument: the source ciphertext. This is preferable to a hidden global cache because the captured resources are visible through direct Python calls.
4. Capture from a prototype
program = CudaGraphProgram.capture(
schedule,
example_inputs=(prototype,),
warmup=capture_warmup,
)2
3
4
5
CudaGraphProgram performs side-stream warmup, allocates fixed dynamic-input storage, captures the evaluator, records the output storage, and derives an input signature.
The prototype determines structure, including:
- value-tree shape;
- value type;
- tensor shape and dtype;
- CKKS depth and caller-recorded parameter provenance;
- polynomial domain, modulus basis, residue representation, scale, and prime IDs.
5. Replay with changing ciphertexts
result = program.replay(
encrypted,
synchronize=True,
)2
3
4
Replay validates the new value before staging its payload into the fixed input allocation. A mismatched depth or representation is rejected rather than silently converted inside the graph wrapper.
The example uses three different encrypted vectors and verifies all three against matrix @ vector.
6. Understand borrowed output storage
borrowed_pointer = result.data.data_ptr()The default result references graph-owned output storage. The next replay can overwrite it. This avoids an extra device copy when the caller consumes the output immediately.
Use:
owned = program.replay(encrypted, copy_output=True)when a result must survive a later replay or escape into another asynchronous lifetime.
7. Separate construction and steady-state cost
The example reports:
- warmup time;
- graph capture time;
- first replay time;
- eager mean latency;
- graph replay mean latency.
Capture is a one-time program-construction cost. Compare steady-state replay only when the same static schedule will execute enough times to amortize it.
Close the program
program.close()Closing releases graph-owned inputs, outputs, and capture state.
Share scratch across two specializations
Add --shared-pool to run the example's optional two-program demonstration:
python examples/18_runtime_cuda_graph.py --device cuda:0 --shared-poolThe same matrix-vector evaluator is captured for two input scales, CudaGraphPool. Each program keeps its own input signature and retained output. The example replays the second program before the first, checks that the second output survives the first replay unchanged, and compares both decrypted results with the clear matrix-vector product. Both programs close when the pool context ends.
Pooled output leaves have independent cloned storage without autograd history or alias relationships to inputs or other output leaves. The shared-storage guide describes stream ordering and the lifetime of persistent resources. The single-program capture and benchmark remain the default example path.
Source
#!/usr/bin/env python3
"""Capture and replay packed CKKS matrix-vector multiplication.
The fixed evaluator computes ``y = A @ x`` with the cyclic-diagonal method.
The matrix, encoded diagonals, rotation keys, and operation schedule are static;
``functools.partial`` binds them into the captured callable.
each replay stages a newly encrypted vector into the program's dynamic input
buffer. Encryption and decryption remain outside capture.
"""
from __future__ import annotations
import argparse
from collections.abc import Callable
from functools import partial
import torch
from common import (
add_engine_args,
error_stats,
make_engine,
print_table,
time_ms,
)
from fhelium import Ciphertext, Plaintext, RotationKey
from fhelium.eager import Engine
from fhelium.runtime import CudaGraphPool, CudaGraphProgram
def matrix_and_vector(
size: int, *, seed: int
) -> tuple[torch.Tensor, torch.Tensor]:
row = torch.arange(size, dtype=torch.float64).view(-1, 1)
column = torch.arange(size, dtype=torch.float64).view(1, -1)
matrix = 0.018 * torch.sin((row + 1) * (column + 2) * 0.17)
matrix += 0.007 * torch.cos((row + column + 1) * 0.23)
generator = torch.Generator(device=torch.get_default_device()).manual_seed(
seed
)
vector = torch.randn(size, generator=generator, dtype=torch.float64) * 0.025
return matrix, vector
def periodic_slots(values: torch.Tensor, num_slots: int) -> torch.Tensor:
return values.repeat(num_slots // values.numel())
def cyclic_diagonal_slots(
matrix: torch.Tensor,
rotation_step: int,
num_slots: int,
) -> torch.Tensor:
"""Return weights aligned with ``torch.roll(x, rotation_step)``."""
size = matrix.size(0)
row = torch.arange(num_slots) % size
column = torch.remainder(row - rotation_step, size)
return matrix[row, column]
def prepare_constants(
engine: Engine,
matrix: torch.Tensor,
) -> tuple[list[Plaintext], dict[int, RotationKey]]:
diagonals = [
engine.prepare_plaintext_for_multiplication(
engine.encode(
cyclic_diagonal_slots(matrix, step, engine.num_slots), depth=0
)
)
for step in range(matrix.size(0))
]
rotation_keys = {
step: engine.rotation_key(step) for step in range(1, matrix.size(0))
}
return diagonals, rotation_keys
def matrix_vector(
source: Ciphertext,
*,
engine: Engine,
diagonals: list[Plaintext],
rotation_keys: dict[int, RotationKey],
) -> Ciphertext:
"""Evaluate one vector with fixed program state."""
rotated_values = []
for step in range(len(diagonals)):
rotated = (
source
if step == 0
else engine.rotate_with_key(source, rotation_keys[step])
)
rotated_values.append(rotated)
rotated_ntt = engine.coefficient_domain_to_ntt_domain(
Ciphertext.stack_batch(rotated_values)
)
diagonal_batch = Plaintext.stack_batch(diagonals)
weighted = engine.multiply_plaintext(rotated_ntt, diagonal_batch)
return engine.rescale_to_next_depth(
engine.ntt_domain_to_coefficient_domain(
engine.sum_ciphertext_batch(weighted)
)
)
def demonstrate_shared_pool(
schedule: Callable[[Ciphertext], Ciphertext],
*,
engine: Engine,
prototype: Ciphertext,
vector: torch.Tensor,
expected: torch.Tensor,
warmup: int,
) -> None:
"""Replay two actual-scale specializations with shared temporary storage."""
prototypes = (
prototype,
engine.encrypt_message(
periodic_slots(vector, engine.num_slots),
scale=2 * prototype.scale,
),
)
with CudaGraphPool(prototype.device) as pool:
programs = [
CudaGraphProgram.capture(
schedule, example_inputs=(value,), warmup=warmup, pool=pool
)
for value in prototypes
]
second = programs[1].replay(prototypes[1], synchronize=True)
retained = second.data.clone()
first = programs[0].replay(prototypes[0], synchronize=True)
torch.testing.assert_close(second.data, retained, rtol=0, atol=0)
rows = []
for source, result in zip(prototypes, (first, second), strict=True):
actual = engine.decrypt_message(result, is_real=True).cpu()[
: expected.numel()
]
error = error_stats(actual, expected)
rows.append([f"{source.scale:.6e}", f"{error['max_abs']:.3e}"])
print(
"\nShared-pool specializations, replayed in reverse capture order:"
)
print_table(["input scale", "max abs error"], rows)
print("Both programs retain their own output storage.")
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
add_engine_args(
parser,
default_preset="slots8192-scale40-depth7-int64",
initial_device="cuda:0",
)
parser.add_argument("--size", type=int, default=8)
parser.add_argument("--capture-warmup", type=int, default=3)
parser.add_argument("--benchmark-warmup", type=int, default=10)
parser.add_argument("--runs", type=int, default=100)
parser.add_argument(
"--shared-pool",
action="store_true",
help="also capture two actual-scale specializations sharing scratch",
)
args = parser.parse_args()
engine = make_engine(args)
if torch.get_default_device().type != "cuda":
parser.error("this CUDA Graph example requires CUDA")
if args.size <= 0 or engine.num_slots % args.size != 0:
parser.error(
f"--size must be positive and divide num_slots={engine.num_slots}"
)
if args.capture_warmup < 0 or args.benchmark_warmup < 0 or args.runs <= 0:
parser.error("warmup counts must be non-negative and --runs positive")
if args.shared_pool and args.capture_warmup == 0:
parser.error("--shared-pool requires at least one capture warmup")
matrix, prototype_vector = matrix_and_vector(args.size, seed=500)
diagonals, rotation_keys = prepare_constants(engine, matrix)
schedule = partial(
matrix_vector,
engine=engine,
diagonals=diagonals,
rotation_keys=rotation_keys,
)
prototype = engine.encrypt_message(
periodic_slots(prototype_vector, engine.num_slots)
)
program = CudaGraphProgram.capture(
schedule,
example_inputs=(prototype,),
warmup=args.capture_warmup,
)
correctness_rows = []
borrowed_pointer = None
for replay_index, seed in enumerate((501, 502, 503)):
_, vector = matrix_and_vector(args.size, seed=seed)
encrypted = engine.encrypt_message(
periodic_slots(vector, engine.num_slots)
)
result = program.replay(encrypted, synchronize=True)
borrowed_pointer = result.data.data_ptr()
actual = engine.decrypt_message(result, is_real=True).cpu()[: args.size]
error = error_stats(actual, matrix @ vector)
correctness_rows.append(
[
replay_index,
f"{error['max_abs']:.3e}",
f"0x{borrowed_pointer:x}",
]
)
print("Changing-input matrix-vector replay correctness:")
print_table(
["replay", "max abs error", "borrowed output"],
correctness_rows,
)
eager_stats, _ = time_ms(
lambda: schedule(prototype),
warmup=args.benchmark_warmup,
runs=args.runs,
device=torch.get_default_device(),
)
graph_stats, _ = time_ms(
lambda: program.replay(prototype),
warmup=args.benchmark_warmup,
runs=args.runs,
device=torch.get_default_device(),
)
print("\nFixed matrix-vector evaluator latency:")
print_table(
["mode", "mean ms", "relative"],
[
["eager", f"{eager_stats['mean_ms']:.4f}", "1.000x"],
[
"CUDA Graph",
f"{graph_stats['mean_ms']:.4f}",
f"{eager_stats['mean_ms'] / graph_stats['mean_ms']:.3f}x",
],
],
)
stats = program.stats
print(
"\nOne-time construction: "
f"warmup={stats.warmup_seconds:.3f}s, "
f"capture={stats.capture_seconds:.3f}s, "
f"first_replay={stats.first_replay_seconds:.6f}s"
)
print(
"Replay returns borrowed storage at "
f"0x{borrowed_pointer:x}; pass copy_output=True when a result must "
"survive the next replay."
)
program.close()
if args.shared_pool:
demonstrate_shared_pool(
schedule,
engine=engine,
prototype=prototype,
vector=prototype_vector,
expected=matrix @ prototype_vector,
warmup=args.capture_warmup,
)
if __name__ == "__main__":
main()2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275