Proto-to-Client Method Mapping¶
Generated 2026-06-11. Authored after the sync-vs-async audit session, which raised the recurring question of "how does a proto method end up as a sync/async client method?"
Covers
virtufin-api,virtufin-websocketmanager,virtufin-workmanager.Related specs: - Sync vs Async Audit — the C#
IEngine/LoadCode/ProcessAsyncdecisions that motivated this spec - Client Testing — cross-language test parity
Purpose¶
Document the end-to-end pipeline that turns a protobuf service definition into a callable client method in each of the three target languages (C#, Python, TypeScript), with particular attention to:
- Where the sync/async decision is made at each layer of the pipeline.
- Why two distinct mappings exist (static stub layer + dynamic dispatch layer).
- Per-language naming conventions that the wrapper layer imposes on top of the generated stubs.
- The known inconsistencies in the existing wrapper layer, and the rationale for leaving them in place.
This spec is descriptive, not prescriptive: it documents the current state of the code, not a desired future state. The follow-up "Known Gaps" section lists items that are deliberately not addressed here.
Architecture: 4 layers¶
The pipeline that turns a .proto file into a client method has four distinct layers.
Each layer makes its own sync-vs-async decision, and the decisions are NOT uniform
across layers or across languages.
┌──────────────────────┐
│ .proto source │
│ (single source of │
│ truth) │
└──────────┬───────────┘
│
┌────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
┌──────────────────┐ ┌──────────────┐ ┌──────────────────┐
│ buf generate │ │ grpc_tools │ │ buf + │
│ (per-protos- │ │ .protoc │ │ protoc-gen-es │
│ repo, C# only) │ │ (Python) │ │ (TypeScript) │
└────────┬─────────┘ └──────┬───────┘ └────────┬─────────┘
│ │ │
▼ ▼ ▼
┌──────────────────┐ ┌──────────────┐ ┌──────────────────┐
│ Gateway.cs │ │ gateway_pb2 │ │ websocketmanager │
│ GatewayGrpc.cs │ │ gateway_pb2_ │ │ _pb.js │
│ (C# stubs) │ │ _grpc.py │ │ (TS stubs) │
└────────┬─────────┘ │ (Python) │ └────────┬─────────┘
│ └──────┬───────┘ │
│ │ │
▼ ▼ ▼
┌──────────────────────────────────────────────────────┐
│ Hand-written wrapper layer (per-language, per-repo) │
│ C#: Virtufin.Api.Client/ApiClient.cs │
│ Py: virtufin-api/client.py │
│ TS: virtufin-*-typescript/src/client.ts │
└────────────────────────┬─────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────┐
│ Dynamic dispatch (runtime) │
│ client.gateway.<service>.<method>() │
│ → Gateway.InvokeJson RPC │
│ → gRPC reflection on backend │
└──────────────────────────────────────────────────────┘
Each layer is documented in detail below.
Layer 1: Proto source¶
The .proto files live in each service repo:
virtufin-api/src/Virtufin.Api.Protos/proto/gateway.protovirtufin-api/src/Virtufin.Api.Protos/proto/config.protovirtufin-api/src/Virtufin.Api.Protos/proto/state.protovirtufin-api/src/Virtufin.Api.Protos/proto/pubsub.protovirtufin-websocketmanager/src/Virtufin.WebSocketManager.Protos/proto/websocketmanager.protovirtufin-workmanager/src/Virtufin.WorkManager.Protos/proto/workmanager.proto
A proto service definition looks like:
service Gateway {
rpc Invoke (InvokeRequest) returns (InvokeResponse);
rpc InvokeJson (InvokeJsonRequest) returns (InvokeJsonResponse);
rpc ListServices (ListServicesRequest) returns (ListServicesResponse);
rpc Subscribe (SubscribeRequest) returns (stream TopicEventRequest);
...
}
Sync/async at this layer: not present. Proto has no notion of sync vs async.
It only specifies:
- The method name (PascalCase by convention; e.g., Invoke, ListServices)
- The request and response message types
- The streaming mode (stream keyword on the request and/or response side)
The streaming mode is the only behavioral attribute of a method that affects how downstream layers see it:
| Proto signature | Streaming mode | What generators see |
|---|---|---|
rpc X (Req) returns (Res); |
Unary | Single request, single response |
rpc X (Req) returns (stream Res); |
Server streaming | Single request, stream of responses |
rpc X (stream Req) returns (Res); |
Client streaming | Stream of requests, single response |
rpc X (stream Req) returns (stream Res); |
Bidirectional | Stream in both directions |
virtufin-api, virtufin-websocketmanager, and virtufin-workmanager use only
unary and server-streaming methods. There are no client-streaming or
bidirectional methods anywhere in the codebase.
The buf.gen.yaml per-protos-repo¶
Each *.Protos/proto/buf.gen.yaml file declares the generator plugins to run:
version: v2
clean: true
inputs:
- directory: .
exclude_paths:
- google
plugins:
- protoc_builtin: csharp
protoc_path: protoc
out: ../Generated
- local: grpc_csharp_plugin
out: ../Generated
Only C# generation is wired up in buf.gen.yaml. Python and TypeScript stubs
are generated by a separate path (Layer 2b) — they are NOT part of the buf generate
pipeline. See the "Build pipeline" section below.
Layer 2: Generator stubs (per language)¶
Each language's gRPC code generator plugin produces stubs with its own naming convention. The choice of sync vs async is the plugin's, not the proto's or the application's.
2a. C# via grpc_csharp_plugin¶
The C# gRPC plugin (grpc_csharp_plugin) is run locally on the dev machine and
on the CI runner (see virtufin-common/.github/workflows/nuget-common.yaml).
For each proto service, it produces a <ServiceName>Client class in
src/Virtufin.<Service>.Protos/Generated/<ServiceName>Grpc.cs.
Per the C# plugin's convention, each proto method X produces two methods
on the client class:
| Method | Signature | Behavior |
|---|---|---|
X |
virtual Res X(Req request, Metadata headers = null, DateTime? deadline = null, CancellationToken cancellationToken = default) |
Sync, blocking. Calls CallInvoker.BlockingUnaryCall and returns the response (or yields a stream iterator for server-streaming). |
XAsync |
virtual AsyncUnaryCall<Res> XAsync(Req request, Metadata headers = null, DateTime? deadline = null, CancellationToken cancellationToken = default) |
Async, callback-based. Returns an AsyncUnaryCall<Res> handle that the caller awaits or attaches callbacks to. |
For server-streaming methods (e.g., Subscribe), the plugin generates a single
method X that returns IAsyncStreamReader<Res> (or AsyncServerStreamingCall<Res>)
— both sync-style and async-style code paths exist on the same call.
The C# plugin's "both" pattern is what the Sync vs Async Audit
refers to when it says "C# gRPC convention: X + XAsync" — this is a property of
the plugin, not of Virtufin's design.
2b. Python via grpc_tools.protoc¶
Python stubs are generated by the GrpcClientGenerator.generate_python_stubs() method
in virtufin-common/src/python/virtufin_dev/grpc_client_generator.py, which calls
grpc_tools.protoc.main() with three output formats:
args = [
"",
f"--proto_path={tmpdir}",
f"--python_out={python_output}", # produces *_pb2.py (message types)
f"--grpc_python_out={python_output}", # produces *_pb2_grpc.py (stubs)
f"--pyi_out={python_output}", # produces *.pyi (type stubs)
*proto_files,
]
The Python gRPC plugin (grpc_python_plugin, bundled with grpcio-tools) produces
a <ServiceName>Stub class in *_pb2_grpc.py with the following convention:
| Method | Signature | Behavior |
|---|---|---|
X |
X(request, target, options=(), channel_credentials=None, call_credentials=None, insecure=False, compression=None, wait_for_ready=None, timeout=None, metadata=None) |
Sync, blocking. Returns the response. The actual call is grpc.experimental.unary_unary(...) for unary methods, which is a thin wrapper around grpc.Channel.unary_unary. |
| (none for async) | — | The Python plugin does not generate a separate async variant. Async-style usage is achieved by passing a grpc.aio.Channel to the Stub.__init__ — the same X(request) method then becomes awaitable. |
The grpc.aio.Channel makes the X method awaitable via Python's __await__
protocol. This is misleading: the generated stub code looks like sync code, but
when used with grpc.aio.Channel, it actually returns a coroutine. This is a
property of the Python plugin and the aio.Channel integration, not of Virtufin's
code.
The Python GatewayStub.Invoke method is sync-shaped (takes a request, returns
a response) but the actual call site in virtufin-api/src/python/virtufin/api/client.py
always uses await self._stub.X(request). There is no native InvokeAsync method
on the Python stub.
The Python stubs are gitignored (src/python/virtufin/*_pb2*.py in
.gitignore). They are regenerated by uv sync in the Python package
build process (via GrpcClientGenerator).
2c. TypeScript via @bufbuild/protoc-gen-es¶
TypeScript stubs are generated by the GrpcClientGenerator.generate_typescript_stubs()
method, which writes a temporary buf.gen.yaml with the protoc-gen-es plugin and
runs buf generate --include-imports:
ts_buf_gen.write_text("""version: v2
managed:
enabled: true
plugins:
- local: protoc-gen-es
out: src/generated
""")
The protoc-gen-es plugin (from @bufbuild/protoc-gen-es, the Buf project) produces
a <service>_pb.ts and <service>_pb.js in src/typescript/src/generated/. It
uses ES module conventions and produces pure TypeScript with no runtime
gRPC dependency — the gRPC-Web / Connect-RPC transport is provided separately by
@connectrpc/connect-web.
| Method | Signature | Behavior |
|---|---|---|
X |
X(request: Req, options?: CallOptions): Promise<Res> |
Async-only. Always returns a Promise. There is no sync variant. |
| (for streaming) | X(request: Req, options?: CallOptions): AsyncIterable<Res> |
Async iterable over the stream of responses. |
The TypeScript stub has no sync variant because the underlying transport
(@connectrpc/connect-web) is browser-oriented and natively async.
The TypeScript stubs are gitignored (src/typescript/src/generated/ in
.gitignore). They are regenerated by npm install (which runs buf generate
via a postinstall hook in the typescript subpackage).
2-summary: per-language stub convention¶
| Language | Sync variant | Async variant | Streaming variant | Generator source |
|---|---|---|---|---|
| C# | X(...) returning Res (blocking) |
XAsync(...) returning Task<Res> |
X(...) returning IAsyncStreamReader<Res> (unary-style API with awaitable backing) |
grpc_csharp_plugin (local) |
| Python | X(...) returning Res (blocking, or await-able when called with grpc.aio.Channel) |
(none) | X(...) returning iterator (sync) or AsyncIterator (with aio.Channel) |
grpc_python_plugin (bundled with grpcio-tools) |
| TypeScript | (none) | X(...) returning Promise<Res> |
X(...) returning AsyncIterable<Res> |
@bufbuild/protoc-gen-es (via Buf) |
The plugin is the source of truth for naming. To change the convention, you
must swap the plugin (e.g., for Python, switching to betterproto would yield a
different async surface). Virtufin does not impose a custom naming layer at the
stub layer; it accepts the plugin's defaults.
Layer 3: Hand-written wrapper layer¶
The raw generator stubs are usable but unfriendly. Virtufin wraps them in a hand-written client library per language. The wrapper layer is where Virtufin's own naming and API-surface decisions are made — and where the sync/async divergences first appear.
3a. C#: Virtufin.Api.Client/ApiClient.cs (and equivalents in the other 2 services)¶
The C# wrapper has three classes:
ApiClient— top-level client; owns theGrpcChanneland the 3 generated stubs (Gateway.GatewayClient,State.StateClient,Pubsub.PubsubClient). Configured in its constructor with the gRPC channel.GatewayClient— wraps theGatewaystub. Inherits fromDynamicObjectso thatclient.Gateway.workmanagerdynamically resolves to aServiceClientinstance for the named backend service. The service is cached per-name.ServiceClient— wraps a single named backend service. Inherits fromDynamicObjectso thatclient.Gateway.workmanager.ListWorkersdynamically resolves to an awaitable that callsgateway.InvokeAsync("workmanager", "ListWorkers", null).
The dynamic resolution is implemented via TryGetMember overrides. The
extracted method name is passed as a string to the Gateway's InvokeJson RPC
(see Layer 4).
Sync/async at this layer: async-only. Every method on the C# wrapper returns
Task<T> or IAsyncEnumerable<T> (for streaming). There is no sync variant
of any public method on ApiClient, GatewayClient, or ServiceClient. The
async-only design matches the C# gRPC plugin's XAsync surface.
Method names use the C# convention: MethodNameAsync for methods returning
Task<T> and MethodName (without Async suffix) for IAsyncEnumerable<T>
or Action<T>-style event handlers. Examples:
Task<Dictionary<string, object?>> InvokeAsync(string service, string method, Dictionary<string, object?>? requestData = null)IAsyncStreamReader<TopicEventRequest> Subscribe(IStreamEventHandler handler)Task<List<string>> ListServicesAsync()
The "Async" suffix is omitted for Subscribe because the result is an
async-stream (the async-ness is in the return type, not the call).
The C# IEngine interface in the workmanager (referenced by the
Sync vs Async Audit §C-tier) follows the same
convention: Task LoadCodeAsync(byte[] code, CancellationToken cancellationToken = default)
and Task<CloudEvent?> ProcessAsync(CloudEvent input, CancellationToken cancellationToken = default).
3b. Python: virtufin-api/src/python/virtufin/api/client.py (and equivalents)¶
The Python wrapper has three classes analogous to the C# ones:
ApiClient(host, grpc_port)— top-level client; owns thegrpc.aio.Channeland the 3 generated stubs (gateway_pb2_grpc.GatewayStub,state_pb2_grpc.StateStub,pubsub_pb2_grpc.PubsubStub). Constructor__aenter__/__aexit__provides async context-manager support.GatewayClient(channel, async_mode=True)— wraps theGatewayStub. Inherits fromobject(noDynamicObjectequivalent); uses__getattr__to dynamically resolvegateway.<service>to aServiceClient. Service cache.ServiceClient(gateway_client, service_name)— wraps a single named backend. Uses__getattr__to dynamically resolve<method>to an awaitable that callsgateway.invoke_json_async(...).
Sync/async at this layer: async-only. All RPC methods on the Python wrapper
are async def. There is no sync variant of any data operation (no
invoke_json, invoke, save_state, delete_state, publish,
subscribe_sync_iter, list_services, or list_methods).
| RPC method | Signature |
|---|---|
list_services_async() |
async def returning List[str] |
list_methods_async(service) |
async def returning List[Dict[str, Any]] |
invoke_json_async(service, method, request_data) |
async def returning Dict[str, Any] |
invoke_async(service, method, request_data) |
async def returning Dict[str, Any] |
invoke_raw_async(service, method, request_data) |
async def returning Dict[str, Any] |
save_state_async(service, key, value, etag, include_value) |
async def returning Dict[str, Any] |
delete_state_async(service, key, include_value) |
async def returning Dict[str, Any] |
get_state_async(service, key) |
async def returning Dict[str, Any] |
get_all_state_async(service) |
async def returning Dict[str, Any] |
register_keys_async(service, keys) |
async def returning Dict[str, Any] |
publish_async(topic, data, metadata) |
async def returning Dict[str, Any] |
publish_with_result_async(topic, data, reply_topic, *, timeout, metadata, correlation_id) |
async def returning PubsubEvent |
subscribe_async_with_handler(handler, services, topics, event_types) |
async def returning None |
subscribe_async_iter(topic) |
async def (async generator) yielding PubsubEvent |
unsubscribe_async(subscription_id) |
async def returning Dict[str, Any] |
The async variants use await self._stub.X(request) and return awaitables.
All 14 RPC methods follow the *_async (or *_async_iter /
*_async_with_handler) suffix convention for consistency.
Renamed in this change (3 non-async methods → *_async):
- GatewayClient.subscribe(...) → GatewayClient.subscribe_async(...) —
formerly returned the raw aio.stub.Subscribe() UnaryStreamCall (which
is itself an AsyncIterator, so callers used async for event in stream
without an outer await). Now async def — callers must await to get
the iterator.
- PubsubClient.subscribe(topic) → PubsubClient.subscribe_async(topic) —
same pattern as above.
- DaprPublisher.publish(topic, data, data_content_type) →
DaprPublisher.publish_async(topic, data, data_content_type) — formerly
a true sync method wrapping the sync dapr.clients.grpc.client.DaprGrpcClient.
Now uses the async dapr.aio.clients.grpc.client.DaprGrpcClientAsync.
The ServiceClient.__getattr__ returns an async def (awaitable) regardless
of call context — the dynamic-dispatch path has always been async-only.
3c. TypeScript: src/client.ts (per typescript subpackage)¶
The TypeScript wrapper is flat — no dynamic dispatch, no nested service proxy. It exposes one class per proto service with one async method per proto RPC:
// virtufin-websocketmanager/src/typescript/src/client.ts
export class WebSocketManagerClient {
async connect(url: string, autoReconnect?: boolean): Promise<ConnectResponse>
async list(): Promise<ListResponse>
async disconnect(id: string): Promise<DisconnectResponse>
async send(id: string, message: Uint8Array, contentType: string, timeoutMs?: number): Promise<SendResponse>
async sendRaw(id: string, message: Uint8Array, contentType: string): Promise<SendRawResponse>
async startPublish(id: string, topic: string): Promise<StartPublishResponse>
async stopPublish(id: string): Promise<StopPublishResponse>
async close(): Promise<void>
}
The virtufin-api/src/typescript/src/client.ts and
virtufin-workmanager/src/typescript/src/client.ts follow the same pattern.
Sync/async at this layer: async-only. No sync variant. The TypeScript
gRPC-Web transport (@connectrpc/connect-web) is inherently async, and
TypeScript projects don't use the aio.Channel trick that Python does, so
there is no path to a sync surface.
Inconsistency note: the TypeScript wrapper does not use the gateway.<service>.<method>()
dynamic-dispatch pattern that the C# and Python wrappers do. It exposes a flat
method-per-RPC surface. Callers who want to call methods on multiple services
must instantiate one client per service. This is a known design wart, listed
in "Known Gaps" below.
Layer 4: Dynamic dispatch¶
Both the C# and Python wrappers use dynamic dispatch to allow the
gateway.<service>.<method>() syntax. The mechanism differs per language
(no __getattr__ in C#, but DynamicObject.TryGetMember is the equivalent).
How a call is resolved at runtime¶
When a Python user writes:
result = await client.gateway.workmanager.CreateWorker({"code_source": ..., "mime_type": "text/x-python"})
…the resolution chain is:
client.gateway— resolves viaApiClient.__getattr__to aGatewayClient(or, depending on the version, directly to aGatewayClientinstance cached per service name)..workmanager— resolves viaGatewayClient.__getattr__to aServiceClientfor the backend service named "workmanager" (cached on first access)..CreateWorker— resolves viaServiceClient.__getattr__to anasync defclosure that captures("workmanager", "CreateWorker")and returns it.- The closure, when called with
({"code_source": ...}), callsself._gateway.invoke_json_async("workmanager", "CreateWorker", {"code_source": ...}). invoke_json_asyncbuilds anInvokeJsonRequest{service="workmanager", method="CreateWorker", request_data='{"code_source": ...}'}and awaitsself._stub.InvokeJson(request).- The
GatewayStub.InvokeJsonmethod (generated by the Python gRPC plugin) invokes the Gateway's gRPC service over the wire. - The Gateway service (
virtufin-api/src/Virtufin.Api/Services/GatewayService.cs) receives theInvokeJsonRequest, parses the method name, uses gRPC reflection on the target backend service to discover the method's request/response types, marshals the JSON into the appropriate Protobuf message, and invokes the backend's gRPC method. - The backend's gRPC method runs (e.g., the workmanager's
CreateWorkerinVirtufin.WorkManager) and returns a Protobuf response, which the Gateway marshals back to JSON and returns to the caller.
The C# flow is the same except for steps 1-4 (uses DynamicObject.TryGetMember
instead of __getattr__).
Why two layers of mapping exist¶
There are two distinct mappings in the call chain:
-
Static mapping (compile-time): the proto method name (e.g.,
CreateWorker) becomes a generated stub method on the client class. This mapping is deterministic and is performed by the gRPC code generator plugin. -
Dynamic mapping (runtime): the method name is passed as a string over the wire to the Gateway service, which uses gRPC reflection to look up and call the actual backend method. This mapping is performed at runtime by the Gateway's
GrpcReflectionServiceandGatewayService.
Why both? The dynamic layer exists so that a client of the API Gateway
(e.g., a Python script) can call methods on a backend service
(e.g., the workmanager) without importing the workmanager's proto definition
or its generated client stub. The client only needs the Gateway's proto and
the GatewayStub; the actual method call is resolved at runtime via the
Gateway's reflection. This decouples clients from the backend proto changes:
a new method added to the workmanager is callable from the Python client
without any changes to the Python package.
The cost is a serialization round-trip (the Gateway marshals JSON ↔ Protobuf
on every call) and a runtime reflection lookup on first call (cached
afterward). For most use cases this is acceptable; for high-throughput
direct-to-backend calls, the user is expected to use the
Virtufin.<Service>.Client library directly, which is generated from
Layer 2 stubs without going through the Gateway.
Sync vs Async decision matrix¶
Putting all four layers together, here's the sync vs async matrix for the common usage patterns:
| Usage pattern | C# surface | Python surface | TypeScript surface |
|---|---|---|---|
| Direct stub call (unary, sync) | stub.Method(req) returning Res (blocking) |
stub.Method(req) returning Res (blocking) |
(n/a — no sync variant) |
| Direct stub call (unary, async) | await stub.MethodAsync(req).ResponseAsync (callback-style) or await stub.MethodAsync(req) (Task-style) |
await stub.Method(req) (using grpc.aio.Channel) |
await client.method(req) |
| Direct stub call (server-streaming) | stub.Subscribe(req) returning IAsyncStreamReader<Res> (await the stream to get results) |
stub.Subscribe(req) returning iterator (sync) or async iterator (with aio.Channel) |
for await (const r of client.subscribe(req)) |
Wrapper call (e.g., ListServices) |
await client.ListServicesAsync() (async-only) |
await client.list_services_async() (async-only) |
(n/a — no Gateway-style wrapper in TS) |
Dynamic dispatch (gateway.<service>.<method>) |
await client.Gateway.<service>.<method>(args) (async-only) |
await client.gateway.<service>.<method>(args) (async-only — the closure is async def regardless) |
(n/a — no dynamic dispatch in TS) |
| Gateway InvokeJson RPC | await client.InvokeAsync(service, method, data) |
await client.invoke_json_async(service, method, data) (async-only) |
(n/a — TS clients don't talk to the Gateway directly) |
The bottom line:
- C# is uniformly async at the wrapper and dynamic-dispatch layers.
The async variant is the only public surface; the C# gRPC plugin's
sync variants (
X) are generated but not used by the wrapper. - Python is uniformly async at the wrapper and dynamic-dispatch layers
(after the rename of
subscribe/subscribe/publish→*_async). The Python wrapper matches the C# and TypeScript wrappers in being strictly async-only at the public surface. - TypeScript is uniformly async at every layer, because the underlying transport is browser-oriented and there is no sync surface available.
Build pipeline¶
The GrpcClientGenerator class in
virtufin-common/src/python/virtufin_dev/grpc_client_generator.py orchestrates
the per-language generation:
| Method | Layer 2 generator used | Output location |
|---|---|---|
generate_python_stubs() |
grpc_tools.protoc with --python_out + --grpc_python_out + --pyi_out |
src/python/virtufin/<service>/ (gitignored) |
generate_typescript_stubs() |
buf generate --include-imports with a temp buf.gen.yaml using protoc-gen-es |
src/typescript/src/generated/ (gitignored) |
| (C#) | buf generate from the per-protos-repo buf.gen.yaml |
src/Virtufin.<Service>.Protos/Generated/ (gitignored) |
The Python GrpcClientGenerator is invoked from per-service build scripts:
virtufin-api/scripts/build_python_client.pyvirtufin-websocketmanager/scripts/build_python_client.py(if present)virtufin-workmanager/scripts/build_python_client.py(if present)
The TypeScript side is invoked by the npm install postinstall hook in each
typescript subpackage.
The C# side is invoked by buf generate from the protos repo (driven by
buf.gen.yaml), and is typically run manually by the developer or by the CI
workflow when the proto changes.
Why C# is in buf.gen.yaml but Python/TS are not¶
The Python and TypeScript stubs are per-service-repo (the Python client for
the workmanager is in virtufin-workmanager/src/python/, not in a separate
generator repo). They are regenerated when the Python or TypeScript package
is built. The C# stubs are in a separate project
(src/Virtufin.<Service>.Protos/Generated/) and need to be regenerated whenever
a downstream consumer of the protos changes, so they live in the protos repo
itself and are version-controlled via the buf.gen.yaml config.
This split is a historical accident (the protos repo was set up first) rather
than a deliberate design. A future cleanup could move all 3 languages to
buf.gen.yaml in the protos repo, but it's not a high-priority item.
Generated files that are gitignored¶
The following patterns are gitignored in each service repo (added in a previous session):
src/Virtufin.*.Protos/Generated/(C#)src/python/*_pb2*.py,src/python/virtufin/*_pb2*.py,src/python/virtufin/<service>/*_pb2*.py(Python)src/typescript/src/generated/(TypeScript)
These files are regenerated on every build. They are NOT part of the version- controlled source.
Known Gaps (deliberately not addressed in this spec)¶
These are design warts that this spec documents but does not propose fixing:
-
TypeScript wrapper has no
gateway.<service>.<method>()pattern. The TypeScript clients are flat (WebSocketManagerClient,WorkManagerClient,ApiClient) with no dynamic dispatch. A caller that wants to invoke methods on multiple services from one place must instantiate one client per service. The C# and Python wrappers have the dynamic dispatch; the TypeScript one doesn't. This is a known inconsistency; the rationale is that@connectrpc/connectdoesn't have a clean equivalent ofDynamicObject/__getattr__for stub-method resolution at runtime. -
The Gateway
InvokeJsonRPC does not support streaming. The dynamic dispatch path (Layer 4) is unary-only — there is noInvokeJsonStreamRPC. Streaming methods (e.g.,Subscribe) cannot be called through the dynamic dispatch; callers must use the direct stub layer for those. This is a functional limitation, not a documentation gap. -
buf.gen.yamlin the protos repos only generates C#. Python and TypeScript are generated by separate paths (per-servicebuild_python_client.pyand per-typescript-subpackagenpm installpostinstall). The C# / Python / TS generation are not unified into a singlebuf generateinvocation. This means the 3 language stubs can drift if the proto changes but only some generators are re-run. -
The Python
GatewayStublooks sync but is actually async when used withgrpc.aio.Channel. A maintainer reading the generated code (gateway_pb2_grpc.py) seesMethod(request, ...)and might assume it's sync. It isn't — the same method is awaitable when the stub is constructed withgrpc.aio.Channel. This is a property of the Python gRPC plugin and is not specific to Virtufin; it's a common gotcha for Python gRPC developers.
Verification: how to test a change to the pipeline¶
If you modify any of the four layers, here is the recommended test sequence:
-
Layer 1 (proto):
cd src/<Service>.Protos/proto && buf generateshould regenerate the C# stubs with no errors. Diff against the previous commit to see the change. -
Layer 2a (C#): After
buf generate, build the .NET solution and run the test suite.Virtufin.<Service>.Client.Testsexercises the generated stubs. -
Layer 2b (Python): From the service repo, run
python3 <(curl -L https://git.haenerconsulting.com/virtufin/virtufin-common/raw/branch/master/scripts/generate-client-tests.py) <proto> --prefix <X>to regenerate the Python client tests. Thencd src/python && uv sync && uv run pytest tests/python/. -
Layer 2c (TypeScript): From the typescript subpackage, run
npm install(which runs the postinstall hook that callsbuf generate). Thennpm test. -
Layer 3 (wrapper): After regenerating the stubs, re-run the integration tests for the wrapper. For Python,
tests/python/test_python_client.pyexercises the wrapper. For C#,Virtufin.<Service>.Client.Tests/IntegrationTestsexercise the wrapper. -
Layer 4 (dynamic dispatch): To test that the dynamic dispatch still works, run an example that uses the
gateway.<service>.<method>()syntax end-to-end. The examplevirtufin-examples/scripts/run_websocket_manager_controller.dotnet.pyexercises the full pipeline: builds a .NET DLL worker, deploys it via the API Gateway, and verifies the dynamic dispatch works by sending commands over Dapr pub/sub.
References¶
- Generated stubs (gitignored, regenerable):
- C#:
src/Virtufin.<Service>.Protos/Generated/<Service>Grpc.cs(3 service repos) - Python:
src/python/virtufin/<service>/*_pb2*.py(api and workmanager only; websocketmanager has no Python stubs) - TypeScript:
src/typescript/src/generated/(3 service repos) - Generator code:
- C#:
grpc_csharp_plugin(external, runs locally and on CI) - Python:
grpc_tools.protoc(called byvirtufin-common/src/python/virtufin_dev/grpc_client_generator.py:124) - TypeScript:
@bufbuild/protoc-gen-es(called byvirtufin-common/src/python/virtufin_dev/grpc_client_generator.py:269) - Wrapper code:
- C#:
virtufin-api/src/Virtufin.Api.Client/ApiClient.cs(and equivalents in the other 2 service repos'<Service>.Clientprojects) - Python:
virtufin-api/src/python/virtufin/api/client.py(and equivalents in the other 2 service repos' Python subpackages) - TypeScript:
src/client.tsin each typescript subpackage - Related specs:
- Sync vs Async Audit
- Client Testing
- API Gateway
- Deployment
- Build pipeline:
virtufin-common/src/python/virtufin_dev/grpc_client_generator.pyvirtufin-api/scripts/build_python_client.pyvirtufin-api/scripts/build_dotnet_client.pyvirtufin-api/scripts/build_typescript_client.py.github/workflows/nuget-common.yaml(CI for C#).github/workflows/pypi-common.yaml(CI for Python).github/workflows/npm-common.yaml(CI for TypeScript)- Proto source:
- 3 service repos'
src/Virtufin.<Service>.Protos/proto/<service>.protofiles buf.gen.yamlin each protos repobuf.yaml+buf.lockfor proto dependency pinning