Design: a from-scratch `protoc` plugin to generate chattolib's Connect stubs

Status: proposed Date: 2026-08-30 Bean: chattolib-zo8b Supersedes: the regen_connect.py regex post-processor (to be deleted)

1. Purpose

Remove the last trace of the connectrpc package from chattolib's build. Today the runtime is hand-rolled (chattolib/_connect.py) but codegen still shells out to protoc-gen-connect-python to produce the *_connect.py stubs, then a regex script strips them down. This design replaces both with a single, purpose-built protoc plugin that emits exactly the stubs chattolib needs — an async client and a thin sync wrapper — with no connectrpc anywhere in the loop.

2. Goals / non-goals

Goals

One codegen path: protoc --chattolib_out emits the service stubs.
Stubs import only from chattolib._connect and the sibling *_pb2 module.
Generated async method names are byte-identical to today's (client.join_room(...), client.post_message(...)) so client.py and the Bot facade are untouched.
A sync client is available, implemented as a wrapper over the async one.
connectrpc and pyqwest are absent from both the runtime and the build.

Non-goals

No server-side / ASGI / WSGI code generation (chattolib is a client).
No streaming RPC support (Chatto's API is unary-only today). If a streaming RPC ever appears, that is a separate design.
No re-implementation of .proto parsing — protoc still parses and validates; the plugin only reads the resulting descriptors.

3. Components

3.1 `chattolib/_connect.py` (exists; small additions)

The hand-rolled runtime, unchanged in behavior:

Code (16-value enum), ConnectError, MethodInfo, _BinaryCodec, google_protobuf_binary_codec().
class ConnectClient — __init__(address, *, codec=None, timeout_ms=None), async execute_unary(*, request, method, headers=None), async close().

Addition: class ConnectClientSync — the sync wrapper (see §4).

3.2 The plugin: `chattolib/_protoc_plugin.py` (new)

A Python script that speaks the protoc plugin protocol:

Reads a CodeGeneratorRequest (protobuf) from stdin.
For each FileDescriptorProto that contains a service, emits one output file: <proto path>_connect.py.
Writes a CodeGeneratorResponse (protobuf) to stdout.

It is installed as a console_scripts entry point named protoc-gen-chattolib, so protoc discovers it on PATH and the invocation is --chattolib_out=<dir>.

3.3 Vendored codegen descriptors (new)

src/chattolib/_pb/google/protobuf/protoc_gen_request_pb2.py and protoc_gen_response_pb2.py — the two stable protoc descriptor stubs the plugin needs to parse its stdin/stdout. Generated once from google/protobuf/*.proto, committed, and not regenerated per-release.

3.4 `scripts/generate_pb.sh` (modified)

Still fetches .proto from the pinned CHATTO_REF tag (unchanged).
Still runs protoc --python_out for the *_pb2.py message files (unchanged).
Replaces --connect-python_out=... with --chattolib_out=....
Drops the protoc-gen-connect-python PATH check / [realtime] pin for it.

3.5 Deleted

scripts/regen_connect.py — the regex post-processor is obsolete.

4. The sync client (the design decision)

ConnectClientSync wraps the async client; it does not re-implement the protocol.

class ConnectClientSync:
    def __init__(self, address, *, codec=None, timeout_ms=None):
        self._async = ConnectClient(address, codec=codec, timeout_ms=timeout_ms)
        self._loop = asyncio.new_event_loop()

    def execute_unary(self, *, request, method, headers=None):
        self._guard_no_running_loop()
        return self._loop.run_until_complete(
            self._async.execute_unary(request=request, method=method, headers=headers)
        )

    def close(self):
        self._loop.run_until_complete(self._async.close())
        self._loop.close()

    def _guard_no_running_loop(self):
        try:
            asyncio.get_running_loop()
        except RuntimeError:
            return  # no running loop: safe to run_until_complete
        raise ChattoError(
            "the sync client cannot be used from within a running event loop; "
            "use the async ConnectClient in async code"
        )

Properties:

Shared protocol — codec, MethodInfo, ConnectError, and the error-envelope parser live only in the async client; the sync path is delegation. One implementation, two entry points.
Zero new dependencies — httpx is already required by the async client.
Loud, not opaque — calling it from inside async def raises a clear ChattoError instead of a nested-loop RuntimeError (decision B).
Each sync client owns a private loop; close() closes both.

5. Generated stub shape

For a service chatto.api.v1.RoomService in rooms.proto, the plugin emits src/chattolib/_pb/chatto/api/v1/rooms_connect.py:

# Generated by chattolib's protoc plugin (protoc-gen-chattolib). DO NOT EDIT.
# source: chatto/api/v1/rooms.proto
import chatto.api.v1.rooms_pb2 as _pb2
from chattolib._connect import ConnectClient, ConnectClientSync, MethodInfo

class RoomServiceClient(ConnectClient):
    async def join_room(self, request: _pb2.JoinRoomRequest, *,
                        headers: dict[str, str] | None = None) -> _pb2.JoinRoomResponse:
        return await self.execute_unary(
            request=request,
            method=MethodInfo(name="JoinRoom",
                               service_name="chatto.api.v1.RoomService",
                               input=_pb2.JoinRoomRequest,
                               output=_pb2.JoinRoomResponse),
            headers=headers,
        )
    # ... one method per RPC ...

class RoomServiceClientSync(ConnectClientSync):
    def join_room(self, request: _pb2.JoinRoomRequest, *,
                  headers: dict[str, str] | None = None) -> _pb2.JoinRoomResponse:
        return self.execute_unary(
            request=request,
            method=MethodInfo(name="JoinRoom",
                               service_name="chatto.api.v1.RoomService",
                               input=_pb2.JoinRoomRequest,
                               output=_pb2.JoinRoomResponse),
            headers=headers,
        )

Notes:

idempotency_level is omitted from the emitted MethodInfo (the runtime defaults it; chattolib never reads it).
use_get is not generated (POST-only; decision from Q3).
Both the async and sync classes are emitted so a sync consumer gets the same method names. A service with N RPCs therefore yields 2N generated methods. The sync classes are additive: build_service_clients still instantiates only the async *ServiceClient, and no existing code imports *ClientSync, so client.py, bot.py, and _transport.py require no changes.
The *_pb2 import uses the proto's own package path (as today), resolved via the existing sys.path shim in _pb/__init__.py.

6. Data flow

.proto (pinned tag)
   │  protoc --proto_path=proto
   ├──► --python_out      →  *_pb2.py            (message classes; unchanged)
   └──► --chattolib_out   →  protoc-gen-chattolib (our plugin)
                              reads CodeGeneratorRequest (stdin)
                              emits *_connect.py   (async + sync clients)
                              via CodeGeneratorResponse (stdout)

chattolib._transport.build_service_clients(base_url)
   instantiates the 23 *ServiceClient (async) as today.

7. Error handling & testing

Unit: a CodeGeneratorRequest fixture (a couple of representative services, incl. one with many methods and one with a oneof-free plain service) → run the plugin → assert the emitted source imports only from chattolib._connect/_pb2, defines both client classes, and has one method per RPC.
Sync-wrapper: a test that (a) a sync call round-trips against a mocked transport, and (b) calling it from inside a running loop raises the clear ChattoError.
Integration (existing): the full tests/ suite stays green, and the live smoke test against the preview server still passes with connectrpc uninstalled.
Golden-file: the 24 regenerated *_connect.py are committed; a CI check re-runs the plugin and fails on any diff (catches plugin regressions).

8. Risks & mitigations

Risk	Mitigation
Plugin emits subtly-wrong stubs on a future proto shape	Golden-file CI diff + unit fixtures covering the method shapes we use.
`run_until_complete` surprises (e.g. signal handlers, `uvloop`)	The guard covers the common case; documented that the sync client is for sync contexts.
Vendored `protoc_gen_*_pb2` drift from a `protoc` version	They are stable; pin and regenerate only if `protoc` changes the codegen schema.
Two clients (async+sync) double the stub size	Acceptable; it's generated code, and it's what enables the sync use-case.

9. Out of scope / follow-ons

A from-scratch replacement for the --python_out message generation (we keep protoc's built-in Python generator — it's not connectrpc-specific).
Streaming RPC support.
Publishing the plugin as a standalone package (it ships inside chattolib).

2026-08-30-chattolib-protoc-plugin-design.md 9.2 KB Verlauf Originalformat

Design: a from-scratch protoc plugin to generate chattolib's Connect stubs