Design: from-scratch protoc plugin for chattolib Connect stubs

Spec for replacing protoc-gen-connect-python with a purpose-built Python
protoc plugin (protoc-gen-chattolib). Emits async + thin sync-wrapping
client stubs; drops connectrpc from the build entirely. See
docs/superpowers/specs/2026-08-30-chattolib-protoc-plugin-design.md

docs/superpowers/specs/2026-08-30-chattolib-protoc-plugin-design.md

@@ -0,0 +1,208 @@
+# Design: a from-scratch `protoc` plugin to generate chattolib's Connect stubs
+
+**Status:** proposed
+**Date:** 2026-08-30
+**Bean:** `chattolib-zo8b`
+**Supersedes:** the `regen_connect.py` regex post-processor (to be deleted)
+
+## 1. Purpose
+
+Remove the last trace of the `connectrpc` package from chattolib's build.
+Today the *runtime* is hand-rolled (`chattolib/_connect.py`) but codegen still
+shells out to `protoc-gen-connect-python` to *produce* the `*_connect.py`
+stubs, then a regex script strips them down. This design replaces both with a
+single, purpose-built `protoc` plugin that emits exactly the stubs chattolib
+needs — an async client **and** a thin sync wrapper — with no `connectrpc`
+anywhere in the loop.
+
+## 2. Goals / non-goals
+
+**Goals**
+- One codegen path: `protoc --chattolib_out` emits the service stubs.
+- Stubs import only from `chattolib._connect` and the sibling `*_pb2` module.
+- Generated **async** method names are byte-identical to today's
+  (`client.join_room(...)`, `client.post_message(...)`) so `client.py` and the
+  `Bot` facade are untouched.
+- A **sync** client is available, implemented as a wrapper over the async one.
+- `connectrpc` and `pyqwest` are absent from both the runtime *and* the build.
+
+**Non-goals**
+- No server-side / ASGI / WSGI code generation (chattolib is a client).
+- No streaming RPC support (Chatto's API is unary-only today). If a streaming
+  RPC ever appears, that is a separate design.
+- No re-implementation of `.proto` parsing — `protoc` still parses and
+  validates; the plugin only reads the resulting descriptors.
+
+## 3. Components
+
+### 3.1 `chattolib/_connect.py` (exists; small additions)
+The hand-rolled runtime, unchanged in behavior:
+- `Code` (16-value enum), `ConnectError`, `MethodInfo`, `_BinaryCodec`,
+  `google_protobuf_binary_codec()`.
+- `class ConnectClient` — `__init__(address, *, codec=None, timeout_ms=None)`,
+  `async execute_unary(*, request, method, headers=None)`, `async close()`.
+
+**Addition:** `class ConnectClientSync` — the sync wrapper (see §4).
+
+### 3.2 The plugin: `chattolib/_protoc_plugin.py` (new)
+A Python script that speaks the `protoc` plugin protocol:
+- Reads a `CodeGeneratorRequest` (protobuf) from **stdin**.
+- For each `FileDescriptorProto` that contains a `service`, emits one output
+  file: `<proto path>_connect.py`.
+- Writes a `CodeGeneratorResponse` (protobuf) to **stdout**.
+
+It is installed as a `console_scripts` entry point named
+**`protoc-gen-chattolib`**, so `protoc` discovers it on `PATH` and the
+invocation is `--chattolib_out=<dir>`.
+
+### 3.3 Vendored codegen descriptors (new)
+`src/chattolib/_pb/google/protobuf/protoc_gen_request_pb2.py` and
+`protoc_gen_response_pb2.py` — the two stable `protoc` descriptor stubs the
+plugin needs to parse its stdin/stdout. Generated once from
+`google/protobuf/*.proto`, committed, and not regenerated per-release.
+
+### 3.4 `scripts/generate_pb.sh` (modified)
+- Still fetches `.proto` from the pinned `CHATTO_REF` tag (unchanged).
+- Still runs `protoc --python_out` for the `*_pb2.py` message files (unchanged).
+- **Replaces** `--connect-python_out=...` with `--chattolib_out=...`.
+- Drops the `protoc-gen-connect-python` PATH check / `[realtime]` pin for it.
+
+### 3.5 Deleted
+- `scripts/regen_connect.py` — the regex post-processor is obsolete.
+
+## 4. The sync client (the design decision)
+
+`ConnectClientSync` wraps the async client; it does **not** re-implement the
+protocol.
+
+```python
+class ConnectClientSync:
+    def __init__(self, address, *, codec=None, timeout_ms=None):
+        self._async = ConnectClient(address, codec=codec, timeout_ms=timeout_ms)
+        self._loop = asyncio.new_event_loop()
+
+    def execute_unary(self, *, request, method, headers=None):
+        self._guard_no_running_loop()
+        return self._loop.run_until_complete(
+            self._async.execute_unary(request=request, method=method, headers=headers)
+        )
+
+    def close(self):
+        self._loop.run_until_complete(self._async.close())
+        self._loop.close()
+
+    def _guard_no_running_loop(self):
+        try:
+            asyncio.get_running_loop()
+        except RuntimeError:
+            return  # no running loop: safe to run_until_complete
+        raise ChattoError(
+            "the sync client cannot be used from within a running event loop; "
+            "use the async ConnectClient in async code"
+        )
+```
+
+Properties:
+- **Shared protocol** — codec, `MethodInfo`, `ConnectError`, and the
+  error-envelope parser live only in the async client; the sync path is
+  delegation. One implementation, two entry points.
+- **Zero new dependencies** — `httpx` is already required by the async client.
+- **Loud, not opaque** — calling it from inside `async def` raises a clear
+  `ChattoError` instead of a nested-loop `RuntimeError` (decision B).
+- Each sync client owns a private loop; `close()` closes both.
+
+## 5. Generated stub shape
+
+For a service `chatto.api.v1.RoomService` in `rooms.proto`, the plugin emits
+`src/chattolib/_pb/chatto/api/v1/rooms_connect.py`:
+
+```python
+# Generated by chattolib's protoc plugin (protoc-gen-chattolib). DO NOT EDIT.
+# source: chatto/api/v1/rooms.proto
+import chatto.api.v1.rooms_pb2 as _pb2
+from chattolib._connect import ConnectClient, ConnectClientSync, MethodInfo
+
+class RoomServiceClient(ConnectClient):
+    async def join_room(self, request: _pb2.JoinRoomRequest, *,
+                        headers: dict[str, str] | None = None) -> _pb2.JoinRoomResponse:
+        return await self.execute_unary(
+            request=request,
+            method=MethodInfo(name="JoinRoom",
+                               service_name="chatto.api.v1.RoomService",
+                               input=_pb2.JoinRoomRequest,
+                               output=_pb2.JoinRoomResponse),
+            headers=headers,
+        )
+    # ... one method per RPC ...
+
+class RoomServiceClientSync(ConnectClientSync):
+    def join_room(self, request: _pb2.JoinRoomRequest, *,
+                  headers: dict[str, str] | None = None) -> _pb2.JoinRoomResponse:
+        return self.execute_unary(
+            request=request,
+            method=MethodInfo(name="JoinRoom",
+                               service_name="chatto.api.v1.RoomService",
+                               input=_pb2.JoinRoomRequest,
+                               output=_pb2.JoinRoomResponse),
+            headers=headers,
+        )
+```
+
+Notes:
+- `idempotency_level` is omitted from the emitted `MethodInfo` (the runtime
+  defaults it; chattolib never reads it).
+- `use_get` is **not** generated (POST-only; decision from Q3).
+- Both the async and sync classes are emitted so a sync consumer gets the same
+  method names. A service with N RPCs therefore yields 2N generated methods.
+  The sync classes are **additive**: `build_service_clients` still instantiates
+  only the async `*ServiceClient`, and no existing code imports `*ClientSync`,
+  so `client.py`, `bot.py`, and `_transport.py` require no changes.
+- The `*_pb2` import uses the proto's own package path (as today), resolved via
+  the existing `sys.path` shim in `_pb/__init__.py`.
+
+## 6. Data flow
+
+```
+.proto (pinned tag)
+   │  protoc --proto_path=proto
+   ├──► --python_out      →  *_pb2.py            (message classes; unchanged)
+   └──► --chattolib_out   →  protoc-gen-chattolib (our plugin)
+                              reads CodeGeneratorRequest (stdin)
+                              emits *_connect.py   (async + sync clients)
+                              via CodeGeneratorResponse (stdout)
+
+chattolib._transport.build_service_clients(base_url)
+   instantiates the 23 *ServiceClient (async) as today.
+```
+
+## 7. Error handling & testing
+
+- **Unit:** a `CodeGeneratorRequest` fixture (a couple of representative
+  services, incl. one with many methods and one with a `oneof`-free plain
+  service) → run the plugin → assert the emitted source imports only from
+  `chattolib._connect`/`_pb2`, defines both client classes, and has one method
+  per RPC.
+- **Sync-wrapper:** a test that (a) a sync call round-trips against a mocked
+  transport, and (b) calling it from inside a running loop raises the clear
+  `ChattoError`.
+- **Integration (existing):** the full `tests/` suite stays green, and the live
+  smoke test against the preview server still passes with `connectrpc`
+  uninstalled.
+- **Golden-file:** the 24 regenerated `*_connect.py` are committed; a CI check
+  re-runs the plugin and fails on any diff (catches plugin regressions).
+
+## 8. Risks & mitigations
+
+| Risk | Mitigation |
+|---|---|
+| Plugin emits subtly-wrong stubs on a future proto shape | Golden-file CI diff + unit fixtures covering the method shapes we use. |
+| `run_until_complete` surprises (e.g. signal handlers, `uvloop`) | The guard covers the common case; documented that the sync client is for sync contexts. |
+| Vendored `protoc_gen_*_pb2` drift from a `protoc` version | They are stable; pin and regenerate only if `protoc` changes the codegen schema. |
+| Two clients (async+sync) double the stub size | Acceptable; it's generated code, and it's what enables the sync use-case. |
+
+## 9. Out of scope / follow-ons
+
+- A from-scratch replacement for the `--python_out` message generation (we keep
+  protoc's built-in Python generator — it's not `connectrpc`-specific).
+- Streaming RPC support.
+- Publishing the plugin as a standalone package (it ships inside chattolib).