fix(stability): stage 1 — timeouts on all TG RPC + resilient download worker + supervision

Eliminates the app-wide hangs where one stuck Telegram RPC holds the single
tg_rpc gate permit forever (freezing every RSS/HTML request) and where the
background download worker dies silently.

1.1 Every RPC under the global gate is now bounded by asyncio.wait_for
    (config tg_rpc_timeout, env TG_RPC_TIMEOUT, default 60). The wait_for wraps
    ONLY the RPC body, never the gate acquire, so queue backpressure stays
    legitimate; the paginated get_chat_history is collected in an inner coroutine
    so the async-for can be bounded. A timeout propagates out of the gate so its
    permit is released (no leak).
1.2 The other live RPC paths get the same treatment: _reply_enrichment's
    get_messages now runs under the gate + timeout; PostParser.get_post is bounded
    at 30s (and a stray print removed); /health's get_me at 10s.
1.3 background_download_worker: queue.get() moved out of the try so task_done()
    in finally balances exactly one get (no ValueError that killed the worker);
    FloodWait is caught BEFORE the generic Exception (sleeps, does not flood);
    download_new_files uses put_nowait so a full queue no longer blocks the
    sweeper forever.
1.4 Both background tasks run under a _supervised() wrapper: a crash or an
    unexpected return is logged CRITICAL and restarted (restarts rate-limited to
    once per 60s so a hard-failing task can't spin), while CancelledError is
    propagated to the child for a clean shutdown.

Tests (tests/test_stage1_hangs.py): gate timeout releases the permit + a second
call succeeds; gate cancel mid-spacing releases the permit; the worker survives
Exception and FloodWait with balanced task_done; and _supervised restarts a
crashing task (rate-limited) and an unexpected return, and propagates cancellation
to its child. 180 passed (174 baseline + 6).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
claude code agent
2026-07-05 06:51:08 +03:00
parent e3b458d774
commit 4daf611f05
7 changed files with 305 additions and 23 deletions
+5 -2
View File
@@ -92,10 +92,13 @@ class PostParser:
post_id: int,
output_type: str = 'json',
debug: bool = False) -> Union[str, Dict[Any, Any], None]:
print(f"Getting post {channel}, {post_id}")
try:
prepared_channel_id: Union[str, int] = self.channel_name_prepare(channel)
message = await self.client.get_messages(prepared_channel_id, post_id)
# Bound the single-post fetch so a hung RPC cannot block the request forever.
message = await asyncio.wait_for(
self.client.get_messages(prepared_channel_id, post_id),
timeout=30,
)
if Config["debug"]: print(message)