fix(stability): stage 1 — timeouts on all TG RPC + resilient download worker + supervision

Eliminates the app-wide hangs where one stuck Telegram RPC holds the single
tg_rpc gate permit forever (freezing every RSS/HTML request) and where the
background download worker dies silently.

1.1 Every RPC under the global gate is now bounded by asyncio.wait_for
    (config tg_rpc_timeout, env TG_RPC_TIMEOUT, default 60). The wait_for wraps
    ONLY the RPC body, never the gate acquire, so queue backpressure stays
    legitimate; the paginated get_chat_history is collected in an inner coroutine
    so the async-for can be bounded. A timeout propagates out of the gate so its
    permit is released (no leak).
1.2 The other live RPC paths get the same treatment: _reply_enrichment's
    get_messages now runs under the gate + timeout; PostParser.get_post is bounded
    at 30s (and a stray print removed); /health's get_me at 10s.
1.3 background_download_worker: queue.get() moved out of the try so task_done()
    in finally balances exactly one get (no ValueError that killed the worker);
    FloodWait is caught BEFORE the generic Exception (sleeps, does not flood);
    download_new_files uses put_nowait so a full queue no longer blocks the
    sweeper forever.
1.4 Both background tasks run under a _supervised() wrapper: a crash or an
    unexpected return is logged CRITICAL and restarted (restarts rate-limited to
    once per 60s so a hard-failing task can't spin), while CancelledError is
    propagated to the child for a clean shutdown.

Tests (tests/test_stage1_hangs.py): gate timeout releases the permit + a second
call succeeds; gate cancel mid-spacing releases the permit; the worker survives
Exception and FloodWait with balanced task_done; and _supervised restarts a
crashing task (rate-limited) and an unexpected return, and propagates cancellation
to its child. 180 passed (174 baseline + 6).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
claude code agent
2026-07-05 06:51:08 +03:00
parent e3b458d774
commit 4daf611f05
7 changed files with 305 additions and 23 deletions
+14 -4
View File
@@ -22,9 +22,12 @@ from typing import Any, Optional, Union, List
from pyrogram import Client
from pyrogram.types import Chat, Message
from tg_throttle import tg_rpc
from config import get_settings
logger = logging.getLogger(__name__)
Config = get_settings()
# Path to cache directory
CACHE_DIR = os.path.join('data', 'tgcache')
@@ -138,10 +141,16 @@ async def cached_get_chat_history(client: Client, channel_id: Union[str, int], l
try:
logger.info(f"history_cache_request: fetching fresh history for channel {channel_id}, limit {limit}")
messages = []
# Hold the global RPC gate only for the live fetch. The paginated `async for`
# cannot be wrapped in wait_for directly, so collect it in an inner coroutine
# and bound THAT with the timeout. wait_for wraps only the RPC body, never the
# gate entry (`async with tg_rpc()`), so legitimate queue backpressure is not
# mistaken for a hang.
async with tg_rpc():
async for message in client.get_chat_history(channel_id, limit=limit):
messages.append(message)
async def _collect():
# Full paginated history; bounded by the outer wait_for.
return [m async for m in client.get_chat_history(channel_id, limit=limit)]
messages = await asyncio.wait_for(_collect(), timeout=Config["tg_rpc_timeout"])
await asyncio.to_thread(_save_history_to_cache, channel_id, messages, limit)
return messages
@@ -214,7 +223,8 @@ async def cached_get_chat(client: Client, channel_id: Union[str, int]) -> Simple
logger.info(f"chatinfo_cache_request: fetching fresh chat info for channel {channel_id}")
async with tg_rpc():
chat = await client.get_chat(channel_id)
# Bound the RPC itself, not the gate entry above it.
chat = await asyncio.wait_for(client.get_chat(channel_id), timeout=Config["tg_rpc_timeout"])
data = {
'id': getattr(chat, 'id', None),