fix(stability): stage 1 — timeouts on all TG RPC + resilient download worker + supervision
Eliminates the app-wide hangs where one stuck Telegram RPC holds the single
tg_rpc gate permit forever (freezing every RSS/HTML request) and where the
background download worker dies silently.
1.1 Every RPC under the global gate is now bounded by asyncio.wait_for
(config tg_rpc_timeout, env TG_RPC_TIMEOUT, default 60). The wait_for wraps
ONLY the RPC body, never the gate acquire, so queue backpressure stays
legitimate; the paginated get_chat_history is collected in an inner coroutine
so the async-for can be bounded. A timeout propagates out of the gate so its
permit is released (no leak).
1.2 The other live RPC paths get the same treatment: _reply_enrichment's
get_messages now runs under the gate + timeout; PostParser.get_post is bounded
at 30s (and a stray print removed); /health's get_me at 10s.
1.3 background_download_worker: queue.get() moved out of the try so task_done()
in finally balances exactly one get (no ValueError that killed the worker);
FloodWait is caught BEFORE the generic Exception (sleeps, does not flood);
download_new_files uses put_nowait so a full queue no longer blocks the
sweeper forever.
1.4 Both background tasks run under a _supervised() wrapper: a crash or an
unexpected return is logged CRITICAL and restarted (restarts rate-limited to
once per 60s so a hard-failing task can't spin), while CancelledError is
propagated to the child for a clean shutdown.
Tests (tests/test_stage1_hangs.py): gate timeout releases the permit + a second
call succeeds; gate cancel mid-spacing releases the permit; the worker survives
Exception and FloodWait with balanced task_done; and _supervised restarts a
crashing task (rate-limited) and an unexpected return, and propagates cancellation
to its child. 180 passed (174 baseline + 6).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
+14
-4
@@ -22,9 +22,12 @@ from typing import Any, Optional, Union, List
|
||||
from pyrogram import Client
|
||||
from pyrogram.types import Chat, Message
|
||||
from tg_throttle import tg_rpc
|
||||
from config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
Config = get_settings()
|
||||
|
||||
# Path to cache directory
|
||||
CACHE_DIR = os.path.join('data', 'tgcache')
|
||||
|
||||
@@ -138,10 +141,16 @@ async def cached_get_chat_history(client: Client, channel_id: Union[str, int], l
|
||||
|
||||
try:
|
||||
logger.info(f"history_cache_request: fetching fresh history for channel {channel_id}, limit {limit}")
|
||||
messages = []
|
||||
# Hold the global RPC gate only for the live fetch. The paginated `async for`
|
||||
# cannot be wrapped in wait_for directly, so collect it in an inner coroutine
|
||||
# and bound THAT with the timeout. wait_for wraps only the RPC body, never the
|
||||
# gate entry (`async with tg_rpc()`), so legitimate queue backpressure is not
|
||||
# mistaken for a hang.
|
||||
async with tg_rpc():
|
||||
async for message in client.get_chat_history(channel_id, limit=limit):
|
||||
messages.append(message)
|
||||
async def _collect():
|
||||
# Full paginated history; bounded by the outer wait_for.
|
||||
return [m async for m in client.get_chat_history(channel_id, limit=limit)]
|
||||
messages = await asyncio.wait_for(_collect(), timeout=Config["tg_rpc_timeout"])
|
||||
await asyncio.to_thread(_save_history_to_cache, channel_id, messages, limit)
|
||||
|
||||
return messages
|
||||
@@ -214,7 +223,8 @@ async def cached_get_chat(client: Client, channel_id: Union[str, int]) -> Simple
|
||||
|
||||
logger.info(f"chatinfo_cache_request: fetching fresh chat info for channel {channel_id}")
|
||||
async with tg_rpc():
|
||||
chat = await client.get_chat(channel_id)
|
||||
# Bound the RPC itself, not the gate entry above it.
|
||||
chat = await asyncio.wait_for(client.get_chat(channel_id), timeout=Config["tg_rpc_timeout"])
|
||||
|
||||
data = {
|
||||
'id': getattr(chat, 'id', None),
|
||||
|
||||
Reference in New Issue
Block a user