#1 [medium] FileResponse with stat_result=None re-stats at send-time and raises a
RuntimeError (-> 500 + stacktrace, escaping get_media's try/except) if the file was
swept between the early exists() check and the send. The old streaming code caught
FileNotFoundError on getsize -> 404. Now prepare_file_response takes ONE
authoritative os.stat (in try/except FileNotFoundError -> 404) after the temp_*
touch and passes it as FileResponse(stat_result=...): restores the 404 semantics,
collapses the double stat into one, and makes the ETag reflect the observed mtime.
The remaining narrow deleted-between-our-stat-and-open window pre-existed.
#2/#3 [doc] Corrected the Content-Disposition comment (FileResponse emits
filename="x" for ASCII, filename*=UTF-8''x only for non-ASCII, via setdefault — a
manual header would override, not double) and the ETag-stability comment (stable
within a 300s window; a longer view costs at most one safe 200 If-Range restart
per interval).
#4 [simplification] Removed the now-dead StreamingResponse import and corrected
get_media's return annotation to -> Response (nothing returns StreamingResponse
after the FileResponse migration).
#5 [test] Added a test for the middleware's non-http (lifespan/websocket) branch —
it only runs in a real deploy, never through TestClient — asserting it delegates
to the inner app and logs nothing.
214 passed (195 baseline + 19).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the hand-rolled media streaming with Starlette FileResponse, drops the
BaseHTTPMiddleware, and enlarges the default threadpool.
3.1 prepare_file_response now returns FileResponse (handles Range/If-Range/206/
416/multipart, sets Accept-Ranges/ETag/Last-Modified, reads efficiently — no
per-64KB to_thread hop that starved the pool). Kept: the early 404 pre-check,
the MIME logic (python-magic + SQLite cache), and every stage-2 behavior — the
temp_* mtime touch (now DEBOUNCED, see below), delete_after -> BackgroundTask
(passed as FileResponse background=), the media_key MIME cache. Removed the
manual Range parsing, file_chunk_generator, and hand-built headers;
Content-Disposition is formed by FileResponse from filename= (no double-set).
206 slices are byte-identical to the old code; accepted RFC-7233 deltas
documented in the tests.
3.2 RequestLoggingMiddleware rewritten as a pure-ASGI class (wraps only send to
observe the status line, never buffers the body, passes non-http scopes
through) — the streaming body flows untouched.
3.3 lifespan sets a larger default executor (ThreadPoolExecutor, IO_THREAD_POOL_SIZE
default 32) and shuts it down on exit.
Review round-1 fixes folded in: the temp_* mtime touch is DEBOUNCED
(TEMP_MTIME_REFRESH_INTERVAL=300s) so FileResponse's mtime-derived ETag stays
stable across a resume/seek session (an every-serve touch broke If-Range resume);
starlette pinned to 0.45.3; the io executor is shut down on lifespan exit; the
ASGI logger includes the query string.
Tests (tests/test_stage3_fileresponse.py, 18): the Range matrix vs FileResponse
with every delta documented; temp_* mtime refreshed when stale AND stable when
fresh (ETag identical); delete_after background runs and removes the file;
media_key MIME cache hit/miss; the ASGI middleware passes the body and logs.
213 passed (195 baseline + 18).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
F1 [WARNING] The HTTP_DOWNLOAD_SEMAPHORE admission-control balance was untested,
and it is a plain asyncio.Semaphore — so an over-release would SILENTLY inflate
the permit count and disable the limiter (no ValueError), a green suite hiding the
exact bug stage 2 fixes. Added two tests (mirroring the stage-1 tg_throttle
balance test): the permit is released exactly once when the download errors (count
back to baseline), and a timed-out acquire (503) releases NOTHING (count
unchanged). Adversarially validated: neutralizing the finally release makes the
first test fail, so it genuinely catches a leak.
F2 [low] Corrected the _download_deduped docstring: the detached task sets the
Future in try/except, and its finally ALWAYS pops the key — the previous wording
attributed the Future-set to the finally, which an auditor reading the finally
block would find contradicted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes flaky static media serving: partial files served as 'ready', truncated
stubs living for an hour, and app-wide hangs under a saturated download path.
2.1 _download_atomic(file_id, final_path, timeout): downloads to a unique
{final}.part.{hex}, validates size>0, and publishes solely via os.rename
(atomic on POSIX); a finally ALWAYS removes the partial (timeout/cancel/
zero-size/race-loser). Every downloader call routes through it, so a file at a
FINAL name ({fid} / temp_{fid}) is GUARANTEED complete (grep-proven: the only
safe_download_media caller passes a .part. path). Big-video timeout scales with
size (min/max/min-speed knobs, documented in dockercompose.yml). The sweeper
regex now matches both .part. and legacy .tmp. stubs.
2.2 In-flight dedup registry: the first request for a key runs the download in a
DETACHED task sharing a Future; the task's finally sets the Future AND pops the
key. Waiters await asyncio.shield(fut) so a client disconnect / waiter timeout
cancels only the waiter, never the download — no hung waiters, no stuck key, a
failed download frees the key for retry.
2.3 FloodWait->429: handler before except RPCError (FloodWait subclasses it),
Retry-After = min(value + rand(1,30), 300); propagates from the detached task
through the Future.
2.4 Serving a temp_* file touches its mtime so the 1h sweeper can't delete a
video out from under a viewer.
2.5 The HTTP download semaphore acquire is bounded (wait_for 30 -> 503 +
Retry-After); the permit is released only if the acquire succeeded. The
request-scoped-permit trade-off (a disconnect can transiently exceed the
download count) is documented inline for stage-7 prod observation.
Tests (tests/test_stage2_static.py, 13): atomic publish/clean on every exit incl.
FloodWait-through-the-finally; concurrent big-video serves no partial; dedup runs
one download, a cancelled waiter doesn't hang others, a failed download frees the
key; FloodWait -> 429; mtime touch; sweeper cleans .part./.tmp./stale temp_ but
keeps fresh files. 193 passed (180 baseline + 13).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
F1 [WARNING] pytest-asyncio was not declared, so the 6 new async stage-1 tests
ERROR on a clean checkout (async def not natively supported) — zero regression
protection, masked by a globally-installed plugin. Added pytest-asyncio to
requirements.txt + asyncio_mode=auto to tests/pytest.ini. Verified on a fresh venv
from requirements alone: the tests collect and run (180 passed).
F2 [WARNING] The /raw_json endpoint's get_messages was the one remaining live RPC
without a timeout, violating the stage-1 DoD ('every Telegram RPC is bounded').
Wrapped it in asyncio.wait_for(..., 30) mirroring PostParser.get_post. (It is not
under the tg_rpc gate, so its blast radius was one request, not the app.)
F3 [WARNING] The worker test globally no-op'd asyncio.sleep, so the dedicated
except FloodWait branch was indistinguishable from the generic handler — deleting
it kept the test green. The sleep stub now records delays and the test asserts the
FloodWait backoff of 6 (=min(1+5,900)), distinct from the success path's 2.
F5 [low] The tricky 'gate outside, timeout inside' nesting was open-coded at 3
sites (each re-deriving the invariant). Extracted tg_rpc_bounded(timeout) into
tg_throttle (using asyncio.timeout()); the 3 sites now use it, so a future call
site cannot silently wrap the gate entry and reopen the hang-under-backpressure.
F4 [low] Documented TG_RPC_TIMEOUT in the dockercompose.yml environment block
next to the other TG_RPC_* knobs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Eliminates the app-wide hangs where one stuck Telegram RPC holds the single
tg_rpc gate permit forever (freezing every RSS/HTML request) and where the
background download worker dies silently.
1.1 Every RPC under the global gate is now bounded by asyncio.wait_for
(config tg_rpc_timeout, env TG_RPC_TIMEOUT, default 60). The wait_for wraps
ONLY the RPC body, never the gate acquire, so queue backpressure stays
legitimate; the paginated get_chat_history is collected in an inner coroutine
so the async-for can be bounded. A timeout propagates out of the gate so its
permit is released (no leak).
1.2 The other live RPC paths get the same treatment: _reply_enrichment's
get_messages now runs under the gate + timeout; PostParser.get_post is bounded
at 30s (and a stray print removed); /health's get_me at 10s.
1.3 background_download_worker: queue.get() moved out of the try so task_done()
in finally balances exactly one get (no ValueError that killed the worker);
FloodWait is caught BEFORE the generic Exception (sleeps, does not flood);
download_new_files uses put_nowait so a full queue no longer blocks the
sweeper forever.
1.4 Both background tasks run under a _supervised() wrapper: a crash or an
unexpected return is logged CRITICAL and restarted (restarts rate-limited to
once per 60s so a hard-failing task can't spin), while CancelledError is
propagated to the child for a clean shutdown.
Tests (tests/test_stage1_hangs.py): gate timeout releases the permit + a second
call succeeds; gate cancel mid-spacing releases the permit; the worker survives
Exception and FloodWait with balanced task_done; and _supervised restarts a
crashing task (rate-limited) and an unexpected return, and propagates cancellation
to its child. 180 passed (174 baseline + 6).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce `tg_watchdog_heartbeat_every` to emit periodic INFO heartbeats.
Add cumulative diagnostics counters and richer logging for watchdog probes,
restart reasons, and disconnect‑flap triggers. Update defaults and tests
accordingly.
Introduce an active watchdog that probes the Telegram client to detect
zombie sessions and restart them in‑process. Add configurable disconnect
flap detection with a sliding window to trigger restarts after repeated
disconnects. New environment variables and config entries are added, and
the Kurigram dependency is now version‑pinned.
Replace MagicMock based text/caption mocks with a lightweight StrWithHtml
subclass that provides the required `.html` property while behaving like a
regular string. This fixes length and strip checks in the parser and
simplifies test setup. Also adds default proxy settings to mock_config for
test completeness.
claude code agent
vvzvlad