Media downloads jammed process-wide: Kurigram serializes them through a single
get_file slot (max_concurrent_transmissions=1), and a zombie media-DC connection
(upload.GetFile timing out) held that slot forever. The main-DC watchdog (get_me)
never noticed, and the 60s cache sweep kept re-queuing the failing file, so all
feed images turned into broken placeholders.
- telegram_client: count consecutive download timeouts; after N (default 5) reuse
_restart_client() to rebuild the media connection — the only recovery signal the
main-DC watchdog cannot provide. Any success resets the streak.
- telegram_client/config: set max_concurrent_transmissions (default 3) so one hung
download is no longer an instant total outage (blast-radius limiter only).
- api_server: negative-cache with exponential backoff for repeatedly-failing files
(skip in background sweep, fast 503+Retry-After in get_media, record/clear in the
dedup runner and background worker; FloodWait excluded).
- dockercompose/tests: document TG_MAX_CONCURRENT_TRANSMISSIONS and
MEDIA_TIMEOUT_RESTART_THRESHOLD; update mock config; add self-heal/backoff tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The container healthcheck hit /rss/...?limit=1 (5s timeout): on a cold cache RSS
generation exceeds 5s, or a hung TG RPC makes it hang, so docker/autoheal restarts
the container mid-download and corrupts temp files. Replace it with /ping, which
reflects process/loop liveness (answers instantly, always) plus TG liveness read
from the watchdog's last-probe data — issuing ZERO Telegram RPC.
- telegram_client: public watchdog_last_ok_age() — seconds since the last successful
watchdog probe (None if never). Pure read of the Stage-1 _wd_last_ok_monotonic
field; no RPC.
- api_server: /ping route (no token, no TG RPC, no SQLite, no fs scan). healthy =
connected and (age is None or age < threshold). age is None right after boot =>
healthy (don't kill before the first probe). connected coerced to bool so the JSON
"connected" field is always a bool (pre-start reports false, never null).
- config: TG_PING_UNHEALTHY_AFTER knob, default interval*(failures+1)+timeout = 250s
(how long until the watchdog itself gives up), env-overridable.
- dockercompose.yml: healthcheck -> curl -sf http://127.0.0.1:80/ping, interval 5m,
timeout 5s, retries 3, start_period 30s. Old /rss check removed, not left behind.
- tests: 10 (healthy/stale/disconnected/fresh-boot/pre-start-null/no-token +
the anti-regression zero-TG-RPC spy across all branches + the accessor).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes flaky static media serving: partial files served as 'ready', truncated
stubs living for an hour, and app-wide hangs under a saturated download path.
2.1 _download_atomic(file_id, final_path, timeout): downloads to a unique
{final}.part.{hex}, validates size>0, and publishes solely via os.rename
(atomic on POSIX); a finally ALWAYS removes the partial (timeout/cancel/
zero-size/race-loser). Every downloader call routes through it, so a file at a
FINAL name ({fid} / temp_{fid}) is GUARANTEED complete (grep-proven: the only
safe_download_media caller passes a .part. path). Big-video timeout scales with
size (min/max/min-speed knobs, documented in dockercompose.yml). The sweeper
regex now matches both .part. and legacy .tmp. stubs.
2.2 In-flight dedup registry: the first request for a key runs the download in a
DETACHED task sharing a Future; the task's finally sets the Future AND pops the
key. Waiters await asyncio.shield(fut) so a client disconnect / waiter timeout
cancels only the waiter, never the download — no hung waiters, no stuck key, a
failed download frees the key for retry.
2.3 FloodWait->429: handler before except RPCError (FloodWait subclasses it),
Retry-After = min(value + rand(1,30), 300); propagates from the detached task
through the Future.
2.4 Serving a temp_* file touches its mtime so the 1h sweeper can't delete a
video out from under a viewer.
2.5 The HTTP download semaphore acquire is bounded (wait_for 30 -> 503 +
Retry-After); the permit is released only if the acquire succeeded. The
request-scoped-permit trade-off (a disconnect can transiently exceed the
download count) is documented inline for stage-7 prod observation.
Tests (tests/test_stage2_static.py, 13): atomic publish/clean on every exit incl.
FloodWait-through-the-finally; concurrent big-video serves no partial; dedup runs
one download, a cancelled waiter doesn't hang others, a failed download frees the
key; FloodWait -> 429; mtime touch; sweeper cleans .part./.tmp./stale temp_ but
keeps fresh files. 193 passed (180 baseline + 13).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce `tg_watchdog_heartbeat_every` to emit periodic INFO heartbeats.
Add cumulative diagnostics counters and richer logging for watchdog probes,
restart reasons, and disconnect‑flap triggers. Update defaults and tests
accordingly.
Introduce an active watchdog that probes the Telegram client to detect
zombie sessions and restart them in‑process. Add configurable disconnect
flap detection with a sliding window to trigger restarts after repeated
disconnects. New environment variables and config entries are added, and
the Kurigram dependency is now version‑pinned.
- Use unique temporary file paths with UUID to avoid race conditions during concurrent downloads.
- Perform atomic rename of the temp file to the final cache location and handle existing concurrent downloads.
- Clean up zero‑size and stale temporary files, including new “.tmp.<hex>” pattern.
- Extend cleanup logic to detect and remove race‑condition temp files.
- Validate and convert environment variables (TG_API_ID, TG_PROXY_PORT, API_PORT) with clear error messages and proper exit handling.
- Flush prints and replace `os._exit` with `sys.exit` for graceful termination.
- Fix URL regex in `PostParser` and guard against missing channel usernames when generating media URLs.
- Deep‑copy message lists in RSS generator to prevent mutation across calls.
- Propagate `FloodWait` exceptions to be handled by the API server instead of retrying internally.
- Enhance `TelegramClient` disconnect handling with a brief pause, shutdown check, and reconnection reset.
- Refine auth retry logic to check for actual `KeyError` instances.
Introduce optional MTProto proxy configuration sourced from environment variables, expose related settings in `config.py`, document the variables in `dockercompose.yml`, and pass the proxy configuration to the Telegram client initialization.
Add a `_shutting_down` flag to `TelegramClient` to suppress disconnect handling during intentional shutdown. The flag is set before sending SIGTERM in `_restart_app` and checked in `_on_disconnect` to ignore those events. Minor formatting adjustments were also applied to `api_server.py` (try block and `uvicorn.run` arguments).
Add queue-based background download system to improve performance and reliability:
- Introduce DOWNLOAD_SEMAPHORE to limit concurrent downloads to 3
- Add asyncio.Queue (maxsize 100) with dedicated worker for background processing
- Implement safe_get_messages and safe_download_media wrappers with timeout protection (30s and 120s)
- Add retry logic for KeyError auth failures with 5s backoff
- Replace synchronous sequential downloads with asynchronous queued processing
- Prevent event loop blocking by queuing files instead of immediate download
vvzvlad
agent_coder
claude code agent