Feed generation blocked the loop (deepcopy of hundreds of Messages + per-message
bleach x4 + str(message) per post). This moves the CPU work off the loop and cuts
it down.
4.1 raw_message is lazy: process_message(include_raw=False) omits str(message)
entirely; only /json and debug-HTML compute it.
4.2 (done BEFORE 4.3) _save_media_file_ids no longer touches asyncio/DB — it
appends to the per-request PostParser._pending_media_ids, and the caller
flushes once via to_thread(upsert_media_file_ids_bulk_sync) (a new executemany
fn in file_io.py). Removed _persist_pending_count + the create_task machinery.
This had to precede 4.3 or get_running_loop() inside the render thread would
raise and silently kill media-id persistence.
4.3 The render pipeline (_create_time_based_media_groups, _create_messages_groups,
_trim_messages_groups, _render_messages_groups) is now plain-sync and runs in
ONE asyncio.to_thread(_render_pipeline) from both feed generators; deepcopy
moved into the thread. The render path is verified free of asyncio primitives.
4.4 Sanitize is now ONE pass per output boundary (removed the 4 internal
per-fragment bleach passes): RSS/HTML feeds sanitize once at the whole-feed
pass; /html and /json sanitize body+footer once in process_message; the debug
<pre> raw dump is html.escape'd. Media embeds in body, reactions in footer, so
both are covered by the boundary pass. XSS tests (<script>/onerror=/javascript:)
confirm all four outputs are clean.
Review round-1: documented that flags are now extracted from the pre-sanitize body
(a 4.4 consequence — non-security, legitimate links unaffected); added
media-caption XSS tests for the exact fragments whose internal passes were removed
(adversarially validated: neutering the sanitizer makes them fail); the media-id
flush is now in a finally so a partial render still persists what it collected.
230 passed (214 baseline + 16).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Eliminates the app-wide hangs where one stuck Telegram RPC holds the single
tg_rpc gate permit forever (freezing every RSS/HTML request) and where the
background download worker dies silently.
1.1 Every RPC under the global gate is now bounded by asyncio.wait_for
(config tg_rpc_timeout, env TG_RPC_TIMEOUT, default 60). The wait_for wraps
ONLY the RPC body, never the gate acquire, so queue backpressure stays
legitimate; the paginated get_chat_history is collected in an inner coroutine
so the async-for can be bounded. A timeout propagates out of the gate so its
permit is released (no leak).
1.2 The other live RPC paths get the same treatment: _reply_enrichment's
get_messages now runs under the gate + timeout; PostParser.get_post is bounded
at 30s (and a stray print removed); /health's get_me at 10s.
1.3 background_download_worker: queue.get() moved out of the try so task_done()
in finally balances exactly one get (no ValueError that killed the worker);
FloodWait is caught BEFORE the generic Exception (sleeps, does not flood);
download_new_files uses put_nowait so a full queue no longer blocks the
sweeper forever.
1.4 Both background tasks run under a _supervised() wrapper: a crash or an
unexpected return is logged CRITICAL and restarted (restarts rate-limited to
once per 60s so a hard-failing task can't spin), while CancelledError is
propagated to the child for a clean shutdown.
Tests (tests/test_stage1_hangs.py): gate timeout releases the permit + a second
call succeeds; gate cancel mid-spacing releases the permit; the worker survives
Exception and FloodWait with balanced task_done; and _supervised restarts a
crashing task (rate-limited) and an unexpected return, and propagates cancellation
to its child. 180 passed (174 baseline + 6).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert delayed_delete_file to async and use asyncio.sleep.
Add pre‑semaphore cache hit to serve cached files without acquiring the download semaphore.
Set SQLite busy_timeout to reduce lock errors.
Introduce a diagnostics counter for pending _persist_media_file_id_async tasks.
Refactor flag extraction to accept pre‑generated HTML and compute html body once.
Batch reply enrichment in RSS generator to minimize API calls.
Store cached messages directly in tg_cache and add fallback for legacy double‑pickle format.
BREAKING CHANGE: delayed_delete_file is now async and must be awaited.
- Use unique temporary file paths with UUID to avoid race conditions during concurrent downloads.
- Perform atomic rename of the temp file to the final cache location and handle existing concurrent downloads.
- Clean up zero‑size and stale temporary files, including new “.tmp.<hex>” pattern.
- Extend cleanup logic to detect and remove race‑condition temp files.
- Validate and convert environment variables (TG_API_ID, TG_PROXY_PORT, API_PORT) with clear error messages and proper exit handling.
- Flush prints and replace `os._exit` with `sys.exit` for graceful termination.
- Fix URL regex in `PostParser` and guard against missing channel usernames when generating media URLs.
- Deep‑copy message lists in RSS generator to prevent mutation across calls.
- Propagate `FloodWait` exceptions to be handled by the API server instead of retrying internally.
- Enhance `TelegramClient` disconnect handling with a brief pause, shutdown check, and reconnection reset.
- Refine auth retry logic to check for actual `KeyError` instances.
Added a guard in `api_server.download_media_file` to detect `None` or empty messages and return a 404 error, preventing downstream failures.
Enhanced `PostParser._extract_reactions` to handle paid reactions, custom emojis, and unknown types, aggregating counts correctly. Updated `_extract_flags` to skip paid and custom emoji reactions, ensuring flag detection works reliably.
Use the original `channel` argument in the error log instead of
`prepared_channel_id` to prevent an UnboundLocalError when an
exception is raised during post parsing.
Replace string comparisons with the MessageMediaType.POLL enum and add the
necessary import. This ensures poll messages are correctly identified and
skipped during media processing.
Update poll handling to extract the `text` attribute from question and option objects when present, falling back to string conversion otherwise. This ensures proper rendering of poll content from Telegram objects that expose a `text` field.
Switch persistence of media file identifiers from a JSON file to a
SQLite database. Introduce DB_PATH, init_db_sync, upsert, update, and
removal functions. Remove json-repair dependency and related locking
logic. Update api_server and post_parser to use the new database
helpers.
BREAKING CHANGE: media_file_ids.json is no longer used; existing
data must be migrated to the new SQLite database.
Add monotonic timing instrumentation around media download, semaphore
wait, HTML sanitization, and media file persistence. Log warnings when
operations exceed defined thresholds and provide diagnostic info for
task queue size. This aids performance monitoring without altering
behaviour.
The changes ensure that when `show_caption_above` is true, text content is rendered before media, and when false, media is rendered before text. This aligns the actual rendering order with the semantic meaning of the flag, fixing a bug where content ordering did not match user expectations.
Add support for Telegram's show_caption_above_media message attribute to conditionally order caption and media content in HTML output. When enabled, media is rendered before the text caption instead of the default text-before-media order.
claude code agent
vvzvlad