Do 1 [test-coverage]: залочил security-инвариант §3.4 (per-post изоляция
фрагмента) именованным тестом test_unbalanced_post_does_not_swallow_next_post_html_feed
(в test_stage4_eventloop.py — гоняет реальный per-post HTML-пайплайн на истории из
2 сообщений). Пост A несёт висячий `<div><b>DANGLING_A` без закрывашек, пост B
целый с маркером. Ассертит: фид бьётся на 2 фрагмента по `<hr post-divider>`;
теги A сбалансированы В ПРЕДЕЛАХ фрагмента A; маркер B в фрагменте B и НЕ в A
(не заглочен). Проверено, что тест краснеет на регрессии «join-before-sanitize»:
при склейке raw-постов до единого whole-feed bleach.clean невайтлистнутый `<hr>`
стрипается (strip=True) → 0 разделителей → 1 фрагмент → первый ассерт падает.
То есть будущий этап, вернувший склейку до санитайза, ловится этим тестом.
Do 2 [documentation]: 4 устаревших коммента про удалённый «whole-feed sanitize
для HTML» обновлены на «per-post в _render_pipeline для обоих (RSS и HTML)»
(post_parser.py process_message docstring / _generate_html_body NOTE /
generate_html_footer coverage-map; rss_generator.py feed-path коммент). Вывод
«санитайзится ровно один раз» сохранён. Исполняемый код не менялся.
Полный сюит 347 passed (+1 тест изоляции); оракул этапа 0 — 11 passed, golden не менялся.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Часть эпика рефакторинга рендер-пайплайна (#34). Один bleach-конфиг на весь
проект, один санитайз на пост, ноль thread-hop'ов в циклах (раньше конфиг был
скопирован трижды и разъехался; RSS-путь делал to_thread + новый CSSSanitizer +
вложенную функцию НА КАЖДЫЙ пост, хотя _render_pipeline уже в worker-треде).
Новый sanitizer.py — единственная bleach-конфигурация: ALLOWED_TAGS (union трёх
старых копий, вкл. s/del), ALLOWED_PROTOCOLS=['http','https','tg'],
sanitize_html() зовёт bleach.clean(..., protocols=..., strip=True) (оба не-дефолт),
fail-CLOSED (html.escape при любом исключении), единое имя лога
html_sanitization_error + log_context. _render_pipeline санитайзит каждый пост
после рендера (в своём worker-треде); rss_generator −71 строка.
Реестр §3:
- §3.1 s/del теперь в фидах (раньше выживали только в одиночном посте);
- §3.2 fail-open → fail-closed (вкл. single-post/JSON путь);
- §3.3 <hr class="post-divider"> реально виден (join ПОСЛЕ санитайза; hr не в
whitelist, поэтому раньше вырезался strip=True);
- §3.4 несбалансированный фрагмент нормализуется в границах своего поста;
- §3.5 гранулярный fail-closed HTML-фида (падает один пост, не весь фид);
- §3.16 три имени санитайз-логов объединены в html_sanitization_error+log_context.
Golden-эталоны обновлены: КАЖДЫЙ изменённый байт привязан к пункту §3 (net-diff
тегов: только +<s>/§3.1 и +<hr>/§3.3; <div> сбалансирован +/- = §3.4;
после снятия этих токенов + нормализатора все 6 голденов побайтово old==new).
Оракул этапа 0 зелёный. Полный сюит 346 passed (+14 sanitizer-тестов).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Upgrade Kurigram 2.2.22 -> 2.2.23 (no breaking API changes; whole suite
green on the new version unchanged).
New in 2.2.23 and now rendered by the bridge:
- LIVE_PHOTO: rendered as a looping video via the /media pipeline
- Poll media (description_media): photo/video/animation/sticker attached
to a poll is rendered above the poll block; find_file_id_in_message
searches poll description_media AND explanation_media instead of
early-returning None for polls
Previously silent media types now rendered:
- STORY: reposted story photo/video via the /media pipeline (render and
URL derive from the same helper-selected object)
- GIVEAWAY / GIVEAWAY_WINNERS: info blocks with quantity/prize/date
- PAID_MEDIA: info block with star amount (content is not downloadable)
- CHECKLIST: title + tasks with done marks
- CONTACT, LOCATION, VENUE (with OSM links), DICE, GAME, INVOICE,
UNSUPPORTED: info blocks, all user-controlled strings html-escaped
Supporting changes:
- titles for all new types in _media_message_title
- no_image flag covers info-block types; poll with media is not no_image;
LIVE_PHOTO counts as video for the video flag
- _save_media_file_ids collects live_photo/story/poll media for the
background cache warmer, honoring the >100MB video guard; paid media
is never collected
- info-block types no longer emit an empty message-media div
- 53 new tests (media URLs with digest, download symmetry, XSS escaping,
flags, size guard); 314 total
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Issue #13: data["html"]["title"] was embedded into the debug HTML of
/post/html without escaping; title never passes through bleach, so a post
whose generated title carries markup (e.g. a poll question) was a reflected
XSS under ?debug=true. Escape it with html.escape, same as raw_message.
Add a regression test with a <script> payload in a poll question.
Issue #17: a bare 'pytest' from the repo root failed 22 tests because the
config in tests/pytest.ini was not picked up (asyncio_mode lost, .venv
collected) and every test module polluted sys.modules['config'] at import
time. Move the config to a root pytest.ini with testpaths=tests, and
centralize the sys.path bootstrap + config mock in tests/conftest.py, which
pytest imports before any test module. Drop the per-module preambles.
Closes#13, closes#17
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#1 [security] The final feed-sanitize try/except was FAIL-OPEN: since 4.4 removed
the per-fragment passes, post['html'] / the concatenated feed html is now raw
channel-controlled HTML, and this is its ONLY sanitize. If bleach itself throws
(RecursionError/OOM on pathological nested HTML — a class already seen in this
project), the except branch returned the RAW payload = stored XSS. Both branches
(RSS per-post, HTML whole-feed) now fail CLOSED via html.escape, so the content
survives as inert text and no live tag ever reaches the client. Added a test that
forces bleach to raise and asserts no live <script>/<img>/<a> reaches the feed
(adversarially validated: reverting to fail-open makes it fail).
#2 [test] Direct test of the new upsert_media_file_ids_bulk_sync on a temp DB:
empty no-op, multi-row insert, and re-upsert-updates-added (the real executemany +
ON CONFLICT, previously only mocked).
#3 [test] Test that media ids collected before a render exception are still
flushed (the flush is in a finally) — removing the finally now turns a test red.
#4 [cleanup] Removed the dead in rss_generator (the
flush is delegated to post_parser._flush_pending_media_ids).
#5 [doc] Corrected the sanitize-map comments: RSS sanitizes per-post, only HTML is
the whole-feed pass.
233 passed (214 baseline + 19).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Feed generation blocked the loop (deepcopy of hundreds of Messages + per-message
bleach x4 + str(message) per post). This moves the CPU work off the loop and cuts
it down.
4.1 raw_message is lazy: process_message(include_raw=False) omits str(message)
entirely; only /json and debug-HTML compute it.
4.2 (done BEFORE 4.3) _save_media_file_ids no longer touches asyncio/DB — it
appends to the per-request PostParser._pending_media_ids, and the caller
flushes once via to_thread(upsert_media_file_ids_bulk_sync) (a new executemany
fn in file_io.py). Removed _persist_pending_count + the create_task machinery.
This had to precede 4.3 or get_running_loop() inside the render thread would
raise and silently kill media-id persistence.
4.3 The render pipeline (_create_time_based_media_groups, _create_messages_groups,
_trim_messages_groups, _render_messages_groups) is now plain-sync and runs in
ONE asyncio.to_thread(_render_pipeline) from both feed generators; deepcopy
moved into the thread. The render path is verified free of asyncio primitives.
4.4 Sanitize is now ONE pass per output boundary (removed the 4 internal
per-fragment bleach passes): RSS/HTML feeds sanitize once at the whole-feed
pass; /html and /json sanitize body+footer once in process_message; the debug
<pre> raw dump is html.escape'd. Media embeds in body, reactions in footer, so
both are covered by the boundary pass. XSS tests (<script>/onerror=/javascript:)
confirm all four outputs are clean.
Review round-1: documented that flags are now extracted from the pre-sanitize body
(a 4.4 consequence — non-security, legitimate links unaffected); added
media-caption XSS tests for the exact fragments whose internal passes were removed
(adversarially validated: neutering the sanitizer makes them fail); the media-id
flush is now in a finally so a partial render still persists what it collected.
230 passed (214 baseline + 16).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Eliminates the app-wide hangs where one stuck Telegram RPC holds the single
tg_rpc gate permit forever (freezing every RSS/HTML request) and where the
background download worker dies silently.
1.1 Every RPC under the global gate is now bounded by asyncio.wait_for
(config tg_rpc_timeout, env TG_RPC_TIMEOUT, default 60). The wait_for wraps
ONLY the RPC body, never the gate acquire, so queue backpressure stays
legitimate; the paginated get_chat_history is collected in an inner coroutine
so the async-for can be bounded. A timeout propagates out of the gate so its
permit is released (no leak).
1.2 The other live RPC paths get the same treatment: _reply_enrichment's
get_messages now runs under the gate + timeout; PostParser.get_post is bounded
at 30s (and a stray print removed); /health's get_me at 10s.
1.3 background_download_worker: queue.get() moved out of the try so task_done()
in finally balances exactly one get (no ValueError that killed the worker);
FloodWait is caught BEFORE the generic Exception (sleeps, does not flood);
download_new_files uses put_nowait so a full queue no longer blocks the
sweeper forever.
1.4 Both background tasks run under a _supervised() wrapper: a crash or an
unexpected return is logged CRITICAL and restarted (restarts rate-limited to
once per 60s so a hard-failing task can't spin), while CancelledError is
propagated to the child for a clean shutdown.
Tests (tests/test_stage1_hangs.py): gate timeout releases the permit + a second
call succeeds; gate cancel mid-spacing releases the permit; the worker survives
Exception and FloodWait with balanced task_done; and _supervised restarts a
crashing task (rate-limited) and an unexpected return, and propagates cancellation
to its child. 180 passed (174 baseline + 6).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert delayed_delete_file to async and use asyncio.sleep.
Add pre‑semaphore cache hit to serve cached files without acquiring the download semaphore.
Set SQLite busy_timeout to reduce lock errors.
Introduce a diagnostics counter for pending _persist_media_file_id_async tasks.
Refactor flag extraction to accept pre‑generated HTML and compute html body once.
Batch reply enrichment in RSS generator to minimize API calls.
Store cached messages directly in tg_cache and add fallback for legacy double‑pickle format.
BREAKING CHANGE: delayed_delete_file is now async and must be awaited.
- Use unique temporary file paths with UUID to avoid race conditions during concurrent downloads.
- Perform atomic rename of the temp file to the final cache location and handle existing concurrent downloads.
- Clean up zero‑size and stale temporary files, including new “.tmp.<hex>” pattern.
- Extend cleanup logic to detect and remove race‑condition temp files.
- Validate and convert environment variables (TG_API_ID, TG_PROXY_PORT, API_PORT) with clear error messages and proper exit handling.
- Flush prints and replace `os._exit` with `sys.exit` for graceful termination.
- Fix URL regex in `PostParser` and guard against missing channel usernames when generating media URLs.
- Deep‑copy message lists in RSS generator to prevent mutation across calls.
- Propagate `FloodWait` exceptions to be handled by the API server instead of retrying internally.
- Enhance `TelegramClient` disconnect handling with a brief pause, shutdown check, and reconnection reset.
- Refine auth retry logic to check for actual `KeyError` instances.
Added a guard in `api_server.download_media_file` to detect `None` or empty messages and return a 404 error, preventing downstream failures.
Enhanced `PostParser._extract_reactions` to handle paid reactions, custom emojis, and unknown types, aggregating counts correctly. Updated `_extract_flags` to skip paid and custom emoji reactions, ensuring flag detection works reliably.
Use the original `channel` argument in the error log instead of
`prepared_channel_id` to prevent an UnboundLocalError when an
exception is raised during post parsing.
Replace string comparisons with the MessageMediaType.POLL enum and add the
necessary import. This ensures poll messages are correctly identified and
skipped during media processing.
Update poll handling to extract the `text` attribute from question and option objects when present, falling back to string conversion otherwise. This ensures proper rendering of poll content from Telegram objects that expose a `text` field.
Switch persistence of media file identifiers from a JSON file to a
SQLite database. Introduce DB_PATH, init_db_sync, upsert, update, and
removal functions. Remove json-repair dependency and related locking
logic. Update api_server and post_parser to use the new database
helpers.
BREAKING CHANGE: media_file_ids.json is no longer used; existing
data must be migrated to the new SQLite database.
Add monotonic timing instrumentation around media download, semaphore
wait, HTML sanitization, and media file persistence. Log warnings when
operations exceed defined thresholds and provide diagnostic info for
task queue size. This aids performance monitoring without altering
behaviour.
The changes ensure that when `show_caption_above` is true, text content is rendered before media, and when false, media is rendered before text. This aligns the actual rendering order with the semantic meaning of the flag, fixing a bug where content ordering did not match user expectations.
Add support for Telegram's show_caption_above_media message attribute to conditionally order caption and media content in HTML output. When enabled, media is rendered before the text caption instead of the default text-before-media order.
agent_coder
vvzvlad
claude code agent