Commit Graph

106 Commits

Author SHA1 Message Date
vvzvlad f0e38b9776 perf(core): improve async handling and caching performance
Convert delayed_delete_file to async and use asyncio.sleep.
Add pre‑semaphore cache hit to serve cached files without acquiring the download semaphore.
Set SQLite busy_timeout to reduce lock errors.
Introduce a diagnostics counter for pending _persist_media_file_id_async tasks.
Refactor flag extraction to accept pre‑generated HTML and compute html body once.
Batch reply enrichment in RSS generator to minimize API calls.
Store cached messages directly in tg_cache and add fallback for legacy double‑pickle format.

BREAKING CHANGE: delayed_delete_file is now async and must be awaited.
2026-05-18 00:08:46 +03:00
vvzvlad ab8f15d49d feat(api): add atomic temp file handling for media downloads
- Use unique temporary file paths with UUID to avoid race conditions during concurrent downloads.
- Perform atomic rename of the temp file to the final cache location and handle existing concurrent downloads.
- Clean up zero‑size and stale temporary files, including new “.tmp.<hex>” pattern.
- Extend cleanup logic to detect and remove race‑condition temp files.
- Validate and convert environment variables (TG_API_ID, TG_PROXY_PORT, API_PORT) with clear error messages and proper exit handling.
- Flush prints and replace `os._exit` with `sys.exit` for graceful termination.
- Fix URL regex in `PostParser` and guard against missing channel usernames when generating media URLs.
- Deep‑copy message lists in RSS generator to prevent mutation across calls.
- Propagate `FloodWait` exceptions to be handled by the API server instead of retrying internally.
- Enhance `TelegramClient` disconnect handling with a brief pause, shutdown check, and reconnection reset.
- Refine auth retry logic to check for actual `KeyError` instances.
2026-05-17 23:49:18 +03:00
vvzvlad dd5999b51f fix: guard against missing messages and reaction issues
Added a guard in `api_server.download_media_file` to detect `None` or empty messages and return a 404 error, preventing downstream failures.

Enhanced `PostParser._extract_reactions` to handle paid reactions, custom emojis, and unknown types, aggregating counts correctly. Updated `_extract_flags` to skip paid and custom emoji reactions, ensuring flag detection works reliably.
2026-05-17 23:12:11 +03:00
vvzvlad 40fa9797e8 feat(api): add trusted proxy handling to local request detection
The `is_local_request` function now supports trusted reverse proxies by:
- Reading `TRUSTED_PROXIES` from configuration.
- Resolving the real client IP from `X-Real-IP` or `X-Forwarded-For` headers when the request comes through a trusted proxy.
- Adding IPv6 loopback support and detailed logging for misconfigurations.
- Updating configuration to expose `trusted_proxies` via the `TRUSTED_PROXIES` environment variable.
2026-05-17 22:55:50 +03:00
vvzvlad 45cd18af99 feat(api): add landing page endpoint
Introduce a root GET route that serves a minimal HTML landing page describing the Pyrogram Bridge service and linking to its GitHub repository. This provides a friendly entry point for users accessing the service via a browser.
2026-05-11 02:54:39 +03:00
vvzvlad cd4e6f0c82 fix(api): use MessageMediaType enum for poll checks
Replace string comparisons with the MessageMediaType.POLL enum and add the
necessary import. This ensures poll messages are correctly identified and
skipped during media processing.
2026-04-16 03:46:03 +03:00
vvzvlad 85cee6b295 refactor(api): comment out redundant digest validation log
The else block that logged a valid digest was commented out to reduce
unnecessary log noise and avoid misleading messages during media
validation.
2026-04-06 23:54:23 +03:00
vvzvlad 57a96044bc feat(db): add mime_type column and caching helpers
Add a new `mime_type` column to the media_file_ids table and provide
functions to get and set cached MIME types. The API now attempts to
retrieve the MIME type from the database before falling back to
python-magic, persisting the detected type for future requests. This
reduces I/O overhead and improves response performance. Also adjust
logging to debug level to reduce noise.
2026-04-06 23:46:22 +03:00
vvzvlad 0f297185e4 feat(api): add range-enabled streaming for media files
Implement HTTP Range request support by replacing FileResponse with StreamingResponse.
Added async chunked generator, proper Content-Range, Accept-Ranges, and filename encoding headers.
Updated imports, endpoint signature, and background cleanup handling.
2026-04-06 23:37:46 +03:00
vvzvlad cf8a2e0ce5 feat(api): add immutable cache header to file responses
Add a Cache‑Control header with `public, max-age=86400, immutable`
to file responses, allowing clients to cache files for a day
2026-04-06 23:29:35 +03:00
vvzvlad c9b0d2ffc9 feat(db): replace media file ID JSON store with SQLite
Switch persistence of media file identifiers from a JSON file to a
SQLite database. Introduce DB_PATH, init_db_sync, upsert, update, and
removal functions. Remove json-repair dependency and related locking
logic. Update api_server and post_parser to use the new database
helpers.

BREAKING CHANGE: media_file_ids.json is no longer used; existing
data must be migrated to the new SQLite database.
2026-04-05 20:20:37 +03:00
vvzvlad 02c29f8692 fix(telegram): add shutdown guard to avoid duplicate restarts
Add a `_shutting_down` flag to `TelegramClient` to suppress disconnect handling during intentional shutdown. The flag is set before sending SIGTERM in `_restart_app` and checked in `_on_disconnect` to ignore those events. Minor formatting adjustments were also applied to `api_server.py` (try block and `uvicorn.run` arguments).
2026-03-16 04:13:39 +03:00
vvzvlad a38a729137 perf(api): add timing diagnostics for media ops
Add monotonic timing instrumentation around media download, semaphore
wait, HTML sanitization, and media file persistence. Log warnings when
operations exceed defined thresholds and provide diagnostic info for
task queue size. This aids performance monitoring without altering
behaviour.
2026-03-16 01:14:24 +03:00
vvzvlad 4ef7b00c16 refactor(api): split download semaphores and remove diagnostic monitoring
Separate live HTTP media requests and background cache workers by introducing `HTTP_DOWNLOAD_SEMAPHORE` and `BACKGROUND_DOWNLOAD_SEMAPHORE`. Remove the previous global `DOWNLOAD_SEMAPHORE`, diagnostic event‑loop monitor, and related logging. Refactor `download_media_file` to handle large videos with temporary files, clean up zero‑size cache entries, and update access timestamps without semaphore tracking. Adjust background worker and request handling to use the new semaphores, simplifying concurrency control and eliminating unnecessary diagnostics.
2026-03-16 01:07:19 +03:00
vvzvlad 8bcce02368 feat(api): add diagnostic monitoring and semaphore tracking
Introduce a periodic event‑loop monitor that logs active tasks, semaphore
state, queue size and thread‑pool activity. Track the number of coroutine
waiters for the download semaphore and emit detailed diagnostic logs in
download handling and the background worker. This aids debugging of
concurrency and resource‑usage issues.
2026-03-16 01:00:31 +03:00
vvzvlad 3c2b4ce544 perf(downloads): implement concurrent download queue with semaphore limiting and retry logic
Add queue-based background download system to improve performance and reliability:
- Introduce DOWNLOAD_SEMAPHORE to limit concurrent downloads to 3
- Add asyncio.Queue (maxsize 100) with dedicated worker for background processing
- Implement safe_get_messages and safe_download_media wrappers with timeout protection (30s and 120s)
- Add retry logic for KeyError auth failures with 5s backoff
- Replace synchronous sequential downloads with asynchronous queued processing
- Prevent event loop blocking by queuing files instead of immediate download
2026-01-01 14:14:26 +03:00
vvzvlad 68a2dbfa04 refactor api_server and post_parser: implement asynchronous JSON file operations with locking mechanism to ensure thread safety, and streamline media file ID persistence logic 2025-09-13 20:47:16 +03:00
vvzvlad 1464ea326e refactor api_server: streamline RSS feed generation by consolidating error handling and removing redundant loop structure 2025-09-13 00:06:36 +03:00
vvzvlad 778aa04904 refactor api_server: implement asynchronous file I/O using asyncio.to_thread for JSON operations, add global python-magic instance for MIME type detection, and enhance delayed file deletion documentation 2025-09-12 23:50:52 +03:00
vvzvlad 43516d6dab refactor api_server and TelegramClient: remove force_exit function and replace with sys.exit for process termination, update restart logic to use SIGTERM instead of SIGKILL for graceful shutdown 2025-06-10 15:03:09 +03:00
vvzvlad 521c3dc394 implement forceful termination in api_server and TelegramClient: add force_exit function for immediate process shutdown, update SIGTERM handler for forceful exit, and adjust restart logic in connection_handler to use SIGKILL 2025-06-10 14:28:51 +03:00
vvzvlad d6c3725b02 implement graceful shutdown in api_server: add SIGTERM handler for proper termination and enhance error handling for server startup issues 2025-06-10 14:06:56 +03:00
vvzvlad 0999e86f2b add timing logs for RSS and HTML generation in api_server and rss_generator 2025-04-27 18:40:31 +04:00
vvzvlad f499dc56a7 refactor api_server: comment out redundant log for cached media files 2025-04-27 18:22:31 +04:00
vvzvlad 422a3925ff fix bug 2025-04-25 16:59:21 +04:00
vvzvlad 386e7e8ce4 improve logging 2025-04-18 23:33:47 +03:00
vvzvlad fe1fee242d fix local req logic 2025-04-18 21:15:35 +03:00
vvzvlad 8c0a88cb84 small fix 2025-04-18 21:10:59 +03:00
vvzvlad 1760ec3485 add types-2 2025-04-18 20:39:57 +03:00
vvzvlad 6aba094a6c add types support 2025-04-18 19:42:54 +03:00
vvzvlad cb419c97af update pylint and pylance disable comments 2025-04-18 18:00:02 +03:00
vvzvlad d520e8c2e8 Update get_raw_post_json 2025-04-13 03:38:00 +03:00
vvzvlad 96419265a3 Update get_raw_post_json to using json.dumps 2025-04-13 03:31:01 +03:00
vvzvlad ff97d1190b Refactor get_raw_post_json 2025-04-13 03:25:28 +03:00
vvzvlad 0b75b5a9b9 Add endpoint to retrieve raw JSON post data 2025-04-13 03:19:10 +03:00
vvzvlad e76fb8606d Remove unnecessary pass statement in ZeroSizeFileError exception class 2025-04-11 18:13:57 +03:00
vvzvlad 55d2b7331a fix linter err 2025-04-11 18:12:56 +03:00
vvzvlad d21a68471d Refactor zero-size file handling 2025-04-11 18:10:21 +03:00
vvzvlad fb73f6b868 fix zero size bug 2025-04-11 18:00:30 +03:00
vvzvlad f42e159ba6 Add flags endpoint 2025-04-11 02:04:58 +03:00
vvzvlad 997b49b4df removal of invalid entries from media_file_ids.json 2025-04-04 22:21:22 +03:00
vvzvlad 129b07abda add logging 2025-04-04 22:09:52 +03:00
vvzvlad de5c3f1fa1 fix bug Invalid digest : None, expected: None 2025-04-04 22:07:47 +03:00
vvzvlad 422e2adc2d Add shebang and encoding declaration to multiple Python files 2025-04-03 01:48:34 +03:00
vvzvlad a45678a852 Add JSON repair functionality 2025-03-26 15:04:24 +03:00
vvzvlad 750c01776a Update API endpoints to include request parameter for token validation, enhancing security by checking client host against allowed addresses. 2025-03-26 15:00:28 +03:00
vvzvlad bbaf2314d5 Remove base64 and URL decoding 2025-03-22 15:26:55 +03:00
vvzvlad 5f5128fed0 Decode base64 parameters if they are encoded 2025-03-22 15:03:53 +03:00
vvzvlad 710ad700e0 URL decoding for exclude_text 2025-03-22 15:01:02 +03:00
vvzvlad d6df00ff8c Refactor RequestLoggingMiddleware to ensure proper logging of request and response details, including error handling during request processing 2025-03-22 14:54:05 +03:00