LinkTracker

How it works

The user adds a link to a GitHub repository or a Stack Overflow question through the Telegram bot. The Bot Service handles the user’s commands and passes subscriptions on to the Scrapper Service over REST and gRPC. On a schedule, the Scrapper polls the external APIs, compares the state and publishes any updates it finds to Kafka. The processing layer picks them up from there, sifts and groups them, and the Bot Service reads what comes out and sends it on to the user. The Scrapper processes links in batches across several concurrent tasks. A failure on one link doesn’t stop the others and is included in the final report.

If Kafka is unavailable, the Scrapper sends the update straight over HTTP — a different route, the same outcome for the user. Messages that could not be processed go to a dead letter queue rather than disappearing. The public REST API is rate-limited per IP, with a separate bucket for each endpoint.

Surviving upstream failures

Every call to someone else’s API is wrapped in timeouts, retries and a circuit breaker. When GitHub or Stack Overflow answer slowly or not at all, requests are cut off on timeout and retried with a delay; if the errors persist the circuit opens, and the service stops hammering a source that is down while continuing to serve the rest.

Sifting and grouping updates

A separate processing layer, implemented as another Kafka consumer, sits between the Scrapper and the bot. It reads raw updates from one topic, drops whatever matches the filtering rules, assigns the survivors a keyword-based priority, shortens over-long texts, groups everything addressed to one chat within a short window, and publishes the result to another topic. The bot reads only what comes out of it.

The layer is deterministic on purpose: filtering, priorities and shortening are rules, not a model. There is no LLM inside. Shortening does sit behind an interface, though, so a rule can be swapped for a model without touching the pipeline itself. The layer imports nothing from the Scrapper and lives as its own package. Lifting it into a standalone service would take no code changes.

The cache, and what it bought

The list of tracked links is served through a two-tier cache: a three-node Valkey cluster with an optional in-process LRU+TTL ahead of it. Keys are hash-tagged by chat ID, so every page for one chat lands on the same cluster slot and a write invalidates them in one pass without touching its neighbours.

The load test was run in three configurations back to back, which is what answers the question of whether the cache was worth the effort:

Configuration	RPS	p50	p99
no cache	146	250 ms	690 ms
Valkey	710	22 ms	250 ms
Valkey + in-process	876	11 ms	230 ms

The first tier does the heavy lifting: nearly five times the throughput and an order of magnitude off the median. The second adds another quarter and halves what latency was left. The gain is modest, because Valkey is already fast over loopback.

Writes, meanwhile, are bottlenecked by PostgreSQL and the cache does nothing for them: POST and DELETE stay within 15% between the second and third scenarios. What improvement they show over the first comes not from the cache itself but from reads no longer competing with them.

Observability

Both services expose /metrics in Prometheus format: RED metrics (request rate, error rate, duration) plus an application counter, links_on_track_total, which reports how many links are currently being tracked. The dashboards live in Grafana, so the state of the system is something you look at rather than something you grep the logs for.

Tests and build

Tests come in four layers: unit, integration, e2e and load. The integration layer brings up real Kafka, PostgreSQL and Valkey through Testcontainers, so what is verified is real behaviour rather than the behaviour of mocks. The load tests produced the figures in the metrics above. All of it runs in GitHub Actions on every push alongside the linters, and the services are packaged in Docker.