In delivery · Observability · Internal platform

Pehredaar

An always-on observer for systems that are already in production — and the last stage of the factory line that built them.

Approval is the midpoint of governance, not the end. A design review tells you whether a system is coherent; a test suite tells you whether it behaves as specified. Neither tells you what production will do to it at 3am. Pehredaar exists to hold that fourth responsibility — post-deployment observation — as an independent job rather than an afterthought bolted on after the first incident.

Shipped systems report into one observer

Measured, not estimated

24,660

Python LOC

69 modules · src/pehredaar

Connector types

uptime · webhook · OTLP · Sentry · OSQuery

Sources watched

of 22 configured

7/7

Products observed

every factory build

Store tables

SQLite · WAL

390

Tests

across 65 files

What it does, and what each part learned the hard way.

Ingest

CustomHTTPConnector · HMAC-SHA256 · constant-time compare

Push-mode error intake

Each monitored product posts structured error payloads to POST /webhook/{source_name}. Every request carries an X-Pehredaar-Signature header — sha256 HMAC over the raw body, verified with hmac.compare_digest so signature checking is constant-time. Per-source field maps use dot-notation to pull error.type, error.message, error.route, error.method, and error.status_code out of whatever shape the product sends.

The connector refuses to start when a source is configured with an empty secret and allow_unauthenticated set to false. An unauthenticated ingest endpoint that silently accepts anything is a backdoor, not a convenience, so the daemon fails loudly at boot instead of quietly accepting forged events.

Probe

HTTPUptimeConnector · GET/HEAD · expected-status assertion

Is it actually serving?

Polls a URL on a fixed interval with a rate-limit floor and asserts the response status against an expected set. DNS is refreshed on its own cadence so a stale record cannot mask a moved host.

The hard lesson here is that reachability is not health. A HEAD against a homepage returns 200 while the login path behind it is failing every request — the marketing page is static, the failure is three layers down. Probes are now pointed at endpoints that assert a correct response, not merely a reachable one: a health route that runs a real database round-trip and answers 503 when the pool is gone.

Dedup

Fingerprint + watermark event store

One incident, not ten thousand lines

Events are normalised into a single EventRecord shape regardless of connector, then fingerprinted so a failure repeating ten thousand times collapses into one tracked incident with an occurrence count. Watermarks track poll position, last status, and consecutive failures per source. 27 tables carry events, issues, alert rules, notification history, traffic, forecasts, and remediation attempts.

Retention is per-source rather than global — 90 days for error streams, 30 for uptime — because the cost of keeping a year of uptime pings is real and the value is not.

Alert

new_fingerprint · spike · repeat

Deciding what deserves a human

Three rule types. A new fingerprint means a failure mode nobody has seen before. A spike means N occurrences inside a window. A repeat means a known issue that will not stay fixed. Each rule carries its own cooldown so a flapping service cannot turn into a pager storm.

Rules are the link most likely to be silently missing. A source can be configured, enabled, polling happily, and still have zero rows in alert_rules — in which case it detects everything and tells nobody. Config and database agreeing is not a given; it has to be checked.

Deliver

Telegram · email · webhook · Slack

Getting it to a person

A dispatcher resolves transport credentials through a secrets layer (never inline in config), builds a message per transport, and records every send in a notifications_sent table with status ok, failed, or rate_limited.

Delivery is recorded, not assumed. A monitoring system that believes it sent an alert is indistinguishable from one that did, right up until the outage nobody heard about.

Integration

The factory's last stage

Wired in before release, not after an incident

Every product the factory ships carries a Pehredaar client — a small module that signs and posts errors from the application's exception handler — plus uptime and error sources registered in the observer's config. All 7 delivered and in-delivery products carry it: URIP, AstroRattan, io-gita, Eagle Tailors, Mushroom Ki Mandi, PP Monitoring, and this site.

Because the wiring is part of the build rather than a post-incident retrofit, observation starts at release instead of starting the first time somebody notices something is wrong. That ordering is the whole point.

Where it actually stands

Pehredaar runs as a daemon against live production systems and has an event store with real history behind it. It is not a finished product. Alert-rule materialisation from config is being hardened, delivery channels beyond Telegram are still being wired, and the observer needs a dead-man’s switch on its own ingest — because the failure mode that matters most for a monitoring system is the one where it stops receiving and does not say so.

That last point is not hypothetical. A pipeline can go quiet for weeks because of a single missing path segment in a webhook URL, while every dashboard stays green and every uptime probe keeps returning 200. The lesson is written into the roadmap rather than out of the story: an observer that cannot detect its own blindness is not yet finished.

← All systems Read the philosophy →