Nginx Log Analyzer
A CLI analyzer for nginx access logs: streaming processing, tolerance for malformed lines, and reports in JSON, Markdown and AsciiDoc.
Stack
- Python
- Docker
Key technical decisions
- Streaming reads instead of loading the file into memory: the whole log never has to be held in RAM.
- A malformed line is skipped rather than fatal, and counting continues over the remaining lines.
- The source can be a local file, a glob pattern or a URL.
- One set of aggregates renders into three formats: JSON for automated processing, Markdown and AsciiDoc to read.
- Nothing external at runtime, only the Python standard library.
- Strict typing checked by mypy, unit tests with pytest, linting, Docker and CI.
A command-line tool that reads nginx access logs and collects statistics from them: the most requested resources, response codes, response-size percentiles and how requests are distributed across dates.
Nginx logs easily grow into gigabytes, and that is what sets the requirements: work without holding the file in memory, and carry on processing even when individual lines do not match the expected format.