Flexible crawling
Start with a website, a list of URLs or an XML sitemap. Configure the depth, page limit and request rate.
PORTFOLIO / WEB & SEO
Deep Crawling.
Explore a website.
Understand its connections.
A project in my portfolio focused on crawling and technical SEO: following pages, reading the signals and making it clearer where improvements are needed.
FROM URL TO FINDINGA fuller view
A fuller view
of your website.
The repository brings together a crawling engine, a web interface and tools for organising the findings. Each feature serves a practical part of the analysis.
SEO, page by page
Inspect titles, descriptions, headings, canonicals, hreflang and directives. Gather the evidence needed to identify gaps and duplicates.
Links & HTTP responses
Map internal and external links and inspect response codes, redirects and URLs that fail to respond.
JavaScript websites
Optional Playwright rendering supports analysis of browser-generated content alongside the initial HTML.
Projects & reports
Save crawls, import and export projects, and work with the findings through reports and formats such as CSV and JSON.
Automation
The CLI and scheduler support recurring crawls. Webhooks and email can deliver notifications when configured.
FROM CONNECTIONS TO DECISIONSCuriosity continues
Curiosity continues
beyond radio.
Deep Crawling adds another area of experimentation to my portfolio: the relationship between software, data and the quality of the web. I am interested in making problems visible and building tools that help people understand them before taking action.
CODE, DOCUMENTATION & CREDITS
A project to explore.
The source is public and the project is evolving. The README describes local setup with Python or Docker. No published releases were available when checked: the repository is the starting point.
The code and documentation retain references to LibreCrawl. The MIT licence preserves the original Phiality copyright; credits and terms are available in the repository.
Sources checked on 31 October 2026: README, source code, licence and feature matrix. A feature being present in the code does not mean it has been verified on every website or external service.