Research methodology
How Linkwake turns Common Crawl into backlink movement
A reproducible overview of the source, normalization, comparisons, page-level enrichment, and limitations behind Linkwake reports and studies.
1. Named source releases
Every analysis begins with a published Common Crawl web-graph release, such as cc-main-2026-mar-apr-may. A graph node represents a registrable domain. A directed edge means at least one crawled page on the source domain linked to the target domain during that release.
2. Domain normalization
Hosts are normalized to registrable domains so subdomains and repeated page links do not inflate the referring-domain relationship. A source domain counts once for a target domain within a release, even when many source pages contain links.
3. Release comparisons
These labels describe observations in two datasets. “Lost” does not prove that a live page removed a link.
4. Page-level evidence
For selected relationships, Linkwake queries Common Crawl indexes, retrieves archived HTML records, extracts matching links, and records source URL, target URL, anchor, link type, and rel attributes. This enrichment is separate from the domain graph and can still be incomplete.
5. Reproducibility and downloads
Research articles identify the release, publish summary datasets as CSV, explain calculations, and provide a citation block. The raw Common Crawl corpus remains available from the source project; Linkwake downloads contain the derived results used in each article rather than republishing the corpus.
Coverage limitations
- Common Crawl is a large sample, not a complete web index.
- Robots rules, crawl selection, errors, and JavaScript can affect coverage.
- Domain relationships do not establish editorial quality or causation.
- Counts from different backlink indexes should not be treated as interchangeable.
Questions or reproduction notes are welcome at support@linkwake.com.