Category

Agents and infrastructure

Covers agents, crawlers and the infrastructure that decides who gets in.

About this section

Covers agents, crawlers and the infrastructure that decides who gets in.

Bot identification by operator and by purpose, access control, declarative signals in robots.txt and what each block costs in visibility. The criterion here is what you lose by blocking, weighed equally with what you protect.