←  All terms
C
SEO & AEO

Crawling

Crawling is the automated retrieval of web pages by bots that discover links and content for search or other indexes.

Crawling is the automated retrieval of web pages by bots that discover links and content for search or other indexes. Sustainable visibility depends on clear information architecture, useful content, and accessible technology; no single measure guarantees rankings. The concept is closely related to Indexing and robots.txt.

Read more

Indexing

robots.txt

sitemap.xml

FAQ

Crawling

Why are some pages not picked up during crawling?
Common causes are robots.txt blocks, missing internal links, error codes, long response times, and content that only loads via JavaScript. Login areas also remain inaccessible. Server logs and crawl reports show which cause applies before content or technology is changed.
When does crawl budget matter for crawling a website?
Crawl budget describes how many URLs a bot fetches within a period. For small and mid-sized sites it is rarely a bottleneck. It becomes relevant for very large sites, many parameter URLs, or slow servers, because new content is then discovered with delay. A tidy structure has more effect than fine-tuning.
How can crawling of a website be controlled?
Through robots.txt, internal linking, XML sitemaps, clean status codes, and avoiding endless URL variants. The robots.txt file governs access, not indexing: blocked URLs can still appear in results if they are linked. For keeping pages out of the index, noindex is the more suitable tool.
How do crawling and indexing differ?
Crawling is the retrieval of a page, while indexing is the decision to include it in the search index. A crawled page is not automatically indexed, for instance with thin content, duplicates, or noindex. Conversely, indexing first requires successful crawling. Diagnosis must therefore check both stages separately.