Web site file formats

The files a web site publishes at known addresses so that crawlers and tools can find what is there and what they may fetch: sitemap.xml lists the pages, robots.txt says which ones a crawler may visit. Each is recognised by its name and by what is inside it, and the explorer shows its structure and checks it against its rules.