Add robots.txt
1 minute read
There is no static/robots.txt in the repo, and no Sitemap: directive published anywhere. Crawlers default to discovery via common
paths, which is fragile.
Tasks
- Create
static/robots.txtwithUser-agent: *,Allow: /, and aSitemap:line pointing atPROJECT_URL + '/sitemap-appendix.xml'
Read next