Test robots.txt: Sites normally listing their scraping guidelines within their robots.txt file. This file tells World wide web crawlers which sections of the location These are allowed or prohibited from scraping. From https://bit.ly/sayyamnrg