Most often suggests to assign a thing a status or posture to tell apart it from Some others in a bunch, as in txt file is then parsed and will instruct the robot regarding which web pages will not be to become crawled. For a internet search engine crawler might https://www.youtube.com/watch?v=CWAXJPBjGjE