Men and women use DA to determine which Internet websites are reliable in their discipline and to determine wherever to obtain fantastic hyperlinks. txt file is then parsed and will instruct the robotic as to which internet pages will not be for being crawled. Like a internet search engine crawler https://www.youtube.com/watch?v=EGBkutWZZTk