This kind of methods are as a result not ordinarily utilized by conforming Net robots or Internet crawlers; some that do not conform are inclined to make requests devoid of regard to context or implications. This guidebook handles a handful of server configuration changes that may be necessary fo