Topic Options
#407436 - 01/03/10 01:41 AM Crafty Spider
Delta Offline
CoS Member

Registered: 12/18/07
Posts: 6750
Loc: Nar
This is kind of disturbing. Is this a Search Engine trying to enable viewing of subscription forums?


Attachments
spider.jpg


_________________________




Top
#407499 - 01/03/10 06:02 PM Re: Crafty Spider [Re: Delta]
TheAbysmal Offline


Registered: 09/22/06
Posts: 1019
At a glance, that is what it appears to be.

This is not a huge cause for alarm, however. MSN, Google, and other big names typically create "well behaved" spiders that pay attention to robots.txt files:

http : // blog . eukhost . com /webhosting/how-to-stop-search-engine-spider-from-crawling-specific-part-on-a-site/

Even if they did not, it is usually a trivial matter to identify them, as your screenshot above shows. From that much, Magister Ventrue can easily setup the website to restrict what these spiders can view, and even serve alternative content. My guess is that he already has these well under control.

In the case of misbehaving spiders, it is only slightly more tricky to identify and cut them off. These exhibit telltale signs, too. A host requesting resources that would not typically be directly accessed, such as a host jumping straight to my profile page, rather than navigating to it as a human would naturally do is one. Other signs might appear in the logs, too:

http : // www . jafsoft . com /searchengines / spider_hunting . html
_________________________

Refuse to die.

Top
#407501 - 01/03/10 07:35 PM Re: Crafty Spider [Re: TheAbysmal]
Delta Offline
CoS Member

Registered: 12/18/07
Posts: 6750
Loc: Nar
Fascinating! Thanks for the information and links, I wonder how long it will be until a spider asks what's wrong with the term "LaVeyan"...
_________________________




Top
#407536 - 01/03/10 11:04 PM Re: Crafty Spider [Re: Delta]
TheAbysmal Offline


Registered: 09/22/06
Posts: 1019
If there were a halfway sensible way to segue from "What's wrong with the term LaVeyan?" to "I am horny. Check me out my webcam!", they already would. wink
_________________________

Refuse to die.

Top


Forum Stats
12095 Members
73 Forums
43862 Topics
405463 Posts

Max Online: 197 @ 10/04/11 06:49 AM
Advertisements