[SpamCop-Geeks] Re: Scooter

[Michael Lefevre]

Chris H. wrote:
> I'm getting some page not found errors so I went through the logs and 
> found:
> 
[snip]
> Additionally 'Scooter' in the past was looking for:
> 
> /cp/scripts/formmail-doc/example_src.html
> /cp/scripts/formmail.html
> /cp/scripts/frames.html
> /cp/scripts/alldetails.html
> /cp/scripts/top.html
> 
> The IP is assigned to Alta Vista.
> 
> The files are completely nonexistent and are not referenced anywhere 
> on my web site.
 
it is weird. that is what spiders do yes.  however, it's not just links on
your site, it's links on any site. and they do remember things they've
seen in the past.  are those files linked from any other site (you could
try using Alta Vista or Google to look for pages that link to your site)?
have those files ever existed on that domain? or on a server on that IP?

> Is this something I should be concerned about, since it's Alta Vista
> or should I be doing something about it???  I currently don't have a
> robots.txt or installed but I don't know if it will do any good for
> them as they don't seem to search for it anyway.

they don't? scooter goes over our site regularly, and it does check the
robots.txt.  Not necessarily on each run - it checks a random bunch of
files at random intervals - it's generally accepted that stuff in
robots.txt may take a week or so to kick in, in the same way that any
changes generally take some time for a crawler to notice them.

the things it's looking for aren't actually what abusers look for -
formmail scripts are not generally called formmail.html, they are called
formmail.cgi or formmail.pl, and those (in the root, or in /cgi/ or in
/cgi-bin/) are what abusers generally scan for.  so it's nothing to be
concerned about security-wise, particularly as it's alta vista doing it.
but it is rather odd.

-- 
Michael