Internet » Cold Fusion » Verity

VERITY SPIDER DOWNLOAD

ColdFusion Spider
Introducing the spider, Teleport Pro Teleport is designed to download Web sites to your hard drive for offline browsing, but we're going to use it to help create our searchable Web spider. To create our searchable Web spider we're going to build a verity collection from a locally stored mirror of a Web site. Creating the Web spider Search Templates Now that we have a verity collection populated with files mirrored from the HTML Goodies Web site, we need to create the templates to receive ...
http://www.sys-con.com/story/?storyid=41800&DE=1 [cache]

How to Defeat Bad Web Robots With Apache
How to Defeat Bad Web Robots With Apache How to Defeat Bad Web Robots With Apache How to Defeat Bad Web Robots With Apache By Lee Killough NOTE: This page has become somewhat outdated :-( I first wrote it in 2001. Create a script which records bad hosts (bad.pl): #!/usr/bin/perl $| = 1; $host = $ENV{'REMOTE_HOST'}; $addr = $ENV{'REMOTE_ADDR'}; $agent = $ENV{'HTTP_USER_AGENT'}; $url = $ENV{'REQUEST_URI'}; $mapfile = "/www/. com$ [NC] RewriteRule ^(/cgi-bin/|/images/) /cgi-bin/bad.pl [L,T=...
http://www.leekillough.com/robots.html [cache]