Robots.txt System Set for Major Revamp

The desire for greater control over how search engines index Web sites is driving an effort by publishers to revise a 13-year-old technology for restricting access

Posted by redwolf on 30 November 2007 | Permalink