diff options
| author | Dries Buytaert <dries@buytaert.net> | 2006-08-14 10:41:52 +0000 |
|---|---|---|
| committer | Dries Buytaert <dries@buytaert.net> | 2006-08-14 10:41:52 +0000 |
| commit | 5b455a2e0fdaec6b5b9c324ffbbe4bddd8c617ee (patch) | |
| tree | 9580f04bc045ae06573679b379b2ea3f340e5cc5 | |
| parent | 20201f4ddc3d9a52671345ffac5eed8655052589 (diff) | |
| download | brdo-5b455a2e0fdaec6b5b9c324ffbbe4bddd8c617ee.tar.gz brdo-5b455a2e0fdaec6b5b9c324ffbbe4bddd8c617ee.tar.bz2 | |
- Patch #75916 by Robert: re-introduced default robot.txt.
| -rw-r--r-- | CHANGELOG.txt | 2 | ||||
| -rw-r--r-- | robots.txt | 57 |
2 files changed, 58 insertions, 1 deletions
diff --git a/CHANGELOG.txt b/CHANGELOG.txt index 55a689b6f..ee539a5fb 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -24,7 +24,7 @@ Drupal x.x.x, xxxx-xx-xx (development version) * optionally allow people to cancel their vote. - distributed authentication: * added default server option. -- fixed critical SQL issue, see SA-2006-005 +- added default robots.txt to control crawlers. - database API: * added db_table_exists(). - blogapi module: diff --git a/robots.txt b/robots.txt new file mode 100644 index 000000000..c41c1a424 --- /dev/null +++ b/robots.txt @@ -0,0 +1,57 @@ +# robots.txt +# +# This file aims to prevent the crawling and idexing of certain parts +# of your site by web crawlers and spiders run by sites like Yahoo! +# and Google. By telling these "robots" where not to go on your site, +# you save bandwidth and server resources. +# +# For more information about the robots.txt standard, see: +# http://www.robotstxt.org/wc/robots.html +# +# See this site for syntax checking: +# http://www.sxw.org.uk/computing/robots/check.html + +User-agent: * +Crawl-delay: 10 +# Directories +Disallow: /files/ +Disallow: /database/ +Disallow: /includes/ +Disallow: /misc/ +Disallow: /modules/ +Disallow: /sites/ +Disallow: /themes/ +Disallow: /scripts/ +Disallow: /updates/ +Disallow: /profiles/ +# Files +Disallow: /xmlrpc.php +Disallow: /cron.php +Disallow: /update.php +Disallow: /install.php +Disallow: /INSTALL.mysql.txt +Disallow: /INSTALL.pgsql.txt +Disallow: /CHANGELOG.txt +Disallow: /MAINTAINERS.txt +Disallow: /LICENSE.txt +Disallow: /UPGRADE.txt +# Paths (clean URLs) +Disallow: /admin/ +Disallow: /aggregator/ +Disallow: /comment/reply/ +Disallow: /contact/ +Disallow: /logout/ +Disallow: /node/add/ +Disallow: /search/ +Disallow: /user/register/ +Disallow: /user/password/ +# Paths (no clean URLs) +Disallow: /?q=admin/ +Disallow: /?q=aggregator/ +Disallow: /?q=comment/reply/ +Disallow: /?q=contact/ +Disallow: /?q=logout/ +Disallow: /?q=node/add/ +Disallow: /?q=search/ +Disallow: /?q=user/password/ +Disallow: /?q=user/register/ |