The Input formats - filter patch has landed. I still need to make update instructions for modules and update the hook docs.

Here's an overview of the changes:
1) Multiple Input formats: they are complete filter configurations (what filters to use, in what order and with which settings). Input formats are admin-definable, and usage of them is role-dependant. For example, you can set it up so that regular users can only use limited HTML, while admins can free HTML without any tag limitations.
The input format can be chosen per content item (nodes, comments, blocks, ...) when you add/edit them. If only a single format is available, there is no choice, and nothing changes with before.

The default install (and the upgrade) contains a basic set of formats which should satisfy the average user's needs.

2) Filters have toggles
Because now you might want to enable a filter only on some input formats, an explicit toggle is provided by the filter system. Modules do not need to worry about it and filters that still have their own on/off switch should get rid of it.

3) Multiple filters per module
This was necessary to accomodate the next change, and it's also a logical extension of the filter system.

4) Embedded PHP is now a filter
Thanks to the multiple input formats, I was able to move the 'embedded PHP' feature from block.module, page.module and book.module into a simple filter which executes PHP code. This filter is part of filter.module, and by default there is an input format 'PHP', restricted to the administrator only, which contains this filter.
This change means that block.module now passes custom block contents through the filter system.
As well as from reducing code duplication and avoiding two type selectors for page/book nodes, you can now combine PHP code with other filters.

5) User-supplied PHP code now requires <?php ?> tags.
This is required for teasers to work with PHP code. Because PHP evaluation is now just another step in the filter process, we can't do this. Also, because teasers are generated before filtering, this would result in errors when the teaser generation would cut off a piece of PHP code.

Also, regular PHP syntax explicitly includes the <?php ?> tags for PHP files, so it makes sense to use the same convention for embedded PHP in Drupal.

6) Filter caching was added.
Benchmarking shows that even for a simple setup (basic html filtering + legacy URL rewriting), filtercache can offer speedups. Unlike the old filtercache, this uses the normal cache table.

7) Filtertips were moved from help into a hook_filter_tips(). This was required to accomodate the fact that there are multiple filters per module, and that filter settings are format dependant. Shoehorning filter tips into _help was ugly and silly. The display of the filter tips is done through the input format selector, so filter_tips_short() no longer exists.

8) A more intelligent linebreak convertor was added, which doesn't stop working if you use block-level tags and which adds <p> tags.

3 files changed, 192 insertions, 5 deletions

diff --git a/database/database.mysql b/database/database.mysql
index b630c1734..2e91654f2 100644
--- a/database/database.mysql
+++ b/database/database.mysql
@@ -135,7 +135,6 @@ CREATE TABLE book (
   nid int(10) unsigned NOT NULL default '0',
   parent int(10) NOT NULL default '0',
   weight tinyint(3) NOT NULL default '0',
-  format tinyint(2) default '0',
   log longtext,
   PRIMARY KEY  (nid),
   KEY parent (parent)
@@ -150,7 +149,7 @@ CREATE TABLE boxes (
   title varchar(64) NOT NULL default '',
   body longtext,
   info varchar(128) NOT NULL default '',
-  type tinyint(2) NOT NULL default '0',
+  format int(4) NOT NULL default '0',
   PRIMARY KEY  (bid),
   UNIQUE KEY title (title),
   UNIQUE KEY info (info)
@@ -184,6 +183,7 @@ CREATE TABLE comments (
   timestamp int(11) NOT NULL default '0',
   score mediumint(9) NOT NULL default '0',
   status tinyint(3) unsigned NOT NULL default '0',
+  format int(4) NOT NULL default '0',
   thread varchar(255) NOT NULL,
   users longtext,
   name varchar(60) default NULL,
@@ -208,13 +208,27 @@ CREATE TABLE directory (
 ) TYPE=MyISAM;
 
 --
+-- Table structure for table 'filter_formats'
+--
+
+CREATE TABLE filter_formats (
+  format int(4) NOT NULL default '0' auto_increment,
+  name varchar(255) NOT NULL default '',
+  roles varchar(255) NOT NULL default '',
+  cache tinyint(2) NOT NULL default '0',
+  PRIMARY KEY format (format)
+) TYPE=MyISAM;
+
+--
 -- Table structure for table 'filters'
 --
 
 CREATE TABLE filters (
+  format int(4) NOT NULL default '0',
   module varchar(64) NOT NULL default '',
+  delta tinyint(2) DEFAULT '0' NOT NULL,
   weight tinyint(2) DEFAULT '0' NOT NULL,
-  KEY module (module)
+  INDEX (weight)
 ) TYPE=MyISAM;
 
 --
@@ -330,6 +344,7 @@ CREATE TABLE node (
   body longtext NOT NULL,
   revisions longtext NOT NULL,
   sticky int(2) NOT NULL default '0',
+  format int(4) NOT NULL default '0',
   PRIMARY KEY  (nid),
   KEY node_type (type(4)),
   KEY node_title_type (title,type(4)),
@@ -362,7 +377,6 @@ CREATE TABLE node_access (
 CREATE TABLE page (
   nid int(10) unsigned NOT NULL default '0',
   link varchar(128) NOT NULL default '',
-  format tinyint(2) NOT NULL default '0',
   description varchar(128) NOT NULL default '',
   PRIMARY KEY  (nid)
 ) TYPE=MyISAM;
@@ -700,3 +714,12 @@ REPLACE blocks SET module = 'user', delta = '1', status = '1';
 INSERT INTO sequences (name, id) VALUES ('menu_mid', 1);
 
 INSERT INTO node_access VALUES (0, 0, 'all', 1, 0, 0);
+
+INSERT INTO filter_formats VALUES (1,'Filtered HTML',',1,2,',1);
+INSERT INTO filter_formats VALUES (2,'PHP code','',0);
+INSERT INTO filter_formats VALUES (3,'Full HTML','',1);
+INSERT INTO filters VALUES (1,'filter',0,0);
+INSERT INTO filters VALUES (1,'filter',3,1);
+INSERT INTO filters VALUES (2,'filter',1,0);
+INSERT INTO filters VALUES (3,'filter',3,0);
+INSERT INTO variable (name,value) VALUES ('filter_html_1','i:1;');
diff --git a/database/database.pgsql b/database/database.pgsql
index 289a2e275..669889f7f 100644
--- a/database/database.pgsql
+++ b/database/database.pgsql
@@ -694,6 +694,16 @@ INSERT INTO permission VALUES (2,'access comments, access content, post comments
 INSERT INTO blocks(module,delta,status) VALUES('user', '0', '1');
 INSERT INTO blocks(module,delta,status) VALUES('user', '1', '1');
 
+INSERT INTO filter_formats VALUES (1,'Filtered HTML',',1,2,',1);
+INSERT INTO filter_formats VALUES (2,'PHP code','',0);
+INSERT INTO filter_formats VALUES (3,'Full HTML','',1));
+INSERT INTO filters VALUES (1,'filter',0,0);
+INSERT INTO filters VALUES (1,'filter',3,1);
+INSERT INTO filters VALUES (2,'filter',1,0);
+INSERT INTO filters VALUES (3,'filter',3,0);
+INSERT INTO variable (name,value) VALUES ('filter_html_1','i:1;');
+
+
 ---
 --- Functions
 ---
diff --git a/database/updates.inc b/database/updates.inc
index e2faae044..be5a59b81 100644
--- a/database/updates.inc
+++ b/database/updates.inc
@@ -69,7 +69,8 @@ $sql_updates = array(
   "2004-07-30" => "update_95",
   "2004-08-04" => "update_96",
   "2004-08-06" => "update_97",
-  "2004-08-07" => "update_98"
+  "2004-08-07" => "update_98",
+  "2004-08-09" => "update_99"
 );
 
 function update_32() {
@@ -1308,6 +1309,159 @@ function update_98() {
   return array();
 }
 
+function update_99() {
+  // Filter patch - Multiple input formats
+  $ret = array();
+
+  /*
+  ** Load the list of PHP book and page nodes.
+  */
+  $php_nodes = array();
+  $res = db_query("SELECT nid FROM {book} WHERE format = 1");
+  while ($book = db_fetch_object($res)) {
+    $php_nodes[] = $book->nid;
+  }
+  $res = db_query("SELECT nid FROM {page} WHERE format = 1");
+  while ($page = db_fetch_object($res)) {
+    $php_nodes[] = $page->nid;
+  }
+
+  /*
+  ** Apply database changes
+  */
+  if ($GLOBALS['db_type'] == 'mysql') {
+    // Filters table
+    $ret[] = update_sql("ALTER TABLE {filters} ADD format int(4) NOT NULL default '0'");
+    $ret[] = update_sql("ALTER TABLE {filters} ADD delta tinyint(2) NOT NULL default '0'");
+
+    // Filter_formats table
+    $ret[] = update_sql("CREATE TABLE {filter_formats} (
+      format int(4) NOT NULL auto_increment,
+      name varchar(255) NOT NULL default '',
+      roles varchar(255) NOT NULL default '',
+      cache tinyint(2) NOT NULL default '1',
+      PRIMARY KEY  (format)
+    )");
+
+    // Store formats in nodes, comments and boxes
+    $ret[] = update_sql("ALTER TABLE {boxes} CHANGE type format int(4) NOT NULL default '0'");
+    $ret[] = update_sql("ALTER TABLE {comments} ADD format int(4) NOT NULL default '0'");
+    $ret[] = update_sql("ALTER TABLE {node} ADD format int(4) NOT NULL default '0'");
+
+    // Get rid of the old book/page type info
+    $ret[] = update_sql("ALTER TABLE {book} DROP format");
+    $ret[] = update_sql("ALTER TABLE {page} DROP format");
+  }
+  else if ($GLOBALS['db_type'] == 'pgsql') {
+    // TODO: add pgsql equivalent. Whoever does this should pay attention that
+    // the keys/indices for the 'filters' table are correct. There was some
+    // inconsistency between the MySQL and PGSQL version before.
+  }
+
+  // Initialize all nodes and comments to the legacy format (see below)
+  $ret[] = update_sql("UPDATE {node} SET format = 1");
+  $ret[] = update_sql("UPDATE {comments} SET format = 1");
+
+  // Set format to PHP for the old PHP book/page nodes.
+  if (count($php_nodes)) {
+    $ret[] = update_sql("UPDATE {node} SET format = 2 WHERE nid IN (". implode(',', $php_nodes) .")");
+  }
+
+  // Boxes now use the filtering system as well.
+  // Type 0 (HTML) maps to Format 3 (Full HTML).
+  // Type 1 (PHP) maps to Format 2 (PHP).
+  $ret[] = update_sql("UPDATE {boxes} SET format = 3 - format");
+
+  /*
+  ** Update PHP content to use <?php ?> tags.
+  */
+  if ($GLOBALS['db_type'] == 'mysql') {
+    $ret[] = update_sql("UPDATE {node} SET teaser = CONCAT('<?php ', teaser) WHERE format = 2");
+    $ret[] = update_sql("UPDATE {node} SET body = CONCAT('<?php ', body) WHERE format = 2");
+    $ret[] = update_sql("UPDATE {boxes} SET body = CONCAT('<?php ', body) WHERE format = 2");
+  }
+  else if ($GLOBALS['db_type'] == 'pgsql') {
+    // TODO: someone needs to verify if this works.
+    $ret[] = update_sql("UPDATE {node} SET teaser = '<?php ' || teaser WHERE format = 2");
+    $ret[] = update_sql("UPDATE {node} SET body = '<?php ' || body WHERE format = 2");
+    $ret[] = update_sql("UPDATE {boxes} SET body = '<?php ' || body WHERE format = 2");
+  }
+
+
+  /*
+  ** We now set up some input formats. One of these is a 'legacy' format which
+  ** tries to preserve as much settings as possible from before the patch.
+  ** The other two are 'PHP code' and 'Full HTML'.
+  */
+
+  // We pick an appropriate name for the legacy format.
+  $old_html_filter = variable_get('filter_html', 0);
+  if ($old_html_filter == FILTER_HTML_ESCAPE) {
+    $default = 'Plain text';
+  }
+  else {
+    $default = 'Filtered HTML';
+  }
+  // Make sure the legacy format is accessible to all roles
+  $all_roles = array_keys(user_roles());
+  $ret[] = update_sql("INSERT INTO {filter_formats} VALUES (1,'$default',',". implode(',', $all_roles) .",',1)");
+
+  // Determine which roles have the old 'create php content' permission.
+  $res = db_query("SELECT rid FROM {permission} WHERE perm LIKE '%create php content%'");
+  $php_roles = array();
+  while ($role = db_fetch_object($res)) {
+    $php_roles[] = $role->rid;
+  }
+  $ret[] = update_sql("INSERT INTO {filter_formats} VALUES (2,'PHP code','". implode(',', $php_roles) .",',0)");
+
+  // This is a 'Full HTML' format which allows all HTML without restrictions.
+  $ret[] = update_sql("INSERT INTO {filter_formats} VALUES (3,'Full HTML','',1)");
+
+  // Set the default format to the legacy format
+  variable_set('filter_default_format', 1);
+
+  // Put the old filters into the legacy format
+  $ret[] = update_sql("UPDATE {filters} SET format = 1");
+
+  // Add filter.module's standard filters (these used to be hardcoded).
+  if (!variable_get('rewrite_old_urls', 0)) {
+    $ret[] = update_sql("DELETE FROM {filters} WHERE module = 'filter'");
+  }
+  else {
+    $ret[] = update_sql("UPDATE {filters} SET delta = 2 WHERE module ='filter'");
+  }
+  if ($old_html_filter != 0) {
+    $ret[] = update_sql("INSERT INTO {filters} (format, module, delta, weight) VALUES (1,'filter',0,0)"); // HTML tag/style filter
+  }
+  $ret[] = update_sql("INSERT INTO {filters} (format, module, delta, weight) VALUES (1,'filter',3,1)"); // Linebreak filter
+  $ret[] = update_sql("INSERT INTO {filters} (format, module, delta, weight) VALUES (2,'filter',1,0)"); // PHP evaluator
+  $ret[] = update_sql("INSERT INTO {filters} (format, module, delta, weight) VALUES (3,'filter',3,0)"); // Linebreak filter
+
+  // Migrate the settings for all core/contrib filtering modules into the legacy
+  // format.
+  $migrate = array('filter_html', // filter.module
+                   'allowed_html',
+                   'filter_style',
+                   'anyfilter_regexps', // anyfilter.module
+                   'htmlcorrector_smartclose', // htmlcorrector.module
+                   'htmlcorrector_xhtmlify',
+                   'htmlcorrector_valueentities',
+                   'project_filter', // project.module
+                   'latex_filter_link' // latex.module
+                   );
+
+  foreach ($migrate as $variable) {
+    $value = variable_get($variable, NULL);
+    if ($value != NULL) {
+      variable_set($variable .'_1', $value);
+      variable_del($variable);
+    }
+  }
+
+  return $ret;
+}
+
+
 function update_sql($sql) {
   $edit = $_POST["edit"];
   $result = db_query($sql);