tokenizer functions
The tokenizer functions provide an interface to the PHP tokenizer embedded in the Zend Engine. Using these functions we can write our own PHP source analyzing or modification tools without having to deal with the language specification at the lexical level.
<?php
if (!defined(’T_ML_COMMENT’)) {
define(’T_ML_COMMENT’, T_COMMENT);
} else {
define(’T_DOC_COMMENT’, T_ML_COMMENT);
}
$source = file_get_contents(’example.php’);
$tokens = token_get_all($source);
foreach ($tokens as $token) {
if (is_string($token)) {
// simple 1-character token
echo $token;
} else {
// token array
list($id, $text) = $token;
switch ($id) {
case T_COMMENT:
case T_ML_COMMENT: // we’ve defined this
case T_DOC_COMMENT: // and this
break;
default:
// anything else -> output “as is”
echo $text;
break;
}
}
}
?>
