Tokenizers
A detector names one tokenizer. The tokenizer supplies grammar classes and fields; the detector's lexicon adds classes to words, and its patterns or fingerprints decide a match (Detectors and patterns). Tokenizers run in linear time and charge their work to the transaction's budget.
Sql
Splits SQL into words, strings, numbers, comments and punctuation, with a libinjection-compatible fingerprint mode. Since 1.0.0-alpha4.
Starting contexts: AsIs, FoldQuotes.
Grammar classes: Comment, Comparison, Equal, LeftParenthesis, Number, RightParenthesis, Semicolon, String, Symbol, Word.
Token fields: none.
Fingerprint mode: yes.
| Option | Allowed values | Default | Range |
|---|---|---|---|
contexts |
AsIs, FoldQuotes |
AsIs |
— |
versionedCommentDigits |
— | 6 |
0–6 |
Html
Tokenizes HTML tags, attributes, text and comments in the chosen starting context, with a libinjection-compatible XSS fingerprint mode. Since 1.0.0-alpha4.
Starting contexts: Data, ValueNoQuote, ValueSingleQuote, ValueDoubleQuote, ValueBackQuote.
Grammar classes: Attribute, Comment, Doctype, Tag, Text.
Token fields: closing, hasValue, localName, name, value.
Fingerprint mode: yes.
| Option | Allowed values | Default | Range |
|---|---|---|---|
contexts |
Data, ValueNoQuote, ValueSingleQuote, ValueDoubleQuote, ValueBackQuote |
Data |
— |
JavaScript
Splits JavaScript into identifiers, strings, templates, numbers, regular expressions and punctuators. Since 1.0.0-alpha4.
Starting contexts: Code, SingleQuoteString, DoubleQuoteString, Template.
Grammar classes: Comment, Identifier, LeftBracket, LeftParenthesis, Number, Punctuator, RegularExpression, RightBracket, RightParenthesis, String, Template.
Token fields: none.
Fingerprint mode: no.
| Option | Allowed values | Default | Range |
|---|---|---|---|
contexts |
Code, SingleQuoteString, DoubleQuoteString, Template |
Code |
— |
Shell
Splits Unix shell input into separators, substitutions, redirects, variables and words. Since 1.0.0-alpha4.
Starting contexts: Command.
Grammar classes: LeftParenthesis, Redirect, Separator, Substitution, Symbol, Variable, Word.
Token fields: basename, nonBlank, target.
Fingerprint mode: no.
| Option | Allowed values | Default | Range |
|---|---|---|---|
contexts |
Command |
Command |
— |
WindowsCmd
Splits Windows cmd input into separators, variables, redirects and words, removing caret escapes. Since 1.0.0-alpha4.
Starting contexts: Command.
Grammar classes: LeftParenthesis, Redirect, Separator, Symbol, Variable, Word.
Token fields: basename, target.
Fingerprint mode: no.
| Option | Allowed values | Default | Range |
|---|---|---|---|
contexts |
Command |
Command |
— |
PowerShell
Splits PowerShell into cmdlets, parameters, variables, strings, invocations and subexpressions. Since 1.0.0-alpha4.
Starting contexts: Command.
Grammar classes: Cmdlet, Invocation, Parameter, Pipe, Separator, String, Subexpression, Symbol, Variable, Word.
Token fields: none.
Fingerprint mode: no.
| Option | Allowed values | Default | Range |
|---|---|---|---|
contexts |
Command |
Command |
— |
LdapFilter
Splits RFC 4515 LDAP filters, including values placed inside an existing filter. Since 1.0.0-alpha4.
Starting contexts: Filter, InsideValue.
Grammar classes: And, Attribute, Escape, FilterType, LeftParenthesis, Not, Or, RightParenthesis, Value, Wildcard.
Token fields: none.
Fingerprint mode: no.
| Option | Allowed values | Default | Range |
|---|---|---|---|
contexts |
Filter, InsideValue |
Filter |
— |
XPath
Splits XPath and XQuery expressions, including values that break out of string literals. Since 1.0.0-alpha4.
Starting contexts: Expression, SingleQuoteString, DoubleQuoteString.
Grammar classes: At, Axis, Comma, DoubleSlash, LeftBracket, LeftParenthesis, Name, Number, Operator, RightBracket, RightParenthesis, Slash, String.
Token fields: none.
Fingerprint mode: no.
| Option | Allowed values | Default | Range |
|---|---|---|---|
contexts |
Expression, SingleQuoteString, DoubleQuoteString |
Expression |
— |
Template
Splits enabled template delimiter families into calls, attribute access, filters and literals. Since 1.0.0-alpha4.
Starting contexts: Text.
Grammar classes: AttributeAccess, Call, Close, Filter, Identifier, Number, Open, Operator, String, Subscript, Text.
Token fields: family.
Fingerprint mode: no.
| Option | Allowed values | Default | Range |
|---|---|---|---|
contexts |
Text |
Text |
— |
families |
Jinja, Erb, FreeMarker, Velocity, Thymeleaf, SpringEl, Ognl, Handlebars, Razor, Smarty |
Jinja, Erb, FreeMarker, Velocity, Thymeleaf, SpringEl, Ognl, Handlebars, Razor, Smarty |
— |