diff options
Diffstat (limited to 'doc/pcre2test.txt')
| -rw-r--r-- | doc/pcre2test.txt | 315 |
1 files changed, 179 insertions, 136 deletions
diff --git a/doc/pcre2test.txt b/doc/pcre2test.txt index d9dbc4d..cfa0baa 100644 --- a/doc/pcre2test.txt +++ b/doc/pcre2test.txt @@ -67,10 +67,10 @@ INPUT ENCODING For maximum portability, therefore, it is safest to avoid non-printing characters in pcre2test input files. There is a facility for specifying - a pattern's characters as hexadecimal pairs, thus making it possible to - include binary zeroes in a pattern for testing purposes. Subject lines - are processed for backslash escapes, which makes it possible to include - any data value. + some or all of a pattern's characters as hexadecimal pairs, thus making + it possible to include binary zeroes in a pattern for testing purposes. + Subject lines are processed for backslash escapes, which makes it pos- + sible to include any data value. COMMAND LINE OPTIONS @@ -138,6 +138,13 @@ COMMAND LINE OPTIONS is done using the pcre2_dfa_match() function instead of the default pcre2_match(). + -error number[,number,...] + Call pcre2_get_error_message() for each of the error numbers + in the comma-separated list, display the resulting messages + on the standard output, then exit with zero exit code. The + numbers may be positive or negative. This is a convenience + facility for PCRE2 maintainers. + -help Output a brief summary these options and then exit. -i Behave as if each pattern has the /info modifier; information @@ -296,10 +303,11 @@ COMMAND LINES wrong file. #pop [<modifiers>] + #popcopy [<modifiers>] - This command is used to manipulate the stack of compiled patterns, as - described in the section entitled "Saving and restoring compiled pat- - terns" below. + These commands are used to manipulate the stack of compiled patterns, + as described in the section entitled "Saving and restoring compiled + patterns" below. #save <filename> @@ -505,7 +513,7 @@ PATTERN MODIFIERS debug same as info,fullbincode fullbincode show binary code with lengths /I info show info about compiled pattern - hex pattern is coded in hexadecimal + hex unquoted characters are hexadecimal jit[=<number>] use JIT jitfast use JIT fast path jitverify verify JIT use @@ -516,7 +524,9 @@ PATTERN MODIFIERS null_context compile with a NULL context parens_nest_limit=<n> set maximum parentheses depth posix use the POSIX API + posix_nosub use the POSIX API with REG_NOSUB push push compiled pattern onto the stack + pushcopy push a copy onto the stack stackguard=<number> test the stackguard feature tables=[0|1|2] select internal tables @@ -591,59 +601,70 @@ PATTERN MODIFIERS testing that pcre2_compile() behaves correctly in this case (it uses default values). - Specifying a pattern in hex + Specifying pattern characters in hexadecimal - The hex modifier specifies that the characters of the pattern are to be - interpreted as pairs of hexadecimal digits. White space is permitted - between pairs. For example: + The hex modifier specifies that the characters of the pattern, except + for substrings enclosed in single or double quotes, are to be inter- + preted as pairs of hexadecimal digits. This feature is provided as a + way of creating patterns that contain binary zeros and other non-print- + ing characters. White space is permitted between pairs of digits. For + example, this pattern contains three characters: /ab 32 59/hex - This feature is provided as a way of creating patterns that contain - binary zero and other non-printing characters. By default, pcre2test - passes patterns as zero-terminated strings to pcre2_compile(), giving - the length as PCRE2_ZERO_TERMINATED. However, for patterns specified in - hexadecimal, the actual length of the pattern is passed. + Parts of such a pattern are taken literally if quoted. This pattern + contains nine characters, only two of which are specified in hexadeci- + mal: + + /ab "literal" 32/hex + + Either single or double quotes may be used. There is no way of includ- + ing the delimiter within a substring. + + By default, pcre2test passes patterns as zero-terminated strings to + pcre2_compile(), giving the length as PCRE2_ZERO_TERMINATED. However, + for patterns specified with the hex modifier, the actual length of the + pattern is passed. Generating long repetitive patterns - Some tests use long patterns that are very repetitive. Instead of cre- - ating a very long input line for such a pattern, you can use a special - repetition feature, similar to the one described for subject lines - above. If the expand modifier is present on a pattern, parts of the + Some tests use long patterns that are very repetitive. Instead of cre- + ating a very long input line for such a pattern, you can use a special + repetition feature, similar to the one described for subject lines + above. If the expand modifier is present on a pattern, parts of the pattern that have the form \[<characters>]{<count>} are expanded before the pattern is passed to pcre2_compile(). For exam- ple, \[AB]{6000} is expanded to "ABAB..." 6000 times. This construction - cannot be nested. An initial "\[" sequence is recognized only if "]{" - followed by decimal digits and "}" is found later in the pattern. If + cannot be nested. An initial "\[" sequence is recognized only if "]{" + followed by decimal digits and "}" is found later in the pattern. If not, the characters remain in the pattern unaltered. - If part of an expanded pattern looks like an expansion, but is really + If part of an expanded pattern looks like an expansion, but is really part of the actual pattern, unwanted expansion can be avoided by giving two values in the quantifier. For example, \[AB]{6000,6000} is not rec- ognized as an expansion item. - If the info modifier is set on an expanded pattern, the result of the + If the info modifier is set on an expanded pattern, the result of the expansion is included in the information that is output. JIT compilation - Just-in-time (JIT) compiling is a heavyweight optimization that can - greatly speed up pattern matching. See the pcre2jit documentation for - details. JIT compiling happens, optionally, after a pattern has been - successfully compiled into an internal form. The JIT compiler converts + Just-in-time (JIT) compiling is a heavyweight optimization that can + greatly speed up pattern matching. See the pcre2jit documentation for + details. JIT compiling happens, optionally, after a pattern has been + successfully compiled into an internal form. The JIT compiler converts this to optimized machine code. It needs to know whether the match-time options PCRE2_PARTIAL_HARD and PCRE2_PARTIAL_SOFT are going to be used, - because different code is generated for the different cases. See the - partial modifier in "Subject Modifiers" below for details of how these + because different code is generated for the different cases. See the + partial modifier in "Subject Modifiers" below for details of how these options are specified for each match attempt. - JIT compilation is requested by the /jit pattern modifier, which may + JIT compilation is requested by the /jit pattern modifier, which may optionally be followed by an equals sign and a number in the range 0 to - 7. The three bits that make up the number specify which of the three + 7. The three bits that make up the number specify which of the three JIT operating modes are to be compiled: 1 compile JIT code for non-partial matching @@ -660,31 +681,31 @@ PATTERN MODIFIERS 6 soft and hard partial matching only 7 all three modes - If no number is given, 7 is assumed. The phrase "partial matching" + If no number is given, 7 is assumed. The phrase "partial matching" means a call to pcre2_match() with either the PCRE2_PARTIAL_SOFT or the - PCRE2_PARTIAL_HARD option set. Note that such a call may return a com- + PCRE2_PARTIAL_HARD option set. Note that such a call may return a com- plete match; the options enable the possibility of a partial match, but - do not require it. Note also that if you request JIT compilation only - for partial matching (for example, /jit=2) but do not set the partial - modifier on a subject line, that match will not use JIT code because + do not require it. Note also that if you request JIT compilation only + for partial matching (for example, /jit=2) but do not set the partial + modifier on a subject line, that match will not use JIT code because none was compiled for non-partial matching. - If JIT compilation is successful, the compiled JIT code will automati- - cally be used when an appropriate type of match is run, except when - incompatible run-time options are specified. For more details, see the - pcre2jit documentation. See also the jitstack modifier below for a way + If JIT compilation is successful, the compiled JIT code will automati- + cally be used when an appropriate type of match is run, except when + incompatible run-time options are specified. For more details, see the + pcre2jit documentation. See also the jitstack modifier below for a way of setting the size of the JIT stack. - If the jitfast modifier is specified, matching is done using the JIT - "fast path" interface, pcre2_jit_match(), which skips some of the san- - ity checks that are done by pcre2_match(), and of course does not work - when JIT is not supported. If jitfast is specified without jit, jit=7 + If the jitfast modifier is specified, matching is done using the JIT + "fast path" interface, pcre2_jit_match(), which skips some of the san- + ity checks that are done by pcre2_match(), and of course does not work + when JIT is not supported. If jitfast is specified without jit, jit=7 is assumed. - If the jitverify modifier is specified, information about the compiled - pattern shows whether JIT compilation was or was not successful. If - jitverify is specified without jit, jit=7 is assumed. If JIT compila- - tion is successful when jitverify is set, the text "(JIT)" is added to + If the jitverify modifier is specified, information about the compiled + pattern shows whether JIT compilation was or was not successful. If + jitverify is specified without jit, jit=7 is assumed. If JIT compila- + tion is successful when jitverify is set, the text "(JIT)" is added to the first output line after a match or non match when JIT-compiled code was actually used in the match. @@ -695,18 +716,18 @@ PATTERN MODIFIERS /pattern/locale=fr_FR The given locale is set, pcre2_maketables() is called to build a set of - character tables for the locale, and this is then passed to pcre2_com- - pile() when compiling the regular expression. The same tables are used + character tables for the locale, and this is then passed to pcre2_com- + pile() when compiling the regular expression. The same tables are used when matching the following subject lines. The /locale modifier applies only to the pattern on which it appears, but can be given in a #pattern - command if a default is needed. Setting a locale and alternate charac- + command if a default is needed. Setting a locale and alternate charac- ter tables are mutually exclusive. Showing pattern memory - The /memory modifier causes the size in bytes of the memory used to - hold the compiled pattern to be output. This does not include the size - of the pcre2_code block; it is just the actual compiled data. If the + The /memory modifier causes the size in bytes of the memory used to + hold the compiled pattern to be output. This does not include the size + of the pcre2_code block; it is just the actual compiled data. If the pattern is subsequently passed to the JIT compiler, the size of the JIT compiled code is also output. Here is an example: @@ -717,31 +738,31 @@ PATTERN MODIFIERS Limiting nested parentheses - The parens_nest_limit modifier sets a limit on the depth of nested - parentheses in a pattern. Breaching the limit causes a compilation - error. The default for the library is set when PCRE2 is built, but - pcre2test sets its own default of 220, which is required for running + The parens_nest_limit modifier sets a limit on the depth of nested + parentheses in a pattern. Breaching the limit causes a compilation + error. The default for the library is set when PCRE2 is built, but + pcre2test sets its own default of 220, which is required for running the standard test suite. Limiting the pattern length - The max_pattern_length modifier sets a limit, in code units, to the + The max_pattern_length modifier sets a limit, in code units, to the length of pattern that pcre2_compile() will accept. Breaching the limit - causes a compilation error. The default is the largest number a + causes a compilation error. The default is the largest number a PCRE2_SIZE variable can hold (essentially unlimited). Using the POSIX wrapper API - The /posix modifier causes pcre2test to call PCRE2 via the POSIX wrap- - per API rather than its native API. This supports only the 8-bit - library. Note that it does not imply POSIX matching semantics; for - more detail see the pcre2posix documentation. When the POSIX API is - being used, the following pattern modifiers set options for the reg- - comp() function: + The /posix and posix_nosub modifiers cause pcre2test to call PCRE2 via + the POSIX wrapper API rather than its native API. When posix_nosub is + used, the POSIX option REG_NOSUB is passed to regcomp(). The POSIX + wrapper supports only the 8-bit library. Note that it does not imply + POSIX matching semantics; for more detail see the pcre2posix documenta- + tion. The following pattern modifiers set options for the regcomp() + function: caseless REG_ICASE multiline REG_NEWLINE - no_auto_capture REG_NOSUB dotall REG_DOTALL ) ungreedy REG_UNGREEDY ) These options are not part of ucp REG_UCP ) the POSIX standard @@ -758,23 +779,24 @@ PATTERN MODIFIERS been set, a large buffer is used. The aftertext and allaftertext subject modifiers work as described - below. All other modifiers cause an error. + below. All other modifiers are either ignored, with a warning message, + or cause an error. Testing the stack guard feature - The /stackguard modifier is used to test the use of pcre2_set_com- - pile_recursion_guard(), a function that is provided to enable stack - availability to be checked during compilation (see the pcre2api docu- - mentation for details). If the number specified by the modifier is + The /stackguard modifier is used to test the use of pcre2_set_com- + pile_recursion_guard(), a function that is provided to enable stack + availability to be checked during compilation (see the pcre2api docu- + mentation for details). If the number specified by the modifier is greater than zero, pcre2_set_compile_recursion_guard() is called to set - up callback from pcre2_compile() to a local function. The argument it - receives is the current nesting parenthesis depth; if this is greater + up callback from pcre2_compile() to a local function. The argument it + receives is the current nesting parenthesis depth; if this is greater than the value given by the modifier, non-zero is returned, causing the compilation to be aborted. Using alternative character tables - The value specified for the /tables modifier must be one of the digits + The value specified for the /tables modifier must be one of the digits 0, 1, or 2. It causes a specific set of built-in character tables to be passed to pcre2_compile(). This is used in the PCRE2 tests to check be- haviour with different character tables. The digit specifies the tables @@ -785,15 +807,15 @@ PATTERN MODIFIERS pcre2_chartables.c.dist 2 a set of tables defining ISO 8859 characters - In table 2, some characters whose codes are greater than 128 are iden- - tified as letters, digits, spaces, etc. Setting alternate character + In table 2, some characters whose codes are greater than 128 are iden- + tified as letters, digits, spaces, etc. Setting alternate character tables and a locale are mutually exclusive. Setting certain match controls The following modifiers are really subject modifiers, and are described - below. However, they may be included in a pattern's modifier list, in - which case they are applied to every subject line that is processed + below. However, they may be included in a pattern's modifier list, in + which case they are applied to every subject line that is processed with that pattern. They may not appear in #pattern commands. These mod- ifiers do not affect the compilation process. @@ -810,21 +832,25 @@ PATTERN MODIFIERS substitute_unknown_unset use PCRE2_SUBSTITUTE_UNKNOWN_UNSET substitute_unset_empty use PCRE2_SUBSTITUTE_UNSET_EMPTY - These modifiers may not appear in a #pattern command. If you want them + These modifiers may not appear in a #pattern command. If you want them as defaults, set them in a #subject command. Saving a compiled pattern - When a pattern with the push modifier is successfully compiled, it is - pushed onto a stack of compiled patterns, and pcre2test expects the - next line to contain a new pattern (or a command) instead of a subject + When a pattern with the push modifier is successfully compiled, it is + pushed onto a stack of compiled patterns, and pcre2test expects the + next line to contain a new pattern (or a command) instead of a subject line. This facility is used when saving compiled patterns to a file, as - described in the section entitled "Saving and restoring compiled pat- - terns" below. The push modifier is incompatible with compilation modi- - fiers such as global that act at match time. Any that are specified are - ignored, with a warning message, except for replace, which causes an - error. Note that, jitverify, which is allowed, does not carry through - to any subsequent matching that uses this pattern. + described in the section entitled "Saving and restoring compiled pat- + terns" below. If pushcopy is used instead of push, a copy of the com- + piled pattern is stacked, leaving the original as current, ready to + match the following input lines. This provides a way of testing the + pcre2_code_copy() function. The push and pushcopy modifiers are + incompatible with compilation modifiers such as global that act at + match time. Any that are specified are ignored (for the stacked copy), + with a warning message, except for replace, which causes an error. Note + that jitverify, which is allowed, does not carry through to any subse- + quent matching that uses a stacked pattern. SUBJECT MODIFIERS @@ -834,12 +860,13 @@ SUBJECT MODIFIERS Setting match options - The following modifiers set options for pcre2_match() or + The following modifiers set options for pcre2_match() or pcre2_dfa_match(). See pcreapi for a description of their effects. anchored set PCRE2_ANCHORED dfa_restart set PCRE2_DFA_RESTART dfa_shortest set PCRE2_DFA_SHORTEST + no_jit set PCRE2_NO_JIT no_utf_check set PCRE2_NO_UTF_CHECK notbol set PCRE2_NOTBOL notempty set PCRE2_NOTEMPTY @@ -848,20 +875,20 @@ SUBJECT MODIFIERS partial_hard (or ph) set PCRE2_PARTIAL_HARD partial_soft (or ps) set PCRE2_PARTIAL_SOFT - The partial matching modifiers are provided with abbreviations because + The partial matching modifiers are provided with abbreviations because they appear frequently in tests. - If the /posix modifier was present on the pattern, causing the POSIX + If the /posix modifier was present on the pattern, causing the POSIX wrapper API to be used, the only option-setting modifiers that have any - effect are notbol, notempty, and noteol, causing REG_NOTBOL, - REG_NOTEMPTY, and REG_NOTEOL, respectively, to be passed to regexec(). - Any other modifiers cause an error. + effect are notbol, notempty, and noteol, causing REG_NOTBOL, + REG_NOTEMPTY, and REG_NOTEOL, respectively, to be passed to regexec(). + The other modifiers are ignored, with a warning message. Setting match controls - The following modifiers affect the matching process or request addi- - tional information. Some of them may also be specified on a pattern - line (see above), in which case they apply to every subject line that + The following modifiers affect the matching process or request addi- + tional information. Some of them may also be specified on a pattern + line (see above), in which case they apply to every subject line that is matched against that pattern. aftertext show text after match @@ -898,6 +925,9 @@ SUBJECT MODIFIERS zero_terminate pass the subject as zero-terminated The effects of these modifiers are described in the following sections. + When matching via the POSIX wrapper API, the aftertext, allaftertext, + and ovector subject modifiers work as described below. All other modi- + fiers are either ignored, with a warning message, or cause an error. Showing more text @@ -951,7 +981,9 @@ SUBJECT MODIFIERS tured parentheses be output after a match. By default, only those up to the highest one actually used in the match are output (corresponding to the return code from pcre2_match()). Groups that did not take part in - the match are output as "<unset>". + the match are output as "<unset>". This modifier is not relevant for + DFA matching (which does no capturing); it is ignored, with a warning + message, if present. Testing callouts @@ -1363,10 +1395,11 @@ CALLOUTS attempt starting at the fourth character of the subject string, when the pointer was at the seventh character, and when the next pattern item was \d. Just one circumflex is output if the start and current - positions are the same. + positions are the same, or if the current position precedes the start + position, which can happen if the callout is in a lookbehind assertion. Callouts numbered 255 are assumed to be automatic callouts, inserted as - a result of the /auto_callout pattern modifier. In this case, instead + a result of the /auto_callout pattern modifier. In this case, instead of showing the callout number, the offset in the pattern, preceded by a plus, is output. For example: @@ -1380,7 +1413,7 @@ CALLOUTS 0: E* If a pattern contains (*MARK) items, an additional line is output when- - ever a change of latest mark is passed to the callout function. For + ever a change of latest mark is passed to the callout function. For example: re> /a(*MARK:X)bc/auto_callout @@ -1394,17 +1427,17 @@ CALLOUTS +12 ^ ^ 0: abc - The mark changes between matching "a" and "b", but stays the same for - the rest of the match, so nothing more is output. If, as a result of - backtracking, the mark reverts to being unset, the text "<unset>" is + The mark changes between matching "a" and "b", but stays the same for + the rest of the match, so nothing more is output. If, as a result of + backtracking, the mark reverts to being unset, the text "<unset>" is output. Callouts with string arguments The output for a callout with a string argument is similar, except that - instead of outputting a callout number before the position indicators, - the callout string and its offset in the pattern string are output - before the reflection of the subject string, and the subject string is + instead of outputting a callout number before the position indicators, + the callout string and its offset in the pattern string are output + before the reflection of the subject string, and the subject string is reflected for each callout. For example: re> /^ab(?C'first')cd(?C"second")ef/ @@ -1421,41 +1454,46 @@ CALLOUTS NON-PRINTING CHARACTERS When pcre2test is outputting text in the compiled version of a pattern, - bytes other than 32-126 are always treated as non-printing characters + bytes other than 32-126 are always treated as non-printing characters and are therefore shown as hex escapes. - When pcre2test is outputting text that is a matched part of a subject - string, it behaves in the same way, unless a different locale has been - set for the pattern (using the /locale modifier). In this case, the - isprint() function is used to distinguish printing and non-printing + When pcre2test is outputting text that is a matched part of a subject + string, it behaves in the same way, unless a different locale has been + set for the pattern (using the /locale modifier). In this case, the + isprint() function is used to distinguish printing and non-printing characters. SAVING AND RESTORING COMPILED PATTERNS - It is possible to save compiled patterns on disc or elsewhere, and + It is possible to save compiled patterns on disc or elsewhere, and reload them later, subject to a number of restrictions. JIT data cannot - be saved. The host on which the patterns are reloaded must be running + be saved. The host on which the patterns are reloaded must be running the same version of PCRE2, with the same code unit width, and must also - have the same endianness, pointer width and PCRE2_SIZE type. Before - compiled patterns can be saved they must be serialized, that is, con- - verted to a stream of bytes. A single byte stream may contain any num- - ber of compiled patterns, but they must all use the same character + have the same endianness, pointer width and PCRE2_SIZE type. Before + compiled patterns can be saved they must be serialized, that is, con- + verted to a stream of bytes. A single byte stream may contain any num- + ber of compiled patterns, but they must all use the same character tables. A single copy of the tables is included in the byte stream (its size is 1088 bytes). - The functions whose names begin with pcre2_serialize_ are used for - serializing and de-serializing. They are described in the pcre2serial- + The functions whose names begin with pcre2_serialize_ are used for + serializing and de-serializing. They are described in the pcre2serial- ize documentation. In this section we describe the features of pcre2test that can be used to test these functions. - When a pattern with push modifier is successfully compiled, it is - pushed onto a stack of compiled patterns, and pcre2test expects the - next line to contain a new pattern (or command) instead of a subject - line. By this means, a number of patterns can be compiled and retained. - The push modifier is incompatible with posix, and control modifiers - that act at match time are ignored (with a message). The jitverify mod- - ifier applies only at compile time. The command + When a pattern with push modifier is successfully compiled, it is + pushed onto a stack of compiled patterns, and pcre2test expects the + next line to contain a new pattern (or command) instead of a subject + line. By contrast, the pushcopy modifier causes a copy of the compiled + pattern to be stacked, leaving the original available for immediate + matching. By using push and/or pushcopy, a number of patterns can be + compiled and retained. These modifiers are incompatible with posix, and + control modifiers that act at match time are ignored (with a message) + for the stacked patterns. The jitverify modifier applies only at com- + pile time. + + The command #save <filename> @@ -1472,9 +1510,10 @@ SAVING AND RESTORING COMPILED PATTERNS matched with the pattern, terminated as usual by an empty line or end of file. This command may be followed by a modifier list containing only control modifiers that act after a pattern has been compiled. In - particular, hex, posix, and push are not allowed, nor are any option- - setting modifiers. The JIT modifiers are, however permitted. Here is - an example that saves and reloads two patterns. + particular, hex, posix, posix_nosub, push, and pushcopy are not + allowed, nor are any option-setting modifiers. The JIT modifiers are, + however permitted. Here is an example that saves and reloads two pat- + terns. /abc/push /xyz/push @@ -1486,9 +1525,13 @@ SAVING AND RESTORING COMPILED PATTERNS #pop jit,bincode abc - If jitverify is used with #pop, it does not automatically imply jit, + If jitverify is used with #pop, it does not automatically imply jit, which is different behaviour from when it is used on a pattern. + The #popcopy command is analagous to the pushcopy modifier in that it + makes current a copy of the topmost stack pattern, leaving the original + still on the stack. + SEE ALSO @@ -1505,5 +1548,5 @@ AUTHOR REVISION - Last updated: 12 December 2015 - Copyright (c) 1997-2015 University of Cambridge. + Last updated: 06 July 2016 + Copyright (c) 1997-2016 University of Cambridge. |