From 68ea654b0f97828c075152d88ae39e4818e87b1e Mon Sep 17 00:00:00 2001 From: John Millaway Date: Wed, 2 Apr 2003 23:20:49 +0000 Subject: Began conversion to DocBook. --- doc/flex.xml | 9635 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 9635 insertions(+) create mode 100644 doc/flex.xml (limited to 'doc') diff --git a/doc/flex.xml b/doc/flex.xml new file mode 100644 index 0000000..5a08ed0 --- /dev/null +++ b/doc/flex.xml @@ -0,0 +1,9635 @@ + + + + +flex: a fast lexical analyzer generator + + + +VernPaxson +W. L.Estes +JohnMillaway + + + + +@defindex hk + +@defindex op +@dircategory Programming +@direntry + + + +This manual describes flex, a tool for generating programs that +perform pattern-matching on text. The manual includes both tutorial and +reference sections. + +This edition of @cite{The flex Manual} documents flex version +@value{VERSION}. It was last updated on @value{UPDATED}. + + + + +Copyright + + + + +The flex manual is placed under the same licensing conditions as the +rest of flex: + + +1990 +1997 +The Regents of the University of California. +All rights reserved. + + + +This code is derived from software contributed to Berkeley by +Vern Paxson. + +The United States Government has rights in this work pursuant +to contract no. DE-AC03-76SF00098 between the United States +Department of Energy and the University of California. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + + + + +Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + + + +Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + + + + +Neither the name of the University nor the names of its contributors +may be used to endorse or promote products derived from this software +without specific prior written permission. + +THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR +IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. + + + + + +Reporting Bugs + + + + +If you have problems with flex or think you have found a bug, +please send mail detailing your problem to +@email{lex-help@@lists.sourceforge.net}. Patches are always welcome. + + + + +Introduction + + +flex is a tool for generating @dfn{scanners}. A scanner is a +program which recognizes lexical patterns in text. The flex +program reads the given input files, or its standard input if no file +names are given, for a description of a scanner to generate. The +description is in the form of pairs of regular expressions and C code, +called @dfn{rules}. flex generates as output a C source file, +lex.yy.c by default, which defines a routine yylex. +This file can be compiled and linked with the flex runtime library to +produce an executable. When the executable is run, it analyzes its +input for occurrences of the regular expressions. Whenever it finds +one, it executes the corresponding C code. + + + + +Some Simple Examples + +First some simple examples to get the flavor of how one uses +flex. + + +The following flex input specifies a scanner which, when it +encounters the string @samp{username} will replace it with the user's +login name: + + + + + + + + + +By default, any text not matched by a flex scanner is copied to +the output, so the net effect of this scanner is to copy its input file +to its output with each occurrence of @samp{username} expanded. In this +input, there is just one rule. @samp{username} is the @dfn{pattern} and +the @samp{printf} is the @dfn{action}. The @samp{%%} symbol marks the +beginning of the rules. + +Here's another simple example: + + + + + + + + +This scanner counts the number of characters and the number of lines in +its input. It produces no output other than the final report on the +character and line counts. The first line declares two globals, +@code{num_lines} and @code{num_chars}, which are accessible both inside +yylex and in the main routine declared after the +second @samp{%%}. There are two rules, one which matches a newline +(@samp{\n}) and increments both the line count and the character count, +and one which matches any character other than a newline (indicated by +the @samp{.} regular expression). + +A somewhat more complicated example: + + + + + + %} + + DIGIT [0-9] + ID [a-z][a-z0-9]* + + %% + + {DIGIT}+ { + printf( "An integer: %s (%d)\n", yytext, + atoi( yytext ) ); + } + + {DIGIT}+"."{DIGIT}* { + printf( "A float: %s (%g)\n", yytext, + atof( yytext ) ); + } + + if|then|begin|end|procedure|function { + printf( "A keyword: %s\n", yytext ); + } + + {ID} printf( "An identifier: %s\n", yytext ); + + "+"|"-"|"*"|"/" printf( "An operator: %s\n", yytext ); + + "{"[\^{}}\n]*"}" /* eat up one-line comments */ + + [ \t\n]+ /* eat up whitespace */ + + . printf( "Unrecognized character: %s\n", yytext ); + + %% + + main( argc, argv ) + int argc; + char **argv; + { + ++argv, --argc; /* skip over program name */ + if ( argc > 0 ) + yyin = fopen( argv[0], "r" ); + else + yyin = stdin; + + yylex(); + } +]]> + + + +This is the beginnings of a simple scanner for a language like Pascal. +It identifies different types of @dfn{tokens} and reports on what it has +seen. + +The details of this example will be explained in the following +sections. + + + + +Format of the Input File + + + + + + + +The flex input file consists of three sections, separated by a +line containing only @samp{%%}. + + + + + + + + + + + +
+Format of the Definitions Section + + + +The @dfn{definitions section} contains declarations of simple @dfn{name} +definitions to simplify the scanner specification, and declarations of +@dfn{start conditions}, which are explained in a later section. + + + +Name definitions have the form: + + + + + + + +The @samp{name} is a word beginning with a letter or an underscore +(@samp{_}) followed by zero or more letters, digits, @samp{_}, or +@samp{-} (dash). The definition is taken to begin at the first +non-whitespace character following the name and continuing to the end of +the line. The definition can subsequently be referred to using +@samp{@{name@}}, which will expand to @samp{(definition)}. For example, + + + + + + + + + +Defines @samp{DIGIT} to be a regular expression which matches a single +digit, and @samp{ID} to be a regular expression which matches a letter +followed by zero-or-more letters-or-digits. A subsequent reference to + + + + + + + + +is identical to + + + + + + + +and matches one-or-more digits followed by a @samp{.} followed by +zero-or-more digits. + + +An unindented comment (i.e., a line +beginning with @samp{/*}) is copied verbatim to the output up +to the next @samp{*/}. + + + + +Any @emph{indented} text or text enclosed in @samp{%@{} and @samp{%@}} +is also copied verbatim to the output (with the %@{ and %@} symbols +removed). The %@{ and %@} symbols must appear unindented on lines by +themselves. + + + +A @code{%top} block is similar to a @samp{%@{} ... @samp{%@}} block, except +that the code in a @code{%top} block is relocated to the @emph{top} of the +generated file, before any flex definitions @footnote{Actually, +@code{yyIN_HEADER} is defined before the @samp{%top} block.}. +The @code{%top} block is useful when you want certain preprocessor macros to be +defined or certain files to be included before the generated code. +The single characters, @samp{@{} and @samp{@}} are used to delimit the +@code{%top} block, as show in the example below: + + + + + #include + } +]]> + + + +Multiple @code{%top} blocks are allowed, and their order is preserved. + +
+ +
+Format of the Rules Section + + + +The @dfn{rules} section of the flex input contains a series of +rules of the form: + + + + + + + +where the pattern must be unindented and the action must begin +on the same line. +@xref{Patterns}, for a further description of patterns and actions. + +In the rules section, any indented or %@{ %@} enclosed text appearing +before the first rule may be used to declare variables which are local +to the scanning routine and (after the declarations) code which is to be +executed whenever the scanning routine is entered. Other indented or +%@{ %@} text in the rule section is still copied to the output, but its +meaning is not well-defined and it may well cause compile-time errors +(this feature is present for @acronym{POSIX} compliance. @xref{Lex and +Posix}, for other such features). + +Any @emph{indented} text or text enclosed in @samp{%@{} and @samp{%@}} +is copied verbatim to the output (with the %@{ and %@} symbols removed). +The %@{ and %@} symbols must appear unindented on lines by themselves. + +
+ +
+Format of the User Code Section + + + +The user code section is simply copied to lex.yy.c verbatim. It +is used for companion routines which call or are called by the scanner. +The presence of this section is optional; if it is missing, the second +@samp{%%} in the input file may be skipped, too. + +
+ +
+Comments in the Input + + +Flex supports C-style comments, that is, anything between /* and */ is +considered a comment. Whenever flex encounters a comment, it copies the +entire comment verbatim to the generated source code. Comments may +appear just about anywhere, but with the following exceptions: + + + + + + +Comments may not appear in the Rules Section wherever flex is expecting +a regular expression. This means comments may not appear at the +beginning of a line, or immediately following a list of scanner states. + + + +Comments may not appear on an @samp{%option} line in the Definitions +Section. + + + + +If you want to follow a simple rule, then always begin a comment on a +new line, with one or more whitespace characters before the initial +@samp{/*}). This rule will work anywhere in the input file. + +All the comments in the following example are valid: + + + + + +{ +ruleC ECHO; +ruleD ECHO; +%{ +/* code block */ +%} +} +%% +/* User Code Section */ + +]]> + + + + + + +Patterns + + + +The patterns in the input (see @ref{Rules Section}) are written using an +extended set of regular expressions. These are: + + + + +x + +match the character 'x' + + + + +. + +any character (byte) except newline + + + + + + + +[xyz] + +a @dfn{character class}; in this case, the pattern +matches either an 'x', a 'y', or a 'z' + + + + + +[abj-oZ] + +a "character class" with a range in it; matches +an 'a', a 'b', any letter from 'j' through 'o', +or a 'Z' + + + + + + +[^A-Z] + +a "negated character class", i.e., any character +but those in the class. In this case, any +character EXCEPT an uppercase letter. + + + + +[^A-Z\n] + +any character EXCEPT an uppercase letter or +a newline + + + + +r* + +zero or more r's, where r is any regular expression + + + + +r+ + +one or more r's + + + + +r? + +zero or one r's (that is, ``an optional r'') + + + + + +r@{2,5@} + +anywhere from two to five r's + + + + +r@{2,@} + +two or more r's + + + + +r@{4@} + +exactly 4 r's + + + + + +@{name@} + +the expansion of the @samp{name} definition +(@pxref{Format}). + + + + + + +"[xyz]\"foo" + +the literal string: @samp{[xyz]"foo} + + + + + +\X + +if X is @samp{a}, @samp{b}, @samp{f}, @samp{n}, @samp{r}, @samp{t}, or +@samp{v}, then the ANSI-C interpretation of @samp{\x}. Otherwise, a +literal @samp{X} (used to escape operators such as @samp{*}) + + + + + +\0 + +a NUL character (ASCII code 0) + + + + + +\123 + +the character with octal value 123 + + + + +\x2a + +the character with hexadecimal value 2a + + + + +(r) + +match an @samp{r}; parentheses are used to override precedence (see below) + + + + + +rs + +the regular expression @samp{r} followed by the regular expression @samp{s}; called +@dfn{concatenation} + + + + +r|s + +either an @samp{r} or an @samp{s} + + + + + +r/s + +an @samp{r} but only if it is followed by an @samp{s}. The text matched by @samp{s} is +included when determining whether this rule is the longest match, but is +then returned to the input before the action is executed. So the action +only sees the text matched by @samp{r}. This type of pattern is called +@dfn{trailing context}. (There are some combinations of @samp{r/s} that flex +cannot match correctly. @xref{Limitations}, regarding dangerous trailing +context.) + + + + + + +^r + +an @samp{r}, but only at the beginning of a line (i.e., +when just starting to scan, or right after a +newline has been scanned). + + + + + + +r$ + +an @samp{r}, but only at the end of a line (i.e., just before a +newline). Equivalent to @samp{r/\n}. + + +Note that flex's notion of ``newline'' is exactly +whatever the C compiler used to compile flex +interprets @samp{\n} as; in particular, on some DOS +systems you must either filter out @samp{\r}s in the +input yourself, or explicitly use @samp{r/\r\n} for @samp{r$}. + + + + + +r + +an @samp{r}, but only in start condition @code{s} (see @ref{Start +Conditions} for discussion of start conditions). + + + + +r + +same, but in any of start conditions @code{s1}, @code{s2}, or @code{s3}. + + + + +<*>r + +an @samp{r} in any start condition, even an exclusive one. + + + + + + +<> + +an end-of-file. + + + + +<> + +an end-of-file when in start condition @code{s1} or @code{s2} + + + + +Note that inside of a character class, all regular expression operators +lose their special meaning except escape (@samp{\}) and the character class +operators, @samp{-}, @samp{]]}, and, at the beginning of the class, @samp{^}. + + +The regular expressions listed above are grouped according to +precedence, from highest precedence at the top to lowest at the bottom. +Those grouped together have equal precedence (see special note on the +precedence of the repeat operator, @samp{@{@}}, under the documentation +for the @samp{--posix} POSIX compliance option). For example, + + + + + + + + +is the same as + + + + + + + +since the @samp{*} operator has higher precedence than concatenation, +and concatenation higher than alternation (@samp{|}). This pattern +therefore matches @emph{either} the string @samp{foo} @emph{or} the +string @samp{ba} followed by zero-or-more @samp{r}'s. To match +@samp{foo} or zero-or-more repetitions of the string @samp{bar}, use: + + + + + + + +And to match a sequence of zero or more repetitions of @samp{foo} and +@samp{bar}: + + + + + + + + + +In addition to characters and ranges of characters, character classes +can also contain @dfn{character class expressions}. These are +expressions enclosed inside @samp{[}: and @samp{:]} delimiters (which +themselves must appear between the @samp{[} and @samp{]} of the +character class. Other elements may occur inside the character class, +too). The valid expressions are: + + + + + + + + +These expressions all designate a set of characters equivalent to the +corresponding standard C @code{isXXX} function. For example, +@samp{[:alnum:]} designates those characters for which isalnum +returns true - i.e., any alphabetic or numeric character. Some systems +don't provide isblank, so flex defines @samp{[:blank:]} as a +blank or a tab. + +For example, the following character classes are all equivalent: + + + + + + + + + +Some notes on patterns are in order. + + + + + + + If your scanner is case-insensitive (the @samp{-i} flag), then +@samp{[:upper:]} and @samp{[:lower:]} are equivalent to +@samp{[:alpha:]}. + +@anchor{case and character ranges} + + + Character classes with ranges, such as @samp{[a-Z]}, should be used with +caution in a case-insensitive scanner if the range spans upper or lowercase +characters. Flex does not know if you want to fold all upper and lowercase +characters together, or if you want the literal numeric range specified (with +no case folding). When in doubt, flex will assume that you meant the literal +numeric range, and will issue a warning. The exception to this rule is a +character range such as @samp{[a-z]} or @samp{[S-W]} where it is obvious that you +want case-folding to occur. Here are some examples with the @samp{-i} flag +enabled: + + + + + + + + + +A negated character class such as the example @samp{[^A-Z]} above +@emph{will} match a newline unless @samp{\n} (or an equivalent escape +sequence) is one of the characters explicitly present in the negated +character class (e.g., @samp{[^A-Z\n]}). This is unlike how many other +regular expression tools treat negated character classes, but +unfortunately the inconsistency is historically entrenched. Matching +newlines means that a pattern like @samp{[^"]*} can match the entire +input unless there's another quote in the input. + + + + + + + +A rule can have at most one instance of trailing context (the @samp{/} operator +or the @samp{$} operator). The start condition, @samp{^}, and @samp{<>} patterns +can only occur at the beginning of a pattern, and, as well as with @samp{/} and @samp{$}, +cannot be grouped inside parentheses. A @samp{^} which does not occur at +the beginning of a rule or a @samp{$} which does not occur at the end of +a rule loses its special properties and is treated as a normal character. + + + + +The following are invalid: + + + + +foobar +]]> + + + +Note that the first of these can be written @samp{foo/bar\n}. + + + + +The following will result in @samp{$} or @samp{^} being treated as a normal character: + + + + + + + + +If the desired meaning is a @samp{foo} or a +@samp{bar}-followed-by-a-newline, the following could be used (the +special @code{|} action is explained below, @pxref{Actions}): + + + + + + + + +A similar trick will work for matching a @samp{foo} or a +@samp{bar}-at-the-beginning-of-a-line. + + + + + + + +How the Input Is Matched + + + + + + + +When the generated scanner is run, it analyzes its input looking for +strings which match any of its patterns. If it finds more than one +match, it takes the one matching the most text (for trailing context +rules, this includes the length of the trailing part, even though it +will then be returned to the input). If it finds two or more matches of +the same length, the rule listed first in the flex input file is +chosen. + + + + +Once the match is determined, the text corresponding to the match +(called the @dfn{token}) is made available in the global character +pointer yytext, and its length in the global integer +yyleng. The @dfn{action} corresponding to the matched pattern is +then executed (@pxref{Actions}), and then the remaining input is scanned +for another match. + + +If no match is found, then the @dfn{default rule} is executed: the next +character in the input is considered matched and copied to the standard +output. Thus, the simplest valid flex input is: + + + + + + + + +which generates a scanner that simply copies its input (one character at +a time) to its output. + + + + + +Note that yytext can be defined in two different ways: either as +a character @emph{pointer} or as a character @emph{array}. You can +control which definition flex uses by including one of the +special directives @code{%pointer} or @code{%array} in the first +(definitions) section of your flex input. The default is +@code{%pointer}, unless you use the @samp{-l} lex compatibility option, +in which case yytext will be an array. The advantage of using +@code{%pointer} is substantially faster scanning and no buffer overflow +when matching very large tokens (unless you run out of dynamic memory). +The disadvantage is that you are restricted in how your actions can +modify yytext (@pxref{Actions}), and calls to the unput +function destroys the present contents of yytext, which can be a +considerable porting headache when moving between different @code{lex} +versions. + + +The advantage of @code{%array} is that you can then modify yytext +to your heart's content, and calls to unput do not destroy +yytext (@pxref{Actions}). Furthermore, existing @code{lex} +programs sometimes access yytext externally using declarations of +the form: + + + + + + + +This definition is erroneous when used with @code{%pointer}, but correct +for @code{%array}. + +The @code{%array} declaration defines yytext to be an array of +@code{YYLMAX} characters, which defaults to a fairly large value. You +can change the size by simply #define'ing @code{YYLMAX} to a different +value in the first section of your flex input. As mentioned +above, with @code{%pointer} yytext grows dynamically to accommodate +large tokens. While this means your @code{%pointer} scanner can +accommodate very large tokens (such as matching entire blocks of +comments), bear in mind that each time the scanner must resize +yytext it also must rescan the entire token from the beginning, +so matching such tokens can prove slow. yytext presently does +@emph{not} dynamically grow if a call to unput results in too +much text being pushed back; instead, a run-time error results. + + +Also note that you cannot use @code{%array} with C++ scanner classes +(@pxref{Cxx}). + + + + +Actions + + +Each pattern in a rule has a corresponding @dfn{action}, which can be +any arbitrary C statement. The pattern ends at the first non-escaped +whitespace character; the remainder of the line is its action. If the +action is empty, then when the pattern is matched the input token is +simply discarded. For example, here is the specification for a program +which deletes all occurrences of @samp{zap me} from its input: + + + + + + + + +This example will copy all other characters in the input to the output +since they will be matched by the default rule. + +Here is a program which compresses multiple blanks and tabs down to a +single blank, and throws away whitespace found at the end of a line: + + + + + + + + + + + + + + +If the action contains a @samp{@}}, then the action spans till the +balancing @samp{@}} is found, and the action may cross multiple lines. +flex knows about C strings and comments and won't be fooled by +braces found within them, but also allows actions to begin with +@samp{%@{} and will consider the action to be all the text up to the +next @samp{%@}} (regardless of ordinary braces inside the action). + + +An action consisting solely of a vertical bar (@samp{|}) means ``same as the +action for the next rule''. See below for an illustration. + +Actions can include arbitrary C code, including @code{return} statements +to return a value to whatever routine called yylex. Each time +yylex is called it continues processing tokens from where it +last left off until it either reaches the end of the file or executes a +return. + + +Actions are free to modify yytext except for lengthening it +(adding characters to its end--these will overwrite later characters in +the input stream). This however does not apply when using @code{%array} +(@pxref{Matching}). In that case, yytext may be freely modified +in any way. + + + +Actions are free to modify yyleng except they should not do so if +the action also includes use of yymore (see below). + + +There are a number of special directives which can be included within an +action: + + + +ECHO + + +copies yytext to the scanner's output. + + + + +BEGIN + + +followed by the name of a start condition places the scanner in the +corresponding start condition (see below). + + + + +REJECT + + +directs the scanner to proceed on to the ``second best'' rule which +matched the input (or a prefix of the input). The rule is chosen as +described above in @ref{Matching}, and yytext and yyleng +set up appropriately. It may either be one which matched as much text +as the originally chosen rule but came later in the flex input +file, or one which matched less text. For example, the following will +both count the words in the input and call the routine special +whenever @samp{frob} is seen: + + + + + + + +Without the @code{REJECT}, any occurences of @samp{frob} in the input +would not be counted as words, since the scanner normally executes only +one action per token. Multiple uses of @code{REJECT} are allowed, each +one finding the next best choice to the currently active rule. For +example, when the following scanner scans the token @samp{abcd}, it will +write @samp{abcdabcaba} to the output: + + + + + + + + + +The first three rules share the fourth's action since they use the +special @samp{|} action. + +@code{REJECT} is a particularly expensive feature in terms of scanner +performance; if it is used in @emph{any} of the scanner's actions it +will slow down @emph{all} of the scanner's matching. Furthermore, +@code{REJECT} cannot be used with the @samp{-Cf} or @samp{-CF} options +(@pxref{Scanner Options}). + +Note also that unlike the other special actions, @code{REJECT} is a +@emph{branch}. code immediately following it in the action will +@emph{not} be executed. + + + + +yymore() + + +tells the scanner that the next time it matches a rule, the +corresponding token should be @emph{appended} onto the current value of +yytext rather than replacing it. For example, given the input +@samp{mega-kludge} the following will write @samp{mega-mega-kludge} to +the output: + + + + + + + + + +First @samp{mega-} is matched and echoed to the output. Then @samp{kludge} +is matched, but the previous @samp{mega-} is still hanging around at the +beginning of +yytext +so the +@code{ECHO} +for the @samp{kludge} rule will actually write @samp{mega-kludge}. + + + + + +Two notes regarding use of yymore. First, yymore +depends on the value of yyleng correctly reflecting the size of +the current token, so you must not modify yyleng if you are using +yymore. Second, the presence of yymore in the +scanner's action entails a minor performance penalty in the scanner's +matching speed. + + +@code{yyless(n)} returns all but the first @code{n} characters of the +current token back to the input stream, where they will be rescanned +when the scanner looks for the next match. yytext and +yyleng are adjusted appropriately (e.g., yyleng will now +be equal to @code{n}). For example, on the input @samp{foobar} the +following will write out @samp{foobarbar}: + + + + + + + + + +An argument of 0 to yyless will cause the entire current input +string to be scanned again. Unless you've changed how the scanner will +subsequently process its input (using @code{BEGIN}, for example), this +will result in an endless loop. + +Note that yyless is a macro and can only be used in the flex +input file, not from other source files. + + + +@code{unput(c)} puts the character @code{c} back onto the input stream. +It will be the next character scanned. The following action will take +the current token and cause it to be rescanned enclosed in parentheses. + + + + + += 0; --i ) + unput( yycopy[i] ); + unput( '(' ); + free( yycopy ); + } +]]> + + + +Note that since each unput puts the given character back at the +@emph{beginning} of the input stream, pushing back strings must be done +back-to-front. + + + +An important potential problem when using unput is that if you +are using @code{%pointer} (the default), a call to unput +@emph{destroys} the contents of yytext, starting with its +rightmost character and devouring one character to the left with each +call. If you need the value of yytext preserved after a call to +unput (as in the above example), you must either first copy it +elsewhere, or build your scanner using @code{%array} instead +(@pxref{Matching}). + + + +Finally, note that you cannot put back @samp{EOF} to attempt to mark the +input stream with an end-of-file. + + +input reads the next character from the input stream. For +example, the following is one way to eat up C comments: + + + + + + + + + + + +(Note that if the scanner is compiled using @code{C++}, then +input is instead referred to as @b{yyinput()}, in order to +avoid a name clash with the @code{C++} stream by the name of +@code{input}.) + + + +YY_FLUSH_BUFFER flushes the scanner's internal buffer so that +the next time the scanner attempts to match a token, it will first +refill the buffer using YY_INPUT (@pxref{Generated Scanner}). +This action is a special case of the more general +yy_flush_buffer function, described below (@pxref{Multiple +Input Buffers}) + + + + + +yyterminate can be used in lieu of a return statement in an +action. It terminates the scanner and returns a 0 to the scanner's +caller, indicating ``all done''. By default, yyterminate is +also called when an end-of-file is encountered. It is a macro and may +be redefined. + + + + +The Generated Scanner + + +The output of flex is the file lex.yy.c, which contains +the scanning routine yylex, a number of tables used by it for +matching tokens, and a number of auxiliary routines and macros. By +default, yylex is declared as follows: + + + + + + + + +(If your environment supports function prototypes, then it will be +@code{int yylex( void )}.) This definition may be changed by defining +the @code{YY_DECL} macro. For example, you could use: + + + + + + + + +to give the scanning routine the name @code{lexscan}, returning a float, +and taking two floats as arguments. Note that if you give arguments to +the scanning routine using a K&R-style/non-prototyped function +declaration, you must terminate the definition with a semi-colon (;). + +flex generates @samp{C99} function definitions by +default. However flex does have the ability to generate obsolete, er, +@samp{traditional}, function definitions. This is to support +bootstrapping gcc on old systems. Unfortunately, traditional +definitions prevent us from using any standard data types smaller than +int (such as short, char, or bool) as function arguments. For this +reason, future versions of flex may generate standard C99 code +only, leaving K&R-style functions to the historians. Currently, if you +do @strong{not} want @samp{C99} definitions, then you must use +@code{%option noansi-definitions}. + + + +Whenever yylex is called, it scans tokens from the global input +file yyin (which defaults to stdin). It continues until it +either reaches an end-of-file (at which point it returns the value 0) or +one of its actions executes a @code{return} statement. + + + + +If the scanner reaches an end-of-file, subsequent calls are undefined +unless either yyin is pointed at a new input file (in which case +scanning continues from that file), or yyrestart is called. +yyrestart takes one argument, a @code{FILE *} pointer (which +can be NULL, if you've set up @code{YY_INPUT} to scan from a source other +than yyin), and initializes yyin for scanning from that +file. Essentially there is no difference between just assigning +yyin to a new input file or using yyrestart to do so; +the latter is available for compatibility with previous versions of +flex, and because it can be used to switch input files in the +middle of scanning. It can also be used to throw away the current input +buffer, by calling it with an argument of yyin; but it would be +better to use @code{YY_FLUSH_BUFFER} (@pxref{Actions}). Note that +yyrestart does @emph{not} reset the start condition to +@code{INITIAL} (@pxref{Start Conditions}). + + +If yylex stops scanning due to executing a @code{return} +statement in one of the actions, the scanner may then be called again +and it will resume scanning where it left off. + + +By default (and for purposes of efficiency), the scanner uses +block-reads rather than simple getc calls to read characters +from yyin. The nature of how it gets its input can be controlled +by defining the @code{YY_INPUT} macro. The calling sequence for +YY_INPUT is @code{YY_INPUT(buf,result,max_size)}. Its action +is to place up to @code{max_size} characters in the character array +@code{buf} and return in the integer variable @code{result} either the +number of characters read or the constant @code{YY_NULL} (0 on Unix +systems) to indicate @samp{EOF}. The default @code{YY_INPUT} reads from +the global file-pointer yyin. + + +Here is a sample definition of @code{YY_INPUT} (in the definitions +section of the input file): + + + + + + + +This definition will change the input processing to occur one character +at a time. + + +When the scanner receives an end-of-file indication from YY_INPUT, it +then checks the yywrap function. If yywrap returns +false (zero), then it is assumed that the function has gone ahead and +set up yyin to point to another input file, and scanning +continues. If it returns true (non-zero), then the scanner terminates, +returning 0 to its caller. Note that in either case, the start +condition remains unchanged; it does @emph{not} revert to +@code{INITIAL}. + + + + +If you do not supply your own version of yywrap, then you must +either use @code{%option noyywrap} (in which case the scanner behaves as +though yywrap returned 1), or you must link with @samp{-lfl} to +obtain the default version of the routine, which always returns 1. + +For scanning from in-memory buffers (e.g., scanning strings), see +@ref{Scanning Strings}. @xref{Multiple Input Buffers}. + + + + +The scanner writes its @code{ECHO} output to the yyout global +(default, stdout), which may be redefined by the user simply by +assigning it to some other @code{FILE} pointer. + + + + +Start Conditions + + +flex provides a mechanism for conditionally activating rules. +Any rule whose pattern is prefixed with @samp{} will only be active +when the scanner is in the @dfn{start condition} named @code{sc}. For +example, + + + + +[^"]* { /* eat up the string body ... */ + ... + } +]]> + + + +will be active only when the scanner is in the @code{STRING} start +condition, and + + + + +\. { /* handle an escape ... */ + ... + } +]]> + + + +will be active only when the current start condition is either +@code{INITIAL}, @code{STRING}, or @code{QUOTE}. + + +Start conditions are declared in the definitions (first) section of the +input using unindented lines beginning with either @samp{%s} or +@samp{%x} followed by a list of names. The former declares +@dfn{inclusive} start conditions, the latter @dfn{exclusive} start +conditions. A start condition is activated using the @code{BEGIN} +action. Until the next @code{BEGIN} action is executed, rules with the +given start condition will be active and rules with other start +conditions will be inactive. If the start condition is inclusive, then +rules with no start conditions at all will also be active. If it is +exclusive, then @emph{only} rules qualified with the start condition +will be active. A set of rules contingent on the same exclusive start +condition describe a scanner which is independent of any of the other +rules in the flex input. Because of this, exclusive start +conditions make it easy to specify ``mini-scanners'' which scan portions +of the input that are syntactically different from the rest (e.g., +comments). + +If the distinction between inclusive and exclusive start conditions +is still a little vague, here's a simple example illustrating the +connection between the two. The set of rules: + + + + +foo do_something(); + + bar something_else(); +]]> + + + +is equivalent to + + + + +foo do_something(); + + bar something_else(); +]]> + + + +Without the @code{} qualifier, the @code{bar} pattern in +the second example wouldn't be active (i.e., couldn't match) when in +start condition @code{example}. If we just used @code{example>} to +qualify @code{bar}, though, then it would only be active in +@code{example} and not in @code{INITIAL}, while in the first example +it's active in both, because in the first example the @code{example} +start condition is an inclusive @code{(%s)} start condition. + + +Also note that the special start-condition specifier +@code{<*>} +matches every start condition. Thus, the above example could also +have been written: + + + + +foo do_something(); + + <*>bar something_else(); +]]> + + + +The default rule (to @code{ECHO} any unmatched character) remains active +in start conditions. It is equivalent to: + + + + +.|\n ECHO; +]]> + + + + + + +@code{BEGIN(0)} returns to the original state where only the rules with +no start conditions are active. This state can also be referred to as +the start-condition @code{INITIAL}, so @code{BEGIN(INITIAL)} is +equivalent to @code{BEGIN(0)}. (The parentheses around the start +condition name are not required but are considered good style.) + +@code{BEGIN} actions can also be given as indented code at the beginning +of the rules section. For example, the following will cause the scanner +to enter the @code{SPECIAL} start condition whenever yylex is +called and the global variable @code{enter_special} is true: + + + + +blahblahblah + ...more rules follow... +]]> + + + +To illustrate the uses of start conditions, here is a scanner which +provides two different interpretations of a string like @samp{123.456}. +By default it will treat it as three tokens, the integer @samp{123}, a +dot (@samp{.}), and the integer @samp{456}. But if the string is +preceded earlier in the line by the string @samp{expect-floats} it will +treat it as a single token, the floating-point number @samp{123.456}: + + + + + + %} + %s expect + + %% + expect-floats BEGIN(expect); + + [0-9]+@samp{.}[0-9]+ { + printf( "found a float, = %f\n", + atof( yytext ) ); + } + \n { + /* that's the end of the line, so + * we need another "expect-number" + * before we'll recognize any more + * numbers + */ + BEGIN(INITIAL); + } + + [0-9]+ { + printf( "found an integer, = %d\n", + atoi( yytext ) ); + } + + "." printf( "found a dot\n" ); +]]> + + + + +Here is a scanner which recognizes (and discards) C comments while +maintaining a count of the current input line. + + + + +[^*\n]* /* eat anything that's not a '*' */ + "*"+[^*/\n]* /* eat up '*'s not followed by '/'s */ + \n ++line_num; + "*"+"/" BEGIN(INITIAL); +]]> + + + +This scanner goes to a bit of trouble to match as much +text as possible with each rule. In general, when attempting to write +a high-speed scanner try to match as much possible in each rule, as +it's a big win. + +Note that start-conditions names are really integer values and +can be stored as such. Thus, the above could be extended in the +following fashion: + + + + + +"/*" { + comment_caller = foo; + BEGIN(comment); + } + + [^*\n]* /* eat anything that's not a '*' */ + "*"+[^*/\n]* /* eat up '*'s not followed by '/'s */ + \n ++line_num; + "*"+"/" BEGIN(comment_caller); +]]> + + + + +Furthermore, you can access the current start condition using the +integer-valued @code{YY_START} macro. For example, the above +assignments to @code{comment_caller} could instead be written + + + + + + + + + +Flex provides @code{YYSTATE} as an alias for @code{YY_START} (since that +is what's used by AT&T @code{lex}). + +For historical reasons, start conditions do not have their own +name-space within the generated scanner. The start condition names are +unmodified in the generated scanner and generated header. +@xref{option-header}. @xref{option-prefix}. + + + +Finally, here's an example of how to match C-style quoted strings using +exclusive start conditions, including expanded escape sequences (but +not including checking for a string that's too long): + + + + +\" { /* saw closing quote - all done */ + BEGIN(INITIAL); + *string_buf_ptr = '\0'; + /* return string constant token type and + * value to parser + */ + } + + \n { + /* error - unterminated string constant */ + /* generate error message */ + } + + \\[0-7]{1,3} { + /* octal escape sequence */ + int result; + + (void) sscanf( yytext + 1, "%o", &result ); + + if ( result > 0xff ) + /* error, constant is out-of-bounds */ + + *string_buf_ptr++ = result; + } + + \\[0-9]+ { + /* generate error - bad escape sequence; something + * like '\48' or '\0777777' + */ + } + + \\n *string_buf_ptr++ = '\n'; + \\t *string_buf_ptr++ = '\t'; + \\r *string_buf_ptr++ = '\r'; + \\b *string_buf_ptr++ = '\b'; + \\f *string_buf_ptr++ = '\f'; + + \\(.|\n) *string_buf_ptr++ = yytext[1]; + + [^\\\n\"]+ { + char *yptr = yytext; + + while ( *yptr ) + *string_buf_ptr++ = *yptr++; + } +]]> + + + + +Often, such as in some of the examples above, you wind up writing a +whole bunch of rules all preceded by the same start condition(s). Flex +makes this a little easier and cleaner by introducing a notion of start +condition @dfn{scope}. A start condition scope is begun with: + + + +{ +]]> + + + +where @code{SCs} is a list of one or more start conditions. Inside the +start condition scope, every rule automatically has the prefix +@code{SCs>} applied to it, until a @samp{@}} which matches the initial +@samp{@{}. So, for example, + + + + +{ + "\\n" return '\n'; + "\\r" return '\r'; + "\\f" return '\f'; + "\\0" return '\0'; + } +]]> + + + +is equivalent to: + + + +"\\n" return '\n'; + "\\r" return '\r'; + "\\f" return '\f'; + "\\0" return '\0'; +]]> + + + +Start condition scopes may be nested. + + + + +The following routines are available for manipulating stacks of start conditions: + +@deftypefun void yy_push_state ( int @code{new_state} ) +pushes the current start condition onto the top of the start condition +stack and switches to +@code{new_state} +as though you had used +@code{BEGIN new_state} +(recall that start condition names are also integers). +@end deftypefun + +@deftypefun void yy_pop_state () +pops the top of the stack and switches to it via +@code{BEGIN}. +@end deftypefun + +@deftypefun int yy_top_state () +returns the top of the stack without altering the stack's contents. +@end deftypefun + + +The start condition stack grows dynamically and so has no built-in size +limitation. If memory is exhausted, program execution aborts. + +To use start condition stacks, your scanner must include a @code{%option +stack} directive (@pxref{Scanner Options}). + + + + +Multiple Input Buffers + + +Some scanners (such as those which support ``include'' files) require +reading from several input streams. As flex scanners do a large +amount of buffering, one cannot control where the next input will be +read from by simply writing a YY_INPUT which is sensitive to +the scanning context. YY_INPUT is only called when the scanner +reaches the end of its buffer, which may be a long time after scanning a +statement such as an @code{include} statement which requires switching +the input source. + +To negotiate these sorts of problems, flex provides a mechanism +for creating and switching between multiple input buffers. An input +buffer is created by using: + + +@deftypefun YY_BUFFER_STATE yy_create_buffer ( FILE *file, int size ) +@end deftypefun + +which takes a @code{FILE} pointer and a size and creates a buffer +associated with the given file and large enough to hold @code{size} +characters (when in doubt, use @code{YY_BUF_SIZE} for the size). It +returns a @code{YY_BUFFER_STATE} handle, which may then be passed to +other routines (see below). + +The @code{YY_BUFFER_STATE} type is a +pointer to an opaque @code{struct yy_buffer_state} structure, so you may +safely initialize @code{YY_BUFFER_STATE} variables to @code{((YY_BUFFER_STATE) +0)} if you wish, and also refer to the opaque structure in order to +correctly declare input buffers in source files other than that of your +scanner. Note that the @code{FILE} pointer in the call to +yy_create_buffer is only used as the value of yyin seen by +@code{YY_INPUT}. If you redefine YY_INPUT so it no longer uses +yyin, then you can safely pass a NULL @code{FILE} pointer to +yy_create_buffer. You select a particular buffer to scan from +using: + +@deftypefun void yy_switch_to_buffer ( YY_BUFFER_STATE new_buffer ) +@end deftypefun + +The above function switches the scanner's input buffer so subsequent tokens +will come from @code{new_buffer}. Note that yy_switch_to_buffer may +be used by yywrap to set things up for continued scanning, instead of +opening a new file and pointing yyin at it. If you are looking for a +stack of input buffers, then you want to use yypush_buffer_state +instead of this function. Note also that switching input sources via either +yy_switch_to_buffer or yywrap does @emph{not} change the +start condition. + + +@deftypefun void yy_delete_buffer ( YY_BUFFER_STATE buffer ) +@end deftypefun + +is used to reclaim the storage associated with a buffer. (@code{buffer} +can be NULL, in which case the routine does nothing.) You can also clear +the current contents of a buffer using: + + + +@deftypefun void yypush_buffer_state ( YY_BUFFER_STATE buffer ) +@end deftypefun + +This function pushes the new buffer state onto an internal stack. The pushed +state becomes the new current state. The stack is maintained by flex and will +grow as required. This function is intended to be used instead of +yy_switch_to_buffer, when you want to change states, but preserve the +current state for later use. + + + +@deftypefun void yypop_buffer_state ( ) +@end deftypefun + +This function removes the current state from the top of the stack, and deletes +it by calling yy_delete_buffer. The next state on the stack, if any, +becomes the new current state. + + + +@deftypefun void yy_flush_buffer ( YY_BUFFER_STATE buffer ) +@end deftypefun + +This function discards the buffer's contents, +so the next time the scanner attempts to match a token from the +buffer, it will first fill the buffer anew using +YY_INPUT. + +@deftypefun YY_BUFFER_STATE yy_new_buffer ( FILE *file, int size ) +@end deftypefun + +is an alias for yy_create_buffer, +provided for compatibility with the C++ use of @code{new} and +@code{delete} for creating and destroying dynamic objects. + + +@code{YY_CURRENT_BUFFER} macro returns a @code{YY_BUFFER_STATE} handle to the +current buffer. It should not be used as an lvalue. + + +Here are two examples of using these features for writing a scanner +which expands include files (the +@code{<>} +feature is discussed below). + +This first example uses yypush_buffer_state and yypop_buffer_state. Flex +maintains the stack internally. + + + + +[ \t]* /* eat the whitespace */ + [^ \t\n]+ { /* got the include file name */ + yyin = fopen( yytext, "r" ); + + if ( ! yyin ) + error( ... ); + + yypush_buffer_state(yy_create_buffer( yyin, YY_BUF_SIZE )); + + BEGIN(INITIAL); + } + + <> { + yypop_buffer_state(); + + if ( !YY_CURRENT_BUFFER ) + { + yyterminate(); + } + } +]]> + + + +The second example, below, does the same thing as the previous example did, but +manages its own input buffer stack manually (instead of letting flex do it). + + + + +[ \t]* /* eat the whitespace */ + [^ \t\n]+ { /* got the include file name */ + if ( include_stack_ptr >= MAX_INCLUDE_DEPTH ) + { + fprintf( stderr, "Includes nested too deeply" ); + exit( 1 ); + } + + include_stack[include_stack_ptr++] = + YY_CURRENT_BUFFER; + + yyin = fopen( yytext, "r" ); + + if ( ! yyin ) + error( ... ); + + yy_switch_to_buffer( + yy_create_buffer( yyin, YY_BUF_SIZE ) ); + + BEGIN(INITIAL); + } + + <> { + if ( --include_stack_ptr 0 ) + { + yyterminate(); + } + + else + { + yy_delete_buffer( YY_CURRENT_BUFFER ); + yy_switch_to_buffer( + include_stack[include_stack_ptr] ); + } + } +]]> + + + +@anchor{Scanning Strings} + +The following routines are available for setting up input buffers for +scanning in-memory strings instead of files. All of them create a new +input buffer for scanning the string, and return a corresponding +@code{YY_BUFFER_STATE} handle (which you should delete with +yy_delete_buffer when done with it). They also switch to the +new buffer using yy_switch_to_buffer, so the next call to +yylex will start scanning the string. + +@deftypefun YY_BUFFER_STATE yy_scan_string ( const char *str ) +scans a NUL-terminated string. +@end deftypefun + +@deftypefun YY_BUFFER_STATE yy_scan_bytes ( const char *bytes, int len ) +scans @code{len} bytes (including possibly @code{NUL}s) starting at location +@code{bytes}. +@end deftypefun + +Note that both of these functions create and scan a @emph{copy} of the +string or bytes. (This may be desirable, since yylex modifies +the contents of the buffer it is scanning.) You can avoid the copy by +using: + + +@deftypefun YY_BUFFER_STATE yy_scan_buffer (char *base, yy_size_t size) +which scans in place the buffer starting at @code{base}, consisting of +@code{size} bytes, the last two bytes of which @emph{must} be +@code{YY_END_OF_BUFFER_CHAR} (ASCII NUL). These last two bytes are not +scanned; thus, scanning consists of @code{base[0]} through +@code{base[size-2]}, inclusive. +@end deftypefun + +If you fail to set up @code{base} in this manner (i.e., forget the final +two @code{YY_END_OF_BUFFER_CHAR} bytes), then yy_scan_buffer +returns a NULL pointer instead of creating a new input buffer. + +@deftp {Data type} yy_size_t +is an integral type to which you can cast an integer expression +reflecting the size of the buffer. +@end deftp + + + + +End-of-File Rules + + +The special rule @code{<>} indicates +actions which are to be taken when an end-of-file is +encountered and yywrap returns non-zero (i.e., indicates +no further files to process). The action must finish +by doing one of the following things: + + + + + + +assigning yyin to a new input file (in previous versions of +flex, after doing the assignment you had to call the special +action @code{YY_NEW_FILE}. This is no longer necessary.) + + + + +executing a @code{return} statement; + + + + +executing the special yyterminate action. + + + + +or, switching to a new buffer using yy_switch_to_buffer as +shown in the example above. + + + + +<> rules may not be used with other patterns; they may only be +qualified with a list of start conditions. If an unqualified <> +rule is given, it applies to @emph{all} start conditions which do not +already have <> actions. To specify an <> rule for only the +initial start condition, use: + + + +<> +]]> + + + +These rules are useful for catching things like unclosed comments. An +example: + + + + +<> { + error( "unterminated quote" ); + yyterminate(); + } + <> { + if ( *++filelist ) + yyin = fopen( *filelist, "r" ); + else + yyterminate(); + } +]]> + + + + + + +Miscellaneous Macros + +@hkindex YY_USER_ACTION +The macro @code{YY_USER_ACTION} can be defined to provide an action +which is always executed prior to the matched rule's action. For +example, it could be #define'd to call a routine to convert yytext to +lower-case. When @code{YY_USER_ACTION} is invoked, the variable +@code{yy_act} gives the number of the matched rule (rules are numbered +starting with 1). Suppose you want to profile how often each of your +rules is matched. The following would do the trick: + + + + + + + + + +where @code{ctr} is an array to hold the counts for the different rules. +Note that the macro @code{YY_NUM_RULES} gives the total number of rules +(including the default rule), even if you use @samp{-s)}, so a correct +declaration for @code{ctr} is: + + + + + + + +@hkindex YY_USER_INIT +The macro @code{YY_USER_INIT} may be defined to provide an action which +is always executed before the first scan (and before the scanner's +internal initializations are done). For example, it could be used to +call a routine to read in a data table or open a logging file. + + +The macro @code{yy_set_interactive(is_interactive)} can be used to +control whether the current buffer is considered @dfn{interactive}. An +interactive buffer is processed more slowly, but must be used when the +scanner's input source is indeed interactive to avoid problems due to +waiting to fill buffers (see the discussion of the @samp{-I} flag in +@ref{Scanner Options}). A non-zero value in the macro invocation marks +the buffer as interactive, a zero value as non-interactive. Note that +use of this macro overrides @code{%option always-interactive} or +@code{%option never-interactive} (@pxref{Scanner Options}). +yy_set_interactive must be invoked prior to beginning to scan +the buffer that is (or is not) to be considered interactive. + + + +The macro @code{yy_set_bol(at_bol)} can be used to control whether the +current buffer's scanning context for the next token match is done as +though at the beginning of a line. A non-zero macro argument makes +rules anchored with @samp{^} active, while a zero argument makes +@samp{^} rules inactive. + + + +The macro YY_AT_BOL returns true if the next token scanned from +the current buffer will have @samp{^} rules active, false otherwise. + + +@hkindex YY_BREAK +In the generated scanner, the actions are all gathered in one large +switch statement and separated using @code{YY_BREAK}, which may be +redefined. By default, it is simply a @code{break}, to separate each +rule's action from the following rule's. Redefining @code{YY_BREAK} +allows, for example, C++ users to #define YY_BREAK to do nothing (while +being very careful that every rule ends with a @code{break}" or a +@code{return}!) to avoid suffering from unreachable statement warnings +where because a rule's action ends with @code{return}, the +@code{YY_BREAK} is inaccessible. + + + + +Values Available To the User + +This chapter summarizes the various values available to the user in the +rule actions. + + + + +char *yytext + +holds the text of the current token. It may be modified but not +lengthened (you cannot append characters to the end). + + + + +If the special directive @code{%array} appears in the first section of +the scanner description, then yytext is instead declared +@code{char yytext[YYLMAX]}, where @code{YYLMAX} is a macro definition +that you can redefine in the first section if you don't like the default +value (generally 8KB). Using @code{%array} results in somewhat slower +scanners, but the value of yytext becomes immune to calls to +unput, which potentially destroy its value when yytext is +a character pointer. The opposite of @code{%array} is @code{%pointer}, +which is the default. + + +You cannot use @code{%array} when generating C++ scanner classes (the +@samp{-+} flag). + + + + + +int yyleng + +holds the length of the current token. + + + + + +FILE *yyin + +is the file which by default flex reads from. It may be +redefined but doing so only makes sense before scanning begins or after +an EOF has been encountered. Changing it in the midst of scanning will +have unexpected results since flex buffers its input; use +yyrestart instead. Once scanning terminates because an +end-of-file has been seen, you can assign yyin at the new input +file and then call the scanner again to continue scanning. + + + + + +void yyrestart( FILE *new_file ) + +may be called to point yyin at the new input file. The +switch-over to the new file is immediate (any previously buffered-up +input is lost). Note that calling yyrestart with yyin +as an argument thus throws away the current input buffer and continues +scanning the same input file. + + + + + +FILE *yyout + +is the file to which @code{ECHO} actions are done. It can be reassigned +by the user. + + + + + +YY_CURRENT_BUFFER + +returns a @code{YY_BUFFER_STATE} handle to the current buffer. + + + + + +YY_START + +returns an integer value corresponding to the current start condition. +You can subsequently use this value with @code{BEGIN} to return to that +start condition. + + + + + + + +Interfacing with <application>Yacc</application> + + + + +One of the main uses of flex is as a companion to the yacc +parser-generator. yacc parsers expect to call a routine named +yylex to find the next input token. The routine is supposed to +return the type of the next token as well as putting any associated +value in the global yylval. To use flex with yacc, +one specifies the @samp{-d} option to yacc to instruct it to +generate the file y.tab.h containing definitions of all the +@code{%tokens} appearing in the yacc input. This file is then +included in the flex scanner. For example, if one of the tokens +is @code{TOK_NUMBER}, part of the scanner might look like: + + + + + + + + + + + +Scanner Options + + + + + +The various flex options are categorized by function in the following +menu. If you want to lookup a particular option by name, @xref{Index of Scanner Options}. + + + + +Even though there are many scanner options, a typical scanner might only +specify the following options: + + + + + + + +The first line specifies the general type of scanner we want. The second line +specifies that we are being careful. The third line asks flex to track line +numbers. The last line tells flex what to name the files. (The options can be +specified in any order. We just dividied them.) + +flex also provides a mechanism for controlling options within the +scanner specification itself, rather than from the flex command-line. +This is done by including @code{%option} directives in the first section +of the scanner specification. You can specify multiple options with a +single @code{%option} directive, and multiple directives in the first +section of your flex input file. + +Most options are given simply as names, optionally preceded by the +word @samp{no} (with no intervening whitespace) to negate their meaning. +The names are the same as their long-option equivalents (but without the +leading @samp{--} ). + +flex scans your rule actions to determine whether you use the +@code{REJECT} or yymore features. The @code{REJECT} and +@code{yymore} options are available to override its decision as to +whether you use the options, either by setting them (e.g., @code{%option +reject)} to indicate the feature is indeed used, or unsetting them to +indicate it actually is not used (e.g., @code{%option noyymore)}. + + +A number of options are available for lint purists who want to suppress +the appearance of unneeded routines in the generated scanner. Each of +the following, if unset (e.g., @code{%option nounput}), results in the +corresponding routine not appearing in the generated scanner: + + + + + + + +(though yy_push_state and friends won't appear anyway unless +you use @code{%option stack)}. + +
+ +
+Options for Specifing Filenames + + + +@anchor{option-header} +@opindex ---header-file +@opindex header-file + +--header-file=FILE, @code{%option header-file="FILE"} + +instructs flex to write a C header to FILE. This file contains +function prototypes, extern variables, and types used by the scanner. +Only the external API is exported by the header file. Many macros that +are usable from within scanner actions are not exported to the header +file. This is due to namespace problems and the goal of a clean +external API. + +While in the header, the macro @code{yyIN_HEADER} is defined, where @samp{yy} +is substituted with the appropriate prefix. + +The @samp{--header-file} option is not compatible with the @samp{--c++} option, +since the C++ scanner provides its own header in yyFlexLexer.h. + + + +@anchor{option-outfile} +@opindex -o +@opindex ---outfile +@opindex outfile + + + +-oFILE, --outfile=FILE, @code{%option outfile="FILE"} + +directs flex to write the scanner to the file FILE instead of +lex.yy.c. If you combine @samp{--outfile} with the @samp{--stdout} option, +then the scanner is written to stdout but its @code{#line} +directives (see the @samp{-l} option above) refer to the file +FILE. + + + +@anchor{option-stdout} +@opindex -t +@opindex ---stdout +@opindex stdout + + + +-t, --stdout, @code{%option stdout} + +instructs flex to write the scanner it generates to standard +output instead of lex.yy.c. + + + +@opindex ---skel + + + +-SFILE, --skel=FILE + +overrides the default skeleton file from which +flex +constructs its scanners. You'll never need this option unless you are doing +flex +maintenance or development. + +@opindex ---tables-file +@opindex tables-file + + + +--tables-file=FILE + +Write serialized scanner dfa tables to FILE. The generated scanner will not +contain the tables, and requires them to be loaded at runtime. +@xref{serialization}. + +@opindex ---tables-verify +@opindex tables-verify + + + +--tables-verify + +This option is for flex development. We document it here in case you stumble +upon it by accident or in case you suspect some inconsistency in the serialized +tables. Flex will serialize the scanner dfa tables but will also generate the +in-code tables as it normally does. At runtime, the scanner will verify that +the serialized tables match the in-code tables, instead of loading them. + + + + + +
+ +
+Options Affecting Scanner Behavior + + +@anchor{option-case-insensitive} +@opindex -i +@opindex ---case-insensitive +@opindex case-insensitive + +-i, --case-insensitive, @code{%option case-insensitive} + +instructs flex to generate a @dfn{case-insensitive} scanner. The +case of letters given in the flex input patterns will be ignored, +and tokens in the input will be matched regardless of case. The matched +text given in yytext will have the preserved case (i.e., it will +not be folded). For tricky behavior, see @ref{case and character ranges}. + + + +@anchor{option-lex-compat} +@opindex -l +@opindex ---lex-compat +@opindex lex-compat + + + +-l, --lex-compat, @code{%option lex-compat} + +turns on maximum compatibility with the original AT&T @code{lex} +implementation. Note that this does not mean @emph{full} compatibility. +Use of this option costs a considerable amount of performance, and it +cannot be used with the @samp{--c++}, @samp{--full}, @samp{--fast}, @samp{-Cf}, or +@samp{-CF} options. For details on the compatibilities it provides, see +@ref{Lex and Posix}. This option also results in the name +@code{YY_FLEX_LEX_COMPAT} being @code{#define}'d in the generated scanner. + + + +@anchor{option-batch} +@opindex -B +@opindex ---batch +@opindex batch + + + +-B, --batch, @code{%option batch} + +instructs flex to generate a @dfn{batch} scanner, the opposite of +@emph{interactive} scanners generated by @samp{--interactive} (see below). In +general, you use @samp{-B} when you are @emph{certain} that your scanner +will never be used interactively, and you want to squeeze a +@emph{little} more performance out of it. If your goal is instead to +squeeze out a @emph{lot} more performance, you should be using the +@samp{-Cf} or @samp{-CF} options, which turn on @samp{--batch} automatically +anyway. + + + +@anchor{option-interactive} +@opindex -I +@opindex ---interactive +@opindex interactive + + + +-I, --interactive, @code{%option interactive} + +instructs flex to generate an @i{interactive} scanner. An +interactive scanner is one that only looks ahead to decide what token +has been matched if it absolutely must. It turns out that always +looking one extra character ahead, even if the scanner has already seen +enough text to disambiguate the current token, is a bit faster than only +looking ahead when necessary. But scanners that always look ahead give +dreadful interactive performance; for example, when a user types a +newline, it is not recognized as a newline token until they enter +@emph{another} token, which often means typing in another whole line. + +flex scanners default to @code{interactive} unless you use the +@samp{-Cf} or @samp{-CF} table-compression options +(@pxref{Performance}). That's because if you're looking for +high-performance you should be using one of these options, so if you +didn't, flex assumes you'd rather trade off a bit of run-time +performance for intuitive interactive behavior. Note also that you +@emph{cannot} use @samp{--interactive} in conjunction with @samp{-Cf} or +@samp{-CF}. Thus, this option is not really needed; it is on by default +for all those cases in which it is allowed. + +You can force a scanner to +@emph{not} +be interactive by using +@samp{--batch} + + + +@anchor{option-7bit} +@opindex -7 +@opindex ---7bit +@opindex 7bit + + + +-7, --7bit, @code{%option 7bit} + +instructs flex to generate a 7-bit scanner, i.e., one which can +only recognize 7-bit characters in its input. The advantage of using +@samp{--7bit} is that the scanner's tables can be up to half the size of +those generated using the @samp{--8bit}. The disadvantage is that such +scanners often hang or crash if their input contains an 8-bit character. + +Note, however, that unless you generate your scanner using the +@samp{-Cf} or @samp{-CF} table compression options, use of @samp{--7bit} +will save only a small amount of table space, and make your scanner +considerably less portable. flex's default behavior is to +generate an 8-bit scanner unless you use the @samp{-Cf} or @samp{-CF}, +in which case flex defaults to generating 7-bit scanners unless +your site was always configured to generate 8-bit scanners (as will +often be the case with non-USA sites). You can tell whether flex +generated a 7-bit or an 8-bit scanner by inspecting the flag summary in +the @samp{--verbose} output as described above. + +Note that if you use @samp{-Cfe} or @samp{-CFe} flex still +defaults to generating an 8-bit scanner, since usually with these +compression options full 8-bit tables are not much more expensive than +7-bit tables. + + + +@anchor{option-8bit} +@opindex -8 +@opindex ---8bit +@opindex 8bit + + + +-8, --8bit, @code{%option 8bit} + +instructs flex to generate an 8-bit scanner, i.e., one which can +recognize 8-bit characters. This flag is only needed for scanners +generated using @samp{-Cf} or @samp{-CF}, as otherwise flex defaults to +generating an 8-bit scanner anyway. + +See the discussion of +@samp{--7bit} +above for flex's default behavior and the tradeoffs between 7-bit +and 8-bit scanners. + + + +@anchor{option-default} +@opindex ---default +@opindex default + + + +--default, @code{%option default} + +generate the default rule. + + + +@anchor{option-always-interactive} +@opindex ---always-interactive +@opindex always-interactive + + + +--always-interactive, @code{%option always-interactive} + +instructs flex to generate a scanner which always considers its input +@emph{interactive}. Normally, on each new input file the scanner calls +isatty in an attempt to determine whether the scanner's input +source is interactive and thus should be read a character at a time. +When this option is used, however, then no such call is made. + + + +@opindex ---never-interactive + + + +--never-interactive, @code{--never-interactive} + +instructs flex to generate a scanner which never considers its input +interactive. This is the opposite of @code{always-interactive}. + + +@anchor{option-posix} +@opindex -X +@opindex ---posix +@opindex posix + + + +-X, --posix, @code{%option posix} + +turns on maximum compatibility with the POSIX 1003.2-1992 definition of +@code{lex}. Since flex was originally designed to implement the +POSIX definition of @code{lex} this generally involves very few changes +in behavior. At the current writing the known differences between +flex and the POSIX standard are: + + + + + +In POSIX and AT&T @code{lex}, the repeat operator, @samp{@{@}}, has lower +precedence than concatenation (thus @samp{ab@{3@}} yields @samp{ababab}). +Most POSIX utilities use an Extended Regular Expression (ERE) precedence +that has the precedence of the repeat operator higher than concatenation +(which causes @samp{ab@{3@}} to yield @samp{abbb}). By default, flex +places the precedence of the repeat operator higher than concatenation +which matches the ERE processing of other POSIX utilities. When either +@samp{--posix} or @samp{-l} are specified, flex will use the +traditional AT&T and POSIX-compliant precedence for the repeat operator +where concatenation has higher precedence than the repeat operator. + + + + + +@anchor{option-stack} +@opindex ---stack +@opindex stack + + + +--stack, @code{%option stack} + +enables the use of +start condition stacks (@pxref{Start Conditions}). + + + +@anchor{option-stdinit} +@opindex ---stdinit +@opindex stdinit + + + +--stdinit, @code{%option stdinit} + +if set (i.e., @b{%option stdinit)} initializes yyin and +yyout to stdin and stdout, instead of the default of +NULL. Some existing @code{lex} programs depend on this behavior, +even though it is not compliant with ANSI C, which does not require +stdin and stdout to be compile-time constant. In a +reentrant scanner, however, this is not a problem since initialization +is performed in yylex_init at runtime. + + + +@anchor{option-yylineno} +@opindex ---yylineno +@opindex yylineno + + + +--yylineno, @code{%option yylineno} + +directs flex to generate a scanner +that maintains the number of the current line read from its input in the +global variable yylineno. This option is implied by @code{%option +lex-compat}. In a reentrant C scanner, the macro yylineno is +accessible regardless of the value of @code{%option yylineno}, however, its +value is not modified by flex unless @code{%option yylineno} is enabled. + + + +@anchor{option-yywrap} +@opindex ---yywrap +@opindex yywrap + + + +--yywrap, @code{%option yywrap} + +if unset (i.e., @code{--noyywrap)}, makes the scanner not call +yywrap upon an end-of-file, but simply assume that there are no +more files to scan (until the user points yyin at a new file and +calls yylex again). + + + + + +
+ +
+Code-Level And API Options + + + +@anchor{option-ansi-definitions} +@opindex ---option-ansi-definitions +@opindex ansi-definitions + +--ansi-definitions, @code{%option ansi-definitions} + +instruct flex to generate ANSI C99 definitions for functions. +This option is enabled by default. +If @code{%option noansi-definitions} is specified, then the obsolete style +is generated. + +@anchor{option-ansi-prototypes} +@opindex ---option-ansi-prototypes +@opindex ansi-prototypes + + + +--ansi-prototypes, @code{%option ansi-prototypes} + +instructs flex to generate ANSI C99 prototypes for functions. +This option is enabled by default. +If @code{noansi-prototypes} is specified, then +prototypes will have empty parameter lists. + +@anchor{option-bison-bridge} +@opindex ---bison-bridge +@opindex bison-bridge + + + +--bison-bridge, @code{%option bison-bridge} + +instructs flex to generate a C scanner that is +meant to be called by a +@code{GNU bison} +parser. The scanner has minor API changes for +bison +compatibility. In particular, the declaration of +yylex +is modified to take an additional parameter, +yylval. +@xref{Bison Bridge}. + +@anchor{option-bison-locations} +@opindex ---bison-locations +@opindex bison-locations + + + +--bison-locations, @code{%option bison-locations} + +instruct flex that +@code{GNU bison} @code{%locations} are being used. +This means yylex will be passed +an additional parameter, yylloc. This option +implies @code{%option bison-bridge}. +@xref{Bison Bridge}. + +@anchor{option-noline} +@opindex -L +@opindex ---noline +@opindex noline + + + +-L, --noline, @code{%option noline} + +instructs +flex +not to generate +@code{#line} +directives. Without this option, +flex +peppers the generated scanner +with @code{#line} directives so error messages in the actions will be correctly +located with respect to either the original +flex +input file (if the errors are due to code in the input file), or +lex.yy.c +(if the errors are +flex's +fault -- you should report these sorts of errors to the email address +given in @ref{Reporting Bugs}). + + + +@anchor{option-reentrant} +@opindex -R +@opindex ---reentrant +@opindex reentrant + + + +-R, --reentrant, @code{%option reentrant} + +instructs flex to generate a reentrant C scanner. The generated scanner +may safely be used in a multi-threaded environment. The API for a +reentrant scanner is different than for a non-reentrant scanner +@pxref{Reentrant}). Because of the API difference between +reentrant and non-reentrant flex scanners, non-reentrant flex +code must be modified before it is suitable for use with this option. +This option is not compatible with the @samp{--c++} option. + +The option @samp{--reentrant} does not affect the performance of +the scanner. + + + +@anchor{option-c++} +@opindex -+ +@opindex ---c++ +@opindex c++ + + + +-+, --c++, @code{%option c++} + +specifies that you want flex to generate a C++ +scanner class. @xref{Cxx}, for +details. + + + +@anchor{option-array} +@opindex ---array +@opindex array + + + +--array, @code{%option array} + +specifies that you want yytext to be an array instead of a char* + + + +@anchor{option-pointer} +@opindex ---pointer +@opindex pointer + + + +--pointer, @code{%option pointer} + +specify that yytext should be a @code{char *}, not an array. +This default is @code{char *}. + + + +@anchor{option-prefix} +@opindex -P +@opindex ---prefix +@opindex prefix + + + +-PPREFIX, --prefix=PREFIX, @code{%option prefix="PREFIX"} + +changes the default @samp{yy} prefix used by flex for all +globally-visible variable and function names to instead be +@samp{PREFIX}. For example, @samp{--prefix=foo} changes the name of +yytext to @code{footext}. It also changes the name of the default +output file from lex.yy.c to lex.foo.c. Here is a partial +list of the names affected: + + + + + + + +(If you are using a C++ scanner, then only yywrap and +@code{yyFlexLexer} are affected.) Within your scanner itself, you can +still refer to the global variables and functions using either version +of their name; but externally, they have the modified name. + +This option lets you easily link together multiple +flex +programs into the same executable. Note, though, that using this +option also renames +yywrap, +so you now +@emph{must} +either +provide your own (appropriately-named) version of the routine for your +scanner, or use +@code{%option noyywrap}, +as linking with +@samp{-lfl} +no longer provides one for you by default. + + + +@anchor{option-main} +@opindex ---main +@opindex main + + + +--main, @code{%option main} + + directs flex to provide a default main program for the +scanner, which simply calls yylex. This option implies +@code{noyywrap} (see below). + + + +@anchor{option-nounistd} +@opindex ---nounistd +@opindex nounistd + + + +--nounistd, @code{%option nounistd} + +suppresses inclusion of the non-ANSI header file unistd.h. This option +is meant to target environments in which unistd.h does not exist. Be aware +that certain options may cause flex to generate code that relies on functions +normally found in unistd.h, (e.g. isatty, read.) +If you wish to use these functions, you will have to inform your compiler where +to find them. +@xref{option-always-interactive}. @xref{option-read}. + + + +@anchor{option-yyclass} +@opindex ---yyclass +@opindex yyclass + + + +--yyclass, @code{%option yyclass="NAME"} + +only applies when generating a C++ scanner (the @samp{--c++} option). It +informs flex that you have derived @code{foo} as a subclass of +@code{yyFlexLexer}, so flex will place your actions in the member +function @code{foo::yylex()} instead of @code{yyFlexLexer::yylex()}. It +also generates a @code{yyFlexLexer::yylex()} member function that emits +a run-time error (by invoking @code{yyFlexLexer::LexerError())} if +called. @xref{Cxx}. + + + + + +
+ +
+Options for Scanner Speed and Size + + + +-C[aefFmr] + +controls the degree of table compression and, more generally, trade-offs +between small scanners and fast scanners. + + +@opindex -C + + +-C + +A lone @samp{-C} specifies that the scanner tables should be compressed +but neither equivalence classes nor meta-equivalence classes should be +used. + +@anchor{option-align} +@opindex -Ca +@opindex ---align +@opindex align + + + +-Ca, --align, @code{%option align} + +(``align'') instructs flex to trade off larger tables in the +generated scanner for faster performance because the elements of +the tables are better aligned for memory access and computation. On some +RISC architectures, fetching and manipulating longwords is more efficient +than with smaller-sized units such as shortwords. This option can +quadruple the size of the tables used by your scanner. + +@anchor{option-ecs} +@opindex -Ce +@opindex ---ecs +@opindex ecs + + + +-Ce, --ecs, @code{%option ecs} + +directs flex to construct @dfn{equivalence classes}, i.e., sets +of characters which have identical lexical properties (for example, if +the only appearance of digits in the flex input is in the +character class ``[0-9]'' then the digits '0', '1', ..., '9' will all be +put in the same equivalence class). Equivalence classes usually give +dramatic reductions in the final table/object file sizes (typically a +factor of 2-5) and are pretty cheap performance-wise (one array look-up +per character scanned). + +@opindex -Cf + + + +-Cf + +specifies that the @dfn{full} scanner tables should be generated - +flex should not compress the tables by taking advantages of +similar transition functions for different states. + +@opindex -CF + + + +-CF + +specifies that the alternate fast scanner representation (described +above under the @samp{--fast} flag) should be used. This option cannot be +used with @samp{--c++}. + +@anchor{option-meta-ecs} +@opindex -Cm +@opindex ---meta-ecs +@opindex meta-ecs + + + +-Cm, --meta-ecs, @code{%option meta-ecs} + +directs +flex +to construct +@dfn{meta-equivalence classes}, +which are sets of equivalence classes (or characters, if equivalence +classes are not being used) that are commonly used together. Meta-equivalence +classes are often a big win when using compressed tables, but they +have a moderate performance impact (one or two @code{if} tests and one +array look-up per character scanned). + +@anchor{option-read} +@opindex -Cr +@opindex ---read +@opindex read + + + +-Cr, --read, @code{%option read} + +causes the generated scanner to @emph{bypass} use of the standard I/O +library (@code{stdio}) for input. Instead of calling fread or +getc, the scanner will use the read system call, +resulting in a performance gain which varies from system to system, but +in general is probably negligible unless you are also using @samp{-Cf} +or @samp{-CF}. Using @samp{-Cr} can cause strange behavior if, for +example, you read from yyin using @code{stdio} prior to calling +the scanner (because the scanner will miss whatever text your previous +reads left in the @code{stdio} input buffer). @samp{-Cr} has no effect +if you define YY_INPUT (@pxref{Generated Scanner}). + + + + + +The options @samp{-Cf} or @samp{-CF} and @samp{-Cm} do not make sense +together - there is no opportunity for meta-equivalence classes if the +table is not being compressed. Otherwise the options may be freely +mixed, and are cumulative. + +The default setting is @samp{-Cem}, which specifies that flex +should generate equivalence classes and meta-equivalence classes. This +setting provides the highest degree of table compression. You can trade +off faster-executing scanners at the cost of larger tables with the +following generally being true: + + + + + + + +Note that scanners with the smallest tables are usually generated and +compiled the quickest, so during development you will usually want to +use the default, maximal compression. + +@samp{-Cfe} is often a good compromise between speed and size for +production scanners. + +@anchor{option-full} +@opindex -f +@opindex ---full +@opindex full + + + +-f, --full, @code{%option full} + +specifies +@dfn{fast scanner}. +No table compression is done and @code{stdio} is bypassed. +The result is large but fast. This option is equivalent to +@samp{--Cfr} + + +@anchor{option-fast} +@opindex -F +@opindex ---fast +@opindex fast + + + +-F, --fast, @code{%option fast} + +specifies that the @emph{fast} scanner table representation should be +used (and @code{stdio} bypassed). This representation is about as fast +as the full table representation @samp{--full}, and for some sets of +patterns will be considerably smaller (and for others, larger). In +general, if the pattern set contains both @emph{keywords} and a +catch-all, @emph{identifier} rule, such as in the set: + + + + + + + +then you're better off using the full table representation. If only +the @emph{identifier} rule is present and you then use a hash table or some such +to detect the keywords, you're better off using +@samp{--fast}. + +This option is equivalent to @samp{-CFr} (see below). It cannot be used +with @samp{--c++}. + + + + + +
+ +
+Debugging Options + + + +@anchor{option-backup} +@opindex -b +@opindex ---backup +@opindex backup + +-b, --backup, @code{%option backup} + +Generate backing-up information to lex.backup. This is a list of +scanner states which require backing up and the input characters on +which they do so. By adding rules one can remove backing-up states. If +@emph{all} backing-up states are eliminated and @samp{-Cf} or @code{-CF} +is used, the generated scanner will run faster (see the @samp{--perf-report} flag). +Only users who wish to squeeze every last cycle out of their scanners +need worry about this option. (@pxref{Performance}). + + + +@anchor{option-debug} +@opindex -d +@opindex ---debug +@opindex debug + + + +-d, --debug, @code{%option debug} + +makes the generated scanner run in @dfn{debug} mode. Whenever a pattern +is recognized and the global variable @code{yy_flex_debug} is non-zero +(which is the default), the scanner will write to stderr a line +of the form: + + + + + + + +The line number refers to the location of the rule in the file defining +the scanner (i.e., the file that was fed to flex). Messages are also +generated when the scanner backs up, accepts the default rule, reaches +the end of its input buffer (or encounters a NUL; at this point, the two +look the same as far as the scanner's concerned), or reaches an +end-of-file. + + + +@anchor{option-perf-report} +@opindex -p +@opindex ---perf-report +@opindex perf-report + + + +-p, --perf-report, @code{%option perf-report} + +generates a performance report to stderr. The report consists of +comments regarding features of the flex input file which will +cause a serious loss of performance in the resulting scanner. If you +give the flag twice, you will also get comments regarding features that +lead to minor performance losses. + +Note that the use of @code{REJECT}, and +variable trailing context (@pxref{Limitations}) entails a substantial +performance penalty; use of yymore, the @samp{^} operator, and +the @samp{--interactive} flag entail minor performance penalties. + + + +@anchor{option-nodefault} +@opindex -s +@opindex ---nodefault +@opindex nodefault + + + +-s, --nodefault, @code{%option nodefault} + +causes the @emph{default rule} (that unmatched scanner input is echoed +to stdout) to be suppressed. If the scanner encounters input +that does not match any of its rules, it aborts with an error. This +option is useful for finding holes in a scanner's rule set. + + + +@anchor{option-trace} +@opindex -T +@opindex ---trace +@opindex trace + + + +-T, --trace, @code{%option trace} + +makes flex run in @dfn{trace} mode. It will generate a lot of +messages to stderr concerning the form of the input and the +resultant non-deterministic and deterministic finite automata. This +option is mostly for use in maintaining flex. + + + +@anchor{option-nowarn} +@opindex -w +@opindex ---nowarn +@opindex nowarn + + + +-w, --nowarn, @code{%option nowarn} + +suppresses warning messages. + + + +@anchor{option-verbose} +@opindex -v +@opindex ---verbose +@opindex verbose + + + +-v, --verbose, @code{%option verbose} + +specifies that flex should write to stderr a summary of +statistics regarding the scanner it generates. Most of the statistics +are meaningless to the casual flex user, but the first line +identifies the version of flex (same as reported by @samp{--version}), +and the next line the flags used when generating the scanner, including +those that are on by default. + + + +@anchor{option-warn} +@opindex ---warn +@opindex warn + + + +--warn, @code{%option warn} + +warn about certain things. In particular, if the default rule can be +matched but no defualt rule has been given, the flex will warn you. +We recommend using this option always. + + + + + +
+ +
+Miscellaneous Options + + +@opindex -c + +-c + +is a do-nothing option included for POSIX compliance. + +@opindex -h +@opindex ---help +generates + + + +-h, -?, --help + +generates a ``help'' summary of flex's options to stdout +and then exits. + +@opindex -n + + + +-n + +is another do-nothing option included only for +POSIX compliance. + +@opindex -V +@opindex ---version + + + +-V, --version + +prints the version number to stdout and exits. + + + + + + + + + +Performance Considerations + + +The main design goal of flex is that it generate high-performance +scanners. It has been optimized for dealing well with large sets of +rules. Aside from the effects on scanner speed of the table compression +@samp{-C} options outlined above, there are a number of options/actions +which degrade performance. These are, from most expensive to least: + + + + + + + + + + +with the first two all being quite expensive and the last two being +quite cheap. Note also that unput is implemented as a routine +call that potentially does quite a bit of work, while yyless is +a quite-cheap macro. So if you are just putting back some excess text +you scanned, use ss. + +@code{REJECT} should be avoided at all costs when performance is +important. It is a particularly expensive option. + +There is one case when @code{%option yylineno} can be expensive. That is when +your patterns match long tokens that could @emph{possibly} contain a newline +character. There is no performance penalty for rules that can not possibly +match newlines, since flex does not need to check them for newlines. In +general, you should avoid rules such as @code{[^f]+}, which match very long +tokens, including newlines, and may possibly match your entire file! A better +approach is to separate @code{[^f]+} into two rules: + + + + + + + +The above scanner does not incur a performance penalty. + + + + +Getting rid of backing up is messy and often may be an enormous amount +of work for a complicated scanner. In principal, one begins by using +the @samp{-b} flag to generate a lex.backup file. For example, +on the input: + + + + + + + + +the file looks like: + + + + + + + +The first few lines tell us that there's a scanner state in which it can +make a transition on an 'o' but not on any other character, and that in +that state the currently scanned text does not match any rule. The +state occurs when trying to match the rules found at lines 2 and 3 in +the input file. If the scanner is in that state and then reads +something other than an 'o', it will have to back up to find a rule +which is matched. With a bit of headscratching one can see that this +must be the state it's in when it has seen @samp{fo}. When this has +happened, if anything other than another @samp{o} is seen, the scanner +will have to back up to simply match the @samp{f} (by the default rule). + +The comment regarding State #8 indicates there's a problem when +@samp{foob} has been scanned. Indeed, on any character other than an +@samp{a}, the scanner will have to back up to accept "foo". Similarly, +the comment for State #9 concerns when @samp{fooba} has been scanned and +an @samp{r} does not follow. + +The final comment reminds us that there's no point going to all the +trouble of removing backing up from the rules unless we're using +@samp{-Cf} or @samp{-CF}, since there's no performance gain doing so +with compressed scanners. + + +The way to remove the backing up is to add ``error'' rules: + + + + + + + + +Eliminating backing up among a list of keywords can also be done using a +``catch-all'' rule: + + + + + + + + +This is usually the best solution when appropriate. + +Backing up messages tend to cascade. With a complicated set of rules +it's not uncommon to get hundreds of messages. If one can decipher +them, though, it often only takes a dozen or so rules to eliminate the +backing up (though it's easy to make a mistake and have an error rule +accidentally match a valid token. A possible future flex feature +will be to automatically add rules to eliminate backing up). + +It's important to keep in mind that you gain the benefits of eliminating +backing up only if you eliminate @emph{every} instance of backing up. +Leaving just one means you gain nothing. + +@emph{Variable} trailing context (where both the leading and trailing +parts do not have a fixed length) entails almost the same performance +loss as @code{REJECT} (i.e., substantial). So when possible a rule +like: + + + + + + + + +is better written: + + + + + + + +or as + + + + + + + +Note that here the special '|' action does @emph{not} provide any +savings, and can even make things worse (@pxref{Limitations}). + +Another area where the user can increase a scanner's performance (and +one that's easier to implement) arises from the fact that the longer the +tokens matched, the faster the scanner will run. This is because with +long tokens the processing of most input characters takes place in the +(short) inner scanning loop, and does not often have to go through the +additional work of setting up the scanning environment (e.g., +yytext) for the action. Recall the scanner for C comments: + + + + +[^*\n]* + "*"+[^*/\n]* + \n ++line_num; + "*"+"/" BEGIN(INITIAL); +]]> + + + +This could be sped up by writing it as: + + + +[^*\n]* + [^*\n]*\n ++line_num; + "*"+[^*/\n]* + "*"+[^*/\n]*\n ++line_num; + "*"+"/" BEGIN(INITIAL); +]]> + + + +Now instead of each newline requiring the processing of another action, +recognizing the newlines is distributed over the other rules to keep the +matched text as long as possible. Note that @emph{adding} rules does +@emph{not} slow down the scanner! The speed of the scanner is +independent of the number of rules or (modulo the considerations given +at the beginning of this section) how complicated the rules are with +regard to operators such as @samp{*} and @samp{|}. + + + +A final example in speeding up a scanner: suppose you want to scan +through a file containing identifiers and keywords, one per line +and with no other extraneous characters, and recognize all the +keywords. A natural first approach is: + + + + + + + + +To eliminate the back-tracking, introduce a catch-all rule: + + + + + + + +Now, if it's guaranteed that there's exactly one word per line, then we +can reduce the total number of matches by a half by merging in the +recognition of newlines with that of the other tokens: + + + + + + + +One has to be careful here, as we have now reintroduced backing up +into the scanner. In particular, while +@emph{we} +know that there will never be any characters in the input stream +other than letters or newlines, +flex +can't figure this out, and it will plan for possibly needing to back up +when it has scanned a token like @samp{auto} and then the next character +is something other than a newline or a letter. Previously it would +then just match the @samp{auto} rule and be done, but now it has no @samp{auto} +rule, only a @samp{auto\n} rule. To eliminate the possibility of backing up, +we could either duplicate all rules but without final newlines, or, +since we never expect to encounter such an input and therefore don't +how it's classified, we can introduce one more catch-all rule, this +one which doesn't include a newline: + + + + + + + +Compiled with @samp{-Cf}, this is about as fast as one can get a +flex scanner to go for this particular problem. + +A final note: flex is slow when matching @code{NUL}s, +particularly when a token contains multiple @code{NUL}s. It's best to +write rules which match @emph{short} amounts of text if it's anticipated +that the text will often include @code{NUL}s. + +Another final note regarding performance: as mentioned in +@ref{Matching}, dynamically resizing yytext to accommodate huge +tokens is a slow process because it presently requires that the (huge) +token be rescanned from the beginning. Thus if performance is vital, +you should attempt to match ``large'' quantities of text but not +``huge'' quantities, where the cutoff between the two is at about 8K +characters per token. + + + + +Generating C++ Scanners + + + +@strong{IMPORTANT}: the present form of the scanning class is @emph{experimental} +and may change considerably between major releases. + + + + +flex provides two different ways to generate scanners for use +with C++. The first way is to simply compile a scanner generated by +flex using a C++ compiler instead of a C compiler. You should +not encounter any compilation errors (@pxref{Reporting Bugs}). You can +then use C++ code in your rule actions instead of C code. Note that the +default input source for your scanner remains yyin, and default +echoing is still done to yyout. Both of these remain @code{FILE +*} variables and not C++ @emph{streams}. + +You can also use flex to generate a C++ scanner class, using the +@samp{-+} option (or, equivalently, @code{%option c++)}, which is +automatically specified if the name of the flex executable ends +in a '+', such as @code{flex++}. When using this option, flex +defaults to generating the scanner to the file lex.yy.cc instead +of lex.yy.c. The generated scanner includes the header file +FlexLexer.h, which defines the interface to two C++ classes. + +The first class, +@code{FlexLexer}, +provides an abstract base class defining the general scanner class +interface. It provides the following member functions: + + + + +const char* YYText() + +returns the text of the most recently matched token, the equivalent of +yytext. + + + + + +int YYLeng() + +returns the length of the most recently matched token, the equivalent of +yyleng. + + + + + +int lineno() const + +returns the current input line number (see @code{%option yylineno)}, or +@code{1} if @code{%option yylineno} was not used. + + + + + +void set_debug( int flag ) + +sets the debugging flag for the scanner, equivalent to assigning to +@code{yy_flex_debug} (@pxref{Scanner Options}). Note that you must build +the scannerusing @code{%option debug} to include debugging information +in it. + + + + + +int debug() const + +returns the current setting of the debugging flag. + + + + +Also provided are member functions equivalent to +yy_switch_to_buffer, yy_create_buffer (though the +first argument is an @code{istream*} object pointer and not a +@code{FILE*)}, yy_flush_buffer, yy_delete_buffer, and +yyrestart (again, the first argument is a @code{istream*} +object pointer). + + + +The second class defined in FlexLexer.h is @code{yyFlexLexer}, +which is derived from @code{FlexLexer}. It defines the following +additional member functions: + + + + +yyFlexLexer( istream* arg_yyin = 0, ostream* arg_yyout = 0 ) + +constructs a @code{yyFlexLexer} object using the given streams for input +and output. If not specified, the streams default to @code{cin} and +@code{cout}, respectively. + + + + + +virtual int yylex() + +performs the same role is yylex does for ordinary flex +scanners: it scans the input stream, consuming tokens, until a rule's +action returns a value. If you derive a subclass @code{S} from +@code{yyFlexLexer} and want to access the member functions and variables +of @code{S} inside yylex, then you need to use @code{%option +yyclass="S"} to inform flex that you will be using that subclass +instead of @code{yyFlexLexer}. In this case, rather than generating +@code{yyFlexLexer::yylex()}, flex generates @code{S::yylex()} +(and also generates a dummy @code{yyFlexLexer::yylex()} that calls +@code{yyFlexLexer::LexerError()} if called). + + + + + +virtual void switch_streams(istream* new_in = 0, ostream* new_out = 0) + +reassigns yyin to @code{new_in} (if non-null) and yyout to +@code{new_out} (if non-null), deleting the previous input buffer if +yyin is reassigned. + + + + +int yylex( istream* new_in, ostream* new_out = 0 ) + +first switches the input streams via @code{switch_streams( new_in, +new_out )} and then returns the value of yylex. + + + + +In addition, @code{yyFlexLexer} defines the following protected virtual +functions which you can redefine in derived classes to tailor the +scanner: + + + + +virtual int LexerInput( char* buf, int max_size ) + +reads up to @code{max_size} characters into @code{buf} and returns the +number of characters read. To indicate end-of-input, return 0 +characters. Note that @code{interactive} scanners (see the @samp{-B} +and @samp{-I} flags in @ref{Scanner Options}) define the macro +@code{YY_INTERACTIVE}. If you redefine LexerInput and need to +take different actions depending on whether or not the scanner might be +scanning an interactive input source, you can test for the presence of +this name via @code{#ifdef} statements. + + + + + +virtual void LexerOutput( const char* buf, int size ) + +writes out @code{size} characters from the buffer @code{buf}, which, while +@code{NUL}-terminated, may also contain internal @code{NUL}s if the +scanner's rules can match text with @code{NUL}s in them. + + + + + + +virtual void LexerError( const char* msg ) + +reports a fatal error message. The default version of this function +writes the message to the stream @code{cerr} and exits. + + + + +Note that a @code{yyFlexLexer} object contains its @emph{entire} +scanning state. Thus you can use such objects to create reentrant +scanners, but see also @ref{Reentrant}. You can instantiate multiple +instances of the same @code{yyFlexLexer} class, and you can also combine +multiple C++ scanner classes together in the same program using the +@samp{-P} option discussed above. + +Finally, note that the @code{%array} feature is not available to C++ +scanner classes; you must use @code{%pointer} (the default). + +Here is an example of a simple C++ scanner: + + + + +flexLexer* lexer = new yyFlexLexer; + while(lexer->yylex() != 0) + ; + return 0; + } +]]> + + + + +If you want to create multiple (different) lexer classes, you use the +@samp{-P} flag (or the @code{prefix=} option) to rename each +@code{yyFlexLexer} to some other @samp{xxFlexLexer}. You then can +include FlexLexer.h> in your other sources once per lexer class, +first renaming @code{yyFlexLexer} as follows: + + + + + + + + + #undef yyFlexLexer + #define yyFlexLexer zzFlexLexer + #include FlexLexer.h> +]]> + + + +if, for example, you used @code{%option prefix="xx"} for one of your +scanners and @code{%option prefix="zz"} for the other. + + + + +Reentrant C Scanners + + +flex has the ability to generate a reentrant C scanner. This is +accomplished by specifying @code{%option reentrant} (@samp{-R}) The generated +scanner is both portable, and safe to use in one or more separate threads of +control. The most common use for reentrant scanners is from within +multi-threaded applications. Any thread may create and execute a reentrant +flex scanner without the need for synchronization with other threads. + + + + +
+ +
+Uses for Reentrant Scanners + +However, there are other uses for a reentrant scanner. For example, you +could scan two or more files simultaneously to implement a @code{diff} at +the token level (i.e., instead of at the character level): + + + + + + + + +Another use for a reentrant scanner is recursion. +(Note that a recursive scanner can also be created using a non-reentrant scanner and +buffer states. @xref{Multiple Input Buffers}.) + +The following crude scanner supports the @samp{eval} command by invoking +another instance of itself. + + + + + + + + +
+ +
+An Overview of the Reentrant API + + +The API for reentrant scanners is different than for non-reentrant +scanners. Here is a quick overview of the API: + + + +@code{%option reentrant} must be specified. + + + +All functions take one additional argument: yyscanner + + + + +All global variables are replaced by their macro equivalents. +(We tell you this because it may be important to you during debugging.) + + + + +yylex_init and yylex_destroy must be called before and +after yylex, respectively. + + + + +Accessor methods (get/set functions) provide access to common +flex variables. + + + + +User-specific data can be stored in @code{yyextra}. + + + + +
+ +
+Reentrant Example + +First, an example of a reentrant scanner: + + + +\n yy_pop_state( yyscanner ); + [^\n]+ fprintf( yyout, "%s\n", yytext); + %% + int main ( int argc, char * argv[] ) + { + yyscan_t scanner; + + yylex_init ( &scanner ); + yylex ( scanner ); + yylex_destroy ( scanner ); + return 0; + } +]]> + + + +
+ +
+The Reentrant API in Detail + +Here are the things you need to do or know to use the reentrant C API of +flex. + + + + +
+ +
+Declaring a Scanner As Reentrant + + %option reentrant (--reentrant) must be specified. + +Notice that @code{%option reentrant} is specified in the above example +(@pxref{Reentrant Example}. Had this option not been specified, +flex would have happily generated a non-reentrant scanner without +complaining. You may explicitly specify @code{%option noreentrant}, if +you do @emph{not} want a reentrant scanner, although it is not +necessary. The default is to generate a non-reentrant scanner. + +
+ +
+The Extra Argument + + + +All functions take one additional argument: yyscanner. + +Notice that the calls to yy_push_state and yy_pop_state +both have an argument, yyscanner , that is not present in a +non-reentrant scanner. Here are the declarations of +yy_push_state and yy_pop_state in the generated scanner: + + + + + + + +Notice that the argument yyscanner appears in the declaration of +both functions. In fact, all flex functions in a reentrant +scanner have this additional argument. It is always the last argument +in the argument list, it is always of type @code{yyscan_t} (which is +typedef'd to @code{void *}) and it is +always named yyscanner. As you may have guessed, +yyscanner is a pointer to an opaque data structure encapsulating +the current state of the scanner. For a list of function declarations, +see @ref{Reentrant Functions}. Note that preprocessor macros, such as +@code{BEGIN}, @code{ECHO}, and @code{REJECT}, do not take this +additional argument. + +
+ +
+Global Variables Replaced By Macros + + +All global variables in traditional flex have been replaced by macro equivalents. + +Note that in the above example, yyout and yytext are +not plain variables. These are macros that will expand to their equivalent lvalue. +All of the familiar flex globals have been replaced by their macro +equivalents. In particular, yytext, yyleng, yylineno, +yyin, yyout, @code{yyextra}, yylval, and yylloc +are macros. You may safely use these macros in actions as if they were plain +variables. We only tell you this so you don't expect to link to these variables +externally. Currently, each macro expands to a member of an internal struct, e.g., + + + +yytext_r) +]]> + + + +One important thing to remember about +yytext +and friends is that +yytext +is not a global variable in a reentrant +scanner, you can not access it directly from outside an action or from +other functions. You must use an accessor method, e.g., +yyget_text, +to accomplish this. (See below). + +
+ +
+Init and Destroy Functions + + + + + + +yylex_init and yylex_destroy must be called before and +after yylex, respectively. + + + + + + + +The function yylex_init must be called before calling any other +function. The argument to yylex_init is the address of an +uninitialized pointer to be filled in by flex. The contents of +@code{ptr_yy_globals} need not be initialized, since flex will +overwrite it anyway. The value stored in @code{ptr_yy_globals} should +thereafter be passed to yylex and @b{yylex_destroy()}. Flex +does not save the argument passed to yylex_init, so it is safe to +pass the address of a local pointer to yylex_init. The function +yylex should be familiar to you by now. The reentrant version +takes one argument, which is the value returned (via an argument) by +yylex_init. Otherwise, it behaves the same as the non-reentrant +version of yylex. + +yylex_init returns 0 (zero) on success, or non-zero on failure, +in which case, errno is set to one of the following values: + + + + + ENOMEM +Memory allocation error. @xref{memory-management}. + + + EINVAL +Invalid argument. + + + + + +The function yylex_destroy should be +called to free resources used by the scanner. After yylex_destroy +is called, the contents of yyscanner should not be used. Of +course, there is no need to destroy a scanner if you plan to reuse it. +A flex scanner (both reentrant and non-reentrant) may be +restarted by calling yyrestart. + +Below is an example of a program that creates a scanner, uses it, then destroys +it when done: + + + + 0) + printf("tok=%d yytext=%s\n", tok, yyget_text(scanner)); + + yylex_destroy(scanner); + return 0; + } +]]> + + + +
+ +
+Accessing Variables with Reentrant Scanners + + +Accessor methods (get/set functions) provide access to common +flex variables. + +Many scanners that you build will be part of a larger project. Portions +of your project will need access to flex values, such as +yytext. In a non-reentrant scanner, these values are global, so +there is no problem accessing them. However, in a reentrant scanner, there are no +global flex values. You can not access them directly. Instead, +you must access flex values using accessor methods (get/set +functions). Each accessor method is named yyget_NAME or +yyset_NAME, where @code{NAME} is the name of the flex +variable you want. For example: + + + + + + + + +The above code may be called from within an action like this: + + + + + + + +You may find that @code{%option header-file} is particularly useful for generating +prototypes of all the accessor functions. @xref{option-header}. + +
+ +
+Extra Data + + + +User-specific data can be stored in @code{yyextra}. + +In a reentrant scanner, it is unwise to use global variables to +communicate with or maintain state between different pieces of your program. +However, you may need access to external data or invoke external functions +from within the scanner actions. +Likewise, you may need to pass information to your scanner +(e.g., open file descriptors, or database connections). +In a non-reentrant scanner, the only way to do this would be through the +use of global variables. +flex allows you to store arbitrary, ``extra'' data in a scanner. +This data is accessible through the accessor methods +yyget_extra +and +yyset_extra +from outside the scanner, and through the shortcut macro +@code{yyextra} +from within the scanner itself. They are defined as follows: + + + + + + + + + + +By default, @code{YY_EXTRA_TYPE} is defined as type @code{void *}. You +will have to cast @code{yyextra} and the return value from +yyget_extra to the appropriate value each time you access the +extra data. To avoid casting, you may override the default type by +defining @code{YY_EXTRA_TYPE} in section 1 of your scanner: + + + + + + #include + #define YY_EXTRA_TYPE struct stat* + %} + %option reentrant + %% + + __filesize__ printf( "%ld", yyextra->st_size ); + __lastmod__ printf( "%ld", yyextra->st_mtime ); + %% + void scan_file( char* filename ) + { + yyscan_t scanner; + struct stat buf; + + yylex_init ( &scanner ); + yyset_in( fopen(filename,"r"), scanner ); + + stat( filename, &buf); + yyset_extra( &buf, scanner ); + yylex ( scanner ); + yylex_destroy( scanner ); + } +]]> + + + + +
+ +
+About yyscan_t + + +@code{yyscan_t} is defined as: + + + + + + + +It is initialized by yylex_init to point to +an internal structure. You should never access this value +directly. In particular, you should never attempt to free it +(use yylex_destroy instead.) + +
+ +
+Functions and Macros Available in Reentrant C Scanners + +The following Functions are available in a reentrant scanner: + + + + + + + + + + + + + + + + + + + + +There are no ``set'' functions for yytext and yyleng. This is intentional. + +The following Macro shortcuts are available in actions in a reentrant +scanner: + + + + + + + + +In a reentrant C scanner, support for yylineno is always present +(i.e., you may access yylineno), but the value is never modified by +flex unless @code{%option yylineno} is enabled. This is to allow +the user to maintain the line count independently of flex. + +@anchor{bison-functions} +The following functions and macros are made available when @code{%option +bison-bridge} (@samp{--bison-bridge}) is specified: + + + + + + + +The following functions and macros are made available +when @code{%option bison-locations} (@samp{--bison-locations}) is specified: + + + + + + + +Support for yylval assumes that @code{YYSTYPE} is a valid type. Support for +yylloc assumes that @code{YYSLYPE} is a valid type. Typically, these types are +generated by bison, and are included in section 1 of the flex +input. + + + + +Incompatibilities with Lex and Posix + + + + +flex is a rewrite of the AT&T Unix @emph{lex} tool (the two +implementations do not share any code, though), with some extensions and +incompatibilities, both of which are of concern to those who wish to +write scanners acceptable to both implementations. flex is fully +compliant with the POSIX @code{lex} specification, except that when +using @code{%pointer} (the default), a call to unput destroys +the contents of yytext, which is counter to the POSIX +specification. In this section we discuss all of the known areas of +incompatibility between flex, AT&T @code{lex}, and the POSIX +specification. flex's @samp{-l} option turns on maximum +compatibility with the original AT&T @code{lex} implementation, at the +cost of a major loss in the generated scanner's performance. We note +below which incompatibilities can be overcome using the @samp{-l} +option. flex is fully compatible with @code{lex} with the +following exceptions: + + + + + +The undocumented @code{lex} scanner internal variable yylineno is +not supported unless @samp{-l} or @code{%option yylineno} is used. + + + + +yylineno should be maintained on a per-buffer basis, rather than +a per-scanner (single global variable) basis. + + + + +yylineno is not part of the POSIX specification. + + + + +The input routine is not redefinable, though it may be called +to read characters following whatever has been matched by a rule. If +input encounters an end-of-file the normal yywrap +processing is done. A ``real'' end-of-file is returned by +input as @code{EOF}. + + + + +Input is instead controlled by defining the YY_INPUT macro. + + + + +The flex restriction that input cannot be redefined is +in accordance with the POSIX specification, which simply does not +specify any way of controlling the scanner's input other than by making +an initial assignment to yyin. + + + + +The unput routine is not redefinable. This restriction is in +accordance with POSIX. + + + + +flex scanners are not as reentrant as @code{lex} scanners. In +particular, if you have an interactive scanner and an interrupt handler +which long-jumps out of the scanner, and the scanner is subsequently +called again, you may get the following message: + + + + +flex scanner internal error--end of buffer missed +]]> + + + +To reenter the scanner, first use: + + + + + + + + +Note that this call will throw away any buffered input; usually this +isn't a problem with an interactive scanner. @xref{Reentrant}, for +flex's reentrant API. + + + + +Also note that flex C++ scanner classes +@emph{are} +reentrant, so if using C++ is an option for you, you should use +them instead. @xref{Cxx}, and @ref{Reentrant} for details. + + + + +output is not supported. Output from the @b{ECHO} macro is +done to the file-pointer yyout (default stdout). + + + + +output is not part of the POSIX specification. + + + + +@code{lex} does not support exclusive start conditions (%x), though they +are in the POSIX specification. + + + + +When definitions are expanded, flex encloses them in parentheses. +With @code{lex}, the following: + + + + + + + + +will not match the string @samp{foo} because when the macro is expanded +the rule is equivalent to @samp{foo[A-Z][A-Z0-9]*?} and the precedence +is such that the @samp{?} is associated with @samp{[A-Z0-9]*}. With +flex, the rule will be expanded to @samp{foo([A-Z][A-Z0-9]*)?} +and so the string @samp{foo} will match. + + + + +Note that if the definition begins with @samp{^} or ends with @samp{$} +then it is @emph{not} expanded with parentheses, to allow these +operators to appear in definitions without losing their special +meanings. But the @samp{}, @samp{/}, and @code{<>} operators +cannot be used in a flex definition. + + + + +Using @samp{-l} results in the @code{lex} behavior of no parentheses +around the definition. + + + + +The POSIX specification is that the definition be enclosed in parentheses. + + + + +Some implementations of @code{lex} allow a rule's action to begin on a +separate line, if the rule's pattern has trailing whitespace: + + + + + + { foobar_action();} +]]> + + + +flex does not support this feature. + + + + +The @code{lex} @code{%r} (generate a Ratfor scanner) option is not +supported. It is not part of the POSIX specification. + + + + +After a call to unput, @emph{yytext} is undefined until the +next token is matched, unless the scanner was built using @code{%array}. +This is not the case with @code{lex} or the POSIX specification. The +@samp{-l} option does away with this incompatibility. + + + + +The precedence of the @samp{@{,@}} (numeric range) operator is +different. The AT&T and POSIX specifications of @code{lex} +interpret @samp{abc@{1,3@}} as match one, two, +or three occurrences of @samp{abc}'', whereas flex interprets it +as ``match @samp{ab} followed by one, two, or three occurrences of +@samp{c}''. The @samp{-l} and @samp{--posix} options do away with this +incompatibility. + + + + +The precedence of the @samp{^} operator is different. @code{lex} +interprets @samp{^foo|bar} as ``match either 'foo' at the beginning of a +line, or 'bar' anywhere'', whereas flex interprets it as ``match +either @samp{foo} or @samp{bar} if they come at the beginning of a +line''. The latter is in agreement with the POSIX specification. + + + + +The special table-size declarations such as @code{%a} supported by +@code{lex} are not required by flex scanners.. flex +ignores them. + + + +The name @code{FLEX_SCANNER} is @code{#define}'d so scanners may be +written for use with either flex or @code{lex}. Scanners also +include @code{YY_FLEX_MAJOR_VERSION}, @code{YY_FLEX_MINOR_VERSION} +and @code{YY_FLEX_SUBMINOR_VERSION} +indicating which version of flex generated the scanner. For +example, for the 2.5.22 release, these defines would be 2, 5 and 22 +respectively. If the version of flex being used is a beta +version, then the symbol @code{FLEX_BETA} is defined. + + + + + + +The following flex features are not included in @code{lex} or the +POSIX specification: + + + + + +C++ scanners + + + +%option + + + +start condition scopes + + + +start condition stacks + + + +interactive/non-interactive scanners + + + +yy_scan_string() and friends + + + +yyterminate() + + + +yy_set_interactive() + + + +yy_set_bol() + + + +YY_AT_BOL() + <> + + + +<*> + + + +YY_DECL + + + +YY_START + + + +YY_USER_ACTION + + + +YY_USER_INIT + + + +#line directives + + + +%@{@}'s around actions + + + +reentrant C API + + + +multiple actions on a line + + + +almost all of the flex command-line options + + + + +The feature ``multiple actions on a line'' +refers to the fact that with flex you can put multiple actions on +the same line, separated with semi-colons, while with @code{lex}, the +following: + + + + + + + +is (rather surprisingly) truncated to + + + + + + + +flex does not truncate the action. Actions that are not enclosed +in braces are simply terminated at the end of the line. + + + + +Memory Management + + +@anchor{memory-management} +This chapter describes how flex handles dynamic memory, and how you can +override the default behavior. + + + + +
+ +
+The Default Memory Management + +Flex allocates dynamic memory during initialization, and once in a while from +within a call to yylex(). Initialization takes place during the first call to +yylex(). Thereafter, flex may reallocate more memory if it needs to enlarge a +buffer. As of version 2.5.9 Flex will clean up all memory when you call yylex_destroy +@xref{faq-memory-leak}. + +Flex allocates dynamic memory for four purposes, listed below @footnote{The +quantities given here are approximate, and may vary due to host architecture, +compiler configuration, or due to future enhancements to flex.} + + + + +16kB for the input buffer. + +Flex allocates memory for the character buffer used to perform pattern +matching. Flex must read ahead from the input stream and store it in a large +character buffer. This buffer is typically the largest chunk of dynamic memory +flex consumes. This buffer will grow if necessary, doubling the size each time. +Flex frees this memory when you call yylex_destroy(). The default size of this +buffer (16384 bytes) is almost always too large. The ideal size for this +buffer is the length of the longest token expected. Flex will allocate a few +extra bytes for housekeeping. + + + + +16kb for the REJECT state. This will only be allocated if you use REJECT. + +The size is the same as the input buffer, so if you override the size of the +input buffer, then you automatically override the size of this buffer as well. + + + + +100 bytes for the start condition stack. + +Flex allocates memory for the start condition stack. This is the stack used +for pushing start states, i.e., with yy_push_state(). It will grow if +necessary. Since the states are simply integers, this stack doesn't consume +much memory. This stack is not present if @code{%option stack} is not +specified. You will rarely need to tune this buffer. The ideal size for this +stack is the maximum depth expected. The memory for this stack is +automatically destroyed when you call yylex_destroy(). @xref{option-stack}. + + + + +40 bytes for each YY_BUFFER_STATE. + +Flex allocates memory for each YY_BUFFER_STATE. The buffer state itself +is about 40 bytes, plus an additional large character buffer (described above.) +The initial buffer state is created during initialization, and with each call +to yy_create_buffer(). You can't tune the size of this, but you can tune the +character buffer as described above. Any buffer state that you explicitly +create by calling yy_create_buffer() is @emph{NOT} destroyed automatically. You +must call yy_delete_buffer() to free the memory. The exception to this rule is +that flex will delete the current buffer automatically when you call +yylex_destroy(). If you delete the current buffer, be sure to set it to NULL. +That way, flex will not try to delete the buffer a second time (possibly +crashing your program!) At the time of this writing, flex does not provide a +growable stack for the buffer states. You have to manage that yourself. +@xref{Multiple Input Buffers}. + + + + +84 bytes for the reentrant scanner guts + +Flex allocates about 84 bytes for the reentrant scanner structure when +you call yylex_init(). It is destroyed when the user calls yylex_destroy(). + + + + + + +
+ +
+Overriding The Default Memory Management + + + + + +Flex calls the functions yyalloc, yyrealloc, and yyfree +when it needs to allocate or free memory. By default, these functions are +wrappers around the standard C functions, @code{malloc}, @code{realloc}, and +@code{free}, respectively. You can override the default implementations by telling +flex that you will provide your own implementations. + +To override the default implementations, you must do two things: + + + + + + Suppress the default implementations by specifying one or more of the +following options: + + + +@opindex noyyalloc + + @code{%option noyyalloc} + + + @code{%option noyyrealloc} + + + @code{%option noyyfree}. + + + + + + + Provide your own implementation of the following functions: @footnote{It +is not necessary to override all (or any) of the memory management routines. +You may, for example, override yyrealloc, but not yyfree or +yyalloc.} + + + + + + + + + + + +In the following example, we will override all three memory routines. We assume +that there is a custom allocator with garbage collection. In order to make this +example interesting, we will use a reentrant scanner, passing a pointer to the +custom allocator through @code{yyextra}. + + + + + + + + + +
+ +
+A Note About yytext And Memory + + + +When flex finds a match, yytext points to the first character of the +match in the input buffer. The string itself is part of the input buffer, and +is @emph{NOT} allocated separately. The value of yytext will be overwritten the next +time yylex() is called. In short, the value of yytext is only valid from within +the matched rule's action. + +Often, you want the value of yytext to persist for later processing, i.e., by a +parser with non-zero lookahead. In order to preserve yytext, you will have to +copy it with strdup() or a similar function. But this introduces some headache +because your parser is now responsible for freeing the copy of yytext. If you +use a yacc or bison parser, (commonly used with flex), you will discover that +the error recovery mechanisms can cause memory to be leaked. + +To prevent memory leaks from strdup'd yytext, you will have to track the memory +somehow. Our experience has shown that a garbage collection mechanism or a +pooled memory mechanism will save you a lot of grief when writing parsers. + + + + +Serialized Tables + + + +@anchor{serialization} +A flex scanner has the ability to save the DFA tables to a file, and +load them at runtime when needed. The motivation for this feature is to reduce +the runtime memory footprint. Traditionally, these tables have been compiled into +the scanner as C arrays, and are sometimes quite large. Since the tables are +compiled into the scanner, the memory used by the tables can never be freed. +This is a waste of memory, especially if an application uses several scanners, +but none of them at the same time. + +The serialization feature allows the tables to be loaded at runtime, before +scanning begins. The tables may be discarded when scanning is finished. + + + + +
+ +
+Creating Serialized Tables + + + +You may create a scanner with serialized tables by specifying: + + + + + + + +These options instruct flex to save the DFA tables to the file @var{FILE}. The tables +will @emph{not} be embedded in the generated scanner. The scanner will not +function on its own. The scanner will be dependent upon the serialized tables. You must +load the tables from this file at runtime before you can scan anything. + +If you do not specify a filename to @code{--tables-file}, the tables will be +saved to lex.yy.tables, where @samp{yy} is the appropriate prefix. + +If your project uses several different scanners, you can concatenate the +serialized tables into one file, and flex will find the correct set of tables, +using the scanner prefix as part of the lookup key. An example follows: + + + + + all.tables +]]> + + + +The above example created two scanners, @samp{cpp}, and @samp{c}. Since we did +not specify a filename, the tables were serialized to lex.c.tables and +lex.cpp.tables, respectively. Then, we concatenated the two files +together into all.tables, which we will distribute with our project. At +runtime, we will open the file and tell flex to load the tables from it. Flex +will find the correct tables automatically. (See next section). + +
+ +
+Loading and Unloading Serialized Tables + + + + + + +If you've built your scanner with @code{%option tables-file}, then you must +load the scanner tables at runtime. This can be accomplished with the following +function: + +@deftypefun int yytables_fload (FILE* @var{fp} [, yyscan_t @var{scanner}]) +Locates scanner tables in the stream pointed to by @var{fp} and loads them. +Memory for the tables is allocated via yyalloc. You must call this +function before the first call to yylex. The argument @var{scanner} +only appears in the reentrant scanner. +This function returns @samp{0} (zero) on success, or non-zero on error. +@end deftypefun + +The loaded tables are @strong{not} automatically destroyed (unloaded) when you +call yylex_destroy. The reason is that you may create several scanners +of the same type (in a reentrant scanner), each of which needs access to these +tables. To avoid a nasty memory leak, you must call the following function: + +@deftypefun int yytables_destroy ([yyscan_t @var{scanner}]) +Unloads the scanner tables. The tables must be loaded again before you can scan +any more data. The argument @var{scanner} only appears in the reentrant +scanner. This function returns @samp{0} (zero) on success, or non-zero on +error. +@end deftypefun + +@strong{The functions yytables_fload and yytables_destroy are not thread-safe.} You must ensure that these functions are called exactly once (for +each scanner type) in a threaded program, before any thread calls yylex. +After the tables are loaded, they are never written to, and no thread +protection is required thereafter -- until you destroy them. + +
+ +
+Tables File Format + + + +This section defines the file format of serialized flex tables. + +The tables format allows for one or more sets of tables to be +specified, where each set corresponds to a given scanner. Scanners are +indexed by name, as described below. The file format is as follows: + + + + + + + +The above diagram shows that a complete set of tables consists of a header +followed by multiple individual tables. Furthermore, multiple complete sets may +be present in the same file, each set with its own header and tables. The sets +are contiguous in the file. The only way to know if another set follows is to +check the next four bytes for the magic number (or check for EOF). The header +and tables sections are padded to 64-bit boundaries. Below we describe each +field in detail. This format does not specify how the scanner will expand the +given data, i.e., data may be serialized as int8, but expanded to an int32 +array at runtime. This is to reduce the size of the serialized data where +possible. Remember, @emph{all integer values are in network byte order}. + +@noindent +Fields of a table header: + + + +th_magic + +Magic number, always 0xF13C57B1. + + + + +th_hsize + +Size of this entire header, in bytes, including all fields plus any padding. + + + + +th_ssize + +Size of this entire set, in bytes, including the header, all tables, plus +any padding. + + + + +th_flags + +Bit flags for this table set. Currently unused. + + + + +th_version[] + +Flex version in NULL-termninated string format. e.g., @samp{2.5.13a}. This is +the version of flex that was used to create the serialized tables. + + + + +th_name[] + +Contains the name of this table set. The default is @samp{yytables}, +and is prefixed accordingly, e.g., @samp{footables}. Must be NULL-terminated. + + + + +th_pad64[] + +Zero or more NULL bytes, padding the entire header to the next 64-bit boundary +as calculated from the beginning of the header. + + + + +@noindent +Fields of a table: + + + +td_id + +Specifies the table identifier. Possible values are: + + +YYTD_ID_ACCEPT (0x01) + +@code{yy_accept} + + + +YYTD_ID_BASE (0x02) + +@code{yy_base} + + + +YYTD_ID_CHK (0x03) + +@code{yy_chk} + + + +YYTD_ID_DEF (0x04) + +@code{yy_def} + + + +YYTD_ID_EC (0x05) + +@code{yy_ec } + + + +YYTD_ID_META (0x06) + +@code{yy_meta} + + + +YYTD_ID_NUL_TRANS (0x07) + +@code{yy_NUL_trans} + + + +YYTD_ID_NXT (0x08) + +@code{yy_nxt}. This array may be two dimensional. See the td_hilen +field below. + + + +YYTD_ID_RULE_CAN_MATCH_EOL (0x09) + +@code{yy_rule_can_match_eol} + + + +YYTD_ID_START_STATE_LIST (0x0A) + +@code{yy_start_state_list}. This array is handled specially because it is an +array of pointers to structs. See the td_flags field below. + + + +YYTD_ID_TRANSITION (0x0B) + +@code{yy_transition}. This array is handled specially because it is an array of +structs. See the td_lolen field below. + + + +YYTD_ID_ACCLIST (0x0C) + +@code{yy_acclist} + + + + + + + +td_flags + +Bit flags describing how to interpret the data in td_data. +The data arrays are one-dimensional by default, but may be +two dimensional as specified in the td_hilen field. + + + +YYTD_DATA8 (0x01) + +The data is serialized as an array of type int8. + + + +YYTD_DATA16 (0x02) + +The data is serialized as an array of type int16. + + + +YYTD_DATA32 (0x04) + +The data is serialized as an array of type int32. + + + +YYTD_PTRANS (0x08) + +The data is a list of indexes of entries in the expanded @code{yy_transition} +array. Each index should be expanded to a pointer to the corresponding entry +in the @code{yy_transition} array. We count on the fact that the +@code{yy_transition} array has already been seen. + + + +YYTD_STRUCT (0x10) + +The data is a list of yy_trans_info structs, each of which consists of +two integers. There is no padding between struct elements or between structs. +The type of each member is determined by the @code{YYTD_DATA*} bits. + + + + + + + +td_lolen + +Specifies the number of elements in the lowest dimension array. If this is +a one-dimensional array, then it is simply the number of elements in this array. +The element size is determined by the td_flags field. + + + + +td_hilen + +If td_hilen is non-zero, then the data is a two-dimensional array. +Otherwise, the data is a one-dimensional array. td_hilen contains the +number of elements in the higher dimensional array, and td_lolen contains +the number of elements in the lowest dimension. + +Conceptually, td_data is either @code{sometype td_data[td_lolen]}, or +@code{sometype td_data[td_hilen][td_lolen]}, where @code{sometype} is specified +by the td_flags field. It is possible for both td_lolen and +td_hilen to be zero, in which case td_data is a zero length +array, and no data is loaded, i.e., this table is simply skipped. Flex does not +currently generate tables of zero length. + + + + +td_data[] + +The table data. This array may be a one- or two-dimensional array, of type +@code{int8}, @code{int16}, @code{int32}, @code{struct yy_trans_info}, or +@code{struct yy_trans_info*}, depending upon the values in the +td_flags, td_lolen, and td_hilen fields. + + + + +td_pad64[] + +Zero or more NULL bytes, padding the entire table to the next 64-bit boundary as +calculated from the beginning of this table. + + + + + + + +Diagnostics + + + + +The following is a list of flex diagnostic messages: + + + + + +@samp{warning, rule cannot be matched} indicates that the given rule +cannot be matched because it follows other rules that will always match +the same text as it. For example, in the following @samp{foo} cannot be +matched because it comes after an identifier ``catch-all'' rule: + + + + + + + + +Using @code{REJECT} in a scanner suppresses this warning. + + + + +@samp{warning, -s option given but default rule can be matched} means +that it is possible (perhaps only in a particular start condition) that +the default rule (match any single character) is the only one that will +match a particular input. Since @samp{-s} was given, presumably this is +not intended. + + + + +@code{reject_used_but_not_detected undefined} or +@code{yymore_used_but_not_detected undefined}. These errors can occur +at compile time. They indicate that the scanner uses @code{REJECT} or +yymore but that flex failed to notice the fact, meaning +that flex scanned the first two sections looking for occurrences +of these actions and failed to find any, but somehow you snuck some in +(via a #include file, for example). Use @code{%option reject} or +@code{%option yymore} to indicate to flex that you really do use +these features. + + + + +@samp{flex scanner jammed}. a scanner compiled with +@samp{-s} has encountered an input string which wasn't matched by any of +its rules. This error can also occur due to internal problems. + + + + +@samp{token too large, exceeds YYLMAX}. your scanner uses @code{%array} +and one of its rules matched a string longer than the @code{YYLMAX} +constant (8K bytes by default). You can increase the value by +#define'ing @code{YYLMAX} in the definitions section of your flex +input. + + + + +@samp{scanner requires -8 flag to use the character 'x'}. Your scanner +specification includes recognizing the 8-bit character @samp{'x'} and +you did not specify the -8 flag, and your scanner defaulted to 7-bit +because you used the @samp{-Cf} or @samp{-CF} table compression options. +See the discussion of the @samp{-7} flag, @ref{Scanner Options}, for +details. + + + + +@samp{flex scanner push-back overflow}. you used unput to push +back so much text that the scanner's buffer could not hold both the +pushed-back text and the current token in yytext. Ideally the +scanner should dynamically resize the buffer in this case, but at +present it does not. + + + + +@samp{input buffer overflow, can't enlarge buffer because scanner uses +REJECT}. the scanner was working on matching an extremely large token +and needed to expand the input buffer. This doesn't work with scanners +that use @code{REJECT}. + + + + +@samp{fatal flex scanner internal error--end of buffer missed}. This can +occur in a scanner which is reentered after a long-jump has jumped out +(or over) the scanner's activation frame. Before reentering the +scanner, use: + + + + + +or, as noted above, switch to using the C++ scanner class. + + + + +@samp{too many start conditions in <> construct!} you listed more start +conditions in a <> construct than exist (so you must have listed at +least one of them twice). + + + + + + + +Limitations + + + +Some trailing context patterns cannot be properly matched and generate +warning messages (@samp{dangerous trailing context}). These are +patterns where the ending of the first part of the rule matches the +beginning of the second part, such as @samp{zx*/xy*}, where the 'x*' +matches the 'x' at the beginning of the trailing context. (Note that +the POSIX draft states that the text matched by such patterns is +undefined.) For some trailing context rules, parts which are actually +fixed-length are not recognized as such, leading to the abovementioned +performance loss. In particular, parts using @samp{|} or @samp{@{n@}} +(such as @samp{foo@{3@}}) are always considered variable-length. +Combining trailing context with the special @samp{|} action can result +in @emph{fixed} trailing context being turned into the more expensive +@emph{variable} trailing context. For example, in the following: + + + + + + + + +Use of unput invalidates yytext and yyleng, unless the +@code{%array} directive or the @samp{-l} option has been used. +Pattern-matching of @code{NUL}s is substantially slower than matching +other characters. Dynamic resizing of the input buffer is slow, as it +entails rescanning all the text matched so far by the current (generally +huge) token. Due to both buffering of input and read-ahead, you cannot +intermix calls to stdio.h routines, such as, @b{getchar()}, +with flex rules and expect it to work. Call input +instead. The total table entries listed by the @samp{-v} flag excludes +the number of table entries needed to determine what rule has been +matched. The number of entries is equal to the number of DFA states if +the scanner does not use @code{REJECT}, and somewhat greater than the +number of states if it does. @code{REJECT} cannot be used with the +@samp{-f} or @samp{-F} options. + +The flex internal algorithms need documentation. + + + + +Additional Reading + +You may wish to read more about the following programs: + + + + lex + + + yacc + + + sed + + + awk + + + + +The following books may contain material of interest: + +John Levine, Tony Mason, and Doug Brown, +@emph{Lex & Yacc}, +O'Reilly and Associates. Be sure to get the 2nd edition. + +M. E. Lesk and E. Schmidt, +@emph{LEX -- Lexical Analyzer Generator} + +Alfred Aho, Ravi Sethi and Jeffrey Ullman, @emph{Compilers: Principles, +Techniques and Tools}, Addison-Wesley (1986). Describes the +pattern-matching techniques used by flex (deterministic finite +automata). + +
+ +
+FAQ + +From time to time, the flex maintainer receives certain +questions. Rather than repeat answers to well-understood problems, we +publish them here. + + + + +
+ +
+When was flex born? + +Vern Paxson took over +the @cite{Software Tools} lex project from Jef Poskanzer in 1982. At that point it +was written in Ratfor. Around 1987 or so, Paxson translated it into C, and +a legend was born :-). + +
+ +
+How do I expand \ escape sequences in C-style quoted strings? + +A key point when scanning quoted strings is that you cannot (easily) write +a single rule that will precisely match the string if you allow things +like embedded escape sequences and newlines. If you try to match strings +with a single rule then you'll wind up having to rescan the string anyway +to find any escape sequences. + +Instead you can use exclusive start conditions and a set of rules, one for +matching non-escaped text, one for matching a single escape, one for +matching an embedded newline, and one for recognizing the end of the +string. Each of these rules is then faced with the question of where to +put its intermediary results. The best solution is for the rules to +append their local value of yytext to the end of a ``string literal'' +buffer. A rule like the escape-matcher will append to the buffer the +meaning of the escape sequence rather than the literal text in yytext. +In this way, yytext does not need to be modified at all. + +
+ +
+Why do flex scanners call fileno if it is not ANSI compatible? + +Flex scanners call fileno in order to get the file descriptor +corresponding to yyin. The file descriptor may be passed to +isatty or read, depending upon which @code{%options} you specified. +If your system does not have fileno support, to get rid of the +read call, do not specify @code{%option read}. To get rid of the isatty +call, you must specify one of @code{%option always-interactive} or +@code{%option never-interactive}. + +
+ +
+Does flex support recursive pattern definitions? + +e.g., + + + + + + + +No. You cannot have recursive definitions. The pattern-matching power of +regular expressions in general (and therefore flex scanners, too) is +limited. In particular, regular expressions cannot ``balance'' parentheses +to an arbitrary degree. For example, it's impossible to write a regular +expression that matches all strings containing the same number of '@{'s +as '@}'s. For more powerful pattern matching, you need a parser, such +as @cite{GNU bison}. + +
+ +
+How do I skip huge chunks of input (tens of megabytes) while using flex? + +Use fseek (or lseek) to position yyin, then call yyrestart. + +
+ +
+Flex is not matching my patterns in the same order that I defined them. + +flex picks the +rule that matches the most text (i.e., the longest possible input string). +This is because flex uses an entirely different matching technique +(``deterministic finite automata'') that actually does all of the matching +simultaneously, in parallel. (Seems impossible, but it's actually a fairly +simple technique once you understand the principles.) + +A side-effect of this parallel matching is that when the input matches more +than one rule, flex scanners pick the rule that matched the @emph{most} text. This +is explained further in the manual, in the section @xref{Matching}. + +If you want flex to choose a shorter match, then you can work around this +behavior by expanding your short +rule to match more text, then put back the extra: + + + + + + + +Another fix would be to make the second rule active only during the +@code{} start condition, and make that start condition exclusive +by declaring it with @code{%x} instead of @code{%s}. + +A final fix is to change the input language so that the ambiguity for +@samp{data_} is removed, by adding characters to it that don't match the +identifier rule, or by removing characters (such as @samp{_}) from the +identifier rule so it no longer matches @samp{data_}. (Of course, you might +also not have the option of changing the input language.) + +
+ +
+My actions are executing out of order or sometimes not at all. + +Most likely, you have (in error) placed the opening @samp{@{} of the action +block on a different line than the rule, e.g., + + + + + + + +flex requires that the opening @samp{@{} of an action associated with a rule +begin on the same line as does the rule. You need instead to write your rules +as follows: + + + + + + + +
+ +
+How can I have multiple input sources feed into the same scanner at the same time? + +If @dots{} + + + + +your scanner is free of backtracking (verified using flex's @samp{-b} flag), + + + +AND you run your scanner interactively (@samp{-I} option; default unless using special table +compression options), + + + +AND you feed it one character at a time by redefining @code{YY_INPUT} to do so, + + + + +then every time it matches a token, it will have exhausted its input +buffer (because the scanner is free of backtracking). This means you +can safely use select at the point and only call yylex for another +token if select indicates there's data available. + +That is, move the select out from the input function to a point where +it determines whether yylex gets called for the next token. + +With this approach, you will still have problems if your input can arrive +piecemeal; select could inform you that the beginning of a token is +available, you call yylex to get it, but it winds up blocking waiting +for the later characters in the token. + +Here's another way: Move your input multiplexing inside of @code{YY_INPUT}. That +is, whenever @code{YY_INPUT} is called, it select's to see where input is +available. If input is available for the scanner, it reads and returns the +next byte. If input is available from another source, it calls whatever +function is responsible for reading from that source. (If no input is +available, it blocks until some input is available.) I've used this technique in an +interpreter I wrote that both reads keyboard input using a flex scanner and +IPC traffic from sockets, and it works fine. + +
+ +
+Can I build nested parsers that work with the same input file? + +This is not going to work without some additional effort. The reason is +that flex block-buffers the input it reads from yyin. This means that the +``outermost'' yylex, when called, will automatically slurp up the first 8K +of input available on yyin, and subsequent calls to other yylex's won't +see that input. You might be tempted to work around this problem by +redefining @code{YY_INPUT} to only return a small amount of text, but it turns out +that that approach is quite difficult. Instead, the best solution is to +combine all of your scanners into one large scanner, using a different +exclusive start condition for each. + +
+ +
+How can I match text only at the end of a file? + +There is no way to write a rule which is ``match this text, but only if +it comes at the end of the file''. You can fake it, though, if you happen +to have a character lying around that you don't allow in your input. +Then you redefine @code{YY_INPUT} to call your own routine which, if it sees +an @samp{EOF}, returns the magic character first (and remembers to return a +real @code{EOF} next time it's called). Then you could write: + + + +(.|\n)*{EOF_CHAR} /* saw comment at EOF */ +]]> + + + +
+ +
+How can I make REJECT cascade across start condition boundaries? + +You can do this as follows. Suppose you have a start condition @samp{A}, and +after exhausting all of the possible matches in @samp{}, you want to try +matches in @samp{}. Then you could use the following: + + + +rule_that_is_long ...; REJECT; +rule ...; REJECT; /* shorter rule */ +etc. +... +.|\n { +/* Shortest and last rule in , so +* cascaded REJECT's will eventually +* wind up matching this rule. We want +* to now switch to the initial state +* and try matching from there instead. +*/ +yyless(0); /* put back matched text */ +BEGIN(INITIAL); +} +]]> + + + +
+ +
+Why can't I use fast or full tables with interactive mode? + +One of the assumptions +flex makes is that interactive applications are inherently slow (they're +waiting on a human after all). +It has to do with how the scanner detects that it must be finished scanning +a token. For interactive scanners, after scanning each character the current +state is looked up in a table (essentially) to see whether there's a chance +of another input character possibly extending the length of the match. If +not, the scanner halts. For non-interactive scanners, the end-of-token test +is much simpler, basically a compare with 0, so no memory bus cycles. Since +the test occurs in the innermost scanning loop, one would like to make it go +as fast as possible. + +Still, it seems reasonable to allow the user to choose to trade off a bit +of performance in this area to gain the corresponding flexibility. There +might be another reason, though, why fast scanners don't support the +interactive option. + +
+ +
+How much faster is -F or -f than -C? + +Much faster (factor of 2-3). + +
+ +
+If I have a simple grammar can't I just parse it with flex? + +Is your grammar recursive? That's almost always a sign that you're +better off using a parser/scanner rather than just trying to use a scanner +alone. + +
+ +
+Why doesn't yyrestart() set the start state back to INITIAL? + +There are two reasons. The first is that there might +be programs that rely on the start state not changing across file changes. +The second is that beginning with flex version 2.4, use of yyrestart is no longer required, +so fixing the problem there doesn't solve the more general problem. + +
+ +
+How can I match C-style comments? + +You might be tempted to try something like this: + + + + + + + +or, worse, this: + + + + + + + +The above rules will eat too much input, and blow up on things like: + + + + + + + +Here is one way which allows you to track line information: + + + +{ +"/*" BEGIN(IN_COMMENT); +} +{ +"*/" BEGIN(INITIAL); +[^*\n]+ // eat comment in chunks +"*" // eat the lone star +\n yylineno++; +} +]]> + + + +
+ +
+The '.' isn't working the way I expected. + +Here are some tips for using @samp{.}: + + + + + +A common mistake is to place the grouping parenthesis AFTER an operator, when +you really meant to place the parenthesis BEFORE the operator, e.g., you +probably want this @code{(foo|bar)+} and NOT this @code{(foo|bar+)}. + +The first pattern matches the words @samp{foo} or @samp{bar} any number of +times, e.g., it matches the text @samp{barfoofoobarfoo}. The +second pattern matches a single instance of @code{foo} or a single instance of +@code{bar} followed by one or more @samp{r}s, e.g., it matches the text @code{barrrr} . + + + +A @samp{.} inside @samp{[]}'s just means a literal@samp{.} (period), +and NOT ``any character except newline''. + + + +Remember that @samp{.} matches any character EXCEPT @samp{\n} (and @samp{EOF}). +If you really want to match ANY character, including newlines, then use @code{(.|\n)} +Beware that the regex @code{(.|\n)+} will match your entire input! + + + +Finally, if you want to match a literal @samp{.} (a period), then use @samp{[.]} or @samp{"."} + + + + +
+ +
+Can I get the flex manual in another format? + +The flex source distribution includes a +texinfo manual. You are free to convert that +texinfo into whatever format you desire. The +texinfo package includes tools for conversion to a +number of formats. + +
+ +
+Does there exist a "faster" NDFA->DFA algorithm? + +There's no way around the potential exponential running time - it +can take you exponential time just to enumerate all of the DFA states. +In practice, though, the running time is closer to linear, or sometimes +quadratic. + +
+ +
+How does flex compile the DFA so quickly? + +There are two big speed wins that flex uses: + + + + + +It analyzes the input rules to construct equivalence classes for those +characters that always make the same transitions. It then rewrites the NFA +using equivalence classes for transitions instead of characters. This cuts +down the NFA->DFA computation time dramatically, to the point where, for +uncompressed DFA tables, the DFA generation is often I/O bound in writing out +the tables. + + + +It maintains hash values for previously computed DFA states, so testing +whether a newly constructed DFA state is equivalent to a previously constructed +state can be done very quickly, by first comparing hash values. + + + + +
+ +
+How can I use more than 8192 rules? + +flex is compiled with an upper limit of 8192 rules per scanner. +If you need more than 8192 rules in your scanner, you'll have to recompile flex +with the following changes in flexdef.h: + + + + #define YY_TRAILING_MASK 0x20000000 +> #define YY_TRAILING_HEAD_MASK 0x40000000 +]]> + + + +This should work okay as long as your C compiler uses 32 bit integers. +But you might want to think about whether using such a huge number of rules +is the best way to solve your problem. + +The following may also be relevant: + +With luck, you should be able to increase the definitions in flexdef.h for: + + + + + + + +recompile everything, and it'll all work. Flex only has these 16-bit-like +values built into it because a long time ago it was developed on a machine +with 16-bit ints. I've given this advice to others in the past but haven't +heard back from them whether it worked okay or not... + +
+ +
+How do I abandon a file in the middle of a scan and switch to a new file? + +Just call @code{yyrestart(newfile)}. Be sure to reset the start state if you want a +``fresh start, since yyrestart does NOT reset the start state back to @code{INITIAL}. + +
+ +
+How do I execute code only during initialization (only before the first scan)? + +You can specify an initial action by defining the macro @code{YY_USER_INIT} (though +note that yyout may not be available at the time this macro is executed). Or you +can add to the beginning of your rules section: + + + + + + + +
+ +
+How do I execute code at termination? + +You can specify an action for the @code{<>} rule. + +
+ +
+Where else can I find help? + +You can find the flex homepage on the web at +@uref{http://lex.sourceforge.net/}. See that page for details about flex +mailing lists as well. + +
+ +
+Can I include comments in the "rules" section of the file? + +Yes, just about anywhere you want to. See the manual for the specific syntax. + +
+ +
+I get an error about undefined yywrap(). + +You must supply a yywrap function of your own, or link to libfl.a +(which provides one), or use + + + + + + + +in your source to say you don't want a yywrap function. + +
+ +
+How can I change the matching pattern at run time? + +You can't, it's compiled into a static table when flex builds the scanner. + +
+ +
+How can I expand macros in the input? + +The best way to approach this problem is at a higher level, e.g., in the parser. + +However, you can do this using multiple input buffers. + + + +> { +if ( expansion_buffer ) +{ +// We were doing an expansion, return to where +// we were. +yy_switch_to_buffer(main_buffer); +yy_delete_buffer(expansion_buffer); +expansion_buffer = 0; +} +else +yyterminate(); +} +]]> + + + +You probably will want a stack of expansion buffers to allow nested macros. +From the above though hopefully the idea is clear. + +
+ +
+How can I build a two-pass scanner? + +One way to do it is to filter the first pass to a temporary file, +then process the temporary file on the second pass. You will probably see a +performance hit, do to all the disk I/O. + +When you need to look ahead far forward like this, it almost always means +that the right solution is to build a parse tree of the entire input, then +walk it after the parse in order to generate the output. In a sense, this +is a two-pass approach, once through the text and once through the parse +tree, but the performance hit for the latter is usually an order of magnitude +smaller, since everything is already classified, in binary format, and +residing in memory. + +
+ +
+How do I match any string not matched in the preceding rules? + +One way to assign precedence, is to place the more specific rules first. If +two rules would match the same input (same sequence of characters) then the +first rule listed in the flex input wins. e.g., + + + + + + + +Note that the rule @code{[a-zA-Z_]+} must come *after* the others. It will match the +same amount of text as the more specific rules, and in that case the +flex scanner will pick the first rule listed in your scanner as the +one to match. + +
+ +
+I am trying to port code from AT&T lex that uses yysptr and yysbuf. + +Those are internal variables pointing into the AT&T scanner's input buffer. I +imagine they're being manipulated in user versions of the input and unput +functions. If so, what you need to do is analyze those functions to figure out +what they're doing, and then replace input with an appropriate definition of +@code{YY_INPUT}. You shouldn't need to (and must not) replace +flex's unput function. + +
+ +
+Is there a way to make flex treat NULL like a regular character? + +Yes, @samp{\0} and @samp{\x00} should both do the trick. Perhaps you have an ancient +version of flex. The latest release is version @value{VERSION}. + +
+ +
+Whenever flex can not match the input it says "flex scanner jammed". + +You need to add a rule that matches the otherwise-unmatched text. +e.g., + + + + + + + +See @code{%option default} for more information. + +
+ +
+Why doesn't flex have non-greedy operators like perl does? + +A DFA can do a non-greedy match by stopping +the first time it enters an accepting state, instead of consuming input until +it determines that no further matching is possible (a ``jam'' state). This +is actually easier to implement than longest leftmost match (which flex does). + +But it's also much less useful than longest leftmost match. In general, +when you find yourself wishing for non-greedy matching, that's usually a +sign that you're trying to make the scanner do some parsing. That's +generally the wrong approach, since it lacks the power to do a decent job. +Better is to either introduce a separate parser, or to split the scanner +into multiple scanners using (exclusive) start conditions. + +You might have +a separate start state once you've seen the @samp{BEGIN}. In that state, you +might then have a regex that will match @samp{END} (to kick you out of the +state), and perhaps @samp{(.|\n)} to get a single character within the chunk ... + +This approach also has much better error-reporting properties. + +
+ +
+Memory leak - 16386 bytes allocated by malloc. +@anchor{faq-memory-leak} + +UPDATED 2002-07-10: As of flex version 2.5.9, this leak means that you did not +call yylex_destroy. If you are using an earlier version of flex, then read +on. + +The leak is about 16426 bytes. That is, (8192 * 2 + 2) for the read-buffer, and +about 40 for @code{struct yy_buffer_state} (depending upon alignment). The leak is in +the non-reentrant C scanner only (NOT in the reentrant scanner, NOT in the C++ +scanner). Since flex doesn't know when you are done, the buffer is never freed. + +However, the leak won't multiply since the buffer is reused no matter how many +times you call yylex. + +If you want to reclaim the memory when you are completely done scanning, then +you might try this: + + + + + + + +Note: @code{yy_init} is an "internal variable", and hasn't been tested in this +situation. It is possible that some other globals may need resetting as well. + +
+ +
+How do I track the byte offset for lseek()? + + + + We thought that it would be possible to have this number through the +> evaluation of the following expression: +> +> seek_position = (no_buffers)*YY_READ_BUF_SIZE + yy_c_buf_p - YY_CURRENT_BUFFER->yy_ch_buf +]]> + + + +While this is the right idea, it has two problems. The first is that +it's possible that flex will request less than @code{YY_READ_BUF_SIZE} during +an invocation of @code{YY_INPUT} (or that your input source will return less +even though @code{YY_READ_BUF_SIZE} bytes were requested). The second problem +is that when refilling its internal buffer, flex keeps some characters +from the previous buffer (because usually it's in the middle of a match, +and needs those characters to construct yytext for the match once it's +done). Because of this, @code{yy_c_buf_p - YY_CURRENT_BUFFER->yy_ch_buf} won't +be exactly the number of characters already read from the current buffer. + +An alternative solution is to count the number of characters you've matched +since starting to scan. This can be done by using @code{YY_USER_ACTION}. For +example, + + + + + + + +(You need to be careful to update your bookkeeping if you use yymore), +yyless, unput, or input.) + +
+ +
+How do I use my own I/O classes in a C++ scanner? + +When the flex C++ scanning class rewrite finally happens, then this sort of thing should become much easier. + + + + + + + +You can do this by passing the various functions (such as LexerInput +and LexerOutput) NULL @code{iostream*}'s, and then +dealing with your own I/O classes surreptitiously (i.e., stashing them in +special member variables). This works because the only assumption about +the lexer regarding what's done with the iostream's is that they're +ultimately passed to LexerInput and LexerOutput, which then do whatever +is necessary with them. + + +
+ +
+How do I skip as many chars as possible? + +How do I skip as many chars as possible -- without interfering with the other +patterns? + +In the example below, we want to skip over characters until we see the phrase +"endskip". The following will @emph{NOT} work correctly (do you see why not?) + + + +startskip BEGIN(SKIP); +... +"endskip" BEGIN(INITIAL); +.* ; +]]> + + + +The problem is that the pattern .* will eat up the word "endskip." +The simplest (but slow) fix is: + + + +"endskip" BEGIN(INITIAL); +. ; +]]> + + + +The fix involves making the second rule match more, without +making it match "endskip" plus something else. So for example: + + + +"endskip" BEGIN(INITIAL); +[^e]+ ; +. ;/* so you eat up e's, too */ +]]> + + + + +
+ +
+deleteme00 + + + + + + + +
+ +
+Are certain equivalent patterns faster than others? + + + +Subject: Re: Flex 2.5.2 performance questions +In-reply-to: Your message of Wed, 18 Sep 96 11:12:17 EDT. +Date: Wed, 18 Sep 96 10:51:02 PDT +From: Vern Paxson + +[Note, the most recent flex release is 2.5.4, which you can get from +ftp.ee.lbl.gov. It has bug fixes over 2.5.2 and 2.5.3.] + +> 1. Using the pattern +> ([Ff](oot)?)?[Nn](ote)?(\.)? +> instead of +> (((F|f)oot(N|n)ote)|((N|n)ote)|((N|n)\.)|((F|f)(N|n)(\.))) +> (in a very complicated flex program) caused the program to slow from +> 300K+/min to 100K/min (no other changes were done). + +These two are not equivalent. For example, the first can match "footnote." +but the second can only match "footnote". This is almost certainly the +cause in the discrepancy - the slower scanner run is matching more tokens, +and/or having to do more backing up. + +> 2. Which of these two are better: [Ff]oot or (F|f)oot ? + +From a performance point of view, they're equivalent (modulo presumably +minor effects such as memory cache hit rates; and the presence of trailing +context, see below). From a space point of view, the first is slightly +preferable. + +> 3. I have a pattern that look like this: +> pats {p1}|{p2}|{p3}|...|{p50} (50 patterns ORd) +> +> running yet another complicated program that includes the following rule: +> {and}/{no4}{bb}{pats} +> +> gets me to "too complicated - over 32,000 states"... + +I can't tell from this example whether the trailing context is variable-length +or fixed-length (it could be the latter if {and} is fixed-length). If it's +variable length, which flex -p will tell you, then this reflects a basic +performance problem, and if you can eliminate it by restructuring your +scanner, you will see significant improvement. + +> so I divided {pats} to {pats1}, {pats2},..., {pats5} each consists of about +> 10 patterns and changed the rule to be 5 rules. +> This did compile, but what is the rule of thumb here ? + +The rule is to avoid trailing context other than fixed-length, in which for +a/b, either the 'a' pattern or the 'b' pattern have a fixed length. Use +of the '|' operator automatically makes the pattern variable length, so in +this case '[Ff]oot' is preferred to '(F|f)oot'. + +> 4. I changed a rule that looked like this: +> {and}{bb}/{ROMAN}[^A-Za-z] { BEGIN... +> +> to the next 2 rules: +> {and}{bb}/{ROMAN}[A-Za-z] { ECHO;} +> {and}{bb}/{ROMAN} { BEGIN... +> +> Again, I understand the using [^...] will cause a great performance loss + +Actually, it doesn't cause any sort of performance loss. It's a surprising +fact about regular expressions that they always match in linear time +regardless of how complex they are. + +> but are there any specific rules about it ? + +See the "Performance Considerations" section of the man page, and also +the example in MISC/fastwc/. + + Vern +]]> + + + + +
+ +
+Is backing up a big deal? + + + +Subject: Re: Flex 2.5.2 performance questions +In-reply-to: Your message of Thu, 19 Sep 96 10:16:04 EDT. +Date: Thu, 19 Sep 96 09:58:00 PDT +From: Vern Paxson + +> a lot about the backing up problem. +> I believe that there lies my biggest problem, and I'll try to improve +> it. + +Since you have variable trailing context, this is a bigger performance +problem. Fixing it is usually easier than fixing backing up, which in a +complicated scanner (yours seems to fit the bill) can be extremely +difficult to do correctly. + +You also don't mention what flags you are using for your scanner. +-f makes a large speed difference, and -Cfe buys you nearly as much +speed but the resulting scanner is considerably smaller. + +> I have an | operator in {and} and in {pats} so both of them are variable +> length. + +-p should have reported this. + +> Is changing one of them to fixed-length is enough ? + +Yes. + +> Is it possible to change the 32,000 states limit ? + +Yes. I've appended instructions on how. Before you make this change, +though, you should think about whether there are ways to fundamentally +simplify your scanner - those are certainly preferable! + + Vern + +To increase the 32K limit (on a machine with 32 bit integers), you increase +the magnitude of the following in flexdef.h: + +#define JAMSTATE -32766 /* marks a reference to the state that always jams */ +#define MAXIMUM_MNS 31999 +#define BAD_SUBSCRIPT -32767 +#define MAX_SHORT 32700 + +Adding a 0 or two after each should do the trick. +]]> + + + + +
+ +
+Can I fake multi-byte character support? + + + + +> I assume as long as my *.l file defines the +> range of expected character code values (in octal format), flex will +> scan the file and read multi-byte characters correctly. But I have no +> confidence in this assumption. + +Your lack of confidence is justified - this won't work. + +Flex has in it a widespread assumption that the input is processed +one byte at a time. Fixing this is on the to-do list, but is involved, +so it won't happen any time soon. In the interim, the best I can suggest +(unless you want to try fixing it yourself) is to write your rules in +terms of pairs of bytes, using definitions in the first section: + + X \xfe\xc2 + ... + %% + foo{X}bar found_foo_fe_c2_bar(); + +etc. Definitely a pain - sorry about that. + +By the way, the email address you used for me is ancient, indicating you +have a very old version of flex. You can get the most recent, 2.5.4, from +ftp.ee.lbl.gov. + + Vern +]]> + + + + +
+ +
+deleteme01 + + + + +Unfortunately flex at the moment has a widespread assumption within it +that characters are processed 8 bits at a time. I don't see any easy +fix for this (other than writing your rules in terms of double characters - +a pain). I also don't know of a wider lex, though you might try surfing +the Plan 9 stuff because I know it's a Unicode system, and also the PCCT +toolkit (try searching say Alta Vista for "Purdue Compiler Construction +Toolkit"). + +Fixing flex to handle wider characters is on the long-term to-do list. +But since flex is a strictly spare-time project these days, this probably +won't happen for quite a while, unless someone else does it first. + + Vern +]]> + + + + +
+ +
+Can you discuss some flex internals? + + + +Subject: Re: translation of flex +In-reply-to: Your message of Sun, 10 Nov 1996 09:16:36 PST. +Date: Mon, 11 Nov 1996 10:33:50 PST +From: Vern Paxson + +> I'm working for the Swedish team translating GNU program, and I'm currently +> working with flex. I have a few questions about some of the messages which +> I hope you can answer. + +All of the things you're wondering about, by the way, concerning flex +internals - probably the only person who understands what they mean in +English is me! So I wouldn't worry too much about getting them right. +That said ... + +> #: main.c:545 +> msgid " %d protos created\n" +> +> Does proto mean prototype? + +Yes - prototypes of state compression tables. + +> #: main.c:539 +> msgid " %d/%d (peak %d) template nxt-chk entries created\n" +> +> Here I'm mainly puzzled by 'nxt-chk'. I guess it means 'next-check'. (?) +> However, 'template next-check entries' doesn't make much sense to me. To be +> able to find a good translation I need to know a little bit more about it. + +There is a scheme in the Aho/Sethi/Ullman compiler book for compressing +scanner tables. It involves creating two pairs of tables. The first has +"base" and "default" entries, the second has "next" and "check" entries. +The "base" entry is indexed by the current state and yields an index into +the next/check table. The "default" entry gives what to do if the state +transition isn't found in next/check. The "next" entry gives the next +state to enter, but only if the "check" entry verifies that this entry is +correct for the current state. Flex creates templates of series of +next/check entries and then encodes differences from these templates as a +way to compress the tables. + +> #: main.c:533 +> msgid " %d/%d base-def entries created\n" +> +> The same problem here for 'base-def'. + +See above. + + Vern +]]> + + + + +
+ +
+unput() messes up yy_at_bol + + + +Subject: Re: FLEX ? +In-reply-to: Your message of Wed, 13 Nov 1996 17:28:38 PST. +Date: Wed, 13 Nov 1996 19:51:54 PST +From: Vern Paxson + +> "unput()" them to input flow, question occurs. If I do this after I scan +> a carriage, the variable "YY_CURRENT_BUFFER->yy_at_bol" is changed. That +> means the carriage flag has gone. + +You can control this by calling yy_set_bol(). It's described in the manual. + +> And if in pre-reading it goes to the end of file, is anything done +> to control the end of curren buffer and end of file? + +No, there's no way to put back an end-of-file. + +> By the way I am using flex 2.5.2 and using the "-l". + +The latest release is 2.5.4, by the way. It fixes some bugs in 2.5.2 and +2.5.3. You can get it from ftp.ee.lbl.gov. + + Vern +]]> + + + + +
+ +
+The | operator is not doing what I want + + + + +> I am not able to use the start condition scope and to use the | (OR) with +> rules having start conditions. + +The problem is that if you use '|' as a regular expression operator, for +example "a|b" meaning "match either 'a' or 'b'", then it must *not* have +any blanks around it. If you instead want the special '|' *action* (which +from your scanner appears to be the case), which is a way of giving two +different rules the same action: + + foo | + bar matched_foo_or_bar(); + +then '|' *must* be separated from the first rule by whitespace and *must* +be followed by a new line. You *cannot* write it as: + + foo | bar matched_foo_or_bar(); + +even though you might think you could because yacc supports this syntax. +The reason for this unfortunately incompatibility is historical, but it's +unlikely to be changed. + +Your problems with start condition scope are simply due to syntax errors +from your use of '|' later confusing flex. + +Let me know if you still have problems. + + Vern +]]> + + + + +
+ +
+Why can't flex understand this variable trailing context pattern? + + + +Subject: Re: flex-2.5.3 bug report +In-reply-to: Your message of Sat, 23 Nov 1996 16:50:09 PST. +Date: Sat, 23 Nov 1996 17:07:32 PST +From: Vern Paxson + +> Enclosed is a lex file that "real" lex will process, but I cannot get +> flex to process it. Could you try it and maybe point me in the right direction? + +Your problem is that some of the definitions in the scanner use the '/' +trailing context operator, and have it enclosed in ()'s. Flex does not +allow this operator to be enclosed in ()'s because doing so allows undefined +regular expressions such as "(a/b)+". So the solution is to remove the +parentheses. Note that you must also be building the scanner with the -l +option for AT&T lex compatibility. Without this option, flex automatically +encloses the definitions in parentheses. + + Vern +]]> + + + + +
+ +
+The ^ operator isn't working + + + +Subject: Re: Flex Bug ? +In-reply-to: Your message of Tue, 26 Nov 1996 14:35:01 PST. +Date: Tue, 26 Nov 1996 11:15:05 PST +From: Vern Paxson + +> In my lexer code, i have the line : +> ^\*.* { } +> +> Thus all lines starting with an astrix (*) are comment lines. +> This does not work ! + +I can't get this problem to reproduce - it works fine for me. Note +though that if what you have is slightly different: + + COMMENT ^\*.* + %% + {COMMENT} { } + +then it won't work, because flex pushes back macro definitions enclosed +in ()'s, so the rule becomes + + (^\*.*) { } + +and now that the '^' operator is not at the immediate beginning of the +line, it's interpreted as just a regular character. You can avoid this +behavior by using the "-l" lex-compatibility flag, or "%option lex-compat". + + Vern +]]> + + + + +
+ +
+Trailing context is getting confused with trailing optional patterns + + + +Subject: Re: Flex 2.5.4 BOF ??? +In-reply-to: Your message of Tue, 26 Nov 1996 16:10:41 PST. +Date: Wed, 27 Nov 1996 10:56:25 PST +From: Vern Paxson + +> Organization(s)?/[a-z] +> +> This matched "Organizations" (looking in debug mode, the trailing s +> was matched with trailing context instead of the optional (s) in the +> end of the word. + +That should only happen with lex. Flex can properly match this pattern. +(That might be what you're saying, I'm just not sure.) + +> Is there a way to avoid this dangerous trailing context problem ? + +Unfortunately, there's no easy way. On the other hand, I don't see why +it should be a problem. Lex's matching is clearly wrong, and I'd hope +that usually the intent remains the same as expressed with the pattern, +so flex's matching will be correct. + + Vern +]]> + + + + +
+ +
+Is flex GNU or not? + + + +Subject: Re: Flex documentation bug +In-reply-to: Your message of Mon, 02 Dec 1996 00:07:08 PST. +Date: Sun, 01 Dec 1996 22:29:39 PST +From: Vern Paxson + +> I'm not sure how or where to submit bug reports (documentation or +> otherwise) for the GNU project stuff ... + +Well, strictly speaking flex isn't part of the GNU project. They just +distribute it because no one's written a decent GPL'd lex replacement. +So you should send bugs directly to me. Those sent to the GNU folks +sometimes find there way to me, but some may drop between the cracks. + +> In GNU Info, under the section 'Start Conditions', and also in the man +> page (mine's dated April '95) is a nice little snippet showing how to +> parse C quoted strings into a buffer, defined to be MAX_STR_CONST in +> size. Unfortunately, no overflow checking is ever done ... + +This is already mentioned in the manual: + +Finally, here's an example of how to match C-style quoted +strings using exclusive start conditions, including expanded +escape sequences (but not including checking for a string +that's too long): + +The reason for not doing the overflow checking is that it will needlessly +clutter up an example whose main purpose is just to demonstrate how to +use flex. + +The latest release is 2.5.4, by the way, available from ftp.ee.lbl.gov. + + Vern +]]> + + + + +
+ +
+ERASEME53 + + + + +> [:alpha:] ([:alnum:] | \\_)* + +If your rule really has embedded blanks as shown above, then it won't +work, as the first blank delimits the rule from the action. (It wouldn't +even compile ...) You need instead: + +[:alpha:]([:alnum:]|\\_)* + +and that should work fine - there's no restriction on what can go inside +of ()'s except for the trailing context operator, '/'. + + Vern +]]> + + + + +
+ +
+I need to scan if-then-else blocks and while loops + + + +Subject: Re: FLEX help +In-reply-to: Your message of Fri, 30 May 1997 13:33:27 PDT. +Date: Fri, 30 May 1997 10:46:35 PDT +From: Vern Paxson + +> We'd like to add "if-then-else", "while", and "for" statements to our +> language ... +> We've investigated many possible solutions. The one solution that seems +> the most reasonable involves knowing the position of a TOKEN in yyin. + +I strongly advise you to instead build a parse tree (abstract syntax tree) +and loop over that instead. You'll find this has major benefits in keeping +your interpreter simple and extensible. + +That said, the functionality you mention for get_position and set_position +have been on the to-do list for a while. As flex is a purely spare-time +project for me, no guarantees when this will be added (in particular, it +for sure won't be for many months to come). + + Vern +]]> + + + + +
+ +
+ERASEME55 + + + +Subject: Re: Flex C++ classes and Bison +In-reply-to: Your message of 09 Aug 1997 17:11:41 PDT. +Date: Fri, 15 Aug 1997 10:48:19 PDT +From: Vern Paxson + +> #define YY_DECL int yylex (YYSTYPE *lvalp, struct parser_control +> *parm) +> +> I have been trying to get this to work as a C++ scanner, but it does +> not appear to be possible (warning that it matches no declarations in +> yyFlexLexer, or something like that). +> +> Is this supposed to be possible, or is it being worked on (I DID +> notice the comment that scanner classes are still experimental, so I'm +> not too hopeful)? + +What you need to do is derive a subclass from yyFlexLexer that provides +the above yylex() method, squirrels away lvalp and parm into member +variables, and then invokes yyFlexLexer::yylex() to do the regular scanning. + + Vern +]]> + + + + +
+ +
+ERASEME56 + + + + +> In that example you show how to count comment lines when using +> C style /* ... */ comments. My question is, shouldn't you take into +> account a scenario where end of a comment marker occurs inside +> character or string literals? + +The scanner certainly needs to also scan character and string literals. +However it does that (there's an example in the man page for strings), the +lexer will recognize the beginning of the literal before it runs across the +embedded "/*". Consequently, it will finish scanning the literal before it +even considers the possibility of matching "/*". + +Example: + + '([^']*|{ESCAPE_SEQUENCE})' + +will match all the text between the ''s (inclusive). So the lexer +considers this as a token beginning at the first ', and doesn't even +attempt to match other tokens inside it. + +I thinnk this subtlety is not worth putting in the manual, as I suspect +it would confuse more people than it would enlighten. + + Vern +]]> + + + + +
+ +
+ERASEME57 + + + +Subject: Re: flex limitations +In-reply-to: Your message of Sat, 06 Sep 1997 11:27:21 PDT. +Date: Mon, 08 Sep 1997 11:38:08 PDT +From: Vern Paxson + +> %% +> [a-zA-Z]+ /* skip a line */ +> { printf("got %s\n", yytext); } +> %% + +What version of flex are you using? If I feed this to 2.5.4, it complains: + + "bug.l", line 5: EOF encountered inside an action + "bug.l", line 5: unrecognized rule + "bug.l", line 5: fatal parse error + +Not the world's greatest error message, but it manages to flag the problem. + +(With the introduction of start condition scopes, flex can't accommodate +an action on a separate line, since it's ambiguous with an indented rule.) + +You can get 2.5.4 from ftp.ee.lbl.gov. + + Vern +]]> + + + + +
+ +
+Is there a repository for flex scanners? + +Not that we know of. You might try asking on comp.compilers. + + +
+ +
+How can I conditionally compile or preprocess my flex input file? + + +Flex doesn't have a preprocessor like C does. You might try using m4, or the C +preprocessor plus a sed script to clean up the result. + + + +
+ +
+Where can I find grammars for lex and yacc? + +In the sources for flex and bison. + + +
+ +
+I get an end-of-buffer message for each character scanned. + +This will happen if your LexerInput() function returns only one character +at a time, which can happen either if you're scanner is "interactive", or +if the streams library on your platform always returns 1 for yyin->gcount(). + +Solution: override LexerInput() with a version that returns whole buffers. + + +
+ +
+unnamed-faq-62 + + + + +> I took a quick look into the flex-sources and altered some #defines in +> flexdefs.h: +> +> #define INITIAL_MNS 64000 +> #define MNS_INCREMENT 1024000 +> #define MAXIMUM_MNS 64000 + +The things to fix are to add a couple of zeroes to: + +#define JAMSTATE -32766 /* marks a reference to the state that always jams */ +#define MAXIMUM_MNS 31999 +#define BAD_SUBSCRIPT -32767 +#define MAX_SHORT 32700 + +and, if you get complaints about too many rules, make the following change too: + + #define YY_TRAILING_MASK 0x200000 + #define YY_TRAILING_HEAD_MASK 0x400000 + +- Vern +]]> + + + + +
+ +
+unnamed-faq-63 + + + + +> stdin_handle = YY_CURRENT_BUFFER; +> ifstream fin( "aFile" ); +> yy_switch_to_buffer( yy_create_buffer( fin, YY_BUF_SIZE ) ); +> +> What I'm wanting to do, is pass the contents of a file thru one set +> of rules and then pass stdin thru another set... It works great if, I +> don't use the C++ classes. But since everything else that I'm doing is +> in C++, I thought I'd be consistent. +> +> The problem is that 'yy_create_buffer' is expecting an istream* as it's +> first argument (as stated in the man page). However, fin is a ifstream +> object. Any ideas on what I might be doing wrong? Any help would be +> appreciated. Thanks!! + +You need to pass &fin, to turn it into an ifstream* instead of an ifstream. +Then its type will be compatible with the expected istream*, because ifstream +is derived from istream. + + Vern +]]> + + + + +
+ +
+unnamed-faq-64 + + + +Subject: Re: Question related to Flex man page? +In-reply-to: Your message of Tue, 16 Dec 1997 15:17:34 PST. +Date: Tue, 16 Dec 1997 14:17:09 PST +From: Vern Paxson + +> Can you explain to me what is ment by a long-jump in relation to flex? + +Using the longjmp() function while inside yylex() or a routine called by it. + +> what is the flex activation frame. + +Just yylex()'s stack frame. + +> As far as I can see yyrestart will bring me back to the sart of the input +> file and using flex++ isnot really an option! + +No, yyrestart() doesn't imply a rewind, even though its name might sound +like it does. It tells the scanner to flush its internal buffers and +start reading from the given file at its present location. + + Vern +]]> + + + + +
+ +
+unnamed-faq-65 + + + + +> /usr/lib/yaccpar: In function `int yyparse()': +> /usr/lib/yaccpar:184: warning: implicit declaration of function `int yylex(...)' +> +> ld: Undefined symbol +> _yylex +> _yyparse +> _yyin + +This is a known problem with Solaris C++ (and/or Solaris yacc). I believe +the fix is to explicitly insert some 'extern "C"' statements for the +corresponding routines/symbols. + + Vern +]]> + + + + +
+ +
+unnamed-faq-66 + + + + +> This is my definition for float and integer types: +> . . . +> NZD [1-9] +> ... +> I've tested my program on other lex version (on UNIX Sun Solaris an HP +> UNIX) and it work well, so I think that my definitions are correct. +> There are any differences between Lex and Flex? + +There are indeed differences, as discussed in the man page. The one +you are probably running into is that when flex expands a name definition, +it puts parentheses around the expansion, while lex does not. There's +an example in the man page of how this can lead to different matching. +Flex's behavior complies with the POSIX standard (or at least with the +last POSIX draft I saw). + + Vern +]]> + + + + +
+ +
+unnamed-faq-67 + + + + +> Thank you very much for your help. I compile and link well with C++ while +> declaring 'yylex ...' extern, But a little problem remains. I get a +> segmentation default when executing ( I linked with lfl library) while it +> works well when using LEX instead of flex. Do you have some ideas about the +> reason for this ? + +The one possible reason for this that comes to mind is if you've defined +yytext as "extern char yytext[]" (which is what lex uses) instead of +"extern char *yytext" (which is what flex uses). If it's not that, then +I'm afraid I don't know what the problem might be. + + Vern +]]> + + + + +
+ +
+unnamed-faq-68 + + + +Subject: Re: flex 2.5: c++ scanners & start conditions +In-reply-to: Your message of Tue, 06 Jan 1998 10:34:21 PST. +Date: Tue, 06 Jan 1998 19:19:30 PST +From: Vern Paxson + +> The problem is that when I do this (using %option c++) start +> conditions seem to not apply. + +The BEGIN macro modifies the yy_start variable. For C scanners, this +is a static with scope visible through the whole file. For C++ scanners, +it's a member variable, so it only has visible scope within a member +function. Your lexbegin() routine is not a member function when you +build a C++ scanner, so it's not modifying the correct yy_start. The +diagnostic that indicates this is that you found you needed to add +a declaration of yy_start in order to get your scanner to compile when +using C++; instead, the correct fix is to make lexbegin() a member +function (by deriving from yyFlexLexer). + + Vern +]]> + + + + +
+ +
+unnamed-faq-69 + + + +Subject: Re: current position in flex buffer +In-reply-to: Your message of Mon, 12 Jan 1998 18:58:23 PST. +Date: Mon, 12 Jan 1998 12:03:15 PST +From: Vern Paxson + +> The problem is how to determine the current position in flex active +> buffer when a rule is matched.... + +You will need to keep track of this explicitly, such as by redefining +YY_USER_ACTION to count the number of characters matched. + +The latest flex release, by the way, is 2.5.4, available from ftp.ee.lbl.gov. + + Vern +]]> + + + + +
+ +
+unnamed-faq-70 + + + + +> That requirement involves knowing +> the character position at which a particular token was matched +> in the lexer. + +The way you have to do this is by explicitly keeping track of where +you are in the file, by counting the number of characters scanned +for each token (available in yyleng). It may prove convenient to +do this by redefining YY_USER_ACTION, as described in the manual. + + Vern +]]> + + + + +
+ +
+unnamed-faq-71 + + + +Subject: Re: flex: how to control start condition from parser? +In-reply-to: Your message of Mon, 26 Jan 1998 05:50:16 PST. +Date: Tue, 27 Jan 1998 22:45:37 PST +From: Vern Paxson + +> It seems useful for the parser to be able to tell the lexer about such +> context dependencies, because then they don't have to be limited to +> local or sequential context. + +One way to do this is to have the parser call a stub routine that's +included in the scanner's .l file, and consequently that has access ot +BEGIN. The only ugliness is that the parser can't pass in the state +it wants, because those aren't visible - but if you don't have many +such states, then using a different set of names doesn't seem like +to much of a burden. + +While generating a .h file like you suggests is certainly cleaner, +flex development has come to a virtual stand-still :-(, so a workaround +like the above is much more pragmatic than waiting for a new feature. + + Vern +]]> + + + + +
+ +
+unnamed-faq-72 + + + +Subject: Re: freebsd flex bug? +In-reply-to: Your message of Fri, 30 Jan 1998 12:00:43 PST. +Date: Fri, 30 Jan 1998 12:42:32 PST +From: Vern Paxson + +> lex.yy.c:1996: parse error before `=' + +This is the key, identifying this error. (It may help to pinpoint +it by using flex -L, so it doesn't generate #line directives in its +output.) I will bet you heavy money that you have a start condition +name that is also a variable name, or something like that; flex spits +out #define's for each start condition name, mapping them to a number, +so you can wind up with: + + %x foo + %% + ... + %% + void bar() + { + int foo = 3; + } + +and the penultimate will turn into "int 1 = 3" after C preprocessing, +since flex will put "#define foo 1" in the generated scanner. + + Vern +]]> + + + + +
+ +
+unnamed-faq-73 + + + +Subject: Re: Lost flex .l file +In-reply-to: Your message of Mon, 02 Feb 1998 14:10:01 PST. +Date: Mon, 02 Feb 1998 11:15:12 PST +From: Vern Paxson + +> I am curious as to +> whether there is a simple way to backtrack from the generated source to +> reproduce the lost list of tokens we are searching on. + +In theory, it's straight-forward to go from the DFA representation +back to a regular-expression representation - the two are isomorphic. +In practice, a huge headache, because you have to unpack all the tables +back into a single DFA representation, and then write a program to munch +on that and translate it into an RE. + +Sorry for the less-than-happy news ... + + Vern +]]> + + + + +
+ +
+unnamed-faq-74 + + + + +> What I have found, is that the smaller the data chunk, the faster the +> program executes. This is the opposite of what I expected. Should this be +> happening this way? + +This is exactly what will happen if your input file has embedded NULs. +From the man page: + +A final note: flex is slow when matching NUL's, particularly +when a token contains multiple NUL's. It's best to write +rules which match short amounts of text if it's anticipated +that the text will often include NUL's. + +So that's the first thing to look for. + + Vern +]]> + + + + +
+ +
+unnamed-faq-75 + + + + +So there are several problems. + +First, to go fast, you want to match as much text as possible, which +your scanners don't in the case that what they're scanning is *not* +a tag. So you want a rule like: + + [^<]+ + +Second, C++ scanners are particularly slow if they're interactive, +which they are by default. Using -B speeds it up by a factor of 3-4 +on my workstation. + +Third, C++ scanners that use the istream interface are slow, because +of how poorly implemented istream's are. I built two versions of +the following scanner: + + %% + .*\n + .* + %% + +and the C version inhales a 2.5MB file on my workstation in 0.8 seconds. +The C++ istream version, using -B, takes 3.8 seconds. + + Vern +]]> + + + + +
+ +
+unnamed-faq-76 + + + +Subject: Re: FLEX 2.5 & THE YEAR 2000 +In-reply-to: Your message of Wed, 03 Jun 1998 11:26:22 PDT. +Date: Wed, 03 Jun 1998 10:22:26 PDT +From: Vern Paxson + +> I am researching the Y2K problem with General Electric R&D +> and need to know if there are any known issues concerning +> the above mentioned software and Y2K regardless of version. + +There shouldn't be, all it ever does with the date is ask the system +for it and then print it out. + + Vern +]]> + + + + +
+ +
+unnamed-faq-77 + + + +Subject: Re: flex problem +In-reply-to: Your message of Wed, 15 Jul 1998 21:30:13 PDT. +Date: Tue, 21 Jul 1998 14:23:34 PDT +From: Vern Paxson + +> To overcome this, I gets() the stdin into a string and lex the string. The +> string is lexed OK except that the end of string isn't lexed properly +> (yy_scan_string()), that is the lexer dosn't recognise the end of string. + +Flex doesn't contain mechanisms for recognizing buffer endpoints. But if +you use fgets instead (which you should anyway, to protect against buffer +overflows), then the final \n will be preserved in the string, and you can +scan that in order to find the end of the string. + + Vern +]]> + + + + +
+ +
+unnamed-faq-78 + + + + +> %{ +> int mylineno = 0; +> %} +> ws [ \t]+ +> alpha [A-Za-z] +> dig [0-9] +> %% +> +> Now you'd expect mylineno to be a member of each instance of class +> yyFlexLexer, but is this the case? A look at the lex.yy.cc file seems to +> indicate otherwise; unless I am missing something the declaration of +> mylineno seems to be outside any class scope. +> +> How will this work if I want to run a multi-threaded application with each +> thread creating a FlexLexer instance? + +Derive your own subclass and make mylineno a member variable of it. + + Vern +]]> + + + + +
+ +
+unnamed-faq-79 + + + +Subject: Re: More than 32K states change hangs +In-reply-to: Your message of Tue, 04 Aug 1998 16:55:39 PDT. +Date: Tue, 04 Aug 1998 22:28:45 PDT +From: Vern Paxson + +> Vern Paxson, +> +> I followed your advice, posted on Usenet bu you, and emailed to me +> personally by you, on how to overcome the 32K states limit. I'm running +> on Linux machines. +> I took the full source of version 2.5.4 and did the following changes in +> flexdef.h: +> #define JAMSTATE -327660 +> #define MAXIMUM_MNS 319990 +> #define BAD_SUBSCRIPT -327670 +> #define MAX_SHORT 327000 +> +> and compiled. +> All looked fine, including check and bigcheck, so I installed. + +Hmmm, you shouldn't increase MAX_SHORT, though looking through my email +archives I see that I did indeed recommend doing so. Try setting it back +to 32700; that should suffice that you no longer need -Ca. If it still +hangs, then the interesting question is - where? + +> Compiling the same hanged program with a out-of-the-box (RedHat 4.2 +> distribution of Linux) +> flex 2.5.4 binary works. + +Since Linux comes with source code, you should diff it against what +you have to see what problems they missed. + +> Should I always compile with the -Ca option now ? even short and simple +> filters ? + +No, definitely not. It's meant to be for those situations where you +absolutely must squeeze every last cycle out of your scanner. + + Vern +]]> + + + + +
+ +
+unnamed-faq-80 + + + +Subject: Re: flex output for static code portion +In-reply-to: Your message of Tue, 11 Aug 1998 11:55:30 PDT. +Date: Mon, 17 Aug 1998 23:57:42 PDT +From: Vern Paxson + +> I would like to use flex under the hood to generate a binary file +> containing the data structures that control the parse. + +This has been on the wish-list for a long time. In principle it's +straight-forward - you redirect mkdata() et al's I/O to another file, +and modify the skeleton to have a start-up function that slurps these +into dynamic arrays. The concerns are (1) the scanner generation code +is hairy and full of corner cases, so it's easy to get surprised when +going down this path :-( ; and (2) being careful about buffering so +that when the tables change you make sure the scanner starts in the +correct state and reading at the right point in the input file. + +> I was wondering if you know of anyone who has used flex in this way. + +I don't - but it seems like a reasonable project to undertake (unlike +numerous other flex tweaks :-). + + Vern +]]> + + + + +
+ +
+unnamed-faq-81 + + +; Thu, 20 Aug 1998 00:47:57 -0700 (PDT) +Received: from hal.cl-ki.uni-osnabrueck.de (hal.cl-ki.Uni-Osnabrueck.DE [131.173.141.2]) + by deimos.rz.uni-osnabrueck.de (8.8.7/8.8.8) with ESMTP id JAA34694 + for ; Thu, 20 Aug 1998 09:47:55 +0200 +Received: (from georg@localhost) by hal.cl-ki.uni-osnabrueck.de (8.6.12/8.6.12) id JAA34834 for vern@ee.lbl.gov; Thu, 20 Aug 1998 09:47:54 +0200 +From: Georg Rehm +Message-Id: <199808200747.JAA34834@hal.cl-ki.uni-osnabrueck.de> +Subject: "flex scanner push-back overflow" +To: vern@ee.lbl.gov +Date: Thu, 20 Aug 1998 09:47:54 +0200 (MEST) +Reply-To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE +X-NoJunk: Do NOT send commercial mail, spam or ads to this address! +X-URL: http://www.cl-ki.uni-osnabrueck.de/~georg/ +X-Mailer: ELM [version 2.4ME+ PL28 (25)] +MIME-Version: 1.0 +Content-Type: text/plain; charset=US-ASCII +Content-Transfer-Encoding: 7bit + +Hi Vern, + +Yesterday, I encountered a strange problem: I use the macro processor m4 +to include some lengthy lists into a .l file. Following is a flex macro +definition that causes some serious pain in my neck: + +AUTHOR ("A. Boucard / L. Boucard"|"A. Dastarac / M. Levent"|"A.Boucaud / L.Boucaud"|"Abderrahim Lamchichi"|"Achmat Dangor"|"Adeline Toullier"|"Adewale Maja-Pearce"|"Ahmed Ziri"|"Akram Ellyas"|"Alain Bihr"|"Alain Gresh"|"Alain Guillemoles"|"Alain Joxe"|"Alain Morice"|"Alain Renon"|"Alain Zecchini"|"Albert Memmi"|"Alberto Manguel"|"Alex De Waal"|"Alfonso Artico"| [...]) + +The complete list contains about 10kB. When I try to "flex" this file +(on a Solaris 2.6 machine, using a modified flex 2.5.4 (I only increased +some of the predefined values in flexdefs.h) I get the error: + +myflex/flex -8 sentag.tmp.l +flex scanner push-back overflow + +When I remove the slashes in the macro definition everything works fine. +As I understand it, the double quotes escape the slash-character so it +really means "/" and not "trailing context". Furthermore, I tried to +escape the slashes with backslashes, but with no use, the same error message +appeared when flexing the code. + +Do you have an idea what's going on here? + +Greetings from Germany, + Georg +-- +Georg Rehm georg@cl-ki.uni-osnabrueck.de +Institute for Semantic Information Processing, University of Osnabrueck, FRG +]]> + + + + +
+ +
+unnamed-faq-82 + + + + +> myflex/flex -8 sentag.tmp.l +> flex scanner push-back overflow + +Flex itself uses a flex scanner. That scanner is running out of buffer +space when it tries to unput() the humongous macro you've defined. When +you remove the '/'s, you make it small enough so that it fits in the buffer; +removing spaces would do the same thing. + +The fix is to either rethink how come you're using such a big macro and +perhaps there's another/better way to do it; or to rebuild flex's own +scan.c with a larger value for + + #define YY_BUF_SIZE 16384 + +- Vern +]]> + + + + +
+ +
+unnamed-faq-83 + + + +Subject: Re: Flex +In-reply-to: Your message of Fri, 04 Sep 1998 12:18:43 +0200. +Date: Sat, 05 Sep 1998 00:59:49 PDT +From: Vern Paxson + +> %% +> +> "TEST1\n" { fprintf(stderr, "TEST1\n"); yyless(5); } +> ^\n { fprintf(stderr, "empty line\n"); } +> . { } +> \n { fprintf(stderr, "new line\n"); } +> +> %% +> -- input --------------------------------------- +> TEST1 +> -- output -------------------------------------- +> TEST1 +> empty line +> ------------------------------------------------ + +IMHO, it's not clear whether or not this is in fact a bug. It depends +on whether you view yyless() as backing up in the input stream, or as +pushing new characters onto the beginning of the input stream. Flex +interprets it as the latter (for implementation convenience, I'll admit), +and so considers the newline as in fact matching at the beginning of a +line, as after all the last token scanned an entire line and so the +scanner is now at the beginning of a new line. + +I agree that this is counter-intuitive for yyless(), given its +functional description (it's less so for unput(), depending on whether +you're unput()'ing new text or scanned text). But I don't plan to +change it any time soon, as it's a pain to do so. Consequently, +you do indeed need to use yy_set_bol() and YY_AT_BOL() to tweak +your scanner into the behavior you desire. + +Sorry for the less-than-completely-satisfactory answer. + + Vern +]]> + + + + +
+ +
+unnamed-faq-84 + + + +Subject: Re: Problems with restarting flex-2.5.2-generated scanner +In-reply-to: Your message of Thu, 24 Sep 1998 10:14:07 PDT. +Date: Thu, 24 Sep 1998 23:28:43 PDT +From: Vern Paxson + +> I am using flex-2.5.2 and bison 1.25 for Solaris and I am desperately +> trying to make my scanner restart with a new file after my parser stops +> with a parse error. When my compiler restarts, the parser always +> receives the token after the token (in the old file!) that caused the +> parser error. + +I suspect the problem is that your parser has read ahead in order +to attempt to resolve an ambiguity, and when it's restarted it picks +up with that token rather than reading a fresh one. If you're using +yacc, then the special "error" production can sometimes be used to +consume tokens in an attempt to get the parser into a consistent state. + + Vern +]]> + + + + +
+ +
+unnamed-faq-85 + + + +Subject: Re: flex 2.5.4a +In-reply-to: Your message of Tue, 27 Oct 1998 16:41:42 PST. +Date: Tue, 27 Oct 1998 16:50:14 PST +From: Vern Paxson + +> This brings up a feature request: How about a command line +> option to specify the filename when reading from stdin? That way one +> doesn't need to create a temporary file in order to get the "#line" +> directives to make sense. + +Use -o combined with -t (per the man page description of -o). + +> P.S., Is there any simple way to use non-blocking IO to parse multiple +> streams? + +Simple, no. + +One approach might be to return a magic character on EWOULDBLOCK and +have a rule + + .* // put back .*, eat magic character + +This is off the top of my head, not sure it'll work. + + Vern +]]> + + + + +
+ +
+unnamed-faq-86 + + + +Subject: Re: Compiling scanners +In-reply-to: Your message of Wed, 13 Jan 1999 10:52:47 PST. +Date: Thu, 14 Jan 1999 00:25:30 PST +From: Vern Paxson + +> It appears that maybe it cannot find the lfl library. + +The Makefile in the distribution builds it, so you should have it. +It's exceedingly trivial, just a main() that calls yylex() and +a yyrap() that always returns 1. + +> %% +> \n ++num_lines; ++num_chars; +> . ++num_chars; + +You can't indent your rules like this - that's where the errors are coming +from. Flex copies indented text to the output file, it's how you do things +like + + int num_lines_seen = 0; + +to declare local variables. + + Vern +]]> + + + + +
+ +
+unnamed-faq-87 + + + +Subject: Re: flex input buffer +In-reply-to: Your message of Tue, 09 Feb 1999 13:53:46 PST. +Date: Tue, 09 Feb 1999 21:03:37 PST +From: Vern Paxson + +> In the flex.skl file the size of the default input buffers is set. Can you +> explain why this size is set and why it is such a high number. + +It's large to optimize performance when scanning large files. You can +safely make it a lot lower if needed. + + Vern +]]> + + + + +
+ +
+unnamed-faq-88 + + + +Subject: Re: Flex error message +In-reply-to: Your message of Wed, 24 Feb 1999 15:31:46 PST. +Date: Thu, 25 Feb 1999 00:11:31 PST +From: Vern Paxson + +> I'm extending a larger scanner written in Flex and I keep running into +> problems. More specifically, I get the error message: +> "flex: input rules are too complicated (>= 32000 NFA states)" + +Increase the definitions in flexdef.h for: + +#define JAMSTATE -32766 /* marks a reference to the state that always j +ams */ +#define MAXIMUM_MNS 31999 +#define BAD_SUBSCRIPT -32767 + +recompile everything, and it should all work. + + Vern +]]> + + + + +
+ +
+unnamed-faq-90 + + + +Subject: Re: FLEX trouble +In-reply-to: Your message of Mon, 31 May 1999 18:44:49 PDT. +Date: Tue, 01 Jun 1999 00:15:07 PDT +From: Vern Paxson + +> I have a trouble with FLEX. Why rule "/*".*"*/" work properly,=20 +> but rule "/*"(.|\n)*"*/" don't work ? + +The second of these will have to scan the entire input stream (because +"(.|\n)*" matches an arbitrary amount of any text) in order to see if +it ends with "*/", terminating the comment. That potentially will overflow +the input buffer. + +> More complex rule "/*"([^*]|(\*/[^/]))*"*/ give an error +> 'unrecognized rule'. + +You can't use the '/' operator inside parentheses. It's not clear +what "(a/b)*" actually means. + +> I now use workaround with state , but single-rule is +> better, i think. + +Single-rule is nice but will always have the problem of either setting +restrictions on comments (like not allowing multi-line comments) and/or +running the risk of consuming the entire input stream, as noted above. + + Vern +]]> + + + + +
+ +
+unnamed-faq-91 + + +; Tue, 15 Jun 1999 08:56:06 -0700 (PDT) +Received: from Unknown/Local ([?.?.?.?]) by my-deja.com; Tue Jun 15 08:55:43 1999 +To: vern@ee.lbl.gov +Date: Tue, 15 Jun 1999 08:55:43 -0700 +From: "Aki Niimura" +Message-ID: +Mime-Version: 1.0 +Cc: +X-Sent-Mail: on +Reply-To: +X-Mailer: MailCity Service +Subject: A question on flex C++ scanner +X-Sender-Ip: 12.72.207.61 +Organization: My Deja Email (http://www.my-deja.com:80) +Content-Type: text/plain; charset=us-ascii +Content-Transfer-Encoding: 7bit + +Dear Dr. Paxon, + +I have been using flex for years. +It works very well on many projects. +Most case, I used it to generate a scanner on C language. +However, one project I needed to generate a scanner +on C++ lanuage. Thanks to your enhancement, flex did +the job. + +Currently, I'm working on enhancing my previous project. +I need to deal with multiple input streams (recursive +inclusion) in this scanner (C++). +I did similar thing for another scanner (C) as you +explained in your documentation. + +The generated scanner (C++) has necessary methods: +- switch_to_buffer(struct yy_buffer_state *b) +- yy_create_buffer(istream *is, int sz) +- yy_delete_buffer(struct yy_buffer_state *b) + +However, I couldn't figure out how to access current +buffer (yy_current_buffer). + +yy_current_buffer is a protected member of yyFlexLexer. +I can't access it directly. +Then, I thought yy_create_buffer() with is = 0 might +return current stream buffer. But it seems not as far +as I checked the source. (flex 2.5.4) + +I went through the Web in addition to Flex documentation. +However, it hasn't been successful, so far. + +It is not my intention to bother you, but, can you +comment about how to obtain the current stream buffer? + +Your response would be highly appreciated. + +Best regards, +Aki Niimura + +--== Sent via Deja.com http://www.deja.com/ ==-- +Share what you know. Learn what you don't. +]]> + + + + +
+ +
+unnamed-faq-92 + + + + +> However, I couldn't figure out how to access current +> buffer (yy_current_buffer). + +Derive your own subclass from yyFlexLexer. + + Vern +]]> + + + + +
+ +
+unnamed-faq-93 + + + +Subject: Re: You're the man to see? +In-reply-to: Your message of Wed, 23 Jun 1999 11:10:29 PDT. +Date: Wed, 23 Jun 1999 09:01:40 PDT +From: Vern Paxson + +> I hope you can help me. I am using Flex and Bison to produce an interpreted +> language. However all goes well until I try to implement an IF statement or +> a WHILE. I cannot get this to work as the parser parses all the conditions +> eg. the TRUE and FALSE conditons to check for a rule match. So I cannot +> make a decision!! + +You need to use the parser to build a parse tree (= abstract syntax trwee), +and when that's all done you recursively evaluate the tree, binding variables +to values at that time. + + Vern +]]> + + + + +
+ +
+unnamed-faq-94 + + + +Subject: Re: flex - question +In-reply-to: Your message of Mon, 28 Jun 1999 19:21:41 PDT. +Date: Fri, 02 Jul 1999 16:52:13 PDT +From: Vern Paxson + +> file, it takes an enormous amount of time. It is funny, because the +> source code has only 12 rules!!! I think it looks like an exponencial +> growth. + +Right, that's the problem - some patterns (those with a lot of +ambiguity, where yours has because at any given time the scanner can +be in the middle of all sorts of combinations of the different +rules) blow up exponentially. + +For your rules, there is an easy fix. Change the ".*" that comes fater +the directory name to "[^ ]*". With that in place, the rules are no +longer nearly so ambiguous, because then once one of the directories +has been matched, no other can be matched (since they all require a +leading blank). + +If that's not an acceptable solution, then you can enter a start state +to pick up the .*\n after each directory is matched. + +Also note that for speed, you'll want to add a ".*" rule at the end, +otherwise rules that don't match any of the patterns will be matched +very slowly, a character at a time. + + Vern +]]> + + + + +
+ +
+unnamed-faq-95 + + + +Subject: Re: Please help. +In-reply-to: Your message of Thu, 08 Jul 1999 13:20:37 PDT. +Date: Thu, 08 Jul 1999 08:20:39 PDT +From: Vern Paxson + +> I was hoping you could help me with my problem. +> +> I tried compiling (gnu)flex on a Solaris 2.4 machine +> but when I ran make (after configure) I got an error. +> +> -------------------------------------------------------------- +> gcc -c -I. -I. -g -O parse.c +> ./flex -t -p ./scan.l >scan.c +> sh: ./flex: not found +> *** Error code 1 +> make: Fatal error: Command failed for target `scan.c' +> ------------------------------------------------------------- +> +> What's strange to me is that I'm only +> trying to install flex now. I then edited the Makefile to +> and changed where it says "FLEX = flex" to "FLEX = lex" +> ( lex: the native Solaris one ) but then it complains about +> the "-p" option. Is there any way I can compile flex without +> using flex or lex? +> +> Thanks so much for your time. + +You managed to step on the bootstrap sequence, which first copies +initscan.c to scan.c in order to build flex. Try fetching a fresh +distribution from ftp.ee.lbl.gov. (Or you can first try removing +".bootstrap" and doing a make again.) + + Vern +]]> + + + + +
+ +
+unnamed-faq-96 + + + +Subject: Re: Please help. +In-reply-to: Your message of Fri, 09 Jul 1999 09:16:14 PDT. +Date: Fri, 09 Jul 1999 00:27:20 PDT +From: Vern Paxson + +> First I removed .bootstrap (and ran make) - no luck. I downloaded the +> software but I still have the same problem. Is there anything else I +> could try. + +Try: + + cp initscan.c scan.c + touch scan.c + make scan.o + +If this last tries to first build scan.c from scan.l using ./flex, then +your "make" is broken, in which case compile scan.c to scan.o by hand. + + Vern +]]> + + + + +
+ +
+unnamed-faq-97 + + + +Subject: Re: Error +In-reply-to: Your message of Mon, 19 Jul 1999 23:08:41 PDT. +Date: Tue, 20 Jul 1999 00:18:26 PDT +From: Vern Paxson + +> I am getting a compilation error. The error is given as "unknown symbol- yylex". + +The parser relies on calling yylex(), but you're instead using the C++ scanning +class, so you need to supply a yylex() "glue" function that calls an instance +scanner of the scanner (e.g., "scanner->yylex()"). + + Vern +]]> + + + + +
+ +
+unnamed-faq-98 + + + + +Well, your problem is the + +switch (yybgin-yysvec-1) { /* witchcraft */ + +at the beginning of lex rules. "witchcraft" == "non-portable". It's +assuming knowledge of the AT&T lex's internal variables. + +For flex, you can probably do the equivalent using a switch on YYSTATE. + + Vern +]]> + + + + +
+ +
+unnamed-faq-99 + + + + +> When we provide the customer with an object code distribution, is it +> necessary for us to provide source +> for the generated C files from flex and bison since they are generated by +> flex and bison ? + +For flex, no. I don't know what the current state of this is for bison. + +> Also, is there any requrirement for us to neccessarily provide source for +> the grammar files which are fed into flex and bison ? + +Again, for flex, no. + +See the file "COPYING" in the flex distribution for the legalese. + + Vern +]]> + + + + +
+ +
+unnamed-faq-100 + + + +Subject: Re: Flex, and self referencing rules +In-reply-to: Your message of Sun, 20 Feb 2000 01:01:21 PST. +Date: Sat, 19 Feb 2000 18:33:16 PST +From: Vern Paxson + +> However, I do not use unput anywhere. I do use self-referencing +> rules like this: +> +> UnaryExpr ({UnionExpr})|("-"{UnaryExpr}) + +You can't do this - flex is *not* a parser like yacc (which does indeed +allow recursion), it is a scanner that's confined to regular expressions. + + Vern +]]> + + + + +
+ +
+unnamed-faq-101 + + + + +If this is exactly your program: + +> digit [0-9] +> digits {digit}+ +> whitespace [ \t\n]+ +> +> %% +> "[" { printf("open_brac\n");} +> "]" { printf("close_brac\n");} +> "+" { printf("addop\n");} +> "*" { printf("multop\n");} +> {digits} { printf("NUMBER = %s\n", yytext);} +> whitespace ; + +then the problem is that the last rule needs to be "{whitespace}" ! + + Vern +]]> + + + +@appendix Appendices + + + + +@appendixsec Makefiles and Flex + + + +In this appendix, we provide tips for writing Makefiles to build your scanners. + +In a traditional build environment, we say that the .c files are the +sources, and the .o files are the intermediate files. When using +flex, however, the .l files are the sources, and the generated +.c files (along with the .o files) are the intermediate files. +This requires you to carefully plan your Makefile. + +Modern @command{make} programs understand that foo.l is intended to +generate lex.yy.c or foo.c, and will behave +accordingly@footnote{GNU @command{make} and GNU @command{automake} are two such +programs that provide implicit rules for flex-generated scanners.}. The +following Makefile does not explicitly instruct @command{make} how to build +foo.c from foo.l. Instead, it relies on the implicit rules of the +@command{make} program to build the intermediate file, scan.c: + + + + + + + + + +For simple cases, the above may be sufficient. For other cases, +you may have to explicitly instruct @command{make} how to build your scanner. +The following is an example of a Makefile containing explicit rules: + + + + + + + + +Notice in the above example that scan.c is in the @code{clean} target. +This is because we consider the file scan.c to be an intermediate file. + +Finally, we provide a realistic example of a flex scanner used with a +bison parser@footnote{This example also applies to yacc parsers.}. +There is a tricky problem we have to deal with. Since a flex scanner +will typically include a header file (e.g., y.tab.h) generated by the +parser, we need to be sure that the header file is generated BEFORE the scanner +is compiled. We handle this case in the following example: + + + + + + + +In the above example, notice the line, + + + + + + + +, which lists the file parse.c (the generated parser) as a dependency of +scan.o. We want to ensure that the parser is created before the scanner +is compiled, and the above line seems to do the trick. Feel free to experiment +with your specific implementation of @command{make}. + + +For more details on writing Makefiles, see @ref{Top, , , make, The +GNU Make Manual}. + +
+ +
+C Scanners with Bison Parsers + + + + + + + +This section describes the flex features useful when integrating +flex with @code{GNU bison}@footnote{The features described here are +purely optional, and are by no means the only way to use flex with bison. +We merely provide some glue to ease development of your parser-scanner pair.}. +Skip this section if you are not using +bison with your scanner. Here we discuss only the flex +half of the flex and bison pair. We do not discuss +bison in any detail. For more information about generating +bison parsers, see @ref{Top, , , bison, the GNU Bison Manual}. + +A compatible bison scanner is generated by declaring @samp{%option +bison-bridge} or by supplying @samp{--bison-bridge} when invoking flex +from the command line. This instructs flex that the macro +yylval may be used. The data type for +yylval, @code{YYSTYPE}, +is typically defined in a header file, included in section 1 of the +flex input file. For a list of functions and macros +available, @xref{bison-functions}. + +The declaration of yylex becomes, + + + + + + + + +If @code{%option bison-locations} is specified, then the declaration +becomes, + + + + + + + + +Note that the macros yylval and yylloc evaluate to pointers. +Support for yylloc is optional in bison, so it is optional in +flex as well. The following is an example of a flex scanner that +is compatible with bison. + + + + +num = atoi(yytext); return NUMBER;} + [[:alnum:]]+ { yylval->str = strdup(yytext); return STRING;} + "="|";" { return yytext[0];} + . {} + % +]]> + + + +As you can see, there really is no magic here. We just use +yylval as we would any other variable. The data type of +yylval is generated by bison, and included in the file +y.tab.h. Here is the corresponding bison parser: + + + + + STRING + %token NUMBER + %% + assignment: + STRING '=' NUMBER ';' { + printf( "(setf %s %d)", $1, $3 ); + } + ; +]]> + + + +
+ +
+M4 Dependency + + +The macro processor m4@footnote{The use of m4 is subject to change in +future revisions of flex.} must be installed wherever flex is installed. +flex invokes @samp{m4}, found by searching the directories in the +@code{PATH} environment variable. Any code you place in section 1 or in the +actions will be sent through m4. Please follow these rules to protect your +code from unwanted m4 processing. + + + + + + Do not use symbols that begin with, @samp{m4_}, such as, @samp{m4_define}, +or @samp{m4_include}, since those are reserved for m4 macro names. + + + + Do not use the strings @samp{[[} or @samp{]]} anywhere in your code. The +former is not valid in C, except within comments, but the latter is valid in +code such as @code{x[y[z]]}. + + + + +m4 is only required at the time you run flex. The generated +scanner is ordinary C or C++, and does @emph{not} require m4. + + +
+ +
+Indices + + + + +
+ +
+Concept Index + +@printindex cp + +
+ +
+Index of Functions and Macros + +This is an index of functions and preprocessor macros that look like functions. +For macros that expand to variables or constants, see @ref{Index of Variables}. + +@printindex fn + +
+ +
+Index of Variables + +This is an index of variables, constants, and preprocessor macros +that expand to variables or constants. + +@printindex vr + +
+ +
+Index of Data Types +@printindex tp + +
+ +
+Index of Hooks + +This is an index of "hooks" that the user may define. These hooks typically correspond +to specific locations in the generated scanner, and may be used to insert arbitrary code. + +@printindex hk + +
+ +
+Index of Scanner Options + +@printindex op + + + -- cgit v1.2.3