FAQ
***

When was flex born?
===================

When was flex born?

Vern Paxson took over the Software Tools lex project from Jef Poskanzer in 1982.  At that point it was written in Ratfor.  Around 1987 or so, Paxson translated it into C, and a legend was born :-).

How do I expand \ escape sequences in C-style quoted strings?
=============================================================

How do I expand \ escape sequences in C-style quoted strings?

A key point when scanning quoted strings is that you cannot (easily) write a single rule that will precisely match the string if you allow things like embedded escape sequences and newlines.  If you try to match strings with a single rule then you'll wind up having to rescan the string anyway to find any escape sequences.

Instead you use exclusive start conditions and a set of rules, one for matching non-escaped text, one for matching a single escape, one for matching an embedded newline, and one for recognizing the end of the string.  Each of these rules is then faced with the question of where to put its intermediary results.  The best solution is for the rules to append their local value of yytext to the end of a "string literal" buffer.  A rule like the escape-matcher will append to the buffer the meaning of the escape sequence rather than the literal text in yytext. In this way, yytext does not need to be modified at all.

Why do flex scanners call fileno if it is not ANSI compatible?
==============================================================

Why do flex scanners call fileno if it is not ANSI compatible?

Flex scanners call fileno() in order to get the file descriptor corresponding to yyin. The file descriptor may be passed to isatty() or read(), depending upon which %options you specified. If your system does not have fileno() support. To get rid of the read() call, do not specify %option read. To get rid of the isatty() call, you must specify one of %option always-interactive or %option never-interactive.

Does flex support recursive pattern definitions?
================================================

Does flex support recursive pattern definitions? e.g.,

     %%
     block   "{"({block}|{statement})*"}"

No. You cannot have recursive definitions.  The pattern-matching power of regular expressions in general (and therefore flex scanners, too) is limited.  In particular, regular expressions cannot "balance" parentheses to an arbitrary degree.  For example, it's impossible to write a regular expression that matches all strings containing the same number of '{'s as '}'s.  For more powerful pattern matching, you need a parser, such as GNU bison.

How do skip huge chunks of input (tens of megabytes) while using flex?
======================================================================

How do skip huge chunks of input (tens of megabytes) while using flex?

Use fseek (or lseek) to position yyin, then call yyrestart().

Flex is not matching my patterns in the same order that I defined them.
=======================================================================

Flex is not matching my patterns in the same order that I defined them.

This is indeed the natural way to expect it to work, however, flex picks the rule that matches the most text (i.e., the longest possible input string). This is because flex uses an entirely different matching technique ("deterministic finite automata") that actually does all of the matching simultaneously, in parallel.  (Seems impossible, but it's actually a fairly simple technique once you understand the principles.)

A side-effect of this parallel matching is that when the input matches more than one rule, flex scanners pick the rule that matched the *most* text. This is explained further in the manual, in the section "How the input is Matched".

If you want flex to choose a shorter match, then you can work around this behavior by expanding your short rule to match more text, then put back the extra:

     data_.*        yyless( 5 ); BEGIN BLOCKIDSTATE;

Another fix would be to make the second rule active only during the <BLOCKIDSTATE> start condition, and make that start condition exclusive by declaring it with %x instead of %s.

A final fix is to change the input language so that the ambiguity for data_ is removed, by adding characters to it that don't match the identifier rule, or by removing characters (such as '_') from the identifier rule so it no longer matches "data_".  (Of course, you might also not have the option of changing the input language ...)

My actions are executing out of order or sometimes not at all.
==============================================================

My actions are executing out of order or sometimes not at all. What's happening?

Most likely, you have (in error) placed the opening { of the action block on a different line than the rule, e.g.,

     ^(foo|bar)
        {  <<<--- WRONG!

        }

flex requires that the opening { of an action associated with a rule begin on the same line as does the rule.  You need instead to write your rules as follows:

     ^(foo|bar)   {  // CORRECT!

         }

How can I have multiple input sources feed into the same scanner at the same time?
==================================================================================

How can I have multiple input sources feed into the same scanner at the same time?

If...

  your scanner is free of backtracking (verified using flex's -b flag),

  AND you run it interactively (-I option; default unless using special table compression options),

  AND you feed it one character at a time by redefining YY_INPUT to do so,

then every time it matches a token, it will have exhausted its input buffer (because the scanner is free of backtracking).  This means you can safely use select() at the point and only call yylex() for another token if select() indicates there's data available.

That is, move the select() out from the input function to a point where it determines whether yylex() gets called for the next token.

With this approach, you will still have problems if your input can arrive piecemeal; select() could inform you that the beginning of a token is available, you call yylex() to get it, but it winds up blocking waiting for the later characters in the token.

Here's another way:  Move your input multiplexing inside of YY_INPUT.  That is, whenever YY_INPUT is called, it select()'s to see where input is available.  If input is available for the scanner, it reads and returns the next byte.  If input is available from another source, it calls whatever function is responsible for reading from that source.  (If no input is available, it blocks until some is.)  I've used this technique in an interpreter I wrote that both reads keyboard input using a flex scanner and IPC traffic from sockets, and it works fine.

Can I build nested parsers that work with the same input file?
==============================================================

Can I build nested parsers that work with the same input file?

This is not going to work without some additional effort.  The reason is that flex block-buffers the input it reads from yyin.  This means that the "outermost" yylex(), when called, will automatically slurp up the first 8K of input available on yyin, and subsequent calls to other yylex()'s won't see that input.  You might be tempted to work around this problem by redefining YY_INPUT to only return a small amount of text, but it turns out that that approach is quite difficult.  Instead, the best solution is to combine all of your scanners into one large scanner, using a different exclusive start condition for each.

How can I match text only at the end of a file?
===============================================

How can I match text only at the end of a file?

There is no way to write a rule which is "match this text, but only if it comes at the end of the file".  You can fake it, though, if you happen to have a character lying around that you don't allow in your input. Then you redefine YY_INPUT to call your own routine which, if it sees an EOF, returns the magic character first (and remembers to return a real EOF next time it's called).  Then you could write:

     <COMMENT>(.|\n)*{EOF_CHAR}    /* saw comment at EOF */

How can I make REJECT cascade across start condition boundaries?
================================================================

How can I make REJECT cascade across start condition boundaries?

You can do this as follows.  Suppose you have a start condition A, and after exhausting all of the possible matches in <A>, you want to try matches in <INITIAL>.  Then you could use the following:

     %x A
     %%
     <A>rule_that_is_long    ...; REJECT;
     <A>rule                 ...; REJECT; /* shorter rule */
     <A>etc.
     ...
     <A>.|\n  {
                 /* Shortest and last rule in <A>, so
                  * cascaded REJECT's will eventually
                  * wind up matching this rule.  We want
                  * to now switch to the initial state
                  * and try matching from there instead.
                  */
                 yyless(0);    /* put back matched text */
                 BEGIN(INITIAL);
              }

Why can't I use fast or full tables with interactive mode?
==========================================================

Why can't I use fast or full tables with interactive mode?

One of the assumptions flex makes is that interactive applications are inherently slow (for just that reason, they're waiting on a human). It has to do with how the scanner detects that it must be finished scanning a token.  For interactive scanners, after scanning each character the current state is looked up in a table (essentially) to see whether there's a chance of another input character possibly extending the length of the match.  If not, the scanner halts.  For non-interactive scanners, the end-of-token test is much simpler, basically a compare with 0, so no memory bus cycles.  Since the test occurs in the innermost scanning loop, one would like to make it go as fast as possible.

Still, it seems reasonable to allow the user to choose to trade off a bit of performance in this area to gain the corresponding flexibility.  There might be another reason, though, why fast scanners don't support the interactive option

How much faster is -F or -f than -C?
====================================

How much faster is -F or -f than -C?

Much faster (factor of 2-3).

If I have a simple grammar can't I just parse it with flex?
===========================================================

If I have a simple grammar, can't I just parse it with flex?

Is your grammar recursive? That's almost always a sign that you're better off using a parser/scanner rather than just trying to use a scanner alone.

Why doesn't yyrestart() set the start state back to INITIAL?
============================================================

Why doesn't yyrestart() set the start state back to INITIAL?

There are two reasons.  The first is that there might be programs that rely on the start state not changing across file changes. The second is that with flex 2.4, use of yyrestart() is no longer required, so fixing the problem there doesn't solve the more general problem.

How can I match C-style comments?
=================================

How can I match C-style comments?

You might be tempted to try something like this:

     "/*".*"*/"       // WRONG!

or, worse, this:

     "/*"(.|\n)"*/"   // WRONG!

The above rules will eat too much input, and blow up on things like:

         /* a comment */ do_my_thing( "oops */" );

Here is one way which allows you to track line information:

     <INITIAL>{
         "/*"              BEGIN(IN_COMMENT);
     }
     <IN_COMMENT>{
         "*/"      BEGIN(INITIAL);
         [^*\n]+   // eat comment in chunks
         "*"       // eat the lone star
         \n        yylineno++;
     }

The '.' isn't working the way I expected.
=========================================

The '.' (dot) isn't working the way I expected.

Here are some tips for using .:

  * A common mistake is to place the grouping parenthesis AFTER an operator, when you really meant to place the parenthesis BEFORE the operator, e.g., you probably want this (foo|bar)+ and NOT this (foo|bar+).

     The first pattern matches the words foo or bar any number of times, e.g., it matches the text barfoofoobarfoo. The second pattern matches a single instance of foo or a single instance of ba followed by one or more rs, e.g., it matches the text barrrr .

  * A . inside []'s just means a literal. (period), and NOT "any character except newline".

  * Remember that . matches any character EXCEPT \n (and EOF). If you really want to match ANY character, including newlines, then use (.|\n) --- Beware that the regex (.|\n)+ will match your entire input!

  * Finally, if you want to match a literal . (a period), then use [.] or "."

Can I get the flex manual in another format?
============================================

Can I get the flex manual in another format?

As of flex 2.5, the manual is distributed in texinfo format. You can use the "texi2*" tools to convert the manual to any format you desire (e.g., texi2html).

Does there exist a "faster" NDFA->DFA algorithm?
================================================

Does there exist a "faster" NDFA->DFA algorithm? Most standard texts (e.g., Aho), imply that NDFA->DFA can take exponential time, since there are exponential number of potential states in NDFA.

There's no way around the potential exponential running time - it can take you exponential time just to enumerate all of the DFA states. In practice, though, the running time is closer to linear, or sometimes quadratic.

How does flex compile the DFA so quickly?
=========================================

How does flex compile the DFA so quickly?

There are two big speed wins that flex uses:

 1. It analyzes the input rules to construct equivalence classes for those characters that always make the same transitions.  It then rewrites the NFA using equivalence classes for transitions instead of characters.  This cuts down the NFA->DFA computation time dramatically, to the point where, for uncompressed DFA tables, the DFA generation is often I/O bound in writing out the tables.

 2. It maintains hash values for previously computed DFA states, so testing whether a newly constructed DFA state is equivalent to a previously constructed state can be done very quickly, by first comparing hash values.

How can I use more than 8192 rules?
===================================

How can I use more than 8192 rules?

Flex is compiled with an upper limit of 8192 rules per scanner. If you need more than 8192 rules in your scanner, you'll have to recompile flex with the following changes in flexdef.h:

     <    #define YY_TRAILING_MASK 0x2000
     <    #define YY_TRAILING_HEAD_MASK 0x4000
     --
     >    #define YY_TRAILING_MASK 0x20000000
     >    #define YY_TRAILING_HEAD_MASK 0x40000000

This should work okay as long as your C compiler uses 32 bit integers. But you might want to think about whether using such a huge number of rules is the best way to solve your problem.

How do I abandon a file in the middle of a scan and switch to a new file?
=========================================================================

How do I abandon a file in the middle of a scan and switch to a new file?

Just all yyrestart(newfile). Be sure to reset the start state if you want a "fresh" start, since yyrestart does NOT reset the start state back to INITIAL.

How do I execute code only during initialization (only before the first scan)?
==============================================================================

How do I execute code only during initialization (only before the first scan)?

You can specify an initial action by defining the macro YY_USER_INIT (though note that yyout may not be available at the time this macro is executed).  Or you can add to the beginning of your rules section:

     %%
         /* Must be indented! */
         static int did_init = 0;

         if ( ! did_init ){
             do_my_init();
             did_init = 1;
         }

How do I execute code at termination?
=====================================

How do I execute code at termination (i.e., only after the last scan?)

You can specifiy an action for the <<EOF>> rule.

Where else can I find help?
===========================

Where else can I find help?

The help-flex email list is served by GNU. See http://www.gnu.org/ for details how to subscribe or search the archives.

Can I include comments in the "rules" section of the file file?
===============================================================

Can I include comments in the "rules" section of the file file?

Yes, just about anywhere you want to. See the manual for the specific syntax.

I get an error about undefined yywrap().
========================================

I get an error about undefined yywrap().

You must supply a yywrap() function of your own, or link to libfl.a (which provides one), or use

%option noyywrap

in your source to say you don't want a yywrap() function. See the manual page for more details concerning yywrap().

How can I change the matching pattern at run time?
==================================================

How can I change the matching pattern at run time?

You can't, it's compiled into a static table when flex builds the scanner.

Is there a way to increase the rules (NFA states to a bigger number?)
=====================================================================

Is there a way to increase the rules (NFA states to a bigger number?)

With luck, you should be able to increase the definitions in flexdef.h for:

     #define JAMSTATE -32766 /* marks a reference to the state that always jams */
     #define MAXIMUM_MNS 31999
     #define BAD_SUBSCRIPT -32767

recompile everything, and it'll all work.  Flex only has these 16-bit-like values built into it because a long time ago it was developed on a machine with 16-bit ints.  I've given this advice to others in the past but haven't heard back from them whether it worked okay or not...

How can I expand macros in the input?
=====================================

How can I expand macros in the input?

The best way to approach this problem is at a higher level, e.g., in the parser.

However, you can do this using multiple input buffers.

     %%
     macro/[a-z]+	{
         /* Saw the macro "macro" followed by extra stuff. */
         main_buffer = YY_CURRENT_BUFFER;
         expansion_buffer = yy_scan_string(expand(yytext));
         yy_switch_to_buffer(expansion_buffer);
         }

     <<EOF>>	{
         if ( expansion_buffer )
             {
             // We were doing an expansion, return to where
             // we were.
             yy_switch_to_buffer(main_buffer);
             yy_delete_buffer(expansion_buffer);
             expansion_buffer = 0;
             }
         else
             yyterminate();
         }

You probably will want a stack of expansion buffers to allow nested macros. From the above though hopefully the idea is clear.

How can I build a two-pass scanner?
===================================

How can I build a two-pass scanner?

One way to do it is to filter the first pass to a temporary file, then process the temporary file on the second pass. You will probably see a performance hit, do to all the disk I/O.

When you need to look ahead far forward like this, it almost always means that the right solution is to build a parse tree of the entire input, then walk it after the parse in order to generate the output.  In a sense, this is a two-pass approach, once through the text and once through the parse tree, but the performance hit for the latter is usually an order of magnitude smaller, since everything is already classified, in binary format, and residing in memory.

How do I match any string not matched in the preceding rules?
=============================================================

How do I match any string not matched in the preceding rules?

One way to assign precedence, is to place the more specific rules first. If two rules would match the same input (same sequence of characters) then the first rule listed in the flex input wins. e.g.,

     %%
     foo[a-zA-Z_]+    return FOO_ID;
     bar[a-zA-Z_]+    return BAR_ID;
     [a-zA-Z_]+       return GENERIC_ID;

Note that the rule [a-zA-Z_]+ must come *after* the others.  It will match the same amount of text as the more specific rules, and in that case the flex scanner will pick the first rule listed in your scanner as the one to match.

I am trying to port code from AT&T lex that uses yysptr and yysbuf.
===================================================================

I am trying to port code from AT&T lex that uses yysptr and yysbuf.

Those are internal variables pointing into the AT&T scanner's input buffer.  I imagine they're being manipulated in user versions of the input() and unput() functions.  If so, what you need to do is analyze those functions to figure out what they're doing, and then replace input() with an appropriate definition of YY_INPUT (see the flex man page).  You shouldn't need to (and must not) replace flex's unput() function.

Is there a way to make flex treat NULL like a regular character?
================================================================

Is there a way to make flex treat NULL like a regular character?

Yes, \0 and \x00 should both do the trick.  Perhaps you have an ancient version of flex.  The latest release is version VERSION.

Whenever flex can not match the input it says "flex scanner jammed".
====================================================================

Whenever flex can not match the input it says "flex scanner jammed".

You need to add a rule that matches the otherwise-unmatched text. e.g.,

     %option yylineno
     %%
     [[a bunch of rules here]]

     .	printf("bad input character '%s' at line %d\n", yytext, yylineno);

See %option default for more information.

Why doesn't flex have non-greedy operators like perl does?
==========================================================

Why doesn't flex have non-greedy operators like perl does?

A DFA can do a non-greedy match by stopping the first time it enters an accepting state, instead of consuming input until it determines that no further matching is possible (a "jam" state).  This is actually easier to implement than longest leftmost match (which flex does).

But it's also much less useful than longest leftmost match.  In general, when you find yourself wishing for non-greedy matching, that's usually a sign that you're trying to make the scanner do some parsing.  That's generally the wrong approach, since it lacks the power to do a decent job. Better is to either introduce a separate parser, or to split the scanner into multiple scanners using (exclusive) start conditions.

You might have a separate start state once you've seen the BEGIN. In that state, you might then have a regex that will match END (to kick you out of the state), and perhaps (.|\n) to get a single character within the chunk ...

This approach also has much better error-reporting properties.

Memory leak - 16386 bytes allocated by malloc.
==============================================

UPDATED 2002-07-10: As of flex version 2.5.9, this leak means that you did not call yylex_destroy(). If you are using an earlier version of flex, then read on.

The leak is about 16426 bytes.  That is, (8192 * 2 + 2) for the read-buffer, and about 40 for struct yy_buffer_state (depending upon alignment). The leak is in the non-reentrant C scanner only (NOT in the reentrant scanner, NOT in the C++ scanner). Since flex doesn't know when you are done, the buffer is never freed.

However, the leak won't multiply since the buffer is reused no matter how many times you call yylex().

If you want to reclaim the memory when you are completely done scanning, then you might try this:

     /* For non-reentrant C scanner only. */
     yy_delete_buffer(yy_current_buffer);
     yy_init = 1;

Note: yy_init is an "internal variable", and hasn't been tested in this situation. It is possible that some other globals may need resetting as well.

How do I track the byte offset for lseek()?
===========================================

     >   We thought that it would be possible to have this number through the
     >   evaluation of the following expression:
     >
     >   seek_position = (no_buffers)*YY_READ_BUF_SIZE + yy_c_buf_p - yy_current_buffer->yy_ch_buf

While this is the right ideas, it has two problems.  The first is that it's possible that flex will request less than YY_READ_BUF_SIZE during an invocation of YY_INPUT (or that your input source will return less even though YY_READ_BUF_SIZE bytes were requested).  The second problem is that when refilling its internal buffer, flex keeps some characters from the previous buffer (because usually it's in the middle of a match, and needs those characters to construct yytext for the match once it's done).  Because of this, yy_c_buf_p - yy_current_buffer->yy_ch_buf won't be exactly the number of characters already read from the current buffer.

An alternative solution is to count the number of characters you've matched since starting to scan.  This can be done by using YY_USER_ACTION.  For example,

#define YY_USER_ACTION num_chars += yyleng;

(You need to be careful to update your bookkeeping if you use yymore(), yyless(), unput(), or input().)

unnamed-faq-16
==============

     To: steves@telebase.com
     Subject: Re: flex C++ question
     In-reply-to: Your message of Thu, 08 Dec 94 13:10:58 EST.
     Date: Wed, 14 Dec 94 16:40:47 PST
     From: Vern Paxson <vern>

     > We'd like to override the provided LexerInput() and LexerOutput()
     > functions, but we'd like to *not* use iostreams.  Instead, we'd like
     > to use some of our own I/O classes.  Is this possible?

     You can do this by passing the various functions nil iostream*'s, and then
     dealing with your own I/O classes surreptitiously (i.e., stashing them in
     special member variables).  This works because the only assumption about
     the lexer regarding what's done with the iostream's is that they're
     ultimately passed to LexerInput and LexerOutput, which then do whatever
     necessary with them.

     When the flex C++ scanning class rewrite finally happens (no date for this
     in sight), then this sort of thing should become much easier.

     		Vern

How do I skip as many chars as possible?
========================================

How do I skip as many chars as possible -- without interfering with the other patterns?

In the example below, we want to skip over characters until we see the phrase "endskip". The following will NOT work correctly (do you see why not?)

       /* INCORRECT SCANNER */
     %x SKIP
     %%
     <INITIAL>startskip   BEGIN(SKIP);
     ...
     <SKIP>"endskip"       BEGIN(INITIAL);
     <SKIP>.*             ;

The problem is that the pattern .* will eat up the word "endskip." The simplest (but slow) fix is:

     <SKIP>"endskip"      BEGIN(INITIAL);
     <SKIP>.              ;

The fix involves making the second rule match more, without making it match "endskip" plus something else.  So for example:

     <SKIP>"endskip"     BEGIN(INITIAL);
     <SKIP>[^e]+         ;
     <SKIP>.		        ;/* so you eat up e's, too */

unnamed-faq-33
==============

     QUESTION:
     When was flex born?

     Vern Paxson took over
     the Software Tools lex project from Jef Poskanzer in 1982.  At that point it
     was written in Ratfor.  Around 1987 or so, Paxson translated it into C, and
     a legend was born :-).

unnamed-faq-42
==============

     To: Adoram Rogel <adoram@orna.hybridge.com>
     Subject: Re: Flex 2.5.2 performance questions
     In-reply-to: Your message of Wed, 18 Sep 96 11:12:17 EDT.
     Date: Wed, 18 Sep 96 10:51:02 PDT
     From: Vern Paxson <vern>

     [Note, the most recent flex release is 2.5.4, which you can get from
      ftp.ee.lbl.gov.  It has bug fixes over 2.5.2 and 2.5.3.]

     > 1. Using the pattern
     >    ([Ff](oot)?)?[Nn](ote)?(\.)?
     >    instead of
     >    (((F|f)oot(N|n)ote)|((N|n)ote)|((N|n)\.)|((F|f)(N|n)(\.)))
     >    (in a very complicated flex program) caused the program to slow from
     >    300K+/min to 100K/min (no other changes were done).

     These two are not equivalent.  For example, the first can match "footnote."
     but the second can only match "footnote".  This is almost certainly the
     cause in the discrepancy - the slower scanner run is matching more tokens,
     and/or having to do more backing up.

     > 2. Which of these two are better: [Ff]oot or (F|f)oot ?

     From a performance point of view, they're equivalent (modulo presumably
     minor effects such as memory cache hit rates; and the presence of trailing
     context, see below).  From a space point of view, the first is slightly
     preferable.

     > 3. I have a pattern that look like this:
     >    pats {p1}|{p2}|{p3}|...|{p50}     (50 patterns ORd)
     >
     >    running yet another complicated program that includes the following rule:
     >    <snext>{and}/{no4}{bb}{pats}
     >
     >    gets me to "too complicated - over 32,000 states"...

     I can't tell from this example whether the trailing context is variable-length
     or fixed-length (it could be the latter if {and} is fixed-length).  If it's
     variable length, which flex -p will tell you, then this reflects a basic
     performance problem, and if you can eliminate it by restructuring your
     scanner, you will see significant improvement.

     >    so I divided {pats} to {pats1}, {pats2},..., {pats5} each consists of about
     >    10 patterns and changed the rule to be 5 rules.
     >    This did compile, but what is the rule of thumb here ?

     The rule is to avoid trailing context other than fixed-length, in which for
     a/b, either the 'a' pattern or the 'b' pattern have a fixed length.  Use
     of the '|' operator automatically makes the pattern variable length, so in
     this case '[Ff]oot' is preferred to '(F|f)oot'.

     > 4. I changed a rule that looked like this:
     >    <snext8>{and}{bb}/{ROMAN}[^A-Za-z] { BEGIN...
     >
     >    to the next 2 rules:
     >    <snext8>{and}{bb}/{ROMAN}[A-Za-z] { ECHO;}
     >    <snext8>{and}{bb}/{ROMAN}         { BEGIN...
     >
     >    Again, I understand the using [^...] will cause a great performance loss

     Actually, it doesn't cause any sort of performance loss.  It's a surprising
     fact about regular expressions that they always match in linear time
     regardless of how complex they are.

     >    but are there any specific rules about it ?

     See the "Performance Considerations" section of the man page, and also
     the example in MISC/fastwc/.

     		Vern

unnamed-faq-43
==============

     To: Adoram Rogel <adoram@hybridge.com>
     Subject: Re: Flex 2.5.2 performance questions
     In-reply-to: Your message of Thu, 19 Sep 96 10:16:04 EDT.
     Date: Thu, 19 Sep 96 09:58:00 PDT
     From: Vern Paxson <vern>

     > a lot about the backing up problem.
     > I believe that there lies my biggest problem, and I'll try to improve
     > it.

     Since you have variable trailing context, this is a bigger performance
     problem.  Fixing it is usually easier than fixing backing up, which in a
     complicated scanner (yours seems to fit the bill) can be extremely
     difficult to do correctly.

     You also don't mention what flags you are using for your scanner.
     -f makes a large speed difference, and -Cfe buys you nearly as much
     speed but the resulting scanner is considerably smaller.

     > I have an | operator in {and} and in {pats} so both of them are variable
     > length.

     -p should have reported this.

     > Is changing one of them to fixed-length is enough ?

     Yes.

     > Is it possible to change the 32,000 states limit ?

     Yes.  I've appended instructions on how.  Before you make this change,
     though, you should think about whether there are ways to fundamentally
     simplify your scanner - those are certainly preferable!

     		Vern

     To increase the 32K limit (on a machine with 32 bit integers), you increase
     the magnitude of the following in flexdef.h:

             #define JAMSTATE -32766 /* marks a reference to the state that always jams */
             #define MAXIMUM_MNS 31999
             #define BAD_SUBSCRIPT -32767
             #define MAX_SHORT 32700

     Adding a 0 or two after each should do the trick.

unnamed-faq-44
==============

     To: Heeman_Lee@hp.com
     Subject: Re: flex - multi-byte support?
     In-reply-to: Your message of Thu, 03 Oct 1996 17:24:04 PDT.
     Date: Fri, 04 Oct 1996 11:42:18 PDT
     From: Vern Paxson <vern>

     >      I assume as long as my *.l file defines the
     >      range of expected character code values (in octal format), flex will
     >      scan the file and read multi-byte characters correctly. But I have no
     >      confidence in this assumption.

     Your lack of confidence is justified - this won't work.

     Flex has in it a widespread assumption that the input is processed
     one byte at a time.  Fixing this is on the to-do list, but is involved,
     so it won't happen any time soon.  In the interim, the best I can suggest
     (unless you want to try fixing it yourself) is to write your rules in
     terms of pairs of bytes, using definitions in the first section:

     	X	\xfe\xc2
     	...
     	%%
     	foo{X}bar	found_foo_fe_c2_bar();

     etc.  Definitely a pain - sorry about that.

     By the way, the email address you used for me is ancient, indicating you
     have a very old version of flex.  You can get the most recent, 2.5.4, from
     ftp.ee.lbl.gov.

     		Vern

unnamed-faq-45
==============

     To: moleary@primus.com
     Subject: Re: Flex / Unicode compatibility question
     In-reply-to: Your message of Tue, 22 Oct 1996 10:15:42 PDT.
     Date: Tue, 22 Oct 1996 11:06:13 PDT
     From: Vern Paxson <vern>

     Unfortunately flex at the moment has a widespread assumption within it
     that characters are processed 8 bits at a time.  I don't see any easy
     fix for this (other than writing your rules in terms of double characters -
     a pain).  I also don't know of a wider lex, though you might try surfing
     the Plan 9 stuff because I know it's a Unicode system, and also the PCCT
     toolkit (try searching say Alta Vista for "Purdue Compiler Construction
     Toolkit").

     Fixing flex to handle wider characters is on the long-term to-do list.
     But since flex is a strictly spare-time project these days, this probably
     won't happen for quite a while, unless someone else does it first.

     		Vern

unnamed-faq-46
==============

     To: Johan Linde <jl@theophys.kth.se>
     Subject: Re: translation of flex
     In-reply-to: Your message of Sun, 10 Nov 1996 09:16:36 PST.
     Date: Mon, 11 Nov 1996 10:33:50 PST
     From: Vern Paxson <vern>

     > I'm working for the Swedish team translating GNU program, and I'm currently
     > working with flex. I have a few questions about some of the messages which
     > I hope you can answer.

     All of the things you're wondering about, by the way, concerning flex
     internals - probably the only person who understands what they mean in
     English is me!  So I wouldn't worry too much about getting them right.
     That said ...

     > #: main.c:545
     > msgid "  %d protos created\n"
     >
     > Does proto mean prototype?

     Yes - prototypes of state compression tables.

     > #: main.c:539
     > msgid "  %d/%d (peak %d) template nxt-chk entries created\n"
     >
     > Here I'm mainly puzzled by 'nxt-chk'. I guess it means 'next-check'. (?)
     > However, 'template next-check entries' doesn't make much sense to me. To be
     > able to find a good translation I need to know a little bit more about it.

     There is a scheme in the Aho/Sethi/Ullman compiler book for compressing
     scanner tables.  It involves creating two pairs of tables.  The first has
     "base" and "default" entries, the second has "next" and "check" entries.
     The "base" entry is indexed by the current state and yields an index into
     the next/check table.  The "default" entry gives what to do if the state
     transition isn't found in next/check.  The "next" entry gives the next
     state to enter, but only if the "check" entry verifies that this entry is
     correct for the current state.  Flex creates templates of series of
     next/check entries and then encodes differences from these templates as a
     way to compress the tables.

     > #: main.c:533
     > msgid "  %d/%d base-def entries created\n"
     >
     > The same problem here for 'base-def'.

     See above.

     		Vern

unnamed-faq-47
==============

     To: Xinying Li <xli@npac.syr.edu>
     Subject: Re: FLEX ?
     In-reply-to: Your message of Wed, 13 Nov 1996 17:28:38 PST.
     Date: Wed, 13 Nov 1996 19:51:54 PST
     From: Vern Paxson <vern>

     > "unput()" them to input flow, question occurs. If I do this after I scan
     > a carriage, the variable "yy_current_buffer->yy_at_bol" is changed. That
     > means the carriage flag has gone.

     You can control this by calling yy_set_bol().  It's described in the manual.

     >      And if in pre-reading it goes to the end of file, is anything done
     > to control the end of curren buffer and end of file?

     No, there's no way to put back an end-of-file.

     >      By the way I am using flex 2.5.2 and using the "-l".

     The latest release is 2.5.4, by the way.  It fixes some bugs in 2.5.2 and
     2.5.3.  You can get it from ftp.ee.lbl.gov.

     		Vern

unnamed-faq-48
==============

     To: Alain.ISSARD@st.com
     Subject: Re: Start condition with FLEX
     In-reply-to: Your message of Mon, 18 Nov 1996 09:45:02 PST.
     Date: Mon, 18 Nov 1996 10:41:34 PST
     From: Vern Paxson <vern>

     > I am not able to use the start condition scope and to use the | (OR) with
     > rules having start conditions.

     The problem is that if you use '|' as a regular expression operator, for
     example "a|b" meaning "match either 'a' or 'b'", then it must *not* have
     any blanks around it.  If you instead want the special '|' *action* (which
     from your scanner appears to be the case), which is a way of giving two
     different rules the same action:

     	foo	|
     	bar	matched_foo_or_bar();

     then '|' *must* be separated from the first rule by whitespace and *must*
     be followed by a new line.  You *cannot* write it as:

     	foo | bar	matched_foo_or_bar();

     even though you might think you could because yacc supports this syntax.
     The reason for this unfortunately incompatibility is historical, but it's
     unlikely to be changed.

     Your problems with start condition scope are simply due to syntax errors
     from your use of '|' later confusing flex.

     Let me know if you still have problems.

     		Vern

unnamed-faq-49
==============

     To: Gregory Margo <gmargo@newton.vip.best.com>
     Subject: Re: flex-2.5.3 bug report
     In-reply-to: Your message of Sat, 23 Nov 1996 16:50:09 PST.
     Date: Sat, 23 Nov 1996 17:07:32 PST
     From: Vern Paxson <vern>

     > Enclosed is a lex file that "real" lex will process, but I cannot get
     > flex to process it.  Could you try it and maybe point me in the right direction?

     Your problem is that some of the definitions in the scanner use the '/'
     trailing context operator, and have it enclosed in ()'s.  Flex does not
     allow this operator to be enclosed in ()'s because doing so allows undefined
     regular expressions such as "(a/b)+".  So the solution is to remove the
     parentheses.  Note that you must also be building the scanner with the -l
     option for AT&T lex compatibility.  Without this option, flex automatically
     encloses the definitions in parentheses.

     		Vern

unnamed-faq-50
==============

     To: Thomas Hadig <hadig@toots.physik.rwth-aachen.de>
     Subject: Re: Flex Bug ?
     In-reply-to: Your message of Tue, 26 Nov 1996 14:35:01 PST.
     Date: Tue, 26 Nov 1996 11:15:05 PST
     From: Vern Paxson <vern>

     > In my lexer code, i have the line :
     > ^\*.*          { }
     >
     > Thus all lines starting with an astrix (*) are comment lines.
     > This does not work !

     I can't get this problem to reproduce - it works fine for me.  Note
     though that if what you have is slightly different:

     	COMMENT	^\*.*
     	%%
     	{COMMENT}	{ }

     then it won't work, because flex pushes back macro definitions enclosed
     in ()'s, so the rule becomes

     	(^\*.*)		{ }

     and now that the '^' operator is not at the immediate beginning of the
     line, it's interpreted as just a regular character.  You can avoid this
     behavior by using the "-l" lex-compatibility flag, or "%option lex-compat".

     		Vern

unnamed-faq-51
==============

     To: Adoram Rogel <adoram@hybridge.com>
     Subject: Re: Flex 2.5.4 BOF ???
     In-reply-to: Your message of Tue, 26 Nov 1996 16:10:41 PST.
     Date: Wed, 27 Nov 1996 10:56:25 PST
     From: Vern Paxson <vern>

     >     Organization(s)?/[a-z]
     >
     > This matched "Organizations" (looking in debug mode, the trailing s
     > was matched with trailing context instead of the optional (s) in the
     > end of the word.

     That should only happen with lex.  Flex can properly match this pattern.
     (That might be what you're saying, I'm just not sure.)

     > Is there a way to avoid this dangerous trailing context problem ?

     Unfortunately, there's no easy way.  On the other hand, I don't see why
     it should be a problem.  Lex's matching is clearly wrong, and I'd hope
     that usually the intent remains the same as expressed with the pattern,
     so flex's matching will be correct.

     		Vern

unnamed-faq-52
==============

     To: Cameron MacKinnon <mackin@interlog.com>
     Subject: Re: Flex documentation bug
     In-reply-to: Your message of Mon, 02 Dec 1996 00:07:08 PST.
     Date: Sun, 01 Dec 1996 22:29:39 PST
     From: Vern Paxson <vern>

     > I'm not sure how or where to submit bug reports (documentation or
     > otherwise) for the GNU project stuff ...

     Well, strictly speaking flex isn't part of the GNU project.  They just
     distribute it because no one's written a decent GPL'd lex replacement.
     So you should send bugs directly to me.  Those sent to the GNU folks
     sometimes find there way to me, but some may drop between the cracks.

     > In GNU Info, under the section 'Start Conditions', and also in the man
     > page (mine's dated April '95) is a nice little snippet showing how to
     > parse C quoted strings into a buffer, defined to be MAX_STR_CONST in
     > size. Unfortunately, no overflow checking is ever done ...

     This is already mentioned in the manual:

          Finally, here's an example of how to  match  C-style  quoted
          strings using exclusive start conditions, including expanded
          escape sequences (but not including checking  for  a  string
          that's too long):

     The reason for not doing the overflow checking is that it will needlessly
     clutter up an example whose main purpose is just to demonstrate how to
     use flex.

     The latest release is 2.5.4, by the way, available from ftp.ee.lbl.gov.

     		Vern

unnamed-faq-53
==============

     To: tsv@cs.UManitoba.CA
     Subject: Re: Flex (reg)..
     In-reply-to: Your message of Thu, 06 Mar 1997 23:50:16 PST.
     Date: Thu, 06 Mar 1997 15:54:19 PST
     From: Vern Paxson <vern>

     > [:alpha:] ([:alnum:] | \\_)*

     If your rule really has embedded blanks as shown above, then it won't
     work, as the first blank delimits the rule from the action.  (It wouldn't
     even compile ...)  You need instead:

     [:alpha:]([:alnum:]|\\_)*

     and that should work fine - there's no restriction on what can go inside
     of ()'s except for the trailing context operator, '/'.

     		Vern

unnamed-faq-54
==============

     To: "Mike Stolnicki" <mstolnic@ford.com>
     Subject: Re: FLEX help
     In-reply-to: Your message of Fri, 30 May 1997 13:33:27 PDT.
     Date: Fri, 30 May 1997 10:46:35 PDT
     From: Vern Paxson <vern>

     > We'd like to add "if-then-else", "while", and "for" statements to our
     > language ...
     > We've investigated many possible solutions.  The one solution that seems
     > the most reasonable involves knowing the position of a TOKEN in yyin.

     I strongly advise you to instead build a parse tree (abstract syntax tree)
     and loop over that instead.  You'll find this has major benefits in keeping
     your interpreter simple and extensible.

     That said, the functionality you mention for get_position and set_position
     have been on the to-do list for a while.  As flex is a purely spare-time
     project for me, no guarantees when this will be added (in particular, it
     for sure won't be for many months to come).

     		Vern

unnamed-faq-55
==============

     To: Colin Paul Adams <colin@colina.demon.co.uk>
     Subject: Re: Flex C++ classes and Bison
     In-reply-to: Your message of 09 Aug 1997 17:11:41 PDT.
     Date: Fri, 15 Aug 1997 10:48:19 PDT
     From: Vern Paxson <vern>

     > #define YY_DECL   int yylex (YYSTYPE *lvalp, struct parser_control
     > *parm)
     >
     > I have been trying  to get this to work as a C++ scanner, but it does
     > not appear to be possible (warning that it matches no declarations in
     > yyFlexLexer, or something like that).
     >
     > Is this supposed to be possible, or is it being worked on (I DID
     > notice the comment that scanner classes are still experimental, so I'm
     > not too hopeful)?

     What you need to do is derive a subclass from yyFlexLexer that provides
     the above yylex() method, squirrels away lvalp and parm into member
     variables, and then invokes yyFlexLexer::yylex() to do the regular scanning.

     		Vern

unnamed-faq-56
==============

     To: Mikael.Latvala@lmf.ericsson.se
     Subject: Re: Possible mistake in Flex v2.5 document
     In-reply-to: Your message of Fri, 05 Sep 1997 16:07:24 PDT.
     Date: Fri, 05 Sep 1997 10:01:54 PDT
     From: Vern Paxson <vern>

     > In that example you show how to count comment lines when using
     > C style /* ... */ comments. My question is, shouldn't you take into
     > account a scenario where end of a comment marker occurs inside
     > character or string literals?

     The scanner certainly needs to also scan character and string literals.
     However it does that (there's an example in the man page for strings), the
     lexer will recognize the beginning of the literal before it runs across the
     embedded "/*".  Consequently, it will finish scanning the literal before it
     even considers the possibility of matching "/*".

     Example:

     	'([^']*|{ESCAPE_SEQUENCE})'

     will match all the text between the ''s (inclusive).  So the lexer
     considers this as a token beginning at the first ', and doesn't even
     attempt to match other tokens inside it.

     I thinnk this subtlety is not worth putting in the manual, as I suspect
     it would confuse more people than it would enlighten.

     		Vern

unnamed-faq-57
==============

     To: "Marty Leisner" <leisner@sdsp.mc.xerox.com>
     Subject: Re: flex limitations
     In-reply-to: Your message of Sat, 06 Sep 1997 11:27:21 PDT.
     Date: Mon, 08 Sep 1997 11:38:08 PDT
     From: Vern Paxson <vern>

     > %%
     > [a-zA-Z]+       /* skip a line */
     >                 {  printf("got %s\n", yytext); }
     > %%

     What version of flex are you using?  If I feed this to 2.5.4, it complains:

     	"bug.l", line 5: EOF encountered inside an action
     	"bug.l", line 5: unrecognized rule
     	"bug.l", line 5: fatal parse error

     Not the world's greatest error message, but it manages to flag the problem.

     (With the introduction of start condition scopes, flex can't accommodate
     an action on a separate line, since it's ambiguous with an indented rule.)

     You can get 2.5.4 from ftp.ee.lbl.gov.

     		Vern

unnamed-faq-58
==============

     To: uocarroll@deagostini.co.uk (Ultan O'Carroll)
     Subject: Re: Flex repositries
     In-reply-to: Your message of Fri, 12 Sep 1997 15:02:28 PDT.
     Date: Fri, 12 Sep 1997 10:31:50 PDT
     From: Vern Paxson <vern>

     >      before I start beavering away I wonder if you know of any
     >      place/libraries for flex
     >      desciption files that might already do this or give me a head start ?

     Unfortunately, no, I don't.  You might try asking on comp.compilers.

     		Vern

unnamed-faq-59
==============

     To: Adoram Rogel <adoram@hybridge.com>
     Subject: Re: Conditional compiling in the definitions section
     In-reply-to: Your message of Thu, 25 Sep 1997 11:22:42 PDT.
     Date: Thu, 25 Sep 1997 10:56:31 PDT
     From: Vern Paxson <vern>

     > I'm trying to combine two large lex files that now differ only in
     > about 10 lines in the definitions section.
     > I would like to have something like this:
     > #ifdef FFF
     > it	\<IT\>
     > #else
     > it	\<I\>
     > #endif
     >
     > Now, I can't add states for these, as I have already too many states
     > and the program is very complicated, and I won't be able to handle
     > 10 or 20 more states.
     >
     > Any trick to do this ?

     You might try using m4, or the C preprocessor plus a sed script to
     clean up the result (strip out the #line's).

     		Vern

unnamed-faq-60
==============

     To: Steve Antoch <SteveAn@visio.com>
     Subject: Re: lex and yacc grammars
     In-reply-to: Your message of Mon, 17 Nov 1997 15:31:25 PST.
     Date: Mon, 17 Nov 1997 15:27:01 PST
     From: Vern Paxson <vern>

     > Would you happen to know where I can find grammars for lex and yacc?

     The flex sources have a grammar for (f)lex.  Dunno about yacc,

     		Vern

unnamed-faq-61
==============

     To: Bryan Housel <bryan@drawcomp.com>
     Subject: Re: Question about Flex v2.5
     In-reply-to: Your message of Tue, 11 Nov 1997 21:30:23 PST.
     Date: Mon, 17 Nov 1997 17:12:21 PST
     From: Vern Paxson <vern>

     > It prints one of those "end of buffer.." messages for each character in the
     > token...

     This will happen if your LexerInput() function returns only one character
     at a time, which can happen either if you're scanner is "interactive", or
     if the streams library on your platform always returns 1 for yyin->gcount().

     Solution: override LexerInput() with a version that returns whole buffers.

     		Vern

unnamed-faq-62
==============

     To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE
     Subject: Re: Flex maximums
     In-reply-to: Your message of Mon, 17 Nov 1997 17:16:06 PST.
     Date: Mon, 17 Nov 1997 17:16:15 PST
     From: Vern Paxson <vern>

     > I took a quick look into the flex-sources and altered some #defines in
     > flexdefs.h:
     >
     > 	#define INITIAL_MNS 64000
     > 	#define MNS_INCREMENT 1024000
     > 	#define MAXIMUM_MNS 64000

     The things to fix are to add a couple of zeroes to:

             #define JAMSTATE -32766 /* marks a reference to the state that always jams */
             #define MAXIMUM_MNS 31999
             #define BAD_SUBSCRIPT -32767
             #define MAX_SHORT 32700

     and, if you get complaints about too many rules, make the following change too:

     	#define YY_TRAILING_MASK 0x200000
     	#define YY_TRAILING_HEAD_MASK 0x400000

     - Vern

unnamed-faq-63
==============

     To: jimmey@lexis-nexis.com (Jimmey Todd)
     Subject: Re: FLEX question regarding istream vs ifstream
     In-reply-to: Your message of Mon, 08 Dec 1997 15:54:15 PST.
     Date: Mon, 15 Dec 1997 13:21:35 PST
     From: Vern Paxson <vern>

     >         stdin_handle = YY_CURRENT_BUFFER;
     >         ifstream fin( "aFile" );
     >         yy_switch_to_buffer( yy_create_buffer( fin, YY_BUF_SIZE ) );
     >
     > What I'm wanting to do, is pass the contents of a file thru one set
     > of rules and then pass stdin thru another set... It works great if, I
     > don't use the C++ classes. But since everything else that I'm doing is
     > in C++, I thought I'd be consistent.
     >
     > The problem is that 'yy_create_buffer' is expecting an istream* as it's
     > first argument (as stated in the man page). However, fin is a ifstream
     > object. Any ideas on what I might be doing wrong? Any help would be
     > appreciated. Thanks!!

     You need to pass &fin, to turn it into an ifstream* instead of an ifstream.
     Then its type will be compatible with the expected istream*, because ifstream
     is derived from istream.

     		Vern

unnamed-faq-64
==============

     To: Enda Fadian <fadiane@piercom.ie>
     Subject: Re: Question related to Flex man page?
     In-reply-to: Your message of Tue, 16 Dec 1997 15:17:34 PST.
     Date: Tue, 16 Dec 1997 14:17:09 PST
     From: Vern Paxson <vern>

     > Can you explain to me what is ment by a long-jump in relation to flex?

     Using the longjmp() function while inside yylex() or a routine called by it.

     > what is the flex activation frame.

     Just yylex()'s stack frame.

     > As far as I can see yyrestart will bring me back to the sart of the input
     > file and using flex++ isnot really an option!

     No, yyrestart() doesn't imply a rewind, even though its name might sound
     like it does.  It tells the scanner to flush its internal buffers and
     start reading from the given file at its present location.

     		Vern

unnamed-faq-65
==============

     To: hassan@larc.info.uqam.ca (Hassan Alaoui)
     Subject: Re: Need urgent Help
     In-reply-to: Your message of Sat, 20 Dec 1997 19:38:19 PST.
     Date: Sun, 21 Dec 1997 21:30:46 PST
     From: Vern Paxson <vern>

     > /usr/lib/yaccpar: In function `int yyparse()':
     > /usr/lib/yaccpar:184: warning: implicit declaration of function `int yylex(...)'
     >
     > ld: Undefined symbol
     >    _yylex
     >    _yyparse
     >    _yyin

     This is a known problem with Solaris C++ (and/or Solaris yacc).  I believe
     the fix is to explicitly insert some 'extern "C"' statements for the
     corresponding routines/symbols.

     		Vern

unnamed-faq-66
==============

     To: mc0307@mclink.it
     Cc: gnu@prep.ai.mit.edu
     Subject: Re: [mc0307@mclink.it: Help request]
     In-reply-to: Your message of Fri, 12 Dec 1997 17:57:29 PST.
     Date: Sun, 21 Dec 1997 22:33:37 PST
     From: Vern Paxson <vern>

     > This is my definition for float and integer types:
     > . . .
     > NZD          [1-9]
     > ...
     > I've tested my program on other lex version (on UNIX Sun Solaris an HP
     > UNIX) and it work well, so I think that my definitions are correct.
     > There are any differences between Lex and Flex?

     There are indeed differences, as discussed in the man page.  The one
     you are probably running into is that when flex expands a name definition,
     it puts parentheses around the expansion, while lex does not.  There's
     an example in the man page of how this can lead to different matching.
     Flex's behavior complies with the POSIX standard (or at least with the
     last POSIX draft I saw).

     		Vern

unnamed-faq-67
==============

     To: hassan@larc.info.uqam.ca (Hassan Alaoui)
     Subject: Re: Thanks
     In-reply-to: Your message of Mon, 22 Dec 1997 16:06:35 PST.
     Date: Mon, 22 Dec 1997 14:35:05 PST
     From: Vern Paxson <vern>

     > Thank you very much for your help. I compile and link well with C++ while
     > declaring 'yylex ...' extern, But a little problem remains. I get a
     > segmentation default when executing ( I linked with lfl library) while it
     > works well when using LEX instead of flex. Do you have some ideas about the
     > reason for this ?

     The one possible reason for this that comes to mind is if you've defined
     yytext as "extern char yytext[]" (which is what lex uses) instead of
     "extern char *yytext" (which is what flex uses).  If it's not that, then
     I'm afraid I don't know what the problem might be.

     		Vern

unnamed-faq-68
==============

     To: "Bart Niswonger" <NISWONGR@almaden.ibm.com>
     Subject: Re: flex 2.5: c++ scanners & start conditions
     In-reply-to: Your message of Tue, 06 Jan 1998 10:34:21 PST.
     Date: Tue, 06 Jan 1998 19:19:30 PST
     From: Vern Paxson <vern>

     > The problem is that when I do this (using %option c++) start
     > conditions seem to not apply.

     The BEGIN macro modifies the yy_start variable.  For C scanners, this
     is a static with scope visible through the whole file.  For C++ scanners,
     it's a member variable, so it only has visible scope within a member
     function.  Your lexbegin() routine is not a member function when you
     build a C++ scanner, so it's not modifying the correct yy_start.  The
     diagnostic that indicates this is that you found you needed to add
     a declaration of yy_start in order to get your scanner to compile when
     using C++; instead, the correct fix is to make lexbegin() a member
     function (by deriving from yyFlexLexer).

     		Vern

unnamed-faq-69
==============

     To: "Boris Zinin" <boris@ippe.rssi.ru>
     Subject: Re: current position in flex buffer
     In-reply-to: Your message of Mon, 12 Jan 1998 18:58:23 PST.
     Date: Mon, 12 Jan 1998 12:03:15 PST
     From: Vern Paxson <vern>

     > The problem is how to determine the current position in flex active
     > buffer when a rule is matched....

     You will need to keep track of this explicitly, such as by redefining
     YY_USER_ACTION to count the number of characters matched.

     The latest flex release, by the way, is 2.5.4, available from ftp.ee.lbl.gov.

     		Vern

unnamed-faq-70
==============

     To: Bik.Dhaliwal@bis.org
     Subject: Re: Flex question
     In-reply-to: Your message of Mon, 26 Jan 1998 13:05:35 PST.
     Date: Tue, 27 Jan 1998 22:41:52 PST
     From: Vern Paxson <vern>

     > That requirement involves knowing
     > the character position at which a particular token was matched
     > in the lexer.

     The way you have to do this is by explicitly keeping track of where
     you are in the file, by counting the number of characters scanned
     for each token (available in yyleng).  It may prove convenient to
     do this by redefining YY_USER_ACTION, as described in the manual.

     		Vern

unnamed-faq-71
==============

     To: Vladimir Alexiev <vladimir@cs.ualberta.ca>
     Subject: Re: flex: how to control start condition from parser?
     In-reply-to: Your message of Mon, 26 Jan 1998 05:50:16 PST.
     Date: Tue, 27 Jan 1998 22:45:37 PST
     From: Vern Paxson <vern>

     > It seems useful for the parser to be able to tell the lexer about such
     > context dependencies, because then they don't have to be limited to
     > local or sequential context.

     One way to do this is to have the parser call a stub routine that's
     included in the scanner's .l file, and consequently that has access ot
     BEGIN.  The only ugliness is that the parser can't pass in the state
     it wants, because those aren't visible - but if you don't have many
     such states, then using a different set of names doesn't seem like
     to much of a burden.

     While generating a .h file like you suggests is certainly cleaner,
     flex development has come to a virtual stand-still :-(, so a workaround
     like the above is much more pragmatic than waiting for a new feature.

     		Vern

unnamed-faq-72
==============

     To: Barbara Denny <denny@3com.com>
     Subject: Re: freebsd flex bug?
     In-reply-to: Your message of Fri, 30 Jan 1998 12:00:43 PST.
     Date: Fri, 30 Jan 1998 12:42:32 PST
     From: Vern Paxson <vern>

     > lex.yy.c:1996: parse error before `='

     This is the key, identifying this error.  (It may help to pinpoint
     it by using flex -L, so it doesn't generate #line directives in its
     output.)  I will bet you heavy money that you have a start condition
     name that is also a variable name, or something like that; flex spits
     out #define's for each start condition name, mapping them to a number,
     so you can wind up with:

     	%x foo
     	%%
     		...
     	%%
     	void bar()
     		{
     		int foo = 3;
     		}

     and the penultimate will turn into "int 1 = 3" after C preprocessing,
     since flex will put "#define foo 1" in the generated scanner.

     		Vern

unnamed-faq-73
==============

     To: Maurice Petrie <mpetrie@infoscigroup.com>
     Subject: Re: Lost flex .l file
     In-reply-to: Your message of Mon, 02 Feb 1998 14:10:01 PST.
     Date: Mon, 02 Feb 1998 11:15:12 PST
     From: Vern Paxson <vern>

     > I am curious as to
     > whether there is a simple way to backtrack from the generated source to
     > reproduce the lost list of tokens we are searching on.

     In theory, it's straight-forward to go from the DFA representation
     back to a regular-expression representation - the two are isomorphic.
     In practice, a huge headache, because you have to unpack all the tables
     back into a single DFA representation, and then write a program to munch
     on that and translate it into an RE.

     Sorry for the less-than-happy news ...

     		Vern

unnamed-faq-74
==============

     To: jimmey@lexis-nexis.com (Jimmey Todd)
     Subject: Re: Flex performance question
     In-reply-to: Your message of Thu, 19 Feb 1998 11:01:17 PST.
     Date: Thu, 19 Feb 1998 08:48:51 PST
     From: Vern Paxson <vern>

     > What I have found, is that the smaller the data chunk, the faster the
     > program executes. This is the opposite of what I expected. Should this be
     > happening this way?

     This is exactly what will happen if your input file has embedded NULs.
     From the man page:

          A final note: flex is slow when matching NUL's, particularly
          when  a  token  contains multiple NUL's.  It's best to write
          rules which match short amounts of text if it's  anticipated
          that the text will often include NUL's.

     So that's the first thing to look for.

     		Vern

unnamed-faq-75
==============

     To: jimmey@lexis-nexis.com (Jimmey Todd)
     Subject: Re: Flex performance question
     In-reply-to: Your message of Thu, 19 Feb 1998 11:01:17 PST.
     Date: Thu, 19 Feb 1998 15:42:25 PST
     From: Vern Paxson <vern>

     So there are several problems.

     First, to go fast, you want to match as much text as possible, which
     your scanners don't in the case that what they're scanning is *not*
     a <RN> tag.  So you want a rule like:

     	[^<]+

     Second, C++ scanners are particularly slow if they're interactive,
     which they are by default.  Using -B speeds it up by a factor of 3-4
     on my workstation.

     Third, C++ scanners that use the istream interface are slow, because
     of how poorly implemented istream's are.  I built two versions of
     the following scanner:

     	%%
     	.*\n
     	.*
     	%%

     and the C version inhales a 2.5MB file on my workstation in 0.8 seconds.
     The C++ istream version, using -B, takes 3.8 seconds.

     		Vern

unnamed-faq-76
==============

     To: "Frescatore, David (CRD, TAD)" <frescatore@exc01crdge.crd.ge.com>
     Subject: Re: FLEX 2.5 & THE YEAR 2000
     In-reply-to: Your message of Wed, 03 Jun 1998 11:26:22 PDT.
     Date: Wed, 03 Jun 1998 10:22:26 PDT
     From: Vern Paxson <vern>

     > I am researching the Y2K problem with General Electric R&D
     > and need to know if there are any known issues concerning
     > the above mentioned software and Y2K regardless of version.

     There shouldn't be, all it ever does with the date is ask the system
     for it and then print it out.

     		Vern

unnamed-faq-77
==============

     To: "Hans Dermot Doran" <htd@ibhdoran.com>
     Subject: Re: flex problem
     In-reply-to: Your message of Wed, 15 Jul 1998 21:30:13 PDT.
     Date: Tue, 21 Jul 1998 14:23:34 PDT
     From: Vern Paxson <vern>

     > To overcome this, I gets() the stdin into a string and lex the string. The
     > string is lexed OK except that the end of string isn't lexed properly
     > (yy_scan_string()), that is the lexer dosn't recognise the end of string.

     Flex doesn't contain mechanisms for recognizing buffer endpoints.  But if
     you use fgets instead (which you should anyway, to protect against buffer
     overflows), then the final \n will be preserved in the string, and you can
     scan that in order to find the end of the string.

     		Vern

unnamed-faq-78
==============

     To: soumen@almaden.ibm.com
     Subject: Re: Flex++ 2.5.3 instance member vs. static member
     In-reply-to: Your message of Mon, 27 Jul 1998 02:10:04 PDT.
     Date: Tue, 28 Jul 1998 01:10:34 PDT
     From: Vern Paxson <vern>

     > %{
     > int mylineno = 0;
     > %}
     > ws      [ \t]+
     > alpha   [A-Za-z]
     > dig     [0-9]
     > %%
     >
     > Now you'd expect mylineno to be a member of each instance of class
     > yyFlexLexer, but is this the case?  A look at the lex.yy.cc file seems to
     > indicate otherwise; unless I am missing something the declaration of
     > mylineno seems to be outside any class scope.
     >
     > How will this work if I want to run a multi-threaded application with each
     > thread creating a FlexLexer instance?

     Derive your own subclass and make mylineno a member variable of it.

     		Vern

unnamed-faq-79
==============

     To: Adoram Rogel <adoram@hybridge.com>
     Subject: Re: More than 32K states change hangs
     In-reply-to: Your message of Tue, 04 Aug 1998 16:55:39 PDT.
     Date: Tue, 04 Aug 1998 22:28:45 PDT
     From: Vern Paxson <vern>

     > Vern Paxson,
     >
     > I followed your advice, posted on Usenet bu you, and emailed to me
     > personally by you, on how to overcome the 32K states limit. I'm running
     > on Linux machines.
     > I took the full source of version 2.5.4 and did the following changes in
     > flexdef.h:
     > #define JAMSTATE -327660
     > #define MAXIMUM_MNS 319990
     > #define BAD_SUBSCRIPT -327670
     > #define MAX_SHORT 327000
     >
     > and compiled.
     > All looked fine, including check and bigcheck, so I installed.

     Hmmm, you shouldn't increase MAX_SHORT, though looking through my email
     archives I see that I did indeed recommend doing so.  Try setting it back
     to 32700; that should suffice that you no longer need -Ca.  If it still
     hangs, then the interesting question is - where?

     > Compiling the same hanged program with a out-of-the-box (RedHat 4.2
     > distribution of Linux)
     > flex 2.5.4 binary works.

     Since Linux comes with source code, you should diff it against what
     you have to see what problems they missed.

     > Should I always compile with the -Ca option now ? even short and simple
     > filters ?

     No, definitely not.  It's meant to be for those situations where you
     absolutely must squeeze every last cycle out of your scanner.

     		Vern

unnamed-faq-80
==============

     To: "Schmackpfeffer, Craig" <Craig.Schmackpfeffer@usa.xerox.com>
     Subject: Re: flex output for static code portion
     In-reply-to: Your message of Tue, 11 Aug 1998 11:55:30 PDT.
     Date: Mon, 17 Aug 1998 23:57:42 PDT
     From: Vern Paxson <vern>

     > I would like to use flex under the hood to generate a binary file
     > containing the data structures that control the parse.

     This has been on the wish-list for a long time.  In principle it's
     straight-forward - you redirect mkdata() et al's I/O to another file,
     and modify the skeleton to have a start-up function that slurps these
     into dynamic arrays.  The concerns are (1) the scanner generation code
     is hairy and full of corner cases, so it's easy to get surprised when
     going down this path :-( ; and (2) being careful about buffering so
     that when the tables change you make sure the scanner starts in the
     correct state and reading at the right point in the input file.

     > I was wondering if you know of anyone who has used flex in this way.

     I don't - but it seems like a reasonable project to undertake (unlike
     numerous other flex tweaks :-).

     		Vern

unnamed-faq-81
==============

     Received: from 131.173.17.11 (131.173.17.11 [131.173.17.11])
     	by ee.lbl.gov (8.9.1/8.9.1) with ESMTP id AAA03838
     	for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 00:47:57 -0700 (PDT)
     Received: from hal.cl-ki.uni-osnabrueck.de (hal.cl-ki.Uni-Osnabrueck.DE [131.173.141.2])
     	by deimos.rz.uni-osnabrueck.de (8.8.7/8.8.8) with ESMTP id JAA34694
     	for <vern@ee.lbl.gov>; Thu, 20 Aug 1998 09:47:55 +0200
     Received: (from georg@localhost) by hal.cl-ki.uni-osnabrueck.de (8.6.12/8.6.12) id JAA34834 for vern@ee.lbl.gov; Thu, 20 Aug 1998 09:47:54 +0200
     From: Georg Rehm <georg@hal.cl-ki.uni-osnabrueck.de>
     Message-Id: <199808200747.JAA34834@hal.cl-ki.uni-osnabrueck.de>
     Subject: "flex scanner push-back overflow"
     To: vern@ee.lbl.gov
     Date: Thu, 20 Aug 1998 09:47:54 +0200 (MEST)
     Reply-To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE
     X-NoJunk: Do NOT send commercial mail, spam or ads to this address!
     X-URL: http://www.cl-ki.uni-osnabrueck.de/~georg/
     X-Mailer: ELM [version 2.4ME+ PL28 (25)]
     MIME-Version: 1.0
     Content-Type: text/plain; charset=US-ASCII
     Content-Transfer-Encoding: 7bit

     Hi Vern,

     Yesterday, I encountered a strange problem: I use the macro processor m4
     to include some lengthy lists into a .l file. Following is a flex macro
     definition that causes some serious pain in my neck:

     AUTHOR           ("A. Boucard / L. Boucard"|"A. Dastarac / M. Levent"|"A.Boucaud / L.Boucaud"|"Abderrahim Lamchichi"|"Achmat Dangor"|"Adeline Toullier"|"Adewale Maja-Pearce"|"Ahmed Ziri"|"Akram Ellyas"|"Alain Bihr"|"Alain Gresh"|"Alain Guillemoles"|"Alain Joxe"|"Alain Morice"|"Alain Renon"|"Alain Zecchini"|"Albert Memmi"|"Alberto Manguel"|"Alex De Waal"|"Alfonso Artico"| [...])

     The complete list contains about 10kB. When I try to "flex" this file
     (on a Solaris 2.6 machine, using a modified flex 2.5.4 (I only increased
     some of the predefined values in flexdefs.h) I get the error:

     myflex/flex -8  sentag.tmp.l
     flex scanner push-back overflow

     When I remove the slashes in the macro definition everything works fine.
     As I understand it, the double quotes escape the slash-character so it
     really means "/" and not "trailing context". Furthermore, I tried to
     escape the slashes with backslashes, but with no use, the same error message
     appeared when flexing the code.

     Do you have an idea what's going on here?

     Greetings from Germany,
     	Georg
     --
     Georg Rehm                                     georg@cl-ki.uni-osnabrueck.de
     Institute for Semantic Information Processing, University of Osnabrueck, FRG

unnamed-faq-82
==============

     To: Georg.Rehm@CL-KI.Uni-Osnabrueck.DE
     Subject: Re: "flex scanner push-back overflow"
     In-reply-to: Your message of Thu, 20 Aug 1998 09:47:54 PDT.
     Date: Thu, 20 Aug 1998 07:05:35 PDT
     From: Vern Paxson <vern>

     > myflex/flex -8  sentag.tmp.l
     > flex scanner push-back overflow

     Flex itself uses a flex scanner.  That scanner is running out of buffer
     space when it tries to unput() the humongous macro you've defined.  When
     you remove the '/'s, you make it small enough so that it fits in the buffer;
     removing spaces would do the same thing.

     The fix is to either rethink how come you're using such a big macro and
     perhaps there's another/better way to do it; or to rebuild flex's own
     scan.c with a larger value for

     	#define YY_BUF_SIZE 16384

     - Vern

unnamed-faq-83
==============

     To: Jan Kort <jan@research.techforce.nl>
     Subject: Re: Flex
     In-reply-to: Your message of Fri, 04 Sep 1998 12:18:43 +0200.
     Date: Sat, 05 Sep 1998 00:59:49 PDT
     From: Vern Paxson <vern>

     > %%
     >
     > "TEST1\n"       { fprintf(stderr, "TEST1\n"); yyless(5); }
     > ^\n             { fprintf(stderr, "empty line\n"); }
     > .               { }
     > \n              { fprintf(stderr, "new line\n"); }
     >
     > %%
     > -- input ---------------------------------------
     > TEST1
     > -- output --------------------------------------
     > TEST1
     > empty line
     > ------------------------------------------------

     IMHO, it's not clear whether or not this is in fact a bug.  It depends
     on whether you view yyless() as backing up in the input stream, or as
     pushing new characters onto the beginning of the input stream.  Flex
     interprets it as the latter (for implementation convenience, I'll admit),
     and so considers the newline as in fact matching at the beginning of a
     line, as after all the last token scanned an entire line and so the
     scanner is now at the beginning of a new line.

     I agree that this is counter-intuitive for yyless(), given its
     functional description (it's less so for unput(), depending on whether
     you're unput()'ing new text or scanned text).  But I don't plan to
     change it any time soon, as it's a pain to do so.  Consequently,
     you do indeed need to use yy_set_bol() and YY_AT_BOL() to tweak
     your scanner into the behavior you desire.

     Sorry for the less-than-completely-satisfactory answer.

     		Vern

unnamed-faq-84
==============

     To: Patrick Krusenotto <krusenot@mac-info-link.de>
     Subject: Re: Problems with restarting flex-2.5.2-generated scanner
     In-reply-to: Your message of Thu, 24 Sep 1998 10:14:07 PDT.
     Date: Thu, 24 Sep 1998 23:28:43 PDT
     From: Vern Paxson <vern>

     > I am using flex-2.5.2 and bison 1.25 for Solaris and I am desperately
     > trying to make my scanner restart with a new file after my parser stops
     > with a parse error. When my compiler restarts, the parser always
     > receives the token after the token (in the old file!) that caused the
     > parser error.

     I suspect the problem is that your parser has read ahead in order
     to attempt to resolve an ambiguity, and when it's restarted it picks
     up with that token rather than reading a fresh one.  If you're using
     yacc, then the special "error" production can sometimes be used to
     consume tokens in an attempt to get the parser into a consistent state.

     		Vern

unnamed-faq-85
==============

     To: Henric Jungheim <junghelh@pe-nelson.com>
     Subject: Re: flex 2.5.4a
     In-reply-to: Your message of Tue, 27 Oct 1998 16:41:42 PST.
     Date: Tue, 27 Oct 1998 16:50:14 PST
     From: Vern Paxson <vern>

     > This brings up a feature request:  How about a command line
     > option to specify the filename when reading from stdin?  That way one
     > doesn't need to create a temporary file in order to get the "#line"
     > directives to make sense.

     Use -o combined with -t (per the man page description of -o).

     > P.S., Is there any simple way to use non-blocking IO to parse multiple
     > streams?

     Simple, no.

     One approach might be to return a magic character on EWOULDBLOCK and
     have a rule

     	.*<magic-character>	// put back .*, eat magic character

     This is off the top of my head, not sure it'll work.

     		Vern

unnamed-faq-86
==============

     To: "Repko, Billy D" <billy.d.repko@intel.com>
     Subject: Re: Compiling scanners
     In-reply-to: Your message of Wed, 13 Jan 1999 10:52:47 PST.
     Date: Thu, 14 Jan 1999 00:25:30 PST
     From: Vern Paxson <vern>

     > It appears that maybe it cannot find the lfl library.

     The Makefile in the distribution builds it, so you should have it.
     It's exceedingly trivial, just a main() that calls yylex() and
     a yyrap() that always returns 1.

     > %%
     >       \n      ++num_lines; ++num_chars;
     >       .       ++num_chars;

     You can't indent your rules like this - that's where the errors are coming
     from.  Flex copies indented text to the output file, it's how you do things
     like

     	int num_lines_seen = 0;

     to declare local variables.

     		Vern

unnamed-faq-87
==============

     To: Erick Branderhorst <Erick.Branderhorst@asml.nl>
     Subject: Re: flex input buffer
     In-reply-to: Your message of Tue, 09 Feb 1999 13:53:46 PST.
     Date: Tue, 09 Feb 1999 21:03:37 PST
     From: Vern Paxson <vern>

     > In the flex.skl file the size of the default input buffers is set.  Can you
     > explain why this size is set and why it is such a high number.

     It's large to optimize performance when scanning large files.  You can
     safely make it a lot lower if needed.

     		Vern

unnamed-faq-88
==============

     To: "Guido Minnen" <guidomi@cogs.susx.ac.uk>
     Subject: Re: Flex error message
     In-reply-to: Your message of Wed, 24 Feb 1999 15:31:46 PST.
     Date: Thu, 25 Feb 1999 00:11:31 PST
     From: Vern Paxson <vern>

     > I'm extending a larger scanner written in Flex and I keep running into
     > problems. More specifically, I get the error message:
     > "flex: input rules are too complicated (>= 32000 NFA states)"

     Increase the definitions in flexdef.h for:

             #define JAMSTATE -32766 /* marks a reference to the state that always j
     ams */
             #define MAXIMUM_MNS 31999
             #define BAD_SUBSCRIPT -32767

     recompile everything, and it should all work.

     		Vern

unnamed-faq-89
==============

     To: John Victor J <vjohn@its.soft.net>
     Subject: Re: flex---is thread safe
     In-reply-to: Your message of Sun, 23 May 1999 12:56:56 +0530.
     Date: Sun, 23 May 1999 00:32:53 PDT
     From: Vern Paxson <vern>

     >      I would like to know whether flex is thread safe???

     I take it you mean the scanners it generates and not flex itself.

     The answer is (still) No, except if you use the -+ option to generate
     a C++ scanning class (and if your stream library is thread-safe).

     		Vern

unnamed-faq-90
==============

     To: "Dmitriy Goldobin" <gold@ems.chel.su>
     Subject: Re: FLEX trouble
     In-reply-to: Your message of Mon, 31 May 1999 18:44:49 PDT.
     Date: Tue, 01 Jun 1999 00:15:07 PDT
     From: Vern Paxson <vern>

     >   I have a trouble with FLEX. Why rule "/*".*"*/" work properly,=20
     > but rule "/*"(.|\n)*"*/" don't work ?

     The second of these will have to scan the entire input stream (because
     "(.|\n)*" matches an arbitrary amount of any text) in order to see if
     it ends with "*/", terminating the comment.  That potentially will overflow
     the input buffer.

     >   More complex rule "/*"([^*]|(\*/[^/]))*"*/ give an error
     > 'unrecognized rule'.

     You can't use the '/' operator inside parentheses.  It's not clear
     what "(a/b)*" actually means.

     >   I now use workaround with state <comment>, but single-rule is
     > better, i think.

     Single-rule is nice but will always have the problem of either setting
     restrictions on comments (like not allowing multi-line comments) and/or
     running the risk of consuming the entire input stream, as noted above.

     		Vern

unnamed-faq-91
==============

     Received: from mc-qout4.whowhere.com (mc-qout4.whowhere.com [209.185.123.18])
     	by ee.lbl.gov (8.9.3/8.9.3) with SMTP id IAA05100
     	for <vern@ee.lbl.gov>; Tue, 15 Jun 1999 08:56:06 -0700 (PDT)
     Received: from Unknown/Local ([?.?.?.?]) by my-deja.com; Tue Jun 15 08:55:43 1999
     To: vern@ee.lbl.gov
     Date: Tue, 15 Jun 1999 08:55:43 -0700
     From: "Aki Niimura" <neko@my-deja.com>
     Message-ID: <KNONDOHDOBGAEAAA@my-deja.com>
     Mime-Version: 1.0
     Cc:
     X-Sent-Mail: on
     Reply-To:
     X-Mailer: MailCity Service
     Subject: A question on flex C++ scanner
     X-Sender-Ip: 12.72.207.61
     Organization: My Deja Email  (http://www.my-deja.com:80)
     Content-Type: text/plain; charset=us-ascii
     Content-Transfer-Encoding: 7bit

     Dear Dr. Paxon,

     I have been using flex for years.
     It works very well on many projects.
     Most case, I used it to generate a scanner on C language.
     However, one project I needed to generate  a scanner
     on C++ lanuage. Thanks to your enhancement, flex did
     the job.

     Currently, I'm working on enhancing my previous project.
     I need to deal with multiple input streams (recursive
     inclusion) in this scanner (C++).
     I did similar thing for another scanner (C) as you
     explained in your documentation.

     The generated scanner (C++) has necessary methods:
     - switch_to_buffer(struct yy_buffer_state *b)
     - yy_create_buffer(istream *is, int sz)
     - yy_delete_buffer(struct yy_buffer_state *b)

     However, I couldn't figure out how to access current
     buffer (yy_current_buffer).

     yy_current_buffer is a protected member of yyFlexLexer.
     I can't access it directly.
     Then, I thought yy_create_buffer() with is = 0 might
     return current stream buffer. But it seems not as far
     as I checked the source. (flex 2.5.4)

     I went through the Web in addition to Flex documentation.
     However, it hasn't been successful, so far.

     It is not my intention to bother you, but, can you
     comment about how to obtain the current stream buffer?

     Your response would be highly appreciated.

     Best regards,
     Aki Niimura

     --== Sent via Deja.com http://www.deja.com/ ==--
     Share what you know. Learn what you don't.

unnamed-faq-92
==============

     To: neko@my-deja.com
     Subject: Re: A question on flex C++ scanner
     In-reply-to: Your message of Tue, 15 Jun 1999 08:55:43 PDT.
     Date: Tue, 15 Jun 1999 09:04:24 PDT
     From: Vern Paxson <vern>

     > However, I couldn't figure out how to access current
     > buffer (yy_current_buffer).

     Derive your own subclass from yyFlexLexer.

     		Vern

unnamed-faq-93
==============

     To: "Stones, Darren" <Darren.Stones@nectech.co.uk>
     Subject: Re: You're the man to see?
     In-reply-to: Your message of Wed, 23 Jun 1999 11:10:29 PDT.
     Date: Wed, 23 Jun 1999 09:01:40 PDT
     From: Vern Paxson <vern>

     > I hope you can help me.  I am using Flex and Bison to produce an interpreted
     > language.  However all goes well until I try to implement an IF statement or
     > a WHILE.  I cannot get this to work as the parser parses all the conditions
     > eg. the TRUE and FALSE conditons to check for a rule match.  So I cannot
     > make a decision!!

     You need to use the parser to build a parse tree (= abstract syntax trwee),
     and when that's all done you recursively evaluate the tree, binding variables
     to values at that time.

     		Vern

unnamed-faq-94
==============

     To: Petr Danecek <petr@ics.cas.cz>
     Subject: Re: flex - question
     In-reply-to: Your message of Mon, 28 Jun 1999 19:21:41 PDT.
     Date: Fri, 02 Jul 1999 16:52:13 PDT
     From: Vern Paxson <vern>

     > file, it takes an enormous amount of time. It is funny, because the
     > source code has only 12 rules!!! I think it looks like an exponencial
     > growth.

     Right, that's the problem - some patterns (those with a lot of
     ambiguity, where yours has because at any given time the scanner can
     be in the middle of all sorts of combinations of the different
     rules) blow up exponentially.

     For your rules, there is an easy fix.  Change the ".*" that comes fater
     the directory name to "[^ ]*".  With that in place, the rules are no
     longer nearly so ambiguous, because then once one of the directories
     has been matched, no other can be matched (since they all require a
     leading blank).

     If that's not an acceptable solution, then you can enter a start state
     to pick up the .*\n after each directory is matched.

     Also note that for speed, you'll want to add a ".*" rule at the end,
     otherwise rules that don't match any of the patterns will be matched
     very slowly, a character at a time.

     		Vern

unnamed-faq-95
==============

     To: Tielman Koekemoer <tielman@spi.co.za>
     Subject: Re: Please help.
     In-reply-to: Your message of Thu, 08 Jul 1999 13:20:37 PDT.
     Date: Thu, 08 Jul 1999 08:20:39 PDT
     From: Vern Paxson <vern>

     > I was hoping you could help me with my problem.
     >
     > I tried compiling (gnu)flex on a Solaris 2.4 machine
     > but when I ran make (after configure) I got an error.
     >
     > --------------------------------------------------------------
     > gcc -c -I. -I. -g -O parse.c
     > ./flex -t -p  ./scan.l >scan.c
     > sh: ./flex: not found
     > *** Error code 1
     > make: Fatal error: Command failed for target `scan.c'
     > -------------------------------------------------------------
     >
     > What's strange to me is that I'm only
     > trying to install flex now. I then edited the Makefile to
     > and changed where it says "FLEX = flex" to "FLEX = lex"
     > ( lex: the native Solaris one ) but then it complains about
     > the "-p" option. Is there any way I can compile flex without
     > using flex or lex?
     >
     > Thanks so much for your time.

     You managed to step on the bootstrap sequence, which first copies
     initscan.c to scan.c in order to build flex.  Try fetching a fresh
     distribution from ftp.ee.lbl.gov.  (Or you can first try removing
     ".bootstrap" and doing a make again.)

     		Vern

unnamed-faq-96
==============

     To: Tielman Koekemoer <tielman@spi.co.za>
     Subject: Re: Please help.
     In-reply-to: Your message of Fri, 09 Jul 1999 09:16:14 PDT.
     Date: Fri, 09 Jul 1999 00:27:20 PDT
     From: Vern Paxson <vern>

     > First I removed .bootstrap (and ran make) - no luck. I downloaded the
     > software but I still have the same problem. Is there anything else I
     > could try.

     Try:

     	cp initscan.c scan.c
     	touch scan.c
     	make scan.o

     If this last tries to first build scan.c from scan.l using ./flex, then
     your "make" is broken, in which case compile scan.c to scan.o by hand.

     		Vern

unnamed-faq-97
==============

     To: Sumanth Kamenani <skamenan@crl.nmsu.edu>
     Subject: Re: Error
     In-reply-to: Your message of Mon, 19 Jul 1999 23:08:41 PDT.
     Date: Tue, 20 Jul 1999 00:18:26 PDT
     From: Vern Paxson <vern>

     > I am getting a compilation error. The error is given as "unknown symbol- yylex".

     The parser relies on calling yylex(), but you're instead using the C++ scanning
     class, so you need to supply a yylex() "glue" function that calls an instance
     scanner of the scanner (e.g., "scanner->yylex()").

     		Vern

unnamed-faq-98
==============

     To: daniel@synchrods.synchrods.COM (Daniel Senderowicz)
     Subject: Re: lex
     In-reply-to: Your message of Mon, 22 Nov 1999 11:19:04 PST.
     Date: Tue, 23 Nov 1999 15:54:30 PST
     From: Vern Paxson <vern>

     Well, your problem is the

             switch (yybgin-yysvec-1) {      /* witchcraft */

     at the beginning of lex rules.  "witchcraft" == "non-portable".  It's
     assuming knowledge of the AT&T lex's internal variables.

     For flex, you can probably do the equivalent using a switch on YYSTATE.

     		Vern

unnamed-faq-99
==============

     To: archow@hss.hns.com
     Subject: Re: Regarding distribution of flex and yacc based grammars
     In-reply-to: Your message of Sun, 19 Dec 1999 17:50:24 +0530.
     Date: Wed, 22 Dec 1999 01:56:24 PST
     From: Vern Paxson <vern>

     > When we provide the customer with an object code distribution, is it
     > necessary for us to provide source
     > for the generated C files from flex and bison since they are generated by
     > flex and bison ?

     For flex, no.  I don't know what the current state of this is for bison.

     > Also, is there any requrirement for us to neccessarily  provide source for
     > the grammar files which are fed into flex and bison ?

     Again, for flex, no.

     See the file "COPYING" in the flex distribution for the legalese.

     		Vern

unnamed-faq-100
===============

     To: Martin Gallwey <gallweym@hyperion.moe.ul.ie>
     Subject: Re: Flex, and self referencing rules
     In-reply-to: Your message of Sun, 20 Feb 2000 01:01:21 PST.
     Date: Sat, 19 Feb 2000 18:33:16 PST
     From: Vern Paxson <vern>

     > However, I do not use unput anywhere. I do use self-referencing
     > rules like this:
     >
     > UnaryExpr               ({UnionExpr})|("-"{UnaryExpr})

     You can't do this - flex is *not* a parser like yacc (which does indeed
     allow recursion), it is a scanner that's confined to regular expressions.

     		Vern

unnamed-faq-101
===============

     To: slg3@lehigh.edu (SAMUEL L. GULDEN)
     Subject: Re: Flex problem
     In-reply-to: Your message of Thu, 02 Mar 2000 12:29:04 PST.
     Date: Thu, 02 Mar 2000 23:00:46 PST
     From: Vern Paxson <vern>

     If this is exactly your program:

     > digit [0-9]
     > digits {digit}+
     > whitespace [ \t\n]+
     >
     > %%
     > "[" { printf("open_brac\n");}
     > "]" { printf("close_brac\n");}
     > "+" { printf("addop\n");}
     > "*" { printf("multop\n");}
     > {digits} { printf("NUMBER = %s\n", yytext);}
     > whitespace ;

     then the problem is that the last rule needs to be "{whitespace}" !

     		Vern