This package contains the implementation of Racket's front end: macro
expander, reader, and module systems. A copy of this implementation is
extracted and built into the Racket executable, so normally this
package's modules are not run directly. The expander or reader can be
run separately, however, and the Racket expander is updated by
modifying this package as it exists in the main Racket Git repository.
Running:
% racket demo.rkt
or
% racket bootstrap-demo.rkt
Runs the examples/tests in "demo.rkt". The tests are not remotely
complete, but they're a quick and useful sanity check. The
"demo.rkt" module uses the somewhat internal interface exported by
`main`, where the expansion, compilation, and evaluation are less
overloaded and more controllable.
Use the "bootstrap-demo.rkt" when running in an older version of
Racket that is not built with this expander (but that version of
Racket must be new enough to provide a primitive '#%linklet module
as a bootstrapping hook).
% racket run.rkt -c
or
% racket bootstrap-run.rkt -c
Runs the expander to load itself from source. Expanded and compiled
modules are stored in , somewhat like bytecode files.
Dependency tracking doesn't take into account the expander itself,
so throw away if the expander changes in a way that you want
reflected in compilation results.
% racket run.rkt -c -l
% racket run.rkt -c -t
Runs the expander to load the specified module (instead of the
default module, which is the expander itself).
When running with a new enough version of Racket that "run.rkt"
works (as opposed to "bootstrap-run.rkt"), the performance of the
expander in this mode should be close to the performance when the
expander is built into the Racket executable. Beware, however, that
"run.rkt" implements just enough of the module loader protocol to
work as a bridge, so module loading and caching can have very
different performance than in an embedding build.
Beware also that the flags above cause bytecode for the target
module to be cached, so running a second time will not test the
expander a second time. Prime the cache directory with modules that
don't change, and then use `-r` to load a module with a read-only
cache.
% racket run.rkt -c -f
Loads the given file as a sequence of top-level forms.
% racket run.rkt -c -e -l
Expands the given file, instead of compiling and running it.
% racket bootstrap-run.rkt -s -c --linklets -l
Compiles the given file to a set of linklets in S-expression form,
instead of compiling and running it.
% racket bootstrap-run.rkt -s -c -x
Checks possibility of converting a module to a stand-alone linklet
with no imports --- used mainly to extract the expander itself.
% racket bootstrap-run.rkt -c -sx -t -o
Expands and extracts as a single linklet to
.
Unless the `--local-rename` flag is also provided to
"bootstrap-run.rkt", an extracted linklet preserves a naming
property of the expander's compilation to linklets, which is that
it uses a distinct symbol for every binding. The symbol--binding
correspondence is useful for some further compiler passes, but
`--local-rename` is useful to minimize syntactic diffs.
% racket bootstrap-run.rkt -c -sx -D -t -o
Expands and extracts as a single linklet, compiles and
decompiles it, then writes the s-expression into .
% racket bootstrap-run.rkt -c -sx -B -t -o
Expands and extracts as a single linklet, compiles it
and then writes the bytecode into .
% zuo . demo
% zuo . run ARGS=" ..."
More shortcuts. Use `zuo . run ...` as a shorthand for `racket
run.rkt -c compiled/cache ...`.
See "main.zuo" for more information and other shortcuts, and see
also "Running zuo" below.
----------------------------------------
Building Racket to use this expander:
After you change the expander, you must perform an explicit build step
to use it when making Racket CS or Racket BC. Normally, for that case,
the expander would be modified as part of a Racket repo checkout.
If you're working with a Racket repo checkout, and if you have a
working `racket` in your `PATH`, use `make derived` in the checkout's
top-level directory to propagate changes for both BC and CS. Then, you
can build CS or BC as usual.
Otherwise:
* For Racket CS, go to "racket/src/cs" in the repo and run `zuo`.
(See also "Running zuo" below.) That will update files in
"racket/src/cs/schemified", including using the new expander to
rebuild the Racket-implemented parts of Racket CS that are in
"../thread", "../io", etc.
After this step, a `make cs` in the top level of the Racket repo
will build Racket CS using the new expander.
* For Racket BC, run `zuo` here, which will update the file
"racket/src/bc/src/startup.inc". Then, when you build Racket BC,
"startup.inc" will be automatically compiled to bytecode for
embedding into the Racket BC executable.
After this step, a `make bc` in the top level of the Racket repo
will build Racket BC using the new expander.
For either of these steps, "main.zuo" will assume that Racket is
already built in the surrounding checkout, so Racket can be run as
`../../bin/racket`. Use `zuo . RACKET=....` to use a different Racket
build, but beware that the version of Racket must be essentially the
same version as the target build.
Before building new CS and BC executables, increment the target Racket
version in "../version/racket_version.h" if you change the
serialization of syntax objects or the linklet protocol. Updating
"racket_version.h" is important both for bytecode files and the
build step that generates the bytecode version of the expander itself.
The `zuo` step for Racket BC generates source files in
"compiled/cache-src". In some cases (hopefully rare), you may have to
manually discard "compiled/cache-src" when things change.
----------------------------------------
Running zuo:
If you don't have `zuo` installed, it may already be built as
"../build/bin/zuo", or you can build it from the sources in "../zuo".
----------------------------------------
Roadmap to the implementation:
read/ - the readers
demo.rkt - simple examples/tests for the reader
syntax/ - syntax-object and binding representation
syntax.rkt - syntax-object structure
scope.rkt - scope sets and binding
binding.rkt - binding representations
binding-table.rkt - managing sets of bindings
namespace/ - namespaces and module instances
expand/ - expander loop and core forms
common/ - utilities
module-path.rkt - [resolved] module path [indexes]
compile/ - from expanded to S-expression linklet
main.rkt - compiler functions called from "eval/main.rkt"
eval/ - evaluation
main.rkt - top-level evaluation, with top-level `module` forms as
an important special case; the `compile-to-linklets`
function compiles to a set of S-expression linklets
api.rkt - wrappers that implement `eval`, `compile`, and `expand`
for `racket/base`
boot/ - internal initialization
handler.rkt - implements the default module name resolver, eval
handler, and compiler handler
...-primitive.rkt - export built-in functions as modules
run/ - helpers to drive the expander; not part of the resulting
expander's implementation
linklet.rkt - a bootstrapping implementation of `linklet` by
compilation into `lambda` plus primitives
extract/ - extracts a module and its dependencies to a single
linklet, especially for extracting the compiler itself
(via "run.rkt"); not part of the resulting expander's
implementation
main.rkt - installs eval handler, etc.; entry point for directly
running the expander/compiler/evaluator, and the provided
variables of this module become the entry points for the
embedded expander
demo.rkt - exercises the expander and compiler (uses "main.rkt")
run.rkt - starts a Racket replacement (uses "main.rkt")
bootstrap-run.rkt - like "run.rkt", but for a host Racket that
does not include linklet support
bootstrap-demo.rkt - like "demo.rkt", but for a host Racket that
does not include linklet support
Beware that names are routinely shadowed when they are provided by
`racket/base` but replaced by the expander's implementation. For
example, `syntax?` is shadowed, and any part of the expander that
needs `syntax?` must import "syntax/syntax.rkt" or
"syntax/checked-syntax.rkt".
----------------------------------------
Performance measurements:
Set the `PLT_EXPANDER_TIMES` environment variable for a summary of
performance information (written via `log-error`, which normally goes
to stderr) on exit. In the output, a category that is nested under
another category contributes to the latter's recorded time and memory
use, but not to its counts. Beware that taking measurements can slow
down the expander slightly.
Set the `PLT_LINKLET_TIMES` environment variable to get similar
information from the underlying runtime system. Except for compile
times, linklet-level times generally will be disjoint from the times
reported by the expander.
----------------------------------------
About structures:
Constructors, predicates, accessors, etc., for a non-transparent
structure type are defined in the implementation of the expander are
compiled/loaded in a way that makes them claim not to be structure
procedures when built into Racket. For example,
(struct-predicate-procedure? syntax?) ; => #f
If a structure type's representation is exported, such as
`exn:fail:syntax`, then operations do claim to be structure operations
(struct-predicate-procedure? exn:fail:syntax?) ; => #t
As a consequence, it's ok to directly expose predicates, accessors,
etc., without exposing an implementation detail.
----------------------------------------
Implementation guidelines:
* Do not rely on more than `racket/base` for code that will be
extracted as the compiler implementation. (Relying on more in
"run/" or "extract/" is allowed.)
* The runtime implementation of the expander must not itself use any
syntax objects or syntax function as provided by the Racket
implementation used to compile the expander. That means, for
example, that the contract system cannot be used in the
implementation of the expander, since the contract system manages
some information with syntax objects at run time. The
expander-extraction process double-checks that the expander is
independent of its host in this way.
* The runtime implementation of the expander can refer (via
`#%kernel`) to reader primitives that are to be implemented by the
reader that is bundled with the expander. The extraction process
simply redirects those references to the implemented variants.
Beware that adjusting parameters from `#%kernel` will not change
the behavior of the bundled reader during bootrstapping of the
expander (i.e., for bootstrapping, always refer to the parameters
from the implementation in the "read" directory).
----------------------------------------
Some naming conventions:
s or stx - a syntax object
sc - a scope
scs - a set or list of scopes
id - an identifier (obviously)
b - a binding; sometimes spelled out as `binding`
m - a result of syntax matching
m - a module
ns - a namespace
ctx - an expansion context (including the expand-time environment)
cctx - a compilation context (including a compile-time environment)
insp - an inspector
mpi - a module path index
mod-name - a resolved module path, usually; sometimes used for other
forms of module reference
c and ec - character and "effective" character (after readtable
mapping) in the reader
- - like , but specifically one for
; for example, `m-ns` is a namespace for some module