This package contains the implementation of Racket's front end: macro expander, reader, and module systems. A copy of this implementation is extracted and built into the Racket executable, so normally this package's modules are not run directly. The expander or reader can be run separately, however, and the Racket expander is updated by modifying this package as it exists in the main Racket Git repository. Running: % racket demo.rkt or % racket bootstrap-demo.rkt Runs the examples/tests in "demo.rkt". The tests are not remotely complete, but they're a quick and useful sanity check. The "demo.rkt" module uses the somewhat internal interface exported by `main`, where the expansion, compilation, and evaluation are less overloaded and more controllable. Use the "bootstrap-demo.rkt" when running in an older version of Racket that is not built with this expander (but that version of Racket must be new enough to provide a primitive '#%linklet module as a bootstrapping hook). % racket run.rkt -c or % racket bootstrap-run.rkt -c Runs the expander to load itself from source. Expanded and compiled modules are stored in , somewhat like bytecode files. Dependency tracking doesn't take into account the expander itself, so throw away if the expander changes in a way that you want reflected in compilation results. % racket run.rkt -c -l % racket run.rkt -c -t Runs the expander to load the specified module (instead of the default module, which is the expander itself). When running with a new enough version of Racket that "run.rkt" works (as opposed to "bootstrap-run.rkt"), the performance of the expander in this mode should be close to the performance when the expander is built into the Racket executable. Beware, however, that "run.rkt" implements just enough of the module loader protocol to work as a bridge, so module loading and caching can have very different performance than in an embedding build. Beware also that the flags above cause bytecode for the target module to be cached, so running a second time will not test the expander a second time. Prime the cache directory with modules that don't change, and then use `-r` to load a module with a read-only cache. % racket run.rkt -c -f Loads the given file as a sequence of top-level forms. % racket run.rkt -c -e -l Expands the given file, instead of compiling and running it. % racket bootstrap-run.rkt -s -c --linklets -l Compiles the given file to a set of linklets in S-expression form, instead of compiling and running it. % racket bootstrap-run.rkt -s -c -x Checks possibility of converting a module to a stand-alone linklet with no imports --- used mainly to extract the expander itself. % racket bootstrap-run.rkt -c -sx -t -o Expands and extracts as a single linklet to . Unless the `--local-rename` flag is also provided to "bootstrap-run.rkt", an extracted linklet preserves a naming property of the expander's compilation to linklets, which is that it uses a distinct symbol for every binding. The symbol--binding correspondence is useful for some further compiler passes, but `--local-rename` is useful to minimize syntactic diffs. % racket bootstrap-run.rkt -c -sx -D -t -o Expands and extracts as a single linklet, compiles and decompiles it, then writes the s-expression into . % racket bootstrap-run.rkt -c -sx -B -t -o Expands and extracts as a single linklet, compiles it and then writes the bytecode into . % zuo . demo % zuo . run ARGS=" ..." More shortcuts. Use `zuo . run ...` as a shorthand for `racket run.rkt -c compiled/cache ...`. See "main.zuo" for more information and other shortcuts, and see also "Running zuo" below. ---------------------------------------- Building Racket to use this expander: After you change the expander, you must perform an explicit build step to use it when making Racket CS or Racket BC. Normally, for that case, the expander would be modified as part of a Racket repo checkout. If you're working with a Racket repo checkout, and if you have a working `racket` in your `PATH`, use `make derived` in the checkout's top-level directory to propagate changes for both BC and CS. Then, you can build CS or BC as usual. Otherwise: * For Racket CS, go to "racket/src/cs" in the repo and run `zuo`. (See also "Running zuo" below.) That will update files in "racket/src/cs/schemified", including using the new expander to rebuild the Racket-implemented parts of Racket CS that are in "../thread", "../io", etc. After this step, a `make cs` in the top level of the Racket repo will build Racket CS using the new expander. * For Racket BC, run `zuo` here, which will update the file "racket/src/bc/src/startup.inc". Then, when you build Racket BC, "startup.inc" will be automatically compiled to bytecode for embedding into the Racket BC executable. After this step, a `make bc` in the top level of the Racket repo will build Racket BC using the new expander. For either of these steps, "main.zuo" will assume that Racket is already built in the surrounding checkout, so Racket can be run as `../../bin/racket`. Use `zuo . RACKET=....` to use a different Racket build, but beware that the version of Racket must be essentially the same version as the target build. Before building new CS and BC executables, increment the target Racket version in "../version/racket_version.h" if you change the serialization of syntax objects or the linklet protocol. Updating "racket_version.h" is important both for bytecode files and the build step that generates the bytecode version of the expander itself. The `zuo` step for Racket BC generates source files in "compiled/cache-src". In some cases (hopefully rare), you may have to manually discard "compiled/cache-src" when things change. ---------------------------------------- Running zuo: If you don't have `zuo` installed, it may already be built as "../build/bin/zuo", or you can build it from the sources in "../zuo". ---------------------------------------- Roadmap to the implementation: read/ - the readers demo.rkt - simple examples/tests for the reader syntax/ - syntax-object and binding representation syntax.rkt - syntax-object structure scope.rkt - scope sets and binding binding.rkt - binding representations binding-table.rkt - managing sets of bindings namespace/ - namespaces and module instances expand/ - expander loop and core forms common/ - utilities module-path.rkt - [resolved] module path [indexes] compile/ - from expanded to S-expression linklet main.rkt - compiler functions called from "eval/main.rkt" eval/ - evaluation main.rkt - top-level evaluation, with top-level `module` forms as an important special case; the `compile-to-linklets` function compiles to a set of S-expression linklets api.rkt - wrappers that implement `eval`, `compile`, and `expand` for `racket/base` boot/ - internal initialization handler.rkt - implements the default module name resolver, eval handler, and compiler handler ...-primitive.rkt - export built-in functions as modules run/ - helpers to drive the expander; not part of the resulting expander's implementation linklet.rkt - a bootstrapping implementation of `linklet` by compilation into `lambda` plus primitives extract/ - extracts a module and its dependencies to a single linklet, especially for extracting the compiler itself (via "run.rkt"); not part of the resulting expander's implementation main.rkt - installs eval handler, etc.; entry point for directly running the expander/compiler/evaluator, and the provided variables of this module become the entry points for the embedded expander demo.rkt - exercises the expander and compiler (uses "main.rkt") run.rkt - starts a Racket replacement (uses "main.rkt") bootstrap-run.rkt - like "run.rkt", but for a host Racket that does not include linklet support bootstrap-demo.rkt - like "demo.rkt", but for a host Racket that does not include linklet support Beware that names are routinely shadowed when they are provided by `racket/base` but replaced by the expander's implementation. For example, `syntax?` is shadowed, and any part of the expander that needs `syntax?` must import "syntax/syntax.rkt" or "syntax/checked-syntax.rkt". ---------------------------------------- Performance measurements: Set the `PLT_EXPANDER_TIMES` environment variable for a summary of performance information (written via `log-error`, which normally goes to stderr) on exit. In the output, a category that is nested under another category contributes to the latter's recorded time and memory use, but not to its counts. Beware that taking measurements can slow down the expander slightly. Set the `PLT_LINKLET_TIMES` environment variable to get similar information from the underlying runtime system. Except for compile times, linklet-level times generally will be disjoint from the times reported by the expander. ---------------------------------------- About structures: Constructors, predicates, accessors, etc., for a non-transparent structure type are defined in the implementation of the expander are compiled/loaded in a way that makes them claim not to be structure procedures when built into Racket. For example, (struct-predicate-procedure? syntax?) ; => #f If a structure type's representation is exported, such as `exn:fail:syntax`, then operations do claim to be structure operations (struct-predicate-procedure? exn:fail:syntax?) ; => #t As a consequence, it's ok to directly expose predicates, accessors, etc., without exposing an implementation detail. ---------------------------------------- Implementation guidelines: * Do not rely on more than `racket/base` for code that will be extracted as the compiler implementation. (Relying on more in "run/" or "extract/" is allowed.) * The runtime implementation of the expander must not itself use any syntax objects or syntax function as provided by the Racket implementation used to compile the expander. That means, for example, that the contract system cannot be used in the implementation of the expander, since the contract system manages some information with syntax objects at run time. The expander-extraction process double-checks that the expander is independent of its host in this way. * The runtime implementation of the expander can refer (via `#%kernel`) to reader primitives that are to be implemented by the reader that is bundled with the expander. The extraction process simply redirects those references to the implemented variants. Beware that adjusting parameters from `#%kernel` will not change the behavior of the bundled reader during bootrstapping of the expander (i.e., for bootstrapping, always refer to the parameters from the implementation in the "read" directory). ---------------------------------------- Some naming conventions: s or stx - a syntax object sc - a scope scs - a set or list of scopes id - an identifier (obviously) b - a binding; sometimes spelled out as `binding` m - a result of syntax matching m - a module ns - a namespace ctx - an expansion context (including the expand-time environment) cctx - a compilation context (including a compile-time environment) insp - an inspector mpi - a module path index mod-name - a resolved module path, usually; sometimes used for other forms of module reference c and ec - character and "effective" character (after readtable mapping) in the reader - - like , but specifically one for ; for example, `m-ns` is a namespace for some module