As many of you already know, I have been working on a replacement for Mesa's current GLSL compiler infrastructure since July. I presented my plans for this work at XDS in September (http://web.cecs.pdx.edu/~idr/publications/xds2008-GLSL_work.pdf). Today I am making the work generally available for review and for patches. The compiler is not "done" by any stretch of the imagination. However, I have been working on it in isolation for much too long, and it's high time to get some review and oversight from the community.
The front-end parser is complete. It has been implemented using flex and bison. Several features specific to those tools have been used, so the compiler does actually require them. Flex and bison implement some extensions for generating re-entrant lexers and parsers. It is probably possible to port to other lex and yacc implementations, but I'm not terribly interested in do it. Patches welcome. The mid-level IR (MIR) has been defined, and numerous transformations have been implemented. The program is simplified to 3-address statements. In each statement operands must be "simple." That is, each operands must be a constant, an variable reference, or an array reference with either a constant index or a variable reference index. Function calls and control-flow statements are also handled. In MIR, high-level constructs like loops and if-statements are directly implemented. Goto-statements are implemented, but they are only used to replace return-statements when a function is copied in-line. The fairly recent addition to the design is the low-level IR (LIR). This is a tree representation of the MIR. The most common way for a compiler to generate machine code is to use dynamic programming perform a least-cost rewrite of a tree. The conversion for MIR to LIR introduces target-architecture dependencies. High-level constructs like loops and if-statements are replaced with lower-level, architecture dependent constructs. The particular dynamic programming system that is being used is lburg. This is the code-generator generator used by LCC (L-burg C Compiler). I had to make a couple small patches to lburg. The code generated by lburg accidentally introduced some LCC dependencies. At some point lburg will need to be replaced. It's a reasonable tool, but it's not a perfect fit. It carries LCC's weird license, so I'm not sure that just modifying it to suit Mesa's needs is a viable option. It's only 942 lines of code, so re-writing it will not be an arduous task. A portion of that work has already been done. There is a substantial list of things that remain to be done. Some of these need to be done before importing the stand-alone compiler into the Mesa tree. Some of it can wait until after this integration. The biggest thing that needs doing, and my next work item, is to finish the machine description for the code-generator. Right now only a small portion of the source LIR is supported. Among other things, if-statements are not yet supported. In addition, loop constructs are not supported from the front-end. Right now the parser just ignores all of the loop constructs. This was intentional. I wanted to see how if-statements would be implemented, top to bottom, before deciding exactly how to implement loops. Many of the implementation details of structures and arrays have been similarly delayed. After the machine description is complete, a proper register allocator will be needed. In its current incarnation, the compiler generates temporaries without bounds. It assumes that there are enough registers to go around. Clearly, this is not a safe assumption. Mesa's current compiler also operates this way. A proper register allocator will help things a lot. The conversion from MIR to LIR treats each MIR statement entirely independently. The LIR is a tree representation. Input values form the leaves, and connections of outputs to inputs form the branches. However, none of these connections are made. As a result, the generated code is not very good. In fact, it can't even generate a dot-product instruction. Performing additional passes over the LIR to make these connections should be straightforward. I honestly think this can wait. To me, it is much more important to get all of the parts complete and generating correct code than it is to generate good code for a subset. Right now the compiler only has a skeletal linker. The infrastructure is mostly in place to combine multiple shaders of the same type (i.e., link multiple vertex shader compilation units together). It just needs to be finished. The linker also needs to perform cross-shader validation. That is, it needs to make sure that vertex shader outputs match fragment shader inputs. It needs to make sure that uniforms are consistently declared across shaders. Etc. Once shaders are properly linked, uniforms need to be placed. The last major missing piece is the GLSL preprocessor. When I was talking to Michal Krol at XDS he pointed out that Nvidia released an open-source GLSL preprocessor. I haven't investigated it very deeply, but I expect we could just use that for the time being. The only copy of the code I could find was in the sample front-end that 3Dlabs released back in 2005 (http://mew.cx/glsl/OpenGLCompilerSept202005.zip). In addition to those major elements, there are a number of minor elements that need to either be written or re-written. As far as I can tell, none of these items require any knowledge of Mesa. Most of them don't even require any compiler expertise. The symbol table currently used is utter rubbish. There are two main problems with it. The biggest problem is that it allocates elements in a large block. When more elements are needed, it attempts to realloc the block. The symbol table routines return pointers to the elements, so having the elements change memory locations is just plain borken. I'm really not sure what I was thinking when I wrote. The other problem is that there is no search acceleration method implemented. It needs a hash table or something! Error logging is also a bit sketchy. Ideally we'd want to be able to generate warnings and errors from various stages of the compiler in arbitrary order. When the application queries the shader log, all of the entries would be sorted by source file and line number. Pretty headers like "In function foo:" would be added at appropriate places. I have some ideas for how this could be implemented. Running valgrind shows pretty quickly that memory management hasn't been a high prioritie thus far. It leaks like a sieve. Some types of objects are fairly short lived. Some types of objects are long lived and are referenced from multiple structures. The short lived objects need to be identified. Some of these objects should be tracked with a pool-based suballocator. When the objects are no longer needed, the entire pool should be destroyed or flushed. The remaining short lived objects should be destroyed as needed. The longer lived objects need to be reference counted and destroyed appropriately. The existing Mesa compiler tokenizes the shading language built-ins during the build process. The tokenized form is compiled at run-time. On of the design goals of the new compiler was to compile the shading language built-ins to the MIR representation during the build process. This requires some sort of out-of-core representation of the MIR. The compiler needs to be able to save and load MIR data. An appropriate representation should be difficult to develop and implement, but it has not be done yet. As an added bonus, such a representation would provide an easy implementation of GL_OES_get_program_binary. Software links: - ftp://ftp.cs.princeton.edu/pub/packages/lcc/lcc-4.2.tar.gz - LCC 4.2. This contains lburg. - http://people.freedesktop.org/~idr/patches/0001-Use-OP_LABEL-and-STATE_LABEL-accessor-macros.patch lburg patch that fixes dependency on generated code to LCC. - My source code repository: git clone http://people.freedesktop.org/~idr/glsl.git glsl ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Mesa3d-dev mailing list Mesa3d-dev@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mesa3d-dev
