Re: HLSL Compiler and d3dcompiler_xx.dll
Am 14.04.2010 um 20:45 schrieb Jason Green: FYI, we encountered a game a while back which used a few shaders that depended on being compiled with a particular version for d3dx9_##.dll. There was a compiler bug which the game engine knew about and accounted for. If you tried to use the compiled shader from a newer d3dx9_##.dll, then the rendering wouldn't look quite right on certain objects. So, there's one argument for identical bytecode compatibility, but it's likely that very few apps will exhibit behavior like this. I think it's only an argument that we may need some specific tests that check for identical bytecode. And by your description it sounds like the compiler bug can be detected by looking at the rendering output. I still think that having the majority of tests check the generated bytecode is a bad idea: * It will be hard to implement the same compiler result * It will make our own optimizations impossible(the MS compiler is very good optimization wise, so that point is mostly moot) * It will be hard to maintain the tests when we're moving them to a newer MS compiler version * There are things that can be translated into different bytecodes, and all are equally valid and optimal, like constants ordering.
Re: HLSL Compiler and d3dcompiler_xx.dll
On Thu, Apr 15, 2010 at 11:23 AM, Stefan Dösinger stefandoesin...@gmx.at wrote: Am 14.04.2010 um 20:45 schrieb Jason Green: FYI, we encountered a game a while back which used a few shaders that depended on being compiled with a particular version for d3dx9_##.dll. There was a compiler bug which the game engine knew about and accounted for. If you tried to use the compiled shader from a newer d3dx9_##.dll, then the rendering wouldn't look quite right on certain objects. So, there's one argument for identical bytecode compatibility, but it's likely that very few apps will exhibit behavior like this. I think it's only an argument that we may need some specific tests that check for identical bytecode. And by your description it sounds like the compiler bug can be detected by looking at the rendering output. I still think that having the majority of tests check the generated bytecode is a bad idea: * It will be hard to implement the same compiler result As long as it is possible, I think we should do. * It will make our own optimizations impossible(the MS compiler is very good optimization wise, so that point is mostly moot) * It will be hard to maintain the tests when we're moving them to a newer MS compiler version Tests should fail if a newer version of MS compiler is used and different code is produced. Or, if you prefer, win_skip if version is newer than what the tests expect. * There are things that can be translated into different bytecodes, and all are equally valid and optimal, like constants ordering. Same as your first concern.
HLSL Compiler and d3dcompiler_xx.dll
Hi, I'm about to implement the HLSL compiler, but I want to make sure I'm taking the right direction. For the compiler: preprocessor(wpp)-hlslparser/compiler(lex/bison)-bytecode writer(same as for the shader assembler). Now the question is where to implement the compiler. It can be called from at least 3 places, d3dx9_xx, d3d10_xx and d3dcompiler_xx. It seems to me that d3dx9_xx and d3d10_xx are forwarding their calls to d3dcompiler_xx. My proposal is: 1) Create the d3dcompiler_xx.dll structure in wine. Is it legal to use a program like dll export viewer to get the exported functions for each dll? 2) Move the assembler to d3dcompiler_xx.dll and forward calls from d3dx9. This is needed because the assembler and compiler can share the same bytecode writer this way. 3) Implement the compiler in d3dcompiler_xx. Regards, Matijn
Re: HLSL Compiler and d3dcompiler_xx.dll
On 14 April 2010 13:08, Matijn Woudt tijn...@gmail.com wrote: Hi, I'm about to implement the HLSL compiler, but I want to make sure I'm taking the right direction. For the compiler: preprocessor(wpp)-hlslparser/compiler(lex/bison)-bytecode writer(same as for the shader assembler). Now the question is where to implement the compiler. It can be called from at least 3 places, d3dx9_xx, d3d10_xx and d3dcompiler_xx. It seems to me that d3dx9_xx and d3d10_xx are forwarding their calls to d3dcompiler_xx. In earlier d3dx9 versions the compiler was part of d3dx9, but at some point Microsoft introduced d3dcompiler. I think it makes sense to switch to the same scheme in Wine. Note that d3dcompiler is (somewhat) documented at http://msdn.microsoft.com/en-us/library/dd607340%28v=VS.85%29.aspx
Re: HLSL Compiler and d3dcompiler_xx.dll
Am 14.04.2010 um 13:08 schrieb Matijn Woudt: 2) Move the assembler to d3dcompiler_xx.dll and forward calls from d3dx9. This is needed because the assembler and compiler can share the same bytecode writer this way. When Mattheo implemented the assembler in the gsoc project we decided to put the assembler into d3dx9 because we assumed MS implements it there as well, and there are no assembler specific exports in d3dcompiler.dll. Using wine-internal exports is generally a bad idea and should only be used if there are clear advantages. (It e.g. makes it impossible to use the native d3dcompiler.dll with builtin d3dx9) Before moving the assembler into d3dcompiler.dll we have to be sure that sharing the bytecode writer is possible and is worth the problems it introduces. 3) Implement the compiler in d3dcompiler_xx. I wrote a basic HLSL compiler as university project in 2008, this is where part of the assembler code came from. Do you have the sources, do you need them?
Re: HLSL Compiler and d3dcompiler_xx.dll
On Wed, Apr 14, 2010 at 3:07 PM, Stefan Dösinger stefandoesin...@gmx.at wrote: Am 14.04.2010 um 13:08 schrieb Matijn Woudt: 2) Move the assembler to d3dcompiler_xx.dll and forward calls from d3dx9. This is needed because the assembler and compiler can share the same bytecode writer this way. When Mattheo implemented the assembler in the gsoc project we decided to put the assembler into d3dx9 because we assumed MS implements it there as well, and there are no assembler specific exports in d3dcompiler.dll. Using wine-internal exports is generally a bad idea and should only be used if there are clear advantages. (It e.g. makes it impossible to use the native d3dcompiler.dll with builtin d3dx9) In the last two versions of d3dcompiler (41 42), there's an undocumented export D3DAssemble, which makes me assume that MS has moved the Assembler there too. We can define D3DAssemble just like D3DXAssembleShader for now, and once Microsoft documents this function (or defines in d3dcompiler.h), we can adjust the definition if needed. Before moving the assembler into d3dcompiler.dll we have to be sure that sharing the bytecode writer is possible and is worth the problems it introduces. I'm not sure why sharing the bytecode writer wouldn't be possible. If the shader assembler is moved along, there shouldn't be problems (or am I missing something?). 3) Implement the compiler in d3dcompiler_xx. I wrote a basic HLSL compiler as university project in 2008, this is where part of the assembler code came from. Do you have the sources, do you need them? Matteo told me about it, and he was going to send your code to me after he checked if it was ok with you. But if you can send them to me directly, that's fine too, of course.
Re: HLSL Compiler and d3dcompiler_xx.dll
On 14 April 2010 15:07, Stefan Dösinger stefandoesin...@gmx.at wrote: 3) Implement the compiler in d3dcompiler_xx. I wrote a basic HLSL compiler as university project in 2008, this is where part of the assembler code came from. Do you have the sources, do you need them? Quite frankly, I also believe that's where some of the issues in the early versions of those assembler patches came from. I don't know if that compiler has seen any work since 2008, but I'd be careful with taking it as too much of an example.
Re: HLSL Compiler and d3dcompiler_xx.dll
Am 14.04.2010 um 15:44 schrieb Henri Verbeet: On 14 April 2010 15:07, Stefan Dösinger stefandoesin...@gmx.at wrote: 3) Implement the compiler in d3dcompiler_xx. I wrote a basic HLSL compiler as university project in 2008, this is where part of the assembler code came from. Do you have the sources, do you need them? Quite frankly, I also believe that's where some of the issues in the early versions of those assembler patches came from. I don't know if that compiler has seen any work since 2008, but I'd be careful with taking it as too much of an example. Yep, I recommend using Mattheo's code if possible. He has rebased my old code and fixed a bunch of issues. Beyond that, my compiler is fairly minimalistic. I think it supports most features except arrays(needs loop unrolling support). However, the optimizer is fairly simplistic. For better optimizations you may want to investigate SSA transformations and related stuff. I guess perfect optimization isn't a requirement for a first version though. Just in case my git tree is here: http://84.112.174.163/~stefan/wine/ . It's not online all the time, and I am not guaranted a fixed IP(although it hasn't changed in years), so check it out as long as you can if you need it.
Re: HLSL Compiler and d3dcompiler_xx.dll
On Wed, Apr 14, 2010 at 4:27 PM, Stefan Dösinger stefandoesin...@gmx.at wrote: Am 14.04.2010 um 15:44 schrieb Henri Verbeet: On 14 April 2010 15:07, Stefan Dösinger stefandoesin...@gmx.at wrote: 3) Implement the compiler in d3dcompiler_xx. I wrote a basic HLSL compiler as university project in 2008, this is where part of the assembler code came from. Do you have the sources, do you need them? Quite frankly, I also believe that's where some of the issues in the early versions of those assembler patches came from. I don't know if that compiler has seen any work since 2008, but I'd be careful with taking it as too much of an example. Yep, I recommend using Mattheo's code if possible. He has rebased my old code and fixed a bunch of issues. Beyond that, my compiler is fairly minimalistic. I think it supports most features except arrays(needs loop unrolling support). However, the optimizer is fairly simplistic. For better optimizations you may want to investigate SSA transformations and related stuff. I guess perfect optimization isn't a requirement for a first version though. First thing would probably be cleaning up and getting something minimal past AJ. Just in case my git tree is here: http://84.112.174.163/~stefan/wine/ . It's not online all the time, and I am not guaranted a fixed IP(although it hasn't changed in years), so check it out as long as you can if you need it. Am I supposed to find your compiler code there? It seems like a git tree of ages ago (where d3dx9_36 only contains math and font code).
Re: HLSL Compiler and d3dcompiler_xx.dll
On Wed, Apr 14, 2010 at 4:38 PM, Matijn Woudt tijn...@gmail.com wrote: On Wed, Apr 14, 2010 at 4:27 PM, Stefan Dösinger stefandoesin...@gmx.at wrote: Am 14.04.2010 um 15:44 schrieb Henri Verbeet: On 14 April 2010 15:07, Stefan Dösinger stefandoesin...@gmx.at wrote: 3) Implement the compiler in d3dcompiler_xx. I wrote a basic HLSL compiler as university project in 2008, this is where part of the assembler code came from. Do you have the sources, do you need them? Quite frankly, I also believe that's where some of the issues in the early versions of those assembler patches came from. I don't know if that compiler has seen any work since 2008, but I'd be careful with taking it as too much of an example. Yep, I recommend using Mattheo's code if possible. He has rebased my old code and fixed a bunch of issues. Beyond that, my compiler is fairly minimalistic. I think it supports most features except arrays(needs loop unrolling support). However, the optimizer is fairly simplistic. For better optimizations you may want to investigate SSA transformations and related stuff. I guess perfect optimization isn't a requirement for a first version though. First thing would probably be cleaning up and getting something minimal past AJ. Just in case my git tree is here: http://84.112.174.163/~stefan/wine/ . It's not online all the time, and I am not guaranted a fixed IP(although it hasn't changed in years), so check it out as long as you can if you need it. Am I supposed to find your compiler code there? It seems like a git tree of ages ago (where d3dx9_36 only contains math and font code). It might also make sense to explore whether compilers like LLVM (used a lot these days including for OpenCL implementations by Apple, Nvidia; and don't forget Gallium3D) and Open64 (used in Cuda) and others. Roderick
Re: HLSL Compiler and d3dcompiler_xx.dll
On Wed, Apr 14, 2010 at 4:51 PM, Roderick Colenbrander thunderbir...@gmail.com wrote: On Wed, Apr 14, 2010 at 4:38 PM, Matijn Woudt tijn...@gmail.com wrote: On Wed, Apr 14, 2010 at 4:27 PM, Stefan Dösinger stefandoesin...@gmx.at wrote: Am 14.04.2010 um 15:44 schrieb Henri Verbeet: On 14 April 2010 15:07, Stefan Dösinger stefandoesin...@gmx.at wrote: 3) Implement the compiler in d3dcompiler_xx. I wrote a basic HLSL compiler as university project in 2008, this is where part of the assembler code came from. Do you have the sources, do you need them? Quite frankly, I also believe that's where some of the issues in the early versions of those assembler patches came from. I don't know if that compiler has seen any work since 2008, but I'd be careful with taking it as too much of an example. Yep, I recommend using Mattheo's code if possible. He has rebased my old code and fixed a bunch of issues. Beyond that, my compiler is fairly minimalistic. I think it supports most features except arrays(needs loop unrolling support). However, the optimizer is fairly simplistic. For better optimizations you may want to investigate SSA transformations and related stuff. I guess perfect optimization isn't a requirement for a first version though. First thing would probably be cleaning up and getting something minimal past AJ. Just in case my git tree is here: http://84.112.174.163/~stefan/wine/ . It's not online all the time, and I am not guaranted a fixed IP(although it hasn't changed in years), so check it out as long as you can if you need it. Am I supposed to find your compiler code there? It seems like a git tree of ages ago (where d3dx9_36 only contains math and font code). It might also make sense to explore whether compilers like LLVM (used a lot these days including for OpenCL implementations by Apple, Nvidia; and don't forget Gallium3D) and Open64 (used in Cuda) and others. Roderick I have thought about using LLVM, but I don't like it because of: 1) Having another wine dependency (~40MB for me on ubuntu) 2) Code generated by LLVM will most likely not generated exactly the same bytecode as native compiler does, even though the shader has same effect on both. This makes it really hard to verify the HLSL compiler is working. My idea was to make wine's hlsl compiler produce exactly the same bytecode as native does. We might even find bugs in the native compiler, and either fix or duplicate these bugs. LLVM might be useful if we needed a really fast compiler, but native compiler isn't really fast AFAIK. Once games are calling it a few hundred times it might be worth the effort.
Re: HLSL Compiler and d3dcompiler_xx.dll
On Wed, Apr 14, 2010 at 5:19 PM, Matijn Woudt tijn...@gmail.com wrote: On Wed, Apr 14, 2010 at 4:51 PM, Roderick Colenbrander thunderbir...@gmail.com wrote: On Wed, Apr 14, 2010 at 4:38 PM, Matijn Woudt tijn...@gmail.com wrote: On Wed, Apr 14, 2010 at 4:27 PM, Stefan Dösinger stefandoesin...@gmx.at wrote: Am 14.04.2010 um 15:44 schrieb Henri Verbeet: On 14 April 2010 15:07, Stefan Dösinger stefandoesin...@gmx.at wrote: 3) Implement the compiler in d3dcompiler_xx. I wrote a basic HLSL compiler as university project in 2008, this is where part of the assembler code came from. Do you have the sources, do you need them? Quite frankly, I also believe that's where some of the issues in the early versions of those assembler patches came from. I don't know if that compiler has seen any work since 2008, but I'd be careful with taking it as too much of an example. Yep, I recommend using Mattheo's code if possible. He has rebased my old code and fixed a bunch of issues. Beyond that, my compiler is fairly minimalistic. I think it supports most features except arrays(needs loop unrolling support). However, the optimizer is fairly simplistic. For better optimizations you may want to investigate SSA transformations and related stuff. I guess perfect optimization isn't a requirement for a first version though. First thing would probably be cleaning up and getting something minimal past AJ. Just in case my git tree is here: http://84.112.174.163/~stefan/wine/ . It's not online all the time, and I am not guaranted a fixed IP(although it hasn't changed in years), so check it out as long as you can if you need it. Am I supposed to find your compiler code there? It seems like a git tree of ages ago (where d3dx9_36 only contains math and font code). It might also make sense to explore whether compilers like LLVM (used a lot these days including for OpenCL implementations by Apple, Nvidia; and don't forget Gallium3D) and Open64 (used in Cuda) and others. Roderick I have thought about using LLVM, but I don't like it because of: 1) Having another wine dependency (~40MB for me on ubuntu) I agree that an extra dependency should be avoided if possible. Remember that if we use some generic framework (if it makes sense) they would do a lot of hard work for us (especially optimizations later on). Further other projects might want to help as well (there is some XNA project for mono which wants a HLSL but there are likely others as well). The 40MB is not a big deal, disk space is really cheap these days. 2) Code generated by LLVM will most likely not generated exactly the same bytecode as native compiler does, even though the shader has same effect on both. This makes it really hard to verify the HLSL compiler is working. My idea was to make wine's hlsl compiler produce exactly the same bytecode as native does. We might even find bugs in the native compiler, and either fix or duplicate these bugs. I don't know much details about the HLSL compiler but I think this is nearly impossible. I wouldn't be surprised if Microsoft has added more and more tweaks to each new revision of the HLSL compiler, so creating 100% the same code is hard. I think we need very strict d3dx9 test cases which don't necessarily check the bytecode (in simple cases you can) but which check whether the shader when fed with data works correctly. LLVM might be useful if we needed a really fast compiler, but native compiler isn't really fast AFAIK. Once games are calling it a few hundred times it might be worth the effort. My main worry is that in the end if we use a compiler framework (remember we don't have a lot of manpower for this) I think it would reduce in more efficient shaders and it prevents some reinventing of the wheel. Roderick
Re: HLSL Compiler and d3dcompiler_xx.dll
2010/4/14 Matijn Woudt tijn...@gmail.com: On Wed, Apr 14, 2010 at 4:27 PM, Stefan Dösinger stefandoesin...@gmx.at wrote: Am 14.04.2010 um 15:44 schrieb Henri Verbeet: On 14 April 2010 15:07, Stefan Dösinger stefandoesin...@gmx.at wrote: 3) Implement the compiler in d3dcompiler_xx. I wrote a basic HLSL compiler as university project in 2008, this is where part of the assembler code came from. Do you have the sources, do you need them? Quite frankly, I also believe that's where some of the issues in the early versions of those assembler patches came from. I don't know if that compiler has seen any work since 2008, but I'd be careful with taking it as too much of an example. Yep, I recommend using Mattheo's code if possible. He has rebased my old code and fixed a bunch of issues. Beyond that, my compiler is fairly minimalistic. I think it supports most features except arrays(needs loop unrolling support). However, the optimizer is fairly simplistic. For better optimizations you may want to investigate SSA transformations and related stuff. I guess perfect optimization isn't a requirement for a first version though. First thing would probably be cleaning up and getting something minimal past AJ. Just in case my git tree is here: http://84.112.174.163/~stefan/wine/ . It's not online all the time, and I am not guaranted a fixed IP(although it hasn't changed in years), so check it out as long as you can if you need it. Am I supposed to find your compiler code there? It seems like a git tree of ages ago (where d3dx9_36 only contains math and font code). Yeah, Stefan's project is based on wine 1.1.0 and you can find the relevant code into dlls/wineshader. One of the things I did was to port the assembler to current wine and move it into d3dx9_36.dll. In the process I changed quite some things all around and I stripped from the bytecode writer what wasn't needed by the assembler. I didn't update the compiler code, and as Henri said, at this point probably it's better to just continue what you are doing and look at Stefan's implementation only as a source for inspiration. As d3dcompiler_xx.dll now seems to contain also the assembler, I'm ok to move the assembler there (once completed) to share the bytecode writer with the compiler. Regarding the optimization, I'd suggest to keep it simple in the beginning. I liked the idea in Stefan's compiler to have a parser which produces an abstract intermediate representation, which is then optimized to generate the bwriter_shader. The optimizer could then be implemented from scratch, or using something third-party as LLVM or such, while as a first step the optimizer could (ideally) simply spit out the code unchanged. I believe that some language features cannot be supported without some optimizations, so this problem can crop up sooner than expected. Lastly, a bit on testing the compiler. I'm not sure trying to get exactly the same bytecode as native is a feasible objective: while for an assembler program there is only one correct translation, this is not true for a compiler. Moreover I expect each iteration of the Microsoft compiler could produce a different output with the same shader program, so there isn't a right implementation to compare with. I agree, this makes testing the compiler much harder...
Re: HLSL Compiler and d3dcompiler_xx.dll
On Wed, Apr 14, 2010 at 5:59 PM, Matteo Bruni matteo.myst...@gmail.com wrote: 2010/4/14 Matijn Woudt tijn...@gmail.com: On Wed, Apr 14, 2010 at 4:27 PM, Stefan Dösinger stefandoesin...@gmx.at wrote: Am 14.04.2010 um 15:44 schrieb Henri Verbeet: On 14 April 2010 15:07, Stefan Dösinger stefandoesin...@gmx.at wrote: 3) Implement the compiler in d3dcompiler_xx. I wrote a basic HLSL compiler as university project in 2008, this is where part of the assembler code came from. Do you have the sources, do you need them? Quite frankly, I also believe that's where some of the issues in the early versions of those assembler patches came from. I don't know if that compiler has seen any work since 2008, but I'd be careful with taking it as too much of an example. Yep, I recommend using Mattheo's code if possible. He has rebased my old code and fixed a bunch of issues. Beyond that, my compiler is fairly minimalistic. I think it supports most features except arrays(needs loop unrolling support). However, the optimizer is fairly simplistic. For better optimizations you may want to investigate SSA transformations and related stuff. I guess perfect optimization isn't a requirement for a first version though. First thing would probably be cleaning up and getting something minimal past AJ. Just in case my git tree is here: http://84.112.174.163/~stefan/wine/ . It's not online all the time, and I am not guaranted a fixed IP(although it hasn't changed in years), so check it out as long as you can if you need it. Am I supposed to find your compiler code there? It seems like a git tree of ages ago (where d3dx9_36 only contains math and font code). Yeah, Stefan's project is based on wine 1.1.0 and you can find the relevant code into dlls/wineshader. One of the things I did was to port the assembler to current wine and move it into d3dx9_36.dll. In the process I changed quite some things all around and I stripped from the bytecode writer what wasn't needed by the assembler. I didn't update the compiler code, and as Henri said, at this point probably it's better to just continue what you are doing and look at Stefan's implementation only as a source for inspiration. dlls/wineshader doesn't exist in that tree? As d3dcompiler_xx.dll now seems to contain also the assembler, I'm ok to move the assembler there (once completed) to share the bytecode writer with the compiler. Regarding the optimization, I'd suggest to keep it simple in the beginning. I liked the idea in Stefan's compiler to have a parser which produces an abstract intermediate representation, which is then optimized to generate the bwriter_shader. The optimizer could then be implemented from scratch, or using something third-party as LLVM or such, while as a first step the optimizer could (ideally) simply spit out the code unchanged. I believe that some language features cannot be supported without some optimizations, so this problem can crop up sooner than expected. That's the direction I wanted to take indeed. The optimizer should at some point optimize according to the flag setting. At first, I would only create test cases with D3DXSHADER_SKIPOPTIMIZATION. Lastly, a bit on testing the compiler. I'm not sure trying to get exactly the same bytecode as native is a feasible objective: while for an assembler program there is only one correct translation, this is not true for a compiler. Moreover I expect each iteration of the Microsoft compiler could produce a different output with the same shader program, so there isn't a right implementation to compare with. I agree, this makes testing the compiler much harder... I think that with a lot of testing, the logic behind can be figured out. Then the compiler can be built around this logic. This could possibly even work for a lot of optimizations. On Wed, Apr 14, 2010 at 5:44 PM, Roderick Colenbrander thunderbir...@gmail.com wrote: My main worry is that in the end if we use a compiler framework (remember we don't have a lot of manpower for this) I think it would reduce in more efficient shaders and it prevents some reinventing of the wheel. I'm still not convinced it's worth the effort. Also, I think that a parser/compiler written in lex/bison is much easier for other developers to understand and maintain once it is in the tree (And probably also a lot easier to review patches).
Re: HLSL Compiler and d3dcompiler_xx.dll
On 14 April 2010 16:19, Matijn Woudt tijn...@gmail.com wrote: I have thought about using LLVM, but I don't like it because of: 1) Having another wine dependency (~40MB for me on ubuntu) 2) Code generated by LLVM will most likely not generated exactly the same bytecode as native compiler does, even though the shader has same effect on both. This makes it really hard to verify the HLSL compiler is working. My idea was to make wine's hlsl compiler produce exactly the same bytecode as native does. We might even find bugs in the native compiler, and either fix or duplicate these bugs. LLVM might be useful if we needed a really fast compiler, but native compiler isn't really fast AFAIK. Once games are calling it a few hundred times it might be worth the effort. My understanding was that LLVM is written in C++, so cannot be used directly in Wine (which only allows C89 code). Therefore, for it to be usable in Wine, the LLVM API will need to be wrapped in a C API. - Reece
Re: HLSL Compiler and d3dcompiler_xx.dll
dlls/wineshader doesn't exist in that tree? Check if you are in the correct branch (should be the compiler branch in his tree). Lastly, a bit on testing the compiler. I'm not sure trying to get exactly the same bytecode as native is a feasible objective: while for an assembler program there is only one correct translation, this is not true for a compiler. Moreover I expect each iteration of the Microsoft compiler could produce a different output with the same shader program, so there isn't a right implementation to compare with. I agree, this makes testing the compiler much harder... I think that with a lot of testing, the logic behind can be figured out. Then the compiler can be built around this logic. This could possibly even work for a lot of optimizations. What I mean is that there could not be a single logic behind, I expect to find differences between the various d3dcompiler_xx dlls, expecially regarding the optimizations. You could just pick a version to compare against and try to stick with that, but maybe this is not the best thing to do. On the other hand, with optimizations disabled maybe this doable (but then you aren't testing the optimizer).
Re: HLSL Compiler and d3dcompiler_xx.dll
On 4/14/10 10:36 AM, Reece Dunn wrote: On 14 April 2010 16:19, Matijn Woudt tijn...@gmail.com wrote: I have thought about using LLVM, but I don't like it because of: 1) Having another wine dependency (~40MB for me on ubuntu) 2) Code generated by LLVM will most likely not generated exactly the same bytecode as native compiler does, even though the shader has same effect on both. This makes it really hard to verify the HLSL compiler is working. My idea was to make wine's hlsl compiler produce exactly the same bytecode as native does. We might even find bugs in the native compiler, and either fix or duplicate these bugs. LLVM might be useful if we needed a really fast compiler, but native compiler isn't really fast AFAIK. Once games are calling it a few hundred times it might be worth the effort. My understanding was that LLVM is written in C++, so cannot be used directly in Wine (which only allows C89 code). Therefore, for it to be usable in Wine, the LLVM API will need to be wrapped in a C API. But LLVM *is* wrapped in a C API. (The headers are in include/llvm-c from the LLVM source root.) The real problem is that if we use LLVM, we'll have to write a backend to generate shader bytecode, and LLVM backends have to be written in C++ (actually, you write description files that get turned into C++ code). Personally, I think that should be a separate project anyway. Chip
Re: HLSL Compiler and d3dcompiler_xx.dll
On Wed, Apr 14, 2010 at 6:47 PM, Matteo Bruni matteo.myst...@gmail.com wrote: dlls/wineshader doesn't exist in that tree? Check if you are in the correct branch (should be the compiler branch in his tree). Right, my mistake. Lastly, a bit on testing the compiler. I'm not sure trying to get exactly the same bytecode as native is a feasible objective: while for an assembler program there is only one correct translation, this is not true for a compiler. Moreover I expect each iteration of the Microsoft compiler could produce a different output with the same shader program, so there isn't a right implementation to compare with. I agree, this makes testing the compiler much harder... I think that with a lot of testing, the logic behind can be figured out. Then the compiler can be built around this logic. This could possibly even work for a lot of optimizations. What I mean is that there could not be a single logic behind, I expect to find differences between the various d3dcompiler_xx dlls, expecially regarding the optimizations. You could just pick a version to compare against and try to stick with that, but maybe this is not the best thing to do. On the other hand, with optimizations disabled maybe this doable (but then you aren't testing the optimizer). It should be same as the latest version available (currently version 42). Changes in the logic when a new version comes out will be caught by test cases. We probably shouldn't directly fail, but perhaps mark it todo_wine when it fails with newer version?
Re: HLSL Compiler and d3dcompiler_xx.dll
On 14 April 2010 17:19, Matijn Woudt tijn...@gmail.com wrote: I have thought about using LLVM, but I don't like it because of: 1) Having another wine dependency (~40MB for me on ubuntu) 2) Code generated by LLVM will most likely not generated exactly the same bytecode as native compiler does, even though the shader has same effect on both. This makes it really hard to verify the HLSL compiler is working. My idea was to make wine's hlsl compiler produce exactly the same bytecode as native does. We might even find bugs in the native compiler, and either fix or duplicate these bugs. LLVM might be useful if we needed a really fast compiler, but native compiler isn't really fast AFAIK. Once games are calling it a few hundred times it might be worth the effort. I think you want to test if the compiled shader works instead of the exact bytecode. Generating the same bytecode is probably way too hard, fragile to test, and most likely not worth the effort. I'm not sure about LLVM. On one hand I don't think we want to be writing and maintaining all the optimizations that LLVM can do inside the d3d compiler dll. On the other hand, it would be a pretty large dependency, and it's not clear if it would gain us anything at all. It's not entirely unlikely that if the driver we're running on has a good GLSL compiler we're not going to see much of a benefit from generating good d3d bytecode. (I can already hear the Mesa people laughing about that one though.) Either way, it would probably already be a good start to have a parser, and generate some basic unoptimized bytecode from that, or even just print an AST.
Re: HLSL Compiler and d3dcompiler_xx.dll
On Wed, Apr 14, 2010 at 7:22 PM, Matijn Woudt tijn...@gmail.com wrote: On Wed, Apr 14, 2010 at 6:47 PM, Matteo Bruni matteo.myst...@gmail.com wrote: dlls/wineshader doesn't exist in that tree? Check if you are in the correct branch (should be the compiler branch in his tree). Right, my mistake. Lastly, a bit on testing the compiler. I'm not sure trying to get exactly the same bytecode as native is a feasible objective: while for an assembler program there is only one correct translation, this is not true for a compiler. Moreover I expect each iteration of the Microsoft compiler could produce a different output with the same shader program, so there isn't a right implementation to compare with. I agree, this makes testing the compiler much harder... I think that with a lot of testing, the logic behind can be figured out. Then the compiler can be built around this logic. This could possibly even work for a lot of optimizations. What I mean is that there could not be a single logic behind, I expect to find differences between the various d3dcompiler_xx dlls, expecially regarding the optimizations. You could just pick a version to compare against and try to stick with that, but maybe this is not the best thing to do. On the other hand, with optimizations disabled maybe this doable (but then you aren't testing the optimizer). It should be same as the latest version available (currently version 42). Changes in the logic when a new version comes out will be caught by test cases. We probably shouldn't directly fail, but perhaps mark it todo_wine when it fails with newer version? Also remember that perhaps based on gpu capabilities (even on different SM3.0 capable GPUs) some different code might get generated. One good reason for not having the same code is vertex / pixel shader constants. For some games d3d9x selects uniforms constants which are close to the SM3.0 limits (wined3d eats a number of constants) and this can cause major issues especially on Geforce6/7 and AMD cards (even modern AMD cards only report SM3.0 limits using glsl; you need UBOs to get more). When we have our own compiler we can 'schedule' the constants in such a way that we might be able to avoid these issues. This leads to different shaders. It is not something to worry about now. I think you just want to focus on rendering tests to see if the shader does the right thing. Roderick
Re: HLSL Compiler and d3dcompiler_xx.dll
On Wed, Apr 14, 2010 at 7:29 PM, Henri Verbeet hverb...@gmail.com wrote: On 14 April 2010 17:19, Matijn Woudt tijn...@gmail.com wrote: I have thought about using LLVM, but I don't like it because of: 1) Having another wine dependency (~40MB for me on ubuntu) 2) Code generated by LLVM will most likely not generated exactly the same bytecode as native compiler does, even though the shader has same effect on both. This makes it really hard to verify the HLSL compiler is working. My idea was to make wine's hlsl compiler produce exactly the same bytecode as native does. We might even find bugs in the native compiler, and either fix or duplicate these bugs. LLVM might be useful if we needed a really fast compiler, but native compiler isn't really fast AFAIK. Once games are calling it a few hundred times it might be worth the effort. I think you want to test if the compiled shader works instead of the exact bytecode. Generating the same bytecode is probably way too hard, fragile to test, and most likely not worth the effort. I think I should go for exact bytecode as long as it is possible, it's the only way we can be sure the implementation is 100% right for each test case. I'm not sure about LLVM. On one hand I don't think we want to be writing and maintaining all the optimizations that LLVM can do inside the d3d compiler dll. On the other hand, it would be a pretty large dependency, and it's not clear if it would gain us anything at all. It's not entirely unlikely that if the driver we're running on has a good GLSL compiler we're not going to see much of a benefit from generating good d3d bytecode. (I can already hear the Mesa people laughing about that one though.) Either way, it would probably already be a good start to have a parser, and generate some basic unoptimized bytecode from that, or even just print an AST. Right, it's probably better to go ahead first, and implement LLVM once we find it necessary (if ever).
Re: HLSL Compiler and d3dcompiler_xx.dll
On Wed, Apr 14, 2010 at 7:30 PM, Roderick Colenbrander thunderbir...@gmail.com wrote: On Wed, Apr 14, 2010 at 7:22 PM, Matijn Woudt tijn...@gmail.com wrote: On Wed, Apr 14, 2010 at 6:47 PM, Matteo Bruni matteo.myst...@gmail.com wrote: dlls/wineshader doesn't exist in that tree? Check if you are in the correct branch (should be the compiler branch in his tree). Right, my mistake. Lastly, a bit on testing the compiler. I'm not sure trying to get exactly the same bytecode as native is a feasible objective: while for an assembler program there is only one correct translation, this is not true for a compiler. Moreover I expect each iteration of the Microsoft compiler could produce a different output with the same shader program, so there isn't a right implementation to compare with. I agree, this makes testing the compiler much harder... I think that with a lot of testing, the logic behind can be figured out. Then the compiler can be built around this logic. This could possibly even work for a lot of optimizations. What I mean is that there could not be a single logic behind, I expect to find differences between the various d3dcompiler_xx dlls, expecially regarding the optimizations. You could just pick a version to compare against and try to stick with that, but maybe this is not the best thing to do. On the other hand, with optimizations disabled maybe this doable (but then you aren't testing the optimizer). It should be same as the latest version available (currently version 42). Changes in the logic when a new version comes out will be caught by test cases. We probably shouldn't directly fail, but perhaps mark it todo_wine when it fails with newer version? Also remember that perhaps based on gpu capabilities (even on different SM3.0 capable GPUs) some different code might get generated. One good reason for not having the same code is vertex / pixel shader constants. For some games d3d9x selects uniforms constants which are close to the SM3.0 limits (wined3d eats a number of constants) and this can cause major issues especially on Geforce6/7 and AMD cards (even modern AMD cards only report SM3.0 limits using glsl; you need UBOs to get more). When we have our own compiler we can 'schedule' the constants in such a way that we might be able to avoid these issues. This leads to different shaders. It is not something to worry about now. I think you just want to focus on rendering tests to see if the shader does the right thing. Roderick I think, for now, we should pass D3DXSHADER_SKIPVALIDATION to the compiler, so the code won't be checked for GPU capabilities. I'm not sure if different code is generated when GPU capabilities are low, but I think it's safe to fail if we request too many constants. In test cases we could check D3DCAPS9 and skip if needed.
Re: HLSL Compiler and d3dcompiler_xx.dll
Am 14.04.2010 um 19:29 schrieb Henri Verbeet: I think you want to test if the compiled shader works instead of the exact bytecode. Generating the same bytecode is probably way too hard, fragile to test, and most likely not worth the effort. Yep, comparing generated bytecode is not going to work. I think I wrote only 3-4 HLSL tests in the compiler, and all those were based on checking the rendering results, similar to the d3d9 visual tests. I'm not sure about LLVM. On one hand I don't think we want to be writing and maintaining all the optimizations that LLVM can do inside the d3d compiler dll. My main concern about LLVM in 2008 was that it wasn't capable of some shader specific things like DDX and DDY. Apple only used it to optimize CPU fallback code, not to optimize code sent to the GPU. This might have changed since then, especially if the Gallium3D developers indeed use LLVM, as they planned in the past. I recommend to contact the Mesa developers about their compiler status and plans. If we can it's certainly an advantage to use an existing compiler engine like LLVM instead of reinventing the wheel. I doubt that the dependency is a big issue. It will probably be required by Mesa as well, so distros will install it anyway. LLVM is available on OSX too. If we're extra-lucky we can implement the parser and bytecode writer ourselves, and just send the parse tree to LLVM and get an optimized tree back. So LLVM could be optional. If you have it great, otherwise things will still work, but shaders will be less efficient. HLSL is both a simple and complicated language. In many ways it is easier than average imperative programming language because it doesn't have pointers, references or similar features. In d3d9 it had pretty much no types(only floats, except in special cases). Then there are issues that make things more complicated, for example the special address registers, vector registers and no memory to push/pop registers. If your register allocator can't fit the program in the existing registers you're out of luck. However, pointers are appearing in GLSL via extensions, and I guess in HLSL it's only a matter of time(maybe dx11 compute shaders...).
Re: HLSL Compiler and d3dcompiler_xx.dll
On Apr 14, 2010, at 1:48 PM, Matijn Woudt wrote: I think you want to test if the compiled shader works instead of the exact bytecode. Generating the same bytecode is probably way too hard, fragile to test, and most likely not worth the effort. I think I should go for exact bytecode as long as it is possible, it's the only way we can be sure the implementation is 100% right for each test case. FYI, we encountered a game a while back which used a few shaders that depended on being compiled with a particular version for d3dx9_##.dll. There was a compiler bug which the game engine knew about and accounted for. If you tried to use the compiled shader from a newer d3dx9_##.dll, then the rendering wouldn't look quite right on certain objects. So, there's one argument for identical bytecode compatibility, but it's likely that very few apps will exhibit behavior like this.
