Re: [sqlite] Next Version of SQLite
[EMAIL PROTECTED] wrote: Joe Wilson [EMAIL PROTECTED] wrote: --- D. Richard Hipp [EMAIL PROTECTED] wrote: Sorry for the confusion. No problem. For what it's worth, I am also curious as to the final form of the VM opcode transformation. The number of opcodes generated by the various SQL statements seems to be roughly the same as the old scheme. At this point without sub-expresssion elimination are you seeing any speed improvement? I have not even looked at performance yet. I'm assuming that performance will drop during the conversion process and that we will have to fight to get it back up to previous levels after the conversion is complete. But consider would can be done with a register machine that would couldn't do with the old stack machine. In a statement like this: SELECT * FROM a NATURAL JOIN b; Suppose tables a and b have column c in common and unique columns a1, a2, a3 and b1, b2, b3. With the stack machine, the algorithm is roughly this: foreach each entry in a: foreach entry in b with b.c==a.c: push a.c push a.a1 push a.a2 push a.a3 push b.b1 push b.b2 push b.b3 return one row of result endforeach endforeach For each result row, all columns had to be pushed onto the stack. Then the OP_Callback opcode would fire, causing sqlite3_step() to return SQLITE_ROW. The result columns would then be available to sqlite3_column_xxx() routines which read those results off of the stack. When sqlite3_step() is called again, all result columns are popped from the stack and execution continues with the first operation after the OP_Callback. In the register VM, result columns are stored in a consecutive sequence of registers. It is no longer necessary to pop the stack of prior results at the start of each sqlite3_step(). So the code can look more like this: foreach entry in a: r1 = a.c r2 = a.a1 r3 = a.a2 r4 = a.a3 foreach entry in b where c=r1: r5 = b.b1 r6 = b.b2 r7 = b.b3 return one row of result endforeach endforeach When result are stored in registers, the computation of the first four columns of the result set can be factored out of the inner loop. If there are 10 matching rows in b for every row in a, this might result in a significant performance boost. Do not look for this improvement right away, though. The first order of business is to get the VM converted over into a register machine. Only after that is successfully accomplished will we look into implementing optimizations such as the above. -- D. Richard Hipp [EMAIL PROTECTED] This architecture replacement is a very significant step for Sqlite from my perspective. It indicates a transition from the simple, more easily implemented technology into a more refined one providing the basis for adding more sophistication in optimization and a generally more efficient core. Such ongoing evolution guarantees that Sqlite will not ossify and become irrelevant. Congratulation Dr Hipp and team for not leaving well alone. - To unsubscribe, send email to [EMAIL PROTECTED] - - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
Joe Wilson [EMAIL PROTECTED] wrote: --- D. Richard Hipp [EMAIL PROTECTED] wrote: Sorry for the confusion. No problem. For what it's worth, I am also curious as to the final form of the VM opcode transformation. The number of opcodes generated by the various SQL statements seems to be roughly the same as the old scheme. At this point without sub-expresssion elimination are you seeing any speed improvement? I have not even looked at performance yet. I'm assuming that performance will drop during the conversion process and that we will have to fight to get it back up to previous levels after the conversion is complete. But consider would can be done with a register machine that would couldn't do with the old stack machine. In a statement like this: SELECT * FROM a NATURAL JOIN b; Suppose tables a and b have column c in common and unique columns a1, a2, a3 and b1, b2, b3. With the stack machine, the algorithm is roughly this: foreach each entry in a: foreach entry in b with b.c==a.c: push a.c push a.a1 push a.a2 push a.a3 push b.b1 push b.b2 push b.b3 return one row of result endforeach endforeach For each result row, all columns had to be pushed onto the stack. Then the OP_Callback opcode would fire, causing sqlite3_step() to return SQLITE_ROW. The result columns would then be available to sqlite3_column_xxx() routines which read those results off of the stack. When sqlite3_step() is called again, all result columns are popped from the stack and execution continues with the first operation after the OP_Callback. In the register VM, result columns are stored in a consecutive sequence of registers. It is no longer necessary to pop the stack of prior results at the start of each sqlite3_step(). So the code can look more like this: foreach entry in a: r1 = a.c r2 = a.a1 r3 = a.a2 r4 = a.a3 foreach entry in b where c=r1: r5 = b.b1 r6 = b.b2 r7 = b.b3 return one row of result endforeach endforeach When result are stored in registers, the computation of the first four columns of the result set can be factored out of the inner loop. If there are 10 matching rows in b for every row in a, this might result in a significant performance boost. Do not look for this improvement right away, though. The first order of business is to get the VM converted over into a register machine. Only after that is successfully accomplished will we look into implementing optimizations such as the above. -- D. Richard Hipp [EMAIL PROTECTED] - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
Bill KING [EMAIL PROTECTED] wrote: A colleague brought up a very good point. At least for the first few revisions, is the old engine/code still going to be available until the new engine code base settles down? You can always pull the old code frm CVS and run it against the newer code. We started making the switch with check-in [4652] on 2008-01-02. So any CVS verious prior to that will be a pure stack-based machine. The changes consist of slowly adding capability to the existing VM so that it can work with operands on the stack or in registers. This continued through check-in [4700] on 2008-01-09. After [4700], all operations can be done using register operands, though the code generator still uses the stack for many things. Since then and moving forward, we are converting the code generator to use registers rather than the stack, and we are removing stack capabilities from the VM as those abilities lapse from use. Toward the end of the process, we will completely remove the operand stack from the VM. During the conversion process, SQLite will likely be slower and use more memory. But all regression tests are suppose to pass after each check-in. So we hope to avoid introducing any bugs. Once the conversion is complete, we hope to be able to reduce the memory footprint of each sqlite3_stmt object and make them run a little faster too. But we need to get a correctly functioning pure register VM first. -- D. Richard Hipp [EMAIL PROTECTED] - To unsubscribe, send email to [EMAIL PROTECTED] -
RE: [sqlite] Next Version of SQLite
-Original Message- From: Nicolas Williams [mailto:[EMAIL PROTECTED] Sent: Monday, January 14, 2008 10:29 AM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Next Version of SQLite On Sun, Jan 13, 2008 at 08:46:03PM -0600, Rick Langschultz wrote: I was wondering what would constitute the creation of SQLite 4.0? IMO major implementation details changes are not necessarily a good rationale for bumping the major version number, not from a user's p.o.v. ... My guess is that the 3.x APIs are plenty good enough that incremental evolution is far better than incompatible revolution, so I bet they are here to stay. Which means that if the major version number changes, then it'll be for marketing purposes. Nico -- Well seeing how SQLite is FREE, it does its own marketing with one word, so to speak. Therefore I doubt marketing will have much impact on versioning. What a quandary develops when one goes against the grain of commercial software deployment :-) Fred - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
On Sun, Jan 13, 2008 at 08:46:03PM -0600, Rick Langschultz wrote: I was wondering what would constitute the creation of SQLite 4.0? IMO major implementation details changes are not necessarily a good rationale for bumping the major version number, not from a user's p.o.v. The authors might think it is though, and from a marketing p.o.v. it really, really might be (perception matters). If changing the major version number need not imply changing the C API prefix, then it's certainly easier to change the major version number. But I think developers are used to sqlite3_ as the prefix now -- changing the major version number without changing that prefix will seem odd, but changing the prefix without substantive changes to the API will be gratuitous and very painful. If the 3.x C APIs became a stone around SQLite's neck and couldn't be incrementally fixed (by deprecating a small number of functions and adding a few new ones), then the time might be right for changing the prefix to sqlite4_. If the DB format had to change radically, then that too might be a good reason to change the major version number. My guess is that the 3.x APIs are plenty good enough that incremental evolution is far better than incompatible revolution, so I bet they are here to stay. Which means that if the major version number changes, then it'll be for marketing purposes. Nico -- - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
On Mon, Jan 14, 2008 at 10:55:42AM -0600, Fred Williams wrote: Which means that if the major version number changes, then it'll be for marketing purposes. Well seeing how SQLite is FREE, it does its own marketing with one word, so to speak. Therefore I doubt marketing will have much impact on versioning. What a quandary develops when one goes against the grain of commercial software deployment :-) I wouldn't knock marketing, even for open source, public domain projects like this one, even where marketing has been of the word-of-mouth type. The question is: will 4.0 scare folks (oh dear, must be a new API, just like 3.0 meant) or entice them (oh cool! must be way better, so I should go learn it now)? I have no idea what the answer to that is. One very good aspect of SQLite 3.x has been stability and evolution (i.e., few if any incompatible changes). This too is very important from a marketing p.o.v. The safest bet is that 3.6.0 is a good enough move. Nico -- - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
[EMAIL PROTECTED] wrote: Bill KING [EMAIL PROTECTED] wrote: A colleague brought up a very good point. At least for the first few revisions, is the old engine/code still going to be available until the new engine code base settles down? You can always pull the old code frm CVS and run it against the newer code. We started making the switch with check-in [4652] on 2008-01-02. So any CVS verious prior to that will be a pure stack-based machine. As expected, will be setting up a set of auto-tests for our particular use-case, but that's just being a good software engineer ;) The changes consist of slowly adding capability to the existing VM so that it can work with operands on the stack or in registers. This continued through check-in [4700] on 2008-01-09. After [4700], all operations can be done using register operands, though the code generator still uses the stack for many things. Since then and moving forward, we are converting the code generator to use registers rather than the stack, and we are removing stack capabilities from the VM as those abilities lapse from use. Toward the end of the process, we will completely remove the operand stack from the VM. One hell of a challenge, but in the end, I do agree with you, there's been a lot of knowledge around for a long time on register based optimisations, and with an infinite register machine, some really amazing optimizations can occur, agreed. Besides, all the cool kids are doing it too ;) During the conversion process, SQLite will likely be slower and use more memory. But all regression tests are suppose to pass after each check-in. So we hope to avoid introducing any bugs. Once the conversion is complete, we hope to be able to reduce the memory footprint of each sqlite3_stmt object and make them run a little faster too. But we need to get a correctly functioning pure register VM first. This part I'm happy with all regression tests are supposed to pass :) I'm still going to wait a point or two before production tho, but that's again just careful management. Looking forward to this tho in the long run. -- D. Richard Hipp [EMAIL PROTECTED] - To unsubscribe, send email to [EMAIL PROTECTED] - -- Bill King, Software Engineer Trolltech, Brisbane Technology Park 26 Brandl St, Eight Mile Plains, QLD, Australia, 4113 Tel + 61 7 3219 9906 (x137) Fax + 61 7 3219 9938 mobile: 0423 532 733 - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
At 9:07 PM -0500 1/12/08, D. Richard Hipp wrote: In case you haven't been watching the timeline (http://www.sqlite.org/cvstrac/timeline) we are in the middle of some major changes. The virtual machine inside of SQLite is being transformed from a stack-based machine into a register-based machine. The whole virtual machine and the code generator is being rewritten. Slowly. Piece by piece. I haven't done an overall line change count yet, but we are looking at some pretty serious code churn. 3.5.4 to 3.5.5 is likely to be the biggest single change in the history of SQLite. I would think something like that is worthy of a 3.6.0 version number. Not just a minor version increase that would be more suitable for minor changes or bugfixes. On the other hand, I can understand if you save the larger increments for changes that are user-visible and not just plumbing. -- Darren Duncan - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
On Sun, 13 Jan 2008, Darren Duncan wrote: I would think something like that is worthy of a 3.6.0 version number. Not just a minor version increase that would be more suitable for minor changes or bugfixes. I agree with Darren that massive changes to the core of the system should be reflected by a major version increase (to 4.0); at a minimum to a minor version increease (to 3.6). A version number change from 3.5.4 to 3.5.5 tells folks that it's a minor bug fix or simple adjustment, not a wholesale rewrite of the system's core. Rich -- Richard B. Shepard, Ph.D. | IntegrityCredibility Applied Ecosystem Services, Inc.|Innovation http://www.appl-ecosys.com Voice: 503-667-4517 Fax: 503-667-8863 - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
What will be the main benefits of the new virtual machine? I mean, it will be just faster or there will be other improvements in the library? --- Marco Bambini http://www.sqlabs.net http://www.sqlabs.net/blog/ http://www.sqlabs.net/realsqlserver/ On Jan 13, 2008, at 3:07 AM, D. Richard Hipp wrote: On Jan 12, 2008, at 7:55 PM, Shawn Wilsher wrote: Hey all, I was wondering when you plan on releasing the next version of SQLite. Mozilla is currently using 3.5.4, but that does not include some OS/2 fixes that were checked in after the release of 3.5.4. Instead of patching our local copy of sqlite, I'd like to use a release version, but at the same time do not want to delay this fix to our OS/2 users very long. The specific checkins we are looking at are 4646, 4647, and 4648. In case you haven't been watching the timeline (http://www.sqlite.org/cvstrac/timeline) we are in the middle of some major changes. The virtual machine inside of SQLite is being transformed from a stack-based machine into a register-based machine. The whole virtual machine and the code generator is being rewritten. Slowly. Piece by piece. I haven't done an overall line change count yet, but we are looking at some pretty serious code churn. 3.5.4 to 3.5.5 is likely to be the biggest single change in the history of SQLite. So you might not want to release product with 3.5.5 embedded. All the regression tests pass, but still If you like, we can set up a special Mozilla branch off of 3.5.4 that includes the OS/2 fixes. On the other hand, if this is not for a release, but rather for general development work, then please build and test with the latest code from CVS. (This applies to *everybody* not just Mozilla.) Please report any problems. The test suite for SQLite is very thorough, but I have found that users can be very creative in stressing SQLite in ways that I would have never imagined, and have not developed tests for. D. Richard Hipp [EMAIL PROTECTED] -- --- To unsubscribe, send email to [EMAIL PROTECTED] -- --- - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
On Jan 13, 2008, at 11:31 AM, Rich Shepard wrote: On Sun, 13 Jan 2008, Darren Duncan wrote: I would think something like that is worthy of a 3.6.0 version number. Not just a minor version increase that would be more suitable for minor changes or bugfixes. I agree with Darren that massive changes to the core of the system should be reflected by a major version increase (to 4.0); at a minimum to a minor version increease (to 3.6). A version number change from 3.5.4 to 3.5.5 tells folks that it's a minor bug fix or simple adjustment, not a wholesale rewrite of the system's core. There are no user-visible changes to the interface. The version numbers in SQLite reflect user-visible changes only. Well, there is one minor user-visible change. The output of EXPLAIN now has 7 columns whereas it used to contain only 5. But the output of EXPLAIN changes from point release to point release all the time anyway, so I do not consider this something worth bumping a version number. I do not expect significant instability with the next release. I want to gain some experience with the new software before I recommend it for millions of deployments. But it should be solid and stable as soon as it is released. For that matter, the current code in CVS (which is well into the conversion to a register machine) has not been giving any problems. There are people on this mailing list (ex: Joe Wilson) who appear to read every line of every change that we make to SQLite, within minutes of making them, and complain if we so much as misspell a word in a comment. And I haven't heard a peep from Joe or anybody else, so I'm thinking the code is still working correctly for everybody despite the massive changes that have already gone in. If you find that the current code in CVS gives problems, or if you see significant problems emerge as we get closer to releasing 3.5.5, then maybe we might consider calling it 3.6.0. But I do not anticipate any serious problems. You should not underestimate the level of detail to which we test SQLite and the thoroughness of the test suite. Not much is likely to slip through the cracks. D. Richard Hipp [EMAIL PROTECTED] - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
On Jan 13, 2008, at 11:40 AM, Marco Bambini wrote: What will be the main benefits of the new virtual machine? Optimizations such as common subexpression elimination and moving subexpressions outside of inner loops will become much easier. The code generator will, in general, be easier to work on and less error prone. An entire class of errors (stack overflow) such as the recent ticket #2832 (which could cause database corruption) will become impossible since the VM will no longer have a stack to overflow. D. Richard Hipp [EMAIL PROTECTED] - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
--- D. Richard Hipp [EMAIL PROTECTED] wrote: There are people on this mailing list (ex: Joe Wilson) who appear to read every line of every change that we make to SQLite, within minutes of making them, and complain if we so much as misspell a word in a comment. And I haven't heard a peep from Joe Wow - what prompted that dig against me? Fixing a spelling mistake in a comment adds no value to the software. None of those spelling mistake tickets are my doing. The majority of the tickets I've reported were legitimate problems in the code - and arguing the case for some of these bugs isn't exactly the easiest thing. But hey, it's cool. I won't file any more tickets. Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
Joe Wilson wrote: --- D. Richard Hipp [EMAIL PROTECTED] wrote: There are people on this mailing list (ex: Joe Wilson) who appear to read every line of every change that we make to SQLite, within minutes of making them, and complain if we so much as misspell a word in a comment. And I haven't heard a peep from Joe Wow - what prompted that dig against me? I took it as a kidding compliment. I saw an invisible smiley after his comment. Your contributions to SQLite are appreciated by all of us. Please continue. Gerry - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
On 1/13/08, Joe Wilson [EMAIL PROTECTED] wrote: --- D. Richard Hipp [EMAIL PROTECTED] wrote: There are people on this mailing list (ex: Joe Wilson) who appear to read every line of every change that we make to SQLite, within minutes of making them, and complain if we so much as misspell a word in a comment. And I haven't heard a peep from Joe Wow - what prompted that dig against me? Fixing a spelling mistake in a comment adds no value to the software. None of those spelling mistake tickets are my doing. The majority of the tickets I've reported were legitimate problems in the code - and arguing the case for some of these bugs isn't exactly the easiest thing. But hey, it's cool. I won't file any more tickets. honestly, coming from Richard, that is most definitely a compliment to an old friend indicative of the value of your contribution. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
On Jan 13, 2008, at 7:53 PM, Gerry Snyder wrote: Joe Wilson wrote: --- D. Richard Hipp [EMAIL PROTECTED] wrote: There are people on this mailing list (ex: Joe Wilson) who appear to read every line of every change that we make to SQLite, within minutes of making them, and complain if we so much as misspell a word in a comment. And I haven't heard a peep from Joe Wow - what prompted that dig against me? I took it as a kidding compliment. I saw an invisible smiley after his comment. Your contributions to SQLite are appreciated by all of us. Exactly. My remark was meant to indicate that I am amazed at how much attention you pay to the code, not any kind of a dig. Sorry for the confusion. You are an important member of the quality assurance team and I hope that you will continue in that role. D. Richard Hipp [EMAIL PROTECTED] - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
I was wondering what would constitute the creation of SQLite 4.0? Since the VDBE is being revamped I would consider this a pretty big revamp of the SQLite code. I am looking forward to testing this out new engine out. I also wanted to know what the difference between stack based and register based is. Unfortunately, i do not mess with a lot of C / C++ code in my work. So I was a little curious. On Jan 13, 2008, at 7:41 PM, D. Richard Hipp wrote: On Jan 13, 2008, at 7:53 PM, Gerry Snyder wrote: Joe Wilson wrote: --- D. Richard Hipp [EMAIL PROTECTED] wrote: There are people on this mailing list (ex: Joe Wilson) who appear to read every line of every change that we make to SQLite, within minutes of making them, and complain if we so much as misspell a word in a comment. And I haven't heard a peep from Joe Wow - what prompted that dig against me? I took it as a kidding compliment. I saw an invisible smiley after his comment. Your contributions to SQLite are appreciated by all of us. Exactly. My remark was meant to indicate that I am amazed at how much attention you pay to the code, not any kind of a dig. Sorry for the confusion. You are an important member of the quality assurance team and I hope that you will continue in that role. D. Richard Hipp [EMAIL PROTECTED] - To unsubscribe, send email to [EMAIL PROTECTED] - smime.p7s Description: S/MIME cryptographic signature
Re: [sqlite] Next Version of SQLite
At 8:46 PM -0600 1/13/08, Rick Langschultz wrote: I was wondering what would constitute the creation of SQLite 4.0? Since the VDBE is being revamped I would consider this a pretty big revamp of the SQLite code. I am looking forward to testing this out new engine out. I also wanted to know what the difference between stack based and register based is. Unfortunately, i do not mess with a lot of C / C++ code in my work. So I was a little curious. Although I would have considered the current large update to be 3.6.0 material, I definitely do *not* consider it to be 4.0.0 material. Major version updates (such as to 4.0) should not be done lightly, and should only be done for things like large or particularly incompatible database file format changes, such as the 2.x to 3.x update was, or for large or incompatible API changes, either way things that users could have no excuse not to notice. So I agree with Richard's current versioning plan concerning this. -- Darren Duncan - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
A colleague brought up a very good point. At least for the first few revisions, is the old engine/code still going to be available until the new engine code base settles down? (via #defines maybe?). It would lead towards a good chance of comparison between the two engines too for people. D. Richard Hipp wrote: On Jan 13, 2008, at 11:31 AM, Rich Shepard wrote: On Sun, 13 Jan 2008, Darren Duncan wrote: I would think something like that is worthy of a 3.6.0 version number. Not just a minor version increase that would be more suitable for minor changes or bugfixes. I agree with Darren that massive changes to the core of the system should be reflected by a major version increase (to 4.0); at a minimum to a minor version increease (to 3.6). A version number change from 3.5.4 to 3.5.5 tells folks that it's a minor bug fix or simple adjustment, not a wholesale rewrite of the system's core. There are no user-visible changes to the interface. The version numbers in SQLite reflect user-visible changes only. Well, there is one minor user-visible change. The output of EXPLAIN now has 7 columns whereas it used to contain only 5. But the output of EXPLAIN changes from point release to point release all the time anyway, so I do not consider this something worth bumping a version number. I do not expect significant instability with the next release. I want to gain some experience with the new software before I recommend it for millions of deployments. But it should be solid and stable as soon as it is released. For that matter, the current code in CVS (which is well into the conversion to a register machine) has not been giving any problems. There are people on this mailing list (ex: Joe Wilson) who appear to read every line of every change that we make to SQLite, within minutes of making them, and complain if we so much as misspell a word in a comment. And I haven't heard a peep from Joe or anybody else, so I'm thinking the code is still working correctly for everybody despite the massive changes that have already gone in. If you find that the current code in CVS gives problems, or if you see significant problems emerge as we get closer to releasing 3.5.5, then maybe we might consider calling it 3.6.0. But I do not anticipate any serious problems. You should not underestimate the level of detail to which we test SQLite and the thoroughness of the test suite. Not much is likely to slip through the cracks. D. Richard Hipp [EMAIL PROTECTED] - To unsubscribe, send email to [EMAIL PROTECTED] - -- Bill King, Software Engineer Trolltech, Brisbane Technology Park 26 Brandl St, Eight Mile Plains, QLD, Australia, 4113 Tel + 61 7 3219 9906 (x137) Fax + 61 7 3219 9938 mobile: 0423 532 733 - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rick Langschultz wrote: I was wondering what would constitute the creation of SQLite 4.0? An incompatible API, or significant behaviour changes. Since the VDBE is being revamped I would consider this a pretty big revamp of the SQLite code. The VDBE is not visible outside of the SQLite library except when EXPLAIN is used. It is immaterial to you as a user of SQLite as long as it is correctly implemented. For example if SQLite started counting internally using roman numerals but still always gave correct (decimal) results then it doesn't matter. I also wanted to know what the difference between stack based and register based is. Unfortunately, i do not mess with a lot of C / C++ code in my work. So I was a little curious. http://en.wikipedia.org/wiki/Virtual_machine#List_of_virtual_machine_software Except for trivial ones, all virtual machines have a stack. For example it is how you implement recursion with the stack growing and shrinking as you enter and leave functions. It is also where intermediate values are stored. Expressions are compiled to work on the stack. For example something like b+2*c would become something like: push b push 2 push c mulitply add To understand this well, I strongly recommend looking up the Forth programming language and a few Forth tutorials. Stack based virtual machines as above are easy to understand and implement. However there is a limit to their performance. Every operation manipulates the stack and you can end up with a lot of stack manipulation code as operations such as add and multiply in the example above only work on the top operands of the stack. You also have to be careful when generating the opcodes that you don't overflow the stack, and that functions return the stack in the same state they found it (it would be disastrous if there were more or less entries on it). Real hardware uses both a stack and registers. Registers are a fast temporary storage location, but you can still use the stack. For example an x86 processor in 32 bit mode has 8 registers. For virtual machines there is no need to limit how many registers there are. http://en.wikipedia.org/wiki/Processor_register Current best practise in compilers is to use registers (and has been for many decades). Look up three address code and SSA as well as the optimisations they allow. It is also easier to JIT register based code. Virtual machines up to and including Java tended to be stack based. .NET, parrot, llvm etc are all register based because of the advantages. In the case of SQLite I suspect that being register based will result in fewer operations per program and each operation will execute faster (no futzing with the stack). You can try it for yourself by using EXPLAIN in 3.5.4 vs CVS. Here is a good description of why the Parrot VM used register rather than stack: http://www.sidhe.org/~dan/blog/archives/000189.html Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHitZnmOOfHg372QQRAljEAJ97yFOuWh6/lOlV572iUjHrHThMfgCfQsjN LMCBk5uiO5IAsUADf96F9lQ= =VNz/ -END PGP SIGNATURE- - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
--- D. Richard Hipp [EMAIL PROTECTED] wrote: Sorry for the confusion. No problem. For what it's worth, I am also curious as to the final form of the VM opcode transformation. The number of opcodes generated by the various SQL statements seems to be roughly the same as the old scheme. At this point without sub-expresssion elimination are you seeing any speed improvement? Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
this was a true complement and nothing else. P Kishor wrote: On 1/13/08, Joe Wilson [EMAIL PROTECTED] wrote: --- D. Richard Hipp [EMAIL PROTECTED] wrote: There are people on this mailing list (ex: Joe Wilson) who appear to read every line of every change that we make to SQLite, within minutes of making them, and complain if we so much as misspell a word in a comment. And I haven't heard a peep from Joe Wow - what prompted that dig against me? Fixing a spelling mistake in a comment adds no value to the software. None of those spelling mistake tickets are my doing. The majority of the tickets I've reported were legitimate problems in the code - and arguing the case for some of these bugs isn't exactly the easiest thing. But hey, it's cool. I won't file any more tickets. honestly, coming from Richard, that is most definitely a compliment to an old friend indicative of the value of your contribution. - To unsubscribe, send email to [EMAIL PROTECTED] - - To unsubscribe, send email to [EMAIL PROTECTED] -
[sqlite] Next Version of SQLite
Hey all, I was wondering when you plan on releasing the next version of SQLite. Mozilla is currently using 3.5.4, but that does not include some OS/2 fixes that were checked in after the release of 3.5.4. Instead of patching our local copy of sqlite, I'd like to use a release version, but at the same time do not want to delay this fix to our OS/2 users very long. The specific checkins we are looking at are 4646, 4647, and 4648. See Bug 411780 for details (https://bugzilla.mozilla.org/show_bug.cgi?id=411780). Cheers, Shawn Wilsher - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
On Jan 12, 2008, at 7:55 PM, Shawn Wilsher wrote: Hey all, I was wondering when you plan on releasing the next version of SQLite. Mozilla is currently using 3.5.4, but that does not include some OS/2 fixes that were checked in after the release of 3.5.4. Instead of patching our local copy of sqlite, I'd like to use a release version, but at the same time do not want to delay this fix to our OS/2 users very long. The specific checkins we are looking at are 4646, 4647, and 4648. In case you haven't been watching the timeline (http://www.sqlite.org/cvstrac/timeline) we are in the middle of some major changes. The virtual machine inside of SQLite is being transformed from a stack-based machine into a register-based machine. The whole virtual machine and the code generator is being rewritten. Slowly. Piece by piece. I haven't done an overall line change count yet, but we are looking at some pretty serious code churn. 3.5.4 to 3.5.5 is likely to be the biggest single change in the history of SQLite. So you might not want to release product with 3.5.5 embedded. All the regression tests pass, but still If you like, we can set up a special Mozilla branch off of 3.5.4 that includes the OS/2 fixes. On the other hand, if this is not for a release, but rather for general development work, then please build and test with the latest code from CVS. (This applies to *everybody* not just Mozilla.) Please report any problems. The test suite for SQLite is very thorough, but I have found that users can be very creative in stressing SQLite in ways that I would have never imagined, and have not developed tests for. D. Richard Hipp [EMAIL PROTECTED] - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] Next Version of SQLite
On Jan 12, 2008 9:07 PM, D. Richard Hipp [EMAIL PROTECTED] wrote: In case you haven't been watching the timeline (http://www.sqlite.org/cvstrac/timeline) we are in the middle of some major changes. The virtual machine inside of SQLite is being transformed from a stack-based machine into a register-based machine. The whole virtual machine and the code generator is being rewritten. Slowly. Piece by piece. I haven't done an overall line change count yet, but we are looking at some pretty serious code churn. 3.5.4 to 3.5.5 is likely to be the biggest single change in the history of SQLite. Out of curiosity, why so many changes for a point release? If you like, we can set up a special Mozilla branch off of 3.5.4 that includes the OS/2 fixes. That'd be awesome if we could get a 3.5.4.1 type of thing going. I'm very much against patching sqlite locally in our tree (harder to upgrade). Cheers, Shawn Wilsher - To unsubscribe, send email to [EMAIL PROTECTED] -