Re: [Monotone-devel] unhexification of revision hashes
Hi, Ludovic Brenta wrote: Sorry to interrupt but in standard SQL there is WHERE id LIKE 'deadbe%' and I happen to use it occasionally on the command line. I'm worried that that won't be possible anymore after the unhexification. Agreed. Unfortunately the '%' doesn't work that well for binary values, so yes, you would loose the ability to query directly via LIKE. The following supposedly still works, provided you use a monotone enhanced sqlite, which provides an unhex() function: WHERE unhex(id) LIKE 'deadbe%' Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] unhexification of revision hashes
Markus Schiltknecht writes: > The prefix_matching_constraint() function prepares an WHERE clause, > which imitates the sqlite'ism named 'GLOB'. Instead of using a clause > like > > WHERE id GLOB 'deadbe*' > > It now prepares a where clause more similar to: > > WHERE id >= 'deadbe' AND id <= 'deadbf' Sorry to interrupt but in standard SQL there is WHERE id LIKE 'deadbe%' and I happen to use it occasionally on the command line. I'm worried that that won't be possible anymore after the unhexification. -- Ludovic Brenta. ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] unhexification of revision hashes
Hi, Zack Weinberg wrote: > 2) On .experiment.encapsulation, selector completion is done totally > differently. I'm not sure what prefix_matching_constraint() needs to > turn into, but I'm sure it won't work in its present form after .e.e > lands. Okay, I've just propagated and fixed everything from n.v.m.e.e to that unhexification branch. For convenience, I've also suspended the other revision, so there's just one un-suspended head now in n.v.m.experiment.db-compaction (44f34ded75a62aceb6eb42b69d3704ede8a69707). All tests succeed on my box again. The prefix_matching_constraint() function prepares an WHERE clause, which imitates the sqlite'ism named 'GLOB'. Instead of using a clause like WHERE id GLOB 'deadbe*' It now prepares a where clause more similar to: WHERE id >= 'deadbe' AND id <= 'deadbf' In a way, this is a work around for the missing support of GLOBing binary fields in SQLite. On the other hand, I like the above boundary think better, because it's more SQL compliant, IIRC. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] unhexification of revision hashes
Hi, Zbigniew Zagórski wrote: Ha... I think it's "Graydon leave effect" ... everybody is now on steroids ;) See releases ? ViewMtn, Guitone, mtteam is also preparing a demo ... Hope this enthusiasm will last for long... I certainly hope so, yes. ;-) Unfortunately I've recompiled head, ran tests and times are the same as with 0.38, so your code is slower. I didn't wanted to bring bad news ... :( No problem, thanks for the performance tests. BTW: please don't drop the mailing list and reply to all. Or can the list admin make the mailing list set a "reply-to" header, or something? Hmm, it's my common mistake to hit "Reply" only but this time I'm innocent. I've noticed later on, sorry. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] unhexification of revision hashes
Markus Schiltknecht pisze: Hello Zbigniew, Zbigniew Zagórski wrote: I don't remember reasons for this change besides db compaction. Hey, thanks, that's great and pretty instant help. ;-) Ha... I think it's "Graydon leave effect" ... everybody is now on steroids ;) See releases ? ViewMtn, Guitone, mtteam is also preparing a demo ... Hope this enthusiasm will last for long... ... Last I checked, I saw very small performance improvements, so these numbers are looking suspicious to me. Can you recheck the CXX flags, please? Unfortunately I've recompiled head, ran tests and times are the same as with 0.38, so your code is slower. I didn't wanted to bring bad news ... :( BTW. All tests on hot DB and repeated few times. BTW: please don't drop the mailing list and reply to all. Or can the list admin make the mailing list set a "reply-to" header, or something? Hmm, it's my common mistake to hit "Reply" only but this time I'm innocent. +1 for Reply To header set by mailing daemon. -- Zbigniew -zbigg- Zagórski / software developer / geek / happy daddy / ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] unhexification of revision hashes
At 2008-01-28T22:15:40+0100, Zbigniew Zagórski wrote: > BTW. What are the cxxflags of official windows build, my build is > "-O2 -g -Wall" stripped, maybe this perf hit is caused by bad flags. We use the defaults from the build process, which end up being: CFLAGS = -g -O2 -fno-strict-aliasing -fexceptions CXXFLAGS = -g -O2 -Wall -W -Wno-unused -fno-strict-aliasing ...which are pretty much what you're using, so the performance regression might be real and needs further investigation. Cheers, -mjg -- Matthew Gregan |/ /|[EMAIL PROTECTED] ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] unhexification of revision hashes
Hi Zack, Zack Weinberg wrote: Ok, I've read this code over. I like it, but unfortunately, there are some blocker issues. Thanks, great! And amazingly quick! 1) There are a lot of places where you changed strings to hexencs. This makes the substantial changes very hard to find. I'd actually like you to have gone further, if possible, and used the DECORATE types wherever possible (revision_id etc) -- but I think you should make those changes on mainline as a separate patch. Hm.. I didn't really realize I was doing that. Was much more like a side effect. I'll check if I can come up with separate patches (branches, that is). 2) On .experiment.encapsulation, selector completion is done totally differently. I'm not sure what prefix_matching_constraint() needs to turn into, but I'm sure it won't work in its present form after .e.e lands. Aha, I didn't realize that. I need to take a look at .e.e. I would like to call the pumpkin for landing .experiment.encapsulation before this or any other branch; I need only another couple days on it, and it's well past the point where large merges from mainline are tractable. Sorry. This also means, please *don't* do the string -> foo_id changes until after .e.e lands. That's fine. I'll try propagating from .e.e to that unhexification branch. 3) Zbigniew's performance issues need to be investigated before merging. Agreed. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] unhexification of revision hashes
Hi, Zack Weinberg wrote: I haven't looked at the code yet, but I suspect it is because IDs are now binary in the database but hexadecimal everywhere else in memory. Thus, we are converting from binary to hex all over the place, which hurts. Uh.. obviously you also got that mail, so I'm taking back all I said about "reply-to-all" and such... sorry for the noise. It *should* be possible to avoid doing that almost everywhere and get back the speed. However, the "almost" is important; the textual representation of revisions, rosters, and changesets has hex IDs in it. It might be that that alone costs so much performance that we're hosed. Yeah, I have hoped that the 10% saving in I/O bandwidth and OS caches would easily be worth the CPU power for hex conversions. Markus - how infeasible would it be to change in-memory *_id from hex to binary before merging this revision? I dunno, but I can give it a try tomorrow. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] unhexification of revision hashes
Hello Zbigniew, Zbigniew Zagórski wrote: I don't remember reasons for this change besides db compaction. Hey, thanks, that's great and pretty instant help. ;-) However I see a small a performance hit on windows: whathex 0.38 log --brief | 46 | 39 graph| 2| 0.8 ls branches | 7.7 | 6 (seconds, all > dev null, monotone db, nvm head) Should it be slower? Faster ? Are those test cases feasible ? Ehm.. in my theories, it should be faster, simply because there's less data to be transfered from rotating storage. At the cost of some encode_hexenc() and decode_hexenc() calls. BTW. What are the cxxflags of official windows build, my build is "-O2 -g -Wall" stripped, maybe this perf hit is caused by bad flags. PS. I see ~10 size loss on two databases (156->137 monotone, 9-7.6 private db) so this db compaction is real. Last I checked, I saw very small performance improvements, so these numbers are looking suspicious to me. Can you recheck the CXX flags, please? BTW: please don't drop the mailing list and reply to all. Or can the list admin make the mailing list set a "reply-to" header, or something? Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] unhexification of revision hashes
On Jan 28, 2008 12:36 PM, Markus Schiltknecht <[EMAIL PROTECTED]> wrote: > I've just committed a revision 47dc584d4ca8799f686b20ce9bf5c59eb69f6d3c > into branch n.v.m.experiment.db-compaction. Todays fixes made it pass > all unit and lua tests, and together with the addition to NEWS, I now > consider that revisions to be ready for landing on mainline. Please review. Ok, I've read this code over. I like it, but unfortunately, there are some blocker issues. 1) There are a lot of places where you changed strings to hexencs. This makes the substantial changes very hard to find. I'd actually like you to have gone further, if possible, and used the DECORATE types wherever possible (revision_id etc) -- but I think you should make those changes on mainline as a separate patch. Because of this, I can't promise that I didn't miss some other glaring problem that will require a third go-round once we can see the smaller diff. 2) On .experiment.encapsulation, selector completion is done totally differently. I'm not sure what prefix_matching_constraint() needs to turn into, but I'm sure it won't work in its present form after .e.e lands. I would like to call the pumpkin for landing .experiment.encapsulation before this or any other branch; I need only another couple days on it, and it's well past the point where large merges from mainline are tractable. Sorry. This also means, please *don't* do the string -> foo_id changes until after .e.e lands. 3) Zbigniew's performance issues need to be investigated before merging. zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] unhexification of revision hashes
On Jan 28, 2008 4:15 PM, Zbigniew Zagórski <[EMAIL PROTECTED]> wrote: > I don't remember reasons for this change besides db compaction. > However I see a small a performance hit on windows: > whathex 0.38 > > log --brief | 46 | 39 > graph| 2| 0.8 > ls branches | 7.7 | 6 > > (seconds, all > dev null, monotone db, nvm head) > > Should it be slower? Faster ? Are those test cases feasible ? I haven't looked at the code yet, but I suspect it is because IDs are now binary in the database but hexadecimal everywhere else in memory. Thus, we are converting from binary to hex all over the place, which hurts. It *should* be possible to avoid doing that almost everywhere and get back the speed. However, the "almost" is important; the textual representation of revisions, rosters, and changesets has hex IDs in it. It might be that that alone costs so much performance that we're hosed. Markus - how infeasible would it be to change in-memory *_id from hex to binary before merging this revision? zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] unhexification of revision hashes
2008/1/28, Markus Schiltknecht <[EMAIL PROTECTED]>: > Hi all, > ... > > I've just committed a revision 47dc584d4ca8799f686b20ce9bf5c59eb69f6d3c > into branch n.v.m.experiment.db-compaction. Todays fixes made it pass > all unit and lua tests, and together with the addition to NEWS, I now > consider that revisions to be ready for landing on mainline. Please review. Hi, I don't remember reasons for this change besides db compaction. However I see a small a performance hit on windows: whathex 0.38 log --brief | 46 | 39 graph| 2| 0.8 ls branches | 7.7 | 6 (seconds, all > dev null, monotone db, nvm head) Should it be slower? Faster ? Are those test cases feasible ? BTW. What are the cxxflags of official windows build, my build is "-O2 -g -Wall" stripped, maybe this perf hit is caused by bad flags. PS. I see ~10 size loss on two databases (156->137 monotone, 9-7.6 private db) so this db compaction is real. Regarding review I'm not familiar with codebase but i'll try. Regards, -- Zbigniew -zbigg- Zagórski / software developer / geek / happy daddy / ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
[Monotone-devel] unhexification of revision hashes
Hi all, as Graydon has officially left the project now, I thought we'd better change into the next gear to keep the project healthy and alive. I've just committed a revision 47dc584d4ca8799f686b20ce9bf5c59eb69f6d3c into branch n.v.m.experiment.db-compaction. Todays fixes made it pass all unit and lua tests, and together with the addition to NEWS, I now consider that revisions to be ready for landing on mainline. Please review. In short, I've changed the database layer to store most, if not all, sha1 hashes in binary, instead of a hexadecimal ASCII compliant notation. This saves a few bytes in your sqlite database and it flattens the way to use binary revision_ids internally as well (which would save us some encode_hexenc() and decode_hexenc() calls). A schema_migration function is provided, but could use some testing. I don't want to break people's databases. Please note that - for some silly reason, don't ask me why - I didn't open up two separate branches, thus there's another head revision in that branch. Regards Markus ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
