Re: [basex-talk] UPDINDEX and ever growing index size

2014-08-15 Thread James Ball
Hi Christian, Thank you for the snapshot — run a few thousand documents in over a number of tests and everything is working. I’ll try to run some longer tests with a more varied data set over the next few days but I’m not expecting that to show anything different. Thank you for all the help.

Re: [basex-talk] UPDINDEX and ever growing index size

2014-08-13 Thread Christian Grün
Hi James, I've just updated the latest snapshot: could you give it another try? Christian __ > On Thu, Aug 7, 2014 at 2:26 AM, James Ball > wrote: >> >> Hi Christian, >> >> I've been following along with your snapshots and I think there's a bug >> still remai

Re: [basex-talk] UPDINDEX and ever growing index size

2014-08-07 Thread Christian Grün
Oh dear, that's not how flagship development looks like... Sorry for that, and thanks for the command scripts. The bug (I believe it's the very same) should be fixed soon. Christian On Thu, Aug 7, 2014 at 2:26 AM, James Ball wrote: > Hi Christian, > > I've been following along with your snaps

Re: [basex-talk] UPDINDEX and ever growing index size

2014-08-06 Thread James Ball
Hi Christian, I’ve been following along with your snapshots and I think there’s a bug still remaining somewhere but I can’t work out what’s causing it which is why I haven’t been back to you. If you use the GUI and the database is open in the GUI then everything works correctly. I can add, rep

Re: [basex-talk] UPDINDEX and ever growing index size

2014-08-04 Thread Christian Grün
A last one for today: I have just uploaded another snapshot which should speed up index updates. Looking forward to your feedback, Christian On Mon, Aug 4, 2014 at 4:01 PM, Christian Grün wrote: > The bug was hidden well [1], but it should be fixed now. Could you > check out the latest snapsho

Re: [basex-talk] UPDINDEX and ever growing index size

2014-08-04 Thread Christian Grün
The bug was hidden well [1], but it should be fixed now. Could you check out the latest snapshot? Christian [1] https://github.com/BaseXdb/basex/commit/429585ce26fca98d124d78fb88216ad7317c52fa On Mon, Aug 4, 2014 at 2:41 PM, Christian Grün wrote: > Hi James, > > I've found a little example fo

Re: [basex-talk] UPDINDEX and ever growing index size

2014-08-04 Thread Christian Grün
Hi James, I've found a little example for the bug (see attached). Sorry for the inconvenience; I'm working on a fix. Christian On Mon, Aug 4, 2014 at 11:30 AM, Christian Grün wrote: > James, > > thanks for testing. We have a bunch of test cases that succeeded for > the rewritten index handl

Re: [basex-talk] UPDINDEX and ever growing index size

2014-08-04 Thread Christian Grün
James, thanks for testing. We have a bunch of test cases that succeeded for the rewritten index handling, but as it seems, we definitely need some more. I'm pretty sure it's a single bug that causes all the error messages (because the code is in itself pretty straightforward), so I would be glad i

Re: [basex-talk] UPDINDEX and ever growing index size

2014-08-04 Thread James Ball
Hi Christian, > I’m glad to tell you that I have now implemented the projected optimizations Thank you for providing the snapshot. I’ve downloaded it and begun running some tests. Unfortunately I’m immediately finding some odd behaviour. I’m using the script I provided in my original issue rep

Re: [basex-talk] UPDINDEX and ever growing index size

2014-08-01 Thread Christian Grün
Hi James, I'm glad to tell you that I have now implemented the projected optimizations: 1. While a database is open, freed slots in the index heap file will now be remembered and refilled with new texts. If this approach proves to be successful, we might make this free slot structure persistent s

Re: [basex-talk] UPDINDEX and ever growing index size

2014-08-01 Thread James Ball
Hi Christian, Thank you for this - looks very promising. I was also having a think and wondered if, assuming a full fix is difficult, a special optimising function would be fast and easy. Instead of rebuilding the index content by reading the database just rebuild the files eliminating the fre

Re: [basex-talk] UPDINDEX and ever growing index size

2014-07-30 Thread Christian Grün
Hi James, I had some first thoughts on possible optimizations for the increasing file size problem, and I may have found a fairly easy solution that covers some of the current problems. It's not implemented yet, but I could at least fix the initial 4096 byte problem [1]. I'll keep you updated, Ch

Re: [basex-talk] UPDINDEX and ever growing index size

2014-07-19 Thread Christian Grün
Hi James, > However the behaviour is different when using db:replace. I think it's doing > a db:delete() and then a db:add(). So first the index file has the ID list > for that attribute value rewritten in place (so the count will go from 2048 > to 2047 for example) with a new value for count a

Re: [basex-talk] UPDINDEX and ever growing index size

2014-07-18 Thread James Ball
Hi Christian, Thank you for coming back so quickly. > One way out (until this has been fixed) is to optimize these databases > in regular time intervals. I’ve been doing this on one of my databases and it does work - it’s just another thing to remember to do! It’s a large database and the inde

Re: [basex-talk] UPDINDEX and ever growing index size

2014-07-15 Thread Christian Grün
Hi James, > The issue I'm seeing is that the size of the index grows by approximately 1MB > with every updating 'transaction' (snapshot?) even if there is no new data > for the index. For example if I have a database with 100,000 files and I > replace one of those files (with itself so there's

[basex-talk] UPDINDEX and ever growing index size

2014-07-14 Thread James Ball
Hello, I’ve finally had some time to look at an issue I’ve been having with databases that have UPDINDEX set to true. I’m now running a BaseX 8.0 beta and was on 7.X when I first encountered this. The issue I’m seeing is that the size of the index grows by approximately 1MB with every updating