Re: initial page allocation versus incremental allocation
Brian Peterson publicay...@verizon.net writes: Won't this still limit the setting to the MAX_PRE_ALLOC_SIZE? Why have the maximum setting? Or, at least, why limit it to just 8 pages? MAX_PRE_ALLOC_SIZE is 1000, it's the default that is just 8 pages, as far as I can see. (And no, I don't know why it is limited to 1000 pages, but I wouldn't expect many objections if a request to increase the limit was filed in JIRA - https://issues.apache.org/jira/browse/DERBY) -- Knut Anders
Re: initial page allocation versus incremental allocation
Whoever decides to work on this should do more than just document, as these params are not tested. They were added in the past for ad-hoc testing and performance debugging. As has been suggested they were never fully implemented/documented as they did not fit in with the original goal of the product - ie. be a zero-admin embedded db for smaller devices. The idea, if they were used at all was for the software to automatically determine what the best settings and up them dynamically rather than force/allow user to do so. As I remember the defaults have something to do with the following: o Start with no preallocation as originally users actually would notice if a 2k page table started out with 16k vs 2k for an empty table. o default to 8 as again the standard user was space limited and did not want to possibly waste space. o 1000 is just a round number, but if it actually gets documented I would see if the code can actually support a number that is more than the number of pages that can be tracked by a single allocation page (or maybe 2). This number is dependent on page sizes, so worst case would be allocation on a 2k page table. Should just add some testing if it is to be supported. Knut Anders Hatlen wrote: Brian Peterson publicay...@verizon.net writes: Won't this still limit the setting to the MAX_PRE_ALLOC_SIZE? Why have the maximum setting? Or, at least, why limit it to just 8 pages? MAX_PRE_ALLOC_SIZE is 1000, it's the default that is just 8 pages, as far as I can see. (And no, I don't know why it is limited to 1000 pages, but I wouldn't expect many objections if a request to increase the limit was filed in JIRA - https://issues.apache.org/jira/browse/DERBY)
Re: initial page allocation versus incremental allocation
idea, if they were used at all was for the software to automatically determine what the best settings and up them dynamically rather than force/allow user to do so. Another idea would be to try to do something adaptive, along the lines of: - if the table is currently smaller than N pages, use an incremental allocation of 8 pages - if the table is larger than N pages, but smaller than 2*N pages, use an incremental allocation of 24 pages - if the table is larger than 2*N pages, but smaller than 10*N pages, use an incremental allocation of 128 pages - if the table is larger than 10*N pages, use an incremental allocation of 256 pages Or something like that. I guess I'm just trying to suggest that it might be possible to be a zero-admin embedded DB and still scale well at page allocation for very large tables. This discussion is probably best suited for derby-dev, and as Knut said it would be nice if there was a JIRA that tracked this for the future. thanks, bryan
Re: initial page allocation versus incremental allocation
Brian Peterson dianeay...@verizon.net writes: I see that there’s a property to allow configuring the number of pages to initially allocate to a table, derby.storage.initialPages, but there isn’t a property to allow for setting the number of pages to allocate when incrementally expanding the file container. It looks like RawStoreFactory might’ve allowed for this with public static final String PRE_ALLOCATE_PAGE = “derby.storage.pagePerAllocation”; but this isn’t reference by anything I can find. I haven't tested that it actually works, but it appears to be referenced in FileContainer.createInfoFromProp(): PreAllocSize = PropertyUtil.getServiceInt(tc, createArgs, RawStoreFactory.PRE_ALLOCATE_PAGE, MIN_PRE_ALLOC_SIZE, MAX_PRE_ALLOC_SIZE, DEFAULT_PRE_ALLOC_SIZE /* default */); If it turns out that setting the property works, we should probably try to get it into the documentation, as it looks like it could be useful. -- Knut Anders
RE: initial page allocation versus incremental allocation
Won't this still limit the setting to the MAX_PRE_ALLOC_SIZE? Why have the maximum setting? Or, at least, why limit it to just 8 pages? If it is configurable, then applications that need the option have it available, and those that don't can continue using the default settings. I'm surprised there's an option to configure the initial allocation but not the increment size. What I've been trying to do is use Derby in a client application that needs to download more data than will fit into heap. The objects get serialized into rows and are indexed by the row number and an additional java.util.UUID that identifies the object. The client gui can analyze the data and display the results and the raw data as required. The loading works great up to the initial allocation of pages, then it crawls, which cascades through and disrupts the whole application. It looks like I could solve this by configuring the page-increment size just like the initial allocation setting. I had hoped to use Derby as a kind of general purpose heap management solution. It has been doing a great job of limiting the amount of heap it uses, and it integrates well with the rest of the application, but the slowdown of the table loading is killing me. Brian -Original Message- From: knut.hat...@sun.com [mailto:knut.hat...@sun.com] Sent: Tuesday, March 03, 2009 3:28 AM To: Derby Discussion Subject: Re: initial page allocation versus incremental allocation Brian Peterson dianeay...@verizon.net writes: I see that there’s a property to allow configuring the number of pages to initially allocate to a table, derby.storage.initialPages, but there isn’t a property to allow for setting the number of pages to allocate when incrementally expanding the file container. It looks like RawStoreFactory might’ve allowed for this with public static final String PRE_ALLOCATE_PAGE = “derby.storage.pagePerAllocation”; but this isn’t reference by anything I can find. I haven't tested that it actually works, but it appears to be referenced in FileContainer.createInfoFromProp(): PreAllocSize = PropertyUtil.getServiceInt(tc, createArgs, RawStoreFactory.PRE_ALLOCATE_PAGE, MIN_PRE_ALLOC_SIZE, MAX_PRE_ALLOC_SIZE, DEFAULT_PRE_ALLOC_SIZE /* default */); If it turns out that setting the property works, we should probably try to get it into the documentation, as it looks like it could be useful. -- Knut Anders
RE: initial page allocation versus incremental allocation
I think that the simple answer is that Cloudscape wasn't designed as a general purpose relational database. Its target niche was embedded and smaller apps. So it sounds like the issue you were having is solved. That is, once you hit your preset size, the default allocations is too small and its impacting your ability to increase your table space. It sounds like an enhancement to me. BTW, if you want to 'hack' this to test the theory out, just change the DEFAULT_PRE_ALLOC_SIZE constant in the File Container to be something like 1000. Then run your test again. _ From: Brian Peterson [mailto:dianeay...@verizon.net] Sent: Monday, March 02, 2009 5:28 PM To: 'Derby Discussion' Subject: initial page allocation versus incremental allocation I see that there's a property to allow configuring the number of pages to initially allocate to a table, derby.storage.initialPages, but there isn't a property to allow for setting the number of pages to allocate when incrementally expanding the file container. It looks like RawStoreFactory might've allowed for this with public static final String PRE_ALLOCATE_PAGE = derby.storage.pagePerAllocation; but this isn't reference by anything I can find. FileContainer fixes the incremental expansion to 8 pages with the DEFAULT_PRE_ALLOC_SIZE constant. What was the reason for not allowing the pre-allocation setting to be configurable? Were there adverse affects on FileContainer if it was increased to something like 100 pages? Brian
