I just think we should not force the uidnext to match the next uid of the insert message.. thats all ;)
Bye, Norman 2010/8/17 Eric Charles <e...@apache.org>: > So, kind-of "let the store create the uid (range) and use it as nextuid" > > In current impl, each mail persistence reads in a locking transaction the > persisted "lastUid" (for jpa), consumes it (+1) and re-persists it. > Each incoming mail queues for this thanks to the locking transaction, giving > performance limitation. > > But we don't want serial, but parallel processing... > > If we let the individual store create the UID for each mail insert > (different threads), I suppose we will still rely on a unique component > responsible to give the UIDNEXT. > This is why I introduced the notion of "cache", but it could be any other > mechanism. > > My point is that if store (db,...) creates the UID (range), it must be > provided back to the imap client reading the mailbox in a managed way. > > So, you told us how you see the UID generation with the auto_increment (for > the database store for example). > Could you sketch us how yo see the UIDNEXT management/generation for a > mailbox, with parallel mails being inserted (and their UID being > automatically generated by the store) ? > > Tks, > > Eric > > > On 17/08/2010 19:40, Norman Maurer wrote: >> >> Well, >> >> I think its not the generation of the UIDNEXT which is a performance >> problem. Its how it is used atm. We currently use it as uid for the >> next message which will get append to the mailbox. It would be more >> performant to use an auto_increment column in jpa for example. Other >> backends have other features which can match the uid generation stuff. >> >> Bye, >> Norman >> >> 2010/8/17 Tim-Christian Mundt<d...@tim-erwin.de>: >>> >>> Eric, >>> >>> you are right about the UIDVALIDITY, the default shouldn't be a random >>> number, but the current timestamp which would guarantee that it won't >>> occur again. >>> I also thought about checking whether the next uid would wrap the uid >>> counter to the negative - which would mean we need to regenerate the >>> uids and create a new UIDVALIDITY. However, it's just to improbable to >>> have more then 9 quintillion mails arriving in a mailbox. >>> >>> UIDVALIDITY + UID as id or not simply means: the combination MUST never >>> refer to any other mail - ever. If for some reason you have to >>> regenerate the uids or do so by default, the UIDVALIDITY MUST change - >>> making the previous uids invalid. The "refer forever" part is in a >>> "STRONGLY ENCOURAGES" section. >>> >>> Caching NEXTUIDS etc sounds really complicated for a simple matter. I'll >>> try to understand why this is such a performance blocker. It's just >>> reading or writing a value, isn't it? Later... >>> >>> Regards >>> Tim >>> >>> Am Dienstag, den 17.08.2010, 17:56 +0200 schrieb Eric Charles: >>>> >>>> Hi Norman, >>>> >>>> I've read the http://www.rfc-editor.org/rfc/rfc3501.txt (section 2.3.1 >>>> Message Numbers) and http://www.rfc-editor.org/rfc/rfc2683.txt (section >>>> 3.4.3. UIDs and UIDVALIDITY) >>>> >>>> A first point is RFC talks about backend server not being able to store >>>> the UIDs. In this case, the UID are to be regenerated each time, with a >>>> different UIDVALIDITY, so a there are no risk to confuse mails. I had >>>> also to read twice the sentence and associated explanation "It seems to >>>> be a common misunderstanding that "the UIDVALIDITY and the UID, taken >>>> together, form a 64-bit identifier that uniquely identifies a message on >>>> a server" ." However, it is said at another place : "The combination of >>>> mailbox name, UIDVALIDITY, and UID must refer to a single immutable >>>> message on that server forever". >>>> >>>> Finally, this may give as requirement that the store API should not >>>> prevent to implement a store that wouldn't be capable of storing UIDs >>>> (seems strange, but considered at numerous places in RFCs). >>>> I think the current store API already allows that ? >>>> >>>> On the UIDVALIDITY, it is now generated for example in JPA with a >>>> Math.abs(RANDOM.nextInt()). RFC states: "A good UIDVALIDITY value to use >>>> in this case is a 32-bit representation of the creation date/time of the >>>> mailbox". >>>> It seems reasonable to provide utility methods such as existing >>>> randomUidValidity() to the store impl, each store having the freedom to >>>> use it or not. >>>> (was just wondering what is the difference between the imap-mailbox and >>>> imap-store projects - not always obvious at first sight to define the >>>> responsibilities of each). >>>> >>>> Coming to the UIDNEXT and as you pointed, I also understand that the >>>> returned UIDNEXT value has nothing to do with the UID that will be given >>>> to the next coming message. That value needs however to be equals or >>>> higher. >>>> I suppose the idea would be to have per mailbox a cache in memory. That >>>> cache would be used to return the UIDNEXT (that would be the current >>>> cache value), but also to assign the UID for coming mails (cache+1). >>>> I am wondering how we can ensure in case of abrupt shutdown that the >>>> last value of the cache be stored. >>>> If we can't ensure that, and this will be probably the case, there we an >>>> have a strategy to init the cache with a value recomputed from the all >>>> the stored UID (something like "give me the highest value from all the >>>> UID of that mailbox). >>>> This would need an initial step when the cache is not initialized but >>>> should not be a penality for a JPA store, even for mailbox with many >>>> mails. Don't know for the other stores (jcr, maildir,...) ? >>>> Should we care on the cache time-to-live? If we don't care about that, >>>> we will have a growing memory, even if for each mailbox, we only need >>>> an Integer (that wouldn't represent much KB). But there is also the >>>> possibility to define a ttl of a few hours, with a scheduled cache >>>> manager that would cleanup things. >>>> >>>> Tks, >>>> >>>> Eric >>>> >>>> >>>> On 15/08/2010 17:25, Norman Maurer wrote: >>>>> >>>>> Hi there, >>>>> >>>>> After looking a bit over the store api again the last days I think >>>>> there is some room for improvements. This improvements will break the >>>>> api (again), so I think we should do it now and after that cut the 0.1 >>>>> release. I will try to explain you why I think there should be some >>>>> improvements made and whats my point of view. Please feel free to >>>>> comment .. >>>>> >>>>> NEXTUID (IMAP-193): >>>>> The NEXTUID generation / house-keeping is just a big performance >>>>> killer. We really guaranteer to use the value of NEXTUID for the next >>>>> message which will get saved. Thats not needed. We just need to >>>>> guaranteer its equal or greater then the value returned by NEXTUID. So >>>>> its prolly more performant to just hold the informations in memory and >>>>> update it every x writes (or something like that). So the >>>>> implementation could use an auto-increment field to generate the >>>>> unique uid when storing the message or just an AtomicInteger for >>>>> generation. Maybe again with a new abstract class called UIDKeeper ? >>>>> >>>>> Does this sound like something which make sense ? >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>> For additional commands, e-mail: server-dev-h...@james.apache.org >>> >>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> For additional commands, e-mail: server-dev-h...@james.apache.org >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
