Yeah. It need to get increment on delivery. Maybe it would worth to re-think our design todo this directly in the underlying storage in "one step (if possible)". Currently we do this:
1) Get mailbox 2) increment lastuid 3) persist changes The problem here is that we need to make sure other threads will not read the mailbox until the persist step was done.. Bye, Norman 2010/5/17 Eric Charles <eric.char...@u-mangate.com>: > Hi Norman, > > As discussed, it may be worth to reread the RFC requirements: > I've copy/paste the RFC paragraph here after. > Can we deduce from that that the uid must be incremented by one for each > mail arriving in a mailbox? > Tks, > Eric > > 2.3.1.1. Unique Identifier (UID) Message Attribute > > A 32-bit value assigned to each message, which when used with the > unique identifier validity value (see below) forms a 64-bit value > that MUST NOT refer to any other message in the mailbox or any > subsequent mailbox with the same name forever. Unique identifiers > are assigned in a strictly ascending fashion in the mailbox; as each > message is added to the mailbox it is assigned a higher UID than the > message(s) which were added previously. Unlike message sequence > numbers, unique identifiers are not necessarily contiguous. > > The unique identifier of a message MUST NOT change during the > session, and SHOULD NOT change between sessions. Any change of > unique identifiers between sessions MUST be detectable using the > UIDVALIDITY mechanism discussed below. Persistent unique identifiers > are required for a client to resynchronize its state from a previous > session with the server (e.g., disconnected or offline access > clients); this is discussed further in [IMAP-DISC]. > > Associated with every mailbox are two values which aid in unique > identifier handling: the next unique identifier value and the unique > identifier validity value. > > The next unique identifier value is the predicted value that will be > assigned to a new message in the mailbox. Unless the unique > identifier validity also changes (see below), the next unique > identifier value MUST have the following two characteristics. First, > the next unique identifier value MUST NOT change unless new messages > are added to the mailbox; and second, the next unique identifier > value MUST change whenever new messages are added to the mailbox, > even if those new messages are subsequently expunged. > > Note: The next unique identifier value is intended to > provide a means for a client to determine whether any > messages have been delivered to the mailbox since the > previous time it checked this value. It is not intended to > provide any guarantee that any message will have this > unique identifier. A client can only assume, at the time > that it obtains the next unique identifier value, that > messages arriving after that time will have a UID greater > than or equal to that value. > > The unique identifier validity value is sent in a UIDVALIDITY > response code in an OK untagged response at mailbox selection time. > If unique identifiers from an earlier session fail to persist in this > session, the unique identifier validity value MUST be greater than > the one used in the earlier session. > > Note: Ideally, unique identifiers SHOULD persist at all > times. Although this specification recognizes that failure > to persist can be unavoidable in certain server > environments, it STRONGLY ENCOURAGES message store > implementation techniques that avoid this problem. For > example: > > 1) Unique identifiers MUST be strictly ascending in the > mailbox at all times. If the physical message store is > re-ordered by a non-IMAP agent, this requires that the > unique identifiers in the mailbox be regenerated, since > the former unique identifiers are no longer strictly > ascending as a result of the re-ordering. > > 2) If the message store has no mechanism to store unique > identifiers, it must regenerate unique identifiers at > each session, and each session must have a unique > UIDVALIDITY value. > > 3) If the mailbox is deleted and a new mailbox with the > same name is created at a later date, the server must > either keep track of unique identifiers from the > previous instance of the mailbox, or it must assign a > new UIDVALIDITY value to the new instance of the > mailbox. A good UIDVALIDITY value to use in this case > is a 32-bit representation of the creation date/time of > the mailbox. It is alright to use a constant such as > 1, but only if it guaranteed that unique identifiers > will never be reused, even in the case of a mailbox > being deleted (or renamed) and a new mailbox by the > same name created at some future time. > > 4) The combination of mailbox name, UIDVALIDITY, and UID > must refer to a single immutable message on that server > forever. In particular, the internal date, [RFC-2822 > <http://www.faqs.org/rfcs/rfc2822.html>] > size, envelope, body structure, and message texts > (RFC822 <http://www.faqs.org/rfcs/rfc822.html>,RFC822 > <http://www.faqs.org/rfcs/rfc822.html>.HEADER,RFC822 > <http://www.faqs.org/rfcs/rfc822.html>.TEXT, and all BODY[...] > fetch data items) must never change. This does not > include message numbers, nor does it include attributes > that can be set by a STORE command (e.g., FLAGS). > > 2.3.1.2. Message Sequence Number Message Attribute > > A relative position from 1 to the number of messages in the mailbox. > This position MUST be ordered by ascending unique identifier. As > each new message is added, it is assigned a message sequence number > that is 1 higher than the number of messages in the mailbox before > that new message was added. > > Message sequence numbers can be reassigned during the session. For > example, when a message is permanently removed (expunged) from the > mailbox, the message sequence number for all subsequent messages is > decremented. The number of messages in the mailbox is also > decremented. Similarly, a new message can be assigned a message > sequence number that was once held by some other message prior to an > expunge. > > In addition to accessing messages by relative position in the > mailbox, message sequence numbers can be used in mathematical > calculations. For example, if an untagged "11 EXISTS" is received, > and previously an untagged "8 EXISTS" was received, three new > messages have arrived with message sequence numbers of 9, 10, and 11. > Another example, if message 287 in a 523 message mailbox has UID > 12345, there are exactly 286 messages which have lesser UIDs and 236 > messages which have greater UIDs. > > > > Read more: http://www.faqs.org/rfcs/rfc3501.html#ixzz0oDUA57mw > > > Read more: http://www.faqs.org/rfcs/rfc3501.html#ixzz0oDTtT7f8 > > > On 05/12/2010 07:46 AM, Norman Maurer wrote: >> >> Hi Eric, >> >> thx to follow up on this. Comments inside >> >> 2010/5/11 Eric Charles<eric.char...@u-mangate.com>: >> >>> >>> Hi, >>> >>> Section 2.3.1.1 of IMAP RFC (http://www.faqs.org/rfcs/rfc3501.html) >>> states >>> that Unique identifiers MUST be strictly ascending in the mailbox at all >>> times. >>> This is currently enforced in >>> org.apache.james.imap.store.mail.model.Mailbox#consumeUid() for the JPA >>> store. >>> >>> A JPAStressTest has been setup by Norman to verify that the uid were >>> correctly generated. >>> This showed that parallel threads could give issues in critical section >>> of >>> org.apache.james.imap.jpa.JPAMailbox#reserveNextUid(mailboxSession) (the >>> section between the transaction begin and commit). >>> >> >> Which brings me to a JCRStressTest which fails :/ >> >> >>> >>> After chatting and patching with Norman, at least 4 strategies are >>> identified: >>> >>> 1. Use locking specific mecanism of each store. This is what is actually >>> implemented for database store (via LockModeType.PESSIMISTIC_WRITE and >>> adequate timeout) on JPAMaibox. The JCR store would need something >>> similar. >>> Simply document the reserveNextUid to indicate that a locking mecanism >>> should be implemented. Currently, this solution works for JPA and uid are >>> correctly generated. >>> >> >> Going this way would allow to use the "most" performant solation per >> implementation. This would also allow us to support clustering in JCR >> etc. That would not be possible when using a "JVM lock". >> >> Locking in JCR is an other problem, but I will do some more research >> first before go into the details >> >> >> >>> >>> 2. Implement some ReentrantLock (or equivalent synchronization) an >>> abstraction level higher. All specific implementations would benefit from >>> this mecanism. Lock would be for all maiboxes. >>> >> >> See above... >> >> >>> >>> 3. Change in a way the api (change >>> org.apache.james.imap.mailbox.Mailbox#appendMessage signature for >>> example) >>> to oblige each implementation to have a threadsafe way of generating the >>> uid. >>> >> >> What is different here from what we have atm in the abstract method of >> StoreMailbox ? >> >> >>> >>> 4. Design and implement a more evolved solution that would, per mailbox, >>> maintain the lastuid and queue all uid generation request per mailbox. >>> >>> >> >> Could you give me some more details about how you think this could be >> done ? At the moment I think about adding an interface called >> UidConsumer. Which only has one method like: >> >> long reserveNextUid(Mailbox mailbox, MailboxSession session); >> >> The instance of the Consumer would get instances in the >> StoreMailboxManager and then passed to the StoreMailbox in the >> constructor. So it would be easy for developers to provide their own >> strategy. >> >> >> >>> >>> So the question is "to which strategy should be go" ? >>> >>> Eric >>> >>> >>> >> >> Bye, >> Norman >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> For additional commands, e-mail: server-dev-h...@james.apache.org >> >> >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
