Re: RV-Predict bugs
Sorry about the vague explanation. But the actual reasons are not the point here. The only thing matters is the Java memory model. And the article I refer to explain exactly why and how this can happen. I agree that such non-thread-safe lazy initialization rarely causes problem. But, guys, please take a look at other issues I reported. There are much more interesting concurrency bugs. I am reporting these minor issues only because they are clearly bugs. Mark, if you have a specific concurrency issue you would like to investigate, the best way is to run RV-Predict against the code that once revealed the problem. Even if the failure occurs rarely, RV-Predict may be able to identify the cause for you. I get all these race reports by running RV-Predict against the test suite so if the small unit test doesn't exercise the problematic code enough you may not get the specific answer you are looking for. Yilong On Sun, Sep 13, 2015 at 1:30 PM, Mark Thomas wrote: > On 13/09/2015 19:02, Rémy Maucherat wrote: > > 2015-09-13 18:45 GMT+02:00 Mark Thomas : > > > >> Yilong, > >> > >> You need to be subscribed to the dev list in order to post to it. > >> > >> On 13/09/2015 14:08, Yilong Li wrote: > >>> Hi Mark, > >>> > >>> On Sun, Sep 13, 2015 at 4:08 AM, Mark Thomas >>> <mailto:ma...@apache.org>> wrote: > >>> > >>> Please do not add any more RV-Predict bug reports to Bugzilla until > >> the > >>> current set have been evaluated. To be honest it would have been > >> better > >>> if you had paused after adding 58367-58379. > >>> > >>> > >>> I am going to stop here because these are all the bugs reported by > >>> RV-Predict in one run of the entire test suite. > >> > >> OK. Lets resolve these and then see where we stand. > >> > >> > > As far as I am concerned, I would prefer batch closing these "issues" > > nobody ever had. > > That is very tempting. > > Looking at the articles by the folks that are the experts in the field > (i.e. the authors of JSR-133) it looks like the issues raised are valid > - even if we don't see then very often / at all. Despite that, I'd be a > lot happier with a clearer explanation of what can go wrong and why/how. > > On the plus side, the issues I've looked at so far have provided > opportunities for clean-up in trunk. > > I'm hoping (but haven't look at all of them yet) that at least one of > these reports is going to identify the root cause of one of those hard > to reproduce / hard to track down concurrency issues. > > Mark > > > - > To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org > For additional commands, e-mail: dev-h...@tomcat.apache.org > >
Re: svn commit: r1703194 - /tomcat/trunk/java/org/apache/tomcat/websocket/WsWebSocketContainer.java
I am not sure declaring this field as volatile is the right way to fix it because the increment is still not atomic. If this counter doesn't have to be precise, I think it's OK to allow data races on this field. Otherwise, it should be declared as atomic. Yilong On Tue, Sep 15, 2015 at 6:43 AM, wrote: > Author: markt > Date: Tue Sep 15 13:43:55 2015 > New Revision: 1703194 > > URL: http://svn.apache.org/r1703194 > Log: > Fix https://bz.apache.org/bugzilla/show_bug.cgi?id=58384 > Calls to backgroundProcess() may come from different threads > > Modified: > tomcat/trunk/java/org/apache/tomcat/websocket/WsWebSocketContainer.java > > Modified: > tomcat/trunk/java/org/apache/tomcat/websocket/WsWebSocketContainer.java > URL: > http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/tomcat/websocket/WsWebSocketContainer.java?rev=1703194&r1=1703193&r2=1703194&view=diff > > == > --- > tomcat/trunk/java/org/apache/tomcat/websocket/WsWebSocketContainer.java > (original) > +++ > tomcat/trunk/java/org/apache/tomcat/websocket/WsWebSocketContainer.java Tue > Sep 15 13:43:55 2015 > @@ -92,7 +92,7 @@ public class WsWebSocketContainer implem > private int maxBinaryMessageBufferSize = > Constants.DEFAULT_BUFFER_SIZE; > private int maxTextMessageBufferSize = Constants.DEFAULT_BUFFER_SIZE; > private volatile long defaultMaxSessionIdleTimeout = 0; > -private int backgroundProcessCount = 0; > +private volatile int backgroundProcessCount = 0; > private int processPeriod = Constants.DEFAULT_PROCESS_PERIOD; > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org > For additional commands, e-mail: dev-h...@tomcat.apache.org > >
Re: RV-Predict bugs
On Tue, Sep 15, 2015 at 7:19 AM, Mark Thomas wrote:
> On 13/09/2015 21:59, Yilong Li wrote:
> > Sorry about the vague explanation. But the actual reasons are not the
> point
> > here.
>
> No, that is exactly the point. When you claim that something that
> appears to be working without issue for many, many users then you need
> to provide a justification for why there is an issue to back up the
> claims you are making.
>
> > The only thing matters is the Java memory model. And the article I
> > refer to explain exactly why and how this can happen.
>
> No, it didn't. I read that article (and parts of the JMM). While the
> article did state there was a problem it did not explain why there was a
> problem.
>
> Long experience has lead us to be sceptical of bugs reported by
> automated tools. When looking at such bugs and we can't see what the
> problem is and the person raising the bug can't provide a clear
> explanation - and continues not to provide a clear explanation when
> asked for one several times - then that is the point where we start to
> think the problem is with the tool.
>
Sorry, I should not assume that the concepts such as happens-before order
and memory model are well understood. Let's talk about how this is allowed
to happen under JMM. Consider the this example again:
if (st_200 == null ) {
st_200 = sm.getString("sc.200");
}
return st_200;
The following is a valid execution trace consists of 5 events:
T1 T2
1 READ null
2 WRITE s
3 READ s
4 READ ???
5 READ s
, where s is the result of sm.getString("sc.200").
T1 sees null in field st_200, initializes it, and return the initialized
value, while T2 sees a non-null value in st_200 and skips the
initialization. The question is what value T2 can read and then return in
event 4. This is addressed in JLS $17.4.5 Happens-before Order (
https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4.5):
We say that a read *r* of a variable *v* is allowed to observe a write *w*
to *v* if, in the *happens-before* partial order of the execution trace:
-
*r* is not ordered before *w* (i.e., it is not the case that *hb(r, w)*),
and
-
there is no intervening write *w*' to *v* (i.e. no write *w*' to *v* such
that *hb(w, w')* and *hb(w', r)*).
Informally, a read *r* is allowed to see the result of a write *w* if there
is no *happens-before* ordering to prevent that read.
The question boils down to: does the write in event 2 prevent event 4 from
reading the initial null value of st_200? No, because there is no
happens-before order involved here. So what kind of constructs introduce
happens-before order? This question is also addressed in the same section
of JLS:
It follows from the above definitions that:
-
An unlock on a monitor *happens-before* every subsequent lock on that
monitor.
-
A write to a volatile field (§8.3.1.4
<https://docs.oracle.com/javase/specs/jls/se8/html/jls-8.html#jls-8.3.1.4>
) *happens-before* every subsequent read of that field.
-
A call to start() on a thread *happens-before* any actions in the
started thread.
-
All actions in a thread *happen-before* any other thread successfully
returns from a join() on that thread.
-
The default initialization of any object *happens-before* any other
actions (other than default-writes) of a program.
> I agree that such non-thread-safe lazy initialization rarely causes
> > problem. But, guys,
>
> 'guys' is not a gender neutral word. Do you really mean to suggest that
> only the male committers look at the other issues? I tend to avoid such
> phraseology where I can and use something like 'all' or 'folks' if
> necessary.
>
I apologize but 'guys' can be gender neutral which is what I meant.
> > please take a look at other issues I reported. There
> > are much more interesting concurrency bugs.
>
> Yes, there are a few more interesting / likely to have an impact bugs.
> The tribes issues I've fixed today look like they were reasonably likely
> to cause issues.
>
> > I am reporting these minor
> > issues only because they are clearly bugs.
>
> They are only clearly bugs once a clear explanation of the concurrency
> issue that underlies many of them has been explained clearly and
> understood.
>
> Opening a large number of bugs (even valid ones) without a clear
> justification is not the way to win friends and influence people. It
> would be far better to open a single, well explained bug, get that
> accepted and fixed and then tell the community that the tool has
> identified more bugs and ask the community how the
Re: RV-Predict bugs
On Tue, Sep 15, 2015 at 10:58 AM, Caldarale, Charles R <
chuck.caldar...@unisys.com> wrote:
> > From: Yilong Li [mailto:yilong...@runtimeverification.com]
> > Subject: Re: RV-Predict bugs
>
> > The following is a valid execution trace consists of 5 events:
> > T1 T2
> > 1 READ null
> > 2 WRITE s
> > 3 READ s
> > 4 READ ???
> > 5 READ s
>
> > where s is the result of sm.getString("sc.200").
>
> > The question is what value T2 can read and then return in event 4.
>
> It's not clear what the READ ??? is intended to represent.
>
I put ??? there because I am trying to reason about what value this read
can see. Sorry for the confusion.
>
> > The question boils down to: does the write in event 2 prevent event 4
> from
> > reading the initial null value of st_200?
>
> It doesn't, but that's not the real problem.
Well, it is the problem (at least part of it) because JLS says "Informally,
a read *r* is allowed to see the result of a write *w* if there is no
*happens-before* ordering to prevent that read." In this case, there is no
HB ordering preventing event 3 to read the initial null value put by JVM.
> Let's expand the example a bit:
>
> T1 T2
> 1 READ null from st_200
> 2a allocate obj
> 2b WRITE obj->field
> 2c WRITE obj into st_200
> 3 READ obj from st_200
> 4 READ obj->field
>
> Since there is no actual ordering of steps 2b (object initialization) and
> 2c (object publication) in T1, T2 can observe them in the reverse order and
> pick up an unitialized value for field.
>
OK, now I see what you are talking about. But consider the following
hashcode example:
public int hashCode() {
if (hashCode == 0) {
hashCode = computeHashCode(); // no object creation involved
}
return hashCode;
}
There is no object initialization/publication involved at all but the
problem still exists: this method could return 0 even if the if-statement
sees a non-zero value.
Yilong
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail and
> its attachments from all computers.
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>
Re: RV-Predict bugs
On Tue, Sep 15, 2015 at 11:32 AM, Caldarale, Charles R <
chuck.caldar...@unisys.com> wrote:
> > From: Yilong Li [mailto:yilong...@runtimeverification.com]
> > Subject: Re: RV-Predict bugs
>
> > Well, it is the problem (at least part of it) because JLS says
> "Informally,
> > a read *r* is allowed to see the result of a write *w* if there is no
> > *happens-before* ordering to prevent that read." In this case, there is
> no
> > HB ordering preventing event 3 to read the initial null value put by JVM.
>
> True, but as Mark previously pointed out, no one cares. All that happens
> in that case is the object is redundantly created and then garbage
> collected later; no damage.
>
No, it's not right. It's possible (although very unlikely) that this method
could return null and results in a null pointer exception later.
>
> > But consider the following hashcode example:
> > public int hashCode() {
> > if (hashCode == 0) {
> > hashCode = computeHashCode(); // no object creation involved
> > }
> > return hashCode;
> > }
>
> > There is no object initialization/publication involved at all but the
> > problem still exists: this method could return 0 even if the if-statement
> > sees a non-zero value.
>
> No, it can't (unless computeHashCode() returns a zero, which presumably it
> won't). This is all action within a single thread, and as such must obey
> intra-thread semantics (i.e., statements are executed in control flow
> order).
>
Yes, it can. You probably misunderstand intra-thread semantics.
Intra-thread semantics says that event 3 (the first read of the field
hashCode) happens before event 4 (the second read of field hashCode) but it
has no effect on the value event 4 can see (because event 3 is also a read
rather than a write).
Yilong
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail and
> its attachments from all computers.
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>
Re: RV-Predict bugs
On Tue, Sep 15, 2015 at 12:06 PM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: Yilong Li [mailto:yilong...@runtimeverification.com] > > Subject: Re: RV-Predict bugs > > > > True, but as Mark previously pointed out, no one cares. All that > happens > > > in that case is the object is redundantly created and then garbage > > > collected later; no damage. > > > No, it's not right. It's possible (although very unlikely) that this > method > > could return null and results in a null pointer exception later. > > Again, that cannot happen; see below. > > > Intra-thread semantics says that event 3 (the first read of the field > > hashCode) happens before event 4 (the second read of field hashCode) but > it > > has no effect on the value event 4 can see (because event 3 is also a > read > > rather than a write). > > You're mixing up inter- and intra-thread semantics and completely ignoring > the write to hashCode when the initially observed value is zero. > Intra-thread semantics require that operations occur in program order, so > the initial zero (or null) check results in a non-zero (or non-null) write > to the field of interest; control flow cannot be ignored here. If your > supposition were correct, a compiler (or CPU) would be free to migrate > actions dependent on a conditional block to a spot prior to that > conditional block - clearly that is nonsense. Nope, I know what I am doing. Let's first see what the expert says. Please check out the last section in this article ( http://jeremymanson.blogspot.com/2008/12/benign-data-races-in-java.html) which starts with "let's break the code". Here, we are talking about T2 in the above example. The second read of the field hashCode is not control-flow-dependent on the first read (the zero-check). Again, intra-thread semantics introduces HB order between the event 3 and event 4. But it doesn't ensure that event 4 must see the same value as event 3. Yilong > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you > received this in error, please contact the sender and delete the e-mail and > its attachments from all computers. > > > - > To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org > For additional commands, e-mail: dev-h...@tomcat.apache.org > >
Re: RV-Predict bugs
On Tue, Sep 15, 2015 at 12:29 PM, Mark Thomas wrote:
> On 15/09/2015 17:59, Yilong Li wrote:
> > On Tue, Sep 15, 2015 at 7:19 AM, Mark Thomas wrote:
>
> >> Long experience has lead us to be sceptical of bugs reported by
> >> automated tools. When looking at such bugs and we can't see what the
> >> problem is and the person raising the bug can't provide a clear
> >> explanation - and continues not to provide a clear explanation when
> >> asked for one several times - then that is the point where we start to
> >> think the problem is with the tool.
> >>
> >
> > Sorry, I should not assume that the concepts such as happens-before order
> > and memory model are well understood. Let's talk about how this is
> allowed
> > to happen under JMM. Consider the this example again:
> >
> > if (st_200 == null ) {
> > st_200 = sm.getString("sc.200");
> > }
> > return st_200;
> >
> > The following is a valid execution trace consists of 5 events:
> > T1 T2
> > 1 READ null
> > 2 WRITE s
> > 3 READ s
> > 4 READ ???
> > 5 READ s
> >
> > , where s is the result of sm.getString("sc.200").
> >
> > T1 sees null in field st_200, initializes it, and return the initialized
> > value, while T2 sees a non-null value in st_200 and skips the
> > initialization. The question is what value T2 can read and then return in
> > event 4. This is addressed in JLS $17.4.5 Happens-before Order (
> > https://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4.5
> ):
> >
> > We say that a read *r* of a variable *v* is allowed to observe a write
> *w*
> > to *v* if, in the *happens-before* partial order of the execution trace:
> >
> >-
> >
> >*r* is not ordered before *w* (i.e., it is not the case that *hb(r,
> w)*),
> >and
> >-
> >
> >there is no intervening write *w*' to *v* (i.e. no write *w*' to *v*
> such
> >that *hb(w, w')* and *hb(w', r)*).
> >
> > Informally, a read *r* is allowed to see the result of a write *w* if
> there
> > is no *happens-before* ordering to prevent that read.
> >
> > The question boils down to: does the write in event 2 prevent event 4
> from
> > reading the initial null value of st_200? No, because there is no
> > happens-before order involved here. So what kind of constructs introduce
> > happens-before order? This question is also addressed in the same section
> > of JLS:
> >
> > It follows from the above definitions that:
> >
> >-
> >
> >An unlock on a monitor *happens-before* every subsequent lock on that
> >monitor.
> >-
> >
> >A write to a volatile field (§8.3.1.4
> ><
> https://docs.oracle.com/javase/specs/jls/se8/html/jls-8.html#jls-8.3.1.4>
> >) *happens-before* every subsequent read of that field.
> >-
> >
> >A call to start() on a thread *happens-before* any actions in the
> >started thread.
> >-
> >
> >All actions in a thread *happen-before* any other thread successfully
> >returns from a join() on that thread.
> >-
> >
> >The default initialization of any object *happens-before* any other
> >actions (other than default-writes) of a program.
>
> Thank you. That helps fill in a few gaps. Putting it into my own words
> to check my understanding:
>
> - The two reads in T2 may be re-ordered because, in T2, there is nothing
> that requires a happens-before relationship between the two reads
>
> - If T2 was executing in isolation the order of the reads wouldn't
> matter
>
> - However, T1 is writing.
>
> So if the writes in T2 are re-ordered and the write from T1 takes place
> between them the T2 read for the line 'st_200 == null' could return a
> non-null value for st_200 (the value after the write from T1) while the
> read for the line 'return st_200;' could return null (the value from
> before the write in T1).
>
> Is my understanding correct?
>
Not really. I have been trying to avoid getting into the discussion about
the low-level stuffs such as instruction reordering and caching. Let's stay
on the abstraction level of JMM. When reasoning on the level of JMM, one
should not care about reordering or speculative read. The happens-before
order is the only thing that matters. I know it could be confusing because
of the name, but happens-bef
Re: RV-Predict bugs
On Tue, Sep 15, 2015 at 1:09 PM, Mark Thomas wrote:
> On 15/09/2015 20:42, Caldarale, Charles R wrote:
> >> From: Mark Thomas [mailto:ma...@apache.org]
> >> Subject: Re: RV-Predict bugs
> >
> >> Putting it into my own words to check my understanding:
> >
> >> - The two reads in T2 may be re-ordered because, in T2, there is nothing
> >> that requires a happens-before relationship between the two reads
> >
> > Depends on what was really meant by the ??? notation. If it were
> another reference to st_200, then the example ignores the potential write
> to st_200.
> >
> >> - If T2 was executing in isolation the order of the reads wouldn't
> >> matter
> >
> > Correct, assuming there is no write in T2's control flow.
> >
> >> - However, T1 is writing.
> >
> >> So if the writes
> >
> > I'll assume you meant "reads" there.
>
> I'd did. Thanks.
>
> >> in T2 are re-ordered and the write from T1 takes place between them the
> T2
> >> read for the line 'st_200 == null' could return a non-null value for
> st_200
> >> (the value after the write from T1) while the read for the line 'return
> >> st_200;' could return null (the value from before the write in T1).
> >
> > No - that would violate program order (intra-thread semantics). The
> compiler cannot legally move the read for "return st_200" to a point before
> the potential write to st_200. The CPU can speculatively read for the
> return value, but must discard such speculation if a write were to occur
> (or use store forwarding to update the read).
>
> Thanks for sticking with me on this. I'm finding it hugely helpful. I
> think I need a few more references to make things clearer so I'm going
> to restate the problem.
>
> st_200 is non-volatile
>
> L1 if (st_200 == null ) {
> L2st_200 = sm.getString("sc.200");
> L3 }
> L4 return st_200;
>
> Ln=Line n
> Tx=Thread x
> Rn=Read at line n
> Wn=Write at line n
>
> So T2 has two reads, T2R1 and T2R4.
> Depending on the value read for T2R1 there may be a write at T2W2.
> If there is a write there is a happens before relationship between T2R1
> and T2R4.
>
> Consider the following sequence
>
> T2R4 (out of order read returns null)
> T1R1 (returns null)
> T1W2 (writes non-null value)
> T1R4 (reads new non-null value)
> T2R1 (reads new non-null value)
>
> Because T2R1 reads a non-null value there is no write in T2.
> Therefore there is no happens-before relationship between T2R1 and T2R4
> because there is no intervening write in that thread (the write happens
> in T1).
> Therefore the re-ordering is allowed to happen.
> And we get the unexpected result.
>
> Or
>
> The write at T1W2 is sufficient to enforce the happens before
> relationship between T2R1 and T2R4. In which case what is the point of
> volatile? What do I gain by using it?
>
Again, as I mentioned in my previous message, I am afraid this is not the
correct way to use HB relationship. To make things more clear, I steal this
transformation from Jeremy Manson's article I sent before:
r1 = hash;
if (hash == 0) {
r1 = hash = // calculate hash
}
return r1;
This is a legal transformation w.r.t JMM (though really impractical) and it
is easy to see that this method can return 0 under certain interleaving.
Yilong
> A still slightly confused,
>
> Mark
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>
Re: RV-Predict bugs
Sorry, I am confused. So you are saying that the author of JMM
misunderstands his own work? Well, I highly doubt that. I don't understand
why you interpret JMM in a way that excludes intra-thread semantics. How is
it possible for JMM to be of any use if it does not take into account the
intra-thread semantics?
On Tue, Sep 15, 2015 at 2:01 PM, Caldarale, Charles R <
chuck.caldar...@unisys.com> wrote:
> > From: Yilong Li [mailto:yilong...@runtimeverification.com]
> > Subject: Re: RV-Predict bugs
>
> > Nope, I know what I am doing. Let's first see what the expert says.
> Please
> > check out the last section in this article (
> > http://jeremymanson.blogspot.com/2008/12/benign-data-races-in-java.html)
> > which starts with "let's break the code". Here, we are talking about T2
> in
> > the above example. The second read of the field hashCode is not
> > control-flow-dependent on the first read (the zero-check). Again,
> > intra-thread semantics introduces HB order between the event 3 and event
> 4.
> > But it doesn't ensure that event 4 must see the same value as event 3.
>
> The above article is limited specifically to the memory model and
> inter-thread semantics. While it's true that the memory model does not
> preclude reordering of the reads, the HB of intra-thread semantics does
> (Chapter 14 of the JLS):
>
> "A block is executed by executing each of the local variable declaration
> statements and other statements in order from first to last (left to
> right)."
>
> Otherwise, a single-thread program using the hashCode() example could
> return zero, and that clearly does not happen.
>
OK, let's do the single-thread case:
if (hashCode == 0) {
hashCode = 1;
}
return hashCode;
T1
L0: WRITE 0 (initial value put by JVM)
L1: READ 0
L2: WRITE 1
L4: READ 1
Let me demonstrate why the last read cannot return 0 according to JMM.
First of all, L0 - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail and
> its attachments from all computers.
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>
Re: RV-Predict bugs
On Tue, Sep 15, 2015 at 2:33 PM, Mark Thomas wrote:
> On 15/09/2015 21:51, Yilong Li wrote:
> > On Tue, Sep 15, 2015 at 1:09 PM, Mark Thomas wrote:
> >
> >> On 15/09/2015 20:42, Caldarale, Charles R wrote:
> >>>> From: Mark Thomas [mailto:ma...@apache.org]
> >>>> Subject: Re: RV-Predict bugs
> >>>
> >>>> Putting it into my own words to check my understanding:
> >>>
> >>>> - The two reads in T2 may be re-ordered because, in T2, there is
> nothing
> >>>> that requires a happens-before relationship between the two reads
> >>>
> >>> Depends on what was really meant by the ??? notation. If it were
> >> another reference to st_200, then the example ignores the potential
> write
> >> to st_200.
> >>>
> >>>> - If T2 was executing in isolation the order of the reads wouldn't
> >>>> matter
> >>>
> >>> Correct, assuming there is no write in T2's control flow.
> >>>
> >>>> - However, T1 is writing.
> >>>
> >>>> So if the writes
> >>>
> >>> I'll assume you meant "reads" there.
> >>
> >> I'd did. Thanks.
> >>
> >>>> in T2 are re-ordered and the write from T1 takes place between them
> the
> >> T2
> >>>> read for the line 'st_200 == null' could return a non-null value for
> >> st_200
> >>>> (the value after the write from T1) while the read for the line
> 'return
> >>>> st_200;' could return null (the value from before the write in T1).
> >>>
> >>> No - that would violate program order (intra-thread semantics). The
> >> compiler cannot legally move the read for "return st_200" to a point
> before
> >> the potential write to st_200. The CPU can speculatively read for the
> >> return value, but must discard such speculation if a write were to occur
> >> (or use store forwarding to update the read).
> >>
> >> Thanks for sticking with me on this. I'm finding it hugely helpful. I
> >> think I need a few more references to make things clearer so I'm going
> >> to restate the problem.
> >>
> >> st_200 is non-volatile
> >>
> >> L1 if (st_200 == null ) {
> >> L2st_200 = sm.getString("sc.200");
> >> L3 }
> >> L4 return st_200;
> >>
> >> Ln=Line n
> >> Tx=Thread x
> >> Rn=Read at line n
> >> Wn=Write at line n
> >>
> >> So T2 has two reads, T2R1 and T2R4.
> >> Depending on the value read for T2R1 there may be a write at T2W2.
> >> If there is a write there is a happens before relationship between T2R1
> >> and T2R4.
> >>
> >> Consider the following sequence
> >>
> >> T2R4 (out of order read returns null)
> >> T1R1 (returns null)
> >> T1W2 (writes non-null value)
> >> T1R4 (reads new non-null value)
> >> T2R1 (reads new non-null value)
> >>
> >> Because T2R1 reads a non-null value there is no write in T2.
> >> Therefore there is no happens-before relationship between T2R1 and T2R4
> >> because there is no intervening write in that thread (the write happens
> >> in T1).
> >> Therefore the re-ordering is allowed to happen.
> >> And we get the unexpected result.
> >>
> >> Or
> >>
> >> The write at T1W2 is sufficient to enforce the happens before
> >> relationship between T2R1 and T2R4. In which case what is the point of
> >> volatile? What do I gain by using it?
> >>
> >
> > Again, as I mentioned in my previous message, I am afraid this is not the
> > correct way to use HB relationship. To make things more clear, I steal
> this
> > transformation from Jeremy Manson's article I sent before:
>
> That doesn't help my improve my understanding at all.
>
> What I'm looking for is an explanation - ideally using my example above
> - of exactly how T2R4 can return null while T2R1 reads a non-null value.
> "Because the JMM says it can" is not a sufficient explanation. I
> believe that the JMM says it can but I don't understand how this can
> actually happen. What I am looking for is the explanation of how. Your
> message at 16.59 (UTC) was heading in the right direction but the
> explanation skipped over the crucial part.
>
Fine. Let's do your example:
T2R4 (out of order read returns null)
T1R1 (r
Re: RV-Predict bugs
On Tue, Sep 15, 2015 at 3:09 PM, Mark Thomas wrote: > On 15/09/2015 22:55, Yilong Li wrote: > > > Fine. Let's do your example: > > T2R4 (out of order read returns null) > > T1R1 (returns null) > > T1W2 (writes non-null value) > > T1R4 (reads new non-null value) > > T2R1 (reads new non-null value) > > > > First of all, when reasoning with JMM, you should not move T2R4 to the > > front. This is forbidden by the intra-thread semantics so I move it back > to > > the bottom: > > T1R1 (returns null) > > T1W2 (writes non-null value) > > T1R4 (reads new non-null value) > > T2R1 (reads new non-null value) > > T2R4 (out of order read returns null) > > > > Intra-thread semantics ensures the following HB order: > > T1R1 < T2W2 < T1R4 > > T2R1 < T2R4 > > I assume you mean: > T1R1 < T*1*W2 < T1R4 > above > correct > > > T2R4 is allowed to read null because 1) T2R4 is not ordered before the > > write of null (let's call it INIT) by JVM during object initialization; > and > > 2) there is no intervening write w' such that INIT < w' < T2R4. You see, > > T1W2 is *not* an intervening write because there is no HB order between > > T1W2 & T2R4. > > > > Let me know if it is still not clear enough. > > Sorry, still not clear enough. > > I understand that the JMM allows this behaviour. I don't understand how > this could happen in a JVM implementation. > > If T2R1 has read a non-null value how could a T2R4, a read from the same > thread that happens at some later point in time, read a previous value > for the same field? > > That would imply that the two reads were implemented differently since > they would have to be looking at different memory locations. What is > going on here? > Please go back and check the compiler transformation I stole from Jeremy's article. That transformation actually introduces a new shared memory location (i.e., field r1). How likely is a compiler going to take advantage of such transformation? I don't know, probably very unlikely. But let me clarify something: I am not trying to prove that this is a very harmful data race in order to justify the importance/usefulness of RV-Predict from the beginning. It's on the your discretion whether you care more about the JLS or the JVM implementation. I am simply trying to explain two things: 1) this *is* a bug according to JMM and 2) there is a correct version that is at least as efficient as the original (buggy) one. Yilong > > Mark > > - > To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org > For additional commands, e-mail: dev-h...@tomcat.apache.org > >
Re: RV-Predict bugs
Your argument seems to assume that reordering is the only code
transformation that can be done by compiler or hardware. I don't agree that
you call this transformation a red herring. It might not be practical but
it's certainly valid. Does it violate the intra-thread semantics you
mentioned in JLS Chaptor 14? No. Is it possible for the transformed method
to return 0? Yes.
Yilong
On Tue, Sep 15, 2015 at 6:39 PM, Caldarale, Charles R <
chuck.caldar...@unisys.com> wrote:
> > From: David Jencks [mailto:david_jen...@yahoo.com.INVALID]
> > Subject: Re: RV-Predict bugs
>
> > I have been having a hard time believing the reordering claims, but
> finally I went to Jeremy's
> > blog post. To recap, he says
>
> > if (hash == 0) {
> > int h = //compute hash
> > hash = h;
> > }
> > return hash;
>
> > can be reordered to
>
> > r1 = hash;
> > if (hash == 0) {
> > r1 = hash = // calculate hash
> > }
> > return r1;
>
> > Obviously this last version is susceptible to returning 0.
>
> > It looks to me like the 2nd version is not a reordering of the java
> statements of the first version.
>
> Correct; it's different logic, and thus a bit of a red herring. It
> introduces a timing window not present in the original code; it's not an
> example of a speculative read.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail and
> its attachments from all computers.
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>
Re: RV-Predict bugs
On Tue, Sep 15, 2015 at 6:29 PM, Caldarale, Charles R <
chuck.caldar...@unisys.com> wrote:
> > From: Yilong Li [mailto:yilong...@runtimeverification.com]
> > Subject: Re: RV-Predict bugs
>
> > So you are saying that the author of JMM misunderstands his own work?
>
> No, I'm saying that he's only looking at things from the point of view of
> 17.4, and ignoring section 14.2 of the JLS. (BTW, prior to Sun being
> acquired by Oracle, I had long discussions with HotSpot designers about the
> C++ difference between "const uint64_t* p" and "uint64_t* const p"; they
> never did get it. Don't assume anyone - Gosling, JSR authors, or me - is
> infallible.)
>
> > I don't understand why you interpret JMM in a way that excludes
> intra-thread semantics.
>
> I'm not excluding intra-thread semantics; section 17.4 is concerned with
> inter-thread semantics while requiring that intra-thread semantics be
> honored.
>
> > How is it possible for JMM to be of any use if it does not take into
> account the intra-
> > thread semantics?
>
> Because those are covered in Chapter 14; the concern of 17 is with
> inter-thread operations ("This chapter describes the semantics of
> multithreaded programs"). Note that there are no single-thread examples in
> Chapter 17.
>
Single-threading is nothing but a degenerate case of multithreading. From
what I have seen right now, JMM works equally well for single-thread
programs.
Yilong
>
> > OK, let's do the single-thread case:
> > if (hashCode == 0) {
> > hashCode = 1;
> > }
> > return hashCode;
>
> > L0: WRITE 0 (initial value put by JVM)
> > L1: READ 0
> > L2: WRITE 1
> > L4: READ 1
>
> A poor example on my part. Both the memory model and section 14.2 of the
> JLS certainly do prevent the return of a zero value when the write occurs
> in a non-concurrent situation. However, even with concurrent execution,
> the JVM cannot legally generate the read at L4 prior to that of L1 due to
> the aforementioned section 14.2. The memory model does theoretically
> permit hardware to move L4 in front of L1 ("The actions of each thread in
> isolation must behave as governed by the semantics of that thread, with the
> exception that the values seen by each read are determined by the memory
> model"), but AFAICT there are no JVM-supported CPUs that actually do that,
> nor can I envision any useful hardware design that would. The x86 spec
> expressly forbids it, other CPUs merge the reads together, with store
> forwarding or check load instructions handling the write the thread may
> generate.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail and
> its attachments from all computers.
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: dev-h...@tomcat.apache.org
>
>
Re: svn commit: r1703866 - in /tomcat/tc8.0.x/trunk: ./ java/org/apache/tomcat/websocket/WsFrameBase.java java/org/apache/tomcat/websocket/WsFrameClient.java java/org/apache/tomcat/websocket/WsSession
Hi Mark, I think these are false positives because the completion handler is only called after the IO operation completes. There is an implicit happens-before order here. I have fixed these false positives recently. That's why I didn't report them in BZ. Could you try the latest version of RV-Predict instead? Sorry for the inconvenience because RV-Predict is also in active development based on the feedback we get from testing it against other projects. Thanks, Yilong On Fri, Sep 18, 2015 at 7:14 AM, wrote: > Author: markt > Date: Fri Sep 18 14:14:18 2015 > New Revision: 1703866 > > URL: http://svn.apache.org/viewvc?rev=1703866&view=rev > Log: > Fix various data races reported by RV-Predict. > Mostly caused by completion handlers that execute in a new thread > > Modified: > tomcat/tc8.0.x/trunk/ (props changed) > tomcat/tc8.0.x/trunk/java/org/apache/tomcat/websocket/WsFrameBase.java > > tomcat/tc8.0.x/trunk/java/org/apache/tomcat/websocket/WsFrameClient.java > tomcat/tc8.0.x/trunk/java/org/apache/tomcat/websocket/WsSession.java > tomcat/tc8.0.x/trunk/webapps/docs/changelog.xml > > Propchange: tomcat/tc8.0.x/trunk/ > > -- > --- svn:mergeinfo (original) > +++ svn:mergeinfo Fri Sep 18 14:14:18 2015 > @@ -1 +1 @@ > > -/tomcat/trunk:1636524,1637156,1637176,1637188,1637331,1637684,1637695,1638720-1638725,1639653,1640010,1640083-1640084,1640088,1640275,1640322,1640347,1640361,1640365,1640403,1640410,1640652,1640655-1640658,1640688,1640700-1640883,1640903,1640976,1640978,1641000,1641026,1641038-1641039,1641051-1641052,1641058,1641064,1641300,1641369,1641374,1641380,1641486,1641634,1641656-1641692,1641704,1641707-1641718,1641720-1641722,1641735,1641981,1642233,1642280,1642554,1642564,1642595,1642606,1642668,1642679,1642697,1642699,1642766,1643002,1643045,1643054-1643055,1643066,1643121,1643128,1643206,1643209-1643210,1643216,1643249,1643270,1643283,1643309-1643310,1643323,1643365-1643366,1643370-1643371,1643465,1643474,1643536,1643570,1643634,1643649,1643651,1643654,1643675,1643731,1643733-1643734,1643761,1643766,1643814,1643937,1643963,1644017,1644169,1644201-1644203,1644321,1644323,1644516,1644523,1644529,1644535,1644730,1644768,1644784-1644785,1644790,1644793,1644815,1644884,1644886,1644890,1644892 > > > ,1644910,1644924,1644929-1644930,1644935,1644989,1645011,1645247,1645355,1645357-1645358,1645455,1645465,1645469,1645471,1645473,1645475,1645486-1645488,1645626,1645641,1645685,1645743,1645763,1645951-1645953,1645955,1645993,1646098-1646106,1646178,1646220,1646302,1646304,1646420,1646470-1646471,1646476,1646559,1646717-1646723,1646773,1647026,1647042,1647530,1647655,1648304,1648815,1648907,1650081,1650365,1651116,1651120,1651280,1651470,1652938,1652970,1653041,1653471,1653550,1653574,1653797,1653815-1653816,1653819,1653840,1653857,1653888,1653972,1654013,1654030,1654050,1654123,1654148,1654159,1654513,1654515,1654517,1654522,1654524,1654725,1654735,1654766,1654785,1654851-1654852,1654978,1655122-1655124,1655126-1655127,1655129-1655130,1655132-1655133,1655312,1655351,1655438,1655441,1655454,168,1656087,1656299,1656319,1656331,1656345,1656350,1656590,1656648-1656650,1656657,1657041,1657054,1657374,1657492,1657510,1657565,1657580,1657584,1657586,1657589,1657592,1657607,1657609,1657 > > > 682,1657907,1658207,1658734,1658781,1658790,1658799,1658802,1658804,1658833,1658840,1658966,1659043,1659053,1659059,1659188-1659189,1659216,1659263,1659293,1659304,1659306-1659307,1659382,1659384,1659428,1659471,1659486,1659505,1659516,1659521,1659524,1659559,1659562,1659803,1659806,1659814,1659833,1659862,1659905,1659919,1659948,1659967,1659983-1659984,1660060,1660074,1660077,1660133,1660168,1660331-1660332,1660353,1660358,1660924,1661386,1661867,1661972,1661990,1662200,1662308-1662309,1662548,1662614,1662736,1662985,1662988-1662989,1663264,1663277,1663298,1663534,1663562,1663676,1663715,1663754,1663768,1663772,1663781,1663893,1663995,1664143,1664163,1664174,1664301,1664317,1664347,1664657,1664659,1664710,1664863-1664864,1664866,1665085,1665292,1665559,1665653,1665661,1665672,1665694,1665697,1665736,1665779,1665976-1665977,1665980-1665981,1665985-1665986,1665989,1665998,1666004,1666008,1666013,1666017,1666024,1666116,1666386-1666387,1666494,1666496,1666552,1666569,1666579,137,1 > > > 49,1666757,1666966,1666972,1666985,1666995,1666997,1667292,1667402,1667406,1667546,1667615,1667630,1667636,1667688,1667764,1667871,1668026,1668135,1668193,1668593,1668596,1668630,1668639,1668843,1669353,1669370,1669451,1669800,1669838,1669876,1669882,1670394,1670433,1670591,1670598-1670600,1670610,1670631,1670719,1670724,1670726,1670730,1670940,1671112,1672272,1672284,1673754,1674294,1675461,1675486,1675594,1675830,1676231,1676250-1676251,1676364,1676381,1676393,1676479,1676525,1676552,1676615,1676630,1676634,1676721,1676926,1676943,1677140,1677802,1678011,1678162,1678174,1678339,1678426-1678427,1678694,1678701,1679534,1679708,1679710,16
Re: svn commit: r1703866 - in /tomcat/tc8.0.x/trunk: ./ java/org/apache/tomcat/websocket/WsFrameBase.java java/org/apache/tomcat/websocket/WsFrameClient.java java/org/apache/tomcat/websocket/WsSession
Are you using the 1.7-SNAPSHOT downloaded recently? It's frequently updated. If that's the case, then it's probably a real race and I have to take a closer look. Thanks, Yilong On Fri, Sep 18, 2015 at 7:43 AM, Mark Thomas wrote: > On 18/09/2015 15:27, Yilong Li wrote: > > Hi Mark, > > > > I think these are false positives because the completion handler is only > > called after the IO operation completes. There is an implicit > > happens-before order here. I have fixed these false positives recently. > > That's why I didn't report them in BZ. Could you try the latest version > of > > RV-Predict instead? Sorry for the inconvenience because RV-Predict is > also > > in active development based on the feedback we get from testing it > against > > other projects. > > I'm using 1.7-SNAPSHOT. > > Mark > > > > > > Thanks, > > Yilong > > > > On Fri, Sep 18, 2015 at 7:14 AM, wrote: > > > >> Author: markt > >> Date: Fri Sep 18 14:14:18 2015 > >> New Revision: 1703866 > >> > >> URL: http://svn.apache.org/viewvc?rev=1703866&view=rev > >> Log: > >> Fix various data races reported by RV-Predict. > >> Mostly caused by completion handlers that execute in a new thread > >> > >> Modified: > >> tomcat/tc8.0.x/trunk/ (props changed) > >> > tomcat/tc8.0.x/trunk/java/org/apache/tomcat/websocket/WsFrameBase.java > >> > >> tomcat/tc8.0.x/trunk/java/org/apache/tomcat/websocket/WsFrameClient.java > >> tomcat/tc8.0.x/trunk/java/org/apache/tomcat/websocket/WsSession.java > >> tomcat/tc8.0.x/trunk/webapps/docs/changelog.xml > >> > >> Propchange: tomcat/tc8.0.x/trunk/ > >> > >> > -- > >> --- svn:mergeinfo (original) > >> +++ svn:mergeinfo Fri Sep 18 14:14:18 2015 > >> @@ -1 +1 @@ > >> > >> > -/tomcat/trunk:1636524,1637156,1637176,1637188,1637331,1637684,1637695,1638720-1638725,1639653,1640010,1640083-1640084,1640088,1640275,1640322,1640347,1640361,1640365,1640403,1640410,1640652,1640655-1640658,1640688,1640700-1640883,1640903,1640976,1640978,1641000,1641026,1641038-1641039,1641051-1641052,1641058,1641064,1641300,1641369,1641374,1641380,1641486,1641634,1641656-1641692,1641704,1641707-1641718,1641720-1641722,1641735,1641981,1642233,1642280,1642554,1642564,1642595,1642606,1642668,1642679,1642697,1642699,1642766,1643002,1643045,1643054-1643055,1643066,1643121,1643128,1643206,1643209-1643210,1643216,1643249,1643270,1643283,1643309-1643310,1643323,1643365-1643366,1643370-1643371,1643465,1643474,1643536,1643570,1643634,1643649,1643651,1643654,1643675,1643731,1643733-1643734,1643761,1643766,1643814,1643937,1643963,1644017,1644169,1644201-1644203,1644321,1644323,1644516,1644523,1644529,1644535,1644730,1644768,1644784-1644785,1644790,1644793,1644815,1644884,1644886,1644890,1644 > 8 > 92 > >> > >> > ,1644910,1644924,1644929-1644930,1644935,1644989,1645011,1645247,1645355,1645357-1645358,1645455,1645465,1645469,1645471,1645473,1645475,1645486-1645488,1645626,1645641,1645685,1645743,1645763,1645951-1645953,1645955,1645993,1646098-1646106,1646178,1646220,1646302,1646304,1646420,1646470-1646471,1646476,1646559,1646717-1646723,1646773,1647026,1647042,1647530,1647655,1648304,1648815,1648907,1650081,1650365,1651116,1651120,1651280,1651470,1652938,1652970,1653041,1653471,1653550,1653574,1653797,1653815-1653816,1653819,1653840,1653857,1653888,1653972,1654013,1654030,1654050,1654123,1654148,1654159,1654513,1654515,1654517,1654522,1654524,1654725,1654735,1654766,1654785,1654851-1654852,1654978,1655122-1655124,1655126-1655127,1655129-1655130,1655132-1655133,1655312,1655351,1655438,1655441,1655454,168,1656087,1656299,1656319,1656331,1656345,1656350,1656590,1656648-1656650,1656657,1657041,1657054,1657374,1657492,1657510,1657565,1657580,1657584,1657586,1657589,1657592,1657607,1657609,1 > 6 > 57 > >> > >> > 682,1657907,1658207,1658734,1658781,1658790,1658799,1658802,1658804,1658833,1658840,1658966,1659043,1659053,1659059,1659188-1659189,1659216,1659263,1659293,1659304,1659306-1659307,1659382,1659384,1659428,1659471,1659486,1659505,1659516,1659521,1659524,1659559,1659562,1659803,1659806,1659814,1659833,1659862,1659905,1659919,1659948,1659967,1659983-1659984,1660060,1660074,1660077,1660133,1660168,1660331-1660332,1660353,1660358,1660924,1661386,1661867,1661972,1661990,1662200,1662308-1662309,1662548,1662614,1662736,1662985,1662988-1662989,1663264,1663277,166329
Re: svn commit: r1703866 - in /tomcat/tc8.0.x/trunk: ./ java/org/apache/tomcat/websocket/WsFrameBase.java java/org/apache/tomcat/websocket/WsFrameClient.java java/org/apache/tomcat/websocket/WsSession
Yes, you are right. These are real. I didn't rerun the entire test suite after fixing the obvious false positives. Yilong On Fri, Sep 18, 2015 at 7:58 AM, Mark Thomas wrote: > On 18/09/2015 15:51, Yilong Li wrote: > > Are you using the 1.7-SNAPSHOT downloaded recently? It's frequently > > updated. If that's the case, then it's probably a real race and I have to > > take a closer look. > > Downloaded at around 10.30 UTC today. > > The races looked plausible. There was an obvious ordering because of the > I/O but assuming the completion handlers are using a thread pool (so a > new thread is not created to handle the completion) I could believe that > a field update in the 'main' thread wasn't immediately visible to the > completion handler thread unless the field was volatile. > > Mark > > > > > > Thanks, > > Yilong > > > > On Fri, Sep 18, 2015 at 7:43 AM, Mark Thomas wrote: > > > >> On 18/09/2015 15:27, Yilong Li wrote: > >>> Hi Mark, > >>> > >>> I think these are false positives because the completion handler is > only > >>> called after the IO operation completes. There is an implicit > >>> happens-before order here. I have fixed these false positives recently. > >>> That's why I didn't report them in BZ. Could you try the latest version > >> of > >>> RV-Predict instead? Sorry for the inconvenience because RV-Predict is > >> also > >>> in active development based on the feedback we get from testing it > >> against > >>> other projects. > >> > >> I'm using 1.7-SNAPSHOT. > >> > >> Mark > >> > >> > >>> > >>> Thanks, > >>> Yilong > >>> > >>> On Fri, Sep 18, 2015 at 7:14 AM, wrote: > >>> > >>>> Author: markt > >>>> Date: Fri Sep 18 14:14:18 2015 > >>>> New Revision: 1703866 > >>>> > >>>> URL: http://svn.apache.org/viewvc?rev=1703866&view=rev > >>>> Log: > >>>> Fix various data races reported by RV-Predict. > >>>> Mostly caused by completion handlers that execute in a new thread > >>>> > >>>> Modified: > >>>> tomcat/tc8.0.x/trunk/ (props changed) > >>>> > >> tomcat/tc8.0.x/trunk/java/org/apache/tomcat/websocket/WsFrameBase.java > >>>> > >>>> > tomcat/tc8.0.x/trunk/java/org/apache/tomcat/websocket/WsFrameClient.java > >>>> > tomcat/tc8.0.x/trunk/java/org/apache/tomcat/websocket/WsSession.java > >>>> tomcat/tc8.0.x/trunk/webapps/docs/changelog.xml > >>>> > >>>> Propchange: tomcat/tc8.0.x/trunk/ > >>>> > >>>> > >> > -- > >>>> --- svn:mergeinfo (original) > >>>> +++ svn:mergeinfo Fri Sep 18 14:14:18 2015 > >>>> @@ -1 +1 @@ > >>>> > >>>> > >> > -/tomcat/trunk:1636524,1637156,1637176,1637188,1637331,1637684,1637695,1638720-1638725,1639653,1640010,1640083-1640084,1640088,1640275,1640322,1640347,1640361,1640365,1640403,1640410,1640652,1640655-1640658,1640688,1640700-1640883,1640903,1640976,1640978,1641000,1641026,1641038-1641039,1641051-1641052,1641058,1641064,1641300,1641369,1641374,1641380,1641486,1641634,1641656-1641692,1641704,1641707-1641718,1641720-1641722,1641735,1641981,1642233,1642280,1642554,1642564,1642595,1642606,1642668,1642679,1642697,1642699,1642766,1643002,1643045,1643054-1643055,1643066,1643121,1643128,1643206,1643209-1643210,1643216,1643249,1643270,1643283,1643309-1643310,1643323,1643365-1643366,1643370-1643371,1643465,1643474,1643536,1643570,1643634,1643649,1643651,1643654,1643675,1643731,1643733-1643734,1643761,1643766,1643814,1643937,1643963,1644017,1644169,1644201-1644203,1644321,1644323,1644516,1644523,1644529,1644535,1644730,1644768,1644784-1644785,1644790,1644793,1644815,1644884,1644886,1644890,1644 > > >> 8 > >> 92 > >>>> > >>>> > >> > ,1644910,1644924,1644929-1644930,1644935,1644989,1645011,1645247,1645355,1645357-1645358,1645455,1645465,1645469,1645471,1645473,1645475,1645486-1645488,1645626,1645641,1645685,1645743,1645763,1645951-1645953,1645955,1645993,1646098-1646106,1646178,1646220,1646302,1646304,1646420,1646470-1646471,1646476,1646559,1646717-1646723,1646773,1647026,1647042,1647530,1647655,1648304,1648815,1648907,1650081,1650365,1651116,1651120,1651280,1651470,1652938,1652970,1653041,1653471,1653550,1
