Build failed in Jenkins: distributedlog-nightly-build #101

[Apache Jenkins Server]

See 

--
[...truncated 149 lines...]
[WARNING] 
:[121,44]
 getBKClientReadTimeout() in 
com.twitter.distributedlog.DistributedLogConfiguration has been deprecated
[WARNING] 
:[365,33]
 isThrow() in com.twitter.util.Future has been deprecated
[INFO] 
[INFO] --- maven-surefire-plugin:2.9:test (default-test) @ distributedlog-core 
---
[INFO] Surefire report directory: 


---
 T E S T S
---

---
 T E S T S
---
Running com.twitter.distributedlog.TestBKDistributedLogManager
Tests run: 30, Failures: 0, Errors: 13, Skipped: 0, Time elapsed: 807.543 sec 
<<< FAILURE!
Running com.twitter.distributedlog.limiter.TestRequestLimiter
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.08 sec
Running com.twitter.distributedlog.TestDistributedLogConfiguration
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.264 sec
Running com.twitter.distributedlog.TestLogSegmentsZK
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.532 sec
Running com.twitter.distributedlog.bk.TestLedgerAllocator
Tests run: 9, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 12.245 sec
Running com.twitter.distributedlog.bk.TestLedgerAllocatorPool
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.973 sec
Running com.twitter.distributedlog.TestAppendOnlyStreamWriter
Tests run: 8, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 322.852 sec <<< 
FAILURE!
Running com.twitter.distributedlog.TestSequenceID
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.994 sec
Running com.twitter.distributedlog.zk.TestZKVersionedSetOp
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.218 sec
Running com.twitter.distributedlog.zk.TestZKWatcherManager
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.142 sec
Running com.twitter.distributedlog.zk.TestZKTransaction
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.458 sec
Running com.twitter.distributedlog.impl.TestZKLogSegmentMetadataStore
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.33 sec
Running com.twitter.distributedlog.impl.TestZKNamespaceWatcher
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.99 sec
Running com.twitter.distributedlog.impl.TestZKLogSegmentFilters
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.172 sec
Running 
com.twitter.distributedlog.impl.federated.TestFederatedZKLogMetadataStore
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.426 sec
Running com.twitter.distributedlog.impl.metadata.TestZKLogMetadata
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.175 sec
Running com.twitter.distributedlog.impl.metadata.TestZKLogMetadataForWriter
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.551 sec
Running 
com.twitter.distributedlog.impl.metadata.TestZKLogMetadataForWriterUtilFunctions
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.201 sec
Running com.twitter.distributedlog.impl.TestZKLogMetadataStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.587 sec
Running com.twitter.distributedlog.TestWriteLimiter
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.153 sec
Running com.twitter.distributedlog.TestTruncate
Running com.twitter.distributedlog.TestAppendOnlyStreamReader
Tests run: 5, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 303.336 sec <<< 
FAILURE!
Running com.twitter.distributedlog.TestNonBlockingReadsMultiReader
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 62.564 sec <<< 
FAILURE!
Running com.twitter.distributedlog.util.TestConfUtils
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.176 sec
Running com.twitter.distributedlog.util.TestTimeSequencer
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.049 sec
Running com.twitter.distributedlog.util.TestFutureUtils
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.381 sec
Running com.twitter.distributedlog.util.TestDLUtils
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.199 sec
Running com.twitter.distributedlog.util.TestPermitManager
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.142 sec
Running com.twitter.distributedlog.util.TestUtils

Re: hundreds of millions of streams?

[Flavio Junqueira]

Perhaps this is an obvious observation, but can't you deploy multiple instances 
to scale to whatever size you like?

-Flavio

> On 28 Oct 2016, at 20:27, Poule Dodue  wrote:
> 
> I think it would cause more concurrency problems in write operations as
> described at page 12 point #2 of this thesis:
> 
> http://www.diva-portal.se/smash/get/diva2:877307/FULLTEXT01.pdf
> 
> 
> 
>> Le 28 oct. 2016 à 11:48, Leigh Stewart  a 
>> écrit :
>> 
>> Got it. We probably can't support that scale at this time.
>> Curious: do you resort to sharing streams among objects with systems that
>> don't support 100s millions of streams? (i.e. partitioning objects across
>> streams?)
>> 
>> On Fri, Oct 28, 2016 at 8:24 AM, Poule Dodue  wrote:
>> 
>>> yes aka ES/CQRS
>>> 
>>> some links:
>>> 
>>> https://msdn.microsoft.com/en-us/library/jj554200.aspx
>>> http://williamverdolini.github.io/2014/08/11/cqrses-architecture/
>>> http://docs.geteventstore.com/introduction/3.9.0/event-sourcing-basics/
>>> 
>>> it needs lot of streams to basically replay events for any entity on a
>>> system.
>>> 
>>> example: i could replay events for all changes that happened in 1 Cart of
>>> 1 User:
>>> 
>>> 
>>> (read events from stream "cart-of-user-233293111" ):
>>> 
>>> 1- added item X
>>> 2- deleted item X
>>> 3- added item Y
>>> 
>>> 
>>> by replaying that stream, I can rebuild a user's cart state
>>> 
>>> 
 Le 28 oct. 2016 à 10:13, Leigh Stewart  a
>>> écrit :
 
 Poule- would you mind sharing some information on Event Sourcing? Are you
 referring to something like
 http://martinfowler.com/eaaDev/EventSourcing.html ?
 
 On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart 
>>> wrote:
 
> DL is not able to handle 100s of millions of streams. 10^5-106 is
>>> probably
> ok.
> ZK is probably the biggest challenge (we are looking at ways to
>>> eliminate
> this as we would like to scale to 10^6-10^7 in the not too distant
>>> future),
> but 100s of millions is so far beyond what we've worked with there would
> likely be other scaling challenges on the way to that point.
> 
> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue 
> wrote:
> 
>> In Event Sourcing, we need to have 1 stream per entity/aggregate so for
>> a typical prod system it means we need hundreds of millions of streams.
>> 
>> Is DL able to handle that or it is limited to, say, few hundreds
>> thousands of streams?
>> 
>> 
>> 
> 
>>> 
>>> 
>