[jira] [Commented] (DL-131) Enable check style for distributedlog-client module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769373#comment-15769373
 ] 

ASF GitHub Bot commented on DL-131:
---

GitHub user xiliuant opened a pull request:

https://github.com/apache/incubator-distributedlog/pull/87

DL-131: Enable checkstyle for distributedlog-client module



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xiliuant/incubator-distributedlog 
xi/checkstyle_client

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-distributedlog/pull/87.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #87


commit e6bc01a0fe084fe4b473546089238f886d7ac6f6
Author: Xi Liu 
Date:   2016-12-22T07:35:04Z

Enable checkstyle for distributedlog-client module




> Enable check style for distributedlog-client module
> ---
>
> Key: DL-131
> URL: https://issues.apache.org/jira/browse/DL-131
> Project: DistributedLog
>  Issue Type: Sub-task
>  Components: distributedlog-client
>Reporter: Xi Liu
>Assignee: Xi Liu
> Fix For: 0.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-distributedlog pull request #87: DL-131: Enable checkstyle for dis...

[xiliuant]

GitHub user xiliuant opened a pull request:

https://github.com/apache/incubator-distributedlog/pull/87

DL-131: Enable checkstyle for distributedlog-client module



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xiliuant/incubator-distributedlog 
xi/checkstyle_client

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-distributedlog/pull/87.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #87


commit e6bc01a0fe084fe4b473546089238f886d7ac6f6
Author: Xi Liu 
Date:   2016-12-22T07:35:04Z

Enable checkstyle for distributedlog-client module




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (DL-145) Fix the flaky testServiceTimeout

[Liang Xie (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769322#comment-15769322
 ] 

Liang Xie commented on DL-145:
--

ping [~si...@apache.org] :)

> Fix the flaky testServiceTimeout
> 
>
> Key: DL-145
> URL: https://issues.apache.org/jira/browse/DL-145
> Project: DistributedLog
>  Issue Type: Test
>  Components: distributedlog-service
>Affects Versions: 0.4.0
>Reporter: Liang Xie
>Assignee: Liang Xie
>
> The TestDistributedLogService#testServiceTimeout case is not stable, e.g. 
> https://builds.apache.org/job/distributedlog-precommit-pullrequest/22/com.twitter$distributedlog-service/testReport/com.twitter.distributedlog.service/TestDistributedLogService/testServiceTimeout/
> It could be reproduced on my box occasionally, and the failures were stable 
> if i tuned the ServiceTimeoutMs from 200 to 150, and always passed if tuned 
> to a larger value, e.g. 1000(btw, my disk is SSD type)
> After digging into it, shows it related with starting a new log segment 
> corner case.
> For a good case, once service time out occurs, steam status : ERROR -> 
> CLOSING -> CLOSED, calling Abortables.asyncAbort will trigger the cached 
> logsegment be aborted, then writeOp will be injected an exception, e.g. write 
> cancel exception.
> For a bad case, since no log records be written before, so there'll be an 
> async start new log segment, once the timeout occurs, the segment starting 
> still not be done, so no cache, then asyncAbort has no change to abort that 
> segment.
> I think change the test timeout value to a larger one should be fine for this 
> special test corner case.
> will attach a minor patch later.  Any suggestions are welcome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DL-155) TestAsyncReaderLock#testReaderLockMultiReadersScenario is flaky

[Liang Xie (JIRA)]

Liang Xie created DL-155:


 Summary: TestAsyncReaderLock#testReaderLockMultiReadersScenario is 
flaky
 Key: DL-155
 URL: https://issues.apache.org/jira/browse/DL-155
 Project: DistributedLog
  Issue Type: Bug
  Components: distributedlog-core
Reporter: Liang Xie
Assignee: Liang Xie


See 
https://builds.apache.org/job/distributedlog-nightly-build/com.twitter$distributedlog-core/162/testReport/junit/com.twitter.distributedlog/TestAsyncReaderLock/testReaderLockMultiReadersScenario/

The error  came from writh path, due to extra control records be transmitted 
sometimes, the entryId probably more than 99 which was expected in original 
case.

DL-151 had a similar issue and gave a solution already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DL-154) Avoid using deprecated methods

[Jay Juma (JIRA)]

Jay Juma created DL-154:
---

 Summary: Avoid using deprecated methods
 Key: DL-154
 URL: https://issues.apache.org/jira/browse/DL-154
 Project: DistributedLog
  Issue Type: Bug
  Components: builds
Reporter: Jay Juma


There are a couple of warnings about using deprecated APIs when building the 
project. It would be good to avoid using deprecated APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DL-17) Ledger allocator pool should be able to use hostname to distinguish write proxies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-17?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769130#comment-15769130
 ] 

ASF GitHub Bot commented on DL-17:
--

GitHub user jayk-juma opened a pull request:

https://github.com/apache/incubator-distributedlog/pull/86

DL-17: Ledger allocator pool should be able to use hostname to distinguish 
write proxies

…guish write proxies

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jayk-juma/incubator-distributedlog 
jay/use_hostname_for_ledger_allocation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-distributedlog/pull/86.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #86


commit 9b86ee41c94e9a9965843fb66205cf7edeaa6643
Author: Jay Juma 
Date:   2016-12-22T05:39:52Z

DL-17: Ledger allocator pool should be able to use hostname to distinguish 
write proxies




> Ledger allocator pool should be able to use hostname to distinguish write 
> proxies
> -
>
> Key: DL-17
> URL: https://issues.apache.org/jira/browse/DL-17
> Project: DistributedLog
>  Issue Type: Task
>  Components: distributedlog-service
>Reporter: Sijie Guo
>Assignee: Jay Juma
>  Labels: help-wanted
> Fix For: 0.4.0
>
>
> Currently write proxy uses shard id to distinguish ledger allocators in the 
> pool. Using shard id is very convenient if deploying using a scheduler like 
> aurora, as scheduler usually be able to assign a unique shard id.
> However, for deployments not using aurora, it might make sense to use just 
> hostname to identify the allocator used by a specific write proxy. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-distributedlog pull request #86: DL-17: Ledger allocator pool shou...

[jayk-juma]

GitHub user jayk-juma opened a pull request:

https://github.com/apache/incubator-distributedlog/pull/86

DL-17: Ledger allocator pool should be able to use hostname to distinguish 
write proxies

…guish write proxies

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jayk-juma/incubator-distributedlog 
jay/use_hostname_for_ledger_allocation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-distributedlog/pull/86.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #86


commit 9b86ee41c94e9a9965843fb66205cf7edeaa6643
Author: Jay Juma 
Date:   2016-12-22T05:39:52Z

DL-17: Ledger allocator pool should be able to use hostname to distinguish 
write proxies




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (DL-17) Ledger allocator pool should be able to use hostname to distinguish write proxies

[Jay Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-17?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Juma reassigned DL-17:
--

Assignee: Jay Juma

> Ledger allocator pool should be able to use hostname to distinguish write 
> proxies
> -
>
> Key: DL-17
> URL: https://issues.apache.org/jira/browse/DL-17
> Project: DistributedLog
>  Issue Type: Task
>  Components: distributedlog-service
>Reporter: Sijie Guo
>Assignee: Jay Juma
>  Labels: help-wanted
> Fix For: 0.4.0
>
>
> Currently write proxy uses shard id to distinguish ledger allocators in the 
> pool. Using shard id is very convenient if deploying using a scheduler like 
> aurora, as scheduler usually be able to assign a unique shard id.
> However, for deployments not using aurora, it might make sense to use just 
> hostname to identify the allocator used by a specific write proxy. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-distributedlog pull request #85: DL-153: typo in distributedlog-be...

[jayk-juma]

GitHub user jayk-juma opened a pull request:

https://github.com/apache/incubator-distributedlog/pull/85

DL-153: typo in distributedlog-benchmark/bin/dbench



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jayk-juma/incubator-distributedlog 
jay/fix_typo_in_benchmark

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-distributedlog/pull/85.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #85


commit 057cbdeb4cce93ae0ac274c9ccc3f8e7b7127850
Author: Jay Juma 
Date:   2016-12-22T05:00:34Z

DL-153: typo in distributedlog-benchmark/bin/dbench




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Is there a REST API for Write and Read proxies?

[Jay Juma]

Asko,

That's something that I am also looking for. I am interested in that.

One comment about "a REST / websockets proxy", are you thinking of running
a separate service or part of current proxy (e.g. listening on a separate
port to handle REST/websockets request)?

- Jay

On Wed, Dec 21, 2016 at 7:40 PM, Asko Kauppi  wrote:

> Thanks for clarification. The question was not meant as criticism, just
> trying to understand the architecture.
>
> Once/if we make a REST API / websockets proxy (of the current proxy), it
> could also take care of access control? I’m thinking of a system where
> access to data is close to the data, not at the services handling the data.
>
> Anyone interested in working on something like that?
>
> Asko Kauppi
> Zalando Tech Helsinki
>
>
> > On 21 Dec 2016, at 19.09, Leigh Stewart 
> wrote:
> >
> > The challenge is the protocol uses a redirection mechanism so theres a
> > protocol beyond thrift/transport.
> >
> > As Jay says we plan to make improvements in this area.
> >
> > On Wed, Dec 21, 2016 at 8:53 AM, Jay Juma  wrote:
> >
> >> Hi Asko,
> >>
> >> I don't think there is a REST api available in the proxy service. The
> API
> >> seems to be thrift-rpc based. I found there is a JIRA to support gPRC
> wire
> >> protocol. It should not be difficult to add a REST api.
> >>
> >> - Jay
> >>
> >> On Wed, Dec 21, 2016 at 4:29 AM, Asko Kauppi 
> >> wrote:
> >>
> >>> I’m reading http://distributedlog.incubator.apache.org/docs/
> >>> latest/user_guide/api/proxy.html  >>> incubator.apache.org/docs/latest/user_guide/api/proxy.html>
> >>>
> >>> Ideally, I wouldn’t need to use a library to talk to a proxy service,
> >>> right? Is there documentation on how to access the proxies as REST
> >>> endpoints / are they such?
> >>>
> >>> My preferred environment is Scala and akka-http.
> >>>
> >>> Asko Kauppi
> >>> Zalando Tech Helsinki
> >>>
> >>>
> >>
>
>

[jira] [Commented] (DL-28) generate java code failed using scrooge

[Tracy Martin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768856#comment-15768856
 ] 

Tracy Martin commented on DL-28:


The build worked fine from home after I removed the ./distributed-service/conf 
file.

The error that I received before removing it was:
ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-assembly-plugin:2.2.1:single (default) on 
project distributedlog-service: Failed to create assembly: Error adding 
directory to archive: 
C:\Projects\apache\incubator-distributedlog\distributedlog-service\conf isn't a 
directory. -> [Help 1]

> generate java code failed using scrooge
> ---
>
> Key: DL-28
> URL: https://issues.apache.org/jira/browse/DL-28
> Project: DistributedLog
>  Issue Type: Bug
>  Components: distributedlog-protocol
>Affects Versions: 0.4.0
> Environment: Apache Maven 3.2.5
> Maven home: C:\apache-maven-3.2.5
> Java version: 1.8.0_60, vendor: Oracle Corporation
> Java home: C:\Java\jdk1.8.0_60\jre
> Default locale: zh_CN, platform encoding: UTF-8
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"
>Reporter: weiwei
>Assignee: Liang Xie
>Priority: Blocker
>  Labels: build
> Fix For: 0.4.0
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] distributedlog . SUCCESS [  0.002 
> s]
> [INFO] DistributedLog Protocol  FAILURE [  1.081 
> s]
> [INFO] DistributedLog Core Library  SKIPPED
> [INFO] DistributedLog Client .. SKIPPED
> [INFO] DistributedLog Service . SKIPPED
> [INFO] DistributedLog Benchmark ... SKIPPED
> [INFO] DistributedLog Example . SKIPPED
> [INFO] DistributedLog Basics .. SKIPPED
> [INFO] DistributedLog Messaging Tutorial .. SKIPPED
> [INFO] DistributedLog Kafka Tutorial .. SKIPPED
> [INFO] DistributedLog MapReduce ... SKIPPED
> [INFO] DistributedLog Tutorials ... SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 1.321 s
> [INFO] Finished at: 2016-08-02T13:26:31+08:00
> [INFO] Final Memory: 12M/150M
> [INFO] 
> 
> [ERROR] Exception parsing: 
> E:\incubator-distributedlog\distributedlog-protocol\src\main\thrift\service.thrift:
>  [3.1] failure: string matching regex `\z' expected but `/' found
> [ERROR] 
> [ERROR] // Response stats codes
> [ERROR] 
> [ERROR] ^
> [ERROR] -> [Help 1]
> com.twitter.scrooge.frontend.FileParseException: Exception parsing: 
> E:\isoftstone\workspacedeis\incubator-distributedlog\distributedlog-protocol\src\main\thrift\service.thrift
>   at 
> com.twitter.scrooge.frontend.ThriftParser$$anonfun$parse$1.apply(ThriftParser.scala:427)
>   at 
> com.twitter.scrooge.frontend.ThriftParser$$anonfun$parse$1.apply(ThriftParser.scala:427)
>   at scala.Option.map(Option.scala:145)
>   at 
> com.twitter.scrooge.frontend.ThriftParser.parse(ThriftParser.scala:420)
>   at 
> com.twitter.scrooge.frontend.ThriftParser.parseFile(ThriftParser.scala:448)
>   at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:66)
>   at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:64)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at 
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>   at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
>   at com.twitter.scrooge.Compiler.run(Compiler.scala:64)
>   at com.twitter.ScroogeRunner.compile(ScroogeRunner.scala:45)
>   at 
> com.twitter.AbstractMavenScroogeMojo.execute(AbstractMavenScroogeMojo.java:239)
>   at 
> com.twitter.MavenScroogeCompileMojo.execute(MavenScroogeCompileMojo.java:22)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> 

[jira] [Updated] (DL-18) Shard id is not optional

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-18?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo updated DL-18:

Labels: help-wanted  (was: )

> Shard id is not optional
> 
>
> Key: DL-18
> URL: https://issues.apache.org/jira/browse/DL-18
> Project: DistributedLog
>  Issue Type: Task
>  Components: documentation
>Reporter: Sijie Guo
>  Labels: help-wanted
> Fix For: 0.4.0
>
>
> http://distributedlog.io/configuration/proxy.html in the documentation, 
> {quote}
> server_shard: The shard id used by the server to identify itself. It is 
> optional but recommended to set. For example, if the write proxy is running 
> in Apache Aurora, you could use the instance id as the shard id. The default 
> value is -1 (unset).
> {quote}
> However, the shard id is not optional. because the ledger allocator pool uses 
> it to distinguish ledger allocators (see DL-17).
> We need to update the documentation to reflect the current state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DL-24) broken quickstart tutorial

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-24?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo resolved DL-24.
-
Resolution: Fixed

> broken quickstart tutorial 
> ---
>
> Key: DL-24
> URL: https://issues.apache.org/jira/browse/DL-24
> Project: DistributedLog
>  Issue Type: Task
>  Components: documentation
>Reporter: Sijie Guo
> Fix For: 0.4.0
>
>
> reported at https://github.com/twitter/distributedlog/issues/40



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DL-17) Ledger allocator pool should be able to use hostname to distinguish write proxies

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-17?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo updated DL-17:

Labels: help-wanted  (was: )

> Ledger allocator pool should be able to use hostname to distinguish write 
> proxies
> -
>
> Key: DL-17
> URL: https://issues.apache.org/jira/browse/DL-17
> Project: DistributedLog
>  Issue Type: Task
>  Components: distributedlog-service
>Reporter: Sijie Guo
>  Labels: help-wanted
> Fix For: 0.4.0
>
>
> Currently write proxy uses shard id to distinguish ledger allocators in the 
> pool. Using shard id is very convenient if deploying using a scheduler like 
> aurora, as scheduler usually be able to assign a unique shard id.
> However, for deployments not using aurora, it might make sense to use just 
> hostname to identify the allocator used by a specific write proxy. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DL-9) Reverse iterator over a log stream #24

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo updated DL-9:
---
Labels: help-wanted  (was: )

> Reverse iterator over a log stream #24
> --
>
> Key: DL-9
> URL: https://issues.apache.org/jira/browse/DL-9
> Project: DistributedLog
>  Issue Type: New Feature
>  Components: distributedlog-core
>Reporter: Sijie Guo
>  Labels: help-wanted
>
> The AsyncLogReader only provides forward iterator over a log stream. 
> Sometimes, a reverse iterator is helpful.
> Cloned from https://github.com/twitter/distributedlog/issues/24



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DL-62) BKSyncLogReader has deadlock

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-62?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo resolved DL-62.
-
   Resolution: Fixed
Fix Version/s: 0.4.0

This has been merged at ebaf145807813c2731f3afbcd30fb526a1d677da

> BKSyncLogReader has deadlock
> 
>
> Key: DL-62
> URL: https://issues.apache.org/jira/browse/DL-62
> Project: DistributedLog
>  Issue Type: Bug
>  Components: distributedlog-core
>Reporter: Khurrum Nasim
>Assignee: Khurrum Nasim
> Fix For: 0.4.0
>
>
> There is a deadlock introduced in DL-32 when [~jderrick] fixed the findbug 
> errors. It is also the root cause of CI failures (DL-59)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DL-61) Open reader starting from a given sequence id

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-61?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo updated DL-61:

Labels: help-wanted  (was: )

> Open reader starting from a given sequence id
> -
>
> Key: DL-61
> URL: https://issues.apache.org/jira/browse/DL-61
> Project: DistributedLog
>  Issue Type: New Feature
>  Components: distributedlog-core
>Reporter: Khurrum Nasim
>Assignee: Khurrum Nasim
>  Labels: help-wanted
>
> Current implementation only supports opening reader by DLSN or TransactionID. 
> DL should support opening reader by SequenceID too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DL-35) Document on how to setup a global DL cluster

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo resolved DL-35.
-
Resolution: Fixed

This has been merged at 
https://github.com/apache/incubator-distributedlog/pull/36

> Document on how to setup a global DL cluster
> 
>
> Key: DL-35
> URL: https://issues.apache.org/jira/browse/DL-35
> Project: DistributedLog
>  Issue Type: Task
>  Components: documentation
>Reporter: Sijie Guo
>Assignee: Leigh Stewart
> Fix For: 0.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DL-121) DL should inherit AutoCloseable

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo updated DL-121:
-
Labels: help-wanted  (was: )

> DL should inherit AutoCloseable
> ---
>
> Key: DL-121
> URL: https://issues.apache.org/jira/browse/DL-121
> Project: DistributedLog
>  Issue Type: Bug
>  Components: distributedlog-core
>Reporter: Gerrit Sundaram
>  Labels: help-wanted
>
> It would be great that DL resources (e.g. namespace, manager, writer and 
> reader) can be auto closable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DL-97) Remove unused methods in BKLogHandler

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo resolved DL-97.
-
Resolution: Fixed

> Remove unused methods in BKLogHandler
> -
>
> Key: DL-97
> URL: https://issues.apache.org/jira/browse/DL-97
> Project: DistributedLog
>  Issue Type: Sub-task
>  Components: distributedlog-core
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 0.4.0
>
>
> Remove unused methods in BKLogHandler



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DL-28) generate java code failed using scrooge

[Sijie Guo (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768308#comment-15768308
 ] 

Sijie Guo commented on DL-28:
-

[~fonan] okay. let me know if you need any help.

> generate java code failed using scrooge
> ---
>
> Key: DL-28
> URL: https://issues.apache.org/jira/browse/DL-28
> Project: DistributedLog
>  Issue Type: Bug
>  Components: distributedlog-protocol
>Affects Versions: 0.4.0
> Environment: Apache Maven 3.2.5
> Maven home: C:\apache-maven-3.2.5
> Java version: 1.8.0_60, vendor: Oracle Corporation
> Java home: C:\Java\jdk1.8.0_60\jre
> Default locale: zh_CN, platform encoding: UTF-8
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"
>Reporter: weiwei
>Assignee: Liang Xie
>Priority: Blocker
>  Labels: build
> Fix For: 0.4.0
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] distributedlog . SUCCESS [  0.002 
> s]
> [INFO] DistributedLog Protocol  FAILURE [  1.081 
> s]
> [INFO] DistributedLog Core Library  SKIPPED
> [INFO] DistributedLog Client .. SKIPPED
> [INFO] DistributedLog Service . SKIPPED
> [INFO] DistributedLog Benchmark ... SKIPPED
> [INFO] DistributedLog Example . SKIPPED
> [INFO] DistributedLog Basics .. SKIPPED
> [INFO] DistributedLog Messaging Tutorial .. SKIPPED
> [INFO] DistributedLog Kafka Tutorial .. SKIPPED
> [INFO] DistributedLog MapReduce ... SKIPPED
> [INFO] DistributedLog Tutorials ... SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 1.321 s
> [INFO] Finished at: 2016-08-02T13:26:31+08:00
> [INFO] Final Memory: 12M/150M
> [INFO] 
> 
> [ERROR] Exception parsing: 
> E:\incubator-distributedlog\distributedlog-protocol\src\main\thrift\service.thrift:
>  [3.1] failure: string matching regex `\z' expected but `/' found
> [ERROR] 
> [ERROR] // Response stats codes
> [ERROR] 
> [ERROR] ^
> [ERROR] -> [Help 1]
> com.twitter.scrooge.frontend.FileParseException: Exception parsing: 
> E:\isoftstone\workspacedeis\incubator-distributedlog\distributedlog-protocol\src\main\thrift\service.thrift
>   at 
> com.twitter.scrooge.frontend.ThriftParser$$anonfun$parse$1.apply(ThriftParser.scala:427)
>   at 
> com.twitter.scrooge.frontend.ThriftParser$$anonfun$parse$1.apply(ThriftParser.scala:427)
>   at scala.Option.map(Option.scala:145)
>   at 
> com.twitter.scrooge.frontend.ThriftParser.parse(ThriftParser.scala:420)
>   at 
> com.twitter.scrooge.frontend.ThriftParser.parseFile(ThriftParser.scala:448)
>   at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:66)
>   at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:64)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at 
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>   at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
>   at com.twitter.scrooge.Compiler.run(Compiler.scala:64)
>   at com.twitter.ScroogeRunner.compile(ScroogeRunner.scala:45)
>   at 
> com.twitter.AbstractMavenScroogeMojo.execute(AbstractMavenScroogeMojo.java:239)
>   at 
> com.twitter.MavenScroogeCompileMojo.execute(MavenScroogeCompileMojo.java:22)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
>

[jira] [Commented] (DL-28) generate java code failed using scrooge

[Tracy Martin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768268#comment-15768268
 ] 

Tracy Martin commented on DL-28:


I was wrong about this.  I will start over tomorrow.

> generate java code failed using scrooge
> ---
>
> Key: DL-28
> URL: https://issues.apache.org/jira/browse/DL-28
> Project: DistributedLog
>  Issue Type: Bug
>  Components: distributedlog-protocol
>Affects Versions: 0.4.0
> Environment: Apache Maven 3.2.5
> Maven home: C:\apache-maven-3.2.5
> Java version: 1.8.0_60, vendor: Oracle Corporation
> Java home: C:\Java\jdk1.8.0_60\jre
> Default locale: zh_CN, platform encoding: UTF-8
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"
>Reporter: weiwei
>Assignee: Liang Xie
>Priority: Blocker
>  Labels: build
> Fix For: 0.4.0
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] distributedlog . SUCCESS [  0.002 
> s]
> [INFO] DistributedLog Protocol  FAILURE [  1.081 
> s]
> [INFO] DistributedLog Core Library  SKIPPED
> [INFO] DistributedLog Client .. SKIPPED
> [INFO] DistributedLog Service . SKIPPED
> [INFO] DistributedLog Benchmark ... SKIPPED
> [INFO] DistributedLog Example . SKIPPED
> [INFO] DistributedLog Basics .. SKIPPED
> [INFO] DistributedLog Messaging Tutorial .. SKIPPED
> [INFO] DistributedLog Kafka Tutorial .. SKIPPED
> [INFO] DistributedLog MapReduce ... SKIPPED
> [INFO] DistributedLog Tutorials ... SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 1.321 s
> [INFO] Finished at: 2016-08-02T13:26:31+08:00
> [INFO] Final Memory: 12M/150M
> [INFO] 
> 
> [ERROR] Exception parsing: 
> E:\incubator-distributedlog\distributedlog-protocol\src\main\thrift\service.thrift:
>  [3.1] failure: string matching regex `\z' expected but `/' found
> [ERROR] 
> [ERROR] // Response stats codes
> [ERROR] 
> [ERROR] ^
> [ERROR] -> [Help 1]
> com.twitter.scrooge.frontend.FileParseException: Exception parsing: 
> E:\isoftstone\workspacedeis\incubator-distributedlog\distributedlog-protocol\src\main\thrift\service.thrift
>   at 
> com.twitter.scrooge.frontend.ThriftParser$$anonfun$parse$1.apply(ThriftParser.scala:427)
>   at 
> com.twitter.scrooge.frontend.ThriftParser$$anonfun$parse$1.apply(ThriftParser.scala:427)
>   at scala.Option.map(Option.scala:145)
>   at 
> com.twitter.scrooge.frontend.ThriftParser.parse(ThriftParser.scala:420)
>   at 
> com.twitter.scrooge.frontend.ThriftParser.parseFile(ThriftParser.scala:448)
>   at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:66)
>   at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:64)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at 
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>   at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
>   at com.twitter.scrooge.Compiler.run(Compiler.scala:64)
>   at com.twitter.ScroogeRunner.compile(ScroogeRunner.scala:45)
>   at 
> com.twitter.AbstractMavenScroogeMojo.execute(AbstractMavenScroogeMojo.java:239)
>   at 
> com.twitter.MavenScroogeCompileMojo.execute(MavenScroogeCompileMojo.java:22)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
>   at 

[jira] [Commented] (DL-28) generate java code failed using scrooge

[Tracy Martin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768248#comment-15768248
 ] 

Tracy Martin commented on DL-28:


I noticed after that the .pom for bookkeeper-stats-provider 4.3.6 once I got it 
installed in my local cache looked the same way.

> generate java code failed using scrooge
> ---
>
> Key: DL-28
> URL: https://issues.apache.org/jira/browse/DL-28
> Project: DistributedLog
>  Issue Type: Bug
>  Components: distributedlog-protocol
>Affects Versions: 0.4.0
> Environment: Apache Maven 3.2.5
> Maven home: C:\apache-maven-3.2.5
> Java version: 1.8.0_60, vendor: Oracle Corporation
> Java home: C:\Java\jdk1.8.0_60\jre
> Default locale: zh_CN, platform encoding: UTF-8
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"
>Reporter: weiwei
>Assignee: Liang Xie
>Priority: Blocker
>  Labels: build
> Fix For: 0.4.0
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] distributedlog . SUCCESS [  0.002 
> s]
> [INFO] DistributedLog Protocol  FAILURE [  1.081 
> s]
> [INFO] DistributedLog Core Library  SKIPPED
> [INFO] DistributedLog Client .. SKIPPED
> [INFO] DistributedLog Service . SKIPPED
> [INFO] DistributedLog Benchmark ... SKIPPED
> [INFO] DistributedLog Example . SKIPPED
> [INFO] DistributedLog Basics .. SKIPPED
> [INFO] DistributedLog Messaging Tutorial .. SKIPPED
> [INFO] DistributedLog Kafka Tutorial .. SKIPPED
> [INFO] DistributedLog MapReduce ... SKIPPED
> [INFO] DistributedLog Tutorials ... SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 1.321 s
> [INFO] Finished at: 2016-08-02T13:26:31+08:00
> [INFO] Final Memory: 12M/150M
> [INFO] 
> 
> [ERROR] Exception parsing: 
> E:\incubator-distributedlog\distributedlog-protocol\src\main\thrift\service.thrift:
>  [3.1] failure: string matching regex `\z' expected but `/' found
> [ERROR] 
> [ERROR] // Response stats codes
> [ERROR] 
> [ERROR] ^
> [ERROR] -> [Help 1]
> com.twitter.scrooge.frontend.FileParseException: Exception parsing: 
> E:\isoftstone\workspacedeis\incubator-distributedlog\distributedlog-protocol\src\main\thrift\service.thrift
>   at 
> com.twitter.scrooge.frontend.ThriftParser$$anonfun$parse$1.apply(ThriftParser.scala:427)
>   at 
> com.twitter.scrooge.frontend.ThriftParser$$anonfun$parse$1.apply(ThriftParser.scala:427)
>   at scala.Option.map(Option.scala:145)
>   at 
> com.twitter.scrooge.frontend.ThriftParser.parse(ThriftParser.scala:420)
>   at 
> com.twitter.scrooge.frontend.ThriftParser.parseFile(ThriftParser.scala:448)
>   at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:66)
>   at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:64)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at 
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>   at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
>   at com.twitter.scrooge.Compiler.run(Compiler.scala:64)
>   at com.twitter.ScroogeRunner.compile(ScroogeRunner.scala:45)
>   at 
> com.twitter.AbstractMavenScroogeMojo.execute(AbstractMavenScroogeMojo.java:239)
>   at 
> com.twitter.MavenScroogeCompileMojo.execute(MavenScroogeCompileMojo.java:22)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
>   at 

[jira] [Commented] (DL-151) TestBKLogReadHandler#testGetFirstDLSNWithOpenLedger is flaky

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768147#comment-15768147
 ] 

ASF GitHub Bot commented on DL-151:
---

GitHub user sijie opened a pull request:

https://github.com/apache/incubator-distributedlog/pull/84

DL-151: TestBKLogReadHandler#testGetFirstDLSNWithOpenLedger is flaky

- Flush/Commit records only after the data are written
- Disable immediate flush
- Reduce num writes from 100 to 10

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sijie/incubator-distributedlog 
sijie/FixTestGetFirstDLSNWithOpenLedger

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-distributedlog/pull/84.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #84


commit 4aa78c2cbbfc1a89c25463a3aa0401595471445a
Author: Sijie Guo 
Date:   2016-12-21T20:55:32Z

DL-151: TestBKLogReadHandler#testGetFirstDLSNWithOpenLedger is flaky

- Flush/Commit records only after the data are written
- Disable immediate flush
- Reduce num writes from 100 to 10




> TestBKLogReadHandler#testGetFirstDLSNWithOpenLedger is flaky
> 
>
> Key: DL-151
> URL: https://issues.apache.org/jira/browse/DL-151
> Project: DistributedLog
>  Issue Type: Task
>  Components: distributedlog-core, tests
>Reporter: Sijie Guo
>Assignee: Sijie Guo
>
> {code}
> Failed tests:   
> testGetFirstDLSNWithOpenLedger(com.twitter.distributedlog.TestBKLogReadHandler):
>  expected: but 
> was:
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-distributedlog pull request #84: DL-151: TestBKLogReadHandler#test...

[sijie]

GitHub user sijie opened a pull request:

https://github.com/apache/incubator-distributedlog/pull/84

DL-151: TestBKLogReadHandler#testGetFirstDLSNWithOpenLedger is flaky

- Flush/Commit records only after the data are written
- Disable immediate flush
- Reduce num writes from 100 to 10

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sijie/incubator-distributedlog 
sijie/FixTestGetFirstDLSNWithOpenLedger

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-distributedlog/pull/84.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #84


commit 4aa78c2cbbfc1a89c25463a3aa0401595471445a
Author: Sijie Guo 
Date:   2016-12-21T20:55:32Z

DL-151: TestBKLogReadHandler#testGetFirstDLSNWithOpenLedger is flaky

- Flush/Commit records only after the data are written
- Disable immediate flush
- Reduce num writes from 100 to 10




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DL-151) TestBKLogReadHandler#testGetFirstDLSNWithOpenLedger is flaky

[Sijie Guo (JIRA)]

Sijie Guo created DL-151:


 Summary: TestBKLogReadHandler#testGetFirstDLSNWithOpenLedger is 
flaky
 Key: DL-151
 URL: https://issues.apache.org/jira/browse/DL-151
 Project: DistributedLog
  Issue Type: Task
  Components: distributedlog-core, tests
Reporter: Sijie Guo
Assignee: Sijie Guo


{code}
Failed tests:   
testGetFirstDLSNWithOpenLedger(com.twitter.distributedlog.TestBKLogReadHandler):
 expected: but 
was:
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DL-149) Docker support

[Sijie Guo (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768060#comment-15768060
 ] 

Sijie Guo commented on DL-149:
--

/cc [~franckcuny]

[~franckcuny] was making distributedlog run on k8s. you guys can coordinate on 
that.

> Docker support
> --
>
> Key: DL-149
> URL: https://issues.apache.org/jira/browse/DL-149
> Project: DistributedLog
>  Issue Type: Improvement
>Reporter: Asko Kauppi
>
> I would like to run DistributedLog in two ways:
> - on my local machine, within a development folder, for a project that uses DL
> - on Kubernetes cluster for production
> Better, modularized Docker support would allow both of these to happen. I 
> could use docker-compose to launch a development setup by a one-liner. In 
> Kubernetes, having separate Zookeeper, BookKeeper, Write Proxy and Read Proxy 
> images would allow free adjustment on how many of these are run, and where.
> I found multiple Docker image projects in the Internet, including one 
> Dockerfile in the distributedlog project itself. However, none of them cover 
> the above needs. So I started this: 
> https://github.com/akauppi/distributedlog-blanket
> The Docker images there don't work. I would ask for help to make it better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DL-28) generate java code failed using scrooge

[Tracy Martin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768031#comment-15768031
 ] 

Tracy Martin commented on DL-28:


mvn clean install -DskipTests=true

> generate java code failed using scrooge
> ---
>
> Key: DL-28
> URL: https://issues.apache.org/jira/browse/DL-28
> Project: DistributedLog
>  Issue Type: Bug
>  Components: distributedlog-protocol
>Affects Versions: 0.4.0
> Environment: Apache Maven 3.2.5
> Maven home: C:\apache-maven-3.2.5
> Java version: 1.8.0_60, vendor: Oracle Corporation
> Java home: C:\Java\jdk1.8.0_60\jre
> Default locale: zh_CN, platform encoding: UTF-8
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"
>Reporter: weiwei
>Assignee: Liang Xie
>Priority: Blocker
>  Labels: build
> Fix For: 0.4.0
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] distributedlog . SUCCESS [  0.002 
> s]
> [INFO] DistributedLog Protocol  FAILURE [  1.081 
> s]
> [INFO] DistributedLog Core Library  SKIPPED
> [INFO] DistributedLog Client .. SKIPPED
> [INFO] DistributedLog Service . SKIPPED
> [INFO] DistributedLog Benchmark ... SKIPPED
> [INFO] DistributedLog Example . SKIPPED
> [INFO] DistributedLog Basics .. SKIPPED
> [INFO] DistributedLog Messaging Tutorial .. SKIPPED
> [INFO] DistributedLog Kafka Tutorial .. SKIPPED
> [INFO] DistributedLog MapReduce ... SKIPPED
> [INFO] DistributedLog Tutorials ... SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 1.321 s
> [INFO] Finished at: 2016-08-02T13:26:31+08:00
> [INFO] Final Memory: 12M/150M
> [INFO] 
> 
> [ERROR] Exception parsing: 
> E:\incubator-distributedlog\distributedlog-protocol\src\main\thrift\service.thrift:
>  [3.1] failure: string matching regex `\z' expected but `/' found
> [ERROR] 
> [ERROR] // Response stats codes
> [ERROR] 
> [ERROR] ^
> [ERROR] -> [Help 1]
> com.twitter.scrooge.frontend.FileParseException: Exception parsing: 
> E:\isoftstone\workspacedeis\incubator-distributedlog\distributedlog-protocol\src\main\thrift\service.thrift
>   at 
> com.twitter.scrooge.frontend.ThriftParser$$anonfun$parse$1.apply(ThriftParser.scala:427)
>   at 
> com.twitter.scrooge.frontend.ThriftParser$$anonfun$parse$1.apply(ThriftParser.scala:427)
>   at scala.Option.map(Option.scala:145)
>   at 
> com.twitter.scrooge.frontend.ThriftParser.parse(ThriftParser.scala:420)
>   at 
> com.twitter.scrooge.frontend.ThriftParser.parseFile(ThriftParser.scala:448)
>   at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:66)
>   at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:64)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at 
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>   at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
>   at com.twitter.scrooge.Compiler.run(Compiler.scala:64)
>   at com.twitter.ScroogeRunner.compile(ScroogeRunner.scala:45)
>   at 
> com.twitter.AbstractMavenScroogeMojo.execute(AbstractMavenScroogeMojo.java:239)
>   at 
> com.twitter.MavenScroogeCompileMojo.execute(MavenScroogeCompileMojo.java:22)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
>   at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
>   at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
>   at 

[GitHub] incubator-distributedlog pull request #83: DL: Ensure namespace path is crea...

[sijie]

GitHub user sijie opened a pull request:

https://github.com/apache/incubator-distributedlog/pull/83

DL: Ensure namespace path is created for TestZKLogMetadataForWriter



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sijie/incubator-distributedlog 
sijie/FixTestCreateLogMetadata

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-distributedlog/pull/83.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #83


commit 87662ce63f8040060de94e23cc74dbd456c3b32e
Author: Sijie Guo 
Date:   2016-12-21T19:51:33Z

Ensure namespace path is created




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (DL-150) TestZKLogMetadataForWriter.testCreateLogMetadata is flaky

[Sijie Guo (JIRA)]

Sijie Guo created DL-150:


 Summary: TestZKLogMetadataForWriter.testCreateLogMetadata is flaky
 Key: DL-150
 URL: https://issues.apache.org/jira/browse/DL-150
 Project: DistributedLog
  Issue Type: Task
  Components: tests
Reporter: Sijie Guo
Assignee: Sijie Guo
 Fix For: 0.4.0


{code}
Error Message

Failed to create log /messaging/distributedlog/testCreateLogMetadata/ 
: NONODE

Stacktrace

com.twitter.distributedlog.exceptions.ZKException: Failed to create log 
/messaging/distributedlog/testCreateLogMetadata/ : NONODE
at 
com.twitter.distributedlog.impl.metadata.ZKLogMetadataForWriter$3.processResult(ZKLogMetadataForWriter.java:277)
at 
org.apache.bookkeeper.zookeeper.ZooKeeperClient$3$1.processResult(ZooKeeperClient.java:566)
at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:687)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:528)

{code}

The problem is the namespace uri is not created before tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DL-28) generate java code failed using scrooge

[Tracy Martin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15767955#comment-15767955
 ] 

Tracy Martin commented on DL-28:


Actually, those WARNINGS were from my attempt to build the bookkeeper 4.3.6
snapshot.

On Wed, Dec 21, 2016 at 2:25 PM, Tracy Martin 



> generate java code failed using scrooge
> ---
>
> Key: DL-28
> URL: https://issues.apache.org/jira/browse/DL-28
> Project: DistributedLog
>  Issue Type: Bug
>  Components: distributedlog-protocol
>Affects Versions: 0.4.0
> Environment: Apache Maven 3.2.5
> Maven home: C:\apache-maven-3.2.5
> Java version: 1.8.0_60, vendor: Oracle Corporation
> Java home: C:\Java\jdk1.8.0_60\jre
> Default locale: zh_CN, platform encoding: UTF-8
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"
>Reporter: weiwei
>Assignee: Liang Xie
>Priority: Blocker
>  Labels: build
> Fix For: 0.4.0
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] distributedlog . SUCCESS [  0.002 
> s]
> [INFO] DistributedLog Protocol  FAILURE [  1.081 
> s]
> [INFO] DistributedLog Core Library  SKIPPED
> [INFO] DistributedLog Client .. SKIPPED
> [INFO] DistributedLog Service . SKIPPED
> [INFO] DistributedLog Benchmark ... SKIPPED
> [INFO] DistributedLog Example . SKIPPED
> [INFO] DistributedLog Basics .. SKIPPED
> [INFO] DistributedLog Messaging Tutorial .. SKIPPED
> [INFO] DistributedLog Kafka Tutorial .. SKIPPED
> [INFO] DistributedLog MapReduce ... SKIPPED
> [INFO] DistributedLog Tutorials ... SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 1.321 s
> [INFO] Finished at: 2016-08-02T13:26:31+08:00
> [INFO] Final Memory: 12M/150M
> [INFO] 
> 
> [ERROR] Exception parsing: 
> E:\incubator-distributedlog\distributedlog-protocol\src\main\thrift\service.thrift:
>  [3.1] failure: string matching regex `\z' expected but `/' found
> [ERROR] 
> [ERROR] // Response stats codes
> [ERROR] 
> [ERROR] ^
> [ERROR] -> [Help 1]
> com.twitter.scrooge.frontend.FileParseException: Exception parsing: 
> E:\isoftstone\workspacedeis\incubator-distributedlog\distributedlog-protocol\src\main\thrift\service.thrift
>   at 
> com.twitter.scrooge.frontend.ThriftParser$$anonfun$parse$1.apply(ThriftParser.scala:427)
>   at 
> com.twitter.scrooge.frontend.ThriftParser$$anonfun$parse$1.apply(ThriftParser.scala:427)
>   at scala.Option.map(Option.scala:145)
>   at 
> com.twitter.scrooge.frontend.ThriftParser.parse(ThriftParser.scala:420)
>   at 
> com.twitter.scrooge.frontend.ThriftParser.parseFile(ThriftParser.scala:448)
>   at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:66)
>   at com.twitter.scrooge.Compiler$$anonfun$run$2.apply(Compiler.scala:64)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at 
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
>   at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
>   at com.twitter.scrooge.Compiler.run(Compiler.scala:64)
>   at com.twitter.ScroogeRunner.compile(ScroogeRunner.scala:45)
>   at 
> com.twitter.AbstractMavenScroogeMojo.execute(AbstractMavenScroogeMojo.java:239)
>   at 
> com.twitter.MavenScroogeCompileMojo.execute(MavenScroogeCompileMojo.java:22)
>   at 
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>   at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
>   at 

[jira] [Commented] (DL-99) Enable thrift mux on DL smoketest reader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15767616#comment-15767616
 ] 

ASF GitHub Bot commented on DL-99:
--

GitHub user sijie opened a pull request:

https://github.com/apache/incubator-distributedlog/pull/82

DL-99 - DL-111

This is a umbrella merge for DL-99 to DL-111. Individual commit in this 
pull request is associated with one DL jira.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sijie/incubator-distributedlog 
squashed_merge/DL-111

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-distributedlog/pull/82.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #82


commit 9ff22f4239754d25fb3c7238cf25cf453f0ebbd6
Author: Dave Rusek 
Date:   2016-07-18T19:23:49Z

DL-99: Enable thriftmux for distributedlog smoketest reader

commit 785ff1cdfb515882f934d4bd79fe34bb39cd95a9
Author: Sijie Guo 
Date:   2016-07-29T04:10:33Z

DL-101: Improve session expire handling on fetching log segments for readers

This change focuses on improving the session expire handling on fetching 
log segments for readers.

The log segment management in DL is now done by 3 parts.

- a LogSegmentMetadataStore (one per namespace instance): it is used for 
fetching the log segments from log segment metadata store (ZooKeeper). it 
doesn't do any caching.
- a LogSegmentMetadataCache (one per namespace instance): it is a guava 
cache based metadata cache. it maintains a mapping between log segment metadata 
path and the log segment metadata. it manages the cache for the log segments 
that will be accessed in this namespace instance. it doesn't manage the 
sequence of the log segments for streams.
- a PerStreamLogSegmentCache for each BKLogHandler. the log segment cache 
is per stream. it maintains the sequence of the log segments.

BKLogWriteHandler doesn't watch the log segment changes. It fetches minimal 
number of log segments when it is created and fetches the full list of log 
segments for truncations. New log segments will be added to the per stream log 
segment cache with log segment rolling.

BKLogReadHandler watch the log segments changes and only notify when the 
list of log segments is changed. the session handling which is specific to the 
metadata store is hidden to the implementations of LogSegmentMetadataStore.

The change tries to cleanup bunch of unused methods in 
BKLog{Read,Write}Handler too.

commit 10ff44510437d4087df5dc4775d968e174df83b0
Author: Leigh Stewart 
Date:   2016-07-29T04:36:49Z

DL-102: Add routing service to write proxy server side

this change is to add getOwner rpc in write proxy. so we can change the 
client side to get owner from write proxy first for routing service. in this 
way, we can start experiementing any resource placement algorithms.

commit 9f80812ab4293de2f7073642f7ef452b6a88f8f6
Author: Sijie Guo 
Date:   2016-07-29T05:02:09Z

DL-103: DL client can use server-side routing service to retrieve 
ownerships for streams

If the dl client configures a routing service finagle name, the client will 
use this to retrieve ownerships for streams.

commit 75774256dbaa5fcccb2e0d319e5d7d272a976901
Author: Jordan Bull 
Date:   2016-08-09T18:36:53Z

DL-104: BKClientReadTimeout can be equal to getReadLACLongPollTimeout so 
current config validates

commit afe64639b871b4dbc564f904fc4f522c0de334d3
Author: Yiming Zang 
Date:   2016-08-10T23:18:53Z

DL-105: Make compression stats available per stream

commit 36a07f6b1d5d3fde395687f97ee10fb9376b5d7b
Author: Yiming Zang 
Date:   2016-08-12T17:30:39Z

DL-106: Use namespace after it is closed will throw AlreadyClosedException

commit d156c1d6223034fceba90750febc42d25e389d5c
Author: Phillip Su 
Date:   2016-08-23T00:47:28Z

DL-107: Added unregistering gauges for distributedlog-core and 
distributedlog-benchmark

commit e18cd48c2f9a9d03dfc7f46e96e74ebaa1471659
Author: Yiming Zang 
Date:   2016-09-17T00:35:51Z

DL-108: Log rate limiting more clearly

commit a743b44f4614154c67cc4d6a98a9003d80062d70
Author: Sijie Guo 
Date:   2016-10-27T17:18:52Z

DL-109: Add a tool to find the stream containing the given ledger

commit 5f50a41b3883e2107befa95f774687deae9d62fe
Author: Sijie Guo 
Date:   2016-11-22T00:17:51Z

DL-110: Write control record if necessary when roll a log segment

commit 0d2e20c7c9189dec7909ae67f4a62443a9aa40bc
Author: Sijie Guo 
Date:   2016-11-22T01:30:13Z

DL-111: 

[jira] [Commented] (DL-102) Add routing service to write proxy server side

[Leigh Stewart (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15767568#comment-15767568
 ] 

Leigh Stewart commented on DL-102:
--

Serverside depends on TwitterRegionResolver

So I guess we'll need to use the generic region resolver here.

> Add routing service to write proxy server side
> --
>
> Key: DL-102
> URL: https://issues.apache.org/jira/browse/DL-102
> Project: DistributedLog
>  Issue Type: Improvement
>Reporter: Sijie Guo
>Assignee: Leigh Stewart
>
> this change is to add getOwner rpc in write proxy. so we can change the 
> client side to get owner from write proxy first for
>  routing service. in this way, we can start experiementing any resource 
> placement algorithms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Is there a REST API for Write and Read proxies?

[Leigh Stewart]

The challenge is the protocol uses a redirection mechanism so theres a
protocol beyond thrift/transport.

As Jay says we plan to make improvements in this area.

On Wed, Dec 21, 2016 at 8:53 AM, Jay Juma  wrote:

> Hi Asko,
>
> I don't think there is a REST api available in the proxy service. The API
> seems to be thrift-rpc based. I found there is a JIRA to support gPRC wire
> protocol. It should not be difficult to add a REST api.
>
> - Jay
>
> On Wed, Dec 21, 2016 at 4:29 AM, Asko Kauppi 
> wrote:
>
> > I’m reading http://distributedlog.incubator.apache.org/docs/
> > latest/user_guide/api/proxy.html  > incubator.apache.org/docs/latest/user_guide/api/proxy.html>
> >
> > Ideally, I wouldn’t need to use a library to talk to a proxy service,
> > right? Is there documentation on how to access the proxies as REST
> > endpoints / are they such?
> >
> > My preferred environment is Scala and akka-http.
> >
> > Asko Kauppi
> > Zalando Tech Helsinki
> >
> >
>

Re: Is there a REST API for Write and Read proxies?

[Jay Juma]

Hi Asko,

I don't think there is a REST api available in the proxy service. The API
seems to be thrift-rpc based. I found there is a JIRA to support gPRC wire
protocol. It should not be difficult to add a REST api.

- Jay

On Wed, Dec 21, 2016 at 4:29 AM, Asko Kauppi  wrote:

> I’m reading http://distributedlog.incubator.apache.org/docs/
> latest/user_guide/api/proxy.html  incubator.apache.org/docs/latest/user_guide/api/proxy.html>
>
> Ideally, I wouldn’t need to use a library to talk to a proxy service,
> right? Is there documentation on how to access the proxies as REST
> endpoints / are they such?
>
> My preferred environment is Scala and akka-http.
>
> Asko Kauppi
> Zalando Tech Helsinki
>
>

[GitHub] incubator-distributedlog pull request #80: Add namespace watch tool

[leighst]

GitHub user leighst opened a pull request:

https://github.com/apache/incubator-distributedlog/pull/80

Add namespace watch tool

Port simple namespace watch tool from downstream

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/leighst/incubator-distributedlog 
lstewart/dlog/watch_tool

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-distributedlog/pull/80.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #80


commit 7401679048f7bade8feba0d38746e8002168c066
Author: Leigh Stewart 
Date:   2016-12-21T15:55:29Z

Add watch tool to DL




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (DL-97) Remove unused methods in BKLogHandler

[Leigh Stewart (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Leigh Stewart updated DL-97:

Assignee: Sijie Guo  (was: Leigh Stewart)

> Remove unused methods in BKLogHandler
> -
>
> Key: DL-97
> URL: https://issues.apache.org/jira/browse/DL-97
> Project: DistributedLog
>  Issue Type: Sub-task
>  Components: distributedlog-core
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 0.4.0
>
>
> Remove unused methods in BKLogHandler



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Is there a REST API for Write and Read proxies?

[Asko Kauppi]

I’m reading 
http://distributedlog.incubator.apache.org/docs/latest/user_guide/api/proxy.html
 


Ideally, I wouldn’t need to use a library to talk to a proxy service, right? Is 
there documentation on how to access the proxies as REST endpoints / are they 
such?

My preferred environment is Scala and akka-http.

Asko Kauppi
Zalando Tech Helsinki

[jira] [Resolved] (DL-146) Fix Stream typo : initiaize

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo resolved DL-146.
--
   Resolution: Fixed
Fix Version/s: 0.4.0

Merge the fix. Thanks [~xieliang007]

> Fix Stream typo : initiaize
> ---
>
> Key: DL-146
> URL: https://issues.apache.org/jira/browse/DL-146
> Project: DistributedLog
>  Issue Type: Improvement
>  Components: distributedlog-service
>Reporter: Liang Xie
>Assignee: Liang Xie
>Priority: Minor
> Fix For: 0.4.0
>
>
> "moved to initiaize()" -> "moved to initialize()"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DL-93) LogSegmentMetadataStore should only notify when the list of log segments is updated

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-93?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo resolved DL-93.
-
Resolution: Fixed

> LogSegmentMetadataStore should only notify when the list of log segments is 
> updated
> ---
>
> Key: DL-93
> URL: https://issues.apache.org/jira/browse/DL-93
> Project: DistributedLog
>  Issue Type: Improvement
>  Components: distributedlog-core
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 0.4.0
>
>
> Currently it notifies the listeners not only when there is a change but also 
> when session expires. it would break the reada
> head loop and cause readers have to wait until it is able to connect to 
> zookeeper again.
> With this change, it would only notify when the list of log segments is 
> updated. If it disconnects to zookeeper, the listen
> er won't be notified and it would keep reading from the log segments it knows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DL-94) Make the zookeeper client used by bookkeeper client retry on session expires

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-94?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo resolved DL-94.
-
Resolution: Fixed

> Make the zookeeper client used by bookkeeper client retry on session expires
> 
>
> Key: DL-94
> URL: https://issues.apache.org/jira/browse/DL-94
> Project: DistributedLog
>  Issue Type: Improvement
>  Components: distributedlog-core
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 0.4.0
>
>
>  * the zookeeper client used by bookkeeper client is purely for metadata 
> accesses, so we should retry on session expires.
> * remove the unnessary zookeeper session handling in bk log handler. as 
> we don't necessary to fail bookkeeper client or log
>  handler when session expires as it would be handled and retried by the 
> zookeeper client.
> * Make the retry infinitely if the retry settings for bkc zookeeper 
> client is set to 0 or negative.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DL-92) Simplify the state transition on stream

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DL-92?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo resolved DL-92.
-
   Resolution: Fixed
Fix Version/s: 0.4.0

> Simplify the state transition on stream
> ---
>
> Key: DL-92
> URL: https://issues.apache.org/jira/browse/DL-92
> Project: DistributedLog
>  Issue Type: Improvement
>  Components: distributedlog-service
>Affects Versions: 0.4.0
>Reporter: Sijie Guo
>Assignee: Leigh Stewart
> Fix For: 0.4.0
>
>
> Simplify the state transition on stream
> * the stream is created on INITIALIZING
> * when the stream is started, it would start transition from INITIALIZING 
> to INITIALIZED
> * it would serve stream operations when the stream is INITIALIZED
> * it would be turned to ERROR when encountered exceptions.
> * the stream would be closed when service operation timeout or 
> encountered any exceptions. it would first be removed from  acquired mapping
> * the stream would be removed from cached mapping depends on probation 
> time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-distributedlog pull request #68: DL-95: Assign host ip client id i...

[sijie]

Github user sijie closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/68


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (DL-93) LogSegmentMetadataStore should only notify when the list of log segments is updated

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766423#comment-15766423
 ] 

ASF GitHub Bot commented on DL-93:
--

Github user sijie closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/66


> LogSegmentMetadataStore should only notify when the list of log segments is 
> updated
> ---
>
> Key: DL-93
> URL: https://issues.apache.org/jira/browse/DL-93
> Project: DistributedLog
>  Issue Type: Improvement
>  Components: distributedlog-core
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 0.4.0
>
>
> Currently it notifies the listeners not only when there is a change but also 
> when session expires. it would break the reada
> head loop and cause readers have to wait until it is able to connect to 
> zookeeper again.
> With this change, it would only notify when the list of log segments is 
> updated. If it disconnects to zookeeper, the listen
> er won't be notified and it would keep reading from the log segments it knows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-distributedlog pull request #67: DL-94: Make the zookeeper client ...

[sijie]

Github user sijie closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/67


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-distributedlog pull request #65: DL-92: Simplify the state transit...

[sijie]

Github user sijie closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/65


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-distributedlog pull request #69: DL-97: Remove unused methods in B...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/69


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (DL-146) Fix Stream typo : initiaize

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DL-146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766417#comment-15766417
 ] 

ASF GitHub Bot commented on DL-146:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/79


> Fix Stream typo : initiaize
> ---
>
> Key: DL-146
> URL: https://issues.apache.org/jira/browse/DL-146
> Project: DistributedLog
>  Issue Type: Improvement
>  Components: distributedlog-service
>Reporter: Liang Xie
>Assignee: Liang Xie
>Priority: Minor
>
> "moved to initiaize()" -> "moved to initialize()"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-distributedlog issue #79: DL-146 : fix Stream typo : initiaize -> ...

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/79
  
+1 LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[incubator-distributedlog] Git Push Summary

[sijie]

Repository: incubator-distributedlog
Updated Branches:
  refs/heads/merge/DL-98 [deleted] 28a8b8ff9

[15/29] incubator-distributedlog git commit: Remove unused methods in BKLogHandler

[sijie]

Remove unused methods in BKLogHandler


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/b7ae590e
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/b7ae590e
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/b7ae590e

Branch: refs/heads/merge/DL-98
Commit: b7ae590e8b819ae63c0139611320dcc0d27e39f7
Parents: f4f633f
Author: Leigh Stewart 
Authored: Mon Dec 12 17:01:43 2016 -0800
Committer: Sijie Guo 
Committed: Mon Dec 12 17:01:43 2016 -0800

--
 .../twitter/distributedlog/BKLogHandler.java| 107 ---
 1 file changed, 107 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b7ae590e/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
index 3b991e2..460de11 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
@@ -32,9 +32,7 @@ import com.twitter.distributedlog.io.AsyncCloseable;
 import com.twitter.distributedlog.logsegment.LogSegmentCache;
 import com.twitter.distributedlog.logsegment.LogSegmentFilter;
 import com.twitter.distributedlog.logsegment.LogSegmentMetadataStore;
-import com.twitter.distributedlog.util.FutureUtils;
 import com.twitter.distributedlog.util.OrderedScheduler;
-import com.twitter.distributedlog.util.Utils;
 import com.twitter.util.Function;
 import com.twitter.util.Future;
 import com.twitter.util.FutureEventListener;
@@ -103,8 +101,6 @@ import java.util.concurrent.atomic.AtomicReference;
 public abstract class BKLogHandler implements Watcher, AsyncCloseable, 
AsyncAbortable {
 static final Logger LOG = LoggerFactory.getLogger(BKLogHandler.class);
 
-private static final int LAYOUT_VERSION = -1;
-
 protected final ZKLogMetadata logMetadata;
 protected final DistributedLogConfiguration conf;
 protected final ZooKeeperClient zooKeeperClient;
@@ -458,57 +454,6 @@ public abstract class BKLogHandler implements Watcher, 
AsyncCloseable, AsyncAbor
 }
 }
 
-public LogRecordWithDLSN getLastLogRecord(boolean recover, boolean 
includeEndOfStream) throws IOException {
-checkLogStreamExists();
-List ledgerList = getFullLedgerListDesc(true, 
true);
-
-for (LogSegmentMetadata metadata: ledgerList) {
-LogRecordWithDLSN record = recoverLastRecordInLedger(metadata, 
recover, false, includeEndOfStream);
-
-if (null != record) {
-assert(!record.isControl());
-LOG.debug("{} getLastLogRecord Returned {}", 
getFullyQualifiedName(), record);
-return record;
-}
-}
-
-throw new LogEmptyException("Log " + getFullyQualifiedName() + " has 
no records");
-}
-
-public long getLastTxId(boolean recover,
-boolean includeEndOfStream) throws IOException {
-checkLogStreamExists();
-return getLastLogRecord(recover, 
includeEndOfStream).getTransactionId();
-}
-
-public DLSN getLastDLSN(boolean recover,
-boolean includeEndOfStream) throws IOException {
-checkLogStreamExists();
-return getLastLogRecord(recover, includeEndOfStream).getDlsn();
-}
-
-public long getLogRecordCount() throws IOException {
-try {
-checkLogStreamExists();
-} catch (LogNotFoundException exc) {
-return 0;
-}
-
-List ledgerList = getFullLedgerList(true, false);
-long count = 0;
-for (LogSegmentMetadata l : ledgerList) {
-if (l.isInProgress()) {
-LogRecord record = recoverLastRecordInLedger(l, false, false, 
false);
-if (null != record) {
-count += record.getLastPositionWithinLogSegment();
-}
-} else {
-count += l.getRecordCount();
-}
-}
-return count;
-}
-
 private Future 
asyncReadFirstUserRecord(LogSegmentMetadata ledger, DLSN beginDLSN) {
 final LedgerHandleCache handleCache =
 
LedgerHandleCache.newBuilder().bkc(bookKeeperClient).conf(conf).build();
@@ -620,15 +565,6 @@ public abstract class BKLogHandler implements Watcher, 
AsyncCloseable, AsyncAbor
 return sum;
 }
 
-public long getFirstTxId() throws IOException {
-checkLogStreamExists();
-

[11/29] incubator-distributedlog git commit: Allow configuring flush interval in micros for distributedlog multi streams writer

[sijie]

Allow configuring flush interval in micros for distributedlog multi streams 
writer

RB_ID=846974


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/f008f751
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/f008f751
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/f008f751

Branch: refs/heads/merge/DL-98
Commit: f008f751ec0a020a670903b01c709007ada87381
Parents: f19e756
Author: Sijie Guo 
Authored: Fri Jul 8 15:22:55 2016 -0700
Committer: Sijie Guo 
Committed: Mon Dec 12 16:52:03 2016 -0800

--

--

[12/29] incubator-distributedlog git commit: LogSegmentMetadataStore should only notify when the list of log segments is updated

[sijie]

LogSegmentMetadataStore should only notify when the list of log segments is 
updated

Currently it notifies the listeners not only when there is a change but also 
when session expires. it would break the readahead loop and cause readers have 
to wait until it is able to connect to zookeeper again.

With this change, it would only notify when the list of log segments is 
updated. If it disconnects to zookeeper, the listener won't be notified and it 
would keep reading from the log segments it knows.

RB_ID=842998


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/72a786e7
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/72a786e7
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/72a786e7

Branch: refs/heads/merge/DL-98
Commit: 72a786e78540fb5a3f8e27f9d71d3616d06d1548
Parents: f008f75
Author: Sijie Guo 
Authored: Mon Jul 11 10:10:55 2016 -0700
Committer: Sijie Guo 
Committed: Mon Dec 12 16:52:18 2016 -0800

--
 .../impl/ZKLogSegmentMetadataStore.java | 94 
 .../impl/TestZKLogSegmentMetadataStore.java | 22 ++---
 2 files changed, 84 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/72a786e7/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/ZKLogSegmentMetadataStore.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/ZKLogSegmentMetadataStore.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/ZKLogSegmentMetadataStore.java
index c0796a1..cb53b23 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/ZKLogSegmentMetadataStore.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/ZKLogSegmentMetadataStore.java
@@ -17,6 +17,7 @@
  */
 package com.twitter.distributedlog.impl;
 
+import com.google.common.collect.ImmutableList;
 import com.twitter.distributedlog.DistributedLogConfiguration;
 import com.twitter.distributedlog.LogSegmentMetadata;
 import com.twitter.distributedlog.ZooKeeperClient;
@@ -45,10 +46,14 @@ import org.apache.zookeeper.Watcher;
 import org.apache.zookeeper.data.Stat;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
+import scala.runtime.AbstractFunction1;
 
 import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
+import java.util.Map;
 import java.util.Set;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ConcurrentMap;
@@ -64,7 +69,9 @@ public class ZKLogSegmentMetadataStore implements 
LogSegmentMetadataStore, Watch
 
 private static final Logger logger = 
LoggerFactory.getLogger(ZKLogSegmentMetadataStore.class);
 
-private static class ReadLogSegmentsTask implements Runnable, 
FutureEventListener {
+private static final List EMPTY_LIST = ImmutableList.of();
+
+private static class ReadLogSegmentsTask implements Runnable, 
FutureEventListener> {
 
 private final String logSegmentsPath;
 private final ZKLogSegmentMetadataStore store;
@@ -78,15 +85,16 @@ public class ZKLogSegmentMetadataStore implements 
LogSegmentMetadataStore, Watch
 }
 
 @Override
-public void onSuccess(final List segments) {
+public void onSuccess(final Versioned segments) {
 // reset the back off after a successful operation
 currentZKBackOffMs = store.minZKBackoffMs;
-final Set listenerSet = 
store.listeners.get(logSegmentsPath);
+final Map listenerSet =
+store.listeners.get(logSegmentsPath);
 if (null != listenerSet) {
 store.submitTask(logSegmentsPath, new Runnable() {
 @Override
 public void run() {
-for (LogSegmentNamesListener listener : listenerSet) {
+for (VersionedLogSegmentNamesListener listener : 
listenerSet.values()) {
 listener.onSegmentsUpdated(segments);
 }
 }
@@ -120,6 +128,48 @@ public class ZKLogSegmentMetadataStore implements 
LogSegmentMetadataStore, Watch
 }
 }
 
+/**
+ * A log segment names listener that keeps tracking the version of list of 
log segments that it has been notified.
+ * It only notify the newer log segments.
+ */
+static class VersionedLogSegmentNamesListener {
+
+private final LogSegmentNamesListener listener;
+

[17/29] incubator-distributedlog git commit: Merge remote-tracking branch 'apache/master' into merge/DL-90

[sijie]

Merge remote-tracking branch 'apache/master' into merge/DL-90


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/c4a6020c
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/c4a6020c
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/c4a6020c

Branch: refs/heads/merge/DL-98
Commit: c4a6020c39d3a91473a3dad4b687320deaf7bc63
Parents: 800b867 132dadc
Author: Sijie Guo 
Authored: Fri Dec 16 23:35:36 2016 -0800
Committer: Sijie Guo 
Committed: Fri Dec 16 23:35:36 2016 -0800

--
 .gitignore  |   4 +
 .travis.yml |   4 +
 Dockerfile  |  10 +-
 Vagrantfile |   5 +-
 distributedlog-benchmark/bin/bundle |   7 +-
 distributedlog-benchmark/bin/dbench |  15 +-
 distributedlog-benchmark/conf/dlogenv.sh|   2 +-
 .../routing/ConsistentHashRoutingService.java   |   2 +-
 distributedlog-core/bin/dlog|  12 +-
 distributedlog-core/conf/write_proxy.conf   |   2 +-
 distributedlog-service/bin/bundle   |   7 +-
 distributedlog-service/bin/dlog |  10 +-
 distributedlog-service/bin/dlog-daemon.sh   | 170 +--
 .../distributedlog-kafka/bin/runner |   1 -
 docker/Dockerfile   |  35 
 docs/admin_guide/vagrant.rst|   4 +-
 docs/basics/introduction.rst|  14 +-
 docs/deployment/cluster.rst |  58 ---
 docs/deployment/docker.rst  |   2 +-
 pom.xml |  11 +-
 scripts/bundle  |   2 -
 scripts/common.sh   |  10 +-
 scripts/integration/smoketest.sh| 113 
 scripts/runner  |  12 +-
 scripts/snapshot|  11 +-
 vagrant/base.sh |  43 ++---
 vagrant/bk.sh   |  64 +++
 vagrant/zk.sh   |   5 +-
 28 files changed, 361 insertions(+), 274 deletions(-)
--

[06/29] incubator-distributedlog git commit: Remove watcher only when it registered watcher

[sijie]

 Remove watcher only when it registered watcher

 (reduce the annoying logging from zookeeper client about "Failed to find 
watcher!"

RB_ID=837073


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/98dc9ab2
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/98dc9ab2
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/98dc9ab2

Branch: refs/heads/merge/DL-98
Commit: 98dc9ab2df8af7af26ca68dbe93be3d420cda417
Parents: b571d3b
Author: Sijie Guo 
Authored: Thu May 26 17:06:31 2016 -0700
Committer: Sijie Guo 
Committed: Mon Dec 12 16:37:27 2016 -0800

--
 .../main/java/com/twitter/distributedlog/BKLogHandler.java| 7 ---
 .../java/com/twitter/distributedlog/BKLogReadHandler.java | 7 ++-
 .../java/com/twitter/distributedlog/BKLogWriteHandler.java| 6 +-
 .../com/twitter/distributedlog/readahead/ReadAheadWorker.java | 2 +-
 .../java/com/twitter/distributedlog/zk/ZKWatcherManager.java  | 4 ++--
 .../com/twitter/distributedlog/zk/TestZKWatcherManager.java   | 2 +-
 6 files changed, 15 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/98dc9ab2/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
index 9aa3465..a6ec318 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
@@ -702,13 +702,6 @@ public abstract class BKLogHandler implements Watcher, 
AsyncCloseable, AsyncAbor
 }
 
 @Override
-public Future asyncClose() {
-// No-op
-
this.zooKeeperClient.getWatcherManager().unregisterChildWatcher(logMetadata.getLogSegmentsPath(),
 this);
-return Future.Void();
-}
-
-@Override
 public Future asyncAbort() {
 return asyncClose();
 }

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/98dc9ab2/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogReadHandler.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogReadHandler.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogReadHandler.java
index 0bf6b84..6a8f90e 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogReadHandler.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogReadHandler.java
@@ -311,7 +311,12 @@ class BKLogReadHandler extends BKLogHandler {
 if (null != handleCache) {
 handleCache.clear();
 }
-return BKLogReadHandler.super.asyncClose();
+// No-op
+zooKeeperClient.getWatcherManager().unregisterChildWatcher(
+logMetadata.getLogSegmentsPath(),
+BKLogReadHandler.this,
+true);
+return Future.Void();
 }
 });
 }

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/98dc9ab2/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogWriteHandler.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogWriteHandler.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogWriteHandler.java
index 573679a..4665ed5 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogWriteHandler.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogWriteHandler.java
@@ -1263,7 +1263,11 @@ class BKLogWriteHandler extends BKLogHandler {
 ).flatMap(new AbstractFunction1() {
 @Override
 public Future apply(Void result) {
-return BKLogWriteHandler.super.asyncClose();
+zooKeeperClient.getWatcherManager().unregisterChildWatcher(
+logMetadata.getLogSegmentsPath(),
+BKLogWriteHandler.this,
+false);
+return Future.Void();
 }
 });
 }

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/98dc9ab2/distributedlog-core/src/main/java/com/twitter/distributedlog/readahead/ReadAheadWorker.java

[24/29] incubator-distributedlog git commit: Merge branch 'merge/DL-95' into merge/DL-96

[sijie]

Merge branch 'merge/DL-95' into merge/DL-96


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/a1a35167
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/a1a35167
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/a1a35167

Branch: refs/heads/merge/DL-98
Commit: a1a351676be9864bbce1e9ba27deadfff79f6b02
Parents: b7ae590 749657f
Author: Sijie Guo 
Authored: Fri Dec 16 23:53:50 2016 -0800
Committer: Sijie Guo 
Committed: Fri Dec 16 23:53:50 2016 -0800

--
 .gitignore  |   4 +
 .travis.yml |   4 +
 Dockerfile  |  10 +-
 Vagrantfile |   5 +-
 distributedlog-benchmark/bin/bundle |   7 +-
 distributedlog-benchmark/bin/dbench |  15 +-
 distributedlog-benchmark/conf/dlogenv.sh|   2 +-
 .../routing/ConsistentHashRoutingService.java   |   2 +-
 distributedlog-core/bin/dlog|  12 +-
 distributedlog-core/conf/write_proxy.conf   |   2 +-
 .../distributedlog/DistributedLogConstants.java |   2 +-
 distributedlog-service/bin/bundle   |   7 +-
 distributedlog-service/bin/dlog |  10 +-
 distributedlog-service/bin/dlog-daemon.sh   | 170 +--
 .../distributedlog-kafka/bin/runner |   1 -
 docker/Dockerfile   |  35 
 docs/admin_guide/vagrant.rst|   4 +-
 docs/basics/introduction.rst|  14 +-
 docs/deployment/cluster.rst |  58 ---
 docs/deployment/docker.rst  |   2 +-
 pom.xml |  11 +-
 scripts/bundle  |   2 -
 scripts/common.sh   |  10 +-
 scripts/integration/smoketest.sh| 113 
 scripts/runner  |  12 +-
 scripts/snapshot|  11 +-
 vagrant/base.sh |  43 ++---
 vagrant/bk.sh   |  64 +++
 vagrant/zk.sh   |   5 +-
 29 files changed, 362 insertions(+), 275 deletions(-)
--

[20/29] incubator-distributedlog git commit: Merge branch 'merge/DL-91' into merge/DL-92

[sijie]

Merge branch 'merge/DL-91' into merge/DL-92


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/adc8ad19
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/adc8ad19
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/adc8ad19

Branch: refs/heads/merge/DL-98
Commit: adc8ad198d59a05dc623818c0948f8f219ce0a8b
Parents: f19e756 4fc6f3d
Author: Sijie Guo 
Authored: Fri Dec 16 23:51:41 2016 -0800
Committer: Sijie Guo 
Committed: Fri Dec 16 23:51:41 2016 -0800

--
 .gitignore  |   4 +
 .travis.yml |   4 +
 Dockerfile  |  10 +-
 Vagrantfile |   5 +-
 distributedlog-benchmark/bin/bundle |   7 +-
 distributedlog-benchmark/bin/dbench |  15 +-
 distributedlog-benchmark/conf/dlogenv.sh|   2 +-
 .../routing/ConsistentHashRoutingService.java   |   2 +-
 distributedlog-core/bin/dlog|  12 +-
 distributedlog-core/conf/write_proxy.conf   |   2 +-
 .../distributedlog/DistributedLogConstants.java |   2 +-
 distributedlog-service/bin/bundle   |   7 +-
 distributedlog-service/bin/dlog |  10 +-
 distributedlog-service/bin/dlog-daemon.sh   | 170 +--
 .../distributedlog-kafka/bin/runner |   1 -
 docker/Dockerfile   |  35 
 docs/admin_guide/vagrant.rst|   4 +-
 docs/basics/introduction.rst|  14 +-
 docs/deployment/cluster.rst |  58 ---
 docs/deployment/docker.rst  |   2 +-
 pom.xml |  11 +-
 scripts/bundle  |   2 -
 scripts/common.sh   |  10 +-
 scripts/integration/smoketest.sh| 113 
 scripts/runner  |  12 +-
 scripts/snapshot|  11 +-
 vagrant/base.sh |  43 ++---
 vagrant/bk.sh   |  64 +++
 vagrant/zk.sh   |   5 +-
 29 files changed, 362 insertions(+), 275 deletions(-)
--

[01/29] incubator-distributedlog git commit: when publishing per partition stats, also publish per stream stats

[sijie]

Repository: incubator-distributedlog
Updated Branches:
  refs/heads/merge/DL-98 [created] 28a8b8ff9


when publishing per partition stats, also publish per stream stats

RB_ID=820062


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/904b8986
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/904b8986
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/904b8986

Branch: refs/heads/merge/DL-98
Commit: 904b8986b46fc908cfd1a0ee05b35dce77d12f5c
Parents: 3fceccc
Author: Jordan Bull 
Authored: Fri May 13 11:27:19 2016 -0700
Committer: Sijie Guo 
Committed: Mon Dec 12 16:16:37 2016 -0800

--
 .../service/DistributedLogServiceImpl.java  | 11 +++
 .../service/stream/BulkWriteOp.java |  8 ++--
 .../service/stream/StreamImpl.java  |  4 ++--
 .../service/stream/StreamOpStats.java   | 20 
 .../distributedlog/service/stream/WriteOp.java  |  8 ++--
 .../service/TestDistributedLogServer.java   |  8 
 .../service/TestDistributedLogService.java  |  1 +
 .../service/stream/TestStreamOp.java|  3 ++-
 8 files changed, 40 insertions(+), 23 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/904b8986/distributedlog-service/src/main/java/com/twitter/distributedlog/service/DistributedLogServiceImpl.java
--
diff --git 
a/distributedlog-service/src/main/java/com/twitter/distributedlog/service/DistributedLogServiceImpl.java
 
b/distributedlog-service/src/main/java/com/twitter/distributedlog/service/DistributedLogServiceImpl.java
index de475ea..751e972 100644
--- 
a/distributedlog-service/src/main/java/com/twitter/distributedlog/service/DistributedLogServiceImpl.java
+++ 
b/distributedlog-service/src/main/java/com/twitter/distributedlog/service/DistributedLogServiceImpl.java
@@ -132,6 +132,7 @@ public class DistributedLogServiceImpl implements 
DistributedLogService.ServiceI
 // Stats
 private final StatsLogger statsLogger;
 private final StatsLogger perStreamStatsLogger;
+private final StreamPartitionConverter streamPartitionConverter;
 private final StreamOpStats streamOpStats;
 private final Counter bulkWritePendingStat;
 private final Counter writePendingStat;
@@ -158,6 +159,7 @@ public class DistributedLogServiceImpl implements 
DistributedLogService.ServiceI
 this.perStreamStatsLogger = perStreamStatsLogger;
 this.dlsnVersion = serverConf.getDlsnVersion();
 this.serverRegionId = serverConf.getRegionId();
+this.streamPartitionConverter = converter;
 int serverPort = serverConf.getServerPort();
 int shard = serverConf.getServerShardId();
 int numThreads = serverConf.getServerThreads();
@@ -396,8 +398,8 @@ public class DistributedLogServiceImpl implements 
DistributedLogService.ServiceI
 public Future writeBulkWithContext(final String stream, 
List data, WriteContext ctx) {
 bulkWritePendingStat.inc();
 receivedRecordCounter.add(data.size());
-BulkWriteOp op = new BulkWriteOp(stream, data, statsLogger, 
perStreamStatsLogger, getChecksum(ctx),
-featureChecksumDisabled, accessControlManager);
+BulkWriteOp op = new BulkWriteOp(stream, data, statsLogger, 
perStreamStatsLogger, streamPartitionConverter,
+getChecksum(ctx), featureChecksumDisabled, accessControlManager);
 executeStreamOp(op);
 return op.result().ensure(new Function0() {
 public BoxedUnit apply() {
@@ -675,8 +677,9 @@ public class DistributedLogServiceImpl implements 
DistributedLogService.ServiceI
ByteBuffer data,
Long checksum,
boolean isRecordSet) {
-return new WriteOp(stream, data, statsLogger, perStreamStatsLogger, 
serverConfig, dlsnVersion,
-checksum, isRecordSet, featureChecksumDisabled, 
accessControlManager);
+return new WriteOp(stream, data, statsLogger, perStreamStatsLogger, 
streamPartitionConverter,
+serverConfig, dlsnVersion, checksum, isRecordSet, 
featureChecksumDisabled,
+accessControlManager);
 }
 
 @VisibleForTesting

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/904b8986/distributedlog-service/src/main/java/com/twitter/distributedlog/service/stream/BulkWriteOp.java
--
diff --git 
a/distributedlog-service/src/main/java/com/twitter/distributedlog/service/stream/BulkWriteOp.java
 

[04/29] incubator-distributedlog git commit: Introduce periodic keepalive control record in writer

[sijie]

Introduce periodic keepalive control record in writer

* so the writer will periodically write 'keepalive' control record to make 
sure the stream is alive. so if the write proxy is disconnected from bookies, 
the control record will fail to write hence write proxy gets a chance to drop 
the ownership.


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/517c77c1
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/517c77c1
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/517c77c1

Branch: refs/heads/merge/DL-98
Commit: 517c77c164cb989ae9829cbd80bf2e492eb8e364
Parents: 7b46a9a
Author: Leigh Stewart 
Authored: Mon Dec 12 16:33:33 2016 -0800
Committer: Sijie Guo 
Committed: Mon Dec 12 16:33:33 2016 -0800

--
 .../distributedlog/BKLogSegmentWriter.java  | 47 -
 .../DistributedLogConfiguration.java| 25 +
 .../distributedlog/DistributedLogConstants.java |  1 +
 .../distributedlog/TestAsyncReaderWriter.java   | 53 
 4 files changed, 125 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/517c77c1/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
index 004b2fb..1b52951 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
@@ -72,6 +72,7 @@ import org.apache.bookkeeper.stats.Counter;
 import org.apache.bookkeeper.stats.Gauge;
 import org.apache.bookkeeper.stats.OpStatsLogger;
 import org.apache.bookkeeper.stats.StatsLogger;
+import org.apache.bookkeeper.util.MathUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import scala.runtime.AbstractFunction1;
@@ -128,6 +129,8 @@ class BKLogSegmentWriter implements LogSegmentWriter, 
AddCallback, Runnable, Siz
 private long numFlushesSinceRestart = 0;
 private long numBytes = 0;
 private long lastEntryId = Long.MIN_VALUE;
+private long lastTransmitNanos = Long.MIN_VALUE;
+private final int periodicKeepAliveMs;
 
 // Indicates whether there are writes that have been successfully 
transmitted that would need
 // a control record to be transmitted to make them visible to the readers 
by updating the last
@@ -137,7 +140,8 @@ class BKLogSegmentWriter implements LogSegmentWriter, 
AddCallback, Runnable, Siz
 private int minDelayBetweenImmediateFlushMs = 0;
 private Stopwatch lastTransmit;
 private boolean streamEnded = false;
-private ScheduledFuture periodicFlushSchedule = null;
+private final ScheduledFuture periodicFlushSchedule;
+private final ScheduledFuture periodicKeepAliveSchedule;
 final private AtomicReference transmitSchedFutureRef = 
new AtomicReference(null);
 final private AtomicReference immFlushSchedFutureRef = 
new AtomicReference(null);
 final private AtomicReference scheduledFlushException = new 
AtomicReference(null);
@@ -312,11 +316,25 @@ class BKLogSegmentWriter implements LogSegmentWriter, 
AddCallback, Runnable, Siz
 if (periodicFlushFrequency > 0 && scheduler != null) {
 periodicFlushSchedule = scheduler.scheduleAtFixedRate(this,
 periodicFlushFrequency/2, periodicFlushFrequency/2, 
TimeUnit.MILLISECONDS);
+} else {
+periodicFlushSchedule = null;
 }
 } else {
 // Min delay heuristic applies only when immediate flush is enabled
 // and transmission threshold is zero
 minDelayBetweenImmediateFlushMs = 
conf.getMinDelayBetweenImmediateFlushMs();
+periodicFlushSchedule = null;
+}
+this.periodicKeepAliveMs = conf.getPeriodicKeepAliveMilliSeconds();
+if (periodicKeepAliveMs > 0 && scheduler != null) {
+periodicKeepAliveSchedule = scheduler.scheduleAtFixedRate(new 
Runnable() {
+@Override
+public void run() {
+keepAlive();
+}
+}, periodicKeepAliveMs, periodicKeepAliveMs, 
TimeUnit.MILLISECONDS);
+} else {
+periodicKeepAliveSchedule = null;
 }
 
 this.conf = conf;
@@ -513,6 +531,13 @@ class BKLogSegmentWriter implements LogSegmentWriter, 
AddCallback, 

[28/29] incubator-distributedlog git commit: Merge branch 'master' into merge/DL-97

[sijie]

Merge branch 'master' into merge/DL-97


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/83a4e092
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/83a4e092
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/83a4e092

Branch: refs/heads/merge/DL-98
Commit: 83a4e092dc9e1ae4e75f085f8ceb932b789df4c2
Parents: f95a0a8 3bd1620
Author: Sijie Guo 
Authored: Tue Dec 20 17:02:40 2016 -0800
Committer: Sijie Guo 
Committed: Tue Dec 20 17:02:40 2016 -0800

--
 dev-support/DL_formatter.xml| 310 -
 distributedlog-build-tools/pom.xml  |  30 ++
 .../resources/distributedlog/checkstyle.xml | 443 +++
 .../resources/distributedlog/suppressions.xml   |  29 ++
 .../main/resources/ide/eclipse/DL_formatter.xml | 310 +
 distributedlog-protocol/pom.xml |  33 ++
 .../java/com/twitter/distributedlog/DLSN.java   |  67 +--
 .../EnvelopedRecordSetReader.java   |  14 +-
 .../EnvelopedRecordSetWriter.java   |  25 +-
 .../com/twitter/distributedlog/LogRecord.java   |  75 ++--
 .../twitter/distributedlog/LogRecordSet.java|  23 +-
 .../distributedlog/LogRecordWithDLSN.java   |  26 +-
 .../twitter/distributedlog/RecordStream.java|   6 +-
 .../annotations/DistributedLogAnnotations.java  |   3 +
 .../annotations/package-info.java   |  21 +
 .../exceptions/AlreadyClosedException.java  |   6 +-
 .../AlreadyTruncatedTransactionException.java   |   4 +-
 .../exceptions/BKTransmitException.java |   6 +-
 .../exceptions/ChecksumFailedException.java |   3 +
 .../exceptions/DLClientClosedException.java |   3 +
 .../distributedlog/exceptions/DLException.java  |   4 +-
 .../exceptions/DLIllegalStateException.java |   3 +
 .../exceptions/EndOfStreamException.java|   3 +
 .../exceptions/FlushException.java  |   4 +-
 .../exceptions/IdleReaderException.java |   5 +
 .../exceptions/InternalServerException.java |   3 +
 .../InvalidEnvelopedEntryException.java |   2 +-
 .../exceptions/InvalidStreamNameException.java  |   3 +
 .../exceptions/LockCancelledException.java  |   3 +
 .../exceptions/LockingException.java|   4 +-
 .../exceptions/LogEmptyException.java   |   4 +-
 .../exceptions/LogExistsException.java  |   2 +-
 .../exceptions/LogNotFoundException.java|   4 +-
 .../exceptions/LogReadException.java|   1 -
 .../exceptions/LogRecordTooLongException.java   |   5 +
 .../exceptions/MetadataException.java   |   3 +
 .../exceptions/NotYetImplementedException.java  |   3 +
 .../exceptions/OverCapacityException.java   |   6 +
 .../OwnershipAcquireFailedException.java|   5 +
 .../exceptions/ReadCancelledException.java  |   3 +
 .../exceptions/RegionUnavailableException.java  |   5 +
 .../exceptions/RequestDeniedException.java  |   3 +
 .../exceptions/RetryableReadException.java  |   3 +
 .../exceptions/ServiceUnavailableException.java |   3 +
 .../exceptions/StreamNotReadyException.java |   3 +
 .../exceptions/StreamUnavailableException.java  |   3 +
 .../exceptions/TooManyStreamsException.java |   3 +
 .../TransactionIdOutOfOrderException.java   |   3 +
 .../exceptions/UnexpectedException.java |   3 +
 .../UnsupportedMetadataVersionException.java|   3 +
 .../exceptions/WriteCancelledException.java |  11 +-
 .../exceptions/WriteException.java  |   6 +-
 .../distributedlog/exceptions/package-info.java |  21 +
 .../distributedlog/io/CompressionCodec.java |   5 +-
 .../distributedlog/io/CompressionUtils.java |  15 +-
 .../io/IdentityCompressionCodec.java|  15 +-
 .../distributedlog/io/LZ4CompressionCodec.java  |  37 +-
 .../twitter/distributedlog/io/package-info.java |  21 +
 .../twitter/distributedlog/package-info.java|  21 +
 .../distributedlog/util/BitMaskUtils.java   |  13 +-
 .../distributedlog/util/ProtocolUtils.java  |   5 +-
 .../distributedlog/util/package-info.java   |  21 +
 .../com/twitter/distributedlog/TestDLSN.java|   7 +-
 .../distributedlog/TestLogRecordSet.java|  17 +-
 pom.xml |   1 +
 65 files changed, 1267 insertions(+), 488 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/83a4e092/pom.xml
--

[21/29] incubator-distributedlog git commit: Merge branch 'merge/DL-92' into merge/DL-93

[sijie]

Merge branch 'merge/DL-92' into merge/DL-93


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/9613f40d
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/9613f40d
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/9613f40d

Branch: refs/heads/merge/DL-98
Commit: 9613f40dec6eeecdb3461d8b997520db76c425b4
Parents: 72a786e adc8ad1
Author: Sijie Guo 
Authored: Fri Dec 16 23:52:19 2016 -0800
Committer: Sijie Guo 
Committed: Fri Dec 16 23:52:19 2016 -0800

--
 .gitignore  |   4 +
 .travis.yml |   4 +
 Dockerfile  |  10 +-
 Vagrantfile |   5 +-
 distributedlog-benchmark/bin/bundle |   7 +-
 distributedlog-benchmark/bin/dbench |  15 +-
 distributedlog-benchmark/conf/dlogenv.sh|   2 +-
 .../routing/ConsistentHashRoutingService.java   |   2 +-
 distributedlog-core/bin/dlog|  12 +-
 distributedlog-core/conf/write_proxy.conf   |   2 +-
 .../distributedlog/DistributedLogConstants.java |   2 +-
 distributedlog-service/bin/bundle   |   7 +-
 distributedlog-service/bin/dlog |  10 +-
 distributedlog-service/bin/dlog-daemon.sh   | 170 +--
 .../distributedlog-kafka/bin/runner |   1 -
 docker/Dockerfile   |  35 
 docs/admin_guide/vagrant.rst|   4 +-
 docs/basics/introduction.rst|  14 +-
 docs/deployment/cluster.rst |  58 ---
 docs/deployment/docker.rst  |   2 +-
 pom.xml |  11 +-
 scripts/bundle  |   2 -
 scripts/common.sh   |  10 +-
 scripts/integration/smoketest.sh| 113 
 scripts/runner  |  12 +-
 scripts/snapshot|  11 +-
 vagrant/base.sh |  43 ++---
 vagrant/bk.sh   |  64 +++
 vagrant/zk.sh   |   5 +-
 29 files changed, 362 insertions(+), 275 deletions(-)
--

[23/29] incubator-distributedlog git commit: Merge branch 'merge/DL-94' into merge/DL-95

[sijie]

Merge branch 'merge/DL-94' into merge/DL-95


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/749657f7
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/749657f7
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/749657f7

Branch: refs/heads/merge/DL-98
Commit: 749657f7dc35e121161d6f162b250dd5b02741a5
Parents: f4f633f 2879090
Author: Sijie Guo 
Authored: Fri Dec 16 23:53:10 2016 -0800
Committer: Sijie Guo 
Committed: Fri Dec 16 23:53:10 2016 -0800

--
 .gitignore  |   4 +
 .travis.yml |   4 +
 Dockerfile  |  10 +-
 Vagrantfile |   5 +-
 distributedlog-benchmark/bin/bundle |   7 +-
 distributedlog-benchmark/bin/dbench |  15 +-
 distributedlog-benchmark/conf/dlogenv.sh|   2 +-
 .../routing/ConsistentHashRoutingService.java   |   2 +-
 distributedlog-core/bin/dlog|  12 +-
 distributedlog-core/conf/write_proxy.conf   |   2 +-
 .../distributedlog/DistributedLogConstants.java |   2 +-
 distributedlog-service/bin/bundle   |   7 +-
 distributedlog-service/bin/dlog |  10 +-
 distributedlog-service/bin/dlog-daemon.sh   | 170 +--
 .../distributedlog-kafka/bin/runner |   1 -
 docker/Dockerfile   |  35 
 docs/admin_guide/vagrant.rst|   4 +-
 docs/basics/introduction.rst|  14 +-
 docs/deployment/cluster.rst |  58 ---
 docs/deployment/docker.rst  |   2 +-
 pom.xml |  11 +-
 scripts/bundle  |   2 -
 scripts/common.sh   |  10 +-
 scripts/integration/smoketest.sh| 113 
 scripts/runner  |  12 +-
 scripts/snapshot|  11 +-
 vagrant/base.sh |  43 ++---
 vagrant/bk.sh   |  64 +++
 vagrant/zk.sh   |   5 +-
 29 files changed, 362 insertions(+), 275 deletions(-)
--

[07/29] incubator-distributedlog git commit: dl: add flag to enable thrift mux on DL Client

[sijie]

dl: add flag to enable thrift mux on DL Client

RB_ID=839555


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/d3a97bc0
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/d3a97bc0
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/d3a97bc0

Branch: refs/heads/merge/DL-98
Commit: d3a97bc0dde0c25516840725599ac46fa03601ab
Parents: 98dc9ab
Author: Dave Rusek 
Authored: Mon Jun 6 16:50:25 2016 -0700
Committer: Sijie Guo 
Committed: Mon Dec 12 16:39:04 2016 -0800

--
 .../distributedlog/service/MonitorService.java  | 28 ++--
 .../service/MonitorServiceApp.java  |  1 +
 2 files changed, 21 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/d3a97bc0/distributedlog-service/src/main/java/com/twitter/distributedlog/service/MonitorService.java
--
diff --git 
a/distributedlog-service/src/main/java/com/twitter/distributedlog/service/MonitorService.java
 
b/distributedlog-service/src/main/java/com/twitter/distributedlog/service/MonitorService.java
index 2683b47..6b58eff 100644
--- 
a/distributedlog-service/src/main/java/com/twitter/distributedlog/service/MonitorService.java
+++ 
b/distributedlog-service/src/main/java/com/twitter/distributedlog/service/MonitorService.java
@@ -85,6 +85,7 @@ public class MonitorService implements NamespaceListener {
 private int heartbeatEveryChecks = 0;
 private int instanceId = -1;
 private int totalInstances = -1;
+private boolean isThriftMux = false;
 
 // Options
 private final Optional uriArg;
@@ -98,6 +99,7 @@ public class MonitorService implements NamespaceListener {
 private final Optional heartbeatEveryChecksArg;
 private final Optional handshakeWithClientInfoArg;
 private final Optional watchNamespaceChangesArg;
+private final Optional isThriftMuxArg;
 
 // Stats
 private final StatsProvider statsProvider;
@@ -224,6 +226,7 @@ public class MonitorService implements NamespaceListener {
Optional heartbeatEveryChecksArg,
Optional handshakeWithClientInfoArg,
Optional watchNamespaceChangesArg,
+   Optional isThriftMuxArg,
StatsReceiver statsReceiver,
StatsProvider statsProvider) {
 // options
@@ -238,6 +241,7 @@ public class MonitorService implements NamespaceListener {
 this.heartbeatEveryChecksArg = heartbeatEveryChecksArg;
 this.handshakeWithClientInfoArg = handshakeWithClientInfoArg;
 this.watchNamespaceChangesArg = watchNamespaceChangesArg;
+this.isThriftMuxArg = isThriftMuxArg;
 
 // Stats
 this.statsReceiver = statsReceiver;
@@ -275,6 +279,7 @@ public class MonitorService implements NamespaceListener {
 }
 handshakeWithClientInfo = handshakeWithClientInfoArg.isPresent();
 watchNamespaceChanges = watchNamespaceChangesArg.isPresent();
+isThriftMux = isThriftMuxArg.isPresent();
 URI uri = URI.create(uriArg.get());
 DistributedLogConfiguration dlConf = new DistributedLogConfiguration();
 if (confFileArg.isPresent()) {
@@ -300,8 +305,22 @@ public class MonitorService implements NamespaceListener {
 ServerSet[] remotes  = new ServerSet[serverSets.length - 1];
 System.arraycopy(serverSets, 1, remotes, 0, remotes.length);
 
+ClientBuilder finagleClientBuilder = ClientBuilder.get()
+.connectTimeout(Duration.fromSeconds(1))
+.tcpConnectTimeout(Duration.fromSeconds(1))
+.requestTimeout(Duration.fromSeconds(2))
+.keepAlive(true)
+.failFast(false);
+
+if (!isThriftMux) {
+finagleClientBuilder = finagleClientBuilder
+.hostConnectionLimit(2)
+.hostConnectionCoresize(2);
+}
+
 dlClient = DistributedLogClientBuilder.newBuilder()
 .name("monitor")
+.thriftmux(isThriftMux)
 .clientId(ClientId$.MODULE$.apply("monitor"))
 .redirectBackoffMaxMs(50)
 .redirectBackoffStartMs(100)
@@ -310,14 +329,7 @@ public class MonitorService implements NamespaceListener {
 .serverSets(local, remotes)
 .streamNameRegex(streamRegex)
 .handshakeWithClientInfo(handshakeWithClientInfo)
-.clientBuilder(ClientBuilder.get()
-.connectTimeout(Duration.fromSeconds(1))
-.tcpConnectTimeout(Duration.fromSeconds(1))
-

[26/29] incubator-distributedlog git commit: Merge remote-tracking branch 'apache/master' into merge/DL-97

[sijie]

Merge remote-tracking branch 'apache/master' into merge/DL-97


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/f95a0a83
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/f95a0a83
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/f95a0a83

Branch: refs/heads/merge/DL-98
Commit: f95a0a830b85e65c4a792223f57e9ccc1d74ea6d
Parents: b8e4c14 26942a9
Author: Sijie Guo 
Authored: Tue Dec 20 00:08:58 2016 -0800
Committer: Sijie Guo 
Committed: Tue Dec 20 00:08:58 2016 -0800

--
 .../benchmark/stream/AsyncReaderBenchmark.java  |   6 +
 .../benchmark/stream/LedgerReadBenchmark.java   |   4 +
 .../benchmark/stream/SyncReaderBenchmark.java   |   6 +
 .../client/routing/TestRoutingService.java  |   4 +-
 .../metadata/ZkMetadataResolver.java|   1 +
 .../com/twitter/distributedlog/DLMTestUtil.java |   4 +-
 .../distributedlog/TestAsyncReaderWriter.java   |   8 +-
 .../TestBKDistributedLogManager.java|  10 +-
 .../TestBKDistributedLogNamespace.java  |  25 +++--
 .../distributedlog/TestDistributedLogBase.java  |  10 +-
 .../distributedlog/TestInterleavedReaders.java  |   2 +-
 .../twitter/distributedlog/TestTruncate.java|   4 +-
 .../distributedlog/TestWriteLimiter.java|  20 ++--
 .../bk/TestLedgerAllocatorPool.java |   2 +-
 .../config/TestConcurrentBaseConfiguration.java |   2 +-
 .../impl/TestZKLogSegmentFilters.java   |   2 +-
 .../limiter/TestRequestLimiter.java |   2 +-
 .../TestLogSegmentMetadataStoreUpdater.java |   2 +-
 .../distributedlog/util/TestConfUtils.java  |   2 +-
 .../distributedlog/util/TestPermitManager.java  |   4 +-
 .../util/TestSafeQueueingFuturePool.java|  10 +-
 .../src/main/thrift/service.thrift  | 111 ++-
 .../service/stream/TestStreamManager.java   |   2 +-
 .../2016-09-19-kafka-vs-distributedlog.md   |   3 +-
 24 files changed, 136 insertions(+), 110 deletions(-)
--

[29/29] incubator-distributedlog git commit: Merge branch 'merge/DL-97' into merge/DL-98

[sijie]

Merge branch 'merge/DL-97' into merge/DL-98


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/28a8b8ff
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/28a8b8ff
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/28a8b8ff

Branch: refs/heads/merge/DL-98
Commit: 28a8b8ff9a01d2862fcedb37bc0f6ce1dc3e1a19
Parents: 9709f9f 83a4e09
Author: Sijie Guo 
Authored: Tue Dec 20 17:04:32 2016 -0800
Committer: Sijie Guo 
Committed: Tue Dec 20 17:04:32 2016 -0800

--
 dev-support/DL_formatter.xml| 310 -
 distributedlog-build-tools/pom.xml  |  30 ++
 .../resources/distributedlog/checkstyle.xml | 443 +++
 .../resources/distributedlog/suppressions.xml   |  29 ++
 .../main/resources/ide/eclipse/DL_formatter.xml | 310 +
 distributedlog-protocol/pom.xml |  33 ++
 .../java/com/twitter/distributedlog/DLSN.java   |  67 +--
 .../EnvelopedRecordSetReader.java   |  14 +-
 .../EnvelopedRecordSetWriter.java   |  25 +-
 .../com/twitter/distributedlog/LogRecord.java   |  75 ++--
 .../twitter/distributedlog/LogRecordSet.java|  23 +-
 .../distributedlog/LogRecordWithDLSN.java   |  26 +-
 .../twitter/distributedlog/RecordStream.java|   6 +-
 .../annotations/DistributedLogAnnotations.java  |   3 +
 .../annotations/package-info.java   |  21 +
 .../exceptions/AlreadyClosedException.java  |   6 +-
 .../AlreadyTruncatedTransactionException.java   |   4 +-
 .../exceptions/BKTransmitException.java |   6 +-
 .../exceptions/ChecksumFailedException.java |   3 +
 .../exceptions/DLClientClosedException.java |   3 +
 .../distributedlog/exceptions/DLException.java  |   4 +-
 .../exceptions/DLIllegalStateException.java |   3 +
 .../exceptions/EndOfStreamException.java|   3 +
 .../exceptions/FlushException.java  |   4 +-
 .../exceptions/IdleReaderException.java |   5 +
 .../exceptions/InternalServerException.java |   3 +
 .../InvalidEnvelopedEntryException.java |   2 +-
 .../exceptions/InvalidStreamNameException.java  |   3 +
 .../exceptions/LockCancelledException.java  |   3 +
 .../exceptions/LockingException.java|   4 +-
 .../exceptions/LogEmptyException.java   |   4 +-
 .../exceptions/LogExistsException.java  |   2 +-
 .../exceptions/LogNotFoundException.java|   4 +-
 .../exceptions/LogReadException.java|   1 -
 .../exceptions/LogRecordTooLongException.java   |   5 +
 .../exceptions/MetadataException.java   |   3 +
 .../exceptions/NotYetImplementedException.java  |   3 +
 .../exceptions/OverCapacityException.java   |   6 +
 .../OwnershipAcquireFailedException.java|   5 +
 .../exceptions/ReadCancelledException.java  |   3 +
 .../exceptions/RegionUnavailableException.java  |   5 +
 .../exceptions/RequestDeniedException.java  |   3 +
 .../exceptions/RetryableReadException.java  |   3 +
 .../exceptions/ServiceUnavailableException.java |   3 +
 .../exceptions/StreamNotReadyException.java |   3 +
 .../exceptions/StreamUnavailableException.java  |   3 +
 .../exceptions/TooManyStreamsException.java |   3 +
 .../TransactionIdOutOfOrderException.java   |   3 +
 .../exceptions/UnexpectedException.java |   3 +
 .../UnsupportedMetadataVersionException.java|   3 +
 .../exceptions/WriteCancelledException.java |  11 +-
 .../exceptions/WriteException.java  |   6 +-
 .../distributedlog/exceptions/package-info.java |  21 +
 .../distributedlog/io/CompressionCodec.java |   5 +-
 .../distributedlog/io/CompressionUtils.java |  15 +-
 .../io/IdentityCompressionCodec.java|  15 +-
 .../distributedlog/io/LZ4CompressionCodec.java  |  37 +-
 .../twitter/distributedlog/io/package-info.java |  21 +
 .../twitter/distributedlog/package-info.java|  21 +
 .../distributedlog/util/BitMaskUtils.java   |  13 +-
 .../distributedlog/util/ProtocolUtils.java  |   5 +-
 .../distributedlog/util/package-info.java   |  21 +
 .../com/twitter/distributedlog/TestDLSN.java|   7 +-
 .../distributedlog/TestLogRecordSet.java|  17 +-
 pom.xml |   1 +
 65 files changed, 1267 insertions(+), 488 deletions(-)
--

[10/29] incubator-distributedlog git commit: Simplify the state transition on stream

[sijie]

Simplify the state transition on stream

* the stream is created on INITIALIZING
* when the stream is started, it would start transition from INITIALIZING to 
INITIALIZED
* it would serve stream operations when the stream is INITIALIZED
* it would be turned to ERROR when encountered exceptions.
* the stream would be closed when service operation timeout or encountered any 
exceptions. it would first be removed from acquired mapping
* the stream would be removed from cached mapping depends on probation time.

RB_ID=848047


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/f19e7564
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/f19e7564
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/f19e7564

Branch: refs/heads/merge/DL-98
Commit: f19e7564ff4a1ec1b5d6f2683db190d739df99bb
Parents: 0a18f56
Author: Leigh Stewart 
Authored: Mon Dec 12 16:49:26 2016 -0800
Committer: Sijie Guo 
Committed: Mon Dec 12 16:49:26 2016 -0800

--
 .../service/DistributedLogServiceImpl.java  |  27 +-
 .../service/stream/StreamImpl.java  | 550 +++
 .../service/stream/StreamManager.java   |   5 +-
 .../service/stream/StreamManagerImpl.java   |  15 +-
 .../service/TestDistributedLogService.java  |  20 +-
 5 files changed, 222 insertions(+), 395 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/f19e7564/distributedlog-service/src/main/java/com/twitter/distributedlog/service/DistributedLogServiceImpl.java
--
diff --git 
a/distributedlog-service/src/main/java/com/twitter/distributedlog/service/DistributedLogServiceImpl.java
 
b/distributedlog-service/src/main/java/com/twitter/distributedlog/service/DistributedLogServiceImpl.java
index 751e972..3a9b904 100644
--- 
a/distributedlog-service/src/main/java/com/twitter/distributedlog/service/DistributedLogServiceImpl.java
+++ 
b/distributedlog-service/src/main/java/com/twitter/distributedlog/service/DistributedLogServiceImpl.java
@@ -90,7 +90,6 @@ import java.util.List;
 import java.util.Map;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.RejectedExecutionException;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.locks.ReentrantReadWriteLock;
 
@@ -378,7 +377,7 @@ public class DistributedLogServiceImpl implements 
DistributedLogService.ServiceI
 // if it is closed, we would not acquire stream again.
 return null;
 }
-writer = streamManager.getOrCreateStream(stream);
+writer = streamManager.getOrCreateStream(stream, true);
 } finally {
 closeLock.readLock().unlock();
 }
@@ -631,26 +630,6 @@ public class DistributedLogServiceImpl implements 
DistributedLogService.ServiceI
 logger.info("Released KeepAlive Latch. Main thread will shut the 
service down.");
 }
 
-@VisibleForTesting
-java.util.concurrent.Future schedule(Runnable runnable, long delayMs) {
-closeLock.readLock().lock();
-try {
-if (serverStatus != ServerStatus.WRITE_AND_ACCEPT) {
-return null;
-} else if (delayMs > 0) {
-return scheduler.schedule(runnable, delayMs, 
TimeUnit.MILLISECONDS);
-} else {
-return scheduler.submit(runnable);
-}
-} catch (RejectedExecutionException ree) {
-logger.error("Failed to schedule task {} in {} ms : ",
-new Object[] { runnable, delayMs, ree });
-return null;
-} finally {
-closeLock.readLock().unlock();
-}
-}
-
 // Test methods.
 
 private DynamicDistributedLogConfiguration getDynConf(String streamName) {
@@ -664,8 +643,8 @@ public class DistributedLogServiceImpl implements 
DistributedLogService.ServiceI
 }
 
 @VisibleForTesting
-Stream newStream(String name) {
-return streamFactory.create(name, getDynConf(name), streamManager);
+Stream newStream(String name) throws IOException {
+return streamManager.getOrCreateStream(name, false);
 }
 
 @VisibleForTesting

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/f19e7564/distributedlog-service/src/main/java/com/twitter/distributedlog/service/stream/StreamImpl.java
--
diff --git 
a/distributedlog-service/src/main/java/com/twitter/distributedlog/service/stream/StreamImpl.java
 

[13/29] incubator-distributedlog git commit: Make the zookeeper client used by bookkeeper client retry on session expires

[sijie]

Make the zookeeper client used by bookkeeper client retry on session expires

* the zookeeper client used by bookkeeper client is purely for metadata 
accesses, so we should retry on session expires.
* remove the unnessary zookeeper session handling in bk log handler. as we 
don't necessary to fail bookkeeper client or log handler when session expires 
as it would be handled and retried by the zookeeper client.
* Make the retry infinitely if the retry settings for bkc zookeeper client is 
set to 0 or negative.

RB_ID=843057


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/f18fe172
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/f18fe172
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/f18fe172

Branch: refs/heads/merge/DL-98
Commit: f18fe172fb4559bcabb7173d1cd0b9d27fc23c93
Parents: 72a786e
Author: Sijie Guo 
Authored: Mon Jul 11 10:21:44 2016 -0700
Committer: Sijie Guo 
Committed: Mon Dec 12 16:58:20 2016 -0800

--
 .../distributedlog/BKAsyncLogReaderDLSN.java| 15 +-
 .../BKDistributedLogNamespace.java  |  5 +-
 .../twitter/distributedlog/BKLogHandler.java| 17 ---
 .../distributedlog/BookKeeperClient.java| 48 
 .../DistributedLogConfiguration.java|  8 +++-
 5 files changed, 17 insertions(+), 76 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/f18fe172/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogReaderDLSN.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogReaderDLSN.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogReaderDLSN.java
index 7d3d53d..b1a9273 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogReaderDLSN.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogReaderDLSN.java
@@ -49,7 +49,6 @@ import java.util.concurrent.atomic.AtomicReference;
 import org.apache.bookkeeper.stats.Counter;
 import org.apache.bookkeeper.stats.OpStatsLogger;
 import org.apache.bookkeeper.stats.StatsLogger;
-import org.apache.zookeeper.Watcher;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import scala.Function1;
@@ -73,7 +72,7 @@ import scala.runtime.AbstractFunction1;
  *  `async_reader`/idle_reader_error: counter. the number idle reader 
errors.
  * 
  */
-class BKAsyncLogReaderDLSN implements 
ZooKeeperClient.ZooKeeperSessionExpireNotifier, AsyncLogReader, Runnable, 
AsyncNotification {
+class BKAsyncLogReaderDLSN implements AsyncLogReader, Runnable, 
AsyncNotification {
 static final Logger LOG = 
LoggerFactory.getLogger(BKAsyncLogReaderDLSN.class);
 
 private static final Function1 
READ_NEXT_MAP_FUNCTION =
@@ -86,7 +85,6 @@ class BKAsyncLogReaderDLSN implements 
ZooKeeperClient.ZooKeeperSessionExpireNoti
 
 protected final BKDistributedLogManager bkDistributedLogManager;
 protected final BKLogReadHandler bkLedgerManager;
-private Watcher sessionExpireWatcher = null;
 private final AtomicReference lastException = new 
AtomicReference();
 private final ScheduledExecutorService executorService;
 private final ConcurrentLinkedQueue pendingRequests = 
new ConcurrentLinkedQueue();
@@ -218,7 +216,6 @@ class BKAsyncLogReaderDLSN implements 
ZooKeeperClient.ZooKeeperSessionExpireNoti
 this.executorService = executorService;
 this.bkLedgerManager = 
bkDistributedLogManager.createReadHandler(subscriberId,
 lockStateExecutor, this, deserializeRecordSet, true);
-sessionExpireWatcher = 
this.bkLedgerManager.registerExpirationHandler(this);
 LOG.debug("Starting async reader at {}", startDLSN);
 this.startDLSN = startDLSN;
 this.scheduleDelayStopwatch = Stopwatch.createUnstarted();
@@ -255,14 +252,6 @@ class BKAsyncLogReaderDLSN implements 
ZooKeeperClient.ZooKeeperSessionExpireNoti
 this.idleReaderTimeoutTask = scheduleIdleReaderTaskIfNecessary();
 }
 
-@Override
-public void notifySessionExpired() {
-// ZK Session notification is an indication to check if this has 
resulted in a fatal error
-// of the underlying reader, in itself this reader doesnt error out 
unless the underlying
-// reader has hit an error
-scheduleBackgroundRead();
-}
-
 private ScheduledFuture scheduleIdleReaderTaskIfNecessary() {
 if (idleErrorThresholdMillis < Integer.MAX_VALUE) {
 // Dont run the task more than once every seconds (for sanity)
@@ -494,8 +483,6 @@ class 

[09/29] incubator-distributedlog git commit: Be able to close the writer within a timeout period

[sijie]

Be able to close the writer within a timeout period

RB_ID=841340


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/0a18f564
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/0a18f564
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/0a18f564

Branch: refs/heads/merge/DL-98
Commit: 0a18f564ba7e4d03f4c6a0859e5478535a59befd
Parents: 800b867
Author: Leigh Stewart 
Authored: Mon Dec 12 16:46:11 2016 -0800
Committer: Sijie Guo 
Committed: Mon Dec 12 16:46:11 2016 -0800

--
 .../distributedlog/BKAsyncLogWriter.java| 33 +---
 .../distributedlog/util/FutureUtils.java| 20 +++-
 .../service/config/ServerConfiguration.java | 25 +++
 .../service/stream/StreamFactoryImpl.java   |  6 +++-
 .../service/stream/StreamImpl.java  | 32 +--
 5 files changed, 99 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0a18f564/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogWriter.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogWriter.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogWriter.java
index f1594f9..79f5f5e 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogWriter.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogWriter.java
@@ -212,8 +212,10 @@ public class BKAsyncLogWriter extends BKAbstractLogWriter 
implements AsyncLogWri
boolean rollLog,
boolean 
allowMaxTxID) {
 Stopwatch stopwatch = Stopwatch.createStarted();
-return doGetLogSegmentWriter(firstTxid, bestEffort, rollLog, 
allowMaxTxID)
-.addEventListener(new 
OpStatsListener(getWriterOpStatsLogger, stopwatch));
+return FutureUtils.stats(
+doGetLogSegmentWriter(firstTxid, bestEffort, rollLog, 
allowMaxTxID),
+getWriterOpStatsLogger,
+stopwatch);
 }
 
 private Future doGetLogSegmentWriter(final long 
firstTxid,
@@ -415,8 +417,10 @@ public class BKAsyncLogWriter extends BKAbstractLogWriter 
implements AsyncLogWri
 @Override
 public Future write(final LogRecord record) {
 final Stopwatch stopwatch = Stopwatch.createStarted();
-return asyncWrite(record, true)
-.addEventListener(new 
OpStatsListener(writeOpStatsLogger, stopwatch));
+return FutureUtils.stats(
+asyncWrite(record, true),
+writeOpStatsLogger,
+stopwatch);
 }
 
 /**
@@ -430,8 +434,10 @@ public class BKAsyncLogWriter extends BKAbstractLogWriter 
implements AsyncLogWri
 @Override
 public Future> writeBulk(final List records) 
{
 final Stopwatch stopwatch = Stopwatch.createStarted();
-return Future.value(asyncWriteBulk(records))
-.addEventListener(new 
OpStatsListener>(bulkWriteOpStatsLogger, stopwatch));
+return FutureUtils.stats(
+Future.value(asyncWriteBulk(records)),
+bulkWriteOpStatsLogger,
+stopwatch);
 }
 
 @Override
@@ -478,12 +484,15 @@ public class BKAsyncLogWriter extends BKAbstractLogWriter 
implements AsyncLogWri
 logSegmentWriterFuture = getLogSegmentWriterForEndOfStream();
 }
 
-return logSegmentWriterFuture.flatMap(new 
AbstractFunction1() {
-@Override
-public Future apply(BKLogSegmentWriter w) {
-return w.markEndOfStream();
-}
-}).addEventListener(new 
OpStatsListener(markEndOfStreamOpStatsLogger, stopwatch));
+return FutureUtils.stats(
+logSegmentWriterFuture.flatMap(new 
AbstractFunction1() {
+@Override
+public Future apply(BKLogSegmentWriter w) {
+return w.markEndOfStream();
+}
+}),
+markEndOfStreamOpStatsLogger,
+stopwatch);
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0a18f564/distributedlog-core/src/main/java/com/twitter/distributedlog/util/FutureUtils.java
--
diff --git 

[16/29] incubator-distributedlog git commit: Remove guage when the log segment writer is removed

[sijie]

Remove guage when the log segment writer is removed

RB_ID=849452


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/bc70b770
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/bc70b770
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/bc70b770

Branch: refs/heads/merge/DL-98
Commit: bc70b7706e2a5cf2af26ab7e92ecf5ec2b1a955e
Parents: b7ae590
Author: Sijie Guo 
Authored: Mon Jul 11 11:11:15 2016 -0700
Committer: Sijie Guo 
Committed: Mon Dec 12 18:27:37 2016 -0800

--
 .../distributedlog/BKLogSegmentWriter.java  | 12 ---
 .../stats/BroadCastStatsLogger.java | 22 
 pom.xml |  2 +-
 3 files changed, 32 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/bc70b770/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
index 1b52951..8276125 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
@@ -156,8 +156,10 @@ class BKLogSegmentWriter implements LogSegmentWriter, 
AddCallback, Runnable, Siz
 
 // stats
 private final StatsLogger envelopeStatsLogger;
+private final StatsLogger transmitOutstandingLogger;
 private final Counter transmitDataSuccesses;
 private final Counter transmitDataMisses;
+private final Gauge transmitOutstandingGauge;
 private final OpStatsLogger transmitDataPacketSize;
 private final Counter transmitControlSuccesses;
 private final Counter pFlushSuccesses;
@@ -255,8 +257,8 @@ class BKLogSegmentWriter implements LogSegmentWriter, 
AddCallback, Runnable, Siz
 pendingWrites = segWriterStatsLogger.getCounter("pending");
 
 // outstanding transmit requests
-StatsLogger transmitOutstandingLogger = 
perLogStatsLogger.scope("transmit").scope("outstanding");
-transmitOutstandingLogger.registerGauge("requests", new 
Gauge() {
+transmitOutstandingLogger = 
perLogStatsLogger.scope("transmit").scope("outstanding");
+transmitOutstandingGauge = new Gauge() {
 @Override
 public Number getDefaultValue() {
 return 0;
@@ -265,7 +267,8 @@ class BKLogSegmentWriter implements LogSegmentWriter, 
AddCallback, Runnable, Siz
 public Number getSample() {
 return outstandingTransmits.get();
 }
-});
+};
+transmitOutstandingLogger.registerGauge("requests", 
transmitOutstandingGauge);
 
 outstandingTransmits = new AtomicInteger(0);
 this.fullyQualifiedLogSegment = streamName + ":" + logSegmentName;
@@ -531,6 +534,9 @@ class BKLogSegmentWriter implements LogSegmentWriter, 
AddCallback, Runnable, Siz
 private void closeInternal(final boolean abort,
final AtomicReference throwExc,
final Promise closePromise) {
+// remove stats
+this.transmitOutstandingLogger.unregisterGauge("requests", 
transmitOutstandingGauge);
+
 // Cancel the periodic keep alive schedule first
 if (null != periodicKeepAliveSchedule) {
 if (!periodicKeepAliveSchedule.cancel(false)) {

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/bc70b770/distributedlog-core/src/main/java/com/twitter/distributedlog/stats/BroadCastStatsLogger.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/stats/BroadCastStatsLogger.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/stats/BroadCastStatsLogger.java
index e29cc47..10a7011 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/stats/BroadCastStatsLogger.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/stats/BroadCastStatsLogger.java
@@ -133,9 +133,26 @@ public class BroadCastStatsLogger {
 }
 
 @Override
+public  void unregisterGauge(String statName, 
Gauge gauge) {
+// no-op
+}
+
+@Override
 public StatsLogger scope(final String scope) {
 return new Two(first.scope(scope), second.scope(scope));
 }
+
+@Override
+public void removeScope(String scope, 

[25/29] incubator-distributedlog git commit: Merge branch 'merge/DL-96' into merge/DL-97

[sijie]

Merge branch 'merge/DL-96' into merge/DL-97


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/b8e4c146
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/b8e4c146
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/b8e4c146

Branch: refs/heads/merge/DL-98
Commit: b8e4c1462a107345f87eb48f82b0ee76f9fc4e4d
Parents: bc70b77 a1a3516
Author: Sijie Guo 
Authored: Fri Dec 16 23:54:12 2016 -0800
Committer: Sijie Guo 
Committed: Fri Dec 16 23:54:12 2016 -0800

--
 .gitignore  |   4 +
 .travis.yml |   4 +
 Dockerfile  |  10 +-
 Vagrantfile |   5 +-
 distributedlog-benchmark/bin/bundle |   7 +-
 distributedlog-benchmark/bin/dbench |  15 +-
 distributedlog-benchmark/conf/dlogenv.sh|   2 +-
 .../routing/ConsistentHashRoutingService.java   |   2 +-
 distributedlog-core/bin/dlog|  12 +-
 distributedlog-core/conf/write_proxy.conf   |   2 +-
 .../distributedlog/DistributedLogConstants.java |   2 +-
 distributedlog-service/bin/bundle   |   7 +-
 distributedlog-service/bin/dlog |  10 +-
 distributedlog-service/bin/dlog-daemon.sh   | 170 +--
 .../distributedlog-kafka/bin/runner |   1 -
 docker/Dockerfile   |  35 
 docs/admin_guide/vagrant.rst|   4 +-
 docs/basics/introduction.rst|  14 +-
 docs/deployment/cluster.rst |  58 ---
 docs/deployment/docker.rst  |   2 +-
 pom.xml |  11 +-
 scripts/bundle  |   2 -
 scripts/common.sh   |  10 +-
 scripts/integration/smoketest.sh| 113 
 scripts/runner  |  12 +-
 scripts/snapshot|  11 +-
 vagrant/base.sh |  43 ++---
 vagrant/bk.sh   |  64 +++
 vagrant/zk.sh   |   5 +-
 29 files changed, 362 insertions(+), 275 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b8e4c146/pom.xml
--

[19/29] incubator-distributedlog git commit: Merge branch 'merge/DL-90' into merge/DL-91

[sijie]

Merge branch 'merge/DL-90' into merge/DL-91


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/4fc6f3d2
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/4fc6f3d2
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/4fc6f3d2

Branch: refs/heads/merge/DL-98
Commit: 4fc6f3d271dc440f324fc537e024fe43d2c285e1
Parents: 0a18f56 76be38e
Author: Sijie Guo 
Authored: Fri Dec 16 23:51:17 2016 -0800
Committer: Sijie Guo 
Committed: Fri Dec 16 23:51:17 2016 -0800

--
 .gitignore  |   4 +
 .travis.yml |   4 +
 Dockerfile  |  10 +-
 Vagrantfile |   5 +-
 distributedlog-benchmark/bin/bundle |   7 +-
 distributedlog-benchmark/bin/dbench |  15 +-
 distributedlog-benchmark/conf/dlogenv.sh|   2 +-
 .../routing/ConsistentHashRoutingService.java   |   2 +-
 distributedlog-core/bin/dlog|  12 +-
 distributedlog-core/conf/write_proxy.conf   |   2 +-
 .../distributedlog/DistributedLogConstants.java |   2 +-
 distributedlog-service/bin/bundle   |   7 +-
 distributedlog-service/bin/dlog |  10 +-
 distributedlog-service/bin/dlog-daemon.sh   | 170 +--
 .../distributedlog-kafka/bin/runner |   1 -
 docker/Dockerfile   |  35 
 docs/admin_guide/vagrant.rst|   4 +-
 docs/basics/introduction.rst|  14 +-
 docs/deployment/cluster.rst |  58 ---
 docs/deployment/docker.rst  |   2 +-
 pom.xml |  11 +-
 scripts/bundle  |   2 -
 scripts/common.sh   |  10 +-
 scripts/integration/smoketest.sh| 113 
 scripts/runner  |  12 +-
 scripts/snapshot|  11 +-
 vagrant/base.sh |  43 ++---
 vagrant/bk.sh   |  64 +++
 vagrant/zk.sh   |   5 +-
 29 files changed, 362 insertions(+), 275 deletions(-)
--

[14/29] incubator-distributedlog git commit: Assign host ip client id if the client id is undefined

[sijie]

Assign host ip client id if the client id is undefined

RB_ID=843132


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/f4f633fe
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/f4f633fe
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/f4f633fe

Branch: refs/heads/merge/DL-98
Commit: f4f633fe655e7146d5aad89c24f7e7107ab65ab2
Parents: f18fe17
Author: Leigh Stewart 
Authored: Mon Dec 12 16:59:38 2016 -0800
Committer: Sijie Guo 
Committed: Mon Dec 12 16:59:38 2016 -0800

--
 .../distributedlog/BKDistributedLogNamespace.java   | 15 ++-
 .../com/twitter/distributedlog/BKLogHandler.java| 16 +---
 2 files changed, 15 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/f4f633fe/distributedlog-core/src/main/java/com/twitter/distributedlog/BKDistributedLogNamespace.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKDistributedLogNamespace.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKDistributedLogNamespace.java
index 7a4fd7f..2df1046 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKDistributedLogNamespace.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKDistributedLogNamespace.java
@@ -75,6 +75,7 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.io.IOException;
+import java.net.InetAddress;
 import java.net.URI;
 import java.util.Collection;
 import java.util.HashMap;
@@ -252,6 +253,14 @@ public class BKDistributedLogNamespace implements 
DistributedLogNamespace {
 }
 }
 
+private static String getHostIpLockClientId() {
+try {
+return InetAddress.getLocalHost().toString();
+} catch(Exception ex) {
+return DistributedLogConstants.UNKNOWN_CLIENT_ID;
+}
+}
+
 private final String clientId;
 private final int regionId;
 private final DistributedLogConfiguration conf;
@@ -326,9 +335,13 @@ public class BKDistributedLogNamespace implements 
DistributedLogNamespace {
 this.featureProvider = featureProvider;
 this.statsLogger = statsLogger;
 this.perLogStatsLogger = perLogStatsLogger;
-this.clientId = clientId;
 this.regionId = regionId;
 this.bkdlConfig = bkdlConfig;
+if (clientId.equals(DistributedLogConstants.UNKNOWN_CLIENT_ID)) {
+this.clientId = getHostIpLockClientId();
+} else {
+this.clientId = clientId;
+}
 
 // Build resources
 StatsLogger schedulerStatsLogger = 
statsLogger.scope("factory").scope("thread_pool");

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/f4f633fe/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
index a84261a..3b991e2 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
@@ -56,7 +56,6 @@ import scala.runtime.AbstractFunction0;
 import scala.runtime.BoxedUnit;
 
 import java.io.IOException;
-import java.net.InetAddress;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.Comparator;
@@ -274,12 +273,7 @@ public abstract class BKLogHandler implements Watcher, 
AsyncCloseable, AsyncAbor
 LOG.debug("Using ZK Path {}", logMetadata.getLogRootPath());
 this.bookKeeperClient = bkcBuilder.build();
 this.metadataStore = metadataStore;
-
-if (lockClientId.equals(DistributedLogConstants.UNKNOWN_CLIENT_ID)) {
-this.lockClientId = getHostIpLockClientId();
-} else {
-this.lockClientId = lockClientId;
-}
+this.lockClientId = lockClientId;
 
 this.getChildrenWatcher = this.zooKeeperClient.getWatcherManager()
 .registerChildWatcher(logMetadata.getLogSegmentsPath(), this);
@@ -316,14 +310,6 @@ public abstract class BKLogHandler implements Watcher, 
AsyncCloseable, AsyncAbor
 return lockClientId;
 }
 
-private String getHostIpLockClientId() {
-try {
-return InetAddress.getLocalHost().toString();
-} catch(Exception ex) {
-return DistributedLogConstants.UNKNOWN_CLIENT_ID;
-}
-}
-
 protected