[GitHub] incubator-distributedlog issue #21: DL-45: DL should allow ByteBuffer based ...

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/21
  
@arvindkandhare @jiazhai I was experimenting using netty-buffer for memory 
allocation and management. I got pretty good performance results on it. I will 
try to share more details with numbers in a few days. /cc @leighst 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSSION] First Release

[Sijie Guo]

cool! Thank you so much for rebasing this.

- Sijie

On Wed, Jan 4, 2017 at 6:20 PM, liang xie  wrote:

> The conflicted DL-167 change has been rebased, please help to review :)
>
> On Thu, Jan 5, 2017 at 8:53 AM, Sijie Guo  wrote:
> > FYI. I merged the repackage script and repackage the namespace under
> > org.apache.distributedlog. All the changes for 0.4.0 are merged (except
> one
> > test case change - waiting for Liang's response). Ready to cut the first
> > release candidate.
> >
> > - Sijie
> >
> > On Thu, Dec 29, 2016 at 8:40 PM, Sijie Guo  wrote:
> >
> >> I've merged all the pull requests that need to be included in 0.4.0 (and
> >> both Jenkins and CI builds are green now). Also I moved inactive jiras
> to
> >> 0.5.0.
> >>
> >> There are a few issues remaining - https://issues.apache.org/
> >> jira/issues/?jql=project%20%3D%20DL%20AND%20resolution%20%
> >> 3D%20Unresolved%20AND%20fixVersion%20%3D%200.4.0%
> >> 20ORDER%20BY%20priority%20DESC
> >>
> >> Most of them already have pull requests (but need to be rebased to
> latest
> >> master). So I can merge them.
> >>
> >> Hopefully I will be able to cut 0.4.0 release in following few days.
> >>
> >> - Sijie
> >>
> >> On Thu, Dec 8, 2016 at 9:16 PM, Sijie Guo  wrote:
> >>
> >>> I will start check the jiras that need to be included in 0.4.0 release
> at
> >>> the weekend. If there is any jiras that need to be included in 0.4.0,
> >>> please comment in the jira.
> >>>
> >>> - Sijie
> >>>
> >>> On Fri, Dec 2, 2016 at 8:28 PM, Sijie Guo  wrote:
> >>>
>  FYI.
> 
>  https://issues.apache.org/jira/browse/DL-81 is created for tracking
>  building the process.
> 
>  https://issues.apache.org/jira/browse/INFRA-13024 and
>  https://issues.apache.org/jira/browse/INFRA-13023 for setting the
> dists
>  for DL.
> 
>  - Sijie
> 
>  On Fri, Dec 2, 2016 at 8:05 PM, Sijie Guo  wrote:
> 
> > Thank you for all the feedbacks. I will drive the release for this
> > version. So that we can have all the procedures documented in the
> wiki.
> >
> > I created the wiki page here for documenting the procedure:
> > https://cwiki.apache.org/confluence/display/DL/Pr
> > eparing+DistributedLog+Releases
> >
> > - Sijie
> >
> >
> > On Mon, Nov 21, 2016 at 7:10 PM, Jia Zhai 
> wrote:
> >
> >> +1
> >> Great to know the commitment of merge twitter's branch. Looking
> >> forward to
> >> see bookkeeper 4.5.
> >>
> >> On Mon, Nov 21, 2016 at 6:36 PM, Flavio Junqueira 
> >> wrote:
> >>
> >> > It does make sense, Sijie, thanks for the update.
> >> >
> >> > -Flavio
> >> >
> >> > > On 17 Nov 2016, at 05:12, Sijie Guo  wrote:
> >> > >
> >> > > Flavio,
> >> > >
> >> > > I totally agreed that not using an official Apache  version is
> not
> >> great
> >> > > for the community. We had a bookkeeper meetup last night. We've
> >> discussed
> >> > > the current situation with the community. We came to a
> commitment
> >> to
> >> > merge
> >> > > Twitter's branch back into bookkeeper 4.5. After that we won't
> >> maintain
> >> > our
> >> > > own branch and switch to 4.5.
> >> > >
> >> > > Hope this make sense.
> >> > >
> >> > > Sijie
> >> > >
> >> > > On Nov 16, 2016 8:04 PM, "Flavio Junqueira" 
> >> wrote:
> >> > >
> >> > > It isn't great that DistributedLog is not using the Apache
> >> BookKeeper
> >> > > release. Essentially anyone using DistributedLog today needs to
> >> use the
> >> > > Twitter branch of BookKeeper, which has diverged from Apache
> >> BookKeeper.
> >> > > I'm sure the changes in the Twitter branch are all great, but
> I'd
> >> be more
> >> > > comfortable being able to rely on the Apache BookKeeper
> releases,
> >> which
> >> > are
> >> > > community driven.
> >> > >
> >> > > I'm not going to block the release on this alone because it is
> >> important
> >> > > for this project to get a first release out soon, but we need to
> >> fix
> >> > DL-2.
> >> > >
> >> > > -Flavio
> >> > >
> >> > >> On 15 Nov 2016, at 19:02, Leigh Stewart
> >> 
> >> > > wrote:
> >> > >>
> >> > >> +1
> >> > >>
> >> > >> On Tue, Nov 15, 2016 at 7:01 PM, Franck Cuny <
> >> franck.c...@gmail.com>
> >> > > wrote:
> >> > >>
> >> > >>> +1 and I agree to not make DL-2 a blocker.
> >> > >>>
> >> > >>> On Tue, Nov 15, 2016 at 2:02 AM, Xi Liu  >
> >> wrote:
> >> > >>>
> >> >  +1 especially on DL-23.
> >> > 
> >> >  - Xi
> >> > 
> >> >  On Sat, Nov 12, 2016 at 2:22 AM, 

[GitHub] incubator-distributedlog pull request #94: DL-167 : fix testCalculateUnequal...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/94


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

incubator-distributedlog git commit: DL-167: fix testCalculateUnequalWeight computing corner case

[sijie]

Repository: incubator-distributedlog
Updated Branches:
  refs/heads/master b44820b50 -> a31782093


DL-167: fix testCalculateUnequalWeight computing corner case

Author: xieliang 

Reviewers: Leigh Stewart 

Closes #94 from xieliang/DL-167-fix-testCalculateUnequalWeight


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/a3178209
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/a3178209
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/a3178209

Branch: refs/heads/master
Commit: a3178209350139af535203e1acff96a151b88013
Parents: b44820b
Author: xieliang 
Authored: Wed Jan 4 22:55:51 2017 -0800
Committer: Sijie Guo 
Committed: Wed Jan 4 22:55:51 2017 -0800

--
 .../service/placement/TestLeastLoadPlacementPolicy.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/a3178209/distributedlog-service/src/test/java/org/apache/distributedlog/service/placement/TestLeastLoadPlacementPolicy.java
--
diff --git 
a/distributedlog-service/src/test/java/org/apache/distributedlog/service/placement/TestLeastLoadPlacementPolicy.java
 
b/distributedlog-service/src/test/java/org/apache/distributedlog/service/placement/TestLeastLoadPlacementPolicy.java
index 4f062f8..5f5ecd4 100644
--- 
a/distributedlog-service/src/test/java/org/apache/distributedlog/service/placement/TestLeastLoadPlacementPolicy.java
+++ 
b/distributedlog-service/src/test/java/org/apache/distributedlog/service/placement/TestLeastLoadPlacementPolicy.java
@@ -147,7 +147,7 @@ public class TestLeastLoadPlacementPolicy {
 assertTrue("Unexpected placement for " + numStreams + " streams to "
 + numSevers + " servers : highest load = " + highestLoadSeen
 + ", lowest load = " + lowestLoadSeen + ", max stream load = " 
+ maxLoad.get(),
-highestLoadSeen - lowestLoadSeen < maxLoad.get());
+highestLoadSeen - lowestLoadSeen <= maxLoad.get());
 }
 
 private Set generateSocketAddresses(int num) {

[GitHub] incubator-distributedlog issue #96: DL-83: Add javadoc to website

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/96
  
@franckcuny ^ 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Git Workflow on Windows

[Sijie Guo]

Chris,

This is very informative and useful. Thank you so much for sharing this. I
will try this out.

Yeah I think most of the issues come from path and symlinks. I think Tracy
already figured the problems out. Looking forward to the contribution.

- Sijie

On Wed, Jan 4, 2017 at 11:24 AM, Chris Nauroth  wrote:

> Microsoft offers free MSDN licenses to Apache committers.  The request
> process is documented here:
>
> https://s.apache.org/BDIK
>
> You'll need to be able to authenticate with your ASF account to access that
> link.
>
> This means that for committers who don't usually work on Windows, you have
> an option for running a Windows VM at no personal cost, in case you want to
> verify a patch running on Windows.  The approval process for the license
> can take a long time though.
>
> From past experience working on Windows compatibility for other projects
> like Hadoop and ZooKeeper, it's very common for software developed
> initially on Linux to assume '/' as the file path separator, and it looks
> like that's what is happening here.  If DistributedLog wants to support
> Windows as a development platform, then the effort would involve tracking
> down incompatibilities like this and fixing them.
>
> This could be a great area for a developer working on Windows to start
> contributing some patches too!
>
>
>
> Chris Nauroth
>
> On Wed, Jan 4, 2017 at 4:56 AM, Tracy Martin 
> wrote:
>
> > Pasted log here:
> >
> > https://gist.github.com/adamtracymartin/2ec2585d9d8efa8f81cbbcaaa2a2ed90
> >
> >
> > -Original Message-
> > From: Franck Cuny [mailto:franck.c...@gmail.com]
> > Sent: Tuesday, January 03, 2017 7:57 PM
> > To: dev@distributedlog.incubator.apache.org
> > Subject: Re: Git Workflow on Windows
> >
> > Hello Tracy,
> >
> > could you paste the complete output of the command you run somewhere ?
> >
> > As far as I know, we haven't tried to build DLog on a windows system, and
> > I don't think we've made any effort to support this so far.
> >
> > On Tue, Jan 3, 2017 at 11:35 AM Tracy Martin 
> > wrote:
> >
> > > I am fixing DL-28 after the merge, and I am following the "Git
> Workflow"
> > > described here:
> > >
> > > https://cwiki.apache.org/confluence/display/DL/Contributing+to+Distrib
> > > utedLog
> > >
> > > I am getting many failures running step 4
> > >
> > > mvn clean apache-rat:check package findbugs:check
> > >
> > > from my Windows desktop.
> > >
> > > I know the error received in DL-28 was related to running the build on
> > > Windows instead of Unix.  Many of the errors that I see in the tests
> > > point to a Windows only problem.  Most of the errors reports are "Path
> > > must start with / character".
> > >
> > > I know when I was first reading about DistributedLog that it required
> > > Unix.  Is the plan to only run from Unix systems?  If so, I will look
> > > at creating a Unix environment for building, testing, and running.
> > > This e-mail and any attachments, contain Genuine Parts Company
> > > confidential information that is proprietary, privileged, and
> > > protected by applicable laws. If you have received this message in
> > > error and are not the intended recipient, you should not retain,
> > > distribute, disclose or use any of this information and you should
> > > destroy this e-mail, any attachments or copies therein forthwith.
> > > Please notify the sender immediately by e-mail if you have received
> this
> > e-mail in error.
> > >
> > > "This e-mail and any attachments, contain Genuine Parts Company
> > > confidential information that is proprietary, privileged, and
> > > protected by applicable laws. If you have received this message in
> > > error and are not the intended recipient, you should not retain,
> > > distribute, disclose or use any of this information and you should
> > > destroy this e-mail, any attachments or copies therein forthwith.
> > > Please notify the sender immediately by e-mail if you have received
> this
> > e-mail in error."
> > >
> > --
> > -franck
> > This e-mail and any attachments, contain Genuine Parts Company
> > confidential information that is proprietary, privileged, and protected
> by
> > applicable laws. If you have received this message in error and are not
> the
> > intended recipient, you should not retain, distribute, disclose or use
> any
> > of this information and you should destroy this e-mail, any attachments
> or
> > copies therein forthwith. Please notify the sender immediately by e-mail
> if
> > you have received this e-mail in error.
> >
> > "This e-mail and any attachments, contain Genuine Parts Company
> > confidential information that is proprietary, privileged, and protected
> by
> > applicable laws. If you have received this message in error and are not
> the
> > intended recipient, you should not retain, distribute, disclose or use
> any
> > of this information and you should destroy this e-mail, any attachments
> or
> > copies 

Re: [DISCUSSION] First Release

[liang xie]

The conflicted DL-167 change has been rebased, please help to review :)

On Thu, Jan 5, 2017 at 8:53 AM, Sijie Guo  wrote:
> FYI. I merged the repackage script and repackage the namespace under
> org.apache.distributedlog. All the changes for 0.4.0 are merged (except one
> test case change - waiting for Liang's response). Ready to cut the first
> release candidate.
>
> - Sijie
>
> On Thu, Dec 29, 2016 at 8:40 PM, Sijie Guo  wrote:
>
>> I've merged all the pull requests that need to be included in 0.4.0 (and
>> both Jenkins and CI builds are green now). Also I moved inactive jiras to
>> 0.5.0.
>>
>> There are a few issues remaining - https://issues.apache.org/
>> jira/issues/?jql=project%20%3D%20DL%20AND%20resolution%20%
>> 3D%20Unresolved%20AND%20fixVersion%20%3D%200.4.0%
>> 20ORDER%20BY%20priority%20DESC
>>
>> Most of them already have pull requests (but need to be rebased to latest
>> master). So I can merge them.
>>
>> Hopefully I will be able to cut 0.4.0 release in following few days.
>>
>> - Sijie
>>
>> On Thu, Dec 8, 2016 at 9:16 PM, Sijie Guo  wrote:
>>
>>> I will start check the jiras that need to be included in 0.4.0 release at
>>> the weekend. If there is any jiras that need to be included in 0.4.0,
>>> please comment in the jira.
>>>
>>> - Sijie
>>>
>>> On Fri, Dec 2, 2016 at 8:28 PM, Sijie Guo  wrote:
>>>
 FYI.

 https://issues.apache.org/jira/browse/DL-81 is created for tracking
 building the process.

 https://issues.apache.org/jira/browse/INFRA-13024 and
 https://issues.apache.org/jira/browse/INFRA-13023 for setting the dists
 for DL.

 - Sijie

 On Fri, Dec 2, 2016 at 8:05 PM, Sijie Guo  wrote:

> Thank you for all the feedbacks. I will drive the release for this
> version. So that we can have all the procedures documented in the wiki.
>
> I created the wiki page here for documenting the procedure:
> https://cwiki.apache.org/confluence/display/DL/Pr
> eparing+DistributedLog+Releases
>
> - Sijie
>
>
> On Mon, Nov 21, 2016 at 7:10 PM, Jia Zhai  wrote:
>
>> +1
>> Great to know the commitment of merge twitter's branch. Looking
>> forward to
>> see bookkeeper 4.5.
>>
>> On Mon, Nov 21, 2016 at 6:36 PM, Flavio Junqueira 
>> wrote:
>>
>> > It does make sense, Sijie, thanks for the update.
>> >
>> > -Flavio
>> >
>> > > On 17 Nov 2016, at 05:12, Sijie Guo  wrote:
>> > >
>> > > Flavio,
>> > >
>> > > I totally agreed that not using an official Apache  version is not
>> great
>> > > for the community. We had a bookkeeper meetup last night. We've
>> discussed
>> > > the current situation with the community. We came to a commitment
>> to
>> > merge
>> > > Twitter's branch back into bookkeeper 4.5. After that we won't
>> maintain
>> > our
>> > > own branch and switch to 4.5.
>> > >
>> > > Hope this make sense.
>> > >
>> > > Sijie
>> > >
>> > > On Nov 16, 2016 8:04 PM, "Flavio Junqueira" 
>> wrote:
>> > >
>> > > It isn't great that DistributedLog is not using the Apache
>> BookKeeper
>> > > release. Essentially anyone using DistributedLog today needs to
>> use the
>> > > Twitter branch of BookKeeper, which has diverged from Apache
>> BookKeeper.
>> > > I'm sure the changes in the Twitter branch are all great, but I'd
>> be more
>> > > comfortable being able to rely on the Apache BookKeeper releases,
>> which
>> > are
>> > > community driven.
>> > >
>> > > I'm not going to block the release on this alone because it is
>> important
>> > > for this project to get a first release out soon, but we need to
>> fix
>> > DL-2.
>> > >
>> > > -Flavio
>> > >
>> > >> On 15 Nov 2016, at 19:02, Leigh Stewart
>> 
>> > > wrote:
>> > >>
>> > >> +1
>> > >>
>> > >> On Tue, Nov 15, 2016 at 7:01 PM, Franck Cuny <
>> franck.c...@gmail.com>
>> > > wrote:
>> > >>
>> > >>> +1 and I agree to not make DL-2 a blocker.
>> > >>>
>> > >>> On Tue, Nov 15, 2016 at 2:02 AM, Xi Liu 
>> wrote:
>> > >>>
>> >  +1 especially on DL-23.
>> > 
>> >  - Xi
>> > 
>> >  On Sat, Nov 12, 2016 at 2:22 AM, Khurrum Nasim <
>> > khurrumnas...@gmail.com>
>> >  wrote:
>> > 
>> > > +1 (non-binding)
>> > >
>> > > I am also interested in participating.
>> > >
>> > > - kn
>> > >
>> > > On Tue, Nov 8, 2016 at 11:08 PM, Sijie Guo 
>> wrote:
>> > >
>> > >> I'd like to start the discussion about the first release.
>> There 

Re: [DISCUSSION] First Release

[Sijie Guo]

FYI. I merged the repackage script and repackage the namespace under
org.apache.distributedlog. All the changes for 0.4.0 are merged (except one
test case change - waiting for Liang's response). Ready to cut the first
release candidate.

- Sijie

On Thu, Dec 29, 2016 at 8:40 PM, Sijie Guo  wrote:

> I've merged all the pull requests that need to be included in 0.4.0 (and
> both Jenkins and CI builds are green now). Also I moved inactive jiras to
> 0.5.0.
>
> There are a few issues remaining - https://issues.apache.org/
> jira/issues/?jql=project%20%3D%20DL%20AND%20resolution%20%
> 3D%20Unresolved%20AND%20fixVersion%20%3D%200.4.0%
> 20ORDER%20BY%20priority%20DESC
>
> Most of them already have pull requests (but need to be rebased to latest
> master). So I can merge them.
>
> Hopefully I will be able to cut 0.4.0 release in following few days.
>
> - Sijie
>
> On Thu, Dec 8, 2016 at 9:16 PM, Sijie Guo  wrote:
>
>> I will start check the jiras that need to be included in 0.4.0 release at
>> the weekend. If there is any jiras that need to be included in 0.4.0,
>> please comment in the jira.
>>
>> - Sijie
>>
>> On Fri, Dec 2, 2016 at 8:28 PM, Sijie Guo  wrote:
>>
>>> FYI.
>>>
>>> https://issues.apache.org/jira/browse/DL-81 is created for tracking
>>> building the process.
>>>
>>> https://issues.apache.org/jira/browse/INFRA-13024 and
>>> https://issues.apache.org/jira/browse/INFRA-13023 for setting the dists
>>> for DL.
>>>
>>> - Sijie
>>>
>>> On Fri, Dec 2, 2016 at 8:05 PM, Sijie Guo  wrote:
>>>
 Thank you for all the feedbacks. I will drive the release for this
 version. So that we can have all the procedures documented in the wiki.

 I created the wiki page here for documenting the procedure:
 https://cwiki.apache.org/confluence/display/DL/Pr
 eparing+DistributedLog+Releases

 - Sijie


 On Mon, Nov 21, 2016 at 7:10 PM, Jia Zhai  wrote:

> +1
> Great to know the commitment of merge twitter's branch. Looking
> forward to
> see bookkeeper 4.5.
>
> On Mon, Nov 21, 2016 at 6:36 PM, Flavio Junqueira 
> wrote:
>
> > It does make sense, Sijie, thanks for the update.
> >
> > -Flavio
> >
> > > On 17 Nov 2016, at 05:12, Sijie Guo  wrote:
> > >
> > > Flavio,
> > >
> > > I totally agreed that not using an official Apache  version is not
> great
> > > for the community. We had a bookkeeper meetup last night. We've
> discussed
> > > the current situation with the community. We came to a commitment
> to
> > merge
> > > Twitter's branch back into bookkeeper 4.5. After that we won't
> maintain
> > our
> > > own branch and switch to 4.5.
> > >
> > > Hope this make sense.
> > >
> > > Sijie
> > >
> > > On Nov 16, 2016 8:04 PM, "Flavio Junqueira" 
> wrote:
> > >
> > > It isn't great that DistributedLog is not using the Apache
> BookKeeper
> > > release. Essentially anyone using DistributedLog today needs to
> use the
> > > Twitter branch of BookKeeper, which has diverged from Apache
> BookKeeper.
> > > I'm sure the changes in the Twitter branch are all great, but I'd
> be more
> > > comfortable being able to rely on the Apache BookKeeper releases,
> which
> > are
> > > community driven.
> > >
> > > I'm not going to block the release on this alone because it is
> important
> > > for this project to get a first release out soon, but we need to
> fix
> > DL-2.
> > >
> > > -Flavio
> > >
> > >> On 15 Nov 2016, at 19:02, Leigh Stewart
> 
> > > wrote:
> > >>
> > >> +1
> > >>
> > >> On Tue, Nov 15, 2016 at 7:01 PM, Franck Cuny <
> franck.c...@gmail.com>
> > > wrote:
> > >>
> > >>> +1 and I agree to not make DL-2 a blocker.
> > >>>
> > >>> On Tue, Nov 15, 2016 at 2:02 AM, Xi Liu 
> wrote:
> > >>>
> >  +1 especially on DL-23.
> > 
> >  - Xi
> > 
> >  On Sat, Nov 12, 2016 at 2:22 AM, Khurrum Nasim <
> > khurrumnas...@gmail.com>
> >  wrote:
> > 
> > > +1 (non-binding)
> > >
> > > I am also interested in participating.
> > >
> > > - kn
> > >
> > > On Tue, Nov 8, 2016 at 11:08 PM, Sijie Guo 
> wrote:
> > >
> > >> I'd like to start the discussion about the first release.
> There are
> > > still a
> > >> few discussions and pull requests outstanding. I think we
> need to
> > >>> pick
> > > up a
> > >> few items and cut the first release and then iterate from
> there.
> > Here
> >  is
> > > a
> > >> 

[51/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

DL-4: Repackage the source under apache namespace


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/b44820b5
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/b44820b5
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/b44820b5

Branch: refs/heads/master
Commit: b44820b50eea9ef2447ec7e136e182ba39e2c269
Parents: 7fab246
Author: Sijie Guo 
Authored: Wed Jan 4 16:44:15 2017 -0800
Committer: Sijie Guo 
Committed: Wed Jan 4 16:46:19 2017 -0800

--
 distributedlog-benchmark/conf/log4j.properties  |8 +-
 .../distributedlog/benchmark/Benchmarker.java   |  468 ---
 .../benchmark/DLWriterWorker.java   |  245 --
 .../distributedlog/benchmark/ReaderWorker.java  |  468 ---
 .../twitter/distributedlog/benchmark/Utils.java |   57 -
 .../distributedlog/benchmark/Worker.java|   26 -
 .../distributedlog/benchmark/WriterWorker.java  |  387 --
 .../distributedlog/benchmark/package-info.java  |   21 -
 .../stream/AbstractReaderBenchmark.java |   70 -
 .../benchmark/stream/AsyncReaderBenchmark.java  |  158 -
 .../benchmark/stream/LedgerBatchReader.java |   82 -
 .../benchmark/stream/LedgerReadBenchmark.java   |  151 -
 .../benchmark/stream/LedgerStreamReader.java|  131 -
 .../benchmark/stream/ReadMode.java  |   28 -
 .../benchmark/stream/StreamBenchmark.java   |  138 -
 .../benchmark/stream/SyncReaderBenchmark.java   |  164 -
 .../benchmark/stream/package-info.java  |   21 -
 .../benchmark/utils/ShiftableRateLimiter.java   |   71 -
 .../benchmark/utils/package-info.java   |   21 -
 .../distributedlog/benchmark/Benchmarker.java   |  468 +++
 .../benchmark/DLWriterWorker.java   |  245 ++
 .../distributedlog/benchmark/ReaderWorker.java  |  468 +++
 .../apache/distributedlog/benchmark/Utils.java  |   57 +
 .../apache/distributedlog/benchmark/Worker.java |   26 +
 .../distributedlog/benchmark/WriterWorker.java  |  387 ++
 .../distributedlog/benchmark/package-info.java  |   21 +
 .../stream/AbstractReaderBenchmark.java |   70 +
 .../benchmark/stream/AsyncReaderBenchmark.java  |  158 +
 .../benchmark/stream/LedgerBatchReader.java |   82 +
 .../benchmark/stream/LedgerReadBenchmark.java   |  151 +
 .../benchmark/stream/LedgerStreamReader.java|  131 +
 .../benchmark/stream/ReadMode.java  |   28 +
 .../benchmark/stream/StreamBenchmark.java   |  138 +
 .../benchmark/stream/SyncReaderBenchmark.java   |  164 +
 .../benchmark/stream/package-info.java  |   21 +
 .../benchmark/utils/ShiftableRateLimiter.java   |   71 +
 .../benchmark/utils/package-info.java   |   21 +
 .../src/main/resources/findbugsExclude.xml  |2 +-
 .../src/main/thrift/loadtest.thrift |2 +-
 distributedlog-client/pom.xml   |2 +-
 .../distributedlog/client/ClientConfig.java |  187 -
 .../client/DistributedLogClientImpl.java| 1200 --
 .../client/DistributedLogMultiStreamWriter.java |  486 ---
 .../client/monitor/MonitorServiceClient.java|   68 -
 .../client/monitor/package-info.java|   21 -
 .../client/ownership/OwnershipCache.java|  235 --
 .../client/ownership/package-info.java  |   21 -
 .../distributedlog/client/package-info.java |   21 -
 .../client/proxy/ClusterClient.java |   51 -
 .../client/proxy/HostProvider.java  |   35 -
 .../client/proxy/ProxyClient.java   |  165 -
 .../client/proxy/ProxyClientManager.java|  362 --
 .../client/proxy/ProxyListener.java |   50 -
 .../client/proxy/package-info.java  |   21 -
 .../client/resolver/DefaultRegionResolver.java  |   85 -
 .../client/resolver/RegionResolver.java |   43 -
 .../client/resolver/package-info.java   |   21 -
 .../routing/ConsistentHashRoutingService.java   |  500 ---
 .../client/routing/NameServerSet.java   |  263 --
 .../client/routing/RegionsRoutingService.java   |  192 -
 .../client/routing/RoutingService.java  |  206 -
 .../client/routing/RoutingServiceProvider.java  |   39 -
 .../client/routing/RoutingUtils.java|   88 -
 .../client/routing/ServerSetRoutingService.java |  274 --
 .../client/routing/ServerSetWatcher.java|   71 -
 .../routing/SingleHostRoutingService.java   |  128 -
 .../distributedlog/client/routing/TestName.java |   49 -
 .../client/routing/TwitterServerSetWatcher.java |   83 -
 .../client/routing/package-info.java|   21 -
 .../client/serverset/DLZkServerSet.java |   91 -
 .../client/serverset/package-info.java  |   21 -
 ...efaultSpeculativeRequestExecutionPolicy.java |  119 -
 .../SpeculativeRequestExecutionPolicy.java  |   34 -
 

[46/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-client/src/main/java/com/twitter/distributedlog/client/proxy/ProxyClient.java
--
diff --git 
a/distributedlog-client/src/main/java/com/twitter/distributedlog/client/proxy/ProxyClient.java
 
b/distributedlog-client/src/main/java/com/twitter/distributedlog/client/proxy/ProxyClient.java
deleted file mode 100644
index d131e28..000
--- 
a/distributedlog-client/src/main/java/com/twitter/distributedlog/client/proxy/ProxyClient.java
+++ /dev/null
@@ -1,165 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.client.proxy;
-
-import com.twitter.distributedlog.client.ClientConfig;
-import com.twitter.distributedlog.client.stats.ClientStats;
-import com.twitter.distributedlog.thrift.service.DistributedLogService;
-import com.twitter.finagle.Service;
-import com.twitter.finagle.ThriftMux;
-import com.twitter.finagle.builder.ClientBuilder;
-import com.twitter.finagle.thrift.ClientId;
-import com.twitter.finagle.thrift.ThriftClientFramedCodec;
-import com.twitter.finagle.thrift.ThriftClientRequest;
-import com.twitter.util.Duration;
-import com.twitter.util.Future;
-import java.net.InetSocketAddress;
-import java.net.SocketAddress;
-import org.apache.thrift.protocol.TBinaryProtocol;
-import scala.Option;
-import scala.runtime.BoxedUnit;
-
-/**
- * Client talks to a single proxy.
- */
-public class ProxyClient {
-
-  /**
-   * Builder to build a proxy client talking to given host 
address.
-   */
-  public interface Builder {
-/**
- * Build a proxy client to address.
- *
- * @param address
- *  proxy address
- * @return proxy client
- */
-ProxyClient build(SocketAddress address);
-}
-
-public static Builder newBuilder(String clientName,
- ClientId clientId,
- ClientBuilder clientBuilder,
- ClientConfig clientConfig,
- ClientStats clientStats) {
-return new DefaultBuilder(clientName, clientId, clientBuilder, 
clientConfig, clientStats);
-}
-
-/**
- * Default Builder for {@link ProxyClient}.
- */
-public static class DefaultBuilder implements Builder {
-
-private final String clientName;
-private final ClientId clientId;
-private final ClientBuilder clientBuilder;
-private final ClientStats clientStats;
-
-private DefaultBuilder(String clientName,
-   ClientId clientId,
-   ClientBuilder clientBuilder,
-   ClientConfig clientConfig,
-   ClientStats clientStats) {
-this.clientName = clientName;
-this.clientId = clientId;
-this.clientStats = clientStats;
-// client builder
-ClientBuilder builder = setDefaultSettings(
-null == clientBuilder ? 
getDefaultClientBuilder(clientConfig) : clientBuilder);
-this.clientBuilder = configureThriftMux(builder, clientId, 
clientConfig);
-}
-
-@SuppressWarnings("unchecked")
-private ClientBuilder configureThriftMux(ClientBuilder builder,
- ClientId clientId,
- ClientConfig clientConfig) {
-if (clientConfig.getThriftMux()) {
-return 
builder.stack(ThriftMux.client().withClientId(clientId));
-} else {
-return 
builder.codec(ThriftClientFramedCodec.apply(Option.apply(clientId)));
-}
-}
-
-private ClientBuilder getDefaultClientBuilder(ClientConfig 
clientConfig) {
-ClientBuilder builder = ClientBuilder.get()
-.tcpConnectTimeout(Duration.fromMilliseconds(200))
-.connectTimeout(Duration.fromMilliseconds(200))
-.requestTimeout(Duration.fromSeconds(1));
-if 

[48/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-benchmark/src/main/java/org/apache/distributedlog/benchmark/Worker.java
--
diff --git 
a/distributedlog-benchmark/src/main/java/org/apache/distributedlog/benchmark/Worker.java
 
b/distributedlog-benchmark/src/main/java/org/apache/distributedlog/benchmark/Worker.java
new file mode 100644
index 000..a948092
--- /dev/null
+++ 
b/distributedlog-benchmark/src/main/java/org/apache/distributedlog/benchmark/Worker.java
@@ -0,0 +1,26 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog.benchmark;
+
+import java.io.Closeable;
+
+/**
+ * Worker to run benchmark.
+ */
+public interface Worker extends Closeable, Runnable {
+}

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-benchmark/src/main/java/org/apache/distributedlog/benchmark/WriterWorker.java
--
diff --git 
a/distributedlog-benchmark/src/main/java/org/apache/distributedlog/benchmark/WriterWorker.java
 
b/distributedlog-benchmark/src/main/java/org/apache/distributedlog/benchmark/WriterWorker.java
new file mode 100644
index 000..9e96765
--- /dev/null
+++ 
b/distributedlog-benchmark/src/main/java/org/apache/distributedlog/benchmark/WriterWorker.java
@@ -0,0 +1,387 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog.benchmark;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.twitter.common.zookeeper.ServerSet;
+import org.apache.distributedlog.DLSN;
+import org.apache.distributedlog.benchmark.utils.ShiftableRateLimiter;
+import org.apache.distributedlog.client.DistributedLogMultiStreamWriter;
+import org.apache.distributedlog.client.serverset.DLZkServerSet;
+import org.apache.distributedlog.exceptions.DLException;
+import org.apache.distributedlog.io.CompressionCodec;
+import org.apache.distributedlog.service.DistributedLogClient;
+import org.apache.distributedlog.service.DistributedLogClientBuilder;
+import org.apache.distributedlog.util.SchedulerUtils;
+import com.twitter.finagle.builder.ClientBuilder;
+import com.twitter.finagle.stats.StatsReceiver;
+import com.twitter.finagle.thrift.ClientId;
+import com.twitter.finagle.thrift.ClientId$;
+import com.twitter.util.Duration$;
+import com.twitter.util.Future;
+import com.twitter.util.FutureEventListener;
+import java.io.IOException;
+import java.net.URI;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import org.apache.bookkeeper.stats.OpStatsLogger;
+import org.apache.bookkeeper.stats.StatsLogger;
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Benchmark for distributedlog proxy client.
+ */
+public class WriterWorker implements Worker {
+
+static final Logger LOG = LoggerFactory.getLogger(WriterWorker.class);
+
+final String streamPrefix;
+final int startStreamId;
+final int endStreamId;
+final int writeConcurrency;
+final int messageSizeBytes;
+final int hostConnectionCoreSize;
+final int hostConnectionLimit;
+final ExecutorService 

[23/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/acl/ZKAccessControlManager.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/acl/ZKAccessControlManager.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/acl/ZKAccessControlManager.java
deleted file mode 100644
index 0c90a50..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/acl/ZKAccessControlManager.java
+++ /dev/null
@@ -1,374 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.impl.acl;
-
-import com.google.common.collect.Sets;
-import com.twitter.distributedlog.DistributedLogConfiguration;
-import com.twitter.distributedlog.ZooKeeperClient;
-import com.twitter.distributedlog.acl.AccessControlManager;
-import com.twitter.distributedlog.exceptions.DLInterruptedException;
-import com.twitter.distributedlog.thrift.AccessControlEntry;
-import com.twitter.util.Await;
-import com.twitter.util.Future;
-import com.twitter.util.FutureEventListener;
-import com.twitter.util.Promise;
-import org.apache.bookkeeper.util.ZkUtils;
-import org.apache.zookeeper.AsyncCallback;
-import org.apache.zookeeper.CreateMode;
-import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.WatchedEvent;
-import org.apache.zookeeper.Watcher;
-import org.apache.zookeeper.ZooKeeper;
-import org.apache.zookeeper.data.Stat;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Set;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicInteger;
-
-/**
- * ZooKeeper Based {@link com.twitter.distributedlog.acl.AccessControlManager}
- */
-public class ZKAccessControlManager implements AccessControlManager, Watcher {
-
-private static final Logger logger = 
LoggerFactory.getLogger(ZKAccessControlManager.class);
-
-private static final int ZK_RETRY_BACKOFF_MS = 500;
-
-protected final DistributedLogConfiguration conf;
-protected final ZooKeeperClient zkc;
-protected final String zkRootPath;
-protected final ScheduledExecutorService scheduledExecutorService;
-
-protected final ConcurrentMap streamEntries;
-protected ZKAccessControl defaultAccessControl;
-protected volatile boolean closed = false;
-
-public ZKAccessControlManager(DistributedLogConfiguration conf,
-  ZooKeeperClient zkc,
-  String zkRootPath,
-  ScheduledExecutorService 
scheduledExecutorService) throws IOException {
-this.conf = conf;
-this.zkc = zkc;
-this.zkRootPath = zkRootPath;
-this.scheduledExecutorService = scheduledExecutorService;
-this.streamEntries = new ConcurrentHashMap();
-try {
-Await.result(fetchDefaultAccessControlEntry());
-} catch (Throwable t) {
-if (t instanceof InterruptedException) {
-throw new DLInterruptedException("Interrupted on getting 
default access control entry for " + zkRootPath, t);
-} else if (t instanceof KeeperException) {
-throw new IOException("Encountered zookeeper exception on 
getting default access control entry for " + zkRootPath, t);
-} else if (t instanceof IOException) {
-throw (IOException) t;
-} else {
-throw new IOException("Encountered unknown exception on 
getting access control entries for " + zkRootPath, t);
-}
-}
-
-try {
-Await.result(fetchAccessControlEntries());
-} catch (Throwable t) {
-if (t instanceof InterruptedException) {
-throw new DLInterruptedException("Interrupted on getting 
access control 

[36/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/BKDistributedLogManager.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKDistributedLogManager.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKDistributedLogManager.java
deleted file mode 100644
index a3959b0..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKDistributedLogManager.java
+++ /dev/null
@@ -1,1106 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Optional;
-import com.google.common.base.Preconditions;
-import com.twitter.distributedlog.callback.LogSegmentListener;
-import com.twitter.distributedlog.config.DynamicDistributedLogConfiguration;
-import com.twitter.distributedlog.exceptions.AlreadyClosedException;
-import com.twitter.distributedlog.exceptions.LogEmptyException;
-import com.twitter.distributedlog.exceptions.LogNotFoundException;
-import com.twitter.distributedlog.exceptions.UnexpectedException;
-import com.twitter.distributedlog.function.CloseAsyncCloseableFunction;
-import com.twitter.distributedlog.function.GetVersionedValueFunction;
-import com.twitter.distributedlog.injector.AsyncFailureInjector;
-import com.twitter.distributedlog.logsegment.LogSegmentEntryStore;
-import com.twitter.distributedlog.logsegment.LogSegmentEntryWriter;
-import com.twitter.distributedlog.metadata.LogMetadataForReader;
-import com.twitter.distributedlog.metadata.LogMetadataForWriter;
-import com.twitter.distributedlog.io.AsyncCloseable;
-import com.twitter.distributedlog.lock.DistributedLock;
-import com.twitter.distributedlog.lock.NopDistributedLock;
-import com.twitter.distributedlog.lock.ZKDistributedLock;
-import com.twitter.distributedlog.logsegment.LogSegmentFilter;
-import com.twitter.distributedlog.logsegment.LogSegmentMetadataCache;
-import com.twitter.distributedlog.metadata.LogStreamMetadataStore;
-import com.twitter.distributedlog.namespace.NamespaceDriver;
-import com.twitter.distributedlog.stats.BroadCastStatsLogger;
-import com.twitter.distributedlog.subscription.SubscriptionsStore;
-import com.twitter.distributedlog.util.Allocator;
-import com.twitter.distributedlog.util.DLUtils;
-import com.twitter.distributedlog.util.FutureUtils;
-import com.twitter.distributedlog.util.MonitoredFuturePool;
-import com.twitter.distributedlog.util.OrderedScheduler;
-import com.twitter.distributedlog.util.PermitLimiter;
-import com.twitter.distributedlog.util.PermitManager;
-import com.twitter.distributedlog.util.SchedulerUtils;
-import com.twitter.distributedlog.util.Utils;
-import com.twitter.util.ExceptionalFunction;
-import com.twitter.util.ExceptionalFunction0;
-import com.twitter.util.Function;
-import com.twitter.util.Future;
-import com.twitter.util.FutureEventListener;
-import com.twitter.util.Promise;
-import org.apache.bookkeeper.feature.FeatureProvider;
-import org.apache.bookkeeper.stats.AlertStatsLogger;
-import org.apache.bookkeeper.stats.StatsLogger;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import scala.runtime.AbstractFunction0;
-import scala.runtime.AbstractFunction1;
-import scala.runtime.BoxedUnit;
-
-import java.io.Closeable;
-import java.io.IOException;
-import java.net.URI;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Set;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.TimeUnit;
-
-import static com.twitter.distributedlog.namespace.NamespaceDriver.Role.READER;
-import static com.twitter.distributedlog.namespace.NamespaceDriver.Role.WRITER;
-
-/**
- * Metrics
- * 
- *  `log_writer/*`: all asynchronous writer related metrics are exposed 
under scope `log_writer`.
- * See {@link BKAsyncLogWriter} for detail stats.
- *  `async_reader/*`: all asyncrhonous reader related metrics are exposed 
under scope `async_reader`.
- * See {@link BKAsyncLogReader} for detail stats.
- *  `writer_future_pool/*`: metrics about the future pools that 

[27/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/ZooKeeperClient.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/ZooKeeperClient.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/ZooKeeperClient.java
deleted file mode 100644
index 74cd6cf..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/ZooKeeperClient.java
+++ /dev/null
@@ -1,402 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog;
-
-import com.google.common.base.Stopwatch;
-import com.twitter.distributedlog.util.FailpointUtils;
-import com.twitter.distributedlog.zk.ZKWatcherManager;
-import org.apache.bookkeeper.stats.NullStatsLogger;
-import org.apache.bookkeeper.stats.StatsLogger;
-import org.apache.bookkeeper.zookeeper.BoundExponentialBackoffRetryPolicy;
-import org.apache.bookkeeper.zookeeper.RetryPolicy;
-import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.WatchedEvent;
-import org.apache.zookeeper.Watcher;
-import org.apache.zookeeper.Watcher.Event.EventType;
-import org.apache.zookeeper.Watcher.Event.KeeperState;
-import org.apache.zookeeper.ZooKeeper;
-import org.apache.zookeeper.ZooDefs;
-import org.apache.zookeeper.data.ACL;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Set;
-import java.util.concurrent.CopyOnWriteArraySet;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
-
-import static com.google.common.base.Charsets.UTF_8;
-
-/**
- * ZooKeeper Client wrapper over {@link 
org.apache.bookkeeper.zookeeper.ZooKeeperClient}.
- * It handles retries on session expires and provides a watcher manager {@link 
ZKWatcherManager}.
- *
- * Metrics
- * 
- *  zookeeper operation stats are exposed under scope zk by
- * {@link org.apache.bookkeeper.zookeeper.ZooKeeperClient}
- *  stats on zookeeper watched events are exposed under scope 
watcher by
- * {@link org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase}
- *  stats about {@link ZKWatcherManager} are exposed under scope 
watcher_manager
- * 
- */
-public class ZooKeeperClient {
-
-public static interface Credentials {
-
-Credentials NONE = new Credentials() {
-@Override
-public void authenticate(ZooKeeper zooKeeper) {
-// noop
-}
-};
-
-void authenticate(ZooKeeper zooKeeper);
-}
-
-public static class DigestCredentials implements Credentials {
-
-String username;
-String password;
-
-public DigestCredentials(String username, String password) {
-this.username = username;
-this.password = password;
-}
-
-@Override
-public void authenticate(ZooKeeper zooKeeper) {
-zooKeeper.addAuthInfo("digest", String.format("%s:%s", username, 
password).getBytes(UTF_8));
-}
-}
-
-public interface ZooKeeperSessionExpireNotifier {
-void notifySessionExpired();
-}
-
-/**
- * Indicates an error connecting to a zookeeper cluster.
- */
-public static class ZooKeeperConnectionException extends IOException {
-private static final long serialVersionUID = 6682391687004819361L;
-
-public ZooKeeperConnectionException(String message) {
-super(message);
-}
-
-public ZooKeeperConnectionException(String message, Throwable cause) {
-super(message, cause);
-}
-}
-
-private static final Logger LOG = 
LoggerFactory.getLogger(ZooKeeperClient.class.getName());
-
-private final String name;
-private final int sessionTimeoutMs;
-private final int defaultConnectionTimeoutMs;
-private final String zooKeeperServers;
-// GuardedBy "this", but still volatile for tests, where we want to be 
able to see writes
-// made from within long synchronized blocks.
-private volatile ZooKeeper zooKeeper = null;
-private final 

[44/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-client/src/main/java/com/twitter/distributedlog/service/DistributedLogClientBuilder.java
--
diff --git 
a/distributedlog-client/src/main/java/com/twitter/distributedlog/service/DistributedLogClientBuilder.java
 
b/distributedlog-client/src/main/java/com/twitter/distributedlog/service/DistributedLogClientBuilder.java
deleted file mode 100644
index 3f65aff..000
--- 
a/distributedlog-client/src/main/java/com/twitter/distributedlog/service/DistributedLogClientBuilder.java
+++ /dev/null
@@ -1,608 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.service;
-
-import static com.google.common.base.Preconditions.checkNotNull;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Optional;
-import com.twitter.common.zookeeper.ServerSet;
-import com.twitter.distributedlog.client.ClientConfig;
-import com.twitter.distributedlog.client.DistributedLogClientImpl;
-import com.twitter.distributedlog.client.monitor.MonitorServiceClient;
-import com.twitter.distributedlog.client.proxy.ClusterClient;
-import com.twitter.distributedlog.client.resolver.DefaultRegionResolver;
-import com.twitter.distributedlog.client.resolver.RegionResolver;
-import com.twitter.distributedlog.client.routing.RegionsRoutingService;
-import com.twitter.distributedlog.client.routing.RoutingService;
-import com.twitter.distributedlog.client.routing.RoutingUtils;
-import com.twitter.distributedlog.thrift.service.DistributedLogService;
-import com.twitter.finagle.Name;
-import com.twitter.finagle.Resolver$;
-import com.twitter.finagle.Service;
-import com.twitter.finagle.ThriftMux;
-import com.twitter.finagle.builder.ClientBuilder;
-import com.twitter.finagle.stats.NullStatsReceiver;
-import com.twitter.finagle.stats.StatsReceiver;
-import com.twitter.finagle.thrift.ClientId;
-import com.twitter.finagle.thrift.ThriftClientFramedCodec;
-import com.twitter.finagle.thrift.ThriftClientRequest;
-import com.twitter.util.Duration;
-import java.net.SocketAddress;
-import java.net.URI;
-import java.util.Random;
-import org.apache.commons.lang.StringUtils;
-import org.apache.thrift.protocol.TBinaryProtocol;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import scala.Option;
-
-/**
- * Builder to build {@link DistributedLogClient}.
- */
-public final class DistributedLogClientBuilder {
-
-private static final Logger logger = 
LoggerFactory.getLogger(DistributedLogClientBuilder.class);
-
-private static final Random random = new 
Random(System.currentTimeMillis());
-
-private String name = null;
-private ClientId clientId = null;
-private RoutingService.Builder routingServiceBuilder = null;
-private ClientBuilder clientBuilder = null;
-private String serverRoutingServiceFinagleName = null;
-private StatsReceiver statsReceiver = new NullStatsReceiver();
-private StatsReceiver streamStatsReceiver = new NullStatsReceiver();
-private ClientConfig clientConfig = new ClientConfig();
-private boolean enableRegionStats = false;
-private final RegionResolver regionResolver = new DefaultRegionResolver();
-
-/**
- * Create a client builder.
- *
- * @return client builder
- */
-public static DistributedLogClientBuilder newBuilder() {
-return new DistributedLogClientBuilder();
-}
-
-/**
- * Create a new client builder from an existing {@code builder}.
- *
- * @param builder the existing builder.
- * @return a new client builder.
- */
-public static DistributedLogClientBuilder 
newBuilder(DistributedLogClientBuilder builder) {
-DistributedLogClientBuilder newBuilder = new 
DistributedLogClientBuilder();
-newBuilder.name = builder.name;
-newBuilder.clientId = builder.clientId;
-newBuilder.clientBuilder = builder.clientBuilder;
-newBuilder.routingServiceBuilder = builder.routingServiceBuilder;
-newBuilder.statsReceiver = builder.statsReceiver;
-newBuilder.streamStatsReceiver = 

[05/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/org/apache/distributedlog/DistributedLogConstants.java
--
diff --git 
a/distributedlog-core/src/main/java/org/apache/distributedlog/DistributedLogConstants.java
 
b/distributedlog-core/src/main/java/org/apache/distributedlog/DistributedLogConstants.java
new file mode 100644
index 000..f951991
--- /dev/null
+++ 
b/distributedlog-core/src/main/java/org/apache/distributedlog/DistributedLogConstants.java
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog;
+
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.zookeeper.ZooDefs.Ids;
+import org.apache.zookeeper.data.ACL;
+
+import static com.google.common.base.Charsets.UTF_8;
+
+public class DistributedLogConstants {
+public static final byte[] EMPTY_BYTES = new byte[0];
+public static final String SCHEME_PREFIX = "distributedlog";
+public static final String BACKEND_BK = "bk";
+public static final long INVALID_TXID = -999;
+public static final long EMPTY_LOGSEGMENT_TX_ID = -99;
+public static final long MAX_TXID = Long.MAX_VALUE;
+public static final long SMALL_LOGSEGMENT_THRESHOLD = 10;
+public static final int LOGSEGMENT_NAME_VERSION = 1;
+public static final int FUTURE_TIMEOUT_IMMEDIATE = 0;
+public static final int FUTURE_TIMEOUT_INFINITE = -1;
+public static final long LOCK_IMMEDIATE = FUTURE_TIMEOUT_IMMEDIATE;
+public static final long LOCK_TIMEOUT_INFINITE = FUTURE_TIMEOUT_INFINITE;
+public static final long LOCK_OP_TIMEOUT_DEFAULT = 120;
+public static final long LOCK_REACQUIRE_TIMEOUT_DEFAULT = 120;
+public static final String UNKNOWN_CLIENT_ID = "Unknown-ClientId";
+public static final int LOCAL_REGION_ID = 0;
+public static final long LOGSEGMENT_DEFAULT_STATUS = 0;
+public static final long UNASSIGNED_LOGSEGMENT_SEQNO = 0;
+public static final long UNASSIGNED_SEQUENCE_ID = -1L;
+public static final long FIRST_LOGSEGMENT_SEQNO = 1;
+public static final long UNRESOLVED_LEDGER_ID = -1;
+public static final long LATENCY_WARN_THRESHOLD_IN_MILLIS = 
TimeUnit.SECONDS.toMillis(1);
+public static final int DL_INTERRUPTED_EXCEPTION_RESULT_CODE = 
Integer.MIN_VALUE + 1;
+public static final int ZK_CONNECTION_EXCEPTION_RESULT_CODE = 
Integer.MIN_VALUE + 2;
+
+public static final String ALLOCATION_POOL_NODE = ".allocation_pool";
+// log segment prefix
+public static final String INPROGRESS_LOGSEGMENT_PREFIX = "inprogress";
+public static final String COMPLETED_LOGSEGMENT_PREFIX = "logrecs";
+public static final String DISALLOW_PLACEMENT_IN_REGION_FEATURE_NAME = 
"disallow_bookie_placement";
+static final byte[] CONTROL_RECORD_CONTENT = "control".getBytes(UTF_8);
+static final byte[] KEEPALIVE_RECORD_CONTENT = "keepalive".getBytes(UTF_8);
+
+// An ACL that gives all permissions to node creators and read permissions 
only to everyone else.
+public static final List EVERYONE_READ_CREATOR_ALL =
+ImmutableList.builder()
+.addAll(Ids.CREATOR_ALL_ACL)
+.addAll(Ids.READ_ACL_UNSAFE)
+.build();
+}

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/org/apache/distributedlog/DistributedLogManager.java
--
diff --git 
a/distributedlog-core/src/main/java/org/apache/distributedlog/DistributedLogManager.java
 
b/distributedlog-core/src/main/java/org/apache/distributedlog/DistributedLogManager.java
new file mode 100644
index 000..7d33e9c
--- /dev/null
+++ 
b/distributedlog-core/src/main/java/org/apache/distributedlog/DistributedLogManager.java
@@ -0,0 +1,308 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this 

[13/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/zk/ZKVersionedSetOp.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/zk/ZKVersionedSetOp.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/zk/ZKVersionedSetOp.java
deleted file mode 100644
index 5b788e2..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/zk/ZKVersionedSetOp.java
+++ /dev/null
@@ -1,71 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.zk;
-
-import com.twitter.distributedlog.util.Transaction.OpListener;
-import org.apache.bookkeeper.meta.ZkVersion;
-import org.apache.bookkeeper.versioning.Version;
-import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.Op;
-import org.apache.zookeeper.OpResult;
-
-import javax.annotation.Nullable;
-
-/**
- * ZooKeeper Operation that plays with {@link 
org.apache.bookkeeper.versioning.Version}
- */
-public class ZKVersionedSetOp extends ZKOp {
-
-private final OpListener listener;
-
-public ZKVersionedSetOp(Op op,
-@Nullable OpListener opListener) {
-super(op);
-this.listener = opListener;
-}
-
-@Override
-protected void commitOpResult(OpResult opResult) {
-assert(opResult instanceof OpResult.SetDataResult);
-OpResult.SetDataResult setDataResult = (OpResult.SetDataResult) 
opResult;
-if (null != listener) {
-listener.onCommit(new 
ZkVersion(setDataResult.getStat().getVersion()));
-}
-}
-
-@Override
-protected void abortOpResult(Throwable t,
- @Nullable OpResult opResult) {
-Throwable cause;
-if (null == opResult) {
-cause = t;
-} else {
-assert (opResult instanceof OpResult.ErrorResult);
-OpResult.ErrorResult errorResult = (OpResult.ErrorResult) opResult;
-if (KeeperException.Code.OK.intValue() == errorResult.getErr()) {
-cause = t;
-} else {
-cause = 
KeeperException.create(KeeperException.Code.get(errorResult.getErr()));
-}
-}
-if (null != listener) {
-listener.onAbort(cause);
-}
-}
-
-}

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/zk/ZKWatcherManager.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/zk/ZKWatcherManager.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/zk/ZKWatcherManager.java
deleted file mode 100644
index 8ef33ea..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/zk/ZKWatcherManager.java
+++ /dev/null
@@ -1,239 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.zk;
-
-import com.twitter.distributedlog.ZooKeeperClient;
-import org.apache.bookkeeper.stats.Gauge;
-import org.apache.bookkeeper.stats.StatsLogger;
-import org.apache.zookeeper.AsyncCallback;
-import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.WatchedEvent;
-import org.apache.zookeeper.Watcher;
-import 

[50/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-benchmark/src/main/java/com/twitter/distributedlog/benchmark/Benchmarker.java
--
diff --git 
a/distributedlog-benchmark/src/main/java/com/twitter/distributedlog/benchmark/Benchmarker.java
 
b/distributedlog-benchmark/src/main/java/com/twitter/distributedlog/benchmark/Benchmarker.java
deleted file mode 100644
index 5b04a05..000
--- 
a/distributedlog-benchmark/src/main/java/com/twitter/distributedlog/benchmark/Benchmarker.java
+++ /dev/null
@@ -1,468 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.benchmark;
-
-import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.base.Preconditions.checkNotNull;
-
-import com.twitter.distributedlog.DistributedLogConfiguration;
-import com.twitter.distributedlog.benchmark.utils.ShiftableRateLimiter;
-import com.twitter.finagle.stats.OstrichStatsReceiver;
-import com.twitter.finagle.stats.StatsReceiver;
-import java.io.File;
-import java.io.IOException;
-import java.net.URI;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.List;
-import java.util.concurrent.TimeUnit;
-import org.apache.bookkeeper.stats.NullStatsProvider;
-import org.apache.bookkeeper.stats.StatsLogger;
-import org.apache.bookkeeper.stats.StatsProvider;
-import org.apache.bookkeeper.util.ReflectionUtils;
-import org.apache.commons.cli.BasicParser;
-import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.HelpFormatter;
-import org.apache.commons.cli.Options;
-import org.apache.commons.lang.StringUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * The launcher for benchmarks.
- */
-public class Benchmarker {
-
-private static final Logger logger = 
LoggerFactory.getLogger(Benchmarker.class);
-
-static final String USAGE = "Benchmarker [-u ] [-c ] [-s 
serverset] [-m (read|write|dlwrite)]";
-
-final String[] args;
-final Options options = new Options();
-
-int rate = 100;
-int maxRate = 1000;
-int changeRate = 100;
-int changeRateSeconds = 1800;
-int concurrency = 10;
-String streamPrefix = "dlog-loadtest";
-int shardId = -1;
-int numStreams = 10;
-List serversetPaths = new ArrayList();
-List finagleNames = new ArrayList();
-int msgSize = 256;
-String mode = null;
-int durationMins = 60;
-URI dlUri = null;
-int batchSize = 0;
-int readersPerStream = 1;
-Integer maxStreamId = null;
-int truncationInterval = 3600;
-Integer startStreamId = null;
-Integer endStreamId = null;
-int hostConnectionCoreSize = 10;
-int hostConnectionLimit = 10;
-boolean thriftmux = false;
-boolean handshakeWithClientInfo = false;
-boolean readFromHead = false;
-int sendBufferSize = 1024 * 1024;
-int recvBufferSize = 1024 * 1024;
-boolean enableBatching = false;
-int batchBufferSize = 256 * 1024;
-int batchFlushIntervalMicros = 2000;
-String routingServiceFinagleNameString;
-
-final DistributedLogConfiguration conf = new DistributedLogConfiguration();
-final StatsReceiver statsReceiver = new OstrichStatsReceiver();
-StatsProvider statsProvider = null;
-
-Benchmarker(String[] args) {
-this.args = args;
-// prepare options
-options.addOption("s", "serverset", true, "Proxy Server Set (separated 
by ',')");
-options.addOption("fn", "finagle-name", true, "Write proxy finagle 
name (separated by ',')");
-options.addOption("c", "conf", true, "DistributedLog Configuration 
File");
-options.addOption("u", "uri", true, "DistributedLog URI");
-options.addOption("i", "shard", true, "Shard Id");
-options.addOption("p", "provider", true, "DistributedLog Stats 
Provider");
-options.addOption("d", "duration", true, "Duration (minutes)");
-options.addOption("sp", "streamprefix", true, "Stream Prefix");
-options.addOption("sc", "streamcount", true, "Number of Streams");
-options.addOption("ms", "messagesize", true, "Message Size 

[04/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/org/apache/distributedlog/LogSegmentMetadata.java
--
diff --git 
a/distributedlog-core/src/main/java/org/apache/distributedlog/LogSegmentMetadata.java
 
b/distributedlog-core/src/main/java/org/apache/distributedlog/LogSegmentMetadata.java
new file mode 100644
index 000..c5050ec
--- /dev/null
+++ 
b/distributedlog-core/src/main/java/org/apache/distributedlog/LogSegmentMetadata.java
@@ -0,0 +1,1125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Comparator;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Objects;
+import org.apache.distributedlog.exceptions.DLInterruptedException;
+import org.apache.distributedlog.exceptions.LogSegmentNotFoundException;
+import 
org.apache.distributedlog.exceptions.UnsupportedMetadataVersionException;
+import org.apache.distributedlog.exceptions.ZKException;
+import org.apache.distributedlog.util.FutureUtils;
+import org.apache.distributedlog.util.Utils;
+import com.twitter.util.Future;
+import com.twitter.util.Promise;
+import org.apache.zookeeper.AsyncCallback;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.data.Stat;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static com.google.common.base.Charsets.UTF_8;
+
+/**
+ * Utility class for storing the metadata associated
+ * with a single edit log segment, stored in a single ledger
+ */
+public class LogSegmentMetadata {
+static final Logger LOG = 
LoggerFactory.getLogger(LogSegmentMetadata.class);
+
+public static enum LogSegmentMetadataVersion {
+VERSION_INVALID(0),
+VERSION_V1_ORIGINAL(1),
+VERSION_V2_LEDGER_SEQNO(2),
+VERSION_V3_MIN_ACTIVE_DLSN(3),
+VERSION_V4_ENVELOPED_ENTRIES(4),
+VERSION_V5_SEQUENCE_ID(5);
+
+public final int value;
+
+private LogSegmentMetadataVersion(int value) {
+this.value = value;
+}
+
+public static LogSegmentMetadataVersion of(int version) {
+switch (version) {
+case 5:
+return VERSION_V5_SEQUENCE_ID;
+case 4:
+return VERSION_V4_ENVELOPED_ENTRIES;
+case 3:
+return VERSION_V3_MIN_ACTIVE_DLSN;
+case 2:
+return VERSION_V2_LEDGER_SEQNO;
+case 1:
+return VERSION_V1_ORIGINAL;
+case 0:
+return VERSION_INVALID;
+default:
+throw new IllegalArgumentException("unknown version " + 
version);
+}
+}
+}
+
+public static enum TruncationStatus {
+ACTIVE (0), PARTIALLY_TRUNCATED(1), TRUNCATED (2);
+private final int value;
+
+private TruncationStatus(int value) {
+this.value = value;
+}
+}
+
+public static class LogSegmentMetadataBuilder {
+protected String zkPath;
+protected long logSegmentId;
+protected LogSegmentMetadataVersion version;
+protected long firstTxId;
+protected int regionId;
+protected long status;
+protected long lastTxId;
+protected long completionTime;
+protected int recordCount;
+protected long logSegmentSequenceNo;
+protected long lastEntryId;
+protected long lastSlotId;
+protected long minActiveEntryId;
+protected long minActiveSlotId;
+protected long startSequenceId;
+protected boolean inprogress;
+
+// This is a derived attribute.
+// Since we overwrite the original version with the target version, 
information that is
+// derived from the original version (e.g. does it support enveloping 
of entries)
+// is lost while parsing.
+// NOTE: This value is not stored in the Metadata store.
+protected 

[06/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/org/apache/distributedlog/DistributedLogConfiguration.java
--
diff --git 
a/distributedlog-core/src/main/java/org/apache/distributedlog/DistributedLogConfiguration.java
 
b/distributedlog-core/src/main/java/org/apache/distributedlog/DistributedLogConfiguration.java
new file mode 100644
index 000..0cb608f
--- /dev/null
+++ 
b/distributedlog-core/src/main/java/org/apache/distributedlog/DistributedLogConfiguration.java
@@ -0,0 +1,3528 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Sets;
+import org.apache.distributedlog.bk.QuorumConfig;
+import org.apache.distributedlog.feature.DefaultFeatureProvider;
+import org.apache.distributedlog.namespace.DistributedLogNamespaceBuilder;
+import org.apache.distributedlog.net.DNSResolverForRacks;
+import org.apache.distributedlog.net.DNSResolverForRows;
+import org.apache.bookkeeper.conf.ClientConfiguration;
+import org.apache.bookkeeper.feature.FeatureProvider;
+import org.apache.bookkeeper.net.DNSToSwitchMapping;
+import org.apache.bookkeeper.stats.StatsLogger;
+import org.apache.bookkeeper.util.ReflectionUtils;
+import org.apache.commons.configuration.CompositeConfiguration;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.configuration.ConfigurationException;
+import org.apache.commons.configuration.PropertiesConfiguration;
+import org.apache.commons.configuration.SystemConfiguration;
+import org.apache.commons.lang.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.Set;
+
+/**
+ * DistributedLog Configuration.
+ * 
+ * DistributedLog configuration is basically a properties based configuration, 
which extends from
+ * Apache commons {@link CompositeConfiguration}. All the DL settings are in 
camel case and prefixed
+ * with a meaningful component name. for example, `zkSessionTimeoutSeconds` 
means SessionTimeoutSeconds
+ * for component `zk`.
+ *
+ * BookKeeper Configuration
+ *
+ * BookKeeper client configuration settings could be loaded via DistributedLog 
configuration. All those
+ * settings are prefixed with `bkc.`. For example, bkc.zkTimeout 
in distributedlog configuration
+ * will be applied as `zkTimeout` in bookkeeper client configuration.
+ *
+ * How to load configuration
+ *
+ * The default distributedlog configuration is constructed by instantiated a 
new instance. This
+ * distributedlog configuration will automatically load the settings that 
specified via
+ * {@link SystemConfiguration}.
+ *
+ * 
+ *  DistributedLogConfiguration conf = new DistributedLogConfiguration();
+ * 
+ *
+ * The recommended way is to load configuration from URL that points to a 
configuration file
+ * ({@link #loadConf(URL)}).
+ *
+ * 
+ *  String configFile = "/path/to/distributedlog/conf/file";
+ *  DistributedLogConfiguration conf = new DistributedLogConfiguration();
+ *  conf.loadConf(new File(configFile).toURI().toURL());
+ * 
+ *
+ * @see org.apache.bookkeeper.conf.ClientConfiguration
+ */
+public class DistributedLogConfiguration extends CompositeConfiguration {
+static final Logger LOG = 
LoggerFactory.getLogger(DistributedLogConfiguration.class);
+
+private static ClassLoader defaultLoader;
+
+static {
+defaultLoader = Thread.currentThread().getContextClassLoader();
+if (null == defaultLoader) {
+defaultLoader = DistributedLogConfiguration.class.getClassLoader();
+}
+}
+
+//
+// ZooKeeper Related Settings
+//
+
+public static final String BKDL_ZK_ACL_ID = "zkAclId";
+public static final String BKDL_ZK_ACL_ID_DEFAULT = null;
+public static final String BKDL_ZK_SESSION_TIMEOUT_SECONDS = 
"zkSessionTimeoutSeconds";
+public static final int BKDL_ZK_SESSION_TIMEOUT_SECONDS_DEFAULT = 30;
+public static final String 

[28/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/ReadAheadEntryReader.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/ReadAheadEntryReader.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/ReadAheadEntryReader.java
deleted file mode 100644
index 40e3930..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/ReadAheadEntryReader.java
+++ /dev/null
@@ -1,992 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Stopwatch;
-import com.google.common.base.Ticker;
-import com.google.common.collect.Lists;
-import com.twitter.distributedlog.callback.LogSegmentListener;
-import 
com.twitter.distributedlog.exceptions.AlreadyTruncatedTransactionException;
-import com.twitter.distributedlog.exceptions.DLIllegalStateException;
-import com.twitter.distributedlog.exceptions.DLInterruptedException;
-import com.twitter.distributedlog.exceptions.EndOfLogSegmentException;
-import com.twitter.distributedlog.exceptions.LogNotFoundException;
-import com.twitter.distributedlog.exceptions.UnexpectedException;
-import com.twitter.distributedlog.io.AsyncCloseable;
-import com.twitter.distributedlog.logsegment.LogSegmentEntryReader;
-import com.twitter.distributedlog.logsegment.LogSegmentEntryStore;
-import com.twitter.distributedlog.logsegment.LogSegmentFilter;
-import com.twitter.distributedlog.util.OrderedScheduler;
-import com.twitter.util.Function0;
-import com.twitter.util.Future;
-import com.twitter.util.FutureEventListener;
-import com.twitter.util.Futures;
-import com.twitter.util.Promise;
-import org.apache.bookkeeper.stats.AlertStatsLogger;
-import org.apache.bookkeeper.versioning.Versioned;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import scala.Function1;
-import scala.runtime.AbstractFunction1;
-import scala.runtime.BoxedUnit;
-
-import java.io.IOException;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.concurrent.CopyOnWriteArraySet;
-import java.util.concurrent.LinkedBlockingQueue;
-import java.util.concurrent.RejectedExecutionException;
-import java.util.concurrent.ScheduledFuture;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicReference;
-
-/**
- * New ReadAhead Reader that uses {@link 
com.twitter.distributedlog.logsegment.LogSegmentEntryReader}.
- *
- * NOTE: all the state changes happen in the same thread. All *unsafe* methods 
should be submitted to the order
- * scheduler using stream name as the key.
- */
-public class ReadAheadEntryReader implements
-AsyncCloseable,
-LogSegmentListener,
-LogSegmentEntryReader.StateChangeListener,
-FutureEventListener {
-
-private static final Logger logger = 
LoggerFactory.getLogger(ReadAheadEntryReader.class);
-
-//
-// Static Functions
-//
-
-private static AbstractFunction1 
START_READER_FUNC = new AbstractFunction1() {
-@Override
-public BoxedUnit apply(LogSegmentEntryReader reader) {
-reader.start();
-return BoxedUnit.UNIT;
-}
-};
-
-//
-// Internal Classes
-//
-
-class SegmentReader implements FutureEventListener {
-
-private LogSegmentMetadata metadata;
-private final long startEntryId;
-private Future openFuture = null;
-private LogSegmentEntryReader reader = null;
-private boolean isStarted = false;
-private boolean isClosed = false;
-
-SegmentReader(LogSegmentMetadata metadata,
-  long startEntryId) {
-this.metadata = metadata;
-this.startEntryId = startEntryId;
-}
-
-synchronized LogSegmentEntryReader getEntryReader() {
-return reader;
-}
-
-synchronized boolean isBeyondLastAddConfirmed() 

[35/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
deleted file mode 100644
index 0cf8ed5..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogHandler.java
+++ /dev/null
@@ -1,715 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog;
-
-import com.google.common.base.Stopwatch;
-import com.twitter.distributedlog.callback.LogSegmentNamesListener;
-import com.twitter.distributedlog.exceptions.LogEmptyException;
-import com.twitter.distributedlog.exceptions.LogSegmentNotFoundException;
-import com.twitter.distributedlog.exceptions.UnexpectedException;
-import com.twitter.distributedlog.logsegment.LogSegmentEntryStore;
-import com.twitter.distributedlog.metadata.LogMetadata;
-import com.twitter.distributedlog.io.AsyncAbortable;
-import com.twitter.distributedlog.io.AsyncCloseable;
-import com.twitter.distributedlog.logsegment.LogSegmentMetadataCache;
-import com.twitter.distributedlog.logsegment.PerStreamLogSegmentCache;
-import com.twitter.distributedlog.logsegment.LogSegmentFilter;
-import com.twitter.distributedlog.logsegment.LogSegmentMetadataStore;
-import com.twitter.distributedlog.metadata.LogStreamMetadataStore;
-import com.twitter.distributedlog.util.FutureUtils;
-import com.twitter.distributedlog.util.OrderedScheduler;
-import com.twitter.util.Function;
-import com.twitter.util.Future;
-import com.twitter.util.FutureEventListener;
-import com.twitter.util.Promise;
-import org.apache.bookkeeper.stats.AlertStatsLogger;
-import org.apache.bookkeeper.stats.OpStatsLogger;
-import org.apache.bookkeeper.stats.StatsLogger;
-import org.apache.bookkeeper.versioning.Version;
-import org.apache.bookkeeper.versioning.Versioned;
-import org.apache.commons.lang3.tuple.Pair;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicInteger;
-import java.util.concurrent.atomic.AtomicReference;
-
-/**
- * The base class about log handler on managing log segments.
- *
- * Metrics
- * The log handler is a base class on managing log segments. so all the metrics
- * here are related to log segments retrieval and exposed under `logsegments`.
- * These metrics are all OpStats, in the format of 
`scope`/logsegments/`op`.
- * 
- * Those operations are:
- * 
- * get_inprogress_segment: time between the inprogress log segment created 
and
- * the handler read it.
- * get_completed_segment: time between a log segment is turned to 
completed and
- * the handler read it.
- * negative_get_inprogress_segment: record the negative values for 
`get_inprogress_segment`.
- * negative_get_completed_segment: record the negative values for 
`get_completed_segment`.
- * recover_last_entry: recovering last entry from a log segment
- * recover_scanned_entries: the number of entries that are scanned during 
recovering.
- * 
- * @see BKLogWriteHandler
- * @see BKLogReadHandler
- */
-public abstract class BKLogHandler implements AsyncCloseable, AsyncAbortable {
-static final Logger LOG = LoggerFactory.getLogger(BKLogHandler.class);
-
-protected final LogMetadata logMetadata;
-protected final DistributedLogConfiguration conf;
-protected final LogStreamMetadataStore streamMetadataStore;
-protected final LogSegmentMetadataStore metadataStore;
-protected final LogSegmentMetadataCache metadataCache;
-protected final LogSegmentEntryStore entryStore;
-protected final int firstNumEntriesPerReadLastRecordScan;
-protected final int maxNumEntriesPerReadLastRecordScan;
-protected volatile long 

[14/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/util/OrderedScheduler.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/util/OrderedScheduler.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/util/OrderedScheduler.java
deleted file mode 100644
index 287bd6d..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/util/OrderedScheduler.java
+++ /dev/null
@@ -1,490 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.util;
-
-import com.google.common.base.Objects;
-import com.google.common.util.concurrent.ThreadFactoryBuilder;
-import com.twitter.distributedlog.stats.BroadCastStatsLogger;
-import com.twitter.util.ExecutorServiceFuturePool;
-import com.twitter.util.FuturePool;
-import com.twitter.util.Time;
-import com.twitter.util.Timer;
-import com.twitter.util.TimerTask;
-import org.apache.bookkeeper.stats.NullStatsLogger;
-import org.apache.bookkeeper.stats.StatsLogger;
-import org.apache.bookkeeper.util.MathUtils;
-import scala.Function0;
-
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.List;
-import java.util.Random;
-import java.util.concurrent.Callable;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.ScheduledFuture;
-import java.util.concurrent.ThreadFactory;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
-
-/**
- * Ordered Scheduler. It is thread pool based {@link 
ScheduledExecutorService}, additionally providing
- * the ability to execute/schedule tasks by key. Hence the tasks 
submitted by same key
- * will be executed in order.
- * 
- * The scheduler is comprised of multiple {@link 
MonitoredScheduledThreadPoolExecutor}s. Each
- * {@link MonitoredScheduledThreadPoolExecutor} is a single thread executor. 
Normal task submissions will
- * be submitted to executors in a random manner to guarantee load balancing. 
Keyed task submissions (e.g
- * {@link OrderedScheduler#apply(Object, Function0)} will be submitted to a 
dedicated executor based on
- * the hash value of submit key.
- *
- * Metrics
- *
- * Per Executor Metrics
- *
- * Metrics about individual executors are exposed via {@link 
Builder#perExecutorStatsLogger}
- * under `scope`/`name`-executor-`id`-0. `name` is the scheduler name 
provided by {@link Builder#name}
- * while `id` is the index of this executor in the pool. And corresponding 
stats of future pool of
- * that executor are exposed under 
`scope`/`name`-executor-`id`-0/futurepool.
- * 
- * See {@link MonitoredScheduledThreadPoolExecutor} and {@link 
MonitoredFuturePool} for per executor metrics
- * exposed.
- *
- * Aggregated Metrics
- * 
- * task_pending_time: opstats. measuring the characteristics about the 
time that tasks spent on
- * waiting being executed.
- * task_execution_time: opstats. measuring the characteristics about the 
time that tasks spent on
- * executing.
- * futurepool/task_pending_time: opstats. measuring the characteristics 
about the time that tasks spent
- * on waiting in future pool being executed.
- * futurepool/task_execution_time: opstats. measuring the characteristics 
about the time that tasks spent
- * on executing.
- * futurepool/task_enqueue_time: opstats. measuring the characteristics 
about the time that tasks spent on
- * submitting to future pool.
- * futurepool/tasks_pending: gauge. how many tasks are pending in this 
future pool.
- * 
- */
-public class OrderedScheduler implements ScheduledExecutorService {
-
-/**
- * Create a builder to build scheduler.
- *
- * @return scheduler builder
- */
-public static Builder newBuilder() {
-return new Builder();
-}
-
-/**
- * Builder for {@link OrderedScheduler}.
- */
-public static class Builder {
-
-private String name = "OrderedScheduler";
-private int corePoolSize = -1;
- 

[24/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/BKNamespaceDriver.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/BKNamespaceDriver.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/BKNamespaceDriver.java
deleted file mode 100644
index 5921233..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/impl/BKNamespaceDriver.java
+++ /dev/null
@@ -1,631 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.impl;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Optional;
-import com.google.common.util.concurrent.ThreadFactoryBuilder;
-import com.twitter.distributedlog.BookKeeperClient;
-import com.twitter.distributedlog.BookKeeperClientBuilder;
-import com.twitter.distributedlog.DistributedLogConfiguration;
-import com.twitter.distributedlog.DistributedLogConstants;
-import com.twitter.distributedlog.MetadataAccessor;
-import com.twitter.distributedlog.ZooKeeperClient;
-import com.twitter.distributedlog.ZooKeeperClientBuilder;
-import com.twitter.distributedlog.acl.AccessControlManager;
-import com.twitter.distributedlog.acl.DefaultAccessControlManager;
-import com.twitter.distributedlog.impl.acl.ZKAccessControlManager;
-import com.twitter.distributedlog.bk.LedgerAllocator;
-import com.twitter.distributedlog.bk.LedgerAllocatorUtils;
-import com.twitter.distributedlog.config.DynamicDistributedLogConfiguration;
-import com.twitter.distributedlog.exceptions.AlreadyClosedException;
-import com.twitter.distributedlog.exceptions.InvalidStreamNameException;
-import com.twitter.distributedlog.impl.federated.FederatedZKLogMetadataStore;
-import com.twitter.distributedlog.impl.logsegment.BKLogSegmentEntryStore;
-import com.twitter.distributedlog.impl.metadata.ZKLogStreamMetadataStore;
-import com.twitter.distributedlog.impl.subscription.ZKSubscriptionsStore;
-import com.twitter.distributedlog.injector.AsyncFailureInjector;
-import com.twitter.distributedlog.logsegment.LogSegmentEntryStore;
-import com.twitter.distributedlog.impl.metadata.BKDLConfig;
-import com.twitter.distributedlog.metadata.LogMetadataForReader;
-import com.twitter.distributedlog.metadata.LogMetadataStore;
-import com.twitter.distributedlog.metadata.LogStreamMetadataStore;
-import com.twitter.distributedlog.namespace.NamespaceDriver;
-import com.twitter.distributedlog.namespace.NamespaceDriverManager;
-import com.twitter.distributedlog.subscription.SubscriptionsStore;
-import com.twitter.distributedlog.util.OrderedScheduler;
-import com.twitter.distributedlog.util.Utils;
-import org.apache.bookkeeper.feature.FeatureProvider;
-import org.apache.bookkeeper.stats.StatsLogger;
-import org.apache.bookkeeper.zookeeper.BoundExponentialBackoffRetryPolicy;
-import org.apache.bookkeeper.zookeeper.RetryPolicy;
-import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.ZooKeeper;
-import org.apache.zookeeper.common.PathUtils;
-import org.apache.zookeeper.data.Stat;
-import org.jboss.netty.channel.socket.ClientSocketChannelFactory;
-import org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory;
-import org.jboss.netty.util.HashedWheelTimer;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.net.URI;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.Executors;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-
-import static com.twitter.distributedlog.util.DLUtils.isReservedStreamName;
-import static com.twitter.distributedlog.util.DLUtils.validateName;
-
-/**
- * Manager for ZooKeeper/BookKeeper based namespace
- */
-public class BKNamespaceDriver implements NamespaceDriver {
-
-private static Logger LOG = 
LoggerFactory.getLogger(BKNamespaceDriver.class);
-
-// register itself
-static {
-
NamespaceDriverManager.registerDriver(DistributedLogConstants.BACKEND_BK, 
BKNamespaceDriver.class);
-   

[40/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-client/src/test/java/com/twitter/distributedlog/client/ownership/TestOwnershipCache.java
--
diff --git 
a/distributedlog-client/src/test/java/com/twitter/distributedlog/client/ownership/TestOwnershipCache.java
 
b/distributedlog-client/src/test/java/com/twitter/distributedlog/client/ownership/TestOwnershipCache.java
deleted file mode 100644
index c0e077b..000
--- 
a/distributedlog-client/src/test/java/com/twitter/distributedlog/client/ownership/TestOwnershipCache.java
+++ /dev/null
@@ -1,207 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.client.ownership;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNull;
-import static org.junit.Assert.assertTrue;
-
-import com.twitter.distributedlog.client.ClientConfig;
-import com.twitter.finagle.stats.NullStatsReceiver;
-import java.net.InetSocketAddress;
-import java.net.SocketAddress;
-import java.util.Map;
-import java.util.Set;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.TestName;
-
-/**
- * Test Case for Ownership Cache.
- */
-public class TestOwnershipCache {
-
-@Rule
-public TestName runtime = new TestName();
-
-private static OwnershipCache createOwnershipCache() {
-ClientConfig clientConfig = new ClientConfig();
-return new OwnershipCache(clientConfig, null,
-  NullStatsReceiver.get(), 
NullStatsReceiver.get());
-}
-
-private static SocketAddress createSocketAddress(int port) {
-return new InetSocketAddress("127.0.0.1", port);
-}
-
-@Test(timeout = 6)
-public void testUpdateOwner() {
-OwnershipCache cache = createOwnershipCache();
-SocketAddress addr = createSocketAddress(1000);
-String stream = runtime.getMethodName();
-
-assertTrue("Should successfully update owner if no owner exists 
before",
-cache.updateOwner(stream, addr));
-assertEquals("Owner should be " + addr + " for stream " + stream,
-addr, cache.getOwner(stream));
-assertTrue("Should successfully update owner if old owner is same",
-cache.updateOwner(stream, addr));
-assertEquals("Owner should be " + addr + " for stream " + stream,
-addr, cache.getOwner(stream));
-}
-
-@Test(timeout = 6)
-public void testRemoveOwnerFromStream() {
-OwnershipCache cache = createOwnershipCache();
-int initialPort = 2000;
-int numProxies = 2;
-int numStreamsPerProxy = 2;
-for (int i = 0; i < numProxies; i++) {
-SocketAddress addr = createSocketAddress(initialPort + i);
-for (int j = 0; j < numStreamsPerProxy; j++) {
-String stream = runtime.getMethodName() + "_" + i + "_" + j;
-cache.updateOwner(stream, addr);
-}
-}
-Map ownershipMap = 
cache.getStreamOwnerMapping();
-assertEquals("There should be " + (numProxies * numStreamsPerProxy) + 
" entries in cache",
-numProxies * numStreamsPerProxy, ownershipMap.size());
-Map ownershipDistribution = 
cache.getStreamOwnershipDistribution();
-assertEquals("There should be " + numProxies + " proxies cached",
-numProxies, ownershipDistribution.size());
-
-String stream = runtime.getMethodName() + "_0_0";
-SocketAddress owner = createSocketAddress(initialPort);
-
-// remove non-existent mapping won't change anything
-SocketAddress nonExistentAddr = createSocketAddress(initialPort + 999);
-cache.removeOwnerFromStream(stream, nonExistentAddr, 
"remove-non-existent-addr");
-assertEquals("Owner " + owner + " should not be removed",
-owner, cache.getOwner(stream));
-ownershipMap = cache.getStreamOwnerMapping();
-assertEquals("There should be " + (numProxies * numStreamsPerProxy) + 
" 

[42/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-client/src/main/java/org/apache/distributedlog/client/routing/ConsistentHashRoutingService.java
--
diff --git 
a/distributedlog-client/src/main/java/org/apache/distributedlog/client/routing/ConsistentHashRoutingService.java
 
b/distributedlog-client/src/main/java/org/apache/distributedlog/client/routing/ConsistentHashRoutingService.java
new file mode 100644
index 000..666fa31
--- /dev/null
+++ 
b/distributedlog-client/src/main/java/org/apache/distributedlog/client/routing/ConsistentHashRoutingService.java
@@ -0,0 +1,500 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog.client.routing;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.common.base.Optional;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.MapDifference;
+import com.google.common.collect.Maps;
+import com.google.common.hash.HashFunction;
+import com.google.common.hash.Hashing;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import com.twitter.common.zookeeper.ServerSet;
+import org.apache.distributedlog.service.DLSocketAddress;
+import com.twitter.finagle.ChannelException;
+import com.twitter.finagle.NoBrokersAvailableException;
+import com.twitter.finagle.stats.Counter;
+import com.twitter.finagle.stats.Gauge;
+import com.twitter.finagle.stats.NullStatsReceiver;
+import com.twitter.finagle.stats.StatsReceiver;
+import com.twitter.util.Function0;
+import java.net.SocketAddress;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.lang3.tuple.Pair;
+import org.jboss.netty.util.HashedWheelTimer;
+import org.jboss.netty.util.Timeout;
+import org.jboss.netty.util.TimerTask;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.collection.Seq;
+
+/**
+ * Consistent Hashing Based {@link RoutingService}.
+ */
+public class ConsistentHashRoutingService extends ServerSetRoutingService {
+
+private static final Logger logger = 
LoggerFactory.getLogger(ConsistentHashRoutingService.class);
+
+@Deprecated
+public static ConsistentHashRoutingService of(ServerSetWatcher 
serverSetWatcher, int numReplicas) {
+return new ConsistentHashRoutingService(serverSetWatcher, numReplicas, 
300, NullStatsReceiver.get());
+}
+
+/**
+ * Builder helper class to build a consistent hash bashed {@link 
RoutingService}.
+ *
+ * @return builder to build a consistent hash based {@link RoutingService}.
+ */
+public static Builder newBuilder() {
+return new Builder();
+}
+
+/**
+ * Builder for building consistent hash based routing service.
+ */
+public static class Builder implements RoutingService.Builder {
+
+private ServerSet serverSet;
+private boolean resolveFromName = false;
+private int numReplicas;
+private int blackoutSeconds = 300;
+private StatsReceiver statsReceiver = NullStatsReceiver.get();
+
+private Builder() {}
+
+public Builder serverSet(ServerSet serverSet) {
+this.serverSet = serverSet;
+return this;
+}
+
+public Builder resolveFromName(boolean enabled) {
+this.resolveFromName = enabled;
+return this;
+}
+
+public Builder numReplicas(int numReplicas) {
+this.numReplicas = numReplicas;
+return this;
+}
+
+public Builder blackoutSeconds(int seconds) {
+this.blackoutSeconds = seconds;
+return this;
+}
+
+public Builder statsReceiver(StatsReceiver statsReceiver) {
+this.statsReceiver = statsReceiver;
+return this;
+}
+
+@Override
+

[19/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/lock/ZKSessionLockFactory.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/lock/ZKSessionLockFactory.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/lock/ZKSessionLockFactory.java
deleted file mode 100644
index bb98e07..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/lock/ZKSessionLockFactory.java
+++ /dev/null
@@ -1,133 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.lock;
-
-import com.twitter.distributedlog.ZooKeeperClient;
-import com.twitter.distributedlog.exceptions.DLInterruptedException;
-import com.twitter.distributedlog.util.OrderedScheduler;
-import com.twitter.util.Future;
-import com.twitter.util.Promise;
-import com.twitter.util.Return;
-import com.twitter.util.Throw;
-import org.apache.bookkeeper.stats.StatsLogger;
-import scala.runtime.BoxedUnit;
-
-import java.io.IOException;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicInteger;
-import java.util.concurrent.atomic.AtomicReference;
-
-/**
- * Factory to create zookeeper based locks.
- */
-public class ZKSessionLockFactory implements SessionLockFactory {
-
-private final ZooKeeperClient zkc;
-private final String clientId;
-private final OrderedScheduler lockStateExecutor;
-private final long lockOpTimeout;
-private final int lockCreationRetries;
-private final long zkRetryBackoffMs;
-
-// Stats
-private final StatsLogger lockStatsLogger;
-
-public ZKSessionLockFactory(ZooKeeperClient zkc,
-String clientId,
-OrderedScheduler lockStateExecutor,
-int lockCreationRetries,
-long lockOpTimeout,
-long zkRetryBackoffMs,
-StatsLogger statsLogger) {
-this.zkc = zkc;
-this.clientId = clientId;
-this.lockStateExecutor = lockStateExecutor;
-this.lockCreationRetries = lockCreationRetries;
-this.lockOpTimeout = lockOpTimeout;
-this.zkRetryBackoffMs = zkRetryBackoffMs;
-
-this.lockStatsLogger = statsLogger.scope("lock");
-}
-
-@Override
-public Future createLock(String lockPath,
-  DistributedLockContext context) {
-AtomicInteger numRetries = new AtomicInteger(lockCreationRetries);
-final AtomicReference interruptedException = new 
AtomicReference(null);
-Promise createPromise =
-new Promise(new 
com.twitter.util.Function() {
-@Override
-public BoxedUnit apply(Throwable t) {
-interruptedException.set(t);
-return BoxedUnit.UNIT;
-}
-});
-createLock(
-lockPath,
-context,
-interruptedException,
-numRetries,
-createPromise,
-0L);
-return createPromise;
-}
-
-void createLock(final String lockPath,
-final DistributedLockContext context,
-final AtomicReference interruptedException,
-final AtomicInteger numRetries,
-final Promise createPromise,
-final long delayMs) {
-lockStateExecutor.schedule(lockPath, new Runnable() {
-@Override
-public void run() {
-if (null != interruptedException.get()) {
-createPromise.updateIfEmpty(new 
Throw(interruptedException.get()));
-return;
-}
-try {
-SessionLock lock = new ZKSessionLock(
-zkc,
-lockPath,
-clientId,
-lockStateExecutor,
-lockOpTimeout,

[38/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-client/src/test/java/org/apache/distributedlog/client/routing/TestRoutingService.java
--
diff --git 
a/distributedlog-client/src/test/java/org/apache/distributedlog/client/routing/TestRoutingService.java
 
b/distributedlog-client/src/test/java/org/apache/distributedlog/client/routing/TestRoutingService.java
new file mode 100644
index 000..d2d61a9
--- /dev/null
+++ 
b/distributedlog-client/src/test/java/org/apache/distributedlog/client/routing/TestRoutingService.java
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog.client.routing;
+
+import static org.junit.Assert.assertEquals;
+
+import org.apache.distributedlog.client.resolver.DefaultRegionResolver;
+import com.twitter.finagle.Address;
+import com.twitter.finagle.Addresses;
+import com.twitter.finagle.addr.WeightedAddress;
+import java.net.InetSocketAddress;
+import java.net.SocketAddress;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test Case for {@link RoutingService}.
+ */
+@RunWith(Parameterized.class)
+public class TestRoutingService {
+
+static final Logger LOG = 
LoggerFactory.getLogger(TestRoutingService.class);
+
+@Parameterized.Parameters
+public static Collection configs() {
+ArrayList list = new ArrayList();
+for (int i = 0; i <= 1; i++) {
+for (int j = 0; j <= 1; j++) {
+for (int k = 0; k <= 1; k++) {
+list.add(new Boolean[] {i == 1, j == 1, k == 1});
+}
+}
+}
+return list;
+}
+
+private final boolean consistentHash;
+private final boolean weightedAddresses;
+private final boolean asyncResolution;
+
+public TestRoutingService(boolean consistentHash, boolean 
weightedAddresses, boolean asyncResolution) {
+this.consistentHash = consistentHash;
+this.weightedAddresses = weightedAddresses;
+this.asyncResolution = asyncResolution;
+}
+
+private List getAddresses(boolean weightedAddresses) {
+ArrayList addresses = new ArrayList();
+addresses.add(Addresses.newInetAddress(new 
InetSocketAddress("127.0.0.1", 3181)));
+addresses.add(Addresses.newInetAddress(new 
InetSocketAddress("127.0.0.2", 3181)));
+addresses.add(Addresses.newInetAddress(new 
InetSocketAddress("127.0.0.3", 3181)));
+addresses.add(Addresses.newInetAddress(new 
InetSocketAddress("127.0.0.4", 3181)));
+addresses.add(Addresses.newInetAddress(new 
InetSocketAddress("127.0.0.5", 3181)));
+addresses.add(Addresses.newInetAddress(new 
InetSocketAddress("127.0.0.6", 3181)));
+addresses.add(Addresses.newInetAddress(new 
InetSocketAddress("127.0.0.7", 3181)));
+
+if (weightedAddresses) {
+ArrayList wAddresses = new ArrayList();
+for (Address address: addresses) {
+wAddresses.add(WeightedAddress.apply(address, 1.0));
+}
+return wAddresses;
+} else {
+return addresses;
+}
+}
+
+private void testRoutingServiceHelper(boolean consistentHash,
+  boolean weightedAddresses,
+  boolean asyncResolution)
+throws Exception {
+ExecutorService executorService = null;
+final List addresses = getAddresses(weightedAddresses);
+final TestName name = new TestName();
+RoutingService routingService;
+if (consistentHash) {
+routingService = ConsistentHashRoutingService.newBuilder()
+.serverSet(new NameServerSet(name))
+.resolveFromName(true)
+.numReplicas(997)
+

[31/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/DistributedLogConfiguration.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/DistributedLogConfiguration.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/DistributedLogConfiguration.java
deleted file mode 100644
index 6da4b8d..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/DistributedLogConfiguration.java
+++ /dev/null
@@ -1,3528 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog;
-
-import com.google.common.base.Optional;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Sets;
-import com.twitter.distributedlog.bk.QuorumConfig;
-import com.twitter.distributedlog.feature.DefaultFeatureProvider;
-import com.twitter.distributedlog.namespace.DistributedLogNamespaceBuilder;
-import com.twitter.distributedlog.net.DNSResolverForRacks;
-import com.twitter.distributedlog.net.DNSResolverForRows;
-import org.apache.bookkeeper.conf.ClientConfiguration;
-import org.apache.bookkeeper.feature.FeatureProvider;
-import org.apache.bookkeeper.net.DNSToSwitchMapping;
-import org.apache.bookkeeper.stats.StatsLogger;
-import org.apache.bookkeeper.util.ReflectionUtils;
-import org.apache.commons.configuration.CompositeConfiguration;
-import org.apache.commons.configuration.Configuration;
-import org.apache.commons.configuration.ConfigurationException;
-import org.apache.commons.configuration.PropertiesConfiguration;
-import org.apache.commons.configuration.SystemConfiguration;
-import org.apache.commons.lang.StringUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.net.URL;
-import java.util.ArrayList;
-import java.util.Iterator;
-import java.util.Set;
-
-/**
- * DistributedLog Configuration.
- * 
- * DistributedLog configuration is basically a properties based configuration, 
which extends from
- * Apache commons {@link CompositeConfiguration}. All the DL settings are in 
camel case and prefixed
- * with a meaningful component name. for example, `zkSessionTimeoutSeconds` 
means SessionTimeoutSeconds
- * for component `zk`.
- *
- * BookKeeper Configuration
- *
- * BookKeeper client configuration settings could be loaded via DistributedLog 
configuration. All those
- * settings are prefixed with `bkc.`. For example, bkc.zkTimeout 
in distributedlog configuration
- * will be applied as `zkTimeout` in bookkeeper client configuration.
- *
- * How to load configuration
- *
- * The default distributedlog configuration is constructed by instantiated a 
new instance. This
- * distributedlog configuration will automatically load the settings that 
specified via
- * {@link SystemConfiguration}.
- *
- * 
- *  DistributedLogConfiguration conf = new DistributedLogConfiguration();
- * 
- *
- * The recommended way is to load configuration from URL that points to a 
configuration file
- * ({@link #loadConf(URL)}).
- *
- * 
- *  String configFile = "/path/to/distributedlog/conf/file";
- *  DistributedLogConfiguration conf = new DistributedLogConfiguration();
- *  conf.loadConf(new File(configFile).toURI().toURL());
- * 
- *
- * @see org.apache.bookkeeper.conf.ClientConfiguration
- */
-public class DistributedLogConfiguration extends CompositeConfiguration {
-static final Logger LOG = 
LoggerFactory.getLogger(DistributedLogConfiguration.class);
-
-private static ClassLoader defaultLoader;
-
-static {
-defaultLoader = Thread.currentThread().getContextClassLoader();
-if (null == defaultLoader) {
-defaultLoader = DistributedLogConfiguration.class.getClassLoader();
-}
-}
-
-//
-// ZooKeeper Related Settings
-//
-
-public static final String BKDL_ZK_ACL_ID = "zkAclId";
-public static final String BKDL_ZK_ACL_ID_DEFAULT = null;
-public static final String BKDL_ZK_SESSION_TIMEOUT_SECONDS = 
"zkSessionTimeoutSeconds";
-public static final int BKDL_ZK_SESSION_TIMEOUT_SECONDS_DEFAULT = 30;
-public static final 

[25/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/bk/package-info.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/bk/package-info.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/bk/package-info.java
deleted file mode 100644
index 697d2e9..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/bk/package-info.java
+++ /dev/null
@@ -1,24 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-/**
- * BookKeeper related util functions.
- * 
- * Ledger Allocator
- *
- */
-package com.twitter.distributedlog.bk;

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/callback/LogSegmentListener.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/callback/LogSegmentListener.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/callback/LogSegmentListener.java
deleted file mode 100644
index 2196245..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/callback/LogSegmentListener.java
+++ /dev/null
@@ -1,42 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.callback;
-
-import com.twitter.distributedlog.LogSegmentMetadata;
-
-import java.util.List;
-
-/**
- * Listener on log segments changes for a given stream used by {@link 
com.twitter.distributedlog.BKLogReadHandler}
- */
-public interface LogSegmentListener {
-
-/**
- * Notified when segments updated. The new sorted log segments
- * list is returned in this method.
- *
- * @param segments
- *  updated list of segments.
- */
-void onSegmentsUpdated(List segments);
-
-/**
- * Notified when the log stream is deleted.
- */
-void onLogStreamDeleted();
-}

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/callback/LogSegmentNamesListener.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/callback/LogSegmentNamesListener.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/callback/LogSegmentNamesListener.java
deleted file mode 100644
index e38f305..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/callback/LogSegmentNamesListener.java
+++ /dev/null
@@ -1,42 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See 

[34/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
deleted file mode 100644
index df64505..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
+++ /dev/null
@@ -1,1348 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.concurrent.Callable;
-import java.util.concurrent.ScheduledFuture;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicInteger;
-import java.util.concurrent.atomic.AtomicReference;
-import java.util.concurrent.locks.ReentrantLock;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Stopwatch;
-import com.twitter.distributedlog.config.DynamicDistributedLogConfiguration;
-import com.twitter.distributedlog.exceptions.BKTransmitException;
-import com.twitter.distributedlog.exceptions.EndOfStreamException;
-import com.twitter.distributedlog.exceptions.FlushException;
-import com.twitter.distributedlog.exceptions.LockingException;
-import com.twitter.distributedlog.exceptions.LogRecordTooLongException;
-import com.twitter.distributedlog.exceptions.TransactionIdOutOfOrderException;
-import com.twitter.distributedlog.exceptions.WriteCancelledException;
-import com.twitter.distributedlog.exceptions.WriteException;
-import com.twitter.distributedlog.exceptions.InvalidEnvelopedEntryException;
-import com.twitter.distributedlog.feature.CoreFeatureKeys;
-import com.twitter.distributedlog.injector.FailureInjector;
-import com.twitter.distributedlog.injector.RandomDelayFailureInjector;
-import com.twitter.distributedlog.io.Buffer;
-import com.twitter.distributedlog.io.CompressionCodec;
-import com.twitter.distributedlog.io.CompressionUtils;
-import com.twitter.distributedlog.lock.DistributedLock;
-import com.twitter.distributedlog.logsegment.LogSegmentEntryWriter;
-import com.twitter.distributedlog.logsegment.LogSegmentWriter;
-import com.twitter.distributedlog.stats.BroadCastStatsLogger;
-import com.twitter.distributedlog.stats.OpStatsListener;
-import com.twitter.distributedlog.util.FailpointUtils;
-import com.twitter.distributedlog.util.FutureUtils;
-import com.twitter.distributedlog.util.OrderedScheduler;
-import com.twitter.distributedlog.util.PermitLimiter;
-import com.twitter.distributedlog.util.SafeQueueingFuturePool;
-import com.twitter.distributedlog.util.SimplePermitLimiter;
-import com.twitter.distributedlog.util.Sizable;
-import com.twitter.util.Function0;
-import com.twitter.util.Future;
-import com.twitter.util.FutureEventListener;
-import com.twitter.util.FuturePool;
-import com.twitter.util.Promise;
-import org.apache.bookkeeper.client.AsyncCallback.AddCallback;
-import org.apache.bookkeeper.client.AsyncCallback.CloseCallback;
-import org.apache.bookkeeper.client.BKException;
-import org.apache.bookkeeper.client.LedgerHandle;
-import org.apache.bookkeeper.feature.Feature;
-import org.apache.bookkeeper.feature.FeatureProvider;
-import org.apache.bookkeeper.stats.AlertStatsLogger;
-import org.apache.bookkeeper.stats.Counter;
-import org.apache.bookkeeper.stats.Gauge;
-import org.apache.bookkeeper.stats.OpStatsLogger;
-import org.apache.bookkeeper.stats.StatsLogger;
-import org.apache.bookkeeper.util.MathUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import scala.runtime.AbstractFunction1;
-import scala.runtime.BoxedUnit;
-
-import static com.google.common.base.Charsets.UTF_8;
-import static com.twitter.distributedlog.LogRecord.MAX_LOGRECORD_SIZE;
-import static com.twitter.distributedlog.LogRecord.MAX_LOGRECORDSET_SIZE;
-
-/**
- * BookKeeper Based Log Segment Writer.
- *
- * Multiple log records are packed into a single bookkeeper
- * entry before sending it over the network. The fact that the log record 
entries
- * are 

[09/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/org/apache/distributedlog/BKLogSegmentWriter.java
--
diff --git 
a/distributedlog-core/src/main/java/org/apache/distributedlog/BKLogSegmentWriter.java
 
b/distributedlog-core/src/main/java/org/apache/distributedlog/BKLogSegmentWriter.java
new file mode 100644
index 000..6b60c77
--- /dev/null
+++ 
b/distributedlog-core/src/main/java/org/apache/distributedlog/BKLogSegmentWriter.java
@@ -0,0 +1,1348 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.concurrent.locks.ReentrantLock;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Stopwatch;
+import org.apache.distributedlog.config.DynamicDistributedLogConfiguration;
+import org.apache.distributedlog.exceptions.BKTransmitException;
+import org.apache.distributedlog.exceptions.EndOfStreamException;
+import org.apache.distributedlog.exceptions.FlushException;
+import org.apache.distributedlog.exceptions.LockingException;
+import org.apache.distributedlog.exceptions.LogRecordTooLongException;
+import org.apache.distributedlog.exceptions.TransactionIdOutOfOrderException;
+import org.apache.distributedlog.exceptions.WriteCancelledException;
+import org.apache.distributedlog.exceptions.WriteException;
+import org.apache.distributedlog.exceptions.InvalidEnvelopedEntryException;
+import org.apache.distributedlog.feature.CoreFeatureKeys;
+import org.apache.distributedlog.injector.FailureInjector;
+import org.apache.distributedlog.injector.RandomDelayFailureInjector;
+import org.apache.distributedlog.io.Buffer;
+import org.apache.distributedlog.io.CompressionCodec;
+import org.apache.distributedlog.io.CompressionUtils;
+import org.apache.distributedlog.lock.DistributedLock;
+import org.apache.distributedlog.logsegment.LogSegmentEntryWriter;
+import org.apache.distributedlog.logsegment.LogSegmentWriter;
+import org.apache.distributedlog.stats.BroadCastStatsLogger;
+import org.apache.distributedlog.stats.OpStatsListener;
+import org.apache.distributedlog.util.FailpointUtils;
+import org.apache.distributedlog.util.FutureUtils;
+import org.apache.distributedlog.util.OrderedScheduler;
+import org.apache.distributedlog.util.PermitLimiter;
+import org.apache.distributedlog.util.SafeQueueingFuturePool;
+import org.apache.distributedlog.util.SimplePermitLimiter;
+import org.apache.distributedlog.util.Sizable;
+import com.twitter.util.Function0;
+import com.twitter.util.Future;
+import com.twitter.util.FutureEventListener;
+import com.twitter.util.FuturePool;
+import com.twitter.util.Promise;
+import org.apache.bookkeeper.client.AsyncCallback.AddCallback;
+import org.apache.bookkeeper.client.AsyncCallback.CloseCallback;
+import org.apache.bookkeeper.client.BKException;
+import org.apache.bookkeeper.client.LedgerHandle;
+import org.apache.bookkeeper.feature.Feature;
+import org.apache.bookkeeper.feature.FeatureProvider;
+import org.apache.bookkeeper.stats.AlertStatsLogger;
+import org.apache.bookkeeper.stats.Counter;
+import org.apache.bookkeeper.stats.Gauge;
+import org.apache.bookkeeper.stats.OpStatsLogger;
+import org.apache.bookkeeper.stats.StatsLogger;
+import org.apache.bookkeeper.util.MathUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.runtime.AbstractFunction1;
+import scala.runtime.BoxedUnit;
+
+import static com.google.common.base.Charsets.UTF_8;
+import static org.apache.distributedlog.LogRecord.MAX_LOGRECORD_SIZE;
+import static org.apache.distributedlog.LogRecord.MAX_LOGRECORDSET_SIZE;
+
+/**
+ * BookKeeper Based Log Segment Writer.
+ *
+ * Multiple log records are packed into a single bookkeeper
+ * entry before sending it over the network. The fact that the log record 
entries
+ * are complete in the bookkeeper entries 

[18/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/metadata/LogSegmentMetadataStoreUpdater.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/metadata/LogSegmentMetadataStoreUpdater.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/metadata/LogSegmentMetadataStoreUpdater.java
deleted file mode 100644
index 0e5e6d4..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/metadata/LogSegmentMetadataStoreUpdater.java
+++ /dev/null
@@ -1,184 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.metadata;
-
-import com.google.common.base.Preconditions;
-import com.twitter.distributedlog.DLSN;
-import com.twitter.distributedlog.DistributedLogConfiguration;
-import com.twitter.distributedlog.LogRecordWithDLSN;
-import com.twitter.distributedlog.LogSegmentMetadata;
-import com.twitter.distributedlog.logsegment.LogSegmentMetadataStore;
-import com.twitter.distributedlog.util.Transaction;
-import com.twitter.util.Future;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import scala.runtime.AbstractFunction1;
-
-public class LogSegmentMetadataStoreUpdater implements MetadataUpdater {
-
-static final Logger LOG = 
LoggerFactory.getLogger(LogSegmentMetadataStoreUpdater.class);
-
-public static MetadataUpdater 
createMetadataUpdater(DistributedLogConfiguration conf,
-
LogSegmentMetadataStore metadataStore) {
-return new LogSegmentMetadataStoreUpdater(conf, metadataStore);
-}
-
-protected final LogSegmentMetadataStore metadataStore;
-protected final LogSegmentMetadata.LogSegmentMetadataVersion 
metadataVersion;
-
-protected LogSegmentMetadataStoreUpdater(DistributedLogConfiguration conf,
- LogSegmentMetadataStore 
metadataStore) {
-this.metadataStore = metadataStore;
-this.metadataVersion = 
LogSegmentMetadata.LogSegmentMetadataVersion.of(conf.getDLLedgerMetadataLayoutVersion());
-}
-
-private String formatLogSegmentSequenceNumber(long logSegmentSeqNo) {
-return String.format("%018d", logSegmentSeqNo);
-}
-
-@Override
-public Transaction transaction() {
-return metadataStore.transaction();
-}
-
-@Override
-public Future updateLastRecord(LogSegmentMetadata 
segment,
-   LogRecordWithDLSN 
record) {
-DLSN dlsn = record.getDlsn();
-Preconditions.checkState(!segment.isInProgress(),
-"Updating last dlsn for an inprogress log segment isn't 
supported.");
-Preconditions.checkArgument(segment.isDLSNinThisSegment(dlsn),
-"DLSN " + dlsn + " doesn't belong to segment " + segment);
-final LogSegmentMetadata newSegment = segment.mutator()
-.setLastDLSN(dlsn)
-.setLastTxId(record.getTransactionId())
-.setRecordCount(record)
-.build();
-return updateSegmentMetadata(newSegment);
-}
-
-@Override
-public Future changeSequenceNumber(LogSegmentMetadata 
segment,
-   long 
logSegmentSeqNo) {
-String newZkPath = segment.getZkPath()
-
.replace(formatLogSegmentSequenceNumber(segment.getLogSegmentSequenceNumber()),
-formatLogSegmentSequenceNumber(logSegmentSeqNo));
-final LogSegmentMetadata newSegment = segment.mutator()
-.setLogSegmentSequenceNumber(logSegmentSeqNo)
-.setZkPath(newZkPath)
-.build();
-return addNewSegmentAndDeleteOldSegment(newSegment, segment);
-}
-
-/**
- * Change the truncation status of a log segment to be active
- *
- * @param segment log segment to change truncation status to active.
- * @return new log segment
- */
-@Override
-public Future setLogSegmentActive(LogSegmentMetadata 
segment) {
-

[01/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

Repository: incubator-distributedlog
Updated Branches:
  refs/heads/master 7fab246d9 -> b44820b50


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/org/apache/distributedlog/auditor/DLAuditor.java
--
diff --git 
a/distributedlog-core/src/main/java/org/apache/distributedlog/auditor/DLAuditor.java
 
b/distributedlog-core/src/main/java/org/apache/distributedlog/auditor/DLAuditor.java
new file mode 100644
index 000..56a4f2e
--- /dev/null
+++ 
b/distributedlog-core/src/main/java/org/apache/distributedlog/auditor/DLAuditor.java
@@ -0,0 +1,630 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog.auditor;
+
+import com.google.common.base.Objects;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import com.google.common.util.concurrent.SettableFuture;
+import org.apache.distributedlog.BookKeeperClient;
+import org.apache.distributedlog.BookKeeperClientBuilder;
+import org.apache.distributedlog.DistributedLogConfiguration;
+import org.apache.distributedlog.DistributedLogManager;
+import org.apache.distributedlog.LogSegmentMetadata;
+import org.apache.distributedlog.impl.BKNamespaceDriver;
+import org.apache.distributedlog.namespace.DistributedLogNamespace;
+import org.apache.distributedlog.ZooKeeperClient;
+import org.apache.distributedlog.ZooKeeperClientBuilder;
+import org.apache.distributedlog.exceptions.DLInterruptedException;
+import org.apache.distributedlog.exceptions.ZKException;
+import org.apache.distributedlog.impl.metadata.BKDLConfig;
+import org.apache.distributedlog.namespace.DistributedLogNamespaceBuilder;
+import org.apache.distributedlog.namespace.NamespaceDriver;
+import org.apache.distributedlog.util.DLUtils;
+import org.apache.bookkeeper.client.BKException;
+import org.apache.bookkeeper.client.BookKeeper;
+import org.apache.bookkeeper.client.BookKeeperAccessor;
+import org.apache.bookkeeper.client.LedgerHandle;
+import org.apache.bookkeeper.meta.LedgerManager;
+import org.apache.bookkeeper.proto.BookkeeperInternalCallbacks;
+import org.apache.bookkeeper.zookeeper.BoundExponentialBackoffRetryPolicy;
+import org.apache.bookkeeper.zookeeper.RetryPolicy;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.zookeeper.AsyncCallback;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.data.Stat;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static com.google.common.base.Charsets.UTF_8;
+
+/**
+ * DL Auditor will audit DL namespace, e.g. find leaked ledger, report disk 
usage by streams.
+ */
+public class DLAuditor {
+
+private static final Logger logger = 
LoggerFactory.getLogger(DLAuditor.class);
+
+private final DistributedLogConfiguration conf;
+
+public DLAuditor(DistributedLogConfiguration conf) {
+this.conf = conf;
+}
+
+private ZooKeeperClient getZooKeeperClient(DistributedLogNamespace 
namespace) {
+NamespaceDriver driver = namespace.getNamespaceDriver();
+assert(driver instanceof BKNamespaceDriver);
+return ((BKNamespaceDriver) driver).getWriterZKC();
+}
+
+private BookKeeperClient getBookKeeperClient(DistributedLogNamespace 
namespace) {
+NamespaceDriver driver = namespace.getNamespaceDriver();
+assert(driver instanceof BKNamespaceDriver);
+return ((BKNamespaceDriver) driver).getReaderBKC();
+}
+
+private String validateAndGetZKServers(List uris) {
+URI firstURI = 

[33/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogWriteHandler.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogWriteHandler.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogWriteHandler.java
deleted file mode 100644
index 2486297..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogWriteHandler.java
+++ /dev/null
@@ -1,1325 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Stopwatch;
-import com.google.common.collect.Lists;
-import com.twitter.distributedlog.config.DynamicDistributedLogConfiguration;
-import com.twitter.distributedlog.exceptions.DLIllegalStateException;
-import com.twitter.distributedlog.exceptions.EndOfStreamException;
-import com.twitter.distributedlog.exceptions.LockingException;
-import com.twitter.distributedlog.exceptions.LogSegmentNotFoundException;
-import com.twitter.distributedlog.exceptions.TransactionIdOutOfOrderException;
-import com.twitter.distributedlog.exceptions.UnexpectedException;
-import com.twitter.distributedlog.function.GetLastTxIdFunction;
-import com.twitter.distributedlog.logsegment.LogSegmentEntryStore;
-import com.twitter.distributedlog.logsegment.LogSegmentEntryWriter;
-import com.twitter.distributedlog.metadata.LogMetadataForWriter;
-import com.twitter.distributedlog.lock.DistributedLock;
-import com.twitter.distributedlog.logsegment.LogSegmentFilter;
-import com.twitter.distributedlog.logsegment.LogSegmentMetadataCache;
-import com.twitter.distributedlog.logsegment.RollingPolicy;
-import com.twitter.distributedlog.logsegment.SizeBasedRollingPolicy;
-import com.twitter.distributedlog.logsegment.TimeBasedRollingPolicy;
-import com.twitter.distributedlog.metadata.LogStreamMetadataStore;
-import com.twitter.distributedlog.metadata.MetadataUpdater;
-import com.twitter.distributedlog.metadata.LogSegmentMetadataStoreUpdater;
-import com.twitter.distributedlog.util.Allocator;
-import com.twitter.distributedlog.util.DLUtils;
-import com.twitter.distributedlog.util.FailpointUtils;
-import com.twitter.distributedlog.util.FutureUtils;
-import com.twitter.distributedlog.util.FutureUtils.FutureEventListenerRunnable;
-import com.twitter.distributedlog.util.OrderedScheduler;
-import com.twitter.distributedlog.util.Transaction;
-import com.twitter.distributedlog.util.PermitLimiter;
-import com.twitter.distributedlog.util.Utils;
-import com.twitter.util.Function;
-import com.twitter.util.Future;
-import com.twitter.util.FutureEventListener;
-import com.twitter.util.Promise;
-import org.apache.bookkeeper.feature.FeatureProvider;
-import org.apache.bookkeeper.stats.AlertStatsLogger;
-import org.apache.bookkeeper.stats.OpStatsLogger;
-import org.apache.bookkeeper.stats.StatsLogger;
-import org.apache.bookkeeper.versioning.Version;
-import org.apache.bookkeeper.versioning.Versioned;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import scala.runtime.AbstractFunction1;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Comparator;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.concurrent.TimeUnit;
-
-import static 
com.twitter.distributedlog.impl.ZKLogSegmentFilters.WRITE_HANDLE_FILTER;
-
-/**
- * Log Handler for Writers.
- *
- * Metrics
- * All the metrics about log write handler are exposed under scope `segments`.
- * 
- *  `segments`/open : opstats. latency characteristics on starting a new 
log segment.
- *  `segments`/close : opstats. latency characteristics on completing an 
inprogress log segment.
- *  `segments`/recover : opstats. latency characteristics on recovering a 
log segment.
- *  `segments`/delete : opstats. latency characteristics on deleting a log 
segment.
- * 
- */
-class BKLogWriteHandler extends BKLogHandler {
-static final Logger LOG = LoggerFactory.getLogger(BKLogReadHandler.class);
-
-private static Transaction.OpListener 
NULL_OP_LISTENER 

[43/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-client/src/main/java/org/apache/distributedlog/client/DistributedLogMultiStreamWriter.java
--
diff --git 
a/distributedlog-client/src/main/java/org/apache/distributedlog/client/DistributedLogMultiStreamWriter.java
 
b/distributedlog-client/src/main/java/org/apache/distributedlog/client/DistributedLogMultiStreamWriter.java
new file mode 100644
index 000..b3f3368
--- /dev/null
+++ 
b/distributedlog-client/src/main/java/org/apache/distributedlog/client/DistributedLogMultiStreamWriter.java
@@ -0,0 +1,486 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog.client;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.distributedlog.LogRecord.MAX_LOGRECORDSET_SIZE;
+import static org.apache.distributedlog.LogRecord.MAX_LOGRECORD_SIZE;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.base.Ticker;
+import com.google.common.collect.Lists;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.distributedlog.DLSN;
+import org.apache.distributedlog.LogRecordSet;
+import org.apache.distributedlog.LogRecordSetBuffer;
+import 
org.apache.distributedlog.client.speculative.DefaultSpeculativeRequestExecutionPolicy;
+import 
org.apache.distributedlog.client.speculative.SpeculativeRequestExecutionPolicy;
+import org.apache.distributedlog.client.speculative.SpeculativeRequestExecutor;
+import org.apache.distributedlog.exceptions.LogRecordTooLongException;
+import org.apache.distributedlog.exceptions.WriteException;
+import org.apache.distributedlog.io.CompressionCodec;
+import org.apache.distributedlog.service.DistributedLogClient;
+import com.twitter.finagle.IndividualRequestTimeoutException;
+import com.twitter.util.Duration;
+import com.twitter.util.Future;
+import com.twitter.util.FutureEventListener;
+import com.twitter.util.Promise;
+import java.nio.ByteBuffer;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Write to multiple streams.
+ */
+public class DistributedLogMultiStreamWriter implements Runnable {
+
+/**
+ * Create a new builder to create a multi stream writer.
+ *
+ * @return a new builder to create a multi stream writer.
+ */
+public static Builder newBuilder() {
+return new Builder();
+}
+
+/**
+ * Builder for the multi stream writer.
+ */
+public static class Builder {
+
+private DistributedLogClient client = null;
+private List streams = null;
+private int bufferSize = 16 * 1024; // 16k
+private long flushIntervalMicros = 2000; // 2ms
+private CompressionCodec.Type codec = CompressionCodec.Type.NONE;
+private ScheduledExecutorService executorService = null;
+private long requestTimeoutMs = 500; // 500ms
+private int firstSpeculativeTimeoutMs = 50; // 50ms
+private int maxSpeculativeTimeoutMs = 200; // 200ms
+private float speculativeBackoffMultiplier = 2;
+private Ticker ticker = Ticker.systemTicker();
+
+private Builder() {}
+
+/**
+ * Set the distributedlog client used for multi stream writer.
+ *
+ * @param client
+ *  distributedlog client
+ * @return builder
+ */
+public Builder client(DistributedLogClient client) {
+this.client = client;
+return this;
+}
+
+/**
+ * Set the list of streams to write to.
+ *
+ * @param streams
+ *  list of streams to write
+ * @return builder
+ */
+public Builder streams(List streams) {
+this.streams = streams;
+return this;
+}
+
+/**
+ * Set the 

[39/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-client/src/test/java/org/apache/distributedlog/client/TestDistributedLogMultiStreamWriter.java
--
diff --git 
a/distributedlog-client/src/test/java/org/apache/distributedlog/client/TestDistributedLogMultiStreamWriter.java
 
b/distributedlog-client/src/test/java/org/apache/distributedlog/client/TestDistributedLogMultiStreamWriter.java
new file mode 100644
index 000..d7494de
--- /dev/null
+++ 
b/distributedlog-client/src/test/java/org/apache/distributedlog/client/TestDistributedLogMultiStreamWriter.java
@@ -0,0 +1,383 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog.client;
+
+import static com.google.common.base.Charsets.UTF_8;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.mockito.Mockito.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import com.google.common.collect.Lists;
+import org.apache.distributedlog.DLSN;
+import org.apache.distributedlog.LogRecord;
+import org.apache.distributedlog.LogRecordSet;
+import org.apache.distributedlog.LogRecordSetBuffer;
+import org.apache.distributedlog.exceptions.LogRecordTooLongException;
+import org.apache.distributedlog.io.CompressionCodec;
+import org.apache.distributedlog.service.DistributedLogClient;
+import com.twitter.finagle.IndividualRequestTimeoutException;
+import com.twitter.util.Await;
+import com.twitter.util.Future;
+import com.twitter.util.Promise;
+import java.nio.ByteBuffer;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+
+/**
+ * Test {@link DistributedLogMultiStreamWriter}.
+ */
+public class TestDistributedLogMultiStreamWriter {
+
+@Test(timeout = 2, expected = IllegalArgumentException.class)
+public void testBuildWithNullStreams() throws Exception {
+DistributedLogMultiStreamWriter.newBuilder()
+.build();
+}
+
+@Test(timeout = 2, expected = IllegalArgumentException.class)
+public void testBuildWithEmptyStreamList() throws Exception {
+DistributedLogMultiStreamWriter.newBuilder()
+.streams(Lists.newArrayList())
+.build();
+}
+
+@Test(timeout = 2, expected = NullPointerException.class)
+public void testBuildWithNullClient() throws Exception {
+DistributedLogMultiStreamWriter.newBuilder()
+.streams(Lists.newArrayList("stream1", "stream2"))
+.build();
+}
+
+@Test(timeout = 2, expected = NullPointerException.class)
+public void testBuildWithNullCodec() throws Exception {
+DistributedLogMultiStreamWriter.newBuilder()
+.streams(Lists.newArrayList("stream1", "stream2"))
+.client(mock(DistributedLogClient.class))
+.compressionCodec(null)
+.build();
+}
+
+@Test(timeout = 2, expected = IllegalArgumentException.class)
+public void testBuildWithInvalidSpeculativeSettings1()
+throws Exception {
+DistributedLogMultiStreamWriter.newBuilder()
+.streams(Lists.newArrayList("stream1", "stream2"))
+.client(mock(DistributedLogClient.class))
+.compressionCodec(CompressionCodec.Type.LZ4)
+.firstSpeculativeTimeoutMs(-1)
+.build();
+}
+
+@Test(timeout = 2, expected = IllegalArgumentException.class)
+public void testBuildWithInvalidSpeculativeSettings2()
+throws Exception {
+DistributedLogMultiStreamWriter.newBuilder()
+.streams(Lists.newArrayList("stream1", "stream2"))
+.client(mock(DistributedLogClient.class))
+

[02/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/org/apache/distributedlog/ZooKeeperClient.java
--
diff --git 
a/distributedlog-core/src/main/java/org/apache/distributedlog/ZooKeeperClient.java
 
b/distributedlog-core/src/main/java/org/apache/distributedlog/ZooKeeperClient.java
new file mode 100644
index 000..e56a22d
--- /dev/null
+++ 
b/distributedlog-core/src/main/java/org/apache/distributedlog/ZooKeeperClient.java
@@ -0,0 +1,402 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog;
+
+import com.google.common.base.Stopwatch;
+import org.apache.distributedlog.util.FailpointUtils;
+import org.apache.distributedlog.zk.ZKWatcherManager;
+import org.apache.bookkeeper.stats.NullStatsLogger;
+import org.apache.bookkeeper.stats.StatsLogger;
+import org.apache.bookkeeper.zookeeper.BoundExponentialBackoffRetryPolicy;
+import org.apache.bookkeeper.zookeeper.RetryPolicy;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.WatchedEvent;
+import org.apache.zookeeper.Watcher;
+import org.apache.zookeeper.Watcher.Event.EventType;
+import org.apache.zookeeper.Watcher.Event.KeeperState;
+import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.ZooDefs;
+import org.apache.zookeeper.data.ACL;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CopyOnWriteArraySet;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import static com.google.common.base.Charsets.UTF_8;
+
+/**
+ * ZooKeeper Client wrapper over {@link 
org.apache.bookkeeper.zookeeper.ZooKeeperClient}.
+ * It handles retries on session expires and provides a watcher manager {@link 
ZKWatcherManager}.
+ *
+ * Metrics
+ * 
+ *  zookeeper operation stats are exposed under scope zk by
+ * {@link org.apache.bookkeeper.zookeeper.ZooKeeperClient}
+ *  stats on zookeeper watched events are exposed under scope 
watcher by
+ * {@link org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase}
+ *  stats about {@link ZKWatcherManager} are exposed under scope 
watcher_manager
+ * 
+ */
+public class ZooKeeperClient {
+
+public static interface Credentials {
+
+Credentials NONE = new Credentials() {
+@Override
+public void authenticate(ZooKeeper zooKeeper) {
+// noop
+}
+};
+
+void authenticate(ZooKeeper zooKeeper);
+}
+
+public static class DigestCredentials implements Credentials {
+
+String username;
+String password;
+
+public DigestCredentials(String username, String password) {
+this.username = username;
+this.password = password;
+}
+
+@Override
+public void authenticate(ZooKeeper zooKeeper) {
+zooKeeper.addAuthInfo("digest", String.format("%s:%s", username, 
password).getBytes(UTF_8));
+}
+}
+
+public interface ZooKeeperSessionExpireNotifier {
+void notifySessionExpired();
+}
+
+/**
+ * Indicates an error connecting to a zookeeper cluster.
+ */
+public static class ZooKeeperConnectionException extends IOException {
+private static final long serialVersionUID = 6682391687004819361L;
+
+public ZooKeeperConnectionException(String message) {
+super(message);
+}
+
+public ZooKeeperConnectionException(String message, Throwable cause) {
+super(message, cause);
+}
+}
+
+private static final Logger LOG = 
LoggerFactory.getLogger(ZooKeeperClient.class.getName());
+
+private final String name;
+private final int sessionTimeoutMs;
+private final int defaultConnectionTimeoutMs;
+private final String zooKeeperServers;
+// GuardedBy "this", but still volatile for tests, where we want to be 
able to see writes
+// made from within long synchronized blocks.
+private volatile ZooKeeper zooKeeper = null;
+private final RetryPolicy 

[37/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogReader.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogReader.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogReader.java
deleted file mode 100644
index aee4103..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAsyncLogReader.java
+++ /dev/null
@@ -1,751 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Optional;
-import com.google.common.base.Stopwatch;
-import com.google.common.base.Ticker;
-import com.twitter.distributedlog.exceptions.DLIllegalStateException;
-import com.twitter.distributedlog.exceptions.DLInterruptedException;
-import com.twitter.distributedlog.exceptions.EndOfStreamException;
-import com.twitter.distributedlog.exceptions.IdleReaderException;
-import com.twitter.distributedlog.exceptions.LogNotFoundException;
-import com.twitter.distributedlog.exceptions.ReadCancelledException;
-import com.twitter.distributedlog.exceptions.UnexpectedException;
-import com.twitter.distributedlog.util.OrderedScheduler;
-import com.twitter.distributedlog.util.Utils;
-import com.twitter.util.Future;
-import com.twitter.util.FutureEventListener;
-import com.twitter.util.Promise;
-import com.twitter.util.Throw;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.concurrent.ConcurrentLinkedQueue;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.ScheduledFuture;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicLong;
-import java.util.concurrent.atomic.AtomicReference;
-
-import org.apache.bookkeeper.stats.Counter;
-import org.apache.bookkeeper.stats.OpStatsLogger;
-import org.apache.bookkeeper.stats.StatsLogger;
-import org.apache.bookkeeper.versioning.Versioned;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import scala.Function1;
-import scala.runtime.AbstractFunction1;
-import scala.runtime.BoxedUnit;
-
-/**
- * BookKeeper based {@link AsyncLogReader} implementation.
- *
- * Metrics
- * All the metrics are exposed under `async_reader`.
- * 
- *  `async_reader`/future_set: opstats. time spent on satisfying futures 
of read requests.
- * if it is high, it means that the caller takes time on processing the result 
of read requests.
- * The side effect is blocking consequent reads.
- *  `async_reader`/schedule: opstats. time spent on scheduling next reads.
- *  `async_reader`/background_read: opstats. time spent on background 
reads.
- *  `async_reader`/read_next_exec: opstats. time spent on executing {@link 
#readNext()}.
- *  `async_reader`/time_between_read_next: opstats. time spent on between 
two consequent {@link #readNext()}.
- * if it is high, it means that the caller is slowing down on calling {@link 
#readNext()}.
- *  `async_reader`/delay_until_promise_satisfied: opstats. total latency 
for the read requests.
- *  `async_reader`/idle_reader_error: counter. the number idle reader 
errors.
- * 
- */
-class BKAsyncLogReader implements AsyncLogReader, Runnable, AsyncNotification {
-static final Logger LOG = LoggerFactory.getLogger(BKAsyncLogReader.class);
-
-private static final Function1 
READ_NEXT_MAP_FUNCTION =
-new AbstractFunction1() {
-@Override
-public LogRecordWithDLSN apply(List 
records) {
-return records.get(0);
-}
-};
-
-private final String streamName;
-protected final BKDistributedLogManager bkDistributedLogManager;
-protected final BKLogReadHandler readHandler;
-private final AtomicReference lastException = new 
AtomicReference();
-private final OrderedScheduler scheduler;
-private final ConcurrentLinkedQueue pendingRequests = 
new ConcurrentLinkedQueue();
-private final 

[30/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/DistributedLogConstants.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/DistributedLogConstants.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/DistributedLogConstants.java
deleted file mode 100644
index e798a0f..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/DistributedLogConstants.java
+++ /dev/null
@@ -1,69 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog;
-
-import java.util.List;
-import java.util.concurrent.TimeUnit;
-
-import com.google.common.collect.ImmutableList;
-import org.apache.zookeeper.ZooDefs.Ids;
-import org.apache.zookeeper.data.ACL;
-
-import static com.google.common.base.Charsets.UTF_8;
-
-public class DistributedLogConstants {
-public static final byte[] EMPTY_BYTES = new byte[0];
-public static final String SCHEME_PREFIX = "distributedlog";
-public static final String BACKEND_BK = "bk";
-public static final long INVALID_TXID = -999;
-public static final long EMPTY_LOGSEGMENT_TX_ID = -99;
-public static final long MAX_TXID = Long.MAX_VALUE;
-public static final long SMALL_LOGSEGMENT_THRESHOLD = 10;
-public static final int LOGSEGMENT_NAME_VERSION = 1;
-public static final int FUTURE_TIMEOUT_IMMEDIATE = 0;
-public static final int FUTURE_TIMEOUT_INFINITE = -1;
-public static final long LOCK_IMMEDIATE = FUTURE_TIMEOUT_IMMEDIATE;
-public static final long LOCK_TIMEOUT_INFINITE = FUTURE_TIMEOUT_INFINITE;
-public static final long LOCK_OP_TIMEOUT_DEFAULT = 120;
-public static final long LOCK_REACQUIRE_TIMEOUT_DEFAULT = 120;
-public static final String UNKNOWN_CLIENT_ID = "Unknown-ClientId";
-public static final int LOCAL_REGION_ID = 0;
-public static final long LOGSEGMENT_DEFAULT_STATUS = 0;
-public static final long UNASSIGNED_LOGSEGMENT_SEQNO = 0;
-public static final long UNASSIGNED_SEQUENCE_ID = -1L;
-public static final long FIRST_LOGSEGMENT_SEQNO = 1;
-public static final long UNRESOLVED_LEDGER_ID = -1;
-public static final long LATENCY_WARN_THRESHOLD_IN_MILLIS = 
TimeUnit.SECONDS.toMillis(1);
-public static final int DL_INTERRUPTED_EXCEPTION_RESULT_CODE = 
Integer.MIN_VALUE + 1;
-public static final int ZK_CONNECTION_EXCEPTION_RESULT_CODE = 
Integer.MIN_VALUE + 2;
-
-public static final String ALLOCATION_POOL_NODE = ".allocation_pool";
-// log segment prefix
-public static final String INPROGRESS_LOGSEGMENT_PREFIX = "inprogress";
-public static final String COMPLETED_LOGSEGMENT_PREFIX = "logrecs";
-public static final String DISALLOW_PLACEMENT_IN_REGION_FEATURE_NAME = 
"disallow_bookie_placement";
-static final byte[] CONTROL_RECORD_CONTENT = "control".getBytes(UTF_8);
-static final byte[] KEEPALIVE_RECORD_CONTENT = "keepalive".getBytes(UTF_8);
-
-// An ACL that gives all permissions to node creators and read permissions 
only to everyone else.
-public static final List EVERYONE_READ_CREATOR_ALL =
-ImmutableList.builder()
-.addAll(Ids.CREATOR_ALL_ACL)
-.addAll(Ids.READ_ACL_UNSAFE)
-.build();
-}

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/DistributedLogManager.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/DistributedLogManager.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/DistributedLogManager.java
deleted file mode 100644
index 34cfb65..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/DistributedLogManager.java
+++ /dev/null
@@ -1,308 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The 

[03/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/org/apache/distributedlog/ReadAheadEntryReader.java
--
diff --git 
a/distributedlog-core/src/main/java/org/apache/distributedlog/ReadAheadEntryReader.java
 
b/distributedlog-core/src/main/java/org/apache/distributedlog/ReadAheadEntryReader.java
new file mode 100644
index 000..0b8c55a
--- /dev/null
+++ 
b/distributedlog-core/src/main/java/org/apache/distributedlog/ReadAheadEntryReader.java
@@ -0,0 +1,992 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Stopwatch;
+import com.google.common.base.Ticker;
+import com.google.common.collect.Lists;
+import org.apache.distributedlog.callback.LogSegmentListener;
+import 
org.apache.distributedlog.exceptions.AlreadyTruncatedTransactionException;
+import org.apache.distributedlog.exceptions.DLIllegalStateException;
+import org.apache.distributedlog.exceptions.DLInterruptedException;
+import org.apache.distributedlog.exceptions.EndOfLogSegmentException;
+import org.apache.distributedlog.exceptions.LogNotFoundException;
+import org.apache.distributedlog.exceptions.UnexpectedException;
+import org.apache.distributedlog.io.AsyncCloseable;
+import org.apache.distributedlog.logsegment.LogSegmentEntryReader;
+import org.apache.distributedlog.logsegment.LogSegmentEntryStore;
+import org.apache.distributedlog.logsegment.LogSegmentFilter;
+import org.apache.distributedlog.util.OrderedScheduler;
+import com.twitter.util.Function0;
+import com.twitter.util.Future;
+import com.twitter.util.FutureEventListener;
+import com.twitter.util.Futures;
+import com.twitter.util.Promise;
+import org.apache.bookkeeper.stats.AlertStatsLogger;
+import org.apache.bookkeeper.versioning.Versioned;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Function1;
+import scala.runtime.AbstractFunction1;
+import scala.runtime.BoxedUnit;
+
+import java.io.IOException;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArraySet;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * New ReadAhead Reader that uses {@link 
org.apache.distributedlog.logsegment.LogSegmentEntryReader}.
+ *
+ * NOTE: all the state changes happen in the same thread. All *unsafe* methods 
should be submitted to the order
+ * scheduler using stream name as the key.
+ */
+public class ReadAheadEntryReader implements
+AsyncCloseable,
+LogSegmentListener,
+LogSegmentEntryReader.StateChangeListener,
+FutureEventListener {
+
+private static final Logger logger = 
LoggerFactory.getLogger(ReadAheadEntryReader.class);
+
+//
+// Static Functions
+//
+
+private static AbstractFunction1 
START_READER_FUNC = new AbstractFunction1() {
+@Override
+public BoxedUnit apply(LogSegmentEntryReader reader) {
+reader.start();
+return BoxedUnit.UNIT;
+}
+};
+
+//
+// Internal Classes
+//
+
+class SegmentReader implements FutureEventListener {
+
+private LogSegmentMetadata metadata;
+private final long startEntryId;
+private Future openFuture = null;
+private LogSegmentEntryReader reader = null;
+private boolean isStarted = false;
+private boolean isClosed = false;
+
+SegmentReader(LogSegmentMetadata metadata,
+  long startEntryId) {
+this.metadata = metadata;
+this.startEntryId = startEntryId;
+}
+
+synchronized LogSegmentEntryReader getEntryReader() {
+return reader;
+}
+
+synchronized boolean isBeyondLastAddConfirmed() {
+return 

[21/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/injector/AsyncFailureInjector.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/injector/AsyncFailureInjector.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/injector/AsyncFailureInjector.java
deleted file mode 100644
index 4145040..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/injector/AsyncFailureInjector.java
+++ /dev/null
@@ -1,131 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.injector;
-
-/**
- * Failure Injector that works in asynchronous way
- */
-public interface AsyncFailureInjector {
-
-AsyncFailureInjector NULL = new AsyncFailureInjector() {
-@Override
-public void injectErrors(boolean enabled) {
-// no-op
-}
-
-@Override
-public boolean shouldInjectErrors() {
-return false;
-}
-
-@Override
-public void injectDelays(boolean enabled) {
-// no-op
-}
-
-@Override
-public boolean shouldInjectDelays() {
-return false;
-}
-
-@Override
-public int getInjectedDelayMs() {
-return 0;
-}
-
-@Override
-public void injectStops(boolean enabled) {
-// no-op
-}
-
-@Override
-public boolean shouldInjectStops() {
-return false;
-}
-
-@Override
-public boolean shouldInjectCorruption(long startEntryId, long 
endEntryId) {
-return false;
-}
-
-@Override
-public String toString() {
-return "NULL";
-}
-};
-
-/**
- * Enable or disable error injection.
- *
- * @param enabled
- *  flag to enable or disable error injection.
- */
-void injectErrors(boolean enabled);
-
-/**
- * Return the flag indicating if should inject errors.
- *
- * @return true to inject errors otherwise false.
- */
-boolean shouldInjectErrors();
-
-/**
- * Enable or disable delay injection.
- *
- * @param enabled
- *  flag to enable or disable delay injection.
- */
-void injectDelays(boolean enabled);
-
-/**
- * Return the flag indicating if should inject delays.
- *
- * @return true to inject delays otherwise false.
- */
-boolean shouldInjectDelays();
-
-/**
- * Return the injected delay in milliseconds.
- *
- * @return the injected delay in milliseconds.
- */
-int getInjectedDelayMs();
-
-/**
- * Enable or disable injecting stops. This could be used
- * for simulating stopping an action.
- */
-void injectStops(boolean enabled);
-
-/**
- * Return the flag indicating if should inject stops.
- *
- * @return true to inject stops otherwise false.
- */
-boolean shouldInjectStops();
-
-/**
- * Return the flag indicating if should inject corruption.
- *
- * @param startEntryId the start entry id
- * @param endEntryId the end entry id
- * @return true to inject corruption otherwise false.
- */
-boolean shouldInjectCorruption(long startEntryId, long endEntryId);
-
-}

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/injector/AsyncRandomFailureInjector.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/injector/AsyncRandomFailureInjector.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/injector/AsyncRandomFailureInjector.java
deleted file mode 100644
index f3bfea9..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/injector/AsyncRandomFailureInjector.java
+++ /dev/null
@@ -1,186 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license 

[15/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/tools/Tool.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/tools/Tool.java 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/tools/Tool.java
deleted file mode 100644
index bb14066..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/tools/Tool.java
+++ /dev/null
@@ -1,243 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.tools;
-
-import org.apache.bookkeeper.util.ReflectionUtils;
-import org.apache.commons.cli.BasicParser;
-import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.HelpFormatter;
-import org.apache.commons.cli.Options;
-import org.apache.commons.cli.ParseException;
-
-import java.util.Map;
-import java.util.TreeMap;
-
-/**
- * A Tool Framework
- */
-public abstract class Tool {
-
-/**
- * Interface of a command to run in a tool.
- */
-protected interface Command {
-String getName();
-String getDescription();
-int runCmd(String[] args) throws Exception;
-void printUsage();
-}
-
-/**
- * {@link org.apache.commons.cli.Options} based command.
- */
-protected abstract static class OptsCommand implements Command {
-
-/**
- * @return options used by this command.
- */
-protected abstract Options getOptions();
-
-/**
- * @return usage of this command.
- */
-protected String getUsage() {
-return cmdName + " [options]";
-}
-
-/**
- * Run given command line commandLine.
- *
- * @param commandLine
- *  command line to run.
- * @return return code of this command.
- * @throws Exception
- */
-protected abstract int runCmd(CommandLine commandLine) throws 
Exception;
-
-protected String cmdName;
-protected String description;
-
-protected OptsCommand(String name, String description) {
-this.cmdName = name;
-this.description = description;
-}
-
-@Override
-public String getName() {
-return cmdName;
-}
-
-@Override
-public String getDescription() {
-return description;
-}
-
-@Override
-public int runCmd(String[] args) throws Exception {
-try {
-BasicParser parser = new BasicParser();
-CommandLine cmdline = parser.parse(getOptions(), args);
-return runCmd(cmdline);
-} catch (ParseException e) {
-printUsage();
-return -1;
-}
-}
-
-@Override
-public void printUsage() {
-HelpFormatter helpFormatter = new HelpFormatter();
-println(cmdName + ": " + getDescription());
-helpFormatter.printHelp(getUsage(), getOptions());
-}
-}
-
-public class HelpCommand implements Command {
-
-@Override
-public String getName() {
-return "help";
-}
-
-@Override
-public String getDescription() {
-return "describe the usage of this tool or its sub-commands.";
-}
-
-@Override
-public int runCmd(String[] args) throws Exception {
-if (args.length == 0) {
-printToolUsage();
-return -1;
-}
-String cmdName = args[0];
-Command command = commands.get(cmdName);
-if (null == command) {
-System.err.println("Unknown command " + cmdName);
-printToolUsage();
-return -1;
-}
-command.printUsage();
-println("");
-return 0;
-}
-
-@Override
-public void printUsage() {
-println(getName() + ": " + getDescription());
-println("");
-println("usage: " + getName() + " ");
-  

[16/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/tools/DistributedLogTool.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/tools/DistributedLogTool.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/tools/DistributedLogTool.java
deleted file mode 100644
index 03d70bd..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/tools/DistributedLogTool.java
+++ /dev/null
@@ -1,2873 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.tools;
-
-import java.io.BufferedReader;
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileNotFoundException;
-import java.io.FileOutputStream;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.InputStreamReader;
-import java.io.OutputStreamWriter;
-import java.io.PrintWriter;
-import java.net.MalformedURLException;
-import java.net.URI;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.Comparator;
-import java.util.Enumeration;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.SortedMap;
-import java.util.TreeMap;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.LinkedBlockingQueue;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicInteger;
-import java.util.concurrent.atomic.AtomicReference;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-import com.google.common.base.Preconditions;
-import com.twitter.distributedlog.BKDistributedLogNamespace;
-import com.twitter.distributedlog.Entry;
-import com.twitter.distributedlog.MetadataAccessor;
-import com.twitter.distributedlog.callback.NamespaceListener;
-import com.twitter.distributedlog.impl.BKNamespaceDriver;
-import com.twitter.distributedlog.logsegment.LogSegmentMetadataStore;
-import com.twitter.distributedlog.namespace.DistributedLogNamespace;
-import com.twitter.distributedlog.namespace.DistributedLogNamespaceBuilder;
-import com.twitter.distributedlog.namespace.NamespaceDriver;
-import com.twitter.distributedlog.util.Utils;
-import org.apache.bookkeeper.client.BKException;
-import org.apache.bookkeeper.client.BookKeeper;
-import org.apache.bookkeeper.client.BookKeeperAccessor;
-import org.apache.bookkeeper.client.BookKeeperAdmin;
-import org.apache.bookkeeper.client.LedgerEntry;
-import org.apache.bookkeeper.client.LedgerHandle;
-import org.apache.bookkeeper.client.LedgerMetadata;
-import org.apache.bookkeeper.client.LedgerReader;
-import org.apache.bookkeeper.net.BookieSocketAddress;
-import org.apache.bookkeeper.proto.BookkeeperInternalCallbacks;
-import org.apache.bookkeeper.util.IOUtils;
-import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.Options;
-import org.apache.commons.cli.ParseException;
-import org.apache.commons.codec.binary.Hex;
-import org.apache.commons.configuration.ConfigurationException;
-import org.apache.commons.lang3.tuple.Pair;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.collect.Lists;
-import com.google.common.collect.Sets;
-import com.google.common.util.concurrent.RateLimiter;
-import com.twitter.distributedlog.AsyncLogReader;
-import com.twitter.distributedlog.AsyncLogWriter;
-import com.twitter.distributedlog.BookKeeperClient;
-import com.twitter.distributedlog.BookKeeperClientBuilder;
-import com.twitter.distributedlog.DLSN;
-import com.twitter.distributedlog.DistributedLogConfiguration;
-import com.twitter.distributedlog.DistributedLogConstants;
-import com.twitter.distributedlog.DistributedLogManager;
-import com.twitter.distributedlog.exceptions.LogNotFoundException;
-import com.twitter.distributedlog.LogReader;
-import com.twitter.distributedlog.LogRecord;
-import 

[17/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/selector/FirstRecordSelector.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/selector/FirstRecordSelector.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/selector/FirstRecordSelector.java
deleted file mode 100644
index a51210f..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/selector/FirstRecordSelector.java
+++ /dev/null
@@ -1,46 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.selector;
-
-import com.twitter.distributedlog.LogRecordWithDLSN;
-
-/**
- * Save the first record processed
- */
-public class FirstRecordSelector implements LogRecordSelector {
-
-final boolean includeControl;
-LogRecordWithDLSN firstRecord;
-
-public FirstRecordSelector(boolean includeControl) {
-this.includeControl = includeControl;
-}
-
-@Override
-public void process(LogRecordWithDLSN record) {
-if (null == this.firstRecord
-&& (includeControl || !record.isControl())) {
-this.firstRecord = record;
-}
-}
-
-@Override
-public LogRecordWithDLSN result() {
-return this.firstRecord;
-}
-}

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/selector/FirstTxIdNotLessThanSelector.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/selector/FirstTxIdNotLessThanSelector.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/selector/FirstTxIdNotLessThanSelector.java
deleted file mode 100644
index 03c2cbb..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/selector/FirstTxIdNotLessThanSelector.java
+++ /dev/null
@@ -1,51 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.selector;
-
-import com.twitter.distributedlog.LogRecordWithDLSN;
-
-/**
- * Save the first record with transaction id not less than the provided 
transaction id.
- * If all records' transaction id is less than provided transaction id, save 
the last record.
- */
-public class FirstTxIdNotLessThanSelector implements LogRecordSelector {
-
-LogRecordWithDLSN result;
-final long txId;
-boolean found = false;
-
-public FirstTxIdNotLessThanSelector(long txId) {
-this.txId = txId;
-}
-
-@Override
-public void process(LogRecordWithDLSN record) {
-if (found) {
-return;
-}
-this.result = record;
-if (record.getTransactionId() >= txId) {
-found = true;
-}
-}
-
-@Override
-public LogRecordWithDLSN result() {
-return this.result;
-}
-}

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/selector/LastRecordSelector.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/selector/LastRecordSelector.java
 

[29/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/LogSegmentMetadata.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/LogSegmentMetadata.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/LogSegmentMetadata.java
deleted file mode 100644
index a8d9e6d..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/LogSegmentMetadata.java
+++ /dev/null
@@ -1,1125 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog;
-
-import java.io.File;
-import java.io.IOException;
-import java.util.Comparator;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Objects;
-import com.twitter.distributedlog.exceptions.DLInterruptedException;
-import com.twitter.distributedlog.exceptions.LogSegmentNotFoundException;
-import 
com.twitter.distributedlog.exceptions.UnsupportedMetadataVersionException;
-import com.twitter.distributedlog.exceptions.ZKException;
-import com.twitter.distributedlog.util.FutureUtils;
-import com.twitter.distributedlog.util.Utils;
-import com.twitter.util.Future;
-import com.twitter.util.Promise;
-import org.apache.zookeeper.AsyncCallback;
-import org.apache.zookeeper.CreateMode;
-import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.data.Stat;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import static com.google.common.base.Charsets.UTF_8;
-
-/**
- * Utility class for storing the metadata associated
- * with a single edit log segment, stored in a single ledger
- */
-public class LogSegmentMetadata {
-static final Logger LOG = 
LoggerFactory.getLogger(LogSegmentMetadata.class);
-
-public static enum LogSegmentMetadataVersion {
-VERSION_INVALID(0),
-VERSION_V1_ORIGINAL(1),
-VERSION_V2_LEDGER_SEQNO(2),
-VERSION_V3_MIN_ACTIVE_DLSN(3),
-VERSION_V4_ENVELOPED_ENTRIES(4),
-VERSION_V5_SEQUENCE_ID(5);
-
-public final int value;
-
-private LogSegmentMetadataVersion(int value) {
-this.value = value;
-}
-
-public static LogSegmentMetadataVersion of(int version) {
-switch (version) {
-case 5:
-return VERSION_V5_SEQUENCE_ID;
-case 4:
-return VERSION_V4_ENVELOPED_ENTRIES;
-case 3:
-return VERSION_V3_MIN_ACTIVE_DLSN;
-case 2:
-return VERSION_V2_LEDGER_SEQNO;
-case 1:
-return VERSION_V1_ORIGINAL;
-case 0:
-return VERSION_INVALID;
-default:
-throw new IllegalArgumentException("unknown version " + 
version);
-}
-}
-}
-
-public static enum TruncationStatus {
-ACTIVE (0), PARTIALLY_TRUNCATED(1), TRUNCATED (2);
-private final int value;
-
-private TruncationStatus(int value) {
-this.value = value;
-}
-}
-
-public static class LogSegmentMetadataBuilder {
-protected String zkPath;
-protected long logSegmentId;
-protected LogSegmentMetadataVersion version;
-protected long firstTxId;
-protected int regionId;
-protected long status;
-protected long lastTxId;
-protected long completionTime;
-protected int recordCount;
-protected long logSegmentSequenceNo;
-protected long lastEntryId;
-protected long lastSlotId;
-protected long minActiveEntryId;
-protected long minActiveSlotId;
-protected long startSequenceId;
-protected boolean inprogress;
-
-// This is a derived attribute.
-// Since we overwrite the original version with the target version, 
information that is
-// derived from the original version (e.g. does it support enveloping 
of entries)
-// is lost while parsing.
-// NOTE: This value is not stored in the Metadata store.
-

[26/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/auditor/DLAuditor.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/auditor/DLAuditor.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/auditor/DLAuditor.java
deleted file mode 100644
index a081606..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/auditor/DLAuditor.java
+++ /dev/null
@@ -1,630 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog.auditor;
-
-import com.google.common.base.Objects;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Lists;
-import com.google.common.util.concurrent.SettableFuture;
-import com.twitter.distributedlog.BookKeeperClient;
-import com.twitter.distributedlog.BookKeeperClientBuilder;
-import com.twitter.distributedlog.DistributedLogConfiguration;
-import com.twitter.distributedlog.DistributedLogManager;
-import com.twitter.distributedlog.LogSegmentMetadata;
-import com.twitter.distributedlog.impl.BKNamespaceDriver;
-import com.twitter.distributedlog.namespace.DistributedLogNamespace;
-import com.twitter.distributedlog.ZooKeeperClient;
-import com.twitter.distributedlog.ZooKeeperClientBuilder;
-import com.twitter.distributedlog.exceptions.DLInterruptedException;
-import com.twitter.distributedlog.exceptions.ZKException;
-import com.twitter.distributedlog.impl.metadata.BKDLConfig;
-import com.twitter.distributedlog.namespace.DistributedLogNamespaceBuilder;
-import com.twitter.distributedlog.namespace.NamespaceDriver;
-import com.twitter.distributedlog.util.DLUtils;
-import org.apache.bookkeeper.client.BKException;
-import org.apache.bookkeeper.client.BookKeeper;
-import org.apache.bookkeeper.client.BookKeeperAccessor;
-import org.apache.bookkeeper.client.LedgerHandle;
-import org.apache.bookkeeper.meta.LedgerManager;
-import org.apache.bookkeeper.proto.BookkeeperInternalCallbacks;
-import org.apache.bookkeeper.zookeeper.BoundExponentialBackoffRetryPolicy;
-import org.apache.bookkeeper.zookeeper.RetryPolicy;
-import org.apache.commons.lang3.tuple.Pair;
-import org.apache.zookeeper.AsyncCallback;
-import org.apache.zookeeper.KeeperException;
-import org.apache.zookeeper.data.Stat;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.net.URI;
-import java.util.ArrayList;
-import java.util.HashSet;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.TreeSet;
-import java.util.concurrent.ConcurrentSkipListMap;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.LinkedBlockingQueue;
-import java.util.concurrent.atomic.AtomicInteger;
-import java.util.concurrent.atomic.AtomicLong;
-
-import static com.google.common.base.Charsets.UTF_8;
-
-/**
- * DL Auditor will audit DL namespace, e.g. find leaked ledger, report disk 
usage by streams.
- */
-public class DLAuditor {
-
-private static final Logger logger = 
LoggerFactory.getLogger(DLAuditor.class);
-
-private final DistributedLogConfiguration conf;
-
-public DLAuditor(DistributedLogConfiguration conf) {
-this.conf = conf;
-}
-
-private ZooKeeperClient getZooKeeperClient(DistributedLogNamespace 
namespace) {
-NamespaceDriver driver = namespace.getNamespaceDriver();
-assert(driver instanceof BKNamespaceDriver);
-return ((BKNamespaceDriver) driver).getWriterZKC();
-}
-
-private BookKeeperClient getBookKeeperClient(DistributedLogNamespace 
namespace) {
-NamespaceDriver driver = namespace.getNamespaceDriver();
-assert(driver instanceof BKNamespaceDriver);
-return ((BKNamespaceDriver) driver).getReaderBKC();
-}
-
-private String validateAndGetZKServers(List uris) {
-URI firstURI = uris.get(0);
-String zkServers = 

[32/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/com/twitter/distributedlog/BookKeeperClient.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BookKeeperClient.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BookKeeperClient.java
deleted file mode 100644
index 8d3c418..000
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BookKeeperClient.java
+++ /dev/null
@@ -1,289 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.twitter.distributedlog;
-
-import com.google.common.base.Optional;
-import com.twitter.distributedlog.ZooKeeperClient.Credentials;
-import com.twitter.distributedlog.ZooKeeperClient.DigestCredentials;
-import com.twitter.distributedlog.exceptions.AlreadyClosedException;
-import com.twitter.distributedlog.exceptions.DLInterruptedException;
-import com.twitter.distributedlog.exceptions.ZKException;
-import com.twitter.distributedlog.net.NetUtils;
-import com.twitter.distributedlog.util.ConfUtils;
-import com.twitter.util.Future;
-import com.twitter.util.Promise;
-import com.twitter.util.Return;
-import com.twitter.util.Throw;
-import org.apache.bookkeeper.client.AsyncCallback;
-import org.apache.bookkeeper.client.BKException;
-import org.apache.bookkeeper.client.BookKeeper;
-import org.apache.bookkeeper.client.LedgerHandle;
-import org.apache.bookkeeper.client.RegionAwareEnsemblePlacementPolicy;
-import org.apache.bookkeeper.conf.ClientConfiguration;
-import org.apache.bookkeeper.feature.FeatureProvider;
-import org.apache.bookkeeper.net.DNSToSwitchMapping;
-import org.apache.bookkeeper.stats.StatsLogger;
-import org.apache.bookkeeper.zookeeper.BoundExponentialBackoffRetryPolicy;
-import org.apache.bookkeeper.zookeeper.RetryPolicy;
-import org.apache.commons.configuration.ConfigurationException;
-import org.apache.zookeeper.KeeperException;
-import org.jboss.netty.channel.socket.ClientSocketChannelFactory;
-import org.jboss.netty.util.HashedWheelTimer;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-
-import static com.google.common.base.Charsets.UTF_8;
-
-/**
- * BookKeeper Client wrapper over {@link BookKeeper}.
- *
- * Metrics
- * 
- *  bookkeeper operation stats are exposed under current scope by {@link 
BookKeeper}
- * 
- */
-public class BookKeeperClient {
-static final Logger LOG = LoggerFactory.getLogger(BookKeeperClient.class);
-
-// Parameters to build bookkeeper client
-private final DistributedLogConfiguration conf;
-private final String name;
-private final String zkServers;
-private final String ledgersPath;
-private final byte[] passwd;
-private final ClientSocketChannelFactory channelFactory;
-private final HashedWheelTimer requestTimer;
-private final StatsLogger statsLogger;
-
-// bookkeeper client state
-private boolean closed = false;
-private BookKeeper bkc = null;
-private ZooKeeperClient zkc;
-private final boolean ownZK;
-// feature provider
-private final Optional featureProvider;
-
-@SuppressWarnings("deprecation")
-private synchronized void commonInitialization(
-DistributedLogConfiguration conf, String ledgersPath,
-ClientSocketChannelFactory channelFactory, StatsLogger 
statsLogger, HashedWheelTimer requestTimer)
-throws IOException, InterruptedException, KeeperException {
-ClientConfiguration bkConfig = new ClientConfiguration();
-bkConfig.setAddEntryTimeout(conf.getBKClientWriteTimeout());
-bkConfig.setReadTimeout(conf.getBKClientReadTimeout());
-bkConfig.setZkLedgersRootPath(ledgersPath);
-bkConfig.setZkTimeout(conf.getBKClientZKSessionTimeoutMilliSeconds());
-bkConfig.setNumWorkerThreads(conf.getBKClientNumberWorkerThreads());
-
bkConfig.setEnsemblePlacementPolicy(RegionAwareEnsemblePlacementPolicy.class);
-bkConfig.setZkRequestRateLimit(conf.getBKClientZKRequestRateLimit());
-

[11/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/org/apache/distributedlog/BKDistributedLogManager.java
--
diff --git 
a/distributedlog-core/src/main/java/org/apache/distributedlog/BKDistributedLogManager.java
 
b/distributedlog-core/src/main/java/org/apache/distributedlog/BKDistributedLogManager.java
new file mode 100644
index 000..00e6b5c
--- /dev/null
+++ 
b/distributedlog-core/src/main/java/org/apache/distributedlog/BKDistributedLogManager.java
@@ -0,0 +1,1106 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.base.Preconditions;
+import org.apache.distributedlog.callback.LogSegmentListener;
+import org.apache.distributedlog.config.DynamicDistributedLogConfiguration;
+import org.apache.distributedlog.exceptions.AlreadyClosedException;
+import org.apache.distributedlog.exceptions.LogEmptyException;
+import org.apache.distributedlog.exceptions.LogNotFoundException;
+import org.apache.distributedlog.exceptions.UnexpectedException;
+import org.apache.distributedlog.function.CloseAsyncCloseableFunction;
+import org.apache.distributedlog.function.GetVersionedValueFunction;
+import org.apache.distributedlog.injector.AsyncFailureInjector;
+import org.apache.distributedlog.logsegment.LogSegmentEntryStore;
+import org.apache.distributedlog.logsegment.LogSegmentEntryWriter;
+import org.apache.distributedlog.metadata.LogMetadataForReader;
+import org.apache.distributedlog.metadata.LogMetadataForWriter;
+import org.apache.distributedlog.io.AsyncCloseable;
+import org.apache.distributedlog.lock.DistributedLock;
+import org.apache.distributedlog.lock.NopDistributedLock;
+import org.apache.distributedlog.lock.ZKDistributedLock;
+import org.apache.distributedlog.logsegment.LogSegmentFilter;
+import org.apache.distributedlog.logsegment.LogSegmentMetadataCache;
+import org.apache.distributedlog.metadata.LogStreamMetadataStore;
+import org.apache.distributedlog.namespace.NamespaceDriver;
+import org.apache.distributedlog.stats.BroadCastStatsLogger;
+import org.apache.distributedlog.subscription.SubscriptionsStore;
+import org.apache.distributedlog.util.Allocator;
+import org.apache.distributedlog.util.DLUtils;
+import org.apache.distributedlog.util.FutureUtils;
+import org.apache.distributedlog.util.MonitoredFuturePool;
+import org.apache.distributedlog.util.OrderedScheduler;
+import org.apache.distributedlog.util.PermitLimiter;
+import org.apache.distributedlog.util.PermitManager;
+import org.apache.distributedlog.util.SchedulerUtils;
+import org.apache.distributedlog.util.Utils;
+import com.twitter.util.ExceptionalFunction;
+import com.twitter.util.ExceptionalFunction0;
+import com.twitter.util.Function;
+import com.twitter.util.Future;
+import com.twitter.util.FutureEventListener;
+import com.twitter.util.Promise;
+import org.apache.bookkeeper.feature.FeatureProvider;
+import org.apache.bookkeeper.stats.AlertStatsLogger;
+import org.apache.bookkeeper.stats.StatsLogger;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.runtime.AbstractFunction0;
+import scala.runtime.AbstractFunction1;
+import scala.runtime.BoxedUnit;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.net.URI;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.distributedlog.namespace.NamespaceDriver.Role.READER;
+import static org.apache.distributedlog.namespace.NamespaceDriver.Role.WRITER;
+
+/**
+ * Metrics
+ * 
+ *  `log_writer/*`: all asynchronous writer related metrics are exposed 
under scope `log_writer`.
+ * See {@link BKAsyncLogWriter} for detail stats.
+ *  `async_reader/*`: all asyncrhonous reader related metrics are exposed 
under scope `async_reader`.
+ * See {@link BKAsyncLogReader} for detail stats.
+ *  `writer_future_pool/*`: metrics about the future pools that used by 
writers are exposed under
+ * scope 

[07/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/org/apache/distributedlog/BookKeeperClient.java
--
diff --git 
a/distributedlog-core/src/main/java/org/apache/distributedlog/BookKeeperClient.java
 
b/distributedlog-core/src/main/java/org/apache/distributedlog/BookKeeperClient.java
new file mode 100644
index 000..a7b17f4
--- /dev/null
+++ 
b/distributedlog-core/src/main/java/org/apache/distributedlog/BookKeeperClient.java
@@ -0,0 +1,289 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog;
+
+import com.google.common.base.Optional;
+import org.apache.distributedlog.ZooKeeperClient.Credentials;
+import org.apache.distributedlog.ZooKeeperClient.DigestCredentials;
+import org.apache.distributedlog.exceptions.AlreadyClosedException;
+import org.apache.distributedlog.exceptions.DLInterruptedException;
+import org.apache.distributedlog.exceptions.ZKException;
+import org.apache.distributedlog.net.NetUtils;
+import org.apache.distributedlog.util.ConfUtils;
+import com.twitter.util.Future;
+import com.twitter.util.Promise;
+import com.twitter.util.Return;
+import com.twitter.util.Throw;
+import org.apache.bookkeeper.client.AsyncCallback;
+import org.apache.bookkeeper.client.BKException;
+import org.apache.bookkeeper.client.BookKeeper;
+import org.apache.bookkeeper.client.LedgerHandle;
+import org.apache.bookkeeper.client.RegionAwareEnsemblePlacementPolicy;
+import org.apache.bookkeeper.conf.ClientConfiguration;
+import org.apache.bookkeeper.feature.FeatureProvider;
+import org.apache.bookkeeper.net.DNSToSwitchMapping;
+import org.apache.bookkeeper.stats.StatsLogger;
+import org.apache.bookkeeper.zookeeper.BoundExponentialBackoffRetryPolicy;
+import org.apache.bookkeeper.zookeeper.RetryPolicy;
+import org.apache.commons.configuration.ConfigurationException;
+import org.apache.zookeeper.KeeperException;
+import org.jboss.netty.channel.socket.ClientSocketChannelFactory;
+import org.jboss.netty.util.HashedWheelTimer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import static com.google.common.base.Charsets.UTF_8;
+
+/**
+ * BookKeeper Client wrapper over {@link BookKeeper}.
+ *
+ * Metrics
+ * 
+ *  bookkeeper operation stats are exposed under current scope by {@link 
BookKeeper}
+ * 
+ */
+public class BookKeeperClient {
+static final Logger LOG = LoggerFactory.getLogger(BookKeeperClient.class);
+
+// Parameters to build bookkeeper client
+private final DistributedLogConfiguration conf;
+private final String name;
+private final String zkServers;
+private final String ledgersPath;
+private final byte[] passwd;
+private final ClientSocketChannelFactory channelFactory;
+private final HashedWheelTimer requestTimer;
+private final StatsLogger statsLogger;
+
+// bookkeeper client state
+private boolean closed = false;
+private BookKeeper bkc = null;
+private ZooKeeperClient zkc;
+private final boolean ownZK;
+// feature provider
+private final Optional featureProvider;
+
+@SuppressWarnings("deprecation")
+private synchronized void commonInitialization(
+DistributedLogConfiguration conf, String ledgersPath,
+ClientSocketChannelFactory channelFactory, StatsLogger 
statsLogger, HashedWheelTimer requestTimer)
+throws IOException, InterruptedException, KeeperException {
+ClientConfiguration bkConfig = new ClientConfiguration();
+bkConfig.setAddEntryTimeout(conf.getBKClientWriteTimeout());
+bkConfig.setReadTimeout(conf.getBKClientReadTimeout());
+bkConfig.setZkLedgersRootPath(ledgersPath);
+bkConfig.setZkTimeout(conf.getBKClientZKSessionTimeoutMilliSeconds());
+bkConfig.setNumWorkerThreads(conf.getBKClientNumberWorkerThreads());
+
bkConfig.setEnsemblePlacementPolicy(RegionAwareEnsemblePlacementPolicy.class);
+bkConfig.setZkRequestRateLimit(conf.getBKClientZKRequestRateLimit());
+

[10/51] [partial] incubator-distributedlog git commit: DL-4: Repackage the source under apache namespace

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/b44820b5/distributedlog-core/src/main/java/org/apache/distributedlog/BKLogHandler.java
--
diff --git 
a/distributedlog-core/src/main/java/org/apache/distributedlog/BKLogHandler.java 
b/distributedlog-core/src/main/java/org/apache/distributedlog/BKLogHandler.java
new file mode 100644
index 000..07ae0ff
--- /dev/null
+++ 
b/distributedlog-core/src/main/java/org/apache/distributedlog/BKLogHandler.java
@@ -0,0 +1,715 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.distributedlog;
+
+import com.google.common.base.Stopwatch;
+import org.apache.distributedlog.callback.LogSegmentNamesListener;
+import org.apache.distributedlog.exceptions.LogEmptyException;
+import org.apache.distributedlog.exceptions.LogSegmentNotFoundException;
+import org.apache.distributedlog.exceptions.UnexpectedException;
+import org.apache.distributedlog.logsegment.LogSegmentEntryStore;
+import org.apache.distributedlog.metadata.LogMetadata;
+import org.apache.distributedlog.io.AsyncAbortable;
+import org.apache.distributedlog.io.AsyncCloseable;
+import org.apache.distributedlog.logsegment.LogSegmentMetadataCache;
+import org.apache.distributedlog.logsegment.PerStreamLogSegmentCache;
+import org.apache.distributedlog.logsegment.LogSegmentFilter;
+import org.apache.distributedlog.logsegment.LogSegmentMetadataStore;
+import org.apache.distributedlog.metadata.LogStreamMetadataStore;
+import org.apache.distributedlog.util.FutureUtils;
+import org.apache.distributedlog.util.OrderedScheduler;
+import com.twitter.util.Function;
+import com.twitter.util.Future;
+import com.twitter.util.FutureEventListener;
+import com.twitter.util.Promise;
+import org.apache.bookkeeper.stats.AlertStatsLogger;
+import org.apache.bookkeeper.stats.OpStatsLogger;
+import org.apache.bookkeeper.stats.StatsLogger;
+import org.apache.bookkeeper.versioning.Version;
+import org.apache.bookkeeper.versioning.Versioned;
+import org.apache.commons.lang3.tuple.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * The base class about log handler on managing log segments.
+ *
+ * Metrics
+ * The log handler is a base class on managing log segments. so all the metrics
+ * here are related to log segments retrieval and exposed under `logsegments`.
+ * These metrics are all OpStats, in the format of 
`scope`/logsegments/`op`.
+ * 
+ * Those operations are:
+ * 
+ * get_inprogress_segment: time between the inprogress log segment created 
and
+ * the handler read it.
+ * get_completed_segment: time between a log segment is turned to 
completed and
+ * the handler read it.
+ * negative_get_inprogress_segment: record the negative values for 
`get_inprogress_segment`.
+ * negative_get_completed_segment: record the negative values for 
`get_completed_segment`.
+ * recover_last_entry: recovering last entry from a log segment
+ * recover_scanned_entries: the number of entries that are scanned during 
recovering.
+ * 
+ * @see BKLogWriteHandler
+ * @see BKLogReadHandler
+ */
+public abstract class BKLogHandler implements AsyncCloseable, AsyncAbortable {
+static final Logger LOG = LoggerFactory.getLogger(BKLogHandler.class);
+
+protected final LogMetadata logMetadata;
+protected final DistributedLogConfiguration conf;
+protected final LogStreamMetadataStore streamMetadataStore;
+protected final LogSegmentMetadataStore metadataStore;
+protected final LogSegmentMetadataCache metadataCache;
+protected final LogSegmentEntryStore entryStore;
+protected final int firstNumEntriesPerReadLastRecordScan;
+protected final int maxNumEntriesPerReadLastRecordScan;
+protected volatile long lastLedgerRollingTimeMillis 

[GitHub] incubator-distributedlog pull request #95: DL-4: Add repackaging script

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/95


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-distributedlog issue #95: DL-4: Add repackaging script

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/95
  
merging this, so I can repackage the namespace.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-distributedlog issue #94: DL-167 : fix testCalculateUnequalWeight ...

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/94
  
@xieliang this change is conflicted with current master. do you mind 
rebasing it again?

@franckcuny I think there is a description in the jira. but I think it 
might be good to ask people include description in the commit (we can probably 
update the contribution guide).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-distributedlog pull request #91: DL-165 : Add TestTimedOutTestsLis...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/91


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

incubator-distributedlog git commit: DL-165: Add TestTimedOutTestsListener to dump timed out cases thread dump

[sijie]

Repository: incubator-distributedlog
Updated Branches:
  refs/heads/master 1a30b0ceb -> ea3c1143f


DL-165: Add TestTimedOutTestsListener to dump timed out cases thread dump

Author: xieliang 

Reviewers: Leigh Stewart 

Closes #91 from xieliang/DL-165-TimedOutTestsListener


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/ea3c1143
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/ea3c1143
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/ea3c1143

Branch: refs/heads/master
Commit: ea3c1143f9e2718d0d86e8b1c8f3a7e51ac19c4d
Parents: 1a30b0c
Author: xieliang 
Authored: Wed Jan 4 16:09:01 2017 -0800
Committer: Sijie Guo 
Committed: Wed Jan 4 16:09:01 2017 -0800

--
 NOTICE  |   9 +
 distributedlog-client/pom.xml   |  15 +-
 distributedlog-core/pom.xml |  15 +-
 distributedlog-protocol/pom.xml |  12 ++
 .../TestTimedOutTestsListener.java  | 183 +++
 .../distributedlog/TimedOutTestsListener.java   | 168 +
 distributedlog-service/pom.xml  |  24 +++
 7 files changed, 424 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/ea3c1143/NOTICE
--
diff --git a/NOTICE b/NOTICE
index 5245e00..7281f12 100644
--- a/NOTICE
+++ b/NOTICE
@@ -10,3 +10,12 @@ The Apache Software Foundation (http://www.apache.org/).
 
 Portions of this software were developed by Twitter.
 Copyright Twitter, 2016
+
+This product contains a modified portion of 'Apache Hadoop', a Java framework 
that
+allows for the distributed processing of large data sets across clusters of 
computers
+using simple programming models., which can be obtained at:
+
+  * LICENSE:
+* LICENSE (Apache License 2.0)
+  * HOMEPAGE:
+* https://hadoop.apache.org/

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/ea3c1143/distributedlog-client/pom.xml
--
diff --git a/distributedlog-client/pom.xml b/distributedlog-client/pom.xml
index 1e88672..aad5093 100644
--- a/distributedlog-client/pom.xml
+++ b/distributedlog-client/pom.xml
@@ -85,6 +85,13 @@
   1.9.5
   test
  
+
+  com.twitter
+  distributedlog-protocol
+  ${project.parent.version}
+  test-jar
+  test
+
   
   
 
@@ -121,12 +128,18 @@
   
 org.apache.maven.plugins
 maven-surefire-plugin
-2.9
+2.19.1
 
   true
   -Xmx3G -Djava.net.preferIPv4Stack=true 
-XX:MaxDirectMemorySize=2G
   always
   1800
+  
+
+  listener
+  com.twitter.distributedlog.TimedOutTestsListener
+
+  
 
   
   

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/ea3c1143/distributedlog-core/pom.xml
--
diff --git a/distributedlog-core/pom.xml b/distributedlog-core/pom.xml
index 4da1849..c5329aa 100644
--- a/distributedlog-core/pom.xml
+++ b/distributedlog-core/pom.xml
@@ -145,6 +145,13 @@
   1.9.5
   test
  
+
+  com.twitter
+  distributedlog-protocol
+  ${project.parent.version}
+  test-jar
+  test
+
   
   
 
@@ -190,12 +197,18 @@
   
 org.apache.maven.plugins
 maven-surefire-plugin
-2.9
+2.19.1
 
   true
   -Xmx3G -Djava.net.preferIPv4Stack=true 
-XX:MaxDirectMemorySize=2G
   always
   1800
+  
+
+  listener
+  com.twitter.distributedlog.TimedOutTestsListener
+
+  
 
   
   

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/ea3c1143/distributedlog-protocol/pom.xml
--
diff --git a/distributedlog-protocol/pom.xml b/distributedlog-protocol/pom.xml
index 3adae05..5e55482 100644
--- a/distributedlog-protocol/pom.xml
+++ b/distributedlog-protocol/pom.xml
@@ -107,6 +107,18 @@
 
   
   
+org.apache.maven.plugins
+maven-jar-plugin
+2.2
+
+  
+
+  test-jar
+
+  
+
+  
+  
 org.codehaus.mojo
 findbugs-maven-plugin
 

RE: Git Workflow on Windows

[Tracy Martin]

Pasted log here:

https://gist.github.com/adamtracymartin/2ec2585d9d8efa8f81cbbcaaa2a2ed90


-Original Message-
From: Franck Cuny [mailto:franck.c...@gmail.com]
Sent: Tuesday, January 03, 2017 7:57 PM
To: dev@distributedlog.incubator.apache.org
Subject: Re: Git Workflow on Windows

Hello Tracy,

could you paste the complete output of the command you run somewhere ?

As far as I know, we haven't tried to build DLog on a windows system, and I 
don't think we've made any effort to support this so far.

On Tue, Jan 3, 2017 at 11:35 AM Tracy Martin  wrote:

> I am fixing DL-28 after the merge, and I am following the "Git Workflow"
> described here:
>
> https://cwiki.apache.org/confluence/display/DL/Contributing+to+Distrib
> utedLog
>
> I am getting many failures running step 4
>
> mvn clean apache-rat:check package findbugs:check
>
> from my Windows desktop.
>
> I know the error received in DL-28 was related to running the build on
> Windows instead of Unix.  Many of the errors that I see in the tests
> point to a Windows only problem.  Most of the errors reports are "Path
> must start with / character".
>
> I know when I was first reading about DistributedLog that it required
> Unix.  Is the plan to only run from Unix systems?  If so, I will look
> at creating a Unix environment for building, testing, and running.
> This e-mail and any attachments, contain Genuine Parts Company
> confidential information that is proprietary, privileged, and
> protected by applicable laws. If you have received this message in
> error and are not the intended recipient, you should not retain,
> distribute, disclose or use any of this information and you should
> destroy this e-mail, any attachments or copies therein forthwith.
> Please notify the sender immediately by e-mail if you have received this 
> e-mail in error.
>
> "This e-mail and any attachments, contain Genuine Parts Company
> confidential information that is proprietary, privileged, and
> protected by applicable laws. If you have received this message in
> error and are not the intended recipient, you should not retain,
> distribute, disclose or use any of this information and you should
> destroy this e-mail, any attachments or copies therein forthwith.
> Please notify the sender immediately by e-mail if you have received this 
> e-mail in error."
>
--
-franck
This e-mail and any attachments, contain Genuine Parts Company confidential 
information that is proprietary, privileged, and protected by applicable laws. 
If you have received this message in error and are not the intended recipient, 
you should not retain, distribute, disclose or use any of this information and 
you should destroy this e-mail, any attachments or copies therein forthwith. 
Please notify the sender immediately by e-mail if you have received this e-mail 
in error.

"This e-mail and any attachments, contain Genuine Parts Company confidential 
information that is proprietary, privileged, and protected by applicable laws. 
If you have received this message in error and are not the intended recipient, 
you should not retain, distribute, disclose or use any of this information and 
you should destroy this e-mail, any attachments or copies therein forthwith. 
Please notify the sender immediately by e-mail if you have received this e-mail 
in error."

RE: Git Workflow on Windows

[Tracy Martin]

I think that most of the issues are in the test cases at this point.  I will 
paste the output later when I am at my desk.

I suspect that bookkeeper has similar issues.  The Readme on gitgub says to use 
a Unix system to build.  I tried to run the same mvn command with tests on 
4.3.6 and received error in the tests.

I can work on making the changes for Windows.  Should we create a separate 
ticket in Jira?  Also, I could not create an issue in Jira.

Sent from my T-Mobile 4G LTE Device


 Original message 
From: Sijie Guo 
Date: 1/4/17 3:29 AM (GMT-05:00)
To: dev@distributedlog.incubator.apache.org
Subject: Re: Git Workflow on Windows

We definitely want to support running distributedlog in other environments.
And in theory, it is a Java system, it should be able to run on different
platforms. But as what Franck mentioned, the major problem is most of us
don't have a windows environment :/ I tried to have a Jenkins job or Travis
CI job to build distributedlog on windows. But unfortunately, it is
inconvenient for me to debug what it failed.

Based on the conversation in the jiras, it seems that the major problems
are a compilation problem and a symlink problem. So fixing these two should
be able to compile and run distributedlog on windows. However, in order to
validate the performance on windows, we might seek for helps from the
people who have more experiences on windows.

- Sijie

On Tue, Jan 3, 2017 at 4:56 PM, Franck Cuny  wrote:

> Hello Tracy,
>
> could you paste the complete output of the command you run somewhere ?
>
> As far as I know, we haven't tried to build DLog on a windows system, and I
> don't think we've made any effort to support this so far.
>
> On Tue, Jan 3, 2017 at 11:35 AM Tracy Martin 
> wrote:
>
> > I am fixing DL-28 after the merge, and I am following the "Git Workflow"
> > described here:
> >
> > https://cwiki.apache.org/confluence/display/DL/
> Contributing+to+DistributedLog
> >
> > I am getting many failures running step 4
> >
> > mvn clean apache-rat:check package findbugs:check
> >
> > from my Windows desktop.
> >
> > I know the error received in DL-28 was related to running the build on
> > Windows instead of Unix.  Many of the errors that I see in the tests
> point
> > to a Windows only problem.  Most of the errors reports are "Path must
> start
> > with / character".
> >
> > I know when I was first reading about DistributedLog that it required
> > Unix.  Is the plan to only run from Unix systems?  If so, I will look at
> > creating a Unix environment for building, testing, and running.
> > This e-mail and any attachments, contain Genuine Parts Company
> > confidential information that is proprietary, privileged, and protected
> by
> > applicable laws. If you have received this message in error and are not
> the
> > intended recipient, you should not retain, distribute, disclose or use
> any
> > of this information and you should destroy this e-mail, any attachments
> or
> > copies therein forthwith. Please notify the sender immediately by e-mail
> if
> > you have received this e-mail in error.
> >
> > "This e-mail and any attachments, contain Genuine Parts Company
> > confidential information that is proprietary, privileged, and protected
> by
> > applicable laws. If you have received this message in error and are not
> the
> > intended recipient, you should not retain, distribute, disclose or use
> any
> > of this information and you should destroy this e-mail, any attachments
> or
> > copies therein forthwith. Please notify the sender immediately by e-mail
> if
> > you have received this e-mail in error."
> >
> --
> -franck
>
This e-mail and any attachments, contain Genuine Parts Company confidential 
information that is proprietary, privileged, and protected by applicable laws. 
If you have received this message in error and are not the intended recipient, 
you should not retain, distribute, disclose or use any of this information and 
you should destroy this e-mail, any attachments or copies therein forthwith. 
Please notify the sender immediately by e-mail if you have received this e-mail 
in error.

"This e-mail and any attachments, contain Genuine Parts Company confidential 
information that is proprietary, privileged, and protected by applicable laws. 
If you have received this message in error and are not the intended recipient, 
you should not retain, distribute, disclose or use any of this information and 
you should destroy this e-mail, any attachments or copies therein forthwith. 
Please notify the sender immediately by e-mail if you have received this e-mail 
in error."

Re: [Discuss] Transaction Support

[Asko Kauppi]

> Beside that, I have one general question - What is the major goal for this
> feature? Are you targeting on building a general XA transaction coordinator
> or just for supporting things like `copy-modify-write' style workflow?

The use case I would have for transactions - at some level of the stack - is 
supporting dynamic configurations.

If a config changes in e.g. three lines, some of the changes may logically 
belong together. E.g. changing both “host” and “port” (if separate entries). 
One shouldn’t be able to read a state, even temporarily, that has new host but 
old port.

I can do this in the application level - it does not need to be part of the DL 
protocol.


Asko Kauppi
Zalando Tech Helsinki

> On 4 Jan 2017, at 9.18, Sijie Guo  wrote:
> 
> Sorry for late response. I think Leigh and you already had some very
> valuable discussions in the doc. I will try to add some of my questions to
> the discussion.
> 
> Beside that, I had a discussion with Leigh today about this. first of all,
> I think it is very good to add transaction support in distributedlog. It is
> one of the primitives that would help building distributed service. But we
> have a concern about making this system become complicated and introduce
> operational overhead when it runs in the large scale system on production.
> There are two major suggestions that I have for this feature -
> 
> Build the 'minimum' logic in core - I think the minimum logic that need to
> be added to the core is -  the special control records (begin, commit and
> abort) and make the reader be able to detect those special control records
> and know what do they mean and how to interrupt with them. Since they are
> special control records, there is not overhead to other readers that
> doesn't require this feature.
> 
> Build the transaction coordinator as a separated proxy service  - I think
> the major concern that we have is putting more complexities into the 'write
> proxy' service. We architected distributedlog in a more microservice-like
> way - we have the core as the stream store, the proxy for serving write and
> read traffic. It would be good that the transaction feature can be done in
> a similar way. So the architecture would be like this -
> 
> *[ write service ] [ read service ] [ transaction coordinator ]*
> *[ stream store
>]*
> 
> if people doesn't need the transaction feature, they can turn if off
> completely without any operational overhead.
> 
> Beside that, I have one general question - What is the major goal for this
> feature? Are you targeting on building a general XA transaction coordinator
> or just for supporting things like `copy-modify-write' style workflow?
> 
> 
> Thanks,
> Sijie
> 
> 
> 
> 
> 
> On Wed, Dec 28, 2016 at 1:12 PM, Xi Liu  wrote:
> 
>> Ping?
>> 
>> On Mon, Dec 19, 2016 at 8:28 AM, Xi Liu  wrote:
>> 
>>> Sijie,
>>> 
>>> No. I thought it might be easier for people to comment on a google doc to
>>> gather the initial feedback. I will put the content back to wiki page
>> once
>>> addressing the comments. Does that sound good to you?
>>> 
>>> And thank you in advance.
>>> 
>>> - Xi
>>> 
>>> 
>>> 
>>> On Sun, Dec 18, 2016 at 8:48 AM, Sijie Guo  wrote:
>>> 
 Hi Xi,
 
 sorry for late response. I will review it soon.
 
 regarding this, a separate question "are we going to use google doc
 instead
 of email thread for any discussion"? I am a bit worried that the
 discussion
 will become lost after moving to google doc. No idea on how other apache
 projects are doing.
 
 - Sijie
 
 On Wed, Dec 14, 2016 at 11:41 PM, Xi Liu  wrote:
 
> Hi all,
> 
> I finalized the first version of the design. This time I used a google
 doc
> so that it is easier for commenting and add a link the wiki page. I
>> will
> update this to the wiki page once we come to the finalized design.
> 
> https://docs.google.com/document/d/14Ns05M8Z5a6DF6fHmWQwISyD5jjeK
> bSIGgSzXuTI5BA/edit
> 
> Let me know if you have any questions. Appreciate your reviews!
> 
> - Xi
> 
> 
> 
> 
> 
> On Fri, Oct 28, 2016 at 7:58 AM, Leigh Stewart
> > wrote:
> 
>> Interesting proposal. A couple quick notes while you continue to
>> flesh
> this
>> out.
>> 
>> a. just to be sure - does this eliminate the need to save seqno with
>> checkpoint?
>> 
>> b. i.e. another way to describe this kind of improvement is "support
>> records (atomic writes) larger than 1MB", iiuc. the advantage being
>> it
>> avoids the baggage of transactions. disadvantages include inability
 to do
>> cross stream transactions, and flexibility (interleaving, etc) (are
 there
>> others?).
>> 
>> c. proxy use case is for supporting multiple writers - have you

Re: vote process for proposals?

[Sijie Guo]

Ping?

Xi, Jon, any updates about this? Do any of you want to drive this?

- Sijie



On Thu, Dec 15, 2016 at 11:18 AM, Sijie Guo 
wrote:

> Xi, Jon, are any of you interested in making a draft about about the
> proposal workflow?
>
> On Wed, Dec 14, 2016 at 6:14 PM, Jon Derrick 
> wrote:
>
> > I think it really worth having a voting proposal, as sometime I might
> lose
> > track of if a proposal is accepted or not and whether it is under
> > development.
> >
> > Beam's process looks promising. You can try to start with that.
> >
> > Another suggestion is it would be awesome if the DL jira queue can have
> new
> > type, called 'Proposal'. Then we can enforce the proposal workflow in the
> > jira.
> >
>
> I think it is possible to ask INFRA team to create a new jira type/workflow
> for us, if we can come up with more details. Can you tell us more about
> your thoughts?
>
> - Sijie
>
>
> >
> > On Tue, Dec 13, 2016 at 1:10 AM, Xi Liu  wrote:
> >
> > > Thank you Sijie. I feel it is good to have a voting process, so that it
> > > would be good to track if a proposal is accepted for developing or
> > > discarded due to any reasons. I will start with my proposal and see how
> > it
> > > is going with the community.
> > >
> > > - Xi
> > >
> > > On Thu, Dec 8, 2016 at 9:11 PM, Sijie Guo  wrote:
> > >
> > > > Xi, thank you for raising this up. I don't think we have a formal
> > process
> > > > for track proposals. I think we can learn the proposals from other
> > apache
> > > > projects. For example, beam has very nice documentation on
> contribution
> > > > guide (http://beam.incubator.apache.org/contribute/contribution-
> guide/
> > ).
> > > > We
> > > > probably can adopt it.
> > > >
> > > > I don't feel strong about the voting process. If it is easier for
> > making
> > > > conclusion on the proposal discussion, let's vote for any discussed
> > > > proposal.
> > > >
> > > > - Sijie
> > > >
> > > >
> > > >
> > > > On Thu, Dec 8, 2016 at 9:10 AM, Xi Liu  wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > It is great that we have a process to track/discuss proposals. but
> > the
> > > > > process is still a bit unclear to me. do we need a vote phase to
> > adopt
> > > > the
> > > > > proposals? and shall we document the process in wiki page?
> > > > >
> > > > > my basic understand about the process is:
> > > > >
> > > > > - create a proposal wiki page to describe the proposal
> > > > > - start the '[discussion]' email thread for the proposal
> > > > > - conversation will happen in the '[discussion]' email thread and
> the
> > > > wiki
> > > > > page will be refined
> > > > >
> > > > > I feel there will be a phase to decide whether this proposal will
> be
> > > > > accepted or discarded and update the state of the proposals. shall
> I
> > > vote
> > > > > DP-2?
> > > > >
> > > > > - Xi
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > - jderrick
> >
>

[GitHub] incubator-distributedlog pull request #89: DL-132: Enable check style for di...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/89


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/4] incubator-distributedlog git commit: DL-132: Enable check style for distributedlog service module.

[sijie]

DL-132: Enable check style for distributedlog service module.

Author: Xi Liu 

Reviewers: Sijie Guo 

Closes #89 from xiliuant/xi/checkstyle_service


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/1a30b0ce
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/1a30b0ce
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/1a30b0ce

Branch: refs/heads/master
Commit: 1a30b0ceb76f33eda08b611d97c150f45f239a95
Parents: 32a52a9
Author: Xi Liu 
Authored: Wed Jan 4 00:43:56 2017 -0800
Committer: Sijie Guo 
Committed: Wed Jan 4 00:43:56 2017 -0800

--
 .../resources/distributedlog/checkstyle.xml |   2 +-
 distributedlog-service/pom.xml  |  33 ++
 .../distributedlog/service/ClientUtils.java |   3 +
 .../service/DistributedLogCluster.java  |  78 ++---
 .../service/DistributedLogServer.java   |  64 ++--
 .../service/DistributedLogServerApp.java|  38 ++-
 .../service/DistributedLogServiceImpl.java  |  48 +--
 .../service/FatalErrorHandler.java  |   7 -
 .../distributedlog/service/MonitorService.java  |  29 +-
 .../service/MonitorServiceApp.java  |  16 +-
 .../service/ServerFeatureKeys.java  |   2 +-
 .../distributedlog/service/StatsFilter.java |   8 +-
 .../service/announcer/Announcer.java|   2 +-
 .../service/announcer/NOPAnnouncer.java |   3 +
 .../service/announcer/ServerSetAnnouncer.java   |  10 +-
 .../service/announcer/package-info.java |  21 ++
 .../service/balancer/Balancer.java  |   5 +
 .../service/balancer/BalancerTool.java  |  42 ++-
 .../service/balancer/ClusterBalancer.java   |  27 +-
 .../balancer/CountBasedStreamChooser.java   |   9 +-
 .../service/balancer/LimitedStreamChooser.java  |  10 +
 .../service/balancer/SimpleBalancer.java|  17 +-
 .../service/balancer/StreamChooser.java |   2 +-
 .../service/balancer/StreamMover.java   |   5 +-
 .../service/balancer/StreamMoverImpl.java   |   2 +-
 .../service/balancer/package-info.java  |  21 ++
 .../config/DefaultStreamConfigProvider.java |  29 +-
 .../service/config/ServerConfiguration.java | 106 ---
 .../config/ServiceStreamConfigProvider.java |  19 +-
 .../service/config/package-info.java|  21 ++
 .../distributedlog/service/package-info.java|  21 ++
 .../service/placement/EqualLoadAppraiser.java   |  26 +-
 .../placement/LeastLoadPlacementPolicy.java | 311 ++-
 .../service/placement/LoadAppraiser.java|  18 +-
 .../service/placement/PlacementPolicy.java  | 194 ++--
 .../placement/PlacementStateManager.java|  80 +++--
 .../service/placement/ServerLoad.java   | 227 +++---
 .../service/placement/StreamLoad.java   | 141 +
 .../placement/ZKPlacementStateManager.java  | 236 +++---
 .../service/placement/package-info.java |  21 ++
 .../service/stream/AbstractStreamOp.java|  23 +-
 .../service/stream/AbstractWriteOp.java |   9 +-
 .../service/stream/BulkWriteOp.java |  29 +-
 .../distributedlog/service/stream/DeleteOp.java |   7 +-
 .../service/stream/HeartbeatOp.java |  11 +-
 .../service/stream/ReleaseOp.java   |   7 +-
 .../distributedlog/service/stream/Stream.java   |   7 +-
 .../service/stream/StreamFactory.java   |   3 +
 .../service/stream/StreamFactoryImpl.java   |   3 +
 .../service/stream/StreamImpl.java  |  34 +-
 .../service/stream/StreamManager.java   |   4 +-
 .../service/stream/StreamManagerImpl.java   |   5 +-
 .../distributedlog/service/stream/StreamOp.java |   2 -
 .../service/stream/StreamOpStats.java   |   2 +-
 .../service/stream/TruncateOp.java  |  11 +-
 .../distributedlog/service/stream/WriteOp.java  |  17 +-
 .../service/stream/WriteOpWithPayload.java  |   3 +
 .../service/stream/admin/AdminOp.java   |   6 +-
 .../service/stream/admin/CreateOp.java  |   7 +-
 .../service/stream/admin/StreamAdminOp.java |   5 +-
 .../service/stream/admin/package-info.java  |  21 ++
 .../stream/limiter/DynamicRequestLimiter.java   |  21 +-
 .../stream/limiter/RequestLimiterBuilder.java   |  24 +-
 .../stream/limiter/ServiceRequestLimiter.java   |   8 +-
 .../stream/limiter/StreamAcquireLimiter.java|   6 +-
 .../stream/limiter/StreamRequestLimiter.java|   5 +-
 .../service/stream/limiter/package-info.java|  21 ++
 .../service/stream/package-info.java|  21 ++
 .../CacheableStreamPartitionConverter.java  |   3 +
 .../DelimiterStreamPartitionConverter.java  |   2 +-
 

[3/4] incubator-distributedlog git commit: DL-132: Enable check style for distributedlog service module.

[sijie]

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/1a30b0ce/distributedlog-service/src/main/java/com/twitter/distributedlog/service/config/ServiceStreamConfigProvider.java
--
diff --git 
a/distributedlog-service/src/main/java/com/twitter/distributedlog/service/config/ServiceStreamConfigProvider.java
 
b/distributedlog-service/src/main/java/com/twitter/distributedlog/service/config/ServiceStreamConfigProvider.java
index 111a874..0ee7db4 100644
--- 
a/distributedlog-service/src/main/java/com/twitter/distributedlog/service/config/ServiceStreamConfigProvider.java
+++ 
b/distributedlog-service/src/main/java/com/twitter/distributedlog/service/config/ServiceStreamConfigProvider.java
@@ -21,26 +21,27 @@ import com.google.common.base.Optional;
 import com.twitter.distributedlog.config.DynamicConfigurationFactory;
 import com.twitter.distributedlog.config.DynamicDistributedLogConfiguration;
 import com.twitter.distributedlog.service.streamset.StreamPartitionConverter;
-import org.apache.commons.configuration.ConfigurationException;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
 import java.io.File;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
+import org.apache.commons.configuration.ConfigurationException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * Provide per stream configuration to DistributedLog service layer.
  */
 public class ServiceStreamConfigProvider implements StreamConfigProvider {
-static final Logger LOG = 
LoggerFactory.getLogger(ServiceStreamConfigProvider.class);
+
+private static final Logger LOG = 
LoggerFactory.getLogger(ServiceStreamConfigProvider.class);
+
+private static final String CONFIG_EXTENSION = "conf";
 
 private final File configBaseDir;
 private final File defaultConfigFile;
 private final StreamPartitionConverter partitionConverter;
 private final DynamicConfigurationFactory configFactory;
 private final DynamicDistributedLogConfiguration defaultDynConf;
-private final static String CONFIG_EXTENSION = "conf";
 
 public ServiceStreamConfigProvider(String configPath,
String defaultConfigPath,
@@ -51,11 +52,13 @@ public class ServiceStreamConfigProvider implements 
StreamConfigProvider {
throws ConfigurationException {
 this.configBaseDir = new File(configPath);
 if (!configBaseDir.exists()) {
-throw new ConfigurationException("Stream configuration base 
directory " + configPath + " does not exist");
+throw new ConfigurationException("Stream configuration base 
directory "
++ configPath + " does not exist");
 }
 this.defaultConfigFile = new File(configPath);
 if (!defaultConfigFile.exists()) {
-throw new ConfigurationException("Stream configuration default 
config " + defaultConfigPath + " does not exist");
+throw new ConfigurationException("Stream configuration default 
config "
++ defaultConfigPath + " does not exist");
 }
 
 // Construct reloading default configuration

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/1a30b0ce/distributedlog-service/src/main/java/com/twitter/distributedlog/service/config/package-info.java
--
diff --git 
a/distributedlog-service/src/main/java/com/twitter/distributedlog/service/config/package-info.java
 
b/distributedlog-service/src/main/java/com/twitter/distributedlog/service/config/package-info.java
new file mode 100644
index 000..bb0026a
--- /dev/null
+++ 
b/distributedlog-service/src/main/java/com/twitter/distributedlog/service/config/package-info.java
@@ -0,0 +1,21 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/**
+ * DistributedLog Server Configurations.
+ */
+package com.twitter.distributedlog.service.config;
\ No newline at end of file

[GitHub] incubator-distributedlog issue #37: Introduce namespace resolver for validat...

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/37
  
@jderrickk ping?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-distributedlog pull request #71: DL-140 : Fix distributedlog-core ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/71


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

incubator-distributedlog git commit: DL-140: Fix distributedlog-core findbug inconsistent synchronization warings

[sijie]

Repository: incubator-distributedlog
Updated Branches:
  refs/heads/master 0711bc2fd -> 32a52a9f7


DL-140: Fix distributedlog-core findbug inconsistent synchronization warings

Author: xieliang 

Reviewers: Leigh Stewart 

Closes #71 from xieliang/DL-140


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/32a52a9f
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/32a52a9f
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/32a52a9f

Branch: refs/heads/master
Commit: 32a52a9f71cc86efeb5fa73576c20d8f392e0ce5
Parents: 0711bc2
Author: xieliang 
Authored: Wed Jan 4 00:41:56 2017 -0800
Committer: Sijie Guo 
Committed: Wed Jan 4 00:41:56 2017 -0800

--
 .../java/com/twitter/distributedlog/BKAbstractLogWriter.java | 4 ++--
 .../main/java/com/twitter/distributedlog/BKLogSegmentWriter.java | 2 +-
 .../java/com/twitter/distributedlog/EnvelopedEntryWriter.java| 2 +-
 .../src/main/java/com/twitter/distributedlog/MaxTxId.java| 2 +-
 .../java/com/twitter/distributedlog/lock/ZKDistributedLock.java  | 4 ++--
 5 files changed, 7 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/32a52a9f/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAbstractLogWriter.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAbstractLogWriter.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAbstractLogWriter.java
index b0cea24..d1c28d7 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAbstractLogWriter.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKAbstractLogWriter.java
@@ -54,7 +54,7 @@ abstract class BKAbstractLogWriter implements Closeable, 
AsyncCloseable, Abortab
 
 // States
 private Promise closePromise = null;
-private boolean forceRolling = false;
+private volatile boolean forceRolling = false;
 private boolean forceRecovery = false;
 
 // Truncation Related
@@ -531,7 +531,7 @@ abstract class BKAbstractLogWriter implements Closeable, 
AsyncCloseable, Abortab
 }
 
 @VisibleForTesting
-public synchronized void setForceRolling(boolean forceRolling) {
+public void setForceRolling(boolean forceRolling) {
 this.forceRolling = forceRolling;
 }
 

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/32a52a9f/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
index 8029f89..df64505 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/BKLogSegmentWriter.java
@@ -420,7 +420,7 @@ class BKLogSegmentWriter implements LogSegmentWriter, 
AddCallback, Runnable, Siz
  *
  * @return position-within-logsegment of the last written log record.
  */
-int getPositionWithinLogSegment() {
+synchronized int getPositionWithinLogSegment() {
 return positionWithinLogSegment;
 }
 

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/32a52a9f/distributedlog-core/src/main/java/com/twitter/distributedlog/EnvelopedEntryWriter.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/EnvelopedEntryWriter.java
 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/EnvelopedEntryWriter.java
index df56288..01a91ab 100644
--- 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/EnvelopedEntryWriter.java
+++ 
b/distributedlog-core/src/main/java/com/twitter/distributedlog/EnvelopedEntryWriter.java
@@ -176,7 +176,7 @@ class EnvelopedEntryWriter implements Writer {
 }
 
 @Override
-public DLSN finalizeTransmit(long lssn, long entryId) {
+public synchronized DLSN finalizeTransmit(long lssn, long entryId) {
 return new DLSN(lssn, entryId, count - 1);
 }
 

http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/32a52a9f/distributedlog-core/src/main/java/com/twitter/distributedlog/MaxTxId.java
--
diff --git 
a/distributedlog-core/src/main/java/com/twitter/distributedlog/MaxTxId.java 

incubator-distributedlog git commit: Improve merge pull request scripts

[sijie]

Repository: incubator-distributedlog
Updated Branches:
  refs/heads/master f607a48ff -> 0711bc2fd


Improve merge pull request scripts

- check if a pull request is approved or not
- reject merging if there is no approval
- handle cases that reviewers don't have name and email configured.

Author: Sijie Guo 

Reviewers: Franck Cuny 

Closes #93 from sijie/sijie/improve_merge_scripts


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/0711bc2f
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/0711bc2f
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/0711bc2f

Branch: refs/heads/master
Commit: 0711bc2fdc2afbb573ce42d9eecccf03e166f006
Parents: f607a48
Author: Sijie Guo 
Authored: Wed Jan 4 00:40:28 2017 -0800
Committer: Sijie Guo 
Committed: Wed Jan 4 00:40:28 2017 -0800

--
 scripts/dev/dl-merge-pr.py | 42 -
 scripts/dev/reviewers  | 18 ++
 2 files changed, 51 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0711bc2f/scripts/dev/dl-merge-pr.py
--
diff --git a/scripts/dev/dl-merge-pr.py b/scripts/dev/dl-merge-pr.py
index ca853d0..6c9d3e8 100755
--- a/scripts/dev/dl-merge-pr.py
+++ b/scripts/dev/dl-merge-pr.py
@@ -84,8 +84,9 @@ TEMP_BRANCH_PREFIX = 'PR_TOOL'
 RELEASE_BRANCH_PREFIX = ''
 
 DEV_BRANCH_NAME = 'master'
+DEFAULT_FIX_VERSION = os.environ.get("DEFAULT_FIX_VERSION", "0.4.0")
 
-def get_json(url):
+def get_json(url, preview_api = False):
   """
   Returns parsed JSON from an API call to the GitHub API.
   """
@@ -93,6 +94,8 @@ def get_json(url):
 request = urllib2.Request(url)
 if GITHUB_OAUTH_KEY:
   request.add_header('Authorization', 'token {0}'.format(GITHUB_OAUTH_KEY))
+if preview_api:
+  request.add_header('Accept', 
'application/vnd.github.black-cat-preview+json')
 return json.load(urllib2.urlopen(request))
   except urllib2.HTTPError as e:
 if 'X-RateLimit-Remaining' in e.headers and 
e.headers['X-RateLimit-Remaining'] == '0':
@@ -139,7 +142,7 @@ def clean_up():
 print('Restoring head pointer to {0}'.format(original_head))
 run_cmd(['git', 'checkout', original_head])
 
-  branches = run_cmd(['git', 'branch']).strip().split('\n')
+  branches = run_cmd(['git', 'branch']).replace(" ", "").split('\n')
 
   for branch in filter(lambda x: x.startswith(TEMP_BRANCH_PREFIX), branches):
 print('Deleting local branch {0}'.format(branch))
@@ -290,6 +293,7 @@ def cherry_pick(pr_num, merge_hash, default_branch):
 def fix_version_from_branch(branch, versions):
   # Note: Assumes this is a sorted (newest->oldest) list of un-released 
versions
   if branch == DEV_BRANCH_NAME:
+versions = filter(lambda x: x == DEFAULT_FIX_VERSION, versions)
 if len(versions) > 0:
   return versions[0]
 else:
@@ -328,9 +332,10 @@ def resolve_jira_issue(merge_branches, comment, jira_id):
 
   if cur_status == 'Resolved' or cur_status == 'Closed':
 fail('JIRA issue {0} already has status \'{1}\''.format(jira_id, 
cur_status))
-print ('=== JIRA {0} ==='.format(jira_id))
-print 
('summary\t\t{0}\nassignee\t{1}\nstatus\t\t{2}\nurl\t\t{3}/{4}\n'.format(
-  cur_summary, cur_assignee, cur_status, JIRA_BASE, jira_id))
+
+  print ('=== JIRA {0} ==='.format(jira_id))
+  print 
('summary\t\t{0}\nassignee\t{1}\nstatus\t\t{2}\nurl\t\t{3}/{4}\n'.format(
+cur_summary, cur_assignee, cur_status, JIRA_BASE, jira_id))
 
   versions = asf_jira.project_versions(CAPITALIZED_PROJECT_NAME)
   versions = sorted(versions, key=lambda x: x.name, reverse=True)
@@ -418,21 +423,40 @@ def get_reviewers(pr_num):
   """
   Gets a candidate list of reviewers that have commented on the PR with '+1' 
or 'LGTM'
   """
-  approval_msgs = ['+1', 'lgtm']
+  reviewers_ids = set()
 
+  approval_msgs = ['+1', 'lgtm']
   pr_comments = get_json('{0}/issues/{1}/comments'.format(GITHUB_API_BASE, 
pr_num))
-
-  reviewers_ids = set()
   for comment in pr_comments:
 for approval_msg in approval_msgs:
   if approval_msg in comment['body'].lower():
 reviewers_ids.add(comment['user']['login'])
 
+  approval_review_states = ['approved']
+  pr_reviews = get_json('{0}/pulls/{1}/reviews'.format(GITHUB_API_BASE, 
pr_num), True)
+  for review in pr_reviews:
+for approval_state in approval_review_states:
+  if approval_state in review['state'].lower():
+reviewers_ids.add(review['user']['login'])
+
+  if len(reviewers_ids) == 0:
+fail("No approvals found in this pull request")
+
+  dir_path = os.path.dirname(os.path.realpath(__file__))
+  

[GitHub] incubator-distributedlog issue #89: DL-132: Enable check style for distribut...

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/89
  
@xiliuant looks good to me. +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[Review] The first release of Apache DistributedLog

[Sijie Guo]

I drafted a blog post for announcing the first apache release. Here is the
draft. Please help review it :D

https://docs.google.com/document/d/1IXVmP2cHkf4ydeUHUJN9p5ZWTpA1auwBhfqMnYBu4A0/edit

- Sijie

Re: Git Workflow on Windows

[Sijie Guo]

We definitely want to support running distributedlog in other environments.
And in theory, it is a Java system, it should be able to run on different
platforms. But as what Franck mentioned, the major problem is most of us
don't have a windows environment :/ I tried to have a Jenkins job or Travis
CI job to build distributedlog on windows. But unfortunately, it is
inconvenient for me to debug what it failed.

Based on the conversation in the jiras, it seems that the major problems
are a compilation problem and a symlink problem. So fixing these two should
be able to compile and run distributedlog on windows. However, in order to
validate the performance on windows, we might seek for helps from the
people who have more experiences on windows.

- Sijie

On Tue, Jan 3, 2017 at 4:56 PM, Franck Cuny  wrote:

> Hello Tracy,
>
> could you paste the complete output of the command you run somewhere ?
>
> As far as I know, we haven't tried to build DLog on a windows system, and I
> don't think we've made any effort to support this so far.
>
> On Tue, Jan 3, 2017 at 11:35 AM Tracy Martin 
> wrote:
>
> > I am fixing DL-28 after the merge, and I am following the "Git Workflow"
> > described here:
> >
> > https://cwiki.apache.org/confluence/display/DL/
> Contributing+to+DistributedLog
> >
> > I am getting many failures running step 4
> >
> > mvn clean apache-rat:check package findbugs:check
> >
> > from my Windows desktop.
> >
> > I know the error received in DL-28 was related to running the build on
> > Windows instead of Unix.  Many of the errors that I see in the tests
> point
> > to a Windows only problem.  Most of the errors reports are "Path must
> start
> > with / character".
> >
> > I know when I was first reading about DistributedLog that it required
> > Unix.  Is the plan to only run from Unix systems?  If so, I will look at
> > creating a Unix environment for building, testing, and running.
> > This e-mail and any attachments, contain Genuine Parts Company
> > confidential information that is proprietary, privileged, and protected
> by
> > applicable laws. If you have received this message in error and are not
> the
> > intended recipient, you should not retain, distribute, disclose or use
> any
> > of this information and you should destroy this e-mail, any attachments
> or
> > copies therein forthwith. Please notify the sender immediately by e-mail
> if
> > you have received this e-mail in error.
> >
> > "This e-mail and any attachments, contain Genuine Parts Company
> > confidential information that is proprietary, privileged, and protected
> by
> > applicable laws. If you have received this message in error and are not
> the
> > intended recipient, you should not retain, distribute, disclose or use
> any
> > of this information and you should destroy this e-mail, any attachments
> or
> > copies therein forthwith. Please notify the sender immediately by e-mail
> if
> > you have received this e-mail in error."
> >
> --
> -franck
>