[GitHub] bookkeeper issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@jvrao yes, that's the idea. 

I am +0 on this. It is fine for me to adding an option, although I still 
don't think it is the right approach to do this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066032#comment-16066032
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@jvrao yes, that's the idea. 

I am +0 on this. It is fine for me to adding an option, although I still 
don't think it is the right approach to do this.


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] bookkeeper issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@jvrao @reddycharan 

one more question, most of the comments are around the performance on 
journal directories. but it turns out this change has nothing to do with the 
journal directories. The change applies only on the ledger directories and 
index directories. if that's the case, I don't see a reason to have this 
configuration. because random I/O anyway exists on ledger and index 
directories, there isn't any performance difference between configuring one 
directory per disk partition or multiple directories per disk partition. 
Filesystem I/O scheduling will handle either case in the same way. so this 
doesn't sound like an operation error as what @jvrao said.

so back to the question, what is the problem that this pull request want to 
address? 

- if the problem is about the mis-reported disk metrics, this can be done 
smarter by de-duplicating the directories within ledger disks and report the 
per disk partition metrics.
- if the problem is about performance, multiple ledger/index disks on same 
partition doesn't have any performance difference with one ledger/index disk 
per disk partition. this doesn't sound like an operation error that needs to be 
fixed.
- if the problem is about performance on journal disks, I am fine with such 
setting. but this pull request need to improve to address the right problem.





---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066056#comment-16066056
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@jvrao @reddycharan 

one more question, most of the comments are around the performance on 
journal directories. but it turns out this change has nothing to do with the 
journal directories. The change applies only on the ledger directories and 
index directories. if that's the case, I don't see a reason to have this 
configuration. because random I/O anyway exists on ledger and index 
directories, there isn't any performance difference between configuring one 
directory per disk partition or multiple directories per disk partition. 
Filesystem I/O scheduling will handle either case in the same way. so this 
doesn't sound like an operation error as what @jvrao said.

so back to the question, what is the problem that this pull request want to 
address? 

- if the problem is about the mis-reported disk metrics, this can be done 
smarter by de-duplicating the directories within ledger disks and report the 
per disk partition metrics.
- if the problem is about performance, multiple ledger/index disks on same 
partition doesn't have any performance difference with one ledger/index disk 
per disk partition. this doesn't sound like an operation error that needs to be 
fixed.
- if the problem is about performance on journal disks, I am fine with such 
setting. but this pull request need to improve to address the right problem.





> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Write retry issues with ZooKeeperClient

[Sijie Guo]

On Tue, Jun 27, 2017 at 4:27 PM, Sam Just  wrote:

> Should EXPIRED be considered a recoverable error for retry purposes?
> Retrying in that case would mean that operations which might have been
> submitted under the assumption that ephemeral nodes were still present
> would be retried after the ephemeral nodes disappeared.  Don't all users
> have special handling for EXPIRED anyway?
>

If a request is failed with expired, that means the request was never
attempted. so it is a recoverable error to retry.
When a request is failed with expired, that means the session that this
request used is also expired and all ephemeral nodes associated with this
session will also be deleted. so it is a safe situation for retries.


> -Sam
>
> On Tue, Jun 27, 2017 at 4:08 PM, Sijie Guo  wrote:
>
> > On Tue, Jun 27, 2017 at 2:36 PM, Sam Just  wrote:
> >
> > > On Tue, Jun 27, 2017 at 2:29 PM, Sijie Guo  wrote:
> > >
> > > > On Tue, Jun 27, 2017 at 10:18 AM, Sam Just 
> > wrote:
> > > >
> > > > > JV: What do you mean by "May not be perfect for negative testing"?
> > > > >
> > > > > I don't think there's anything inevitable about this particular
> class
> > > of
> > > > > behavior.  ZK could have chosen to avoid this problem entirely by
> > doing
> > > > > duplicate op detection server-side with a per-session transaction
> > log.
> > > > >
> > > > > Since it doesn't, we'll need to solve it ourselves.
> > > > > ZkLedgerUnderreplicationManager relies on either getting success
> or
> > > > > NodeExists on the ephemeral lock node to determine whether a
> > particular
> > > > > ReplicationWorker is responsible for replicating a particular
> ledger.
> > > I
> > > > > haven't reproduced this one, but it seems to me that two workers
> > could
> > > > both
> > > > > try to replicate the same ledger and *both get NodeExists* on the
> > lock
> > > > > node.  This would leave the ledger locked for replication until
> > > whichever
> > > > > one actually wrote the node restarts.
> > > > >
> > > > > Users like the above are pretty easy to fix.  One option would be
> to
> > > > simply
> > > > > include with the write a payload including a nonce for the client.
> > > Upon
> > > > a
> > > > > ConnectionLoss event, we read the node and determine whether we
> > > "won".  I
> > > > > think ledger creation probably falls into the same category, the
> > > metadata
> > > > > could include an identifier for the creator.
> > > > >
> > > >
> > > > for ephemeral znode, you don't have to add extra payload.
> > > >
> > > > in the retry logic,
> > > >
> > > > - catch NodeExists exception
> > > > - call exists to check the znode. you can get the Stat (
> > > > https://zookeeper.apache.org/doc/r3.4.6/api/org/apache/
> > > > zookeeper/data/Stat.html#getEphemeralOwner()
> > > > ) of this znode.
> > > > - you can compare the Stat#getEphmeralOwner with the client's current
> > > > session id. if they match, the node is created by this session,
> > otherwise
> > > > this node is created by other session.
> > > >
> > > >
> > > Ah, I'll do that then, much easier.
> > >
> > >
> > > >
> > > > >
> > > > > I'm less sure about AbstractZkLedgerManager.writeLedgerMetadata.
> It
> > > > > relies
> > > > > on setData success/BadVersion atomicity to ensure consistent rmw.
> > > >
> > > > It's
> > > > > certainly solvable by including a version vector, but I think
> there's
> > > > > probably a simpler way to do it.  I'm not sure under what
> > circumstances
> > > > > there even *can* be racing writes to the metadata -- is the only
> case
> > > > that
> > > > > matters concurrent updates between the writer and a single
> > replication
> > > > > worker?
> > > > >
> > > >
> > > > I don't think it can be addressed by a version vector, there is no
> > public
> > > > *history* about who changed what version.
> > > > But the question is why do you need to address this
> > #writeLedgerMetadata
> > > > inside the ZooKeeperClient retry loop. I think there is already a
> logic
> > > on
> > > > #writeLedgerMetadata to resolve conflicts due to BadVersion.
> > > > The LedgerHandle will re-read the ledger metadata when encountering
> > > version
> > > > conflicts.
> > > >
> > > >
> > > That's the part I'm unsure of.  I know it has handling for BadVersion,
> I
> > > don't know
> > > whether it's still correct if you get a BadVersion even though your
> write
> > > actually
> > > succeeded, I need to look into that more.  Also, I'm not at all
> > suggesting
> > > we handle
> > > this in the retry loop, I'm suggesting that for non-idempotent writes,
> we
> > > should not
> > > retry in ZooKeeperClient and instead propagate the ConnectionLoss error
> > and
> > > let the caller deal with it.
> > >
> >
> > +1 I agree with no retrying non-idempotent writes.
> >
> >
> > > -Sam
> > >
> > >
> > > >
> > > > >
> > > > > The outline is probably:
> > > > > 1) Add a way to do retries automatically on *unconditional* creates
> > and
> > > > > deletes (there are some users that don't care).
> > > > > 2) Au

[GitHub] bookkeeper issue #210: BOOKKEEPER-1100: Add Http Server for Bookkeeper

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/210
  
regarding the servlet, I agree with @eolivelli , we probably should define 
a standard API and allow it running with different web container. @yzang : do 
finagle and vertx support servlet?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-1100) Add Http Endpoint for Bookkeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066090#comment-16066090
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1100:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/210
  
regarding the servlet, I agree with @eolivelli , we probably should define 
a standard API and allow it running with different web container. @yzang : do 
finagle and vertx support servlet?


> Add Http Endpoint for Bookkeeper
> 
>
> Key: BOOKKEEPER-1100
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1100
> Project: Bookkeeper
>  Issue Type: New Feature
>  Components: bookkeeper-server
>Affects Versions: 4.5.0
>Reporter: Yiming Zang
>Assignee: Yiming Zang
> Fix For: 4.5.0
>
>
> Add a Http Server Component for Bookkeeper.
> It would be great that Bookkeeper can have an Http Server to expose some 
> useful APIs. We can expose and manage the server status, server 
> configuration, lifecycle states, query or trigger auto recovery through HTTP 
> Endpoint, or even trigger GC and compaction over Http Endpoint.
> This ticket is mainly to add a Http framework for Bookkeeper, so that we can 
> extend more useful APIs based on the existing Http framework.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

BookKeeper repo has been moved into gitbox

[Jia Zhai]

Hi All,
Thanks to the help from INFRA team, now bookkeeper repo has been moved into
gitbox.

The new URL (on our side) is
https://gitbox.apache.org/repos/asf/bookkeeper.git
For committers, please go through the setup at
https://gitbox.apache.org/setup/

Thanks.

Re: BookKeeper repo has been moved into gitbox

[Enrico Olivelli]

Hi Jia,
this is a very important change !!

Do you now what will happen if any committers uses
git-ws-us.apache.org as usual ?
is there any chance to break things ?

we should immediately update the wiki page
https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission

let's wait for Sijie acknowledge

Thank you
-- Enrico



2017-06-28 12:24 GMT+02:00 Jia Zhai :
> Hi All,
> Thanks to the help from INFRA team, now bookkeeper repo has been moved into
> gitbox.
>
> The new URL (on our side) is
> https://gitbox.apache.org/repos/asf/bookkeeper.git
> For committers, please go through the setup at
> https://gitbox.apache.org/setup/
>
> Thanks.

Re: BookKeeper repo has been moved into gitbox

[Enrico Olivelli]

This is great
it seems that we have also a GitHub team for bookkeeper committers
https://github.com/orgs/apache/teams/bookkeeper-committers/members

-- Enrico


2017-06-28 12:29 GMT+02:00 Enrico Olivelli :
> Hi Jia,
> this is a very important change !!
>
> Do you now what will happen if any committers uses
> git-ws-us.apache.org as usual ?
> is there any chance to break things ?
>
> we should immediately update the wiki page
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission
>
> let's wait for Sijie acknowledge
>
> Thank you
> -- Enrico
>
>
>
> 2017-06-28 12:24 GMT+02:00 Jia Zhai :
>> Hi All,
>> Thanks to the help from INFRA team, now bookkeeper repo has been moved into
>> gitbox.
>>
>> The new URL (on our side) is
>> https://gitbox.apache.org/repos/asf/bookkeeper.git
>> For committers, please go through the setup at
>> https://gitbox.apache.org/setup/
>>
>> Thanks.

Re: BookKeeper repo has been moved into gitbox

[Enrico Olivelli]

I am going on with trials in order to discover our new space before
hitting bad problems.

If you try to access the old git repo you will get

[enrico.olivelli@localhost bookkeeper]$ git pull apache master
fatal: repository
'https://git-wip-us.apache.org/repos/asf/bookkeeper.git/' not found

As the old repo is not working anymore

I have updated the master Jenkins job
https://builds.apache.org/job/bookkeeper-master/

I have just updated the wiki page as well
https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission

-- Enrico

2017-06-28 13:20 GMT+02:00 Enrico Olivelli :
> This is great
> it seems that we have also a GitHub team for bookkeeper committers
> https://github.com/orgs/apache/teams/bookkeeper-committers/members
>
> -- Enrico
>
>
> 2017-06-28 12:29 GMT+02:00 Enrico Olivelli :
>> Hi Jia,
>> this is a very important change !!
>>
>> Do you now what will happen if any committers uses
>> git-ws-us.apache.org as usual ?
>> is there any chance to break things ?
>>
>> we should immediately update the wiki page
>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission
>>
>> let's wait for Sijie acknowledge
>>
>> Thank you
>> -- Enrico
>>
>>
>>
>> 2017-06-28 12:24 GMT+02:00 Jia Zhai :
>>> Hi All,
>>> Thanks to the help from INFRA team, now bookkeeper repo has been moved into
>>> gitbox.
>>>
>>> The new URL (on our side) is
>>> https://gitbox.apache.org/repos/asf/bookkeeper.git
>>> For committers, please go through the setup at
>>> https://gitbox.apache.org/setup/
>>>
>>> Thanks.

Build failed in Jenkins: bookkeeper-master #1798

[Apache Jenkins Server]

See 

--
[...truncated 1.02 MB...]
Running org.apache.bookkeeper.test.LedgerCreateDeleteTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.559 sec - in 
org.apache.bookkeeper.test.LedgerCreateDeleteTest
Running org.apache.bookkeeper.test.ConfigurationTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.382 sec - in 
org.apache.bookkeeper.test.ConfigurationTest
Running org.apache.bookkeeper.test.BookieFailureTest
Tests run: 80, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.47 sec - in 
org.apache.bookkeeper.test.BookieFailureTest
Running org.apache.bookkeeper.test.LocalBookiesRegistryTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.275 sec - in 
org.apache.bookkeeper.test.LocalBookiesRegistryTest
Running org.apache.bookkeeper.test.ConditionalSetTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.228 sec - in 
org.apache.bookkeeper.test.ConditionalSetTest
Running org.apache.bookkeeper.test.BookieJournalRollingTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.647 sec - in 
org.apache.bookkeeper.test.BookieJournalRollingTest
Running org.apache.bookkeeper.test.BookieZKExpireTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.481 sec - in 
org.apache.bookkeeper.test.BookieZKExpireTest
Running 
org.apache.bookkeeper.client.BookKeeperDiskSpaceWeightedLedgerPlacementTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.932 sec - in 
org.apache.bookkeeper.client.BookKeeperDiskSpaceWeightedLedgerPlacementTest
Running org.apache.bookkeeper.client.TestSpeculativeRead
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.727 sec - in 
org.apache.bookkeeper.client.TestSpeculativeRead
Running org.apache.bookkeeper.client.TestBookieHealthCheck
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.021 sec - in 
org.apache.bookkeeper.client.TestBookieHealthCheck
Running org.apache.bookkeeper.client.TestGetBookieInfoTimeout
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.818 sec - in 
org.apache.bookkeeper.client.TestGetBookieInfoTimeout
Running org.apache.bookkeeper.client.RoundRobinDistributionScheduleTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.286 sec - in 
org.apache.bookkeeper.client.RoundRobinDistributionScheduleTest
Running org.apache.bookkeeper.client.UpdateLedgerCmdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.457 sec - in 
org.apache.bookkeeper.client.UpdateLedgerCmdTest
Running org.apache.bookkeeper.client.TestLedgerFragmentReplication
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.102 sec - in 
org.apache.bookkeeper.client.TestLedgerFragmentReplication
Running org.apache.bookkeeper.client.BookieRecoveryTest
Tests run: 120, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 92.909 sec - 
in org.apache.bookkeeper.client.BookieRecoveryTest
Running org.apache.bookkeeper.client.LedgerCloseTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.646 sec - in 
org.apache.bookkeeper.client.LedgerCloseTest
Running org.apache.bookkeeper.client.LedgerRecoveryTest
Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.926 sec - 
in org.apache.bookkeeper.client.LedgerRecoveryTest
Running org.apache.bookkeeper.client.TestTryReadLastConfirmed
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.46 sec - in 
org.apache.bookkeeper.client.TestTryReadLastConfirmed
Running org.apache.bookkeeper.client.UpdateLedgerOpTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.355 sec - in 
org.apache.bookkeeper.client.UpdateLedgerOpTest
Running org.apache.bookkeeper.client.TestWeightedRandomSelection
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.933 sec - in 
org.apache.bookkeeper.client.TestWeightedRandomSelection
Running org.apache.bookkeeper.client.TestBookieWatcher
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.583 sec - in 
org.apache.bookkeeper.client.TestBookieWatcher
Running org.apache.bookkeeper.client.ParallelLedgerRecoveryTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.393 sec - 
in org.apache.bookkeeper.client.ParallelLedgerRecoveryTest
Running org.apache.bookkeeper.client.BookieWriteLedgerTest
Tests run: 120, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 108.308 sec - 
in org.apache.bookkeeper.client.BookieWriteLedgerTest
Running org.apache.bookkeeper.client.GenericEnsemblePlacementPolicyTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.902 sec - in 
org.apache.bookkeeper.client.GenericEnsemblePlacementPolicyTest
Running org.apache.bookkeeper.client.TestAddEntryQuorumTimeout
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.386 sec - in 
org.apache.bookkeeper.client.TestAddEntryQuorumTimeout
Running 

Re: BookKeeper repo has been moved into gitbox

[Jia Zhai]

Thanks Enrico for the confirm. Yes, we may need some changes, especially
the repo address.

On Wed, Jun 28, 2017 at 8:07 PM, Enrico Olivelli 
wrote:

> I am going on with trials in order to discover our new space before
> hitting bad problems.
>
> If you try to access the old git repo you will get
>
> [enrico.olivelli@localhost bookkeeper]$ git pull apache master
> fatal: repository
> 'https://git-wip-us.apache.org/repos/asf/bookkeeper.git/' not found
>
> As the old repo is not working anymore
>
> I have updated the master Jenkins job
> https://builds.apache.org/job/bookkeeper-master/
>
> I have just updated the wiki page as well
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission
>
> -- Enrico
>
> 2017-06-28 13:20 GMT+02:00 Enrico Olivelli :
> > This is great
> > it seems that we have also a GitHub team for bookkeeper committers
> > https://github.com/orgs/apache/teams/bookkeeper-committers/members
> >
> > -- Enrico
> >
> >
> > 2017-06-28 12:29 GMT+02:00 Enrico Olivelli :
> >> Hi Jia,
> >> this is a very important change !!
> >>
> >> Do you now what will happen if any committers uses
> >> git-ws-us.apache.org as usual ?
> >> is there any chance to break things ?
> >>
> >> we should immediately update the wiki page
> >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission
> >>
> >> let's wait for Sijie acknowledge
> >>
> >> Thank you
> >> -- Enrico
> >>
> >>
> >>
> >> 2017-06-28 12:24 GMT+02:00 Jia Zhai :
> >>> Hi All,
> >>> Thanks to the help from INFRA team, now bookkeeper repo has been moved
> into
> >>> gitbox.
> >>>
> >>> The new URL (on our side) is
> >>> https://gitbox.apache.org/repos/asf/bookkeeper.git
> >>> For committers, please go through the setup at
> >>> https://gitbox.apache.org/setup/
> >>>
> >>> Thanks.
>

Re: BookKeeper repo has been moved into gitbox

[Enrico Olivelli]

Jia,
I already did it.
I'm not aware of other places to change the link to the repo.
Do you have other ideas for the switch ?
Cheers
-- Enrico


2017-06-28 14:51 GMT+02:00 Jia Zhai :
> Thanks Enrico for the confirm. Yes, we may need some changes, especially
> the repo address.
>
> On Wed, Jun 28, 2017 at 8:07 PM, Enrico Olivelli 
> wrote:
>
>> I am going on with trials in order to discover our new space before
>> hitting bad problems.
>>
>> If you try to access the old git repo you will get
>>
>> [enrico.olivelli@localhost bookkeeper]$ git pull apache master
>> fatal: repository
>> 'https://git-wip-us.apache.org/repos/asf/bookkeeper.git/' not found
>>
>> As the old repo is not working anymore
>>
>> I have updated the master Jenkins job
>> https://builds.apache.org/job/bookkeeper-master/
>>
>> I have just updated the wiki page as well
>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission
>>
>> -- Enrico
>>
>> 2017-06-28 13:20 GMT+02:00 Enrico Olivelli :
>> > This is great
>> > it seems that we have also a GitHub team for bookkeeper committers
>> > https://github.com/orgs/apache/teams/bookkeeper-committers/members
>> >
>> > -- Enrico
>> >
>> >
>> > 2017-06-28 12:29 GMT+02:00 Enrico Olivelli :
>> >> Hi Jia,
>> >> this is a very important change !!
>> >>
>> >> Do you now what will happen if any committers uses
>> >> git-ws-us.apache.org as usual ?
>> >> is there any chance to break things ?
>> >>
>> >> we should immediately update the wiki page
>> >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission
>> >>
>> >> let's wait for Sijie acknowledge
>> >>
>> >> Thank you
>> >> -- Enrico
>> >>
>> >>
>> >>
>> >> 2017-06-28 12:24 GMT+02:00 Jia Zhai :
>> >>> Hi All,
>> >>> Thanks to the help from INFRA team, now bookkeeper repo has been moved
>> into
>> >>> gitbox.
>> >>>
>> >>> The new URL (on our side) is
>> >>> https://gitbox.apache.org/repos/asf/bookkeeper.git
>> >>> For committers, please go through the setup at
>> >>> https://gitbox.apache.org/setup/
>> >>>
>> >>> Thanks.
>>

Re: BookKeeper repo has been moved into gitbox

[Venkateswara Rao Jujjuri]

What is this change about? Can we still access and use
https://github.com/apache/bookkeeper.git ?

On Wed, Jun 28, 2017 at 5:53 AM, Enrico Olivelli 
wrote:

> Jia,
> I already did it.
> I'm not aware of other places to change the link to the repo.
> Do you have other ideas for the switch ?
> Cheers
> -- Enrico
>
>
> 2017-06-28 14:51 GMT+02:00 Jia Zhai :
> > Thanks Enrico for the confirm. Yes, we may need some changes, especially
> > the repo address.
> >
> > On Wed, Jun 28, 2017 at 8:07 PM, Enrico Olivelli 
> > wrote:
> >
> >> I am going on with trials in order to discover our new space before
> >> hitting bad problems.
> >>
> >> If you try to access the old git repo you will get
> >>
> >> [enrico.olivelli@localhost bookkeeper]$ git pull apache master
> >> fatal: repository
> >> 'https://git-wip-us.apache.org/repos/asf/bookkeeper.git/' not found
> >>
> >> As the old repo is not working anymore
> >>
> >> I have updated the master Jenkins job
> >> https://builds.apache.org/job/bookkeeper-master/
> >>
> >> I have just updated the wiki page as well
> >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission
> >>
> >> -- Enrico
> >>
> >> 2017-06-28 13:20 GMT+02:00 Enrico Olivelli :
> >> > This is great
> >> > it seems that we have also a GitHub team for bookkeeper committers
> >> > https://github.com/orgs/apache/teams/bookkeeper-committers/members
> >> >
> >> > -- Enrico
> >> >
> >> >
> >> > 2017-06-28 12:29 GMT+02:00 Enrico Olivelli :
> >> >> Hi Jia,
> >> >> this is a very important change !!
> >> >>
> >> >> Do you now what will happen if any committers uses
> >> >> git-ws-us.apache.org as usual ?
> >> >> is there any chance to break things ?
> >> >>
> >> >> we should immediately update the wiki page
> >> >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> Patch+Sumission
> >> >>
> >> >> let's wait for Sijie acknowledge
> >> >>
> >> >> Thank you
> >> >> -- Enrico
> >> >>
> >> >>
> >> >>
> >> >> 2017-06-28 12:24 GMT+02:00 Jia Zhai :
> >> >>> Hi All,
> >> >>> Thanks to the help from INFRA team, now bookkeeper repo has been
> moved
> >> into
> >> >>> gitbox.
> >> >>>
> >> >>> The new URL (on our side) is
> >> >>> https://gitbox.apache.org/repos/asf/bookkeeper.git
> >> >>> For committers, please go through the setup at
> >> >>> https://gitbox.apache.org/setup/
> >> >>>
> >> >>> Thanks.
> >>
>



-- 
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi

Re: BookKeeper repo has been moved into gitbox

[Enrico Olivelli]

JV,
GitHub at the moment is only a mirror for the official Git repo.
When committing we must push to gitbox now
For clones/readonly work/pull requests we are using GitHub as before

-- Enrico

2017-06-28 15:18 GMT+02:00 Venkateswara Rao Jujjuri :
> What is this change about? Can we still access and use
> https://github.com/apache/bookkeeper.git ?
>
> On Wed, Jun 28, 2017 at 5:53 AM, Enrico Olivelli 
> wrote:
>
>> Jia,
>> I already did it.
>> I'm not aware of other places to change the link to the repo.
>> Do you have other ideas for the switch ?
>> Cheers
>> -- Enrico
>>
>>
>> 2017-06-28 14:51 GMT+02:00 Jia Zhai :
>> > Thanks Enrico for the confirm. Yes, we may need some changes, especially
>> > the repo address.
>> >
>> > On Wed, Jun 28, 2017 at 8:07 PM, Enrico Olivelli 
>> > wrote:
>> >
>> >> I am going on with trials in order to discover our new space before
>> >> hitting bad problems.
>> >>
>> >> If you try to access the old git repo you will get
>> >>
>> >> [enrico.olivelli@localhost bookkeeper]$ git pull apache master
>> >> fatal: repository
>> >> 'https://git-wip-us.apache.org/repos/asf/bookkeeper.git/' not found
>> >>
>> >> As the old repo is not working anymore
>> >>
>> >> I have updated the master Jenkins job
>> >> https://builds.apache.org/job/bookkeeper-master/
>> >>
>> >> I have just updated the wiki page as well
>> >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission
>> >>
>> >> -- Enrico
>> >>
>> >> 2017-06-28 13:20 GMT+02:00 Enrico Olivelli :
>> >> > This is great
>> >> > it seems that we have also a GitHub team for bookkeeper committers
>> >> > https://github.com/orgs/apache/teams/bookkeeper-committers/members
>> >> >
>> >> > -- Enrico
>> >> >
>> >> >
>> >> > 2017-06-28 12:29 GMT+02:00 Enrico Olivelli :
>> >> >> Hi Jia,
>> >> >> this is a very important change !!
>> >> >>
>> >> >> Do you now what will happen if any committers uses
>> >> >> git-ws-us.apache.org as usual ?
>> >> >> is there any chance to break things ?
>> >> >>
>> >> >> we should immediately update the wiki page
>> >> >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>> Patch+Sumission
>> >> >>
>> >> >> let's wait for Sijie acknowledge
>> >> >>
>> >> >> Thank you
>> >> >> -- Enrico
>> >> >>
>> >> >>
>> >> >>
>> >> >> 2017-06-28 12:24 GMT+02:00 Jia Zhai :
>> >> >>> Hi All,
>> >> >>> Thanks to the help from INFRA team, now bookkeeper repo has been
>> moved
>> >> into
>> >> >>> gitbox.
>> >> >>>
>> >> >>> The new URL (on our side) is
>> >> >>> https://gitbox.apache.org/repos/asf/bookkeeper.git
>> >> >>> For committers, please go through the setup at
>> >> >>> https://gitbox.apache.org/setup/
>> >> >>>
>> >> >>> Thanks.
>> >>
>>
>
>
>
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi

Re: Write retry issues with ZooKeeperClient

[Venkateswara Rao Jujjuri]

Current retry logic is transparent to the type of the operation and is down
the stack which has no understanding of the operation type. If we want to
differentiate between idempotent and non-idempotent, we may need to insert
that logic into the retry logic. Other option is to handle retry at higher
level were we have operational context. Both seems little messy. Looks like
we can take care of ephemeral node issue as explained above. For ledger
nodes and metadata updates we need to come up with possible failures and
ramifications of getting node exists then we can figure out how to handle
that.

On Tue, Jun 27, 2017 at 4:08 PM, Sijie Guo  wrote:

> On Tue, Jun 27, 2017 at 2:36 PM, Sam Just  wrote:
>
> > On Tue, Jun 27, 2017 at 2:29 PM, Sijie Guo  wrote:
> >
> > > On Tue, Jun 27, 2017 at 10:18 AM, Sam Just 
> wrote:
> > >
> > > > JV: What do you mean by "May not be perfect for negative testing"?
> > > >
> > > > I don't think there's anything inevitable about this particular class
> > of
> > > > behavior.  ZK could have chosen to avoid this problem entirely by
> doing
> > > > duplicate op detection server-side with a per-session transaction
> log.
> > > >
> > > > Since it doesn't, we'll need to solve it ourselves.
> > > > ZkLedgerUnderreplicationManager relies on either getting success or
> > > > NodeExists on the ephemeral lock node to determine whether a
> particular
> > > > ReplicationWorker is responsible for replicating a particular ledger.
> > I
> > > > haven't reproduced this one, but it seems to me that two workers
> could
> > > both
> > > > try to replicate the same ledger and *both get NodeExists* on the
> lock
> > > > node.  This would leave the ledger locked for replication until
> > whichever
> > > > one actually wrote the node restarts.
> > > >
> > > > Users like the above are pretty easy to fix.  One option would be to
> > > simply
> > > > include with the write a payload including a nonce for the client.
> > Upon
> > > a
> > > > ConnectionLoss event, we read the node and determine whether we
> > "won".  I
> > > > think ledger creation probably falls into the same category, the
> > metadata
> > > > could include an identifier for the creator.
> > > >
> > >
> > > for ephemeral znode, you don't have to add extra payload.
> > >
> > > in the retry logic,
> > >
> > > - catch NodeExists exception
> > > - call exists to check the znode. you can get the Stat (
> > > https://zookeeper.apache.org/doc/r3.4.6/api/org/apache/
> > > zookeeper/data/Stat.html#getEphemeralOwner()
> > > ) of this znode.
> > > - you can compare the Stat#getEphmeralOwner with the client's current
> > > session id. if they match, the node is created by this session,
> otherwise
> > > this node is created by other session.
> > >
> > >
> > Ah, I'll do that then, much easier.
> >
> >
> > >
> > > >
> > > > I'm less sure about AbstractZkLedgerManager.writeLedgerMetadata.  It
> > > > relies
> > > > on setData success/BadVersion atomicity to ensure consistent rmw.
> > >
> > > It's
> > > > certainly solvable by including a version vector, but I think there's
> > > > probably a simpler way to do it.  I'm not sure under what
> circumstances
> > > > there even *can* be racing writes to the metadata -- is the only case
> > > that
> > > > matters concurrent updates between the writer and a single
> replication
> > > > worker?
> > > >
> > >
> > > I don't think it can be addressed by a version vector, there is no
> public
> > > *history* about who changed what version.
> > > But the question is why do you need to address this
> #writeLedgerMetadata
> > > inside the ZooKeeperClient retry loop. I think there is already a logic
> > on
> > > #writeLedgerMetadata to resolve conflicts due to BadVersion.
> > > The LedgerHandle will re-read the ledger metadata when encountering
> > version
> > > conflicts.
> > >
> > >
> > That's the part I'm unsure of.  I know it has handling for BadVersion, I
> > don't know
> > whether it's still correct if you get a BadVersion even though your write
> > actually
> > succeeded, I need to look into that more.  Also, I'm not at all
> suggesting
> > we handle
> > this in the retry loop, I'm suggesting that for non-idempotent writes, we
> > should not
> > retry in ZooKeeperClient and instead propagate the ConnectionLoss error
> and
> > let the caller deal with it.
> >
>
> +1 I agree with no retrying non-idempotent writes.
>
>
> > -Sam
> >
> >
> > >
> > > >
> > > > The outline is probably:
> > > > 1) Add a way to do retries automatically on *unconditional* creates
> and
> > > > deletes (there are some users that don't care).
> > > > 2) Audit and modify existing users to either use the methods
> introduced
> > > in
> > > > 1 or handle the ConnectionLoss events explicitly.
> > > > 3) Switch ZooKeeperClient to not retry conditional writes instead
> > > > propagating the ConnectionLoss event, that should make its semantics
> > > match
> > > > the vanilla ZK client.
> > > > -Sam
> > > >
> > > > On Tue, Jun 27, 2017 at 8:59 AM, Si

Build failed in Jenkins: bookkeeper-master #1799

[Apache Jenkins Server]

See 

--
[...truncated 1.03 MB...]
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.433 sec - in 
org.apache.bookkeeper.client.BookKeeperDiskSpaceWeightedLedgerPlacementTest
Running org.apache.bookkeeper.client.TestSpeculativeRead
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.206 sec - in 
org.apache.bookkeeper.client.TestSpeculativeRead
Running org.apache.bookkeeper.client.TestBookieHealthCheck
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.006 sec - in 
org.apache.bookkeeper.client.TestBookieHealthCheck
Running org.apache.bookkeeper.client.TestGetBookieInfoTimeout
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.173 sec - in 
org.apache.bookkeeper.client.TestGetBookieInfoTimeout
Running org.apache.bookkeeper.client.RoundRobinDistributionScheduleTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.224 sec - in 
org.apache.bookkeeper.client.RoundRobinDistributionScheduleTest
Running org.apache.bookkeeper.client.UpdateLedgerCmdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.174 sec - in 
org.apache.bookkeeper.client.UpdateLedgerCmdTest
Running org.apache.bookkeeper.client.TestLedgerFragmentReplication
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.088 sec - in 
org.apache.bookkeeper.client.TestLedgerFragmentReplication
Running org.apache.bookkeeper.client.BookieRecoveryTest
Tests run: 120, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 95.018 sec - 
in org.apache.bookkeeper.client.BookieRecoveryTest
Running org.apache.bookkeeper.client.LedgerCloseTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.733 sec - in 
org.apache.bookkeeper.client.LedgerCloseTest
Running org.apache.bookkeeper.client.LedgerRecoveryTest
Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.43 sec - in 
org.apache.bookkeeper.client.LedgerRecoveryTest
Running org.apache.bookkeeper.client.TestTryReadLastConfirmed
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.506 sec - in 
org.apache.bookkeeper.client.TestTryReadLastConfirmed
Running org.apache.bookkeeper.client.UpdateLedgerOpTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.93 sec - in 
org.apache.bookkeeper.client.UpdateLedgerOpTest
Running org.apache.bookkeeper.client.TestWeightedRandomSelection
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.456 sec - in 
org.apache.bookkeeper.client.TestWeightedRandomSelection
Running org.apache.bookkeeper.client.TestBookieWatcher
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.462 sec - in 
org.apache.bookkeeper.client.TestBookieWatcher
Running org.apache.bookkeeper.client.ParallelLedgerRecoveryTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.936 sec - 
in org.apache.bookkeeper.client.ParallelLedgerRecoveryTest
Running org.apache.bookkeeper.client.BookieWriteLedgerTest
Tests run: 120, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 118.815 sec - 
in org.apache.bookkeeper.client.BookieWriteLedgerTest
Running org.apache.bookkeeper.client.GenericEnsemblePlacementPolicyTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.804 sec - in 
org.apache.bookkeeper.client.GenericEnsemblePlacementPolicyTest
Running org.apache.bookkeeper.client.TestAddEntryQuorumTimeout
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.821 sec - in 
org.apache.bookkeeper.client.TestAddEntryQuorumTimeout
Running org.apache.bookkeeper.client.BookKeeperCloseTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.532 sec - 
in org.apache.bookkeeper.client.BookKeeperCloseTest
Running org.apache.bookkeeper.client.LocalBookKeeperTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.39 sec - in 
org.apache.bookkeeper.client.LocalBookKeeperTest
Running 
org.apache.bookkeeper.client.TestRackawareEnsemblePlacementPolicyUsingScript
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.583 sec - in 
org.apache.bookkeeper.client.TestRackawareEnsemblePlacementPolicyUsingScript
Running org.apache.bookkeeper.client.TestWatchEnsembleChange
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.335 sec - in 
org.apache.bookkeeper.client.TestWatchEnsembleChange
Running org.apache.bookkeeper.client.TestReadEntryListener
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.494 sec - in 
org.apache.bookkeeper.client.TestReadEntryListener
Running org.apache.bookkeeper.client.BookieWriteLedgersWithDifferentDigestsTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.57 sec - in 
org.apache.bookkeeper.client.BookieWriteLedgersWithDifferentDigestsTest
Running org.apache.bookkeeper.client.TestLedgerChecker
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.88 sec - in 
org.apache.boo

[GitHub] jvrao commented on issue #210: BOOKKEEPER-1100: Add Http Server for Bookkeeper

[git]

jvrao commented on issue #210: BOOKKEEPER-1100: Add Http Server for Bookkeeper
URL: https://github.com/apache/bookkeeper/pull/210#issuecomment-311660233
 
 
   We are adding REST endpoint to BookKeeper. I need to check if they overlap. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper issue #210: BOOKKEEPER-1100: Add Http Server for Bookkeeper

[jvrao]

Github user jvrao commented on the issue:

https://github.com/apache/bookkeeper/pull/210
  
We are adding REST endpoint to BookKeeper. I need to check if they overlap. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-1100) Add Http Endpoint for Bookkeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066479#comment-16066479
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1100:


Github user jvrao commented on the issue:

https://github.com/apache/bookkeeper/pull/210
  
We are adding REST endpoint to BookKeeper. I need to check if they overlap. 


> Add Http Endpoint for Bookkeeper
> 
>
> Key: BOOKKEEPER-1100
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1100
> Project: Bookkeeper
>  Issue Type: New Feature
>  Components: bookkeeper-server
>Affects Versions: 4.5.0
>Reporter: Yiming Zang
>Assignee: Yiming Zang
> Fix For: 4.5.0
>
>
> Add a Http Server Component for Bookkeeper.
> It would be great that Bookkeeper can have an Http Server to expose some 
> useful APIs. We can expose and manage the server status, server 
> configuration, lifecycle states, query or trigger auto recovery through HTTP 
> Endpoint, or even trigger GC and compaction over Http Endpoint.
> This ticket is mainly to add a Http framework for Bookkeeper, so that we can 
> extend more useful APIs based on the existing Http framework.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] bookkeeper issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication

[jvrao]

Github user jvrao commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@sijie @reddycharan Let me step back and discuss this.

1. Are there any legitimate use cases of specifying multiple ledger dirs on 
the same disk partition? if not, in production I would like to flag this as 
error, or at least a warning to ignore other ledger dirs. I don't think code 
need to work hard to dedup and skip these things. This is not a naive user 
mistake, configurations are done by system administrator and they need to be 
crisp IMO.
2. Currently we have only one active entrylog, so having multiple dirs on 
the same disk, at the most may fill one disk partition faster/quicker than 
other. (Say three ledger dirs on disk1 and one ledger dir on disk2). If disk1 
and disk2 are of same sizes, disk1 gets used more. but if disk1 is bigger than 
disk2 then this feature may be handy.
3. When we move to multiple active entrylogs, we are going to distribute 
entrylogs across ledger dirs. When this feature is introduced it surely hits 
perf because one disk may be having multiple entrylogs which gets 
written/scyned unless dedup is implemented.

@sijie I am not sure what is the biggest concern about this. Are you 
worried about too many configuration parameters? or something else?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066501#comment-16066501
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user jvrao commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@sijie @reddycharan Let me step back and discuss this.

1. Are there any legitimate use cases of specifying multiple ledger dirs on 
the same disk partition? if not, in production I would like to flag this as 
error, or at least a warning to ignore other ledger dirs. I don't think code 
need to work hard to dedup and skip these things. This is not a naive user 
mistake, configurations are done by system administrator and they need to be 
crisp IMO.
2. Currently we have only one active entrylog, so having multiple dirs on 
the same disk, at the most may fill one disk partition faster/quicker than 
other. (Say three ledger dirs on disk1 and one ledger dir on disk2). If disk1 
and disk2 are of same sizes, disk1 gets used more. but if disk1 is bigger than 
disk2 then this feature may be handy.
3. When we move to multiple active entrylogs, we are going to distribute 
entrylogs across ledger dirs. When this feature is introduced it surely hits 
perf because one disk may be having multiple entrylogs which gets 
written/scyned unless dedup is implemented.

@sijie I am not sure what is the biggest concern about this. Are you 
worried about too many configuration parameters? or something else?


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] jvrao commented on issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication

[git]

jvrao commented on issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication
URL: https://github.com/apache/bookkeeper/pull/189#issuecomment-311665630
 
 
   @sijie @reddycharan Let me step back and discuss this.
   
   1. Are there any legitimate use cases of specifying multiple ledger dirs on 
the same disk partition? if not, in production I would like to flag this as 
error, or at least a warning to ignore other ledger dirs. I don't think code 
need to work hard to dedup and skip these things. This is not a naive user 
mistake, configurations are done by system administrator and they need to be 
crisp IMO.
   2. Currently we have only one active entrylog, so having multiple dirs on 
the same disk, at the most may fill one disk partition faster/quicker than 
other. (Say three ledger dirs on disk1 and one ledger dir on disk2). If disk1 
and disk2 are of same sizes, disk1 gets used more. but if disk1 is bigger than 
disk2 then this feature may be handy.
   3. When we move to multiple active entrylogs, we are going to distribute 
entrylogs across ledger dirs. When this feature is introduced it surely hits 
perf because one disk may be having multiple entrylogs which gets 
written/scyned unless dedup is implemented.
   
   @sijie I am not sure what is the biggest concern about this. Are you worried 
about too many configuration parameters? or something else?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jvrao commented on a change in pull request #200: Issue 209: Introduce Speculative Read/Read LAC policy

[git]

jvrao commented on a change in pull request #200: Issue 209: Introduce 
Speculative Read/Read LAC policy
URL: https://github.com/apache/bookkeeper/pull/200#discussion_r124549893
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingReadOp.java
 ##
 @@ -565,7 +559,7 @@ protected void submitCallback(int code) {
 }
 }
 LOG.error("Read of ledger entry failed: L{} E{}-E{}, Heard from 
{}. First unread entry is {}",
-new Object[] { lh.getId(), startEntryId, endEntryId, 
heardFromHosts, firstUnread });
+new Object[] { lh.getId(), startEntryId, endEntryId, 
heardFromHostsBitSet, firstUnread });
 
 Review comment:
   I would still like to get the bookie inet address as we use it extensively 
for debugging. With this we get the index only right?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jvrao commented on issue #200: Issue 209: Introduce Speculative Read/Read LAC policy

[git]

jvrao commented on issue #200: Issue 209: Introduce Speculative Read/Read LAC 
policy
URL: https://github.com/apache/bookkeeper/pull/200#issuecomment-311673084
 
 
   @sijie what is the major drive of this change? Current method of trying 
speculative reads has any major flaw? I understand this is better organized as 
speculative reads has its own policy, but trying to understand what use-case 
drove you to this change at Twitter?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper issue #200: Issue 209: Introduce Speculative Read/Read LAC policy

[jvrao]

Github user jvrao commented on the issue:

https://github.com/apache/bookkeeper/pull/200
  
@sijie what is the major drive of this change? Current method of trying 
speculative reads has any major flaw? I understand this is better organized as 
speculative reads has its own policy, but trying to understand what use-case 
drove you to this change at Twitter?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: BookKeeper repo has been moved into gitbox

[Sijie Guo]

I think the GitBox change is related to BP-9: Using Github Issues. Gitbox
is the solution provided by ASF to managing Github repos, linking the
Github accounts with Apache counts.

Before this, the bookkeeper repo was setup with dural-sync, when the
source-of-truth is the git-wp-us.apache.org/repos/asf/bookkeeper.git and
the Github repo github.com/apache/bookkeeper is the readonly mirror.
It is hard to proceed BP-9 with permissions on readonly-mirror.

After this, the github repo will become the source of truth, and the gitbox
repo is the mirror of Github repo. All the bookkeeper committers will be
part of the bookkeeper-committers team in Github and have the permissions
on the Github mirror. In this way, we have the permissions to setup labels,
milestones for BP-9.

There is no changes to use https://github.com/apache/bookkeeper.git . The
original repo https://git-wip-us.apache.org/repos/asf/bookkeeper.git/ is
changed to https://gitbox.apache.org/repos/asf/bookkeeper.git (I think
Enrico just updated the wiki about that).

- Sijie

On Wed, Jun 28, 2017 at 6:18 AM, Venkateswara Rao Jujjuri  wrote:

> What is this change about? Can we still access and use
> https://github.com/apache/bookkeeper.git ?
>
> On Wed, Jun 28, 2017 at 5:53 AM, Enrico Olivelli 
> wrote:
>
> > Jia,
> > I already did it.
> > I'm not aware of other places to change the link to the repo.
> > Do you have other ideas for the switch ?
> > Cheers
> > -- Enrico
> >
> >
> > 2017-06-28 14:51 GMT+02:00 Jia Zhai :
> > > Thanks Enrico for the confirm. Yes, we may need some changes,
> especially
> > > the repo address.
> > >
> > > On Wed, Jun 28, 2017 at 8:07 PM, Enrico Olivelli 
> > > wrote:
> > >
> > >> I am going on with trials in order to discover our new space before
> > >> hitting bad problems.
> > >>
> > >> If you try to access the old git repo you will get
> > >>
> > >> [enrico.olivelli@localhost bookkeeper]$ git pull apache master
> > >> fatal: repository
> > >> 'https://git-wip-us.apache.org/repos/asf/bookkeeper.git/' not found
> > >>
> > >> As the old repo is not working anymore
> > >>
> > >> I have updated the master Jenkins job
> > >> https://builds.apache.org/job/bookkeeper-master/
> > >>
> > >> I have just updated the wiki page as well
> > >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> Patch+Sumission
> > >>
> > >> -- Enrico
> > >>
> > >> 2017-06-28 13:20 GMT+02:00 Enrico Olivelli :
> > >> > This is great
> > >> > it seems that we have also a GitHub team for bookkeeper committers
> > >> > https://github.com/orgs/apache/teams/bookkeeper-committers/members
> > >> >
> > >> > -- Enrico
> > >> >
> > >> >
> > >> > 2017-06-28 12:29 GMT+02:00 Enrico Olivelli :
> > >> >> Hi Jia,
> > >> >> this is a very important change !!
> > >> >>
> > >> >> Do you now what will happen if any committers uses
> > >> >> git-ws-us.apache.org as usual ?
> > >> >> is there any chance to break things ?
> > >> >>
> > >> >> we should immediately update the wiki page
> > >> >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> > Patch+Sumission
> > >> >>
> > >> >> let's wait for Sijie acknowledge
> > >> >>
> > >> >> Thank you
> > >> >> -- Enrico
> > >> >>
> > >> >>
> > >> >>
> > >> >> 2017-06-28 12:24 GMT+02:00 Jia Zhai :
> > >> >>> Hi All,
> > >> >>> Thanks to the help from INFRA team, now bookkeeper repo has been
> > moved
> > >> into
> > >> >>> gitbox.
> > >> >>>
> > >> >>> The new URL (on our side) is
> > >> >>> https://gitbox.apache.org/repos/asf/bookkeeper.git
> > >> >>> For committers, please go through the setup at
> > >> >>> https://gitbox.apache.org/setup/
> > >> >>>
> > >> >>> Thanks.
> > >>
> >
>
>
>
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi
>

[GitHub] sijie commented on a change in pull request #200: Issue 209: Introduce Speculative Read/Read LAC policy

[git]

sijie commented on a change in pull request #200: Issue 209: Introduce 
Speculative Read/Read LAC policy
URL: https://github.com/apache/bookkeeper/pull/200#discussion_r124584618
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingReadOp.java
 ##
 @@ -565,7 +559,7 @@ protected void submitCallback(int code) {
 }
 }
 LOG.error("Read of ledger entry failed: L{} E{}-E{}, Heard from 
{}. First unread entry is {}",
-new Object[] { lh.getId(), startEntryId, endEntryId, 
heardFromHosts, firstUnread });
+new Object[] { lh.getId(), startEntryId, endEntryId, 
heardFromHostsBitSet, firstUnread });
 
 Review comment:
   I will add the address information in the error messages
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper issue #200: Issue 209: Introduce Speculative Read/Read LAC policy

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/200
  
- in the upcoming changes of long poll requests, we need same speculative 
policy for long poll requests. abstract the logic as a policy will make things 
easier and we don't have to duplicate the code for both reads and long poll 
reads
- abstracting the logic as a policy will help us improving the policy 
without changing the read code.
- we also extend the default policy to have exponential backup in place.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] sijie commented on issue #200: Issue 209: Introduce Speculative Read/Read LAC policy

[git]

sijie commented on issue #200: Issue 209: Introduce Speculative Read/Read LAC 
policy
URL: https://github.com/apache/bookkeeper/pull/200#issuecomment-311708998
 
 
   - in the upcoming changes of long poll requests, we need same speculative 
policy for long poll requests. abstract the logic as a policy will make things 
easier and we don't have to duplicate the code for both reads and long poll 
reads
   - abstracting the logic as a policy will help us improving the policy 
without changing the read code.
   - we also extend the default policy to have exponential backoff in place.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #200: Issue 209: Introduce Speculative Read/Read LAC policy

[git]

sijie commented on issue #200: Issue 209: Introduce Speculative Read/Read LAC 
policy
URL: https://github.com/apache/bookkeeper/pull/200#issuecomment-311708998
 
 
   - in the upcoming changes of long poll requests, we need same speculative 
policy for long poll requests. abstract the logic as a policy will make things 
easier and we don't have to duplicate the code for both reads and long poll 
reads
   - abstracting the logic as a policy will help us improving the policy 
without changing the read code.
   - we also extend the default policy to have exponential backup in place.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #210: BOOKKEEPER-1100: Add Http Server for Bookkeeper

[git]

sijie commented on issue #210: BOOKKEEPER-1100: Add Http Server for Bookkeeper
URL: https://github.com/apache/bookkeeper/pull/210#issuecomment-311709678
 
 
   @jvrao do you mind sending out your proposal. we might need to merge the 
efforts here, and also @eolivelli has some ideas on how to implement this with 
standard servlets.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper issue #210: BOOKKEEPER-1100: Add Http Server for Bookkeeper

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/210
  
@jvrao do you mind sending out your proposal. we might need to merge the 
efforts here, and also @eolivelli has some ideas on how to implement this with 
standard servlets.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-1100) Add Http Endpoint for Bookkeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066794#comment-16066794
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1100:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/210
  
@jvrao do you mind sending out your proposal. we might need to merge the 
efforts here, and also @eolivelli has some ideas on how to implement this with 
standard servlets.


> Add Http Endpoint for Bookkeeper
> 
>
> Key: BOOKKEEPER-1100
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1100
> Project: Bookkeeper
>  Issue Type: New Feature
>  Components: bookkeeper-server
>Affects Versions: 4.5.0
>Reporter: Yiming Zang
>Assignee: Yiming Zang
> Fix For: 4.5.0
>
>
> Add a Http Server Component for Bookkeeper.
> It would be great that Bookkeeper can have an Http Server to expose some 
> useful APIs. We can expose and manage the server status, server 
> configuration, lifecycle states, query or trigger auto recovery through HTTP 
> Endpoint, or even trigger GC and compaction over Http Endpoint.
> This ticket is mainly to add a Http framework for Bookkeeper, so that we can 
> extend more useful APIs based on the existing Http framework.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] bookkeeper issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
>> re 1.

I am not aware if they are users configuring in this way. but I am not sure 
if this is the case for flagging as error. because in current implementation, 
such configuration doesn't have any performance impacts. if this flag is 
necessarily for multiple entry log files, I would defer adding this flag when 
reviewing the multiple-active-entry-log-files change. because it is not very 
clear about the problem and benefits.

>> re 2>

this sounds like a policy problem - how to choose the directory to place 
the entry log files to achieve balance between disk partitions? currently 
bookie asks the ledgerdirs manager for a ledger directory. the ledger disks can 
do a better job for balancing the placement of entry log files.

>> re 3.

if we have a policy abstraction in LedgerDirsManager, we can have a better 
policy on how to pick up the entry log files for write. in this way, we can 
balance the disk usage and I/O usage.

I am seeing a trend of adding more and more configuration settings into the 
bookies. some are strongly required. some doesn't seem to be really needed. In 
this pull request, the configuration doesn't seem to be the right solution that 
address the problems that you guys are listing. it seems to be me a flag to 
mask the underneath problems and this flag will most likely be used anymore 
when we have the correct solution in place.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066813#comment-16066813
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
>> re 1.

I am not aware if they are users configuring in this way. but I am not sure 
if this is the case for flagging as error. because in current implementation, 
such configuration doesn't have any performance impacts. if this flag is 
necessarily for multiple entry log files, I would defer adding this flag when 
reviewing the multiple-active-entry-log-files change. because it is not very 
clear about the problem and benefits.

>> re 2>

this sounds like a policy problem - how to choose the directory to place 
the entry log files to achieve balance between disk partitions? currently 
bookie asks the ledgerdirs manager for a ledger directory. the ledger disks can 
do a better job for balancing the placement of entry log files.

>> re 3.

if we have a policy abstraction in LedgerDirsManager, we can have a better 
policy on how to pick up the entry log files for write. in this way, we can 
balance the disk usage and I/O usage.

I am seeing a trend of adding more and more configuration settings into the 
bookies. some are strongly required. some doesn't seem to be really needed. In 
this pull request, the configuration doesn't seem to be the right solution that 
address the problems that you guys are listing. it seems to be me a flag to 
mask the underneath problems and this flag will most likely be used anymore 
when we have the correct solution in place.


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] sijie commented on issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication

[git]

sijie commented on issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication
URL: https://github.com/apache/bookkeeper/pull/189#issuecomment-311713578
 
 
   >> re 1.
   
   I am not aware if they are users configuring in this way. but I am not sure 
if this is the case for flagging as error. because in current implementation, 
such configuration doesn't have any performance impacts. if this flag is 
necessarily for multiple entry log files, I would defer adding this flag when 
reviewing the multiple-active-entry-log-files change. because it is not very 
clear about the problem and benefits.
   
   >> re 2>
   
   this sounds like a policy problem - how to choose the directory to place the 
entry log files to achieve balance between disk partitions? currently bookie 
asks the ledgerdirs manager for a ledger directory. the ledger disks can do a 
better job for balancing the placement of entry log files.
   
   >> re 3.
   
   if we have a policy abstraction in LedgerDirsManager, we can have a better 
policy on how to pick up the entry log files for write. in this way, we can 
balance the disk usage and I/O usage.
   
   I am seeing a trend of adding more and more configuration settings into the 
bookies. some are strongly required. some doesn't seem to be really needed. In 
this pull request, the configuration doesn't seem to be the right solution that 
address the problems that you guys are listing. it seems to be me a flag to 
mask the underneath problems and this flag will most likely be used anymore 
when we have the correct solution in place.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication

[jvrao]

Github user jvrao commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
Multiple entrylog change is coming in and Charan is using this for that. So
you can think of this as setting stage for that.
If you don't want to have the flag/configuration parameter we can get rid
of it; That is not really needed for what we need.

If you feel strongly against this configuration param, we can flag a
warning instead of error if user configured multiple entrylogs on same
disk. Its a bit of code change and another commit not a biggie.



On Wed, Jun 28, 2017 at 9:25 AM, Sijie Guo  wrote:

> re 1.
>
> I am not aware if they are users configuring in this way. but I am not
> sure if this is the case for flagging as error. because in current
> implementation, such configuration doesn't have any performance impacts. 
if
> this flag is necessarily for multiple entry log files, I would defer 
adding
> this flag when reviewing the multiple-active-entry-log-files change.
> because it is not very clear about the problem and benefits.
>
> re 2>
>
> this sounds like a policy problem - how to choose the directory to place
> the entry log files to achieve balance between disk partitions? currently
> bookie asks the ledgerdirs manager for a ledger directory. the ledger 
disks
> can do a better job for balancing the placement of entry log files.
>
> re 3.
>
> if we have a policy abstraction in LedgerDirsManager, we can have a better
> policy on how to pick up the entry log files for write. in this way, we 
can
> balance the disk usage and I/O usage.
>
> I am seeing a trend of adding more and more configuration settings into
> the bookies. some are strongly required. some doesn't seem to be really
> needed. In this pull request, the configuration doesn't seem to be the
> right solution that address the problems that you guys are listing. it
> seems to be me a flag to mask the underneath problems and this flag will
> most likely be used anymore when we have the correct solution in place.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 

> .
>



-- 
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] jvrao commented on issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication

[git]

jvrao commented on issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication
URL: https://github.com/apache/bookkeeper/pull/189#issuecomment-311716056
 
 
   Multiple entrylog change is coming in and Charan is using this for that. So
   you can think of this as setting stage for that.
   If you don't want to have the flag/configuration parameter we can get rid
   of it; That is not really needed for what we need.
   
   If you feel strongly against this configuration param, we can flag a
   warning instead of error if user configured multiple entrylogs on same
   disk. Its a bit of code change and another commit not a biggie.
   
   
   
   On Wed, Jun 28, 2017 at 9:25 AM, Sijie Guo  wrote:
   
   > re 1.
   >
   > I am not aware if they are users configuring in this way. but I am not
   > sure if this is the case for flagging as error. because in current
   > implementation, such configuration doesn't have any performance impacts. if
   > this flag is necessarily for multiple entry log files, I would defer adding
   > this flag when reviewing the multiple-active-entry-log-files change.
   > because it is not very clear about the problem and benefits.
   >
   > re 2>
   >
   > this sounds like a policy problem - how to choose the directory to place
   > the entry log files to achieve balance between disk partitions? currently
   > bookie asks the ledgerdirs manager for a ledger directory. the ledger disks
   > can do a better job for balancing the placement of entry log files.
   >
   > re 3.
   >
   > if we have a policy abstraction in LedgerDirsManager, we can have a better
   > policy on how to pick up the entry log files for write. in this way, we can
   > balance the disk usage and I/O usage.
   >
   > I am seeing a trend of adding more and more configuration settings into
   > the bookies. some are strongly required. some doesn't seem to be really
   > needed. In this pull request, the configuration doesn't seem to be the
   > right solution that address the problems that you guys are listing. it
   > seems to be me a flag to mask the underneath problems and this flag will
   > most likely be used anymore when we have the correct solution in place.
   >
   > ?
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > ,
   > or mute the thread
   > 

   > .
   >
   
   
   
   -- 
   Jvrao
   ---
   First they ignore you, then they laugh at you, then they fight you, then
   you win. - Mahatma Gandhi
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066823#comment-16066823
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user jvrao commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
Multiple entrylog change is coming in and Charan is using this for that. So
you can think of this as setting stage for that.
If you don't want to have the flag/configuration parameter we can get rid
of it; That is not really needed for what we need.

If you feel strongly against this configuration param, we can flag a
warning instead of error if user configured multiple entrylogs on same
disk. Its a bit of code change and another commit not a biggie.



On Wed, Jun 28, 2017 at 9:25 AM, Sijie Guo  wrote:

> re 1.
>
> I am not aware if they are users configuring in this way. but I am not
> sure if this is the case for flagging as error. because in current
> implementation, such configuration doesn't have any performance impacts. 
if
> this flag is necessarily for multiple entry log files, I would defer 
adding
> this flag when reviewing the multiple-active-entry-log-files change.
> because it is not very clear about the problem and benefits.
>
> re 2>
>
> this sounds like a policy problem - how to choose the directory to place
> the entry log files to achieve balance between disk partitions? currently
> bookie asks the ledgerdirs manager for a ledger directory. the ledger 
disks
> can do a better job for balancing the placement of entry log files.
>
> re 3.
>
> if we have a policy abstraction in LedgerDirsManager, we can have a better
> policy on how to pick up the entry log files for write. in this way, we 
can
> balance the disk usage and I/O usage.
>
> I am seeing a trend of adding more and more configuration settings into
> the bookies. some are strongly required. some doesn't seem to be really
> needed. In this pull request, the configuration doesn't seem to be the
> right solution that address the problems that you guys are listing. it
> seems to be me a flag to mask the underneath problems and this flag will
> most likely be used anymore when we have the correct solution in place.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 

> .
>



-- 
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi



> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] bookkeeper issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@jvrao I am fine with having a configuration setting as what I said before.

The comments are mostly for people knowing different aspects on the 
solutions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] sijie commented on issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication

[git]

sijie commented on issue #189: BOOKKEEPER-1033: Handle DirsPartitionDuplication
URL: https://github.com/apache/bookkeeper/pull/189#issuecomment-311757496
 
 
   @jvrao I am fine with having a configuration setting as what I said before.
   
   The comments are mostly for people knowing different aspects on the 
solutions.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067060#comment-16067060
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@jvrao I am fine with having a configuration setting as what I said before.

The comments are mostly for people knowing different aspects on the 
solutions.


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: BookKeeper repo has been moved into gitbox

[Sijie Guo]

Enrico, Jia,

I have confirmed that I have the permission on creating label/milestones
and also I see the 'merge button' in the pull requests as well.

Following the BP-9, I have created corresponding labels and milestones for
4.5.0 and 4.6.0. Let me know if they look good to you.

For the 'merge button', according to the discussion in BP-9, we want to
'squash merging' and disable 'merge commits' and 'rebase merging', right?
We might need to ask INFRA for doing that? Any thoughts?

- Sijie

On Wed, Jun 28, 2017 at 6:20 AM, Enrico Olivelli 
wrote:

> JV,
> GitHub at the moment is only a mirror for the official Git repo.
> When committing we must push to gitbox now
> For clones/readonly work/pull requests we are using GitHub as before
>
> -- Enrico
>
> 2017-06-28 15:18 GMT+02:00 Venkateswara Rao Jujjuri :
> > What is this change about? Can we still access and use
> > https://github.com/apache/bookkeeper.git ?
> >
> > On Wed, Jun 28, 2017 at 5:53 AM, Enrico Olivelli 
> > wrote:
> >
> >> Jia,
> >> I already did it.
> >> I'm not aware of other places to change the link to the repo.
> >> Do you have other ideas for the switch ?
> >> Cheers
> >> -- Enrico
> >>
> >>
> >> 2017-06-28 14:51 GMT+02:00 Jia Zhai :
> >> > Thanks Enrico for the confirm. Yes, we may need some changes,
> especially
> >> > the repo address.
> >> >
> >> > On Wed, Jun 28, 2017 at 8:07 PM, Enrico Olivelli  >
> >> > wrote:
> >> >
> >> >> I am going on with trials in order to discover our new space before
> >> >> hitting bad problems.
> >> >>
> >> >> If you try to access the old git repo you will get
> >> >>
> >> >> [enrico.olivelli@localhost bookkeeper]$ git pull apache master
> >> >> fatal: repository
> >> >> 'https://git-wip-us.apache.org/repos/asf/bookkeeper.git/' not found
> >> >>
> >> >> As the old repo is not working anymore
> >> >>
> >> >> I have updated the master Jenkins job
> >> >> https://builds.apache.org/job/bookkeeper-master/
> >> >>
> >> >> I have just updated the wiki page as well
> >> >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> Patch+Sumission
> >> >>
> >> >> -- Enrico
> >> >>
> >> >> 2017-06-28 13:20 GMT+02:00 Enrico Olivelli :
> >> >> > This is great
> >> >> > it seems that we have also a GitHub team for bookkeeper committers
> >> >> > https://github.com/orgs/apache/teams/bookkeeper-committers/members
> >> >> >
> >> >> > -- Enrico
> >> >> >
> >> >> >
> >> >> > 2017-06-28 12:29 GMT+02:00 Enrico Olivelli :
> >> >> >> Hi Jia,
> >> >> >> this is a very important change !!
> >> >> >>
> >> >> >> Do you now what will happen if any committers uses
> >> >> >> git-ws-us.apache.org as usual ?
> >> >> >> is there any chance to break things ?
> >> >> >>
> >> >> >> we should immediately update the wiki page
> >> >> >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> >> Patch+Sumission
> >> >> >>
> >> >> >> let's wait for Sijie acknowledge
> >> >> >>
> >> >> >> Thank you
> >> >> >> -- Enrico
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> 2017-06-28 12:24 GMT+02:00 Jia Zhai :
> >> >> >>> Hi All,
> >> >> >>> Thanks to the help from INFRA team, now bookkeeper repo has been
> >> moved
> >> >> into
> >> >> >>> gitbox.
> >> >> >>>
> >> >> >>> The new URL (on our side) is
> >> >> >>> https://gitbox.apache.org/repos/asf/bookkeeper.git
> >> >> >>> For committers, please go through the setup at
> >> >> >>> https://gitbox.apache.org/setup/
> >> >> >>>
> >> >> >>> Thanks.
> >> >>
> >>
> >
> >
> >
> > --
> > Jvrao
> > ---
> > First they ignore you, then they laugh at you, then they fight you, then
> > you win. - Mahatma Gandhi
>

[GitHub] sijie closed issue #216: Remove useless and misleading '&= true'

[git]

sijie closed issue #216: Remove useless and misleading '&= true'
URL: https://github.com/apache/bookkeeper/issues/216
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper issue #216: Remove useless and misleading '&= true'

[sijie]

Github user sijie closed the issue at:

https://github.com/apache/bookkeeper/issues/216


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] sijie closed pull request #215: Issue 216: Remove useless and misleading '&= true'

[git]

sijie closed pull request #215: Issue 216: Remove useless and misleading '&= 
true'
URL: https://github.com/apache/bookkeeper/pull/215
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper pull request #215: Issue 216: Remove useless and misleading '&= t...

[sijie]

Github user sijie closed the pull request at:

https://github.com/apache/bookkeeper/pull/215


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: BookKeeper repo has been moved into gitbox

[Sijie Guo]

Hi Enrico,

The gitbox is the readonly mirror of the github repo. The github repo is
the source of truth. So we should use the github repo for jenkins build and
also for patch submissions.

The merge script uses 'apache' remote as the destination for pushing the
squashed changes and uses 'apache-github' remote as the source for fetching
pull requests. so we have to use the github repo for both 'apache' and
'apache-github' remotes, then the merge script can work without changing
anything.

- Sijie

On Wed, Jun 28, 2017 at 5:07 AM, Enrico Olivelli 
wrote:

> I am going on with trials in order to discover our new space before
> hitting bad problems.
>
> If you try to access the old git repo you will get
>
> [enrico.olivelli@localhost bookkeeper]$ git pull apache master
> fatal: repository
> 'https://git-wip-us.apache.org/repos/asf/bookkeeper.git/' not found
>
> As the old repo is not working anymore
>
> I have updated the master Jenkins job
> https://builds.apache.org/job/bookkeeper-master/
>
> I have just updated the wiki page as well
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission
>
> -- Enrico
>
> 2017-06-28 13:20 GMT+02:00 Enrico Olivelli :
> > This is great
> > it seems that we have also a GitHub team for bookkeeper committers
> > https://github.com/orgs/apache/teams/bookkeeper-committers/members
> >
> > -- Enrico
> >
> >
> > 2017-06-28 12:29 GMT+02:00 Enrico Olivelli :
> >> Hi Jia,
> >> this is a very important change !!
> >>
> >> Do you now what will happen if any committers uses
> >> git-ws-us.apache.org as usual ?
> >> is there any chance to break things ?
> >>
> >> we should immediately update the wiki page
> >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission
> >>
> >> let's wait for Sijie acknowledge
> >>
> >> Thank you
> >> -- Enrico
> >>
> >>
> >>
> >> 2017-06-28 12:24 GMT+02:00 Jia Zhai :
> >>> Hi All,
> >>> Thanks to the help from INFRA team, now bookkeeper repo has been moved
> into
> >>> gitbox.
> >>>
> >>> The new URL (on our side) is
> >>> https://gitbox.apache.org/repos/asf/bookkeeper.git
> >>> For committers, please go through the setup at
> >>> https://gitbox.apache.org/setup/
> >>>
> >>> Thanks.
>

[GitHub] sijie closed pull request #190: BOOKKEEPER-1034: Bookie start in RO when diskfull

[git]

sijie closed pull request #190: BOOKKEEPER-1034: Bookie start in RO when 
diskfull
URL: https://github.com/apache/bookkeeper/pull/190
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper pull request #190: BOOKKEEPER-1034: Bookie start in RO when diskf...

[sijie]

Github user sijie closed the pull request at:

https://github.com/apache/bookkeeper/pull/190


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-1034) When all disks are full, start Bookie in RO mode if RO mode is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067253#comment-16067253
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1034:


Github user sijie closed the pull request at:

https://github.com/apache/bookkeeper/pull/190


> When all disks are full, start Bookie in RO mode if RO mode is enabled 
> ---
>
> Key: BOOKKEEPER-1034
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1034
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
> Fix For: 4.5.0
>
>
> - When all disks are full start Bookie in RO mode if RO mode is enabled
> - This will work only if "isForceGCAllowWhenNoSpace" is allowed, since 
> LedgerDirsManager.getWritableLedgerDirsForNewLog will be able to find new 
> writableLedgerDir even when all disks are full.
> - If bookie has died abruptly then it may have missed flushing EntryMemtable 
> and
> IndexInMemoryPageManager. So next time when it starts when disc is full
> it fails to create index file and it shuts down. So Bookie should be able to 
> create index file though it has reached the diskusagethreshold, while 
> starting the Bookie in Readonly Mode. But ofcourse there should be some 
> config to safeguard when disk usable space is so low.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BOOKKEEPER-1034) When all disks are full, start Bookie in RO mode if RO mode is enabled

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo resolved BOOKKEEPER-1034.
---
   Resolution: Fixed
Fix Version/s: 4.5.0

Issue resolved by merging pull request 190
[https://github.com/apache/bookkeeper/pull/190]

{noformat}
commit 9ddd9e6f9e48b03a57a2c78ec2630303abd49782
Author: Charan Reddy Guttapalem 
AuthorDate: Wed Jun 28 14:24:05 2017 -0700
Commit: Sijie Guo 
CommitDate: Wed Jun 28 14:24:05 2017 -0700

BOOKKEEPER-1034: Bookie start in RO when diskfull

When the disk is above threshold, Bookie goes to RO. If we have to restart 
the
bookie, on the way back, bookie tries to create new entrylog and other 
files,
which will fail because disk usage is above threshold,
hence bookie refuses to come up. So with this fix we will try to start in RO
mode if RO is enabled.

Also, if bookie has died abruptly then it may missed flushing EntryMemtable 
and
IndexInMemoryPageManager. So next time when it starts when disc is full
it is failing to create index file and it is shutting down, though we 
expect it
to start in readonlymode. So Bookie should be able to create index file
though it has reached the diskusagethreshold, while starting the Bookie in
Readonly Mode. But ofcourse there should be some config to safeguard when
disk usable space is so low.

Minor fixes in shutdown logic of Bookie and Bookieserver.

Author: Charan Reddy Guttapalem 

Reviewers: Jia Zhai , Sijie Guo 

This closes #190 from reddycharan/bookiestartinreadonlywhendiskfull

{noformat}


> When all disks are full, start Bookie in RO mode if RO mode is enabled 
> ---
>
> Key: BOOKKEEPER-1034
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1034
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
> Fix For: 4.5.0
>
>
> - When all disks are full start Bookie in RO mode if RO mode is enabled
> - This will work only if "isForceGCAllowWhenNoSpace" is allowed, since 
> LedgerDirsManager.getWritableLedgerDirsForNewLog will be able to find new 
> writableLedgerDir even when all disks are full.
> - If bookie has died abruptly then it may have missed flushing EntryMemtable 
> and
> IndexInMemoryPageManager. So next time when it starts when disc is full
> it fails to create index file and it shuts down. So Bookie should be able to 
> create index file though it has reached the diskusagethreshold, while 
> starting the Bookie in Readonly Mode. But ofcourse there should be some 
> config to safeguard when disk usable space is so low.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] sijie commented on issue #81: BOOKKEEPER-753: Bookie should run garbage collection before startup when all directories became full

[git]

sijie commented on issue #81: BOOKKEEPER-753: Bookie should run garbage 
collection before startup when all directories became full
URL: https://github.com/apache/bookkeeper/pull/81#issuecomment-311796693
 
 
   @jvrao @reddycharan #190 is merged. let me know what is your thought of this 
pull request.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper issue #81: BOOKKEEPER-753: Bookie should run garbage collection b...

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/81
  
@jvrao @reddycharan #190 is merged. let me know what is your thought of 
this pull request.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-753) Bookie should run garbage collection before startup when all directories became full

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067258#comment-16067258
 ] 

ASF GitHub Bot commented on BOOKKEEPER-753:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/81
  
@jvrao @reddycharan #190 is merged. let me know what is your thought of 
this pull request.


> Bookie should run garbage collection before startup when all directories 
> became full
> 
>
> Key: BOOKKEEPER-753
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-753
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> otherwise, a bookie that ran-out-of disk would never be started up again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] sijie closed pull request #193: BOOKKEEPER-1086: ZkUnderreplicationManager cache watcher

[git]

sijie closed pull request #193: BOOKKEEPER-1086: ZkUnderreplicationManager 
cache watcher
URL: https://github.com/apache/bookkeeper/pull/193
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper pull request #193: BOOKKEEPER-1086: ZkUnderreplicationManager cac...

[sijie]

Github user sijie closed the pull request at:

https://github.com/apache/bookkeeper/pull/193


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-1086) Ledger Recovery - Refactor PendingReadOp

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067265#comment-16067265
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1086:


Github user sijie closed the pull request at:

https://github.com/apache/bookkeeper/pull/193


> Ledger Recovery - Refactor PendingReadOp
> 
>
> Key: BOOKKEEPER-1086
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1086
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
>  bookkeeper recovery improvement (part-1): refactor PendingReadOp
> this change is the first part of improving bookkeeper recovery. it is 
> basically a refactor change, which:
> - abstract an interface for LedgerEntryRequest in PendingReadOp
> - rename current implementation to SequenceReadRequest, which read the 
> entry in the sequence of quorum.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] sijie commented on issue #195: Update JMX documentation

[git]

sijie commented on issue #195: Update JMX documentation
URL: https://github.com/apache/bookkeeper/issues/195#issuecomment-311797998
 
 
   @eolivelli I assume this should be go into 4.5.0 release?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (BOOKKEEPER-1098) ZkUnderreplicationManager can build up an unbounded number of watchers

[Sijie Guo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo resolved BOOKKEEPER-1098.
---
   Resolution: Fixed
 Assignee: Samuel Just
Fix Version/s: 4.5.0

> ZkUnderreplicationManager can build up an unbounded number of watchers
> --
>
> Key: BOOKKEEPER-1098
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1098
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-auto-recovery
>Reporter: Samuel Just
>Assignee: Samuel Just
>Priority: Minor
> Fix For: 4.5.0
>
>
> getLedgerToReplicate leaves watches each time it traverses the
> tree until it finds a suitable replication target.  Since we don't have
> a way of canceling watches, these watches tend to get abandoned,
> particularly on interior nodes, which aren't changed much.  Thus,
> over time, some nodes would build up a very large number of watches.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] bookkeeper issue #193: BOOKKEEPER-1098: ZkUnderreplicationManager cache watc...

[athanatos]

Github user athanatos commented on the issue:

https://github.com/apache/bookkeeper/pull/193
  
@sijie Oops, I just noticed the comments.  I guess you just changed the 
title?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] bookkeeper issue #195: Update JMX documentation

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/issues/195
  
@eolivelli I assume this should be go into 4.5.0 release?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] athanatos commented on issue #193: BOOKKEEPER-1098: ZkUnderreplicationManager cache watcher

[git]

athanatos commented on issue #193: BOOKKEEPER-1098: ZkUnderreplicationManager 
cache watcher
URL: https://github.com/apache/bookkeeper/pull/193#issuecomment-311798072
 
 
   @sijie Oops, I just noticed the comments.  I guess you just changed the 
title?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (BOOKKEEPER-1098) ZkUnderreplicationManager can build up an unbounded number of watchers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067267#comment-16067267
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1098:


Github user athanatos commented on the issue:

https://github.com/apache/bookkeeper/pull/193
  
@sijie Oops, I just noticed the comments.  I guess you just changed the 
title?


> ZkUnderreplicationManager can build up an unbounded number of watchers
> --
>
> Key: BOOKKEEPER-1098
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1098
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-auto-recovery
>Reporter: Samuel Just
>Assignee: Samuel Just
>Priority: Minor
> Fix For: 4.5.0
>
>
> getLedgerToReplicate leaves watches each time it traverses the
> tree until it finds a suitable replication target.  Since we don't have
> a way of canceling watches, these watches tend to get abandoned,
> particularly on interior nodes, which aren't changed much.  Thus,
> over time, some nodes would build up a very large number of watches.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] athanatos commented on a change in pull request #193: BOOKKEEPER-1098: ZkUnderreplicationManager cache watcher

[git]

athanatos commented on a change in pull request #193: BOOKKEEPER-1098: 
ZkUnderreplicationManager cache watcher
URL: https://github.com/apache/bookkeeper/pull/193#discussion_r124662485
 
 

 ##
 File path: 
bookkeeper-server/src/test/java/org/apache/bookkeeper/util/SubTreeCacheTest.java
 ##
 @@ -0,0 +1,321 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.bookkeeper.util;
+
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.WatchedEvent;
+import org.apache.zookeeper.Watcher;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.SortedSet;
+import java.util.TreeSet;
+
+public class SubTreeCacheTest {
+class TestTreeProvider implements SubTreeCache.TreeProvider {
+class Node {
+Watcher watcher = null;
+public Map children = new HashMap<>();
+}
+
+final Node root = new Node();
+
+Node getNode(String path) throws KeeperException {
+String[] pathSegments = path.split("/");
+Node cur = root;
+for (String segment : pathSegments) {
+if (segment.length() == 0)
+continue; // ignore leading empty one for leading /
+if (cur.children.containsKey(segment)) {
+cur = cur.children.get(segment);
+} else {
+throw KeeperException.create(KeeperException.Code.NONODE);
+}
+}
+return cur;
+}
+
+@Override
+public List getChildren(
+String path, Watcher watcher) throws InterruptedException, 
KeeperException {
+Node node = getNode(path);
+
+/* Enforce only one live watch per node */
+Assert.assertTrue(null == node.watcher);
+
+node.watcher = watcher;
+return new ArrayList(node.children.keySet());
+}
+
+public void createNode(String path) throws KeeperException {
+String[] segments = path.split("/");
+if (segments.length == 0) {
+throw KeeperException.create(KeeperException.Code.NONODE);
+}
+String child = segments[segments.length - 1];
+String[] parentSegments = Arrays.copyOfRange(segments, 0, 
segments.length - 1);
+Node parent = getNode(String.join("/", parentSegments));
+if (parent.children.containsKey(child)) {
+throw KeeperException.create(KeeperException.Code.NODEEXISTS);
+} else {
+parent.children.put(child, new Node());
+if (null != parent.watcher) {
+parent.watcher.process(
+new WatchedEvent(
+Watcher.Event.EventType.NodeCreated,
+Watcher.Event.KeeperState.SyncConnected,
+path));
+parent.watcher = null;
+}
+}
+}
+
+public void removeNode(String path) throws KeeperException {
+String[] segments = path.split("/");
+if (segments.length == 0) {
+throw KeeperException.create(KeeperException.Code.NONODE);
+}
+String child = segments[segments.length - 1];
+String[] parentSegments = Arrays.copyOfRange(segments, 0, 
segments.length - 1);
+String parentPath = String.join("/", parentSegments);
+Node parent = getNode(parentPath);
+if (!parent.children.containsKey(child)) {
+throw KeeperException.create(KeeperException.Code.NONODE);
+} else {
+Node cNode = parent.children.get(child);
+if (!cNode.children.isEmpty()) {
+throw 
KeeperException.create(KeeperException.Code.NOTEMPTY);
+} else {
+if (null != cNode.watcher) {
+cNode.watcher.process(
+ 

[GitHub] bookkeeper issue #193: BOOKKEEPER-1098: ZkUnderreplicationManager cache watc...

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/193
  
@athanatos yeah, I just changed the title for the correct jira number. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] sijie commented on issue #193: BOOKKEEPER-1098: ZkUnderreplicationManager cache watcher

[git]

sijie commented on issue #193: BOOKKEEPER-1098: ZkUnderreplicationManager cache 
watcher
URL: https://github.com/apache/bookkeeper/pull/193#issuecomment-311799215
 
 
   @athanatos yeah, I just changed the title for the correct jira number. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (BOOKKEEPER-1098) ZkUnderreplicationManager can build up an unbounded number of watchers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067278#comment-16067278
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1098:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/193
  
@athanatos yeah, I just changed the title for the correct jira number. 


> ZkUnderreplicationManager can build up an unbounded number of watchers
> --
>
> Key: BOOKKEEPER-1098
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1098
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-auto-recovery
>Reporter: Samuel Just
>Assignee: Samuel Just
>Priority: Minor
> Fix For: 4.5.0
>
>
> getLedgerToReplicate leaves watches each time it traverses the
> tree until it finds a suitable replication target.  Since we don't have
> a way of canceling watches, these watches tend to get abandoned,
> particularly on interior nodes, which aren't changed much.  Thus,
> over time, some nodes would build up a very large number of watches.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] bookkeeper issue #127: BOOKKEEPER-1028 and BOOKKEEPER-1029

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/127
  
@reddycharan there is a findbugs error introduced in this pull request. Can 
you fix that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] sijie commented on issue #127: BOOKKEEPER-1028 and BOOKKEEPER-1029

[git]

sijie commented on issue #127: BOOKKEEPER-1028 and BOOKKEEPER-1029
URL: https://github.com/apache/bookkeeper/pull/127#issuecomment-311799683
 
 
   @reddycharan there is a findbugs error introduced in this pull request. Can 
you fix that?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (BOOKKEEPER-1028) inc/excl opts listunderreplicate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067280#comment-16067280
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1028:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/127
  
@reddycharan there is a findbugs error introduced in this pull request. Can 
you fix that?


> inc/excl opts listunderreplicate
> 
>
> Key: BOOKKEEPER-1028
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1028
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - Introduce including and excluding BookieId options
> for listunderreplicatedLedgers



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] sijie commented on issue #183: BOOKKEEPER-588 SSL Support for Bookkeeper

[git]

sijie commented on issue #183: BOOKKEEPER-588 SSL Support for Bookkeeper
URL: https://github.com/apache/bookkeeper/pull/183#issuecomment-311799940
 
 
   @eolivelli I will try to go through this again. 
   
   /cc @merlimat  for reviewing as well
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper issue #183: BOOKKEEPER-588 SSL Support for Bookkeeper

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/183
  
@eolivelli I will try to go through this again. 

/cc @merlimat  for reviewing as well


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067282#comment-16067282
 ] 

ASF GitHub Bot commented on BOOKKEEPER-588:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/183
  
@eolivelli I will try to go through this again. 

/cc @merlimat  for reviewing as well


> SSL support
> ---
>
> Key: BOOKKEEPER-588
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-588
> Project: Bookkeeper
>  Issue Type: Sub-task
>Reporter: Ivan Kelly
>Assignee: Enrico Olivelli
> Fix For: 4.5.0
>
> Attachments: 0001-MutualTLS-for-Bookkeeper.patch, 
> 0004-BOOKKEEPER-588-SSL-support-for-bookkeeper.patch
>
>
> SSL support using startTLS



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] sijie commented on a change in pull request #197: BOOKKEEPER-974: Add an official bookkeeper docker image

[git]

sijie commented on a change in pull request #197: BOOKKEEPER-974: Add an 
official bookkeeper docker image
URL: https://github.com/apache/bookkeeper/pull/197#discussion_r124664167
 
 

 ##
 File path: docker/4.4.0-alpine/Dockerfile
 ##
 @@ -0,0 +1,59 @@
+FROM java:openjdk-8-jre-alpine
+MAINTAINER Francesco Caliumi 
+
+# Install required packages
+RUN apk add --no-cache \
+bash \
+su-exec
+
+ENV ZK_SERVERS= \
+BK_USER=bookkeeper \
+BK_PORT= \
+BK_BUILD_PORT=3181 \
+BOOKIE_OPTS="" \
+BK_JOURNAL_DIR=/data/journal \
+BK_LEDGER_DIR=/data/ledger \
+BK_INDEX_DIR=/data/index \
+BK_LEDGERS_PATH=
+
+# Add a user and make dirs
+RUN set -x \
+&& adduser -D "${BK_USER}" \
+&& mkdir -p "${BK_JOURNAL_DIR}" "${BK_LEDGER_DIR}" "${BK_INDEX_DIR}" \
+&& chown "$BK_USER:$BK_USER" "${BK_JOURNAL_DIR}" "${BK_LEDGER_DIR}" 
"${BK_INDEX_DIR}"
+
+ARG GPG_KEY=D0BC8D8A4E90A40AFDFC43B3E22A746A68E327C1
+ARG BK_VERSION=4.4.0
+
+ENV DISTRO_NAME=bookkeeper-server-${BK_VERSION}-bin
+   
+# Download Apache Bookkeeper, verify its PGP signature, untar and clean up
+RUN set -x \
+&& apk add --no-cache --virtual .build-deps \
+gnupg \
+wget \
+&& mkdir -pv /opt \
+&& cd /opt \
+&& wget -q 
"https://archive.apache.org/dist/bookkeeper/bookkeeper-${BK_VERSION}/${DISTRO_NAME}.tar.gz";
 \
+&& wget -q 
"https://archive.apache.org/dist/bookkeeper/bookkeeper-${BK_VERSION}/${DISTRO_NAME}.tar.gz.asc";
 \
+&& export GNUPGHOME="$(mktemp -d)" \
+#&& gpg --keyserver ha.pool.sks-keyservers.net --recv-key "$GPG_KEY" \
+#&& gpg --batch --verify "$DISTRO_NAME.tar.gz.asc" "$DISTRO_NAME.tar.gz" \
+&& tar -xzf "$DISTRO_NAME.tar.gz" \
+&& rm -r "$GNUPGHOME" "$DISTRO_NAME.tar.gz" "$DISTRO_NAME.tar.gz.asc" \
+&& apk del .build-deps
+
+ENV BK_DIR=/opt/bookkeeper-server-${BK_VERSION}
 
 Review comment:
   +1 for what @merlimat said.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #197: BOOKKEEPER-974: Add an official bookkeeper docker image

[git]

sijie commented on issue #197: BOOKKEEPER-974: Add an official bookkeeper 
docker image
URL: https://github.com/apache/bookkeeper/pull/197#issuecomment-311800374
 
 
   @caiok any updates on this pull request?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper issue #197: BOOKKEEPER-974: Add an official bookkeeper docker ima...

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/197
  
@caiok any updates on this pull request?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067288#comment-16067288
 ] 

ASF GitHub Bot commented on BOOKKEEPER-974:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/197
  
@caiok any updates on this pull request?


> make pushing a docker image as part of the release procedure
> 
>
> Key: BOOKKEEPER-974
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Jia Zhai
>
> make pushing a docker image as part of the release procedure.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] sijie commented on a change in pull request #213: Issue 212: change github PR template to adjust both issue# and jira#

[git]

sijie commented on a change in pull request #213: Issue 212: change github PR 
template to adjust both issue# and jira#
URL: https://github.com/apache/bookkeeper/pull/213#discussion_r124664720
 
 

 ##
 File path: .github/PULL_REQUEST_TEMPLATE.md
 ##
 @@ -1,15 +1,16 @@
 Descriptions of the changes in this PR:
 
-xx
+(PR description content here)...
 
 ---
 Be sure to do all of the following to help us incorporate your contribution
 quickly and easily:
 
 - [ ] Make sure the PR title is formatted like:
-`: Description of pull request`
+`: Description of pull request`
 
 Review comment:
   @eolivelli does this address your comment? do we need to call out 
'BOOKKEEPER-#' explicility?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper issue #183: BOOKKEEPER-588 SSL Support for Bookkeeper

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/183
  
fyi, I added @merlimat and me to the reviewers list, making sure we have 
other eyes on reviewing this change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] sijie commented on issue #183: BOOKKEEPER-588 SSL Support for Bookkeeper

[git]

sijie commented on issue #183: BOOKKEEPER-588 SSL Support for Bookkeeper
URL: https://github.com/apache/bookkeeper/pull/183#issuecomment-311802299
 
 
   fyi, I added @merlimat and me to the reviewers list, making sure we have 
other eyes on reviewing this change.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067297#comment-16067297
 ] 

ASF GitHub Bot commented on BOOKKEEPER-588:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/183
  
fyi, I added @merlimat and me to the reviewers list, making sure we have 
other eyes on reviewing this change.


> SSL support
> ---
>
> Key: BOOKKEEPER-588
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-588
> Project: Bookkeeper
>  Issue Type: Sub-task
>Reporter: Ivan Kelly
>Assignee: Enrico Olivelli
> Fix For: 4.5.0
>
> Attachments: 0001-MutualTLS-for-Bookkeeper.patch, 
> 0004-BOOKKEEPER-588-SSL-support-for-bookkeeper.patch
>
>
> SSL support using startTLS



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] sijie commented on issue #198: TestBackwardCompat.testCompat410 often fails due to io.netty.util.IllegalReferenceCountException

[git]

sijie commented on issue #198: TestBackwardCompat.testCompat410 often fails due 
to io.netty.util.IllegalReferenceCountException
URL: https://github.com/apache/bookkeeper/issues/198#issuecomment-311804836
 
 
   marked this as a blocker for 4.5.0, since it seems to indicate some kind of 
reference leaking in current code base.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] bookkeeper issue #198: TestBackwardCompat.testCompat410 often fails due to i...

[sijie]

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/issues/198
  
marked this as a blocker for 4.5.0, since it seems to indicate some kind of 
reference leaking in current code base.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] bookkeeper issue #127: BOOKKEEPER-1028 and BOOKKEEPER-1029

[reddycharan]

Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/127
  
@sijie  my bad, missed checking findbugs error after previous push. Fixed 
them now and it passed.

Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BOOKKEEPER-1028) inc/excl opts listunderreplicate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067419#comment-16067419
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1028:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/127
  
@sijie  my bad, missed checking findbugs error after previous push. Fixed 
them now and it passed.

Thanks.


> inc/excl opts listunderreplicate
> 
>
> Key: BOOKKEEPER-1028
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1028
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - Introduce including and excluding BookieId options
> for listunderreplicatedLedgers



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] reddycharan commented on issue #127: BOOKKEEPER-1028 and BOOKKEEPER-1029

[git]

reddycharan commented on issue #127: BOOKKEEPER-1028 and BOOKKEEPER-1029
URL: https://github.com/apache/bookkeeper/pull/127#issuecomment-311815591
 
 
   @sijie  my bad, missed checking findbugs error after previous push. Fixed 
them now and it passed.
   
   Thanks.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on a change in pull request #200: Issue 209: Introduce Speculative Read/Read LAC policy

[git]

sijie commented on a change in pull request #200: Issue 209: Introduce 
Speculative Read/Read LAC policy
URL: https://github.com/apache/bookkeeper/pull/200#discussion_r124678337
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingReadOp.java
 ##
 @@ -565,7 +559,7 @@ protected void submitCallback(int code) {
 }
 }
 LOG.error("Read of ledger entry failed: L{} E{}-E{}, Heard from 
{}. First unread entry is {}",
-new Object[] { lh.getId(), startEntryId, endEntryId, 
heardFromHosts, firstUnread });
+new Object[] { lh.getId(), startEntryId, endEntryId, 
heardFromHostsBitSet, firstUnread });
 
 Review comment:
   @jvrao I added the heardFromHosts set back in e93a5cb . so the error 
messages will include the host information. let me know if it is okay to you. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: BookKeeper repo has been moved into gitbox

[Jia Zhai]

Hi Sijie,
opened a ticket https://issues.apache.org/jira/browse/INFRA-14470 to only
"squash merge"
Thanks

On Thu, Jun 29, 2017 at 5:17 AM, Sijie Guo  wrote:

> Hi Enrico,
>
> The gitbox is the readonly mirror of the github repo. The github repo is
> the source of truth. So we should use the github repo for jenkins build and
> also for patch submissions.
>
> The merge script uses 'apache' remote as the destination for pushing the
> squashed changes and uses 'apache-github' remote as the source for fetching
> pull requests. so we have to use the github repo for both 'apache' and
> 'apache-github' remotes, then the merge script can work without changing
> anything.
>
> - Sijie
>
> On Wed, Jun 28, 2017 at 5:07 AM, Enrico Olivelli 
> wrote:
>
> > I am going on with trials in order to discover our new space before
> > hitting bad problems.
> >
> > If you try to access the old git repo you will get
> >
> > [enrico.olivelli@localhost bookkeeper]$ git pull apache master
> > fatal: repository
> > 'https://git-wip-us.apache.org/repos/asf/bookkeeper.git/' not found
> >
> > As the old repo is not working anymore
> >
> > I have updated the master Jenkins job
> > https://builds.apache.org/job/bookkeeper-master/
> >
> > I have just updated the wiki page as well
> > https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission
> >
> > -- Enrico
> >
> > 2017-06-28 13:20 GMT+02:00 Enrico Olivelli :
> > > This is great
> > > it seems that we have also a GitHub team for bookkeeper committers
> > > https://github.com/orgs/apache/teams/bookkeeper-committers/members
> > >
> > > -- Enrico
> > >
> > >
> > > 2017-06-28 12:29 GMT+02:00 Enrico Olivelli :
> > >> Hi Jia,
> > >> this is a very important change !!
> > >>
> > >> Do you now what will happen if any committers uses
> > >> git-ws-us.apache.org as usual ?
> > >> is there any chance to break things ?
> > >>
> > >> we should immediately update the wiki page
> > >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> Patch+Sumission
> > >>
> > >> let's wait for Sijie acknowledge
> > >>
> > >> Thank you
> > >> -- Enrico
> > >>
> > >>
> > >>
> > >> 2017-06-28 12:24 GMT+02:00 Jia Zhai :
> > >>> Hi All,
> > >>> Thanks to the help from INFRA team, now bookkeeper repo has been
> moved
> > into
> > >>> gitbox.
> > >>>
> > >>> The new URL (on our side) is
> > >>> https://gitbox.apache.org/repos/asf/bookkeeper.git
> > >>> For committers, please go through the setup at
> > >>> https://gitbox.apache.org/setup/
> > >>>
> > >>> Thanks.
> >
>

[GitHub] bookkeeper issue #205: Issue 208: Improve ledger fence logic

[jiazhai]

Github user jiazhai commented on the issue:

https://github.com/apache/bookkeeper/pull/205
  
Seems there is conflict files.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] jiazhai commented on issue #205: Issue 208: Improve ledger fence logic

[git]

jiazhai commented on issue #205: Issue 208: Improve ledger fence logic
URL: https://github.com/apache/bookkeeper/pull/205#issuecomment-311847825
 
 
   Seems there is conflict files.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jiazhai commented on issue #205: Issue 208: Improve ledger fence logic

[git]

jiazhai commented on issue #205: Issue 208: Improve ledger fence logic
URL: https://github.com/apache/bookkeeper/pull/205#issuecomment-311847825
 
 
   Seems there are conflict files.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: BookKeeper repo has been moved into gitbox

[Enrico Olivelli]

2017-06-29 4:06 GMT+02:00 Jia Zhai :
> Hi Sijie,
> opened a ticket https://issues.apache.org/jira/browse/INFRA-14470 to only
> "squash merge"
> Thanks
>
> On Thu, Jun 29, 2017 at 5:17 AM, Sijie Guo  wrote:
>
>> Hi Enrico,
>>
>> The gitbox is the readonly mirror of the github repo. The github repo is
>> the source of truth. So we should use the github repo for jenkins build and
>> also for patch submissions.
>>
>> The merge script uses 'apache' remote as the destination for pushing the
>> squashed changes and uses 'apache-github' remote as the source for fetching
>> pull requests. so we have to use the github repo for both 'apache' and
>> 'apache-github' remotes, then the merge script can work without changing
>> anything.

Did you try to use the "merge " button for merging recent PRs ?
Honestly I prefer the merge script anyway because it gives use the
chance to check our QA standards and tracks reviewers and the original
contributor in the git logs.
I used the "merge button" in other projects but sometimes (for long
PRs) the final result was not "awesome"

@Sijie
are you going to fix jenkins jobs and the wiki ? I can to it either

I think that we should make a clear recap about GIT repos on the dev@
list and in a wiki page

I will try to use the merge script next time I merge some pull request

-- Enrico

>>
>> - Sijie
>>
>> On Wed, Jun 28, 2017 at 5:07 AM, Enrico Olivelli 
>> wrote:
>>
>> > I am going on with trials in order to discover our new space before
>> > hitting bad problems.
>> >
>> > If you try to access the old git repo you will get
>> >
>> > [enrico.olivelli@localhost bookkeeper]$ git pull apache master
>> > fatal: repository
>> > 'https://git-wip-us.apache.org/repos/asf/bookkeeper.git/' not found
>> >
>> > As the old repo is not working anymore
>> >
>> > I have updated the master Jenkins job
>> > https://builds.apache.org/job/bookkeeper-master/
>> >
>> > I have just updated the wiki page as well
>> > https://cwiki.apache.org/confluence/display/BOOKKEEPER/Patch+Sumission
>> >
>> > -- Enrico
>> >
>> > 2017-06-28 13:20 GMT+02:00 Enrico Olivelli :
>> > > This is great
>> > > it seems that we have also a GitHub team for bookkeeper committers
>> > > https://github.com/orgs/apache/teams/bookkeeper-committers/members
>> > >
>> > > -- Enrico
>> > >
>> > >
>> > > 2017-06-28 12:29 GMT+02:00 Enrico Olivelli :
>> > >> Hi Jia,
>> > >> this is a very important change !!
>> > >>
>> > >> Do you now what will happen if any committers uses
>> > >> git-ws-us.apache.org as usual ?
>> > >> is there any chance to break things ?
>> > >>
>> > >> we should immediately update the wiki page
>> > >> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>> Patch+Sumission
>> > >>
>> > >> let's wait for Sijie acknowledge
>> > >>
>> > >> Thank you
>> > >> -- Enrico
>> > >>
>> > >>
>> > >>
>> > >> 2017-06-28 12:24 GMT+02:00 Jia Zhai :
>> > >>> Hi All,
>> > >>> Thanks to the help from INFRA team, now bookkeeper repo has been
>> moved
>> > into
>> > >>> gitbox.
>> > >>>
>> > >>> The new URL (on our side) is
>> > >>> https://gitbox.apache.org/repos/asf/bookkeeper.git
>> > >>> For committers, please go through the setup at
>> > >>> https://gitbox.apache.org/setup/
>> > >>>
>> > >>> Thanks.
>> >
>>