[jira] [Commented] (BOOKKEEPER-1095) Long Poll - Client side changes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069689#comment-16069689
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1095:


Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/220
  
@sijie awesome
I hope that "simple clients" in the future will be able to leverage such 
powerful feature without having to add new "coordination services middlewares". 
Let's discuss this in a separate thread.
I have never tried to use "really" DistributedLog, I will give it a try.
I am really happy with having it merged in BK ! Thank you


> Long Poll - Client side changes
> ---
>
> Key: BOOKKEEPER-1095
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1095
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client, bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Robin Dhamankar
> Fix For: 4.5.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-816) use native fallocate & sync_file_range to improve journal allocation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069670#comment-16069670
 ] 

ASF GitHub Bot commented on BOOKKEEPER-816:
---

Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/80
  
@sijie do you have some cycle to go on with this issue ?
I apologize for being "annoying" on this topic


> use native fallocate & sync_file_range to improve journal allocation
> 
>
> Key: BOOKKEEPER-816
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-816
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> it'd better to leverage filesystem fallocate & sync_file_range for journal 
> performance.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1095) Long Poll - Client side changes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069659#comment-16069659
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1095:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/220
  
@eolivelli I think the best way to do 'tailing' is to using the 
distributedlog library. it handles fetch and deals with the switch between 
normal reads and long poll reads when detecting lac is advancing. once this 
change is in, distributedlog should be compatible with current community master.


> Long Poll - Client side changes
> ---
>
> Key: BOOKKEEPER-1095
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1095
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client, bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Robin Dhamankar
> Fix For: 4.5.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1095) Long Poll - Client side changes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069289#comment-16069289
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1095:


GitHub user sijie opened a pull request:

https://github.com/apache/bookkeeper/pull/220

BOOKKEEPER-1095: Server and Client Side Changes

Descriptions of the changes in this PR:

- changes on FileInfo to support notifications on LAC changes
- a new ReadEntryLongPollV3 processor to process readEntry requests with 
long poll flags
- a new public API for long poll: readLastConfirmedAndEntry. if it is 
reading beyond the LAC, it will become a long poll request and wait for 
advancing LAC on bookie side; if it isn't reading beyond the LAC it will be 
normal reads.
- also have a speculative mechanism for long poll reads.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sijie/bookkeeper BOOKKEEPER-1094

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bookkeeper/pull/220.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #220


commit 48e88fc1be0c2efee858144ce7c1288d947a2b15
Author: Sijie Guo 
Date:   2017-06-21T21:12:45Z

First attempt for porting long poll changes on client and server

commit 87fc9989d3fe3858c9c006413e028b0f78122ddb
Author: Sijie Guo 
Date:   2017-06-29T22:28:53Z

Merge branch 'master' into BOOKKEEPER-1094

commit 973e5be4c4f636c9f9ddb7898ed0794a3891103e
Author: Sijie Guo 
Date:   2017-06-30T01:02:32Z

BOOKKEEPER-1094: Long Poll - Server and Client Side Changes




> Long Poll - Client side changes
> ---
>
> Key: BOOKKEEPER-1095
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1095
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client, bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Robin Dhamankar
> Fix For: 4.5.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-759) bookkeeper: delay ensemble change if it doesn't break ack quorum requirement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069081#comment-16069081
 ] 

ASF GitHub Bot commented on BOOKKEEPER-759:
---

Github user sijie closed the pull request at:

https://github.com/apache/bookkeeper/pull/202


> bookkeeper: delay ensemble change if it doesn't break ack quorum requirement
> 
>
> Key: BOOKKEEPER-759
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-759
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> flag to allow delay ensemble change. if that is set to change, will not do 
> ensemble change until it breaks ack quorum requirement



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068720#comment-16068720
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@jvrao @reddycharan - that's one of my comments as well. because I don't 
have a clear picture what directories are concerned. 


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068658#comment-16068658
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
with this change they introduced multiplejournals/journaldirectories - 
https://github.com/apache/bookkeeper/commit/123eccd435a4a96a9147ed4a24efbe9025fe79ba


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068642#comment-16068642
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user jvrao commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
In my comments I called journal dirs. but infact they are ledger dirs. 
Typos from my side.
As far as journals are concerned, we can have only one journal dir right? 
@reddycharan what do you mean by " Will add check for journaldirectories " ?


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068636#comment-16068636
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
and yes, I didn't add check for journaldirectories. Will add check for 
journaldirectories also. Thanks for pointing that out.


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1028) inc/excl opts listunderreplicate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067419#comment-16067419
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1028:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/127
  
@sijie  my bad, missed checking findbugs error after previous push. Fixed 
them now and it passed.

Thanks.


> inc/excl opts listunderreplicate
> 
>
> Key: BOOKKEEPER-1028
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1028
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - Introduce including and excluding BookieId options
> for listunderreplicatedLedgers



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067297#comment-16067297
 ] 

ASF GitHub Bot commented on BOOKKEEPER-588:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/183
  
fyi, I added @merlimat and me to the reviewers list, making sure we have 
other eyes on reviewing this change.


> SSL support
> ---
>
> Key: BOOKKEEPER-588
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-588
> Project: Bookkeeper
>  Issue Type: Sub-task
>Reporter: Ivan Kelly
>Assignee: Enrico Olivelli
> Fix For: 4.5.0
>
> Attachments: 0001-MutualTLS-for-Bookkeeper.patch, 
> 0004-BOOKKEEPER-588-SSL-support-for-bookkeeper.patch
>
>
> SSL support using startTLS



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067288#comment-16067288
 ] 

ASF GitHub Bot commented on BOOKKEEPER-974:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/197
  
@caiok any updates on this pull request?


> make pushing a docker image as part of the release procedure
> 
>
> Key: BOOKKEEPER-974
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Jia Zhai
>
> make pushing a docker image as part of the release procedure.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067282#comment-16067282
 ] 

ASF GitHub Bot commented on BOOKKEEPER-588:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/183
  
@eolivelli I will try to go through this again. 

/cc @merlimat  for reviewing as well


> SSL support
> ---
>
> Key: BOOKKEEPER-588
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-588
> Project: Bookkeeper
>  Issue Type: Sub-task
>Reporter: Ivan Kelly
>Assignee: Enrico Olivelli
> Fix For: 4.5.0
>
> Attachments: 0001-MutualTLS-for-Bookkeeper.patch, 
> 0004-BOOKKEEPER-588-SSL-support-for-bookkeeper.patch
>
>
> SSL support using startTLS



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1028) inc/excl opts listunderreplicate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067280#comment-16067280
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1028:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/127
  
@reddycharan there is a findbugs error introduced in this pull request. Can 
you fix that?


> inc/excl opts listunderreplicate
> 
>
> Key: BOOKKEEPER-1028
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1028
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - Introduce including and excluding BookieId options
> for listunderreplicatedLedgers



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1098) ZkUnderreplicationManager can build up an unbounded number of watchers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067278#comment-16067278
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1098:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/193
  
@athanatos yeah, I just changed the title for the correct jira number. 


> ZkUnderreplicationManager can build up an unbounded number of watchers
> --
>
> Key: BOOKKEEPER-1098
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1098
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-auto-recovery
>Reporter: Samuel Just
>Assignee: Samuel Just
>Priority: Minor
> Fix For: 4.5.0
>
>
> getLedgerToReplicate leaves watches each time it traverses the
> tree until it finds a suitable replication target.  Since we don't have
> a way of canceling watches, these watches tend to get abandoned,
> particularly on interior nodes, which aren't changed much.  Thus,
> over time, some nodes would build up a very large number of watches.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1098) ZkUnderreplicationManager can build up an unbounded number of watchers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067267#comment-16067267
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1098:


Github user athanatos commented on the issue:

https://github.com/apache/bookkeeper/pull/193
  
@sijie Oops, I just noticed the comments.  I guess you just changed the 
title?


> ZkUnderreplicationManager can build up an unbounded number of watchers
> --
>
> Key: BOOKKEEPER-1098
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1098
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-auto-recovery
>Reporter: Samuel Just
>Assignee: Samuel Just
>Priority: Minor
> Fix For: 4.5.0
>
>
> getLedgerToReplicate leaves watches each time it traverses the
> tree until it finds a suitable replication target.  Since we don't have
> a way of canceling watches, these watches tend to get abandoned,
> particularly on interior nodes, which aren't changed much.  Thus,
> over time, some nodes would build up a very large number of watches.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-753) Bookie should run garbage collection before startup when all directories became full

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067258#comment-16067258
 ] 

ASF GitHub Bot commented on BOOKKEEPER-753:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/81
  
@jvrao @reddycharan #190 is merged. let me know what is your thought of 
this pull request.


> Bookie should run garbage collection before startup when all directories 
> became full
> 
>
> Key: BOOKKEEPER-753
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-753
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> otherwise, a bookie that ran-out-of disk would never be started up again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1034) When all disks are full, start Bookie in RO mode if RO mode is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067253#comment-16067253
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1034:


Github user sijie closed the pull request at:

https://github.com/apache/bookkeeper/pull/190


> When all disks are full, start Bookie in RO mode if RO mode is enabled 
> ---
>
> Key: BOOKKEEPER-1034
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1034
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
> Fix For: 4.5.0
>
>
> - When all disks are full start Bookie in RO mode if RO mode is enabled
> - This will work only if "isForceGCAllowWhenNoSpace" is allowed, since 
> LedgerDirsManager.getWritableLedgerDirsForNewLog will be able to find new 
> writableLedgerDir even when all disks are full.
> - If bookie has died abruptly then it may have missed flushing EntryMemtable 
> and
> IndexInMemoryPageManager. So next time when it starts when disc is full
> it fails to create index file and it shuts down. So Bookie should be able to 
> create index file though it has reached the diskusagethreshold, while 
> starting the Bookie in Readonly Mode. But ofcourse there should be some 
> config to safeguard when disk usable space is so low.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067060#comment-16067060
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@jvrao I am fine with having a configuration setting as what I said before.

The comments are mostly for people knowing different aspects on the 
solutions.


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066823#comment-16066823
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user jvrao commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
Multiple entrylog change is coming in and Charan is using this for that. So
you can think of this as setting stage for that.
If you don't want to have the flag/configuration parameter we can get rid
of it; That is not really needed for what we need.

If you feel strongly against this configuration param, we can flag a
warning instead of error if user configured multiple entrylogs on same
disk. Its a bit of code change and another commit not a biggie.



On Wed, Jun 28, 2017 at 9:25 AM, Sijie Guo  wrote:

> re 1.
>
> I am not aware if they are users configuring in this way. but I am not
> sure if this is the case for flagging as error. because in current
> implementation, such configuration doesn't have any performance impacts. 
if
> this flag is necessarily for multiple entry log files, I would defer 
adding
> this flag when reviewing the multiple-active-entry-log-files change.
> because it is not very clear about the problem and benefits.
>
> re 2>
>
> this sounds like a policy problem - how to choose the directory to place
> the entry log files to achieve balance between disk partitions? currently
> bookie asks the ledgerdirs manager for a ledger directory. the ledger 
disks
> can do a better job for balancing the placement of entry log files.
>
> re 3.
>
> if we have a policy abstraction in LedgerDirsManager, we can have a better
> policy on how to pick up the entry log files for write. in this way, we 
can
> balance the disk usage and I/O usage.
>
> I am seeing a trend of adding more and more configuration settings into
> the bookies. some are strongly required. some doesn't seem to be really
> needed. In this pull request, the configuration doesn't seem to be the
> right solution that address the problems that you guys are listing. it
> seems to be me a flag to mask the underneath problems and this flag will
> most likely be used anymore when we have the correct solution in place.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 

> .
>



-- 
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi



> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066813#comment-16066813
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
>> re 1.

I am not aware if they are users configuring in this way. but I am not sure 
if this is the case for flagging as error. because in current implementation, 
such configuration doesn't have any performance impacts. if this flag is 
necessarily for multiple entry log files, I would defer adding this flag when 
reviewing the multiple-active-entry-log-files change. because it is not very 
clear about the problem and benefits.

>> re 2>

this sounds like a policy problem - how to choose the directory to place 
the entry log files to achieve balance between disk partitions? currently 
bookie asks the ledgerdirs manager for a ledger directory. the ledger disks can 
do a better job for balancing the placement of entry log files.

>> re 3.

if we have a policy abstraction in LedgerDirsManager, we can have a better 
policy on how to pick up the entry log files for write. in this way, we can 
balance the disk usage and I/O usage.

I am seeing a trend of adding more and more configuration settings into the 
bookies. some are strongly required. some doesn't seem to be really needed. In 
this pull request, the configuration doesn't seem to be the right solution that 
address the problems that you guys are listing. it seems to be me a flag to 
mask the underneath problems and this flag will most likely be used anymore 
when we have the correct solution in place.


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1100) Add Http Endpoint for Bookkeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066479#comment-16066479
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1100:


Github user jvrao commented on the issue:

https://github.com/apache/bookkeeper/pull/210
  
We are adding REST endpoint to BookKeeper. I need to check if they overlap. 


> Add Http Endpoint for Bookkeeper
> 
>
> Key: BOOKKEEPER-1100
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1100
> Project: Bookkeeper
>  Issue Type: New Feature
>  Components: bookkeeper-server
>Affects Versions: 4.5.0
>Reporter: Yiming Zang
>Assignee: Yiming Zang
> Fix For: 4.5.0
>
>
> Add a Http Server Component for Bookkeeper.
> It would be great that Bookkeeper can have an Http Server to expose some 
> useful APIs. We can expose and manage the server status, server 
> configuration, lifecycle states, query or trigger auto recovery through HTTP 
> Endpoint, or even trigger GC and compaction over Http Endpoint.
> This ticket is mainly to add a Http framework for Bookkeeper, so that we can 
> extend more useful APIs based on the existing Http framework.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1100) Add Http Endpoint for Bookkeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066090#comment-16066090
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1100:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/210
  
regarding the servlet, I agree with @eolivelli , we probably should define 
a standard API and allow it running with different web container. @yzang : do 
finagle and vertx support servlet?


> Add Http Endpoint for Bookkeeper
> 
>
> Key: BOOKKEEPER-1100
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1100
> Project: Bookkeeper
>  Issue Type: New Feature
>  Components: bookkeeper-server
>Affects Versions: 4.5.0
>Reporter: Yiming Zang
>Assignee: Yiming Zang
> Fix For: 4.5.0
>
>
> Add a Http Server Component for Bookkeeper.
> It would be great that Bookkeeper can have an Http Server to expose some 
> useful APIs. We can expose and manage the server status, server 
> configuration, lifecycle states, query or trigger auto recovery through HTTP 
> Endpoint, or even trigger GC and compaction over Http Endpoint.
> This ticket is mainly to add a Http framework for Bookkeeper, so that we can 
> extend more useful APIs based on the existing Http framework.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066056#comment-16066056
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@jvrao @reddycharan 

one more question, most of the comments are around the performance on 
journal directories. but it turns out this change has nothing to do with the 
journal directories. The change applies only on the ledger directories and 
index directories. if that's the case, I don't see a reason to have this 
configuration. because random I/O anyway exists on ledger and index 
directories, there isn't any performance difference between configuring one 
directory per disk partition or multiple directories per disk partition. 
Filesystem I/O scheduling will handle either case in the same way. so this 
doesn't sound like an operation error as what @jvrao said.

so back to the question, what is the problem that this pull request want to 
address? 

- if the problem is about the mis-reported disk metrics, this can be done 
smarter by de-duplicating the directories within ledger disks and report the 
per disk partition metrics.
- if the problem is about performance, multiple ledger/index disks on same 
partition doesn't have any performance difference with one ledger/index disk 
per disk partition. this doesn't sound like an operation error that needs to be 
fixed.
- if the problem is about performance on journal disks, I am fine with such 
setting. but this pull request need to improve to address the right problem.





> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066032#comment-16066032
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@jvrao yes, that's the idea. 

I am +0 on this. It is fine for me to adding an option, although I still 
don't think it is the right approach to do this.


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065761#comment-16065761
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user jvrao commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
Right. So if there are two dirs /journal1 and /journal2 both were created
on the same disk.
With the dedup, we will choose only one say, /journal1 and /journal2 is
always kept empty. Am I getting it right?
We do the dedup logic even while getting the disk capacity etc etc. So here
we are masking/managing mis=configuration

But our point is, configuring two directories under same disk partition is
itself wrong, unless user wanted to do it explicitly
for some test reasons. That is why we have that configuration variable,
that forces operator error, unless it is set explicitly " I know what I am
doing" kind.

Makes sense?

On Tue, Jun 27, 2017 at 2:10 PM, Sijie Guo  wrote:

> @jvrao 
>
> sorry, what is the performance problem behind this? is there anything I
> missed in this conversation?
>
> If a user configures multiple journal directories, can we dedup and use
> one journal directory per disk partition? If we dedup, there is no
> performance degradation and there is no backward compatibility issue, 
right?
>
> For example, we open a entrylog file by calling
> *ledgerDirsManager.getWritableLedgerDirsForNewLog()* , the dirs manager
> can handle the duplication (make sure one directory per disk partition is
> used) and return the right directory. By doing this, all the 
directory/disk
> partition related complexity is hidden and managed and people don't have
> the know the details.
>
> I don't have any strong opinion adding another setting. My feeling is that
> we are not taking the right approach to address the problem here.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 

> .
>



-- 
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi



> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065468#comment-16065468
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@jvrao 

sorry, what is the performance problem behind this? is there anything I 
missed in this conversation?

If a user configures multiple journal directories, can we dedup and use one 
journal directory per disk partition? If we dedup, there is no performance 
degradation and there is no backward compatibility issue, right?

For example, we open a entrylog file by calling 
**ledgerDirsManager.getWritableLedgerDirsForNewLog()** , the dirs manager can 
handle the duplication (make sure one directory per disk partition is used) and 
return the right directory. By doing this, all the directory/disk partition 
related complexity is hidden and managed and people don't have the know the 
details.

I don't have any strong opinion adding another setting. My feeling is that 
we are not taking the right approach to address the problem here.



> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-816) use native fallocate & sync_file_range to improve journal allocation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065446#comment-16065446
 ] 

ASF GitHub Bot commented on BOOKKEEPER-816:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/80
  
@eolivelli thank you


> use native fallocate & sync_file_range to improve journal allocation
> 
>
> Key: BOOKKEEPER-816
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-816
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> it'd better to leverage filesystem fallocate & sync_file_range for journal 
> performance.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-816) use native fallocate & sync_file_range to improve journal allocation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065434#comment-16065434
 ] 

ASF GitHub Bot commented on BOOKKEEPER-816:
---

Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/80
  
@sijie I am going to use this improvement in order to make a necessary 
change in order to support java9. I would like to make 4.5 runnable on java9 as 
we are starting to test all of our apps and bookies cannot run on java9 without 
 hacks


> use native fallocate & sync_file_range to improve journal allocation
> 
>
> Key: BOOKKEEPER-816
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-816
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> it'd better to leverage filesystem fallocate & sync_file_range for journal 
> performance.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065218#comment-16065218
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user jvrao commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
The flag is to allow backward compatibility and warn the user if they 
configured multiple journal dirs on the same partition by mistake. Per @sijie's 
point, yes we can handle them by monitoring the disk partition, and mark all 
dirs on that dir as non-writable. But this doesn't address the perf problem 
@reddycharan mentioned. I think we need to look at this in two folds.
1. Add correct metrics for reporting capacity, and keep existing behavior. 
This is what @reddycharan did.
2. @sijie suggesting enhancing existing behavior, which can be handeled 
with another Jira item.

Thoughts? 


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1028) inc/excl opts listunderreplicate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063975#comment-16063975
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1028:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/127
  
@sijie addressed your comments -
- move logic from BKShell to BookkeeperAdmin
- added functional testcases
- make DistributionSchedule and RoundRobinDistributionSchedule non-public


> inc/excl opts listunderreplicate
> 
>
> Key: BOOKKEEPER-1028
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1028
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - Introduce including and excluding BookieId options
> for listunderreplicatedLedgers



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1034) When all disks are full, start Bookie in RO mode if RO mode is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063961#comment-16063961
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1034:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/190
  
@sijie fixed tailing spaces


> When all disks are full, start Bookie in RO mode if RO mode is enabled 
> ---
>
> Key: BOOKKEEPER-1034
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1034
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - When all disks are full start Bookie in RO mode if RO mode is enabled
> - This will work only if "isForceGCAllowWhenNoSpace" is allowed, since 
> LedgerDirsManager.getWritableLedgerDirsForNewLog will be able to find new 
> writableLedgerDir even when all disks are full.
> - If bookie has died abruptly then it may have missed flushing EntryMemtable 
> and
> IndexInMemoryPageManager. So next time when it starts when disc is full
> it fails to create index file and it shuts down. So Bookie should be able to 
> create index file though it has reached the diskusagethreshold, while 
> starting the Bookie in Readonly Mode. But ofcourse there should be some 
> config to safeguard when disk usable space is so low.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062816#comment-16062816
 ] 

ASF GitHub Bot commented on BOOKKEEPER-588:
---

Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/183
  
@sijie for me this patch is OK, I don't know if other developers/committer 
want to add other comments

@kishorekasi can you change the description of the PR which usually it 
written to "git log" to remove the (wrong) reference to my github account and 
add a reference to @ivankelly which was the original author of the patch ?

We will need to add the documentation to the security docs, maybe we can do 
it after committing the general introduction to "security" which I am working on


> SSL support
> ---
>
> Key: BOOKKEEPER-588
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-588
> Project: Bookkeeper
>  Issue Type: Sub-task
>Reporter: Ivan Kelly
>Assignee: Enrico Olivelli
> Fix For: 4.5.0
>
> Attachments: 0001-MutualTLS-for-Bookkeeper.patch, 
> 0004-BOOKKEEPER-588-SSL-support-for-bookkeeper.patch
>
>
> SSL support using startTLS



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1100) Add Http Endpoint for Bookkeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062613#comment-16062613
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1100:


Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/210
  
Some comments:

1) is it possible NOT to bundle all the implementations in 
bookkeeper-server project ?
Or at least mark the dependencies as "provided" o "runtime" so that we do 
not make clients import transitively all that new jars. Maybe we can do as for 
stats-providers
2) we are giving access to Bookie configuration, what about security ? 
Maybe an user can provide a custom implementation of the HttpServer which 
requires auth ?
3) @reddycharan I think that this could be the base for implementing the 
dropped JMX-based features, we could (in another issue)
4) I think that this proposal does not 'force' a clear HTTP API to 
implementations, that is that from the various implementations it seems that we 
want to provide a REST-like API but it is really up to the implementations. 
This actually means that no one can write other tools to integrate with the 
Bookie, as the API will depend on the actual provider .
It would be better to decide a standard API and document it.
I am really in favor to make the low level implementation "pluggable": for 
instance I run my Bookies in the same process of a Tomcat and/or with Jetty.

I think that a better approach would be to provide a standard HttpServlet 
which could be deployed to any standard Web Container. This way we (BookKeeper 
community) will be driving the API and the security aspects of the Bookie



> Add Http Endpoint for Bookkeeper
> 
>
> Key: BOOKKEEPER-1100
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1100
> Project: Bookkeeper
>  Issue Type: New Feature
>  Components: bookkeeper-server
>Affects Versions: 4.5.0
>Reporter: Yiming Zang
>Assignee: Yiming Zang
> Fix For: 4.5.0
>
>
> Add a Http Server Component for Bookkeeper.
> It would be great that Bookkeeper can have an Http Server to expose some 
> useful APIs. We can expose and manage the server status, server 
> configuration, lifecycle states, query or trigger auto recovery through HTTP 
> Endpoint, or even trigger GC and compaction over Http Endpoint.
> This ticket is mainly to add a Http framework for Bookkeeper, so that we can 
> extend more useful APIs based on the existing Http framework.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062537#comment-16062537
 ] 

ASF GitHub Bot commented on BOOKKEEPER-588:
---

Github user kishorekasi commented on the issue:

https://github.com/apache/bookkeeper/pull/183
  
Updated the pull request. I am not able to reproduced the previous jenkins 
failure locally. With this new update I will wait for the jenkins run to check 
if the failures are repeatable.


> SSL support
> ---
>
> Key: BOOKKEEPER-588
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-588
> Project: Bookkeeper
>  Issue Type: Sub-task
>Reporter: Ivan Kelly
>Assignee: Enrico Olivelli
> Fix For: 4.5.0
>
> Attachments: 0001-MutualTLS-for-Bookkeeper.patch, 
> 0004-BOOKKEEPER-588-SSL-support-for-bookkeeper.patch
>
>
> SSL support using startTLS



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1100) Add Http Endpoint for Bookkeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062262#comment-16062262
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1100:


Github user jiazhai commented on the issue:

https://github.com/apache/bookkeeper/pull/210
  
@yzang, Would you please help change the last part of this PR description a 
little?  If you did the checks, put an "x" in the "[ ]".


> Add Http Endpoint for Bookkeeper
> 
>
> Key: BOOKKEEPER-1100
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1100
> Project: Bookkeeper
>  Issue Type: New Feature
>  Components: bookkeeper-server
>Affects Versions: 4.5.0
>Reporter: Yiming Zang
>Assignee: Yiming Zang
> Fix For: 4.5.0
>
>
> Add a Http Server Component for Bookkeeper.
> It would be great that Bookkeeper can have an Http Server to expose some 
> useful APIs. We can expose and manage the server status, server 
> configuration, lifecycle states, query or trigger auto recovery through HTTP 
> Endpoint, or even trigger GC and compaction over Http Endpoint.
> This ticket is mainly to add a Http framework for Bookkeeper, so that we can 
> extend more useful APIs based on the existing Http framework.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1086) Ledger Recovery - Refactor PendingReadOp

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059451#comment-16059451
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1086:


Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/193
  
@athanatos can you change the title to "BOOKKEEPER-1098: 
ZkUnderreplicationManager cache watcher"
our bot is binding comments to the wrong issue


> Ledger Recovery - Refactor PendingReadOp
> 
>
> Key: BOOKKEEPER-1086
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1086
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
>  bookkeeper recovery improvement (part-1): refactor PendingReadOp
> this change is the first part of improving bookkeeper recovery. it is 
> basically a refactor change, which:
> - abstract an interface for LedgerEntryRequest in PendingReadOp
> - rename current implementation to SequenceReadRequest, which read the 
> entry in the sequence of quorum.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1086) Ledger Recovery - Refactor PendingReadOp

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059436#comment-16059436
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1086:


Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/193
  
@sijie I think this change can be merged


> Ledger Recovery - Refactor PendingReadOp
> 
>
> Key: BOOKKEEPER-1086
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1086
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
>  bookkeeper recovery improvement (part-1): refactor PendingReadOp
> this change is the first part of improving bookkeeper recovery. it is 
> basically a refactor change, which:
> - abstract an interface for LedgerEntryRequest in PendingReadOp
> - rename current implementation to SequenceReadRequest, which read the 
> entry in the sequence of quorum.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-759) bookkeeper: delay ensemble change if it doesn't break ack quorum requirement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058542#comment-16058542
 ] 

ASF GitHub Bot commented on BOOKKEEPER-759:
---

GitHub user sijie opened a pull request:

https://github.com/apache/bookkeeper/pull/202

BOOKKEEPER-759: Delay Ensemble Change & Disable Ensemble Change

Descriptions of the changes in this PR:

This pull request contains the changes around ensemble change.

- Delay Ensemble Change: Provide a flag to allow delay ensemble change. if 
that is set to change, will not do ensemble change until it breaks ack quorum 
requirements.
- Disable Ensemble Change: Provide a runtime feature flag to allow 
disabling ensemble change. The ensemble change behavior can be disabled 
on-the-fly via the FeatureProvider.

---
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

- [x] Make sure the PR title is formatted like:
`: Description of pull request`
`e.g. Issue 123: Description ...`
- [x] Make sure tests pass via `mvn clean apache-rat:check install 
findbugs:check`.
- [x] Replace `` in the title with the actual Issue number, if 
there is one.

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sijie/bookkeeper 
client_changes/delay_ensemble_changes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bookkeeper/pull/202.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #202


commit b75dbf32f0831457980d976c50bb1df1a26726de
Author: Sijie Guo 
Date:   2014-03-11T23:09:38Z

bookkeeper: delay ensemble change if it doesn't break ack quorum requirement

- flag to allow delay ensemble change. if that is set to change, will not 
do ensemble change until it breaks ack quorum requirements
- add test case for delaying ensemble changes.




> bookkeeper: delay ensemble change if it doesn't break ack quorum requirement
> 
>
> Key: BOOKKEEPER-759
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-759
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> flag to allow delay ensemble change. if that is set to change, will not do 
> ensemble change until it breaks ack quorum requirement



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1028) inc/excl opts listunderreplicate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058049#comment-16058049
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1028:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/127
  
@sijie if you dont have anymore comments, this commit also should be good 
to go.


> inc/excl opts listunderreplicate
> 
>
> Key: BOOKKEEPER-1028
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1028
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - Introduce including and excluding BookieId options
> for listunderreplicatedLedgers



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1097) GC test when no WritableDirs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058038#comment-16058038
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1097:


Github user asfgit closed the pull request at:

https://github.com/apache/bookkeeper/pull/188


> GC test when no WritableDirs
> 
>
> Key: BOOKKEEPER-1097
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1097
> Project: Bookkeeper
>  Issue Type: Test
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
> Fix For: 4.5.0
>
>
> 
> - Functional test validating that Compaction takes place even if there
> are no writableledgerdir but there are ledgerdirs according to
> LedgerDirsManager.getWritableLedgerDirsForNewLog
> 
> - end-to-end testcase of Bookie recovery, incase of Bookie ledgerdir 
> reaching
> the threshold, and recovering by forcing the gc/compaction



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1092) Ledger Recovery - Add Test Case for Parallel Ledger Recovery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058029#comment-16058029
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1092:


Github user asfgit closed the pull request at:

https://github.com/apache/bookkeeper/pull/182


> Ledger Recovery - Add Test Case for Parallel Ledger Recovery
> 
>
> Key: BOOKKEEPER-1092
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1092
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> Add test case for parallel ledger recovery



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058018#comment-16058018
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
The concern I have is - people don't have to care about this flag.

User should still configure the directories in the way how they are 
configuring right now. The bookie can be smart to figure about the directory 
and disk-partition layout. You can build the Map in 
LedgerDirManager. So the LedgerDirManager does the de-duplication and handle 
disk filled up correctly. When a DiskPartition fills up, all the directories on 
this disk partition should be marked as non-writable.

If the concern is about the duplicated disk partitions fill the ledger 
directory checking. we should just fix the behavior, rather than exposing 
another configuration setting to the users. 


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1093) Piggyback LAC on ReadResponse

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057932#comment-16057932
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1093:


Github user asfgit closed the pull request at:

https://github.com/apache/bookkeeper/pull/180


> Piggyback LAC on ReadResponse
> -
>
> Key: BOOKKEEPER-1093
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1093
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client, bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> Currently LAC is only updated when the reader explicitly calls 
> #readLastAddConfirmed(). In tailing-read use cases, it will not wise to keep 
> calling #readLastAddConfirmed, especially when the traffic is huge.
> The idea here is piggy-back LAC along with the read responses. so the client 
> will get advanced LAC along with read responses. so it will reduce calling 
> #readLastAddConfirmed. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1034) When all disks are full, start Bookie in RO mode if RO mode is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057925#comment-16057925
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1034:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/190
  
@reddycharan I don't have more comments. please fix the tailing spaces.


> When all disks are full, start Bookie in RO mode if RO mode is enabled 
> ---
>
> Key: BOOKKEEPER-1034
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1034
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - When all disks are full start Bookie in RO mode if RO mode is enabled
> - This will work only if "isForceGCAllowWhenNoSpace" is allowed, since 
> LedgerDirsManager.getWritableLedgerDirsForNewLog will be able to find new 
> writableLedgerDir even when all disks are full.
> - If bookie has died abruptly then it may have missed flushing EntryMemtable 
> and
> IndexInMemoryPageManager. So next time when it starts when disc is full
> it fails to create index file and it shuts down. So Bookie should be able to 
> create index file though it has reached the diskusagethreshold, while 
> starting the Bookie in Readonly Mode. But ofcourse there should be some 
> config to safeguard when disk usable space is so low.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1089) Ledger Recovery - allow batch reads in ledger recovery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057916#comment-16057916
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1089:


Github user asfgit closed the pull request at:

https://github.com/apache/bookkeeper/pull/181


> Ledger Recovery - allow batch reads in ledger recovery
> --
>
> Key: BOOKKEEPER-1089
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1089
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> bookkeeper recovery improvement (part-4): allow batch reading in ledger 
> recovery
> - enable batch read in ledger recovery, so we could parallel reading to 
> improve recovery time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057420#comment-16057420
 ] 

ASF GitHub Bot commented on BOOKKEEPER-974:
---

Github user jiazhai commented on the issue:

https://github.com/apache/bookkeeper/pull/197
  
There seems be an empty file: "docker/4.4.0/.keep", what is the reason to 
have it?


> make pushing a docker image as part of the release procedure
> 
>
> Key: BOOKKEEPER-974
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Jia Zhai
>
> make pushing a docker image as part of the release procedure.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1097) GC test when no WritableDirs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054617#comment-16054617
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1097:


Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/188
  
@reddycharan the patch is ok. I think @sijie will try merge current patch 
list in the best order


> GC test when no WritableDirs
> 
>
> Key: BOOKKEEPER-1097
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1097
> Project: Bookkeeper
>  Issue Type: Test
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> 
> - Functional test validating that Compaction takes place even if there
> are no writableledgerdir but there are ledgerdirs according to
> LedgerDirsManager.getWritableLedgerDirsForNewLog
> 
> - end-to-end testcase of Bookie recovery, incase of Bookie ledgerdir 
> reaching
> the threshold, and recovering by forcing the gc/compaction



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1097) GC test when no WritableDirs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054605#comment-16054605
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1097:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/188
  
Thanks. I think this commit should be good to go.


> GC test when no WritableDirs
> 
>
> Key: BOOKKEEPER-1097
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1097
> Project: Bookkeeper
>  Issue Type: Test
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> 
> - Functional test validating that Compaction takes place even if there
> are no writableledgerdir but there are ledgerdirs according to
> LedgerDirsManager.getWritableLedgerDirsForNewLog
> 
> - end-to-end testcase of Bookie recovery, incase of Bookie ledgerdir 
> reaching
> the threshold, and recovering by forcing the gc/compaction



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054604#comment-16054604
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@sijie @eolivelli anymore comments/concerns?


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1034) When all disks are full, start Bookie in RO mode if RO mode is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054603#comment-16054603
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1034:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/190
  
@sijie if you dont have anymore comments, I can fix the tailing spaces and 
send new CR.


> When all disks are full, start Bookie in RO mode if RO mode is enabled 
> ---
>
> Key: BOOKKEEPER-1034
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1034
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - When all disks are full start Bookie in RO mode if RO mode is enabled
> - This will work only if "isForceGCAllowWhenNoSpace" is allowed, since 
> LedgerDirsManager.getWritableLedgerDirsForNewLog will be able to find new 
> writableLedgerDir even when all disks are full.
> - If bookie has died abruptly then it may have missed flushing EntryMemtable 
> and
> IndexInMemoryPageManager. So next time when it starts when disc is full
> it fails to create index file and it shuts down. So Bookie should be able to 
> create index file though it has reached the diskusagethreshold, while 
> starting the Bookie in Readonly Mode. But ofcourse there should be some 
> config to safeguard when disk usable space is so low.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052515#comment-16052515
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
Yes @eolivelli.

I want the option to be available for users to choose whatever layout they 
want and provide backward compatibility, but at the same time provide a config 
to restrict the users not to inadvertently configure multiple ledgerdirs in the 
same partition if it is not desired. To satisfy all needs this seems to be 
easier.


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051035#comment-16051035
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@sijie as far as I can see the flag only adds an additional check on the 
configuration and it does not affect the way we are going to calculate disk 
space (which now it will be correct).

@reddycharan Do I understand correctly ?



> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051005#comment-16051005
 ] 

ASF GitHub Bot commented on BOOKKEEPER-588:
---

Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/183
  
Overall is ok,
 but rat is failing and on jenkins one of the new tests failed, the mixed 
cluster one

Can you fix it?


> SSL support
> ---
>
> Key: BOOKKEEPER-588
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-588
> Project: Bookkeeper
>  Issue Type: Sub-task
>Reporter: Ivan Kelly
>Assignee: Enrico Olivelli
> Fix For: 4.5.0
>
> Attachments: 0001-MutualTLS-for-Bookkeeper.patch, 
> 0004-BOOKKEEPER-588-SSL-support-for-bookkeeper.patch
>
>
> SSL support using startTLS



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1086) Ledger Recovery - Refactor PendingReadOp

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050856#comment-16050856
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1086:


GitHub user athanatos opened a pull request:

https://github.com/apache/bookkeeper/pull/193

BOOKKEEPER-1086: ZkUnderreplicationManager cache watcher

Previously, getLedgerToReplicate left watches each time it traversed the
tree until it found a suitable replication target.  Since we don't have
a way of canceling watches, these watches tended to get abandoned,
particularly on interior nodes, which aren't changed much.  Thus,
over time, some nodes would build up a very large number of watch.

Instead, introduce a caching mechanism to remember outstanding watches
and avoid ever creating two watches on the same node.

Author: Samuel Just 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/athanatos/bookkeeper 
forupstream/BOOKKEEPER-1098

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bookkeeper/pull/193.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #193


commit cb3c7a2f4e256de00268740c37fc2a2ecd9b4e31
Author: Samuel Just 
Date:   2017-03-29T19:06:56Z

BOOKKEEPER-1086: ZkUnderreplicationManager cache watcher

Previously, getLedgerToReplicate left watches each time it traversed the
tree until it found a suitable replication target.  Since we don't have
a way of canceling watches, these watches tended to get abandoned,
particularly on interior nodes, which aren't changed much.  Thus,
over time, some nodes would build up a very large number of watch.

Instead, introduce a caching mechanism to remember outstanding watches
and avoid ever creating two watches on the same node.

Author: Samuel Just 




> Ledger Recovery - Refactor PendingReadOp
> 
>
> Key: BOOKKEEPER-1086
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1086
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
>  bookkeeper recovery improvement (part-1): refactor PendingReadOp
> this change is the first part of improving bookkeeper recovery. it is 
> basically a refactor change, which:
> - abstract an interface for LedgerEntryRequest in PendingReadOp
> - rename current implementation to SequenceReadRequest, which read the 
> entry in the sequence of quorum.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1034) When all disks are full, start Bookie in RO mode if RO mode is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048380#comment-16048380
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1034:


GitHub user reddycharan opened a pull request:

https://github.com/apache/bookkeeper/pull/190

BOOKKEEPER-1034: Bookie start in RO when diskfull

When the disk is above threshold, Bookie goes to RO. If we have to restart 
the
bookie, on the way back, bookie tries to create new entrylog and other 
files,
which will fail because disk usage is above threshold,
hence bookie refuses to come up. So with this fix we will try to start in RO
mode if RO is enabled.

Also, if bookie has died abruptly then it may missed flushing EntryMemtable 
and
IndexInMemoryPageManager. So next time when it starts when disc is full
it is failing to create index file and it is shutting down, though we 
expect it
to start in readonlymode. So Bookie should be able to create index file
though it has reached the diskusagethreshold, while starting the Bookie in
Readonly Mode. But ofcourse there should be some config to safeguard when
disk usable space is so low.

Minor fixes in shutdown logic of Bookie and Bookieserver.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reddycharan/bookkeeper 
bookiestartinreadonlywhendiskfull

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bookkeeper/pull/190.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #190


commit 4c3dccbb747c1403793c535760013007f633f2ca
Author: Charan Reddy Guttapalem 
Date:   2017-03-30T01:30:22Z

BOOKKEEPER-1034: Bookie start in RO when diskfull

When the disk is above threshold, Bookie goes to RO. If we have to restart 
the
bookie, on the way back, bookie tries to create new entrylog and other 
files,
which will fail because disk usage is above threshold,
hence bookie refuses to come up. So with this fix we will try to start in RO
mode if RO is enabled.

Also, if bookie has died abruptly then it may missed flushing EntryMemtable 
and
IndexInMemoryPageManager. So next time when it starts when disc is full
it is failing to create index file and it is shutting down, though we 
expect it
to start in readonlymode. So Bookie should be able to create index file
though it has reached the diskusagethreshold, while starting the Bookie in
Readonly Mode. But ofcourse there should be some config to safeguard when
disk usable space is so low.

Minor fixes in shutdown logic of Bookie and Bookieserver.




> When all disks are full, start Bookie in RO mode if RO mode is enabled 
> ---
>
> Key: BOOKKEEPER-1034
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1034
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - When all disks are full start Bookie in RO mode if RO mode is enabled
> - This will work only if "isForceGCAllowWhenNoSpace" is allowed, since 
> LedgerDirsManager.getWritableLedgerDirsForNewLog will be able to find new 
> writableLedgerDir even when all disks are full.
> - If bookie has died abruptly then it may have missed flushing EntryMemtable 
> and
> IndexInMemoryPageManager. So next time when it starts when disc is full
> it fails to create index file and it shuts down. So Bookie should be able to 
> create index file though it has reached the diskusagethreshold, while 
> starting the Bookie in Readonly Mode. But ofcourse there should be some 
> config to safeguard when disk usable space is so low.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048369#comment-16048369
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@sijie do you mean to say that remove the config and make it mandatory? I 
dont think we can do that since it breaks backward compatibility and for 
development and testing purpose there should be way to configure multiple 
ledgers even on machines which have single drive.


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048359#comment-16048359
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@reddycharan gotcha. If that's the case, is the configuration flag really 
needed here?


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048354#comment-16048354
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
1) Because of this we would get accurate disk usage/availability metrics if 
multiple ledgerdirs are configured in the same drive. This is important for 
DiskWeightBasedPlacementPolicy.
2) Also, internally we would like to not to let users to inadvertently 
configure multiple ledgerdirs in the same drive , since having multiple 
ledgerdirs in the same drive is not going to improve performance.

This issues was initially raised by @revans2 in one of my earlier code 
review comments.


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048296#comment-16048296
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/189
  
@reddycharan I can understand this code change but I don't see a strong 
reason about this change. What is the issue if you configure multiple same 
directories? Also the option doesn't seem to be necessary here.


> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-753) Bookie should run garbage collection before startup when all directories became full

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048174#comment-16048174
 ] 

ASF GitHub Bot commented on BOOKKEEPER-753:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/81
  
Sure


> Bookie should run garbage collection before startup when all directories 
> became full
> 
>
> Key: BOOKKEEPER-753
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-753
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> otherwise, a bookie that ran-out-of disk would never be started up again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1028) inc/excl opts listunderreplicate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047950#comment-16047950
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1028:


Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/127
  
jenkins job crashed, I restarted it
https://builds.apache.org/job/bookkeeper-precommit-pullrequest/54/console

CI completed with 2 tests failures, I think they are not releated to the 
patch

Failed tests:
TestSpeculativeRead.testSpeculativeReadFirstReadCompleteIsOk:260
TestBackwardCompat.testCompat410:647 Shouldn't be able to write

Tests run: 1167, Failures: 2, Errors: 0, Skipped: 0




> inc/excl opts listunderreplicate
> 
>
> Key: BOOKKEEPER-1028
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1028
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - Introduce including and excluding BookieId options
> for listunderreplicatedLedgers



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-753) Bookie should run garbage collection before startup when all directories became full

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047948#comment-16047948
 ] 

ASF GitHub Bot commented on BOOKKEEPER-753:
---

Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/81
  
sorry my comments are about #127, not this issue


> Bookie should run garbage collection before startup when all directories 
> became full
> 
>
> Key: BOOKKEEPER-753
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-753
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> otherwise, a bookie that ran-out-of disk would never be started up again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-753) Bookie should run garbage collection before startup when all directories became full

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047896#comment-16047896
 ] 

ASF GitHub Bot commented on BOOKKEEPER-753:
---

Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/81
  
CI complete with due tests failures, I think they are not releated to the 
patch

Failed tests: 
  TestSpeculativeRead.testSpeculativeReadFirstReadCompleteIsOk:260
  TestBackwardCompat.testCompat410:647 Shouldn't be able to write

Tests run: 1167, Failures: 2, Errors: 0, Skipped: 0


> Bookie should run garbage collection before startup when all directories 
> became full
> 
>
> Key: BOOKKEEPER-753
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-753
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> otherwise, a bookie that ran-out-of disk would never be started up again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1033) Handle DirsPartitionDuplication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047512#comment-16047512
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1033:


GitHub user reddycharan opened a pull request:

https://github.com/apache/bookkeeper/pull/189

BOOKKEEPER-1033: Handle DirsPartitionDuplication

- Provide configuration for allowDirsPartitionDuplication
- while calculating total disk metrics account Partition Duplication

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reddycharan/bookkeeper 
dirspartitionduplication

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bookkeeper/pull/189.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #189


commit e4c165d43e1da42214aa23cc4c652774aa1f1e68
Author: cguttapalem 
Date:   2017-04-11T19:37:16Z

BOOKKEEPER-1033: Handle DirsPartitionDuplication

- Provide configuration for allowDirsPartitionDuplication
- while calculating total disk metrics account Partition Duplication




> Handle DirsPartitionDuplication
> ---
>
> Key: BOOKKEEPER-1033
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1033
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> - Provide configuration for allowDirsPartitionDuplication
> - while calculating total disk metrics account Partition Duplication



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1097) GC test when no WritableDirs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047305#comment-16047305
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1097:


GitHub user reddycharan opened a pull request:

https://github.com/apache/bookkeeper/pull/188

BOOKKEEPER-1097: GC test when no WritableDirs

- Functional test validating that Compaction takes place even if there
are no writableledgerdir but there are ledgerdirs according to
LedgerDirsManager.getWritableLedgerDirsForNewLog

- end-to-end testcase of Bookie recovery, incase of Bookie ledgerdir 
reaching
the threshold, and recovering by forcing the gc/compaction

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reddycharan/bookkeeper gctestnowritabledirs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bookkeeper/pull/188.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #188


commit df6f761ac1888919da6d0fbac941b38546997da1
Author: Charan Reddy Guttapalem 
Date:   2016-10-04T20:33:21Z

BOOKKEEPER-1097: GC test when no WritableDirs

- Functional test validating that Compaction takes place even if there
are no writableledgerdir but there are ledgerdirs according to
LedgerDirsManager.getWritableLedgerDirsForNewLog

- end-to-end testcase of Bookie recovery, incase of Bookie ledgerdir 
reaching
the threshold, and recovering by forcing the gc/compaction




> GC test when no WritableDirs
> 
>
> Key: BOOKKEEPER-1097
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1097
> Project: Bookkeeper
>  Issue Type: Test
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> 
> - Functional test validating that Compaction takes place even if there
> are no writableledgerdir but there are ledgerdirs according to
> LedgerDirsManager.getWritableLedgerDirsForNewLog
> 
> - end-to-end testcase of Bookie recovery, incase of Bookie ledgerdir 
> reaching
> the threshold, and recovering by forcing the gc/compaction



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-753) Bookie should run garbage collection before startup when all directories became full

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047193#comment-16047193
 ] 

ASF GitHub Bot commented on BOOKKEEPER-753:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/81
  
@reddycharan sure. it happened to us before, that's why the code exists. I 
am trying to reach a consensus on this change. If you guys are okay with this 
change, I am going to rebase my change.


> Bookie should run garbage collection before startup when all directories 
> became full
> 
>
> Key: BOOKKEEPER-753
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-753
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> otherwise, a bookie that ran-out-of disk would never be started up again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-753) Bookie should run garbage collection before startup when all directories became full

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047169#comment-16047169
 ] 

ASF GitHub Bot commented on BOOKKEEPER-753:
---

Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/81
  
@sijie ok, it is bit of an extreme case for it to happen (since 
BookieStorageThreshold will be generally configured at safe levels) and even in 
the case of eventuality (by misconfiguring or when bookie receives very high 
traffic before next DISK_CHECK during the DISK_CHECK_INTERVAL), we should be ok 
to lose a bookie because of write quorum.

Being said that, I don't have a strong reason to say no for providing extra 
safety mechanism / recoverability option. It should be ok, but we have to work 
on conflicts.


> Bookie should run garbage collection before startup when all directories 
> became full
> 
>
> Key: BOOKKEEPER-753
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-753
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> otherwise, a bookie that ran-out-of disk would never be started up again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1028) inc/excl opts listunderreplicate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047122#comment-16047122
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1028:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/127
  
@eolivelli I fixed it. It seems this time it has passed. Boxing/unboxing 
checks is new thing to me. Thanks for the follow up.


> inc/excl opts listunderreplicate
> 
>
> Key: BOOKKEEPER-1028
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1028
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - Introduce including and excluding BookieId options
> for listunderreplicatedLedgers



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-753) Bookie should run garbage collection before startup when all directories became full

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047104#comment-16047104
 ] 

ASF GitHub Bot commented on BOOKKEEPER-753:
---

Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/81
  
your changes might not be required, because we are letting Bookie to start 
in RO only mode when disks are full and within gcWaitTime period it will do gc 
anyhow. Also, with your change it might impact Bookie startup time, since gc is 
being done as part of startup synchronously. 

I can send my CR within a day, if you want to evaluate it.


> Bookie should run garbage collection before startup when all directories 
> became full
> 
>
> Key: BOOKKEEPER-753
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-753
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> otherwise, a bookie that ran-out-of disk would never be started up again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-753) Bookie should run garbage collection before startup when all directories became full

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046982#comment-16046982
 ] 

ASF GitHub Bot commented on BOOKKEEPER-753:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/81
  
@reddycharan yes. I think the features can co-exist. I think the question 
is - are you guys okay with this change? I am trying to move on merging 
twitter's branch.


> Bookie should run garbage collection before startup when all directories 
> became full
> 
>
> Key: BOOKKEEPER-753
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-753
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> otherwise, a bookie that ran-out-of disk would never be started up again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-753) Bookie should run garbage collection before startup when all directories became full

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046967#comment-16046967
 ] 

ASF GitHub Bot commented on BOOKKEEPER-753:
---

Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/81
  
How we handle when ledgerdirs are full when Bookie is started is little 
different
- if isReadOnlyModeEnabled is true, then we try to bring up bookie in 
ReadOnlyMode and provide ability to create EntryLog file or Index file for 
replaying Journal or for doing compaction, though storage has reached threshold.

In your change, you try to do force GC in this case and if still it hasn't 
come below threshold then you fail, if it has come below threshold then you 
start Bookie in RW mode.

So I believe, theoretically speaking both the features can coexist. Like 
following
- it tries to do force GC if storage is full during start
- if still it cannt find space, then start Bookie in RO mode and provide 
ability to create entrylog/index file for replaying Journal or for compaction.

But for pushing my code on top of yours I've to deal with bunch of 
conflicts since we are touching the same code components. 

Also, I dont think your commits are in sync with current repo code. You 
need to incorporate changes for handling LedgerDirsMonitor/LedgerDirsManager 
decoupling which we did with this change - 
https://github.com/apache/bookkeeper/commit/efd8ec26926d2e75f51d6d576543a0dcc116b83f
 .

@sijie @eolivelli @jvrao 


> Bookie should run garbage collection before startup when all directories 
> became full
> 
>
> Key: BOOKKEEPER-753
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-753
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> otherwise, a bookie that ran-out-of disk would never be started up again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1089) Ledger Recovery - allow batch reads in ledger recovery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046685#comment-16046685
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1089:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/181
  
rebased to latest master, waiting for CI to complete


> Ledger Recovery - allow batch reads in ledger recovery
> --
>
> Key: BOOKKEEPER-1089
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1089
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> bookkeeper recovery improvement (part-4): allow batch reading in ledger 
> recovery
> - enable batch read in ledger recovery, so we could parallel reading to 
> improve recovery time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1096) When ledger is deleted, along with leaf node all the eligible branch nodes also should be deleted in ZooKeeper.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046667#comment-16046667
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1096:


Github user asfgit closed the pull request at:

https://github.com/apache/bookkeeper/pull/186


> When ledger is deleted, along with leaf node all the eligible branch nodes 
> also should be deleted in ZooKeeper.
> ---
>
> Key: BOOKKEEPER-1096
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1096
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
> Fix For: 4.5.0
>
>
> Currently when we delete a ledger, we delete just the leaf node in the ZK but 
> we ignore about the branch nodes. This is ok for FlatLedgerManager, but for 
> HierarchicalLedgerManagers, especially for LongHierarchicalLedgerManager, the 
> number of internal nodes gets blown up over time and we would get into ZK 
> capacity limitations. When ZK reaches the capacity limits, it will manifest 
> in very severe performance and stability issues of cluster. So for 
> HierarchicalLedgerManagers, when we delete a ledger we should optimistically 
> recursively delete the parent znodes as well if they don’t have anymore child 
> znodes. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1089) Ledger Recovery - allow batch reads in ledger recovery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045907#comment-16045907
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1089:


Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/181
  
@sijie this patch has been reviewed and all previous commits have been 
merged, can you rebase ? 


> Ledger Recovery - allow batch reads in ledger recovery
> --
>
> Key: BOOKKEEPER-1089
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1089
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> bookkeeper recovery improvement (part-4): allow batch reading in ledger 
> recovery
> - enable batch read in ledger recovery, so we could parallel reading to 
> improve recovery time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1017) Create documentation for ZooKeeper ACLs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045830#comment-16045830
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1017:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/185
  
@eolivelli if it is WIP, it is good to add change the PR title to be "WIP - 
"


> Create documentation for ZooKeeper ACLs
> ---
>
> Key: BOOKKEEPER-1017
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1017
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client, bookkeeper-server, Documentation
>Affects Versions: 4.5.0
>Reporter: Enrico Olivelli
>Assignee: Enrico Olivelli
>Priority: Blocker
> Fix For: 4.5.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1093) Piggyback LAC on ReadResponse

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045829#comment-16045829
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1093:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/180
  
@merlimat ??


> Piggyback LAC on ReadResponse
> -
>
> Key: BOOKKEEPER-1093
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1093
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client, bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> Currently LAC is only updated when the reader explicitly calls 
> #readLastAddConfirmed(). In tailing-read use cases, it will not wise to keep 
> calling #readLastAddConfirmed, especially when the traffic is huge.
> The idea here is piggy-back LAC along with the read responses. so the client 
> will get advanced LAC along with read responses. so it will reduce calling 
> #readLastAddConfirmed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1088) Ledger Recovery - Add a ReadEntryListener to callback on individual request

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045500#comment-16045500
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1088:


Github user asfgit closed the pull request at:

https://github.com/apache/bookkeeper/pull/187


> Ledger Recovery - Add a ReadEntryListener to callback on individual request
> ---
>
> Key: BOOKKEEPER-1088
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1088
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> bookkeeper recovery improvement (part-3): add a ReadEntryListener to callback 
> on individual request.
> - add read entry listener which allow doing batch read, but callback on 
> individual entries in sequence. so in recovery op, we could issue batch 
> reads, then on each individual callback do add entry and stop when received 
> NoSuchEntry.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1088) Ledger Recovery - Add a ReadEntryListener to callback on individual request

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045430#comment-16045430
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1088:


GitHub user sijie opened a pull request:

https://github.com/apache/bookkeeper/pull/187

CompactionTest tests are broken because of BOOKKEEPER-1088

Problem:

5fe86525a9c823f79b3e97fd82ea4aa1c75c79eb this commit broken the master. It 
is because the merge/rebase introduce duplicated lines

Solution:

This change is to remove the duplicated lines introduced at 
5fe86525a9c823f79b3e97fd82ea4aa1c75c79eb

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sijie/bookkeeper sijie/fix_merge_issue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bookkeeper/pull/187.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #187


commit 45d2c539abd64f7d5ef98b94acc61e651c41a45b
Author: Sijie Guo 
Date:   2017-06-10T03:52:39Z

Remove duplicated lines introduced when merging pull requests




> Ledger Recovery - Add a ReadEntryListener to callback on individual request
> ---
>
> Key: BOOKKEEPER-1088
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1088
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> bookkeeper recovery improvement (part-3): add a ReadEntryListener to callback 
> on individual request.
> - add read entry listener which allow doing batch read, but callback on 
> individual entries in sequence. so in recovery op, we could issue batch 
> reads, then on each individual callback do add entry and stop when received 
> NoSuchEntry.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1096) When ledger is deleted, along with leaf node all the eligible branch nodes also should be deleted in ZooKeeper.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045197#comment-16045197
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1096:


Github user reddycharan commented on the issue:

https://github.com/apache/bookkeeper/pull/186
  
@revans2 and @knusbaum FYI, LongHierarchicalLedgerManager needs this fix 
for https://issues.apache.org/jira/browse/BOOKKEEPER-1096


> When ledger is deleted, along with leaf node all the eligible branch nodes 
> also should be deleted in ZooKeeper.
> ---
>
> Key: BOOKKEEPER-1096
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1096
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> Currently when we delete a ledger, we delete just the leaf node in the ZK but 
> we ignore about the branch nodes. This is ok for FlatLedgerManager, but for 
> HierarchicalLedgerManagers, especially for LongHierarchicalLedgerManager, the 
> number of internal nodes gets blown up over time and we would get into ZK 
> capacity limitations. When ZK reaches the capacity limits, it will manifest 
> in very severe performance and stability issues of cluster. So for 
> HierarchicalLedgerManagers, when we delete a ledger we should optimistically 
> recursively delete the parent znodes as well if they don’t have anymore child 
> znodes. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1096) When ledger is deleted, along with leaf node all the eligible branch nodes also should be deleted in ZooKeeper.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045193#comment-16045193
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1096:


GitHub user reddycharan opened a pull request:

https://github.com/apache/bookkeeper/pull/186

BOOKKEEPER-1096: recursive znode delete

When ledger is deleted, along with leaf node
all the eligible branch nodes should be
deleted in ZooKeeper.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reddycharan/bookkeeper recursiveznodedelete

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bookkeeper/pull/186.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #186


commit d1feabb7e2c95bbd4beb8d8a68956deeb28894aa
Author: Charan Reddy Guttapalem 
Date:   2017-06-08T02:54:35Z

BOOKKEEPER-1096: recursive znode delete

When ledger is deleted, along with leaf node
all the eligible branch nodes should be
deleted in ZooKeeper.




> When ledger is deleted, along with leaf node all the eligible branch nodes 
> also should be deleted in ZooKeeper.
> ---
>
> Key: BOOKKEEPER-1096
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1096
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>
> Currently when we delete a ledger, we delete just the leaf node in the ZK but 
> we ignore about the branch nodes. This is ok for FlatLedgerManager, but for 
> HierarchicalLedgerManagers, especially for LongHierarchicalLedgerManager, the 
> number of internal nodes gets blown up over time and we would get into ZK 
> capacity limitations. When ZK reaches the capacity limits, it will manifest 
> in very severe performance and stability issues of cluster. So for 
> HierarchicalLedgerManagers, when we delete a ledger we should optimistically 
> recursively delete the parent znodes as well if they don’t have anymore child 
> znodes. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1093) Piggyback LAC on ReadResponse

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044624#comment-16044624
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1093:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/180
  
you still need to call "readLastAddConfirmed". This change is not "long 
poll", it is "piggy-back lac". It is an optimization on reducing the times of 
calling readLastAddConfirmed. In a tailing case, most of the time you should 
call getLastAddConfirmed, you keep reading until your current read entry is 
moving beyond the value that #getLastAddConfirmed, which means you caught up 
with the writer then you switch to call #readLastAddConfirmed. since lac is 
piggy-back, so the value returned by #readLastConfirmed is advancing, hence you 
don't need to call #readLastAddConfirmed that often.


> Piggyback LAC on ReadResponse
> -
>
> Key: BOOKKEEPER-1093
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1093
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client, bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> Currently LAC is only updated when the reader explicitly calls 
> #readLastAddConfirmed(). In tailing-read use cases, it will not wise to keep 
> calling #readLastAddConfirmed, especially when the traffic is huge.
> The idea here is piggy-back LAC along with the read responses. so the client 
> will get advanced LAC along with read responses. so it will reduce calling 
> #readLastAddConfirmed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1017) Create documentation for ZooKeeper ACLs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044552#comment-16044552
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1017:


GitHub user eolivelli opened a pull request:

https://github.com/apache/bookkeeper/pull/185

BOOKKEEPER-1017 Create documentation for ZooKeeper ACLs

This is the documentation for ZooKeeper security and there is an intro 
about security in general.
It is work-in-progress, I created this PR in order to make it visible and 
gather suggestions while writing
 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eolivelli/bookkeeper 
BOOKKEEPER-1017-zookeeper-docs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bookkeeper/pull/185.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #185


commit 5e241b99035426320c07bf52851470de13661bae
Author: Enrico Olivelli 
Date:   2017-06-09T15:34:05Z

BOOKKEEPER-1017 Create documentation for ZooKeeper ACLs




> Create documentation for ZooKeeper ACLs
> ---
>
> Key: BOOKKEEPER-1017
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1017
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client, bookkeeper-server, Documentation
>Affects Versions: 4.5.0
>Reporter: Enrico Olivelli
>Assignee: Enrico Olivelli
>Priority: Blocker
> Fix For: 4.5.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-753) Bookie should run garbage collection before startup when all directories became full

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044534#comment-16044534
 ] 

ASF GitHub Bot commented on BOOKKEEPER-753:
---

Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/81
  
@sijie @jvrao @reddycharan 
do you think we can proceed with this patch ? or is there any conflict with 
new patches @reddycharan  is going to push ?




> Bookie should run garbage collection before startup when all directories 
> became full
> 
>
> Key: BOOKKEEPER-753
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-753
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> otherwise, a bookie that ran-out-of disk would never be started up again.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1028) inc/excl opts listunderreplicate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044357#comment-16044357
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1028:


Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/127
  
I am sorry @reddycharan but findbugs is still failing

[INFO] Boxing/unboxing to parse a primitive 
org.apache.bookkeeper.bookie.BookieShell$LostBookieRecoveryDelayCmd.runCmd(CommandLine)
 [org.apache.bookkeeper.bookie.BookieShell$LostBookieRecoveryDelayCmd] At 
BookieShell.java:[line 1403] DM_BOXED_PRIMITIVE_FOR_PARSING



> inc/excl opts listunderreplicate
> 
>
> Key: BOOKKEEPER-1028
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1028
> Project: Bookkeeper
>  Issue Type: New Feature
>Reporter: Charan Reddy Guttapalem
>Assignee: Charan Reddy Guttapalem
>Priority: Minor
>
> - Introduce including and excluding BookieId options
> for listunderreplicatedLedgers



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1093) Piggyback LAC on ReadResponse

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044344#comment-16044344
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1093:


Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/180
  
@sijie I did a more deep review and it is all great for me.

Do you think it would be useful to add a test for  readUnconfirmedEntries ? 
Actually the implementation is shared with readEntries so maybe we can live 
without such new test

b.q. Just to understand better  
How are you using this feature to improve speed of "tailing reads" ? I see 
that the LAC is advanced without calling readLastAddConfirmed but there is no 
guarantee that it will actually advance so sometimes you will need to call 
readLastAddConfirmed anyway, isn't it ?.
How can you distinguish the case of dead writer/no more data/closed ledger 
? by catching exceptions ?



> Piggyback LAC on ReadResponse
> -
>
> Key: BOOKKEEPER-1093
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1093
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client, bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> Currently LAC is only updated when the reader explicitly calls 
> #readLastAddConfirmed(). In tailing-read use cases, it will not wise to keep 
> calling #readLastAddConfirmed, especially when the traffic is huge.
> The idea here is piggy-back LAC along with the read responses. so the client 
> will get advanced LAC along with read responses. so it will reduce calling 
> #readLastAddConfirmed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044138#comment-16044138
 ] 

ASF GitHub Bot commented on BOOKKEEPER-588:
---

Github user eolivelli closed the pull request at:

https://github.com/apache/bookkeeper/pull/97


> SSL support
> ---
>
> Key: BOOKKEEPER-588
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-588
> Project: Bookkeeper
>  Issue Type: Sub-task
>Reporter: Ivan Kelly
>Assignee: Enrico Olivelli
> Fix For: 4.5.0
>
> Attachments: 0001-MutualTLS-for-Bookkeeper.patch, 
> 0004-BOOKKEEPER-588-SSL-support-for-bookkeeper.patch
>
>
> SSL support using startTLS



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044137#comment-16044137
 ] 

ASF GitHub Bot commented on BOOKKEEPER-588:
---

Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/97
  
closing this PR in favour of #183 from @kishorekasi 


> SSL support
> ---
>
> Key: BOOKKEEPER-588
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-588
> Project: Bookkeeper
>  Issue Type: Sub-task
>Reporter: Ivan Kelly
>Assignee: Enrico Olivelli
> Fix For: 4.5.0
>
> Attachments: 0001-MutualTLS-for-Bookkeeper.patch, 
> 0004-BOOKKEEPER-588-SSL-support-for-bookkeeper.patch
>
>
> SSL support using startTLS



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-588) SSL support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044071#comment-16044071
 ] 

ASF GitHub Bot commented on BOOKKEEPER-588:
---

GitHub user kishorekasi opened a pull request:

https://github.com/apache/bookkeeper/pull/183

BOOKKEEPER-588 SSL Support for Bookkeeper

+ Merged changes from @eoliville
+ Mutual Authentication

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kishorekasi/bookkeeper BOOKKEEPER-588-kishore

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bookkeeper/pull/183.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #183


commit 1105ea825707c2b8301285de7ec0db8f06dcfa09
Author: Kishore Kasi Udayashankar 
Date:   2017-02-13T19:59:22Z

BOOKKEEPER-588 SSL Support for Bookkeeper

+ Merged changes from @eoliville
+ Mutual Authentication




> SSL support
> ---
>
> Key: BOOKKEEPER-588
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-588
> Project: Bookkeeper
>  Issue Type: Sub-task
>Reporter: Ivan Kelly
>Assignee: Enrico Olivelli
> Fix For: 4.5.0
>
> Attachments: 0001-MutualTLS-for-Bookkeeper.patch, 
> 0004-BOOKKEEPER-588-SSL-support-for-bookkeeper.patch
>
>
> SSL support using startTLS



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-748) Move fence requests out of read threads

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037525#comment-16037525
 ] 

ASF GitHub Bot commented on BOOKKEEPER-748:
---

Github user asfgit closed the pull request at:

https://github.com/apache/bookkeeper/pull/174


> Move fence requests out of read threads
> ---
>
> Key: BOOKKEEPER-748
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-748
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> move fence requests out of read threads to address TODO.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1092) Ledger Recovery - Add Test Case for Parallel Ledger Recovery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037517#comment-16037517
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1092:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/182
  
@merlimat can you review this as well?


> Ledger Recovery - Add Test Case for Parallel Ledger Recovery
> 
>
> Key: BOOKKEEPER-1092
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1092
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> Add test case for parallel ledger recovery



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1089) Ledger Recovery - allow batch reads in ledger recovery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037515#comment-16037515
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1089:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/181
  
@merlimat ?


> Ledger Recovery - allow batch reads in ledger recovery
> --
>
> Key: BOOKKEEPER-1089
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1089
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> bookkeeper recovery improvement (part-4): allow batch reading in ledger 
> recovery
> - enable batch read in ledger recovery, so we could parallel reading to 
> improve recovery time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1093) Piggyback LAC on ReadResponse

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037514#comment-16037514
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1093:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/180
  
@merlimat review?


> Piggyback LAC on ReadResponse
> -
>
> Key: BOOKKEEPER-1093
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1093
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client, bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> Currently LAC is only updated when the reader explicitly calls 
> #readLastAddConfirmed(). In tailing-read use cases, it will not wise to keep 
> calling #readLastAddConfirmed, especially when the traffic is huge.
> The idea here is piggy-back LAC along with the read responses. so the client 
> will get advanced LAC along with read responses. so it will reduce calling 
> #readLastAddConfirmed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1088) Ledger Recovery - Add a ReadEntryListener to callback on individual request

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037502#comment-16037502
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1088:


Github user asfgit closed the pull request at:

https://github.com/apache/bookkeeper/pull/178


> Ledger Recovery - Add a ReadEntryListener to callback on individual request
> ---
>
> Key: BOOKKEEPER-1088
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1088
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> bookkeeper recovery improvement (part-3): add a ReadEntryListener to callback 
> on individual request.
> - add read entry listener which allow doing batch read, but callback on 
> individual entries in sequence. so in recovery op, we could issue batch 
> reads, then on each individual callback do add entry and stop when received 
> NoSuchEntry.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1087) Ledger Recovery - Add a parallel reading request in PendingReadOp

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037467#comment-16037467
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1087:


Github user asfgit closed the pull request at:

https://github.com/apache/bookkeeper/pull/177


> Ledger Recovery - Add a parallel reading request in PendingReadOp
> -
>
> Key: BOOKKEEPER-1087
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1087
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> bookkeeper recovery improvement (part-2): add a parallel reading request 
> in PendingReadOp
> - add a parallel reading request in PendingReadOp
> - allow PendingReadOp to configure whether to do parallel reading or not
> - add flag in ClientConfiguration to allow configuring whether to do 
> parallel reading in LedgerRecoveryOp or not.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1087) Ledger Recovery - Add a parallel reading request in PendingReadOp

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037463#comment-16037463
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1087:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/177
  
All CI passed. Merging this now.


> Ledger Recovery - Add a parallel reading request in PendingReadOp
> -
>
> Key: BOOKKEEPER-1087
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1087
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> bookkeeper recovery improvement (part-2): add a parallel reading request 
> in PendingReadOp
> - add a parallel reading request in PendingReadOp
> - allow PendingReadOp to configure whether to do parallel reading or not
> - add flag in ClientConfiguration to allow configuring whether to do 
> parallel reading in LedgerRecoveryOp or not.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-748) Move fence requests out of read threads

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037358#comment-16037358
 ] 

ASF GitHub Bot commented on BOOKKEEPER-748:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/174
  
rebased to latest master and wait CI to complete.


> Move fence requests out of read threads
> ---
>
> Key: BOOKKEEPER-748
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-748
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> move fence requests out of read threads to address TODO.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-944) Multiple issues and improvements to BK Compaction.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037335#comment-16037335
 ] 

ASF GitHub Bot commented on BOOKKEEPER-944:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/108
  
merged this. thank you @reddycharan @dlg99 


> Multiple issues and improvements to BK Compaction.
> --
>
> Key: BOOKKEEPER-944
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-944
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-server
>Affects Versions: 4.4.0
>Reporter: Venkateswararao Jujjuri (JV)
>Assignee: Venkateswararao Jujjuri (JV)
> Fix For: 4.5.0
>
>
> We have identified multiple issues with BK compaction.
> This issue is to list all of them in one Jira ticket.
> 1.
> MajorCompaction and MinorCompaction are very basic. Either they do it or 
> won’t do it. Proposal is to  add Low Water Mark(LWM) and High Water Mark(HWM) 
> to the disk space. Have different compaction frequency and re-claim %s when 
> the disk space is < low water mark  ,  >  LWM < HWM, > HWM.
> 2.
> MajorCompaction and Minor Compactions are strictly frequency based. They 
> should at least time of the day based, and also run during low system load, 
> and if the system load raises, reduce the compaction depending on the disk 
> availability 
> 3.
> Current code disables compaction when disk space grows beyond configured 
> threshold. There is no exit from this point. Have an option to keep reserved 
> space for compaction, at least 2 entryLog file sizes when 
> isForceGCAllowWhenNoSpace enabled.
> 4.
> Current code toggles READONLY status of the bookie as soon as it falls below 
> the disk storage threshold. Imagine if we keep 95% as the threshold, Bookie 
> becomes RW as soon as it falls below 95 % and few more writes pushes it above 
> 95 and it turns back to RONLY. Use a set of defines (another set of LWM/HWM?) 
> where Bookie turns RO on high end and won't become RW until it hits low end.
> 5.
> Current code never checks if the compaction is enabled or disabled once the 
> major/minor compaction is started. If the bookie goes > disk threshold (95%) 
> and at that compaction is going on, it never checks until it finishes but 
> there may not be disk available for compaction to take place. So check if 
> compaction is enabled after processing every EntryLog.
> 6.
> Current code changes the Bookie Cookie value even when new storage is added. 
> When the cookie changes Bookie becomes a new one, and BK cluster treats it as 
> new bookie. If we have mechanism to keep valid cookie even after adding 
> additional disk space, we may have a chance to bring the bookie back to 
> healthy mode and have compaction going.
> 7. Bug
> CheckPoint was never attempted to complete after once sync failure. There is 
> a TODO in the code for this area.
> 8.
> When the disk is above threshold, Bookie goes to RO. If we have to restart 
> the bookie, on the way back, bookie tries to create new entrylog and other 
> files, which will fail because disk usage is above threshold, hence bookie 
> refuses to come up.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-944) Multiple issues and improvements to BK Compaction.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037333#comment-16037333
 ] 

ASF GitHub Bot commented on BOOKKEEPER-944:
---

Github user asfgit closed the pull request at:

https://github.com/apache/bookkeeper/pull/108


> Multiple issues and improvements to BK Compaction.
> --
>
> Key: BOOKKEEPER-944
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-944
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-server
>Affects Versions: 4.4.0
>Reporter: Venkateswararao Jujjuri (JV)
>Assignee: Venkateswararao Jujjuri (JV)
> Fix For: 4.5.0
>
>
> We have identified multiple issues with BK compaction.
> This issue is to list all of them in one Jira ticket.
> 1.
> MajorCompaction and MinorCompaction are very basic. Either they do it or 
> won’t do it. Proposal is to  add Low Water Mark(LWM) and High Water Mark(HWM) 
> to the disk space. Have different compaction frequency and re-claim %s when 
> the disk space is < low water mark  ,  >  LWM < HWM, > HWM.
> 2.
> MajorCompaction and Minor Compactions are strictly frequency based. They 
> should at least time of the day based, and also run during low system load, 
> and if the system load raises, reduce the compaction depending on the disk 
> availability 
> 3.
> Current code disables compaction when disk space grows beyond configured 
> threshold. There is no exit from this point. Have an option to keep reserved 
> space for compaction, at least 2 entryLog file sizes when 
> isForceGCAllowWhenNoSpace enabled.
> 4.
> Current code toggles READONLY status of the bookie as soon as it falls below 
> the disk storage threshold. Imagine if we keep 95% as the threshold, Bookie 
> becomes RW as soon as it falls below 95 % and few more writes pushes it above 
> 95 and it turns back to RONLY. Use a set of defines (another set of LWM/HWM?) 
> where Bookie turns RO on high end and won't become RW until it hits low end.
> 5.
> Current code never checks if the compaction is enabled or disabled once the 
> major/minor compaction is started. If the bookie goes > disk threshold (95%) 
> and at that compaction is going on, it never checks until it finishes but 
> there may not be disk available for compaction to take place. So check if 
> compaction is enabled after processing every EntryLog.
> 6.
> Current code changes the Bookie Cookie value even when new storage is added. 
> When the cookie changes Bookie becomes a new one, and BK cluster treats it as 
> new bookie. If we have mechanism to keep valid cookie even after adding 
> additional disk space, we may have a chance to bring the bookie back to 
> healthy mode and have compaction going.
> 7. Bug
> CheckPoint was never attempted to complete after once sync failure. There is 
> a TODO in the code for this area.
> 8.
> When the disk is above threshold, Bookie goes to RO. If we have to restart 
> the bookie, on the way back, bookie tries to create new entrylog and other 
> files, which will fail because disk usage is above threshold, hence bookie 
> refuses to come up.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-816) use native fallocate & sync_file_range to improve journal allocation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036267#comment-16036267
 ] 

ASF GitHub Bot commented on BOOKKEEPER-816:
---

Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/80
  
I just got into this interesting reading about fallocate
https://groups.google.com/forum/m/#!topic/mechanical-sympathy/UMrKt75yOmg

I am ok with the current patch and the jni approach


> use native fallocate & sync_file_range to improve journal allocation
> 
>
> Key: BOOKKEEPER-816
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-816
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-server
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> it'd better to leverage filesystem fallocate & sync_file_range for journal 
> performance.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)