Re: [VOTE] Graduation resolution proposal

2018-09-10 Thread Ivan Kelly
+1 On Sat, Sep 8, 2018 at 2:21 AM, Matteo Merli wrote: > Please vote on the proposal for Pulsar graduation to TLP to submit to > the Incubator PMC. > > This vote will stay open for at least 72 hours. > > > > > Establish the Apache Pulsar Project > > WHEREA

Re: PIP 20: Mechanism to revoke TLS authentication

2018-09-06 Thread Ivan Kelly
That said, it is stored in Json so it wouldn't be much work to sort one of the values as a list of strings. -Ivan On Thu, Sep 6, 2018 at 10:55 AM, Ivan Kelly wrote: > PIP updated. The one thing that isn't 100% clean with the dynamic > configuration keys is that they only support

Re: PIP 20: Mechanism to revoke TLS authentication

2018-09-06 Thread Ivan Kelly
PIP updated. The one thing that isn't 100% clean with the dynamic configuration keys is that they only support string as value, so the list of revoked keys needs some sort of serialization. -Ivan On Thu, Sep 6, 2018 at 10:41 AM, Ivan Kelly wrote: > On Tue, Aug 21, 2018 at 3:57 A

Re: PIP 20: Mechanism to revoke TLS authentication

2018-09-06 Thread Ivan Kelly
On Tue, Aug 21, 2018 at 3:57 AM, Rajan Dhabalia wrote: >> I will consider both this and system topics and update the PIP. > > I think we should avoid more complexity by introducing system topic to > store "subject key" and then process it. Instead we can store it to > global/config zk as it requir

Re: [DRAFT] Podling report

2018-09-06 Thread Ivan Kelly
ubmitted to the wiki and >> discussed in the mailing list. >> >> PIP 23: Message Tracing By Interceptors >> PIP 22: Pulsar Dead Letter Topic >> PIP 21: Pulsar Edge Component >> PIP 20: Mechanism to revoke TLS authentication >> PIP 19: Pulsar SQL >> >> >> How would you assess the podling's maturity? >> Please feel free to add your own commentary. >> >> [ ] Initial setup >> [ ] Working towards first release >> [ ] Community building >> [X] Nearing graduation >> [ ] Other: >> >> Date of last release: >> 2018-08-02, 2.1.0-incubating >> >> When were the last committers or PPMC members elected? >> >> 2018-06-11 - Ivan Kelly >> 2018-06-11 - Jia Zhai >> >> >> -- >> Matteo Merli >> >>

Re: [DRAFT] Podling report

2018-09-06 Thread Ivan Kelly
that should be +1 On Thu, Sep 6, 2018 at 10:38 AM, Ivan Kelly wrote: > lgtm +! > > On Thu, Sep 6, 2018 at 9:52 AM, Sijie Guo wrote: >> +1 looks good >> >> - Sijie >> >> On Wed, Sep 5, 2018 at 5:36 PM Matteo Merli wrote: >> >>> Sorry

Re: [discuss] how to label milestone

2018-09-05 Thread Ivan Kelly
> The pain comes from issues marked for a bug release milestone but it is > merged directly to master and never cherry-picked to branch for bugfix > releases. > So the release manager has to compare branches and master and also have to > compare labels, milestones and even look into the details of

Re: [DRAFT] [UPDATED] Graduation resolution proposal

2018-09-03 Thread Ivan Kelly
+1 I wish to remain part of PMC. Regarding BookKeeper bylaws, BookKeeper has them because ZooKeeper had them when we branched off, and my understanding is that zookeeper had them, because hadoop had them when they branched off. As far as I'm concerned, they're unnecessary given the default asf by

Re: [discuss] how to label milestone

2018-09-03 Thread Ivan Kelly
> All the bug fixes are deferred to release manager cherry-picking to bug fix > branches. Since master and branches are quickly diverged with changes, > it is very painful for cherry-picks later and there is no clear process for > cherry-pick and labelling. What is painful here? If it's the confli

Re: Accidently pushed a change to branch-1.22

2018-08-28 Thread Ivan Kelly
include it. > > > > > On Tue, Aug 28, 2018 at 5:28 AM Ivan Kelly wrote: > >> Hi folks, >> >> I accidently pushed a change to branch-1.22 to fix #2401. >> >> >> https://github.com/apache/incubator-pulsar/commit/92b809784f631bc6b72cc1116761e8fac02ef41c

Accidently pushed a change to branch-1.22

2018-08-28 Thread Ivan Kelly
Hi folks, I accidently pushed a change to branch-1.22 to fix #2401. https://github.com/apache/incubator-pulsar/commit/92b809784f631bc6b72cc1116761e8fac02ef41c Should I revert immediately? It's a pretty straightforward change given what the bug was. -Ivan

Re: PIP 20: Mechanism to revoke TLS authentication

2018-08-20 Thread Ivan Kelly
> Here, we would like to configure "subject-key-identifier" at every broker in the cluster dynamically. We also want to perform certain actions once this configuration-value has been changed. Ah, i didn't know this existed. Very useful. I will consider both this and system topics and update the PI

Re: PIP 20: Mechanism to revoke TLS authentication

2018-08-20 Thread Ivan Kelly
> Instead of using zookeeper, can we consider using a managed ledger or a system topic for keeping all these revoked keys? Do we already have an example of a system topic in use? Is there a custom namespace that's restricted to the admin role? -Ivan

PIP 20: Mechanism to revoke TLS authentication

2018-08-07 Thread Ivan Kelly
Hi folks, This is a PIP to add a mechanism to block TLS client certs from accessing Pulsar if they have been compromised. This is a relatively small change, but I thought it best to put it to the community before moving ahead with it, as people may have opinions on the approach. The PIP is here:

Re: [VOTE] Pulsar Release 2.1.0-incubating Candidate 5

2018-07-23 Thread Ivan Kelly
The git error can be fixed by adding false It already has failOnUnableToExtractRepoInfo as false, which I assumed would cover this case. -Ivan On Mon, Jul 23, 2018 at 6:05 AM, Sijie Guo wrote: > any more review comments? > > - Sijie > > On Fri, Jul 20, 2018 at 1:13 PM Sijie Guo wrote: > >> Oka

Re: Release 2.1

2018-07-17 Thread Ivan Kelly
The second change is https://github.com/apache/incubator-pulsar/pull/2177 -Ivan On Tue, Jul 17, 2018 at 12:45 AM, Sijie Guo wrote: > Cool. I will wait for these two issues to be merged. > > - Sijie > > On Mon, Jul 16, 2018 at 3:25 PM Ivan Kelly wrote: > >> I&#

Re: Release 2.1

2018-07-16 Thread Ivan Kelly
se candidate number that I > stopped last time. > > If you have any questions regarding 2.1 release, feel free to ping me here. > > - Sijie > > > > On Mon, Jul 9, 2018 at 2:44 AM Ivan Kelly wrote: > >> It would be good to get tls proxy auth fixes into 2.1 s

Re: Release 2.1

2018-07-09 Thread Ivan Kelly
It would be good to get tls proxy auth fixes into 2.1 since there's time to do so now. Without these changes auth + tls do not work if using a proxy. -Ivan On Thu, Jul 5, 2018 at 10:13 PM, Sijie Guo wrote: > Hi all, > > Due the connector packaging issue, we delayed 2.1 release on fixing that. >

Re: [VOTE] Maintaining "Apache Pulsar" name

2018-06-21 Thread Ivan Kelly
+1 On Thu, Jun 21, 2018 at 2:06 AM, Matteo Merli wrote: > Following the previous discussion at > https://lists.apache.org/thread.html/fe60c21b7cdca00918fe25e7ddf9772677342063b9da24c9b84b3329@%3Cdev.pulsar.apache.org%3E > I am calling a formal vote for keep using "Apache Pulsar" name for the > pro

Re: Test library migration proposal

2018-06-18 Thread Ivan Kelly
> We do have access to the docker client via test containers. > > There is no binding between test containers and any test framework all the > underlying methods and classes and fully exposed so they can be mapped to > any test harness. In that case, the first thing to do is try to migrate the smo

Re: Test library migration proposal

2018-06-15 Thread Ivan Kelly
Thanks for starting this conversation Ali. The current tests use arquillian, but there's nothing in the tests that's intimately tied to it, so if we were to remove it there wouldn't be that much that needs to change. On Thu, Jun 14, 2018 at 8:21 PM, Ali Ahmed wrote: > I am proposing we move our c

Re: Weirdness in jenkins build (tests not isolated as expected)

2018-06-04 Thread Ivan Kelly
>> Two concurrent builds on the same machine will have different >> workspaces (@2 or @3 will be appended to $WORKSPACE). > > I am not sure about that. If taking a look at the jenkins job output, the > workspace is " > /home/jenkins/jenkins-slave/workspace/pulsar_precommit_integrationtests" It onl

Re: Weirdness in jenkins build (tests not isolated as expected)

2018-06-04 Thread Ivan Kelly
> 1) the job is using `$WORKSPACE/.repository` as the maven local repo. > 2) concurrent build is enabled > 3) it is a freestyle build, including two maven runs. Two concurrent builds on the same machine will have different workspaces (@2 or @3 will be appended to $WORKSPACE). We should disable con

Weirdness in jenkins build (tests not isolated as expected)

2018-06-04 Thread Ivan Kelly
Hi folks, I've just come across something very strange in a jenkins build that others should be aware of in case they hit across the same thing. https://builds.apache.org/job/pulsar_precommit_integrationtests/1782/ This build is a bulld of the branch 69b5976d8b40789f7f139185ffc1e5cc0bbe4f4b http

Re: [VOTE] Pulsar Release 2.0.0-rc1-incubating Candidate 3

2018-05-11 Thread Ivan Kelly
+1 (non-binding) - Signatures and hashes are good - LICENSEs and NOTICEs are good - Rat and tests are clean (some env issues locally, but fine on a clean VM) - nit: the source distribution is missing the dockerfile and scripts to build the testing docker image, so I couldn't run integration tests

Re: [VOTE] Pulsar Release 2.0.0-rc1-incubating Candidate 2

2018-05-08 Thread Ivan Kelly
> Created PR at https://github.com/apache/incubator-pulsar/pull/1748 I'll take a look, but it may be tomorrow before I get the time. > In some cases, the feedback for the NOTICE file has been to avoid inclusion > on what was not strictly required, in particular, regarding non ASL > licensed proje

Re: [VOTE] Pulsar Release 2.0.0-rc1-incubating Candidate 2

2018-05-08 Thread Ivan Kelly
-1 There's quite a few issues with LICENSE and NOTICE https://gist.githubusercontent.com/ivankelly/a93f06d2a3075575c43a0dd44165bac1/raw/83551daf958432f532ee53cccaa6397f0c1af7bf/pulsar-libs.org Some stuff is missing completely. Others are incomplete. Note that this only applies to the binary dis

Re: Plan for Pulsar 2.0 release

2018-04-19 Thread Ivan Kelly
On Thu, Apr 19, 2018 at 10:07 AM, Dave Fisher wrote: > Is it possible to have 2.0.0alpha on a Maven Central release?> Zookeeper do it all the time. http://search.maven.org/#artifactdetails%7Corg.apache.zookeeper%7Czookeeper%7C3.5.3-beta%7Cpom -Ivan

Re: Plan for Pulsar 2.0 release

2018-04-19 Thread Ivan Kelly
Why not call it 2.0.0Alpha1, or 2.0.0Beta1? Calling it RC will be confusing for people who haven't read this conversation. The only problem with alpha or beta is that you won't get a 2.0.0 release in the end, but a 2.0.1. Many people don't trust '.0' releases anyhow though. -Ivan On Thu, Apr 19,

Re: PIP review process - improvement

2018-04-09 Thread Ivan Kelly
> It will keep the discussion public, convenient, archived, maintain history, > and will be mailed out (with github notifications). > > What do you all say? +1 for this. This is how BK does it, and I've found it much easier to use. -Ivan

[DISCUSS] PIP-16: Tiered storage

2018-04-05 Thread Ivan Kelly
Hi folks, This proposal is to add tiered storage to pulsar. Storing backlogs on bookies for a long time can get expensive. If there are other cheaper forms of storage available (S3/HDFS), capex can be reduced by moving older data to this storage. The proposed design proposes an interface to do t

Re: Pulsar now using standard Apache BookKeeper

2018-03-05 Thread Ivan Kelly
> Keep in mind the next Pulsar release needs to be based on a particular > BookKeeper release. An Apache project cannot release against another > project's SNAPSHOT since that has not been validated by their release process. The bookkeeper 4.7.0 release process should be starting soon, so it sho

Re: Pulsar now using standard Apache BookKeeper

2018-03-02 Thread Ivan Kelly
t through. There were 246 >> > commits to merge into a codebase that slightly changed in a 4 years >> > timespan. For the curious, this is the spreadsheet we used to track the >> > merging. >> > https://docs.google.com/spreadsheets/d/1jAy3EfjViqNEKpCKpWiRv- >>

Re: Fine grained control over batching in tests

2018-03-01 Thread Ivan Kelly
Mon, Feb 26, 2018 at 9:53 PM, Ivan Kelly wrote: > Hi folks, > > I'd like to do some tests with batching and compaction. So far the > only way I've found to control batching is by setting > BatchingMaxMessages and MaxPublishingDelay, but this doesn't seem 100% > det

Fine grained control over batching in tests

2018-02-26 Thread Ivan Kelly
Hi folks, I'd like to do some tests with batching and compaction. So far the only way I've found to control batching is by setting BatchingMaxMessages and MaxPublishingDelay, but this doesn't seem 100% deterministic to me. Does anyone have a better way to control this, or would it make more sense

Integration testing for Pulsar

2018-02-23 Thread Ivan Kelly
Hi folks, I've just pushed the first[1] in a series of pull requests to add integration testing to pulsar. The tests will run with a framework called Arquillian Cube[2], which can be thought of as a way to run docker-compose within testng. A lot of the setup and code I'll be pushing comes from B

Re: [DISCUSS] PIP 14: Pulsar Schema Registry

2018-02-14 Thread Ivan Kelly
Couldn't the schema for a topic be stored in another topic with infinite retention? So if we have a topic /my-prop/my-namespace/topic-foo, we could have a topic /my-prop/my-namespace/topic-foo/schema which contains the different versions of the schema for the original? To get the schema, you just

Re: [DISCUSS] PIP-10: Remove cluster for namespace and topic names

2018-01-11 Thread Ivan Kelly
On Wed, Jan 10, 2018 at 7:31 PM, Joe F wrote: > While we are at it, can we also think of "persistent://"? PIP-11 covers > this only for limited use cases. > > Maybe shorten it to p://? That will be 9MB for a million topics, in ZK > strings (more with utf8). You could even get rid of the //. http

Re: [DISCUSS] Topic Compaction for Pulsar

2017-12-12 Thread Ivan Kelly
_ >> > 差出人: Dave Fisher >> > 送信日時: 2017年12月12日 8:49:13 >> > 宛先: dev@pulsar.incubator.apache.org >> > 件名: Re: [DISCUSS] Topic Compaction for Pulsar >> > >> > Any thoughts Pulsar devs? >> > >> > Sent f

Re: [DISCUSS] Topic Compaction for Pulsar

2017-12-11 Thread Ivan Kelly
Any comments/feedback on this? -Ivan On Mon, Dec 4, 2017 at 11:54 PM, Ivan Kelly wrote: > Hi all, > > I've created PIP to add topic compaction to Pulsar. The PIP only > discusses the compaction mechanism and serving from a compacted topic. > I've left compaction sche

Re: [VOTE] Pulsar 1.21.0-incubating Release Candidate 2

2017-12-05 Thread Ivan Kelly
e Fisher wrote: > >> Hi - >> >> If you build from source do you have this same issue? >> >> Where was your discussion with Matteo? If not on the mailing list then it >> ought to be for full exposure and informed decision making. >> >> Regards, >> D

Re: [VOTE] Pulsar 1.21.0-incubating Release Candidate 2

2017-12-05 Thread Ivan Kelly
-1 on this release. There's a bunch of netty jars being shipped with the binary distribution which makes publication fail with an error such as: 2017-12-05 15:31:07,922 - WARN - [pulsar-client-io-2-1:ClientCnx@156] - [id: 0xd1059600, L:/10.142.0.5:34890 ! R:pulsar3-us-east.c.ivan-184610.internal/

[DISCUSS] Topic Compaction for Pulsar

2017-12-04 Thread Ivan Kelly
Hi all, I've created PIP to add topic compaction to Pulsar. The PIP only discusses the compaction mechanism and serving from a compacted topic. I've left compaction scheduling to be solved later. The proposed solution should be kicked off manually. The master issue is: https://github.com/apache/i

Re: [VOTE] Pulsar 1.20.0-incubating Release Candidate 0

2017-10-05 Thread Ivan Kelly
+1 lgtm Tests done (debian 9): - sha1, md5, gpg - apache rat - unit tests - NOTICE & LICENSE - setup cross colo install, tested a few global topics, tested some local topics, all work well. Good work! -Ivan On Sun, Oct 1, 2017 at 10:55 PM, Joe F wrote: > This is the first release candidate for