Re: [VOTE] Apache Kudu 1.8.0-RC2

[Dan Burkert]

+1

Built and ran C++ tests on MacOS.  Had three failures (fs_manager-test,
kudu-tool-test, and trace-test), but I believe they are all MacOS-specific,
and shouldn't be release blockers.

- Dan

On Fri, Oct 19, 2018 at 11:48 PM Andrew Wong  wrote:

> +1
>
> Built and ran all C++ tests on Centos 6.6 in debug and release modes. A
> couple failed but, upon retrying, passed.
> Ran all Java tests on Centos 6.6 via maven and gradle in release mode. All
> passed.
>
> Built and ran all C++ tests on Centos 7.3 in release mode. All passed.
> Ran all Java tests on Centos 7.3 via gradle in release mode. A number of
> tests failed at first:
>
>- TestKuduBackup.testSimpleBackupAndRestore and
>TestKuduBackup.testRandomBackupAndRestore (both passed on retry,
> flakiness
>is tracked as KUDU-2548 <
> https://issues.apache.org/jira/browse/KUDU-2584>
>)
>- ITClientStressTest.testManyShortClientsGeneratingScanTokens (too many
>open files led to ClassNotFoundException on my shared machine that is
>running Kudu and Impala)
>- TestKuduClient.testGetAuthnToken (passed on retry, some flakiness of
>TestKuduClient tracked in KUDU-2236
>)
>- TestMasterFailover.testKillLeaderBeforeOpenTable (passed on retry,
>flakiness tracked as KUDU-2592
>)
>- DefaultSourceTest.testTableScanWithProjectionAndPredicateLong (passed
>on retry, flakiness of DefaultSourceTest is tracked in KUDU-2029
> and KUDU-2599
>)
>- DefaultSourceTest.testBasicSparkSQLWithInListPredicate (passed on
>retry, ditto)
>
> The takeaway for me is that while we've been OK about tracking flakiness in
> our C++ tests (there exists a flaky test tracking server for them), we
> should do a better job at this for Java tests. Within the last release, the
> pre-commit gerrit job was updated to retry failed Java tests up to 3x and,
> for the sake of pre-commits passing, has costed the introduction of a
> number of new flakes. This doesn't seem release-blocking, but we should be
> mindful to address this moving forward.
>
> On Tue, Oct 16, 2018 at 2:47 AM Attila Bukor  wrote:
>
> > Hi Devs,
> >
> > As suggested on the RC1 vote thread, I've included the complete fix for
> > KUDU-2463 and created a new release candidate for Apache Kudu 1.8.0.
> >
> > Apache Kudu 1.8.0 is a minor release that offers many improvements and
> > fixes
> > since the prior release.
> >
> > The is a source-only release. The artifacts have been staged here:
> > https://dist.apache.org/repos/dist/dev/kudu/1.8.0-RC2/
> >
> > Java convenience binaries in the form of a Maven repository are staged
> > here:
> > https://repository.apache.org/content/repositories/orgapachekudu-1027
> >
> > It is tagged in Git as 1.8.0-RC2 and the corresponding hash is the
> > following:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=cbbf7b580c4ab4fdf6621e4ee5ab1ddc5f03cb4e
> >
> > The release notes can be found here:
> > https://github.com/apache/kudu/blob/1.8.0-RC2/docs/release_notes.adoc
> >
> > The KEYS file to verify the artifact signatures can be found here:
> > https://dist.apache.org/repos/dist/release/kudu/KEYS
> >
> > I'd suggest going through the README and the release notes, building
> > Kudu, and running the unit tests. Testing out the Maven repo would also
> > be appreciated.
> >
> > The vote will run at least until Friday, October 19th at 11:59 AM CEST.
> >
> > Thank you,
> > Attila
> >
>

Re: Proposal: Gradle as the primary build tool

[Dan Burkert]

Sounds great, thanks for chasing this down!

- Dan

On Wed, Oct 3, 2018 at 7:20 PM Grant Henke 
wrote:

> Hi Kudu dev community,
>
> I wanted to provide a progress and plan update to switching over to Gradle
> as the primary build tool.
>
> First, the following steps have already been completed:
>
>- Fixed outstanding Gradle build issues
>- Added dist-test support to Gradle
>- Started using Gradle as the default build in Jenkins pre-commit
>- Changed the Java README to reference Gradle instead of Maven
>- Changed the releasing documentation to reference Gradle instead of
>Maven
>
> Next, the plan is to release Kudu 1.8 using Gradle and remove the maven
> build shortly after branching.
>
> There are two motivations for removing the Maven build now. The primary
> reason is to reduce the workload of maintaining two builds. The secondary
> reason is to simplify some of the test dependencies and break the test
> utilities out into their own module. This is much more straightforward in
> Gradle than it is in Maven.
>
> If there are any concerns or issues with this plan please let me know.
>
> Thank you,
> Grant
>
> On Fri, Apr 27, 2018 at 5:07 PM Adar Lieber-Dembo 
> wrote:
>
> > I am also in favor of this switch. I have nothing to add that you're
> > not already aware of (i.e. need to implement some sort of failed test
> > retry to make the pre-commit change).
> >
> > On Fri, Apr 27, 2018 at 10:05 AM, Grant Henke 
> wrote:
> > > A recent patch has evaluated and addressed any remaining Artifact
> issues.
> > > See the commit message here for details:
> > >
> >
> https://github.com/apache/kudu/commit/7f04873975d73bb335e09ee2a99fd4b0cd89
> > >
> > > That effort improved both the Maven and the Gradle artifacts.
> > >
> > > I wouldn't say the two are exactly the same. I would say that the
> Gradle
> > > artifacts are better.
> > >
> > >- The manifest information in the jars is better.
> > >   - Including an accurate NOTICE.txt and LICENSE.txt
> > >- The pom's are more simplified.
> > >   - They only include what a user needs to know, not all the
> complex
> > >   build details.
> > >
> > > I will walkthrough the full release process as a part of #3 in my
> > proposal
> > > well before the 1.8 release to verify there are no other issues.
> > >
> > > Thank you,
> > > Grant
> > >
> > > On Fri, Apr 27, 2018 at 11:56 AM, Todd Lipcon 
> wrote:
> > >
> > >> I'm on board with this.
> > >>
> > >> Have we already tested the artifact publishing process with gradle?
> Does
> > >> the resulting pom look identical to the pom that we publish with
> maven?
> > I
> > >> assume so, but want to make sure we don't miss this part of the
> process
> > >> which we don't do except for at release time.
> > >>
> > >> -Todd
> > >>
> > >> On Fri, Apr 27, 2018 at 9:31 AM, Grant Henke 
> > wrote:
> > >>
> > >> > Hi Kudu dev community,
> > >> >
> > >> > For some time now the Java build has had an experimental Gradle
> build
> > (
> > >> > KUDU-2066 ) along
> > side
> > >> > the
> > >> > primary Maven build. Since then the Gradle build has been improved
> to
> > >> > support all of the necessary functionality of the Maven build and
> > more.
> > >> >
> > >> > There are many benefits to using Gradle as listed here:
> > >> > https://gradle.org/maven-vs-gradle/
> > >> >
> > >> > Some of the benefits specific to Kudu have been:
> > >> >
> > >> >- Shading Control
> > >> >   - We have had many issues in Maven because modules can not
> > depend
> > >> on
> > >> >   the shaded artifacts of other modules. This resulted in the
> > >> > classpath at
> > >> >   compile and test time being different from the classpath in
> our
> > >> > deployed
> > >> >   artifacts. (KUDU-2241
> > >> >   , KUDU-1780
> > >> >   )
> > >> >- Reliable Incremental Builds
> > >> >   - Maven can pull module dependencies from the local repository
> > >> during
> > >> >   incremental builds. Because of some of the shading issues
> above,
> > >> > this could
> > >> >   result in different resulting artifacts depending on
> > how/when/where
> > >> > the
> > >> >   build was run. (KUDU-2043
> > >> >   )
> > >> >   - Mini-Cluster discovery: Because Maven can use locally
> > installed
> > >> >   module jars in incremental builds, modules other than
> > kudu-client
> > >> > can not
> > >> >   discover the kudu binaries. This happens because the search is
> > >> > started from
> > >> >   the ~/.m2 directory where the installed kudu-client jar is
> > instead
> > >> > of the
> > >> >   project directory. See here
> > >> >    > >> > client/src/test/java/org/apache/kudu/client/TestUtils.java#L93-L98>
> > >> >   for 

Re: Kudu 1.8.0 Release

[Dan Burkert]

Sounds good to me.

- Dan

On Fri, Sep 21, 2018 at 11:14 AM Andrew Wong  wrote:

> I've been discussing with members of the dev community and I think it makes
> sense to push out branching, to let some in-progress work finish and bake,
> and to let some devs come back from time off. So I'm now proposing we
> branch Monday, *Oct. 8*, and we'll vote a couple days after.
>
> Also, it seems other devs had plans specific to this release, so I'm happy
> to hand RM duties if others want to take over. If not, I'll continue
> driving it.
>
> Please let me know if you agree/disagree with the new plan.
>
>
> Andrew
>
> On Fri, Sep 21, 2018 at 7:31 AM Andrew Wong  wrote:
>
> > Hello Kudu developers!
> >
> > It's been just over three months since we released 1.7.1, and it's been
> > almost six months since we released 1.7.0. In that time, we've accrued
> many
> > important improvements and bug fixes (a whopping ~521 since Kudu 1.7.0!).
> > Now seems like as good a time as any to start thinking about a Kudu 1.8.0
> > release, and I'm volunteering to RM it.
> >
> > I'd like to propose we cut the branch for 1.8.x on Tuesday, *Oct. 2*,
> > which gives us a bit over a week to prepare, and then we'll start a vote
> a
> > couple of days after that.
> >
> > *What does this mean for you, *you ask? If this sound good to you all,
> > please start thinking about and writing up release notes for notable
> > changes that have landed in this release, and get them checked in before
> > Oct. 2.
> >
> > If you're struggling to remember that far back, here's a command you can
> > run to check out what you've been up to since Kudu 1.7.0's release:
> >
> > $ git log 472ae66565d9a8efdc81f16568249d7c4b108251..master --oneline
> > --graph --no-merges --first-parent --author=
> >
> > Please let me know if you agree with this plan, have comments, questions,
> > concerns, etc.
> >
> >
> > Thanks!
> > Andrew
> >
> >
> >
> >
>

KUDU-428 Sentry Integration Design Doc

[Dan Burkert]

Hi Kudu developers,

I've drafted a design doc for KUDU-428 Sentry Integration [1].  This is the
next major step under the security umbrella.  The high level goal is table
and column level access control and authorization in Kudu by integrating
with Apache Sentry.  Please take a look at the doc -- comments and
suggestions welcome!

[1]:
https://docs.google.com/document/d/1SEBtgWwBFqij5CuCZwhOqDNSDVViC0WERq6RzsPCWjQ/edit?usp=sharing

[ANNOUNCE] Recognizing the newest Apache Kudu committers

[Dan Burkert]

Hi all,

I'm pleased to announce that the Kudu PMC has voted to add Attila Bukor and
Sailesh Mukil as committers and PMC members.

Attila has contributed many supportability, build, docs, and quality of
life improvements.  In addition, Attila has been very active helping users
on our Slack and email lists.

Sailesh has put a lot of work into Kudu's RPC layer, especially focusing on
security and encryption features. Besides KRPC, Sailesh has put a lot of
work into improving the integration between Apache Impala and Kudu.

I know I speak for the rest of the PMC when I congratulate our newest
members and encourage them to keep up the good work!

- Dan

Re: [VOTE] Apache Kudu 1.7.1 RC2

[Dan Burkert]

+1

Built on macOS 10.12, all tests passed.

- Dan

On Tue, Jun 5, 2018 at 5:21 PM, William Berkeley 
wrote:

> +1
>
> Built on el7.
> Ran tests. Passed.
> Put on a 4-node cluster with some existing data written by 1.5. Was able to
> scan the data back (same results as before).
>
> -Will
>
> On Wed, May 30, 2018 at 8:24 PM, Grant Henke  wrote:
>
> > Thank you for all the validation Attila.
> >
> > Good reminder on the release notes. I will update those. That doesn't
> > require a new release candidate.
> >
> > Thank you,
> > Grant
> >
> > On Wed, May 30, 2018 at 4:44 PM, Attila Bukor 
> wrote:
> >
> > > +1 (non-binding)
> > >
> > > * Artifact's sha1 checksum and gpg signature is valid.
> > > * C++ release build on el7 succeeded, all tests passed.
> > > * Java build also succeeded on el7, 2 tests failed in kudu-client, both
> > > due to environmental issues (stress tests maxed out fd count, don't
> have
> > > permissinos to change it on this machine). Removing these two tests
> > allowed
> > > all other tests to pass on the other Java projects.
> > > * C++ release build fails on MacOS 10.13.2 with Apple LLVM version
> 9.1.0
> > > (clang-902.0.39.1) build fails, but as MacOS support is experimental, I
> > > believe this is not an issue and the build fails on 1.7.0 and 1.7.1-RC1
> > so
> > > this is not a new regression. After backporting 61d3fff, the build
> > succeeds
> > > and all tests pass. I cherry-picked this commit on branch-1.7.x in case
> > > there will be a 1.7.2 or a 1.7.1-RC3.
> > > * Java build on MacOS succeeds, all tests passed.
> > > * Tested Maven repo, works fine.
> > > * Skimmed through README, looks fine to me, but I'm not sure what to
> > check
> > > here.
> > > * Release notes hasn't been updated since RC1, I believe at least
> > > KUDU-2443 and ColumnSchema.toString NPE fix should be mentioned, but I
> > also
> > > think this doesn't warrant a new release candidate. I'm not sure about
> > the
> > > policy on the release notes though.
> > >
> > > Thanks for this RC, Grant!
> > >
> > > Attila
> > >
> > >
> > > > On 2018. May 30., at 15:44, Grant Henke  wrote:
> > > >
> > > > Hi,
> > > >
> > > > The Apache Kudu team is happy to announce the first release candidate
> > for
> > > > Apache Kudu 1.7.1.
> > > >
> > > > Apache Kudu 1.7.1 is a bug-fix release which fixes critical issues in
> > > Kudu
> > > > 1.7.0.
> > > >
> > > > The is a source-only release. The artifacts have been staged here:
> > > > https://dist.apache.org/repos/dist/dev/kudu/1.7.1-RC2/
> > > >
> > > > Java convenience binaries in the form of a Maven repository are
> staged
> > > here:
> > > > https://repository.apache.org/content/repositories/
> orgapachekudu-1021/
> > > >
> > > > It is tagged in Git as 1.7.1-RC1 and the corresponding git hash is
> the
> > > > following:
> > > > *https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > > 5418bfcbbfc6c1809cc869e0119f003e8fb66e37
> > > >  > > 5418bfcbbfc6c1809cc869e0119f003e8fb66e37>*
> > > >
> > > > The release notes can be found here:
> > > > https://github.com/apache/kudu/blob/1.7.1-RC2/docs/
> release_notes.adoc
> > > >
> > > > The KEYS file to verify the artifact signatures can be found here:
> > > > https://dist.apache.org/repos/dist/release/kudu/KEYS
> > > >
> > > > I'd suggest going through the README and the release notes, building
> > > Kudu,
> > > > and running the unit tests. Testing out the Maven repo would also be
> > > > appreciated.
> > > >
> > > > The vote will run until Monday, June 4th at 11PM PDT.
> > > >
> > > > Thank you,
> > > > Grant
> > > >
> > > > --
> > > > Grant Henke
> > > > Software Engineer | Cloudera
> > > > gr...@cloudera.com | twitter.com/gchenke |
> linkedin.com/in/granthenke
> > >
> > >
> > >
> >
> >
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
>

Re: Adding examples

[Dan Burkert]

Sounds great to me.  Having examples in the repository proper means they
can be hooked into the build, and prevented from going stale.  Having that
kind of integration with CI is critical to making sure things don't bit-rot.

- Dan

On Tue, Mar 27, 2018 at 2:45 PM, William Berkeley 
wrote:

> Hi Kudu devs. I'm thinking about adding examples of Kudu client usage to
> the main Kudu repo. We already have examples for C++ found in
> src/kudu/client/samples, so these examples would be for Java and Spark. I'm
> not planning to add Python at first but it should go in there eventually.
>
> The current proposal is to have a new top level directory /examples, with
> the Java and Spark example code completely separate from the Java and Spark
> client code. This way the examples can also serve as examples of how to
> build, as well as how to code.
>
> I'd appreciate any thoughts about this.
>
> Thanks,
> -Will
>

Re: [VOTE] Apache Kudu 1.6.0 RC1

[Dan Burkert]

+1 built and ran tests on macos.

- Dan

On Tue, Dec 5, 2017 at 7:06 PM, Mike Percy  wrote:

> +1
>
> I built 1.6.0-RC1 on RHEL 7 in RELEASE mode.
>
> - Checksums and sigs match
> - README.txt and LICENSE.txt look good
> - RAT check passed
> - All tests passed except for one flaky test (delete_table-itest) and one
> environment-related test issue (on kudu-tool-test), the latter of which I
> filed as KUDU-2234
> - Tested the kudu-client Maven repository jar against one of the old
> kudu-examples example programs
> - Performed some manual failover testing on a 10-node cluster with flash
> disks and everything looked solid
>
> Mike
>
> On Fri, Dec 1, 2017 at 11:10 PM, Mike Percy  wrote:
>
> > Hi,
> >
> > The Apache Kudu team is happy to announce the first release candidate for
> > Apache Kudu 1.6.0.
> >
> > Apache Kudu 1.6.0 is a minor release that offers many improvements and
> > fixes since the prior release.
> >
> > The is a source-only release. The artifacts have been staged here:
> > https://dist.apache.org/repos/dist/dev/kudu/1.6.0-RC1/
> >
> > Java convenience binaries in the form of a Maven repository are staged
> > here:
> > https://repository.apache.org/content/repositories/orgapachekudu-1017
> >
> > It is tagged in Git as 1.6.0-RC1 and the corresponding git hash is the
> > following:
> > https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > 7e28dfc728e54e82d6cbc11fdcd4b9e62f2dfd66
> >
> > The release notes can be found here:
> > https://github.com/apache/kudu/blob/1.6.0-RC1/docs/release_notes.adoc
> >
> > The KEYS file to verify the artifact signatures can be found here:
> > https://dist.apache.org/repos/dist/release/kudu/KEYS
> >
> > I'd suggest going through the README and the release notes, building
> Kudu,
> > and running the unit tests. Testing out the Maven repo would also be
> > appreciated.
> >
> > The vote will run until Tuesday, December 5th at 11PM PDT. We'd normally
> > run a vote for 3 full days but since we're headed into the weekend now
> > let's do 4 days instead.
> >
> > Thanks,
> > Mike
> >
> >
>

Re: INT128 Column Support Interest

[Dan Burkert]

I think it would be useful.  As far as I've seen the main costs in carrying
data types are in writing performant encoders, and updating integrations to
work with them.  I'm guessing with 128 bit integers there would be some
integrations that can't or won't support it, which might be a cause for
confusion.  Overall, though, I think the upsides of efficiency and
decreased storage space are compelling.   Do you have a sense yet of what
encodings are going to be supported down the road (will we get to full
parity with 32/64)?

- Dan

On Thu, Nov 16, 2017 at 2:19 PM, Grant Henke  wrote:

> Hi all,
>
> As a part of adding DECIMAL support to Kudu it was necessary to add
> internal support for 128 bit integers. Taking that one step further and
> supporting public columns and APIs for 128 bit integers would not be too
> much additional work. However, I wanted to gauge the interest from the
> community.
>
> My initial thoughts are that having an INT128 column type could be useful
> for things like UUIDs, IPv6 addresses, MD5 hashes and other similar types
> of data.
>
> Is there any interest or uses for a INT128 column type? Is anyone
> currently using a STRING or BINARY column for 128 bit data?
>
> Thank you,
> Grant
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>

Re: RFC: Canceling RPC in mid-transmission

[Dan Burkert]

I'm still hung up on the premise that if the network is down, cancellation
should still be expected to complete in bounded time.  If the network is
partitioned or congested to the point of non-delivery, how is the
coordinator going to deliver the cancellation messages to the fragments in
the first place?  If we're allowing for arbitrary network delays, there is
no way to make this guarantee.  We're laser focussed on this issue of
cancelling mid-transmission RPCs, but transmission is a very minor slice of
the whole RPC lifecycle, and I don't believe it's worth complicating the
RPC protocol or lifecycle over.

- Dan

On Fri, Oct 13, 2017 at 11:01 AM, Sailesh Mukil 
wrote:

> On Fri, Oct 13, 2017 at 9:37 AM, Michael Ho  wrote:
>
> > There were some recent discussions about the proposed mechanism to cancel
> > an RPC in mid-transmission. I am writing this email to summarize the
> > problem cancellation is trying to solve and some proposed solutions.
> >
> > As part of IMPALA-2567  jira/browse/IMPALA-2567>,
> > Impala is porting its RPC facility to utilize the Kudu RPC library. One
> > major feature of KRPC is asynchronous RPC. This prevents the need for
> > Impala to create a separate thread for each communication channel between
> > fragment instances running on different Impala backends. When an
> > asynchronous RPC is in progress, the KRPC code needs to hold on to the
> RPC
> > payload provided and owned by a fragment instance. We have a singleton
> > messenger instance per Impalad node. The connection between two hosts is
> > shared by different fragment instances.
> >
> > At any point when the RPC is in-progress, the coordinator can cancel all
> > fragment instances of a query for various reasons. In which case, the
> > fragment execution threads need to cancel their in-flight RPC and free up
> > the RPC payloads (e.g. which are passed as sidecars of the RPC request)
> > within a reasonable amount of time (e.g. 500ms). Currently in the KRPC
> > code, when a cancellation is requested when an outbound call is in
> SENDING
> > state (i.e. it has potentially sent some or the whole header to the
> remote
> > server), it will wait for the entire payload to be sent before invoking
> the
> > completion callback specified in the async RPC request. At which point,
> the
> > KRPC client can assume that the RPC payload is no longer referenced by
> the
> > KRPC code and it can be freed.
> >
> > When the network connections between Impalad backends is functioning
> well,
> > it shouldn't be long before the payload is done sending as Impalad
> > internally tries to bound the payload size to 8MB (but not necessarily
> > guaranteed in some degenerated case e.g. row with 1GB string). However,
> > when the network gets congested or is very lossy, there is in theory no
> > bound on how long it takes to finish sending the payload. In other words,
> > the cancellation of the RPC can take unbounded amount of time to complete
> > and thus the cancellation of a fragment instance can also take infinitely
> > long. This is a chicken-and-egg problem. Usually, the user cancels the
> > query because the poor network performance caused it to run for too long.
> > With the unbounded wait, the user cannot even cancel the query within a
> > bounded period of time.
> >
> > Similar problem arises when an RPC times out. As the KRPC code stands
> > today, there is no guarantee that the payload is not referenced by the
> KRPC
> > code when the completion callback is invoked due to timeout. The client
> > doesn't get any notification on when the payload can be freed. In theory,
> > this can be solved by deferring the callback until the payload has been
> > completely sent. However, this may not always be desirable due to the
> > unbounded wait mentioned above. For example, if a client requests a
> timeout
> > of 60s, it's not desirable to invoke the callback say 300s after the
> > timeout.
> >
> > If you agree that the above are indeed problems, we need a RPC
> > cancellation mechanism which meets the following requirements:
> >
> > - the RPC payload is no longer referenced by the KRPC code when the
> > completion callback is invoked.
> > - the time between the cancellation request and the invocation of the
> > completion callback should be reasonably bounded.
> >
> > The followings are two different proposals floated:
> >
> > 1. Upgrade the KRPC protocol to insert a footer at the very end of each
> > outbound call transfer (i.e. after header, serialized protobuf request,
> > sidecars etc). The footer contains a flag which is parsed by the server
> to
> > determine whether the inbound call should be discarded. Normally, this
> flag
> > will be cleared. When an outbound call is cancelled while it's still in
> the
> > SENDING state, the footer is modified in place to have the flag set. In
> > addition, the sidecars of the outbound call transfers are modified to
> point
> > to 

Re: [VOTE] Apache Kudu 1.5.0 RC3

[Dan Burkert]

+1

Built and ran tests on macos, no unexpected failures.

- Dan

On Thu, Sep 7, 2017 at 8:24 PM, Jean-Daniel Cryans <jdcry...@apache.org>
wrote:

> +1
>
> Built on CentOS 7.3.1611 in release mode, ran all unit tests, also ran the
> Java unit tests.
>
> J-D
>
> On Tue, Sep 5, 2017 at 10:40 AM, Dan Burkert <danburk...@apache.org>
> wrote:
>
> > Hi,
> >
> > The Apache Kudu team is happy to announce the third release candidate for
> > Apache Kudu 1.5.0.
> >
> > Apache Kudu 1.5.0 is a minor release which offers many improvements and
> > fixes since the prior release.
> >
> > The is a source-only release. The artifacts are staged here:
> > https://dist.apache.org/repos/dist/dev/kudu/1.5.0-RC3
> >
> > Java convenience binaries in the form of a Maven repository are staged
> > here: https://repository.apache.org/content/repositories/
> > orgapachekudu-1015.
> >
> > It is built from this tag:
> > https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > 9101f85fa4ec465b490720629eb5a0caba2563dd
> >
> > KEYS file: http://www.apache.org/dist/kudu/KEYS
> >
> > I suggest going through the README, building Kudu, and running the unit
> > tests, and testing the Maven artifacts.
> >
> > The vote will run until Friday, September 8th at 12pm PDT.
> >
> > Thanks,
> > Dan
> >
>

Re: [VOTE] Apache Kudu 1.5.0 RC2

[Dan Burkert]

Thanks Hao, and Adar and Mike for reviewing.  Glad we caught this issue
before the release!  I've just created an RC 3 with Hao's patch which will
run until Friday.  This vote is abandoned.

- Dan

On Mon, Sep 4, 2017 at 10:54 PM, Todd Lipcon <t...@cloudera.com> wrote:

> On Mon, Sep 4, 2017 at 10:50 PM, Hao Hao <hao@cloudera.com> wrote:
>
> > Thanks Adar, Mike a lot for the quick review! Apologize again for not
> > caught it in an earlier stage.
> >
> > @Todd, I believe this LIFO change only controls which container the
> copied
> > tablet should reside inside a single disk. Whether the tablets will
> spread
> > across
> > multiple disks is still decided by the flag
> > '--fs_target_data_dirs_per_tablet'
> > introduced in the 'TServer disk failure handling' feature. But maybe you
> > can
> > elaborate a bit more on what kind of regression you are talking about?
> >
>
> Ah, I was initially worried that it would choose to re-use an open
> container on disk A before it would use any container from disk B. I forgot
> that the disk selection behavior happens before the container selection
> behavior, so it will still randomize (or round robin) across the assigned
> data disks. Thanks for clarifying.
>
> -Todd
>
>
> > Thanks!
> >
> > Best,
> > Hao
> >
> > On Mon, Sep 4, 2017 at 10:10 PM, Todd Lipcon <t...@cloudera.com> wrote:
> >
> > > Quick question on the fix: does the lifo behavior now cause the
> entirety
> > of
> > > the copied tablet to end up in one container (hence one disk) if the
> > tablet
> > > is less than a few gb? If so that also seems like a regression to
> > consider.
> > >
> > > (Sorry for hijacking this thread rather than commenting on Gerrit. On
> my
> > > phone)
> > >
> > > Todd
> > >
> > > On Sep 4, 2017 9:21 PM, "Adar Lieber-Dembo" <a...@cloudera.com> wrote:
> > >
> > > > -1
> > > >
> > > > I agree with Hao's assessment that KUDU-2131 should be considered a
> > > > blocker for 1.5. Below is some more color:
> > > >
> > > > Before the fix for KUDU-1726, tablet copies fsynced more than
> > > > necessary but reliably kept their sessions alive by amortizing the
> > > > cost of the fsync in each downloaded block.
> > > >
> > > > With the fix, the number of fsyncs in a tablet copy is no longer
> > > > amortized to each downloaded block. In theory, this is good: by
> > > > batching up all of the fsyncs, we can find opportunities to coalesce
> > > > them, or omit all of them if the copy is aborted. But, if the number
> > > > of fsyncs turns out to be very high (because it's a large tablet and
> > > > the tserver already has a high number of containers), the
> > > > non-amortized cost can reliably cause the tablet copy session to time
> > > > out. I view this as a regression to the overall reliability of tablet
> > > > copies.
> > > >
> > > > Hao authored a fix for KUDU-2131. Mike and I reviewed it, and I just
> > > > merged it to master. I'm cherry picking it to branch-1.5.x and I
> think
> > > > we should spin up an RC3 that includes it.
> > > >
> > > >
> > > > On Fri, Sep 1, 2017 at 11:00 PM, Hao Hao <hao@cloudera.com>
> wrote:
> > > > > Unfortunately, I think I just found a blocker for 1.5 (KUDU-2131
> > > > > <https://issues.apache.org/jira/browse/KUDU-2131>).
> > > > >
> > > > > I am not sure if we should revert KUDU-1726 or do a quick fix which
> > > adds
> > > > a
> > > > > user
> > > > > facing flag to config tablet copy session expire time (such as
> > > > > --tablet_copy_idle_timeout_ms).
> > > > > Apologize for any inconvenience!
> > > > >
> > > > > Best,
> > > > > Hao
> > > > >
> > > > > On Fri, Sep 1, 2017 at 6:44 PM, Mike Percy <mpe...@apache.org>
> > wrote:
> > > > >
> > > > >> +1 on 1.5.0 RC2.
> > > > >>
> > > > >> Thanks for the quick turnaround on RC1, Dan.
> > > > >>
> > > > >> Sigs and checksums match. LICENSE and NOTICE files look good.
> > > > >>
> > > > >> I built the C++ code in RELEASE mode on Ubuntu xenial and all the
> > C++
> > > > tests
> > > &

[VOTE] Apache Kudu 1.5.0 RC3

[Dan Burkert]

Hi,

The Apache Kudu team is happy to announce the third release candidate for
Apache Kudu 1.5.0.

Apache Kudu 1.5.0 is a minor release which offers many improvements and
fixes since the prior release.

The is a source-only release. The artifacts are staged here:
https://dist.apache.org/repos/dist/dev/kudu/1.5.0-RC3

Java convenience binaries in the form of a Maven repository are staged
here: https://repository.apache.org/content/repositories/orgapachekudu-1015.

It is built from this tag:
https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=9101f85fa4ec465b490720629eb5a0caba2563dd

KEYS file: http://www.apache.org/dist/kudu/KEYS

I suggest going through the README, building Kudu, and running the unit
tests, and testing the Maven artifacts.

The vote will run until Friday, September 8th at 12pm PDT.

Thanks,
Dan

[VOTE] Apache Kudu 1.5.0 RC1

[Dan Burkert]

Hi,

The Apache Kudu team is happy to announce the second release candidate for
Apache Kudu 1.5.0.

Apache Kudu 1.5.0 is a minor release which offers many improvements and
fixes since the prior release.

The is a source-only release. The artifacts are staged here:
https://dist.apache.org/repos/dist/dev/kudu/1.5.0-RC2/

Java convenience binaries in the form of a Maven repository are staged
here: https://repository.apache.org/content/repositories/orgapachekudu-1014.

It is built from this tag:
https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=9f2e3253319d7d9733cf764a63e308b91787538a

KEYS file: http://www.apache.org/dist/kudu/KEYS

I suggest going through the README, building Kudu, and running the unit
tests, and testing the Maven artifacts.

The vote will run until Wednesday, September 6th at 5pm PDT.

Thanks,
Dan

Re: [VOTE] Apache Kudu 1.5.0 RC1

[Dan Burkert]

OK I'm going to sink this in favor of RC2.  Thanks for the thorough
investigation, Mike.

- Dan

On Fri, Sep 1, 2017 at 2:40 PM, Mike Percy <mpe...@apache.org> wrote:

> -1 on 1.5.0-RC1 for licensing reasons.
>
> I ran this command to look at the new files added between tags:
> git diff --name-status 1.4.0 1.5.0-RC1 | grep ^A
>
> Then I grepped those changed files for ASF license headers. Of the ones
> that didn't have the usual ASF license headers, I looked for a mention of
> them in LICENSE.txt
>
> 1. The following files are missing from LICENSE.txt and need to be added.
> We can just add an entry for the whole directory, I think:
>
> build-support/iwyu/iwyu_tool.py: LLVM Release License (only
> thirdparty/llvm-* is mentioned as being covered by this license in the
> thirdparty/LICENSE.txt file)
> build-support/iwyu/mappings/boost-all-private.imp: LLVM Release License
> build-support/iwyu/mappings/boost-all.imp: LLVM Release License
>
> We can simply say the files in build-support/iwyu/ are licensed under the
> LLVM Release License, a full copy of which can be found in
> thirdparty/LICENSE.txt
>
> 2. The license header for these files is wrong:
>
> java/kudu-client-tools/src/main/java/org/apache/kudu/
> mapreduce/tools/ExportCsv.java
> java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/
> ExportCsvMapper.java
> java/kudu-client-tools/src/main/java/org/apache/kudu/
> mapreduce/tools/ImportParquet.java
> java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/
> ImportParquetMapper.java
> java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/
> ParquetReadSupport.java
> java/kudu-client/src/test/java/org/apache/kudu/client/
> TestAuthnTokenReacquire.java
> java/kudu-client/src/test/java/org/apache/kudu/client/
> TestAuthnTokenReacquireOpen.java
>
> They are using the ASL2.0 "short header" but because this is an ASF project
> they need to use the standard ASF header [1] that starts with:
> "// Licensed to the Apache Software Foundation (ASF) under one or more
> contributor license agreements."
>
> I am assuming that each of the above files is an original contribution from
> the respective authors to the Kudu project, and not imported from
> elsewhere. We need to check that. If the former is the case, we simply
> change to the proper header. If the latter is the case, the files need to
> be added to the LICENSE.txt file.
>
> Mike
>
> [1] https://www.apache.org/legal/src-headers.html
>
> On Fri, Sep 1, 2017 at 9:53 AM, Dan Burkert <danburk...@apache.org> wrote:
>
> > Hi,
> >
> > The Apache Kudu team is happy to announce the first release candidate for
> > Apache Kudu 1.5.0.
> >
> > Apache Kudu 1.5.0 is a minor release which offers many improvements and
> > fixes since the prior release.
> >
> > The is a source-only release. The artifacts are staged here:
> > https://dist.apache.org/repos/dist/dev/kudu/1.5.0-RC1/
> >
> > It is built from this tag:
> > https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > 40012e318401c920dc83c222d6aeba395ed2a6a4
> >
> > KEYS file: http://www.apache.org/dist/kudu/KEYS
> >
> > I suggest going through the README, building Kudu, and running the unit
> > tests.
> >
> > The vote will run until Wednesday, September 6th at 10AM PDT.
> >
> > Thanks,
> > Dan
> >
>

Re: IWYU configuration for gerrit pre-commit checks

[Dan Burkert]

Thanks, Alexey!

- Dan

On Thu, Aug 24, 2017 at 4:46 PM, Mike Percy  wrote:

> Thanks for setting this up, Alexey!
>
> Mike
>
> On Thu, Aug 24, 2017 at 11:18 AM, Alexey Serbin 
> wrote:
>
> > Hi,
> >
> > Today I enabled the IWYU (include-what-you-use) configuration
> > for the pre-commit Jenkins job at https://gerrit.cloudera.org
> >
> > The newly introduced automated check runs the IWYU tool [1]
> > (include-what-you-use is the name of the binary)
> > to help us keeping the Kudu source code cleaner: [2].
> >
> > As you would expect, Jenkins automatically starts a job for
> > the IWYU configuration along with jobs for other configs
> > (DEBUG, RELEASE, ..., LINT).  The IWYU job runs the tool
> > against the files modified by the changelists in question.
> > If the IWYU jobs fails, you should look at the job's console
> > output and update your code in accordance with IWYU recommendations.
> > I assume your changelists are already synchronized with
> > the trunk to include a couple of IWYU-related updates
> > that the IWYU Jenkins job depends on.
> >
> > As a side note, I want to mention that the include-what-you-use
> > tool is still in alpha quality phase and there might be some quirks;
> > e.g. the tool might suggest something that breaks compilation, etc.
> > I put some effort to minimize such mishaps, but if you hit any of
> > those, please let me know -- I'll help you to resolve those.
> >
> > If you want to run the verification locally
> > before submitting your patch for review,
> > in case of using GNU make you just run
> >
> >   make iwyu
> >
> > (the same as you would do to run the lint: 'make lint')
> >
> > If you have any question or concerns, please let me know.
> >
> >
> > Kind regards,
> >
> > Alexey
> >
> >
> > References:
> >   [1] https://github.com/include-what-you-use/include-what-you-use
> >   [2] https://github.com/include-what-you-use/include-what-you-use
> > /blob/master/docs/WhyIWYU.md
> >
> >
> >
>

Upcoming 1.5 release

[Dan Burkert]

Hey everyone,

I think it's time to get the ball rolling on a 1.5 release.  It's been a
bit over 2 months since 1.4, and we've had ~250 commits to master since
then.  I volunteer to RM, unless someone else wants the honor.

I propose we branch 1.5 this coming Monday, August 28th, with the aim to
start a vote on Wednesday, August 30th or the following day.

If you have authored any major improvements since 1.4, now is the time to
start writing release not entries, or let me know so I can do that.

- Dan

Re: Java 7 and Spark Support

[Dan Burkert]

I'll preface my response by saying I don't think there are any hard and
fast rules here, but I'd like us to try
and continue following SemVer rules as much as possible.

On Tue, Aug 15, 2017 at 2:03 PM, Grant Henke  wrote:
>
>
>- Should/can we drop Spark 1 support in the next minor release?
>

My interpretation is that it's permissible to stop shipping releases of an
artifact at any point (in this case kudu-spark1_2.10),
so I'm all for dropping Spark 1 as soon as we feel there are a sufficiently
low number of users.


>- Should/can we drop Java 7 support in the next minor release? Does it
>need to be a major release?
>

My interpretation of SemVer is that we can't drop JRE 7 support without a
major version bump. That being said,
I do think we're quickly approaching the time in which it would be
appropriate to make this step.


>- How should we support Spark 2.2.0 if we don't drop Java 7? Should we
>only require Java 1.8 for the Spark 2 modules?
>

Spark has put us in a difficult position here - either kudu-spark2_2.11
remains JRE 7 compatible
and is capped at Spark 2.1, or we make an exception for kudu-spark2_2.11,
drop
JRE 7 compatibility, and continue floating the Spark version against the
latest 2.x release.  I think given the
velocity of the Spark project and the fact that Spark itself doesn't seem
to have any qualms about
breaking SemVer, we should do the latter.


> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>

Re: Preconditions vs assert in the Kudu Java client

[Dan Burkert]

By the way - I do think your points about exception safety WRT cleaning up
illegal intermediate states is extremely interesting.  That is, in my
opinion, a principal reason why the exceptions do _not_ result in robust
software, as proponents claim, but instead in software which works well in
most situations, but which fail catastrophically and unpredictably due to
failure to handle black-swan error cases.

- Dan

On Tue, Jul 11, 2017 at 1:24 PM, Dan Burkert <danburk...@apache.org> wrote:

> My general rule of thumb is that I use 'assert' whenever I'm claiming that
> something must be true given static (non-runtime) knowledge local to the
> context (usually a method or class).  If the assert fails, it indicates
> that there is irrefutably a bug in that context.  I use the Preconditions
> helpers for things that should be true, but might be outside the control of
> a local context, or might be runtime-dependent.  In extreme situations I
> might use an explicit if/throw new AssertionException if the consequences
> of a hypothetical bug are so extreme that it doesn't make sense to ever
> disable the assert (e.g. in a fireMissiles method).
>
> In terms of assert/Preconditions vs DCHECK/CHECK: I think it's not exactly
> a 1:1 comparison.  We quite frequently return error Status's for things
> that we use Preconditions for in Java (illegal argument, illegal state). I
> think that's appropriate, given the different error propagation paradigms
> of Java vs. our flavor of C++.
>
> Sorry if this doesn't actually suggest a resolution, I think it's a 'know
> it when you see it' situation - both Preconditions and asserts have
> appropriate uses.
>
> - Dan
>
> On Tue, Jul 11, 2017 at 12:49 PM, Alexey Serbin <aser...@cloudera.com>
> wrote:
>
>> Hi,
>>
>> While working on a small refactoring in the Kudu Java client, I found
>> that sometimes we use asserts
>> and sometimes Preconditions [2] from the guava library to assert on the
>> consistency of the code.
>>
>> As Todd suggested, I'm starting this thread to clarify on what is the
>> best way to do perform
>> the consistency checks in our Kudu Java client code.  Ideally, the
>> desired outcome from the
>> anticipated discussion would be some sort of guidance on assertion of the
>> consistency constrains
>> and invariants in our Java client code.  Once we have the guidance, we
>> could put it as an
>> additional section of our 'Contributing' page.
>>
>> The issue with the Java asserts is that they are not applicable unless
>> the JVM is run with '-ea'
>> flag, so it's not possible to explicitly stop processing further
>> instructions in the context
>> when an inconsistency is detected.
>>
>> To me, we have more clarity with the consistency checks and invariant
>> assertion in our C++ code.
>> We have CHECK- and DCHECK- derived macros to perform various types of
>> consistency checks.  In short,
>> the CHECK-related are always there, and the DCHECK-related ones are
>> present only in debug builds.
>> We don't turn on/off any of those dynamically in runtime.  Also, those
>> C++ asserts are kind of 'hard'
>> ones -- the whole program is terminated and you don't need to think how
>> to deal with the mess which
>> isn't possible to recover from.  The latter one is a luxury which is not
>> available in Java
>> (correct me if I'm wrong).
>>
>> Putting the explicit termination of the whole program (with an optional
>> coredump) aside,
>> the Java's assert looks like DCHECK() but with the ability to turn it
>> on/off using the run-time switch.
>> Using guava's Preconditions gives us an option to address that and have a
>> way
>> to assert the consistency in our code even if code is run without the
>> '-ea' JVM flag.
>>
>> So, to me it looks natural to use the Preconditions-derived checks for
>> asserting on invariants
>> even while running in production environment, while using asserts only in
>> debug/test mode,
>> asserting on programmatic errors and other non-crucial invariants which
>> might be handled
>> by the code even if those checks are removed.
>>
>> However, this simple approach contradicts to some points of [1] (which,
>> by my understanding,
>> is contradictory and very confusing as is).  Also, I'm not a Java
>> programmer,
>> so I might be missing some crucial points here.
>>
>> What do you think?  Your feedback is highly appreciated.
>>
>>
>> Thanks,
>>
>> Alexey
>>
>>
>> References:
>>   [1] http://docs.oracle.com/javase/8/docs/technotes/guides/langua
>> ge/assert.html
>>   [2] https://github.com/google/guava/wiki/PreconditionsExplained
>>
>>
>>
>>
>

Re: Preconditions vs assert in the Kudu Java client

[Dan Burkert]

My general rule of thumb is that I use 'assert' whenever I'm claiming that
something must be true given static (non-runtime) knowledge local to the
context (usually a method or class).  If the assert fails, it indicates
that there is irrefutably a bug in that context.  I use the Preconditions
helpers for things that should be true, but might be outside the control of
a local context, or might be runtime-dependent.  In extreme situations I
might use an explicit if/throw new AssertionException if the consequences
of a hypothetical bug are so extreme that it doesn't make sense to ever
disable the assert (e.g. in a fireMissiles method).

In terms of assert/Preconditions vs DCHECK/CHECK: I think it's not exactly
a 1:1 comparison.  We quite frequently return error Status's for things
that we use Preconditions for in Java (illegal argument, illegal state). I
think that's appropriate, given the different error propagation paradigms
of Java vs. our flavor of C++.

Sorry if this doesn't actually suggest a resolution, I think it's a 'know
it when you see it' situation - both Preconditions and asserts have
appropriate uses.

- Dan

On Tue, Jul 11, 2017 at 12:49 PM, Alexey Serbin 
wrote:

> Hi,
>
> While working on a small refactoring in the Kudu Java client, I found that
> sometimes we use asserts
> and sometimes Preconditions [2] from the guava library to assert on the
> consistency of the code.
>
> As Todd suggested, I'm starting this thread to clarify on what is the best
> way to do perform
> the consistency checks in our Kudu Java client code.  Ideally, the desired
> outcome from the
> anticipated discussion would be some sort of guidance on assertion of the
> consistency constrains
> and invariants in our Java client code.  Once we have the guidance, we
> could put it as an
> additional section of our 'Contributing' page.
>
> The issue with the Java asserts is that they are not applicable unless the
> JVM is run with '-ea'
> flag, so it's not possible to explicitly stop processing further
> instructions in the context
> when an inconsistency is detected.
>
> To me, we have more clarity with the consistency checks and invariant
> assertion in our C++ code.
> We have CHECK- and DCHECK- derived macros to perform various types of
> consistency checks.  In short,
> the CHECK-related are always there, and the DCHECK-related ones are
> present only in debug builds.
> We don't turn on/off any of those dynamically in runtime.  Also, those C++
> asserts are kind of 'hard'
> ones -- the whole program is terminated and you don't need to think how to
> deal with the mess which
> isn't possible to recover from.  The latter one is a luxury which is not
> available in Java
> (correct me if I'm wrong).
>
> Putting the explicit termination of the whole program (with an optional
> coredump) aside,
> the Java's assert looks like DCHECK() but with the ability to turn it
> on/off using the run-time switch.
> Using guava's Preconditions gives us an option to address that and have a
> way
> to assert the consistency in our code even if code is run without the
> '-ea' JVM flag.
>
> So, to me it looks natural to use the Preconditions-derived checks for
> asserting on invariants
> even while running in production environment, while using asserts only in
> debug/test mode,
> asserting on programmatic errors and other non-crucial invariants which
> might be handled
> by the code even if those checks are removed.
>
> However, this simple approach contradicts to some points of [1] (which, by
> my understanding,
> is contradictory and very confusing as is).  Also, I'm not a Java
> programmer,
> so I might be missing some crucial points here.
>
> What do you think?  Your feedback is highly appreciated.
>
>
> Thanks,
>
> Alexey
>
>
> References:
>   [1] http://docs.oracle.com/javase/8/docs/technotes/guides/langua
> ge/assert.html
>   [2] https://github.com/google/guava/wiki/PreconditionsExplained
>
>
>
>

Re: [VOTE] Apache Kudu 1.4.0 RC2

[Dan Burkert]

+1

Built and ran tests on OS X 10.10.

- Dan

On Mon, Jun 12, 2017 at 9:49 AM, Brock Noland  wrote:

> +1 (non-binding)
>
> Built and ran tests
>
> On Mon, Jun 12, 2017 at 9:49 AM, Jean-Daniel Cryans 
> wrote:
> > Err sorry I meant to vote on Friday but then Happy Hour happened :)
> >
> > +1, ran the same verifications as for RC1.
> >
> > FWIW one of my test clusters hit the issue that killed RC1 after I had
> > voted for it, so I'm glad we caught it.
> >
> > J-D
> >
> > On Mon, Jun 12, 2017 at 12:38 AM, Todd Lipcon  wrote:
> >
> >> Hmm, no votes so far? Let's extend the vote to tomorrow night (I guess
> it
> >> wasn't so nice to have most of the voting period fall on a weekend).
> I'll
> >> try to vote myself tomorrow as well.
> >>
> >> -Todd
> >>
> >> On Thu, Jun 8, 2017 at 7:15 PM, Todd Lipcon  wrote:
> >>
> >> > Hi,
> >> >
> >> > The Apache Kudu team is happy to announce the first release candidate
> for
> >> > Apache Kudu 1.4.0.
> >> >
> >> > Apache Kudu 1.4.0 is a minor release which offers many improvements
> and
> >> > fixes since the prior release. This second release candidate includes
> the
> >> > release notes as well as two bug fixes for issues discovered during
> the
> >> RC1
> >> > voting period.
> >> >
> >> > The is a source-only release. The artifacts were staged here:
> >> > https://dist.apache.org/repos/dist/dev/kudu/1.4.0-RC2
> >> >
> >> > Java convenience binaries in the form of a Maven repository are staged
> >> > here:
> >> > https://repository.apache.org/content/repositories/
> orgapachekudu-1010/
> >> > (note that the Java artifacts did not change since the prior RC, so
> this
> >> > is the same staged repository)
> >> >
> >> > It was built from this tag:
> >> > https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> >> > 1ed03d710ebf06afa35ee0f7813649d661f90829
> >> >
> >> > KEYS file:
> >> > http://www.apache.org/dist/kudu/KEYS
> >> >
> >> > I'd suggest going through the README, building Kudu, and running the
> >> > unit tests. Testing out the Maven repo would also be appreciated.
> >> >
> >> > The vote will run until Sunday, June 8th at 7PM PDT.
> >> >
> >> > Thanks,
> >> > Todd
> >> >
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
>

Re: Multi-word Flag Style

[Dan Burkert]

OK it sounds like the consensus is to move to dash-style flags.  I've filed
KUDU-1984 <https://issues.apache.org/jira/browse/KUDU-1984> to track this.
I don't have immediate time to work on it, but if no one picks it up I'll
probably circle back when I have less on my plate.

- Dan

On Mon, Apr 10, 2017 at 1:24 PM, Alexey Serbin <aser...@cloudera.com> wrote:

> Oops, sure -- we cannot use dashes when referring to the variables in our
> C++ code.
>
>
> On 4/10/17 12:40 PM, Dan Burkert wrote:
>
>> On Mon, Apr 10, 2017 at 11:56 AM, Adar Dembo <a...@cloudera.com> wrote:
>>
>>> But, I don't know whether gflags can be coerced to programmatically
>>> emit flags with dashes (i.e. when invoked with --help) without a patch
>>> or two.
>>>
>>
>> Yah, I was thinking the same.  Gflags just landed support for dashes
>> upstream in 2.2, so I imagine they would be open to a patch to configure
>> dashes in help output.
>>
>>
>> Certainly in the code we would want to retain the use of
>>> underscores when referring to flag variables; FLAGS_foo_bar conforms
>>> to our coding style more than something like FLAGS-foo-bar.
>>>
>>> I agree - I don't think dashes are valid identifier characters in C++
>> anyway.
>>
>> - Dan
>>
>>
>>
>>> On Mon, Apr 10, 2017 at 11:00 AM, Alexey Serbin <aser...@cloudera.com>
>>> wrote:
>>>
>>>> I think it's a good move.  It would be nice to add a notice about that
>>>> in
>>>> the user-facing docs.
>>>>
>>>> Also, I think it would be more consistent to convert those flags
>>>>
>>> altogether
>>>
>>>> at some point to be in dash-ish form, both the code and the docs.
>>>> Maybe,
>>>> 1.4 is a good point to do that.
>>>>
>>>>
>>>> Kind regards,
>>>>
>>>> Alexey
>>>>
>>>>
>>>>
>>>> On 4/10/17 10:42 AM, William Berkeley wrote:
>>>>
>>>>> I agree, for the reason you gave: dashes are the norm in Unix, so they
>>>>> "feel right" for flag names.
>>>>>
>>>>> -Will
>>>>>
>>>>> On Mon, Apr 10, 2017 at 1:38 PM, Dan Burkert <danburk...@apache.org>
>>>>> wrote:
>>>>>
>>>>> Hi all,
>>>>>>
>>>>>> As of Kudu 1.3, multi-word flags can use a dash '-' separator in lieu
>>>>>>
>>>>> of
>>>
>>>> the underscore '_' separator.  For example,  --memory_limit_hard_bytes
>>>>>> can
>>>>>> now be specified as --memory-limit-hard-bytes, or even
>>>>>> --memory_limit-hard_bytes.  Of the people I've talked to, most seem to
>>>>>> prefer dashes to underscores in flag names, since that's been the Unix
>>>>>> norm
>>>>>> for a long time.
>>>>>>
>>>>>> Going forward, I'd like to propose that we document flag names using
>>>>>> dashes
>>>>>> wherever possible.  We would continue accepting underscores
>>>>>>
>>>>> indefinitely,
>>>
>>>> since to stop doing so would break compatibility. For the most part,
>>>>>>
>>>>> this
>>>
>>>> means incrementally switching the documentation to use dashes, and
>>>>>> getting
>>>>>> glog to output dashes in --help output.
>>>>>>
>>>>>> Any thoughts?
>>>>>>
>>>>>> - Dan
>>>>>>
>>>>>>
>

Re: [VOTE] Apache Kudu 1.3.1 RC1

[Dan Burkert]

+1

Built and ran tests on OS X 10.10.  delete_tablet-itest fails, but it
appears to fail on master as well, all they way back to when it was
introduced about a month ago.  Not a blocker IMO.

- Dan

On Mon, Apr 17, 2017 at 4:32 PM, Todd Lipcon  wrote:

> +1
>
> Built the release build bits on CentOS 6.8 and CentOS 7.3 (from stock VM
> images, following the docs).
>
> Since no read/write path or consensus changes were made since 1.3.0 I
> didn't bother rerunning any larger cluster tests or benchmarks. Instead I
> focused on more targeted testing of the changes.
>
> Ran release mode tests on both, no unexpected failures (couple timing
> flakes due to overloaded VMs but succeeded on retry).
>
> I also ran through the repro steps from KUDU-1968 a couple of times and
> verified that no blocks were lost, servers could restart correctly, and
> ksck with checksum_scan mode passed.
>
> I also successfully ran RAT as part of building the release tarball.
>
> -Todd
>
> On Thu, Apr 13, 2017 at 2:35 PM, Adar Dembo  wrote:
>
> > +1
> >
> > I downloaded, built on Ubuntu 16.04 for TSAN, and all the C++ tests
> passed.
> >
> > I recently filed KUDU-1975 which is (IMHO) an annoying regression, but
> > I don't think it should hold up the release.
> >
> >
> > On Thu, Apr 13, 2017 at 9:09 AM, Jean-Daniel Cryans  >
> > wrote:
> > > +1
> > >
> > > I downloaded the bits, verified the signature, md5, sha1. Built
> > everything
> > > on CentOS 6.6 in release mode, ran all the tests, they passed. Same for
> > > Java.
> > >
> > > J-D
> > >
> > > On Wed, Apr 12, 2017 at 10:15 PM, Todd Lipcon  wrote:
> > >
> > >> Hi,
> > >>
> > >> The Apache Kudu team is happy to announce the first release candidate
> > for
> > >> Apache Kudu 1.3.1.
> > >>
> > >> Apache Kudu 1.3.1 is a bug fix release which fixes critical issues
> > >> discovered in Apache Kudu 1.3.0. Please see the release notes for
> > details.
> > >>
> > >> The is a source-only release. The artifacts were staged here:
> > >> https://dist.apache.org/repos/dist/dev/kudu/1.3.1-RC1/
> > >>
> > >> Java convenience binaries in the form of a Maven repository are staged
> > >> here:
> > >> https://repository.apache.org/content/repositories/
> orgapachekudu-1009/
> > >>
> > >> It was built from this tag:
> > >> *https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > >> afc0f479ba6e04ec84c3c10c55be940c0784c8e1
> > >>  > >> afc0f479ba6e04ec84c3c10c55be940c0784c8e1>*
> > >>
> > >> The release notes can be found here:
> > >> *https://github.com/apache/kudu/blob/branch-1.3.x/docs/
> > release_notes.adoc
> > >>  > release_notes.adoc
> > >> >*
> > >>
> > >> KEYS file:
> > >> http://www.apache.org/dist/kudu/KEYS
> > >>
> > >> I'd suggest going through the README, building Kudu, and running the
> > >> unit tests. Testing out the Maven repo would also be appreciated.
> > >>
> > >> The vote will run until Monday, April 17th at 6PM PDT (more than the
> > >> required minimum 72 hours so folks have three weekdays and don't feel
> > >> pressured to vote over the weekend).
> > >>
> > >> Thanks,
> > >> Todd
> > >>
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: Multi-word Flag Style

[Dan Burkert]

On Mon, Apr 10, 2017 at 11:56 AM, Adar Dembo <a...@cloudera.com> wrote:
>
> But, I don't know whether gflags can be coerced to programmatically
> emit flags with dashes (i.e. when invoked with --help) without a patch
> or two.


Yah, I was thinking the same.  Gflags just landed support for dashes
upstream in 2.2, so I imagine they would be open to a patch to configure
dashes in help output.


> Certainly in the code we would want to retain the use of
> underscores when referring to flag variables; FLAGS_foo_bar conforms
> to our coding style more than something like FLAGS-foo-bar.
>

I agree - I don't think dashes are valid identifier characters in C++
anyway.

- Dan


>
>
> On Mon, Apr 10, 2017 at 11:00 AM, Alexey Serbin <aser...@cloudera.com>
> wrote:
> > I think it's a good move.  It would be nice to add a notice about that in
> > the user-facing docs.
> >
> > Also, I think it would be more consistent to convert those flags
> altogether
> > at some point to be in dash-ish form, both the code and the docs.  Maybe,
> > 1.4 is a good point to do that.
> >
> >
> > Kind regards,
> >
> > Alexey
> >
> >
> >
> > On 4/10/17 10:42 AM, William Berkeley wrote:
> >>
> >> I agree, for the reason you gave: dashes are the norm in Unix, so they
> >> "feel right" for flag names.
> >>
> >> -Will
> >>
> >> On Mon, Apr 10, 2017 at 1:38 PM, Dan Burkert <danburk...@apache.org>
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> As of Kudu 1.3, multi-word flags can use a dash '-' separator in lieu
> of
> >>> the underscore '_' separator.  For example,  --memory_limit_hard_bytes
> >>> can
> >>> now be specified as --memory-limit-hard-bytes, or even
> >>> --memory_limit-hard_bytes.  Of the people I've talked to, most seem to
> >>> prefer dashes to underscores in flag names, since that's been the Unix
> >>> norm
> >>> for a long time.
> >>>
> >>> Going forward, I'd like to propose that we document flag names using
> >>> dashes
> >>> wherever possible.  We would continue accepting underscores
> indefinitely,
> >>> since to stop doing so would break compatibility. For the most part,
> this
> >>> means incrementally switching the documentation to use dashes, and
> >>> getting
> >>> glog to output dashes in --help output.
> >>>
> >>> Any thoughts?
> >>>
> >>> - Dan
> >>>
> >
>

Multi-word Flag Style

[Dan Burkert]

Hi all,

As of Kudu 1.3, multi-word flags can use a dash '-' separator in lieu of
the underscore '_' separator.  For example,  --memory_limit_hard_bytes can
now be specified as --memory-limit-hard-bytes, or even
--memory_limit-hard_bytes.  Of the people I've talked to, most seem to
prefer dashes to underscores in flag names, since that's been the Unix norm
for a long time.

Going forward, I'd like to propose that we document flag names using dashes
wherever possible.  We would continue accepting underscores indefinitely,
since to stop doing so would break compatibility. For the most part, this
means incrementally switching the documentation to use dashes, and getting
glog to output dashes in --help output.

Any thoughts?

- Dan

Re: [VOTE] Apache Kudu 1.3.0 RC1

[Dan Burkert]

The macOS compile issue backport just landed in branch-1.3.x.  The macOS
test issue backport is in review <https://gerrit.cloudera.org/#/c/6353/>.

- Dan

On Fri, Mar 10, 2017 at 4:59 PM, Todd Lipcon <t...@cloudera.com> wrote:

> Alright, sounds fair. Are these issues being actively worked on at the
> moment by you/Dan? Would like to get a new RC out ASAP.
>
> -Todd
>
> On Fri, Mar 10, 2017 at 4:51 PM, Adar Dembo <a...@cloudera.com> wrote:
>
> > -1
> >
> > env_util-test.cc and env-test.cc both fail to compile on Ubuntu 16.04
> > with the default gcc (5.4.0). The errors look like this:
> >
> > /tmp/apache-kudu-1.3.0/src/kudu/util/env_util-test.cc: In lambda
> function:
> > /tmp/apache-kudu-1.3.0/src/kudu/util/env_util-test.cc:65:71: error:
> > label ‘gtest_label_testnofatal_67’ used but not defined
> > /tmp/apache-kudu-1.3.0/src/kudu/util/env_util-test.cc:65:108: warning:
> > label ‘gtest_label_testnofatal_73’ defined but not used
> > [-Wunused-label]
> > /tmp/apache-kudu-1.3.0/src/kudu/util/env_util-test.cc: In member
> > function ‘virtual void
> > kudu::env_util::EnvUtilTest_TestDiskSpaceCheck_Test::TestBody()’:
> > /tmp/apache-kudu-1.3.0/src/kudu/util/env_util-test.cc:65:71: error:
> > label ‘gtest_label_testnofatal_71’ used but not defined
> > /tmp/apache-kudu-1.3.0/src/kudu/util/env_util-test.cc:65:108: warning:
> > label ‘gtest_label_testnofatal_73’ defined but not used
> > [-Wunused-label]
> > src/kudu/util/CMakeFiles/env_util-test.dir/build.make:62: recipe for
> > target 'src/kudu/util/CMakeFiles/env_util-test.dir/env_util-test.cc.o'
> > failed
> >
> > I realize this is just test code, but AFAIK we have yet to ship a
> > release whose default build configuration (i.e. without NO_TESTS=1)
> > fails to build on a commonly used platform, and I'd like to avoid
> > setting that precedent. Between this and the macOS build breakages Dan
> > pointed out, I think we have enough reason to do an RC1 that builds
> > properly.
> >
> > On Fri, Mar 10, 2017 at 6:00 PM, Dan Burkert <danburk...@apache.org>
> > wrote:
> > > +0
> > >
> > > - env_util-test.cc fails to build on OS X 10.10 (fixed on master in
> > 97831ead
> > > <https://github.com/apache/kudu/commit/97831ead>)
> > > - external_mini_cluster-test and security-itest fail on OS X 10.10
> (fixed
> > > on master in c5ec0ddb0
> > > <https://github.com/apache/kudu/commit/c5ec0ddb01da87de4a037a4879a4ef
> > 92c434930f>
> > > )
> > > - verified signature
> > >
> > > I don't think we should hold the release over macOS test issues, but if
> > we
> > > end up needing to sink it for another reason we should backport those
> two
> > > patches.
> > >
> > > - Dan
> > >
> > > On Thu, Mar 9, 2017 at 10:37 PM, Todd Lipcon <t...@apache.org> wrote:
> > >
> > >> On Thu, Mar 9, 2017 at 10:36 PM, Todd Lipcon <t...@apache.org> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > The Apache Kudu team is happy to announce the first release
> candidate
> > for
> > >> > Apache Kudu 1.3.0.
> > >> >
> > >> > Apache Kudu 1.3 is a minor release which adds various new features,
> > >> > improvements, bug fixes, and optimizations on top of Kudu 1.2.
> > Highlights
> > >> > include significantly improved support for security, garbage
> > collection
> > >> of
> > >> > historical data, and lower space consumption in default
> > configurations.
> > >> > Please see the release notes for details.
> > >> >
> > >> > Thanks to the 25 developers who contributed code or documentation to
> > this
> > >> > release!
> > >> >
> > >> > The is a source-only release. The artifacts were staged here:
> > >> > https://dist.apache.org/repos/dist/dev/kudu/1.3.0-RC1/
> > >> >
> > >> > Java convenience binaries in the form of a Maven repository are
> staged
> > >> > here:
> > >> > https://repository.apache.org/content/repositories/
> > orgapachekudu-1008/
> > >> >
> > >> > It was built from this tag:
> > >> > https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > >> > dfd9b491f635a89b75889f8277316c9b78143883
> > >> >
> > >> > The release notes can be found here (some links from this document
> > will
> > >> > only work when this version is released):
> > >> > *https://github.com/apache/kudu/blob/branch-1.3.x/docs/relea
> > >> se_notes.adoc
> > >> > <https://github.com/apache/kudu/blob/branch-1.3.x/docs/relea
> > >> se_notes.adoc>*
> > >> >
> > >> > KEYS file:
> > >> > http://www.apache.org/dist/kudu/KEYS
> > >> >
> > >> > I'd suggest going through the README, building Kudu, and running the
> > >> > unit tests. Testing out the Maven repo would also be appreciated.
> > >> >
> > >> > The vote will run until Monday, 3/12 at 11PM PST. This is a bit more
> > than
> > >> > the minimum 72 hours, but given the weekend, I figured it's worth
> > giving
> > >> > two full business days for people to vote.
> > >> >
> > >> Oops, just noticed the typo right after sending. This should read
> > Monday,
> > >> 3/13 at 11pm PST.
> > >>
> > >> -Todd
> > >>
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: [VOTE] Apache Kudu 1.3.0 RC1

[Dan Burkert]

+0

- env_util-test.cc fails to build on OS X 10.10 (fixed on master in 97831ead
)
- external_mini_cluster-test and security-itest fail on OS X 10.10 (fixed
on master in c5ec0ddb0

)
- verified signature

I don't think we should hold the release over macOS test issues, but if we
end up needing to sink it for another reason we should backport those two
patches.

- Dan

On Thu, Mar 9, 2017 at 10:37 PM, Todd Lipcon  wrote:

> On Thu, Mar 9, 2017 at 10:36 PM, Todd Lipcon  wrote:
>
> > Hi,
> >
> > The Apache Kudu team is happy to announce the first release candidate for
> > Apache Kudu 1.3.0.
> >
> > Apache Kudu 1.3 is a minor release which adds various new features,
> > improvements, bug fixes, and optimizations on top of Kudu 1.2. Highlights
> > include significantly improved support for security, garbage collection
> of
> > historical data, and lower space consumption in default configurations.
> > Please see the release notes for details.
> >
> > Thanks to the 25 developers who contributed code or documentation to this
> > release!
> >
> > The is a source-only release. The artifacts were staged here:
> > https://dist.apache.org/repos/dist/dev/kudu/1.3.0-RC1/
> >
> > Java convenience binaries in the form of a Maven repository are staged
> > here:
> > https://repository.apache.org/content/repositories/orgapachekudu-1008/
> >
> > It was built from this tag:
> > https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > dfd9b491f635a89b75889f8277316c9b78143883
> >
> > The release notes can be found here (some links from this document will
> > only work when this version is released):
> > *https://github.com/apache/kudu/blob/branch-1.3.x/docs/relea
> se_notes.adoc
> >  se_notes.adoc>*
> >
> > KEYS file:
> > http://www.apache.org/dist/kudu/KEYS
> >
> > I'd suggest going through the README, building Kudu, and running the
> > unit tests. Testing out the Maven repo would also be appreciated.
> >
> > The vote will run until Monday, 3/12 at 11PM PST. This is a bit more than
> > the minimum 72 hours, but given the weekend, I figured it's worth giving
> > two full business days for people to vote.
> >
> Oops, just noticed the typo right after sending. This should read Monday,
> 3/13 at 11pm PST.
>
> -Todd
>

Re: Proposal for 1.3 branch and release

[Dan Burkert]

+1 SGTM, thanks!

- Dan

On Tue, Feb 21, 2017 at 11:19 AM, Hao Hao  wrote:

> +1 Thanks a lot Todd for putting this together and volunteered to be the
> RM.
>
> Best,
> Hao
>
> On Tue, Feb 21, 2017 at 10:40 AM, Dave Wang  wrote:
>
> > Strong +1 on your proposal Todd.
> >
> > - Dave
> >
> > On Tue, Feb 21, 2017 at 10:39 AM, Jean-Daniel Cryans <
> jdcry...@apache.org>
> > wrote:
> >
> > > +1 on the plan and +1 on you being the RM since you're so close to all
> > the
> > > security-related patches.
> > >
> > > J-D
> > >
> > > On Tue, Feb 21, 2017 at 10:32 AM, Todd Lipcon  wrote:
> > >
> > > > Hi folks,
> > > >
> > > > I don't think the topic of Kudu 1.3 has been brought up on the dev
> list
> > > yet
> > > > (though I just hinted at it on a user@ thread). I wanted to open
> this
> > > > topic
> > > > to discussion by proposing a release plan.
> > > >
> > > > Back in December, the discussion around the 1.2 release was to hold
> off
> > > on
> > > > the security features, and push them to 1.3. So, obviously security
> has
> > > > been a big focus of development over the last month or so. I've been
> > > pretty
> > > > involved with this development and it seems from my perspective that
> > the
> > > > initial phase of development (authentication without authorization)
> is
> > > > wrapping up, and would be good to get out to users.
> > > >
> > > > Aside from security, there are a bunch of good bug fixes and
> > > improvements.
> > > > From a quick scan of the git log, a few things of note are:
> > > >
> > > > - fixes to various cases where more disk space was consumed than
> > actually
> > > > used for data
> > > > - support for using Breakpad for crash analysis
> > > > - new default encodings/compression to save space and increase
> > > performance
> > > > - compression of the WAL and deltafiles (performance and space)
> > > > - support for IS NULL and IS NOT NULL predicates
> > > > - improved hash-partition pruning for IN-list predicates
> > > >
> > > > ...plus a few nice bug fixes and stability improvements.
> > > >
> > > > All of the above seems like it should make for a compelling 1.3
> > release.
> > > >
> > > > Following the pattern of 1.2, I think we should plan to branch this
> > week,
> > > > and then spend a week or two on stabilization and testing before
> making
> > > the
> > > > release candidate.
> > > >
> > > > I'll volunteer to be the RM again unles anyone else feels like
> taking a
> > > > turn.
> > > >
> > > > Any comments or concerns?
> > > >
> > > > -Todd
> > > >
> > > > --
> > > > Todd Lipcon
> > > > Software Engineer, Cloudera
> > > >
> > >
> >
>

Re: TLS for localhost connections

[Dan Burkert]

On Fri, Feb 10, 2017 at 10:02 AM, Todd Lipcon  wrote:
>
> Yea, I figured that both sides would check the remote peer, and if they
> both agree that the other side is local, they'd offer TLS_NEGOTIATION_ONLY
> or somesuch. This could also be used in a scenario where a user needs
> authentication but fully trusts their network and therefore doesn't
> need/want encryption, so I'd call it NEGOTIATION_ONLY or AUTH_ONLY rather
> than LOCAL.
>
>
SGTM.


>
> Yea, but still the best number here is 685MB/sec. Assuming 2ghz, that's
> around 3 cycles/byte (~25x slower than crc32). According to Intel, AES
> encryption with AESNI can be around 1.3 cycles/byte:
> https://software.intel.com/sites/default/files/m/d/4/1/d/
> 8/10TB24_Breakthrough_AES_Performance_with_Intel_AES_
> New_Instructions.final.secure.pdf
>
>
Sorry I was a little opaque there, but my point was we should expect the
integrity overhead to be even *higher* on Centos6, since it ships with an
older
OpenSSL than you tested on (although not as old as my numbers from 0.9.8).


> -Todd
>
>
> >
> > On Thu, Feb 9, 2017 at 10:22 PM, Todd Lipcon  wrote:
> >
> > > Hey folks,
> > >
> > > For those not following along, we're very close to the point where
> we'll
> > be
> > > enabling TLS for all wire communication done by a Kudu cluster (at
> least
> > > when security features are enabled). One thing we've decided is
> important
> > > is to preserve good performance for applications like Spark and Impala
> > > which typically schedule tasks local to the data on the tablet servers,
> > and
> > > we think that enabling TLS for these localhost connections will have an
> > > unacceptable performance hit.
> > >
> > > Our thinking was to continue to use TLS *authentication* to prevent
> MITM
> > > attacks (possible because we typically don't bind to low ports). But,
> we
> > > don't need TLS *encryption*.
> > >
> > > This is possible using the various TLS "NULL" ciphers -- we can have
> both
> > > the client and server notice that the remote peer is local and enable
> the
> > > NULL cipher suite. However, I did some research this evening and it
> looks
> > > like the NULL ciphers disable encryption but don't disable the MAC
> > > integrity portion of TLS. Best I can tell, there is no API to do so.
> > >
> > > I did some brief checks using openssl s_client and s_server on my
> laptop
> > > (openssl 1.0.2g, haswell), and got the following numbers for
> transferring
> > > 5GB:
> > >
> > > ADH-AES128-SHA
> > > Client: 42.2M cycles
> > > Server: 35.3M cycles
> > >
> > > AECDH-NULL-SHA: (closest NULL I could find to the above)
> > > Client: 36.2M cycles
> > > Server: 28.6M cycles
> > >
> > > no TLS at all (using netcat to a local TCP port):
> > > Client: 20.8M cycles
> > > Server: 10.0M cycles
> > >
> > > baseline: iperf -n 5000M localhost
> > > Client: 2.3M cycles
> > > Server: 1.8M cycles
> > > [not sure why this is so much faster than netcat - I guess because with
> > > netcat I was piping to /dev/null which still requires more syscalls?]
> > >
> > > (note that the client in all of these cases includes the 'dd' command
> to
> > > generate the data, which probably explains why it's 7-10M cycles more
> > than
> > > the server in every case)
> > >
> > > To summarize, just disabling encryption has not much improvement, given
> > > that Intel chips now optimize AES. The checksumming itself adds more
> > > significant overhead than the encryption. This agrees with numbers I've
> > > seen around the web that crypto-strength checksums only go 1GB/sec or
> so
> > > max, typically much slower.
> > >
> > > Thinking about the best solution here, I think we should consider using
> > TLS
> > > during negotiation, and then just completely dropping the TLS (i.e not
> > > wrapping the sockets in TlsSockets). I think this still gives us the
> > > protection against the localhost MITM (because the handshake would
> fail)
> > > and be trivially zero-overhead. Am I missing any big issues with this
> > idea?
> > > Anyone got a better one?
> > >
> > > -Todd
> > > --
> > > Todd Lipcon
> > > Software Engineer, Cloudera
> > >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: TLS for localhost connections

[Dan Burkert]

A couple thoughts:

* You had to explicitly turn on the ADH and AECDH ciphers, right?  We
shouldn't be using those in any circumstances, but I don't think it would
change the results of your test.

* We've discussed adding checksums to the RPC system in the past, but
punted since we would get it 'for free' with TLS.  Perhaps we would want to
turn off checksums in the localhost scenario anyway, though?

* The solution of not wrapping the socket sounds reasonable.  We may need
to add a new RPC feature flag (e.g. TLS_LOCAL) so that we can be sure that
both sides recognize they are on a localhost connection.  Or perhaps this
isn't a problem and a connection can always be determined to be to the
localhost?

* The overhead of crypto hashes is highly dependent on the implementation.
For instance, some tests on my laptop (openssl speed md5 sha1 sha256 sha512
):

OpenSSL 0.9.8zg

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
md5  37815.63k   116197.14k   251336.60k   358376.77k
414424.95k
sha1 44766.06k   130830.68k   275829.53k   392799.78k
452215.48k
sha256   31386.52k73018.50k   132622.47k   161953.92k
175492.62k
sha512   20567.70k84586.04k   160368.90k   240929.09k
273687.68k

OpenSSL 1.0.2k

The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192
bytes
md5  55295.46k   158955.86k   360768.81k   510628.52k
580591.62k
sha1 66130.74k   188498.24k   443250.86k   685325.87k
824453.80k
sha256   69510.92k   153928.53k   275164.76k   349643.09k
379147.13k
sha512   44970.85k   175308.13k   304458.49k   448055.48k
524516.17k

sha1 is almost 50% faster, and sha512 is almost 100% faster.



On Thu, Feb 9, 2017 at 10:22 PM, Todd Lipcon  wrote:

> Hey folks,
>
> For those not following along, we're very close to the point where we'll be
> enabling TLS for all wire communication done by a Kudu cluster (at least
> when security features are enabled). One thing we've decided is important
> is to preserve good performance for applications like Spark and Impala
> which typically schedule tasks local to the data on the tablet servers, and
> we think that enabling TLS for these localhost connections will have an
> unacceptable performance hit.
>
> Our thinking was to continue to use TLS *authentication* to prevent MITM
> attacks (possible because we typically don't bind to low ports). But, we
> don't need TLS *encryption*.
>
> This is possible using the various TLS "NULL" ciphers -- we can have both
> the client and server notice that the remote peer is local and enable the
> NULL cipher suite. However, I did some research this evening and it looks
> like the NULL ciphers disable encryption but don't disable the MAC
> integrity portion of TLS. Best I can tell, there is no API to do so.
>
> I did some brief checks using openssl s_client and s_server on my laptop
> (openssl 1.0.2g, haswell), and got the following numbers for transferring
> 5GB:
>
> ADH-AES128-SHA
> Client: 42.2M cycles
> Server: 35.3M cycles
>
> AECDH-NULL-SHA: (closest NULL I could find to the above)
> Client: 36.2M cycles
> Server: 28.6M cycles
>
> no TLS at all (using netcat to a local TCP port):
> Client: 20.8M cycles
> Server: 10.0M cycles
>
> baseline: iperf -n 5000M localhost
> Client: 2.3M cycles
> Server: 1.8M cycles
> [not sure why this is so much faster than netcat - I guess because with
> netcat I was piping to /dev/null which still requires more syscalls?]
>
> (note that the client in all of these cases includes the 'dd' command to
> generate the data, which probably explains why it's 7-10M cycles more than
> the server in every case)
>
> To summarize, just disabling encryption has not much improvement, given
> that Intel chips now optimize AES. The checksumming itself adds more
> significant overhead than the encryption. This agrees with numbers I've
> seen around the web that crypto-strength checksums only go 1GB/sec or so
> max, typically much slower.
>
> Thinking about the best solution here, I think we should consider using TLS
> during negotiation, and then just completely dropping the TLS (i.e not
> wrapping the sockets in TlsSockets). I think this still gives us the
> protection against the localhost MITM (because the handshake would fail)
> and be trivially zero-overhead. Am I missing any big issues with this idea?
> Anyone got a better one?
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: Monitoring Use Cases for Kudu

[Dan Burkert]

Hi Senthil,

Kudu masters and tservers expose internal metrics in JSON format at the
/metrics endpoint of the web UI.  You could start by scraping this endpoint
every 10 seconds or so, and recording that.  I think that's more or less
how Cloudera Manager exposes Kudu metrics.  Our docs
 have more
info about the metrics collected, and other ways they are exposed.

- Dan

On Mon, Jan 30, 2017 at 9:26 PM, Senthil Kumar 
wrote:

> [BCC - Kudu Dev Team]
>
> Hello Team ,  Recently we start Kudu Production Cluster , but i feel like
> Monitoring Kudu Components is critical in Production to avoid failures.
>
> I'm thinking below two components are good to start with
>
> 1)  Masters
> 2) Tablet Servers
>
>
> I welcome everyone's suggestion here to list down the requirements for
> monitoring Kudu Cluster ..
>
>
> Cheers,
> Senthil
>

Re: TokenPB contents

[Dan Burkert]

On Wed, Jan 25, 2017 at 3:33 PM, Alexey Serbin <aser...@cloudera.com> wrote:
>
> Or it's proposed that if new
> fields are found in a token then don't accept the token at all?  In that
> case it might be an issue during rolling upgrade, I think.
>

That's what I was thinking, but that's a good reason not to do it that
way.  Our normal required feature flags mechanism should work in that case,
where the field is not critical/required.


>
>
> Best regards,
>
> Alexey
>
> On Wed, Jan 25, 2017 at 3:14 PM, Dan Burkert <danburk...@apache.org>
> wrote:
>
> > That's an interesting idea - say if the format evolved to have an
> > additional field which restricts use of the token?  Could we use the
> > protobuf unknown fields API to recognize if this happened?
> >
> > - Dan
> >
> > On Wed, Jan 25, 2017 at 3:03 PM, Alexey Serbin <aser...@cloudera.com>
> > wrote:
> >
> >> I like this idea.
> >>
> >> Probably, that's too early at this point, but consider adding notion of
> >> compt/non-compat feature flags into tokens.  Imagine the new version of
> >> token has some additional restriction field, which older code does not
> >> understand.  In that case, even if the token signature is correct, the
> >> older code should not accept the new token since unsupported non-compat
> >> feature flags is present.
> >>
> >> But in the essence this looks great, IMO.
> >>
> >>
> >> Best regards,
> >>
> >> Alexey
> >>
> >> On Wed, Jan 25, 2017 at 12:52 PM, Todd Lipcon <t...@cloudera.com>
> wrote:
> >>
> >>> Actually had one more idea... how about:
> >>> message AuthenticationToken {
> >>> };
> >>>
> >>> message AuthorizationToken {
> >>> };
> >>>
> >>> message TokenPB {
> >>>   // The time at which this token expires, in seconds since the
> >>>   // unix epoch.
> >>>   optional int64 expire_unix_epoch_seconds = 1;
> >>>
> >>>   oneof token {
> >>> AuthenticationToken authn = 2;
> >>> AuthenticationToken authz = 3;
> >>>   }
> >>> };
> >>>
> >>> message SignedTokenPB {
> >>>   // The actual token contents. This is a serialized TokenPB protobuf.
> >>> However, we use a
> >>>   // 'bytes' field, since protobuf doesn't guarantee that if two
> >>> implementations serialize
> >>>   // a protobuf, they'll necessary get bytewise identical results,
> >>> particularly in the
> >>>   // presence of unknown fields.
> >>>   optional bytes token_contents = 2;
> >>>
> >>>   // The cryptographic signature of 'token_contents'.
> >>>   optional bytes signature = 3;
> >>>
> >>>   // The sequence number of the key which produced 'signature'.
> >>>   optional int64 signing_key_seq_num = 4;
> >>> };
> >>>
> >>>
> >>>
> >>> On Wed, Jan 25, 2017 at 12:44 PM, Todd Lipcon <t...@cloudera.com>
> wrote:
> >>>
> >>>> On Wed, Jan 25, 2017 at 12:40 PM, Dan Burkert <d...@cloudera.com>
> wrote:
> >>>>
> >>>>> I think it must go in the 'token_contents' itself, otherwise it can
> be
> >>>>> modified by a malicious client.  Other than that, looks good.
> >>>>>
> >>>>>
> >>>> well, if it went into a separate field, then we'd have something like:
> >>>>
> >>>> optional bytes token_contents = 1;
> >>>> optional int64 expiration_timestamp = 2;
> >>>>
> >>>> // Signature of the string: '<32-bit big-endian length of
> >>>> token_contents> token_contents <64-bit big-endian expiration>'
> >>>> optional bytes signature = 3;
> >>>>
> >>>> so they could try to modify it, but the signature would fail.
> >>>>
> >>>>
> >>>>
> >>>>> On Wed, Jan 25, 2017 at 12:37 PM, Todd Lipcon <t...@cloudera.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hey folks
> >>>>>>
> >>>>>> I'm working on the token signing/verification stuff at the moment.
> >>>>>> Curious to solicit some opinions on this:
> >>>>>>
> >>>>>>
> >>>>>> message 

Re: TokenPB contents

[Dan Burkert]

That's an interesting idea - say if the format evolved to have an
additional field which restricts use of the token?  Could we use the
protobuf unknown fields API to recognize if this happened?

- Dan

On Wed, Jan 25, 2017 at 3:03 PM, Alexey Serbin <aser...@cloudera.com> wrote:

> I like this idea.
>
> Probably, that's too early at this point, but consider adding notion of
> compt/non-compat feature flags into tokens.  Imagine the new version of
> token has some additional restriction field, which older code does not
> understand.  In that case, even if the token signature is correct, the
> older code should not accept the new token since unsupported non-compat
> feature flags is present.
>
> But in the essence this looks great, IMO.
>
>
> Best regards,
>
> Alexey
>
> On Wed, Jan 25, 2017 at 12:52 PM, Todd Lipcon <t...@cloudera.com> wrote:
>
>> Actually had one more idea... how about:
>> message AuthenticationToken {
>> };
>>
>> message AuthorizationToken {
>> };
>>
>> message TokenPB {
>>   // The time at which this token expires, in seconds since the
>>   // unix epoch.
>>   optional int64 expire_unix_epoch_seconds = 1;
>>
>>   oneof token {
>> AuthenticationToken authn = 2;
>> AuthenticationToken authz = 3;
>>   }
>> };
>>
>> message SignedTokenPB {
>>   // The actual token contents. This is a serialized TokenPB protobuf.
>> However, we use a
>>   // 'bytes' field, since protobuf doesn't guarantee that if two
>> implementations serialize
>>   // a protobuf, they'll necessary get bytewise identical results,
>> particularly in the
>>   // presence of unknown fields.
>>   optional bytes token_contents = 2;
>>
>>   // The cryptographic signature of 'token_contents'.
>>   optional bytes signature = 3;
>>
>>   // The sequence number of the key which produced 'signature'.
>>   optional int64 signing_key_seq_num = 4;
>> };
>>
>>
>>
>> On Wed, Jan 25, 2017 at 12:44 PM, Todd Lipcon <t...@cloudera.com> wrote:
>>
>>> On Wed, Jan 25, 2017 at 12:40 PM, Dan Burkert <d...@cloudera.com> wrote:
>>>
>>>> I think it must go in the 'token_contents' itself, otherwise it can be
>>>> modified by a malicious client.  Other than that, looks good.
>>>>
>>>>
>>> well, if it went into a separate field, then we'd have something like:
>>>
>>> optional bytes token_contents = 1;
>>> optional int64 expiration_timestamp = 2;
>>>
>>> // Signature of the string: '<32-bit big-endian length of
>>> token_contents> token_contents <64-bit big-endian expiration>'
>>> optional bytes signature = 3;
>>>
>>> so they could try to modify it, but the signature would fail.
>>>
>>>
>>>
>>>> On Wed, Jan 25, 2017 at 12:37 PM, Todd Lipcon <t...@cloudera.com>
>>>> wrote:
>>>>
>>>>> Hey folks
>>>>>
>>>>> I'm working on the token signing/verification stuff at the moment.
>>>>> Curious to solicit some opinions on this:
>>>>>
>>>>>
>>>>> message TokenPB {
>>>>>   // The actual token contents. This is typically a serialized
>>>>>   // protobuf of its own. However, we use a 'bytes' field, since
>>>>>   // protobuf doesn't guarantee that if two implementations serialize
>>>>>   // a protobuf, they'll necessary get bytewise identical results,
>>>>>   // particularly in the presence of unknown fields.
>>>>>   optional bytes token_contents = 1;
>>>>>
>>>>>   // The cryptographic signature of 'token_contents'.
>>>>>   optional bytes signature = 2;
>>>>>
>>>>>   // The sequence number of the key which produced 'signature'.
>>>>>   optional int64_t signing_key_seq_num = 3;
>>>>> };
>>>>>
>>>>> The thing that's currently missing is an expiration timestamp of the
>>>>> signature. I have two options here:
>>>>>
>>>>> *Option A*) say that the TokenPB itself doesn't capture expiration,
>>>>> and if a particular type of token needs expiration, it would have to put 
>>>>> an
>>>>> 'expiration time' in its token contents itself.
>>>>>
>>>>> *pros:*
>>>>> - token signing/verification is just a simple operation on the
>>>>> 'token_contents' string
>>>>>
>>>>> *Cons:*
>>>>> - would likely end up with redundant code between AuthN and AuthZ
>>>>> tokens, both of which need expiration. However, that code isn't very
>>>>> complicated (just a timestamp comparison) so maybe not a big deal?
>>>>>
>>>>> *Option B)* add an expiration timestamp field to TokenPB
>>>>> *pros:*
>>>>> - consolidate the expiration checking code into TokenVerifier
>>>>> *cons:*
>>>>> - now in order to sign/verify a token, we actually need to be signing
>>>>> something like a concatenation of 'token_contents + signature'. Not too
>>>>> hard to construct this concatenation, but it does add some complexity.
>>>>>
>>>>> Any strong opinions either way?
>>>>>
>>>>> -Todd
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>

Re: TokenPB contents

[Dan Burkert]

On Wed, Jan 25, 2017 at 12:52 PM, Todd Lipcon <t...@cloudera.com> wrote:

> Actually had one more idea... how about:
> message AuthenticationToken {
> };
>
> message AuthorizationToken {
> };
>
> message TokenPB {
>   // The time at which this token expires, in seconds since the
>   // unix epoch.
>   optional int64 expire_unix_epoch_seconds = 1;
>
>   oneof token {
> AuthenticationToken authn = 2;
> AuthenticationToken authz = 3;
>   }
> };
>
> message SignedTokenPB {
>   // The actual token contents. This is a serialized TokenPB protobuf.
> However, we use a
>   // 'bytes' field, since protobuf doesn't guarantee that if two
> implementations serialize
>   // a protobuf, they'll necessary get bytewise identical results,
> particularly in the
>   // presence of unknown fields.
>   optional bytes token_contents = 2;
>
>   // The cryptographic signature of 'token_contents'.
>   optional bytes signature = 3;
>
>   // The sequence number of the key which produced 'signature'.
>   optional int64 signing_key_seq_num = 4;
> };
>
>
This looks great to me.


>
>
> On Wed, Jan 25, 2017 at 12:44 PM, Todd Lipcon <t...@cloudera.com> wrote:
>
>> On Wed, Jan 25, 2017 at 12:40 PM, Dan Burkert <d...@cloudera.com> wrote:
>>
>>> I think it must go in the 'token_contents' itself, otherwise it can be
>>> modified by a malicious client.  Other than that, looks good.
>>>
>>>
>> well, if it went into a separate field, then we'd have something like:
>>
>> optional bytes token_contents = 1;
>> optional int64 expiration_timestamp = 2;
>>
>> // Signature of the string: '<32-bit big-endian length of token_contents>
>> token_contents <64-bit big-endian expiration>'
>> optional bytes signature = 3;
>>
>> so they could try to modify it, but the signature would fail.
>>
>>
>>
>>> On Wed, Jan 25, 2017 at 12:37 PM, Todd Lipcon <t...@cloudera.com> wrote:
>>>
>>>> Hey folks
>>>>
>>>> I'm working on the token signing/verification stuff at the moment.
>>>> Curious to solicit some opinions on this:
>>>>
>>>>
>>>> message TokenPB {
>>>>   // The actual token contents. This is typically a serialized
>>>>   // protobuf of its own. However, we use a 'bytes' field, since
>>>>   // protobuf doesn't guarantee that if two implementations serialize
>>>>   // a protobuf, they'll necessary get bytewise identical results,
>>>>   // particularly in the presence of unknown fields.
>>>>   optional bytes token_contents = 1;
>>>>
>>>>   // The cryptographic signature of 'token_contents'.
>>>>   optional bytes signature = 2;
>>>>
>>>>   // The sequence number of the key which produced 'signature'.
>>>>   optional int64_t signing_key_seq_num = 3;
>>>> };
>>>>
>>>> The thing that's currently missing is an expiration timestamp of the
>>>> signature. I have two options here:
>>>>
>>>> *Option A*) say that the TokenPB itself doesn't capture expiration,
>>>> and if a particular type of token needs expiration, it would have to put an
>>>> 'expiration time' in its token contents itself.
>>>>
>>>> *pros:*
>>>> - token signing/verification is just a simple operation on the
>>>> 'token_contents' string
>>>>
>>>> *Cons:*
>>>> - would likely end up with redundant code between AuthN and AuthZ
>>>> tokens, both of which need expiration. However, that code isn't very
>>>> complicated (just a timestamp comparison) so maybe not a big deal?
>>>>
>>>> *Option B)* add an expiration timestamp field to TokenPB
>>>> *pros:*
>>>> - consolidate the expiration checking code into TokenVerifier
>>>> *cons:*
>>>> - now in order to sign/verify a token, we actually need to be signing
>>>> something like a concatenation of 'token_contents + signature'. Not too
>>>> hard to construct this concatenation, but it does add some complexity.
>>>>
>>>> Any strong opinions either way?
>>>>
>>>> -Todd
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>>
>>>
>>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: TokenPB contents

[Dan Burkert]

I think it must go in the 'token_contents' itself, otherwise it can be
modified by a malicious client.  Other than that, looks good.

- Dan

On Wed, Jan 25, 2017 at 12:37 PM, Todd Lipcon  wrote:

> Hey folks
>
> I'm working on the token signing/verification stuff at the moment. Curious
> to solicit some opinions on this:
>
>
> message TokenPB {
>   // The actual token contents. This is typically a serialized
>   // protobuf of its own. However, we use a 'bytes' field, since
>   // protobuf doesn't guarantee that if two implementations serialize
>   // a protobuf, they'll necessary get bytewise identical results,
>   // particularly in the presence of unknown fields.
>   optional bytes token_contents = 1;
>
>   // The cryptographic signature of 'token_contents'.
>   optional bytes signature = 2;
>
>   // The sequence number of the key which produced 'signature'.
>   optional int64_t signing_key_seq_num = 3;
> };
>
> The thing that's currently missing is an expiration timestamp of the
> signature. I have two options here:
>
> *Option A*) say that the TokenPB itself doesn't capture expiration, and
> if a particular type of token needs expiration, it would have to put an
> 'expiration time' in its token contents itself.
>
> *pros:*
> - token signing/verification is just a simple operation on the
> 'token_contents' string
>
> *Cons:*
> - would likely end up with redundant code between AuthN and AuthZ tokens,
> both of which need expiration. However, that code isn't very complicated
> (just a timestamp comparison) so maybe not a big deal?
>
> *Option B)* add an expiration timestamp field to TokenPB
> *pros:*
> - consolidate the expiration checking code into TokenVerifier
> *cons:*
> - now in order to sign/verify a token, we actually need to be signing
> something like a concatenation of 'token_contents + signature'. Not too
> hard to construct this concatenation, but it does add some complexity.
>
> Any strong opinions either way?
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: Use of boost non-header-only functionality

[Dan Burkert]

Is there an equivalent library for the JVM?

- Dan

On Thu, Jan 19, 2017 at 4:33 PM, Dan Burkert <danburk...@apache.org> wrote:

> I think for a long time we resisted vendoring boost, and so avoiding the
> dynamically linked parts of boost was advantageous to avoid
> incompatibilities in system boost versions.  I recall we are vendoring
> boost now, so that should be less of a concern, I would think.
>
> - Dan
>
> On Thu, Jan 19, 2017 at 4:16 PM, Todd Lipcon <t...@cloudera.com> wrote:
>
>> Hey folks,
>>
>> I'm starting to look at implementing a 96-bit timestamp type that would be
>> compatible with Impala and Parquet's TIMESTAMP. Internally, the 96 bits
>> are
>> a 32-bit day number (Julian day) and a 64-bit time on that day
>> (nanoseconds).
>>
>> This maps nicely to the boost::date_time::ptime data type for the purposes
>> of stringification, etc. Doing manual stringification without using a
>> library is a bit messy, since it has to account for leap years, etc, when
>> mapping the day number to a year/month/day.
>>
>> Unfortunately, we currently only use boost in a header-only fashion, and
>> it
>> appears the necessary functionality is not header-only. I can't quite
>> recall why we have the header-only restriction - it seems we should be
>> able
>> to static-link boost like any other library. Does anyone have a problem
>> with making that change? Was there some issue that I'm forgetting about?
>>
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>

Re: Use of boost non-header-only functionality

[Dan Burkert]

I think for a long time we resisted vendoring boost, and so avoiding the
dynamically linked parts of boost was advantageous to avoid
incompatibilities in system boost versions.  I recall we are vendoring
boost now, so that should be less of a concern, I would think.

- Dan

On Thu, Jan 19, 2017 at 4:16 PM, Todd Lipcon  wrote:

> Hey folks,
>
> I'm starting to look at implementing a 96-bit timestamp type that would be
> compatible with Impala and Parquet's TIMESTAMP. Internally, the 96 bits are
> a 32-bit day number (Julian day) and a 64-bit time on that day
> (nanoseconds).
>
> This maps nicely to the boost::date_time::ptime data type for the purposes
> of stringification, etc. Doing manual stringification without using a
> library is a bit messy, since it has to account for leap years, etc, when
> mapping the day number to a year/month/day.
>
> Unfortunately, we currently only use boost in a header-only fashion, and it
> appears the necessary functionality is not header-only. I can't quite
> recall why we have the header-only restriction - it seems we should be able
> to static-link boost like any other library. Does anyone have a problem
> with making that change? Was there some issue that I'm forgetting about?
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: [VOTE] Apache Kudu 1.2.0 RC1

[Dan Burkert]

Little bit late to the party, but release LGTM.  Compiled and passed all
tests on OS X 10.10.

- Dan

On Wed, Jan 18, 2017 at 8:48 AM, William Berkeley 
wrote:

> Yeah I'm ok with passing it. Just a thought since it would be a quick fix.
>
> -Will
>
> On Wed, Jan 18, 2017 at 5:05 AM, Dinesh Bhat  wrote:
>
> > +1
> > - built on OS X debug/release bits.
> > - All tests passed except env-test (seems to be failing with setrlimit
> > failure, not a blocker).
> >
> > Thanks,
> > Dinesh.
> > > On Jan 12, 2017, at 8:24 PM, Todd Lipcon  wrote:
> > >
> > > Hi,
> > >
> > > The Apache Kudu team is happy to announce the first release candidate
> for
> > > Apache Kudu 1.2.0.
> > >
> > > Apache Kudu 1.2 is a minor release which adds various new features,
> > > improvements, bug fixes, and optimizations on top of Kudu 1.1. Please
> see
> > > the release notes for details. Thanks to the 18 developers who
> > contributed
> > > code or documentation to this release!
> > >
> > > The is a source-only release. The artifacts were staged here:
> > > https://dist.apache.org/repos/dist/dev/kudu/1.2.0-RC1/
> > >
> > > Java convenience binaries in the form of a Maven repository are staged
> > here:
> > > https://repository.apache.org/content/repositories/orgapachekudu-1006/
> > >
> > > It was built from this tag:
> > > https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > c3276651dcb4a4995957fe99cd75b0e174dd7e90
> > >
> > > The release notes can be found here (some links from this document will
> > > only work when this version is released):
> > > https://github.com/apache/kudu/blob/branch-1.2
> > >  > release_notes.adoc>
> > > .x/docs/release_notes.adoc
> > >  > release_notes.adoc>
> > >
> > > KEYS file:
> > > http://www.apache.org/dist/kudu/KEYS
> > >
> > > I'd suggest going through the README, building Kudu, and running the
> > > unit tests. Testing out the Maven repo would also be appreciated.
> > >
> > > Given the upcoming holiday weekend in the US, I think we should extend
> > the
> > > voting period beyond the minimum-required 72 hours. This vote will
> close
> > > 11:59pm PST on next Tuesday, Jan 17.
> > >
> > > Thanks,
> > > Todd
> >
> >
>

Re: initial draft of security scoping document/roadmap

[Dan Burkert]

I've added a new section to the Authentication Design scope doc
<https://docs.google.com/document/d/1Yu4iuIhaERwug1vS95yWDd_WzrNRIKvvVGUb31y-_mY/edit?usp=sharing>
called
"TLS Negotiation" which has a specific recommendation for how Kudu will
handle TLS connection upgrade in a backwards compatible manner.  Comments
welcome!

- Dan

On Wed, Oct 26, 2016 at 5:05 PM, Todd Lipcon <t...@cloudera.com> wrote:

> I made some edits in Dan's doc linked above to turn it into more of a
> design doc about our current plan of attack. Please take a look at the
> updated doc.
>
> On Mon, Oct 17, 2016 at 6:48 PM, William Berkeley <wdberke...@cloudera.com
> >
> wrote:
>
> > Fixed link:
> > https://docs.google.com/document/d/1Yu4iuIhaERwug1vS95yWDd_
> > WzrNRIKvvVGUb31y-_mY/edit#
> >
> > -Will
> >
> > On Mon, Oct 17, 2016 at 9:42 PM, Dan Burkert <danburk...@apache.org>
> > wrote:
> >
> > > Hi all,
> > >
> > > We've been thinking about some of the particular details laid out in
> the
> > > scope doc, and have come up with some alternatives.  You can see the
> > > alternative and discussion here:
> > >
> > > https://docs.google.com/document/d/1Yu4iuIhaERwug1vS95yWDd_
> > > WzrNRIKvvVGUb31y-_mY/edit#
> > >
> > > - Dan
> > >
> > > On Mon, Oct 10, 2016 at 9:07 PM, Todd Lipcon <t...@cloudera.com>
> wrote:
> > >
> > > > Hey folks,
> > > >
> > > > I'm hoping to start breaking down some tasks from this document in
> the
> > > next
> > > > couple days. If you were planning to take a look, please try to do so
> > > soon.
> > > >
> > > > -Todd
> > > >
> > > > On Mon, Oct 3, 2016 at 5:33 PM, Todd Lipcon <t...@cloudera.com>
> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Now that we've reached 1.0, one of the next big barriers to Kudu
> > > adoption
> > > > > is its lack of security features. I've started working on a
> document
> > > > which
> > > > > tries to scope out some initial support for authentication,
> > > > authorization,
> > > > > and data security:
> > > > >
> > > > > https://docs.google.com/document/d/1cPNDTpVkIUo676RlszpTF1gHZ8l0T
> > > > > dbB7zFBAuOuYUw/edit#heading=h.gsibhnd5dyem
> > > > >
> > > > > The aim here is to lay out a roadmap that has some milestones that
> > are
> > > > > achievable in the next few upcoming releases. As noted in the last
> > > > section
> > > > > of the document, there will be more work to follow, but the goal
> here
> > > is
> > > > to
> > > > > get us started with the base infrastructure.
> > > > >
> > > > > It's rough, yet, but please take a look and leave comments,
> > questions,
> > > > and
> > > > > suggestions.
> > > > >
> > > > > Thanks
> > > > > -Todd
> > > > > --
> > > > > Todd Lipcon
> > > > > Software Engineer, Cloudera
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Todd Lipcon
> > > > Software Engineer, Cloudera
> > > >
> > >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: checkstyle on console during precommit builds

[Dan Burkert]

Thanks, Jordan!

On Mon, Nov 28, 2016 at 10:17 AM, Jordan Birdsell <jtbirds...@apache.org>
wrote:

> This has gone in, you will need to rebase any patches if you want to avoid
> the extra large build output.
>
> On Fri, Nov 25, 2016, 4:25 PM Jordan Birdsell <jordantbirds...@gmail.com>
> wrote:
>
> > This appears to have done it https://gerrit.cloudera.org/#/c/5225/.
> >
> > Used a SuppressionFilter module in the style config and that seemed to
> > work.
> >
> > On Fri, Nov 25, 2016 at 12:02 PM Mike Percy <mpe...@apache.org> wrote:
> >
> > I am familiar with checkstyle on Maven and could potentially take a look
> at
> > this, but it won't be before Tuesday.
> >
> > Mike
> >
> > On Wed, Nov 23, 2016 at 8:08 PM, Dan Burkert <d...@cloudera.com> wrote:
> >
> > > Nevermind, that didn't fix it.
> > >
> > > On Wed, Nov 23, 2016 at 11:59 AM, Dan Burkert <d...@cloudera.com>
> wrote:
> > >
> > > > Still double checking, but this seems to have done the trick:
> > > > https://gerrit.cloudera.org/#/c/5206/.  Not sure why, probably never
> > > will.
> > > >
> > > > - Dan
> > > >
> > > > On Wed, Nov 23, 2016 at 11:50 AM, Todd Lipcon <t...@cloudera.com>
> > wrote:
> > > >
> > > >> On Wed, Nov 23, 2016 at 11:13 AM, Dan Burkert <d...@cloudera.com>
> > wrote:
> > > >>
> > > >>> I'm looking into this a bit.  It does get disabled on generated
> > sources
> > > >>> when I run it locally, and there is a line in the pom.xml
> > > >>> <https://github.com/apache/kudu/blob/master/java/pom.xml#L260>
> > calling
> > > >>> that out.  I haven't been able to pinpoint what is different on the
> > > build
> > > >>> slaves yet.  The version of checkstyle appears to match.
> > > >>>
> > > >>
> > > >> hrm, that's odd... are we pinning the maven plugin version? I doubt
> > JVM
> > > >> version makes a difference, but perhaps...
> > > >>
> > > >> When you run locally using the same invocation as
> > > >> build-support/jenkins/build-and-test.sh it does the same?
> > > >>
> > > >>
> > > >>>
> > > >>> - Dan
> > > >>>
> > > >>> On Wed, Nov 23, 2016 at 10:45 AM, William Berkeley <
> > > >>> wdberke...@cloudera.com> wrote:
> > > >>>
> > > >>>> Sorry. Will get to it asap. Away from home right now. Should be
> able
> > > to
> > > >>>> disable on generated source.
> > > >>>>
> > > >>>> On Wed, Nov 23, 2016 at 12:59 Todd Lipcon <t...@cloudera.com>
> > wrote:
> > > >>>>
> > > >>>> > Ping?
> > > >>>> >
> > > >>>> > On Sun, Nov 20, 2016 at 7:53 PM, Todd Lipcon <t...@cloudera.com
> >
> > > >>>> wrote:
> > > >>>> >
> > > >>>> > Hey folks,
> > > >>>> >
> > > >>>> > It seems after we enabled checkstyle on java builds, there's
> now a
> > > >>>> ton of
> > > >>>> > spew in the log for precommit builds, eg:
> > > >>>> > http://104.196.14.100/job/kudu-gerrit/4670/BUILD_TYPE=TSAN/c
> > > >>>> onsoleText
> > > >>>> >
> > > >>>> > the build log is now 12MB, about 11MB of which appears to be
> > 52,045
> > > >>>> > checkstyle warnings. Of those, 51962 appear to be in generated
> > code
> > > :)
> > > >>>> >
> > > >>>> > So, a few questions:
> > > >>>> > 1) can we get the checkstyle output to go to a separate file
> > instead
> > > >>>> of
> > > >>>> > the console?
> > > >>>> > 2) can we disable checkstyle on generated sources?
> > > >>>> >
> > > >>>> > -Todd
> > > >>>> > --
> > > >>>> > Todd Lipcon
> > > >>>> > Software Engineer, Cloudera
> > > >>>> >
> > > >>>> >
> > > >>>> >
> > > >>>> >
> > > >>>> > --
> > > >>>> > Todd Lipcon
> > > >>>> > Software Engineer, Cloudera
> > > >>>> >
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Todd Lipcon
> > > >> Software Engineer, Cloudera
> > > >>
> > > >
> > > >
> > >
> >
> >
>

Re: checkstyle on console during precommit builds

[Dan Burkert]

I'm looking into this a bit.  It does get disabled on generated sources
when I run it locally, and there is a line in the pom.xml
 calling that
out.  I haven't been able to pinpoint what is different on the build slaves
yet.  The version of checkstyle appears to match.

- Dan

On Wed, Nov 23, 2016 at 10:45 AM, William Berkeley 
wrote:

> Sorry. Will get to it asap. Away from home right now. Should be able to
> disable on generated source.
>
> On Wed, Nov 23, 2016 at 12:59 Todd Lipcon  wrote:
>
> > Ping?
> >
> > On Sun, Nov 20, 2016 at 7:53 PM, Todd Lipcon  wrote:
> >
> > Hey folks,
> >
> > It seems after we enabled checkstyle on java builds, there's now a ton of
> > spew in the log for precommit builds, eg:
> > http://104.196.14.100/job/kudu-gerrit/4670/BUILD_TYPE=TSAN/consoleText
> >
> > the build log is now 12MB, about 11MB of which appears to be 52,045
> > checkstyle warnings. Of those, 51962 appear to be in generated code :)
> >
> > So, a few questions:
> > 1) can we get the checkstyle output to go to a separate file instead of
> > the console?
> > 2) can we disable checkstyle on generated sources?
> >
> > -Todd
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>

Re: [VOTE] Apache Kudu 1.1.0 RC1

[Dan Burkert]

+1

- Dan

On Thu, Nov 17, 2016 at 9:20 PM, Todd Lipcon <t...@cloudera.com> wrote:

> OK, given it's not a regression does anyone strongly oppose moving forward
> with 1.1.0 and try to squeeze 1.1.1 in in a couple weeks? In general my
> philosophy is that each release should be at least as stable as previous
> ones, but if we get into the habit of blocking releases because there's one
> more bug found that's not in the release, then it will be hard to keep our
> schedules.
>
> If no one -1s before tomorrow morning I'll close the vote and post the
> release probably by mid-afternoon tomorrow PST.
>
> -Todd
>
> On Thu, Nov 17, 2016 at 8:26 PM, Jordan Birdsell <
> jordantbirds...@gmail.com>
> wrote:
>
> > Agreed with proceeding and doing a 1.1.1 release.
> >
> > On Thu, Nov 17, 2016, 5:11 PM Dan Burkert <danburk...@apache.org> wrote:
> >
> > > Not a regression, it's been there since the add/drop range partitions
> > > feature landed.
> > >
> > > - Dan
> > >
> > > On Thu, Nov 17, 2016 at 4:58 PM, Todd Lipcon <t...@cloudera.com>
> wrote:
> > >
> > > > Sorry for radio silence on the vote thread so far -- been traveling
> > this
> > > > week.
> > > >
> > > > Is the range partition bug a regression or also present in 1.0? If
> it's
> > > not
> > > > a regression i think it's reasonable to continue with the release and
> > try
> > > > to fix in a 1.1.1 along with a couple other late discovered bugs.
> > > >
> > > > Todd
> > > >
> > > > On Nov 17, 2016 6:50 PM, "Dan Burkert" <danburk...@apache.org>
> wrote:
> > > >
> > > >  Dimitris Tsirogiannis just uncovered a bug that allows unintended
> > range
> > > > partitions to be dropped: KUDU-1750
> > > > <https://issues.apache.org/jira/browse/KUDU-1750>.  The bug allows a
> > > range
> > > > partition to be dropped if either the lower or upper bound range
> > > partition
> > > > key matches, instead of requiring both to match (it boils down to an
> ||
> > > > where an && ought to be).  I'm not sure whether this is high priority
> > > > enough to sink the vote, but we should consider it.  This could be
> > > > construed as 'unintended data loss if the API is not used carefully'.
> > > Fix
> > > > is in review.
> > > >
> > > > - Dan
> > > >
> > > > On Thu, Nov 17, 2016 at 11:58 AM, Brock Noland <br...@phdata.io>
> > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Built on 7.2 and ran tests
> > > > >
> > > > > On Monday, November 14, 2016, Todd Lipcon <t...@apache.org> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > The Apache Kudu team is happy to announce the first release
> > candidate
> > > > for
> > > > > > Apache Kudu 1.1.0.
> > > > > >
> > > > > > Apache Kudu 1.1 is a minor release which adds various new
> features,
> > > > > > improvements, bug fixes, and optimizations on top of Kudu 1.0.
> > Please
> > > > see
> > > > > > the release notes for details.
> > > > > >
> > > > > > The is a source-only release. The artifacts were staged here:
> > > > > > https://dist.apache.org/repos/dist/dev/kudu/1.1.0-RC1/
> > > > > >
> > > > > > Java convenience binaries in the form of a Maven repository are
> > > staged
> > > > > > here:
> > > > > >
> > > https://repository.apache.org/content/repositories/orgapachekudu-1005/
> > > > > >
> > > > > > It was built from this tag:
> > > > > > https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > > > > > 3f64b68724ded320a31b69467c5a5f7744b07a50
> > > > > >
> > > > > > The release notes can be found here (some links from this
> document
> > > will
> > > > > > only work when this version is released):
> > > > > > https://github.com/apache/kudu/blob/branch-1.1.x/docs/
> > > > release_notes.adoc
> > > > > >
> > > > > > KEYS file:
> > > > > > http://www.apache.org/dist/kudu/KEYS
> > > > > >
> > > > > > I'd suggest going through the README, building Kudu, and running
> > the
> > > > > > unit tests. Testing out the Maven repo would also be appreciated.
> > > > > >
> > > > > > While voting, please note that there are some known test failures
> > on
> > > > RHEL
> > > > > > 6:
> > > > > > - sasl_rpc-test fails in ASAN builds due to a memory leak in
> > > libkrb5.so
> > > > > > - external_mini_cluster-test fails in RELEASE builds due to a bug
> > in
> > > > MIT
> > > > > > Kerberos 1.10 (the workaround is in progress)
> > > > > > These are test-only issues related to in-progress security work
> > which
> > > > is
> > > > > > not yet supported in Kudu 1.1. Please disregard such failures
> when
> > > > > voting.
> > > > > >
> > > > > > Please try the release and vote; the vote will end in 72 hours
> (on
> > > > > > 11/17/2016 at 4:30pm PST).
> > > > > >
> > > > > > Thanks,
> > > > > > Todd
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: [VOTE] Apache Kudu 1.1.0 RC1

[Dan Burkert]

Not a regression, it's been there since the add/drop range partitions
feature landed.

- Dan

On Thu, Nov 17, 2016 at 4:58 PM, Todd Lipcon <t...@cloudera.com> wrote:

> Sorry for radio silence on the vote thread so far -- been traveling this
> week.
>
> Is the range partition bug a regression or also present in 1.0? If it's not
> a regression i think it's reasonable to continue with the release and try
> to fix in a 1.1.1 along with a couple other late discovered bugs.
>
> Todd
>
> On Nov 17, 2016 6:50 PM, "Dan Burkert" <danburk...@apache.org> wrote:
>
>  Dimitris Tsirogiannis just uncovered a bug that allows unintended range
> partitions to be dropped: KUDU-1750
> <https://issues.apache.org/jira/browse/KUDU-1750>.  The bug allows a range
> partition to be dropped if either the lower or upper bound range partition
> key matches, instead of requiring both to match (it boils down to an ||
> where an && ought to be).  I'm not sure whether this is high priority
> enough to sink the vote, but we should consider it.  This could be
> construed as 'unintended data loss if the API is not used carefully'.  Fix
> is in review.
>
> - Dan
>
> On Thu, Nov 17, 2016 at 11:58 AM, Brock Noland <br...@phdata.io> wrote:
>
> > +1 (non-binding)
> >
> > Built on 7.2 and ran tests
> >
> > On Monday, November 14, 2016, Todd Lipcon <t...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > The Apache Kudu team is happy to announce the first release candidate
> for
> > > Apache Kudu 1.1.0.
> > >
> > > Apache Kudu 1.1 is a minor release which adds various new features,
> > > improvements, bug fixes, and optimizations on top of Kudu 1.0. Please
> see
> > > the release notes for details.
> > >
> > > The is a source-only release. The artifacts were staged here:
> > > https://dist.apache.org/repos/dist/dev/kudu/1.1.0-RC1/
> > >
> > > Java convenience binaries in the form of a Maven repository are staged
> > > here:
> > > https://repository.apache.org/content/repositories/orgapachekudu-1005/
> > >
> > > It was built from this tag:
> > > https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > > 3f64b68724ded320a31b69467c5a5f7744b07a50
> > >
> > > The release notes can be found here (some links from this document will
> > > only work when this version is released):
> > > https://github.com/apache/kudu/blob/branch-1.1.x/docs/
> release_notes.adoc
> > >
> > > KEYS file:
> > > http://www.apache.org/dist/kudu/KEYS
> > >
> > > I'd suggest going through the README, building Kudu, and running the
> > > unit tests. Testing out the Maven repo would also be appreciated.
> > >
> > > While voting, please note that there are some known test failures on
> RHEL
> > > 6:
> > > - sasl_rpc-test fails in ASAN builds due to a memory leak in libkrb5.so
> > > - external_mini_cluster-test fails in RELEASE builds due to a bug in
> MIT
> > > Kerberos 1.10 (the workaround is in progress)
> > > These are test-only issues related to in-progress security work which
> is
> > > not yet supported in Kudu 1.1. Please disregard such failures when
> > voting.
> > >
> > > Please try the release and vote; the vote will end in 72 hours (on
> > > 11/17/2016 at 4:30pm PST).
> > >
> > > Thanks,
> > > Todd
> > >
> >
>

Re: [VOTE] Apache Kudu 1.1.0 RC1

[Dan Burkert]

 Dimitris Tsirogiannis just uncovered a bug that allows unintended range
partitions to be dropped: KUDU-1750
.  The bug allows a range
partition to be dropped if either the lower or upper bound range partition
key matches, instead of requiring both to match (it boils down to an ||
where an && ought to be).  I'm not sure whether this is high priority
enough to sink the vote, but we should consider it.  This could be
construed as 'unintended data loss if the API is not used carefully'.  Fix
is in review.

- Dan

On Thu, Nov 17, 2016 at 11:58 AM, Brock Noland  wrote:

> +1 (non-binding)
>
> Built on 7.2 and ran tests
>
> On Monday, November 14, 2016, Todd Lipcon  wrote:
>
> > Hi,
> >
> > The Apache Kudu team is happy to announce the first release candidate for
> > Apache Kudu 1.1.0.
> >
> > Apache Kudu 1.1 is a minor release which adds various new features,
> > improvements, bug fixes, and optimizations on top of Kudu 1.0. Please see
> > the release notes for details.
> >
> > The is a source-only release. The artifacts were staged here:
> > https://dist.apache.org/repos/dist/dev/kudu/1.1.0-RC1/
> >
> > Java convenience binaries in the form of a Maven repository are staged
> > here:
> > https://repository.apache.org/content/repositories/orgapachekudu-1005/
> >
> > It was built from this tag:
> > https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > 3f64b68724ded320a31b69467c5a5f7744b07a50
> >
> > The release notes can be found here (some links from this document will
> > only work when this version is released):
> > https://github.com/apache/kudu/blob/branch-1.1.x/docs/release_notes.adoc
> >
> > KEYS file:
> > http://www.apache.org/dist/kudu/KEYS
> >
> > I'd suggest going through the README, building Kudu, and running the
> > unit tests. Testing out the Maven repo would also be appreciated.
> >
> > While voting, please note that there are some known test failures on RHEL
> > 6:
> > - sasl_rpc-test fails in ASAN builds due to a memory leak in libkrb5.so
> > - external_mini_cluster-test fails in RELEASE builds due to a bug in MIT
> > Kerberos 1.10 (the workaround is in progress)
> > These are test-only issues related to in-progress security work which is
> > not yet supported in Kudu 1.1. Please disregard such failures when
> voting.
> >
> > Please try the release and vote; the vote will end in 72 hours (on
> > 11/17/2016 at 4:30pm PST).
> >
> > Thanks,
> > Todd
> >
>

Re: [VOTE] Apache Kudu 1.1.0 RC1

[Dan Burkert]

+1

Built and tested on OS X 10.10 in debug mode, and checked hashes and
signature.

A few of the tests failed, but these are expected for this release (and
have already been fixed on master):

* client_samples-test
* external_mini_cluster-test
* sasl_rpc-test
* pstack_watcher-test

sasl_rpc-test hangs, so I ran the tests using 'ctest -E sasl_rpc-test' to
work around it.

- Dan

On Mon, Nov 14, 2016 at 4:26 PM, Todd Lipcon  wrote:

> Hi,
>
> The Apache Kudu team is happy to announce the first release candidate for
> Apache Kudu 1.1.0.
>
> Apache Kudu 1.1 is a minor release which adds various new features,
> improvements, bug fixes, and optimizations on top of Kudu 1.0. Please see
> the release notes for details.
>
> The is a source-only release. The artifacts were staged here:
> https://dist.apache.org/repos/dist/dev/kudu/1.1.0-RC1/
>
> Java convenience binaries in the form of a Maven repository are staged
> here:
> https://repository.apache.org/content/repositories/orgapachekudu-1005/
>
> It was built from this tag:
> https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> 3f64b68724ded320a31b69467c5a5f7744b07a50
>
> The release notes can be found here (some links from this document will
> only work when this version is released):
> https://github.com/apache/kudu/blob/branch-1.1.x/docs/release_notes.adoc
>
> KEYS file:
> http://www.apache.org/dist/kudu/KEYS
>
> I'd suggest going through the README, building Kudu, and running the
> unit tests. Testing out the Maven repo would also be appreciated.
>
> While voting, please note that there are some known test failures on RHEL
> 6:
> - sasl_rpc-test fails in ASAN builds due to a memory leak in libkrb5.so
> - external_mini_cluster-test fails in RELEASE builds due to a bug in MIT
> Kerberos 1.10 (the workaround is in progress)
> These are test-only issues related to in-progress security work which is
> not yet supported in Kudu 1.1. Please disregard such failures when voting.
>
> Please try the release and vote; the vote will end in 72 hours (on
> 11/17/2016 at 4:30pm PST).
>
> Thanks,
> Todd
>

Re: 1.1.0 rc coming today

[Dan Burkert]

Sounds good.

On Mon, Nov 14, 2016 at 3:09 PM, Jordan Birdsell 
wrote:

> Sounds reasonable to me
>
> On Mon, Nov 14, 2016, 5:06 PM Todd Lipcon  wrote:
>
> > Hey folks,
> >
> > I'm wrapping up the creation of 1.1.0 RC0 in the next few minutes. I
> > noticed during my review of the branch that there are still a few items
> > missing on the release notes (eg the wire compatibility section just says
> > 'XXX').
> >
> > Rather than delay the vote to finish these up, I'd like to suggest that
> we
> > build an RC0 as is (with the work-in-progress rel notes) and people can
> > start their testing/voting/etc. Then we can finish up the rel notes and
> any
> > other doc changes in parallel and do an abbreviated "re-vote" on an RC1
> > incorporating these changes. We can easily verify that the two RCs are
> > identical modulo the docs/ directory so we shouldn't need to re-build or
> > re-test for this RC1.
> >
> > I propose we keep the normal 72-hour total voting period across both RCs,
> > but that everyone be required to "propagate" their vote to RC1 when it is
> > available. For example, I'll send out RC0 for vote this afternoon, then
> > send out RC1 within 36 hours. We'd still close the vote 72 hours after
> RC0
> > was made available (Thurs afternoon). Does that sound acceptable?
> >
> > -Todd
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>

Re: Patches that need release notes for Kudu 1.1.0

[Dan Burkert]

'pruning fails with IS NOT NULL' and 'DNS resolves killing tservers' were
both fixed in 1.0.1.  I'll write up notes for IN LIST.

- Dan

On Tue, Nov 8, 2016 at 9:58 AM, Jean-Daniel Cryans 
wrote:

> Hey devs,
>
> I'm doing a pass over the stuff committed since 1.0.1
>  and here's a list
> of patches that *may* need release notes (feel free to disagree):
>
> - Todd's WAL retention change
> https://github.com/apache/kudu/commit/127438af30356f1afedb86
> 2166c907ff754d1c55
> - Todd's fix to UPSERTS with only PKs specified:
> https://github.com/apache/kudu/commit/9911c489c45b3a261ee50a
> d1f837387b4953421b
> - Todd's leader pre-elections
> https://github.com/apache/kudu/commit/45c1512fcc42772173490b
> e4e7aefcf45129fed2
> - Todd's fsync'ing of tombstoned tablets fix
> https://github.com/apache/kudu/commit/141de3377e6bdd95b8a455
> d262c00066f23bc021
> - Todd's crash fix when deleting tablets
> https://github.com/apache/kudu/commit/cf976a40e05389dd2f3de0
> 98835c09c50a7526b5
>
> - Dan's Partition pruning fails with IS NOT NULL
> https://github.com/apache/kudu/commit/ce17a9c4eb34dcfe63c8d4
> 321d38d18a0cb8c5c2
> - Dan and Sameer's IN LIST predicate
> https://github.com/apache/kudu/commit/9b50bd585f62834f9d6578
> 7344e545bd6846404c
> - Dan's bug fix for DNS resolves killing tservers
> https://github.com/apache/kudu/commit/fc5e6a3e7921b9a4758481
> 17063e2d7db56f8ee4
>
> - Dinesh's leader step down tool:
> https://github.com/apache/kudu/commit/833abea787d3ac4bde3e52
> 92c82b3408aad9ba92
>
> - Alexey's deprecation of CountBufferedOperations
> https://github.com/apache/kudu/commit/8613415b2eafd5457a68a9
> 1c1c0e22b96e8c6f1d
>
> - Maxim's dumping of version info when starting servers
> https://github.com/apache/kudu/commit/ba36a0b7d122545ae99fb2
> a2e7e9c7e0035669b1
>
> - Adar's preallocation improvement (since we're already release noting it
> as a problem)
> https://github.com/apache/kudu/commit/abea8c69f786cc884c50be
> 810a7eee7200caec03
>
> I'm about to post a patch for the Java client changes so I won't list
> those. There's also all the security-related stuff that I'm not sure what
> could require release notes.
>
> J-D
>

Heads Up: new (test-only) system dependencies

[Dan Burkert]

Hi all,

As you can see in the recent commit history, work has gotten started on
adding security features to Kudu.  As part of this, we are now relying on
MIT Kerberos (krb5), OpenSSL, and the GSSAPI SASL plugin.  Since these are
security-sensitive dependencies, we are relying on the system to provide
them (meaning they need to be pre-installed).

Currently the dependencies are only needed for tests, but eventually they
will be required for running Kudu, at least in secure mode.  At that point
installation.adoc and other user documentation will be updated accordingly.

OS X install instructions (Homebrew):
--
brew install homebrew/dupes/krb5

Ubuntu install instructions:
---
apt-get install openssl
apt-get install krb5-admin-server krb5-user krb5-kdc
apt-get install libsasl2-modules-gssapi-mit

RHEL install instructions:
-
yum install openssl
yum install krb5-workstation krb5-server
yum install cyrus-sasl-gssapi

- Dan

Re: Plan for upcoming releases

[Dan Burkert]

Sounds good to me, thanks for volunteering.

- Dan

On Mon, Oct 17, 2016 at 5:27 PM, Jordan Birdsell 
wrote:

> Seems reasonable to me, have a few things for Python left I'd like to get
> in for 1.1: range partition implementation and alter table capabilities,
> that should be enough time.
>
> Thanks Todd,
> Jordan
>
> On Mon, Oct 17, 2016, 8:23 PM Todd Lipcon  wrote:
>
> > Hey folks,
> >
> > I'd like to propose a release plan for the next few months.
> >
> > My suggestion is that we aim for Kudu 1.1 some time around mid-November
> --
> > hopefully with enough time before Thanksgiving that no one here in the US
> > needs to worry about an impending release over the holiday. So, I think
> > that would be some time the week of 11/14.
> >
> > That gives us a month from now to make some progress on in-flight
> features
> > and fixes. My preference is that this should be a time-based rather than
> > scope-based release, but I expect that some of the items that people are
> > working on should be ready by then. Speaking for myself, the pre-election
> > patch series can improve stability significantly in some cases and it
> will
> > be great to get that out to our users. I'm sure others have improvements
> > pending that will be ready by then too.
> >
> > Following 1.1, I think we should plan for a 1.2 release early in January
> > (6-8 weeks after 1.1). My hope is that we can get a good chunk of the
> > security features that folks have started looking at done by then, as
> well
> > as various other work that people have been circulating design docs for.
> >
> > I'll also volunteer to RM these two releases.
> >
> > Thanks
> > -Todd
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>

[RESULT] [VOTE] Apache Kudu 1.0.1 RC1

[Dan Burkert]

The vote is now closed. It passes with:

3 binding +1s: Alexey, Dan, JD
1 binding +0: David
2 non-binding +1s: Brock, Dinesh
0 -1s

I will be uploading the sources and jars soon. Thanks to everyone that
participated!

- Dan


On Mon, Oct 10, 2016 at 4:52 PM, Jean-Daniel Cryans <jdcry...@apache.org>
wrote:

> +1
>
> I downloaded the tarball, compiled  everything and ran the tests on RHEL6
> (100% passed).
>
> I also deployed custom packages on a 4 nodes EC2 cluster, modified YCSB to
> use the 1.0.1 jar, inserted 10M rows, ran a combination of
> inserts/reads/updates over 100K rows, looks good.
>
> J-D
>
> On Fri, Oct 7, 2016 at 2:00 PM, Dan Burkert <danburk...@apache.org> wrote:
>
> > Hi,
> >
> > We're happy to announce the first release candidate for Apache Kudu
> 1.0.1.
> >
> > This release includes bug fixes and documentation updates since the 1.0.0
> > release.
> >
> > The is a source-only release. The artifacts were staged here:
> > https://dist.apache.org/repos/dist/dev/kudu/1.0.1-RC1/
> >
> > It was built from this tag:
> > https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > e60b610253f4303b24d41575f7bafbc5d69edddb
> >
> > The release notes can be found here (the release notes on
> kudu.apache.org
> > will be updated with the release):
> > https://github.com/apache/kudu/blob/branch-1.0.x/docs/release_notes.adoc
> >
> > KEYS file:
> > https://www.apache.org/dist/kudu/KEYS
> >
> > I'd suggest going through the README, building Kudu, and running the
> > unit tests.
> >
> > Please try the release and vote; vote will be open for at least 72 hours.
> >
> > Thanks,
> >
> > - Dan
> >
>

Re: [VOTE] Apache Kudu 1.0.1 RC1

[Dan Burkert]

+1

* Compiled and tested release mode on OS X 10.10, with no unexpected
failures.
* Compiled and tested Debug/ASAN/TSAN/Release on Centos 6, no failures.
* Checked signatures and licenses (RAT)

- Dan

On Mon, Oct 10, 2016 at 12:27 PM, Alexey Serbin <aser...@cloudera.com>
wrote:

> +1
>
> I compiled the project from source using the 1.0.1-RC1 tag (release and
> debug modes) and ran the test suite using 'ctest -j4' at CentOS 6.6 (kernel
> 2.6.32-504.30.3.el6.x86_64).  All tests passed.
>
> Also, I compiled the project from source using the 1.0.1-RC1 tag (release
> and debug modes) and ran the tests using 'ctest -j2' at MacOS X 10.11.5.
> All tests passed except for pstack_watcher-test, which is a known issue for
> Kudu on MacOS X.
>
> Besides, I run 'kudu test loadgen' tool against the newly built binaries
> multiple times, inserting several million of rows and it passed.
>
>
> Best regards,
>
> Alexey
>
>
> On Fri, Oct 7, 2016 at 2:00 PM, Dan Burkert <danburk...@apache.org> wrote:
>
> > Hi,
> >
> > We're happy to announce the first release candidate for Apache Kudu
> 1.0.1.
> >
> > This release includes bug fixes and documentation updates since the 1.0.0
> > release.
> >
> > The is a source-only release. The artifacts were staged here:
> > https://dist.apache.org/repos/dist/dev/kudu/1.0.1-RC1/
> >
> > It was built from this tag:
> > https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=
> > e60b610253f4303b24d41575f7bafbc5d69edddb
> >
> > The release notes can be found here (the release notes on
> kudu.apache.org
> > will be updated with the release):
> > https://github.com/apache/kudu/blob/branch-1.0.x/docs/release_notes.adoc
> >
> > KEYS file:
> > https://www.apache.org/dist/kudu/KEYS
> >
> > I'd suggest going through the README, building Kudu, and running the
> > unit tests.
> >
> > Please try the release and vote; vote will be open for at least 72 hours.
> >
> > Thanks,
> >
> > - Dan
> >
>

[VOTE] Apache Kudu 1.0.1 RC1

[Dan Burkert]

Hi,

We're happy to announce the first release candidate for Apache Kudu 1.0.1.

This release includes bug fixes and documentation updates since the 1.0.0
release.

The is a source-only release. The artifacts were staged here:
https://dist.apache.org/repos/dist/dev/kudu/1.0.1-RC1/

It was built from this tag:
https://git-wip-us.apache.org/repos/asf?p=kudu.git;a=commit;h=e60b610253f4303b24d41575f7bafbc5d69edddb

The release notes can be found here (the release notes on kudu.apache.org
will be updated with the release):
https://github.com/apache/kudu/blob/branch-1.0.x/docs/release_notes.adoc

KEYS file:
https://www.apache.org/dist/kudu/KEYS

I'd suggest going through the README, building Kudu, and running the
unit tests.

Please try the release and vote; vote will be open for at least 72 hours.

Thanks,

- Dan

Re: Fault tolerant scanner returning out-of-order partitions

[Dan Burkert]

The first three rows (including the out of order row) all fall in the same
range partition, so the issue is likely that the intra-tablet scan returned
out of order results (as opposed to the client scanning tablets out of
order). I'm under the same impression about SetFaultTolerant(), which is
why the test explicitly sets it.  How often is this happening? Back when
this test was committed a few months ago I ran it a few thousand times and
never saw anything like this.

On Mon, Oct 3, 2016 at 10:35 PM, Todd Lipcon  wrote:

> Hey Dan (+CC dev in case anyone else knows about this too)
>
> I'm debugging some flakiness in alter_table-randomized-test, and ti seems
> like it's failing because the verification scan is returning some out of
> order rows, despite using "SetFaultTolerant()". Granted, fault tolerance
> isn't publicly guaranteed to return rows in order, but I was under the
> impression that, with range partitioned tablets, it would always do so.
>
> The scan result I'm seeing has the following sequence within it:
>
> (int32 key=537424064, int32 c945=NULL, int32 c79=234639860, int32
> c990=NULL)
>  OUT OF ORDER ROW
> (int32 key=552025439, int32 c945=NULL, int32 c79=234639860, int32
> c990=NULL)
>  BACK TO NORMAL ORDER
> (int32 key=539314778, int32 c945=1708089980, int32 c79=-878787336, int32
> c990=829302644)
> (int32 key=541817227, int32 c945=2064952224, int32 c79=2064952224, int32
> c990=NULL)
> (int32 key=546056206, int32 c945=26527696, int32 c79=26527696, int32
> c990=26527696)
> (int32 key=601960253, int32 c945=NULL, int32 c79=1088757503, int32
> c990=NULL)
> (int32 key=677154987, int32 c945=823764490, int32 c79=823764490, int32
> c990=823764490)
>
> The prior alter was:
> I1004 05:17:48.192611 28113 alter_table-randomized-test.cc:481] Dropping
> range partition: [805306356, 872415219) resulting partitions: (134217726,
> 201326589], (268435452, 335544315], (335544315, 402653178], (402653178,
> 469762041], (536870904, 603979767], (671088630, 738197493], (738197493,
> 805306356], (939524082, 1006632945], (1006632945, 1073741808], (1275068397,
> 1342177260], (1342177260, 1409286123], (1409286123, 1476394986],
> (1610612712, 1677721575], (1879048164, 1946157027], (2013265890,
> 2080374753], (2080374753, 2147483616)
> I1004 05:17:48.193013 28113 alter_table-randomized-test.cc:406]
> Committing Alterations
>
> The whole log is available here:
> https://gist.githubusercontent.com/toddlipcon/
> 466976caf973f496885da9efc2f7246c/raw/f9baf418dad4ad07f33961b131c86e
> 84803815a8/alter_table-randomized-test.txt
>
> Any ideas what might be causing this out-of-order result? Is the test
> making some incorrect assumptions or might we have a bug?
>
> -Todd
>
>
>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: Kudu 1.0.1 patch release

[Dan Burkert]

Fixes have landed for KUDU-1651
<https://github.com/apache/kudu/commit/1f40913de6d3427847f3d435e1b84fb928a6f6a9>
and KUDU-1652
<https://github.com/apache/kudu/commit/ce17a9c4eb34dcfe63c8d4321d38d18a0cb8c5c2>,
so every outstanding known issue
that we want to address has a committed fix in master. Since many people are
otherwise busy with the Strata conference this week, I think we should
stick to
the original plan to cut an RC on October 7th, with a potential release
date the
following week. If any further bugs arise before the 7th we can consider
them
for inclusion.  Thanks all.

- Dan

On Mon, Sep 26, 2016 at 11:18 AM, Adar Dembo <a...@cloudera.com> wrote:
> +1 to a patch release.
>
> My only opinion on the nominated patches is that I'm not sure if we
> need to include the KUDU-1090 fix. IIRC it was very rare and generally
> only contributed to test flakiness. But, I'm fine with including it if
> others feel strongly.
>
> On Mon, Sep 26, 2016 at 10:55 AM, Todd Lipcon <t...@cloudera.com> wrote:
>> +1 for a bug fix release, especially to address the known crash bugs.
I'll
>> add one more to the running:
>>
>> https://gerrit.cloudera.org/#/c/4535/
>>
>> This fixes a crash when running on single-core systems. Again not too
>> common (even most VMs have two cores these days) but it can be a blocker
if
>> you need to run on a single-core.
>>
>> -Todd
>>
>> On Mon, Sep 26, 2016 at 10:49 AM, Dan Burkert <danburk...@apache.org>
wrote:
>>
>>> Hi all,
>>>
>>> Since the 1.0.0 release there have been a few issues found which may
>>> warrant a bug fix on the 1.0.x line.  I'd like to get the ball rolling
and
>>> figure out what we might want to include in a bug release, if we decide
to
>>> go ahead with a bug release fix.
>>>
>>> Potential inclusions:
>>>
>>> KUDU-1652 <https://issues.apache.org/jira/browse/KUDU-1652>: Partition
>>> pruning / scan optimization fails with IS NOT NULL predicate on PK
column
>>> KUDU-1651 <https://issues.apache.org/jira/browse/KUDU-1651>: tserver
crash
>>> when pushing predicate on dict encoded block with all null values
>>> 8fc75a5c65
>>> <https://github.com/apache/kudu/commit/8fc75a5c654e100871316e61878b14
>>> 1df4707d0e>:
>>> [java
>>> client] Fix an NPE in KuduException
>>> 9911c489
>>> <https://github.com/apache/kudu/commit/9911c489c45b3a261ee50ad1f83738
>>> 7b4953421b>
>>>  KUDU-1623 <https://issues.apache.org/jira/browse/KUDU-1623>: Properly
>>> handle UPSERTS that only include PK column
>>> b0b273e8
>>> <https://github.com/apache/kudu/commit/b0b273e8271752b6eb04ba163981aa
>>> d1c792e413>:
>>> [java client] make DateFormat safe to use
>>> 1eb2418
>>> <https://github.com/apache/kudu/commit/1eb24183a540f4e3bbbc8a399e440e
>>> cf905f6129>:
>>> consensus: properly truncate all state when aborting operations
>>> KUDU-1090 <https://issues.apache.org/jira/browse/KUDU-1090> 4b9d2f6
>>> <https://github.com/apache/kudu/commit/4b9d2f6976f45ea57e9a2c2648f31b
>>> 3a0941a569>:
>>> relax
>>> MemTracker uniqueness constraint
>>>
>>> It would also be nice to fix the Java client's client2tablet
>>> synchronization / memory leak issue, but I'm not sure of the proper set
of
>>> patches to backport.  JD/David, do you have any insight on that?
>>>
>>> Please reply with any other commits that you would like to include.
>>>
>>> Most of these issues are extremely rare, or easily worked around (or
both),
>>> but I think in aggregate they represent enough sharp edges that waiting
for
>>> a 1.1 release may be painful.
>>>
>>> Since a few of these issues don't have a patch committed or even in
review
>>> yet, an RC probably can't be cut before the end of next week (October
>>> 7th).  I volunteer to RM this one.  What does everyone think?
>>>
>>> - Dan
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera

Re: Kudu Build: Unsupported options error

[Dan Burkert]

Valencia,

If these are just issues with building libstdc++ in thirdparty, then the
effect on Kudu should be negligible.  libstdc++ is *only* used (linked-to)
when building with thread sanitizer (TSAN) support.  Right now we
unconditionally build thirdparty libraries for TSAN when compiling on Linux
(including libstdc++), but you could easily opt-out by changing the
conditionals here:
https://github.com/apache/kudu/blob/master/thirdparty/build-thirdparty.sh#L86.
It should also be noted that in the next few days Adar will be landing a
big reorganization to the thirdparty build that will make thirdparty
libraries build for TSAN when requested.  It's also going to switch out
libstdc++ for libc++ in thirdparty (still only used for TSAN).

- Dan

On Fri, Sep 23, 2016 at 2:47 AM, Valencia Serrao  wrote:

>
> Hi All,
>
> I am building Kudu on ppc64le. However, while building Kudu's thirdparty
> "libstdc++-v3", I have encountered following errors:
> Error1: clang-3.8: error: unsupported option '-print-multi-os-directory'
> Error2: clang-3.8: error: unknown argument: '-mlong-double-64'
>
> I couldn't find any documentation to find the equivalent options to be used
> on ppc64le. To get around the Error1, , I have removed the
> -print-multi-os-directory option and continued with the build.
>
> The build has surely progressed with this work-around, however, I need to
> know:
> 1. What is the impact of removing "-print-multi-os-directory" ? Will it
> significantly affect the functionality  for Impala ?
> 2. Can I take the same approach for the Error2 ? What will be the impact on
> functionality ?
> 3. If removing the above two options has a significant impact, then, could
> you share the documentation/equivalent options to be used for ppc64le ?
>
> Any pointers on this issue will be helpful.
>
> Regards,
> Valencia,
>

Re: [ANNOUNCE] Two new Kudu committer/PMC members

[Dan Burkert]

Congrats!

On Mon, Sep 12, 2016 at 4:27 PM, Jordan Birdsell 
wrote:

> Congrats
>
> On Mon, Sep 12, 2016 at 7:09 PM Brock Noland  wrote:
>
>> Congratulations!!
>>
>> On Mon, Sep 12, 2016 at 6:06 PM, David Alves 
>> wrote:
>>
>>> Congrats Alexey and Will!!
>>>
>>> On Mon, Sep 12, 2016 at 3:55 PM, Todd Lipcon  wrote:
>>>
>>> > Hi Kudu community,
>>> >
>>> > It's my great pleasure to announce that the PMC has voted to add both
>>> > Alexey Serbin and Will Berkeley as committers and PMC members.
>>> >
>>> > Alexey has been contributing for a few months, including developing
>>> some
>>> > pretty meaty (and tricky) additions. Two of note are the addition of
>>> > doxygen for our client APIs, as well as the implementation of
>>> > AUTO_FLUSH_BACKGROUND in C++. He has also been quite active in reviews
>>> > recently, having reviewed 40+ patches in the last couple months. He
>>> also
>>> > contributed by testing and voting on the recent 0.10 release.
>>> >
>>> > Will has been a great contributor as well, spending a lot of time in
>>> areas
>>> > that don't get as much love from others. In particular, he's made
>>> several
>>> > fixes to the web UIs, has greatly improved the Flume integration, and
>>> has
>>> > been burning down a lot of long-standing bugs recently. Will also
>>> spends a
>>> > lot of his time outside of Kudu working with users and always has good
>>> > input on what our user community will think of a feature. Like Alexey,
>>> Will
>>> > also participated in the 0.10 release process.
>>> >
>>> > Both of these community members have already been "acting the part"
>>> > through their contributions detailed above, and the PMC is excited to
>>> > continue working with them in their expanded roles.
>>> >
>>> > Please join me in congratulating them!
>>> >
>>> > -Todd
>>> >
>>>
>>
>>

Re: Deploying to the ASF maven repo

[Dan Burkert]

Both the staging and nightly repos look good to me, I was able to use them
from a thirdparty maven project.

  

  apache.snapshots
  Apache Snapshot Repository
  https://repository.apache.org/snapshots


  apache.kudu.staging
  Apache Kudu Staging Repository
  
https://repository.apache.org/content/repositories/orgapachekudu-1000/

  


- Dan

On Wed, Aug 24, 2016 at 3:36 PM, Todd Lipcon  wrote:

> Looks like Dan actually got this set up a couple months ago while we were
> in the incubator.
>
> I wrote a small patch http://gerrit.cloudera.org:8080/4118 to change our
> parent pom to the ASF parent. With this I was able to deploy a
> 1.0.0-SNAPSHOT to the ASF snapshots repo:
> https://repository.apache.org/content/repositories/
> snapshots/org/apache/kudu/
>
> I also cherry-picked that patch on top of the 0.10.0 release tag and was
> able to stage the 0.10.0 repository here:
> https://repository.apache.org/content/repositories/orgapachekudu-1000/
>
> Let's run a lazy consensus vote on the above staging repository - if
> someone else can try building against it and decides it looks good, I'll
> figure out how to "promote" this.
>
> Next steps would be to update our release documentation to include mvn
> deploys, and to set up a Jenkins job that auto-deploys snapshot builds on a
> nightly (or per-commit) basis.
>
> -Todd
>
>
> On Tue, Aug 23, 2016 at 1:11 PM, Mike Percy  wrote:
>
> > Woops, good catch.
> >
> > Mike
> >
> >
> > On Tue, Aug 23, 2016 at 12:01 PM, Todd Lipcon  wrote:
> >
> > > Hey folks
> > >
> > > Currently our pom is set up to deploy to Cloudera repos, but given that
> > we
> > > are an ASF project, we should be deploying to ASF maven.
> > >
> > > I filed https://issues.apache.org/jira/browse/INFRA-12479 to get us
> set
> > up
> > > in the ASF Nexus. Expect a gerrit review some time soon to change over
> > our
> > > pom, and then I'll figure out how to publish our most recent
> release(s).
> > >
> > > -Todd
> > >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>