Re: [ANNOUNCE] Apache Kudu 1.0.0 release

[Benjamin Kim]

Todd,

Thanks. I’ll look into those.

Cheers,
Ben


> On Sep 20, 2016, at 12:11 AM, Todd Lipcon  wrote:
> 
> The Apache Kudu team is happy to announce the release of Kudu 1.0.0!
> 
> Kudu is an open source storage engine for structured data which supports 
> low-latency random access together with efficient analytical access patterns. 
> It is designed within the context of the Apache Hadoop ecosystem and supports 
> many integrations with other data analytics projects both inside and outside 
> of the Apache Software Foundation.
> 
> This latest version adds several new features, including:
> 
> - Removal of multiversion concurrency control (MVCC) history is now 
> supported. This allows Kudu to reclaim disk space, where previously Kudu 
> would keep a full history of all changes made to a given table since the 
> beginning of time.
> 
> - Most of Kudu’s command line tools have been consolidated under a new 
> top-level "kudu" tool. This reduces the number of large binaries distributed 
> with Kudu and also includes much-improved help output.
> 
> - Administrative tools including "kudu cluster ksck" now support running 
> against multi-master Kudu clusters.
> 
> - The C++ client API now supports writing data in AUTO_FLUSH_BACKGROUND mode. 
> This can provide higher throughput for ingest workloads.
> 
> This release also includes many bug fixes, optimizations, and other 
> improvements, detailed in the release notes available at:
> http://kudu.apache.org/releases/1.0.0/docs/release_notes.html 
> 
> 
> Download the source release here:
> http://kudu.apache.org/releases/1.0.0/ 
> 
> 
> Convenience binary artifacts for the Java client and various Java 
> integrations (eg Spark, Flume) are also now available via the ASF Maven 
> repository.
> 
> Enjoy the new release!
> 
> - The Apache Kudu team

Re: [ANNOUNCE] Apache Kudu 1.0.0 release

[Todd Lipcon]

-announce


On Tue, Sep 20, 2016 at 11:34 AM, Benjamin Kim  wrote:

> This is awesome!!! Great!!!
>
> Do you know if any improvements were also made to the Spark plugin jar?
>

Looks like a few changes based on the git log:
https://gist.github.com/4fa3ccc3b9be787227fed89c1bd42837

as well as a number of changes to the Java client (which gets pulled into
the Spark jar):
https://gist.github.com/e2a8ca78e51773fabb70aae34207199f


In particular, I think the partition pruning work in the Java client should
reduce the number of Spark partitions if you have predicates on your data
frames. (though I haven't personally verified it)

-Todd



> On Sep 20, 2016, at 12:11 AM, Todd Lipcon  wrote:
>
> The Apache Kudu team is happy to announce the release of Kudu 1.0.0!
>
> Kudu is an open source storage engine for structured data which supports
> low-latency random access together with efficient analytical access
> patterns. It is designed within the context of the Apache Hadoop ecosystem
> and supports many integrations with other data analytics projects both
> inside and outside of the Apache Software Foundation.
>
> This latest version adds several new features, including:
>
> - Removal of multiversion concurrency control (MVCC) history is now
> supported. This allows Kudu to reclaim disk space, where previously Kudu
> would keep a full history of all changes made to a given table since the
> beginning of time.
>
> - Most of Kudu’s command line tools have been consolidated under a new
> top-level "kudu" tool. This reduces the number of large binaries
> distributed with Kudu and also includes much-improved help output.
>
> - Administrative tools including "kudu cluster ksck" now support running
> against multi-master Kudu clusters.
>
> - The C++ client API now supports writing data in AUTO_FLUSH_BACKGROUND
> mode. This can provide higher throughput for ingest workloads.
>
> This release also includes many bug fixes, optimizations, and other
> improvements, detailed in the release notes available at:
> http://kudu.apache.org/releases/1.0.0/docs/release_notes.html
>
> Download the source release here:
> http://kudu.apache.org/releases/1.0.0/
>
> Convenience binary artifacts for the Java client and various Java
> integrations (eg Spark, Flume) are also now available via the ASF Maven
> repository.
>
> Enjoy the new release!
>
> - The Apache Kudu team
>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Re: RFC: removing raft_consensus-test (mock-based)

[Adar Dembo]

Does this mean we should also remove gmock from thirdparty and go back to
plain gtest? Or are we still making use of other gmock features (not
related to mocking, if any)?

On Tuesday, September 20, 2016, Todd Lipcon  wrote:

> Great, glad you agree. It also feels good to delete 700 lines of code :)
>
> I included the removal in this patch: https://gerrit.cloudera.org/#/
> c/4476/
>
> Planning on moving it from "WIP" state to commitable hopefully today, but
> already worth a cursory look to make sure the direction of the refactor is
> agreeable.
>
> -Todd
>
> On Tue, Sep 20, 2016 at 10:59 AM, David Alves  > wrote:
>
> > +1, we've since broadened our coverage in other tests that those cases
> > are covered elsewhere. Plus I agree that with the queue taking more
> > responsibility mocking it becomes a hardship.
> >
> > -david
> >
> >
> > > I'm doing a bit of Consensus refactoring, and finding that the
> > raft_consensus-test mock-based test is a bit of a roadblock to
> refactoring.
> > We don't use gmock anywhere else, and so whenever I look at this code I
> > quickly develop a head-ache trying to understand what it's doing. Looking
> > through the test cases themselves, I believe all of the code paths they
> try
> > to cover are equally well covered by other test cases in
> > raft_consensus-itest. Rather than spend several hours trying to figure
> out
> > why the mock isn't behaving like the real code, I'd rather spend that
> time
> > on adding more test cases for the real code. Thoughts?
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

RFC: removing raft_consensus-test (mock-based)

[David Alves]

+1, we've since broadened our coverage in other tests that those cases
are covered elsewhere. Plus I agree that with the queue taking more
responsibility mocking it becomes a hardship.

-david


> I'm doing a bit of Consensus refactoring, and finding that the 
> raft_consensus-test mock-based test is a bit of a roadblock to refactoring. 
> We don't use gmock anywhere else, and so whenever I look at this code I 
> quickly develop a head-ache trying to understand what it's doing. Looking 
> through the test cases themselves, I believe all of the code paths they try 
> to cover are equally well covered by other test cases in 
> raft_consensus-itest. Rather than spend several hours trying to figure out 
> why the mock isn't behaving like the real code, I'd rather spend that time on 
> adding more test cases for the real code. Thoughts?

Re: [ANNOUNCE] Apache Kudu 1.0.0 release

[Aminul Islam]

Congrats
On Sep 20, 2016 9:35 PM, "Jacques Nadeau"  wrote:

> Congrats to everyone. This is a great accomplishment!
>
> On Tue, Sep 20, 2016 at 12:11 AM, Todd Lipcon  wrote:
>
>> The Apache Kudu team is happy to announce the release of Kudu 1.0.0!
>>
>> Kudu is an open source storage engine for structured data which supports
>> low-latency random access together with efficient analytical access
>> patterns. It is designed within the context of the Apache Hadoop ecosystem
>> and supports many integrations with other data analytics projects both
>> inside and outside of the Apache Software Foundation.
>>
>> This latest version adds several new features, including:
>>
>> - Removal of multiversion concurrency control (MVCC) history is now
>> supported. This allows Kudu to reclaim disk space, where previously Kudu
>> would keep a full history of all changes made to a given table since the
>> beginning of time.
>>
>> - Most of Kudu’s command line tools have been consolidated under a new
>> top-level "kudu" tool. This reduces the number of large binaries
>> distributed with Kudu and also includes much-improved help output.
>>
>> - Administrative tools including "kudu cluster ksck" now support running
>> against multi-master Kudu clusters.
>>
>> - The C++ client API now supports writing data in AUTO_FLUSH_BACKGROUND
>> mode. This can provide higher throughput for ingest workloads.
>>
>> This release also includes many bug fixes, optimizations, and other
>> improvements, detailed in the release notes available at:
>> http://kudu.apache.org/releases/1.0.0/docs/release_notes.html
>>
>> Download the source release here:
>> http://kudu.apache.org/releases/1.0.0/
>>
>> Convenience binary artifacts for the Java client and various Java
>> integrations (eg Spark, Flume) are also now available via the ASF Maven
>> repository.
>>
>> Enjoy the new release!
>>
>> - The Apache Kudu team
>>
>
>