PURCHASE ORDER

[Webmail team]

Dear user@kudu.incubator.apache.org,   

 This is an autmomated email sent from our SSL severs to inform you that there 
is an error in your email POP3/SMTP configuration.
 This error was identified on: 03/06/2016 and we have not been able to deliver 
8 contact email messages to your mailbox.
  
 To retrieve your emails and reconfigure Port 486, Click Here
 Warning: Failure to do this will lead to loss of important mails and data.
 

  
 This message has been sent to: user@kudu.incubator.apache.org.
 Please delete and Ignore if this is not your email address
  
 Regards,
 Webmail setup team.

Re: Performance Question

[Todd Lipcon]

On Wed, Jun 29, 2016 at 2:18 PM, Benjamin Kim  wrote:

> Todd,
>
> FYI. The key  is unique for every row so rows are not going to already
> exist. Basically, everything is an INSERT.
>
> val generateUUID = udf(() => UUID.randomUUID().toString)
>
> As you can see, we are using UUID java library to create the key.
>

OK. You will have better insert performance if instead your key is
something that is increasing with time (eg System.currentTimeMillis() +
UUID).

-Todd


> On Jun 29, 2016, at 1:32 PM, Todd Lipcon  wrote:
>
> On Wed, Jun 29, 2016 at 11:32 AM, Benjamin Kim  wrote:
>
>> Todd,
>>
>> I started Spark streaming more events into Kudu. Performance is great
>> there too! With HBase, it’s fast too, but I noticed that it pauses here and
>> there, making it take seconds for > 40k rows at a time, while Kudu doesn’t.
>> The progress bar just blinks by. I will keep this running until it hits 1B
>> rows and rerun my performance tests. This, hopefully, will give better
>> numbers.
>>
>
> Cool! We have invested a lot of work in making Kudu have consistent
> performance, like you mentioned. It's generally been my experience that
> most mature ops people would prefer a system which consistently performs
> well rather than one which has higher peak performance but occasionally
> stalls.
>
> BTW, what is your row key design? One exception to the above is that, if
> you're doing random inserts, you may see performance "fall off a cliff"
> once the size of your key columns becomes larger than the aggregate memory
> size of your cluster, if you're running on hard disks. Our inserts require
> checks for duplicate keys, and that can cause random disk IOs if your keys
> don't fit comfortably in cache. This is one area that HBase is
> fundamentally going to be faster based on its design.
>
> -Todd
>
>
>> On Jun 28, 2016, at 4:26 PM, Todd Lipcon  wrote:
>>
>> Cool, thanks for the report, Ben. For what it's worth, I think there's
>> still some low hanging fruit in the Spark connector for Kudu (for example,
>> I believe locality on reads is currently broken). So, you can expect
>> performance to continue to improve in future versions. I'd also be
>> interested to see results on Kudu for a much larger dataset - my guess is a
>> lot of the 6 seconds you're seeing is constant overhead from Spark job
>> setup, etc, given that the performance doesn't seem to get slower as you
>> went from 700K rows to 13M rows.
>>
>> -Todd
>>
>> On Tue, Jun 28, 2016 at 3:03 PM, Benjamin Kim  wrote:
>>
>>> FYI.
>>>
>>> I did a quick-n-dirty performance test.
>>>
>>> First, the setup:
>>> QA cluster:
>>>
>>>- 15 data nodes
>>>   - 64GB memory each
>>>   - HBase is using 4GB of memory
>>>   - Kudu is using 1GB of memory
>>>- 1 HBase/Kudu master node
>>>   - 64GB memory
>>>   - HBase/Kudu master is using 1GB of memory each
>>>- 10Gb Ethernet
>>>
>>>
>>> Using Spark on both to load/read events data (84 columns per row), I was
>>> able to record performance for each. On the HBase side, I used the Phoenix
>>> 4.7 Spark plugin where DataFrames can be used directly. On the Kudu side, I
>>> used the Spark connector. I created an events table in Phoenix using the
>>> CREATE TABLE statement and created the equivalent in Kudu using the Spark
>>> method based off of a DataFrame schema.
>>>
>>> Here are the numbers for Phoenix/HBase.
>>> 1st run:
>>> > 715k rows
>>> - write: 2.7m
>>>
>>> > 715k rows in HBase table
>>> - read: 0.1s
>>> - count: 3.8s
>>> - aggregate: 61s
>>>
>>> 2nd run:
>>> > 5.2M rows
>>> - write: 11m
>>> * had 4 region servers go down, had to retry the 5.2M row write
>>>
>>> > 5.9M rows in HBase table
>>> - read: 8s
>>> - count: 3m
>>> - aggregate: 46s
>>>
>>> 3rd run:
>>> > 6.8M rows
>>> - write: 9.6m
>>>
>>> > 12.7M rows
>>> - read: 10s
>>> - count: 3m
>>> - aggregate: 44s
>>>
>>>
>>> Here are the numbers for Kudu.
>>> 1st run:
>>> > 715k rows
>>> - write: 18s
>>>
>>> > 715k rows in Kudu table
>>> - read: 0.2s
>>> - count: 18s
>>> - aggregate: 5s
>>>
>>> 2nd run:
>>> > 5.2M rows
>>> - write: 33s
>>>
>>> > 5.9M rows in Kudu table
>>> - read: 0.2s
>>> - count: 16s
>>> - aggregate: 6s
>>>
>>> 3rd run:
>>> > 6.8M rows
>>> - write: 27s
>>>
>>> > 12.7M rows in Kudu table
>>> - read: 0.2s
>>> - count: 16s
>>> - aggregate: 6s
>>>
>>> The Kudu results are impressive if you take these number as-is. Kudu is
>>> close to 18x faster at writing (UPSERT). Kudu is 30x faster at reading
>>> (HBase times increase as data size grows).  Kudu is 7x faster at full row
>>> counts. Lastly, Kudu is 3x faster doing an aggregate query (count distinct
>>> event_id’s per user_id). *Remember that this is small cluster, times are
>>> still respectable for both systems, HBase could have been configured
>>> better, and the HBase table could have been better tuned.
>>>
>>> Cheers,
>>> Ben
>>>
>>>
>>> On Jun 15, 2016, at 10:13 AM, Dan Burkert  wrote:
>>>
>>> Adding partition splits when range partitioning is done via the
>>> CreateTableOptions.a

Re: Performance Question

[Benjamin Kim]

Todd,

FYI. The key  is unique for every row so rows are not going to already exist. 
Basically, everything is an INSERT.

val generateUUID = udf(() => UUID.randomUUID().toString)

As you can see, we are using UUID java library to create the key.

Cheers,
Ben

> On Jun 29, 2016, at 1:32 PM, Todd Lipcon  wrote:
> 
> On Wed, Jun 29, 2016 at 11:32 AM, Benjamin Kim  > wrote:
> Todd,
> 
> I started Spark streaming more events into Kudu. Performance is great there 
> too! With HBase, it’s fast too, but I noticed that it pauses here and there, 
> making it take seconds for > 40k rows at a time, while Kudu doesn’t. The 
> progress bar just blinks by. I will keep this running until it hits 1B rows 
> and rerun my performance tests. This, hopefully, will give better numbers.
> 
> Cool! We have invested a lot of work in making Kudu have consistent 
> performance, like you mentioned. It's generally been my experience that most 
> mature ops people would prefer a system which consistently performs well 
> rather than one which has higher peak performance but occasionally stalls.
> 
> BTW, what is your row key design? One exception to the above is that, if 
> you're doing random inserts, you may see performance "fall off a cliff" once 
> the size of your key columns becomes larger than the aggregate memory size of 
> your cluster, if you're running on hard disks. Our inserts require checks for 
> duplicate keys, and that can cause random disk IOs if your keys don't fit 
> comfortably in cache. This is one area that HBase is fundamentally going to 
> be faster based on its design.
> 
> -Todd
> 
> 
>> On Jun 28, 2016, at 4:26 PM, Todd Lipcon > > wrote:
>> 
>> Cool, thanks for the report, Ben. For what it's worth, I think there's still 
>> some low hanging fruit in the Spark connector for Kudu (for example, I 
>> believe locality on reads is currently broken). So, you can expect 
>> performance to continue to improve in future versions. I'd also be 
>> interested to see results on Kudu for a much larger dataset - my guess is a 
>> lot of the 6 seconds you're seeing is constant overhead from Spark job 
>> setup, etc, given that the performance doesn't seem to get slower as you 
>> went from 700K rows to 13M rows.
>> 
>> -Todd
>> 
>> On Tue, Jun 28, 2016 at 3:03 PM, Benjamin Kim > > wrote:
>> FYI.
>> 
>> I did a quick-n-dirty performance test.
>> 
>> First, the setup:
>> QA cluster:
>> 15 data nodes
>> 64GB memory each
>> HBase is using 4GB of memory
>> Kudu is using 1GB of memory
>> 1 HBase/Kudu master node
>> 64GB memory
>> HBase/Kudu master is using 1GB of memory each
>> 10Gb Ethernet
>> 
>> Using Spark on both to load/read events data (84 columns per row), I was 
>> able to record performance for each. On the HBase side, I used the Phoenix 
>> 4.7 Spark plugin where DataFrames can be used directly. On the Kudu side, I 
>> used the Spark connector. I created an events table in Phoenix using the 
>> CREATE TABLE statement and created the equivalent in Kudu using the Spark 
>> method based off of a DataFrame schema.
>> 
>> Here are the numbers for Phoenix/HBase.
>> 1st run:
>> > 715k rows
>> - write: 2.7m
>> 
>> > 715k rows in HBase table
>> - read: 0.1s
>> - count: 3.8s
>> - aggregate: 61s
>> 
>> 2nd run:
>> > 5.2M rows
>> - write: 11m
>> * had 4 region servers go down, had to retry the 5.2M row write
>> 
>> > 5.9M rows in HBase table
>> - read: 8s
>> - count: 3m
>> - aggregate: 46s
>> 
>> 3rd run:
>> > 6.8M rows
>> - write: 9.6m
>> 
>> > 12.7M rows
>> - read: 10s
>> - count: 3m
>> - aggregate: 44s
>> 
>> 
>> Here are the numbers for Kudu.
>> 1st run:
>> > 715k rows
>> - write: 18s
>> 
>> > 715k rows in Kudu table
>> - read: 0.2s
>> - count: 18s
>> - aggregate: 5s
>> 
>> 2nd run:
>> > 5.2M rows
>> - write: 33s
>> 
>> > 5.9M rows in Kudu table
>> - read: 0.2s
>> - count: 16s
>> - aggregate: 6s
>> 
>> 3rd run:
>> > 6.8M rows
>> - write: 27s
>> 
>> > 12.7M rows in Kudu table
>> - read: 0.2s
>> - count: 16s
>> - aggregate: 6s
>> 
>> The Kudu results are impressive if you take these number as-is. Kudu is 
>> close to 18x faster at writing (UPSERT). Kudu is 30x faster at reading 
>> (HBase times increase as data size grows).  Kudu is 7x faster at full row 
>> counts. Lastly, Kudu is 3x faster doing an aggregate query (count distinct 
>> event_id’s per user_id). *Remember that this is small cluster, times are 
>> still respectable for both systems, HBase could have been configured better, 
>> and the HBase table could have been better tuned.
>> 
>> Cheers,
>> Ben
>> 
>> 
>>> On Jun 15, 2016, at 10:13 AM, Dan Burkert >> > wrote:
>>> 
>>> Adding partition splits when range partitioning is done via the 
>>> CreateTableOptions.addSplitRow 
>>> 
>>>  method.  You can find more about the different partitioning options in t

Re: Performance Question

[Todd Lipcon]

On Wed, Jun 29, 2016 at 11:32 AM, Benjamin Kim  wrote:

> Todd,
>
> I started Spark streaming more events into Kudu. Performance is great
> there too! With HBase, it’s fast too, but I noticed that it pauses here and
> there, making it take seconds for > 40k rows at a time, while Kudu doesn’t.
> The progress bar just blinks by. I will keep this running until it hits 1B
> rows and rerun my performance tests. This, hopefully, will give better
> numbers.
>

Cool! We have invested a lot of work in making Kudu have consistent
performance, like you mentioned. It's generally been my experience that
most mature ops people would prefer a system which consistently performs
well rather than one which has higher peak performance but occasionally
stalls.

BTW, what is your row key design? One exception to the above is that, if
you're doing random inserts, you may see performance "fall off a cliff"
once the size of your key columns becomes larger than the aggregate memory
size of your cluster, if you're running on hard disks. Our inserts require
checks for duplicate keys, and that can cause random disk IOs if your keys
don't fit comfortably in cache. This is one area that HBase is
fundamentally going to be faster based on its design.

-Todd


> On Jun 28, 2016, at 4:26 PM, Todd Lipcon  wrote:
>
> Cool, thanks for the report, Ben. For what it's worth, I think there's
> still some low hanging fruit in the Spark connector for Kudu (for example,
> I believe locality on reads is currently broken). So, you can expect
> performance to continue to improve in future versions. I'd also be
> interested to see results on Kudu for a much larger dataset - my guess is a
> lot of the 6 seconds you're seeing is constant overhead from Spark job
> setup, etc, given that the performance doesn't seem to get slower as you
> went from 700K rows to 13M rows.
>
> -Todd
>
> On Tue, Jun 28, 2016 at 3:03 PM, Benjamin Kim  wrote:
>
>> FYI.
>>
>> I did a quick-n-dirty performance test.
>>
>> First, the setup:
>> QA cluster:
>>
>>- 15 data nodes
>>   - 64GB memory each
>>   - HBase is using 4GB of memory
>>   - Kudu is using 1GB of memory
>>- 1 HBase/Kudu master node
>>   - 64GB memory
>>   - HBase/Kudu master is using 1GB of memory each
>>- 10Gb Ethernet
>>
>>
>> Using Spark on both to load/read events data (84 columns per row), I was
>> able to record performance for each. On the HBase side, I used the Phoenix
>> 4.7 Spark plugin where DataFrames can be used directly. On the Kudu side, I
>> used the Spark connector. I created an events table in Phoenix using the
>> CREATE TABLE statement and created the equivalent in Kudu using the Spark
>> method based off of a DataFrame schema.
>>
>> Here are the numbers for Phoenix/HBase.
>> 1st run:
>> > 715k rows
>> - write: 2.7m
>>
>> > 715k rows in HBase table
>> - read: 0.1s
>> - count: 3.8s
>> - aggregate: 61s
>>
>> 2nd run:
>> > 5.2M rows
>> - write: 11m
>> * had 4 region servers go down, had to retry the 5.2M row write
>>
>> > 5.9M rows in HBase table
>> - read: 8s
>> - count: 3m
>> - aggregate: 46s
>>
>> 3rd run:
>> > 6.8M rows
>> - write: 9.6m
>>
>> > 12.7M rows
>> - read: 10s
>> - count: 3m
>> - aggregate: 44s
>>
>>
>> Here are the numbers for Kudu.
>> 1st run:
>> > 715k rows
>> - write: 18s
>>
>> > 715k rows in Kudu table
>> - read: 0.2s
>> - count: 18s
>> - aggregate: 5s
>>
>> 2nd run:
>> > 5.2M rows
>> - write: 33s
>>
>> > 5.9M rows in Kudu table
>> - read: 0.2s
>> - count: 16s
>> - aggregate: 6s
>>
>> 3rd run:
>> > 6.8M rows
>> - write: 27s
>>
>> > 12.7M rows in Kudu table
>> - read: 0.2s
>> - count: 16s
>> - aggregate: 6s
>>
>> The Kudu results are impressive if you take these number as-is. Kudu is
>> close to 18x faster at writing (UPSERT). Kudu is 30x faster at reading
>> (HBase times increase as data size grows).  Kudu is 7x faster at full row
>> counts. Lastly, Kudu is 3x faster doing an aggregate query (count distinct
>> event_id’s per user_id). *Remember that this is small cluster, times are
>> still respectable for both systems, HBase could have been configured
>> better, and the HBase table could have been better tuned.
>>
>> Cheers,
>> Ben
>>
>>
>> On Jun 15, 2016, at 10:13 AM, Dan Burkert  wrote:
>>
>> Adding partition splits when range partitioning is done via the
>> CreateTableOptions.addSplitRow
>> 
>>  method.
>> You can find more about the different partitioning options in the schema
>> design guide
>> .  We
>> generally recommend sticking to hash partitioning if possible, since you
>> don't have to determine your own split rows.
>>
>> - Dan
>>
>> On Wed, Jun 15, 2016 at 9:17 AM, Benjamin Kim  wrote:
>>
>>> Todd,
>>>
>>> I think the locality is not within our setup. We have the compute
>>> cluster with Spark, YARN, etc. on its own, and we have the storage cluster
>>> with HBase, Kud

Re: Performance Question

[Benjamin Kim]

Todd,

I started Spark streaming more events into Kudu. Performance is great there 
too! With HBase, it’s fast too, but I noticed that it pauses here and there, 
making it take seconds for > 40k rows at a time, while Kudu doesn’t. The 
progress bar just blinks by. I will keep this running until it hits 1B rows and 
rerun my performance tests. This, hopefully, will give better numbers.

Thanks,
Ben


> On Jun 28, 2016, at 4:26 PM, Todd Lipcon  wrote:
> 
> Cool, thanks for the report, Ben. For what it's worth, I think there's still 
> some low hanging fruit in the Spark connector for Kudu (for example, I 
> believe locality on reads is currently broken). So, you can expect 
> performance to continue to improve in future versions. I'd also be interested 
> to see results on Kudu for a much larger dataset - my guess is a lot of the 6 
> seconds you're seeing is constant overhead from Spark job setup, etc, given 
> that the performance doesn't seem to get slower as you went from 700K rows to 
> 13M rows.
> 
> -Todd
> 
> On Tue, Jun 28, 2016 at 3:03 PM, Benjamin Kim  > wrote:
> FYI.
> 
> I did a quick-n-dirty performance test.
> 
> First, the setup:
> QA cluster:
> 15 data nodes
> 64GB memory each
> HBase is using 4GB of memory
> Kudu is using 1GB of memory
> 1 HBase/Kudu master node
> 64GB memory
> HBase/Kudu master is using 1GB of memory each
> 10Gb Ethernet
> 
> Using Spark on both to load/read events data (84 columns per row), I was able 
> to record performance for each. On the HBase side, I used the Phoenix 4.7 
> Spark plugin where DataFrames can be used directly. On the Kudu side, I used 
> the Spark connector. I created an events table in Phoenix using the CREATE 
> TABLE statement and created the equivalent in Kudu using the Spark method 
> based off of a DataFrame schema.
> 
> Here are the numbers for Phoenix/HBase.
> 1st run:
> > 715k rows
> - write: 2.7m
> 
> > 715k rows in HBase table
> - read: 0.1s
> - count: 3.8s
> - aggregate: 61s
> 
> 2nd run:
> > 5.2M rows
> - write: 11m
> * had 4 region servers go down, had to retry the 5.2M row write
> 
> > 5.9M rows in HBase table
> - read: 8s
> - count: 3m
> - aggregate: 46s
> 
> 3rd run:
> > 6.8M rows
> - write: 9.6m
> 
> > 12.7M rows
> - read: 10s
> - count: 3m
> - aggregate: 44s
> 
> 
> Here are the numbers for Kudu.
> 1st run:
> > 715k rows
> - write: 18s
> 
> > 715k rows in Kudu table
> - read: 0.2s
> - count: 18s
> - aggregate: 5s
> 
> 2nd run:
> > 5.2M rows
> - write: 33s
> 
> > 5.9M rows in Kudu table
> - read: 0.2s
> - count: 16s
> - aggregate: 6s
> 
> 3rd run:
> > 6.8M rows
> - write: 27s
> 
> > 12.7M rows in Kudu table
> - read: 0.2s
> - count: 16s
> - aggregate: 6s
> 
> The Kudu results are impressive if you take these number as-is. Kudu is close 
> to 18x faster at writing (UPSERT). Kudu is 30x faster at reading (HBase times 
> increase as data size grows).  Kudu is 7x faster at full row counts. Lastly, 
> Kudu is 3x faster doing an aggregate query (count distinct event_id’s per 
> user_id). *Remember that this is small cluster, times are still respectable 
> for both systems, HBase could have been configured better, and the HBase 
> table could have been better tuned.
> 
> Cheers,
> Ben
> 
> 
>> On Jun 15, 2016, at 10:13 AM, Dan Burkert > > wrote:
>> 
>> Adding partition splits when range partitioning is done via the 
>> CreateTableOptions.addSplitRow 
>> 
>>  method.  You can find more about the different partitioning options in the 
>> schema design guide 
>> .  We generally 
>> recommend sticking to hash partitioning if possible, since you don't have to 
>> determine your own split rows.
>> 
>> - Dan
>> 
>> On Wed, Jun 15, 2016 at 9:17 AM, Benjamin Kim > > wrote:
>> Todd,
>> 
>> I think the locality is not within our setup. We have the compute cluster 
>> with Spark, YARN, etc. on its own, and we have the storage cluster with 
>> HBase, Kudu, etc. on another. We beefed up the hardware specs on the compute 
>> cluster and beefed up storage capacity on the storage cluster. We got this 
>> setup idea from the Databricks folks. I do have a question. I created the 
>> table to use range partition on columns. I see that if I use hash partition 
>> I can set the number of splits, but how do I do that using range (50 nodes * 
>> 10 = 500 splits)?
>> 
>> Thanks,
>> Ben
>> 
>> 
>>> On Jun 15, 2016, at 9:11 AM, Todd Lipcon >> > wrote:
>>> 
>>> Awesome use case. One thing to keep in mind is that spark parallelism will 
>>> be limited by the number of tablets. So, you might want to split into 10 or 
>>> so buckets per node to get the best query throughput.
>>> 
>>> Usually if you run top on some machines while running the query you can see 
>>> if it is fully utilizing the cores.

Re: Kundera - JPA compliant Object Datastore Mapper for Kudu

[Jean-Daniel Cryans]

Hi Karthik!

Thanks for sharing this.

I see that you've written most of the code so I wonder, do you have any
feedback on Kudu's APIs? Any weird things you noticed? Any gotchas?

We're getting close to 1.0, so we still have some time to make (potentially
breaking) changes.

Thanks!

J-D

On Wed, Jun 29, 2016 at 3:48 AM, Karthik Prasad Manchala <
karthikp.manch...@impetus.co.in> wrote:

> Hi all,
>
>
> Kundera  being one of the
> most popular JPA provider for NoSql datastores has added support for basic
> CRUD operations and Select queries on Kudu. Please feel free to explore
> more using the below link.
>
>
> - https://github.com/impetus-opensource/Kundera/wiki/Kundera-with-Kudu
>
>
> Thanks and regards,
>
> Team Kundera.
>
> --
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>

Kundera - JPA compliant Object Datastore Mapper for Kudu

[Karthik Prasad Manchala]

Hi all,


Kundera being one of the most 
popular JPA provider for NoSql datastores has added support for basic CRUD 
operations and Select queries on Kudu. Please feel free to explore more using 
the below link.


- https://github.com/impetus-opensource/Kundera/wiki/Kundera-with-Kudu


Thanks and regards,

Team Kundera.








NOTE: This message may contain information that is confidential, proprietary, 
privileged or otherwise protected by law. The message is intended solely for 
the named addressee. If received in error, please destroy and notify the 
sender. Any use of this email is prohibited when received in error. Impetus 
does not represent, warrant and/or guarantee, that the integrity of this 
communication has been maintained nor that the communication is free of errors, 
virus, interception or interference.