[ANNOUNCE] Apache Kudu 1.6.0 released

[Mike Percy]

The Apache Kudu team is happy to announce the release of Kudu 1.6.0.

Kudu is an open source storage engine for structured data that supports
low-latency random access together with efficient analytical access
patterns.
It is designed within the context of the Apache Hadoop ecosystem and
supports
many integrations with other data analytics projects both inside and
outside of
the Apache Software Foundation.

Apache Kudu 1.6.0 is a minor release that offers several new features,
improvements, optimizations, and bug fixes. Please see the release notes for
details.

Download it here: https://kudu.apache.org/releases/1.6.0/
Full release notes:
https://kudu.apache.org/releases/1.6.0/docs/release_notes.html

Regards,
The Apache Kudu team

Re: Data inconsistency after restart

[David Alves]

Hi Petter

   I'd like to clarify what exactly happened and exactly what are you
referring to as "inconsistency".
   From what I understand of the first error you observed, the Kudu was
underprovisioned, memory wise, and the ingest jobs/queries failed. Is that
right? Since Kudu doesn't have atomic multi-row writes, it's currently
expected in this case that you'll end up with partially written data.
   If you tried the same job again, and it succeeded, for certain types of
operation (UPSERT, INSERT IGNORE) then the remaining rows would be written
and all the data would be there as expected.
   I'd like to distinguish this lack of atomicity on multi-row transactions
from "inconsistency", which is what you might observe if an operation
didn't fail, but you couldn't see all the data. For this latter case there
are options you can choose to avoid any inconsistency.

Best
David



On Wed, Dec 6, 2017 at 4:26 AM, Petter von Dolwitz (Hem) <
petter.von.dolw...@gmail.com> wrote:

> Thanks for your reply Andrew!
>
> >How did you verify that all the data was inserted and how did you find
> some data missing?
> This was done using Impala. We counted the rows for groups representing
> the chunks we inserted.
>
> >Following up on what I posted, take a look at https://kudu.apache.org/doc
> s/transaction_semantics.html#_read_operations_scans. It seems definitely
> possible that not all of the rows had finished inserting when counting, or
> that the scans were sent to a stale replica.
> Before we shut down we could only see the following in the logs. I.e., no
> sign that ingestion was still ongoing.
>
> kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
> 07:27:35.010694 90793 maintenance_manager.cc:383] P
> a38902afefca4a85a5469d149df9b4cb: we have exceeded our soft memory limit
> (current capacity is 67.52%).  However, there are no ops currently runnable
> which would free memory.
>
> Also the (cloudera) metric total_kudu_rows_inserted_rate_across_kudu_replicas
> showed zero.
>
> Still it seems like some data became inconsistent after restart. But if
> the maintenance_manager performs important jobs that are required to ensure
> that all data is inserted then I can understand why we ended up with
> inconsistent data. But, if I understand you correct,  you are saying that
> these jobs are not critical for ingestion. In the link you provided I read
> "Impala scans are currently performed as READ_LATEST and have no
> consistency guarantees.". I would assume this means that it does not
> guarantee consistency if new data is inserted but should give valid (and
> same) results if no new data is inserted?
>
> I have not tried the ksck tool yet. Thank you for reminding. I will have a
> look.
>
> Br,
> Petter
>
>
> 2017-12-06 1:31 GMT+01:00 Andrew Wong :
>
>> How did you verify that all the data was inserted and how did you find
>>> some data missing? I'm wondering if it's possible that the initial
>>> "missing" data was data that Kudu was still in the process of inserting
>>> (albeit slowly, due to memory backpressure or somesuch).
>>>
>>
>> Following up on what I posted, take a look at https://kudu.apache.org/doc
>> s/transaction_semantics.html#_read_operations_scans. It seems definitely
>> possible that not all of the rows had finished inserting when counting, or
>> that the scans were sent to a stale replica.
>>
>> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong  wrote:
>>
>>> Hi Petter,
>>>
>>> When we verified that all data was inserted we found that some data was
 missing. We added this missing data and on some chunks we got the
 information that all rows were already present, i.e impala says something
 like Modified: 0 rows, nnn errors. Doing the verification again now
 shows that the Kudu table is complete. So, even though we did not insert
 any data on some chunks, a count(*) operation over these chunks now returns
 a different value.
>>>
>>>
>>> How did you verify that all the data was inserted and how did you find
>>> some data missing? I'm wondering if it's possible that the initial
>>> "missing" data was data that Kudu was still in the process of inserting
>>> (albeit slowly, due to memory backpressure or somesuch).
>>>
>>> Now to my question. Will data be inconsistent if we recycle Kudu after
 seeing soft memory limit warnings?
>>>
>>>
>>> Your data should be consistently written, even with those warnings.
>>> AFAIK they would cause a bit of slowness, not incorrect results.
>>>
>>> Is there a way to tell when it is safe to restart Kudu to avoid these
 issues? Should we use any special procedure when restarting (e.g. only
 restart the tablet servers, only restart one tablet server at a time or
 something like that)?
>>>
>>>
>>> In general, you can use the `ksck` tool to check the health of your
>>> cluster. See https://kudu.apache.org/docs/command_line_tools_referenc
>>> e.html#cluster-ksck for more details. For 

Re: Data inconsistency after restart

[David Alves]

(re-sending as apparently a previous version of this message got lost)

Hi Petter

   I'd like to clarify what exactly happened and exactly what are you
referring to as "inconsistency".
   From what I understand of the first error you observed, the Kudu
was underprovisioned, memory wise, and the ingest jobs/queries failed.
Is that right? Since Kudu doesn't have atomic multi-row writes, it's
currently expected in this case that you'll end up with partially
written data.
   If you tried the same job again, and it succeeded, for certain
types of operation (UPSERT, INSERT IGNORE) then the remaining rows
would be written and all the data would be there as expected. Which
kind of operation are you using to insert the data?
   I'd like to understand the precise context better. Particularly I'd
like to distinguish this lack of atomicity on multi-row transactions
from "inconsistency" (in the CAP sense), which is what you might
observe if an operation didn't fail, but you couldn't see all the
data. For this latter case there are options you can choose, and we
can go through those, if that's the case.

Best

On Thu, Dec 7, 2017 at 12:34 PM, Petter von Dolwitz (Hem)
 wrote:
> Hi Andrew and Alexey,
>
> thanks for taking the time to answer.
>
>>Interesting. Just to be sure, was that seen on one tserver, or did you see
>> them across all of them?
> I did not look at all tservers at the time but it was one or two out of five
> so this could be a reason if one of more tservers had remaining work. The
> inserted_rows metric was steady at zero though.
>
>>Did you verified the client didn't report any errors during data ingestion?
>> Most likely you did, but I just wanted to make sure. BTW, what kind of
>> client did you use for the data ingestion?
> I don't remember seeing any errors at the client side. I used 4 parallell
> impala jobs as clients. When memory was not constrained I could see
> ingestion rates steady around 1.5 million rows / second. After a while
> (couple of hours perhaps), the ingestion rate dropped significantly so I
> decided to stop ingesting and restart kudu with more memory (after the logs
> stopped spinning at the tserver(s) I was looking at). I guess this was due
> to a backlog building up and that kudu throttled the incoming traffic.
>
> I could try to repeat the exercise and try to record result more
> scientifically.
>
> Br,
> Petter
>
>
>
>
> 2017-12-07 20:48 GMT+01:00 Alexey Serbin :
>>
>> Hi Petter,
>>
>> Before going too deep in attempts to find the place where the data was
>> lost, I just wanted to make sure we definitely know that the data was
>> delivered from the client to the server side.
>>
>> Did you verified the client didn't report any errors during data
>> ingestion?  Most likely you did, but I just wanted to make sure. BTW, what
>> kind of client did you use for the data ingestion?
>>
>> Thanks!
>>
>>
>> Kind regards,
>>
>> Alexey
>>
>>
>> On 12/6/17 3:56 PM, Andrew Wong wrote:
>>>
>>> Hi Petter,
>>>
>>> Before we shut down we could only see the following in the logs.
>>> I.e., no sign that ingestion was still ongoing.
>>>
>>>
>>> Interesting. Just to be sure, was that seen on one tserver, or did you
>>> see them across all of them?
>>>
>>> But if the maintenance_manager performs important jobs that are
>>> required to ensure that all data is inserted then I can understand
>>> why we ended up with inconsistent data.
>>>
>>>
>>> The maintenance manager's role is somewhat orthogonal to writes: data is
>>> first written to the on-disk write-ahead log and also kept in-memory to be
>>> accessible by scans. The maintenance manager periodically shuttles this
>>> in-memory data to disk, among various other tasks like cleaning up WAL
>>> segments, compacting rowsets, etc. Given that, a lack of maintenance ops
>>> shouldn't cause incorrectness in data, even after restarting.
>>>
>>> I would assume this means that it does not guarantee consistency
>>> if new data is inserted but should give valid (and same) results
>>> if no new data is inserted?
>>>
>>>
>>> Right, if /all/ tservers a truly caught up and done processing the
>>> writes, with no tablet copies going on, and with no new data coming in, then
>>> the results should be consistent.
>>>
>>>
>>> Hope this helped,
>>> Andrew
>>>
>>> On Wed, Dec 6, 2017 at 7:33 AM, Boris Tyukin >> > wrote:
>>>
>>> this is smart, we are doing the same thing but the best part that
>>> attracts me to Kudu is replacing our main HDFS storage with Kudu
>>> to enable near RT use cases and not to deal with HBase and a
>>> Lambda architecture mess so reliability and scalability is a big
>>> deal for us as we are looking to move most of our data to Kudu.
>>>
>>> On Wed, Dec 6, 2017 at 9:58 AM, Petter von Dolwitz (Hem)
>>> >> > 

Re: Data inconsistency after restart

[Alexey Serbin]

Hi Petter,

Before going too deep in attempts to find the place where the data was 
lost, I just wanted to make sure we definitely know that the data was 
delivered from the client to the server side.


Did you verified the client didn't report any errors during data 
ingestion?  Most likely you did, but I just wanted to make sure. BTW, 
what kind of client did you use for the data ingestion?


Thanks!


Kind regards,

Alexey


On 12/6/17 3:56 PM, Andrew Wong wrote:

Hi Petter,

Before we shut down we could only see the following in the logs.
I.e., no sign that ingestion was still ongoing.


Interesting. Just to be sure, was that seen on one tserver, or did you 
see them across all of them?


But if the maintenance_manager performs important jobs that are
required to ensure that all data is inserted then I can understand
why we ended up with inconsistent data.


The maintenance manager's role is somewhat orthogonal to writes: data 
is first written to the on-disk write-ahead log and also kept 
in-memory to be accessible by scans. The maintenance manager 
periodically shuttles this in-memory data to disk, among various other 
tasks like cleaning up WAL segments, compacting rowsets, etc. Given 
that, a lack of maintenance ops shouldn't cause incorrectness in data, 
even after restarting.


I would assume this means that it does not guarantee consistency
if new data is inserted but should give valid (and same) results
if no new data is inserted?


Right, if /all/ tservers a truly caught up and done processing the 
writes, with no tablet copies going on, and with no new data coming 
in, then the results should be consistent.



Hope this helped,
Andrew

On Wed, Dec 6, 2017 at 7:33 AM, Boris Tyukin > wrote:


this is smart, we are doing the same thing but the best part that
attracts me to Kudu is replacing our main HDFS storage with Kudu
to enable near RT use cases and not to deal with HBase and a
Lambda architecture mess so reliability and scalability is a big
deal for us as we are looking to move most of our data to Kudu.

On Wed, Dec 6, 2017 at 9:58 AM, Petter von Dolwitz (Hem)
> wrote:

Hi Boris,

we do not have a Cloudera contract at the moment. Until we
gained more Kudu experience we keep our master data in parquet
format so that we can rebuild Kudu-tables upon errors. We are
still in the early learning phase.

Br,
Petter



2017-12-06 14:35 GMT+01:00 Boris Tyukin >:

this is definitely concerning thread for us looking to use
Impala for storing mission-critical company data. Petter,
are you paid Cloudera customer btw? I wonder if you opened
support ticket as well

On Wed, Dec 6, 2017 at 7:26 AM, Petter von Dolwitz (Hem)
> wrote:

Thanks for your reply Andrew!

>How did you verify that all the data was inserted and
how did you find some data missing?
This was done using Impala. We counted the rows for
groups representing the chunks we inserted.

>Following up on what I posted, take a look at

https://kudu.apache.org/docs/transaction_semantics.html#_read_operations_scans

.
It seems definitely possible that not all of the rows
had finished inserting when counting, or that the
scans were sent to a stale replica.
Before we shut down we could only see the following in
the logs. I.e., no sign that ingestion was still ongoing.


kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
07:27:35.010694 90793 maintenance_manager.cc:383] P
a38902afefca4a85a5469d149df9b4cb: we have exceeded our
soft memory limit (current capacity is 67.52%). 
However, there are no ops currently runnable which

would free memory.

Also the (cloudera) metric
total_kudu_rows_inserted_rate_across_kudu_replicas
showed zero.

Still it seems like some data became inconsistent
after restart. But if the maintenance_manager performs
important jobs that are required to ensure that all
data is inserted then I can understand why we ended up
with inconsistent data. But, if I understand you
correct,  you are saying that these jobs are not
critical