[jira] [Commented] (KUDU-2416) Incorrect fallthrough in Java PartialRow.setMin for DECIMAL times
[
https://issues.apache.org/jira/browse/KUDU-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451552#comment-16451552
]
Grant Henke commented on KUDU-2416:
---
I think we should likely add some more PartitionPruner test coverage, but I put
a pointed patch to get in sooner than later.
> Incorrect fallthrough in Java PartialRow.setMin for DECIMAL times
> -
>
> Key: KUDU-2416
> URL: https://issues.apache.org/jira/browse/KUDU-2416
> Project: Kudu
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.0
>Reporter: Todd Lipcon
>Assignee: Grant Henke
>Priority: Critical
>
> There's a missing 'break' statement in the following code:
> {code:java}
> case DECIMAL:
> ColumnTypeAttributes typeAttributes = column.getTypeAttributes();
> addDecimal(index,
> DecimalUtil.minValue(typeAttributes.getPrecision(),
> typeAttributes.getScale()));
> case STRING:
> addStringUtf8(index, AsyncKuduClient.EMPTY_ARRAY);
> break;
> {code}
> which I think could cause incorrect results for range partition pruning on
> decimal columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KUDU-2416) Incorrect fallthrough in Java PartialRow.setMin for DECIMAL times
[
https://issues.apache.org/jira/browse/KUDU-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451541#comment-16451541
]
Grant Henke commented on KUDU-2416:
---
A quick look at PartialRow.setMin shows it is used in 3 places:
* *_PartialRow.decodeRangePartitionKey_*
** When the buffer being decoded doesn't contain the column
** This is used in _Partition.formatRangePartition which is only used by
Impala and should always have all the columns in the buffer given_
* *_PartitionPruner.pushPredsIntoLowerBoundRangeKey &
PartitionPruner.pushPredsIntoUpperBoundRangeKey_*:
** When the column is part of the range partition but not a part of the
predicate
** These are only used in _PartitionPruner.create which is used in the scanner
APIs_
In all cases when the decimal case falls through and addStringUtf8 is called it
will result in an IllegalArgumentException when checking the expected column
type. This is not great, but it's better than returning the wrong rows.
However, when adding a unit test for PartialRow.setMin I found that since 1.0.0
([4fd0572|https://github.com/apache/kudu/commit/4fd0572]) we have been setting
Integer.MIN_VALUE on LONG and UNIXTIME_MICROS/TIMESTAMP columns. This is likely
a much bigger issue given that could result in silently pruning the wrong
partitions in the case that a LONG column is a part of the range partition but
not a part of the predicate. I need to look at _PartitionPruner.create_ code
more to understand the true impact.
I will post this patch with this test and then investigate further.
> Incorrect fallthrough in Java PartialRow.setMin for DECIMAL times
> -
>
> Key: KUDU-2416
> URL: https://issues.apache.org/jira/browse/KUDU-2416
> Project: Kudu
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.0
>Reporter: Todd Lipcon
>Assignee: Grant Henke
>Priority: Critical
>
> There's a missing 'break' statement in the following code:
> {code:java}
> case DECIMAL:
> ColumnTypeAttributes typeAttributes = column.getTypeAttributes();
> addDecimal(index,
> DecimalUtil.minValue(typeAttributes.getPrecision(),
> typeAttributes.getScale()));
> case STRING:
> addStringUtf8(index, AsyncKuduClient.EMPTY_ARRAY);
> break;
> {code}
> which I think could cause incorrect results for range partition pruning on
> decimal columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KUDU-2416) Incorrect fallthrough in Java PartialRow.setMin for DECIMAL times
[
https://issues.apache.org/jira/browse/KUDU-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451437#comment-16451437
]
Todd Lipcon commented on KUDU-2416:
---
[~granthenke] mind taking a look at this? I can add the missing 'break' but not
sure of the ramifications here and what test coverage we're missing.
> Incorrect fallthrough in Java PartialRow.setMin for DECIMAL times
> -
>
> Key: KUDU-2416
> URL: https://issues.apache.org/jira/browse/KUDU-2416
> Project: Kudu
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.0
>Reporter: Todd Lipcon
>Assignee: Grant Henke
>Priority: Critical
>
> There's a missing 'break' statement in the following code:
> {code:java}
> case DECIMAL:
> ColumnTypeAttributes typeAttributes = column.getTypeAttributes();
> addDecimal(index,
> DecimalUtil.minValue(typeAttributes.getPrecision(),
> typeAttributes.getScale()));
> case STRING:
> addStringUtf8(index, AsyncKuduClient.EMPTY_ARRAY);
> break;
> {code}
> which I think could cause incorrect results for range partition pruning on
> decimal columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KUDU-2416) Incorrect fallthrough in Java PartialRow.setMin for DECIMAL times
Todd Lipcon created KUDU-2416:
-
Summary: Incorrect fallthrough in Java PartialRow.setMin for
DECIMAL times
Key: KUDU-2416
URL: https://issues.apache.org/jira/browse/KUDU-2416
Project: Kudu
Issue Type: Bug
Components: java
Affects Versions: 1.7.0
Reporter: Todd Lipcon
Assignee: Grant Henke
There's a missing 'break' statement in the following code:
{code:java}
case DECIMAL:
ColumnTypeAttributes typeAttributes = column.getTypeAttributes();
addDecimal(index,
DecimalUtil.minValue(typeAttributes.getPrecision(),
typeAttributes.getScale()));
case STRING:
addStringUtf8(index, AsyncKuduClient.EMPTY_ARRAY);
break;
{code}
which I think could cause incorrect results for range partition pruning on
decimal columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KUDU-2352) Add an API to allow bounded staleness scan
[ https://issues.apache.org/jira/browse/KUDU-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Hao reassigned KUDU-2352: - Assignee: Hao Hao > Add an API to allow bounded staleness scan > -- > > Key: KUDU-2352 > URL: https://issues.apache.org/jira/browse/KUDU-2352 > Project: Kudu > Issue Type: Improvement > Components: api >Affects Versions: 1.7.0 >Reporter: Hao Hao >Assignee: Hao Hao >Priority: Major > > It would be nice to have an API that allows clients specify a timestamp so > that in either READ_AT_SNAPSHOT or READ_YOUR_WRITES mode the chosen snapshot > timestamp is guaranteed to be higher than the given bound. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KUDU-2415) READ_YOUR_WRITES scan with no prior operation fails
[ https://issues.apache.org/jira/browse/KUDU-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Hao reassigned KUDU-2415: - Assignee: Hao Hao > READ_YOUR_WRITES scan with no prior operation fails > --- > > Key: KUDU-2415 > URL: https://issues.apache.org/jira/browse/KUDU-2415 > Project: Kudu > Issue Type: Bug > Components: client, tserver >Affects Versions: 1.7.0 >Reporter: Todd Lipcon >Assignee: Hao Hao >Priority: Major > > If I create a new Java client, and then perform a scan in READ_YOUR_WRITES > mode without having done any prior operations from that client, it sends an > RPC with read_mode=READ_YOUR_WRITES but without any propagated or snapshot > timestamp field set. The server seems to interpret this as a value '0' and > then fails with the error: > Snapshot timestamp is earlier than the ancient history mark. Consider > increasing the value of the configuration parameter > --tablet_history_max_age_sec. Snapshot timestamp: P: 0 usec, L: 1 Ancient > History Mark: P: 1524607330402072 usec, L: 0 Physical time difference: > -1524607330.402s -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KUDU-2415) READ_YOUR_WRITES scan with no prior operation fails
[ https://issues.apache.org/jira/browse/KUDU-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451414#comment-16451414 ] Hao Hao commented on KUDU-2415: --- I think one way to mitigate this is KUDU-2352. > READ_YOUR_WRITES scan with no prior operation fails > --- > > Key: KUDU-2415 > URL: https://issues.apache.org/jira/browse/KUDU-2415 > Project: Kudu > Issue Type: Bug > Components: client, tserver >Affects Versions: 1.7.0 >Reporter: Todd Lipcon >Priority: Major > > If I create a new Java client, and then perform a scan in READ_YOUR_WRITES > mode without having done any prior operations from that client, it sends an > RPC with read_mode=READ_YOUR_WRITES but without any propagated or snapshot > timestamp field set. The server seems to interpret this as a value '0' and > then fails with the error: > Snapshot timestamp is earlier than the ancient history mark. Consider > increasing the value of the configuration parameter > --tablet_history_max_age_sec. Snapshot timestamp: P: 0 usec, L: 1 Ancient > History Mark: P: 1524607330402072 usec, L: 0 Physical time difference: > -1524607330.402s -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KUDU-2415) READ_YOUR_WRITES scan with no prior operation fails
[
https://issues.apache.org/jira/browse/KUDU-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451395#comment-16451395
]
Todd Lipcon commented on KUDU-2415:
---
One other detail that looks important: this only happens if the tablet has
never been written to. In that case the MvccManager has state like:
{cur_snap_=MvccSnapshot[committed={T|T < 1}], in_flight=0, safe=0,
earliest=18446744073709551615}
and so the clean time is getting set to 0. This is as of commit
c4b8804ac16ebfe373efa33f82d86a0e3d677e72
> READ_YOUR_WRITES scan with no prior operation fails
> ---
>
> Key: KUDU-2415
> URL: https://issues.apache.org/jira/browse/KUDU-2415
> Project: Kudu
> Issue Type: Bug
> Components: client, tserver
>Affects Versions: 1.7.0
>Reporter: Todd Lipcon
>Priority: Major
>
> If I create a new Java client, and then perform a scan in READ_YOUR_WRITES
> mode without having done any prior operations from that client, it sends an
> RPC with read_mode=READ_YOUR_WRITES but without any propagated or snapshot
> timestamp field set. The server seems to interpret this as a value '0' and
> then fails with the error:
> Snapshot timestamp is earlier than the ancient history mark. Consider
> increasing the value of the configuration parameter
> --tablet_history_max_age_sec. Snapshot timestamp: P: 0 usec, L: 1 Ancient
> History Mark: P: 1524607330402072 usec, L: 0 Physical time difference:
> -1524607330.402s
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KUDU-2415) READ_YOUR_WRITES scan with no prior operation fails
[
https://issues.apache.org/jira/browse/KUDU-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451395#comment-16451395
]
Todd Lipcon edited comment on KUDU-2415 at 4/24/18 11:26 PM:
-
One other detail that looks important: this only happens if the tablet has
never been written to. In that case the MvccManager has state like:
{code}
{cur_snap_=MvccSnapshot[committed={T|T < 1}], in_flight=0, safe=0,
earliest=18446744073709551615}
{code}
and so the clean time is getting set to 0. This is as of commit
c4b8804ac16ebfe373efa33f82d86a0e3d677e72
was (Author: tlipcon):
One other detail that looks important: this only happens if the tablet has
never been written to. In that case the MvccManager has state like:
{cur_snap_=MvccSnapshot[committed={T|T < 1}], in_flight=0, safe=0,
earliest=18446744073709551615}
and so the clean time is getting set to 0. This is as of commit
c4b8804ac16ebfe373efa33f82d86a0e3d677e72
> READ_YOUR_WRITES scan with no prior operation fails
> ---
>
> Key: KUDU-2415
> URL: https://issues.apache.org/jira/browse/KUDU-2415
> Project: Kudu
> Issue Type: Bug
> Components: client, tserver
>Affects Versions: 1.7.0
>Reporter: Todd Lipcon
>Priority: Major
>
> If I create a new Java client, and then perform a scan in READ_YOUR_WRITES
> mode without having done any prior operations from that client, it sends an
> RPC with read_mode=READ_YOUR_WRITES but without any propagated or snapshot
> timestamp field set. The server seems to interpret this as a value '0' and
> then fails with the error:
> Snapshot timestamp is earlier than the ancient history mark. Consider
> increasing the value of the configuration parameter
> --tablet_history_max_age_sec. Snapshot timestamp: P: 0 usec, L: 1 Ancient
> History Mark: P: 1524607330402072 usec, L: 0 Physical time difference:
> -1524607330.402s
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KUDU-2415) READ_YOUR_WRITES scan with no prior operation fails
Todd Lipcon created KUDU-2415: - Summary: READ_YOUR_WRITES scan with no prior operation fails Key: KUDU-2415 URL: https://issues.apache.org/jira/browse/KUDU-2415 Project: Kudu Issue Type: Bug Components: client, tserver Affects Versions: 1.7.0 Reporter: Todd Lipcon If I create a new Java client, and then perform a scan in READ_YOUR_WRITES mode without having done any prior operations from that client, it sends an RPC with read_mode=READ_YOUR_WRITES but without any propagated or snapshot timestamp field set. The server seems to interpret this as a value '0' and then fails with the error: Snapshot timestamp is earlier than the ancient history mark. Consider increasing the value of the configuration parameter --tablet_history_max_age_sec. Snapshot timestamp: P: 0 usec, L: 1 Ancient History Mark: P: 1524607330402072 usec, L: 0 Physical time difference: -1524607330.402s -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KUDU-1366) Consider switching to jemalloc
[ https://issues.apache.org/jira/browse/KUDU-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved KUDU-1366. --- Resolution: Won't Fix Fix Version/s: n/a With recent improvements in tcmalloc I don't think this is worth pursuing > Consider switching to jemalloc > -- > > Key: KUDU-1366 > URL: https://issues.apache.org/jira/browse/KUDU-1366 > Project: Kudu > Issue Type: Bug > Components: perf >Reporter: Todd Lipcon >Priority: Major > Fix For: n/a > > Attachments: Kudu Benchmarks.pdf, Kudu Benchmarks.pdf > > > We spend a fair amount of time in the allocator. While we could spend some > time trying to use arenas more, it's also worth considering switching > allocators. I ran a few quick tests with jemalloc 4.1 and it seems like it > might be better than the version of tcmalloc that we use (and has much more > active development) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
