[Impala-ASF-CR] IMPALA-5235: Initialize resourceProfile with a dummy value

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5235: Initialize resourceProfile_ with a dummy value
..


Patch Set 1: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/6750
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I02de2ca1f627d7f61921840722661a5323e91579
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: anujphadke 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5235: Initialize resourceProfile with a dummy value

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-5235: Initialize resourceProfile_ with a dummy value
..


IMPALA-5235: Initialize resourceProfile_ with a dummy value

resourceProfile_ is initialized to NULL and gets set later while
finalizing the plan fragment. resourceProfile_ is accessed in
createHashJoinFragment before it gets set. Accessing it before
throws a NullPointerException. This change initializes it with a dummy
value instead of null to avoid this.
This code path gets executed only with log_level=3

Change-Id: I02de2ca1f627d7f61921840722661a5323e91579
Reviewed-on: http://gerrit.cloudera.org:8080/6750
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins
---
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
1 file changed, 3 insertions(+), 2 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Alex Behm: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/6750
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I02de2ca1f627d7f61921840722661a5323e91579
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: anujphadke 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
..


IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter

HdfsSequenceTableWriter::ConsumeRow() function dereferenced a pointer
that pointed to a previously deallocated memory (which belonged to an
out of scope string object). This caused the ASAN build to fail.

The fix was verified by running TestTableWriters.test_seq_writer and
TestTableWriters.test_seq_writer_hive_compatibility end-to-end tests
against the ASAN build. These tests consistently crashed impalad
before the fix.

Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b
Reviewed-on: http://gerrit.cloudera.org:8080/6762
Reviewed-by: Michael Ho 
Tested-by: Impala Public Jenkins
---
M be/src/exec/hdfs-sequence-table-writer.cc
1 file changed, 4 insertions(+), 3 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Michael Ho: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/6762
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Attila Jeges 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho 

[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
..


Patch Set 2: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/6762
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Attila Jeges 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when 
mem_limit exceeded
..


IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded

We need to check for AllocateLocal() returning NULL. CopyFrom() takes
care of that for us.  Also adjust a few other places in the code base
that didn't have the check.

The new test reproduces the crash, but in order to get this test file to
execute, I had to move the xfail to be a function decorator. Apparently
xfail as a statement causes the test to not run at all. We should run
all of these queries even if they are non-determistic to at least verify
that impalad does not crash.

Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6
Reviewed-on: http://gerrit.cloudera.org:8080/6761
Reviewed-by: Dan Hecht 
Tested-by: Impala Public Jenkins
---
M be/src/exprs/aggregate-functions-ir.cc
M be/src/exprs/hive-udf-call.cc
M be/src/exprs/udf-builtins-ir.cc
M be/src/udf/uda-test.cc
M be/src/udf/udf-test.cc
M be/src/udf_samples/uda-sample.cc
M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-update.test
M tests/custom_cluster/test_alloc_fail.py
8 files changed, 27 insertions(+), 25 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Dan Hecht: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/6761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Matthew Jacobs 
Gerrit-Reviewer: Michael Ho 

[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when 
mem_limit exceeded
..


Patch Set 4: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/6761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Matthew Jacobs 
Gerrit-Reviewer: Michael Ho 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5235: Initialize resourceProfile with a dummy value

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5235: Initialize resourceProfile_ with a dummy value
..


Patch Set 1:

Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/519/

-- 
To view, visit http://gerrit.cloudera.org:8080/6750
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I02de2ca1f627d7f61921840722661a5323e91579
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: anujphadke 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
..


Patch Set 2:

Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/518/

-- 
To view, visit http://gerrit.cloudera.org:8080/6762
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Attila Jeges 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter

[Michael Ho (Code Review)]

Michael Ho has posted comments on this change.

Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
..


Patch Set 2: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/6762
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Attila Jeges 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Michael Ho 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when 
mem_limit exceeded
..


Patch Set 4:

Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/517/

-- 
To view, visit http://gerrit.cloudera.org:8080/6761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Matthew Jacobs 
Gerrit-Reviewer: Michael Ho 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded

[Dan Hecht (Code Review)]

Dan Hecht has posted comments on this change.

Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when 
mem_limit exceeded
..


Patch Set 4: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/6761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Matthew Jacobs 
Gerrit-Reviewer: Michael Ho 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5162,IMPALA-5163: stress test support on secure clusters

[Michael Brown (Code Review)]

Michael Brown has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/6763

Change subject: IMPALA-5162,IMPALA-5163: stress test support on secure clusters
..

IMPALA-5162,IMPALA-5163: stress test support on secure clusters

This patch adds support for running the stress test
(concurrent_select.py) and loading nested data (load_nested.py) into a
Kerberized, SSL-enabled Impala cluster. It assumes the calling user
already has a valid Kerberos ticket. One way to do that is:

1. Get access to a keytab and krb5.config
2. Set KRB5_CONFIG and KRB5CCNAME appropriately
3. Run kinit(1)
4. Run load_nested.py and/or concurrent_select.py within this
   environment.

Because our Python clients already support Kerberos and SSL, we simply
need to make sure to use the correct options when calling the entry
points and initializing the clients:

Impala: Impyla
Hive: Impyla
HDFS: hdfs.ext.kerberos.KerberosClient

With this patch, I was able to manually do a short concurrent_select.py
run against a secure cluster without connection or auth errors, and I
was able to do the same with load_nested.py for a cluster that already
had TPC-H loaded.

Follow-ons for future cleanup work:

IMPALA-5263: support CA bundles when running stress test against SSL'd
 Impala

IMPALA-5264: fix InsecurePlatformWarning under stress test with SSL

Change-Id: I0daad57bb8ceeb5071b75125f11c1997ed7e0179
---
M testdata/bin/load_nested.py
M tests/comparison/cli_options.py
M tests/comparison/cluster.py
M tests/comparison/db_connection.py
M tests/stress/concurrent_select.py
5 files changed, 61 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/6763/1
-- 
To view, visit http://gerrit.cloudera.org:8080/6763
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I0daad57bb8ceeb5071b75125f11c1997ed7e0179
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Michael Brown 

[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded

[Michael Ho (Code Review)]

Michael Ho has posted comments on this change.

Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when 
mem_limit exceeded
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/6761/3/be/src/exprs/aggregate-functions-ir.cc
File be/src/exprs/aggregate-functions-ir.cc:

PS3, Line 1295: reinterpret_cast
> oops, i had to add this cast. Michael do you still prefer to call CopyFrom 
Hmm...it still looks cleaner to call CopyFrom() if possible.


-- 
To view, visit http://gerrit.cloudera.org:8080/6761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Dan Hecht 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Matthew Jacobs 
Gerrit-Reviewer: Michael Ho 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter

[Attila Jeges (Code Review)]

Attila Jeges has posted comments on this change.

Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
..


Patch Set 1:

(1 comment)

> (1 comment)
 > 
 > I assumed you have verified the fix with ASAN build, right ?

Yes, added testing details to the commit message.

http://gerrit.cloudera.org:8080/#/c/6762/1/be/src/exec/hdfs-sequence-table-writer.cc
File be/src/exec/hdfs-sequence-table-writer.cc:

Line 306: value_length = text.size();
> DCHECK_EQ(value_length, row_buf_.Size());
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/6762
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Attila Jeges 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Michael Ho 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3742: Partitions and sort INSERTs for Kudu tables

[Marcel Kornacker (Code Review)]

Marcel Kornacker has posted comments on this change.

Change subject: IMPALA-3742: Partitions and sort INSERTs for Kudu tables
..


Patch Set 7: Code-Review+2

(2 comments)

http://gerrit.cloudera.org:8080/#/c/6559/7/fe/src/main/java/org/apache/impala/analysis/KuduPartitionExpr.java
File fe/src/main/java/org/apache/impala/analysis/KuduPartitionExpr.java:

Line 37:  * a given row. The children of this Expr produce the values for the 
partition columns.
is it documented in some class header that values outside the legal range 
return a -1? probably most appropriate in the .h file.


Line 74:   if (i != 0) {
single line


-- 
To view, visit http://gerrit.cloudera.org:8080/6559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I84ce0032a1b10958fdf31faef225372c5c38fdc4
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Marcel Kornacker 
Gerrit-Reviewer: Matthew Jacobs 
Gerrit-Reviewer: Mostafa Mokhtar 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5188: Add slot sorting in TupleDescriptor::LayoutEquals()

[Taras Bobrovytsky (Code Review)]

Taras Bobrovytsky has posted comments on this change.

Change subject: IMPALA-5188: Add slot sorting in TupleDescriptor::LayoutEquals()
..


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/6610/6/be/src/runtime/descriptors.cc
File be/src/runtime/descriptors.cc:

Line 678:   for (SlotDescriptor* slot: slots()) sorted_slots[slot->slot_idx_] = 
slot;
It turns out that the reason why the GVO for this patch is failing is that 
slot_idx does not always start with 0. The following query produces a tuple 
with a single slot with a slot_idx=1:

select c_custkey, v1.cnt
from tpch_nested_parquet.customer c
inner join
  (select count(*) cnt from c.c_orders
   where false) v1
where c_custkey < 10

The tuple looks like this:
Tuple(id=0 size=25 slots=[Slot(id=3 type=BIGINT col_path=[0] offset=16 
null=(offset=24 mask=2) slot_idx=1 field_idx=-1)] tuple_path=[])


-- 
To view, visit http://gerrit.cloudera.org:8080/6610
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I426ad244678dbfe517262dfb7bbf4adc0247a35e
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4318: Kudu support for CREATE EXTERNAL TABLE AS SELECT

[Zach Amsden (Code Review)]

Zach Amsden has abandoned this change.

Change subject: IMPALA-4318:  Kudu support for CREATE EXTERNAL TABLE AS SELECT
..


Abandoned

Not doing at this time.

-- 
To view, visit http://gerrit.cloudera.org:8080/6261
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: abandon
Gerrit-Change-Id: I9aa82809a6c0c5e6386827314b7e5b520c1a6633
Gerrit-PatchSet: 14
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Zach Amsden 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Marcel Kornacker 
Gerrit-Reviewer: Matthew Jacobs 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Zach Amsden 

[Impala-ASF-CR] IMPALA-5180: Don't use non-deterministic exprs in partition pruning

[Zach Amsden (Code Review)]

Zach Amsden has uploaded a new patch set (#8).

Change subject: IMPALA-5180: Don't use non-deterministic exprs in partition 
pruning
..

IMPALA-5180: Don't use non-deterministic exprs in partition pruning

Non-deterministic exprs which evaluate as constant should not be
used during HDFS partition pruning.  We consider Exprs which have no
SlotRefs as bound by default, and thus we end up trying to apply
them indisrciminately.  Constant propagation makes this situation
easier to run into and the behavior is rather unexpected.

The fix for now is to explicitly disallow non-deterministic Exprs
in partition pruning.

Change-Id: I91054c6bf017401242259a1eff5e859085285546
---
M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/hdfs.test
4 files changed, 35 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/6575/8
-- 
To view, visit http://gerrit.cloudera.org:8080/6575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I91054c6bf017401242259a1eff5e859085285546
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Zach Amsden 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Zach Amsden 

[Impala-ASF-CR] IMPALA-5246: UDF's Close() should handle Expr's preparation failure

[Michael Ho (Code Review)]

Michael Ho has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/6757

Change subject: IMPALA-5246: UDF's Close() should handle Expr's preparation 
failure
..

IMPALA-5246: UDF's Close() should handle Expr's preparation failure

UDF may fail to initialize due low memory limit or
other reasons In which case, its Prepare() function may
not have been called and its thread local state may not be
initialized.

The Close() functions of some of the built-in and test-udf
made the wrong assumption that the thread local states are
always initialized. This may lead to de-referencing null
pointer in Close(). This change fixes this issue by
checking the thread local state is not null and returns
early if so.

Change-Id: Id2c689246ed4f8dd38f104fa35904f3926a7039c
---
M be/src/exprs/case-expr.cc
M be/src/exprs/math-functions-ir.cc
M be/src/exprs/string-functions-ir.cc
M be/src/exprs/timestamp-functions.cc
M be/src/exprs/utility-functions.cc
M be/src/testutil/test-udfs.cc
6 files changed, 17 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/6757/1
-- 
To view, visit http://gerrit.cloudera.org:8080/6757
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id2c689246ed4f8dd38f104fa35904f3926a7039c
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Michael Ho 

[Impala-ASF-CR] IMPALA-5180: Don't use non-deterministic exprs in partition pruning

[Zach Amsden (Code Review)]

Zach Amsden has uploaded a new patch set (#7).

Change subject: IMPALA-5180: Don't use non-deterministic exprs in partition 
pruning
..

IMPALA-5180: Don't use non-deterministic exprs in partition pruning

Non-deterministic exprs which evaluate as constant should not be
used during HDFS partition pruning.  We consider Exprs which have no
SlotRefs as bound by default, and thus we end up trying to apply
them indisrciminately.  Constant propagation makes this situation
easier to run into and the behavior is rather unexpected.

The fix for now is to explicitly disallow non-deterministic Exprs
in partition pruning.

Change-Id: I91054c6bf017401242259a1eff5e859085285546
---
M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/hdfs.test
4 files changed, 36 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/6575/7
-- 
To view, visit http://gerrit.cloudera.org:8080/6575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I91054c6bf017401242259a1eff5e859085285546
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Zach Amsden 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Zach Amsden 

[Impala-ASF-CR] IMPALA-3742: Partitions and sort INSERTs for Kudu tables

[Thomas Tauber-Marshall (Code Review)]

Thomas Tauber-Marshall has uploaded a new patch set (#7).

Change subject: IMPALA-3742: Partitions and sort INSERTs for Kudu tables
..

IMPALA-3742: Partitions and sort INSERTs for Kudu tables

Bulk DMLs (INSERT, UPSERT, UPDATE, and DELETE) for Kudu
are currently painful because we just send rows randomly,
which creates a lot of work for Kudu since it partitions
and sorts data before writing, causing writes to be slow
and leading to timeouts.

We can alleviate this by sending the rows to Kudu already
partitioned and sorted. This patch partitions and sorts
rows according to Kudu's partitioning scheme for INSERTs
and UPSERTs. A followup patch will handle UPDATE and DELETE.

It accomplishes this by inserting an exchange node and a sort
node into the plan before the operation. Both the exchange and
the sort are given a KuduPartitionExpr which takes a row and
calls into the Kudu client to return its partition number.

It also disallows INSERT hints for Kudu tables, since the
hints that we support (SHUFFLE, CLUSTER, SORTBY), so longer
make sense.

Testing:
- Updated planner tests.
- Ran the Kudu functional tests.
- Ran performance tests demonstrating that we can now handle much
  larger inserts without having timeouts.

Change-Id: I84ce0032a1b10958fdf31faef225372c5c38fdc4
---
M be/src/exec/kudu-table-sink.cc
M be/src/exec/kudu-util.cc
M be/src/exec/kudu-util.h
M be/src/exprs/CMakeLists.txt
M be/src/exprs/expr-context.h
M be/src/exprs/expr.cc
A be/src/exprs/kudu-partition-expr.cc
A be/src/exprs/kudu-partition-expr.h
M be/src/runtime/coordinator.cc
M be/src/runtime/data-stream-sender.cc
M be/src/runtime/data-stream-sender.h
M be/src/scheduling/scheduler.cc
M bin/impala-config.sh
M common/thrift/Exprs.thrift
M common/thrift/Partitions.thrift
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
A fe/src/main/java/org/apache/impala/analysis/KuduPartitionExpr.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/planner/DataPartition.java
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/TableSink.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeUpsertStmtTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/kudu-upsert.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test
27 files changed, 618 insertions(+), 171 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6559/7
-- 
To view, visit http://gerrit.cloudera.org:8080/6559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I84ce0032a1b10958fdf31faef225372c5c38fdc4
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Marcel Kornacker 
Gerrit-Reviewer: Matthew Jacobs 
Gerrit-Reviewer: Mostafa Mokhtar 
Gerrit-Reviewer: Thomas Tauber-Marshall 

[Impala-ASF-CR] IMPALA-5257: test seq writer hive compatibility fails on local file system build

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5257: test_seq_writer_hive_compatibility fails on local 
file system build
..


Patch Set 1: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/6746
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic7dbe2529818865f871b66d78642ed956d1ee039
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Attila Jeges 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5257: test seq writer hive compatibility fails on local file system build

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-5257: test_seq_writer_hive_compatibility fails on local 
file system build
..


IMPALA-5257: test_seq_writer_hive_compatibility fails on local file
system build

TestTableWriters.test_seq_writer_hive_compatibility test introduced in
IMPALA-3079 had to be skipped for non-HDFS filesystems.

Change-Id: Ic7dbe2529818865f871b66d78642ed956d1ee039
Reviewed-on: http://gerrit.cloudera.org:8080/6746
Reviewed-by: Michael Ho 
Tested-by: Impala Public Jenkins
---
M tests/query_test/test_compressed_formats.py
1 file changed, 3 insertions(+), 0 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Michael Ho: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/6746
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic7dbe2529818865f871b66d78642ed956d1ee039
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Attila Jeges 
Gerrit-Reviewer: Attila Jeges 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho