[Impala-ASF-CR] IMPALA-5235: Initialize resourceProfile with a dummy value
Impala Public Jenkins has posted comments on this change. Change subject: IMPALA-5235: Initialize resourceProfile_ with a dummy value .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/6750 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I02de2ca1f627d7f61921840722661a5323e91579 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: anujphadke Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5235: Initialize resourceProfile with a dummy value
Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-5235: Initialize resourceProfile_ with a dummy value .. IMPALA-5235: Initialize resourceProfile_ with a dummy value resourceProfile_ is initialized to NULL and gets set later while finalizing the plan fragment. resourceProfile_ is accessed in createHashJoinFragment before it gets set. Accessing it before throws a NullPointerException. This change initializes it with a dummy value instead of null to avoid this. This code path gets executed only with log_level=3 Change-Id: I02de2ca1f627d7f61921840722661a5323e91579 Reviewed-on: http://gerrit.cloudera.org:8080/6750 Reviewed-by: Alex Behm Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/planner/PlanNode.java 1 file changed, 3 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Verified Alex Behm: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/6750 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I02de2ca1f627d7f61921840722661a5323e91579 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: anujphadke Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter .. IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter HdfsSequenceTableWriter::ConsumeRow() function dereferenced a pointer that pointed to a previously deallocated memory (which belonged to an out of scope string object). This caused the ASAN build to fail. The fix was verified by running TestTableWriters.test_seq_writer and TestTableWriters.test_seq_writer_hive_compatibility end-to-end tests against the ASAN build. These tests consistently crashed impalad before the fix. Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b Reviewed-on: http://gerrit.cloudera.org:8080/6762 Reviewed-by: Michael Ho Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-sequence-table-writer.cc 1 file changed, 4 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Verified Michael Ho: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/6762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
Impala Public Jenkins has posted comments on this change. Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/6762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded We need to check for AllocateLocal() returning NULL. CopyFrom() takes care of that for us. Also adjust a few other places in the code base that didn't have the check. The new test reproduces the crash, but in order to get this test file to execute, I had to move the xfail to be a function decorator. Apparently xfail as a statement causes the test to not run at all. We should run all of these queries even if they are non-determistic to at least verify that impalad does not crash. Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Reviewed-on: http://gerrit.cloudera.org:8080/6761 Reviewed-by: Dan Hecht Tested-by: Impala Public Jenkins --- M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/hive-udf-call.cc M be/src/exprs/udf-builtins-ir.cc M be/src/udf/uda-test.cc M be/src/udf/udf-test.cc M be/src/udf_samples/uda-sample.cc M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-update.test M tests/custom_cluster/test_alloc_fail.py 8 files changed, 27 insertions(+), 25 deletions(-) Approvals: Impala Public Jenkins: Verified Dan Hecht: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Impala Public Jenkins has posted comments on this change. Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2550: Switch to per-query exec rpc
Dan Hecht has posted comments on this change.
Change subject: IMPALA-2550: Switch to per-query exec rpc
..
Patch Set 10:
(71 comments)
http://gerrit.cloudera.org:8080/#/c/6535/10/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:
PS10, Line 327: // if we get an error while trying to get a connection to the
backend,
: // keep going
:
what is this talking about?
PS10, Line 352: keep on cancelling the other fragments
seems misplaced now
PS10, Line 354: eturn true;
should this return false? we didn't actually succeed at sending the rpc, right?
http://gerrit.cloudera.org:8080/#/c/6535/10/be/src/runtime/coordinator-backend-state.h
File be/src/runtime/coordinator-backend-state.h:
PS10, Line 61: status
GetStatus()?
http://gerrit.cloudera.org:8080/#/c/6535/10/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
PS10, Line 178: close
i'm not sure what this is talking about. is it referring to CloseConsumer() or
Close()?
PS10, Line 179: it should
: // cancel itself when it receives an error status after
reporting its profile.
I also don't see what this is trying to explain.
oh, after reading FIS::Cancel() I guess this is referring to that code in there
that will do CloseConsumer(). This is pretty confusing since I wouldn't assume
that cancel on the FIS side is going to CloseConsumer.
PS10, Line 183: Grab executor
what is this referring to?
PS10, Line 184: order to get a reference to coord_instance_
: // so that coord_sink_ remains valid throughout query
lifetime.
how does getting a reference to coord_instance_ affect the validity of
coord_sink_? The FIS isn't ref counted, the QES is. But we already have the QES
reference, so i'm not sure what this is talking about.
PS10, Line 189: at this point, the query is done with the Prepare phase,
why do we know that? Oh, because GetFinstanceState() blocks until Prepare?
That's pretty subtle.
PS10, Line 204: GetPrepareStatus
this is a dangerous side-effect to have inside of a DCHECK -- it creates a
barrier only present in debug builds. while this isn't bug, it means that
release builds have a pretty big difference in behavior.
Maybe we should rename GetPrepareStatus() to WaitForPrepare() or similar to
highlight that side-effect and prevent this from happening.
Line 263: }
DCHECK(!has_coord_fragment || coord_backend_idx_ != -1) ?
PS10, Line 413: backend_state->GetStatus();
what is this trying to do? Oh, I guess get the status of BackendState::Exec()?
But this might also pick up the execution status of the FIS if it raced ahead
fast enough, right?
also is it really true this won't be CANCELLED? Couldn't we have FIS_0 hit an
error during QIS::Prepare(), which sets its preapred_status_, which QES notices
and calls QES::CancelInternal(), which calls FIS_1::Cancel(), which cases FIS_1
to send a report with CANCELLED as the status, which could arrive before the
report for FIS_0 which contains the real error?
PS10, Line 424: DCHECK(query_status_.ok()); // nobody should have been able to
cancel
why? because the query handle hasn't yet been returned? But what about via the
debug webpage?
PS10, Line 964: #if 0
: do we really want this?
what are you going to do with this?
http://gerrit.cloudera.org:8080/#/c/6535/10/be/src/runtime/coordinator.h
File be/src/runtime/coordinator.h:
PS10, Line 210: query
query execution? (as opposed to the query being closed)
PS10, Line 299: /// True if execution has completed, false otherwise.
: bool execution_completed_ = false;
doesn't appear to be used
http://gerrit.cloudera.org:8080/#/c/6535/10/be/src/runtime/descriptors.cc
File be/src/runtime/descriptors.cc:
PS10, Line 547: ReleaseResources
why is this called ReleaseResources and not Close? What are we saying the
difference between those two terms are?
http://gerrit.cloudera.org:8080/#/c/6535/10/be/src/runtime/fragment-instance-state.cc
File be/src/runtime/fragment-instance-state.cc:
Like QES, I think the lifecycle state machine could be more clearly defined for
FIS, but that can be refined later.
PS10, Line 95: if (!status.ok() && !status.IsCancelled() &&
!status.IsMemLimitExceeded()) {
: // Log error message in addition to returning in Status.
Queries that do not fetch
: // results (e.g. insert) may not receive the message
directly and can only retrieve
: // the log.
: runtime_state_->LogError(status.msg());
: }
you should delete this when rebasing e0d1db5
PS10, Line 104: // TODO: deal with final report failure, otherwise the
coordinator doesn't know
: // we're done
is that a regression or something that didn't work before as well?
PS10, Line 213: // TODO: why is this here? FragmentComplete() al
[Impala-ASF-CR] IMPALA-5235: Initialize resourceProfile with a dummy value
Impala Public Jenkins has posted comments on this change. Change subject: IMPALA-5235: Initialize resourceProfile_ with a dummy value .. Patch Set 1: Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/519/ -- To view, visit http://gerrit.cloudera.org:8080/6750 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I02de2ca1f627d7f61921840722661a5323e91579 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: anujphadke Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
Impala Public Jenkins has posted comments on this change. Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter .. Patch Set 2: Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/518/ -- To view, visit http://gerrit.cloudera.org:8080/6762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
Michael Ho has posted comments on this change. Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/6762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Michael Ho Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Impala Public Jenkins has posted comments on this change. Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. Patch Set 4: Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/517/ -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Dan Hecht has posted comments on this change. Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5162,IMPALA-5163: stress test support on secure clusters
Michael Brown has uploaded a new change for review. http://gerrit.cloudera.org:8080/6763 Change subject: IMPALA-5162,IMPALA-5163: stress test support on secure clusters .. IMPALA-5162,IMPALA-5163: stress test support on secure clusters This patch adds support for running the stress test (concurrent_select.py) and loading nested data (load_nested.py) into a Kerberized, SSL-enabled Impala cluster. It assumes the calling user already has a valid Kerberos ticket. One way to do that is: 1. Get access to a keytab and krb5.config 2. Set KRB5_CONFIG and KRB5CCNAME appropriately 3. Run kinit(1) 4. Run load_nested.py and/or concurrent_select.py within this environment. Because our Python clients already support Kerberos and SSL, we simply need to make sure to use the correct options when calling the entry points and initializing the clients: Impala: Impyla Hive: Impyla HDFS: hdfs.ext.kerberos.KerberosClient With this patch, I was able to manually do a short concurrent_select.py run against a secure cluster without connection or auth errors, and I was able to do the same with load_nested.py for a cluster that already had TPC-H loaded. Follow-ons for future cleanup work: IMPALA-5263: support CA bundles when running stress test against SSL'd Impala IMPALA-5264: fix InsecurePlatformWarning under stress test with SSL Change-Id: I0daad57bb8ceeb5071b75125f11c1997ed7e0179 --- M testdata/bin/load_nested.py M tests/comparison/cli_options.py M tests/comparison/cluster.py M tests/comparison/db_connection.py M tests/stress/concurrent_select.py 5 files changed, 61 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/6763/1 -- To view, visit http://gerrit.cloudera.org:8080/6763 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0daad57bb8ceeb5071b75125f11c1997ed7e0179 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Michael Brown
[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
Attila Jeges has uploaded a new patch set (#2). Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter .. IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter HdfsSequenceTableWriter::ConsumeRow() function dereferenced a pointer that pointed to a previously deallocated memory (which belonged to an out of scope string object). This caused the ASAN build to fail. The fix was verified by running TestTableWriters.test_seq_writer and TestTableWriters.test_seq_writer_hive_compatibility end-to-end tests against the ASAN build. These tests consistently crashed impalad before the fix. Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b --- M be/src/exec/hdfs-sequence-table-writer.cc 1 file changed, 4 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/6762/2 -- To view, visit http://gerrit.cloudera.org:8080/6762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
Hello Michael Ho, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6762 to look at the new patch set (#2). Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter .. IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter HdfsSequenceTableWriter::ConsumeRow() function dereferenced a pointer that pointed to a previously deallocated memory (which belonged to an out of scope string object). This caused the ASAN build to fail. The fix was verified by running TestTableWriters.test_seq_writer and TestTableWriters.test_seq_writer_hive_compatibility end-to-end tests against the ASAN build. These tests consistently crashed impalad before the fix. Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b --- M be/src/exec/hdfs-sequence-table-writer.cc 1 file changed, 4 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/6762/2 -- To view, visit http://gerrit.cloudera.org:8080/6762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Michael Ho has posted comments on this change. Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/6761/3/be/src/exprs/aggregate-functions-ir.cc File be/src/exprs/aggregate-functions-ir.cc: PS3, Line 1295: reinterpret_cast > oops, i had to add this cast. Michael do you still prefer to call CopyFrom Hmm...it still looks cleaner to call CopyFrom() if possible. -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
Attila Jeges has posted comments on this change. Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter .. Patch Set 1: (1 comment) > (1 comment) > > I assumed you have verified the fix with ASAN build, right ? Yes, added testing details to the commit message. http://gerrit.cloudera.org:8080/#/c/6762/1/be/src/exec/hdfs-sequence-table-writer.cc File be/src/exec/hdfs-sequence-table-writer.cc: Line 306: value_length = text.size(); > DCHECK_EQ(value_length, row_buf_.Size()); Done -- To view, visit http://gerrit.cloudera.org:8080/6762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Michael Ho Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Dan Hecht has posted comments on this change. Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/6761/3/be/src/exprs/aggregate-functions-ir.cc File be/src/exprs/aggregate-functions-ir.cc: PS3, Line 1295: reinterpret_cast oops, i had to add this cast. Michael do you still prefer to call CopyFrom here? http://gerrit.cloudera.org:8080/#/c/6761/3/be/src/exprs/udf-builtins-ir.cc File be/src/exprs/udf-builtins-ir.cc: PS3, Line 512: reinterpret_cast(str.c_str()) and here. -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Hello Michael Ho, Matthew Jacobs, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6761 to look at the new patch set (#3). Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded We need to check for AllocateLocal() returning NULL. CopyFrom() takes care of that for us. Also adjust a few other places in the code base that didn't have the check. The new test reproduces the crash, but in order to get this test file to execute, I had to move the xfail to be a function decorator. Apparently xfail as a statement causes the test to not run at all. We should run all of these queries even if they are non-determistic to at least verify that impalad does not crash. Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 --- M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/hive-udf-call.cc M be/src/exprs/udf-builtins-ir.cc M be/src/udf/uda-test.cc M be/src/udf/udf-test.cc M be/src/udf_samples/uda-sample.cc M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-update.test M tests/custom_cluster/test_alloc_fail.py 8 files changed, 27 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/6761/3 -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Matthew Jacobs has posted comments on this change. Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Matthew Jacobs has posted comments on this change. Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/6761/1/be/src/udf_samples/uda-sample.cc File be/src/udf_samples/uda-sample.cc: Line 86: void StringConcatUpdate(FunctionContext* context, const StringVal& arg1, > How do make changes to that? I think you should have access to push to that repo. Github also has an editor so you can click on the edit icon directly on the page and make edits, then commit it. We should kill this repo soon though this is clearly burdensome. -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Michael Ho has posted comments on this change. Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
Michael Ho has posted comments on this change. Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter .. Patch Set 1: Code-Review+2 (1 comment) I assumed you have verified the fix with ASAN build, right ? http://gerrit.cloudera.org:8080/#/c/6762/1/be/src/exec/hdfs-sequence-table-writer.cc File be/src/exec/hdfs-sequence-table-writer.cc: Line 306: value_length = text.size(); DCHECK_EQ(value_length, row_buf_.Size()); -- To view, visit http://gerrit.cloudera.org:8080/6762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Michael Ho Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Dan Hecht has posted comments on this change. Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/6761/1//COMMIT_MSG Commit Message: PS1, Line 10: a few other places > Thanks for doing it. Please feel free to ignore but there are a few places I did the ones where we can get rid of the is_null check at the callsite. For some of them, we still need the is_null check, so it seems clearest to not use CopyFrom() in those cases. PS1, Line 14: marker > I believe in python this is called a decorator. Done http://gerrit.cloudera.org:8080/#/c/6761/1/be/src/udf_samples/uda-sample.cc File be/src/udf_samples/uda-sample.cc: Line 86: void StringConcatUpdate(FunctionContext* context, const StringVal& arg1, > until we remove the impala-udf-sample repo, we should probably keep that co How do make changes to that? -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Hello Michael Ho, Matthew Jacobs, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6761 to look at the new patch set (#2). Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded We need to check for AllocateLocal() returning NULL. CopyFrom() takes care of that for us. Also adjust a few other places in the code base that didn't have the check. The new test reproduces the crash, but in order to get this test file to execute, I had to move the xfail to be a function decorator. Apparently xfail as a statement causes the test to not run at all. We should run all of these queries even if they are non-determistic to at least verify that impalad does not crash. Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 --- M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/hive-udf-call.cc M be/src/exprs/udf-builtins-ir.cc M be/src/udf/uda-test.cc M be/src/udf/udf-test.cc M be/src/udf_samples/uda-sample.cc M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-update.test M tests/custom_cluster/test_alloc_fail.py 8 files changed, 25 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/6761/2 -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-4864 Speed up single slot predicates with dictionaries
Zach Amsden has uploaded a new patch set (#4). Change subject: IMPALA-4864 Speed up single slot predicates with dictionaries .. IMPALA-4864 Speed up single slot predicates with dictionaries When dictionaries are present we can pre-evaluate conjuncts against the dictionary values and simply look up the result. Status of this diff: Compiles and starts. Bitmap tests for new functionality pass. FE and BE tests passing. Going to rebase to try to get past a bug that is firing consistently on my branch as a few fixes have gone in. Change-Id: I65981c89e5292086809ec1268f5a273f4c1fe054 --- M be/src/exec/hdfs-parquet-scanner-ir.cc M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-parquet-scanner.h M be/src/exec/parquet-column-readers.cc M be/src/exec/parquet-column-readers.h M be/src/exec/parquet-scratch-tuple-batch.h M be/src/util/bitmap-test.cc M be/src/util/bitmap.h M be/src/util/dict-encoding.h M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test 11 files changed, 422 insertions(+), 182 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/6726/4 -- To view, visit http://gerrit.cloudera.org:8080/6726 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I65981c89e5292086809ec1268f5a273f4c1fe054 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Zach Amsden
[Impala-ASF-CR] IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter
Attila Jeges has uploaded a new change for review. http://gerrit.cloudera.org:8080/6762 Change subject: IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter .. IMPALA-5261: Heap use-after-free in HdfsSequenceTableWriter HdfsSequenceTableWriter::ConsumeRow() function dereferenced a pointer that pointed to a previously deallocated memory (which belonged to an out of scope string object). This caused the ASAN build to fail. Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b --- M be/src/exec/hdfs-sequence-table-writer.cc 1 file changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/6762/1 -- To view, visit http://gerrit.cloudera.org:8080/6762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id339247f892710529d8ad56dd1e98eadbf32900b Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Attila Jeges
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Michael Ho has posted comments on this change. Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. Patch Set 1: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/6761/1//COMMIT_MSG Commit Message: PS1, Line 10: a few other places Thanks for doing it. Please feel free to ignore but there are a few places in the code which can be converted to using StringVal::CopyFrom() even though they have the check: AggregateFunctions::ReservoirSampleFinalize(), CastFunctions::CastToChar(), udf-builtins-ir.cc, ? -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Michael Ho Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Matthew Jacobs has posted comments on this change. Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. Patch Set 1: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/6761/1//COMMIT_MSG Commit Message: PS1, Line 14: marker I believe in python this is called a decorator. http://gerrit.cloudera.org:8080/#/c/6761/1/be/src/udf_samples/uda-sample.cc File be/src/udf_samples/uda-sample.cc: Line 86: void StringConcatUpdate(FunctionContext* context, const StringVal& arg1, until we remove the impala-udf-sample repo, we should probably keep that code up to date as well: https://github.com/cloudera/impala-udf-samples/blob/master/uda-sample.cc -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Matthew Jacobs Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem limit exceeded
Dan Hecht has uploaded a new change for review. http://gerrit.cloudera.org:8080/6761 Change subject: IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded .. IMPALA-5252: Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded We need to check for AllocateLocal() returning NULL. CopyFrom() takes care of that for us. Also adjust a few other places in the code base that didn't have the check. The new test reproduces the crash, but in order to get this test file to execute, I had to move the xfail to be a function marker. Apparently xfail as a statement causes the test to not run at all. We should run all of these queries even if they are non-determistic to at least verify that impalad does not crash. Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 --- M be/src/exprs/hive-udf-call.cc M be/src/udf/uda-test.cc M be/src/udf/udf-test.cc M be/src/udf_samples/uda-sample.cc M testdata/workloads/functional-query/queries/QueryTest/alloc-fail-update.test M tests/custom_cluster/test_alloc_fail.py 6 files changed, 23 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/6761/1 -- To view, visit http://gerrit.cloudera.org:8080/6761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iafefef24479164cc4d2b99191d2de28eb8b311b6 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Dan Hecht
[Impala-ASF-CR] Bump Kudu version to 238249c
Matthew Jacobs has posted comments on this change. Change subject: Bump Kudu version to 238249c .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/6718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I92587a8061ce70ecd9dac4889bda550636982767 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Matthew Jacobs Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3742: Partitions and sort INSERTs for Kudu tables
Marcel Kornacker has posted comments on this change.
Change subject: IMPALA-3742: Partitions and sort INSERTs for Kudu tables
..
Patch Set 7: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/6559/7/fe/src/main/java/org/apache/impala/analysis/KuduPartitionExpr.java
File fe/src/main/java/org/apache/impala/analysis/KuduPartitionExpr.java:
Line 37: * a given row. The children of this Expr produce the values for the
partition columns.
is it documented in some class header that values outside the legal range
return a -1? probably most appropriate in the .h file.
Line 74: if (i != 0) {
single line
--
To view, visit http://gerrit.cloudera.org:8080/6559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I84ce0032a1b10958fdf31faef225372c5c38fdc4
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall
Gerrit-Reviewer: Alex Behm
Gerrit-Reviewer: Dimitris Tsirogiannis
Gerrit-Reviewer: Marcel Kornacker
Gerrit-Reviewer: Matthew Jacobs
Gerrit-Reviewer: Mostafa Mokhtar
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Mention Kerberos and TLS for Kudu security
Matthew Jacobs has posted comments on this change. Change subject: [DOCS] Mention Kerberos and TLS for Kudu security .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/6634 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I1266ad38468ef2e987aff54db35e6cafdacc Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: John Russell Gerrit-Reviewer: Ambreen Kazi Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: John Russell Gerrit-Reviewer: Matthew Jacobs Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5188: Add slot sorting in TupleDescriptor::LayoutEquals()
Taras Bobrovytsky has posted comments on this change. Change subject: IMPALA-5188: Add slot sorting in TupleDescriptor::LayoutEquals() .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/6610/6/be/src/runtime/descriptors.cc File be/src/runtime/descriptors.cc: Line 678: for (SlotDescriptor* slot: slots()) sorted_slots[slot->slot_idx_] = slot; It turns out that the reason why the GVO for this patch is failing is that slot_idx does not always start with 0. The following query produces a tuple with a single slot with a slot_idx=1: select c_custkey, v1.cnt from tpch_nested_parquet.customer c inner join (select count(*) cnt from c.c_orders where false) v1 where c_custkey < 10 The tuple looks like this: Tuple(id=0 size=25 slots=[Slot(id=3 type=BIGINT col_path=[0] offset=16 null=(offset=24 mask=2) slot_idx=1 field_idx=-1)] tuple_path=[]) -- To view, visit http://gerrit.cloudera.org:8080/6610 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I426ad244678dbfe517262dfb7bbf4adc0247a35e Gerrit-PatchSet: 6 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Taras Bobrovytsky Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4318: Kudu support for CREATE EXTERNAL TABLE AS SELECT
Zach Amsden has abandoned this change. Change subject: IMPALA-4318: Kudu support for CREATE EXTERNAL TABLE AS SELECT .. Abandoned Not doing at this time. -- To view, visit http://gerrit.cloudera.org:8080/6261 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I9aa82809a6c0c5e6386827314b7e5b520c1a6633 Gerrit-PatchSet: 14 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Zach Amsden Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Marcel Kornacker Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Zach Amsden
[Impala-ASF-CR] IMPALA-5180: Don't use non-deterministic exprs in partition pruning
Zach Amsden has uploaded a new patch set (#8). Change subject: IMPALA-5180: Don't use non-deterministic exprs in partition pruning .. IMPALA-5180: Don't use non-deterministic exprs in partition pruning Non-deterministic exprs which evaluate as constant should not be used during HDFS partition pruning. We consider Exprs which have no SlotRefs as bound by default, and thus we end up trying to apply them indisrciminately. Constant propagation makes this situation easier to run into and the behavior is rather unexpected. The fix for now is to explicitly disallow non-deterministic Exprs in partition pruning. Change-Id: I91054c6bf017401242259a1eff5e859085285546 --- M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/hdfs.test 4 files changed, 35 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/6575/8 -- To view, visit http://gerrit.cloudera.org:8080/6575 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91054c6bf017401242259a1eff5e859085285546 Gerrit-PatchSet: 8 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Zach Amsden Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Zach Amsden
[Impala-ASF-CR] IMPALA-5246: UDF's Close() should handle Expr's preparation failure
Michael Ho has uploaded a new change for review. http://gerrit.cloudera.org:8080/6757 Change subject: IMPALA-5246: UDF's Close() should handle Expr's preparation failure .. IMPALA-5246: UDF's Close() should handle Expr's preparation failure UDF may fail to initialize due low memory limit or other reasons In which case, its Prepare() function may not have been called and its thread local state may not be initialized. The Close() functions of some of the built-in and test-udf made the wrong assumption that the thread local states are always initialized. This may lead to de-referencing null pointer in Close(). This change fixes this issue by checking the thread local state is not null and returns early if so. Change-Id: Id2c689246ed4f8dd38f104fa35904f3926a7039c --- M be/src/exprs/case-expr.cc M be/src/exprs/math-functions-ir.cc M be/src/exprs/string-functions-ir.cc M be/src/exprs/timestamp-functions.cc M be/src/exprs/utility-functions.cc M be/src/testutil/test-udfs.cc 6 files changed, 17 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/6757/1 -- To view, visit http://gerrit.cloudera.org:8080/6757 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id2c689246ed4f8dd38f104fa35904f3926a7039c Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Michael Ho
[Impala-ASF-CR] IMPALA-5180: Don't use non-deterministic exprs in partition pruning
Zach Amsden has uploaded a new patch set (#7). Change subject: IMPALA-5180: Don't use non-deterministic exprs in partition pruning .. IMPALA-5180: Don't use non-deterministic exprs in partition pruning Non-deterministic exprs which evaluate as constant should not be used during HDFS partition pruning. We consider Exprs which have no SlotRefs as bound by default, and thus we end up trying to apply them indisrciminately. Constant propagation makes this situation easier to run into and the behavior is rather unexpected. The fix for now is to explicitly disallow non-deterministic Exprs in partition pruning. Change-Id: I91054c6bf017401242259a1eff5e859085285546 --- M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/hdfs.test 4 files changed, 36 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/6575/7 -- To view, visit http://gerrit.cloudera.org:8080/6575 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91054c6bf017401242259a1eff5e859085285546 Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Zach Amsden Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Zach Amsden
[Impala-ASF-CR] IMPALA-3742: Partitions and sort INSERTs for Kudu tables
Thomas Tauber-Marshall has uploaded a new patch set (#7). Change subject: IMPALA-3742: Partitions and sort INSERTs for Kudu tables .. IMPALA-3742: Partitions and sort INSERTs for Kudu tables Bulk DMLs (INSERT, UPSERT, UPDATE, and DELETE) for Kudu are currently painful because we just send rows randomly, which creates a lot of work for Kudu since it partitions and sorts data before writing, causing writes to be slow and leading to timeouts. We can alleviate this by sending the rows to Kudu already partitioned and sorted. This patch partitions and sorts rows according to Kudu's partitioning scheme for INSERTs and UPSERTs. A followup patch will handle UPDATE and DELETE. It accomplishes this by inserting an exchange node and a sort node into the plan before the operation. Both the exchange and the sort are given a KuduPartitionExpr which takes a row and calls into the Kudu client to return its partition number. It also disallows INSERT hints for Kudu tables, since the hints that we support (SHUFFLE, CLUSTER, SORTBY), so longer make sense. Testing: - Updated planner tests. - Ran the Kudu functional tests. - Ran performance tests demonstrating that we can now handle much larger inserts without having timeouts. Change-Id: I84ce0032a1b10958fdf31faef225372c5c38fdc4 --- M be/src/exec/kudu-table-sink.cc M be/src/exec/kudu-util.cc M be/src/exec/kudu-util.h M be/src/exprs/CMakeLists.txt M be/src/exprs/expr-context.h M be/src/exprs/expr.cc A be/src/exprs/kudu-partition-expr.cc A be/src/exprs/kudu-partition-expr.h M be/src/runtime/coordinator.cc M be/src/runtime/data-stream-sender.cc M be/src/runtime/data-stream-sender.h M be/src/scheduling/scheduler.cc M bin/impala-config.sh M common/thrift/Exprs.thrift M common/thrift/Partitions.thrift M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java A fe/src/main/java/org/apache/impala/analysis/KuduPartitionExpr.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/planner/DataPartition.java M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeUpsertStmtTest.java M testdata/workloads/functional-planner/queries/PlannerTest/kudu-upsert.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test 27 files changed, 618 insertions(+), 171 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6559/7 -- To view, visit http://gerrit.cloudera.org:8080/6559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I84ce0032a1b10958fdf31faef225372c5c38fdc4 Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Marcel Kornacker Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] Marcel's review 2.0
Thomas Tauber-Marshall has abandoned this change. Change subject: Marcel's review 2.0 .. Abandoned Forgot to squash -- To view, visit http://gerrit.cloudera.org:8080/6753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Ieb788780ad570b84d5660a6f6ca0c9c24f3eb016 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Alex Behm
