[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4568/ -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 28 Jun 2019 05:47:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8689: test hive impala interop failing with "Timeout >7200s"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13755 ) Change subject: IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 Gerrit-Change-Number: 13755 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 28 Jun 2019 05:42:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8689: test hive impala interop failing with "Timeout >7200s"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13755 ) Change subject: IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4569/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 Gerrit-Change-Number: 13755 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 28 Jun 2019 05:42:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8689: test hive impala interop failing with "Timeout >7200s"
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13755 ) Change subject: IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" .. Patch Set 2: Looked like some maven flakiness -- To view, visit http://gerrit.cloudera.org:8080/13755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 Gerrit-Change-Number: 13755 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 28 Jun 2019 05:42:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8689: test hive impala interop failing with "Timeout >7200s"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13755 ) Change subject: IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4567/ -- To view, visit http://gerrit.cloudera.org:8080/13755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 Gerrit-Change-Number: 13755 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 28 Jun 2019 03:37:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8713: fix stack overflow in unhex()
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13743 ) Change subject: IMPALA-8713: fix stack overflow in unhex() .. IMPALA-8713: fix stack overflow in unhex() Write the results into the output heap buffer instead of into a temporary stack buffer. No additional memory is used because AnyValUtil::FromBuffer() allocated a temporary buffer anyway. Testing: Added a targeted test to expr-test that caused a crash before this fix. Change-Id: Ie0c1760511a04c0823fc465cf6e529e9681b2488 Reviewed-on: http://gerrit.cloudera.org:8080/13743 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exprs/expr-test.cc M be/src/exprs/math-functions-ir.cc 2 files changed, 6 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie0c1760511a04c0823fc465cf6e529e9681b2488 Gerrit-Change-Number: 13743 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8713: fix stack overflow in unhex()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13743 ) Change subject: IMPALA-8713: fix stack overflow in unhex() .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0c1760511a04c0823fc465cf6e529e9681b2488 Gerrit-Change-Number: 13743 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Jun 2019 03:02:00 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-8484: Run queries on disjoint executor groups
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/13550 )
Change subject: WIP IMPALA-8484: Run queries on disjoint executor groups
..
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13550/7/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/13550/7/be/src/scheduling/admission-controller.cc@1336
PS7, Line 1336: for (auto& it: queue_node->per_group_schedules) {
> Are we still considering groups in a deterministic order? Is it deliberate
Yes, per_group_schedules is a map, so the order is deterministic and this is
what I intended, so that an executor group we can free executor groups easier
if we find them underutilized. I'm happy to evaluate this more if you feel that
we should consider balancing the work evenly.
--
To view, visit http://gerrit.cloudera.org:8080/13550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b
Gerrit-Change-Number: 13550
Gerrit-PatchSet: 7
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 28 Jun 2019 00:56:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-8484: Run queries on disjoint executor groups
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13550 ) Change subject: WIP IMPALA-8484: Run queries on disjoint executor groups .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/13550/7/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/13550/7/be/src/scheduling/admission-controller.cc@496 PS7, Line 496: const int64_t max_requests = pool_cfg.max_requests; > This code looks weird in that most other code now uses GetMaxRequestsForPoo I agree that this still needs some thought. Should we use GetMaxRequestsForPool(group_size) here instead? I generally feel that admitting more concurrent queries to larger clusters might not increase the throughput beyond a certain point. To model this, we could also add a field "admit_num_limit" to TBackendDescriptor, make it default to num_cores and expose it via a flag. -- To view, visit http://gerrit.cloudera.org:8080/13550 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b Gerrit-Change-Number: 13550 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 28 Jun 2019 00:54:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-8484: Run queries on disjoint executor groups
Andrew Sherman has posted comments on this change. (
http://gerrit.cloudera.org:8080/13550 )
Change subject: WIP IMPALA-8484: Run queries on disjoint executor groups
..
Patch Set 7:
(3 comments)
a few early comments/questions
http://gerrit.cloudera.org:8080/#/c/13550/7/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/13550/7/be/src/scheduling/admission-controller.cc@496
PS7, Line 496: const int64_t max_requests = pool_cfg.max_requests;
This code looks weird in that most other code now uses GetMaxRequestsForPool()
rather then looking at pool config fields. It also seems inconsistent in that
we are ignoring max_running_queries_multiple here, but are still using
GetMaxMemForPool() in HasAvailableMemResources().
http://gerrit.cloudera.org:8080/#/c/13550/7/be/src/scheduling/admission-controller.cc@1336
PS7, Line 1336: for (auto& it: queue_node->per_group_schedules) {
Are we still considering groups in a deterministic order? Is it deliberate that
we would then tend to fill some groups first? Or do we want to balance work
across groups?
http://gerrit.cloudera.org:8080/#/c/13550/7/be/src/scheduling/executor-group.h
File be/src/scheduling/executor-group.h:
http://gerrit.cloudera.org:8080/#/c/13550/7/be/src/scheduling/executor-group.h@93
PS7, Line 93: /// executors running on the same host (e.g. during tests) are
counted individually.
Maybe no need to mention backend descriptors here
--
To view, visit http://gerrit.cloudera.org:8080/13550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b
Gerrit-Change-Number: 13550
Gerrit-PatchSet: 7
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 28 Jun 2019 00:05:04 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-8484: Run queries on disjoint executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13550 ) Change subject: WIP IMPALA-8484: Run queries on disjoint executor groups .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3777/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13550 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b Gerrit-Change-Number: 13550 Gerrit-PatchSet: 8 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 23:52:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8636: Implement INSERT for insert-only ACID tables
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13559 )
Change subject: IMPALA-8636: Implement INSERT for insert-only ACID tables
..
Patch Set 10:
(13 comments)
I had to refresh my memory on how the query lifecycle worked for inserts, but I
think I paged enough back in to have some valid thoughts on it.
The biggest problem I think is the one that you identified - that the
transaction commit, Impala catalog and HMS updates are not atomic from the
point of view of Impala, so we're opening ourselves up to various anomalies. I
haven't really thought through exactly what anomalies are possible, but it
would be good to avoid them entirely, if possible.
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/exec/hdfs-table-sink.h
File be/src/exec/hdfs-table-sink.h:
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/exec/hdfs-table-sink.h@264
PS10, Line 264: long
int64_t?
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/exec/hdfs-table-sink.cc
File be/src/exec/hdfs-table-sink.cc:
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/exec/hdfs-table-sink.cc@500
PS10, Line 500: // However, for transactional tables we should create a new
empty base directory.
Why? I assume there is some good reason but it's not immediately obvious to me.
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/exec/hdfs-table-sink.cc@700
PS10, Line 700: if (IsTransactional()) return true;
This one seems more obvious to me, but makes me think that the class comment
should have a brief expectation of the directory layout and behaviour of ACID
inserts. Or a pointer to something that explains it.
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/runtime/coordinator.cc@571
PS10, Line 571: FinalizeHdfsInsert
We should maybe rename this to FinalizeHdfsDml(), now or later.
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/runtime/coordinator.cc@571
PS10, Line 571: Status Coordinator::FinalizeHdfsInsert() {
I think we should probably do the transaction abort in this function, since it
will happen asynchronously and not depend on the client unregistering the
query. I think it fits conceptually with removing the staging directory and
that cleanup.
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/client-request-state.h@307
PS10, Line 307: /// True if there is an open transaction.
Hive ACID transaction, just to be clear.
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/client-request-state.h@308
PS10, Line 308: bool in_transaction_ = false;
I think this would best fit in DmlExecState.
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/client-request-state.cc@720
PS10, Line 720: DCHECK(exec_request().__isset.transaction_id);
I think we would prefer to abort the transaction earlier in the query
lifecycle. Query unregistration does not necessarily happen in a timely fashion
because it depends on client RPCs. Coordinator::FinalizeHdfsInsert() is maybe
the right place.
This could also be a helper, maybe - Done() is getting to the point where it
doesn't fit on a screen.
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/client-request-state.cc@820
PS10, Line 820: //TODO: HMS is not updated at this point, only in
UpdateCatalog(). But
Zoli explained this to me out-of-band. I think I agree that having the catalog
commit the transaction is the right approach, since we'd want transaction
commit and the Impala catalog update to be an atomic operation from the
point-of-view of impalads.
It looks like there's an add_dynamic_partitions() method that takes a
transaction ID, so I think in theory we could solve that partition creation
problem by having the impalad create the partitions in the transaction, but
that would still leave a window of inconsistency.
It's a little asymmetrical to have the Impala start the transaction and
catalogd commit it, but it seems less weird than the consistency issues.
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/frontend.h
File be/src/service/frontend.h:
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/frontend.h@170
PS10, Line 170: /// Commits transaction with the given transaction id.
Are there any invariants we should document for these methods? I guess we're
just assuming that this coordinator had opened a transaction previous.
Are there any interesting failure modes to document?
http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/frontend.h@171
PS10, Line 171: long
I think we want to use int64_t, to match the thr
[Impala-ASF-CR] WIP IMPALA-8484: Run queries on disjoint executor groups
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13550 ) Change subject: WIP IMPALA-8484: Run queries on disjoint executor groups .. Patch Set 7: (21 comments) PS8 is to please pyflake and clang-tidy http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py File bin/start-impala-cluster.py: http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@274 PS7, Line 274: > flake8: E251 unexpected spaces around keyword / parameter equals Done http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@274 PS7, Line 274: > flake8: E251 unexpected spaces around keyword / parameter equals Done http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@439 PS7, Line 439: > flake8: E251 unexpected spaces around keyword / parameter equals Done http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@439 PS7, Line 439: > flake8: E251 unexpected spaces around keyword / parameter equals Done http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@453 PS7, Line 453: > flake8: E251 unexpected spaces around keyword / parameter equals Done http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@453 PS7, Line 453: > flake8: E251 unexpected spaces around keyword / parameter equals Done http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@713 PS7, Line 713: ; > flake8: E703 statement ends with a semicolon Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/custom_cluster/test_auto_scaling.py File tests/custom_cluster/test_auto_scaling.py: http://gerrit.cloudera.org:8080/#/c/13550/7/tests/custom_cluster/test_auto_scaling.py@21 PS7, Line 21: import json > flake8: F401 'json' imported but unused Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/custom_cluster/test_auto_scaling.py@22 PS7, Line 22: import requests > flake8: F401 'requests' imported but unused Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/custom_cluster/test_auto_scaling.py@31 PS7, Line 31: class TestAutoScaling(CustomClusterTestSuite): > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/custom_cluster/test_auto_scaling.py@62 PS7, Line 62: 3 > flake8: E225 missing whitespace around operator Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/auto_scaler.py File tests/util/auto_scaler.py: http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/auto_scaler.py@31 PS7, Line 31: class AutoScaler(object): > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/auto_scaler.py@181 PS7, Line 181: = > flake8: E711 comparison to None should be 'if cond is None:' Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/auto_scaler.py@293 PS7, Line 293: if __name__ == "__main__": > flake8: E305 expected 2 blank lines after class or function definition, fou Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py File tests/util/concurrent_workload.py: http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@24 PS7, Line 24: import signal > flake8: F401 'signal' imported but unused Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@27 PS7, Line 27: import _strptime > flake8: F401 '_strptime' imported but unused Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@35 PS7, Line 35: class ConcurrentWorkload(object): > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@68 PS7, Line 68: e > flake8: F841 local variable 'e' is assigned to but never used Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@109 PS7, Line 109: > flake8: E201 whitespace after '[' Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@118 PS7, Line 118: = > flake8: E711 comparison to None should be 'if cond is None:' Done http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@149 PS7, Line 149: if __name__ == "__main__": > flake8: E305 expected 2 blank lines after class or function definition, fou Done -- To view, visit http://gerrit.cloudera.org:8080/13550 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b Gerrit-Change-Number: 13550 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 23:13:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-8484: Run queries on disjoint executor groups
Hello Andrew Sherman, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13550 to look at the new patch set (#8). Change subject: WIP IMPALA-8484: Run queries on disjoint executor groups .. WIP IMPALA-8484: Run queries on disjoint executor groups This change adds support running queries of a single admission control pool on one of several, disjoint sets of executors called 'executor groups'. This change also re-schedules queries before admitting them if the admission controller detects that the cluster membership has changed. TODO: Elaborate on change and design Known limitations: - When using executor groups, only a single coordinator and a single AC pool are supported. Open questions and issues: - Executor groups can become unhealthy when one or more of their executors fail. We should build support to heal or decommission unhealthy groups in a subsequent change. Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b --- M be/src/runtime/exec-env.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/cluster-membership-mgr-test.cc M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/runtime-profile.h M bin/start-impala-cluster.py M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/StatestoreService.thrift M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/common/resource_pool_config.py M tests/custom_cluster/test_admission_controller.py A tests/custom_cluster/test_auto_scaling.py A tests/util/auto_scaler.py A tests/util/concurrent_workload.py M www/backends.tmpl 35 files changed, 1,523 insertions(+), 432 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/13550/8 -- To view, visit http://gerrit.cloudera.org:8080/13550 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b Gerrit-Change-Number: 13550 Gerrit-PatchSet: 8 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3776/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 22:57:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 22:24:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4568/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 22:24:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. IMPALA-8612: Fix sporadic NPE when dropping an authorized table In the analyze() function of DropTableOrViewStmt it's possible that serverName_ is not set when analyzer.getTable() throws. As a result when the Catalog executes the drop table DDL it runs into a failing Precondition check and throws a NullPointerException when updating user privileges. Note, to run into the NPE it's required to have authorization enabled. Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Reviewed-on: http://gerrit.cloudera.org:8080/13508 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java 1 file changed, 9 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 4: (9 comments) Cleaned it up a bit with better naming for user-facing configs. I think this has reasonable test coverage and is ready for review (all the core tests pass on this patch). http://gerrit.cloudera.org:8080/#/c/13746/3/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/13746/3/shell/impala_client.py@375 PS3, Line 375: > flake8: E261 at least two spaces before inline comment Done http://gerrit.cloudera.org:8080/#/c/13746/3/shell/impala_client.py@379 PS3, Line 379: > flake8: E222 multiple spaces after operator Done http://gerrit.cloudera.org:8080/#/c/13746/3/shell/impala_client.py@384 PS3, Line 384: : > flake8: E203 whitespace before ':' Done http://gerrit.cloudera.org:8080/#/c/13746/3/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/13746/3/shell/impala_shell.py@537 PS3, Line 537: > flake8: E501 line too long (93 > 90 characters) Done http://gerrit.cloudera.org:8080/#/c/13746/3/shell/impala_shell.py@797 PS3, Line 797: > flake8: E501 line too long (102 > 90 characters) Done http://gerrit.cloudera.org:8080/#/c/13746/3/shell/option_parser.py File shell/option_parser.py: http://gerrit.cloudera.org:8080/#/c/13746/3/shell/option_parser.py@266 PS3, Line 266: > flake8: E501 line too long (91 > 90 characters) Done http://gerrit.cloudera.org:8080/#/c/13746/3/tests/common/test_dimensions.py File tests/common/test_dimensions.py: http://gerrit.cloudera.org:8080/#/c/13746/3/tests/common/test_dimensions.py@120 PS3, Line 120: > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/13746/3/tests/common/test_dimensions.py@123 PS3, Line 123: > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/13746/3/tests/shell/util.py File tests/shell/util.py: http://gerrit.cloudera.org:8080/#/c/13746/3/tests/shell/util.py@153 PS3, Line 153: > flake8: E501 line too long (93 > 90 characters) Done -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 27 Jun 2019 22:21:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13746 to look at the new patch set (#4). Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. IMPALA-8717: impala-shell support for HS2 HTTP endpoint Adds impala-shell support to connect to HiveServer2 HTTP endpoint. Relies on toolchain change at https://gerrit.cloudera.org/#/c/13725/. Adds a new shell flag --http that tells it to pick the appropriate client transport. Example usages: --- impala-shell --protocol='hs2' --http (No auth) impala-shell --protocol='hs2' --http --ldap -u. (PLAIN auth) impala-shell --protocol-'hs2' --http --ssl --ca_cert... (TLS) impala-shell --protocol='hs2' --http --ldap --ssl --ca_cert... (LDAP + TLS) Limitations: --- - Does not work with the default --protocol='beeswax' due to lack of server side support. Appropriate error message is thrown. - Does not support Kerberos (-k) due to lack ot SPNEGO support. Testing: - Parameterized existing shell tests to support this combination. - Manually tested plain LDAP and LDAP + TLS combinations due to lack of LDAP support in tests. Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 --- M be/src/service/impala-server.cc M bin/impala-config.sh M shell/impala_client.py M shell/impala_shell.py M shell/option_parser.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/conftest.py M tests/custom_cluster/test_client_ssl.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 13 files changed, 134 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/13746/4 -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8689: test hive impala interop failing with "Timeout >7200s"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13755 ) Change subject: IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3775/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 Gerrit-Change-Number: 13755 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 22:08:08 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-8484: Run queries on disjoint executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13550 ) Change subject: WIP IMPALA-8484: Run queries on disjoint executor groups .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3774/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13550 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b Gerrit-Change-Number: 13550 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 22:03:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8689: test hive impala interop failing with "Timeout >7200s"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13755 ) Change subject: IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3773/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 Gerrit-Change-Number: 13755 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 21:54:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8689: test hive impala interop failing with "Timeout >7200s"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13755 ) Change subject: IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4567/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 Gerrit-Change-Number: 13755 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 21:48:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8689: test hive impala interop failing with "Timeout >7200s"
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/13755 ) Change subject: IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 Gerrit-Change-Number: 13755 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 21:45:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8689: test hive impala interop failing with "Timeout >7200s"
Abhishek Rawat has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13755 ) Change subject: IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" .. IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" The newly added Hive<->Impala interop test fails due to unexpected wrong results when reading TimeStamp column value written by Hive. The short term measure is to remove TimeStamp column from the interop tests. The original issue will be fixed by IMPALA-8721. Testing: Ran the testcase N number of times on both upstream and downstream code base. Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 --- M tests/custom_cluster/test_hive_parquet_codec_interop.py 1 file changed, 4 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/13755/2 -- To view, visit http://gerrit.cloudera.org:8080/13755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 Gerrit-Change-Number: 13755 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] WIP IMPALA-8484: Run queries on disjoint executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13550 ) Change subject: WIP IMPALA-8484: Run queries on disjoint executor groups .. Patch Set 7: (21 comments) http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py File bin/start-impala-cluster.py: http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@274 PS7, Line 274: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@274 PS7, Line 274: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@439 PS7, Line 439: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@439 PS7, Line 439: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@453 PS7, Line 453: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@453 PS7, Line 453: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/13550/7/bin/start-impala-cluster.py@713 PS7, Line 713: ; flake8: E703 statement ends with a semicolon http://gerrit.cloudera.org:8080/#/c/13550/7/tests/custom_cluster/test_auto_scaling.py File tests/custom_cluster/test_auto_scaling.py: http://gerrit.cloudera.org:8080/#/c/13550/7/tests/custom_cluster/test_auto_scaling.py@21 PS7, Line 21: import json flake8: F401 'json' imported but unused http://gerrit.cloudera.org:8080/#/c/13550/7/tests/custom_cluster/test_auto_scaling.py@22 PS7, Line 22: import requests flake8: F401 'requests' imported but unused http://gerrit.cloudera.org:8080/#/c/13550/7/tests/custom_cluster/test_auto_scaling.py@31 PS7, Line 31: class TestAutoScaling(CustomClusterTestSuite): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/13550/7/tests/custom_cluster/test_auto_scaling.py@62 PS7, Line 62: 3 flake8: E225 missing whitespace around operator http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/auto_scaler.py File tests/util/auto_scaler.py: http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/auto_scaler.py@31 PS7, Line 31: class AutoScaler(object): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/auto_scaler.py@181 PS7, Line 181: = flake8: E711 comparison to None should be 'if cond is None:' http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/auto_scaler.py@293 PS7, Line 293: if __name__ == "__main__": flake8: E305 expected 2 blank lines after class or function definition, found 1 http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py File tests/util/concurrent_workload.py: http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@24 PS7, Line 24: import signal flake8: F401 'signal' imported but unused http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@27 PS7, Line 27: import _strptime flake8: F401 '_strptime' imported but unused http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@35 PS7, Line 35: class ConcurrentWorkload(object): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@68 PS7, Line 68: e flake8: F841 local variable 'e' is assigned to but never used http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@109 PS7, Line 109: flake8: E201 whitespace after '[' http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@118 PS7, Line 118: = flake8: E711 comparison to None should be 'if cond is None:' http://gerrit.cloudera.org:8080/#/c/13550/7/tests/util/concurrent_workload.py@149 PS7, Line 149: if __name__ == "__main__": flake8: E305 expected 2 blank lines after class or function definition, found 1 -- To view, visit http://gerrit.cloudera.org:8080/13550 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b Gerrit-Change-Number: 13550 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 21:25:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-8484: Run queries on disjoint executor groups
Hello Andrew Sherman, Tim Armstrong, Bikramjeet Vig, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13550 to look at the new patch set (#7). Change subject: WIP IMPALA-8484: Run queries on disjoint executor groups .. WIP IMPALA-8484: Run queries on disjoint executor groups This change adds support running queries of a single admission control pool on one of several, disjoint sets of executors called 'executor groups'. This change also re-schedules queries before admitting them if the admission controller detects that the cluster membership has changed. TODO: Elaborate on change and design Known limitations: - When using executor groups, only a single coordinator and a single AC pool are supported. Open questions and issues: - Executor groups can become unhealthy when one or more of their executors fail. We should build support to heal or decommission unhealthy groups in a subsequent change. Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b --- M be/src/runtime/exec-env.cc M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/cluster-membership-mgr-test.cc M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/scheduling/cluster-membership-test-util.cc M be/src/scheduling/executor-group-test.cc M be/src/scheduling/executor-group.cc M be/src/scheduling/executor-group.h M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/runtime-profile.h M bin/start-impala-cluster.py M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/StatestoreService.thrift M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/common/resource_pool_config.py M tests/custom_cluster/test_admission_controller.py A tests/custom_cluster/test_auto_scaling.py A tests/util/auto_scaler.py A tests/util/concurrent_workload.py M www/backends.tmpl 35 files changed, 1,522 insertions(+), 432 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/13550/7 -- To view, visit http://gerrit.cloudera.org:8080/13550 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b Gerrit-Change-Number: 13550 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] WIP IMPALA-8484: Run queries on disjoint executor groups
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13550 ) Change subject: WIP IMPALA-8484: Run queries on disjoint executor groups .. Patch Set 7: (15 comments) http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.h@347 PS3, Line 347: boost::mutex admission_ctrl_lock_; > For the #running queries metrics, one thing to keep in mind is that the ne As we discussed in person, I removed group stats altogether and added the number of admitted queries to the /backends debug page. http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.h@578 PS3, Line 578: > This data structure some some significance for the policy, since the iterat Changed it to an ordered map and added a comment at the typedef. http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.cc@470 PS3, Line 470:<< " needs=" << PrintBytes(p > pool_max_mem is the max mem across the cluster, so we should use pool_stats Done http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.cc@530 PS3, Line 530: xRequestsForPool(pool > pool_cfg.max_requests would also scale with the cluster, so if we decide to I switched this to a slot based model that allows pool_cfg.max_requests concurrent queries per executor. http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.cc@672 PS3, Line 672: schedule.per_backend_exec_params().size(), cluster_thread_reservation, : query_opts.thread_reservation_aggregate_limit); : return true; : } > I think we should be logging all reasons, otherwise we wont know whats hold I change the relevant code to log every reason (the profiles still only contain the last one). Let me know if you'd like us to find a way to include all non-admissions in the profile, too. http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.cc@689 PS3, Line 689: int64_t cluster_mem_to_admit = schedule.GetClusterMemoryToAdmit(); > this should be outside this scope, that is, before the lock. otherwise it w Done http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.cc@692 PS3, Line 692: Substitute(REASON_REQ_OVER_POOL_MEM, PrintBytes(cluster_mem_to_admit), > If we iterate over the groups, will we tend to admit to the first group? So Good point, made it an ordered map and added a comment to the header. http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.cc@712 PS3, Line 712: ics_.pool_max_queued->SetValue(pool_cfg.max_ > I would also like to avoid adding a flag. Agree there's a good chance we ca I split up the checks in per-cluster and per-group rejection tests, but since the cluster size can change between admission attempts, it still seems beneficial to run both checks for every admission attempt. I added code to only run the tests when the cluster membership changes but we don't have versioning/notification for changes to the pool configs so that approach might not work (see new comment in DequeueLoop()). http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.cc@999 PS3, Line 999: DCHECK_GE(remote_pool_stats.num_admitted_running, 0); > Nit: I think this method should have a different name as the Scheduler stil Done http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.cc@1009 PS3, Line 1009: > nit: maybe add a dcheck to make sure this in-consistency ( < ) never happen Done http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/admission-controller.cc@1177 PS3, Line 1177: _GE(stats->agg_num_queued(), stats->local_stats().num_queued); : > this is fine, its what our current behavior is. If we let queries after thi Yeah, this TODO carried over from old code. Should we just remove it for now and file a jira instead? http://gerrit.cloudera.org:8080/#/c/13550/6/be/src/scheduling/cluster-membership-mgr.h File be/src/scheduling/cluster-membership-mgr.h: http://gerrit.cloudera.org:8080/#/c/13550/6/be/src/scheduling/cluster-membership-mgr.h@98 PS6, Line 98: /// The version of this Snapshot. It is incremented every time the cluster membership This probably needs at least some smoke testing http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/cluster-membership-mgr.h File be/src/scheduling/cluster-membership-mgr.h: http://gerrit.cloudera.org:8080/#/c/13550/3/be/src/scheduling/cluster-membership-mgr.h@97 PS3, Line 97: > Nit: add a description Done http://gerrit.cloudera.org:8080
[Impala-ASF-CR] IMPALA-8713: fix stack overflow in unhex()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13743 ) Change subject: IMPALA-8713: fix stack overflow in unhex() .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4566/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0c1760511a04c0823fc465cf6e529e9681b2488 Gerrit-Change-Number: 13743 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 27 Jun 2019 21:21:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8713: fix stack overflow in unhex()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13743 ) Change subject: IMPALA-8713: fix stack overflow in unhex() .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0c1760511a04c0823fc465cf6e529e9681b2488 Gerrit-Change-Number: 13743 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 27 Jun 2019 21:21:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8689: test hive impala interop failing with "Timeout >7200s"
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13755 ) Change subject: IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13755/1/tests/custom_cluster/test_hive_parquet_codec_interop.py File tests/custom_cluster/test_hive_parquet_codec_interop.py: http://gerrit.cloudera.org:8080/#/c/13755/1/tests/custom_cluster/test_hive_parquet_codec_interop.py@52 PS1, Line 52: self.execute_query_expect_success(self.client, Maybe leave a TODO: with a JIRA in it to add the coverage back in. -- To view, visit http://gerrit.cloudera.org:8080/13755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 Gerrit-Change-Number: 13755 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 21:20:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8689: test hive impala interop failing with "Timeout >7200s"
Abhishek Rawat has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13755 Change subject: IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" .. IMPALA-8689: test_hive_impala_interop failing with "Timeout >7200s" The newly added Hive<->Impala interop test fails due to unexpected wrong results when reading TimeStamp column value written by Hive. The short term measure is to remove TimeStamp column from the interop tests. The original issue will be fixed by IMPALA-8721. Testing: Ran the testcase N number of times on both upstream and downstream code base. Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 --- M tests/custom_cluster/test_hive_parquet_codec_interop.py 1 file changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/13755/1 -- To view, visit http://gerrit.cloudera.org:8080/13755 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I148c79a31f9aada1b75614390434462d1e483f28 Gerrit-Change-Number: 13755 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Add ext-py/bitarray-0.9.0 in .gitignore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13745 ) Change subject: Add ext-py/bitarray-0.9.0 in .gitignore .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 Gerrit-Change-Number: 13745 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 20:51:14 + Gerrit-HasComments: No
[Impala-ASF-CR] Add ext-py/bitarray-0.9.0 in .gitignore
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13745 ) Change subject: Add ext-py/bitarray-0.9.0 in .gitignore .. Add ext-py/bitarray-0.9.0 in .gitignore Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 Reviewed-on: http://gerrit.cloudera.org:8080/13745 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M shell/.gitignore 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 Gerrit-Change-Number: 13745 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8673: Add query option to force plan hints for insert queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13753 ) Change subject: IMPALA-8673: Add query option to force plan hints for insert queries .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3772/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c3f213402b8e4d1940f96738ad21edf800fa43a Gerrit-Change-Number: 13753 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 19:15:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8713: fix stack overflow in unhex()
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/13743 ) Change subject: IMPALA-8713: fix stack overflow in unhex() .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13743 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0c1760511a04c0823fc465cf6e529e9681b2488 Gerrit-Change-Number: 13743 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 27 Jun 2019 19:00:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8673: Add query option to force plan hints for insert queries
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13753 )
Change subject: IMPALA-8673: Add query option to force plan hints for insert
queries
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13753/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java:
http://gerrit.cloudera.org:8080/#/c/13753/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@2042
PS1, Line 2042:
insertCtx.getQueryOptions().setDefault_hints_insert_statement("NOCLUSTERED:noshuffle");
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13753
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1c3f213402b8e4d1940f96738ad21edf800fa43a
Gerrit-Change-Number: 13753
Gerrit-PatchSet: 1
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 27 Jun 2019 18:37:19 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8673: Add query option to force plan hints for insert queries
Abhishek Rawat has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13753 Change subject: IMPALA-8673: Add query option to force plan hints for insert queries .. IMPALA-8673: Add query option to force plan hints for insert queries IMPALA-5293 enabled the pre-insert clustering by default. This could cause performance regression and this change provides a query option for setting default hints for INSERT statement. New query option 'DEFAULT_HINTS_INSERT_STATEMENT' was added. It also supports adding multiple supported hints when separated by ':' set DEFAULT_HINTS_INSERT_STATEMENT=[clustered|noclustered]; set DEFAULT_HINTS_INSERT_STATEMENT=[shuffle|noshuffle]; set DEFAULT_HINTS_INSERT_STATEMENT= [clustered|noclustered]:[shuffle|noshuffle]; If a given insert statement already has plan hints in the query text, the default hints, if any, are all ignored. This is because, if a query has plan hints specified by the user, we don't want to override it. When a default hint is set, and there is an INSERT statement without any plan hints in the query text, the default hints have the same affect as they would have had, if they were applied as plan hints in the query text. So these default hints have the same application and restrictions as the existing plan hints for INSERT statement. Testing: - Added unit tests in AnalyzeDDLTest for CTAS. - Added unit tests in AnalyzeStmtsTest for insert statements. - Added unit tests in PlannerTest validating the plan for various scenarios involving different combinations of default hints. Change-Id: I1c3f213402b8e4d1940f96738ad21edf800fa43a --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/insert-default-clustered-noshuffle.test A testdata/workloads/functional-planner/queries/PlannerTest/insert-default-clustered-shuffle.test A testdata/workloads/functional-planner/queries/PlannerTest/insert-default-clustered.test A testdata/workloads/functional-planner/queries/PlannerTest/insert-default-noclustered-noshuffle.test A testdata/workloads/functional-planner/queries/PlannerTest/insert-default-noclustered-shuffle.test A testdata/workloads/functional-planner/queries/PlannerTest/insert-default-noclustered.test A testdata/workloads/functional-planner/queries/PlannerTest/insert-default-noshuffle.test A testdata/workloads/functional-planner/queries/PlannerTest/insert-default-shuffle.test 16 files changed, 1,094 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/13753/1 -- To view, visit http://gerrit.cloudera.org:8080/13753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1c3f213402b8e4d1940f96738ad21edf800fa43a Gerrit-Change-Number: 13753 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8585: Insert data into ACID table during dataload
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13633 ) Change subject: IMPALA-8585: Insert data into ACID table during dataload .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3771/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13633 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id459519d5b963ea760c44719c12736bf104938f9 Gerrit-Change-Number: 13633 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 27 Jun 2019 18:02:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8585: Insert data into ACID table during dataload
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13633 ) Change subject: IMPALA-8585: Insert data into ACID table during dataload .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3770/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13633 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id459519d5b963ea760c44719c12736bf104938f9 Gerrit-Change-Number: 13633 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 27 Jun 2019 17:57:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8585: Insert data into ACID table during dataload
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13633 ) Change subject: IMPALA-8585: Insert data into ACID table during dataload .. Patch Set 3: (2 comments) Please ignore patch set 2, it ignored some additional stuff by mistake. http://gerrit.cloudera.org:8080/#/c/13633/1/testdata/datasets/functional/functional_schema_template.sql File testdata/datasets/functional/functional_schema_template.sql: http://gerrit.cloudera.org:8080/#/c/13633/1/testdata/datasets/functional/functional_schema_template.sql@2137 PS1, Line 2137: -- Compactions could be also interesting, but seem to be sporadically very slow. > mind filing this JIRA and adding a TODO here? then we can commit this with Done http://gerrit.cloudera.org:8080/#/c/13633/2/tests/query_test/test_acid.py File tests/query_test/test_acid.py: http://gerrit.cloudera.org:8080/#/c/13633/2/tests/query_test/test_acid.py@55 PS2, Line 55: # Tests that should run for all (HDFS based) table formats, not just text. > flake8: E302 expected 2 blank lines, found 1 Done -- To view, visit http://gerrit.cloudera.org:8080/13633 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id459519d5b963ea760c44719c12736bf104938f9 Gerrit-Change-Number: 13633 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 27 Jun 2019 17:24:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8585: Insert data into ACID table during dataload
Hello Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13633 to look at the new patch set (#3). Change subject: IMPALA-8585: Insert data into ACID table during dataload .. IMPALA-8585: Insert data into ACID table during dataload Insert some data into insert_only_transactional_table during dataload and add tests that read it. Unlike current ACID tests, this new test is ran for every file format, not just text. Change-Id: Id459519d5b963ea760c44719c12736bf104938f9 --- M testdata/datasets/functional/functional_schema_template.sql A testdata/workloads/functional-query/queries/QueryTest/acid-all-formats.test M tests/query_test/test_acid.py 3 files changed, 40 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/13633/3 -- To view, visit http://gerrit.cloudera.org:8080/13633 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id459519d5b963ea760c44719c12736bf104938f9 Gerrit-Change-Number: 13633 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8630: Hash the full path when calculating consistent remote placement
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13545 ) Change subject: IMPALA-8630: Hash the full path when calculating consistent remote placement .. IMPALA-8630: Hash the full path when calculating consistent remote placement Consistent remote placement currently uses the relative filename within a partition for the consistent hash. If the relative filenames for different partitions have a simple naming scheme, then multiple partitions may have files of the same name. This is true for some tables written by Hive (e.g. in our minicluster the tpcds.store_sales has this problem). This can lead to unbalanced placement of remote ranges. This adds a partition_path_hash to the THdfsFileSplit and THdfsFileSplitGeneratorSpec, calculated in the frontend (which has all of the partition information). The scheduler hashes this in addition to the relative path. Testing: - Added several new scheduler tests that verify the consistent remote scheduling sees blocks with different relative paths, partition paths, or offsets as distinct. - Ran core tests Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Reviewed-on: http://gerrit.cloudera.org:8080/13545 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java 7 files changed, 332 insertions(+), 55 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13545 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I46c739fc31af539af2b3509e2a161f4e29f44d7b Gerrit-Change-Number: 13545 Gerrit-PatchSet: 14 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8585: Insert data into ACID table during dataload
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13633 ) Change subject: IMPALA-8585: Insert data into ACID table during dataload .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13633/2/tests/query_test/test_acid.py File tests/query_test/test_acid.py: http://gerrit.cloudera.org:8080/#/c/13633/2/tests/query_test/test_acid.py@55 PS2, Line 55: class TestAcidAllFormats(ImpalaTestSuite): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/13633 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id459519d5b963ea760c44719c12736bf104938f9 Gerrit-Change-Number: 13633 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 27 Jun 2019 17:19:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8585: Insert data into ACID table during dataload
Hello Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13633 to look at the new patch set (#2). Change subject: IMPALA-8585: Insert data into ACID table during dataload .. IMPALA-8585: Insert data into ACID table during dataload Insert some data into insert_only_transactional_table during dataload and add tests that read it. Unlike current ACID tests, this new test is ran for every file format, not just text. Change-Id: Id459519d5b963ea760c44719c12736bf104938f9 --- M testdata/datasets/functional/functional_schema_template.sql A testdata/workloads/functional-query/queries/QueryTest/acid-all-formats.test M tests/query_test/test_acid.py 3 files changed, 39 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/13633/2 -- To view, visit http://gerrit.cloudera.org:8080/13633 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id459519d5b963ea760c44719c12736bf104938f9 Gerrit-Change-Number: 13633 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3769/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 17:19:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4565/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 16:43:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 16:43:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 16:40:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13508/4/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java File fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java: http://gerrit.cloudera.org:8080/#/c/13508/4/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@131 PS4, Line 131: tableName_); > Just an FYI usually it's better to not call toString() explicitly in the lo Thanks for sharing! I removed the toString() call even though it doesn't matter here. -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 16:40:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Hello Fredy Wijaya, Zoltan Borok-Nagy, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13508 to look at the new patch set (#5). Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. IMPALA-8612: Fix sporadic NPE when dropping an authorized table In the analyze() function of DropTableOrViewStmt it's possible that serverName_ is not set when analyzer.getTable() throws. As a result when the Catalog executes the drop table DDL it runs into a failing Precondition check and throws a NullPointerException when updating user privileges. Note, to run into the NPE it's required to have authorization enabled. Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 --- M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java 1 file changed, 9 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/13508/5 -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8648: Add a simple stress test for reading ACID tables
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13751 ) Change subject: IMPALA-8648: Add a simple stress test for reading ACID tables .. Patch Set 2: Code-Review-2 I ran it a bit more and hit the following issue: Query aborted:Disk I/O error on csringhofer-desktop:22001: Failed to open HDFS file hdfs://localhost:20500/test-warehouse/test_read_hive_inserts_9016893c.db/test_read_hive_inserts/_tmp.base_001/00_0.manifest Error(2): No such file or directory This should be fixed if the .tmp file skipping logic goes in: https://gerrit.cloudera.org/#/c/13665/ Until then this test is flaky. -- To view, visit http://gerrit.cloudera.org:8080/13751 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57d7ab5bf39e9225e50add0a234849ed20d8b3f8 Gerrit-Change-Number: 13751 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 27 Jun 2019 16:12:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8648: Add a simple stress test for reading ACID tables
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13751 to look at the new patch set (#2). Change subject: IMPALA-8648: Add a simple stress test for reading ACID tables .. IMPALA-8648: Add a simple stress test for reading ACID tables Add a test that runs several INSERT INTO/OVERWRITE to a single table in Hive and reads in from a parallel process using Impala. Change-Id: I57d7ab5bf39e9225e50add0a234849ed20d8b3f8 --- A tests/stress/test_acid_stress.py 1 file changed, 117 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/13751/2 -- To view, visit http://gerrit.cloudera.org:8080/13751 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I57d7ab5bf39e9225e50add0a234849ed20d8b3f8 Gerrit-Change-Number: 13751 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8648: Add a simple stress test fpr reading ACID tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13751 ) Change subject: IMPALA-8648: Add a simple stress test fpr reading ACID tables .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3768/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13751 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57d7ab5bf39e9225e50add0a234849ed20d8b3f8 Gerrit-Change-Number: 13751 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 27 Jun 2019 15:40:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8585: Insert data into ACID table during dataload
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13633 ) Change subject: IMPALA-8585: Insert data into ACID table during dataload .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13633/1/testdata/datasets/functional/functional_schema_template.sql File testdata/datasets/functional/functional_schema_template.sql: http://gerrit.cloudera.org:8080/#/c/13633/1/testdata/datasets/functional/functional_schema_template.sql@2137 PS1, Line 2137: -- Compactions could be also interesting, but seem to be sporadically very slow. > can you file a JIRA for this issue that you mentioned? mind filing this JIRA and adding a TODO here? then we can commit this with the workaround for now but also track the fix -- To view, visit http://gerrit.cloudera.org:8080/13633 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id459519d5b963ea760c44719c12736bf104938f9 Gerrit-Change-Number: 13633 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 27 Jun 2019 15:25:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 4: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13508/4/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java File fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java: http://gerrit.cloudera.org:8080/#/c/13508/4/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@131 PS4, Line 131: tableName_.toString() Just an FYI usually it's better to not call toString() explicitly in the logger since toString() will be lazily called by the logger. For example if we had a very expensive toString() call, calling LOG.info() on a different log level such as WARN or ERROR would not incur an expensive toString() call. I don't think it matters for this, though. -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 15:18:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Add ext-py/bitarray-0.9.0 in .gitignore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13745 ) Change subject: Add ext-py/bitarray-0.9.0 in .gitignore .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4564/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 Gerrit-Change-Number: 13745 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 15:14:22 + Gerrit-HasComments: No
[Impala-ASF-CR] Add ext-py/bitarray-0.9.0 in .gitignore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13745 ) Change subject: Add ext-py/bitarray-0.9.0 in .gitignore .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 Gerrit-Change-Number: 13745 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 15:14:21 + Gerrit-HasComments: No
[Impala-ASF-CR] Add ext-py/bitarray-0.9.0 in .gitignore
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13745 ) Change subject: Add ext-py/bitarray-0.9.0 in .gitignore .. Patch Set 2: > Patch Set 2: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4559/ Looks like flaky Maven issue. Retrying the merge. 02:24:28 [ERROR] Failed to execute goal on project impala-minimal-hive-exec: Could not resolve dependencies for project org.apache.impala:impala-minimal-hive-exec:jar:0.1-SNAPSHOT: Could not transfer artifact io.dropwizard.metrics:metrics-core:jar:3.1.0 from/to impala.cdp.repo (https://native-toolchain.s3.amazonaws.com/build/cdp_components/1153860/maven): Access denied to: https://native-toolchain.s3.amazonaws.com/build/cdp_components/1153860/maven/io/dropwizard/metrics/metrics-core/3.1.0/metrics-core-3.1.0.jar , ReasonPhrase:Forbidden. -> [Help 1] 02:24:28 [ERROR] 02:24:28 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 02:24:28 [ERROR] Re-run Maven using the -X switch to enable full debug logging. 02:24:28 [ERROR] 02:24:28 [ERROR] For more information about the errors and possible solutions, please read the following articles: 02:24:28 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException 02:24:28 mvn -U -B install -DskipTests exited with code 0 02:24:28 shaded-deps/CMakeFiles/shaded-deps.dir/build.make:57: recipe for target 'shaded-deps/CMakeFiles/shaded-deps' failed 02:24:28 make[2]: *** [shaded-deps/CMakeFiles/shaded-deps] Error 1 02:24:28 CMakeFiles/Makefile2:14411: recipe for target 'shaded-deps/CMakeFiles/shaded-deps.dir/all' failed 02:24:28 make[1]: *** [shaded-deps/CMakeFiles/shaded-deps.dir/all] Error 2 02:24:28 Makefile:94: recipe for target 'all' failed 02:24:28 make: *** [all] Error -- To view, visit http://gerrit.cloudera.org:8080/13745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 Gerrit-Change-Number: 13745 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 15:14:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8648: Add a simple stress test fpr reading ACID tables
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13751 Change subject: IMPALA-8648: Add a simple stress test fpr reading ACID tables .. IMPALA-8648: Add a simple stress test fpr reading ACID tables Add a test that runs several INSERT INTO/OVERWRITE to a single table in Hive and reads in from a parallel process using Impala. Change-Id: I57d7ab5bf39e9225e50add0a234849ed20d8b3f8 --- A tests/stress/test_acid_stress.py 1 file changed, 117 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/13751/1 -- To view, visit http://gerrit.cloudera.org:8080/13751 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I57d7ab5bf39e9225e50add0a234849ed20d8b3f8 Gerrit-Change-Number: 13751 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3767/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 14:47:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8665:Include extra info in error message when date cast fails
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/13680 )
Change subject: IMPALA-8665:Include extra info in error message when date cast
fails
..
IMPALA-8665:Include extra info in error message when date cast fails
This change extends the error message Impala yields when casting STRING
to DATE (explicitly or implicitly) fails. The new error message includes
the violating string value.
Testing:
changes -> date-partitioning.test & date.test
query_test/test_date_queries.py test passed
Example:
select cast('20' as date);
ERROR: UDF ERROR: String to Date parse failed. Invalid string val: "20"
Change-Id: If800b7696515cd61afee27220c55ff2440a86f04
Reviewed-on: http://gerrit.cloudera.org:8080/13680
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
M be/src/exprs/cast-functions-ir.cc
M testdata/workloads/functional-query/queries/QueryTest/date-partitioning.test
M testdata/workloads/functional-query/queries/QueryTest/date.test
3 files changed, 12 insertions(+), 8 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/13680
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If800b7696515cd61afee27220c55ff2440a86f04
Gerrit-Change-Number: 13680
Gerrit-PatchSet: 10
Gerrit-Owner: Jiawei Wang
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jiawei Wang
[Impala-ASF-CR] IMPALA-8665:Include extra info in error message when date cast fails
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13680 ) Change subject: IMPALA-8665:Include extra info in error message when date cast fails .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If800b7696515cd61afee27220c55ff2440a86f04 Gerrit-Change-Number: 13680 Gerrit-PatchSet: 9 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Comment-Date: Thu, 27 Jun 2019 14:39:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8636: Implement INSERT for insert-only ACID tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13559 ) Change subject: IMPALA-8636: Implement INSERT for insert-only ACID tables .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3766/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad Gerrit-Change-Number: 13559 Gerrit-PatchSet: 10 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 14:26:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/13508 )
Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table
..
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13508/3/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
File fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java:
http://gerrit.cloudera.org:8080/#/c/13508/3/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@40
PS3, Line 40: Drop
> Shouldn't this be DropTableOrViewStmt?
Oh, copy-paste issue. Thanks for spotting!
Done
http://gerrit.cloudera.org:8080/#/c/13508/3/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@131
PS3, Line 131: "Ignoring TableLoadingException for {}", tableName_.toString());
> nit: it's usually preferable to use the SLF4J log format, e.g. LOG.info("Ig
Done
http://gerrit.cloudera.org:8080/#/c/13508/3/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@134
PS3, Line 134: ring());
> tableName_.toString() may print "db.tbl": https://github.com/apache/impala/
If you drop the table like "drop table tbl;" then tableName_.toString() won't
print "default.tbl" just "tbl". However, providing default as the DB name for
the drop table statement will result in printing "default.tbl". I think this is
fine and I rewrote the code to use TableName.toString().
Done
--
To view, visit http://gerrit.cloudera.org:8080/13508
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952
Gerrit-Change-Number: 13508
Gerrit-PatchSet: 4
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 27 Jun 2019 14:07:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Hello Fredy Wijaya, Zoltan Borok-Nagy, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13508 to look at the new patch set (#4). Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. IMPALA-8612: Fix sporadic NPE when dropping an authorized table In the analyze() function of DropTableOrViewStmt it's possible that serverName_ is not set when analyzer.getTable() throws. As a result when the Catalog executes the drop table DDL it runs into a failing Precondition check and throws a NullPointerException when updating user privileges. Note, to run into the NPE it's required to have authorization enabled. Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 --- M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java 1 file changed, 9 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/13508/4 -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8636: Implement INSERT for insert-only ACID tables
Hello Csaba Ringhofer, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13559 to look at the new patch set (#10). Change subject: IMPALA-8636: Implement INSERT for insert-only ACID tables .. IMPALA-8636: Implement INSERT for insert-only ACID tables This commit adds INSERT support for insert-only ACID tables. The Frontend opens a transaction for queries that refer to transactional tables. For INSERT statements that write insert-only ACID tables it also allocates a write ID. The Frontend aborts the transaction if an error occurs during analysis/planning. The Backend gets the transaction id in TExecRequestState and the write id is set for the HDFS table sinks. The sinks write the files at their final destination which is an ACID base/delta directory. There is no need for finalization of transactional INSERTS. ClientRequestState commits the transaction in WaitInternal() if everything went well. If the transaction is still open in Done(), it means there was an error, therefore the transaction needs to be aborted. The Backend commits/aborts the transaction by calling the Frontend via JNI. Testing: * added new tables during dataload * added acid-insert.test file with INSERT statements against the new tables * added integration test with Hive to test_hms_integration.py. The test inserts data with Impala and reads with Hive. (These integration tests only run with exhaustive exploration strategy) TODO in following commits: * add locks and heartbeats * implement TRUNCATE (maybe in another commit) * CTAS creates files in the 'root' directory of the table/partition. It is handled correctly during SELECT, but would be better to create a base directory from the beginning. Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad --- M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/runtime/coordinator.cc M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/util/jni-util.h M common/thrift/DataSinks.thrift M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/common/TransactionException.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-query/queries/QueryTest/acid-insert.test M tests/metadata/test_hms_integration.py M tests/query_test/test_insert.py 25 files changed, 731 insertions(+), 119 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/13559/10 -- To view, visit http://gerrit.cloudera.org:8080/13559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad Gerrit-Change-Number: 13559 Gerrit-PatchSet: 10 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13665 ) Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa Gerrit-Change-Number: 13665 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 27 Jun 2019 12:59:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8253: Draft - Parquet delta encoding and decoding.
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12621 ) Change subject: IMPALA-8253: Draft - Parquet delta encoding and decoding. .. Patch Set 13: We decided to split this into multiple subtasks. A part of this change (with some modifications) is https://gerrit.cloudera.org/#/c/13737/. -- To view, visit http://gerrit.cloudera.org:8080/12621 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie7378ac1a490a6c89a0a4349aae86cbc0fbc80f8 Gerrit-Change-Number: 12621 Gerrit-PatchSet: 13 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 12:09:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4561/ -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 3 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 27 Jun 2019 12:08:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8665:Include extra info in error message when date cast fails
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13680 ) Change subject: IMPALA-8665:Include extra info in error message when date cast fails .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4563/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If800b7696515cd61afee27220c55ff2440a86f04 Gerrit-Change-Number: 13680 Gerrit-PatchSet: 9 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Comment-Date: Thu, 27 Jun 2019 09:07:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8665:Include extra info in error message when date cast fails
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13680 ) Change subject: IMPALA-8665:Include extra info in error message when date cast fails .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If800b7696515cd61afee27220c55ff2440a86f04 Gerrit-Change-Number: 13680 Gerrit-PatchSet: 9 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Comment-Date: Thu, 27 Jun 2019 09:07:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8665:Include extra info in error message when date cast fails
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/13680 ) Change subject: IMPALA-8665:Include extra info in error message when date cast fails .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If800b7696515cd61afee27220c55ff2440a86f04 Gerrit-Change-Number: 13680 Gerrit-PatchSet: 8 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Comment-Date: Thu, 27 Jun 2019 09:05:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13665 ) Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3765/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa Gerrit-Change-Number: 13665 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 27 Jun 2019 08:31:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13665 ) Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4562/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa Gerrit-Change-Number: 13665 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 27 Jun 2019 07:52:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Vihang Karajgaonkar has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/13665 ) Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories .. IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories The FileMetadataLoader is used to load the file information in when the table is loaded. By default, it lists all the files in the table/partition directory. Currently, it only skips the filenames which are invalid (hidden files and ones starting with "_" etc). However, it does not skip the directories which are temporary or hidden. In case of Hive when data is inserted into a table, it creates a temporary staging directory which is a hidden directory under the table location. When the insert in hive is completed, such staging directories are removed. But if there is a refresh called during that time, FileMetadataLoader will add the files in the staging directory as well. Not only this could cause temporary invalid results but it causes table to go in a bad state when these temporary directories are removed. The only work-around in such a case to issue a refresh on the table again. This patch adds logic in the filemetadataloader to ignore such temporary staging directories. Unfortunately, hadoop does not provide a API which can recursively list files in a directory and skip certain directories. This patch addes this logic of filtering into existing RecursingIterator in FileSystemUtil. In case of S3 we use the listFiles API to natively recurse using the S3AFileSystem. This case needs a special handling to check if the files returned are contained within a ignored directory. The patch adds this to the recursing iterator using a new util method in FileSystemUtil. Also, the existing code to recover partitions implements its own recursion logic which includes path validation. This already skips such hidden directories since they do not conform to the partition spec. The patch does a minor modification to this method by directly calling the listStatusIterator instead of going through FileSystemUtil#listStatus whiche uses the filtering remote iterator now. Testing: 1. Added a new tests as well as modified existing ones which were related to cover interesting cases. 2. Ran concurrent inserts from Hive while issuing refresh in a loop on Impala side. Earlier this would cause the table to go into a bad state. Now, it works fine for the staging directories. It still runs into a FileNotFoundException from the impalad when there are insert overwrite statements in Hive Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java A fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java M tests/metadata/test_recursive_listing.py 6 files changed, 275 insertions(+), 51 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/13665/11 -- To view, visit http://gerrit.cloudera.org:8080/13665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa Gerrit-Change-Number: 13665 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13665 ) Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/13665/11/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java: http://gerrit.cloudera.org:8080/#/c/13665/11/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@734 PS11, Line 734: if (isRecursive_ && listingType_.equals(ListingType.LIST_FILES) && isInIgnoredDirectory( line too long (94 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa Gerrit-Change-Number: 13665 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 27 Jun 2019 07:52:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Add ext-py/bitarray-0.9.0 in .gitignore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13745 ) Change subject: Add ext-py/bitarray-0.9.0 in .gitignore .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4559/ -- To view, visit http://gerrit.cloudera.org:8080/13745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 Gerrit-Change-Number: 13745 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 07:52:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13665 ) Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories .. Patch Set 8: (6 comments) http://gerrit.cloudera.org:8080/#/c/13665/8/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java: http://gerrit.cloudera.org:8080/#/c/13665/8/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@541 PS8, Line 541: temporary directories > the code now seems to also skip any files that match this pattern, not just The latest patchset changed it so that it only skips temp directories now. I saw code in FileMetadataLoader which keeps stats related to hidden files. Didn't want to mess up those, since temp/hidden files in valid directories seem intentional and long-lived, while temp directories most are transient in most cases. http://gerrit.cloudera.org:8080/#/c/13665/8/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@545 PS8, Line 545: all underlying files (except which are :* in the ignored directories) > does this mean we no longer yield directories, and only yield files? does t Actually, I added this line in the doc since I noticed that the RecursingIterator yields files not directories even without the patch. I thought that was important and hence added it here. These semantics are not changed after the patch. http://gerrit.cloudera.org:8080/#/c/13665/8/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@568 PS8, Line 568: isS3AFileSystem(p) > why'd you change from isS3AFileSystem(fs) to isS3AFileSystem(p)? In the cas this was unintentional change. Didn't realize there are two isS3AFileSystem methods and I used the wrong one. http://gerrit.cloudera.org:8080/#/c/13665/8/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@568 PS8, Line 568: isS3AFileSystem > perhaps we can add a @VisibleForTesting way we can make this path get used Sure. Do you have any suggestions on how can we do this? static methods are hard to mock. Are you thinking of using a test-only flag? http://gerrit.cloudera.org:8080/#/c/13665/8/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@626 PS8, Line 626: LISTING_TYPE > style nit: enums should be named like classes (ListingType) Done http://gerrit.cloudera.org:8080/#/c/13665/8/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@699 PS8, Line 699: // if the current file is on a ignored path return early : if (isIgnoredPath(fileStatus)) return; > how does this prevent recursion into tmp dirs in the recursive listFiles ca yeah, I realized that when one of my own tests failed on the last patch set. The latest patch set addresses this. -- To view, visit http://gerrit.cloudera.org:8080/13665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa Gerrit-Change-Number: 13665 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 27 Jun 2019 07:40:58 + Gerrit-HasComments: Yes
