[Impala-ASF-CR] IMPALA-7329: Blacklist CDH Maven snapshots repository
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10999 Change subject: IMPALA-7329: Blacklist CDH Maven snapshots repository .. IMPALA-7329: Blacklist CDH Maven snapshots repository Impala depends on CDH Maven snapshots which transitively pull dependencies from other repositories which can cause the build to be non-reproducible, e.g. IMPALA-7316. This patch makes the build to be reproducible by blacklisting cdh.snapshots.repo so that Maven does not accidentally downloads the latest CDH snapshots when running a build, which can cause incompatibility issues. Testing: - Ran core tests with CDH_BUILD_NUMBER=422770 and did not hit the issue described in IMPALA-7316 Change-Id: Id945bc2769f92f3df3bb4f617b00db77a6502ff3 --- M impala-parent/pom.xml 1 file changed, 11 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/10999/2 -- To view, visit http://gerrit.cloudera.org:8080/10999 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id945bc2769f92f3df3bb4f617b00db77a6502ff3 Gerrit-Change-Number: 10999 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-6857: Add Jvm pause/GC Monitor utility
Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10998 Change subject: IMPALA-6857: Add Jvm pause/GC Monitor utility .. IMPALA-6857: Add Jvm pause/GC Monitor utility This commit adds a stripped down version of Hadoop's JvmPauseMonitor class (https://bit.ly/2O6qSwm) . The core implementaion is borrowed from hadoop-common project and the hadoop dependencies are removed. - Removed dependency on AbstractService. - Not relying on Hadoop's Configuration object for reading confs. - Switched to Guava's implementation of Stopwatch. This utility class can detect both GC/non-GC pauses. In case of GC pauses, the difference of GC metrics before and after pause are logged. Sample Output: = Detected pause in JVM or host machine (eg GC): pause of approximately 2356ms GC pool 'PS MarkSweep' had collection(s): count=1 time=2241ms GC pool 'PS Scavenge' had collection(s): count=3 time=352ms Detected pause in JVM or host machine (eg GC): pause of approximately 1964ms GC pool 'PS MarkSweep' had collection(s): count=1 time=2082ms GC pool 'PS Scavenge' had collection(s): count=1 time=251ms Detected pause in JVM or host machine (eg GC): pause of approximately 2120ms GC pool 'PS MarkSweep' had collection(s): count=1 time=2454ms Detected pause in JVM or host machine (eg GC): pause of approximately 2238ms GC pool 'PS MarkSweep' had collection(s): count=5 time=13464ms ^CDetected pause in JVM or host machine (eg GC): pause of approximately 2233ms GC pool 'PS MarkSweep' had collection(s): count=1 time=2733ms Testing: === - Tested it manually with kill -SIGSTOP/-SIGCONT . Made sure that the non-GC JVM pauses are logged. - This class' functionality is tested manually by invoking it's main() - Injected a memory leak into the Catalog server code and made sure the GC is detected. Change-Id: I30d897b7e063846ad6d8f88243e2c04264da0341 --- M be/src/common/init.cc M be/src/util/jni-util.cc M be/src/util/jni-util.h M fe/src/main/java/org/apache/impala/common/JniUtil.java A fe/src/main/java/org/apache/impala/util/JvmPauseMonitor.java 5 files changed, 228 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/10998/2 -- To view, visit http://gerrit.cloudera.org:8080/10998 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I30d897b7e063846ad6d8f88243e2c04264da0341 Gerrit-Change-Number: 10998 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada
[Impala-ASF-CR] IMPALA-6677: [DOCS] Document the next day function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10893 ) Change subject: IMPALA-6677: [DOCS] Document the next_day function .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 Gerrit-Change-Number: 10893 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 20 Jul 2018 02:19:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6677: [DOCS] Document the next day function
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10893 ) Change subject: IMPALA-6677: [DOCS] Document the next_day function .. IMPALA-6677: [DOCS] Document the next_day function Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 Reviewed-on: http://gerrit.cloudera.org:8080/10893 Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins --- M docs/topics/impala_datetime_functions.xml 1 file changed, 58 insertions(+), 15 deletions(-) Approvals: Vuk Ercegovac: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 Gerrit-Change-Number: 10893 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6677: [DOCS] Document the next day function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10893 ) Change subject: IMPALA-6677: [DOCS] Document the next_day function .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/352/ -- To view, visit http://gerrit.cloudera.org:8080/10893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 Gerrit-Change-Number: 10893 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 20 Jul 2018 02:09:33 + Gerrit-HasComments: No
[Impala-ASF-CR] KUDU-2492: Make the use of SO REUSEPORT conditional on it being defined
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10994 ) Change subject: KUDU-2492: Make the use of SO_REUSEPORT conditional on it being defined .. KUDU-2492: Make the use of SO_REUSEPORT conditional on it being defined A recent commit to Kudu, "rpc: add experimental rpc_reuseport flag", added the use of the rpc flag SO_REUSEREPORT. This flag is not available with older versions of Linux, resulting in a compiler error. This patch avoids the compiler error with a macro that checks if SO_REUSEPORT is defined, and if it's not attempting to set it will return an error. -- IMPALA-7302: This is cherry-picked to fix builds breaking on CentOS 6.4. Since some of our Jenkins machines are CentOS 6.4, and upgrading them to our new minimum supported OS of CentOS 6.8 is non-trivial, we cherry- pick this patch to temporarily unblock these builds until the Jenkins AMIs are upgraded. Change-Id: I042125cdaedafa5ebbc09e5a3c12112dfeec59a1 Reviewed-on: http://gerrit.cloudera.org:8080/10994 Reviewed-by: Thomas Marshall Tested-by: Impala Public Jenkins --- M be/src/kudu/util/net/socket.cc 1 file changed, 8 insertions(+), 4 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I042125cdaedafa5ebbc09e5a3c12112dfeec59a1 Gerrit-Change-Number: 10994 Gerrit-PatchSet: 2 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] KUDU-2492: Make the use of SO REUSEPORT conditional on it being defined
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10994 ) Change subject: KUDU-2492: Make the use of SO_REUSEPORT conditional on it being defined .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I042125cdaedafa5ebbc09e5a3c12112dfeec59a1 Gerrit-Change-Number: 10994 Gerrit-PatchSet: 1 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Fri, 20 Jul 2018 01:59:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6677: [DOCS] Document the next day function
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10893 ) Change subject: IMPALA-6677: [DOCS] Document the next_day function .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 Gerrit-Change-Number: 10893 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 20 Jul 2018 00:55:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6299: Modify IRBuilder to use LLVM's CPU features
Pooja Nilangekar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/10979 ) Change subject: IMPALA-6299: Modify IRBuilder to use LLVM's CPU features .. IMPALA-6299: Modify IRBuilder to use LLVM's CPU features Previously, the IRBuilder of the LlvmCodeGen class used CpuInfo's list of enabled features to determine the validity of certain instructions. It did not consider the whitelist which passed while initilaizing the LlvmCodeGen class. Now, the IRBuilder inspects its own cpu attributes before emitting instruction. This change also adds functionality to modify the cpu attributes of the LlvmCodeGen class for testing. Testing: Verified that the current tests which use and modify CpuInfo produce expected results. Change-Id: Ifece8949c143146d2a1b38d72d21c2d733bed90f --- M be/src/benchmarks/bloom-filter-benchmark.cc M be/src/codegen/llvm-codegen-test.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/filter-context.cc M be/src/util/cpu-info.cc 6 files changed, 102 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/10979/3 -- To view, visit http://gerrit.cloudera.org:8080/10979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifece8949c143146d2a1b38d72d21c2d733bed90f Gerrit-Change-Number: 10979 Gerrit-PatchSet: 3 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6677: [DOCS] Document the next day function
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10893 ) Change subject: IMPALA-6677: [DOCS] Document the next_day function .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/10893/4/docs/topics/impala_datetime_functions.xml File docs/topics/impala_datetime_functions.xml: http://gerrit.cloudera.org:8080/#/c/10893/4/docs/topics/impala_datetime_functions.xml@1941 PS4, Line 1941: SELECT NEXT_DAY > I've followed a little bit the discussion about moving towards this style. Done -- To view, visit http://gerrit.cloudera.org:8080/10893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 Gerrit-Change-Number: 10893 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 20 Jul 2018 00:30:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6677: [DOCS] Document the next day function
Hello Bharath Vissapragada, Lars Volker, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10893 to look at the new patch set (#5). Change subject: IMPALA-6677: [DOCS] Document the next_day function .. IMPALA-6677: [DOCS] Document the next_day function Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 --- M docs/topics/impala_datetime_functions.xml 1 file changed, 58 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/10893/5 -- To view, visit http://gerrit.cloudera.org:8080/10893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 Gerrit-Change-Number: 10893 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 7: How about having an overloaded method, dropPartitions(list, boolean removeCacheDirective). It can true by default, but we make it false in the dirtyPatitions case. Is it more complicated than that? -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 20 Jul 2018 00:14:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6677: [DOCS] Document the next day function
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10893 ) Change subject: IMPALA-6677: [DOCS] Document the next_day function .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/10893/4/docs/topics/impala_datetime_functions.xml File docs/topics/impala_datetime_functions.xml: http://gerrit.cloudera.org:8080/#/c/10893/4/docs/topics/impala_datetime_functions.xml@1941 PS4, Line 1941: SELECT NEXT_DAY I've followed a little bit the discussion about moving towards this style. However, I'd prefer that we retain file-level consistency, then move to the new style at the file granularity. I'm concerned that an inconsistent style for a given file will be worse than a consistent style, even it we're moving away from it. This file looks like it uses lower-case for both keywords and functions. -- To view, visit http://gerrit.cloudera.org:8080/10893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 Gerrit-Change-Number: 10893 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 23:58:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7256: Aggregator mem usage isn't reflected in summary
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10989 ) Change subject: IMPALA-7256: Aggregator mem usage isn't reflected in summary .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10989 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba6ef207bed47810fc742aec3481db5f313cf97f Gerrit-Change-Number: 10989 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Jul 2018 23:46:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6677: [DOCS] Document the next day function
Hello Bharath Vissapragada, Lars Volker, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10893 to look at the new patch set (#4). Change subject: IMPALA-6677: [DOCS] Document the next_day function .. IMPALA-6677: [DOCS] Document the next_day function Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 --- M docs/topics/impala_datetime_functions.xml 1 file changed, 59 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/10893/4 -- To view, visit http://gerrit.cloudera.org:8080/10893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 Gerrit-Change-Number: 10893 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6677: [DOCS] Document the next day function
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10893 ) Change subject: IMPALA-6677: [DOCS] Document the next_day function .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/10893/3/docs/topics/impala_datetime_functions.xml File docs/topics/impala_datetime_functions.xml: http://gerrit.cloudera.org:8080/#/c/10893/3/docs/topics/impala_datetime_functions.xml@1941 PS3, Line 1941: one a week later > the "a" should be removed. Done http://gerrit.cloudera.org:8080/#/c/10893/3/docs/topics/impala_datetime_functions.xml@1951 PS3, Line 1951: 07-12 > perhaps include the example referenced on L1940? Done -- To view, visit http://gerrit.cloudera.org:8080/10893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 Gerrit-Change-Number: 10893 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 23:44:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7256: Aggregator mem usage isn't reflected in summary
Hello Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10989 to look at the new patch set (#2). Change subject: IMPALA-7256: Aggregator mem usage isn't reflected in summary .. IMPALA-7256: Aggregator mem usage isn't reflected in summary This patch fixes a bug where memory used by an Aggregator wasn't being reflected in the exec summary for the corresponding aggregation node by ensuring that the Aggregator's MemTracker is a child of the node's MemTracker. Testing: - Manually ran a query and checked that all memory used by the Aggregator is accounted for in the exec summary. Change-Id: Iba6ef207bed47810fc742aec3481db5f313cf97f --- M be/src/exec/aggregator.cc M be/src/exec/aggregator.h M be/src/exec/exec-node.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/runtime/reservation-manager.cc M be/src/runtime/reservation-manager.h 7 files changed, 23 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/10989/2 -- To view, visit http://gerrit.cloudera.org:8080/10989 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iba6ef207bed47810fc742aec3481db5f313cf97f Gerrit-Change-Number: 10989 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7256: Aggregator mem usage isn't reflected in summary
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/10989 ) Change subject: IMPALA-7256: Aggregator mem usage isn't reflected in summary .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10989/1/be/src/exec/exec-node.cc File be/src/exec/exec-node.cc: http://gerrit.cloudera.org:8080/#/c/10989/1/be/src/exec/exec-node.cc@113 PS1, Line 113: expr_mem_tracker_.reset(new MemTracker(-1, "Exprs", mem_tracker_.get(), false)); > The parallel MemTracker/ReservationTracker hierarchy is pretty annoying her Done -- To view, visit http://gerrit.cloudera.org:8080/10989 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba6ef207bed47810fc742aec3481db5f313cf97f Gerrit-Change-Number: 10989 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Jul 2018 23:38:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] bad python27 set usage
Philip Zeyliger has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10996 Change subject: bad python27 set usage .. bad python27 set usage Change-Id: Ia749dca006485d9bc4a22aee692196f4d0b8409b --- M tests/util/thrift_util.py 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/10996/1 -- To view, visit http://gerrit.cloudera.org:8080/10996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia749dca006485d9bc4a22aee692196f4d0b8409b Gerrit-Change-Number: 10996 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] bad python27 set usage
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10996 ) Change subject: bad python27 set usage .. Patch Set 1: My apologies; I clicked on publish by accident. I'm testing a Jenkins job. -- To view, visit http://gerrit.cloudera.org:8080/10996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia749dca006485d9bc4a22aee692196f4d0b8409b Gerrit-Change-Number: 10996 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 19 Jul 2018 23:01:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 7: The problem is that dropPartition() is also called on dirty partitions, in which case we should not change anything in the namenode. Maybe we should further differentiate these two cases, but that's non-trivial code change. -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Thu, 19 Jul 2018 22:41:38 + Gerrit-HasComments: No
[Impala-ASF-CR] KUDU-2492: Make the use of SO REUSEPORT conditional on it being defined
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10994 ) Change subject: KUDU-2492: Make the use of SO_REUSEPORT conditional on it being defined .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2843/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I042125cdaedafa5ebbc09e5a3c12112dfeec59a1 Gerrit-Change-Number: 10994 Gerrit-PatchSet: 1 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 19 Jul 2018 22:39:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10995 Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions In the prior code, the authorization checker for the masked privilege requests skips the check for system database access. As a result, certain commands, such as SHOW CREATE VIEW that references built-in database requires permission to access to the built-in database where accessing built-in database should always be allowed. The patch fixes it by using the authorizePrivilegeRequest() method that does a check on the system database similar to how other authorization checks are performed. Testing: - Added new authorization test - Ran all FE tests Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 3 files changed, 44 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/10995/2 -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] KUDU-2492: Make the use of SO REUSEPORT conditional on it being defined
Sailesh Mukil has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10994 Change subject: KUDU-2492: Make the use of SO_REUSEPORT conditional on it being defined .. KUDU-2492: Make the use of SO_REUSEPORT conditional on it being defined A recent commit to Kudu, "rpc: add experimental rpc_reuseport flag", added the use of the rpc flag SO_REUSEREPORT. This flag is not available with older versions of Linux, resulting in a compiler error. This patch avoids the compiler error with a macro that checks if SO_REUSEPORT is defined, and if it's not attempting to set it will return an error. -- IMPALA-7302: This is cherry-picked to fix builds breaking on CentOS 6.4. Since some of our Jenkins machines are CentOS 6.4, and upgrading them to our new minimum supported OS of CentOS 6.8 is non-trivial, we cherry- pick this patch to temporarily unblock these builds until the Jenkins AMIs are upgraded. Change-Id: I042125cdaedafa5ebbc09e5a3c12112dfeec59a1 --- M be/src/kudu/util/net/socket.cc 1 file changed, 8 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/10994/1 -- To view, visit http://gerrit.cloudera.org:8080/10994 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I042125cdaedafa5ebbc09e5a3c12112dfeec59a1 Gerrit-Change-Number: 10994 Gerrit-PatchSet: 1 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7256: Aggregator mem usage isn't reflected in summary
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10989 ) Change subject: IMPALA-7256: Aggregator mem usage isn't reflected in summary .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/10989/1/be/src/exec/exec-node.cc File be/src/exec/exec-node.cc: http://gerrit.cloudera.org:8080/#/c/10989/1/be/src/exec/exec-node.cc@113 PS1, Line 113: reservation_tracker_.reset(new ReservationTracker); The parallel MemTracker/ReservationTracker hierarchy is pretty annoying here. The reservation reporting won't work properly if ExecNode::reservation_tracker_ and ExecNode::buffer_pool_client_ are both in use because MemTracker assumes it only has one linked ReservationTracker. The design of the trackers feels suboptimal to me but we ended up in this place for some valid reasons. Anyway, I'm thinking it's probably best if we just add reservation_tracker_ to the aggregation nodes where it's actually used so that it's more obvious that this potential wonkiness doesn't happen. -- To view, visit http://gerrit.cloudera.org:8080/10989 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba6ef207bed47810fc742aec3481db5f313cf97f Gerrit-Change-Number: 10989 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Jul 2018 21:18:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6299: Modify IRBuilder to use LLVM's CPU features
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/10979 ) Change subject: IMPALA-6299: Modify IRBuilder to use LLVM's CPU features .. Patch Set 2: (5 comments) http://gerrit.cloudera.org:8080/#/c/10979/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10979/2//COMMIT_MSG@10 PS2, Line 10: while nit: to http://gerrit.cloudera.org:8080/#/c/10979/2//COMMIT_MSG@10 PS2, Line 10: certian nit:typo http://gerrit.cloudera.org:8080/#/c/10979/2/be/src/codegen/llvm-codegen.h File be/src/codegen/llvm-codegen.h: http://gerrit.cloudera.org:8080/#/c/10979/2/be/src/codegen/llvm-codegen.h@737 PS2, Line 737: current_cpu_attrs_ > Maybe enabled_cpu_attrs_? I agree with Tim http://gerrit.cloudera.org:8080/#/c/10979/2/be/src/codegen/llvm-codegen.cc File be/src/codegen/llvm-codegen.cc: http://gerrit.cloudera.org:8080/#/c/10979/2/be/src/codegen/llvm-codegen.cc@252 PS2, Line 252: auto cpu_flag_it = cpu_flag_mappings_.find(CpuInfo::SSE4_2); : DCHECK(cpu_flag_it != cpu_flag_mappings_.end()); : if (current_cpu_attrs_.find(cpu_flag_it->second) != current_cpu_attrs_.end( can we use IsCPUFeatureEnabled() here? http://gerrit.cloudera.org:8080/#/c/10979/2/be/src/codegen/llvm-codegen.cc@1581 PS2, Line 1581: auto cpu_flag_it = cpu_flag_mappings_.find(CpuInfo::SSE4_2); : DCHECK(cpu_flag_it != cpu_flag_mappings_.end()); : if (current_cpu_attrs_.find(cpu_flag_it->second) != current_cpu_attrs_.end( same here, IsCPUFeatureEnabled can be used -- To view, visit http://gerrit.cloudera.org:8080/10979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifece8949c143146d2a1b38d72d21c2d733bed90f Gerrit-Change-Number: 10979 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Jul 2018 20:40:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] test recover partitions.py had asserts that were always true.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10993 ) Change subject: test_recover_partitions.py had asserts that were always true. .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10993 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id49171304b853f15c43c8cfca066b6694c4a669f Gerrit-Change-Number: 10993 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 20:35:30 + Gerrit-HasComments: No
[Impala-ASF-CR] test recover partitions.py had asserts that were always true.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10993 ) Change subject: test_recover_partitions.py had asserts that were always true. .. test_recover_partitions.py had asserts that were always true. Running "python -m compileall" discovered some assertions that were always true. I've re-instated them to their true spirit. Change-Id: Id49171304b853f15c43c8cfca066b6694c4a669f Reviewed-on: http://gerrit.cloudera.org:8080/10993 Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins --- M tests/metadata/test_recover_partitions.py 1 file changed, 5 insertions(+), 6 deletions(-) Approvals: Vuk Ercegovac: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10993 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id49171304b853f15c43c8cfca066b6694c4a669f Gerrit-Change-Number: 10993 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7217: Incorrect UPDATE/DELETE authorization privilege
Adam Holley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10966 ) Change subject: IMPALA-7217: Incorrect UPDATE/DELETE authorization privilege .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10966 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69d451f727a7df6c41166a15cf1ed6f5334dc739 Gerrit-Change-Number: 10966 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 19:42:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6994:Avoid reloading a table's HMS data for file-only operations
Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/10587 ) Change subject: IMPALA-6994:Avoid reloading a table's HMS data for file-only operations .. Patch Set 2: (8 comments) http://gerrit.cloudera.org:8080/#/c/10587/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10587/2//COMMIT_MSG@7 PS2, Line 7: IMPALA-6994:Avoid reloading a table's HMS data for file-only operations > nit: add a space Done http://gerrit.cloudera.org:8080/#/c/10587/2//COMMIT_MSG@10 PS2, Line 10: unecessary > nit: check spelling and remove extra spaces. Done http://gerrit.cloudera.org:8080/#/c/10587/2//COMMIT_MSG@9 PS2, Line 9: unpartitioned HDFS table, : or to an existing partition > simplify to just HDFS table, or is there some case that's being excluded? Done http://gerrit.cloudera.org:8080/#/c/10587/2//COMMIT_MSG@21 PS2, Line 21: Testing: Ran core test without failure. > Is there any new scenario that you want to test for this change? If existin Yes I want to test the scenario where concurrent removal of partition and insertion of row to the same partition happens. a) Impala is inserting the row to an existing partition b) Hive is removing the partition at the same time. http://gerrit.cloudera.org:8080/#/c/10587/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10587/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1252 PS2, Line 1252: nonewPartitionHint > needs a comment. Done http://gerrit.cloudera.org:8080/#/c/10587/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1252 PS2, Line 1252: nonewPartitionHint > needs a comment. Added the comment http://gerrit.cloudera.org:8080/#/c/10587/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1365 PS2, Line 1365: loadMetadataAndDiskIds(partitionsToUpdateFileMdByPath, true); > any way to refactor this function so you don't need to duplicate this code Refactored the code and added a new function to reload the partition. http://gerrit.cloudera.org:8080/#/c/10587/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/10587/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@640 PS2, Line 640: private void loadTableMetadata(Table tbl, long newCatalogVersion, > the arguments here are getting quite unwieldly, and it seems that the two e Made changes to the code to use enum MetadataLoadFlag -- To view, visit http://gerrit.cloudera.org:8080/10587 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331bb0371fde287f43a85b025b4f98cb45f3eb3c Gerrit-Change-Number: 10587 Gerrit-PatchSet: 2 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 18:27:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6994: Avoid reloading a table's HMS data for file-only operations
Hello Bharath Vissapragada, Todd Lipcon, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10587 to look at the new patch set (#3). Change subject: IMPALA-6994: Avoid reloading a table's HMS data for file-only operations .. IMPALA-6994: Avoid reloading a table's HMS data for file-only operations The problem is that while inserting a new row to an existing partition of a HDFS table or to an unpartitioned table, catalogd makes an unnecessary call to Hive MetaStore to do a getTable() and list the partitions of the table. In such a case where a row is being inserted to an existing partition or to an unpartitioned table only the file metadata for HDFS tables needs to be updated. This change avoids calling the Hive Meta Store in updatePartitionsFromHms() by providing a hint from the caller, to skip loading the partitions. Testing: Ran core test without failure. Change-Id: I331bb0371fde287f43a85b025b4f98cb45f3eb3c --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 2 files changed, 126 insertions(+), 52 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/10587/3 -- To view, visit http://gerrit.cloudera.org:8080/10587 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I331bb0371fde287f43a85b025b4f98cb45f3eb3c Gerrit-Change-Number: 10587 Gerrit-PatchSet: 3 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] test recover partitions.py had asserts that were always true.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10993 ) Change subject: test_recover_partitions.py had asserts that were always true. .. Patch Set 1: Code-Review+2 there was a sweep of this not long ago. thanks for fixing the missing ones. -- To view, visit http://gerrit.cloudera.org:8080/10993 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id49171304b853f15c43c8cfca066b6694c4a669f Gerrit-Change-Number: 10993 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 17:27:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6490: Reconnect shell when remote restarts
Le Minh Nghia has posted comments on this change. (
http://gerrit.cloudera.org:8080/10992 )
Change subject: IMPALA-6490: Reconnect shell when remote restarts
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10992/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10992/1//COMMIT_MSG@9
PS1, Line 9: If the remote impalad died while the shell waited for
> nit: can you wrap lines in git commits at 60 chars? formatting in git tools
I did not know that, really useful, thanks!
http://gerrit.cloudera.org:8080/#/c/10992/1/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/10992/1/shell/impala_client.py@233
PS1, Line 233: """Checks to see if the current Impala connection is still
alive. If not, an exception
: will be raised."""
> Comment is out of date now.
I think it makes sense now?
http://gerrit.cloudera.org:8080/#/c/10992/1/shell/impala_client.py@235
PS1, Line 235: if self.connected:
: return self.imp_service.PingImpalaService()
> Since test_connection() now returns a True/False value, maybe we should cat
Done
http://gerrit.cloudera.org:8080/#/c/10992/1/tests/shell/test_shell_interactive.py
File tests/shell/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/10992/1/tests/shell/test_shell_interactive.py@157
PS1, Line 157: @pytest.mark.execute_serially
: def test_auto_reconnect(self):
: """Test reconnect after restarting the remote impalad
without using connect;"""
: kill_cluster = " ".join([START_CLUSTER, "--kill"])
: pexpect.run(kill_cluster)
: pexpect.run(START_CLUSTER)
: proc = pexpect.spawn(SHELL_CMD)
: proc.expect(":21000] default>")
: # Disconnect
: pexpect.run(kill_cluster)
: proc.sendline("show tables;")
: proc.expect("[Not connected]")
: # Restarting Impalad
: pexpect.run(START_CLUSTER)
: # Check reconnect
: proc.sendline("show tables;")
: proc.expect("large_table")
> Please check ./tests/custom_cluster/test_shell_interactive_reconnect.py.
Done
--
To view, visit http://gerrit.cloudera.org:8080/10992
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia13365a9696886f01294e98054cf4e7cd66ab712
Gerrit-Change-Number: 10992
Gerrit-PatchSet: 1
Gerrit-Owner: Le Minh Nghia
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Le Minh Nghia
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 19 Jul 2018 17:19:39 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] test recover partitions.py had asserts that were always true.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10993 ) Change subject: test_recover_partitions.py had asserts that were always true. .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2842/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10993 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id49171304b853f15c43c8cfca066b6694c4a669f Gerrit-Change-Number: 10993 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 19 Jul 2018 17:17:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6490: Reconnect shell when remote restarts
Hello Zoltan Borok-Nagy, Attila Jeges, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10992 to look at the new patch set (#2). Change subject: IMPALA-6490: Reconnect shell when remote restarts .. IMPALA-6490: Reconnect shell when remote restarts If the remote impad died while the shell waited for a command to complete, the shell disconnected. Previously after restarting the remote impalad, we need to run "connect;" to reconnect, noew the shell will automatically reconnect. Testing: Added test_auto_connect_after_impalad_died in test_shell_interactive_reconnect.py Change-Id: Ia13365a9696886f01294e98054cf4e7cd66ab712 --- M shell/impala_client.py M shell/impala_shell.py M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/shell/test_shell_interactive.py 5 files changed, 51 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10992/2 -- To view, visit http://gerrit.cloudera.org:8080/10992 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia13365a9696886f01294e98054cf4e7cd66ab712 Gerrit-Change-Number: 10992 Gerrit-PatchSet: 2 Gerrit-Owner: Le Minh Nghia Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6299: Modify IRBuilder to use LLVM's CPU features
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10979 ) Change subject: IMPALA-6299: Modify IRBuilder to use LLVM's CPU features .. Patch Set 2: I made the mistake of only looking at the code that was changed. I think this is missing some updates to codegen functions. E.g. there's a CpuInfo::IsSupported(CpuInfo::AVX2) in FilterContext::CodegenInsert() that should be converted to the new API. I did a quick grep and I think that might be the only place, but you should also verify that we don't need to update any more callsites of CpuInfo::IsSupported -- To view, visit http://gerrit.cloudera.org:8080/10979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifece8949c143146d2a1b38d72d21c2d733bed90f Gerrit-Change-Number: 10979 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 19 Jul 2018 16:50:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3825: [WIP]Distributed RuntimeFiltering Work in progress patch for Distributing runtime filtering aggregation.
Sailesh Mukil has posted comments on this change. (
http://gerrit.cloudera.org:8080/10937 )
Change subject: IMPALA-3825: [WIP]Distributed RuntimeFiltering Work in progress
patch for Distributing runtime filtering aggregation.
..
Patch Set 1:
(9 comments)
I added some high level comments. After addressing these high level comments,
you can clean up the code so we can do a second round of reviews.
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/common/global-flags.cc@132
PS1, Line 132: DEFINE_bool(enable_distributed_filter_aggregation, true,
: "Enables aggregation of filters"
: "in a distributed manner, set to false to revert to
coordinator based aggregation");
After looking through the code, it seems better to avoid having this flag.
Let's get rid of coordinator-filter-state.* completely, and go with the new
approach you've done. When we need to compare the 2 approaches for benchmarking
purposes, we can use 2 branches.
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/runtime/coordinator-backend-state.cc@93
PS1, Line 93: x.second.ToThrift();
Move inside if(), since it's only needed in that case.
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/runtime/coordinator-backend-state.cc@94
PS1, Line 94: AggregatorRoutingTable::const_iterator it =
aggregator_routing_table.find(x.first);
: if (it != aggregator_routing_table.end()) {
: tfs.__set_aggregator_address(it->second);
: rpc_params->filter_routing_table.insert(
: std::pair(x.first, tfs));
: }
Wouldn't all filters that are present in 'filter_routing_table' be present in
'aggregator_routing_table' ?
If that's the case, we're doing the same work multiple times here (once per
BackendState), and we would probably be better off not having the
'aggregator_routing_table' at all and just set the aggregator address in
'FilterState' as part of InitFilterRoutingTable().
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/runtime/coordinator-backend-state.cc@101
PS1, Line 101:
We also don't need to send the filter routing table to the nodes that don't
aggregate or don't produce right? If it's not simple to do, you can leave a
TODO for now.
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/runtime/coordinator.cc@284
PS1, Line 284:
We can set the aggregator address in the FilterState here, and not have the
aggregator_routing_table.
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/runtime/query-exec-mgr.h
File be/src/runtime/query-exec-mgr.h:
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/runtime/query-exec-mgr.h@22
PS1, Line 22: #include
None of these new #includes should be needed since you haven't added any new
code to this file.
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/runtime/query-state.cc@559
PS1, Line 559:
Add a TODO here to call PublishFilterFromAggregator() asynchronously using a
thread pool.
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/runtime/runtime-filter-bank.cc
File be/src/runtime/runtime-filter-bank.cc:
http://gerrit.cloudera.org:8080/#/c/10937/1/be/src/runtime/runtime-filter-bank.cc@205
PS1, Line 205: SendFilterToAggregator(
What if the aggregator and the producer are on the same node? We can consider
short-circuiting that path. Add a TODO for that.
http://gerrit.cloudera.org:8080/#/c/10937/1/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:
http://gerrit.cloudera.org:8080/#/c/10937/1/bin/bootstrap_toolchain.py@430
PS1, Line 430: "flatbuffers", "gcc", "gflags", "glog",
I'm guessing this was a mistake?
--
To view, visit http://gerrit.cloudera.org:8080/10937
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0289ee39787d8e9c13a16c3d7d64f471bc554536
Gerrit-Change-Number: 10937
Gerrit-PatchSet: 1
Gerrit-Owner: Rahul Shivu Mahadev
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Comment-Date: Thu, 19 Jul 2018 16:42:32 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7277. Support INSERT and LOAD DATA statements in LocalCatalog
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10914 ) Change subject: IMPALA-7277. Support INSERT and LOAD DATA statements in LocalCatalog .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10914 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4ae47a0b58022ed77abe51d2596c2b1d0111fae3 Gerrit-Change-Number: 10914 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 16:20:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Fredy Wijaya has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10939 ) Change subject: IMPALA-7259: Improve Impala shell performance .. IMPALA-7259: Improve Impala shell performance This patch fixes the slow performance in Impala shell, especially for large queries by replacing all calls to sqlparse.format(sql_string, strip_comments=True) with the custom implementation of strip comments that does not use grouping. The code to strip leading comments was also refactored to not use grouping. * Benchmark running a query with 12K columns * Before the patch: $ time impala-shell.sh -f large.sql --quiet real2m4.154s user2m0.536s sys 0m0.088s After the patch: $ time impala-shell.sh -f large.sql --quiet real0m3.885s user0m1.516s sys 0m0.048s Testing: - Added a new test to test the Impala shell performance - Ran all shell tests on Python 2.6 and Python 2.7 Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Reviewed-on: http://gerrit.cloudera.org:8080/10939 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M shell/impala_shell.py M tests/shell/test_shell_commandline.py 2 files changed, 70 insertions(+), 15 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 14 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6490: Reconnect shell when remote restarts
Attila Jeges has posted comments on this change. (
http://gerrit.cloudera.org:8080/10992 )
Change subject: IMPALA-6490: Reconnect shell when remote restarts
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10992/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10992/1//COMMIT_MSG@9
PS1, Line 9: If the remote impalad died while the shell waited for
nit: can you wrap lines in git commits at 60 chars? formatting in git tools
prefer this
If you edit commit messages with git you can have it auto set this w/ this in
your vimrc:
au FileType gitcommit set tw=60
http://gerrit.cloudera.org:8080/#/c/10992/1/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/10992/1/shell/impala_client.py@233
PS1, Line 233: """Checks to see if the current Impala connection is still
alive. If not, an exception
: will be raised."""
Comment is out of date now.
http://gerrit.cloudera.org:8080/#/c/10992/1/shell/impala_client.py@235
PS1, Line 235: if self.connected:
: return self.imp_service.PingImpalaService()
Since test_connection() now returns a True/False value, maybe we should catch
exceptions here (and return False) instead of requiring the caller to deal with
it.
Also, it would make the code cleaner if we returned False explicitly if
self.conneced is False.
http://gerrit.cloudera.org:8080/#/c/10992/1/tests/shell/test_shell_interactive.py
File tests/shell/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/10992/1/tests/shell/test_shell_interactive.py@157
PS1, Line 157: @pytest.mark.execute_serially
: def test_auto_reconnect(self):
: """Test reconnect after restarting the remote impalad
without using connect;"""
: kill_cluster = " ".join([START_CLUSTER, "--kill"])
: pexpect.run(kill_cluster)
: pexpect.run(START_CLUSTER)
: proc = pexpect.spawn(SHELL_CMD)
: proc.expect(":21000] default>")
: # Disconnect
: pexpect.run(kill_cluster)
: proc.sendline("show tables;")
: proc.expect("[Not connected]")
: # Restarting Impalad
: pexpect.run(START_CLUSTER)
: # Check reconnect
: proc.sendline("show tables;")
: proc.expect("large_table")
Please check ./tests/custom_cluster/test_shell_interactive_reconnect.py.
Probably this test should be moved there.
--
To view, visit http://gerrit.cloudera.org:8080/10992
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia13365a9696886f01294e98054cf4e7cd66ab712
Gerrit-Change-Number: 10992
Gerrit-PatchSet: 1
Gerrit-Owner: Le Minh Nghia
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 19 Jul 2018 15:23:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6490: Reconnect shell when remote restarts
Le Minh Nghia has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10992 Change subject: IMPALA-6490: Reconnect shell when remote restarts .. IMPALA-6490: Reconnect shell when remote restarts If the remote impalad died while the shell waited for a command to complete, the shell disconnected. Previously after restarting the remote impalad, we need to run "connect;" to reconnect, now the shell will automatically reconnect. Testing: Added test_auto_connect Change-Id: Ia13365a9696886f01294e98054cf4e7cd66ab712 --- M shell/impala_client.py M shell/impala_shell.py M tests/shell/test_shell_interactive.py 3 files changed, 32 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/10992/1 -- To view, visit http://gerrit.cloudera.org:8080/10992 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia13365a9696886f01294e98054cf4e7cd66ab712 Gerrit-Change-Number: 10992 Gerrit-PatchSet: 1 Gerrit-Owner: Le Minh Nghia
[Impala-ASF-CR] IMPALA-7315: fix test update with clear entries flag race
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10986 ) Change subject: IMPALA-7315: fix test_update_with_clear_entries_flag race .. IMPALA-7315: fix test_update_with_clear_entries_flag race We need to wait for the subscriber to process the second update in order to guarantee that the first update for that subscriber has been applied. Otherwise there is a race window where the second subscriber may see the older version of the statestore topic. Change-Id: I2be2b61b6deb0228fbc5a242e43076beb8871454 Reviewed-on: http://gerrit.cloudera.org:8080/10986 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/statestore/test_statestore.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2be2b61b6deb0228fbc5a242e43076beb8871454 Gerrit-Change-Number: 10986 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7315: fix test update with clear entries flag race
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10986 ) Change subject: IMPALA-7315: fix test_update_with_clear_entries_flag race .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2be2b61b6deb0228fbc5a242e43076beb8871454 Gerrit-Change-Number: 10986 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 09:49:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7315: fix test update with clear entries flag race
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10986 ) Change subject: IMPALA-7315: fix test_update_with_clear_entries_flag race .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2be2b61b6deb0228fbc5a242e43076beb8871454 Gerrit-Change-Number: 10986 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 06:35:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7315: fix test update with clear entries flag race
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10986 ) Change subject: IMPALA-7315: fix test_update_with_clear_entries_flag race .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2841/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2be2b61b6deb0228fbc5a242e43076beb8871454 Gerrit-Change-Number: 10986 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 19 Jul 2018 06:35:35 + Gerrit-HasComments: No
