[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Vuk Ercegovac has abandoned this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 5 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 5: decided to assert that all plans should be the same on all fs's. the change then will be to the configs to ensure that. the known parameter that causes changes is block size, so we should separately test for that type of change (not indirectly via the choice of fs). -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 5 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Feb 2018 19:32:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Dimitris Tsirogiannis has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 5: What's the consensus on this one? If we're not moving forward, let's abandon this change. -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 5 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 07 Feb 2018 19:25:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 5: noticed a discrepancy in the s3 ranges from run-to-run for parquet files. before merging, I'd like to figure out why its unstable and what to do about it. -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 5 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 12 Jan 2018 20:06:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 5: alex, pls have a look and let me know if you have additional comments. -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 5 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 11 Jan 2018 20:01:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Dimitris Tsirogiannis has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 5: Code-Review+2 I am fine with the change to unblock the tests. Feel free to check with Alex in case he has any comments/concerns. -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 5 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 11 Jan 2018 19:54:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Hello Thomas Tauber-Marshall, Taras Bobrovytsky, Lars Volker, Dimitris Tsirogiannis, Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8890 to look at the new patch set (#5). Change subject: IMPALA-3526: update FE tests to pass on S3 .. IMPALA-3526: update FE tests to pass on S3 Plans for some queries/same data may differ when the same data is stored on S3 vs. HDFS. This is due to block size differences used to enumerate range scans on the different file systems. As a result, FE tests have been disabled for S3 configurations. This has also led to staleness in the tests that were specific to S3. Most of the broken tests are due to staleness, but several are due to true plan differences. This change fixes stale tests and separates out those test queries whose plans differ across S3 and HDFS. Tests: - ran FE tests on S3 Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 --- M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/S3PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/join-order-hdfs.test A testdata/workloads/functional-planner/queries/PlannerTest/join-order-s3.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/s3.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-hdfs.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-s3.test R testdata/workloads/functional-planner/queries/PlannerTest/tpch.test 9 files changed, 2,180 insertions(+), 1,065 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/8890/5 -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 5 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 4: (1 comment) yes, missed adding the file when renaming. added now. http://gerrit.cloudera.org:8080/#/c/8890/4/fe/src/test/java/org/apache/impala/planner/PlannerTest.java File fe/src/test/java/org/apache/impala/planner/PlannerTest.java: http://gerrit.cloudera.org:8080/#/c/8890/4/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@132 PS4, Line 132: // Tests queries that are insensitive to all filesystems. > maybe also say "whose plans are insensitive to the underlying filesystem" Done. changed the wording a bit in all places. -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 4 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 11 Jan 2018 00:37:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Dimitris Tsirogiannis has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 4: (1 comment) Is there a file missing (tpch.test)? http://gerrit.cloudera.org:8080/#/c/8890/4/fe/src/test/java/org/apache/impala/planner/PlannerTest.java File fe/src/test/java/org/apache/impala/planner/PlannerTest.java: http://gerrit.cloudera.org:8080/#/c/8890/4/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@132 PS4, Line 132: // Tests queries that are insensitive to all filesystems. maybe also say "whose plans are insensitive to the underlying filesystem" -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 4 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 11 Jan 2018 00:01:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/8890 )
Change subject: IMPALA-3526: update FE tests to pass on S3
..
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/8890/3/fe/src/test/java/org/apache/impala/planner/PlannerTest.java
File fe/src/test/java/org/apache/impala/planner/PlannerTest.java:
http://gerrit.cloudera.org:8080/#/c/8890/3/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@130
PS3, Line 130: @Test
: public void testJoinOrder() {
: runPlannerTestFile("join-order");
: }
:
: @Test
: public void testJoinOrderHdfs() {
: runPlannerTestFile("join-order-hdfs");
: }
> Just by reading this section, it's not clear what we're testing in join-ord
added more comments.
http://gerrit.cloudera.org:8080/#/c/8890/3/testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test:
http://gerrit.cloudera.org:8080/#/c/8890/3/testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test@1
PS3, Line 1: # TPCH-Q1
> Shouldn't this be renamed to tpch.test to be consistent with the other rena
renamed. I was a bit on the fence for this one to preserve the existing "all"--
there are other tests for views, nested, etc-- but I'm fine with changing the
name.
--
To view, visit http://gerrit.cloudera.org:8080/8890
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141
Gerrit-Change-Number: 8890
Gerrit-PatchSet: 3
Gerrit-Owner: Vuk Ercegovac
Gerrit-Reviewer: Alex Behm
Gerrit-Reviewer: Dimitris Tsirogiannis
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Taras Bobrovytsky
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Wed, 10 Jan 2018 23:30:33 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Hello Thomas Tauber-Marshall, Taras Bobrovytsky, Lars Volker, Dimitris Tsirogiannis, Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8890 to look at the new patch set (#4). Change subject: IMPALA-3526: update FE tests to pass on S3 .. IMPALA-3526: update FE tests to pass on S3 Plans for some queries/same data may differ when the same data is stored on S3 vs. HDFS. This is due to block size differences used to enumerate range scans on the different file systems. As a result, FE tests have been disabled for S3 configurations. This has also led to staleness in the tests that were specific to S3. Most of the broken tests are due to staleness, but several are due to true plan differences. This change fixes stale tests and separates out those test queries whose plans differ across S3 and HDFS. Tests: - ran FE tests on S3 Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 --- M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/S3PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/join-order-hdfs.test A testdata/workloads/functional-planner/queries/PlannerTest/join-order-s3.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/s3.test D testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-hdfs.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-s3.test 9 files changed, 2,180 insertions(+), 4,428 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/8890/4 -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 4 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Dimitris Tsirogiannis has posted comments on this change. (
http://gerrit.cloudera.org:8080/8890 )
Change subject: IMPALA-3526: update FE tests to pass on S3
..
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/8890/3/fe/src/test/java/org/apache/impala/planner/PlannerTest.java
File fe/src/test/java/org/apache/impala/planner/PlannerTest.java:
http://gerrit.cloudera.org:8080/#/c/8890/3/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@130
PS3, Line 130: @Test
: public void testJoinOrder() {
: runPlannerTestFile("join-order");
: }
:
: @Test
: public void testJoinOrderHdfs() {
: runPlannerTestFile("join-order-hdfs");
: }
Just by reading this section, it's not clear what we're testing in join-order
vs join-order-hdfs (same comment applies to the s3planner tests). Maybe add a
comment to explain what is going on with these tests and why the separation is
needed.
Also, another thought would be to rename join-order to join-order-all-fs or
join-order-common (or something along these lines).
http://gerrit.cloudera.org:8080/#/c/8890/3/testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test:
http://gerrit.cloudera.org:8080/#/c/8890/3/testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test@1
PS3, Line 1: # TPCH-Q1
Shouldn't this be renamed to tpch.test to be consistent with the other renames
you did?
--
To view, visit http://gerrit.cloudera.org:8080/8890
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141
Gerrit-Change-Number: 8890
Gerrit-PatchSet: 3
Gerrit-Owner: Vuk Ercegovac
Gerrit-Reviewer: Alex Behm
Gerrit-Reviewer: Dimitris Tsirogiannis
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Taras Bobrovytsky
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Wed, 10 Jan 2018 21:08:16 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 2: changed the test query file names. -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 08 Jan 2018 20:28:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Hello Thomas Tauber-Marshall, Taras Bobrovytsky, Lars Volker, Dimitris Tsirogiannis, Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8890 to look at the new patch set (#3). Change subject: IMPALA-3526: update FE tests to pass on S3 .. IMPALA-3526: update FE tests to pass on S3 Plans for some queries/same data may differ when the same data is stored on S3 vs. HDFS. This is due to block size differences used to enumerate range scans on the different file systems. As a result, FE tests have been disabled for S3 configurations. This has also led to staleness in the tests that were specific to S3. Most of the broken tests are due to staleness, but several are due to true plan differences. This change fixes stale tests and separates out those test queries whose plans differ across S3 and HDFS. Tests: - ran FE tests on S3 Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 --- M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/S3PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/join-order-hdfs.test A testdata/workloads/functional-planner/queries/PlannerTest/join-order-s3.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/s3.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-hdfs.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-s3.test 9 files changed, 2,171 insertions(+), 1,064 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/8890/3 -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 2: For an explanation for the plan difference, pls have a look at this comment I posted on the JIRA: https://issues.apache.org/jira/browse/IMPALA-3526?focusedCommentId=16296008&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16296008 I think the "all" is there to contrast with other tpch tests (kudu, nested, views). I may have misread that one though. My take is that there is a separate design discussion for this issue (see the JIRA comment). This patch is just for enabling FE tests for S3. I'm also keeping ADLS testing as separate for now. -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 08 Jan 2018 19:28:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Dimitris Tsirogiannis has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 2: Thanks for working on that. I am not sure I understand why the plan changes for some queries (e.g. TPC-H Q2). Any ideas what causes it? Also, it's kind of weird that some files are now called *-all.test without having *all* the queries :). We may need to rethink how to move forward, I wouldn't want to create another set of files if for some reason plans change again in ALDS. -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 03 Jan 2018 00:49:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Taras Bobrovytsky has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 2: I agree that it's probably best to unblock first. Making it more general can be done in a future patch. -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 21 Dec 2017 23:39:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 2: I thought about it, but since it was relatively localized, yet blocking all FE tests on S3, I went for this fairly manual change. I think we should unblock first, turn these tests on, then make it more general. I'm fine with trying ideas along the lines you suggest so let me know if you have a preference for how to stage this. -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 21 Dec 2017 23:15:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Taras Bobrovytsky has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 1: It might be worth thinking about how to address the problem of having different plans on different storage systems more generally. For example, maybe we could extend the testing framework so that we could have multiple PlAN sections for each QUERY section. The PLAN section could be annotated with storage information. For example: PLAN PLAN DISTRIBUTEDPLAN -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 21 Dec 2017 20:59:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Hello Thomas Tauber-Marshall, Taras Bobrovytsky, Lars Volker, Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8890 to look at the new patch set (#2). Change subject: IMPALA-3526: update FE tests to pass on S3 .. IMPALA-3526: update FE tests to pass on S3 Plans for some queries/same data may differ when the same data is stored on S3 vs. HDFS. This is due to block size differences used to enumerate range scans on the different file systems. As a result, FE tests have been disabled for S3 configurations. This has also led to staleness in the tests that were specific to S3. Most of the broken tests are due to staleness, but several are due to true plan differences. This change fixes stale tests and separates out those test queries whose plans differ across S3 and HDFS. Tests: - ran FE tests on S3 Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 --- M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/S3PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/join-order-hdfs.test A testdata/workloads/functional-planner/queries/PlannerTest/join-order-s3.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/s3.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-all-hdfs.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-all-s3.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test 9 files changed, 2,171 insertions(+), 1,064 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/8890/2 -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/8890/1/testdata/workloads/functional-planner/queries/PlannerTest/join-order-hdfs.test File testdata/workloads/functional-planner/queries/PlannerTest/join-order-hdfs.test: http://gerrit.cloudera.org:8080/#/c/8890/1/testdata/workloads/functional-planner/queries/PlannerTest/join-order-hdfs.test@1 PS1, Line 1: S3 > hdfs Done -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 21 Dec 2017 20:42:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/8890 ) Change subject: IMPALA-3526: update FE tests to pass on S3 .. Patch Set 1: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/8890/1/testdata/workloads/functional-planner/queries/PlannerTest/join-order-hdfs.test File testdata/workloads/functional-planner/queries/PlannerTest/join-order-hdfs.test: http://gerrit.cloudera.org:8080/#/c/8890/1/testdata/workloads/functional-planner/queries/PlannerTest/join-order-hdfs.test@1 PS1, Line 1: S3 hdfs -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 21 Dec 2017 19:49:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3526: update FE tests to pass on S3
Vuk Ercegovac has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8890 Change subject: IMPALA-3526: update FE tests to pass on S3 .. IMPALA-3526: update FE tests to pass on S3 Plans for some queries/same data may differ when the same data is stored on S3 vs. HDFS. This is due to block size differences used to enumerate range scans on the different file systems. As a result, FE tests have been disabled for S3 configurations. This has also led to staleness in the tests that were specific to S3. Most of the broken tests are due to staleness, but several are due to true plan differences. This change fixes stale tests and separates out those test queries whose plans differ across S3 and HDFS. Tests: - ran FE tests on S3 Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 --- M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/S3PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/join-order-hdfs.test A testdata/workloads/functional-planner/queries/PlannerTest/join-order-s3.test M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/functional-planner/queries/PlannerTest/s3.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-all-hdfs.test A testdata/workloads/functional-planner/queries/PlannerTest/tpch-all-s3.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test 9 files changed, 2,171 insertions(+), 1,064 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/8890/1 -- To view, visit http://gerrit.cloudera.org:8080/8890 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4c8221949e76b0a0b9192e6b56c4da5eeae04141 Gerrit-Change-Number: 8890 Gerrit-PatchSet: 1 Gerrit-Owner: Vuk Ercegovac
