[Impala-ASF-CR] IMPALA-7010: don't run memory usage tests on non-HDFS

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10370 )

Change subject: IMPALA-7010: don't run memory usage tests on non-HDFS
..


Patch Set 5:

This change did not cherrypick successfully into branch 2.x. To resolve this, 
please do the cherry-pick manually and submit it to Gerrit at refs/for/2.x or 
add an exception to the branch 2.x copy of bin/ignored_commits.json. Thanks, 
your friendly bot at https://jenkins.impala.io/job/cherrypick-2.x-and-test/482/ 
.


--
To view, visit http://gerrit.cloudera.org:8080/10370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
Gerrit-Change-Number: 10370
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Sat, 12 May 2018 01:30:05 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7010: don't run memory usage tests on non-HDFS

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10370 )

Change subject: IMPALA-7010: don't run memory usage tests on non-HDFS
..


Patch Set 4: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/10370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
Gerrit-Change-Number: 10370
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 11 May 2018 22:41:48 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7010: don't run memory usage tests on non-HDFS

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/10370 )

Change subject: IMPALA-7010: don't run memory usage tests on non-HDFS
..

IMPALA-7010: don't run memory usage tests on non-HDFS

Moved a number of tests with tuned mem_limits. In some cases
this required separating the tests from non-tuned functional
tests.

TestQueryMemLimit used very high and very low limits only, so seemed
safe to run in all configurations.

Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
Reviewed-on: http://gerrit.cloudera.org:8080/10370
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins 
---
A testdata/workloads/functional-query/queries/QueryTest/insert-mem-limit.test
M testdata/workloads/functional-query/queries/QueryTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test
A 
testdata/workloads/functional-query/queries/QueryTest/kudu_insert_mem_limit.test
A 
testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch-mem-limit.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test
M testdata/workloads/functional-query/queries/QueryTest/spilling-aggs.test
R 
testdata/workloads/functional-query/queries/QueryTest/spilling-regression-exhaustive.test
M tests/common/skip.py
M tests/query_test/test_insert.py
M tests/query_test/test_kudu.py
M tests/query_test/test_mem_usage_scaling.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_sort.py
M tests/query_test/test_spilling.py
15 files changed, 196 insertions(+), 156 deletions(-)

Approvals:
  Tim Armstrong: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/10370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
Gerrit-Change-Number: 10370
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-7010: don't run memory usage tests on non-HDFS

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10370 )

Change subject: IMPALA-7010: don't run memory usage tests on non-HDFS
..


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2454/


--
To view, visit http://gerrit.cloudera.org:8080/10370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
Gerrit-Change-Number: 10370
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 11 May 2018 19:14:16 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7010: don't run memory usage tests on non-HDFS

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10370 )

Change subject: IMPALA-7010: don't run memory usage tests on non-HDFS
..


Patch Set 4: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/10370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
Gerrit-Change-Number: 10370
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 11 May 2018 19:14:05 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7010: don't run memory usage tests on non-HDFS

[Dan Hecht (Code Review)]

Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10370 )

Change subject: IMPALA-7010: don't run memory usage tests on non-HDFS
..


Patch Set 3: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10370/2/tests/common/skip.py
File tests/common/skip.py:

http://gerrit.cloudera.org:8080/#/c/10370/2/tests/common/skip.py@130
PS2, Line 130:   qualified_path = pytest.mark.skipif(IS_LOCAL,
 :   reason="Tests rely on HDFS qualified paths")
> Done. The local filesystem tests run with a single impalad, so it ends up w
Ah, didn't realize that's that we only started one node for local fs tests. I'm 
not sure why that is, but okay.



--
To view, visit http://gerrit.cloudera.org:8080/10370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
Gerrit-Change-Number: 10370
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 11 May 2018 18:54:36 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7010: don't run memory usage tests on non-HDFS

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10370 )

Change subject: IMPALA-7010: don't run memory usage tests on non-HDFS
..


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/10370/2/testdata/workloads/functional-query/queries/QueryTest/insert-mem-limit.test
File 
testdata/workloads/functional-query/queries/QueryTest/insert-mem-limit.test:

http://gerrit.cloudera.org:8080/#/c/10370/2/testdata/workloads/functional-query/queries/QueryTest/insert-mem-limit.test@3
PS2, Line 3: # Check that hdfs writers respects mem_limit.
> maybe add comments to the test files saying the mem limit is tuned for 3 no
Done


http://gerrit.cloudera.org:8080/#/c/10370/2/tests/common/skip.py
File tests/common/skip.py:

http://gerrit.cloudera.org:8080/#/c/10370/2/tests/common/skip.py@130
PS2, Line 130:   mem_usage_different = pytest.mark.skipif(IS_LOCAL,
 :   reason="Memory limit too low when running single node")
> do we still need that? should all of these tests be "tuned_for_minicluster"
Done. The local filesystem tests run with a single impalad, so it ends up with 
3x the memory consumption in some cases.



--
To view, visit http://gerrit.cloudera.org:8080/10370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
Gerrit-Change-Number: 10370
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 11 May 2018 18:41:43 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7010: don't run memory usage tests on non-HDFS

[Tim Armstrong (Code Review)]

Hello Sailesh Mukil, Impala Public Jenkins, Dan Hecht,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/10370

to look at the new patch set (#3).

Change subject: IMPALA-7010: don't run memory usage tests on non-HDFS
..

IMPALA-7010: don't run memory usage tests on non-HDFS

Moved a number of tests with tuned mem_limits. In some cases
this required separating the tests from non-tuned functional
tests.

TestQueryMemLimit used very high and very low limits only, so seemed
safe to run in all configurations.

Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
---
A testdata/workloads/functional-query/queries/QueryTest/insert-mem-limit.test
M testdata/workloads/functional-query/queries/QueryTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test
A 
testdata/workloads/functional-query/queries/QueryTest/kudu_insert_mem_limit.test
A 
testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch-mem-limit.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test
M testdata/workloads/functional-query/queries/QueryTest/spilling-aggs.test
R 
testdata/workloads/functional-query/queries/QueryTest/spilling-regression-exhaustive.test
M tests/common/skip.py
M tests/query_test/test_insert.py
M tests/query_test/test_kudu.py
M tests/query_test/test_mem_usage_scaling.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_sort.py
M tests/query_test/test_spilling.py
15 files changed, 196 insertions(+), 156 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/10370/3
--
To view, visit http://gerrit.cloudera.org:8080/10370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
Gerrit-Change-Number: 10370
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sailesh Mukil 

[Impala-ASF-CR] IMPALA-7010: don't run memory usage tests on non-HDFS

[Dan Hecht (Code Review)]

Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10370 )

Change subject: IMPALA-7010: don't run memory usage tests on non-HDFS
..


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/10370/2/testdata/workloads/functional-query/queries/QueryTest/insert-mem-limit.test
File 
testdata/workloads/functional-query/queries/QueryTest/insert-mem-limit.test:

http://gerrit.cloudera.org:8080/#/c/10370/2/testdata/workloads/functional-query/queries/QueryTest/insert-mem-limit.test@3
PS2, Line 3: # Check that hdfs writers respects mem_limit.
maybe add comments to the test files saying the mem limit is tuned for 3 node 
hdfs cluster, to help prevent these from accidentally being added back to the 
full matrix.


http://gerrit.cloudera.org:8080/#/c/10370/2/tests/common/skip.py
File tests/common/skip.py:

http://gerrit.cloudera.org:8080/#/c/10370/2/tests/common/skip.py@130
PS2, Line 130:   mem_usage_different = pytest.mark.skipif(IS_LOCAL,
 :   reason="Memory limit too low when running single node")
do we still need that? should all of these tests be "tuned_for_minicluster".

And I don't understand the reason here anyway - local fs tests still use the 
minicluster (as do hdfs and isilon for that matter). They just use a different 
default filesystem.



--
To view, visit http://gerrit.cloudera.org:8080/10370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
Gerrit-Change-Number: 10370
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Fri, 11 May 2018 17:36:38 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7010: don't run memory usage tests on non-HDFS

[Tim Armstrong (Code Review)]

Tim Armstrong has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10370


Change subject: IMPALA-7010: don't run memory usage tests on non-HDFS
..

IMPALA-7010: don't run memory usage tests on non-HDFS

Moved a number of tests with tuned mem_limits. In some cases
this required separating the tests from non-tuned functional
tests.

TestQueryMemLimit used very high and very low limits only, so seemed
safe to run in all configurations.

Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
---
A testdata/workloads/functional-query/queries/QueryTest/insert-mem-limit.test
M testdata/workloads/functional-query/queries/QueryTest/insert.test
M testdata/workloads/functional-query/queries/QueryTest/kudu_insert.test
A 
testdata/workloads/functional-query/queries/QueryTest/kudu_insert_mem_limit.test
A 
testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch-mem-limit.test
M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test
M testdata/workloads/functional-query/queries/QueryTest/spilling-aggs.test
R 
testdata/workloads/functional-query/queries/QueryTest/spilling-regression-exhaustive.test
M tests/common/skip.py
M tests/query_test/test_insert.py
M tests/query_test/test_kudu.py
M tests/query_test/test_mem_usage_scaling.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_sort.py
M tests/query_test/test_spilling.py
15 files changed, 183 insertions(+), 150 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/10370/1
--
To view, visit http://gerrit.cloudera.org:8080/10370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9686195a29dde2d87b19ef8bb0e93e08f8bee662
Gerrit-Change-Number: 10370
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins