[Impala-ASF-CR] IMPALA-10171: Create query options for local time related flags
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16469 to look at the new patch set (#5). Change subject: IMPALA-10171: Create query options for local time related flags .. IMPALA-10171: Create query options for local time related flags convert_legacy_hive_parquet_utc_timestamps and use_local_tz_for_unix_timestamp_conversions were controllable only by flags until now. After this change the old flags are only used on the Coordinator to set the defaults for the query options. If default_query_options also sets these query options, it will take precedence over the old flags. Possible follow up work: - the old flags could be deprecated, as default_query_options can be used to set this on a server level - testing these functionalities no longer needs custom cluster tests - rewriting the existing tests could speed up test execution Testing: - expr-test was rewritten to use the query option instead of the flag - extended the custom cluster tests for the old flags to also use the query options - ran related tests Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 --- M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/parquet-column-readers.cc M be/src/exprs/expr-test.cc M be/src/exprs/timestamp-functions.h M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/workloads/functional-query/queries/QueryTest/utc-timestamp-functions.test M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py M tests/custom_cluster/test_local_tz_conversion.py 14 files changed, 183 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/16469/5 -- To view, visit http://gerrit.cloudera.org:8080/16469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 Gerrit-Change-Number: 16469 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10171: Create query options for local time related flags
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16469 to look at the new patch set (#6). Change subject: IMPALA-10171: Create query options for local time related flags .. IMPALA-10171: Create query options for local time related flags convert_legacy_hive_parquet_utc_timestamps and use_local_tz_for_unix_timestamp_conversions were controllable only by flags until now. After this change the old flags are only used on the Coordinator to set the defaults for the query options. If default_query_options also sets these query options, it will take precedence over the old flags. Possible follow up work: - the old flags could be deprecated, as default_query_options can be used to set this on a server level - testing these functionalities no longer needs custom cluster tests - rewriting the existing tests could speed up test execution Testing: - expr-test was rewritten to use the query option instead of the flag - extended the custom cluster tests for the old flags to also use the query options - ran related tests Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 --- M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/parquet-column-readers.cc M be/src/exprs/expr-test.cc M be/src/exprs/timestamp-functions.h M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/workloads/functional-query/queries/QueryTest/utc-timestamp-functions.test M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py M tests/custom_cluster/test_local_tz_conversion.py 14 files changed, 183 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/16469/6 -- To view, visit http://gerrit.cloudera.org:8080/16469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 Gerrit-Change-Number: 16469 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10171: Create query options for local time related flags
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16469 ) Change subject: IMPALA-10171: Create query options for local time related flags .. Patch Set 7: (3 comments) In path 5 and 6 I didn't rebase with a fresh enough version, please ignore them. http://gerrit.cloudera.org:8080/#/c/16469/4/be/src/exprs/timestamp-functions.h File be/src/exprs/timestamp-functions.h: http://gerrit.cloudera.org:8080/#/c/16469/4/be/src/exprs/timestamp-functions.h@112 PS4, Line 112: /// query option use_local_tz_for_unix_timestamp_conversions. If it is true, the > These references to the flag need updating. Done http://gerrit.cloudera.org:8080/#/c/16469/4/common/thrift/ImpalaInternalService.thrift File common/thrift/ImpalaInternalService.thrift: http://gerrit.cloudera.org:8080/#/c/16469/4/common/thrift/ImpalaInternalService.thrift@452 PS4, Line 452: > nit: extra space. Done http://gerrit.cloudera.org:8080/#/c/16469/4/testdata/workloads/functional-query/queries/QueryTest/utc-timestamp-functions.test File testdata/workloads/functional-query/queries/QueryTest/utc-timestamp-functions.test: http://gerrit.cloudera.org:8080/#/c/16469/4/testdata/workloads/functional-query/queries/QueryTest/utc-timestamp-functions.test@20 PS4, Line 20: oca > local-timestamp-functions.test, no? oops, done -- To view, visit http://gerrit.cloudera.org:8080/16469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 Gerrit-Change-Number: 16469 Gerrit-PatchSet: 7 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 08:13:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10171: Create query options for local time related flags
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16469 ) Change subject: IMPALA-10171: Create query options for local time related flags .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7192/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 Gerrit-Change-Number: 16469 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 08:17:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10171: Create query options for local time related flags
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16469 ) Change subject: IMPALA-10171: Create query options for local time related flags .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7193/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 Gerrit-Change-Number: 16469 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 08:18:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9652: CTAS doesn't respect transactional properties
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16472 Change subject: IMPALA-9652: CTAS doesn't respect transactional properties .. IMPALA-9652: CTAS doesn't respect transactional properties Because of a bug the INSERT part of a CTAS statement didn't run in a transaction. Testing: * Added CTAS statements and SHOW FILES queries to acid-insert.test Check if the files are created in a delta directory, if so, then the INSERT must have been running in a transaction. Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 --- M fe/src/main/java/org/apache/impala/service/Frontend.java M testdata/workloads/functional-query/queries/QueryTest/acid-insert.test 2 files changed, 56 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/16472/1 -- To view, visit http://gerrit.cloudera.org:8080/16472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 Gerrit-Change-Number: 16472 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9652: CTAS doesn't respect transactional properties
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16472 ) Change subject: IMPALA-9652: CTAS doesn't respect transactional properties .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6443/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 Gerrit-Change-Number: 16472 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Sep 2020 10:53:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9652: CTAS doesn't respect transactional properties
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16472 ) Change subject: IMPALA-9652: CTAS doesn't respect transactional properties .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7194/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 Gerrit-Change-Number: 16472 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Sep 2020 11:08:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9652: CTAS doesn't respect transactional properties
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16472 ) Change subject: IMPALA-9652: CTAS doesn't respect transactional properties .. Patch Set 1: Code-Review+2 (3 comments) Thanks for fixing these bugs, I have only nits. http://gerrit.cloudera.org:8080/#/c/16472/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16472/1//COMMIT_MSG@9 PS1, Line 9: Because of a bug the INSERT part of a CTAS statement didn't Can you add more info about how we worked before the change, and how we will work after it? E.g. before the change the files written by CTAS were put the the root directory as original files. After the change the files will go to a delta directory like for insert and the table will be locked during the operation. http://gerrit.cloudera.org:8080/#/c/16472/1//COMMIT_MSG@10 PS1, Line 10: transaction Do I understand well that we are still not transactional in a multi table sense? So while the source tables will be locked, no refresh is done on them and the new table can be created based on their state at a previous transaction id. http://gerrit.cloudera.org:8080/#/c/16472/1/testdata/workloads/functional-query/queries/QueryTest/acid-insert.test File testdata/workloads/functional-query/queries/QueryTest/acid-insert.test: http://gerrit.cloudera.org:8080/#/c/16472/1/testdata/workloads/functional-query/queries/QueryTest/acid-insert.test@65 PS1, Line 65: : QUERY : drop table insertonly_nopart_ctas; Is there a reason why we do the cleanup here? The whole DB is dropped after running the test. -- To view, visit http://gerrit.cloudera.org:8080/16472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 Gerrit-Change-Number: 16472 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Sep 2020 11:23:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10177: Generate test-classpath.txt
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16450 to look at the new patch set (#3). Change subject: IMPALA-10177: Generate test-classpath.txt .. IMPALA-10177: Generate test-classpath.txt Currently the hive-jdbc jar is added under the scope 'test'. Therefore it won't be included in the build-classpath.txt because it only includes jars from the 'runtime' scope. Because of this run-hive-jdbc.sh throws ClassNotFoundException exception. This commit adds a new execution 'write-test-classpath' to pom.xml. It creates a test-classpath.txt. This can be used by scripts that need classes from the 'test' scope. Modified set-classpath.sh as well. If its first argument is 'test', then it also adds the contents of test-classpath.txt to the CLASSPATH. Now run-jdbc-client.sh sources 'set-classpath.sh test'. Testing: * executed core tests, turned out test_describe_formatted is broken, see IMPALA-10176 for details. Disabling that test for now. Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc --- M bin/run-jdbc-client.sh M bin/set-classpath.sh M fe/pom.xml M tests/metadata/test_metadata_query_statements.py 4 files changed, 20 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/16450/3 -- To view, visit http://gerrit.cloudera.org:8080/16450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc Gerrit-Change-Number: 16450 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
Gabor Kaszab has uploaded this change for review. (
http://gerrit.cloudera.org:8080/16473
Change subject: IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
..
IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
The previous error message contained only the input string but
including the format string as well would make debugging easier.
Extended error message:
SELECT CAST('0;367' as date format 'YY;DDD');
String to Date parse failed. Input '0;367' doesn't match with
format 'YY;IDD'
Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622
---
M be/src/exprs/cast-functions-ir.cc
M tests/query_test/test_cast_with_format.py
2 files changed, 17 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/16473/1
--
To view, visit http://gerrit.cloudera.org:8080/16473
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622
Gerrit-Change-Number: 16473
Gerrit-PatchSet: 1
Gerrit-Owner: Gabor Kaszab
[Impala-ASF-CR] Generate test-classpath.txt
Gabor Kaszab has uploaded a new patch set (#4) to the change originally created by Zoltan Borok-Nagy. ( http://gerrit.cloudera.org:8080/16450 ) Change subject: Generate test-classpath.txt .. Generate test-classpath.txt Currently the hive-jdbc jar is added under the scope 'test'. Therefore it won't be included in the build-classpath.txt because it only includes jars from the 'runtime' scope. Because of this run-hive-jdbc.sh throws ClassNotFoundException exception. This commit adds a new execution 'write-test-classpath' to pom.xml. It creates a test-classpath.txt. This can be used by scripts that need classes from the 'test' scope. Modified set-classpath.sh as well. If its first argument is 'test', then it also adds the contents of test-classpath.txt to the CLASSPATH. Now run-jdbc-client.sh sources 'set-classpath.sh test'. Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc --- M bin/run-jdbc-client.sh M bin/set-classpath.sh M fe/pom.xml 3 files changed, 18 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/16450/4 -- To view, visit http://gerrit.cloudera.org:8080/16450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc Gerrit-Change-Number: 16450 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] Generate test-classpath.txt
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16450 ) Change subject: Generate test-classpath.txt .. Patch Set 3: The pre-commit job failed because without this fix test_describe_formatted was broken, i.e. it compared two empty result sets. I disabled test_describe_formatted and opened IMPALA-10176 for it. -- To view, visit http://gerrit.cloudera.org:8080/16450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc Gerrit-Change-Number: 16450 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 12:12:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16473 ) Change subject: IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6444/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622 Gerrit-Change-Number: 16473 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Sep 2020 12:12:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16473 ) Change subject: IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16473/1/tests/query_test/test_cast_with_format.py File tests/query_test/test_cast_with_format.py: http://gerrit.cloudera.org:8080/#/c/16473/1/tests/query_test/test_cast_with_format.py@1461 PS1, Line 1461: ' flake8: E501 line too long (91 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/16473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622 Gerrit-Change-Number: 16473 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Sep 2020 12:13:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16473
to look at the new patch set (#2).
Change subject: IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
..
IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
The previous error message contained only the input string but
including the format string as well would make debugging easier.
Extended error message:
SELECT CAST('0;367' as date format 'YY;DDD');
String to Date parse failed. Input '0;367' doesn't match with
format 'YY;IDD'
Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622
---
M be/src/exprs/cast-functions-ir.cc
M tests/query_test/test_cast_with_format.py
2 files changed, 17 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/16473/2
--
To view, visit http://gerrit.cloudera.org:8080/16473
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622
Gerrit-Change-Number: 16473
Gerrit-PatchSet: 2
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10177: Generate test-classpath.txt
Hello Gabor Kaszab, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16450 to look at the new patch set (#5). Change subject: IMPALA-10177: Generate test-classpath.txt .. IMPALA-10177: Generate test-classpath.txt Currently the hive-jdbc jar is added under the scope 'test'. Therefore it won't be included in the build-classpath.txt because it only includes jars from the 'runtime' scope. Therefore run-hive-jdbc.sh throws ClassNotFoundException exception. This commit adds a new execution 'write-test-classpath' to pom.xml. It creates a test-classpath.txt. This can be used by scripts that need classes from the 'test' scope. Modified set-classpath.sh as well. If its first argument is 'test', then it also adds the contents of test-classpath.txt to the CLASSPATH. Now run-jdbc-client.sh sources 'set-classpath.sh test'. Testing: * executed core tests, turned out test_describe_formatted is broken, see IMPALA-10176 for details. Disabling that test for now. Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc --- M bin/run-jdbc-client.sh M bin/set-classpath.sh M fe/pom.xml M tests/metadata/test_metadata_query_statements.py 4 files changed, 20 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/16450/5 -- To view, visit http://gerrit.cloudera.org:8080/16450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc Gerrit-Change-Number: 16450 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10177: Generate test-classpath.txt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16450 ) Change subject: IMPALA-10177: Generate test-classpath.txt .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7195/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc Gerrit-Change-Number: 16450 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 12:30:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10177: Generate test-classpath.txt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16450 ) Change subject: IMPALA-10177: Generate test-classpath.txt .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7196/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc Gerrit-Change-Number: 16450 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 12:33:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16473 ) Change subject: IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7197/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622 Gerrit-Change-Number: 16473 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Sep 2020 12:34:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9652: CTAS doesn't respect transactional properties
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16472 to look at the new patch set (#2). Change subject: IMPALA-9652: CTAS doesn't respect transactional properties .. IMPALA-9652: CTAS doesn't respect transactional properties Because of a bug the INSERT part of a CTAS statement didn't run in a transaction and it just put the new files under the root directory of the table. This didn't cause too much problems because there couldn't be any concurrent operations as the table was under construction. However, this behavior is not working particularly well in the context of replication, as the notification event needs a transaction id. With this fix the INSERT operation runs in a transaction and the new files are created under a delta directory. Testing: * Added CTAS statements and SHOW FILES queries to acid-insert.test Check if the files are created in a delta directory, if so, then the INSERT must have been running in a transaction. Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 --- M fe/src/main/java/org/apache/impala/service/Frontend.java M testdata/workloads/functional-query/queries/QueryTest/acid-insert.test 2 files changed, 56 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/16472/2 -- To view, visit http://gerrit.cloudera.org:8080/16472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 Gerrit-Change-Number: 16472 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16473 ) Change subject: IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7198/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622 Gerrit-Change-Number: 16473 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Sep 2020 12:43:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10177: Generate test-classpath.txt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16450 ) Change subject: IMPALA-10177: Generate test-classpath.txt .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7199/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc Gerrit-Change-Number: 16450 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 12:44:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9652: CTAS doesn't respect transactional properties
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16472 ) Change subject: IMPALA-9652: CTAS doesn't respect transactional properties .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/16472/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16472/1//COMMIT_MSG@9 PS1, Line 9: Because of a bug the INSERT part of a CTAS statement didn't > Can you add more info about how we worked before the change, and how we wil Done http://gerrit.cloudera.org:8080/#/c/16472/1//COMMIT_MSG@10 PS1, Line 10: transaction > Do I understand well that we are still not transactional in a multi table s Yes, it works like the INSERT statements. IMPALA-8788 is tracking the transactional behavior that you described. It would be a bigger chunk of work, and with event polling we almost always use the latest snapshots of the tables. But I agree that it should be fixed ideally. http://gerrit.cloudera.org:8080/#/c/16472/1/testdata/workloads/functional-query/queries/QueryTest/acid-insert.test File testdata/workloads/functional-query/queries/QueryTest/acid-insert.test: http://gerrit.cloudera.org:8080/#/c/16472/1/testdata/workloads/functional-query/queries/QueryTest/acid-insert.test@65 PS1, Line 65: : QUERY : drop table insertonly_nopart_ctas; > Is there a reason why we do the cleanup here? The whole DB is dropped after I find it good practice to DROP tables after their last use. E.g. if you are copy pasting some tests it's better to get an error message than to use the wrong table. -- To view, visit http://gerrit.cloudera.org:8080/16472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 Gerrit-Change-Number: 16472 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 12:45:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10177: Generate test-classpath.txt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16450 ) Change subject: IMPALA-10177: Generate test-classpath.txt .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6445/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc Gerrit-Change-Number: 16450 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 12:46:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9652: CTAS doesn't respect transactional properties
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16472 ) Change subject: IMPALA-9652: CTAS doesn't respect transactional properties .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 Gerrit-Change-Number: 16472 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 12:54:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9652: CTAS doesn't respect transactional properties
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16472 ) Change subject: IMPALA-9652: CTAS doesn't respect transactional properties .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6446/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 Gerrit-Change-Number: 16472 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 12:55:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9652: CTAS doesn't respect transactional properties
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16472 ) Change subject: IMPALA-9652: CTAS doesn't respect transactional properties .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 Gerrit-Change-Number: 16472 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 12:55:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16473 ) Change subject: IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE .. Patch Set 3: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/16473/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16473/3//COMMIT_MSG@15 PS3, Line 15: IDD DDD? -- To view, visit http://gerrit.cloudera.org:8080/16473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622 Gerrit-Change-Number: 16473 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 13:10:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16473
to look at the new patch set (#4).
Change subject: IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
..
IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
The previous error message contained only the input string but
including the format string as well would make debugging easier.
Extended error message:
SELECT CAST('0;367' as date format 'YY;DDD');
String to Date parse failed. Input '0;367' doesn't match with
format 'YY;DDD'
Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622
---
M be/src/exprs/cast-functions-ir.cc
M tests/query_test/test_cast_with_format.py
2 files changed, 17 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/16473/4
--
To view, visit http://gerrit.cloudera.org:8080/16473
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622
Gerrit-Change-Number: 16473
Gerrit-PatchSet: 4
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16473 ) Change subject: IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE .. Patch Set 4: (2 comments) Carry +2 from Zoltan http://gerrit.cloudera.org:8080/#/c/16473/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16473/3//COMMIT_MSG@15 PS3, Line 15: DDD > DDD? Done http://gerrit.cloudera.org:8080/#/c/16473/1/tests/query_test/test_cast_with_format.py File tests/query_test/test_cast_with_format.py: http://gerrit.cloudera.org:8080/#/c/16473/1/tests/query_test/test_cast_with_format.py@1461 PS1, Line 1461: > flake8: E501 line too long (91 > 90 characters) Done -- To view, visit http://gerrit.cloudera.org:8080/16473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622 Gerrit-Change-Number: 16473 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 13:48:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16473 ) Change subject: IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6447/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622 Gerrit-Change-Number: 16473 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 13:49:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9652: CTAS doesn't respect transactional properties
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16472 ) Change subject: IMPALA-9652: CTAS doesn't respect transactional properties .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 Gerrit-Change-Number: 16472 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 16:05:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10171: Create query options for local time related flags
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16469 ) Change subject: IMPALA-10171: Create query options for local time related flags .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 Gerrit-Change-Number: 16469 Gerrit-PatchSet: 7 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 16:15:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10177: Generate test-classpath.txt
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/16450 ) Change subject: IMPALA-10177: Generate test-classpath.txt .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc Gerrit-Change-Number: 16450 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 16:46:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10171: Create query options for local time related flags
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16469 ) Change subject: IMPALA-10171: Create query options for local time related flags .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 Gerrit-Change-Number: 16469 Gerrit-PatchSet: 8 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 16:48:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10171: Create query options for local time related flags
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16469 ) Change subject: IMPALA-10171: Create query options for local time related flags .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6448/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 Gerrit-Change-Number: 16469 Gerrit-PatchSet: 8 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 16:48:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9930 (part 1): Initial refactor for admission control service
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/16411 ) Change subject: IMPALA-9930 (part 1): Initial refactor for admission control service .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/16411/2/be/src/rpc/sidecar-util.h File be/src/rpc/sidecar-util.h: http://gerrit.cloudera.org:8080/#/c/16411/2/be/src/rpc/sidecar-util.h@83 PS2, Line 83: Status In the original code in Coordinator::BackendState::ExecAsync, this Status was Status::Expected. Should we preserve that? -- To view, visit http://gerrit.cloudera.org:8080/16411 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7974a979cf05ed569f31e1ab20694e29fd3e4508 Gerrit-Change-Number: 16411 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 18 Sep 2020 17:25:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10076: Reduce partition level update logs
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16375 ) Change subject: IMPALA-10076: Reduce partition level update logs .. Patch Set 4: Code-Review+2 (1 comment) Thanks for making the suggested changes. +1. Carrying forward Tim's +1 from earlier. http://gerrit.cloudera.org:8080/#/c/16375/3/fe/src/main/java/org/apache/impala/catalog/PartitionMetaSummary.java File fe/src/main/java/org/apache/impala/catalog/PartitionMetaSummary.java: http://gerrit.cloudera.org:8080/#/c/16375/3/fe/src/main/java/org/apache/impala/catalog/PartitionMetaSummary.java@126 PS3, Line 126: // Init the lexico > Sure. Sorry to make you confused. I refactor this part to be more readable Thanks, it much easier to understand now. -- To view, visit http://gerrit.cloudera.org:8080/16375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Gerrit-Change-Number: 16375 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 18 Sep 2020 17:33:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10177: Generate test-classpath.txt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16450 ) Change subject: IMPALA-10177: Generate test-classpath.txt .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6445/ -- To view, visit http://gerrit.cloudera.org:8080/16450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc Gerrit-Change-Number: 16450 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 18:03:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16473 ) Change subject: IMPALA-10175: Extend error msg when CAST(FORMAT) fails for DATE .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6447/ -- To view, visit http://gerrit.cloudera.org:8080/16473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e379f0f112e83e1511edb170bbe41f903972622 Gerrit-Change-Number: 16473 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 18:05:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9652: CTAS doesn't respect transactional properties
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16472 ) Change subject: IMPALA-9652: CTAS doesn't respect transactional properties .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6446/ -- To view, visit http://gerrit.cloudera.org:8080/16472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ed96aeadbcead9fdc548da5922a066460ff9f77 Gerrit-Change-Number: 16472 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 18:13:39 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10178 (Run-time profile shall report skews)
Qifan Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16474 Change subject: WIP IMPALA-10178 (Run-time profile shall report skews) .. WIP IMPALA-10178 (Run-time profile shall report skews) This fix addresses the current limitation in runtime profile that skews existing in certain operators such as the rows read counter (RowsRead) are not directly report. With the fix, any such skew is reported within the average profile. Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 --- M be/src/runtime/coordinator-backend-state.cc M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M be/src/util/stat-util.h 5 files changed, 94 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/16474/1 -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 1 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Qifan Chen has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 Run-time profile shall report skews .. WIP IMPALA-10178 Run-time profile shall report skews This fix addresses the current limitation in runtime profile that skews existing in certain operators such as the rows read counter (RowsRead) are not directly report. With the fix, any such skew is reported within the average profile. Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 --- M be/src/runtime/coordinator-backend-state.cc M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M be/src/util/stat-util.h 5 files changed, 94 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/16474/2 -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] WIP IMPALA-10178 (Run-time profile shall report skews)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 (Run-time profile shall report skews) .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16474/1/be/src/util/runtime-profile.cc File be/src/util/runtime-profile.cc: http://gerrit.cloudera.org:8080/#/c/16474/1/be/src/util/runtime-profile.cc@634 PS1, Line 634: // For each counter within this profile that can be of skew potentially, verify line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 1 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 18 Sep 2020 21:02:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 Run-time profile shall report skews .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/16474/2/be/src/util/runtime-profile.cc File be/src/util/runtime-profile.cc: http://gerrit.cloudera.org:8080/#/c/16474/2/be/src/util/runtime-profile.cc@634 PS2, Line 634: // For each counter within this profile that can be of skew potentially, verify line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 18 Sep 2020 21:04:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Qifan Chen has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 Run-time profile shall report skews .. WIP IMPALA-10178 Run-time profile shall report skews This fix addresses the current limitation in runtime profile that skews existing in certain operators such as the rows read counter (RowsRead) are not directly reported. With the fix, any such skew is reported in the average counter profile. Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 --- M be/src/runtime/coordinator-backend-state.cc M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M be/src/util/stat-util.h 5 files changed, 94 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/16474/3 -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 Run-time profile shall report skews .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16474/3/be/src/util/runtime-profile.cc File be/src/util/runtime-profile.cc: http://gerrit.cloudera.org:8080/#/c/16474/3/be/src/util/runtime-profile.cc@634 PS3, Line 634: // For each counter within this profile that can be of skew potentially, verify line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 18 Sep 2020 21:06:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Qifan Chen has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 Run-time profile shall report skews .. WIP IMPALA-10178 Run-time profile shall report skews This fix addresses the current limitation in runtime profile that skews existing in certain operators such as the rows read counter (RowsRead) are not directly reported. With the fix, any such skew is reported in the average counter profile. Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 --- M be/src/runtime/coordinator-backend-state.cc M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M be/src/util/stat-util.h 5 files changed, 94 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/16474/4 -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 Run-time profile shall report skews .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7200/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 1 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 18 Sep 2020 21:23:48 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 Run-time profile shall report skews .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7201/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 18 Sep 2020 21:25:10 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 Run-time profile shall report skews .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7203/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 18 Sep 2020 21:27:51 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 Run-time profile shall report skews .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7202/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 18 Sep 2020 21:26:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9227: Add coverage for query retries caused by RPC failure
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16456 ) Change subject: IMPALA-9227: Add coverage for query retries caused by RPC failure .. IMPALA-9227: Add coverage for query retries caused by RPC failure Added two new unit tests for query retries triggered by RPC failures. The RPC failures are simulated by using debug_action to inject RPC fault. Testing: - Passed test_query_retries.py, including the new test cases. Change-Id: I6c800a764b22c72d5c408e14fd1bdea5336d7d82 --- M tests/custom_cluster/test_query_retries.py 1 file changed, 127 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/16456/2 -- To view, visit http://gerrit.cloudera.org:8080/16456 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6c800a764b22c72d5c408e14fd1bdea5336d7d82 Gerrit-Change-Number: 16456 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16474 )
Change subject: WIP IMPALA-10178 Run-time profile shall report skews
..
Patch Set 4:
(6 comments)
Read through out of curiousity. Cool!
http://gerrit.cloudera.org:8080/#/c/16474/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16474/4//COMMIT_MSG@13
PS4, Line 13:
Can you include an example of what the output looks like?
http://gerrit.cloudera.org:8080/#/c/16474/4/be/src/util/runtime-profile-counters.h
File be/src/util/runtime-profile-counters.h:
http://gerrit.cloudera.org:8080/#/c/16474/4/be/src/util/runtime-profile-counters.h@431
PS4, Line 431: bool HasSkew() {
nit: move to .cc file to reduce compilation time.
I guess this is gonna conflict with https://gerrit.cloudera.org/#/c/15798/ a
bit cause I changed the AveragedCounter data structures.
http://gerrit.cloudera.org:8080/#/c/16474/4/be/src/util/runtime-profile.cc
File be/src/util/runtime-profile.cc:
http://gerrit.cloudera.org:8080/#/c/16474/4/be/src/util/runtime-profile.cc@611
PS4, Line 611: const std::string RuntimeProfile::PROFILE_INFO_KEY_SKEW_SUMMARY
= "FRAGMENT_SKEW_SUMMARY";
nit: put constants up the top of the file.
http://gerrit.cloudera.org:8080/#/c/16474/4/be/src/util/runtime-profile.cc@613
PS4, Line 613: std::string
can omit std:: from std::string, std::vector in .cc files where common/names.h
is included.
http://gerrit.cloudera.org:8080/#/c/16474/4/be/src/util/runtime-profile.cc@620
PS4, Line 620: Chilren
Children
http://gerrit.cloudera.org:8080/#/c/16474/4/be/src/util/runtime-profile.cc@648
PS4, Line 648: auto
auto& to avoid an implied copy
--
To view, visit http://gerrit.cloudera.org:8080/16474
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1
Gerrit-Change-Number: 16474
Gerrit-PatchSet: 4
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 18 Sep 2020 22:11:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10171: Create query options for local time related flags
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16469 ) Change subject: IMPALA-10171: Create query options for local time related flags .. IMPALA-10171: Create query options for local time related flags convert_legacy_hive_parquet_utc_timestamps and use_local_tz_for_unix_timestamp_conversions were controllable only by flags until now. After this change the old flags are only used on the Coordinator to set the defaults for the query options. If default_query_options also sets these query options, it will take precedence over the old flags. Possible follow up work: - the old flags could be deprecated, as default_query_options can be used to set this on a server level - testing these functionalities no longer needs custom cluster tests - rewriting the existing tests could speed up test execution Testing: - expr-test was rewritten to use the query option instead of the flag - extended the custom cluster tests for the old flags to also use the query options - ran related tests Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 Reviewed-on: http://gerrit.cloudera.org:8080/16469 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/parquet-column-readers.cc M be/src/exprs/expr-test.cc M be/src/exprs/timestamp-functions.h M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/workloads/functional-query/queries/QueryTest/utc-timestamp-functions.test M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py M tests/custom_cluster/test_local_tz_conversion.py 14 files changed, 183 insertions(+), 37 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 Gerrit-Change-Number: 16469 Gerrit-PatchSet: 9 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10171: Create query options for local time related flags
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16469 ) Change subject: IMPALA-10171: Create query options for local time related flags .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4252d1c8f8e224c1d0e8234f09374bcc0c6f68 Gerrit-Change-Number: 16469 Gerrit-PatchSet: 8 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 22:12:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9227: Add coverage for query retries caused by RPC failure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16456 ) Change subject: IMPALA-9227: Add coverage for query retries caused by RPC failure .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7204/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16456 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c800a764b22c72d5c408e14fd1bdea5336d7d82 Gerrit-Change-Number: 16456 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 18 Sep 2020 22:24:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 ) Change subject: IMPALA-9664: Support hive replication .. Patch Set 2: (17 comments) http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java: http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@1131 PS2, Line 1131:* @param insertEventDataList A list of insert event encapsulating the information needed line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4662 PS2, Line 4662: makeInsertEventData((HdfsTable) table, partVals, newFiles, isInsertOverwrite)); line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@746 PS2, Line 746: runInsertTest(tableToInsertPart, tableToInsertMulPart, numberOfSelfEventsBefore, false); line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@764 PS2, Line 764: runInsertTest(tableToInsertPart, tableToInsertMulPart, numberOfSelfEventsBefore, true); line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/16439/2/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@869 PS2, Line 869: unpartTable.getFileSystem(), new Path(unpartTable.getHdfsBaseDir()), overwrite, "copy_"); line too long (97 > 90) http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py File tests/custom_cluster/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@224 PS2, Line 224: _ flake8: E501 line too long (96 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@226 PS2, Line 226: l flake8: E501 line too long (93 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@228 PS2, Line 228: ] flake8: E501 line too long (92 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@246 PS2, Line 246: ] flake8: E501 line too long (92 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@254 PS2, Line 254: _ flake8: E501 line too long (96 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@256 PS2, Line 256: l flake8: E501 line too long (93 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@265 PS2, Line 265: ] flake8: E501 line too long (92 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@274 PS2, Line 274: _ flake8: E501 line too long (96 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@276 PS2, Line 276: l flake8: E501 line too long (93 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@285 PS2, Line 285: ] flake8: E501 line too long (92 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@328 PS2, Line 328: ] flake8: E501 line too long (92 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16439/2/tests/custom_cluster/test_event_processing.py@350 PS2, Line 350: e flake8: E722 do not use bare except' -- To view, visit http://gerrit.cloudera.org:8080/16439 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a Gerrit-Change-Number: 16439 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 22:36:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16439 to look at the new patch set (#2). Change subject: IMPALA-9664: Support hive replication .. IMPALA-9664: Support hive replication This patch makes some improvements to the INSERT event generated by Impala. Specifically, the INSERT event will now include new file information when Impala inserts into a table. This information can be used by external tools like Hive Replication to replicate the changes made by Impala in their source databases. Additionally, this patch modifies the truncate table execution so that it uses HMS API to truncate the table instead of deleting the files directly on the filesystem. Following changes were made. 1. Fires insert events for insert overwrite. 2. Has the names of the new files in the events. In case of insert overwrite, this is just a list of files which were added by the insert overwrite operation. 3. In case of ACID tables, fires transactional notification API for all the partitions in which data is inserted. 4. For tables which have replication enabled, the truncate table operation now uses a HMS API to truncate the table. This is necessary since HMS API moves the files to a replication change manager location if needed. Additionally, it generates ALTER_TABLE events with truncate flag set to true. TODO: 1. For external tables, replication does not seem to work in the dev environment. This will be done as a followup. Testing: 1. Created a new test in test_events_processing.py which inserts into managed tables which are being replicated. It makes sure that hive replication detects the new rows which are added into the tables. The test also exercises insert overwrite and truncate statements and makes sure that the table is replicated correctly. Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/HiveStorageDescriptorFactory.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java M fe/src/test/resources/hive-site.xml.py M tests/custom_cluster/test_event_processing.py 13 files changed, 781 insertions(+), 184 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/16439/2 -- To view, visit http://gerrit.cloudera.org:8080/16439 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a Gerrit-Change-Number: 16439 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Qifan Chen has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 Run-time profile shall report skews .. WIP IMPALA-10178 Run-time profile shall report skews This fix addresses the current limitation in runtime profile that skews existing in certain operators such as the rows read counter (RowsRead) are not directly reported. With the fix, any such skew is reported in the average counter profile. TODO: 1. Find out if RowsRead is really rows produced or rows processed. Rows processed is more relevant to skws; 2. Add rows processed counters for joins; 2. Add some unit tests for the skwe sections; 3. Run core tests. Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 --- M be/src/runtime/coordinator-backend-state.cc M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M be/src/util/stat-util.h 5 files changed, 94 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/16474/5 -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 Run-time profile shall report skews .. Patch Set 5: Hi Tim, Thanks a lot for the quick review! If you have any other counters to include, please let me know. -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 22:49:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9664: Support hive replication
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16439 ) Change subject: IMPALA-9664: Support hive replication .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/7205/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/16439 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icaf3fe0adff755ff853960f270ceb45b11a84f0a Gerrit-Change-Number: 16439 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Sep 2020 22:51:02 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-10178 Run-time profile shall report skews
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16474 ) Change subject: WIP IMPALA-10178 Run-time profile shall report skews .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7206/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1 Gerrit-Change-Number: 16474 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 23:05:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9382: part 1: transposed profile prototype
Hello Kurt Deschler, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15798 to look at the new patch set (#22). Change subject: IMPALA-9382: part 1: transposed profile prototype .. IMPALA-9382: part 1: transposed profile prototype This adds an experimental profile representation that is denser than the traditional representation. Counters, info strings and other information for all instances of a fragment are merged into a single tree. Descriptive stats (min, max, mean) are shown for each counter, along with the values for each instance. It can be enabled by setting --gen_experimental_profile=true. The default behaviour is unchanged, aside from including a few extra counters in existing profiles. An example of the pretty-printed profile is attached to the JIRA. The thrift representation of the profile is extended so that all instances of a fragment can be merged together into a single "aggregated" fragment, with vectors of counters. The in-memory representation is transformed in a similar way. The RuntimeProfile class is restructured so that there is a common RuntimeProfileBase class, with RuntimeProfile and AggregatedRuntimeProfile subclasses. Execution fills in counters in RuntimeProfile for each instances, then these are aggregated together into an AggregatedRuntimeProfile on the coordinator. This replaces the "averaged" profile concept with an abstraction that more clearly distinguishes what operations apply to aggregated and unaggregated profiles. In a future change, we could use AggregatedRuntimeProfile for status reports so that less data needs to be sent to the coordinator, and the coordinator needs to do less processing. The new profile removes the bad practice of including aggregated stats as strings from the new profile. These stats can now be automatically as aggregations of counters. The legacy uses of InfoString are preserved so as to not lose information but can be removed when we switch to the transposed profile. Also make TotalTime and InactiveTime behave like other counters - they are pretty-printed the same as other counters. Inactive time is also now subtracted from local time in the averaged profile, which fixes IMPALA-2794. TODO in later patches for IMPALA-9382: These will need to be fixed before this can be considered production ready. * The JSON profile generation is not fully implemented for aggregated profiles. * Not all counter times are included in aggregated profile, e.g. time series counters. * The pretty-printing of the various profile counters will need to be improved to be more readable, e.g. grouping by host, improving formatting. * The aggregated profile is only updated at the end of the query. We need to support live updating. * Consider how to show local time per instance - make it a first-class counter in the profile? Possible extensions: * We could better highlight outliers when pretty-printing the profile. Testing: * I diffed the text profile of TPC-DS Q1 to make sure there were no unexpected changes. * Added unit test for stats computation in AveragedCounter. * Passed core tests. * exhaustive tests * ASAN tests * Ran some tests locally with TSAN Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a --- M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/fragment-instance-state.cc M be/src/util/dummy-runtime-profile.h M be/src/util/pretty-printer-test.cc M be/src/util/pretty-printer.h M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile-test.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/RuntimeProfile.thrift 13 files changed, 1,633 insertions(+), 687 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/15798/22 -- To view, visit http://gerrit.cloudera.org:8080/15798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a Gerrit-Change-Number: 15798 Gerrit-PatchSet: 22 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9382: part 1: transposed profile prototype
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15798 ) Change subject: IMPALA-9382: part 1: transposed profile prototype .. Patch Set 23: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a Gerrit-Change-Number: 15798 Gerrit-PatchSet: 23 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 23:25:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9382: part 1: transposed profile prototype
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15798 ) Change subject: IMPALA-9382: part 1: transposed profile prototype .. Patch Set 23: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6449/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a Gerrit-Change-Number: 15798 Gerrit-PatchSet: 23 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 23:25:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9382: part 1: transposed profile prototype
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15798 ) Change subject: IMPALA-9382: part 1: transposed profile prototype .. Patch Set 22: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a Gerrit-Change-Number: 15798 Gerrit-PatchSet: 22 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 23:24:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9382: part 1: transposed profile prototype
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15798 ) Change subject: IMPALA-9382: part 1: transposed profile prototype .. Patch Set 22: The bugfix was trivial -- To view, visit http://gerrit.cloudera.org:8080/15798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a Gerrit-Change-Number: 15798 Gerrit-PatchSet: 22 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 23:25:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9382: part 1: transposed profile prototype
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15798 ) Change subject: IMPALA-9382: part 1: transposed profile prototype .. Patch Set 22: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7207/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a Gerrit-Change-Number: 15798 Gerrit-PatchSet: 22 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Sep 2020 23:45:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10076: Reduce partition level update logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16375 ) Change subject: IMPALA-10076: Reduce partition level update logs .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6450/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Gerrit-Change-Number: 16375 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 19 Sep 2020 00:36:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10076: Reduce partition level update logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16375 ) Change subject: IMPALA-10076: Reduce partition level update logs .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Gerrit-Change-Number: 16375 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 19 Sep 2020 00:36:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9382: part 1: transposed profile prototype
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15798 ) Change subject: IMPALA-9382: part 1: transposed profile prototype .. Patch Set 23: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6449/ -- To view, visit http://gerrit.cloudera.org:8080/15798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a Gerrit-Change-Number: 15798 Gerrit-PatchSet: 23 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 19 Sep 2020 04:45:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10076: Reduce partition level update logs
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16375 ) Change subject: IMPALA-10076: Reduce partition level update logs .. IMPALA-10076: Reduce partition level update logs Catalogd and the legacy mode coordinator log catalog topic items one by a line. A table can have more than 100K partitions. It's too verbose to log all partition level updates. This patch aggregates the partition level updates of each table and only logs the aggregated info in a line. For instance, here is an aggregated log: Collected 24 partition update(s): 1:HDFS_PARTITION:test.tbl:(p=1,p=2,...,p=24), version=1451, original size=(avg=646, min=646, max=648, sum=15516), compressed size=(avg=478, min=475, max=485, sum=11487) If there are more than 3 partitions, we only show the lexicographically smallest, the second smallest and the largest partition names. If there are only one partition update, the log format is the same as before: Collected 1 partition update(s): 1:HDFS_PARTITION:test.tbl:p=0, version=1451, original size=648, compressed size=475 To support these, this patch implements a helper class, PartitionMetaSummary, to aggregate these information and get a summary for logging. The compressed size is calculated by BE. So the jni method for NativeAddPendingTopicItem is extended to return the actual compressed size. Tests: - Added unit tests for PartitionMetaSummary. - Start the cluster in mixed catalog mode and manually verified the logs. Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Reviewed-on: http://gerrit.cloudera.org:8080/16375 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/service/fe-support.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java A fe/src/main/java/org/apache/impala/catalog/PartitionMetaSummary.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java A fe/src/test/java/org/apache/impala/catalog/PartitionMetaSummaryTest.java 11 files changed, 438 insertions(+), 26 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Gerrit-Change-Number: 16375 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-10076: Reduce partition level update logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16375 ) Change subject: IMPALA-10076: Reduce partition level update logs .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Gerrit-Change-Number: 16375 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 19 Sep 2020 06:03:56 + Gerrit-HasComments: No
