[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12635 ) Change subject: IMPALA-8254: Fix error when running compute stats with compression_codec set .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2296/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12635 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2 Gerrit-Change-Number: 12635 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 28 Feb 2019 07:04:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala
Zoltan Ivanfi has posted comments on this change. ( http://gerrit.cloudera.org:8080/12559 ) Change subject: IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala .. Patch Set 4: > Thank you, Zoltan! > How do I get +2? Gerrit only allows me to give a +1, one of the other reviewers can give you a +2. (I'm not an Impala committer myself, I just coordinate cross-component timestamp efforts.) -- To view, visit http://gerrit.cloudera.org:8080/12559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id134036026876238622cb182f790ac0f46654654 Gerrit-Change-Number: 12559 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Thu, 28 Feb 2019 06:50:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12559 ) Change subject: IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala .. Patch Set 4: Thank you, Zoltan! How do I get +2? -- To view, visit http://gerrit.cloudera.org:8080/12559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id134036026876238622cb182f790ac0f46654654 Gerrit-Change-Number: 12559 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Thu, 28 Feb 2019 06:45:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala
Zoltan Ivanfi has posted comments on this change. ( http://gerrit.cloudera.org:8080/12559 ) Change subject: IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala .. Patch Set 4: Code-Review+1 LGTM, thanks! -- To view, visit http://gerrit.cloudera.org:8080/12559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id134036026876238622cb182f790ac0f46654654 Gerrit-Change-Number: 12559 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Thu, 28 Feb 2019 06:31:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8254: Fix error when running compute stats with compression codec set
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12635 Change subject: IMPALA-8254: Fix error when running compute stats with compression_codec set .. IMPALA-8254: Fix error when running compute stats with compression_codec set This patch fixes an error when running compute stats with compression_codec set. This patch also updates missing compression codecs from the compression_codec query option. Testing: - Updated BE query-options-test - Added test_compute.stats.py - Ran all tests in test_compute.stats.py Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2 --- M be/src/service/query-options-test.cc M be/src/service/query-options.cc M tests/metadata/test_compute_stats.py 3 files changed, 35 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/12635/3 -- To view, visit http://gerrit.cloudera.org:8080/12635 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2cb546fbd3d2a02e0ed30d85a33a04852bed9dd2 Gerrit-Change-Number: 12635 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] [DOCS] Add HADOOP-15720 to the list of Known Issues
Pooja Nilangekar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12631 ) Change subject: [DOCS] Add HADOOP-15720 to the list of Known Issues .. Patch Set 1: (3 comments) Hi Alex, Thanks for doing this! I have requested some changes that would affect the correctness. Thanks, Pooja http://gerrit.cloudera.org:8080/#/c/12631/1/docs/topics/impala_known_issues.xml File docs/topics/impala_known_issues.xml: http://gerrit.cloudera.org:8080/#/c/12631/1/docs/topics/impala_known_issues.xml@555 PS1, Line 555: NameNode host http://gerrit.cloudera.org:8080/#/c/12631/1/docs/topics/impala_known_issues.xml@557 PS1, Line 557: 5 hdfs_operation_timeout_sec http://gerrit.cloudera.org:8080/#/c/12631/1/docs/topics/impala_known_issues.xml@565 PS1, Line 565: NameNode host in the bad state. -- To view, visit http://gerrit.cloudera.org:8080/12631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5 Gerrit-Change-Number: 12631 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Comment-Date: Thu, 28 Feb 2019 06:02:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8259: regenerate cluster configs automatically
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12629 ) Change subject: IMPALA-8259: regenerate cluster configs automatically .. IMPALA-8259: regenerate cluster configs automatically Testing: Tested manually in my dev environment with the following steps: ./bin/clean.sh ./buildall.sh -noclean -skiptests -ninja -notests start-impala-cluster.py Change-Id: Id4a1dcc2018bf7fc600f6d490bb1d7a898cccbb4 Reviewed-on: http://gerrit.cloudera.org:8080/12629 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/start-impala-cluster.py 1 file changed, 7 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id4a1dcc2018bf7fc600f6d490bb1d7a898cccbb4 Gerrit-Change-Number: 12629 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8259: regenerate cluster configs automatically
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12629 ) Change subject: IMPALA-8259: regenerate cluster configs automatically .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4a1dcc2018bf7fc600f6d490bb1d7a898cccbb4 Gerrit-Change-Number: 12629 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 28 Feb 2019 05:50:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3848/ -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 28 Feb 2019 05:25:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8100 (WIP): Add initial support for Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 ) Change subject: IMPALA-8100 (WIP): Add initial support for Ranger .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2295/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 Gerrit-Change-Number: 12632 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 28 Feb 2019 05:06:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8100 (WIP): Add initial support for Ranger
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 ) Change subject: IMPALA-8100 (WIP): Add initial support for Ranger .. Patch Set 3: Make the rat happy. -- To view, visit http://gerrit.cloudera.org:8080/12632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 Gerrit-Change-Number: 12632 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 28 Feb 2019 04:41:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8100 (WIP): Add initial support for Ranger
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12632 ) Change subject: IMPALA-8100 (WIP): Add initial support for Ranger .. IMPALA-8100 (WIP): Add initial support for Ranger This patch adds an initial support for Ranger that can be enabled via the following flags in both impalad and catalogd to do enforcement. - ranger_service_type=impala - ranger_app_id=some_app_id - authorization_factory_class=\ org.apache.impala.authorization.ranger.RangerAuthorizationFactory Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M bin/rat_exclude_files.txt M common/thrift/BackendGflags.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java A fe/src/test/resources/ranger-impala-audit.xml A fe/src/test/resources/ranger-impala-security.xml M impala-parent/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/.gitignore A testdata/cluster/ranger/setup/impala_service.json A testdata/cluster/ranger/setup/impala_servicedef.json A testdata/cluster/ranger/setup/impala_user.json.template 21 files changed, 651 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/3 -- To view, visit http://gerrit.cloudera.org:8080/12632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 Gerrit-Change-Number: 12632 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8100 (WIP): Add initial support for Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 ) Change subject: IMPALA-8100 (WIP): Add initial support for Ranger .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2294/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 Gerrit-Change-Number: 12632 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 28 Feb 2019 04:27:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12559 ) Change subject: IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala .. Patch Set 4: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/237/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id134036026876238622cb182f790ac0f46654654 Gerrit-Change-Number: 12559 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Thu, 28 Feb 2019 04:04:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8100 (WIP): Add initial support for Ranger
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12632 Change subject: IMPALA-8100 (WIP): Add initial support for Ranger .. IMPALA-8100 (WIP): Add initial support for Ranger This patch adds an initial support for Ranger that can be enabled via the following flags to do enforcement. - ranger_service_type=impala - ranger_app_id=some_app_id - authorization_factory_class=\ org.apache.impala.authorization.ranger.RangerAuthorizationFactory Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java A fe/src/test/resources/ranger-impala-audit.xml A fe/src/test/resources/ranger-impala-security.xml M impala-parent/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/.gitignore A testdata/cluster/ranger/setup/impala_service.json A testdata/cluster/ranger/setup/impala_servicedef.json A testdata/cluster/ranger/setup/impala_user.json.template 20 files changed, 616 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/2 -- To view, visit http://gerrit.cloudera.org:8080/12632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 Gerrit-Change-Number: 12632 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12591 )
Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates
..
Patch Set 5:
(16 comments)
Mostly general comments.
I think the code in CatalogOpExecutor is in dire need of some refactoring
(outside the scope of this patch). We should probably factor out all the common
stuff into CatalogOpCtx and then pass it around instead of adding new
parameters for every method.
http://gerrit.cloudera.org:8080/#/c/12591/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12591/5//COMMIT_MSG@19
PS5, Line 19: and the catalog version number. The uuid is generated for each
catalogservice when
nit: line overflow
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@189
PS5, Line 189: // unique identifier of this catalog service
: private static final String SERVICE_UUID =
UUID.randomUUID().toString();
Why not use the catalogServiceId_ below? Use TUniqueIdUtil#printId() to convert
it to a string.
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@741
PS5, Line 741: if (!isExternalEventProcessingEnabled()) return result;
Isn't this a preconditions check?
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@785
PS5, Line 785: ersion
version
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@805
PS5, Line 805: tbl.addToVersionsForInflightEvents(versionNumber);
Looks like this can silently fail. How about logging something in that case?
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/Table.java@20
PS5, Line 20: import java.util.ArrayDeque;
unused?
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/Table.java@24
PS5, Line 24: import java.util.Deque;
unused?
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/Table.java@117
PS5, Line 117: private static final int MAX_NUMBER_OF_INFLIGHT_EVENTS = 10;
doc
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/Table.java@120
PS5, Line 120: in seconds to initial
?
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@889
PS5, Line 889: protected List pendingVersionNumbersFromCatalog_ =
Collections.EMPTY_LIST;
Add a doc? I think this crucial to the logic here.
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@900
PS5, Line 900: * This method detects if this event is self-generated or
not. In order to
May be add a pointer to the MetastoreEvents class where you defined what a
self-event is?
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1147
PS5, Line 1147: /**
:* Adds self-event identifiers in the table parameters
:*/
: public static void addCatalogServiceIdentifiers(
: org.apache.hadoop.hive.metastore.api.Table msTbl, long
catalogVersionId) {
: msTbl.putToParameters(
: CATALOG_SERVICE_ID_PROP_KEY,
CatalogServiceCatalog.getServiceUUID());
: msTbl.putToParameters(CATALOG_VERSION_PROP_KEY,
String.valueOf(catalogVersionId));
: }
:
: /**
:* Adds self-event identifiers in the partition parameters
:*/
: public static void addCatalogServiceIdentifiers(
: org.apache.hadoop.hive.metastore.api.Partition partition,
long catalogVersionId) {
: partition.putToParameters(
: CATALOG_SERVICE_ID_PROP_KEY,
CatalogServiceCatalog.getServiceUUID());
: partition.putToParameters(CATALOG_VERSION_PROP_KEY,
String.valueOf(catalogVersionId));
: }
Should these be no-ops if the event processing is not enabled?
http://gerrit.cloudera.org:8080/#/c/12591/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor
[Impala-ASF-CR] IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12559 ) Change subject: IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala .. Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/12559/3/docs/topics/impala_parquet.xml File docs/topics/impala_parquet.xml: http://gerrit.cloudera.org:8080/#/c/12559/3/docs/topics/impala_parquet.xml@1270 PS3, Line 1270: BINAR annotated with the UTF8 OriginalType > A "Y" is missing from the end of "BINAR[Y]" Done http://gerrit.cloudera.org:8080/#/c/12559/3/docs/topics/impala_parquet.xml@1314 PS3, Line 1314: INT64 annotated with the TIMESTAMP_MICROS > "OriginalType" missing from "INT64 annotated with the TIMESTAMP_MICROS [Ori Done http://gerrit.cloudera.org:8080/#/c/12559/3/docs/topics/impala_timestamp.xml File docs/topics/impala_timestamp.xml: http://gerrit.cloudera.org:8080/#/c/12559/3/docs/topics/impala_timestamp.xml@221 PS3, Line 221: values with : the OriginalType annotation > Change Done http://gerrit.cloudera.org:8080/#/c/12559/3/docs/topics/impala_timestamp.xml@224 PS3, Line 224: with the LogicalType annotation specifies whether UTC to local > Change Done http://gerrit.cloudera.org:8080/#/c/12559/3/docs/topics/impala_timestamp.xml@262 PS3, Line 262: default for a performance reason, to avoid unexpected incompatibility problems > This sentence is hard to understand, I would break into two sentences like Done -- To view, visit http://gerrit.cloudera.org:8080/12559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id134036026876238622cb182f790ac0f46654654 Gerrit-Change-Number: 12559 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Thu, 28 Feb 2019 03:48:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala
Hello Zoltan Borok-Nagy, Zoltan Ivanfi, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12559 to look at the new patch set (#4). Change subject: IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala .. IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala Change-Id: Id134036026876238622cb182f790ac0f46654654 --- M docs/shared/impala_common.xml M docs/topics/impala_bigint.xml M docs/topics/impala_parquet.xml M docs/topics/impala_timestamp.xml 4 files changed, 746 insertions(+), 501 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/12559/4 -- To view, visit http://gerrit.cloudera.org:8080/12559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id134036026876238622cb182f790ac0f46654654 Gerrit-Change-Number: 12559 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi
[Impala-ASF-CR] IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12559 ) Change subject: IMPALA-7725: [DOCS] Support for Parquet INT64 Timestamp in Impala .. Patch Set 4: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/237/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id134036026876238622cb182f790ac0f46654654 Gerrit-Change-Number: 12559 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Thu, 28 Feb 2019 03:48:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8100 (WIP): Add initial support for Ranger
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 ) Change subject: IMPALA-8100 (WIP): Add initial support for Ranger .. Patch Set 2: This is a sneak-peak of initial support for Ranger that depends on this CR: https://gerrit.cloudera.org/c/12542 to illustrate the use of new authorization interfaces to implement Ranger. -- To view, visit http://gerrit.cloudera.org:8080/12632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 Gerrit-Change-Number: 12632 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Comment-Date: Thu, 28 Feb 2019 03:46:18 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Add HADOOP-15720 to the list of Known Issues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12631 ) Change subject: [DOCS] Add HADOOP-15720 to the list of Known Issues .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/236/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5 Gerrit-Change-Number: 12631 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Comment-Date: Thu, 28 Feb 2019 03:18:02 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Add HADOOP-15720 to the list of Known Issues
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12631 ) Change subject: [DOCS] Add HADOOP-15720 to the list of Known Issues .. Patch Set 1: Please let me know what has been added to make the to troubleshoot this easier in 3.2. -- To view, visit http://gerrit.cloudera.org:8080/12631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5 Gerrit-Change-Number: 12631 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Comment-Date: Thu, 28 Feb 2019 03:16:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12483 ) Change subject: IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files. .. IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files. Add functions in CatalogUsageMonitor to monitor and report the catalog usage of the tables have the most number of files List the tables with the most number of files in Catalog server web-ui and sorted by the files number. Testing: Add tests to check table usage information is in the catalogd's catalog page. Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b Reviewed-on: http://gerrit.cloudera.org:8080/12483 Reviewed-by: Bharath Vissapragada Tested-by: Impala Public Jenkins --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogUsageMonitor.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M tests/webserver/test_web_pages.py M www/catalog.tmpl 9 files changed, 162 insertions(+), 6 deletions(-) Approvals: Bharath Vissapragada: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b Gerrit-Change-Number: 12483 Gerrit-PatchSet: 10 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12483 ) Change subject: IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files. .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b Gerrit-Change-Number: 12483 Gerrit-PatchSet: 9 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 28 Feb 2019 03:15:00 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Add HADOOP-15720 to the list of Known Issues
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12631 Change subject: [DOCS] Add HADOOP-15720 to the list of Known Issues .. [DOCS] Add HADOOP-15720 to the list of Known Issues Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5 --- M docs/topics/impala_known_issues.xml 1 file changed, 18 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/12631/1 -- To view, visit http://gerrit.cloudera.org:8080/12631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5 Gerrit-Change-Number: 12631 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] [DOCS] Add HADOOP-15720 to the list of Known Issues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12631 ) Change subject: [DOCS] Add HADOOP-15720 to the list of Known Issues .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/236/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9497d337cb79c38add85bb0de9d07660617c76b5 Gerrit-Change-Number: 12631 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 28 Feb 2019 03:12:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8261: Enhance create-test-configuration.sh to not fail when FE has not been built
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12630 ) Change subject: IMPALA-8261: Enhance create-test-configuration.sh to not fail when FE has not been built .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2293/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6536dcffc1124e79c1ed111ad92d257493cc8feb Gerrit-Change-Number: 12630 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 28 Feb 2019 02:49:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8261: Enhance create-test-configuration.sh to not fail when FE has not been built
Fredy Wijaya has uploaded this change for review. (
http://gerrit.cloudera.org:8080/12630
Change subject: IMPALA-8261: Enhance create-test-configuration.sh to not fail
when FE has not been built
..
IMPALA-8261: Enhance create-test-configuration.sh to not fail when FE has not
been built
This patch updates create-test-configuration.sh to not fail due to
missing PostgreSQL JDBC driver when FE has not been built by
downloading it from Maven Central instead. When the JDBC driver
already exists in ${POSTGRES_JDBC_DRIVER}, it will use that instead.
Testing:
Manually ran create-test-configuration.sh with and without FE built.
Change-Id: I6536dcffc1124e79c1ed111ad92d257493cc8feb
---
M bin/create-test-configuration.sh
1 file changed, 8 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/12630/1
--
To view, visit http://gerrit.cloudera.org:8080/12630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6536dcffc1124e79c1ed111ad92d257493cc8feb
Gerrit-Change-Number: 12630
Gerrit-PatchSet: 1
Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8259: regenerate cluster configs automatically
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12629 ) Change subject: IMPALA-8259: regenerate cluster configs automatically .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2292/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4a1dcc2018bf7fc600f6d490bb1d7a898cccbb4 Gerrit-Change-Number: 12629 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 28 Feb 2019 02:06:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5397: Set query's end time us when the operation completes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12583 ) Change subject: IMPALA-5397: Set query's end_time_us_ when the operation completes .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa9a1b443df3dbd95f9f297c2f923ad795b14745 Gerrit-Change-Number: 12583 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 28 Feb 2019 02:05:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5397: Set query's end time us when the operation completes
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12583 ) Change subject: IMPALA-5397: Set query's end_time_us_ when the operation completes .. IMPALA-5397: Set query's end_time_us_ when the operation completes Previously, a query's end time was set only when UnregisterQuery() was called. This was misleading because the query could have completed earlier. After this change, if the query has a coordinator, the end time is set when the query releases its admission control resources. For queries without coordinators, (e.g. DDL queries), the end time would still be set when UnregisterQuery() is called. Testing: Added a test to ensure that the end time is set before the query is closed for a query with a coordinator. It also ensures that for a query without a coordinator, the end time is set only when the query is closed. Tested queries with various settings of --idle-query-timeout. Ran exhaustive end-to-end tests and the stress test without any failures. Change-Id: Iaa9a1b443df3dbd95f9f297c2f923ad795b14745 Reviewed-on: http://gerrit.cloudera.org:8080/12583 Reviewed-by: Bikramjeet Vig Tested-by: Impala Public Jenkins --- M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M tests/common/impala_test_suite.py M tests/query_test/test_observability.py 6 files changed, 78 insertions(+), 16 deletions(-) Approvals: Bikramjeet Vig: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iaa9a1b443df3dbd95f9f297c2f923ad795b14745 Gerrit-Change-Number: 12583 Gerrit-PatchSet: 8 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2291/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 28 Feb 2019 02:01:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8259: regenerate cluster configs automatically
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12629 ) Change subject: IMPALA-8259: regenerate cluster configs automatically .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3849/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4a1dcc2018bf7fc600f6d490bb1d7a898cccbb4 Gerrit-Change-Number: 12629 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 28 Feb 2019 01:50:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12591 ) Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2290/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353 Gerrit-Change-Number: 12591 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 28 Feb 2019 01:52:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8259: regenerate cluster configs automatically
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12629 ) Change subject: IMPALA-8259: regenerate cluster configs automatically .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4a1dcc2018bf7fc600f6d490bb1d7a898cccbb4 Gerrit-Change-Number: 12629 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 28 Feb 2019 01:50:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8259: regenerate cluster configs automatically
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12629 ) Change subject: IMPALA-8259: regenerate cluster configs automatically .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4a1dcc2018bf7fc600f6d490bb1d7a898cccbb4 Gerrit-Change-Number: 12629 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 28 Feb 2019 01:46:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8259: regenerate cluster configs automatically
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12629 Change subject: IMPALA-8259: regenerate cluster configs automatically .. IMPALA-8259: regenerate cluster configs automatically Testing: Tested manually in my dev environment with the following steps: ./bin/clean.sh ./buildall.sh -noclean -skiptests -ninja -notests start-impala-cluster.py Change-Id: Id4a1dcc2018bf7fc600f6d490bb1d7a898cccbb4 --- M bin/start-impala-cluster.py 1 file changed, 7 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/12629/1 -- To view, visit http://gerrit.cloudera.org:8080/12629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id4a1dcc2018bf7fc600f6d490bb1d7a898cccbb4 Gerrit-Change-Number: 12629 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] Usability fixups for the unified backend test executable
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12620 ) Change subject: Usability fixups for the unified backend test executable .. Patch Set 4: Code-Review+2 Thanks! -- To view, visit http://gerrit.cloudera.org:8080/12620 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic206044c8f119b6ee164eb2e6121ec67e71dafa8 Gerrit-Change-Number: 12620 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 28 Feb 2019 01:39:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2288/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 28 Feb 2019 01:27:22 + Gerrit-HasComments: No
[Impala-ASF-CR] Usability fixups for the unified backend test executable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12620 ) Change subject: Usability fixups for the unified backend test executable .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2289/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12620 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic206044c8f119b6ee164eb2e6121ec67e71dafa8 Gerrit-Change-Number: 12620 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 28 Feb 2019 01:33:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. IMPALA-8240: Event processor should keep trying when metastore is unavailable. When metastore is unavailable, EventProcessor state is changed to ERROR when there is a MetastoreFetchNotificationException. After this change, the exception handler will not change the state and EventProcessor continues trying when metastore is unavailable. Testing: Added test in MetastoreEventProcessorTest to check event processor state is active even after multiple NotificationFetchExceptions Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 50 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12601/8 -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates
Vihang Karajgaonkar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12591 ) Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates .. IMPALA-7972 Detect self-events to avoid unnecessary invalidates This patch adds support to detect self-generated events from catalog. This is used to avoid unnecessary invalidates to the tables from such self-events. Currently, alter_table, alter_partition, add_partition and drop_partition event types can invalidate the table metadata. Originally, we planned to have a global version number support from metastore (see HIVE-21115). But since that is still not complete, we rely on a combination of other identifiers to determine if a event is self-generated or not. These self-event identifiers consists of values from the table/partition parameters. A catalog service uuid and the catalog version number. The uuid is generated for each catalogservice when it comes up and it adds it to the table/partition parameters with the key "impala.CatalogServiceId". The catalog version number is added with the key "impala.CatalogVersion". When catalog executes a DDL operation it appends the current catalog version to the list of version numbers for the in-flight events for the table. Events processor clears this version when the corresponding version number identified by serviceId is received in the event. This is needed since it is possible that a external non-Impala system which generates the event presents the same serviceId and version number later on. The algorithm to detect a self-event is as below. 1. Add the service id and expected catalog version to table/partition parameters when executing the DDL operation. When the HMS operation is successful, add the version number to the list of version for in-flight events at table level. 2. When the event is received, the first time you see the combination of serviceId and version number, event processor clears the version number from table's list and determines the event as self-generated (and hence ignored) 3. If the event data presents a unknown serviceId or if the version number is not present in the list of in-flight versions, event is not a self-event and needs to be processed. In order to limit the total memory footprint, only 10 version numbers are stored at the table. Since the event processor is expected to poll every few seconds this should be a reasonable bound which satisfies most use-cases. Otherwise, event processor may wrongly process a self-event to invalidate the table. In such a case, its a performance penalty not a correctness issue. In case of drop_partition event, the partition object is not available in the event. Hence we cannot determine if its a self-event. In such cases currently we always issue a invalidate command. This is a known limitation and will be improved in IMPALA-7973 Patch adds new tests to trigger alter table/partition DDLs from impala and makes sure that the table is not invalidated. Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 6 files changed, 1,127 insertions(+), 202 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/12591/5 -- To view, visit http://gerrit.cloudera.org:8080/12591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353 Gerrit-Change-Number: 12591 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3848/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 28 Feb 2019 01:21:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 28 Feb 2019 01:21:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. Patch Set 8: (3 comments) http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@38 PS6, Line 38: import org.apache.hadoop.hive.metastore.api.CurrentNotificationEventId; > nit, unused import. RRest looks good to me. Done http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@38 PS6, Line 38: import org.apache.hadoop.hive.metastore > remove Done http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@542 PS6, Line 542: T > nit: space if ( Done -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 28 Feb 2019 01:18:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@38 PS6, Line 38: import org.apache.calcite.avatica.Meta; nit, unused import. RRest looks good to me. -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 28 Feb 2019 01:12:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. Patch Set 6: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java: http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@38 PS6, Line 38: import org.apache.calcite.avatica.Meta; remove http://gerrit.cloudera.org:8080/#/c/12601/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@542 PS6, Line 542: (r nit: space if ( -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Thu, 28 Feb 2019 01:12:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Usability fixups for the unified backend test executable
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12620 ) Change subject: Usability fixups for the unified backend test executable .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2287/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12620 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic206044c8f119b6ee164eb2e6121ec67e71dafa8 Gerrit-Change-Number: 12620 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 28 Feb 2019 01:06:11 + Gerrit-HasComments: No
[Impala-ASF-CR] Usability fixups for the unified backend test executable
Joe McDonnell has uploaded a new patch set (#4) to the change originally created by Lars Volker. ( http://gerrit.cloudera.org:8080/12620 ) Change subject: Usability fixups for the unified backend test executable .. Usability fixups for the unified backend test executable This fixes the error messages in bin/validate-unified-backend-test-filters.py to be clearer. It also modifies the dependency graph so that "make $TESTNAME" will run validation on the unified backend executable. Change-Id: Ic206044c8f119b6ee164eb2e6121ec67e71dafa8 --- M be/CMakeLists.txt M bin/validate-unified-backend-test-filters.py 2 files changed, 15 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/12620/4 -- To view, visit http://gerrit.cloudera.org:8080/12620 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic206044c8f119b6ee164eb2e6121ec67e71dafa8 Gerrit-Change-Number: 12620 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/12591 )
Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates
..
Patch Set 4:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@738
PS4, Line 738: public Collection getInFlightVersionsForEvents(String
dbName, String tblName)
> Should be a list: evens are ordered and must be ignored in the order that t
Done
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@765
PS4, Line 765: public void removeFromInFlightVersionsForEvents(String dbName,
String tblName,
> This form introduces a race condition. Can you do something like:
Discussed this with Paul and agreed that this race condition may not be a
problem as of now, since any other thread which can possible modify this list
will either add a new version or invalidate the table. In both the cases, the
code works as expected. I have added a TODO in MetastoreEvents class to improve
this as a followup item later.
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@889
PS4, Line 889: protected Collection pendingVersionNumbersFromCatalog_
= Collections.EMPTY_LIST;
> Again, this introduces a race condition if some other thread changes the ta
Will be done as a followup since it is not critical as of now. This is related
to invalidateOrIgnore(versionNumber) approach which was suggested.
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@919
PS4, Line 919: protected boolean isSelfEvent() throws CatalogException {
> I see. You want to identify outside of a lock if this is a self event. The
Thanks for suggestion, As discussed this can be done as a followup in a
separate patch.
--
To view, visit http://gerrit.cloudera.org:8080/12591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353
Gerrit-Change-Number: 12591
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Thu, 28 Feb 2019 00:49:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. IMPALA-8240: Event processor should keep trying when metastore is unavailable. When metastore is unavailable, EventProcessor state is changed to ERROR when there is a MetastoreFetchNotificationException. After this change, the exception handler will not change the state and EventProcessor continues trying when metastore is unavailable. Testing: Added test in MetastoreEventProcessorTest to check event processor state is active even after multiple NotificationFetchExceptions Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 --- M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 2 files changed, 52 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12601/6 -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] Make error message for backend test validation more clear
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/12620 ) Change subject: Make error message for backend test validation more clear .. Patch Set 3: Let's flip this around. I'll do an upload and you can review. -- To view, visit http://gerrit.cloudera.org:8080/12620 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic206044c8f119b6ee164eb2e6121ec67e71dafa8 Gerrit-Change-Number: 12620 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 28 Feb 2019 00:37:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12601 )
Change subject: IMPALA-8240: Event processor should keep trying when metastore
is unavailable.
..
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@534
PS5, Line 534: @Override
> nit: newline
Done
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@556
PS5, Line 556: // Roughly half of the time an exception is thrown. Make sure
the event processor
: // is still active.
: for(int i = 0 ; i < 11 ; i++) {
: fetchProcessor.processEvents();
: assertEquals(EventProcessorStatus.ACTIVE,
fetchProcessor.getStatus());
: }
> right. But since the exception is thrown at random, it is possible that the
Understood. Thanks for clarifying.
--
To view, visit http://gerrit.cloudera.org:8080/12601
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023
Gerrit-Change-Number: 12601
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Thu, 28 Feb 2019 00:37:01 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12601 )
Change subject: IMPALA-8240: Event processor should keep trying when metastore
is unavailable.
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@556
PS5, Line 556: // Roughly half of the time an exception is thrown. Make sure
the event processor
: // is still active.
: for(int i = 0 ; i < 11 ; i++) {
: fetchProcessor.processEvents();
: assertEquals(EventProcessorStatus.ACTIVE,
fetchProcessor.getStatus());
: }
> I'm sorry, not sure what you mean here. The test class makes sure that exce
right. But since the exception is thrown at random, it is possible that the
above loop can finish without a single exception being thrown. Essentially the
test is not doing what is meant to do.
What I meant above is to loop until an exception is thrown, break out of loop
and assert that the status is still ACTIVE. Did I get something wrong?
--
To view, visit http://gerrit.cloudera.org:8080/12601
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023
Gerrit-Change-Number: 12601
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Thu, 28 Feb 2019 00:33:52 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12601 )
Change subject: IMPALA-8240: Event processor should keep trying when metastore
is unavailable.
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@556
PS5, Line 556: // Roughly half of the time an exception is thrown. Make sure
the event processor
: // is still active.
: for(int i = 0 ; i < 11 ; i++) {
: fetchProcessor.processEvents();
: assertEquals(EventProcessorStatus.ACTIVE,
fetchProcessor.getStatus());
: }
> I'm sorry, not sure what you mean here. The test class makes sure that exce
I think I see your point here. I will make the change. Thanks for the comments.
--
To view, visit http://gerrit.cloudera.org:8080/12601
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023
Gerrit-Change-Number: 12601
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Thu, 28 Feb 2019 00:31:42 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Anurag Mantripragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12601 )
Change subject: IMPALA-8240: Event processor should keep trying when metastore
is unavailable.
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@556
PS5, Line 556: // Roughly half of the time an exception is thrown. Make sure
the event processor
: // is still active.
: for(int i = 0 ; i < 11 ; i++) {
: fetchProcessor.processEvents();
: assertEquals(EventProcessorStatus.ACTIVE,
fetchProcessor.getStatus());
: }
> Looks like this can even pass when no exceptions are thrown. I think we sho
I'm sorry, not sure what you mean here. The test class makes sure that
exceptions are thrown roughly half of the times this method is called..This is
actually an overkill. I want to test the status has not changed to ERROR after
exceptions. This was the behavior before the change - Even a single exception
would change the status to ERROR which would prevent the EventProcessor to run
the next time.
--
To view, visit http://gerrit.cloudera.org:8080/12601
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023
Gerrit-Change-Number: 12601
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Thu, 28 Feb 2019 00:20:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Make error message for backend test validation more clear
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/12620 )
Change subject: Make error message for backend test validation more clear
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12620/2/bin/validate-unified-backend-test-filters.py
File bin/validate-unified-backend-test-filters.py:
http://gerrit.cloudera.org:8080/#/c/12620/2/bin/validate-unified-backend-test-filters.py@77
PS2, Line 77: print("FAILED: Some test filters specified by\n"
: "
ADD_UNIFIED_BE_TEST/ADD_UNIFIED_BE_LSAN_TEST macros in\n"
: "CMakeLists.txt do not match any tests in
the unified backend\n"
: "test executable. This can happen if there
is a bogus filter or\n"
: "if some tests
> There are two different errors.
Done
--
To view, visit http://gerrit.cloudera.org:8080/12620
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic206044c8f119b6ee164eb2e6121ec67e71dafa8
Gerrit-Change-Number: 12620
Gerrit-PatchSet: 3
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Lars Volker
Gerrit-Comment-Date: Thu, 28 Feb 2019 00:20:08 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Make error message for backend test validation more clear
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12620 to look at the new patch set (#3). Change subject: Make error message for backend test validation more clear .. Make error message for backend test validation more clear Change-Id: Ic206044c8f119b6ee164eb2e6121ec67e71dafa8 --- M bin/validate-unified-backend-test-filters.py 1 file changed, 10 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/12620/3 -- To view, visit http://gerrit.cloudera.org:8080/12620 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic206044c8f119b6ee164eb2e6121ec67e71dafa8 Gerrit-Change-Number: 12620 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] Bump debug web page header size to prevent wrapping
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12596 ) Change subject: Bump debug web page header size to prevent wrapping .. Bump debug web page header size to prevent wrapping Before this change the header would wrap between the daemon name and the pages list for the impalad debug page. Change-Id: Icc9c97c41ef12016f797101b25a49464de233034 Reviewed-on: http://gerrit.cloudera.org:8080/12596 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M www/common-header.tmpl 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12596 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icc9c97c41ef12016f797101b25a49464de233034 Gerrit-Change-Number: 12596 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Bump debug web page header size to prevent wrapping
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12596 ) Change subject: Bump debug web page header size to prevent wrapping .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12596 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc9c97c41ef12016f797101b25a49464de233034 Gerrit-Change-Number: 12596 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 28 Feb 2019 00:08:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 ) Change subject: IMPALA-8185: Abstract out real/mock file system operations .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2286/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d Gerrit-Change-Number: 12437 Gerrit-PatchSet: 6 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Wed, 27 Feb 2019 23:56:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8182: Add single-node plan to PlanCtx
Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/12436 ) Change subject: IMPALA-8182: Add single-node plan to PlanCtx .. Abandoned Will do as part of an actual test later. -- To view, visit http://gerrit.cloudera.org:8080/12436 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I983473f259df8af17d8dd38e497fe33e839cd0dc Gerrit-Change-Number: 12436 Gerrit-PatchSet: 2 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers
[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations
Hello Bharath Vissapragada, Philip Zeyliger, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12437 to look at the new patch set (#6). Change subject: IMPALA-8185: Abstract out real/mock file system operations .. IMPALA-8185: Abstract out real/mock file system operations The recent addition of the test case builder introduced the idea of running a query in "test case mode" on a system different from the on on which the test case was created. A number of if-statements were used to mock file system operations which depend on the file system to exist. This patch abstracts out the mock vs. real operations into a set of "facade" classes. The HDFS facade performs the real opeations, the "mock" facade mocks the operations well enough to fool the planner. Selection of the facade depends on the recently-added test case mode query option. Tests: Reran all tests: both those that depend on a real file system and the test case builder tests that depend on the mock file system. Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java A fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java A fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java M fe/src/main/java/org/apache/impala/analysis/HdfsUri.java A fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java 7 files changed, 352 insertions(+), 137 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/12437/6 -- To view, visit http://gerrit.cloudera.org:8080/12437 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d Gerrit-Change-Number: 12437 Gerrit-PatchSet: 6 Gerrit-Owner: Paul Rogers Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/12624 ) Change subject: IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/12624/1/be/src/rpc/impala-service-pool.cc File be/src/rpc/impala-service-pool.cc: http://gerrit.cloudera.org:8080/#/c/12624/1/be/src/rpc/impala-service-pool.cc@119 PS1, Line 119: "The service queue is full; it has $3 items; memory " > It's full when either the max queue length or the memory consumption limit If we want to keep the single message then maybe a message like "$0 request on $1 from $2 dropped due to backpressure. The service queue contains $3 items out of a maximum of $4; memory consumption is $5" I suppose if we really wanted to be more precise we could have the caller pass in the exact reason for the rejection. -- To view, visit http://gerrit.cloudera.org:8080/12624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98 Gerrit-Change-Number: 12624 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 27 Feb 2019 23:19:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12483 ) Change subject: IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files. .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2285/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b Gerrit-Change-Number: 12483 Gerrit-PatchSet: 9 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 27 Feb 2019 23:15:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates
Paul Rogers has posted comments on this change. (
http://gerrit.cloudera.org:8080/12591 )
Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates
..
Patch Set 4:
(7 comments)
Identified one more potential race condition and suggested a solution.
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@738
PS4, Line 738: public Collection getInFlightVersionsForEvents(String
dbName, String tblName)
Should be a list: evens are ordered and must be ignored in the order that they
were added to the list.
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@765
PS4, Line 765: public void removeFromInFlightVersionsForEvents(String dbName,
String tblName,
This form introduces a race condition. Can you do something like:
void invaliateOrIgnore(int versionNo)
- lock
- if list is non-empty, and first entry is versionNo, remove that first entry
- Else, invalidate the table
- unlock
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@889
PS4, Line 889: protected Collection pendingVersionNumbersFromCatalog_
= Collections.EMPTY_LIST;
Again, this introduces a race condition if some other thread changes the table
since we obtained the list.
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@919
PS4, Line 919: protected boolean isSelfEvent() throws CatalogException {
I see. You want to identify outside of a lock if this is a self event. The
proposed non-racy solution makes self-event detection an integral part of the
invalidate, avoiding race conditions.
The logic would be:
if server id not set, or does not match
- invalidate
else
- invalidOrIgnore(event version)
Can even be simplified to:
eventVersion = -1
if serverId is set and matches then evenVersion = version from event
invalidateOrIgnore(eventVersion)
I think this completely eliminates all race condition paths
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@957
PS4, Line 957: if (isSelfEvent()) {
To enable this logging, have invalidOrIgnore return true if ignored, false if
invalidated (or visa-versa, you choose)
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1100
PS4, Line 1100: protected boolean isSelfEvent() {
With the new system, we don't know if it is a self event until we lock and
look. All we can tell is that, when we checked, it WAS a self event.
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/12591/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@148
PS4, Line 148: * its a performance penalty not a correctness issue.
Thanks for the explanation!
--
To view, visit http://gerrit.cloudera.org:8080/12591
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353
Gerrit-Change-Number: 12591
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Wed, 27 Feb 2019 23:13:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12591 ) Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2284/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353 Gerrit-Change-Number: 12591 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 27 Feb 2019 23:12:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12483 ) Change subject: IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files. .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3846/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b Gerrit-Change-Number: 12483 Gerrit-PatchSet: 9 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 27 Feb 2019 23:08:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12483 ) Change subject: IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files. .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b Gerrit-Change-Number: 12483 Gerrit-PatchSet: 9 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 27 Feb 2019 23:07:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12483 ) Change subject: IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files. .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2283/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b Gerrit-Change-Number: 12483 Gerrit-PatchSet: 8 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 27 Feb 2019 23:00:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12624 ) Change subject: IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2282/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98 Gerrit-Change-Number: 12624 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 27 Feb 2019 22:55:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files.
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12483 to look at the new patch set (#8). Change subject: IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files. .. IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files. Add functions in CatalogUsageMonitor to monitor and report the catalog usage of the tables have the most number of files List the tables with the most number of files in Catalog server web-ui and sorted by the files number. Testing: Add tests to check table usage information is in the catalogd's catalog page. Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogUsageMonitor.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M tests/webserver/test_web_pages.py M www/catalog.tmpl 9 files changed, 162 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/12483/8 -- To view, visit http://gerrit.cloudera.org:8080/12483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b Gerrit-Change-Number: 12483 Gerrit-PatchSet: 8 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files.
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12483 to look at the new patch set (#9). Change subject: IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files. .. IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files. Add functions in CatalogUsageMonitor to monitor and report the catalog usage of the tables have the most number of files List the tables with the most number of files in Catalog server web-ui and sorted by the files number. Testing: Add tests to check table usage information is in the catalogd's catalog page. Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogUsageMonitor.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M tests/webserver/test_web_pages.py M www/catalog.tmpl 9 files changed, 162 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/12483/9 -- To view, visit http://gerrit.cloudera.org:8080/12483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b Gerrit-Change-Number: 12483 Gerrit-PatchSet: 9 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates
Vihang Karajgaonkar has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/12591 ) Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates .. IMPALA-7972 Detect self-events to avoid unnecessary invalidates This patch adds support to detect self-generated events from catalog. This is used to avoid unnecessary invalidates to the tables from such self-events. Currently, alter_table, alter_partition, add_partition and drop_partition event types can invalidate the table metadata. Originally, we planned to have a global version number support from metastore (see HIVE-21115). But since that is still not complete, we rely on a combination of other identifiers to determine if a event is self-generated or not. These self-event identifiers consists of values from the table/partition parameters. A catalog service uuid and the catalog version number. The uuid is generated for each catalogservice when it comes up and it adds it to the table/partition parameters with the key "impala.CatalogServiceId". The catalog version number is added with the key "impala.CatalogVersion". When catalog executes a DDL operation it appends the current catalog version to the list of version numbers for the in-flight events for the table. Events processor clears this version when the corresponding version number identified by serviceId is received in the event. This is needed since it is possible that a external non-Impala system which generates the event presents the same serviceId and version number later on. The algorithm to detect a self-event is as below. 1. Add the service id and expected catalog version to table/partition parameters when executing the DDL operation. When the HMS operation is successful, add the version number to the list of version for in-flight events at table level. 2. When the event is received, the first time you see the combination of serviceId and version number, event processor clears the version number from table's list and determines the event as self-generated (and hence ignored) 3. If the event data presents a unknown serviceId or if the version number is not present in the list of in-flight versions, event is not a self-event and needs to be processed. In order to limit the total memory footprint, only 10 version numbers are stored at the table. Since the event processor is expected to poll every few seconds this should be a reasonable bound which satisfies most use-cases. Otherwise, event processor may wrongly process a self-event to invalidate the table. In such a case, its a performance penalty not a correctness issue. In case of drop_partition event, the partition object is not available in the event. Hence we cannot determine if its a self-event. In such cases currently we always issue a invalidate command. This is a known limitation and will be improved in IMPALA-7973 Patch adds new tests to trigger alter table/partition DDLs from impala and makes sure that the table is not invalidated. Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 6 files changed, 1,142 insertions(+), 203 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/12591/4 -- To view, visit http://gerrit.cloudera.org:8080/12591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353 Gerrit-Change-Number: 12591 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12483 ) Change subject: IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files. .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/12483/8/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/12483/8/tests/webserver/test_web_pages.py@27 PS8, Line 27: import re flake8: F811 redefinition of unused 're' from line 25 -- To view, visit http://gerrit.cloudera.org:8080/12483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b Gerrit-Change-Number: 12483 Gerrit-PatchSet: 8 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 27 Feb 2019 22:31:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5397: Set query's end time us when the operation completes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12583 ) Change subject: IMPALA-5397: Set query's end_time_us_ when the operation completes .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2281/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa9a1b443df3dbd95f9f297c2f923ad795b14745 Gerrit-Change-Number: 12583 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 27 Feb 2019 22:31:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12579 )
Change subject: IMPALA-7800: Reject new connections after --fe_service_threads
..
Patch Set 3:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/12579/3/be/src/rpc/TAcceptQueueServer.cpp
File be/src/rpc/TAcceptQueueServer.cpp:
http://gerrit.cloudera.org:8080/#/c/12579/3/be/src/rpc/TAcceptQueueServer.cpp@28
PS3, Line 28: 120,
Will it be safer to have a higher default timeout (e.g. 300 seconds) to avoid
users' complaint given the current behavior is that we don't have a timeout ?
Please also add a remark that setting it to 0 means there is no timeout.
http://gerrit.cloudera.org:8080/#/c/12579/3/be/src/rpc/TAcceptQueueServer.cpp@151
PS3, Line 151: const string& error)
nit: we tend to put the constant parameters followed by non-constant parameters.
http://gerrit.cloudera.org:8080/#/c/12579/3/be/src/rpc/TAcceptQueueServer.cpp@193
PS3, Line 193: 0LL
nit: 'LL' not needed.
http://gerrit.cloudera.org:8080/#/c/12579/3/be/src/rpc/TAcceptQueueServer.cpp@196
PS3, Line 196: if (entry->expiration_time_)
nit: if (entry->expiration_time_ != 0) {
http://gerrit.cloudera.org:8080/#/c/12579/3/be/src/rpc/TAcceptQueueServer.cpp@197
PS3, Line 197: wait_time = entry->expiration_time_ - MonotonicMillis();
Shouldn't this be inside the while loop below ?
http://gerrit.cloudera.org:8080/#/c/12579/3/be/src/rpc/TAcceptQueueServer.cpp@199
PS3, Line 199: LOG(INFO) << "All " << maxTasks_ << " server threads are
in use. "
: << "Waiting for " << wait_time << " msecs.";
Will this lead to log spam ?
http://gerrit.cloudera.org:8080/#/c/12579/3/be/src/rpc/TAcceptQueueServer.cpp@204
PS3, Line 204: Timing out connection request.";
Any chance we can print some details (e.g. IP address, port) here ?
http://gerrit.cloudera.org:8080/#/c/12579/3/be/src/rpc/TAcceptQueueServer.cpp@239
PS3, Line 239: shared_ptr
This can be "unique_ptr" now that TTransport lives inside TAcceptQueueEntry.
This makes ownership easier to understand.
http://gerrit.cloudera.org:8080/#/c/12579/3/be/src/rpc/TAcceptQueueServer.cpp@260
PS3, Line 260: if (FLAGS_accepted_cnxn_timeout)
nit: if (FLAGS_accepted_cnxn_timeout != 0) {
http://gerrit.cloudera.org:8080/#/c/12579/3/be/src/rpc/TAcceptQueueServer.cpp@261
PS3, Line 261: MILLIS_PER_SEC;
long line.
--
To view, visit http://gerrit.cloudera.org:8080/12579
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64
Gerrit-Change-Number: 12579
Gerrit-PatchSet: 3
Gerrit-Owner: Zoram Thanga
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Wed, 27 Feb 2019 22:26:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6897: Catalog web-ui exposes top-n tables with most num of files.
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12483 )
Change subject: IMPALA-6897: Catalog web-ui exposes top-n tables with most num
of files.
..
Patch Set 7: Code-Review+2
(1 comment)
If you could rebase on to the master, I can submit it for a GVO.
http://gerrit.cloudera.org:8080/#/c/12483/7/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/12483/7/fe/src/main/java/org/apache/impala/catalog/Table.java@174
PS7, Line 174: public void initMetrics() {
newline
--
To view, visit http://gerrit.cloudera.org:8080/12483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04df5756641bb10dbb86d813b8001c4a04d7dc9b
Gerrit-Change-Number: 12483
Gerrit-PatchSet: 7
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Comment-Date: Wed, 27 Feb 2019 22:18:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/12624 ) Change subject: IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/12624/1/be/src/rpc/impala-service-pool.cc File be/src/rpc/impala-service-pool.cc: http://gerrit.cloudera.org:8080/#/c/12624/1/be/src/rpc/impala-service-pool.cc@119 PS1, Line 119: "The service queue is full; it has $3 items; memory " > Isn't the message "The service queue is full" misleading? It's full when either the max queue length or the memory consumption limit is reached. Definitely open to better suggestion for the message. I considered "The service queue has reached its capacity limit" but that doesn't seem any different. http://gerrit.cloudera.org:8080/#/c/12624/1/be/src/rpc/impala-service-pool.cc@120 PS1, Line 120: "consumption is $4", > Nit: needs terminating "." Done http://gerrit.cloudera.org:8080/#/c/12624/1/be/src/rpc/impala-service-pool.cc@125 PS1, Line 125: PrettyPrinter::Print(service_mem_tracker_->consumption(), TUnit::BYTES)); > There's a similar message in service-pool.cc As you mentioned, that's the Kudu's implementation. We forked it to allow us to enforce memory limit too. So, in the Kudu's implementation, the queue is only full because it has reached the maximum queue length. -- To view, visit http://gerrit.cloudera.org:8080/12624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98 Gerrit-Change-Number: 12624 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 27 Feb 2019 22:08:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()
Hello Andrew Sherman, Thomas Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12624 to look at the new patch set (#2). Change subject: IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() .. IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() An incoming request to a RPC service can be rejected due to either exceeding the memory limit or maximum allowed queue length. It's unclear from the current error message which of those factors contributes to the failure as neither the actual queue length nor the memory consumption is printed. This patch fixes the problem by printing the estimated queue length and memory consumption when a RPC request is dropped. Testing done: verified the new error message with test_rpc_timeout.py Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98 --- M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h 2 files changed, 7 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/12624/2 -- To view, visit http://gerrit.cloudera.org:8080/12624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98 Gerrit-Change-Number: 12624 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-5397: Set query's end time us when the operation completes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12583 ) Change subject: IMPALA-5397: Set query's end_time_us_ when the operation completes .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3845/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa9a1b443df3dbd95f9f297c2f923ad795b14745 Gerrit-Change-Number: 12583 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 27 Feb 2019 22:01:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12601 )
Change subject: IMPALA-8240: Event processor should keep trying when metastore
is unavailable.
..
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@534
PS5, Line 534: @Override
nit: newline
http://gerrit.cloudera.org:8080/#/c/12601/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@556
PS5, Line 556: // Roughly half of the time an exception is thrown. Make sure
the event processor
: // is still active.
: for(int i = 0 ; i < 11 ; i++) {
: fetchProcessor.processEvents();
: assertEquals(EventProcessorStatus.ACTIVE,
fetchProcessor.getStatus());
: }
Looks like this can even pass when no exceptions are thrown. I think we should
do something like,
while (exceptionNotThrown) {
fetchProcessor.processEvents();
}
assertEquals(ACTIVE, fetchProcessor.status());
--
To view, visit http://gerrit.cloudera.org:8080/12601
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023
Gerrit-Change-Number: 12601
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Wed, 27 Feb 2019 21:57:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5397: Set query's end time us when the operation completes
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/12583 ) Change subject: IMPALA-5397: Set query's end_time_us_ when the operation completes .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa9a1b443df3dbd95f9f297c2f923ad795b14745 Gerrit-Change-Number: 12583 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 27 Feb 2019 21:53:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5397: Set query's end time us when the operation completes
Pooja Nilangekar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/12583 ) Change subject: IMPALA-5397: Set query's end_time_us_ when the operation completes .. IMPALA-5397: Set query's end_time_us_ when the operation completes Previously, a query's end time was set only when UnregisterQuery() was called. This was misleading because the query could have completed earlier. After this change, if the query has a coordinator, the end time is set when the query releases its admission control resources. For queries without coordinators, (e.g. DDL queries), the end time would still be set when UnregisterQuery() is called. Testing: Added a test to ensure that the end time is set before the query is closed for a query with a coordinator. It also ensures that for a query without a coordinator, the end time is set only when the query is closed. Tested queries with various settings of --idle-query-timeout. Ran exhaustive end-to-end tests and the stress test without any failures. Change-Id: Iaa9a1b443df3dbd95f9f297c2f923ad795b14745 --- M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M tests/common/impala_test_suite.py M tests/query_test/test_observability.py 6 files changed, 78 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/12583/7 -- To view, visit http://gerrit.cloudera.org:8080/12583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaa9a1b443df3dbd95f9f297c2f923ad795b14745 Gerrit-Change-Number: 12583 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5397: Set query's end time us when the operation completes
Pooja Nilangekar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12583 ) Change subject: IMPALA-5397: Set query's end_time_us_ when the operation completes .. Patch Set 7: (2 comments) http://gerrit.cloudera.org:8080/#/c/12583/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12583/1//COMMIT_MSG@13 PS1, Line 13: admission control resources > we can look at that in a separate patch, regardless of that, this patch is Ack http://gerrit.cloudera.org:8080/#/c/12583/6/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/12583/6/be/src/service/client-request-state.cc@1313 PS6, Line 1313: // Update the query's end ti > we can probably use AtomicInt for the end_time_us and use CompareAndSwap an Done -- To view, visit http://gerrit.cloudera.org:8080/12583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa9a1b443df3dbd95f9f297c2f923ad795b14745 Gerrit-Change-Number: 12583 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 27 Feb 2019 21:47:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/12624 ) Change subject: IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() .. Patch Set 1: Oh I see, the service-pool.cc version does not reject messages because of lack of memory, doh. -- To view, visit http://gerrit.cloudera.org:8080/12624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98 Gerrit-Change-Number: 12624 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 27 Feb 2019 21:16:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12624 ) Change subject: IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2280/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98 Gerrit-Change-Number: 12624 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 27 Feb 2019 21:11:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/12624 ) Change subject: IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() .. Patch Set 1: (3 comments) Change looks basically good, I have a few nits. http://gerrit.cloudera.org:8080/#/c/12624/1/be/src/rpc/impala-service-pool.cc File be/src/rpc/impala-service-pool.cc: http://gerrit.cloudera.org:8080/#/c/12624/1/be/src/rpc/impala-service-pool.cc@119 PS1, Line 119: "The service queue is full; it has $3 items; memory " Isn't the message "The service queue is full" misleading? http://gerrit.cloudera.org:8080/#/c/12624/1/be/src/rpc/impala-service-pool.cc@120 PS1, Line 120: "consumption is $4", Nit: needs terminating "." http://gerrit.cloudera.org:8080/#/c/12624/1/be/src/rpc/impala-service-pool.cc@125 PS1, Line 125: PrettyPrinter::Print(service_mem_tracker_->consumption(), TUnit::BYTES)); There's a similar message in service-pool.cc Is it right that you are not changing the message there because there is no service_mem_tracker in that class? -- To view, visit http://gerrit.cloudera.org:8080/12624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98 Gerrit-Change-Number: 12624 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 27 Feb 2019 21:11:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4568: Cache parquet footer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12622 ) Change subject: IMPALA-4568: Cache parquet footer .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2278/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib30dcc4a13050c11f011491dec3776ba9e402c78 Gerrit-Change-Number: 12622 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Feb 2019 20:58:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12623 ) Change subject: IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2279/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b40b8e2077b61e2f7e244b87c946f30803438bc Gerrit-Change-Number: 12623 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Feb 2019 20:59:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5397: Set query's end time us when the operation completes
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/12583 ) Change subject: IMPALA-5397: Set query's end_time_us_ when the operation completes .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/12583/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12583/1//COMMIT_MSG@13 PS1, Line 13: admission control resources > I once came across a case where a query was cancelled (wherein the state wa we can look at that in a separate patch, regardless of that, this patch is definitely useful, so lets get this in first. http://gerrit.cloudera.org:8080/#/c/12583/6/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/12583/6/be/src/service/client-request-state.cc@1313 PS6, Line 1313: unique_lock l(lock_); we can probably use AtomicInt for the end_time_us and use CompareAndSwap and do away with holding lock_ every time just for this. Another reason is that I am a bit concerned about holding parent lock in the coordinator since it is called by a bunch of CRS methods, I dont see any deadlocks now, but it definitely increases the probability of adding one in some future change if not careful -- To view, visit http://gerrit.cloudera.org:8080/12583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa9a1b443df3dbd95f9f297c2f923ad795b14745 Gerrit-Change-Number: 12583 Gerrit-PatchSet: 6 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 27 Feb 2019 20:41:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5397: Set query's end time us when the operation completes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12583 ) Change subject: IMPALA-5397: Set query's end_time_us_ when the operation completes .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2277/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa9a1b443df3dbd95f9f297c2f923ad795b14745 Gerrit-Change-Number: 12583 Gerrit-PatchSet: 6 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 27 Feb 2019 20:39:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8240: Event processor should keep trying when metastore is unavailable.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12601 ) Change subject: IMPALA-8240: Event processor should keep trying when metastore is unavailable. .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2276/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I570a52462e6d3b634b2c227dfcb98e20ad2a0023 Gerrit-Change-Number: 12601 Gerrit-PatchSet: 5 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Wed, 27 Feb 2019 20:38:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()
Michael Ho has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12624 Change subject: IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() .. IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() An incoming request to a RPC service can be rejected due to either exceeding the memory limit or maximum allowed queue length. It's unclear from the current error message which of those factors contributes to the failure as neither the actual queue length nor the memory consumption is printed. This patch fixes the problem by printing the estimated queue length and memory consumption when a RPC request is dropped. Testing done: verified the new error message with test_rpc_timeout.py Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98 --- M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h 2 files changed, 7 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/12624/1 -- To view, visit http://gerrit.cloudera.org:8080/12624 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If0297658acf2b23823dcb7d2bdff5d8e4475bb98 Gerrit-Change-Number: 12624 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho
[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()
Michael Ho has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12623 Change subject: IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() .. IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() An incoming request to a RPC service can be rejected either due to exceeding the memory limit or maximum allowed queue length. It's unclear from the current error message which of those factors contributes to the failure as neither the actual queue length nor the memory consumption is printed. This patch fixes the problem by printing the estimated queue length and memory consumption when a RPC request is dropped. Change-Id: I5b40b8e2077b61e2f7e244b87c946f30803438bc --- M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h 2 files changed, 7 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/12623/1 -- To view, visit http://gerrit.cloudera.org:8080/12623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5b40b8e2077b61e2f7e244b87c946f30803438bc Gerrit-Change-Number: 12623 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho
[Impala-ASF-CR] IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy()
Michael Ho has abandoned this change. ( http://gerrit.cloudera.org:8080/12623 ) Change subject: IMPALA-8256: Better error message for ImpalaServicePool::RejectTooBusy() .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/12623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I5b40b8e2077b61e2f7e244b87c946f30803438bc Gerrit-Change-Number: 12623 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-4568: Cache parquet footer
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/12622 ) Change subject: IMPALA-4568: Cache parquet footer .. Patch Set 1: Sorry, pushing the wrong branch. This is WIP. Definitely not meant for review. -- To view, visit http://gerrit.cloudera.org:8080/12622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib30dcc4a13050c11f011491dec3776ba9e402c78 Gerrit-Change-Number: 12622 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 27 Feb 2019 20:28:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4568: Cache parquet footer
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/12622 )
Change subject: IMPALA-4568: Cache parquet footer
..
Patch Set 1:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/hdfs-scanner.cc
File be/src/exec/hdfs-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/hdfs-scanner.cc@828
PS1, Line 828:BufferOpts(cache_footer ?
BufferOpts::EXT_CACHED : BufferOpts::UNCACHED, -1),
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/hdfs-text-scanner.cc
File be/src/exec/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/hdfs-text-scanner.cc@120
PS1, Line 120:BufferOpts(split->cache_tags(), files[i]->mtime));
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/hdfs-parquet-scanner.cc
File be/src/exec/parquet/hdfs-parquet-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@1200
PS1, Line 1200: // We're processing the scan range issued in
IssueInitialRanges(). The scan range should
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@1202
PS1, Line 1202: // expect. Note we can't detect if the file is larger than
we expect without attempting
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@1237
PS1, Line 1237: string(reinterpret_cast(magic_number_ptr),
sizeof(PARQUET_VERSION_NUMBER)),
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@1265
PS1, Line 1265: "footer: $1 bytes. File size: $2 bytes.",
filename(), metadata_size, file_len));
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@1271
PS1, Line 1271: return
scan_node_->mem_tracker()->MemLimitExceeded(state_, details, metadata_size);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/parquet-footer-cache.cc
File be/src/exec/parquet/parquet-footer-cache.cc:
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/parquet-footer-cache.cc@76
PS1, Line 76: void ParquetFooterCache::Partition::Insert(const string& fname,
const FileMetaData& footer) {
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/runtime/exec-env.h
File be/src/runtime/exec-env.h:
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/runtime/exec-env.h@222
PS1, Line 222:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/runtime/io/request-ranges.h
File be/src/runtime/io/request-ranges.h:
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/runtime/io/request-ranges.h@275
PS1, Line 275:
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/12622
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30dcc4a13050c11f011491dec3776ba9e402c78
Gerrit-Change-Number: 12622
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Wed, 27 Feb 2019 20:28:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4568: Cache parquet footer
Michael Ho has abandoned this change. ( http://gerrit.cloudera.org:8080/12622 ) Change subject: IMPALA-4568: Cache parquet footer .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/12622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ib30dcc4a13050c11f011491dec3776ba9e402c78 Gerrit-Change-Number: 12622 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-4568: Cache parquet footer
Michael Ho has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12622 Change subject: IMPALA-4568: Cache parquet footer .. IMPALA-4568: Cache parquet footer Change-Id: Ib30dcc4a13050c11f011491dec3776ba9e402c78 --- M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-text-scanner.cc M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/parquet/parquet-column-readers.cc A be/src/exec/parquet/parquet-footer-cache.cc A be/src/exec/parquet/parquet-footer-cache.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/io/request-context.cc M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc 15 files changed, 443 insertions(+), 126 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/12622/1 -- To view, visit http://gerrit.cloudera.org:8080/12622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib30dcc4a13050c11f011491dec3776ba9e402c78 Gerrit-Change-Number: 12622 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho
