[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14307 ) Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4834/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 19 Oct 2019 05:55:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9070: Include table location in lineage for 'CREATE EXTERNAL TABLE' DDL.
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/14515 )
Change subject: IMPALA-9070: Include table location in lineage for 'CREATE
EXTERNAL TABLE' DDL.
..
Patch Set 1:
(1 comment)
Had one question about whether "" is the right value for non-external tables.
http://gerrit.cloudera.org:8080/#/c/14515/1/be/src/util/lineage-util.h
File be/src/util/lineage-util.h:
http://gerrit.cloudera.org:8080/#/c/14515/1/be/src/util/lineage-util.h@119
PS1, Line 119: writer.String("tableLocation");
I think this writes it out even if it wasn't set, i.e. for external tables. It
seems like we should omit the key/value or set it to null (not sure what atlas
expects). I.e.
if (lineage.__isset.table_location) {
...
}
--
To view, visit http://gerrit.cloudera.org:8080/14515
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If02b0cc16d52c1956298171628f5737cab62ce9f
Gerrit-Change-Number: 14515
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Sat, 19 Oct 2019 05:36:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14307 ) Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. Patch Set 7: Thanks Vihang's comment! There is a bug when getCatalogDelta() runs concurrently with reset(). If reset() finishes first, it'll update lastResetCatalogVersion_ (now renamed to lastResetStartVersion_) and this got propagated at the end of getCatalogDelta(). We should keep the value of lastResetCatalogVersion_ when creating GetCatalogDeltaContext and use it instead. Fixed this bug. -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 19 Oct 2019 05:14:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Hello Bharath Vissapragada, Vihang Karajgaonkar, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14307 to look at the new patch set (#7). Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. IMPALA-7506: support global INVALIDATE METADATA in local catalog mode The minimal catalog object version of valid catalog objects is used to implement global invalidate metadata in legacy catalog mode. Coordinator sends DDL RPC to catalogd for global invalidate metadata and gets the expected min catalog version in the response. It's the version when catalogd starts to reset the entire catalog, which means when the reset is done, all valid catalog objects should be associated with a catalog version larger than it. Coordinator will wait until its min catalog version exceeds this value, which means it has processed all the updates of the reset propagated from the catalogd via statestored. If SYNC_DDL is set, the coordinator will also wait until other coordinators reach the same statestore topic version with it, so they have also processed the same updates and had the latest catalog after reset. In local catalog mode, the coordinator does not cache all the metadata. Instead, it caches them on-demand (based on query requests), and removes them based on the Guava cache configurations (size or TTL) or explicit invalidation from the catalog topic updates. So it's hard to track the minimal catalog object version correctly. This patch adds a new field (lastResetCatalogVersion) in TCatalog to propagate the catalog version when catalogd starts to reset the entire metadata. Each time when catalogd generates a new topic update, it will generate a TCatalogObject of CATALOG type containing the state of the catalog which includes this new field. To make all changes of the reset being added in the same topic update with this TCatalog object. Rapidly changed tables that have catalog version exceeding the version range of this update will also be included. When coordinator receives a new value of lastResetCatalogVersion in a topic update, it means catalogd has reset the entire catalog and all the relative updates are whether included in the same or previous topic updates. This is guaranteed by three facts: 1) In catalogd, lastResetCatalogVersion_ is only updated at the end of reset() and is protected by versionLock_. 2) In catalogd's update gathering thread, reading lastResetCatalogVersion_ requires the read lock of versionLock_. The value is then saved into GetCatalogDeltaContext and finally got propagated. 3) If reset() is done, the next update won't skip any tables. The update gathering thread will collect all catalog objects. Thus, all metadata with catalog version <= lastResetCatalogVersion can be considered stale after coordinator finish processing the topic update. lastResetCatalogVersion + 1 is the lower bound (included) of min catalog object version of a coordinator. To avoid catalogd's update collector thread being blocked by concurrent DDLs that holding the table locks, this patch also fixes IMPALA-9062. In local catalog mode, we just need to propagate the table name of a changed table, so don't need to acquire table lock to get a full TTable object. This patch also exposes the min catalog object version of coordinator via a new metric "catalog.min-catalog-object-version" to ease debugging. Tests: - Recover all existing tests that have been disabled due to this missing feature Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 --- M be/src/service/impala-server.cc M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M common/thrift/CatalogObjects.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M tests/authorization/test_grant_revoke.py M tests/authorization/test_ranger.py M tests/common/skip.py M tests/custom_cluster/test_local_catalog.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py 14 files changed, 187 insertions(+), 109 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/14307/7 -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5114/ -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 19 Oct 2019 04:31:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5113/ -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 19 Oct 2019 03:34:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5111/ -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Sat, 19 Oct 2019 02:39:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/14307 )
Change subject: IMPALA-7506: support global INVALIDATE METADATA in local
catalog mode
..
Patch Set 6:
(4 comments)
Thanks Vihang's comments!
> I looks like you are sending all the objects in the catalogd when you detect
> the reset. Is it possible to not send the objects and just invalidate the
> whole local catalog cache when you detect that the last reset version is
> changed?
Reseting coordinator's local catalog is ok. But coordinator needs to know when
the reset() in catalogd finishs. For SYNC_DDL, it also needs to know the
catalog topic version (not catalog version, it's the version of the statestore
topic) so it can wait for other coordinators to be ready. I have an explanation
with code links in "Why can’t we simply reset the cache of local catalog mode
Coordinator for global INVALIDATE METADATA?" in the doc:
https://docs.google.com/document/d/1-AoigzsQPgSGosW4vtVP8E7TiwoJJfLkAq3Rd6KvKBg
http://gerrit.cloudera.org:8080/#/c/14307/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14307/5//COMMIT_MSG@41
PS5, Line 41: 1) No topic updates are sent from catalogd when the write lock of
: versionLock is held in CatalogServiceCatalog.reset(). Note that
the
: update thread requires holding the read lock of versionLock.
: 2) Authz changes before holding the write lock can only be sent
in a
: previous topic update or in the next topic update after reset().
: 3) No catalog objects are skipped in the topic update right after
: reset(). See changes in GetCatalogDeltaContext
> Did you consider the case when a reset is executed during execution of getC
Oh, it's possible when reset() runs again after reset(), so getCatalogDelta
thread is running with collectFullUpdates being set and will collect everything.
In normal cases if getCatalogDelta() and reset() run concurrently, it's ok
since getCatalogDelta runs with collectFullUpdates unset so will only collect
updates in range of (fromVersion, toVersion]. After reset() holding the write
lock, the updates are with version larger than toVersion.
I'll update the proof here. It's ok for a previous update to contain some
results of the reset(). The most important update, CATALOG type TCatalog
object, can only get the correct lastResetCatalogVersion_ after reset()
finishes. Before that, it just gets an old value of lastResetCatalogVersion_.
(lastResetCatalogVersion_ is updated at the end of reset())
http://gerrit.cloudera.org:8080/#/c/14307/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/14307/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@721
PS5, Line 721: public long getCatalogDelta(long nativeCatalogServerPtr, long
fromVersion) throws
: TException {
: long toVersion;
: boolean collectFullUpdates;
: versionLock_.readLock().lock();
: try {
: toVersion = catalogVersion_;
:
> what happens if the reset thread takes a write lock after this code block?
Then toVersion is small enough that won't cover updates after reset() acquiring
the write lock. However, if reset() runs again after reset(), hasResetCatalog
is true so all updates will be collected. As in the above comment, I think it's
ok for a previous update to contain some results of a running reset() as long
as it don't propagate the lastResetCatalogVersion_.
BTW, with the changes in addTableToCatalogDeltaHelper(), in local catalog mode
we no longer need to acquire the table locks. So collecting all table updates
won't be blocked by concurrent DDLs.
http://gerrit.cloudera.org:8080/#/c/14307/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1104
PS5, Line 1104:* which would also violate the semantics of SYNC_DDL.
> Does the collectFullUpdates flag come in play in v1 as well? If yes, that s
Yes, it should be used only in MINIMAL topic mode. I'll constrain this.
http://gerrit.cloudera.org:8080/#/c/14307/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1601
PS5, Line 1601: sing for id just befo
> It looks like the currentCatalogVersion may not be the right reset version
Sorry, I should rename it to "lastResetCatalogVersion" or
"lastResetBeginVersion" to avoid confusion. It's the same version that we
return to the coordinator, meaning that all catalog objects with versions <=
this will be invalidated. Coordinator first gets this version in the RPC
response, then when receiving this again in the CATALOG type TCatalog object
update (via statestore topic update), it knows that reset() has finished and
all
[Impala-ASF-CR] IMPALA-9070: Include table location in lineage for 'CREATE EXTERNAL TABLE' DDL.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14515 ) Change subject: IMPALA-9070: Include table location in lineage for 'CREATE EXTERNAL TABLE' DDL. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4833/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14515 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If02b0cc16d52c1956298171628f5737cab62ce9f Gerrit-Change-Number: 14515 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Sat, 19 Oct 2019 01:16:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8999: make union scheduling work with mt dop
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14384 ) Change subject: IMPALA-8999: make union scheduling work with mt_dop .. Patch Set 11: (9 comments) http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.h File be/src/scheduling/scheduler.h: http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.h@381 PS11, Line 381: For HDFS, this attempts to load balance among instances by computing the average : /// number of bytes per instances and then in a single pass assigning scan ranges to : /// each instance to roughly meet that average. nit: looks like we forgot to update this in the commit for IMPALA-9015 http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc File be/src/scheduling/scheduler.cc: http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@291 PS11, Line 291: one or more scan nodes for the cases with non-union fragments, we are now considering the parallelism of all scan nodes in the fragment as opposed to the left most scan node, what are the cases that this can happen in a plan? Maybe add that as a test case too http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@332 PS11, Line 332: Instance selection for an interior fragment what does instance selection and interior fragment mean here? http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@360 PS11, Line 360: hosts nit: instances_per_host http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@363 PS11, Line 363: only factor in nit: got confused with this, read it as "only factor... in" vs the intended meaning of "only... factor in". maybe say "only consider parallelism" http://gerrit.cloudera.org:8080/#/c/14384/11/be/src/scheduling/scheduler.cc@368 PS11, Line 368: input_fragment_hosts nit: input_fragment_instances_per_host http://gerrit.cloudera.org:8080/#/c/14384/11/fe/src/main/java/org/apache/impala/planner/UnionNode.java File fe/src/main/java/org/apache/impala/planner/UnionNode.java: http://gerrit.cloudera.org:8080/#/c/14384/11/fe/src/main/java/org/apache/impala/planner/UnionNode.java@122 PS11, Line 122: Union fragments are scheduled on the union of hosts that the child fragments run : // on. nit: Union fragments are scheduled on the union of hosts of all scans in the fragment as well as the hosts of all its input fragments http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test File testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test: http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test@52 PS11, Line 52: it nit: if http://gerrit.cloudera.org:8080/#/c/14384/11/testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-scheduling.test@133 PS11, Line 133: add a test for a union fragment where the scan ranges for scan node are more than 4 but the max instances gets bounded by mt_dop -- To view, visit http://gerrit.cloudera.org:8080/14384 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d2e9c86b530da3053e49d42b837dca0b1348ff2 Gerrit-Change-Number: 14384 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 19 Oct 2019 00:54:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9070: Include table location in lineage for 'CREATE EXTERNAL TABLE' DDL.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14515 Change subject: IMPALA-9070: Include table location in lineage for 'CREATE EXTERNAL TABLE' DDL. .. IMPALA-9070: Include table location in lineage for 'CREATE EXTERNAL TABLE' DDL. Atlas needs table location to establish lineage between a newly created external table and its table location. The table location information is not available until the createTable catalog op succeeds. After this change, location information is sent to the backend in the TDDLExecResponse message which adds it to the lineage graph. This information is sent only for create external table queries. Testing: Added a test to verify the tableLocation field is populated for a create external table query lineage. Also, modified the lineage.test file to include location information for all lineages. Change-Id: If02b0cc16d52c1956298171628f5737cab62ce9f --- M be/src/service/client-request-state.cc M be/src/util/lineage-util.h M common/thrift/CatalogService.thrift M common/thrift/LineageGraph.thrift M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M testdata/workloads/functional-query/queries/QueryTest/lineage.test M tests/custom_cluster/test_lineage.py 7 files changed, 920 insertions(+), 449 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/14515/1 -- To view, visit http://gerrit.cloudera.org:8080/14515 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If02b0cc16d52c1956298171628f5737cab62ce9f Gerrit-Change-Number: 14515 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4832/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 19 Oct 2019 00:05:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4831/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 18 Oct 2019 23:56:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5113/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 18 Oct 2019 23:15:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/14397/7/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java: http://gerrit.cloudera.org:8080/#/c/14397/7/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@343 PS7, Line 343: if (Boolean.parseBoolean( > line too long (96 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 18 Oct 2019 23:18:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5114/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 18 Oct 2019 23:19:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Hao Hao has uploaded a new patch set (#8) to the change originally created by Vihang Karajgaonkar. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. IMPALA-9030: Handle translated external Kudu tables In HMS 3.1 there is a default tranformer introduced which checks the client capabilities and transforms a table before creating it. Additionally, it also makes sure that any managed table which is created is transactional. If a user creates a managed table which is not transactional, it automatically converts such table as external and sets certain table properties to mark such transformed tables. This presents a problem for managed Kudu tables in Impala since managed and external tables are handled differently in Kudu. Specifically, if a Kudu table is managed, certain operations like drop table, rename table, alter table are performed on the Kudu side along with updating the catalog. If the Kudu table is external, the Kudu operations are skipped and only catalog side operations are performed. When the user creates a managed Kudu table, user expects that drop table, rename table should be updated by Impala automatically in Kudu as well. But since HMS 3 transforms such managed tables into external, currently Impala does not perform the Kudu side operations breaking the semantics for the user. This patch makes changes to Catalog so that it can detect such transformed external tables and perform Kudu side operations similar to what it was doing for managed Kudu table when talking with previous HMS versions. Note that this change is in preparation of bumping up the CDP build which will be done in a separate change. For the current CDP build number the patch is essentially a no-op. Testing: 1. Bumped up the CDP build number in a private build so that the HMS translation logic is pulled in. Ran all the tests. Without the patch there are many Kudu tests which were failing. After the patch none of the Kudu tests fail. There were additional Ranger tests which failed due to the CDP bump but those were unrelated to this patch and should be fixed as part of a separate change when the CDP build number is bumped up. Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java 8 files changed, 94 insertions(+), 40 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/14397/8 -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/14397 )
Change subject: IMPALA-9030: Handle translated external Kudu tables
..
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14397/7/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java:
http://gerrit.cloudera.org:8080/#/c/14397/7/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@343
PS7, Line 343: if
(Boolean.parseBoolean(getTblProperties().get(KuduTable.TBL_PROP_EXTERNAL_TABLE_PURGE)))
{
line too long (96 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/14397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36
Gerrit-Change-Number: 14397
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Greg Solovyev
Gerrit-Reviewer: Hao Hao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Fri, 18 Oct 2019 23:13:41 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/14397/6/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/14397/6/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@75 PS6, Line 75: !KuduTable.isExternalTable(msTbl) > That said, if someone more familiar with Impala metadata feels that option I also thinks option 1 is more safe, and as no one so far says it is safe with option 2. I just pushed a patch with option 1. -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 18 Oct 2019 23:14:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Hao Hao has uploaded a new patch set (#7) to the change originally created by Vihang Karajgaonkar. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. IMPALA-9030: Handle translated external Kudu tables In HMS 3.1 there is a default tranformer introduced which checks the client capabilities and transforms a table before creating it. Additionally, it also makes sure that any managed table which is created is transactional. If a user creates a managed table which is not transactional, it automatically converts such table as external and sets certain table properties to mark such transformed tables. This presents a problem for managed Kudu tables in Impala since managed and external tables are handled differently in Kudu. Specifically, if a Kudu table is managed, certain operations like drop table, rename table, alter table are performed on the Kudu side along with updating the catalog. If the Kudu table is external, the Kudu operations are skipped and only catalog side operations are performed. When the user creates a managed Kudu table, user expects that drop table, rename table should be updated by Impala automatically in Kudu as well. But since HMS 3 transforms such managed tables into external, currently Impala does not perform the Kudu side operations breaking the semantics for the user. This patch makes changes to Catalog so that it can detect such transformed external tables and perform Kudu side operations similar to what it was doing for managed Kudu table when talking with previous HMS versions. Note that this change is in preparation of bumping up the CDP build which will be done in a separate change. For the current CDP build number the patch is essentially a no-op. Testing: 1. Bumped up the CDP build number in a private build so that the HMS translation logic is pulled in. Ran all the tests. Without the patch there are many Kudu tests which were failing. After the patch none of the Kudu tests fail. There were additional Ranger tests which failed due to the CDP bump but those were unrelated to this patch and should be fixed as part of a separate change when the CDP build number is bumped up. Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java 8 files changed, 93 insertions(+), 40 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/14397/7 -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5111/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 18 Oct 2019 22:24:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 5: I'm going to run the precommit tests just to make sure that the new test passes :) -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 18 Oct 2019 22:24:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9061: Update ant version for centos in bootstrap system.sh
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14482 ) Change subject: IMPALA-9061: Update ant version for centos in bootstrap_system.sh .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Gerrit-Change-Number: 14482 Gerrit-PatchSet: 4 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Oct 2019 22:08:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9061: Update ant version for centos in bootstrap system.sh
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14482 ) Change subject: IMPALA-9061: Update ant version for centos in bootstrap_system.sh .. IMPALA-9061: Update ant version for centos in bootstrap_system.sh bootstrap_system.sh currently use ant 1.9.13 on CentOS/Redhat environment. this release cannot be accessed, the earliest version was 1.9.14. please see https://www-us.apache.org/dist/ant/binaries/. upgrade version to 1.9.14 Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Reviewed-on: http://gerrit.cloudera.org:8080/14482 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M bin/bootstrap_system.sh 1 file changed, 4 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/14482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Gerrit-Change-Number: 14482 Gerrit-PatchSet: 5 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/14397/6/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/14397/6/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@75 PS6, Line 75: !KuduTable.isExternalTable(msTbl) > Hmm, it will be translated to 'Preconditions.checkState(KuduTable.isSynchro I agree, ideally for Kudu external+purge tables would be treated the same as managed table, even in creation. Otherwise, I imagine it being quite confusing how to handle the rename or deletion of an external+purge table that doesn't exist in Kudu (e.g. drop table would fail because the table doesn't exist in Kudu). I think we should explore doing number 2, but I'm uncertain whether there are other repercussions in Impala surrounding creating an external table when a managed table was requested. As such, for now, let's just go with option 1 -- it seems easier to me, and simpler to reason about and document than the current behavior. -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 18 Oct 2019 21:22:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9030: Handle translated external Kudu tables
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14397 ) Change subject: IMPALA-9030: Handle translated external Kudu tables .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/14397/6/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/14397/6/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@75 PS6, Line 75: !KuduTable.isExternalTable(msTbl) > I agree, ideally for Kudu external+purge tables would be treated the same a That said, if someone more familiar with Impala metadata feels that option 2 is safe, I'd prefer going down that route. -- To view, visit http://gerrit.cloudera.org:8080/14397 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I324523361c923b7d291cb4d0f1028b1a5b653b36 Gerrit-Change-Number: 14397 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Greg Solovyev Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 18 Oct 2019 21:23:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9061: Update ant version for centos in bootstrap system.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14482 ) Change subject: IMPALA-9061: Update ant version for centos in bootstrap_system.sh .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Gerrit-Change-Number: 14482 Gerrit-PatchSet: 4 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Oct 2019 20:33:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9067: [DOCS] Impala ports doc update
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14501 ) Change subject: IMPALA-9067: [DOCS] Impala_ports doc update .. IMPALA-9067: [DOCS] Impala_ports doc update - Corrected the table headings. - Updated descriptions. - Replaced the field names with the startup flags. - Removed the unused ports for Llama. - Removed one unused keydef, logs_managing, from impala_keydefs.ditamap Change-Id: I0f86958a645ab476416f4f11dc2df400bfc62e79 Reviewed-on: http://gerrit.cloudera.org:8080/14501 Tested-by: Impala Public Jenkins Reviewed-by: Thomas Tauber-Marshall --- M docs/impala_keydefs.ditamap M docs/topics/impala_ports.xml 2 files changed, 68 insertions(+), 411 deletions(-) Approvals: Impala Public Jenkins: Verified Thomas Tauber-Marshall: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/14501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0f86958a645ab476416f4f11dc2df400bfc62e79 Gerrit-Change-Number: 14501 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9067: [DOCS] Impala ports doc update
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14501 ) Change subject: IMPALA-9067: [DOCS] Impala_ports doc update .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14501 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0f86958a645ab476416f4f11dc2df400bfc62e79 Gerrit-Change-Number: 14501 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 18 Oct 2019 18:00:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1523729
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14508 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1523729 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4830/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c4ddc846ca879eae6ca4bce9bac77d7a9a3f43e Gerrit-Change-Number: 14508 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Fri, 18 Oct 2019 17:26:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/14307 )
Change subject: IMPALA-7506: support global INVALIDATE METADATA in local
catalog mode
..
Patch Set 6:
(4 comments)
I looks like you are sending all the objects in the catalogd when you detect
the reset. Is it possible to not send the objects and just invalidate the whole
local catalog cache when you detect that the last reset version is changed?
http://gerrit.cloudera.org:8080/#/c/14307/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14307/5//COMMIT_MSG@41
PS5, Line 41: 1) No topic updates are sent from catalogd when the write lock of
: versionLock is held in CatalogServiceCatalog.reset(). Note that
the
: update thread requires holding the read lock of versionLock.
: 2) Authz changes before holding the write lock can only be sent
in a
: previous topic update or in the next topic update after reset().
: 3) No catalog objects are skipped in the topic update right after
: reset(). See changes in GetCatalogDeltaContext
Did you consider the case when a reset is executed during execution of
getCatalogDelta? Looks like it is possible for this to happen since the
getCatalogDelta takes a read lock and releases in the for loop. So the reset
thread can take a write lock in between the getCatalogDelta which effectively
means that the topic may have some state before reset and some state after the
reset.
http://gerrit.cloudera.org:8080/#/c/14307/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/14307/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@721
PS5, Line 721: public long getCatalogDelta(long nativeCatalogServerPtr, long
fromVersion) throws
: TException {
: long toVersion;
: boolean collectFullUpdates;
: versionLock_.readLock().lock();
: try {
: toVersion = catalogVersion_;
:
what happens if the reset thread takes a write lock after this code block?
http://gerrit.cloudera.org:8080/#/c/14307/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1104
PS5, Line 1104:* which would also violate the semantics of SYNC_DDL.
> I see what you are saying, nice find. Agree that we shouldn't take any lock
Does the collectFullUpdates flag come in play in v1 as well? If yes, that seems
unnecessary.
http://gerrit.cloudera.org:8080/#/c/14307/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1601
PS5, Line 1601: sing for id just befo
It looks like the currentCatalogVersion may not be the right reset version
since once we assign currentCatalogVersion and the place where we take the
write lock is not atomic. May be you should directly use (catalogVersion_-1)
here since you already hold the write lock
--
To view, visit http://gerrit.cloudera.org:8080/14307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549
Gerrit-Change-Number: 14307
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Fri, 18 Oct 2019 17:00:05 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1523729
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14508 Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1523729 .. IMPALA-9047: Bump CDP_BUILD_NUMBER to 1523729 This patch bumps CDP_BUILD_NUMBER to 1523729. Some test cases would fail due to this change. For example, some Ranger related E2E tests and FE tests would fail when we are using a newer version of Ranger (e.g., ranger-2.0.0.7.0.2.0-138 instead of ranger-1.2.0.7.1.0.0-33) due to the changes to the default Ranger policies described at https://issues.apache.org/jira/browse/RANGER-2536. To address this issue, this patch temporarily disables those affected Ranger tests. Specifically, the affected tests in the following test files are disabled for now. 1. test_authorized_proxy.py 2. test_ranger.py 3. AuthorizationStmtTest.java 4. RangerAuditLogTest.java Testing: - This patch passes the exhaustive tests. Change-Id: I9c4ddc846ca879eae6ca4bce9bac77d7a9a3f43e --- M bin/impala-config.sh M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M tests/authorization/test_authorized_proxy.py M tests/authorization/test_ranger.py 5 files changed, 101 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/14508/1 -- To view, visit http://gerrit.cloudera.org:8080/14508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9c4ddc846ca879eae6ca4bce9bac77d7a9a3f43e Gerrit-Change-Number: 14508 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-9061: Update ant version for centos in bootstrap system.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14482 ) Change subject: IMPALA-9061: Update ant version for centos in bootstrap_system.sh .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5108/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Gerrit-Change-Number: 14482 Gerrit-PatchSet: 4 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Oct 2019 16:16:10 + Gerrit-HasComments: No
[Impala-ASF-CR] MPALA-9061: Update ant version for centos in bootstrap system.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14482 ) Change subject: MPALA-9061: Update ant version for centos in bootstrap_system.sh .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Gerrit-Change-Number: 14482 Gerrit-PatchSet: 3 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Oct 2019 16:15:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8755: Backend support for Z-ordering
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14080 ) Change subject: IMPALA-8755: Backend support for Z-ordering .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4829/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0200748ce3e65ebc5d3530f794c0f80aa335a2ab Gerrit-Change-Number: 14080 Gerrit-PatchSet: 9 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Oct 2019 16:18:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9061: Update ant version for centos in bootstrap system.sh
Tim Armstrong has uploaded a new patch set (#4) to the change originally created by Fucun Chu. ( http://gerrit.cloudera.org:8080/14482 ) Change subject: IMPALA-9061: Update ant version for centos in bootstrap_system.sh .. IMPALA-9061: Update ant version for centos in bootstrap_system.sh bootstrap_system.sh currently use ant 1.9.13 on CentOS/Redhat environment. this release cannot be accessed, the earliest version was 1.9.14. please see https://www-us.apache.org/dist/ant/binaries/. upgrade version to 1.9.14 Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb --- M bin/bootstrap_system.sh 1 file changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/14482/4 -- To view, visit http://gerrit.cloudera.org:8080/14482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Gerrit-Change-Number: 14482 Gerrit-PatchSet: 4 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9061: Update ant version for centos in bootstrap system.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14482 ) Change subject: IMPALA-9061: Update ant version for centos in bootstrap_system.sh .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5107/ -- To view, visit http://gerrit.cloudera.org:8080/14482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Gerrit-Change-Number: 14482 Gerrit-PatchSet: 3 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Oct 2019 16:15:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/1 ) Change subject: IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4828/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I066652bfa7d924742af01aef8df4512e00620c7d Gerrit-Change-Number: 1 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Oct 2019 16:14:39 + Gerrit-HasComments: No
[Impala-ASF-CR] MPALA-9061: Update ant version for centos in bootstrap system.sh
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14482 ) Change subject: MPALA-9061: Update ant version for centos in bootstrap_system.sh .. Patch Set 2: Code-Review+2 Thanks for your contribution! Much appreciated. -- To view, visit http://gerrit.cloudera.org:8080/14482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Gerrit-Change-Number: 14482 Gerrit-PatchSet: 2 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Oct 2019 16:14:48 + Gerrit-HasComments: No
[Impala-ASF-CR] MPALA-9061: Update ant version for centos in bootstrap system.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14482 ) Change subject: MPALA-9061: Update ant version for centos in bootstrap_system.sh .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5107/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Gerrit-Change-Number: 14482 Gerrit-PatchSet: 3 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 18 Oct 2019 16:15:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/1 ) Change subject: IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/1/2/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/1/2/tests/common/impala_test_suite.py@219 PS2, Line 219: # flake8: E265 block comment should start with '# ' http://gerrit.cloudera.org:8080/#/c/1/2/tests/stress/test_acid_stress.py File tests/stress/test_acid_stress.py: http://gerrit.cloudera.org:8080/#/c/1/2/tests/stress/test_acid_stress.py@48 PS2, Line 48: def run_tasks(tasks): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I066652bfa7d924742af01aef8df4512e00620c7d Gerrit-Change-Number: 1 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Oct 2019 15:34:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8755: Backend support for Z-ordering
Norbert Luksa has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/14080 ) Change subject: IMPALA-8755: Backend support for Z-ordering .. IMPALA-8755: Backend support for Z-ordering This change depends on gerrit.cloudera.org/#/c/13955/ (Frontend support for Z-ordering) The commit adds a Comperator based on Z-ordering. See in detail: https://en.wikipedia.org/wiki/Z-order_curve The comperator instead of calculating the Z-values of the rows, looks for the column with the most significant dimension, and compares the values of this column only. The most significant dimension will be the one where the compared values have the highest different bits. The algorithm requires values of the same binary representation, but this can be relaxed. Currently, strings, varchars, floats and doubles are not supported. Testing: * Added unit tests. * Run manual tests, comparing 4-column values with 4-bit integers, for all possible combinations. Checked the result by calculating the Z-value for each comparison. * Tested performance on various data, getting great results. Change-Id: I0200748ce3e65ebc5d3530f794c0f80aa335a2ab --- M be/src/exec/exchange-node.cc M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/partial-sort-node.cc M be/src/exec/partial-sort-node.h M be/src/exec/sort-node.cc M be/src/exec/sort-node.h M be/src/exec/topn-node.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/sorter.cc M be/src/runtime/sorter.h M be/src/util/CMakeLists.txt A be/src/util/tuple-row-compare-test.cc M be/src/util/tuple-row-compare.cc M be/src/util/tuple-row-compare.h 16 files changed, 784 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/14080/9 -- To view, visit http://gerrit.cloudera.org:8080/14080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0200748ce3e65ebc5d3530f794c0f80aa335a2ab Gerrit-Change-Number: 14080 Gerrit-PatchSet: 9 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/1 to look at the new patch set (#2). Change subject: IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs .. IMPALA-8648: Add stress tests for ACID INSERTs/SELECTs Added different stress tests for ACID operations. We already had one stress test where an Impala client issued SELECTs while another process inserted data through Hive. The test checked wether some invariants are held. I added the following tests: * one Impala client inserts data, one Impala client reads and checks invariants * multiple Impala clients issue INSERTs and TRUNCATEs, multiple Impala clients issue SELECTs and check invariants * multiple Impala clients insert data, sometimes failures are injected during inserts to create aborted transactions. In the meantime multiple Impala clients read the data and check that if they see data from aborted transactions During the tests I found and fixed a temporal deadlock. CatalogOpExecutor.truncateTable() locked catalog_.versionLock_ and the HMS ACID lock in a different order than the INSERT statements, it could lead to a deadlock for 30 seconds, after that TRUNCATE failed. Also, in the Frontend I switched the order of write id allocation and locking. From now on at first we lock the table then allocate a write id. Change-Id: I066652bfa7d924742af01aef8df4512e00620c7d --- M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/common/TransactionKeepalive.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/common/impala_test_suite.py M tests/stress/test_acid_stress.py 6 files changed, 356 insertions(+), 98 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/1/2 -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I066652bfa7d924742af01aef8df4512e00620c7d Gerrit-Change-Number: 1 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14307 ) Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4827/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 18 Oct 2019 14:45:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14307 ) Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. Patch Set 6: Refactored some codes and add the fix for IMPALA-9062 that don't need to acquire table lock in gathering topic updates in minimal topic mode. Thanks Bharath's comments! Todd, Vihang, hope can hear your opinions for this approach. -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 18 Oct 2019 14:05:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Hello Bharath Vissapragada, Vihang Karajgaonkar, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14307 to look at the new patch set (#6). Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. IMPALA-7506: support global INVALIDATE METADATA in local catalog mode The minimal catalog object version of valid catalog objects is used to implement global invalidate metadata in legacy catalog mode. Coordinator sends DDL RPC to catalogd for global invalidate metadata and gets the expected min catalog version in the response. It's the version when catalogd starts to reset the entire catalog, which means when the reset is done, all valid catalog objects should be associated with a catalog version larger than it. Coordinator will wait until its min catalog version exceeds this value, which means it has processed all the updates of the reset propagated from the catalogd via statestored. If SYNC_DDL is set, the coordinator will also wait until other coordinators reach the same statestore topic version with it, so they have also processed the same updates and had the latest catalog after reset. In local catalog mode, the coordinator does not cache all the metadata. Instead, it caches them on-demand (based on query requests), and removes them based on the Guava cache configurations (size or TTL) or explicit invalidation from the catalog topic updates. So it's hard to track the minimal catalog object version correctly. This patch adds a new field (lastResetCatalogVersion) in TCatalog to propagate the catalog version when catalogd starts to reset the entire metadata. Each time when catalogd generates a new topic update, it will generate a TCatalogObject of CATALOG type containing the state of the catalog which includes this new field. To make all changes of the reset being added in the same topic update with this TCatalog object. Rapidly changed tables that have catalog version exceeding the version range of this update will also be included. When coordinator receives a new value of lastResetCatalogVersion in a topic update, it means catalogd has reset the entire catalog and all the relative updates are whether included in the same or previous topic updates. This is guaranteed by three facts: 1) No topic updates are sent from catalogd when the write lock of versionLock is held in CatalogServiceCatalog.reset(). Note that the update thread requires holding the read lock of versionLock. 2) Authz changes before holding the write lock can only be sent in a previous topic update or in the next topic update after reset(). 3) No catalog objects are skipped in the topic update right after reset(). See changes in GetCatalogDeltaContext. Thus, all metadata with catalog version <= lastResetCatalogVersion can be considered stale after coordinator finish processing the topic update. lastResetCatalogVersion + 1 is the lower bound (included) of min catalog object version of a coordinator. To avoid catalogd's update collector thread being blocked by concurrent DDLs that holding the table locks, this patch also fixes IMPALA-9062. In local catalog mode, we just need to propagate the table name of a changed table, so don't need to acquire table lock to get a full TTable object. This patch also exposes the min catalog object version of coordinator via a new metric "catalog.min-catalog-object-version" to ease debugging. Tests: - Recover all existing tests that have been disabled due to this missing feature Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 --- M be/src/service/impala-server.cc M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M common/thrift/CatalogObjects.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M tests/authorization/test_grant_revoke.py M tests/authorization/test_ranger.py M tests/common/skip.py M tests/custom_cluster/test_local_catalog.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py 14 files changed, 180 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/14307/6 -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] WIP: IMPALA-3357: change sorter to use three-way partitioning
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14373 ) Change subject: WIP: IMPALA-3357: change sorter to use three-way partitioning .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4826/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad72d82090ea5b5b19f62163cf01aa630e15f1f2 Gerrit-Change-Number: 14373 Gerrit-PatchSet: 1 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Oct 2019 13:17:18 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-3357: change sorter to use three-way partitioning
Norbert Luksa has posted comments on this change. ( http://gerrit.cloudera.org:8080/14373 ) Change subject: WIP: IMPALA-3357: change sorter to use three-way partitioning .. Patch Set 1: Code-Review-1 Did not improve performance significantly. -- To view, visit http://gerrit.cloudera.org:8080/14373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad72d82090ea5b5b19f62163cf01aa630e15f1f2 Gerrit-Change-Number: 14373 Gerrit-PatchSet: 1 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Oct 2019 12:37:40 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-3357: change sorter to use three-way partitioning
Norbert Luksa has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14373 Change subject: WIP: IMPALA-3357: change sorter to use three-way partitioning .. WIP: IMPALA-3357: change sorter to use three-way partitioning Sorter's quicksort implementation is very suboptimal for duplicate keys. The quicksort implementation in the sorter is based on dividing the input into two partitions: <= pivot and >= pivot. This commit introduces a middle partition, where the input is equal to the pivot (== pivot). This way it doesn't need to recurse on the middle partition. This means it can sort a partition full of duplicate values in a single pass over the input. Tests: * TODO Change-Id: Iad72d82090ea5b5b19f62163cf01aa630e15f1f2 --- M be/src/runtime/sorter-internal.h M be/src/runtime/sorter-ir.cc 2 files changed, 64 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/14373/1 -- To view, visit http://gerrit.cloudera.org:8080/14373 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iad72d82090ea5b5b19f62163cf01aa630e15f1f2 Gerrit-Change-Number: 14373 Gerrit-PatchSet: 1 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6501: Optimize count(star) for Kudu scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14347 ) Change subject: IMPALA-6501: Optimize count(star) for Kudu scans .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4825/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14347 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic99e0f954d0ca65779bd531ca79ace1fcb066fb9 Gerrit-Change-Number: 14347 Gerrit-PatchSet: 11 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 18 Oct 2019 12:27:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
Attila Jeges has posted comments on this change. (
http://gerrit.cloudera.org:8080/14291 )
Change subject: IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
..
Patch Set 10:
(10 comments)
Thanks for making the changes. Few more comments:
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-parser.h
File be/src/runtime/datetime-iso-sql-format-parser.h:
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-parser.h@81
PS10, Line 81: As a side effect moves '*format' to the next character in the
: // format.
It doesn't move *format to the next character, it moves it to the last
character of the escape sequence. If *format doesn't point at an escape
sequence, *format is not changed.
Maybe something like this:
"
If '*format' points at a beginning of an escape sequence, '*format' is moved to
the last character of the escape sequence. Otherwise, '*format' is not changed.
"
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-parser.cc
File be/src/runtime/datetime-iso-sql-format-parser.cc:
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-parser.cc@57
PS10, Line 57: ==
'>=' might be safer to use here
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-parser.cc@251
PS10, Line 251: // If we reached the end of input or the end of token
sequence, we can return.
: if (*current_pos >= end_pos || *current_tok_idx >=
dt_ctx.toks.size()) {
: return (*current_pos >= end_pos && *current_tok_idx >=
dt_ctx.toks.size());
: }
What if we reached the end of input but dt_ctx.toks still contains some empty
TEXT tokens?
select cast('1985-12-09-' as date format '-MM-DD-""');
I think this corner-case should be handled here, instead of just returning
false.
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-tokenizer.h
File be/src/runtime/datetime-iso-sql-format-tokenizer.h:
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-tokenizer.h@91
PS10, Line 91: function
string functions
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-tokenizer.h@128
PS10, Line 128: bool IsStartOfTextToken(const char* current_pos) const;
This should probably be a static function instead of const.
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-tokenizer.h@137
PS10, Line 137: start_str
str_start, here and elsewhere in the comment.
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-tokenizer.h@141
PS10, Line 141: const char* FindEndOfTextToken(const char* str_start, const
char* str_end,
: bool is_escaped);
This should be a static function too.
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-tokenizer.cc
File be/src/runtime/datetime-iso-sql-format-tokenizer.cc:
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-tokenizer.cc@124
PS10, Line 124: if (token->second.type == FX_MODIFIER) {
: if (used_tokens_.size() > 0) return
MISPLACED_FX_MODIFIER_ERROR;
: dt_ctx_->fx_modifier = true;
: *current_pos += curr_token_size;
: return SUCCESS;
: }
: if (token->second.type == FM_MODIFIER) {
: fm_modifier_active_ = true;
: *current_pos += curr_token_size;
: return SUCCESS;
: }
This allows weird format strings too, e.g.: 'FXFMFMFX-MM-DD'
Probably these should return an error.
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-tokenizer.cc@251
PS10, Line 251: DCHECK(str_begin < str_end);
nit: DCHECK(str_begin <= *current_pos && *current_pos < str_end);
http://gerrit.cloudera.org:8080/#/c/14291/10/be/src/runtime/datetime-iso-sql-format-tokenizer.cc@254
PS10, Line 254: (is_escaped)
nit: no need to put is_ecaped inside parentheses.
--
To view, visit http://gerrit.cloudera.org:8080/14291
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I30d2f6656054371476aaa8bd0d51f572b9369855
Gerrit-Change-Number: 14291
Gerrit-PatchSet: 10
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Fri, 18 Oct 2019 12:05:26 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
Attila Jeges has posted comments on this change. (
http://gerrit.cloudera.org:8080/14291 )
Change subject: IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
..
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14291/7/tests/query_test/test_cast_with_format.py
File tests/query_test/test_cast_with_format.py:
http://gerrit.cloudera.org:8080/#/c/14291/7/tests/query_test/test_cast_with_format.py@782
PS7, Line 782: # Strict separator matching.
: result = self.client.execute("select cast('2001-03-02
03:10:15' as timestamp format"
: "'FX MM-DD HH12:MI:SS')")
: assert result.data == ["NULL"]
:
: result = self.client.execute("select cast('2001-03-03
03:10:15' as timestamp format"
: "'FX-MM-DD HH12::MI:SS')")
: assert result.data == ["NULL"]
:
: result = self.client.execute("select cast('2001-03-04'
as timestamp format"
: "'FX-MM-DD ')")
: assert result.data == ["NULL"]
:
: # Strict token length matching.
: result = self.client.execute("select cast('2001-3-05' as
timestamp format "
: "'FX-MM-DD')")
: assert result.data == ["NULL"]
> L778 is a positive test for FX modifier. Does that cover what you ask for?
Ok, thanks.
--
To view, visit http://gerrit.cloudera.org:8080/14291
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I30d2f6656054371476aaa8bd0d51f572b9369855
Gerrit-Change-Number: 14291
Gerrit-PatchSet: 10
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Fri, 18 Oct 2019 12:10:23 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6501: Optimize count(star) for Kudu scans
Norbert Luksa has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/14347 ) Change subject: IMPALA-6501: Optimize count(star) for Kudu scans .. IMPALA-6501: Optimize count(star) for Kudu scans IMPALA-5036 added an optimisation for count(star) in Parquet scans that avoids materialising dummy rows. This change provides similar optimization for Kudu tables. Instead of materializing empty rows when computing count star, we use the NumRows field from the Kudu API. The Kudu scanner tuple is modified to have one slot into which we will write the num rows statistic. The aggregate function is changed from count to a special sum function that gets initialized to 0. Tests: * Added end-to-end tests ̣* Added planner tests * Run performance tests on tpch.lineitem Kudu table with 25 set as scaling factor, on 1 node, with mt_dop set to 1, just to measure the speedup gained when scanning. Counting the rows before the optimization took around 400ms, and around 170ms after. Change-Id: Ic99e0f954d0ca65779bd531ca79ace1fcb066fb9 --- M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scanner.cc M be/src/exec/kudu-scanner.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test A testdata/workloads/functional-planner/queries/PlannerTest/kudu-stats-agg.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-stats-agg.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test A testdata/workloads/functional-query/queries/QueryTest/kudu-stats-agg.test M tests/query_test/test_aggregation.py 18 files changed, 583 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/14347/11 -- To view, visit http://gerrit.cloudera.org:8080/14347 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic99e0f954d0ca65779bd531ca79ace1fcb066fb9 Gerrit-Change-Number: 14347 Gerrit-PatchSet: 11 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6501: Optimize count(star) for Kudu scans
Norbert Luksa has posted comments on this change. (
http://gerrit.cloudera.org:8080/14347 )
Change subject: IMPALA-6501: Optimize count(star) for Kudu scans
..
Patch Set 11:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/14347/10/be/src/exec/kudu-scanner.cc
File be/src/exec/kudu-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/14347/10/be/src/exec/kudu-scanner.cc@110
PS10, Line 110: while (scanner_->HasMoreRows()) {
: RETURN_IF_CANCELLED(state_);
: RETURN_IF_ERROR(GetNextScannerBatch());
:
: cur_kudu_batch_n
> optional: I think that this code would be even clearer by doing the loop fi
Done
http://gerrit.cloudera.org:8080/#/c/14347/10/be/src/exec/kudu-scanner.cc@130
PS10, Line 130: dst_row->SetTuple(0, tuple
> This seems unnecessary.
Done
http://gerrit.cloudera.org:8080/#/c/14347/10/be/src/exec/kudu-scanner.cc@132
PS10, Line 132:
> *eos must be true at this point, so I would replace this with a DCHECK
Since it has to be true, I'd simplify the loop even more and set it to true
after.
http://gerrit.cloudera.org:8080/#/c/14347/10/be/src/exec/kudu-scanner.cc@140
PS10, Line 140: if (scan_node_->optimize_count_star()) {
: return GetNextWithCountStarOptimization(row_batch, eos);
> possible further optimization: It would be enough to allocate a buffer for
Done
http://gerrit.cloudera.org:8080/#/c/14347/10/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
File fe/src/main/java/org/apache/impala/planner/KuduScanNode.java:
http://gerrit.cloudera.org:8080/#/c/14347/10/fe/src/main/java/org/apache/impala/planner/KuduScanNode.java@263
PS10, Line 263: for (SlotDescriptor desc:
getTupleDesc().getSlotsOrderedByOffset()) {
> Nit: Maybe we could invert the if and avoid using continue in this case:
Done
http://gerrit.cloudera.org:8080/#/c/14347/10/fe/src/main/java/org/apache/impala/planner/ScanNode.java
File fe/src/main/java/org/apache/impala/planner/ScanNode.java:
http://gerrit.cloudera.org:8080/#/c/14347/10/fe/src/main/java/org/apache/impala/planner/ScanNode.java@68
PS10, Line 68: // the count(*) optimization can be applied.
> Typo: 'can be applied'.
Done
http://gerrit.cloudera.org:8080/#/c/14347/10/fe/src/main/java/org/apache/impala/planner/ScanNode.java@114
PS10, Line 114: protected boolean
isCountStarOptimizationDescriptor(SlotDescriptor desc) {
> We could use a constant instead of the string literal especially as the sam
Done
http://gerrit.cloudera.org:8080/#/c/14347/10/tests/query_test/test_aggregation.py
File tests/query_test/test_aggregation.py:
http://gerrit.cloudera.org:8080/#/c/14347/10/tests/query_test/test_aggregation.py@273
PS10, Line 273:vector.get_value('table_format').compression_codec !=
'none'):
> Nit: Add one more space of indentation.
That would result in:
flake8: E129 visually indented line with same indent as next logical line
(see PS4)
--
To view, visit http://gerrit.cloudera.org:8080/14347
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic99e0f954d0ca65779bd531ca79ace1fcb066fb9
Gerrit-Change-Number: 14347
Gerrit-PatchSet: 11
Gerrit-Owner: Norbert Luksa
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Norbert Luksa
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 18 Oct 2019 11:46:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4824/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 18 Oct 2019 09:32:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Hello Quanlong Huang, Hao Hao, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14398 to look at the new patch set (#5). Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config When catalogd connected to mysql/postgresql directly instead of use HMS, a kudu managerd table created DDL would failed due to Preconditions checked failed. The patch fixed the bug by using the if condition instead of the Preconditions functions which lead to exception when lack of HMS config. Tests: * Add test for create/drop kudu managerd table without HMS config * Ran all front-end tests Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 --- M fe/src/main/java/org/apache/impala/catalog/KuduTable.java A fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java A fe/src/test/resources/hive-site-without-hms.xml.py 3 files changed, 177 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/14398/5 -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 5 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4822/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 3 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 18 Oct 2019 08:09:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9055: Impala shouldn't set expiration to NEVER for cache directives.
Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14485 ) Change subject: IMPALA-9055: Impala shouldn't set expiration to NEVER for cache directives. .. IMPALA-9055: Impala shouldn't set expiration to NEVER for cache directives. In HdfsCachingUtil we set the expiration of cache directives to never. This works well until the cache pool has max TTL set. Once max TTL is set Impala will get an exception when it tries to add caching for tables or partitions. I changed HdfsCachingUtil to not set the expiration. This way the cache directive inherits the expiration from the cache pool. Testing Added e2e test that creates a table in a cache pool that has max TTL. Change-Id: I475b92704b19e337b2e62f766e5b978585bf6583 Reviewed-on: http://gerrit.cloudera.org:8080/14485 Reviewed-by: Csaba Ringhofer Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java M testdata/bin/setup-hdfs-env.sh M testdata/workloads/functional-query/queries/QueryTest/hdfs-caching.test M tests/query_test/test_hdfs_caching.py 4 files changed, 19 insertions(+), 9 deletions(-) Approvals: Csaba Ringhofer: Looks good to me, but someone else must approve Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14485 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I475b92704b19e337b2e62f766e5b978585bf6583 Gerrit-Change-Number: 14485 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4823/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 4 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 18 Oct 2019 08:13:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4821/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 2 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 18 Oct 2019 08:07:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 4: > Is it possible to reproduce this in a dev environment? Would be > good to include a regression test we we don't break it again. Hi Tim, I've already add a fe test to validate this patch. -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 4 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 18 Oct 2019 07:38:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Hello Quanlong Huang, Hao Hao, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14398 to look at the new patch set (#4). Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config When catalogd connected to mysql/postgresql directly instead of use HMS, a kudu managerd table created DDL would failed due to Preconditions checked failed. The patch fixed the bug by using the if condition instead of the Preconditions functions which lead to exception when lack of HMS config. Tests: * Add test for create/drop kudu managerd table without HMS config * Ran all front-end tests Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 --- M fe/src/main/java/org/apache/impala/catalog/KuduTable.java A fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java A fe/src/test/resources/hive-site-without-hms.xml.py 3 files changed, 160 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/14398/4 -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 4 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/14398/3/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java File fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java: http://gerrit.cloudera.org:8080/#/c/14398/3/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java@26 PS3, Line 26: private static String HIVE_SITE_WITHOUT_HMS = HIVE_CONF_DIR + line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 3 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 18 Oct 2019 07:29:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Hello Quanlong Huang, Hao Hao, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14398 to look at the new patch set (#3). Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config When catalogd connected to mysql/postgresql directly instead of use HMS, a kudu managerd table created DDL would failed due to Preconditions checked failed. The patch fixed the bug by using the if condition instead of the Preconditions functions which lead to exception when lack of HMS config. Tests: * Add test for create/drop kudu managerd table without HMS config * Ran all front-end tests Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 --- M fe/src/main/java/org/apache/impala/catalog/KuduTable.java A fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java A fe/src/test/resources/hive-site-without-hms.xml.py 3 files changed, 160 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/14398/3 -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 3 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/14398/2/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java File fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java: http://gerrit.cloudera.org:8080/#/c/14398/2/fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java@26 PS2, Line 26: private static String HIVE_SITE_WITHOUT_HMS = HIVE_CONF_DIR + "/hive-site-without-hms.xml"; line too long (95 > 90) -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 2 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 18 Oct 2019 07:26:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config
Hello Quanlong Huang, Hao Hao, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14398 to look at the new patch set (#2). Change subject: IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config .. IMPALA-8974: Fixed a bug when create kudu managerd table without HMS config When catalogd connected to mysql/postgresql directly instead of use HMS, a kudu managerd table created DDL would failed due to Preconditions checked failed. The patch fixed the bug by using the if condition instead of the Preconditions functions which lead to exception when lack of HMS config. Tests: * Add test for create/drop kudu managerd table without HMS config * Ran all front-end tests Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 --- M fe/src/main/java/org/apache/impala/catalog/KuduTable.java A fe/src/test/java/org/apache/impala/customcluster/CreateKuduTableWithoutHMSTest.java A fe/src/test/resources/hive-site-without-hms.xml.py 3 files changed, 159 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/14398/2 -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 2 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] MPALA-9061: Update ant version for centos in bootstrap system.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14482 ) Change subject: MPALA-9061: Update ant version for centos in bootstrap_system.sh .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4820/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Gerrit-Change-Number: 14482 Gerrit-PatchSet: 2 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Oct 2019 07:13:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14291 ) Change subject: IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2 .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4819/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14291 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30d2f6656054371476aaa8bd0d51f572b9369855 Gerrit-Change-Number: 14291 Gerrit-PatchSet: 10 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Oct 2019 07:06:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
Hello Attila Jeges, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14291 to look at the new patch set (#10). Change subject: IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2 .. IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2 This patch adds additional datetime format tokens on top of what was introduced with Milestone 1 (IMPALA-8703). The tokens introduced: - Free text token: Surrounded by double quotes, a free text section can be given in the format where the same text is expected in the input without the surrounding double quotes. - FX modifier: This modifier has to be given at the beginning of the format and is valid for the whole format. In a string to datetime conversion this forces strict separator matching and expects all the tokens in the input to have the same length as their maximum length. E.g. A month has to be of length 2 prefixed by zero if needed. This is the default in a datetime to string conversion. - FM modifier: This modifier affects only the following token and overrides the FX modifier. Using this the value of a token can be shorter than the max length if followed by a separator. E.g. 1-digit month, less than 4-digit year, etc. This is the default behaviour in a string to datetime conversion. Change-Id: I30d2f6656054371476aaa8bd0d51f572b9369855 --- M be/src/runtime/date-parse-util.cc M be/src/runtime/datetime-iso-sql-format-parser.cc M be/src/runtime/datetime-iso-sql-format-parser.h M be/src/runtime/datetime-iso-sql-format-tokenizer.cc M be/src/runtime/datetime-iso-sql-format-tokenizer.h M be/src/runtime/datetime-parser-common.cc M be/src/runtime/datetime-parser-common.h M be/src/runtime/timestamp-parse-util.cc M tests/query_test/test_cast_with_format.py 9 files changed, 631 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/14291/10 -- To view, visit http://gerrit.cloudera.org:8080/14291 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I30d2f6656054371476aaa8bd0d51f572b9369855 Gerrit-Change-Number: 14291 Gerrit-PatchSet: 10 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] MPALA-9061: Update ant version for centos in bootstrap system.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14482 ) Change subject: MPALA-9061: Update ant version for centos in bootstrap_system.sh .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/14482/2/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/14482/2/bin/bootstrap_system.sh@241 PS2, Line 241: redhat sha512sum -c - <<< '487dbd1d7f678a92924ba884a57e910ccb4fe565c554278795a8fdfc80c4e88d81ebc2ccecb5a8f353f0b2076572bb921499a2cadb064e0f44fc406a3c31da20 apache-ant-1.9.14-bin.tar.gz' line too long (186 > 90) -- To view, visit http://gerrit.cloudera.org:8080/14482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Gerrit-Change-Number: 14482 Gerrit-PatchSet: 2 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Oct 2019 06:30:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] MPALA-9061: Update ant version for centos in bootstrap system.sh
Fucun Chu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14482 Change subject: MPALA-9061: Update ant version for centos in bootstrap_system.sh .. MPALA-9061: Update ant version for centos in bootstrap_system.sh bootstrap_system.sh currently use ant 1.9.13 on CentOS/Redhat environment. this release cannot be accessed, the earliest version was 1.9.14. please see https://www-us.apache.org/dist/ant/binaries/. upgrade version to 1.9.14 Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb --- M bin/bootstrap_system.sh 1 file changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/14482/2 -- To view, visit http://gerrit.cloudera.org:8080/14482 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1dc224e7afb16f95abc9262f094c4ff7aa465dfb Gerrit-Change-Number: 14482 Gerrit-PatchSet: 2 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14291 ) Change subject: IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2 .. Patch Set 10: (4 comments) http://gerrit.cloudera.org:8080/#/c/14291/10/tests/query_test/test_cast_with_format.py File tests/query_test/test_cast_with_format.py: http://gerrit.cloudera.org:8080/#/c/14291/10/tests/query_test/test_cast_with_format.py@743 PS10, Line 743: assert result.data == [r'''1985 text tab used for whitespace http://gerrit.cloudera.org:8080/#/c/14291/10/tests/query_test/test_cast_with_format.py@747 PS10, Line 747: assert result.data == [r'''1985 text tab used for whitespace http://gerrit.cloudera.org:8080/#/c/14291/10/tests/query_test/test_cast_with_format.py@751 PS10, Line 751: assert result.data == [r'''1985 text tab used for whitespace http://gerrit.cloudera.org:8080/#/c/14291/10/tests/query_test/test_cast_with_format.py@755 PS10, Line 755: assert result.data == [r'''1985 text tab used for whitespace -- To view, visit http://gerrit.cloudera.org:8080/14291 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30d2f6656054371476aaa8bd0d51f572b9369855 Gerrit-Change-Number: 14291 Gerrit-PatchSet: 10 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 18 Oct 2019 06:24:15 + Gerrit-HasComments: Yes
