[Impala-ASF-CR](hadoop-next) IMPALA-4277: bump Hadoop component versions except for Hadoop itself
Tim Armstrong has submitted this change and it was merged. Change subject: IMPALA-4277: bump Hadoop component versions except for Hadoop itself .. IMPALA-4277: bump Hadoop component versions except for Hadoop itself The Hive version bump required some minor changes to deal with renames and added function arguments. The other components required no changes to build the frontend against their API (I haven't been able to test the JNI backend stuff). IMPALA-4172 is blocking the hadoop version bump. This builds if I manually put hive_metastore.thrift in the right place and set SKIP_TOOLCHAIN_BOOTSTRAP=false. Tests can't run since we don't have the special tarballs available that are used for the test cluster. I manually constructed the Hive build dependency by taking hive_metastore.thrift and putting it at $CDH_COMPONENTS_HOME/hive-2.1.0-cdh6.x-SNAPSHOT/src/metastore/if/hive_metastore.thrift The Hive API changes were: * org.apache.hive.service.cli moved to org.apache.hive.service.rpc * MetaStoreUtils.validateName() and MetaStoreUtils.updatePartitionStatsFast() have additional context arguments. * HiveConf.ConfVars.METASTORE_BATCH_RETRIEVE_TABLE_PARTITION_MAX was renamed to METASTORE_BATCH_RETRIEVE_OBJECTS_MAX Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 Reviewed-on: http://gerrit.cloudera.org:8080/4698 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M bin/impala-config.sh M common/thrift/TCLIService.thrift M common/thrift/cli_service.thrift M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/Subquery.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/analysis/TypeDef.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/service/FrontendTest.java 16 files changed, 37 insertions(+), 39 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/4698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: hadoop-next Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Charlie Helin Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](hadoop-next) IMPALA-4277: bump Hadoop component versions except for Hadoop itself
Tim Armstrong has posted comments on this change. Change subject: IMPALA-4277: bump Hadoop component versions except for Hadoop itself .. Patch Set 3: Code-Review+2 Verified+1 Rebase. Confirmed that it compiles, but can't run tests yet. -- To view, visit http://gerrit.cloudera.org:8080/4698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: hadoop-next Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Charlie Helin Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: No
[Impala-ASF-CR](hadoop-next) IMPALA-4277: bump Hadoop component versions except for Hadoop itself
Hello Sailesh Mukil, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4698 to look at the new patch set (#3). Change subject: IMPALA-4277: bump Hadoop component versions except for Hadoop itself .. IMPALA-4277: bump Hadoop component versions except for Hadoop itself The Hive version bump required some minor changes to deal with renames and added function arguments. The other components required no changes to build the frontend against their API (I haven't been able to test the JNI backend stuff). IMPALA-4172 is blocking the hadoop version bump. This builds if I manually put hive_metastore.thrift in the right place and set SKIP_TOOLCHAIN_BOOTSTRAP=false. Tests can't run since we don't have the special tarballs available that are used for the test cluster. I manually constructed the Hive build dependency by taking hive_metastore.thrift and putting it at $CDH_COMPONENTS_HOME/hive-2.1.0-cdh6.x-SNAPSHOT/src/metastore/if/hive_metastore.thrift The Hive API changes were: * org.apache.hive.service.cli moved to org.apache.hive.service.rpc * MetaStoreUtils.validateName() and MetaStoreUtils.updatePartitionStatsFast() have additional context arguments. * HiveConf.ConfVars.METASTORE_BATCH_RETRIEVE_TABLE_PARTITION_MAX was renamed to METASTORE_BATCH_RETRIEVE_OBJECTS_MAX Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 --- M bin/impala-config.sh M common/thrift/TCLIService.thrift M common/thrift/cli_service.thrift M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/Subquery.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/analysis/TypeDef.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/service/FrontendTest.java 16 files changed, 37 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/4698/3 -- To view, visit http://gerrit.cloudera.org:8080/4698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: hadoop-next Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Charlie Helin Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](hadoop-next) IMPALA-4277: bump Hadoop component versions except for Hadoop itself
Sailesh Mukil has posted comments on this change. Change subject: IMPALA-4277: bump Hadoop component versions except for Hadoop itself .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/4698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: hadoop-next Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Charlie Helin Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: No
[Impala-ASF-CR](hadoop-next) IMPALA-4277: bump Hadoop component versions except for Hadoop itself
Charlie Helin has posted comments on this change. Change subject: IMPALA-4277: bump Hadoop component versions except for Hadoop itself .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/4698/1//COMMIT_MSG Commit Message: PS1, Line 16: This builds if I manually put hive_metastore.thrift in the right place > We don't have the special component tarballs for CDH6.x, so I just did this Ohh I see that looks better! -- To view, visit http://gerrit.cloudera.org:8080/4698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: hadoop-next Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Charlie Helin Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: Yes
[Impala-ASF-CR](hadoop-next) IMPALA-4277: bump Hadoop component versions except for Hadoop itself
Tim Armstrong has uploaded a new patch set (#2). Change subject: IMPALA-4277: bump Hadoop component versions except for Hadoop itself .. IMPALA-4277: bump Hadoop component versions except for Hadoop itself The Hive version bump required some minor changes to deal with renames and added function arguments. The other components required no changes to build the frontend against their API (I haven't been able to test the JNI backend stuff). IMPALA-4172 is blocking the hadoop version bump. This builds if I manually put hive_metastore.thrift in the right place and set SKIP_TOOLCHAIN_BOOTSTRAP=false. Tests can't run since we don't have the special tarballs available that are used for the test cluster. I manually constructed the Hive build dependency by taking hive_metastore.thrift and putting it at $CDH_COMPONENTS_HOME/hive-2.1.0-cdh6.x-SNAPSHOT/src/metastore/if/hive_metastore.thrift The Hive API changes were: * org.apache.hive.service.cli moved to org.apache.hive.service.rpc * MetaStoreUtils.validateName() and MetaStoreUtils.updatePartitionStatsFast() have additional context arguments. * HiveConf.ConfVars.METASTORE_BATCH_RETRIEVE_TABLE_PARTITION_MAX was renamed to METASTORE_BATCH_RETRIEVE_OBJECTS_MAX Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 --- M bin/impala-config.sh M common/thrift/TCLIService.thrift M common/thrift/cli_service.thrift M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/Subquery.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/analysis/TypeDef.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/service/FrontendTest.java 16 files changed, 38 insertions(+), 40 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/4698/2 -- To view, visit http://gerrit.cloudera.org:8080/4698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: hadoop-next Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Charlie Helin Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](hadoop-next) IMPALA-4277: bump Hadoop component versions except for Hadoop itself
Tim Armstrong has posted comments on this change. Change subject: IMPALA-4277: bump Hadoop component versions except for Hadoop itself .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/4698/1//COMMIT_MSG Commit Message: PS1, Line 16: This builds if I manually put hive_metastore.thrift in the right place > I agree with the question above. Would it not be better to use the env vari See https://gerrit.cloudera.org/#/c/4720/ PS1, Line 16: This builds if I manually put hive_metastore.thrift in the right place > Why does this need to manually be put in the right place? We don't have the special component tarballs for CDH6.x, so I just did this to validate that it can build. Line 23: > Since there are only a few API changes, I think it would be good if you cou Done -- To view, visit http://gerrit.cloudera.org:8080/4698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: hadoop-next Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Charlie Helin Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: Yes
[Impala-ASF-CR](hadoop-next) IMPALA-4277: bump Hadoop component versions except for Hadoop itself
Charlie Helin has posted comments on this change. Change subject: IMPALA-4277: bump Hadoop component versions except for Hadoop itself .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/4698/1//COMMIT_MSG Commit Message: PS1, Line 16: This builds if I manually put hive_metastore.thrift in the right place > Why does this need to manually be put in the right place? I agree with the question above. Would it not be better to use the env variables to point to any necessary location instead and then manually assign the variables such that it builds? That way you could separate the build into two separate steps 1) Retrieve/locate the proper artifacts 2) Point to the location where the artifact resides -- To view, visit http://gerrit.cloudera.org:8080/4698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: hadoop-next Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Charlie Helin Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: Yes
[Impala-ASF-CR](hadoop-next) IMPALA-4277: bump Hadoop component versions except for Hadoop itself
Sailesh Mukil has posted comments on this change. Change subject: IMPALA-4277: bump Hadoop component versions except for Hadoop itself .. Patch Set 1: (2 comments) Sorry for the delay. http://gerrit.cloudera.org:8080/#/c/4698/1//COMMIT_MSG Commit Message: PS1, Line 16: This builds if I manually put hive_metastore.thrift in the right place Why does this need to manually be put in the right place? Line 23: Since there are only a few API changes, I think it would be good if you could add a link here (if one exists) to the function signatures that changed, or just list them out. It will be good to refer back to this in the case of any issues related to these new APIs we see in testing in the future. -- To view, visit http://gerrit.cloudera.org:8080/4698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: hadoop-next Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Charlie Helin Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: Yes
[Impala-ASF-CR](hadoop-next) IMPALA-4277: bump Hadoop component versions except for Hadoop itself
Tim Armstrong has posted comments on this change. Change subject: IMPALA-4277: bump Hadoop component versions except for Hadoop itself .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/4698/1/bin/impala-config.sh File bin/impala-config.sh: Line 299 > Should this not also be bumped? It's bumped in https://gerrit.cloudera.org/#/c/4701/ - I wanted that patch to be separate because it adds some nasty hacks. I'll look at making that change on master, I think it's a good suggestion but we really need to sort out other issues before we can build against arbitrary versions without doing gymnastics to build component tarballs. -- To view, visit http://gerrit.cloudera.org:8080/4698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: hadoop-next Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Charlie Helin Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-HasComments: Yes
[Impala-ASF-CR](hadoop-next) IMPALA-4277: bump Hadoop component versions except for Hadoop itself
Charlie Helin has posted comments on this change.
Change subject: IMPALA-4277: bump Hadoop component versions except for Hadoop
itself
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/4698/1/bin/impala-config.sh
File bin/impala-config.sh:
Line 299
Should this not also be bumped?
Also I would make these configurable using something like:
export FOO_VERSION=${FOO_VERSION:-2.0.0-...}
--
To view, visit http://gerrit.cloudera.org:8080/4698
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: hadoop-next
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Charlie Helin
Gerrit-Reviewer: Sailesh Mukil
Gerrit-HasComments: Yes
[Impala-ASF-CR](hadoop-next) IMPALA-4277: bump Hadoop component versions except for Hadoop itself
Tim Armstrong has uploaded a new change for review. http://gerrit.cloudera.org:8080/4698 Change subject: IMPALA-4277: bump Hadoop component versions except for Hadoop itself .. IMPALA-4277: bump Hadoop component versions except for Hadoop itself The Hive version bump required some minor changes to deal with renames and added function arguments. The other components required no changes to build the frontend against their API (I haven't been able to test the JNI backend stuff). IMPALA-4172 is blocking the hadoop version bump. This builds if I manually put hive_metastore.thrift in the right place and set SKIP_TOOLCHAIN_BOOTSTRAP=false. Tests can't run since we don't have the special tarballs available that are used for the test cluster. I manually constructed the Hive build dependency by taking hive_metastore.thrift and putting it at $CDH_COMPONENTS_HOME/hive-2.1.0-cdh6.x-SNAPSHOT/src/metastore/if/hive_metastore.thrift Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 --- M bin/impala-config.sh M common/thrift/TCLIService.thrift M common/thrift/cli_service.thrift M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/Subquery.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/analysis/TypeDef.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/service/FrontendTest.java 16 files changed, 38 insertions(+), 40 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/4698/1 -- To view, visit http://gerrit.cloudera.org:8080/4698 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia0be6bbe76d929ceaeb2fa9ac4ebba1820c4dab7 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: hadoop-next Gerrit-Owner: Tim Armstrong
