[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0
[ https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653217#comment-16653217 ] Hudson commented on SQOOP-3381: --- FAILURE: Integrated in Jenkins build Sqoop-hadoop200 #1216 (See [https://builds.apache.org/job/Sqoop-hadoop200/1216/]) SQOOP-3381: Upgrade the Parquet library from 1.6.0 to 1.9.0 (vasas: [https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=5dd8c8aad1c7732754fae190eb5424371ed6fef4]) * (edit) src/test/org/apache/sqoop/util/ParquetReader.java * (add) src/java/org/apache/sqoop/mapreduce/hcat/DerbyPolicy.java * (edit) src/test/org/apache/sqoop/TestParquetExport.java * (edit) src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java * (edit) gradle/sqoop-package.gradle * (edit) src/java/org/apache/sqoop/avro/AvroUtil.java * (edit) src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java * (edit) src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java * (edit) src/test/org/apache/sqoop/TestParquetImport.java * (edit) src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java * (edit) ivy.xml * (edit) src/java/org/apache/sqoop/hive/HiveImport.java * (edit) src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java * (edit) src/test/org/apache/sqoop/hive/minicluster/HiveMiniCluster.java * (edit) build.gradle * (edit) gradle.properties * (edit) src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java * (edit) testdata/hcatalog/conf/hive-site.xml * (edit) ivy/libraries.properties > Upgrade the Parquet library from 1.6.0 to 1.9.0 > --- > > Key: SQOOP-3381 > URL: https://issues.apache.org/jira/browse/SQOOP-3381 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > Fix For: 3.0.0 > > > As we will need to register a data supplier in the fix for parquet decimal > support, we will need a version that contains PARQUET-243. > We need to upgrade the Parquet library to a version that contains this fix > and is compatible with Hadoop. Most probably, the newest version will be > adequate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0
[ https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653172#comment-16653172 ] ASF subversion and git services commented on SQOOP-3381: Commit 5dd8c8aad1c7732754fae190eb5424371ed6fef4 in sqoop's branch refs/heads/trunk from [~vasas] [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=5dd8c8a ] SQOOP-3381: Upgrade the Parquet library from 1.6.0 to 1.9.0 (Fero Szabo via Szabolcs Vasas) > Upgrade the Parquet library from 1.6.0 to 1.9.0 > --- > > Key: SQOOP-3381 > URL: https://issues.apache.org/jira/browse/SQOOP-3381 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > Fix For: 3.0.0 > > > As we will need to register a data supplier in the fix for parquet decimal > support, we will need a version that contains PARQUET-243. > We need to upgrade the Parquet library to a version that contains this fix > and is compatible with Hadoop. Most probably, the newest version will be > adequate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0
[ https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650311#comment-16650311 ] Fero Szabo commented on SQOOP-3381: --- Hi [~dvoros], Thanks for letting me know. Anyway, I've just updated my patch on Reviewboard. I encountered the same security policy related issue as you did with the Hadoop upgrade in SQOOP-3305, so I've incorporated the DerbyPolicy and the related code changes in mine. I hope you approve, and can have a look at it. :) I've decided to go for an older version of Hive, 2.1.1, since that suffices for this parquet upgrade as well. > Upgrade the Parquet library from 1.6.0 to 1.9.0 > --- > > Key: SQOOP-3381 > URL: https://issues.apache.org/jira/browse/SQOOP-3381 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > Fix For: 3.0.0 > > > As we will need to register a data supplier in the fix for parquet decimal > support, we will need a version that contains PARQUET-243. > We need to upgrade the Parquet library to a version that contains this fix > and is compatible with Hadoop. Most probably, the newest version will be > adequate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0
[ https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639659#comment-16639659 ] Daniel Voros commented on SQOOP-3381: - With SQOOP-3305 I've decided to hold off until there's an HBase release that supports Hadoop 3.x. I don't think Hive 3.1.0 would help in this regard, since parquet classes are still shaded in hive-exec:3.1.0. > Upgrade the Parquet library from 1.6.0 to 1.9.0 > --- > > Key: SQOOP-3381 > URL: https://issues.apache.org/jira/browse/SQOOP-3381 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > Fix For: 3.0.0 > > > As we will need to register a data supplier in the fix for parquet decimal > support, we will need a version that contains PARQUET-243. > We need to upgrade the Parquet library to a version that contains this fix > and is compatible with Hadoop. Most probably, the newest version will be > adequate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0
[ https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639554#comment-16639554 ] Fero Szabo commented on SQOOP-3381: --- Hi [~dvoros], Thanks for your comment and sorry for the late answer! I've been pretty busy in the last few weeks with other issues, but now am ready to continue working on this one. Thanks for pointing me to the shaded parquet-hadoop-bundle. A few Hive tests are failing because of it... It makes me wonder if you've made progress with SQOOP-3305 in the meantime, and if upgrading Hive to 3.1.0 would solve this problem? Because of the failing Hive tests, I haven't tested on a cluster, yet, but will certainly do so. > Upgrade the Parquet library from 1.6.0 to 1.9.0 > --- > > Key: SQOOP-3381 > URL: https://issues.apache.org/jira/browse/SQOOP-3381 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > Fix For: 3.0.0 > > > As we will need to register a data supplier in the fix for parquet decimal > support, we will need a version that contains PARQUET-243. > We need to upgrade the Parquet library to a version that contains this fix > and is compatible with Hadoop. Most probably, the newest version will be > adequate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0
[ https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16612402#comment-16612402 ] Daniel Voros commented on SQOOP-3381: - Hey [~fero], thanks for keeping that in mind. What I've seen during the hadoop3 upgrade, is that Avro is added to the MR classpath from under hadoop. So where this could lead to issues is conflicting versions of Avro in hadoop and Parquet shipped with Sqoop. Could you try your patch (having new parquet jar in lib/) on a cluster with current Hadoop versions? I don't think we should bother with testing with Hadoop 3, we'll face that in the Hadoop 3 patch. (One more thing to keep in mind, is that parquet-hadoop-bundle is also shaded into the hive-exec artifact. However, I think the classes involved in PARQUET-243 are not bundled there.) > Upgrade the Parquet library from 1.6.0 to 1.9.0 > --- > > Key: SQOOP-3381 > URL: https://issues.apache.org/jira/browse/SQOOP-3381 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > Fix For: 3.0.0 > > > As we will need to register a data supplier in the fix for parquet decimal > support, we will need a version that contains PARQUET-243. > We need to upgrade the Parquet library to a version that contains this fix > and is compatible with Hadoop. Most probably, the newest version will be > adequate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0
[ https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16612271#comment-16612271 ] Fero Szabo commented on SQOOP-3381: --- Hi [~dvoros], I think this change might affect the Hadoop 3.0 upgrade... Can you perhaps comment on this? Thanks! Fero > Upgrade the Parquet library from 1.6.0 to 1.9.0 > --- > > Key: SQOOP-3381 > URL: https://issues.apache.org/jira/browse/SQOOP-3381 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > Fix For: 3.0.0 > > > As we will need to register a data supplier in the fix for parquet decimal > support, we will need a version that contains PARQUET-243. > We need to upgrade the Parquet library to a version that contains this fix > and is compatible with Hadoop. Most probably, the newest version will be > adequate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
