[jira] [Commented] (SQOOP-3381) Upgrade the Parquet library from 1.6.0 to 1.9.0
[ https://issues.apache.org/jira/browse/SQOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650311#comment-16650311 ] Fero Szabo commented on SQOOP-3381: --- Hi [~dvoros], Thanks for letting me know. Anyway, I've just updated my patch on Reviewboard. I encountered the same security policy related issue as you did with the Hadoop upgrade in SQOOP-3305, so I've incorporated the DerbyPolicy and the related code changes in mine. I hope you approve, and can have a look at it. :) I've decided to go for an older version of Hive, 2.1.1, since that suffices for this parquet upgrade as well. > Upgrade the Parquet library from 1.6.0 to 1.9.0 > --- > > Key: SQOOP-3381 > URL: https://issues.apache.org/jira/browse/SQOOP-3381 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > Fix For: 3.0.0 > > > As we will need to register a data supplier in the fix for parquet decimal > support, we will need a version that contains PARQUET-243. > We need to upgrade the Parquet library to a version that contains this fix > and is compatible with Hadoop. Most probably, the newest version will be > adequate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3384) Document import into external Hive table backed by S3
[ https://issues.apache.org/jira/browse/SQOOP-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650292#comment-16650292 ] Hudson commented on SQOOP-3384: --- FAILURE: Integrated in Jenkins build Sqoop-hadoop200 #1215 (See [https://builds.apache.org/job/Sqoop-hadoop200/1215/]) SQOOP-3384: Document import into external Hive table backed by S3 (vasas: [https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=c329f360dd08ef3b9bd82897fcd611e7431d32c8]) * (edit) src/docs/user/s3.txt > Document import into external Hive table backed by S3 > - > > Key: SQOOP-3384 > URL: https://issues.apache.org/jira/browse/SQOOP-3384 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Boglarka Egyed >Assignee: Boglarka Egyed >Priority: Major > Fix For: 3.0.0 > > Attachments: SQOOP-3384.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 68687: SQOOP-3381 Upgrade the Parquet library
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68687/ --- (Updated Oct. 15, 2018, 2:13 p.m.) Review request for Sqoop, Boglarka Egyed, daniel voros, and Szabolcs Vasas. Bugs: SQOOP-3381 https://issues.apache.org/jira/browse/SQOOP-3381 Repository: sqoop-trunk Description (updated) --- This change upgrades our parquet library to the newest version and a whole lot of libraries to newer versions with it. As we will need to register a data supplier in the fix for parquet decimal support (SQOOP-3382), we will need a version that contains PARQUET-243. We need to upgrade the Parquet library to a version that contains this fix and is compatible with Hadoop 3.0. A few things to note: - hadoop's version is still 2.8.0 - hive is upgraded to 2.1.1 - the rest of the dependency changes are required for the hive version bump. There is are a few changes in the codebase, but of course no new functionality at all: - in the TestParquetImport class, the new implementation returns a Utf8 object for Strings written out. - Added the security policy and related code changes from the patch for SQOOP-3305 (upgrade hadoop) written by Daniel Voros. - modified HiveMiniCluster config so it won't try to start a web ui (it's unnecessary during tests anyway) Diffs (updated) - build.gradle fc7fc0c4 gradle.properties 0d30378d gradle/sqoop-package.gradle 1a8d994d ivy.xml 670cb32d ivy/libraries.properties 8f3dab2b src/java/org/apache/sqoop/avro/AvroUtil.java 1663b1d1 src/java/org/apache/sqoop/hive/HiveImport.java 48800366 src/java/org/apache/sqoop/mapreduce/hcat/DerbyPolicy.java PRE-CREATION src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 784b5f2a src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetExportJobConfigurator.java 2180cc20 src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetImportJobConfigurator.java 90b910a3 src/java/org/apache/sqoop/mapreduce/parquet/hadoop/HadoopParquetMergeJobConfigurator.java 66ebc5b8 src/test/org/apache/sqoop/TestParquetExport.java be1d8164 src/test/org/apache/sqoop/TestParquetImport.java 2810e318 src/test/org/apache/sqoop/TestParquetIncrementalImportMerge.java adad0cc1 src/test/org/apache/sqoop/hive/TestHiveServer2ParquetImport.java b55179a4 src/test/org/apache/sqoop/hive/minicluster/HiveMiniCluster.java 9dd54486 src/test/org/apache/sqoop/util/ParquetReader.java f1c2fe10 testdata/hcatalog/conf/hive-site.xml 8a84a5d3 Diff: https://reviews.apache.org/r/68687/diff/4/ Changes: https://reviews.apache.org/r/68687/diff/3-4/ Testing (updated) --- Ant unit and 3rd party tests were successful. gradlew test and thirdpartytest were succesful as well. Thanks, Fero Szabo
[jira] [Commented] (SQOOP-3384) Document import into external Hive table backed by S3
[ https://issues.apache.org/jira/browse/SQOOP-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650210#comment-16650210 ] ASF subversion and git services commented on SQOOP-3384: Commit c329f360dd08ef3b9bd82897fcd611e7431d32c8 in sqoop's branch refs/heads/trunk from [~vasas] [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=c329f36 ] SQOOP-3384: Document import into external Hive table backed by S3 (Boglarka Egyed via Szabolcs Vasas) > Document import into external Hive table backed by S3 > - > > Key: SQOOP-3384 > URL: https://issues.apache.org/jira/browse/SQOOP-3384 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Boglarka Egyed >Assignee: Boglarka Egyed >Priority: Major > Attachments: SQOOP-3384.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 68979: SQOOP-3384: Document import into external Hive table backed by S3
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68979/#review209540 --- Ship it! Ship It! - Fero Szabo On Oct. 10, 2018, 3:42 p.m., Boglarka Egyed wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68979/ > --- > > (Updated Oct. 10, 2018, 3:42 p.m.) > > > Review request for Sqoop, Fero Szabo and Szabolcs Vasas. > > > Bugs: SQOOP-3384 > https://issues.apache.org/jira/browse/SQOOP-3384 > > > Repository: sqoop-trunk > > > Description > --- > > Document import into external Hive table backed by S3 > > > Diffs > - > > src/docs/user/s3.txt 3724454d7efda6b390a5984d9be44d20c404f766 > > > Diff: https://reviews.apache.org/r/68979/diff/1/ > > > Testing > --- > > ant clean docs > ./gradlew docs > > > Thanks, > > Boglarka Egyed > >
Re: Review Request 68979: SQOOP-3384: Document import into external Hive table backed by S3
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68979/#review209535 --- Ship it! Hi Bogi, Thank you for documenting this feature too! and docs and gradle docs are successful on my side too. Szabolcs - Szabolcs Vasas On Oct. 10, 2018, 3:42 p.m., Boglarka Egyed wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68979/ > --- > > (Updated Oct. 10, 2018, 3:42 p.m.) > > > Review request for Sqoop, Fero Szabo and Szabolcs Vasas. > > > Bugs: SQOOP-3384 > https://issues.apache.org/jira/browse/SQOOP-3384 > > > Repository: sqoop-trunk > > > Description > --- > > Document import into external Hive table backed by S3 > > > Diffs > - > > src/docs/user/s3.txt 3724454d7efda6b390a5984d9be44d20c404f766 > > > Diff: https://reviews.apache.org/r/68979/diff/1/ > > > Testing > --- > > ant clean docs > ./gradlew docs > > > Thanks, > > Boglarka Egyed > >
[jira] [Commented] (SQOOP-3378) Error during direct Netezza import/export can interrupt process in uncontrolled ways
[
https://issues.apache.org/jira/browse/SQOOP-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649977#comment-16649977
]
Szabolcs Vasas commented on SQOOP-3378:
---
Hi [~dvoros],
It seems TestNetezzaExternalTableExportMapper is still failing on the Apache
Jenkins server and I have also managed to reproduce the issue locally by
running the unit tests in 1 JVM.
Do you have any idea what can cause this hanging issue?
I have seen "Need to open FIFO for reading, otherwise writing would hang"
comment in the patch so I thought you might have encountered similar thing
during the development.
The branch I used to reproduce the issue is this:
[https://github.com/szvasas/sqoop/commits/SQOOP-3104]
If you execute ./gradlew unitTest then it would hang when executing
TestNetezzaExternalTableExportMapper.
> Error during direct Netezza import/export can interrupt process in
> uncontrolled ways
>
>
> Key: SQOOP-3378
> URL: https://issues.apache.org/jira/browse/SQOOP-3378
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Major
> Fix For: 1.5.0, 3.0.0
>
> Attachments: SQOOP-3378.2.patch
>
>
> SQLException during JDBC operation in direct Netezza import/export signals
> parent thread to fail fast by interrupting it (see
> [here|https://github.com/apache/sqoop/blob/c814e58348308b05b215db427412cd6c0b21333e/src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java#L92]).
> We're [trying to process the interrupt in the
> parent|https://github.com/apache/sqoop/blob/c814e58348308b05b215db427412cd6c0b21333e/src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java#L232]
> (main) thread, but there's no guarantee that we're not in some blocking
> internal call that will process the interrupted flag and reset it before
> we're able to check.
> It is also possible that the parent thread has passed the "checking part"
> when it gets interrupted. In case of {{NetezzaExternalTableExportMapper}}
> this can interrupt the upload of log files.
> I'd recommend using some other means of communication between the threads
> than interrupts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3392) Document metadata-transaction-isolation-level option
[
https://issues.apache.org/jira/browse/SQOOP-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boglarka Egyed updated SQOOP-3392:
--
Description: Option {{metadata-transaction-isolation-level}} has been added
in SQOOP-2349 however documentation, example commands are missing. (was:
Option {{metadata-transaction-isolation-level}} has been added in SQOOP-2349
however documentation, example command are missing.)
> Document metadata-transaction-isolation-level option
>
>
> Key: SQOOP-3392
> URL: https://issues.apache.org/jira/browse/SQOOP-3392
> Project: Sqoop
> Issue Type: Task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Priority: Major
>
> Option {{metadata-transaction-isolation-level}} has been added in SQOOP-2349
> however documentation, example commands are missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SQOOP-3392) Document metadata-transaction-isolation-level option
Boglarka Egyed created SQOOP-3392:
-
Summary: Document metadata-transaction-isolation-level option
Key: SQOOP-3392
URL: https://issues.apache.org/jira/browse/SQOOP-3392
Project: Sqoop
Issue Type: Task
Affects Versions: 1.4.7
Reporter: Boglarka Egyed
Option {{metadata-transaction-isolation-level}} has been added in SQOOP-2349
however documentation, example command are missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
