[jira] [Updated] (SQOOP-3363) Test incremental import with S3
[ https://issues.apache.org/jira/browse/SQOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boglarka Egyed updated SQOOP-3363: -- Attachment: SQOOP-3363.patch > Test incremental import with S3 > --- > > Key: SQOOP-3363 > URL: https://issues.apache.org/jira/browse/SQOOP-3363 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Boglarka Egyed >Assignee: Boglarka Egyed >Priority: Major > Attachments: SQOOP-3363.patch, SQOOP-3363.patch, SQOOP-3363.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 68475: SQOOP-3363: Test incremental import with S3
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68475/ --- (Updated Aug. 23, 2018, 6:18 p.m.) Review request for Sqoop, daniel voros, Fero Szabo, Nguyen Truong, and Szabolcs Vasas. Changes --- Added temporary rootdir clean-up logic at the end of every test case Bugs: SQOOP-3363 https://issues.apache.org/jira/browse/SQOOP-3363 Repository: sqoop-trunk Description --- * Added new test cases for Parquet import into S3 as it was still missing * Added new test cases for incremental append import into S3 in Text, Avro, Sequence and Parquet file format * Added new test cases for incremental merge import into S3 in Text and Parquet file format * Updated some previously added logic in S3 util and test classes Diffs (updated) - src/test/org/apache/sqoop/s3/TestS3AvroImport.java e130c42104b86e854d45babc009a5f1409a74a48 src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3ParquetImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java c17c1c54918df0b4d1ecbaef4e381975d72756ae src/test/org/apache/sqoop/s3/TestS3TextImport.java 60e2cd3025e67ecd43bdfb6b30d1b8d69a50da86 src/test/org/apache/sqoop/testutil/AvroTestUtils.java 04a8494a5d1d8a5020d5a3b629bbab62d3c09ffd src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 1730698e80cc77395f8a296b7bf01c104533e10b src/test/org/apache/sqoop/testutil/ParquetFileTestUtils.java PRE-CREATION src/test/org/apache/sqoop/testutil/S3TestUtils.java ceaff3b3a2bfd031b9772c9b43afdfa670c23718 src/test/org/apache/sqoop/testutil/SequenceFileTestUtils.java ad7576dbb2447423c677429f24163031a9d39b5f src/test/org/apache/sqoop/testutil/TextFileTestUtils.java df19cb8be7a633a6f1e1e3f9bc7d0dbc268aa90a Diff: https://reviews.apache.org/r/68475/diff/3/ Changes: https://reviews.apache.org/r/68475/diff/2-3/ Testing --- ant clean test -Ds3.bucket.url= -Ds3.generator.command= Thanks, Boglarka Egyed
Re: Review Request 68470: SQOOP-3366 Improve unit tests to be able to execute them in a single JVM
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68470/#review207820 --- Ship it! Ship It! - Fero Szabo On Aug. 23, 2018, 3:54 p.m., Nguyen Truong wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68470/ > --- > > (Updated Aug. 23, 2018, 3:54 p.m.) > > > Review request for Sqoop. > > > Bugs: SQOOP-3366 > https://issues.apache.org/jira/browse/SQOOP-3366 > > > Repository: sqoop-trunk > > > Description > --- > > Fix the unit tests which changed the state of the JVM so that other tests > will not be affected when they are run on a single JVM. > > > Diffs > - > > src/test/org/apache/sqoop/TestIncrementalImport.java e1faf351f > src/test/org/apache/sqoop/TestSqoopOptions.java d0591ad0d > > src/test/org/apache/sqoop/metastore/TestMetastoreConfigurationParameters.java > 0f1eb890b > src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 1730698e8 > src/test/org/apache/sqoop/testutil/HsqldbTestServer.java c63a8f2dc > src/test/org/apache/sqoop/tool/TestMainframeImportTool.java c2edc5347 > > > Diff: https://reviews.apache.org/r/68470/diff/3/ > > > Testing > --- > > No test has been added. > > > Thanks, > > Nguyen Truong > >
Re: Review Request 68470: SQOOP-3366 Improve unit tests to be able to execute them in a single JVM
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68470/#review207818 --- Ship it! Hi Nguyen, Thanks for submitting this patch! I have applied it, disabled the kerberized unit tests (TestKerberosAuthenticator, TestHiveMiniCluster, TestHiveServer2TextImport) and ./gradlew test ran successfully with forkEvery parameter set to 0. The test execution time has reduced to 8.5 minutes from ~20 minutes! - Szabolcs Vasas On Aug. 23, 2018, 3:54 p.m., Nguyen Truong wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68470/ > --- > > (Updated Aug. 23, 2018, 3:54 p.m.) > > > Review request for Sqoop. > > > Bugs: SQOOP-3366 > https://issues.apache.org/jira/browse/SQOOP-3366 > > > Repository: sqoop-trunk > > > Description > --- > > Fix the unit tests which changed the state of the JVM so that other tests > will not be affected when they are run on a single JVM. > > > Diffs > - > > src/test/org/apache/sqoop/TestIncrementalImport.java e1faf351f > src/test/org/apache/sqoop/TestSqoopOptions.java d0591ad0d > > src/test/org/apache/sqoop/metastore/TestMetastoreConfigurationParameters.java > 0f1eb890b > src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 1730698e8 > src/test/org/apache/sqoop/testutil/HsqldbTestServer.java c63a8f2dc > src/test/org/apache/sqoop/tool/TestMainframeImportTool.java c2edc5347 > > > Diff: https://reviews.apache.org/r/68470/diff/3/ > > > Testing > --- > > No test has been added. > > > Thanks, > > Nguyen Truong > >
[jira] [Updated] (SQOOP-3363) Test incremental import with S3
[ https://issues.apache.org/jira/browse/SQOOP-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boglarka Egyed updated SQOOP-3363: -- Attachment: SQOOP-3363.patch > Test incremental import with S3 > --- > > Key: SQOOP-3363 > URL: https://issues.apache.org/jira/browse/SQOOP-3363 > Project: Sqoop > Issue Type: Sub-task >Affects Versions: 1.4.7 >Reporter: Boglarka Egyed >Assignee: Boglarka Egyed >Priority: Major > Attachments: SQOOP-3363.patch, SQOOP-3363.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 68475: SQOOP-3363: Test incremental import with S3
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68475/ --- (Updated Aug. 23, 2018, 3:56 p.m.) Review request for Sqoop, daniel voros, Fero Szabo, Nguyen Truong, and Szabolcs Vasas. Changes --- Eliminated issues with parallel test execution Bugs: SQOOP-3363 https://issues.apache.org/jira/browse/SQOOP-3363 Repository: sqoop-trunk Description --- * Added new test cases for Parquet import into S3 as it was still missing * Added new test cases for incremental append import into S3 in Text, Avro, Sequence and Parquet file format * Added new test cases for incremental merge import into S3 in Text and Parquet file format * Updated some previously added logic in S3 util and test classes Diffs (updated) - src/test/org/apache/sqoop/s3/TestS3AvroImport.java e130c42104b86e854d45babc009a5f1409a74a48 src/test/org/apache/sqoop/s3/TestS3IncrementalAppendAvroImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3IncrementalAppendParquetImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3IncrementalAppendSequenceFileImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3IncrementalAppendTextImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3IncrementalMergeParquetImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3IncrementalMergeTextImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3ParquetImport.java PRE-CREATION src/test/org/apache/sqoop/s3/TestS3SequenceFileImport.java c17c1c54918df0b4d1ecbaef4e381975d72756ae src/test/org/apache/sqoop/s3/TestS3TextImport.java 60e2cd3025e67ecd43bdfb6b30d1b8d69a50da86 src/test/org/apache/sqoop/testutil/AvroTestUtils.java 04a8494a5d1d8a5020d5a3b629bbab62d3c09ffd src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 1730698e80cc77395f8a296b7bf01c104533e10b src/test/org/apache/sqoop/testutil/ParquetFileTestUtils.java PRE-CREATION src/test/org/apache/sqoop/testutil/S3TestUtils.java ceaff3b3a2bfd031b9772c9b43afdfa670c23718 src/test/org/apache/sqoop/testutil/SequenceFileTestUtils.java ad7576dbb2447423c677429f24163031a9d39b5f src/test/org/apache/sqoop/testutil/TextFileTestUtils.java df19cb8be7a633a6f1e1e3f9bc7d0dbc268aa90a Diff: https://reviews.apache.org/r/68475/diff/2/ Changes: https://reviews.apache.org/r/68475/diff/1-2/ Testing --- ant clean test -Ds3.bucket.url= -Ds3.generator.command= Thanks, Boglarka Egyed
Re: Review Request 68470: SQOOP-3366 Improve unit tests to be able to execute them in a single JVM
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68470/ --- (Updated Aug. 23, 2018, 3:54 p.m.) Review request for Sqoop. Changes --- Sorry, the previous one was not up-to-date with trunk. Bugs: SQOOP-3366 https://issues.apache.org/jira/browse/SQOOP-3366 Repository: sqoop-trunk Description --- Fix the unit tests which changed the state of the JVM so that other tests will not be affected when they are run on a single JVM. Diffs (updated) - src/test/org/apache/sqoop/TestIncrementalImport.java e1faf351f src/test/org/apache/sqoop/TestSqoopOptions.java d0591ad0d src/test/org/apache/sqoop/metastore/TestMetastoreConfigurationParameters.java 0f1eb890b src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 1730698e8 src/test/org/apache/sqoop/testutil/HsqldbTestServer.java c63a8f2dc src/test/org/apache/sqoop/tool/TestMainframeImportTool.java c2edc5347 Diff: https://reviews.apache.org/r/68470/diff/3/ Changes: https://reviews.apache.org/r/68470/diff/2-3/ Testing --- No test has been added. Thanks, Nguyen Truong
[jira] [Assigned] (SQOOP-3371) Fix tests using HiveMiniCluster
[ https://issues.apache.org/jira/browse/SQOOP-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas reassigned SQOOP-3371: - Assignee: Szabolcs Vasas > Fix tests using HiveMiniCluster > --- > > Key: SQOOP-3371 > URL: https://issues.apache.org/jira/browse/SQOOP-3371 > Project: Sqoop > Issue Type: Test >Affects Versions: 1.4.7 >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas >Priority: Major > > It seems that SQOOP-3360 broke our tests which use HiveMiniCluster because > org.apache.calcite is not present in the hive-exec:core JAR but this > dependency seems to be needed by these tests. > I am not sure why our Jenkins job did not catch the issue earlier but I get > consistent failures when I run these tests with a clean ivy cache and ant so > the dependency issue needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3371) Fix tests using HiveMiniCluster
Szabolcs Vasas created SQOOP-3371: - Summary: Fix tests using HiveMiniCluster Key: SQOOP-3371 URL: https://issues.apache.org/jira/browse/SQOOP-3371 Project: Sqoop Issue Type: Test Affects Versions: 1.4.7 Reporter: Szabolcs Vasas It seems that SQOOP-3360 broke our tests which use HiveMiniCluster because org.apache.calcite is not present in the hive-exec:core JAR but this dependency seems to be needed by these tests. I am not sure why our Jenkins job did not catch the issue earlier but I get consistent failures when I run these tests with a clean ivy cache and ant so the dependency issue needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3224) Mainframe FTP transfer should have an option to use binary mode for transfer
[ https://issues.apache.org/jira/browse/SQOOP-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590384#comment-16590384 ] Hudson commented on SQOOP-3224: --- SUCCESS: Integrated in Jenkins build Sqoop-hadoop200 #1199 (See [https://builds.apache.org/job/Sqoop-hadoop200/1199/]) SQOOP-3224: Mainframe FTP transfer should have an option to use binary (vasas: [https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0d6c455e5bbfc092d4a90f352eb262f347758132]) * (edit) src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java * (add) src/java/org/apache/sqoop/mapreduce/mainframe/AbstractMainframeDatasetImportMapper.java * (add) src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetBinaryRecord.java * (add) src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java * (edit) src/test/org/apache/sqoop/tool/TestMainframeImportTool.java * (edit) src/docs/user/import-mainframe.txt * (edit) src/java/org/apache/sqoop/tool/BaseSqoopTool.java * (add) src/java/org/apache/sqoop/mapreduce/ByteKeyOutputFormat.java * (edit) src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java * (add) src/java/org/apache/sqoop/mapreduce/KeyRecordWriters.java * (edit) build.xml * (edit) src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java * (edit) src/test/org/apache/sqoop/manager/mainframe/MainframeTestUtil.java * (edit) src/java/org/apache/sqoop/tool/ImportTool.java * (edit) src/java/org/apache/sqoop/SqoopOptions.java * (edit) src/java/org/apache/sqoop/tool/MainframeImportTool.java * (edit) src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java * (edit) src/test/org/apache/sqoop/manager/mainframe/MainframeManagerImportTest.java * (edit) src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java * (edit) src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java * (add) src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetBinaryImportMapper.java > Mainframe FTP transfer should have an option to use binary mode for transfer > > > Key: SQOOP-3224 > URL: https://issues.apache.org/jira/browse/SQOOP-3224 > Project: Sqoop > Issue Type: Improvement >Reporter: Chris Teoh >Assignee: Chris Teoh >Priority: Minor > Fix For: 3.0.0 > > > Currently the mainframe FTP module is hard coded to use ascii transfer mode. > Propose a mainframe module flag to be able to change modes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 68470: SQOOP-3366 Improve unit tests to be able to execute them in a single JVM
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68470/ --- (Updated Aug. 23, 2018, 3:23 p.m.) Review request for Sqoop. Bugs: SQOOP-3366 https://issues.apache.org/jira/browse/SQOOP-3366 Repository: sqoop-trunk Description --- Fix the unit tests which changed the state of the JVM so that other tests will not be affected when they are run on a single JVM. Diffs (updated) - src/test/org/apache/sqoop/TestIncrementalImport.java e1faf351f src/test/org/apache/sqoop/TestSqoopOptions.java d0591ad0d src/test/org/apache/sqoop/metastore/TestMetastoreConfigurationParameters.java 0f1eb890b src/test/org/apache/sqoop/testutil/BaseSqoopTestCase.java 1730698e8 src/test/org/apache/sqoop/testutil/HsqldbTestServer.java c63a8f2dc src/test/org/apache/sqoop/tool/TestMainframeImportTool.java 0b0c6c342 Diff: https://reviews.apache.org/r/68470/diff/2/ Changes: https://reviews.apache.org/r/68470/diff/1-2/ Testing --- No test has been added. Thanks, Nguyen Truong
[jira] [Comment Edited] (SQOOP-3224) Mainframe FTP transfer should have an option to use binary mode for transfer
[ https://issues.apache.org/jira/browse/SQOOP-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590360#comment-16590360 ] Szabolcs Vasas edited comment on SQOOP-3224 at 8/23/18 3:10 PM: Hi [~chris.t...@gmail.com], Your patch is committed, thank you for your contribution! Please close the review board request and attach the final patch file to this JIRA. Thanks and regards, Szabolcs was (Author: vasas): Hi [~chris.t...@gmail.com], Your patch is committed, thank you for your contribution! Please close the review board request. Regards, Szabolcs > Mainframe FTP transfer should have an option to use binary mode for transfer > > > Key: SQOOP-3224 > URL: https://issues.apache.org/jira/browse/SQOOP-3224 > Project: Sqoop > Issue Type: Improvement >Reporter: Chris Teoh >Assignee: Chris Teoh >Priority: Minor > Fix For: 3.0.0 > > > Currently the mainframe FTP module is hard coded to use ascii transfer mode. > Propose a mainframe module flag to be able to change modes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 62492: SQOOP-3224: Mainframe FTP transfer should have an option to use binary mode for transfer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62492/#review207814 --- Ship it! Ship It! - Szabolcs Vasas On Aug. 23, 2018, 9:51 a.m., Chris Teoh wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62492/ > --- > > (Updated Aug. 23, 2018, 9:51 a.m.) > > > Review request for Sqoop. > > > Bugs: SQOOP-3224 > https://issues.apache.org/jira/browse/SQOOP-3224 > > > Repository: sqoop-trunk > > > Description > --- > > Added --as-binaryfile and --buffersize to support FTP transfer mode switching. > > > Diffs > - > > build.xml 084823cf > src/docs/user/import-mainframe.txt abeb7cde > src/java/org/apache/sqoop/SqoopOptions.java f97dbfdf > src/java/org/apache/sqoop/mapreduce/ByteKeyOutputFormat.java PRE-CREATION > src/java/org/apache/sqoop/mapreduce/KeyRecordWriters.java PRE-CREATION > src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java fec34f21 > > src/java/org/apache/sqoop/mapreduce/mainframe/AbstractMainframeDatasetImportMapper.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java > ea54b07f > > src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetBinaryImportMapper.java > PRE-CREATION > > src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetBinaryRecord.java > PRE-CREATION > > src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java > 1f78384b > > src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java > 0b7b5b85 > src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java > 8ef30d38 > src/java/org/apache/sqoop/tool/BaseSqoopTool.java 9dcbdd59 > src/java/org/apache/sqoop/tool/ImportTool.java 478f1748 > src/java/org/apache/sqoop/tool/MainframeImportTool.java cdd9d6d0 > src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java 95bc0ecb > src/test/org/apache/sqoop/manager/mainframe/MainframeManagerImportTest.java > 041dfb78 > src/test/org/apache/sqoop/manager/mainframe/MainframeTestUtil.java f28ff36c > > src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java > PRE-CREATION > src/test/org/apache/sqoop/tool/TestMainframeImportTool.java 0b0c6c34 > > > Diff: https://reviews.apache.org/r/62492/diff/22/ > > > Testing > --- > > Unit tests. > > Functional testing on mainframe. > > > Thanks, > > Chris Teoh > >
[jira] [Commented] (SQOOP-3224) Mainframe FTP transfer should have an option to use binary mode for transfer
[ https://issues.apache.org/jira/browse/SQOOP-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590359#comment-16590359 ] ASF subversion and git services commented on SQOOP-3224: Commit 0d6c455e5bbfc092d4a90f352eb262f347758132 in sqoop's branch refs/heads/trunk from [~vasas] [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=0d6c455 ] SQOOP-3224: Mainframe FTP transfer should have an option to use binary mode for transfer (Chris Teoh via Szabolcs Vasas) > Mainframe FTP transfer should have an option to use binary mode for transfer > > > Key: SQOOP-3224 > URL: https://issues.apache.org/jira/browse/SQOOP-3224 > Project: Sqoop > Issue Type: Improvement >Reporter: Chris Teoh >Assignee: Chris Teoh >Priority: Minor > > Currently the mainframe FTP module is hard coded to use ascii transfer mode. > Propose a mainframe module flag to be able to change modes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SQOOP-3370) Investigate sqoop import + --incremental + lastmodified + --as-sequencefile use case
Boglarka Egyed created SQOOP-3370: - Summary: Investigate sqoop import + --incremental + lastmodified + --as-sequencefile use case Key: SQOOP-3370 URL: https://issues.apache.org/jira/browse/SQOOP-3370 Project: Sqoop Issue Type: Improvement Affects Versions: 1.4.7 Reporter: Boglarka Egyed I checked both the [User Guide|https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_incremental_imports] and the Apache Sqoop Cookbook and it seems that there is no documentation regarding the supported file formats in case of the sqoop import + --incremental + lastmodified use case. There is no guideline for sqoop import + --incremental + lastmodified + --as-sequencefile either. I also checked the code and couldn't find any test to cover this nor any guideline how to do incremental merge import properly with Sequence file however it should be checked, tested and well documented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SQOOP-3369) Investigate sqoop import + --incremental + lastmodified + --as-avrodatafile use case
[
https://issues.apache.org/jira/browse/SQOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boglarka Egyed updated SQOOP-3369:
--
Issue Type: Improvement (was: Bug)
> Investigate sqoop import + --incremental + lastmodified + --as-avrodatafile
> use case
>
>
> Key: SQOOP-3369
> URL: https://issues.apache.org/jira/browse/SQOOP-3369
> Project: Sqoop
> Issue Type: Improvement
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Priority: Major
>
> Currently the sqoop import + --incremental + lastmodified + --as-avrodatafile
> command fails fast with the following message:
> {noformat}
> WARN tool.BaseSqoopTool: Setting your password on the command-line is
> insecure. Consider using -P instead.
> --incremental lastmodified cannot be used in conjunction with
> --as-avrodatafile.{noformat}
> However the {{merge-tool}} supports AvroData file format too, see
> [https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
> It should be checked whether the fail fast is still valid or --incremental
> lastmodified work with AvroData file format too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3369) Investigate sqoop import + --incremental + lastmodified + --as-avrodatafile use case
[
https://issues.apache.org/jira/browse/SQOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boglarka Egyed updated SQOOP-3369:
--
Summary: Investigate sqoop import + --incremental + lastmodified +
--as-avrodatafile use case (was: Investigate sqoop import + --incremental +
lastmodified + --as-avrofile use casedata)
> Investigate sqoop import + --incremental + lastmodified + --as-avrodatafile
> use case
>
>
> Key: SQOOP-3369
> URL: https://issues.apache.org/jira/browse/SQOOP-3369
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Priority: Major
>
> Currently the sqoop import + --incremental + lastmodified + --as-avrodatafile
> command fails fast with the following message:
> {noformat}
> WARN tool.BaseSqoopTool: Setting your password on the command-line is
> insecure. Consider using -P instead.
> --incremental lastmodified cannot be used in conjunction with
> --as-avrodatafile.{noformat}
> However the {{merge-tool}} supports AvroData file format too, see
> [https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
> It should be checked whether the fail fast is still valid or --incremental
> lastmodified work with AvroData file format too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3369) Investigate sqoop import + --incremental + lastmodified + --as-avrofile use casedata
[
https://issues.apache.org/jira/browse/SQOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boglarka Egyed updated SQOOP-3369:
--
Summary: Investigate sqoop import + --incremental + lastmodified +
--as-avrofile use casedata (was: Investigate sqoop import + --incremental +
lastmodified + --as-avrofile use case)
> Investigate sqoop import + --incremental + lastmodified + --as-avrofile use
> casedata
>
>
> Key: SQOOP-3369
> URL: https://issues.apache.org/jira/browse/SQOOP-3369
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Priority: Major
>
> Currently the sqoop import + --incremental + lastmodified + --as-avrodatafile
> command fails fast with the following message:
> {noformat}
> WARN tool.BaseSqoopTool: Setting your password on the command-line is
> insecure. Consider using -P instead.
> --incremental lastmodified cannot be used in conjunction with
> --as-avrodatafile.{noformat}
> However the {{merge-tool}} supports AvroData file format too, see
> [https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
> It should be checked whether the fail fast is still valid or --incremental
> lastmodified work with AvroData file format too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3369) Investigate sqoop import + --incremental + lastmodified + --as-avrofile use case
[
https://issues.apache.org/jira/browse/SQOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boglarka Egyed updated SQOOP-3369:
--
Description:
Currently the sqoop import + --incremental + lastmodified + --as-avrodatafile
command fails fast with the following message:
{noformat}
WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
--incremental lastmodified cannot be used in conjunction with
--as-avrodatafile.{noformat}
However the {{merge-tool}} supports AvroData file format too, see
[https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
It should be checked whether the fail fast is still valid or --incremental
lastmodified work with AvroData file format too.
was:
Currently the sqoop import + -incremental lastmodified + -as-avrodatafile
command fails fast with the following message:
{noformat}
WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
--incremental lastmodified cannot be used in conjunction with
--as-avrodatafile.{noformat}
However the {{merge-tool}} supports AvroData file format too, see
[https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
It should be checked whether the fail fast is still valid or --incremental
lastmodified work with AvroData file format too.
> Investigate sqoop import + --incremental + lastmodified + --as-avrofile use
> case
>
>
> Key: SQOOP-3369
> URL: https://issues.apache.org/jira/browse/SQOOP-3369
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Priority: Major
>
> Currently the sqoop import + --incremental + lastmodified + --as-avrodatafile
> command fails fast with the following message:
> {noformat}
> WARN tool.BaseSqoopTool: Setting your password on the command-line is
> insecure. Consider using -P instead.
> --incremental lastmodified cannot be used in conjunction with
> --as-avrodatafile.{noformat}
> However the {{merge-tool}} supports AvroData file format too, see
> [https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
> It should be checked whether the fail fast is still valid or --incremental
> lastmodified work with AvroData file format too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3369) Investigate sqoop import + --incremental + lastmodified + --as-avrofile use case
[
https://issues.apache.org/jira/browse/SQOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boglarka Egyed updated SQOOP-3369:
--
Description:
Currently the sqoop import + -incremental lastmodified + -as-avrodatafile
command fails fast with the following message:
{noformat}
WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
--incremental lastmodified cannot be used in conjunction with
--as-avrodatafile.{noformat}
However the {{merge-tool}} supports AvroData file format too, see
[https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
It should be checked whether the fail fast is still valid or --incremental
lastmodified work with AvroData file format too.
was:
Currently the {{sqoop import}} + {{--incremental lastmodified}} +
{{--as-avrodatafile}} command fails fast with the following message:
{noformat}
WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
--incremental lastmodified cannot be used in conjunction with
--as-avrodatafile.{noformat}
However the {{merge-tool}} supports AvroData file format too, see
[https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
It should be checked whether the fail fast is still valid or {{--incremental
lastmodified}} work with AvroData file format too.
> Investigate sqoop import + --incremental + lastmodified + --as-avrofile use
> case
>
>
> Key: SQOOP-3369
> URL: https://issues.apache.org/jira/browse/SQOOP-3369
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Priority: Major
>
> Currently the sqoop import + -incremental lastmodified + -as-avrodatafile
> command fails fast with the following message:
> {noformat}
> WARN tool.BaseSqoopTool: Setting your password on the command-line is
> insecure. Consider using -P instead.
> --incremental lastmodified cannot be used in conjunction with
> --as-avrodatafile.{noformat}
> However the {{merge-tool}} supports AvroData file format too, see
> [https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
> It should be checked whether the fail fast is still valid or --incremental
> lastmodified work with AvroData file format too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3369) Investigate sqoop import + --incremental + lastmodified + --as-avrofile use case
[
https://issues.apache.org/jira/browse/SQOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boglarka Egyed updated SQOOP-3369:
--
Description:
Currently the {{sqoop import}} + {{--incremental {{lastmodified +
{{-as-avrodatafile}} command fails fast with the following message:
{noformat}
WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
--incremental lastmodified cannot be used in conjunction with
--as-avrodatafile.{noformat}
However the {{merge-tool}} supports AvroData file format too, see
[https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
It should be checked whether the fail fast is still valid or {{--incremental
lastmodified}} work with AvroData file format too.
was:
Currently the \{{sqoop import}} + \{{--incremental}} + {{lastmodified}} +
{{--as-avrodatafile}} command fails fast with the following message:
{noformat}
WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
--incremental lastmodified cannot be used in conjunction with
--as-avrodatafile.{noformat}
However the {{merge-tool}} supports AvroData file format too, see
[https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
It should be checked whether the fail fast is still valid or {{--incremental
lastmodified}} work with AvroData file format too.
> Investigate sqoop import + --incremental + lastmodified + --as-avrofile use
> case
>
>
> Key: SQOOP-3369
> URL: https://issues.apache.org/jira/browse/SQOOP-3369
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Priority: Major
>
> Currently the {{sqoop import}} + {{--incremental {{lastmodified +
> {{-as-avrodatafile}} command fails fast with the following message:
> {noformat}
> WARN tool.BaseSqoopTool: Setting your password on the command-line is
> insecure. Consider using -P instead.
> --incremental lastmodified cannot be used in conjunction with
> --as-avrodatafile.{noformat}
> However the {{merge-tool}} supports AvroData file format too, see
> [https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
> It should be checked whether the fail fast is still valid or {{--incremental
> lastmodified}} work with AvroData file format too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3369) Investigate sqoop import + --incremental + lastmodified + --as-avrofile use case
[
https://issues.apache.org/jira/browse/SQOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boglarka Egyed updated SQOOP-3369:
--
Description:
Currently the {{sqoop import}} + {{--incremental lastmodified}} +
{{--as-avrodatafile}} command fails fast with the following message:
{noformat}
WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
--incremental lastmodified cannot be used in conjunction with
--as-avrodatafile.{noformat}
However the {{merge-tool}} supports AvroData file format too, see
[https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
It should be checked whether the fail fast is still valid or {{--incremental
lastmodified}} work with AvroData file format too.
was:
Currently the {{sqoop import}} + {{--incremental lastmodified}} +
{{-as-avrodatafile}} command fails fast with the following message:
{noformat}
WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
--incremental lastmodified cannot be used in conjunction with
--as-avrodatafile.{noformat}
However the {{merge-tool}} supports AvroData file format too, see
[https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
It should be checked whether the fail fast is still valid or {{--incremental
lastmodified}} work with AvroData file format too.
> Investigate sqoop import + --incremental + lastmodified + --as-avrofile use
> case
>
>
> Key: SQOOP-3369
> URL: https://issues.apache.org/jira/browse/SQOOP-3369
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Priority: Major
>
> Currently the {{sqoop import}} + {{--incremental lastmodified}} +
> {{--as-avrodatafile}} command fails fast with the following message:
> {noformat}
> WARN tool.BaseSqoopTool: Setting your password on the command-line is
> insecure. Consider using -P instead.
> --incremental lastmodified cannot be used in conjunction with
> --as-avrodatafile.{noformat}
> However the {{merge-tool}} supports AvroData file format too, see
> [https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
> It should be checked whether the fail fast is still valid or {{--incremental
> lastmodified}} work with AvroData file format too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SQOOP-3369) Investigate sqoop import + --incremental + lastmodified + --as-avrofile use case
[
https://issues.apache.org/jira/browse/SQOOP-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boglarka Egyed updated SQOOP-3369:
--
Description:
Currently the {{sqoop import}} + {{--incremental lastmodified}} +
{{-as-avrodatafile}} command fails fast with the following message:
{noformat}
WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
--incremental lastmodified cannot be used in conjunction with
--as-avrodatafile.{noformat}
However the {{merge-tool}} supports AvroData file format too, see
[https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
It should be checked whether the fail fast is still valid or {{--incremental
lastmodified}} work with AvroData file format too.
was:
Currently the {{sqoop import}} + {{--incremental {{lastmodified +
{{-as-avrodatafile}} command fails fast with the following message:
{noformat}
WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
--incremental lastmodified cannot be used in conjunction with
--as-avrodatafile.{noformat}
However the {{merge-tool}} supports AvroData file format too, see
[https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
It should be checked whether the fail fast is still valid or {{--incremental
lastmodified}} work with AvroData file format too.
> Investigate sqoop import + --incremental + lastmodified + --as-avrofile use
> case
>
>
> Key: SQOOP-3369
> URL: https://issues.apache.org/jira/browse/SQOOP-3369
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Priority: Major
>
> Currently the {{sqoop import}} + {{--incremental lastmodified}} +
> {{-as-avrodatafile}} command fails fast with the following message:
> {noformat}
> WARN tool.BaseSqoopTool: Setting your password on the command-line is
> insecure. Consider using -P instead.
> --incremental lastmodified cannot be used in conjunction with
> --as-avrodatafile.{noformat}
> However the {{merge-tool}} supports AvroData file format too, see
> [https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
> It should be checked whether the fail fast is still valid or {{--incremental
> lastmodified}} work with AvroData file format too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SQOOP-3369) Investigate sqoop import + --incremental + lastmodified + --as-avrofile use case
Boglarka Egyed created SQOOP-3369:
-
Summary: Investigate sqoop import + --incremental + lastmodified +
--as-avrofile use case
Key: SQOOP-3369
URL: https://issues.apache.org/jira/browse/SQOOP-3369
Project: Sqoop
Issue Type: Bug
Affects Versions: 1.4.7
Reporter: Boglarka Egyed
Currently the \{{sqoop import}} + \{{--incremental}} + {{lastmodified}} +
{{--as-avrodatafile}} command fails fast with the following message:
{noformat}
WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
--incremental lastmodified cannot be used in conjunction with
--as-avrodatafile.{noformat}
However the {{merge-tool}} supports AvroData file format too, see
[https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html#_literal_sqoop_merge_literal.]
It should be checked whether the fail fast is still valid or {{--incremental
lastmodified}} work with AvroData file format too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 62492: SQOOP-3224: Mainframe FTP transfer should have an option to use binary mode for transfer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62492/ --- (Updated Aug. 23, 2018, 7:51 p.m.) Review request for Sqoop. Changes --- Based on feedback. Bugs: SQOOP-3224 https://issues.apache.org/jira/browse/SQOOP-3224 Repository: sqoop-trunk Description --- Added --as-binaryfile and --buffersize to support FTP transfer mode switching. Diffs (updated) - build.xml 084823cf src/docs/user/import-mainframe.txt abeb7cde src/java/org/apache/sqoop/SqoopOptions.java f97dbfdf src/java/org/apache/sqoop/mapreduce/ByteKeyOutputFormat.java PRE-CREATION src/java/org/apache/sqoop/mapreduce/KeyRecordWriters.java PRE-CREATION src/java/org/apache/sqoop/mapreduce/RawKeyTextOutputFormat.java fec34f21 src/java/org/apache/sqoop/mapreduce/mainframe/AbstractMainframeDatasetImportMapper.java PRE-CREATION src/java/org/apache/sqoop/mapreduce/mainframe/MainframeConfiguration.java ea54b07f src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetBinaryImportMapper.java PRE-CREATION src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetBinaryRecord.java PRE-CREATION src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetFTPRecordReader.java 1f78384b src/java/org/apache/sqoop/mapreduce/mainframe/MainframeDatasetImportMapper.java 0b7b5b85 src/java/org/apache/sqoop/mapreduce/mainframe/MainframeImportJob.java 8ef30d38 src/java/org/apache/sqoop/tool/BaseSqoopTool.java 9dcbdd59 src/java/org/apache/sqoop/tool/ImportTool.java 478f1748 src/java/org/apache/sqoop/tool/MainframeImportTool.java cdd9d6d0 src/java/org/apache/sqoop/util/MainframeFTPClientUtils.java 95bc0ecb src/test/org/apache/sqoop/manager/mainframe/MainframeManagerImportTest.java 041dfb78 src/test/org/apache/sqoop/manager/mainframe/MainframeTestUtil.java f28ff36c src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetBinaryRecord.java PRE-CREATION src/test/org/apache/sqoop/tool/TestMainframeImportTool.java 0b0c6c34 Diff: https://reviews.apache.org/r/62492/diff/22/ Changes: https://reviews.apache.org/r/62492/diff/21-22/ Testing --- Unit tests. Functional testing on mainframe. Thanks, Chris Teoh
