[jira] [Commented] (SQOOP-3409) Fix temporary rootdir clean up in Sqoop-S3 tests

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696133#comment-16696133
 ] 

Hudson commented on SQOOP-3409:
---

FAILURE: Integrated in Jenkins build Sqoop-hadoop200 #1233 (See 
[https://builds.apache.org/job/Sqoop-hadoop200/1233/])
SQOOP-3409: Fix temporary rootdir clean up in Sqoop-S3 tests (vasas: 
[https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=264cddbecf6d99a793600885e4c0ded72158695c])
* (edit) src/test/org/apache/sqoop/testutil/S3TestUtils.java


> Fix temporary rootdir clean up in Sqoop-S3 tests
> 
>
> Key: SQOOP-3409
> URL: https://issues.apache.org/jira/browse/SQOOP-3409
> Project: Sqoop
>  Issue Type: Task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: SQOOP-3409.patch
>
>
> Temporary root directory clean up doesn't work as expected, many generated 
> temprootdirs are being kept in the used bucket after test runs. Clean up 
> logic should be checked and fixed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69433: Setting up Travis CI using Gradle test categories

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69433/#review210809
---



Hi Szabi,

The whole change looks good to me, haven't spotted any mistakes, though still 
need to run tests.

Just some questions to clarify my understanding of the change:

I see the build.xml contains the default values for the connection strings. 
1. How do these get picked up by the docker images?
I'm guessing that I can see portforwarding in the yml that's the input for 
docker compose, this would answer it.

2. And how does gradle pick 'em up?
I think this is why you've modified the util classes throughout Sqoop. Is that 
correct?

So, what are the modifications in the build.xml needed for?


build.xml
Line 193 (original), 197 (patched)


I guess localhost could have stayed (just the port had to be added), or was 
there a problem with it?



src/test/org/apache/sqoop/manager/mysql/MySQLTestUtils.java
Line 56 (original), 56 (patched)


uppercase / lowercase typo :)

mysqlDbNAme > mysqlDbName


- Fero Szabo


On Nov. 22, 2018, 3:59 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69433/
> ---
> 
> (Updated Nov. 22, 2018, 3:59 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3289
> https://issues.apache.org/jira/browse/SQOOP-3289
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> The patch includes the following changes:
> - Changed the default DB connection parameters to Docker image defaults so 
> the test tasks can be started without specifying connection parameters
> - Connection parameter settings duplications are removed
> - Most of the JDBC drivers are downloaded from Maven repositories the only 
> exception is Oracle. Contributors have to upload ojdbc6.jar to a public drive 
> and make it available to the CI job by setting the ORACLE_DRIVER_URL in Travis
> - Introduced separate test tasks for each databases
> - An Oracle Express Edition Docker image is added to 
> sqoop-thirdpartytest-db-services.yml so Oracle tests which does not require 
> Oracle EE features can be executed much easier
> - The ports for MySQL and PostgreSQL Docker containers are changed because 
> the default ones were used in the Travis VM already.
> - Introduced OracleEe test category for tests requiring Oracle EE database. 
> These tests won't be executed on Travis. The good news is that only a few 
> tests require Oracle EE
> 
> Documentation is still coming feel free to provide a feedback!
> 
> 
> Diffs
> -
> 
>   .travis.yml PRE-CREATION 
>   build.gradle efe980d67 
>   build.xml a0e25191e 
>   gradle.properties 722bc8bb2 
>   src/scripts/thirdpartytest/docker-compose/oraclescripts/ee-healthcheck.sh 
> PRE-CREATION 
>   src/scripts/thirdpartytest/docker-compose/oraclescripts/healthcheck.sh 
> fb7800efe 
>   
> src/scripts/thirdpartytest/docker-compose/sqoop-thirdpartytest-db-services.yml
>  b4cf48863 
>   src/test/org/apache/sqoop/manager/cubrid/CubridTestUtils.java 4fd522bae 
>   
> src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManualTest.java
>  ed949b98f 
>   src/test/org/apache/sqoop/manager/db2/DB2ManagerImportManualTest.java 
> 32dfc5eb2 
>   src/test/org/apache/sqoop/manager/db2/DB2TestUtils.java PRE-CREATION 
>   src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java 
> 494c75b08 
>   src/test/org/apache/sqoop/manager/mysql/MySQLTestUtils.java be205c877 
>   src/test/org/apache/sqoop/manager/oracle/ExportTest.java a60168719 
>   src/test/org/apache/sqoop/manager/oracle/ImportTest.java 5db9fe34e 
>   src/test/org/apache/sqoop/manager/oracle/OraOopTestCase.java 1598813d8 
>   src/test/org/apache/sqoop/manager/oracle/OraOopTypesTest.java 1f67c4697 
>   src/test/org/apache/sqoop/manager/oracle/OracleConnectionFactoryTest.java 
> 34e182f4c 
>   src/test/org/apache/sqoop/manager/oracle/TimestampDataTest.java be086c5c2 
>   src/test/org/apache/sqoop/manager/oracle/util/OracleUtils.java 14b57f91a 
>   
> src/test/org/apache/sqoop/manager/postgresql/DirectPostgreSQLExportManualTest.java
>  7dd6efcf9 
>   
> src/test/org/apache/sqoop/manager/postgresql/PGBulkloadManagerManualTest.java 
> 1fe264456 
>   src/test/org/apache/sqoop/manager/postgresql/PostgresqlExportTest.java 
> eb798fa99 
>   
> src/test/org/apache/sqoop/manager/postgresql/PostgresqlExternalTableImportTest.java
>  8c3d2fd90 
>   src/test/org/apache/sqoop/manager/postgresql/PostgresqlTestUtil.java 
> e9705e5da 
>   src/test/org/apache/sqoop/manager/sqlserver/MSSQLTestUtils.java bd12c5566 
>   

[jira] [Commented] (SQOOP-3408) Introduce a Gradle build parameter to set the default forkEvery value for the tests

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696060#comment-16696060
 ] 

Hudson commented on SQOOP-3408:
---

FAILURE: Integrated in Jenkins build Sqoop-hadoop200 #1232 (See 
[https://builds.apache.org/job/Sqoop-hadoop200/1232/])
SQOOP-3408 Introduce a Gradle build parameter to set the default (fero: 
[https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=74252163f0455747ed2a99eb6c2c9d0e48f3dedc])
* (edit) COMPILING.txt
* (edit) build.gradle


> Introduce a Gradle build parameter to set the default forkEvery value for the 
> tests
> ---
>
> Key: SQOOP-3408
> URL: https://issues.apache.org/jira/browse/SQOOP-3408
> Project: Sqoop
>  Issue Type: Test
>Affects Versions: 1.4.7
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
> Attachments: SQOOP-3408.patch
>
>
> The [forkEvery 
> parameter|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery]
>  of the Gradle test tasks is currently set to 0 which means that all of the 
> tests run in a single JVM (the only exception is the kerberizedTest task 
> which requires a new JVM for every test class).
> The benefit of this setup is that the test tasks finish much faster since the 
> JVM creation is a slow operation. However the Sqoop test framework seems to 
> consume/leak too much memory which can lead to an OutOfMemoryError during the 
> build if there is not enough memory on the machine running the tests.
> The goal of this JIRA is to introduce a new parameter to the Gradle build 
> which can be used to set the default forkEvery parameter and thus prevent the 
> JVM running out of memory.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 69434: SQOOP-3410: Test S3 import with fs.s3a.security.credential.provider.path

[Boglarka Egyed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69434/
---

Review request for Sqoop, Fero Szabo and Szabolcs Vasas.


Bugs: SQOOP-3410
https://issues.apache.org/jira/browse/SQOOP-3410


Repository: sqoop-trunk


Description
---

Based on 
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Configure_the_hadoop.security.credential.provider.path_property
 property fs.s3a.security.credential.provider.path can also be used for passing 
the location of the credential store. This should be also tested and documented.


Diffs
-

  src/docs/user/s3.txt 6ff828c497e0711a2394f768ed5d61ecaf9ec273 
  src/java/org/apache/sqoop/util/password/CredentialProviderHelper.java 
4e79f0ae252969c4a426d1ff69072695eb37b7a6 
  src/test/org/apache/sqoop/credentials/TestPassingSecurePassword.java 
dca3195b8051048c5c7c2fb3bf30774e9d19eda8 
  src/test/org/apache/sqoop/s3/TestS3ImportWithHadoopCredProvider.java 
e1d7cbda2c65aa59a89715adff52b85fb3730477 


Diff: https://reviews.apache.org/r/69434/diff/1/


Testing
---

ant clean test
./gradlew -Ds3.bucket.url= 
-Ds3.generator.command= s3Test --tests 
TestS3ImportWithHadoopCredProvider

ant clean docs
./gradlew docs


Thanks,

Boglarka Egyed

[jira] [Updated] (SQOOP-3410) Test S3 import with fs.s3a.security.credential.provider.path

[Boglarka Egyed (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3410:
--
Attachment: SQOOP-3410.patch

> Test S3 import with fs.s3a.security.credential.provider.path
> 
>
> Key: SQOOP-3410
> URL: https://issues.apache.org/jira/browse/SQOOP-3410
> Project: Sqoop
>  Issue Type: Improvement
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Major
> Attachments: SQOOP-3410.patch
>
>
> Based on 
> [https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Configure_the_hadoop.security.credential.provider.path_property]
>  property fs.s3a.security.credential.provider.path can also be used for 
> passing the location of the credential store. This should be also tested and 
> documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (SQOOP-3289) Add .travis.yml

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas updated SQOOP-3289:
--
Issue Type: Sub-task  (was: Task)
Parent: SQOOP-3228

> Add .travis.yml
> ---
>
> Key: SQOOP-3289
> URL: https://issues.apache.org/jira/browse/SQOOP-3289
> Project: Sqoop
>  Issue Type: Sub-task
>  Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: SQOOP-3289.patch
>
>
> Adding a .travis.yml would enable running builds/tests on travis-ci.org. 
> Currently if you wish to use Travis for testing your changes, you have to 
> manually add a .travis.yml to your branch. Having it committed to trunk would 
> save us this extra step.
> I currently have an example 
> [{{.travis.yml}}|https://github.com/dvoros/sqoop/blob/93a4c06c1a3da1fd5305c99e379484507797b3eb/.travis.yml]
>  on my travis branch running unit tests for every commit and every pull 
> request: https://travis-ci.org/dvoros/sqoop/builds
> Later we could add the build status to the project readme as well, see: 
> https://github.com/dvoros/sqoop/tree/travis
> Also, an example of a pull request: https://github.com/dvoros/sqoop/pull/1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3289) Add .travis.yml

[Szabolcs Vasas (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696056#comment-16696056
 ] 

Szabolcs Vasas commented on SQOOP-3289:
---

Hi All,

This initiative was abandoned for a while, after the Gradle upgrade I have 
managed to create a Travis job which can execute almost all of the third party 
tests with Travis.
Please see my RB request attached: https://reviews.apache.org/r/69433/



> Add .travis.yml
> ---
>
> Key: SQOOP-3289
> URL: https://issues.apache.org/jira/browse/SQOOP-3289
> Project: Sqoop
>  Issue Type: Task
>  Components: build
>Affects Versions: 1.4.7
>Reporter: Daniel Voros
>Assignee: Daniel Voros
>Priority: Minor
> Fix For: 1.5.0
>
>
> Adding a .travis.yml would enable running builds/tests on travis-ci.org. 
> Currently if you wish to use Travis for testing your changes, you have to 
> manually add a .travis.yml to your branch. Having it committed to trunk would 
> save us this extra step.
> I currently have an example 
> [{{.travis.yml}}|https://github.com/dvoros/sqoop/blob/93a4c06c1a3da1fd5305c99e379484507797b3eb/.travis.yml]
>  on my travis branch running unit tests for every commit and every pull 
> request: https://travis-ci.org/dvoros/sqoop/builds
> Later we could add the build status to the project readme as well, see: 
> https://github.com/dvoros/sqoop/tree/travis
> Also, an example of a pull request: https://github.com/dvoros/sqoop/pull/1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 69433: Setting up Travis CI using Gradle test categories

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69433/
---

Review request for Sqoop.


Bugs: SQOOP-3289
https://issues.apache.org/jira/browse/SQOOP-3289


Repository: sqoop-trunk


Description
---

The patch includes the following changes:
- Changed the default DB connection parameters to Docker image defaults so the 
test tasks can be started without specifying connection parameters
- Connection parameter settings duplications are removed
- Most of the JDBC drivers are downloaded from Maven repositories the only 
exception is Oracle. Contributors have to upload ojdbc6.jar to a public drive 
and make it available to the CI job by setting the ORACLE_DRIVER_URL in Travis
- Introduced separate test tasks for each databases
- An Oracle Express Edition Docker image is added to 
sqoop-thirdpartytest-db-services.yml so Oracle tests which does not require 
Oracle EE features can be executed much easier
- The ports for MySQL and PostgreSQL Docker containers are changed because the 
default ones were used in the Travis VM already.
- Introduced OracleEe test category for tests requiring Oracle EE database. 
These tests won't be executed on Travis. The good news is that only a few tests 
require Oracle EE

Documentation is still coming feel free to provide a feedback!


Diffs
-

  .travis.yml PRE-CREATION 
  build.gradle efe980d67 
  build.xml a0e25191e 
  gradle.properties 722bc8bb2 
  src/scripts/thirdpartytest/docker-compose/oraclescripts/ee-healthcheck.sh 
PRE-CREATION 
  src/scripts/thirdpartytest/docker-compose/oraclescripts/healthcheck.sh 
fb7800efe 
  
src/scripts/thirdpartytest/docker-compose/sqoop-thirdpartytest-db-services.yml 
b4cf48863 
  src/test/org/apache/sqoop/manager/cubrid/CubridTestUtils.java 4fd522bae 
  
src/test/org/apache/sqoop/manager/db2/DB2ImportAllTableWithSchemaManualTest.java
 ed949b98f 
  src/test/org/apache/sqoop/manager/db2/DB2ManagerImportManualTest.java 
32dfc5eb2 
  src/test/org/apache/sqoop/manager/db2/DB2TestUtils.java PRE-CREATION 
  src/test/org/apache/sqoop/manager/db2/DB2XmlTypeImportManualTest.java 
494c75b08 
  src/test/org/apache/sqoop/manager/mysql/MySQLTestUtils.java be205c877 
  src/test/org/apache/sqoop/manager/oracle/ExportTest.java a60168719 
  src/test/org/apache/sqoop/manager/oracle/ImportTest.java 5db9fe34e 
  src/test/org/apache/sqoop/manager/oracle/OraOopTestCase.java 1598813d8 
  src/test/org/apache/sqoop/manager/oracle/OraOopTypesTest.java 1f67c4697 
  src/test/org/apache/sqoop/manager/oracle/OracleConnectionFactoryTest.java 
34e182f4c 
  src/test/org/apache/sqoop/manager/oracle/TimestampDataTest.java be086c5c2 
  src/test/org/apache/sqoop/manager/oracle/util/OracleUtils.java 14b57f91a 
  
src/test/org/apache/sqoop/manager/postgresql/DirectPostgreSQLExportManualTest.java
 7dd6efcf9 
  src/test/org/apache/sqoop/manager/postgresql/PGBulkloadManagerManualTest.java 
1fe264456 
  src/test/org/apache/sqoop/manager/postgresql/PostgresqlExportTest.java 
eb798fa99 
  
src/test/org/apache/sqoop/manager/postgresql/PostgresqlExternalTableImportTest.java
 8c3d2fd90 
  src/test/org/apache/sqoop/manager/postgresql/PostgresqlTestUtil.java 
e9705e5da 
  src/test/org/apache/sqoop/manager/sqlserver/MSSQLTestUtils.java bd12c5566 
  src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerExportTest.java 
ab1e8ff2d 
  src/test/org/apache/sqoop/manager/sqlserver/SQLServerManagerImportTest.java 
3c5bb327e 
  src/test/org/apache/sqoop/metastore/db2/DB2JobToolTest.java 81ef5fce6 
  
src/test/org/apache/sqoop/metastore/db2/DB2MetaConnectIncrementalImportTest.java
 5403908e2 
  src/test/org/apache/sqoop/metastore/db2/DB2SavedJobsTest.java b41eda110 
  src/test/org/apache/sqoop/metastore/postgres/PostgresJobToolTest.java 
59ea151a5 
  
src/test/org/apache/sqoop/metastore/postgres/PostgresMetaConnectIncrementalImportTest.java
 afc6bd232 
  src/test/org/apache/sqoop/metastore/postgres/PostgresSavedJobsTest.java 
9f9e865b9 
  src/test/org/apache/sqoop/testcategories/thirdpartytest/OracleEeTest.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/69433/diff/1/


Testing
---

The testing was done in my own Sqoop fork with Travis: 
https://travis-ci.org/szvasas/sqoop/builds/458464720


Thanks,

Szabolcs Vasas

[jira] [Created] (SQOOP-3410) Test S3 import with fs.s3a.security.credential.provider.path

[Boglarka Egyed (JIRA)]

Boglarka Egyed created SQOOP-3410:
-

 Summary: Test S3 import with 
fs.s3a.security.credential.provider.path
 Key: SQOOP-3410
 URL: https://issues.apache.org/jira/browse/SQOOP-3410
 Project: Sqoop
  Issue Type: Improvement
Reporter: Boglarka Egyed
Assignee: Boglarka Egyed


Based on 
[https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Configure_the_hadoop.security.credential.provider.path_property]
 property fs.s3a.security.credential.provider.path can also be used for passing 
the location of the credential store. This should be also tested and documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3409) Fix temporary rootdir clean up in Sqoop-S3 tests

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696026#comment-16696026
 ] 

ASF subversion and git services commented on SQOOP-3409:


Commit 264cddbecf6d99a793600885e4c0ded72158695c in sqoop's branch 
refs/heads/trunk from [~vasas]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=264cddb ]

SQOOP-3409: Fix temporary rootdir clean up in Sqoop-S3 tests

(Boglarka Egyed via Szabolcs Vasas)


> Fix temporary rootdir clean up in Sqoop-S3 tests
> 
>
> Key: SQOOP-3409
> URL: https://issues.apache.org/jira/browse/SQOOP-3409
> Project: Sqoop
>  Issue Type: Task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Major
> Attachments: SQOOP-3409.patch
>
>
> Temporary root directory clean up doesn't work as expected, many generated 
> temprootdirs are being kept in the used bucket after test runs. Clean up 
> logic should be checked and fixed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69430: SQOOP-3409: Fix temporary rootdir clean up in Sqoop-S3 tests

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69430/#review210803
---


Ship it!




Ship It!

- Fero Szabo


On Nov. 22, 2018, 1:37 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69430/
> ---
> 
> (Updated Nov. 22, 2018, 1:37 p.m.)
> 
> 
> Review request for Sqoop, Fero Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3409
> https://issues.apache.org/jira/browse/SQOOP-3409
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Temporary root directory clean up doesn't work as expected, many generated 
> temprootdirs are being kept in the used bucket after test runs. This was 
> caused as the target directory cleanup and name reset happened before the 
> temprootdir cleanup however the temprootdir name depends on the target dir 
> name in the tests.
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/testutil/S3TestUtils.java 
> 2fc606115196a7a2b6088be104e2a421888f8798 
> 
> 
> Diff: https://reviews.apache.org/r/69430/diff/1/
> 
> 
> Testing
> ---
> 
> ./gradlew s3Test -Ds3.bucket.url= 
> -Ds3.generator.command=, all the used 
> temprootdirs have been cleaned up
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

Re: Review Request 69430: SQOOP-3409: Fix temporary rootdir clean up in Sqoop-S3 tests

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69430/#review210802
---


Ship it!




Ship It!

- Szabolcs Vasas


On Nov. 22, 2018, 1:37 p.m., Boglarka Egyed wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69430/
> ---
> 
> (Updated Nov. 22, 2018, 1:37 p.m.)
> 
> 
> Review request for Sqoop, Fero Szabo and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3409
> https://issues.apache.org/jira/browse/SQOOP-3409
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Temporary root directory clean up doesn't work as expected, many generated 
> temprootdirs are being kept in the used bucket after test runs. This was 
> caused as the target directory cleanup and name reset happened before the 
> temprootdir cleanup however the temprootdir name depends on the target dir 
> name in the tests.
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/testutil/S3TestUtils.java 
> 2fc606115196a7a2b6088be104e2a421888f8798 
> 
> 
> Diff: https://reviews.apache.org/r/69430/diff/1/
> 
> 
> Testing
> ---
> 
> ./gradlew s3Test -Ds3.bucket.url= 
> -Ds3.generator.command=, all the used 
> temprootdirs have been cleaned up
> 
> 
> Thanks,
> 
> Boglarka Egyed
> 
>

[jira] [Commented] (SQOOP-3405) Refactor: break up Parameterized tests on a per database basis

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696017#comment-16696017
 ] 

Hudson commented on SQOOP-3405:
---

FAILURE: Integrated in Jenkins build Sqoop-hadoop200 #1231 (See 
[https://builds.apache.org/job/Sqoop-hadoop200/1231/])
SQOOP-3405: Refactor: break up Parameterized tests on a per database (vasas: 
[https://git-wip-us.apache.org/repos/asf?p=sqoop.git=commit=0a7407613b12a4bb25e737506ef0f091d3a7dae1])
* (delete) 
src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
* (delete) 
src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
* (delete) 
src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java
* (add) src/test/org/apache/sqoop/importjob/splitby/SplitByImportTestBase.java
* (add) 
src/test/org/apache/sqoop/importjob/configuration/SqlServerImportJobTestConfiguration.java
* (add) 
src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
* (add) src/test/org/apache/sqoop/testutil/adapter/SqlServerDatabaseAdapter.java
* (delete) src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java
* (delete) src/test/org/apache/sqoop/importjob/SplitByImportTest.java
* (add) 
src/test/org/apache/sqoop/importjob/numerictypes/NumericTypesImportTestBase.java
* (add) 
src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
* (add) 
src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
* (add) 
src/test/org/apache/sqoop/importjob/configuration/MysqlImportJobTestConfiguration.java
* (add) 
src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java
* (add) 
src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java
* (add) src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java
* (add) src/test/org/apache/sqoop/testutil/adapter/MysqlDatabaseAdapter.java
* (delete) src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java
* (add) 
src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
* (add) src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java
* (add) src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java


> Refactor: break up Parameterized tests on a per database basis
> --
>
> Key: SQOOP-3405
> URL: https://issues.apache.org/jira/browse/SQOOP-3405
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> Follow the example of the abstract class SavedJobsTestBase and it's 
> subclasses!
> We need this to be able to add test categories (so for Travis integration) as 
> well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3408) Introduce a Gradle build parameter to set the default forkEvery value for the tests

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696016#comment-16696016
 ] 

ASF subversion and git services commented on SQOOP-3408:


Commit 74252163f0455747ed2a99eb6c2c9d0e48f3dedc in sqoop's branch 
refs/heads/trunk from [~fero]
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=7425216 ]

SQOOP-3408 Introduce a Gradle build parameter to set the default forkEvery 
value for the tests

(Szabolcs Vasas via Fero Szabo)


> Introduce a Gradle build parameter to set the default forkEvery value for the 
> tests
> ---
>
> Key: SQOOP-3408
> URL: https://issues.apache.org/jira/browse/SQOOP-3408
> Project: Sqoop
>  Issue Type: Test
>Affects Versions: 1.4.7
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
> Attachments: SQOOP-3408.patch
>
>
> The [forkEvery 
> parameter|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery]
>  of the Gradle test tasks is currently set to 0 which means that all of the 
> tests run in a single JVM (the only exception is the kerberizedTest task 
> which requires a new JVM for every test class).
> The benefit of this setup is that the test tasks finish much faster since the 
> JVM creation is a slow operation. However the Sqoop test framework seems to 
> consume/leak too much memory which can lead to an OutOfMemoryError during the 
> build if there is not enough memory on the machine running the tests.
> The goal of this JIRA is to introduce a new parameter to the Gradle build 
> which can be used to set the default forkEvery parameter and thus prevent the 
> JVM running out of memory.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69429: Introduce a Gradle build parameter to set the default forkEvery value for the tests

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69429/#review210799
---


Ship it!




Ship It!

- Fero Szabo


On Nov. 22, 2018, 1:11 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69429/
> ---
> 
> (Updated Nov. 22, 2018, 1:11 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3408
> https://issues.apache.org/jira/browse/SQOOP-3408
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Documented forkEvery.default in COMPILING.txt.
> Needed to move the definition of the kerberizedTest task below 
> tasks.withType(Test) block to preserve forkEvery 1 setting.
> 
> 
> Diffs
> -
> 
>   COMPILING.txt 0383707f689102a3a543d94646cfaaf21710 
>   build.gradle 954935daeaaaf45e1b2fd83f74e11f5ed2d58377 
> 
> 
> Diff: https://reviews.apache.org/r/69429/diff/1/
> 
> 
> Testing
> ---
> 
> ./gradlew test : runs the test task with forkEvery=0
> ./gradlew -DforkEvery.default=5 test : runs the test taks with forkEvery=5
> 
> ./gradlew kerberizedTest : runs the kerberizedTest task with forkEvery=1
> ./gradlew -DforkEvery.default=5 kerberizedTest : runs the kerberizedTest task 
> with forkEvery=1, so the forkEvery.default parameter does not affect 
> kerberizedTest
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

Re: Review Request 69429: Introduce a Gradle build parameter to set the default forkEvery value for the tests

[Boglarka Egyed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69429/#review210798
---


Ship it!




Thanks Szabolcs for this improvement and the thorough documentation around it! 
Your change LGTM.

- Boglarka Egyed


On Nov. 22, 2018, 1:11 p.m., Szabolcs Vasas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69429/
> ---
> 
> (Updated Nov. 22, 2018, 1:11 p.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3408
> https://issues.apache.org/jira/browse/SQOOP-3408
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Documented forkEvery.default in COMPILING.txt.
> Needed to move the definition of the kerberizedTest task below 
> tasks.withType(Test) block to preserve forkEvery 1 setting.
> 
> 
> Diffs
> -
> 
>   COMPILING.txt 0383707f689102a3a543d94646cfaaf21710 
>   build.gradle 954935daeaaaf45e1b2fd83f74e11f5ed2d58377 
> 
> 
> Diff: https://reviews.apache.org/r/69429/diff/1/
> 
> 
> Testing
> ---
> 
> ./gradlew test : runs the test task with forkEvery=0
> ./gradlew -DforkEvery.default=5 test : runs the test taks with forkEvery=5
> 
> ./gradlew kerberizedTest : runs the kerberizedTest task with forkEvery=1
> ./gradlew -DforkEvery.default=5 kerberizedTest : runs the kerberizedTest task 
> with forkEvery=1, so the forkEvery.default parameter does not affect 
> kerberizedTest
> 
> 
> Thanks,
> 
> Szabolcs Vasas
> 
>

[jira] [Resolved] (SQOOP-3405) Refactor: break up Parameterized tests on a per database basis

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas resolved SQOOP-3405.
---
   Resolution: Fixed
Fix Version/s: 3.0.0

Hi [~fero],

Your patch is now committed, thank you for your contribution!



> Refactor: break up Parameterized tests on a per database basis
> --
>
> Key: SQOOP-3405
> URL: https://issues.apache.org/jira/browse/SQOOP-3405
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
> Fix For: 3.0.0
>
>
> Follow the example of the abstract class SavedJobsTestBase and it's 
> subclasses!
> We need this to be able to add test categories (so for Travis integration) as 
> well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SQOOP-3405) Refactor: break up Parameterized tests on a per database basis

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SQOOP-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695958#comment-16695958
 ] 

ASF subversion and git services commented on SQOOP-3405:


Commit 0a7407613b12a4bb25e737506ef0f091d3a7dae1 in sqoop's branch 
refs/heads/trunk from root
[ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=0a74076 ]

SQOOP-3405: Refactor: break up Parameterized tests on a per database basis

(Fero Szabo via Szabolcs Vasas)


> Refactor: break up Parameterized tests on a per database basis
> --
>
> Key: SQOOP-3405
> URL: https://issues.apache.org/jira/browse/SQOOP-3405
> Project: Sqoop
>  Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
>
> Follow the example of the abstract class SavedJobsTestBase and it's 
> subclasses!
> We need this to be able to add test categories (so for Travis integration) as 
> well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Boglarka Egyed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/#review210796
---


Ship it!




Ship It!

- Boglarka Egyed


On Nov. 22, 2018, 1:39 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69407/
> ---
> 
> (Updated Nov. 22, 2018, 1:39 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3405
> https://issues.apache.org/jira/browse/SQOOP-3405
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Breaking up the parameterized test classes into a per database basis. 
> Provides better readability, needed for proper test categorization (and thus, 
> for travis integration).
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cbe2 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbbd3 
>   
> src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
>  4ad7defe1 
>   
> src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
>  fbcbdebeb 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
> 22567162d 
>   src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
> ebd014688 
> 
> 
> Diff: https://reviews.apache.org/r/69407/diff/7/
> 
> 
> Testing
> ---
> 
> unit and 3rd party tests.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/#review210795
---


Ship it!




Ship It!

- Szabolcs Vasas


On Nov. 22, 2018, 1:39 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69407/
> ---
> 
> (Updated Nov. 22, 2018, 1:39 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3405
> https://issues.apache.org/jira/browse/SQOOP-3405
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Breaking up the parameterized test classes into a per database basis. 
> Provides better readability, needed for proper test categorization (and thus, 
> for travis integration).
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cbe2 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbbd3 
>   
> src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
>  4ad7defe1 
>   
> src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
>  fbcbdebeb 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
> 22567162d 
>   src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
> ebd014688 
> 
> 
> Diff: https://reviews.apache.org/r/69407/diff/7/
> 
> 
> Testing
> ---
> 
> unit and 3rd party tests.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

[jira] [Updated] (SQOOP-3409) Fix temporary rootdir clean up in Sqoop-S3 tests

[Boglarka Egyed (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boglarka Egyed updated SQOOP-3409:
--
Attachment: SQOOP-3409.patch

> Fix temporary rootdir clean up in Sqoop-S3 tests
> 
>
> Key: SQOOP-3409
> URL: https://issues.apache.org/jira/browse/SQOOP-3409
> Project: Sqoop
>  Issue Type: Task
>Affects Versions: 1.4.7
>Reporter: Boglarka Egyed
>Assignee: Boglarka Egyed
>Priority: Major
> Attachments: SQOOP-3409.patch
>
>
> Temporary root directory clean up doesn't work as expected, many generated 
> temprootdirs are being kept in the used bucket after test runs. Clean up 
> logic should be checked and fixed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Fero Szabo via Review Board]

> On Nov. 22, 2018, 8:29 a.m., Boglarka Egyed wrote:
> > src/test/org/apache/sqoop/importjob/configuration/MysqlImportJobTestConfiguration.java
> > Lines 24 (patched)
> > 
> >
> > Renamed files are shown as new files now which compromises the diff. 
> > Could you please take a look and regenerate the diff?
> 
> Boglarka Egyed wrote:
> Sorry, I wanted to say that it corrupts the diff.

Fixed with black-belt git magic, with the help of Szabi. ;)


- Fero


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/#review210785
---


On Nov. 22, 2018, 1:39 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69407/
> ---
> 
> (Updated Nov. 22, 2018, 1:39 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3405
> https://issues.apache.org/jira/browse/SQOOP-3405
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Breaking up the parameterized test classes into a per database basis. 
> Provides better readability, needed for proper test categorization (and thus, 
> for travis integration).
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cbe2 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbbd3 
>   
> src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
>  4ad7defe1 
>   
> src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
>  fbcbdebeb 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
> 22567162d 
>   src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
> ebd014688 
> 
> 
> Diff: https://reviews.apache.org/r/69407/diff/7/
> 
> 
> Testing
> ---
> 
> unit and 3rd party tests.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Fero Szabo via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/
---

(Updated Nov. 22, 2018, 1:39 p.m.)


Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.


Bugs: SQOOP-3405
https://issues.apache.org/jira/browse/SQOOP-3405


Repository: sqoop-trunk


Description
---

Breaking up the parameterized test classes into a per database basis. Provides 
better readability, needed for proper test categorization (and thus, for travis 
integration).


Diffs (updated)
-

  src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java PRE-CREATION 
  src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cbe2 
  src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbbd3 
  
src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
 4ad7defe1 
  
src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
 fbcbdebeb 
  
src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
 PRE-CREATION 
  
src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
 PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
PRE-CREATION 
  src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
22567162d 
  src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
ebd014688 


Diff: https://reviews.apache.org/r/69407/diff/7/

Changes: https://reviews.apache.org/r/69407/diff/6-7/


Testing
---

unit and 3rd party tests.


Thanks,

Fero Szabo

Review Request 69430: SQOOP-3409: Fix temporary rootdir clean up in Sqoop-S3 tests

[Boglarka Egyed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69430/
---

Review request for Sqoop, Fero Szabo and Szabolcs Vasas.


Bugs: SQOOP-3409
https://issues.apache.org/jira/browse/SQOOP-3409


Repository: sqoop-trunk


Description
---

Temporary root directory clean up doesn't work as expected, many generated 
temprootdirs are being kept in the used bucket after test runs. This was caused 
as the target directory cleanup and name reset happened before the temprootdir 
cleanup however the temprootdir name depends on the target dir name in the 
tests.


Diffs
-

  src/test/org/apache/sqoop/testutil/S3TestUtils.java 
2fc606115196a7a2b6088be104e2a421888f8798 


Diff: https://reviews.apache.org/r/69430/diff/1/


Testing
---

./gradlew s3Test -Ds3.bucket.url= 
-Ds3.generator.command=, all the used 
temprootdirs have been cleaned up


Thanks,

Boglarka Egyed

Review Request 69429: Introduce a Gradle build parameter to set the default forkEvery value for the tests

[Szabolcs Vasas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69429/
---

Review request for Sqoop.


Bugs: SQOOP-3408
https://issues.apache.org/jira/browse/SQOOP-3408


Repository: sqoop-trunk


Description
---

Documented forkEvery.default in COMPILING.txt.
Needed to move the definition of the kerberizedTest task below 
tasks.withType(Test) block to preserve forkEvery 1 setting.


Diffs
-

  COMPILING.txt 0383707f689102a3a543d94646cfaaf21710 
  build.gradle 954935daeaaaf45e1b2fd83f74e11f5ed2d58377 


Diff: https://reviews.apache.org/r/69429/diff/1/


Testing
---

./gradlew test : runs the test task with forkEvery=0
./gradlew -DforkEvery.default=5 test : runs the test taks with forkEvery=5

./gradlew kerberizedTest : runs the kerberizedTest task with forkEvery=1
./gradlew -DforkEvery.default=5 kerberizedTest : runs the kerberizedTest task 
with forkEvery=1, so the forkEvery.default parameter does not affect 
kerberizedTest


Thanks,

Szabolcs Vasas

[jira] [Created] (SQOOP-3409) Fix temporary rootdir clean up in Sqoop-S3 tests

[Boglarka Egyed (JIRA)]

Boglarka Egyed created SQOOP-3409:
-

 Summary: Fix temporary rootdir clean up in Sqoop-S3 tests
 Key: SQOOP-3409
 URL: https://issues.apache.org/jira/browse/SQOOP-3409
 Project: Sqoop
  Issue Type: Task
Affects Versions: 1.4.7
Reporter: Boglarka Egyed
Assignee: Boglarka Egyed


Temporary root directory clean up doesn't work as expected, many generated 
temprootdirs are being kept in the used bucket after test runs. Clean up logic 
should be checked and fixed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (SQOOP-3408) Introduce a Gradle build parameter to set the default forkEvery value for the tests

[Szabolcs Vasas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SQOOP-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Vasas reassigned SQOOP-3408:
-

Assignee: Szabolcs Vasas

> Introduce a Gradle build parameter to set the default forkEvery value for the 
> tests
> ---
>
> Key: SQOOP-3408
> URL: https://issues.apache.org/jira/browse/SQOOP-3408
> Project: Sqoop
>  Issue Type: Test
>Affects Versions: 1.4.7
>Reporter: Szabolcs Vasas
>Assignee: Szabolcs Vasas
>Priority: Major
>
> The [forkEvery 
> parameter|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery]
>  of the Gradle test tasks is currently set to 0 which means that all of the 
> tests run in a single JVM (the only exception is the kerberizedTest task 
> which requires a new JVM for every test class).
> The benefit of this setup is that the test tasks finish much faster since the 
> JVM creation is a slow operation. However the Sqoop test framework seems to 
> consume/leak too much memory which can lead to an OutOfMemoryError during the 
> build if there is not enough memory on the machine running the tests.
> The goal of this JIRA is to introduce a new parameter to the Gradle build 
> which can be used to set the default forkEvery parameter and thus prevent the 
> JVM running out of memory.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (SQOOP-3408) Introduce a Gradle build parameter to set the default forkEvery value for the tests

[Szabolcs Vasas (JIRA)]

Szabolcs Vasas created SQOOP-3408:
-

 Summary: Introduce a Gradle build parameter to set the default 
forkEvery value for the tests
 Key: SQOOP-3408
 URL: https://issues.apache.org/jira/browse/SQOOP-3408
 Project: Sqoop
  Issue Type: Test
Affects Versions: 1.4.7
Reporter: Szabolcs Vasas


The [forkEvery 
parameter|https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery]
 of the Gradle test tasks is currently set to 0 which means that all of the 
tests run in a single JVM (the only exception is the kerberizedTest task which 
requires a new JVM for every test class).
The benefit of this setup is that the test tasks finish much faster since the 
JVM creation is a slow operation. However the Sqoop test framework seems to 
consume/leak too much memory which can lead to an OutOfMemoryError during the 
build if there is not enough memory on the machine running the tests.
The goal of this JIRA is to introduce a new parameter to the Gradle build which 
can be used to set the default forkEvery parameter and thus prevent the JVM 
running out of memory.
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Boglarka Egyed]

> On Nov. 22, 2018, 8:29 a.m., Boglarka Egyed wrote:
> > src/test/org/apache/sqoop/importjob/configuration/MysqlImportJobTestConfiguration.java
> > Lines 24 (patched)
> > 
> >
> > Renamed files are shown as new files now which compromises the diff. 
> > Could you please take a look and regenerate the diff?

Sorry, I wanted to say that it corrupts the diff.


- Boglarka


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/#review210785
---


On Nov. 21, 2018, 3 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69407/
> ---
> 
> (Updated Nov. 21, 2018, 3 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3405
> https://issues.apache.org/jira/browse/SQOOP-3405
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Breaking up the parameterized test classes into a per database basis. 
> Provides better readability, needed for proper test categorization (and thus, 
> for travis integration).
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cb 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbb 
>   
> src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
>  4ad7def 
>   
> src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
>  fbcbdeb 
>   
> src/test/org/apache/sqoop/importjob/configuration/MysqlImportJobTestConfiguration.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/configuration/SqlServerImportJobTestConfiguration.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/NumericTypesImportTestBase.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/SplitByImportTestBase.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
> 2256716 
>   src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
> ebd0146 
>   src/test/org/apache/sqoop/testutil/adapter/MysqlDatabaseAdapter.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/adapter/SqlServerDatabaseAdapter.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69407/diff/6/
> 
> 
> Testing
> ---
> 
> unit and 3rd party tests.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>

Re: Review Request 69407: Refactor: break up Parameterized tests on a per database basis

[Boglarka Egyed]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69407/#review210785
---




src/test/org/apache/sqoop/importjob/configuration/MysqlImportJobTestConfiguration.java
Lines 24 (patched)


Renamed files are shown as new files now which compromises the diff. Could 
you please take a look and regenerate the diff?


- Boglarka Egyed


On Nov. 21, 2018, 3 p.m., Fero Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69407/
> ---
> 
> (Updated Nov. 21, 2018, 3 p.m.)
> 
> 
> Review request for Sqoop, Boglarka Egyed and Szabolcs Vasas.
> 
> 
> Bugs: SQOOP-3405
> https://issues.apache.org/jira/browse/SQOOP-3405
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> ---
> 
> Breaking up the parameterized test classes into a per database basis. 
> Provides better readability, needed for proper test categorization (and thus, 
> for travis integration).
> 
> 
> Diffs
> -
> 
>   src/test/org/apache/sqoop/importjob/DatabaseAdapterFactory.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/NumericTypesImportTest.java af310cb 
>   src/test/org/apache/sqoop/importjob/SplitByImportTest.java 90b7cbb 
>   
> src/test/org/apache/sqoop/importjob/configuration/MSSQLServerImportJobTestConfiguration.java
>  4ad7def 
>   
> src/test/org/apache/sqoop/importjob/configuration/MySQLImportJobTestConfiguration.java
>  fbcbdeb 
>   
> src/test/org/apache/sqoop/importjob/configuration/MysqlImportJobTestConfiguration.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/configuration/SqlServerImportJobTestConfiguration.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/MysqlNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/NumericTypesImportTestBase.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/OracleNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/PostgresNumericTypesImportTest.java
>  PRE-CREATION 
>   
> src/test/org/apache/sqoop/importjob/numerictypes/SqlServerNumericTypesImportTest.java
>  PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/MysqlSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/OracleSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/PostgresSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/SplitByImportTestBase.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/importjob/splitby/SqlServerSplitByImportTest.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/adapter/MSSQLServerDatabaseAdapter.java 
> 2256716 
>   src/test/org/apache/sqoop/testutil/adapter/MySqlDatabaseAdapter.java 
> ebd0146 
>   src/test/org/apache/sqoop/testutil/adapter/MysqlDatabaseAdapter.java 
> PRE-CREATION 
>   src/test/org/apache/sqoop/testutil/adapter/SqlServerDatabaseAdapter.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69407/diff/6/
> 
> 
> Testing
> ---
> 
> unit and 3rd party tests.
> 
> 
> Thanks,
> 
> Fero Szabo
> 
>