[jira] [Updated] (SQOOP-3197) Database credentials cannot be configured in PostgreSQL third party tests
[ https://issues.apache.org/jira/browse/SQOOP-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3197: -- Attachment: SQOOP-3197.patch > Database credentials cannot be configured in PostgreSQL third party tests > - > > Key: SQOOP-3197 > URL: https://issues.apache.org/jira/browse/SQOOP-3197 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas > Attachments: SQOOP-3197.patch > > > PostgreSQL third party tests look like they support configurable database > parameters (e.g. they load the database user, name and password from a system > property) but the property names in the test cases do not match the property > names in the build.xml and because of that these parameters cannot be > specified from the command line. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Review Request 59844: Database credentials cannot be configured in PostgreSQL third party tests
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59844/ --- Review request for Sqoop. Bugs: SQOOP-3197 https://issues.apache.org/jira/browse/SQOOP-3197 Repository: sqoop-trunk Description --- PostgreSQL third party tests look like they support configurable database parameters (e.g. they load the database user, name and password from a system property) but the property names in the test cases do not match the property names in the build.xml and because of that these parameters cannot be specified from the command line. Diffs - build.xml af43c47 src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java 38d6ba6 src/test/com/cloudera/sqoop/manager/PostgresqlExternalTableImportTest.java 0515e09 src/test/com/cloudera/sqoop/manager/PostgresqlImportTest.java 5872d25 Diff: https://reviews.apache.org/r/59844/diff/1/ Testing --- Ran unit and third party test cases. Thanks, Szabolcs Vasas
[jira] [Created] (SQOOP-3197) Database credentials cannot be configured in PostgreSQL third party tests
Szabolcs Vasas created SQOOP-3197: - Summary: Database credentials cannot be configured in PostgreSQL third party tests Key: SQOOP-3197 URL: https://issues.apache.org/jira/browse/SQOOP-3197 Project: Sqoop Issue Type: Test Reporter: Szabolcs Vasas Assignee: Szabolcs Vasas PostgreSQL third party tests look like they support configurable database parameters (e.g. they load the database user, name and password from a system property) but the property names in the test cases do not match the property names in the build.xml and because of that these parameters cannot be specified from the command line. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SQOOP-3196) Modify MySQLAuthTest to use configurable test database parameters
[ https://issues.apache.org/jira/browse/SQOOP-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3196: -- Attachment: SQOOP-3196.patch > Modify MySQLAuthTest to use configurable test database parameters > -- > > Key: SQOOP-3196 > URL: https://issues.apache.org/jira/browse/SQOOP-3196 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas > Attachments: SQOOP-3196.patch > > > Most of the MySQL third party test cases use MySQLTestUtils to get the test > database parameters because these parameters can be configured using system > properties. > One exception is MySQLAuthTest which uses a predefined database name, > username and password which cannot be configured. > The goal of this JIRA is to change MySQLAuthTest to use MySQLTestUtils since > it would make setting up a new test database easier and make the MySQL third > party test suite more consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Review Request 59843: Modify MySQLAuthTest to use configurable test database parameters
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59843/ --- Review request for Sqoop. Bugs: SQOOP-3196 https://issues.apache.org/jira/browse/SQOOP-3196 Repository: sqoop-trunk Description --- Most of the MySQL third party test cases use MySQLTestUtils to get the test database parameters because these parameters can be configured using system properties. One exception is MySQLAuthTest which uses a predefined database name, username and password which cannot be configured. The goal of this JIRA is to change MySQLAuthTest to use MySQLTestUtils since it would make setting up a new test database easier and make the MySQL third party test suite more consistent. After changing MySQLAuthTest to use MySQLTestUtils I realized that MySQLAllTablesTest was failing because MySQLAuthTest did not drop all the tables it created so I needed to modify it to make sure all the created tables are dropped. I did not want to add more code duplication to this test class so I did some refactoring on it before addig the extra drop logic. Diffs - src/test/com/cloudera/sqoop/manager/DirectMySQLExportTest.java ec7b84a src/test/com/cloudera/sqoop/manager/MySQLAuthTest.java d5cca5d src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java 6310a39 Diff: https://reviews.apache.org/r/59843/diff/1/ Testing --- Executed unit and third party test cases. Thanks, Szabolcs Vasas
[jira] [Created] (SQOOP-3196) Modify MySQLAuthTest to use configurable test database parameters
Szabolcs Vasas created SQOOP-3196: - Summary: Modify MySQLAuthTest to use configurable test database parameters Key: SQOOP-3196 URL: https://issues.apache.org/jira/browse/SQOOP-3196 Project: Sqoop Issue Type: Test Reporter: Szabolcs Vasas Assignee: Szabolcs Vasas Most of the MySQL third party test cases use MySQLTestUtils to get the test database parameters because these parameters can be configured using system properties. One exception is MySQLAuthTest which uses a predefined database name, username and password which cannot be configured. The goal of this JIRA is to change MySQLAuthTest to use MySQLTestUtils since it would make setting up a new test database easier and make the MySQL third party test suite more consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Review Request 59833: SQLServerDatatypeImportDelimitedFileTest can fail in some environments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59833/#review177044 --- Ship it! Hi Szabi, I ran the unit and the third party tests successfully with your patch. Thanks for the improvement! Bogi - Boglarka Egyed On June 6, 2017, 10:29 a.m., Szabolcs Vasas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59833/ > --- > > (Updated June 6, 2017, 10:29 a.m.) > > > Review request for Sqoop. > > > Bugs: sqoop-3195 > https://issues.apache.org/jira/browse/sqoop-3195 > > > Repository: sqoop-trunk > > > Description > --- > > I tried to execute the SQLServer third party tests with two different SQL > Server versions. With SQL Server 2014 Express edition all the tests were > successful however with SQL Server 2017 Developer edition I got the following > error: > [junit] Test > org.apache.sqoop.manager.sqlserver.SQLServerDatatypeImportDelimitedFileTest > FAILED > Failure for following Test Data : > FLOAT > SCALE : null > PREC : null > TO_INSERT : 1.7976931348623157 > DB_READBACK : 1.7976931348623155 > HDFS_READBACK : 1.7976931348623155 > NEG_POS_FLAG : POS > OFFSET : 8 > --- > Exception details : > expected a different string expected:<1.797693134862315[5]> but > was:<1.797693134862315[7]> > By looking at the test case I have found that it inserts 1.7976931348623157 > into the database but it expects 1.7976931348623155 (the last digit is 5 > instead of 7) probably because float is an approximate numeric data types on > MSSQL and on earlier versions this is how it worked but. > I suggest using a less precise float number in this test case to avoid > flakyness. > > > Diffs > - > > testdata/DatatypeTestData-import-lite.txt a4b5c75 > > > Diff: https://reviews.apache.org/r/59833/diff/1/ > > > Testing > --- > > I ran SQL server third party tests with MSSQL 2014 and MSSQL 2017 too. > > > Thanks, > > Szabolcs Vasas > >
Re: Review Request 59831: HCatalogExportTest fails because of column escaping problems
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59831/#review177041 --- Ship it! Hi Szabi, I ran the unit and the third party tests with your patch successfully. Thanks for the fix, nice catch! Thanks, Bogi - Boglarka Egyed On June 6, 2017, 10:04 a.m., Szabolcs Vasas wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59831/ > --- > > (Updated June 6, 2017, 10:04 a.m.) > > > Review request for Sqoop. > > > Bugs: SQOOP-3194 > https://issues.apache.org/jira/browse/SQOOP-3194 > > > Repository: sqoop-trunk > > > Description > --- > > org.apache.sqoop.hcat.HCatalogTestUtils#assertSqlColValForRowId: the colname > should be always escaped, since this is the approach taken in the creation > SQL command as well in > org.apache.sqoop.hcat.HCatalogTestUtils#getSqlCreateTableStatement. > > > Diffs > - > > src/test/org/apache/sqoop/hcat/HCatalogExportTest.java 32add56 > src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java 4301a33 > > > Diff: https://reviews.apache.org/r/59831/diff/1/ > > > Testing > --- > > I ran all the Sqoop tests (unit tests and third party tests) with the patch. > > > Thanks, > > Szabolcs Vasas > >
[jira] [Commented] (SQOOP-3193) Sqoop comamnd to overwrite the table
[ https://issues.apache.org/jira/browse/SQOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038721#comment-16038721 ] Jayanthi R commented on SQOOP-3193: --- Yes, you are correct Bogi ! It's Version 1. > Sqoop comamnd to overwrite the table > > > Key: SQOOP-3193 > URL: https://issues.apache.org/jira/browse/SQOOP-3193 > Project: Sqoop > Issue Type: Bug > Components: sqoop2-shell >Affects Versions: 1.99.7 > Environment: linux >Reporter: Jayanthi R > Labels: features > Fix For: no-release > > > I am using sqoop import command and I want to overwrite the data when I use > it for second or third time (except the first time) > sqoop import --connect jdbc:postgresql://reuxeuls677.bp.com/auditdb > --username audit_user --password theaudituserloginpassword --table > test_automation --hive-overwrite --hive-table sqoophive -m 1 --hive-import > --hive-database 'test_automation_db' --create-hive-table --hive-table > test_automation; > But, I am getting the following error: > 17/06/06 09:34:13 ERROR tool.ImportTool: Encountered IOException running > import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output > directory hdfs://HDPEMDCPROD/user/dluser/test_automation already exists -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (SQOOP-3193) Sqoop comamnd to overwrite the table
[ https://issues.apache.org/jira/browse/SQOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038587#comment-16038587 ] Boglarka Egyed edited comment on SQOOP-3193 at 6/6/17 11:45 AM: Hi [~jayanthir], The version info doesn't seem to be correct as it refers to Sqoop2, however, the command you have provided is definitely a Sqoop1 as that one is the command line tool (Sqoop2 works differently). Could you please review it? Based on the Sqoop User Guide --hive-overwrite will overwrite your table using a CREATE TABLE statement so I think the --create-hive-table command in your example is unnecessary and causes the already exists error message, please see: https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_importing_data_into_hive Cheers, Bogi was (Author: boglarkaegyed): Hi [~jayanthir], The version info doesn't seem to be correct as it refers to Sqoop2, however, the command you have provided is definitely a Sqoop1 as that one is the command line tool (Sqoop2 works differently). Based on the Sqoop User Guide --hive-overwrite will overwrite your table using a CREATE TABLE statement so I think the --create-hive-table command in your example is unnecessary and causes the already exists error message, please see: https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_importing_data_into_hive Cheers, Bogi > Sqoop comamnd to overwrite the table > > > Key: SQOOP-3193 > URL: https://issues.apache.org/jira/browse/SQOOP-3193 > Project: Sqoop > Issue Type: Bug > Components: sqoop2-shell >Affects Versions: 1.99.7 > Environment: linux >Reporter: Jayanthi R > Labels: features > Fix For: no-release > > > I am using sqoop import command and I want to overwrite the data when I use > it for second or third time (except the first time) > sqoop import --connect jdbc:postgresql://reuxeuls677.bp.com/auditdb > --username audit_user --password theaudituserloginpassword --table > test_automation --hive-overwrite --hive-table sqoophive -m 1 --hive-import > --hive-database 'test_automation_db' --create-hive-table --hive-table > test_automation; > But, I am getting the following error: > 17/06/06 09:34:13 ERROR tool.ImportTool: Encountered IOException running > import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output > directory hdfs://HDPEMDCPROD/user/dluser/test_automation already exists -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SQOOP-3193) Sqoop comamnd to overwrite the table
[ https://issues.apache.org/jira/browse/SQOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038587#comment-16038587 ] Boglarka Egyed commented on SQOOP-3193: --- Hi [~jayanthir], The version info doesn't seem to be correct as it refers to Sqoop2, however, the command you have provided is definitely a Sqoop1 as that one is the command line tool (Sqoop2 works differently). Based on the Sqoop User Guide --hive-overwrite will overwrite your table using a CREATE TABLE statement so I think the --create-hive-table command in your example is unnecessary and causes the already exists error message, please see: https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_importing_data_into_hive Cheers, Bogi > Sqoop comamnd to overwrite the table > > > Key: SQOOP-3193 > URL: https://issues.apache.org/jira/browse/SQOOP-3193 > Project: Sqoop > Issue Type: Bug > Components: sqoop2-shell >Affects Versions: 1.99.7 > Environment: linux >Reporter: Jayanthi R > Labels: features > Fix For: no-release > > > I am using sqoop import command and I want to overwrite the data when I use > it for second or third time (except the first time) > sqoop import --connect jdbc:postgresql://reuxeuls677.bp.com/auditdb > --username audit_user --password theaudituserloginpassword --table > test_automation --hive-overwrite --hive-table sqoophive -m 1 --hive-import > --hive-database 'test_automation_db' --create-hive-table --hive-table > test_automation; > But, I am getting the following error: > 17/06/06 09:34:13 ERROR tool.ImportTool: Encountered IOException running > import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output > directory hdfs://HDPEMDCPROD/user/dluser/test_automation already exists -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SQOOP-3195) SQLServerDatatypeImportDelimitedFileTest can fail in some environments
[ https://issues.apache.org/jira/browse/SQOOP-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3195: -- Attachment: SQOOP-3195.patch > SQLServerDatatypeImportDelimitedFileTest can fail in some environments > -- > > Key: SQOOP-3195 > URL: https://issues.apache.org/jira/browse/SQOOP-3195 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas > Attachments: SQOOP-3195.patch > > > I tried to execute the SQLServer third party tests with two different SQL > Server versions. With SQL Server 2014 Express edition all the tests were > successful however with SQL Server 2017 Developer edition I got the following > error: > [junit] Test > org.apache.sqoop.manager.sqlserver.SQLServerDatatypeImportDelimitedFileTest > FAILED > Failure for following Test Data : > FLOAT > SCALE : null > PREC : null > TO_INSERT : 1.7976931348623157 > DB_READBACK : 1.7976931348623155 > HDFS_READBACK : 1.7976931348623155 > NEG_POS_FLAG : POS > OFFSET : 8 > --- > Exception details : >expected a different string expected:<1.797693134862315[5]> but > was:<1.797693134862315[7]> > By looking at the test case I have found that it inserts 1.7976931348623157 > into the database but it expects 1.7976931348623155 (the last digit is 5 > instead of 7) probably because float is an approximate numeric data types on > MSSQL and on earlier versions this is how it worked but. > I suggest using a less precise float number in this test case to avoid > flakyness. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Review Request 59833: SQLServerDatatypeImportDelimitedFileTest can fail in some environments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59833/ --- Review request for Sqoop. Bugs: sqoop-3195 https://issues.apache.org/jira/browse/sqoop-3195 Repository: sqoop-trunk Description --- I tried to execute the SQLServer third party tests with two different SQL Server versions. With SQL Server 2014 Express edition all the tests were successful however with SQL Server 2017 Developer edition I got the following error: [junit] Test org.apache.sqoop.manager.sqlserver.SQLServerDatatypeImportDelimitedFileTest FAILED Failure for following Test Data : FLOAT SCALE : null PREC : null TO_INSERT : 1.7976931348623157 DB_READBACK : 1.7976931348623155 HDFS_READBACK : 1.7976931348623155 NEG_POS_FLAG : POS OFFSET : 8 --- Exception details : expected a different string expected:<1.797693134862315[5]> but was:<1.797693134862315[7]> By looking at the test case I have found that it inserts 1.7976931348623157 into the database but it expects 1.7976931348623155 (the last digit is 5 instead of 7) probably because float is an approximate numeric data types on MSSQL and on earlier versions this is how it worked but. I suggest using a less precise float number in this test case to avoid flakyness. Diffs - testdata/DatatypeTestData-import-lite.txt a4b5c75 Diff: https://reviews.apache.org/r/59833/diff/1/ Testing --- I ran SQL server third party tests with MSSQL 2014 and MSSQL 2017 too. Thanks, Szabolcs Vasas
[jira] [Created] (SQOOP-3195) SQLServerDatatypeImportDelimitedFileTest can fail in some environments
Szabolcs Vasas created SQOOP-3195: - Summary: SQLServerDatatypeImportDelimitedFileTest can fail in some environments Key: SQOOP-3195 URL: https://issues.apache.org/jira/browse/SQOOP-3195 Project: Sqoop Issue Type: Test Reporter: Szabolcs Vasas Assignee: Szabolcs Vasas I tried to execute the SQLServer third party tests with two different SQL Server versions. With SQL Server 2014 Express edition all the tests were successful however with SQL Server 2017 Developer edition I got the following error: [junit] Test org.apache.sqoop.manager.sqlserver.SQLServerDatatypeImportDelimitedFileTest FAILED Failure for following Test Data : FLOAT SCALE : null PREC : null TO_INSERT : 1.7976931348623157 DB_READBACK : 1.7976931348623155 HDFS_READBACK : 1.7976931348623155 NEG_POS_FLAG : POS OFFSET : 8 --- Exception details : expected a different string expected:<1.797693134862315[5]> but was:<1.797693134862315[7]> By looking at the test case I have found that it inserts 1.7976931348623157 into the database but it expects 1.7976931348623155 (the last digit is 5 instead of 7) probably because float is an approximate numeric data types on MSSQL and on earlier versions this is how it worked but. I suggest using a less precise float number in this test case to avoid flakyness. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SQOOP-3194) HCatalogExportTest fails because of column escaping problems
[ https://issues.apache.org/jira/browse/SQOOP-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szabolcs Vasas updated SQOOP-3194: -- Attachment: SQOOP-3194.patch > HCatalogExportTest fails because of column escaping problems > > > Key: SQOOP-3194 > URL: https://issues.apache.org/jira/browse/SQOOP-3194 > Project: Sqoop > Issue Type: Test >Reporter: Szabolcs Vasas >Assignee: Szabolcs Vasas > Attachments: SQOOP-3194.patch > > > It seems there were some changes in the escaping logic in the test framework > which broke HCatalogExportTest. Since it is a third party test ant clean test > did not spot the problem earlier. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Review Request 59831: HCatalogExportTest fails because of column escaping problems
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59831/ --- Review request for Sqoop. Bugs: SQOOP-3194 https://issues.apache.org/jira/browse/SQOOP-3194 Repository: sqoop-trunk Description --- org.apache.sqoop.hcat.HCatalogTestUtils#assertSqlColValForRowId: the colname should be always escaped, since this is the approach taken in the creation SQL command as well in org.apache.sqoop.hcat.HCatalogTestUtils#getSqlCreateTableStatement. Diffs - src/test/org/apache/sqoop/hcat/HCatalogExportTest.java 32add56 src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java 4301a33 Diff: https://reviews.apache.org/r/59831/diff/1/ Testing --- I ran all the Sqoop tests (unit tests and third party tests) with the patch. Thanks, Szabolcs Vasas
[jira] [Created] (SQOOP-3194) HCatalogExportTest fails because of column escaping problems
Szabolcs Vasas created SQOOP-3194: - Summary: HCatalogExportTest fails because of column escaping problems Key: SQOOP-3194 URL: https://issues.apache.org/jira/browse/SQOOP-3194 Project: Sqoop Issue Type: Test Reporter: Szabolcs Vasas Assignee: Szabolcs Vasas It seems there were some changes in the escaping logic in the test framework which broke HCatalogExportTest. Since it is a third party test ant clean test did not spot the problem earlier. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SQOOP-3193) Sqoop comamnd to overwrite the table
Jayanthi R created SQOOP-3193: - Summary: Sqoop comamnd to overwrite the table Key: SQOOP-3193 URL: https://issues.apache.org/jira/browse/SQOOP-3193 Project: Sqoop Issue Type: Bug Components: sqoop2-shell Affects Versions: 1.99.7 Environment: linux Reporter: Jayanthi R Fix For: no-release I am using sqoop import command and I want to overwrite the data when I use it for second or third time (except the first time) sqoop import --connect jdbc:postgresql://reuxeuls677.bp.com/auditdb --username audit_user --password theaudituserloginpassword --table test_automation --hive-overwrite --hive-table sqoophive -m 1 --hive-import --hive-database 'test_automation_db' --create-hive-table --hive-table test_automation; But, I am getting the following error: 17/06/06 09:34:13 ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://HDPEMDCPROD/user/dluser/test_automation already exists -- This message was sent by Atlassian JIRA (v6.3.15#6346)
