[jira] [Updated] (SQOOP-2272) Import decimal columns from mysql to hive 0.14
[ https://issues.apache.org/jira/browse/SQOOP-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Lin updated SQOOP-2272: Attachment: SQOOP-2272.patch Allowing sqoop to convert DECIMAL to DECIMAL in Hive > Import decimal columns from mysql to hive 0.14 > -- > > Key: SQOOP-2272 > URL: https://issues.apache.org/jira/browse/SQOOP-2272 > Project: Sqoop > Issue Type: Bug > Components: sqoop2-shell >Affects Versions: 1.4.5 >Reporter: Pawan Pawar >Assignee: Eric Lin > Attachments: SQOOP-2272.patch > > > I am importing data from mysql to hive. several columns in source table are > of type decimal. but sqoop convert this types into the double. How can I > import that table with same precision and scale in hive. > My query is: > sqoop import --connect > jdbc:mysql://localhost:3306/SourceDataBase?zeroDateTimeBehavior=convertToNull > --username root --password root --hive-table MyHiveDatabaseName.MyTableName > --hive-import --hive-table MyHiveDatabaseName.MyTableName --query 'select * > from MyTableName where $CONDITIONS' -m 1 --target-dir > /user/hive/warehouse/MyHiveDatabaseName/MyTableName -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Review Request 57551: SQOOP-2272 - Import decimal columns from mysql to hive 0.14
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57551/ --- Review request for Sqoop, Attila Szabo and Szabolcs Vasas. Repository: sqoop-trunk Description --- Currently Sqoop converts DECIMAL from RDMS into DOUBLE in Hive, which is not correct as user will lose precisions. Since Hive supports DECIMAL long ago, we should support DECIMAL to DECIMAL conversion from Sqoop to Hive. Diffs - src/java/org/apache/sqoop/hive/HiveTypes.java ad00535 src/java/org/apache/sqoop/hive/TableDefWriter.java 32fcca3 src/test/com/cloudera/sqoop/hive/TestTableDefWriter.java 4db629f Diff: https://reviews.apache.org/r/57551/diff/1/ Testing --- Test case + maunaul test Thanks, Eric Lin
[jira] [Commented] (SQOOP-2272) Import decimal columns from mysql to hive 0.14
[ https://issues.apache.org/jira/browse/SQOOP-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15907106#comment-15907106 ] Eric Lin commented on SQOOP-2272: - Review: https://reviews.apache.org/r/57551/ > Import decimal columns from mysql to hive 0.14 > -- > > Key: SQOOP-2272 > URL: https://issues.apache.org/jira/browse/SQOOP-2272 > Project: Sqoop > Issue Type: Bug > Components: sqoop2-shell >Affects Versions: 1.4.5 >Reporter: Pawan Pawar >Assignee: Eric Lin > Attachments: SQOOP-2272.patch > > > I am importing data from mysql to hive. several columns in source table are > of type decimal. but sqoop convert this types into the double. How can I > import that table with same precision and scale in hive. > My query is: > sqoop import --connect > jdbc:mysql://localhost:3306/SourceDataBase?zeroDateTimeBehavior=convertToNull > --username root --password root --hive-table MyHiveDatabaseName.MyTableName > --hive-import --hive-table MyHiveDatabaseName.MyTableName --query 'select * > from MyTableName where $CONDITIONS' -m 1 --target-dir > /user/hive/warehouse/MyHiveDatabaseName/MyTableName -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SQOOP-3152) --map-column-hive to support DECIMAL(xx,xx)
Eric Lin created SQOOP-3152: --- Summary: --map-column-hive to support DECIMAL(xx,xx) Key: SQOOP-3152 URL: https://issues.apache.org/jira/browse/SQOOP-3152 Project: Sqoop Issue Type: Bug Reporter: Eric Lin Assignee: Eric Lin Priority: Minor The following command: sqoop import --connect jdbc:mysql://localhost/test --username root --password 'cloudera' --table decimal_table -m 1 --driver com.mysql.jdbc.Driver --verbose --hive-import --hive-database default --hive-table decimal_table --hive-overwrite --map-column-hive a='DECIMAL(10,4)' will fail with below error: {code} 17/03/13 18:42:09 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping should be the form key=value[,key=value]* java.lang.IllegalArgumentException: Malformed mapping. Column mapping should be the form key=value[,key=value]* at org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1333) at org.apache.sqoop.SqoopOptions.setMapColumnHive(SqoopOptions.java:1349) at org.apache.sqoop.tool.BaseSqoopTool.applyHiveOptions(BaseSqoopTool.java:1198) at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011) at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435) at org.apache.sqoop.Sqoop.run(Sqoop.java:135) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) Malformed mapping. Column mapping should be the form key=value[,key=value]* {code} --map-column-hive should support DECIMAL(10,5) format. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SQOOP-3153) Sqoop export with spec file formats error message could be more verbose
Boglarka Egyed created SQOOP-3153: - Summary: Sqoop export with spec file formats error message could be more verbose Key: SQOOP-3153 URL: https://issues.apache.org/jira/browse/SQOOP-3153 Project: Sqoop Issue Type: Improvement Affects Versions: 1.4.6 Reporter: Boglarka Egyed Priority: Minor Currently the error message during an export with --as-sequencefile or --as-avrodatafile or --as-parquetfile is the following: {code} ERROR tool.BaseSqoopTool: Error parsing arguments for export: ERROR tool.BaseSqoopTool: Unrecognized argument: --as-avrodatafile {code} Based on the result of SQOOP-3151 this error message could be more verbose as it is a bit misleading now. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SQOOP-3153) Sqoop export with --as- error message could be more verbose
[ https://issues.apache.org/jira/browse/SQOOP-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boglarka Egyed updated SQOOP-3153: -- Summary: Sqoop export with --as- error message could be more verbose (was: Sqoop export with spec file formats error message could be more verbose) > Sqoop export with --as- error message could be more verbose > - > > Key: SQOOP-3153 > URL: https://issues.apache.org/jira/browse/SQOOP-3153 > Project: Sqoop > Issue Type: Improvement >Affects Versions: 1.4.6 >Reporter: Boglarka Egyed >Priority: Minor > > Currently the error message during an export with --as-sequencefile or > --as-avrodatafile or --as-parquetfile is the following: > {code} > ERROR tool.BaseSqoopTool: Error parsing arguments for export: > ERROR tool.BaseSqoopTool: Unrecognized argument: --as-avrodatafile > {code} > Based on the result of SQOOP-3151 this error message could be more verbose as > it is a bit misleading now. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (SQOOP-3152) --map-column-hive to support DECIMAL(xx,xx)
[ https://issues.apache.org/jira/browse/SQOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Lin updated SQOOP-3152: Attachment: SQOOP-3152.patch Patch added. > --map-column-hive to support DECIMAL(xx,xx) > --- > > Key: SQOOP-3152 > URL: https://issues.apache.org/jira/browse/SQOOP-3152 > Project: Sqoop > Issue Type: Bug >Affects Versions: 1.4.6 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: SQOOP-3152.patch > > > The following command: > sqoop import --connect jdbc:mysql://localhost/test --username root --password > 'cloudera' --table decimal_table -m 1 --driver com.mysql.jdbc.Driver > --verbose --hive-import --hive-database default --hive-table decimal_table > --hive-overwrite --map-column-hive a='DECIMAL(10,4)' > will fail with below error: > {code} > 17/03/13 18:42:09 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping > should be the form key=value[,key=value]* > java.lang.IllegalArgumentException: Malformed mapping. Column mapping should > be the form key=value[,key=value]* > at > org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1333) > at > org.apache.sqoop.SqoopOptions.setMapColumnHive(SqoopOptions.java:1349) > at > org.apache.sqoop.tool.BaseSqoopTool.applyHiveOptions(BaseSqoopTool.java:1198) > at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011) > at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435) > at org.apache.sqoop.Sqoop.run(Sqoop.java:135) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) > at org.apache.sqoop.Sqoop.main(Sqoop.java:252) > Malformed mapping. Column mapping should be the form key=value[,key=value]* > {code} > --map-column-hive should support DECIMAL(10,5) format. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Review Request 57576: SQOOP-3152 - --map-column-hive to support DECIMAL(xx, xx)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57576/ --- Review request for Sqoop, Attila Szabo and Szabolcs Vasas. Repository: sqoop-trunk Description --- The following command: sqoop import --connect jdbc:mysql://localhost/test --username root --password 'cloudera' --table decimal_table -m 1 --driver com.mysql.jdbc.Driver --verbose --hive-import --hive-database default --hive-table decimal_table --hive-overwrite --map-column-hive a='DECIMAL(10,4)' will fail with below error: {code} 17/03/13 18:42:09 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping should be the form key=value[,key=value]* java.lang.IllegalArgumentException: Malformed mapping. Column mapping should be the form key=value[,key=value]* at org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1333) at org.apache.sqoop.SqoopOptions.setMapColumnHive(SqoopOptions.java:1349) at org.apache.sqoop.tool.BaseSqoopTool.applyHiveOptions(BaseSqoopTool.java:1198) at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011) at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435) at org.apache.sqoop.Sqoop.run(Sqoop.java:135) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) Malformed mapping. Column mapping should be the form key=value[,key=value]* {code} --map-column-hive should support DECIMAL(10,5) format. Diffs - src/java/org/apache/sqoop/SqoopOptions.java b33b54b src/test/com/cloudera/sqoop/hive/TestTableDefWriter.java 4db629f Diff: https://reviews.apache.org/r/57576/diff/1/ Testing --- Added test case + manual testing Thanks, Eric Lin
[jira] [Commented] (SQOOP-3152) --map-column-hive to support DECIMAL(xx,xx)
[ https://issues.apache.org/jira/browse/SQOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923261#comment-15923261 ] Eric Lin commented on SQOOP-3152: - Review link: https://reviews.apache.org/r/57576/ > --map-column-hive to support DECIMAL(xx,xx) > --- > > Key: SQOOP-3152 > URL: https://issues.apache.org/jira/browse/SQOOP-3152 > Project: Sqoop > Issue Type: Bug >Affects Versions: 1.4.6 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: SQOOP-3152.patch > > > The following command: > sqoop import --connect jdbc:mysql://localhost/test --username root --password > 'cloudera' --table decimal_table -m 1 --driver com.mysql.jdbc.Driver > --verbose --hive-import --hive-database default --hive-table decimal_table > --hive-overwrite --map-column-hive a='DECIMAL(10,4)' > will fail with below error: > {code} > 17/03/13 18:42:09 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping > should be the form key=value[,key=value]* > java.lang.IllegalArgumentException: Malformed mapping. Column mapping should > be the form key=value[,key=value]* > at > org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1333) > at > org.apache.sqoop.SqoopOptions.setMapColumnHive(SqoopOptions.java:1349) > at > org.apache.sqoop.tool.BaseSqoopTool.applyHiveOptions(BaseSqoopTool.java:1198) > at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011) > at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435) > at org.apache.sqoop.Sqoop.run(Sqoop.java:135) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) > at org.apache.sqoop.Sqoop.main(Sqoop.java:252) > Malformed mapping. Column mapping should be the form key=value[,key=value]* > {code} > --map-column-hive should support DECIMAL(10,5) format. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (SQOOP-3152) --map-column-hive to support DECIMAL(xx,xx)
[ https://issues.apache.org/jira/browse/SQOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923260#comment-15923260 ] Eric Lin commented on SQOOP-3152: - After the patch, you will still see the following WARNING message: {code} 17/03/14 10:53:53 WARN hive.TableDefWriter: Column a had to be cast to a less precise type in Hive {code} This will be addressed in SQOOP-2272 > --map-column-hive to support DECIMAL(xx,xx) > --- > > Key: SQOOP-3152 > URL: https://issues.apache.org/jira/browse/SQOOP-3152 > Project: Sqoop > Issue Type: Bug >Affects Versions: 1.4.6 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: SQOOP-3152.patch > > > The following command: > sqoop import --connect jdbc:mysql://localhost/test --username root --password > 'cloudera' --table decimal_table -m 1 --driver com.mysql.jdbc.Driver > --verbose --hive-import --hive-database default --hive-table decimal_table > --hive-overwrite --map-column-hive a='DECIMAL(10,4)' > will fail with below error: > {code} > 17/03/13 18:42:09 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping > should be the form key=value[,key=value]* > java.lang.IllegalArgumentException: Malformed mapping. Column mapping should > be the form key=value[,key=value]* > at > org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1333) > at > org.apache.sqoop.SqoopOptions.setMapColumnHive(SqoopOptions.java:1349) > at > org.apache.sqoop.tool.BaseSqoopTool.applyHiveOptions(BaseSqoopTool.java:1198) > at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011) > at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435) > at org.apache.sqoop.Sqoop.run(Sqoop.java:135) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) > at org.apache.sqoop.Sqoop.main(Sqoop.java:252) > Malformed mapping. Column mapping should be the form key=value[,key=value]* > {code} > --map-column-hive should support DECIMAL(10,5) format. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Support object store
Hi Expert, Does Sqoop support object store? For example Amazon S3? Angela