[
https://issues.apache.org/jira/browse/SQOOP-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Senior updated SQOOP-3087:
--------------------------------
Description:
Created an Oracle table called {{SH_$.TEST}} and attempt to Sqoop import as
Avro ({{--as-avrodatafile}}) throws:
{code}
16/12/21 16:54:58 ERROR sqoop.Sqoop: Got exception running Sqoop:
org.apache.avro.SchemaParseException: Illegal character in: SH_$.TEST
org.apache.avro.SchemaParseException: Illegal character in: SH_$.TEST
at org.apache.avro.Schema.validateName(Schema.java:1142)
at org.apache.avro.Schema.access$200(Schema.java:80)
at org.apache.avro.Schema$Name.<init>(Schema.java:483)
at org.apache.avro.Schema.createRecord(Schema.java:160)
at
org.apache.sqoop.orm.AvroSchemaGenerator.generate(AvroSchemaGenerator.java:97)
at
org.apache.sqoop.mapreduce.DataDrivenImportJob.generateAvroSchema(DataDrivenImportJob.java:154)
at
org.apache.sqoop.mapreduce.DataDrivenImportJob.configureMapper(DataDrivenImportJob.java:92)
at
org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:260)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673)
at
org.apache.sqoop.manager.oracle.OraOopConnManager.importTable(OraOopConnManager.java:284)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:507)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
{code}
I believe this is because it is trying to use the value {{SH_$_TEST}} for the
{{"name"}} field in the Avro schema definition in the Avro data files. It seems
that this is not allowed as per the Avro specification on names:
https://avro.apache.org/docs/1.7.7/spec.html#Names
Command line is:
{code}
sqoop import --connect jdbc:oracle:thin:@localhost:1521/ORA11204 --username xxx
--password xxx --table SH_$.TEST --target-dir=/user/oracle/sh.db/test
--delete-target-dir -m2 --direct --fetch-size=5000 --as-avrodatafile
--outdir=/tmp
{code}
We know we can work around this by using the {{--query}} option for Sqoop, but
this means we are not able to benefit from the OraOop optimisations with direct
mode.
Can functionality (such as a schema conversion parameter) be added to Sqoop to
remove the {{$}} (and any other unsupported characters) from the {{name}} field
stored in the Avro schema in the Avro data file?
was:
Created an Oracle table called {{SH_$.TEST}} and attempt to Sqoop import as
Avro ({{--as-avrodatafile}}) throws:
{code}
16/12/21 11:56:09 ERROR sqoop.Sqoop: Got exception running Sqoop:
org.apache.avro.SchemaParseException: Illegal character in: SH_$_TEST
org.apache.avro.SchemaParseException: Illegal character in: SH_$_TEST
{code}
I believe this is because it is trying to use the value {{SH_$_TEST}} for the
{{"name"}} field in the Avro schema definition in the Avro data files. It seems
that this is not allowed as per the Avro specification on names:
https://avro.apache.org/docs/1.7.7/spec.html#Names
Command line is:
{code}
sqoop import --connect jdbc:oracle:thin:@localhost:1521/ORA11204 --username xxx
--password xxx --table SH_$.TEST --target-dir=/user/oracle/sh.db/test
--delete-target-dir -m2 --direct --fetch-size=5000 --as-avrodatafile
--outdir=/tmp
{code}
We know we can work around this by using the {{--query}} option for Sqoop, but
this means we are not able to benefit from the OraOop optimisations with direct
mode.
Can functionality (such as a schema conversion parameter) be added to Sqoop to
remove the {{$}} (and any other unsupported characters) from the {{name}} field
stored in the Avro schema in the Avro data file?
> Dollar ($) in Oracle schema name causes sqoop to avro to throw exception
> ------------------------------------------------------------------------
>
> Key: SQOOP-3087
> URL: https://issues.apache.org/jira/browse/SQOOP-3087
> Project: Sqoop
> Issue Type: Bug
> Affects Versions: 1.4.6
> Reporter: Steve Senior
>
> Created an Oracle table called {{SH_$.TEST}} and attempt to Sqoop import as
> Avro ({{--as-avrodatafile}}) throws:
> {code}
> 16/12/21 16:54:58 ERROR sqoop.Sqoop: Got exception running Sqoop:
> org.apache.avro.SchemaParseException: Illegal character in: SH_$.TEST
> org.apache.avro.SchemaParseException: Illegal character in: SH_$.TEST
> at org.apache.avro.Schema.validateName(Schema.java:1142)
> at org.apache.avro.Schema.access$200(Schema.java:80)
> at org.apache.avro.Schema$Name.<init>(Schema.java:483)
> at org.apache.avro.Schema.createRecord(Schema.java:160)
> at
> org.apache.sqoop.orm.AvroSchemaGenerator.generate(AvroSchemaGenerator.java:97)
> at
> org.apache.sqoop.mapreduce.DataDrivenImportJob.generateAvroSchema(DataDrivenImportJob.java:154)
> at
> org.apache.sqoop.mapreduce.DataDrivenImportJob.configureMapper(DataDrivenImportJob.java:92)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:260)
> at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673)
> at
> org.apache.sqoop.manager.oracle.OraOopConnManager.importTable(OraOopConnManager.java:284)
> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:507)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> {code}
> I believe this is because it is trying to use the value {{SH_$_TEST}} for the
> {{"name"}} field in the Avro schema definition in the Avro data files. It
> seems that this is not allowed as per the Avro specification on names:
> https://avro.apache.org/docs/1.7.7/spec.html#Names
> Command line is:
> {code}
> sqoop import --connect jdbc:oracle:thin:@localhost:1521/ORA11204 --username
> xxx --password xxx --table SH_$.TEST --target-dir=/user/oracle/sh.db/test
> --delete-target-dir -m2 --direct --fetch-size=5000 --as-avrodatafile
> --outdir=/tmp
> {code}
> We know we can work around this by using the {{--query}} option for Sqoop,
> but this means we are not able to benefit from the OraOop optimisations with
> direct mode.
> Can functionality (such as a schema conversion parameter) be added to Sqoop
> to remove the {{$}} (and any other unsupported characters) from the {{name}}
> field stored in the Avro schema in the Avro data file?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
