Hello Mark, I will try to test this but, am curious if you tried:
sqoop import --connect jdbc:mysql://DATABASE_ENDPOINT --query "select * from ci_84adea33-9194-4753-925f-529a87656048 where \$CONDITIONS" --as-parquetfile --class-name mydata1 --username USERNAME -P --map-column-java <CHAR_COL>=String Markus Kemper Customer Operations Engineer [image: www.cloudera.com] <http://www.cloudera.com> On Thu, Aug 4, 2016 at 3:48 PM, Mark Wagoner <[email protected]> wrote: > Hi Markus, > > Thanks for the feedback. > > I was able to use a slight variation of your suggestion to get past the > table name issue, but then encountered a new error related apparently to > the class of the column type I am importing (varchar), which is still > unresolved as of today (https://issues.apache.org/jira/browse/SQOOP-2408) > so I'll likely be importing in another format and then converting until > this bug is fixed. > > [hadoop@ip-172-31-25-10 ~]$ sqoop import --connect > jdbc:mysql://DATABASE_ENDPOINT --query 'select * from > `ci_84adea33-9194-4753-925f-529a87656048` t1 where $CONDITIONS' > --class-name mydata0 --as-parquetfile --target-dir /home/hadoop/mydata0 > --split-by t1.rsid --username USERNAME -P > Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will > fail. > Please set $ACCUMULO_HOME to the root of your Accumulo installation. > 16/08/04 19:14:00 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 > Enter password: > 16/08/04 19:14:08 INFO manager.MySQLManager: Preparing to use a MySQL > streaming resultset. > 16/08/04 19:14:08 INFO tool.CodeGenTool: Beginning code generation > 16/08/04 19:14:08 INFO manager.SqlManager: Executing SQL statement: select > * from `ci_84adea33-9194-4753-925f-529a87656048` t1 where (1 = 0) > 16/08/04 19:14:08 INFO manager.SqlManager: Executing SQL statement: select > * from `ci_84adea33-9194-4753-925f-529a87656048` t1 where (1 = 0) > 16/08/04 19:14:08 INFO manager.SqlManager: Executing SQL statement: select > * from `ci_84adea33-9194-4753-925f-529a87656048` t1 where (1 = 0) > 16/08/04 19:14:08 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is > /usr/lib/hadoop-mapreduce > Note: /tmp/sqoop-hadoop/compile/d06024b7489332878d1144702c6a7923/mydata0.java > uses or overrides a deprecated API. > Note: Recompile with -Xlint:deprecation for details. > 16/08/04 19:14:12 INFO orm.CompilationManager: Writing jar file: > /tmp/sqoop-hadoop/compile/d06024b7489332878d1144702c6a7923/mydata0.jar > 16/08/04 19:14:12 INFO mapreduce.ImportJobBase: Beginning query import. > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/ > slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/ > log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 16/08/04 19:14:13 INFO Configuration.deprecation: mapred.jar is > deprecated. Instead, use mapreduce.job.jar > 16/08/04 19:14:13 INFO manager.SqlManager: Executing SQL statement: select > * from `ci_84adea33-9194-4753-925f-529a87656048` t1 where (1 = 0) > 16/08/04 19:14:13 INFO manager.SqlManager: Executing SQL statement: select > * from `ci_84adea33-9194-4753-925f-529a87656048` t1 where (1 = 0) > 16/08/04 19:14:15 INFO Configuration.deprecation: mapred.map.tasks is > deprecated. Instead, use mapreduce.job.maps > 16/08/04 19:14:16 INFO impl.TimelineClientImpl: Timeline service address: > http://ip-172-31-25-105.us-west-2.compute.internal:8188/ws/v1/timeline/ > 16/08/04 19:14:16 INFO client.RMProxy: Connecting to ResourceManager at > ip-172-31-25-105.us-west-2.compute.internal/172.31.25.105:8032 > 16/08/04 19:14:17 INFO db.DBInputFormat: Using read commited transaction > isolation > 16/08/04 19:14:17 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: > SELECT MIN(t1.rsid), MAX(t1.rsid) FROM (select * from > `ci_84adea33-9194-4753-925f-529a87656048` t1 where (1 = 1) ) AS t1 > 16/08/04 19:14:18 WARN db.TextSplitter: Generating splits for a textual > index column. > 16/08/04 19:14:18 WARN db.TextSplitter: If your database sorts in a > case-insensitive order, this may result in a partial import or duplicate > records. > 16/08/04 19:14:18 WARN db.TextSplitter: You are strongly encouraged to > choose an integral split column. > 16/08/04 19:14:18 INFO mapreduce.JobSubmitter: number of splits:5 > 16/08/04 19:14:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: > job_1470337090744_0001 > 16/08/04 19:14:19 INFO impl.YarnClientImpl: Submitted application > application_1470337090744_0001 > 16/08/04 19:14:19 INFO mapreduce.Job: The url to track the job: > http://ip-172-31-25-105.us-west-2.compute.internal:20888/ > proxy/application_1470337090744_0001/ > 16/08/04 19:14:19 INFO mapreduce.Job: Running job: job_1470337090744_0001 > 16/08/04 19:14:30 INFO mapreduce.Job: Job job_1470337090744_0001 running > in uber mode : false > 16/08/04 19:14:30 INFO mapreduce.Job: map 0% reduce 0% > 16/08/04 19:14:43 INFO mapreduce.Job: map 20% reduce 0% > 16/08/04 19:14:43 INFO mapreduce.Job: Task Id : > attempt_1470337090744_0001_m_000000_0, Status : FAILED > Error: org.apache.avro.AvroRuntimeException: Not a Specific class: char > at org.apache.avro.specific.SpecificData.createSchema( > SpecificData.java:213) > at org.apache.avro.reflect.ReflectData.createSchema( > ReflectData.java:303) > at org.apache.avro.reflect.ReflectData.createFieldSchema( > ReflectData.java:430) > at org.kitesdk.data.spi.DataModelUtil$AllowNulls.createFieldSchema( > DataModelUtil.java:55) > at org.apache.avro.reflect.ReflectData.createSchema( > ReflectData.java:354) > at org.apache.avro.reflect.ReflectData.createFieldSchema( > ReflectData.java:430) > at org.kitesdk.data.spi.DataModelUtil$AllowNulls.createFieldSchema( > DataModelUtil.java:55) > at org.apache.avro.reflect.ReflectData.createSchema( > ReflectData.java:354) > at org.apache.avro.reflect.ReflectData.createFieldSchema( > ReflectData.java:430) > at org.kitesdk.data.spi.DataModelUtil$AllowNulls.createFieldSchema( > DataModelUtil.java:55) > at org.apache.avro.reflect.ReflectData.createSchema( > ReflectData.java:354) > at org.apache.avro.specific.SpecificData.getSchema( > SpecificData.java:154) > at org.kitesdk.data.spi.DataModelUtil.getReaderSchema( > DataModelUtil.java:171) > at org.kitesdk.data.spi.DataModelUtil.resolveType( > DataModelUtil.java:148) > at org.kitesdk.data.spi.AbstractDataset.<init>( > AbstractDataset.java:44) > at org.kitesdk.data.spi.filesystem.FileSystemDataset.< > init>(FileSystemDataset.java:85) > at org.kitesdk.data.spi.filesystem.FileSystemDataset.< > init>(FileSystemDataset.java:115) > at org.kitesdk.data.spi.filesystem.FileSystemDataset$Builder.build( > FileSystemDataset.java:541) > at org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.load( > FileSystemDatasetRepository.java:194) > at org.kitesdk.data.spi.AbstractDatasetRepository.load( > AbstractDatasetRepository.java:40) > at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.loadJobDataset( > DatasetKeyOutputFormat.java:544) > at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat. > loadOrCreateTaskAttemptDataset(DatasetKeyOutputFormat.java:555) > at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat. > loadOrCreateTaskAttemptView(DatasetKeyOutputFormat.java:568) > at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.getRecordWriter( > DatasetKeyOutputFormat.java:426) > at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.< > init>(MapTask.java:656) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:776) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > > Best, > Mark > > On Wed, Aug 3, 2016 at 4:10 PM, Markus Kemper <[email protected]> wrote: > >> Hello Mark, >> >> Have you tried the following: >> >> sqoop import --connect jdbc:mysql://DATABASE_ENDPOINT --query "select * >> from ci_84adea33-9194-4753-925f-529a87656048 where \$CONDITIONS" >> --as-parquetfile --class-name >> mydata1 --username USERNAME -P >> >> >> Markus Kemper >> Customer Operations Engineer >> [image: www.cloudera.com] <http://www.cloudera.com> >> >> >> On Tue, Aug 2, 2016 at 4:59 PM, Mark Wagoner <[email protected]> >> wrote: >> >>> hi, >>> >>> I need to be able to import table names with hypens but keep getting the >>> following error. Is there any way to specify a table alias to rename a >>> table replacing hypens with underscores? >>> >>> sqoop import --connect jdbc:mysql://DATABASE_ENDPOINT --table >>> ci_84adea33-9194-4753-925f-529a87656048 --as-parquetfile --class-name >>> mydata1 --username USERNAME -P >>> >>> >>> 16/08/02 20:11:22 INFO manager.SqlManager: Executing SQL statement: >>> SELECT t.* FROM `ci_84adea33-9194-4753-925f-529a87656048` AS t LIMIT 1 >>> 16/08/02 20:11:23 ERROR sqoop.Sqoop: Got exception running Sqoop: >>> org.apache.avro.SchemaParseException: Illegal character in: >>> ci_84adea33-9194-4753-925f-529a87656048 >>> org.apache.avro.SchemaParseException: Illegal character in: >>> ci_84adea33-9194-4753-925f-529a87656048 >>> at org.apache.avro.Schema.validateName(Schema.java:1042) >>> at org.apache.avro.Schema.access$200(Schema.java:78) >>> at org.apache.avro.Schema$Name.<init>(Schema.java:431) >>> at org.apache.avro.Schema.createRecord(Schema.java:144) >>> at >>> org.apache.sqoop.orm.AvroSchemaGenerator.generate(AvroSchema >>> Generator.java:83) >>> at >>> org.apache.sqoop.mapreduce.DataDrivenImportJob.generateAvroS >>> chema(DataDrivenImportJob.java:133) >>> at >>> org.apache.sqoop.mapreduce.DataDrivenImportJob.configureMapp >>> er(DataDrivenImportJob.java:106) >>> at >>> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJob >>> Base.java:260) >>> at org.apache.sqoop.manager.SqlManager.importTable(SqlManager. >>> java:673) >>> at >>> org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118) >>> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497) >>> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) >>> at org.apache.sqoop.Sqoop.run(Sqoop.java:143) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) >>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) >>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) >>> at org.apache.sqoop.Sqoop.main(Sqoop.java:236) >>> >>> Thanks, >>> Mark >>> >> >> >
