Hi Andy, I'm glad to hear that Sqoop has started working for you! There is great article on Cloudera blog [1] that describes what sort of logs is Hadoop generating and how to get them. The task logs are describe there as well.
Jarcec Links: 1: http://blog.cloudera.com/blog/2009/09/apache-hadoop-log-files-where-to-find-them-in-cdh-and-what-info-they-contain/ On Thu, May 23, 2013 at 06:08:47AM -0700, Han Sen Tey wrote: > Good morning Jarcec, > > Thank you very much for your insight. I would have never attempted to > remove all single quote from my sqoop command. Upon removing all quotes from > my sqoop command, my Oozie workflow that comprised of a pig-, hive- and > sqoop- actions flow like a river. > > export --verbose --connect > jdbc:mysql://ec2-23-22-223-47.compute-1.amazonaws.com:3306/test --table > CHRYSLER_Anametrix_Aggregated --update-key > VisitorID,VisitorDay,VisitorMonth,VisitorYear,VisitorSession --update-mode > allowinsert --export-dir /user/hive/warehouse/chrysler_anametrix_aggregated > --username root -m 1 --input-fields-terminated-by | --input-null-string \\N > --input-null-non-string \\N > > > One question to help with my future debugging. Where or how can I find the > map task log in CDH4 ? > > Thanks, > Andy > > > > > > ________________________________ > From: Jarek Jarcec Cecho <[email protected]> > To: [email protected]; Han Sen Tey <[email protected]> > Sent: Thursday, May 23, 2013 4:45:55 AM > Subject: Re: SQOOP export from Hadoop Hive to MySQL > > > Hi Han, > would you mind sharing entire Sqoop log generated with parameter --verbose as > an attachment? It might be also beneficial to share the map task log. > > Oozie is not using bash nor any other shell to execute Sqoop, so you should > remove any escaping introduced especially for shell. Looking at your example, > I think that it should be something like: > > ... --innput-fields-terminated-by | --input-null-string \\N ... > > Jarcec > > On Tue, May 21, 2013 at 05:56:16AM -0700, Han Sen Tey wrote: > > Greeting experts, > > > > I have a table "CHRYSLER_Anametrix_Aggregated" in Hive and in MySQL. > > Similarly I can find this table in hdfs -- > > /user/hive/warehouse/chrysler_anametrix_aggregated. > > > > > > I can successfully execute the following SQOOP command from namenode CLI > > - > > > > sqoop export --verbose --connect > > jdbc:mysql://ec2-23-22-223-47.compute-1.amazonaws.com:3306/test --table > > CHRYSLER_Anametrix_Aggregated --update-key > > VisitorID,VisitorDay,VisitorMonth,VisitorYear,VisitorSession > > --update-mode allowinsert --export-dir > > /user/hive/warehouse/chrysler_anametrix_aggregated --username root -m 1 > > --input-fields-terminated-by '|' --input-null-string '\\N' > > --input-null-non-string '\\N' > > > > However, when I want to run the above as part of Oozie workflow action > > ... > > > > > > <action name="sqoop-node"> > > <sqoop xmlns="uri:oozie:sqoop-action:0.2"> > > <job-tracker>${jobTracker}</job-tracker> > > <name-node>${nameNode}</name-node> > > > > <configuration> > > <property> > > <name>sqoop.connection.factories</name> > > > > <value>com.cloudera.sqoop.manager.DefaultManagerFactory</value> > > </property> > > <property> > > > > <name>mapred.job.queue.name</name> > > <value>${queueName}</value> > > </property> > > <property> > > <name>mapred.compress.map.output</name> > > <value>true</value> > > </property> > > > > <property> > > > > <name>oozie.service.WorkflowAppService.system.libpath</name> > > <value>${nameNode}/user/oozie/share/lib/sqoop</value> > > </property> > > </configuration> > > > > <command>export --verbose --connect > > jdbc:mysql://ec2-23-22-223-47.compute-1.amazonaws.com:3306/test --table > > CHRYSLER_Anametrix_Aggregated --update-key > > VisitorID,VisitorDay,VisitorMonth,VisitorYear,VisitorSession > > --update-mode allowinsert --export-dir > > /user/hive/warehouse/chrysler_anametrix_aggregated --username root -m 1 > > --input-fields-terminated-by '|' --input-null-string '\\N' > > --input-null-non-string '\\N'</command> > > </sqoop> > > <ok to="end"/> > > <error to="fail"/> > > </action> > > > > I get the following error message. I have > > "mysql-connector-java.5.1.24-bin.jar" shared library - > > /user/oozie/share/lib/sqoop. > > > > Hadoop version - 2.00 - CDH4.2.0 > > Sqoop version - 1.4.2. - CDH4.2.0 > > > > > > Can you experts please guide me ... a rookie to Hadoop and its projects. > > > > 2013-05-21 00:48:35,135 WARN mapreduce.Counters: Group > > org.apache.hadoop.mapred.Task$Counter is deprecated. Use > > org.apache.hadoop.mapreduce.TaskCounter instead 2013-05-21 00:48:37,394 > > WARN org.apache.hadoop.conf.Configuration: session.id is deprecated. > > Instead, use dfs.metrics.session-id 2013-05-21 00:48:37,408 INFO > > org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with > > processName=MAP, sessionId= 2013-05-21 00:48:38,042 INFO > > org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0 > > 2013-05-21 00:48:38,058 INFO org.apache.hadoop.mapred.Task: Using > > ResourceCalculatorPlugin : > > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5dce1bea 2013-05-21 > > 00:48:38,423 INFO org.apache.hadoop.mapred.MapTask: Processing split: > > hdfs://ip-10-147-191-48.ec2.internal:8020/user/ubuntu/oozie-oozi/0000042-130520131050057-oozie-oozi-W/sqoop-node--sqoop/input/dummy.txt:0+5 > > 2013-05-21 00:48:38,453 WARN mapreduce.Counters: Counter name > MAP_INPUT_BYTES > > is deprecated. Use FileInputFormatCounters as group name and BYTES_READ > >as counter name instead 2013-05-21 00:48:38,459 INFO > >org.apache.hadoop.mapred.MapTask: numReduceTasks: 0 2013-05-21 00:48:39,743 > >WARN org.apache.sqoop.tool.SqoopTool: $SQOOP_CONF_DIR has not been set in > >the environment. Cannot check for additional configuration. 2013-05-21 > >00:48:40,032 DEBUG org.apache.sqoop.tool.BaseSqoopTool: Enabled debug > >logging. 2013-05-21 00:48:40,036 WARN org.apache.sqoop.SqoopOptions: > >Character argument '|' has multiple characters; only the first will be used. > >2013-05-21 00:48:40,084 DEBUG org.apache.sqoop.ConnFactory: > >sqoop.connection.factories is set; ignoring managers.d 2013-05-21 > >00:48:40,139 DEBUG org.apache.sqoop.ConnFactory: Loaded manager factory: > >com.cloudera.sqoop.manager.DefaultManagerFactory 2013-05-21 00:48:40,139 > >DEBUG org.apache.sqoop.ConnFactory: Trying ManagerFactory: > >com.cloudera.sqoop.manager.DefaultManagerFactory 2013-05-21 > 00:48:40,140 > > DEBUG org.apache.sqoop.manager.DefaultManagerFactory: Trying with scheme: > >jdbc:mysql: 2013-05-21 00:48:40,183 INFO > >org.apache.sqoop.manager.MySQLManager: Preparing to use a MySQL streaming > >resultset. 2013-05-21 00:48:40,183 DEBUG org.apache.sqoop.ConnFactory: > >Instantiated ConnManager org.apache.sqoop.manager.MySQLManager@795e0c2b > >2013-05-21 00:48:40,184 INFO org.apache.sqoop.tool.CodeGenTool: Beginning > >code generation 2013-05-21 00:48:40,233 DEBUG > >org.apache.sqoop.manager.SqlManager: No connection paramenters specified. > >Using regular API for making connection. 2013-05-21 00:48:40,911 DEBUG > >org.apache.sqoop.manager.SqlManager: Using fetchSize for next query: > >-2147483648 2013-05-21 00:48:40,911 INFO > >org.apache.sqoop.manager.SqlManager: Executing SQL statement: SELECT t.* > >FROM `CHRYSLER_Anametrix_Aggregated` AS t LIMIT 1 2013-05-21 00:48:40,930 > >DEBUG org.apache.sqoop.orm.ClassWriter: selected columns: 2013-05-21 > >00:48:40,931 DEBUG > > org.apache.sqoop.orm.ClassWriter: VisitorID 2013-05-21 00:48:40,931 > >DEBUG org.apache.sqoop.orm.ClassWriter: VisitorDay 2013-05-21 00:48:40,931 > >DEBUG org.apache.sqoop.orm.ClassWriter: VisitorMonth 2013-05-21 > >00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter: VisitorYear > >2013-05-21 00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter: > >VisitorSession 2013-05-21 00:48:40,931 DEBUG > >org.apache.sqoop.orm.ClassWriter: Brand 2013-05-21 00:48:40,931 DEBUG > >org.apache.sqoop.orm.ClassWriter: DeviceType 2013-05-21 00:48:40,931 DEBUG > >org.apache.sqoop.orm.ClassWriter: DeviceBrowser 2013-05-21 00:48:40,931 > >DEBUG org.apache.sqoop.orm.ClassWriter: TrafficSource 2013-05-21 > >00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter: SiteID 2013-05-21 > >00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter: BuyID 2013-05-21 > >00:48:40,931 DEBUG org.apache.sqoop.orm.ClassWriter: AdvertisementID > >2013-05-21 00:48:40,931 DEBUG > org.apache.sqoop.orm.ClassWriter: > > PlacementID 2013-05-21 00:48:40,934 DEBUG > >org.apache.sqoop.orm.ClassWriter: CreativeID 2013-05-21 00:48:40,934 DEBUG > >org.apache.sqoop.orm.ClassWriter: PaidKeyword 2013-05-21 00:48:40,934 > >DEBUG org.apache.sqoop.orm.ClassWriter: Model200 2013-05-21 00:48:40,934 > >DEBUG org.apache.sqoop.orm.ClassWriter: Model200Conv 2013-05-21 > >00:48:40,934 DEBUG org.apache.sqoop.orm.ClassWriter: Model300 2013-05-21 > >00:48:40,934 DEBUG org.apache.sqoop.orm.ClassWriter: Model300TownCountry > >2013-05-21 00:48:40,934 DEBUG org.apache.sqoop.orm.ClassWriter: BMOLineup > >2013-05-21 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: BMOModels > >2013-05-21 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: BMOColors > >2013-05-21 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: BMOOptions > >2013-05-21 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: BMOSummary > >2013-05-21 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: > >BrochureLineup > 2013-05-21 > > 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: BrochureForm > >2013-05-21 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: > >BrochureMailLink 2013-05-21 00:48:40,935 DEBUG > >org.apache.sqoop.orm.ClassWriter: BrochureDownloadLink 2013-05-21 > >00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: PreDL 2013-05-21 > >00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: DealerLocate > >2013-05-21 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: DLHours > >2013-05-21 00:48:40,935 DEBUG org.apache.sqoop.orm.ClassWriter: DLServices > >2013-05-21 00:48:40,936 DEBUG org.apache.sqoop.orm.ClassWriter: > >DLDealerSite 2013-05-21 00:48:40,936 DEBUG org.apache.sqoop.orm.ClassWriter: > > PreSNI 2013-05-21 00:48:40,936 DEBUG org.apache.sqoop.orm.ClassWriter: > >SearchNewInventory 2013-05-21 00:48:40,936 DEBUG > >org.apache.sqoop.orm.ClassWriter: SNIVIN 2013-05-21 00:48:40,936 DEBUG > >org.apache.sqoop.orm.ClassWriter: SNIDetails 2013-05-21 00:48:40,936 > DEBUG > > org.apache.sqoop.orm.ClassWriter: SNIDealer 2013-05-21 00:48:40,936 > >DEBUG org.apache.sqoop.orm.ClassWriter: TestDriveForm 2013-05-21 > >00:48:40,936 DEBUG org.apache.sqoop.orm.ClassWriter: TestDriveComplete > >2013-05-21 00:48:40,936 DEBUG org.apache.sqoop.orm.ClassWriter: Inventives > >2013-05-21 00:48:40,937 DEBUG org.apache.sqoop.orm.ClassWriter: PreCal > >2013-05-21 00:48:40,937 DEBUG org.apache.sqoop.orm.ClassWriter: > >PaymentCalculator 2013-05-21 00:48:40,937 DEBUG > >org.apache.sqoop.orm.ClassWriter: PreGAQ 2013-05-21 00:48:40,937 DEBUG > >org.apache.sqoop.orm.ClassWriter: GetAQuote 2013-05-21 00:48:40,937 DEBUG > >org.apache.sqoop.orm.ClassWriter: LandingPage 2013-05-21 00:48:40,937 > >DEBUG org.apache.sqoop.orm.ClassWriter: Models 2013-05-21 00:48:40,937 > >DEBUG org.apache.sqoop.orm.ClassWriter: PhotosGallery 2013-05-21 > >00:48:40,937 DEBUG org.apache.sqoop.orm.ClassWriter: Interior 2013-05-21 > >00:48:40,937 DEBUG > org.apache.sqoop.orm.ClassWriter: > > Exterior 2013-05-21 00:48:40,937 DEBUG org.apache.sqoop.orm.ClassWriter: > >Performance 2013-05-21 00:48:40,938 DEBUG org.apache.sqoop.orm.ClassWriter: > > Safety 2013-05-21 00:48:40,938 DEBUG org.apache.sqoop.orm.ClassWriter: > >Specifications 2013-05-21 00:48:40,938 DEBUG > >org.apache.sqoop.orm.ClassWriter: ModelHighlights 2013-05-21 00:48:40,938 > >DEBUG org.apache.sqoop.orm.ClassWriter: Dimensions 2013-05-21 00:48:40,938 > >DEBUG org.apache.sqoop.orm.ClassWriter: Warranty 2013-05-21 00:48:40,965 > >DEBUG org.apache.sqoop.manager.SqlManager: Using fetchSize for next query: > >-2147483648 2013-05-21 00:48:40,965 INFO > >org.apache.sqoop.manager.SqlManager: Executing SQL statement: SELECT t.* > >FROM `CHRYSLER_Anametrix_Aggregated` AS t LIMIT 1 2013-05-21 00:48:40,990 > >DEBUG org.apache.sqoop.orm.ClassWriter: Writing source file: > >/tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/CHRYSLER_Anametrix_Aggregated.java > > 2013-05-21 00:48:40,990 DEBUG > > org.apache.sqoop.orm.ClassWriter: Table name: > >CHRYSLER_Anametrix_Aggregated 2013-05-21 00:48:40,990 DEBUG > >org.apache.sqoop.orm.ClassWriter: Columns: VisitorID:12, VisitorDay:12, > >VisitorMonth:12, VisitorYear:12, VisitorSession:4, Brand:12, DeviceType:12, > >DeviceBrowser:12, TrafficSource:12, SiteID:12, BuyID:12, AdvertisementID:12, > >PlacementID:12, CreativeID:12, PaidKeyword:12, Model200:4, Model200Conv:4, > >Model300:4, Model300TownCountry:4, BMOLineup:4, BMOModels:4, BMOColors:4, > >BMOOptions:4, BMOSummary:4, BrochureLineup:4, BrochureForm:4, > >BrochureMailLink:4, BrochureDownloadLink:4, PreDL:4, DealerLocate:4, > >DLHours:4, DLServices:4, DLDealerSite:4, PreSNI:4, SearchNewInventory:4, > >SNIVIN:4, SNIDetails:4, SNIDealer:4, TestDriveForm:4, TestDriveComplete:4, > >Inventives:4, PreCal:4, PaymentCalculator:4, PreGAQ:4, GetAQuote:4, > >LandingPage:4, Models:4, PhotosGallery:4, Interior:4, Exterior:4, > >Performance:4, Safety:4, Specifications:4, ModelHighlights:4, > > Dimensions:4, Warranty:4, 2013-05-21 00:48:40,990 DEBUG > >org.apache.sqoop.orm.ClassWriter: sourceFilename is > >CHRYSLER_Anametrix_Aggregated.java 2013-05-21 00:48:40,991 DEBUG > >org.apache.sqoop.orm.CompilationManager: Found existing > >/tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/ 2013-05-21 > >00:48:40,992 INFO org.apache.sqoop.orm.CompilationManager: > >$HADOOP_MAPRED_HOME is not set 2013-05-21 00:48:41,098 DEBUG > >org.apache.sqoop.orm.CompilationManager: Adding source file: > >/tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/CHRYSLER_Anametrix_Aggregated.java > > 2013-05-21 00:48:41,098 DEBUG org.apache.sqoop.orm.CompilationManager: > >Invoking javac with args: 2013-05-21 00:48:41,098 DEBUG > >org.apache.sqoop.orm.CompilationManager: -sourcepath 2013-05-21 > >00:48:41,098 DEBUG org.apache.sqoop.orm.CompilationManager: > >/tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/ 2013-05-21 > >00:48:41,098 DEBUG org.apache.sqoop.orm.CompilationManager: > -d > > 2013-05-21 00:48:41,099 DEBUG org.apache.sqoop.orm.CompilationManager: > >/tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/ 2013-05-21 > >00:48:41,099 DEBUG org.apache.sqoop.orm.CompilationManager: -classpath > >2013-05-21 00:48:41,099 DEBUG org.apache.sqoop.orm.CompilationManager: > > > > /run/cloudera-scm-agent/process/76-mapreduce-TASKTRACKER:/usr/lib/jvm/j2sdk1.6-oracle/lib/tools.jar:/usr/lib/hadoop-0.20-mapreduce:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.2.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop- > > 0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.0.0-mr1-cdh4.2.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/j > > asper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jline-0.9.94.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-map > > reduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/zookeeper-3.4.5-cdh4.2.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar:/usr/share/cmf/lib/plugins/navigator-plugin-4.5.1-shaded.jar:/usr/share/cmf/lib/plugins/event-publish-4.5.1-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-4.5.1.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.cloudera.2.jar: > > /usr/lib/hadoop-hdfs/lib/jersey-core-1.8.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/jline-0.9.94.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.4.5-cdh4.2.0.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.3.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.8.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/ > > usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.2.0.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.0.0-cdh4.2.0-tests.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/avro-1.7.3.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.2.jar:/usr/lib/hadoop/lib/jersey-core-1.8.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jline-0.9.94.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-c > > onfiguration-1.6.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.5-cdh4.2.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hado > > op/lib/hue-plugins-2.2.0-cdh4.2.0.jar:/usr/lib/hadoop/lib/kfs-0.3.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/jersey-server-1.8.jar:/usr/lib/hadoop/lib/jersey-json-1.8.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-auth-2.0.0-cdh4.2.0.jar:/usr/lib/hadoop/hadoop-annotations.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.2.0-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.0.0-cdh4.2.0.jar:/usr/lib/hadoop/hadoop-common-2.0.0-cdh4.2.0.jar:/usr/lib/hadoop/hadoop-common.jar:/mnt/mapred/local/taskTracker/distcache/-3359360379488511723_1277082233_1150217957/ip-10-147-191-48.ec2.internal/user/ubuntu/oozie-oozi/0000042-130520131050057-oozie-oozi-W/sqoop-node--sqoop/sqoop-launcher.jar:/mnt/mapred/local/taskTracker/distcache/-3926870669093555941_811939984_1868217431/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoo > > p/avro-ipc-1.7.3-tests.jar:/mnt/mapred/local/taskTracker/distcache/4298483340499219542_2138859044_1868217453/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/avro-ipc-1.7.3.jar:/mnt/mapred/local/taskTracker/distcache/-1975491928978283532_-1185784129_1868217475/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/avro-mapred-1.7.3.jar:/mnt/mapred/local/taskTracker/distcache/8448357694523829347_-1491317642_1868217500/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/commons-beanutils-1.7.0.jar:/mnt/mapred/local/taskTracker/distcache/-334079317866666788_-232552041_1868217523/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/commons-beanutils-core-1.8.0.jar:/mnt/mapred/local/taskTracker/distcache/-644448586073609072_-372575484_1868217547/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/commons-configuration-1.6.jar:/mnt/mapred/local/taskTracker/distcache/246054495199880993_357051411_1868217589/ip-10-147-191-48.ec2.internal/ > > user/oozie/share/lib/sqoop/commons-digester-1.8.jar:/mnt/mapred/local/taskTracker/distcache/-4144140590487253933_766503300_1868217614/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/commons-io-2.1.jar:/mnt/mapred/local/taskTracker/distcache/5753546228468760323_-553874526_1868217665/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/guava-11.0.2.jar:/mnt/mapred/local/taskTracker/distcache/8662866019753074519_-1003108238_1868217924/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/hbase-0.94.2-cdh4.2.0.jar:/mnt/mapred/local/taskTracker/distcache/-4840377369495417802_845918870_1868217954/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/high-scale-lib-1.1.1.jar:/mnt/mapred/local/taskTracker/distcache/-142623586016294918_445946554_1868217996/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/hsqldb-1.8.0.7.jar:/mnt/mapred/local/taskTracker/distcache/5142583573507929188_-105788043_1868218025/ip-10-147-191-48.ec2.internal/us > > er/oozie/share/lib/sqoop/httpclient-4.1.3.jar:/mnt/mapred/local/taskTracker/distcache/-6413369533723682352_-180904991_1868218051/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/httpcore-4.1.3.jar:/mnt/mapred/local/taskTracker/distcache/3446550291828223401_633399508_1868218088/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/jsr305-1.3.9.jar:/mnt/mapred/local/taskTracker/distcache/-6492962697058022370_-616322509_1868218133/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/libthrift-0.9.0.jar:/mnt/mapred/local/taskTracker/distcache/7049151581973813525_2017354882_1868218156/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/metrics-core-2.1.2.jar:/mnt/mapred/local/taskTracker/distcache/623583253106481693_846556431_1142287025/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/mysql-connector-java-5.1.24-bin.jar:/mnt/mapred/local/taskTracker/distcache/-7982675208196719520_-1207761111_1868218197/ip-10-147-191-48.ec2.interna > > l/user/oozie/share/lib/sqoop/netty-3.4.0.Final.jar:/mnt/mapred/local/taskTracker/distcache/-4577237135972373202_-1630140361_1868218251/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/sqoop-1.4.2-cdh4.2.0.jar:/mnt/mapred/local/taskTracker/distcache/-412267918101123238_-517736257_1868215515/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/oozie/json-simple-1.1.jar:/mnt/mapred/local/taskTracker/ubuntu/jobcache/job_201305131705_0181/attempt_201305131705_0181_m_000000_0/work:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.2.0.jar:/mnt/mapred/local/taskTracker/distcache/-4577237135972373202_-1630140361_1868218251/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/sqoop-1.4.2-cdh4.2.0.jar > > 2013-05-21 00:48:48,206 INFO org.apache.sqoop.orm.CompilationManager: > > Writing jar file: > > /tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/CHRYSLER_Anametrix_Aggregated.jar > > 2013-05-21 00:48:48,210 DEBUG > > org.apache.sqoop.orm.CompilationManager: Scanning for .class files in > >directory: /tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca > >2013-05-21 00:48:48,221 DEBUG org.apache.sqoop.orm.CompilationManager: Got > >classfile: > >/tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/CHRYSLER_Anametrix_Aggregated.class > > -> CHRYSLER_Anametrix_Aggregated.class 2013-05-21 00:48:48,236 DEBUG > >org.apache.sqoop.orm.CompilationManager: Finished writing jar file > >/tmp/sqoop-mapred/compile/f9c22cb042b8b020c4190df2f8e30bca/CHRYSLER_Anametrix_Aggregated.jar > > 2013-05-21 00:48:48,243 WARN org.apache.sqoop.manager.MySQLManager: MySQL > >Connector upsert functionality is using INSERT ON 2013-05-21 00:48:48,243 > >WARN org.apache.sqoop.manager.MySQLManager: DUPLICATE KEY UPDATE clause that > >relies on table's unique key. 2013-05-21 00:48:48,243 WARN > >org.apache.sqoop.manager.MySQLManager: Insert/update distinction is > >therefore independent on column 2013-05-21 00:48:48,243 WARN > > org.apache.sqoop.manager.MySQLManager: names specified in --update-key > >parameter. Please see MySQL 2013-05-21 00:48:48,245 WARN > >org.apache.sqoop.manager.MySQLManager: documentation for additional > >limitations. 2013-05-21 00:48:48,290 INFO > >org.apache.sqoop.mapreduce.ExportJobBase: Beginning export of > >CHRYSLER_Anametrix_Aggregated 2013-05-21 00:48:48,319 DEBUG > >org.apache.sqoop.mapreduce.JobBase: Using InputFormat: class > >org.apache.sqoop.mapreduce.ExportInputFormat 2013-05-21 00:48:48,575 DEBUG > >org.apache.sqoop.mapreduce.JobBase: Adding to job classpath: > >file:/mnt/mapred/local/taskTracker/distcache/-4577237135972373202_-1630140361_1868218251/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/sqoop-1.4.2-cdh4.2.0.jar > > 2013-05-21 00:48:48,581 DEBUG org.apache.sqoop.mapreduce.JobBase: Adding to > >job classpath: > > > >file:/mnt/mapred/local/taskTracker/distcache/623583253106481693_846556431_1142287025/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/mysql-connector-java-5.1.24-bin.jar > > 2013-05-21 00:48:48,585 DEBUG org.apache.sqoop.mapreduce.JobBase: Adding to > >job classpath: > >file:/mnt/mapred/local/taskTracker/distcache/-4577237135972373202_-1630140361_1868218251/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/sqoop-1.4.2-cdh4.2.0.jar > > 2013-05-21 00:48:48,588 DEBUG org.apache.sqoop.mapreduce.JobBase: Adding to > >job classpath: > >file:/mnt/mapred/local/taskTracker/distcache/-4577237135972373202_-1630140361_1868218251/ip-10-147-191-48.ec2.internal/user/oozie/share/lib/sqoop/sqoop-1.4.2-cdh4.2.0.jar > > 2013-05-21 00:48:48,588 WARN org.apache.sqoop.mapreduce.JobBase: SQOOP_HOME > >is unset. May not be able to find all job dependencies. 2013-05-21 > >00:48:48,739 WARN org.apache.hadoop.mapred.JobClient: Use > >GenericOptionsParser for parsing the arguments. Applications > > should implement Tool for the same. 2013-05-21 00:48:49,058 INFO > >org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to > >process : 1 2013-05-21 00:48:49,059 DEBUG > >org.apache.sqoop.mapreduce.ExportInputFormat: Target numMapTasks=1 > >2013-05-21 00:48:49,059 DEBUG org.apache.sqoop.mapreduce.ExportInputFormat: > >Total input bytes=8187428 2013-05-21 00:48:49,059 DEBUG > >org.apache.sqoop.mapreduce.ExportInputFormat: maxSplitSize=8187428 > >2013-05-21 00:48:49,063 INFO > >org.apache.hadoop.mapreduce.lib.input.FileInputFormat: Total input paths to > >process : 1 2013-05-21 00:48:49,073 DEBUG > >org.apache.sqoop.mapreduce.ExportInputFormat: Generated splits: 2013-05-21 > >00:48:49,073 DEBUG org.apache.sqoop.mapreduce.ExportInputFormat: > >Paths:/user/hive/warehouse/chrysler_anametrix_aggregated/part-AG-00000-1.txt:0+8187428 > > Locations:ip-10-147-191-48.ec2.internal:; 2013-05-21 00:48:49,403 INFO > >org.apache.hadoop.mapred.JobClient: Running job: > job_201305131705_0182 > > 2013-05-21 00:48:50,417 INFO org.apache.hadoop.mapred.JobClient: map 0% > >reduce 0% 2013-05-21 00:49:06,478 INFO org.apache.hadoop.mapred.JobClient: > >Task Id : attempt_201305131705_0182_m_000000_0, Status : FAILED 2013-05-21 > >00:49:16,703 INFO org.apache.hadoop.mapred.JobClient: Task Id : > >attempt_201305131705_0182_m_000000_1, Status : FAILED 2013-05-21 > >00:49:25,861 INFO org.apache.hadoop.mapred.JobClient: Task Id : > >attempt_201305131705_0182_m_000000_2, Status : FAILED 2013-05-21 > >00:49:42,012 INFO org.apache.hadoop.mapred.JobClient: Job complete: > >job_201305131705_0182 2013-05-21 00:49:42,015 INFO > >org.apache.hadoop.mapred.JobClient: Counters: 7 2013-05-21 00:49:42,020 INFO > >org.apache.hadoop.mapred.JobClient: Job Counters 2013-05-21 00:49:42,023 > >INFO org.apache.hadoop.mapred.JobClient: Failed map tasks=1 2013-05-21 > >00:49:42,023 INFO org.apache.hadoop.mapred.JobClient: Launched map > >tasks=4 2013-05-21 00:49:42,023 INFO > > org.apache.hadoop.mapred.JobClient: Data-local map tasks=4 2013-05-21 > >00:49:42,024 INFO org.apache.hadoop.mapred.JobClient: Total time spent > >by all maps in occupied slots (ms)=50187 2013-05-21 00:49:42,024 INFO > >org.apache.hadoop.mapred.JobClient: Total time spent by all reduces in > >occupied slots (ms)=0 2013-05-21 00:49:42,024 INFO > >org.apache.hadoop.mapred.JobClient: Total time spent by all maps waiting > >after reserving slots (ms)=0 2013-05-21 00:49:42,024 INFO > >org.apache.hadoop.mapred.JobClient: Total time spent by all reduces > >waiting after reserving slots (ms)=0 2013-05-21 00:49:42,032 WARN > >mapreduce.Counters: Group FileSystemCounters is deprecated. Use > >org.apache.hadoop.mapreduce.FileSystemCounter instead 2013-05-21 > >00:49:42,033 INFO org.apache.sqoop.mapreduce.ExportJobBase: Transferred 0 > >bytes in 53.4336 seconds (0 bytes/sec) 2013-05-21 00:49:42,035 WARN > >mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter > is deprecated. > > Use org.apache.hadoop.mapreduce.TaskCounter instead 2013-05-21 > >00:49:42,035 INFO org.apache.sqoop.mapreduce.ExportJobBase: Exported 0 > >records. 2013-05-21 00:49:42,035 ERROR org.apache.sqoop.tool.ExportTool: > >Error during export: Export job failed! 2013-05-21 00:49:42,063 INFO > >org.apache.hadoop.mapred.Task: Task:attempt_201305131705_0181_m_000000_0 is > >done. And is in the process of commiting 2013-05-21 00:49:43,124 INFO > >org.apache.hadoop.mapred.Task: Task attempt_201305131705_0181_m_000000_0 is > >allowed to commit now 2013-05-21 00:49:43,292 INFO > >org.apache.hadoop.mapred.FileOutputCommitter: Saved output of task > >'attempt_201305131705_0181_m_000000_0' to > >hdfs://ip-10-147-191-48.ec2.internal:8020/user/ubuntu/oozie-oozi/0000042-130520131050057-oozie-oozi-W/sqoop-node--sqoop/output > > 2013-05-21 00:49:43,300 INFO org.apache.hadoop.mapred.Task: Task > >'attempt_201305131705_0181_m_000000_0' done. 2013-05-21 00:49:43,309 INFO > org.apache.hadoop.mapred.TaskLogsTruncater: > > Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 > > > > Regards, > > Andy Tey > > Student !!
signature.asc
Description: Digital signature
