[bcc: [email protected] (deprecated), to: [email protected]]
Kevin, The project page for Sqoop at Apache Incubator is at [1]. Also, information regarding relevant mailing lists that you should direct your comments to is at [2]. >From the information you have provided below, it appears that the data transfer was completed successfully but invocation of Hive client to populate that into a table failed for some reason. What version of Hive are you using? It would be good if you could try manually creating a table in Hive to make sure that it is working as expected. [1] http://incubator.apache.org/projects/sqoop.html [2] https://cwiki.apache.org/confluence/display/SQOOP/Mailing+Lists Thanks, Arvind On Fri, Aug 5, 2011 at 6:21 PM, Kevin <[email protected]> wrote: > http://incubator.apache.org/sqoop/ seems to be down, so I guess I'll > have to continue the post here. Sorry Arvind! > > But I ended up doing my project on a new machine, so I had to start > again from scratch. The problem I was having from before doesn't > occur > anymore. > > Instead, I encountered some other problems. > > I ran: > ./sqoop import --verbose --fields-terminated-by , --hive-drop-import- > delims --direct --connect jdbc:postgresql://query-4.redfintest.com: > 5432/stingray_6_5_d --username redfin_oltp -P --table brokerages -- > hive-import > > And got: > 11/08/05 17:47:05 DEBUG util.DirectImportUtils: Writing to filesystem: > file:/// > 11/08/05 17:47:05 DEBUG util.DirectImportUtils: Creating destination > directory brokerages > 11/08/05 17:47:05 DEBUG io.SplittingOutputStream: Opening next output > file: brokerages/data-00000 > 11/08/05 17:47:05 WARN util.NativeCodeLoader: Unable to load native- > hadoop library for your platform... using builtin-java classes where > applicable > 11/08/05 17:47:05 DEBUG manager.DirectPostgresqlManager: Starting > stream sink > 11/08/05 17:47:05 DEBUG manager.DirectPostgresqlManager: Waiting for > process completion > 11/08/05 17:47:05 INFO manager.DirectPostgresqlManager: Transfer loop > complete. > 11/08/05 17:47:05 INFO manager.DirectPostgresqlManager: Transferred > 79.166 KB in 0.0401 seconds (1.9274 MB/sec) > 11/08/05 17:47:05 INFO hive.HiveImport: Loading uploaded data into > Hive > 11/08/05 17:47:05 DEBUG hive.HiveImport: Hive.inputTable: brokerages > 11/08/05 17:47:05 DEBUG hive.HiveImport: Hive.outputTable: brokerages > 11/08/05 17:47:05 DEBUG manager.SqlManager: Using fetchSize for next > query: 1000 > 11/08/05 17:47:05 INFO manager.SqlManager: Executing SQL statement: > SELECT t.* FROM "brokerages" AS t LIMIT 1 > 11/08/05 17:47:05 DEBUG manager.SqlManager: Using fetchSize for next > query: 1000 > 11/08/05 17:47:05 INFO manager.SqlManager: Executing SQL statement: > SELECT t.* FROM "brokerages" AS t LIMIT 1 > 11/08/05 17:47:05 WARN hive.TableDefWriter: Column created_date had to > be cast to a less precise type in Hive > 11/08/05 17:47:05 DEBUG hive.TableDefWriter: Create statement: CREATE > TABLE IF NOT EXISTS `brokerages` ( `brokerage_id` BIGINT, > `brokerage_name` STRING, `managing_broker` STRING, `phone` STRING, > `created_date` STRING) COMMENT 'Imported by sqoop on 2011/08/05 > 17:47:05' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\054' LINES > TERMINATED BY '\012' STORED AS TEXTFILE > 11/08/05 17:47:05 DEBUG hive.TableDefWriter: Load statement: LOAD DATA > INPATH 'file:/usr/lib/sqoop-1.3.0/bin/brokerages' INTO TABLE > `brokerages` > 11/08/05 17:47:05 DEBUG hive.HiveImport: Using external Hive process. > 11/08/05 17:47:06 INFO hive.HiveImport: hive-log4j.properties not > found > 11/08/05 17:47:06 INFO hive.HiveImport: Hive history file=/tmp/root/ > hive_job_log_root_201108051747_1516665945.txt > 11/08/05 17:47:06 INFO hive.HiveImport: 11/08/05 17:47:06 INFO > exec.HiveHistory: Hive history file=/tmp/root/ > hive_job_log_root_201108051747_1516665945.txt > 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO > parse.ParseDriver: Parsing command: CREATE TABLE IF NOT EXISTS > `brokerages` ( `brokerage_id` BIGINT, `brokerage_name` STRING, > `managing_broker` STRING, `phone` STRING, `created_date` STRING) > COMMENT 'Imported by sqoop on 2011/08/05 17:47:05' ROW FORMAT > DELIMITED FIELDS TERMINATED BY '\054' LINES TERMINATED BY '\012' > STORED AS TEXTFILE > 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO > parse.ParseDriver: Parse Completed > 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO > parse.SemanticAnalyzer: Starting Semantic Analysis > 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO > parse.SemanticAnalyzer: Creating table brokerages position=27 > 11/08/05 17:47:07 INFO hive.HiveImport: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.IllegalArgumentException: URI: does not have a scheme > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java: > 919) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java: > 904) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java: > 7074) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java: > 6573) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java: > 238) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.Driver.run(Driver.java:736) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:262) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:269) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:430) > 11/08/05 17:47:07 INFO hive.HiveImport: at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > 11/08/05 17:47:07 INFO hive.HiveImport: at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 39) > 11/08/05 17:47:07 INFO hive.HiveImport: at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: > 25) > 11/08/05 17:47:07 INFO hive.HiveImport: at > java.lang.reflect.Method.invoke(Method.java:597) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.util.RunJar.main(RunJar.java:186) > 11/08/05 17:47:07 INFO hive.HiveImport: Caused by: > java.lang.IllegalArgumentException: URI: does not have a scheme > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java: > 127) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java: > 1855) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:1865) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java: > 917) > 11/08/05 17:47:07 INFO hive.HiveImport: ... 16 more > 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO > ql.Driver: Semantic Analysis Completed > 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO > ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, > properties:null) > 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO > ql.Driver: Starting command: CREATE TABLE IF NOT EXISTS `brokerages` > ( `brokerage_id` BIGINT, `brokerage_name` STRING, `managing_broker` > STRING, `phone` STRING, `created_date` STRING) COMMENT 'Imported by > sqoop on 2011/08/05 17:47:05' ROW FORMAT DELIMITED FIELDS TERMINATED > BY '\054' LINES TERMINATED BY '\012' STORED AS TEXTFILE > 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO > exec.DDLTask: Default to LazySimpleSerDe for table brokerages > 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 INFO > hive.log: DDL: struct brokerages { i64 brokerage_id, string > brokerage_name, string managing_broker, string phone, string > created_date} > 11/08/05 17:47:07 INFO hive.HiveImport: FAILED: Error in metadata: > java.lang.IllegalArgumentException: URI: does not have a scheme > 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 ERROR > exec.DDLTask: FAILED: Error in metadata: > java.lang.IllegalArgumentException: URI: does not have a scheme > 11/08/05 17:47:07 INFO hive.HiveImport: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.IllegalArgumentException: URI: does not have a scheme > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:476) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3146) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:213) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java: > 57) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.Driver.run(Driver.java:748) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:262) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:269) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:430) > 11/08/05 17:47:07 INFO hive.HiveImport: at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > 11/08/05 17:47:07 INFO hive.HiveImport: at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 39) > 11/08/05 17:47:07 INFO hive.HiveImport: at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: > 25) > 11/08/05 17:47:07 INFO hive.HiveImport: at > java.lang.reflect.Method.invoke(Method.java:597) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.util.RunJar.main(RunJar.java:186) > 11/08/05 17:47:07 INFO hive.HiveImport: Caused by: > java.lang.IllegalArgumentException: URI: does not have a scheme > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java: > 127) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java: > 1855) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:1865) > 11/08/05 17:47:07 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:470) > 11/08/05 17:47:07 INFO hive.HiveImport: ... 17 more > 11/08/05 17:47:07 INFO hive.HiveImport: > 11/08/05 17:47:07 INFO hive.HiveImport: FAILED: Execution Error, > return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask > 11/08/05 17:47:07 INFO hive.HiveImport: 11/08/05 17:47:07 ERROR > ql.Driver: FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask > 11/08/05 17:47:07 ERROR tool.ImportTool: Encountered IOException > running import job: java.io.IOException: Hive exited with status 9 > at > com.cloudera.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java: > 326) > at > com.cloudera.sqoop.hive.HiveImport.executeScript(HiveImport.java:276) > at > com.cloudera.sqoop.hive.HiveImport.importTable(HiveImport.java:218) > at > com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:362) > at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:423) > at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180) > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:219) > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:228) > at com.cloudera.sqoop.Sqoop.main(Sqoop.java:237) > > Data seems to have transferred correctly, but it transferred to file:/// > which is my local machine. I thought by default to should transfer to > hdfs:///? Is there a way for me to force it to transfer hdfs? On the > other hand, I'm having troubles invoking Hive once again. It seems as > though Sqoop doesn't recognize Hive even though I have set $HIVE_HOME > to the correct directory. I'm running Hive 0.7.0. > Thanks Arvind, I greatly appreciate your responses! > > On Aug 4, 6:10 pm, "[email protected]" <[email protected]> wrote: >> [bcc:[email protected], to:[email protected]] >> >> Kevin, >> >> It looks like the transfer of data was successful but there was a >> problem in invoking Hive. What version of Hive are you using? >> >> Hive by default generates a session log file in /tmp. Search for >> hive.log in /tmp and let us know what you find in there. From the >> looks of it, it appears to be a classpath issue. >> >> Thanks, >> Arvind >> >> Note: Please subscribe to [email protected] and direct >> further repsonses there. >> >> >> >> >> >> >> >> On Thu, Aug 4, 2011 at 5:38 PM, Kevin <[email protected]> wrote: >> > I appreciate your response Arvind. I tried the --direct route you >> > mentioned and it seems to have fixed the problems I mentioned earlier. >> > Unfortunately, I haven't been successful with Sqoop yet. I'm running >> > into this problem: >> >> > After executing: >> > sqoop import --direct --connect jdbc:postgresql:// >> > query-4.redfintest.com:5432/stingray_6_5_d --username redfin_oltp -P -- >> > table brokerages --hive-import --hive-home /usr/share/brisk/hive/ -- >> > target-dir /data/qa-metrics/ >> >> > I get: >> > 11/08/04 17:26:46 INFO tool.BaseSqoopTool: Using Hive-specific >> > delimiters for output. You can override >> > 11/08/04 17:26:46 INFO tool.BaseSqoopTool: delimiters with --fields- >> > terminated-by, etc. >> > 11/08/04 17:26:46 INFO manager.SqlManager: Using default fetchSize of >> > 1000 >> > 11/08/04 17:26:46 INFO tool.CodeGenTool: Beginning code generation >> > 11/08/04 17:26:47 INFO manager.SqlManager: Executing SQL statement: >> > SELECT t.* FROM "brokerages" AS t LIMIT 1 >> > 11/08/04 17:26:47 INFO manager.SqlManager: Executing SQL statement: >> > SELECT t.* FROM "brokerages" AS t LIMIT 1 >> > 11/08/04 17:26:47 INFO orm.CompilationManager: HADOOP_HOME is /usr/lib/ >> > hadoop >> > 11/08/04 17:26:47 INFO orm.CompilationManager: Found hadoop core jar >> > at: /usr/lib/hadoop/hadoop-0.20.2-cdh3u1-core.jar >> > 11/08/04 17:26:48 INFO orm.CompilationManager: Writing jar file: /tmp/ >> > sqoop-root/compile/8d97f121b1707576d1574cb5ba4653b0/brokerages.jar >> > 11/08/04 17:26:48 INFO manager.DirectPostgresqlManager: Beginning psql >> > fast path import >> > 11/08/04 17:26:48 INFO manager.SqlManager: Executing SQL statement: >> > SELECT t.* FROM "brokerages" AS t LIMIT 1 >> > 11/08/04 17:26:48 INFO manager.DirectPostgresqlManager: Performing >> > import of table brokerages from database stingray_6_5_d >> > 11/08/04 17:26:48 WARN util.NativeCodeLoader: Unable to load native- >> > hadoop library for your platform... using builtin-java classes where >> > applicable >> > 11/08/04 17:26:48 INFO manager.DirectPostgresqlManager: Transfer loop >> > complete. >> > 11/08/04 17:26:48 INFO manager.DirectPostgresqlManager: Transferred >> > 78.8574 KB in 0.0396 seconds (1.9445 MB/sec) >> > 11/08/04 17:26:48 INFO hive.HiveImport: Loading uploaded data into >> > Hive >> > 11/08/04 17:26:48 INFO manager.SqlManager: Executing SQL statement: >> > SELECT t.* FROM "brokerages" AS t LIMIT 1 >> > 11/08/04 17:26:48 INFO manager.SqlManager: Executing SQL statement: >> > SELECT t.* FROM "brokerages" AS t LIMIT 1 >> > 11/08/04 17:26:48 WARN hive.TableDefWriter: Column created_date had to >> > be cast to a less precise type in Hive >> > 11/08/04 17:26:48 INFO hive.HiveImport: Exception in thread "main" >> > java.lang.NoClassDefFoundError: jline/ArgumentCompletor >> > $ArgumentDelimiter >> > 11/08/04 17:26:48 INFO hive.HiveImport: at >> > java.lang.Class.forName0(Native Method) >> > 11/08/04 17:26:48 INFO hive.HiveImport: at >> > java.lang.Class.forName(Class.java:247) >> > 11/08/04 17:26:48 INFO hive.HiveImport: at >> > org.apache.hadoop.util.RunJar.main(RunJar.java:179) >> > 11/08/04 17:26:48 INFO hive.HiveImport: Caused by: >> > java.lang.ClassNotFoundException: jline.ArgumentCompletor >> > $ArgumentDelimiter >> > 11/08/04 17:26:48 INFO hive.HiveImport: at >> > java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> > 11/08/04 17:26:48 INFO hive.HiveImport: at >> > java.security.AccessController.doPrivileged(Native Method) >> > 11/08/04 17:26:48 INFO hive.HiveImport: at >> > java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> > 11/08/04 17:26:48 INFO hive.HiveImport: at >> > java.lang.ClassLoader.loadClass(ClassLoader.java:307) >> > 11/08/04 17:26:48 INFO hive.HiveImport: at sun.misc.Launcher >> > $AppClassLoader.loadClass(Launcher.java:301) >> > 11/08/04 17:26:48 INFO hive.HiveImport: at >> > java.lang.ClassLoader.loadClass(ClassLoader.java:248) >> > 11/08/04 17:26:48 INFO hive.HiveImport: ... 3 more >> > 11/08/04 17:26:48 ERROR tool.ImportTool: Encountered IOException >> > running import job: java.io.IOException: Hive exited with status 1 >> > at >> > com.cloudera.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.jav >> > a: >> > 326) >> > at >> > com.cloudera.sqoop.hive.HiveImport.executeScript(HiveImport.java:276) >> > at >> > com.cloudera.sqoop.hive.HiveImport.importTable(HiveImport.java:218) >> > at >> > com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:362) >> > at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:423) >> > at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144) >> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> > at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180) >> > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:219) >> > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:228) >> > at com.cloudera.sqoop.Sqoop.main(Sqoop.java:237) >> >> > It seems to be a Hive issue. I haven't had luck trying to figure out a >> > solution. Is it possible that my Hive is corrupt? >> >> > On Aug 2, 6:18 pm, "[email protected]" <[email protected]> wrote: >> >> [bcc:[email protected], to:[email protected]. >> >> Please move the conversation over to Apache mailing list.] >> >> >> Kevin, >> >> >> The OOM error you pointed out is raised when the proportion of the VM >> >> time spent is GC crosses a high threshold that should normally not be >> >> reached. This could happen if your existing heap space for the map >> >> task is small enough that it compares with the record size you are >> >> dealing with. You could try increasing your heap space by specifying >> >> the property mapred.child.java.opts to something like -Xmx4096m, >> >> assuming your nodes have that much memory to spare. You can also add >> >> another switch to this property -XX:-UseGCOverheadLimit which will >> >> disable the policy of the VM that results in OOM errors like you are >> >> seeing, however doing that may not be of any help. >> >> >> Alternatively, you could try using the direct mode of import from >> >> PostgreSQL server by specifying --direct option during the import. >> >> This option will require that you have the PostgreSQL client (psql) >> >> installed on the nodes where the map task will be executed. >> >> >> Thanks, >> >> Arvin >> >> >> On Tue, Aug 2, 2011 at 5:09 PM, Kevin <[email protected]> wrote: >> >> > Hi all, >> >> >> > I am trying to use Sqoop alongside Brisk. For those who don't know, >> >> > Brisk is DataStax's is a Hadoop/Hive distribution powered by >> >> > Cassandra.http://www.datastax.com/products/brisk >> >> >> > I'm attempting to use Sqoop to transfer data from a PostgreSQL db to >> >> > Hive. >> >> > I used this command: sqoop import --connect jdbc:postgresql:// >> >> > idb.corp.redfin.com:5432/metrics --username redfin_readonly -P --table >> >> > metrics --target-dir /data/qatest --hive-import >> >> >> > A end of the output is: >> >> >> > 11/08/02 16:50:15 INFO mapred.LocalJobRunner: >> >> > 11/08/02 16:50:16 INFO mapred.JobClient: map 100% reduce 0% >> >> > 11/08/02 16:50:18 INFO mapreduce.AutoProgressMapper: Auto-progress >> >> > thread is finished. keepGoing=false >> >> > 11/08/02 16:50:18 WARN mapred.LocalJobRunner: job_local_0001 >> >> > java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager: >> >> > Deleted path /tmp/hadoop-root/mapred/local/archive/ >> >> > 6026311161398729600_27875872_1046092330/file/usr/lib/sqoop/lib/ant- >> >> > contrib-1.0b3.jar >> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager: >> >> > Deleted path /tmp/hadoop-root/mapred/local/ >> >> > archive/-2957435182115348485_1795484550_1046091330/file/usr/lib/sqoop/ >> >> > lib/ant-eclipse-1.0-jvm1.2.jar >> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager: >> >> > Deleted path /tmp/hadoop-root/mapred/local/archive/ >> >> > 8769703156041621920_1132758636_1046087330/file/usr/lib/sqoop/lib/ >> >> > avro-1.5.1.jar >> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager: >> >> > Deleted path /tmp/hadoop-root/mapred/local/ >> >> > archive/-9111846482501201198_-633786885_1046093330/file/usr/lib/sqoop/ >> >> > lib/avro-ipc-1.5.1.jar >> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager: >> >> > Deleted path /tmp/hadoop-root/mapred/local/ >> >> > archive/-7340432634222599452_756368084_1046091330/file/usr/lib/sqoop/ >> >> > lib/avro-mapred-1.5.1.jar >> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager: >> >> > Deleted path /tmp/hadoop-root/mapred/local/ >> >> > archive/-5046079240639376542_-1808425119_1046090330/file/usr/lib/sqoop/ >> >> > lib/commons-io-1.4.jar >> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager: >> >> > Deleted path /tmp/hadoop-root/mapred/local/archive/ >> >> > 8537290295187062884_-810674145_1046086330/file/usr/lib/sqoop/lib/ >> >> > ivy-2.0.0-rc2.jar >> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager: >> >> > Deleted path /tmp/hadoop-root/mapred/local/ >> >> > archive/-3739620623688167588_-1832479804_1046082330/file/usr/lib/sqoop/ >> >> > lib/jackson-core-asl-1.7.3.jar >> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager: >> >> > Deleted path /tmp/hadoop-root/mapred/local/archive/ >> >> > 3083352659231038596_-1724007002_1046089330/file/usr/lib/sqoop/lib/ >> >> > jackson-mapper-asl-1.7.3.jar >> >> > 11/08/02 16:50:18 INFO filecache.TrackerDistributedCacheManager: >> >> > Deleted path /tmp/hadoop-root/mapred/local/archive/ >> >> > 2334745090627744860_-1029425194_1046082330/file/usr/lib/sqoop/lib/jopt- >> >> > simple-3.2.jar >> >> ... >> >> read more » > > -- > NOTE: The mailing list [email protected] is deprecated in favor of > Apache Sqoop mailing list [email protected]. Please subscribe > to it by sending an email to [email protected]. >
