Flume - multiple import to Hbase
Hi, We are using the cloudera's HBase sink to import data to Hbase using flume. Can anybody tell how to do a multiple file import using the same? Regards, Shobha M __ Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
RE: Open source of HiBench 2.1 (a Hadoop benchmark suite)
Jason, HiBench looks very interesting, and I'm downloading the data files now to try it out. HiTune looks extremely useful as well, but it looks like it is still at a prerelease level (0.9), and has been a year between commits. Is there ongoing development / maintenance on this project, and if so, will those updates be released to the public? Thanks! - Tim. From: Dai, Jason [jason@intel.com] Sent: Wednesday, June 13, 2012 9:27 PM To: common-user@hadoop.apache.org Subject: Open source of HiBench 2.1 (a Hadoop benchmark suite) Hi, HiBench, a Hadoop benchmark suite constructed by Intel, is used intensively for Hadoop benchmarking, tuning optimizations both inside Intel and by our customers/partners. It consists of a set of representative Hadoop programs including both micro-benchmarks and more real world applications (e.g., search, machine learning and Hive queries). We have made HiBench 2.1 available under Apache License 2.0 at https://github.com/hibench/HiBench-2.1, and would like to get your feedbacks on how it can be further improved. BTW, please stop by the Intel booth if you are at Hadoop summit, so that we can have more interactive discussions on both HiBench and HiTune (our Hadoop performance analyzer open sourced at https://github.com/hitune/hitune). Thanks, -Jason The information contained in this email is intended only for the personal and confidential use of the recipient(s) named above. The information and any attached documents contained in this message may be Exar confidential and/or legally privileged. If you are not the intended recipient, you are hereby notified that any review, use, dissemination or reproduction of this message is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by return email and delete the original message.
use hdfs as normal file system?
Hello, I have installed hdfs to use it with hadoop and hbase. I am wondering if i can use it as a normal file system too that just connects several HDDs ? For example i can see the files i have stored through the webinterface (browse filesystem) or with the dfs commands. However if I go to the exact location in the hdfs directory there is nothing. Everything is stored in an hdfs format. Is it possible to go to /app/hadoop/tmp/data/text.txt for example in some way? Can i use it in such way? I just want to avoid to install NFS servers as i already have in place hdfs. Best, PA
Re: use hdfs as normal file system?
Hi Pierre, According to my knowledge HDFS is not compliant as a normal filesystem. But you can use Fuse as a workaround to mount it. This is what we do in our company, but we were experiencing some stability issues. Ruslan On Sat, Jun 16, 2012 at 12:22 AM, Pierre Antoine DuBoDeNa pad...@gmail.com wrote: Hello, I have installed hdfs to use it with hadoop and hbase. I am wondering if i can use it as a normal file system too that just connects several HDDs ? For example i can see the files i have stored through the webinterface (browse filesystem) or with the dfs commands. However if I go to the exact location in the hdfs directory there is nothing. Everything is stored in an hdfs format. Is it possible to go to /app/hadoop/tmp/data/text.txt for example in some way? Can i use it in such way? I just want to avoid to install NFS servers as i already have in place hdfs. Best, PA
Re: Map works well, but Redue failed
Most probably you have a network problem. Check your hostname and IP address mapping From: Yongwei Xing jdxyw2...@gmail.com To: common-user@hadoop.apache.org Sent: Thursday, June 14, 2012 10:15 AM Subject: Map works well, but Redue failed Hi all I run a simple sort program, however, I meet such error like below. 12/06/15 01:13:17 WARN mapred.JobClient: Error reading task outputServer returned HTTP response code: 403 for URL: http://192.168.1.106:50060/tasklog?plaintext=trueattemptid=attempt_201206150102_0002_m_01_1filter=stdout 12/06/15 01:13:18 WARN mapred.JobClient: Error reading task outputServer returned HTTP response code: 403 for URL: http://192.168.1.106:50060/tasklog?plaintext=trueattemptid=attempt_201206150102_0002_m_01_1filter=stderr 12/06/15 01:13:20 INFO mapred.JobClient: map 50% reduce 0% 12/06/15 01:13:23 INFO mapred.JobClient: map 100% reduce 0% 12/06/15 01:14:19 INFO mapred.JobClient: Task Id : attempt_201206150102_0002_m_00_2, Status : FAILED Too many fetch-failures 12/06/15 01:14:20 WARN mapred.JobClient: Error reading task outputServer returned HTTP response code: 403 for URL: http://192.168.1.106:50060/tasklog?plaintext=trueattemptid=attempt_201206150102_0002_m_00_2filter=stdout Does anyone know what's the reason and how to resolve it? Best Regards, -- Welcome to my ET Blog http://www.jdxyw.com
RE: Open source of HiBench 2.1 (a Hadoop benchmark suite)
Hi Tim, The data files are only for nutchindexing and bayes; other workloads can run without the data files. There are still on-going development on HiTune, with the current focus on its ease of deployment (i.e., eliminating the additional Hadoop cluster used by Chukwa). And we would like to get your feedbacks on how to improve HiTune and HiTune after you try them out. Thanks, -Jason From: Tim Broberg tim.brob...@exar.commailto:tim.brob...@exar.com Date: Fri, Jun 15, 2012 at 1:07 PM Subject: RE: Open source of HiBench 2.1 (a Hadoop benchmark suite) To: common-user@hadoop.apache.orgmailto:common-user@hadoop.apache.org common-user@hadoop.apache.orgmailto:common-user@hadoop.apache.org Jason, HiBench looks very interesting, and I'm downloading the data files now to try it out. HiTune looks extremely useful as well, but it looks like it is still at a prerelease level (0.9), and has been a year between commits. Is there ongoing development / maintenance on this project, and if so, will those updates be released to the public? Thanks! - Tim. From: Dai, Jason [jason@intel.commailto:jason@intel.com] Sent: Wednesday, June 13, 2012 9:27 PM To: common-user@hadoop.apache.orgmailto:common-user@hadoop.apache.org Subject: Open source of HiBench 2.1 (a Hadoop benchmark suite) Hi, HiBench, a Hadoop benchmark suite constructed by Intel, is used intensively for Hadoop benchmarking, tuning optimizations both inside Intel and by our customers/partners. It consists of a set of representative Hadoop programs including both micro-benchmarks and more real world applications (e.g., search, machine learning and Hive queries). We have made HiBench 2.1 available under Apache License 2.0 at https://github.com/hibench/HiBench-2.1, and would like to get your feedbacks on how it can be further improved. BTW, please stop by the Intel booth if you are at Hadoop summit, so that we can have more interactive discussions on both HiBench and HiTune (our Hadoop performance analyzer open sourced at https://github.com/hitune/hitune). Thanks, -Jason The information contained in this email is intended only for the personal and confidential use of the recipient(s) named above. The information and any attached documents contained in this message may be Exar confidential and/or legally privileged. If you are not the intended recipient, you are hereby notified that any review, use, dissemination or reproduction of this message is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by return email and delete the original message.
Re: use hdfs as normal file system?
Thanks for your reply!! This is the only think i found related on the web.. but i don't see much support let's say for it.. so i am not sure if it's the best choice.. What kind of issues you get in your company? (What company? if i may ask.. I mean I deal with scientific documents mostly.. the files most of the time will be static we just have to read them several times to apply algorithms). 2012/6/15 Ruslan Al-Fakikh metarus...@gmail.com Hi Pierre, According to my knowledge HDFS is not compliant as a normal filesystem. But you can use Fuse as a workaround to mount it. This is what we do in our company, but we were experiencing some stability issues. Ruslan On Sat, Jun 16, 2012 at 12:22 AM, Pierre Antoine DuBoDeNa pad...@gmail.com wrote: Hello, I have installed hdfs to use it with hadoop and hbase. I am wondering if i can use it as a normal file system too that just connects several HDDs ? For example i can see the files i have stored through the webinterface (browse filesystem) or with the dfs commands. However if I go to the exact location in the hdfs directory there is nothing. Everything is stored in an hdfs format. Is it possible to go to /app/hadoop/tmp/data/text.txt for example in some way? Can i use it in such way? I just want to avoid to install NFS servers as i already have in place hdfs. Best, PA
