Re: input split for hbase mapreduce

Please take a look at map() method of Mapper classes in the code base. e.g. hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/GroupingTableMapper.java On Tue, May 9, 2017 at 2:51 AM, Rajeshkumar J <rajeshkumarit8...@gmail.com> wrote: > Hi > >If I am running map

input split for hbase mapreduce

Hi If I am running mapreduce on hbase tables what will be the input to mapper function Thanks

Re: HBase mapreduce job crawls on final 25% of maps

It appears that my issue was caused by the missing sections I mentioned in the second post. I ran a job with these settings, and my job finished in < 6 hours. Thanks for your suggestions because I have further ideas regarding issues moving forward. scan.setCaching(500);// 1 is the default

Re: HBase mapreduce job crawls on final 25% of maps

Hi Chien, 4. From 50-150k per * second * to 100-150k per * minute *, as stated above, so reads went *DOWN* significantly. I think you must have misread. I will take into account some of your other suggestions. Thanks, Colin On Tue, Apr 12, 2016 at 8:19 PM, Chien Le wrote:

Re: HBase mapreduce job crawls on final 25% of maps

Some things I would look at: 1. Node statistics, both the mapper and regionserver nodes. Make sure they're on fully healthy nodes (no disk issues, no half duplex, etc) and that they're not already saturated from other jobs. 2. Is there a common regionserver behind the remaining mappers/regions? If

Re: HBase mapreduce job crawls on final 25% of maps

I've noticed that I've omitted scan.setCaching(500);// 1 is the default in Scan, which will be bad for MapReduce jobs scan.setCacheBlocks(false); // don't set to true for MR jobs which appear to be suggestions from examples. Still I am not sure if this explains the significant request

Re: HBase mapreduce job crawls on final 25% of maps

Excuse my double post. I thought I deleted my draft, and then constructed a cleaner, more detailed, more readable mail. On Tue, Apr 12, 2016 at 10:26 PM, Colin Kincaid Williams wrote: > After trying to get help with distcp on hadoop-user and cdh-user > mailing lists, I've given

HBase mapreduce job crawls on final 25% of maps

After trying to get help with distcp on hadoop-user and cdh-user mailing lists, I've given up on trying to use distcp and exporttable to migrate my hbase from .92.1 cdh4.1.3 to .98 on cdh5.3.0 I've been working on an hbase map reduce job to serialize my entries and insert them into kafka. Then I

HBASE mapReduce stoppage

We are bulk loading 1 billion rows into hbase. The 1 billion file was split into 20 files of ~22.5GB. Ingesting the file to hdfs took ~2min. Ingesting the first file to hbase took ~3 hours. The next took ~5hours, then it is increasing. By the sixth or seventh file the ingestion just stops

Re: HBASE mapReduce stoppage

Hi Dilon, Sounds like your table was not pre-split from the behavior that you are describing, but when you say that you are bulk loading the data using MR is this a MR job that does Put(s) into HBase or just generating HFiles (if using importtsv you have both options) that are later on bulk

Re: HBase MapReduce in Kerberized cluster

I searched (current) 0.98 and branch-1 where I found: ./hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java Looking at both 0.98[1] and 0.98.6[2] on github I see TokenUtil as part of hbase-server. Is it necessary for us to add this call to TokenUtil to all MR jobs

Re: HBase MapReduce in Kerberized cluster

Please take a look at HBASE-12493 User class should provide a way to re-use existing token which went into 0.98.9 FYI On Thu, May 14, 2015 at 8:37 AM, Edward C. Skoviak edward.skov...@gmail.com wrote: I searched (current) 0.98 and branch-1 where I found:

HBase MapReduce in Kerberized cluster

I'm attempting to write a Crunch pipeline to read various rows from a table in HBase and then do processing on these results. I am doing this from a cluster deployed using CDH 5.3.2 running Kerberos and YARN. I was hoping to get an answer on what is considered the best approach to authenticate to

Re: HBase MapReduce in Kerberized cluster

bq. it has been moved to be a part of the hbase-server package I searched (current) 0.98 and branch-1 where I found: ./hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java FYI On Wed, May 13, 2015 at 11:45 AM, Edward C. Skoviak edward.skov...@gmail.com wrote: I'm

A solution for data skew issue in HBase-Mapreduce jobs

Hi, all, I submit a new patch to fix the data skew issue in HBase-Mapreduce jobs. Would you please take a look at this new patch and give me some advice? https://issues.apache.org/jira/browse/HBASE-12590 Example: yeweichen2...@gmail.com

Re: A solution for data skew issue in HBase-Mapreduce jobs

Did you attach a screenshot ? The attachment shows up as grey area. Probably you can attach the image to JIRA. Cheers On Sun, Nov 30, 2014 at 6:57 PM, yeweichen2...@gmail.com yeweichen2...@gmail.com wrote: Hi, all, I submit a new patch to fix the data skew issue in HBase-Mapreduce

Re: Re: A solution for data skew issue in HBase-Mapreduce jobs

To: user@hbase.apache.org Subject: Re: A solution for data skew issue in HBase-Mapreduce jobs Did you attach a screenshot ? The attachment shows up as grey area. Probably you can attach the image to JIRA. Cheers On Sun, Nov 30, 2014 at 6:57 PM, yeweichen2...@gmail.com yeweichen2...@gmail.com wrote

Re: Hbase Mapreduce API - Reduce to a file is not working properly.

(); } context.write(key, new IntWritable(sum)); } } = Regards, Parkirat Bagga. -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Hbase-Mapreduce-API-Reduce-to-a-file-is-not-working-properly

Re: Hbase Mapreduce API - Reduce to a file is not working properly.

/Hbase-Mapreduce-API-Reduce-to-a-file-is-not-working-properly-tp4062141p4062240.html Sent from the HBase User mailing list archive at Nabble.com.

Re: Hbase Mapreduce API - Reduce to a file is not working properly.

(); } context.write(key, new IntWritable(sum)); } } = Regards, Parkirat Bagga. -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Hbase-Mapreduce-API-Reduce-to-a-file-is-not-working-properly-tp4062141p4062240

Re: Hbase Mapreduce API - Reduce to a file is not working properly.

in reducer. I am using Hbase Version Version 0.94.5.23. *Problem:* Now in my job, my mapper output a key as text and value as text, but my reducer output key as text and value as nullwritable, but it seems *hbase mapreduce api dont consider reducer*, and outputs both key and value as text

Re: Hbase Mapreduce API - Reduce to a file is not working properly.

or 1 so 1 test1 test1 this1 to 1 to 1 works 1 Regards, Parkirat Bagga -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Hbase-Mapreduce-API-Reduce-to-a-file-is-not-working-properly

Re: Hbase Mapreduce API - Reduce to a file is not working properly.

to 1 works 1 Regards, Parkirat Bagga -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Hbase-Mapreduce-API-Reduce-to-a-file-is-not-working-properly-tp4062141p406.html Sent from the HBase User mailing list

Hbase Mapreduce API - Reduce to a file is not working properly.

and value as nullwritable, but it seems *hbase mapreduce api dont consider reducer*, and outputs both key and value as text. Moreover if the same key comes twice, it goes to the file twice, even if my reducer want to log it only once. Could anybody help me with this problem? Regards, Parkirat Singh

Re: Hbase Mapreduce API - Reduce to a file is not working properly.

. *Problem:* Now in my job, my mapper output a key as text and value as text, but my reducer output key as text and value as nullwritable, but it seems *hbase mapreduce api dont consider reducer*, and outputs both key and value as text. Moreover if the same key comes twice, it goes

Re: HBase MapReduce problem

Hi Ted, I am trying your solution. But I got the same error message. Thanks

Re: HBase MapReduce problem

Did you create the table prior to launching your program ? If so, when you scan hbase:meta table, do you see row(s) for it ? Cheers On Feb 4, 2014, at 12:53 AM, Murali muralidha...@veradistech.com wrote: Hi Ted, I am trying your solution. But I got the same error message. Thanks

Re: HBase MapReduce problem

Hi Ted, I am using HBase 0.96 version. But I am also getting the below error message 14/02/03 10:18:32 ERROR mapreduce.TableOutputFormat: org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for after 35 tries. Exception in thread main

Re: HBase MapReduce problem

Murali: Are you using 0.96.1.1 ? Can you show us the command line you used ? Meanwhile I assume the HBase cluster is functional - you can use shell to insert data. Cheers On Mon, Feb 3, 2014 at 8:33 PM, Murali muralidha...@veradistech.com wrote: Hi Ted, I am using HBase 0.96 version.

Re: HBase MapReduce problem

Hi Ted Thanks for your reply. I am using HBase version 0.96.0. I can insert a record using shell command. I am running the below command to run my MapReduce job. It is a word count example. Reading a text file from hdfs file path and insert the counts to HBase table. hadoop jar hb.jar

Re: HBase MapReduce problem

See the sample command in http://hbase.apache.org/book.html#trouble.mapreduce : HADOOP_CLASSPATH=`hbase classpath` hadoop jar On Mon, Feb 3, 2014 at 9:33 PM, Murali muralidha...@veradistech.com wrote: Hi Ted Thanks for your reply. I am using HBase version 0.96.0. I can insert a record

HBase MapReduce with setup function problem

Dear all, I am writing a MapReduce application processing HBase table. In each map, it needs to read data from another HBase table, so i use the 'setup' function to initialize the HTable instance like this: @Override public void setup(Context context){ Configuration conf =

Re: HBase MapReduce with setup function problem

Have you considered using MultiTableInputFormat ? Cheers On Mon, Jan 27, 2014 at 9:14 AM, daidong daidon...@gmail.com wrote: Dear all, I am writing a MapReduce application processing HBase table. In each map, it needs to read data from another HBase table, so i use the 'setup' function

Re: HBase MapReduce with setup function problem

I agree that we should find the cause for why initialization got stuck. I noticed empty catch block: } catch (IOException e) { } Can you add some logging there to see what might have gone wrong ? Thanks On Mon, Jan 27, 2014 at 11:56 AM, daidong daidon...@gmail.com wrote: Dear

HBase MapReduce problem

Dear all, I have a simple HBase MapReduce application and try to run it on a 12-node cluster using this command: HADOOP_CLASSPATH=`bin/hbase classpath` ~/hadoop-1.1.2/bin/hadoop jar .jar org.test.WordCount HBase version is 0.95.0. But i got this error: java.lang.RuntimeException

Re: HBase MapReduce problem

Why do you use 0.95 which was a developer release ? See http://hbase.apache.org/book.html#d243e520 Cheers On Fri, Jan 24, 2014 at 8:40 AM, daidong daidon...@gmail.com wrote: Dear all, I have a simple HBase MapReduce application and try to run it on a 12-node cluster using this command

Re: HBase MapReduce problem

/book.html#d243e520 Cheers On Fri, Jan 24, 2014 at 8:40 AM, daidong daidon...@gmail.com wrote: Dear all, I have a simple HBase MapReduce application and try to run it on a 12-node cluster using this command: HADOOP_CLASSPATH=`bin/hbase classpath` ~/hadoop-1.1.2/bin/hadoop jar

FILE_BYTES_READ counter missing for HBase mapreduce job

Hi, Basically I have a mapreduce job to scan a hbase table and do some processing. After the job finishes, I only got three filesystem counters: HDFS_BYTES_READ, HDFS_BYTES_WRITTEN and FILE_BYTES_WRITTEN. The value of HDFS_BYTES_READ is not very useful here because it shows the size of the .META

Re: FILE_BYTES_READ counter missing for HBase mapreduce job

Addition info: The mapreduce job I run is a map-only job. It does not have reducers and it write data directly to hdfs in the mapper. Could this be the reason why there's no value for file_bytes_read? If so, is there any easy way to get the total input data size? Thanks Haijia On Thu, Sep 5,

issure about DNS error in running hbase mapreduce

i use hadoop-dns-checker check the dns problem ,seems all ok,but when i run MR task in hbase,it report problem,anyone have good idea? # ./run-on-cluster.sh hosts1 CH22 The authenticity of host 'ch22 (192.168.10.22)' can't be established. RSA key fingerprint is

HBase mapreduce job: unable to find region for a table

I am running a very simple MR HBase job (reading from a tiny HBase table and outputs nothing). I run it on a pseudo-distributed HBase cluster on my local machine which uses a pseudo-distributed HDFS (on local machine again). When I run it, I get the following exception: Unable to find region

Re: HBase mapreduce job: unable to find region for a table

Hi, Is your table properly served? Are you able to see it on the Web UI? Is you HBCK reporting everything correctly? JM 2013/7/11 S. Zhou myx...@yahoo.com I am running a very simple MR HBase job (reading from a tiny HBase table and outputs nothing). I run it on a pseudo-distributed HBase

Re: HBase mapreduce job: unable to find region for a table

Yes, I can see the table through hbase shell and web ui (localhost:60010). hbck reports ok From: Jean-Marc Spaggiari jean-m...@spaggiari.org To: user@hbase.apache.org; S. Zhou myx...@yahoo.com Sent: Thursday, July 11, 2013 11:01 AM Subject: Re: HBase

Re: HBase mapreduce job: unable to find region for a table

-Marc Spaggiari jean-m...@spaggiari.org *To:* user@hbase.apache.org; S. Zhou myx...@yahoo.com *Sent:* Thursday, July 11, 2013 11:01 AM *Subject:* Re: HBase mapreduce job: unable to find region for a table Hi, Is your table properly served? Are you able to see it on the Web UI? Is you HBCK

hbase + mapreduce

Hello: I'm working in a proyect, and i'm using hbase for storage the data, y have this method that work great but without the performance i'm looking for, so i want is to make the same but using mapreduce. public ArrayListMyObject findZ(String z) throws IOException {

Re: hbase + mapreduce

Here you have several examples: http://hbase.apache.org/book/mapreduce.example.html http://sujee.net/tech/articles/hadoop/hbase-map-reduce-freq-counter/ http://bigdataprocessing.wordpress.com/2012/07/27/hadoop-hbase-mapreduce-examples/ http://stackoverflow.com/questions/12215313/load-data

hbase + mapreduce

Hello: I'm working in a proyect, and i'm using hbase for storage the data, y have this method that work great but without the performance i'm looking for, so i want is to make the same but using mapreduce. public ArrayListMyObject findZ(String z) throws IOException {

Re: Hbase Mapreduce- Problem in using arrayList of pust in MapFunction

Tnx,But I don't know why when the client.buffer.size is increased, I've got bad result,does it related to other parameters ? and I give 8 gb heap to each regionserver. On Mon, Jan 21, 2013 at 12:34 PM, Harsh J ha...@cloudera.com wrote: Hi Farrokh, This isn't a HDFS question - please ask these

Re: Hbase Mapreduce- Problem in using arrayList of pust in MapFunction

Give put(ListPut puts) a shot and see if it works for you. Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com On Mon, Jan 21, 2013 at 11:41 AM, Farrokh Shahriari mohandes.zebeleh...@gmail.com wrote: Hi there Is there any way to use arrayList of Puts in map function to

RE: Hbase Mapreduce- Problem in using arrayList of pust in MapFunction

- From: Farrokh Shahriari [mohandes.zebeleh...@gmail.com] Sent: Monday, January 21, 2013 11:41 AM To: user@hbase.apache.org Subject: Hbase Mapreduce- Problem in using arrayList of pust in MapFunction Hi there Is there any way to use arrayList of Puts in map function to insert data

Re: Hbase MapReduce

/2012 01:33 PM Subject:RE: Hbase MapReduce It 's weird that hbase aggregate functions don't use MapReduce, this means that the performance will be very poor. Is it a must to use coprocessors? Is there a much easier way to improve the functions' performance ? CC: user@hbase.apache.org

Hbase MapReduce

Dear all, I wanted to ask a question.. Do Hbase Aggregate Functions such as rowcount, getMax, get Average use MapReduce to execute those functions? Thanks :D

Re: Hbase MapReduce

Regards, Dalia. You have to use MapReduce for that. In the HBase in Practice´s book, there are lot of great examples for this. On 11/24/2012 12:15 PM, Dalia Sobhy wrote: Dear all, I wanted to ask a question.. Do Hbase Aggregate Functions such as rowcount, getMax, get Average use MapReduce to

Re: Hbase MapReduce

Hi, but you do not need to us M/R. You could also use coprocessors. See this site: https://blogs.apache.org/hbase/entry/coprocessor_introduction - in the section Endpoints An aggregation coprocessor ships with hbase that should match your requirements. You just need to load it and eventually

Re: Hbase MapReduce

Do you think it would be a good idea to temper the use of CoProcessors? This kind of reminds me of when people first started using stored procedures... Sent from a remote device. Please excuse any typos... Mike Segel On Nov 24, 2012, at 11:46 AM, tom t...@arcor.de wrote: Hi, but you do not

RE: Hbase MapReduce

It 's weird that hbase aggregate functions don't use MapReduce, this means that the performance will be very poor. Is it a must to use coprocessors? Is there a much easier way to improve the functions' performance ? CC: user@hbase.apache.org From: michael_se...@hotmail.com Subject: Re: Hbase

RE: Hbase MapReduce

To: user@hbase.apache.org user@hbase.apache.org, Date: 11/24/2012 01:33 PM Subject:RE: Hbase MapReduce It 's weird that hbase aggregate functions don't use MapReduce, this means that the performance will be very poor. Is it a must to use coprocessors? Is there a much easier way

Re: Query regarding HBase Mapreduce

not sure what you mean by HBase mapreduce on small files. If you are using MapReduce with HBase as a source, you are not dealing with files directly. If you are using HBase as a sink, then the lots of small files is a problem which is orthogonal to the use of HBase. I don't think

Re: Query regarding HBase Mapreduce

Hi amit I am starting with Hbase and MR so my opinion ismore about what I read than real world. However the documentation says Hadoop will deal better with a set of large files than a lot of small ones. regards amit bohra bohra.a@... writes:

Re: Query regarding HBase Mapreduce

handle billions of files. 2012/10/25 amit bohra bohr...@gmail.com Hi, We are working on processing of lot of small files. For processing them we are using HBase Mapreduce as of now. Currently we are working with files in the range for around few millions, but over the period of time it would

HBase MapReduce - Using mutiple tables as source

Hi, While writing a MapReduce job for HBase, can I use multiple tables as input? I think TableMapReduceUtil.initTableMapperJob() takes a single table as parameter. For my requirement, I want to specify multiple tables and scan instances. I read about MultiTableInputCollection in the document

Re: HBase MapReduce - Using mutiple tables as source

Hello Amlan, Issue is still unresolved...Will get fixed in 0.96.0. Regards, Mohammad Tariq On Mon, Aug 6, 2012 at 5:01 PM, Amlan Roy amlan@cleartrip.com wrote: Hi, While writing a MapReduce job for HBase, can I use multiple tables as input? I think

Re: HBase MapReduce - Using mutiple tables as source

Hi, Isn't that the case that you can always initiate a scanner inside a map job (referring to another table from which had been set into the configuration of TableMapReduceUtil.initTableMapperJob(...) ) ? Hope this serves as temporary solution. On 08/06/2012 02:35 PM, Mohammad Tariq wrote:

Re: HBase MapReduce - Using mutiple tables as source

Hi Amlan, I think if you share your usecase regarding two tables as inputs, people on the mailing list may be able to help you better. For example, are you looking at joining the two tables? What are the sizes of the tables etc? Best Regards, Sonal Crux: Reporting for HBase

RE: HBase MapReduce - Using mutiple tables as source

. What is the best solution available in 0.92.0 (I understand the best solution is coming in version 0.96.0). Regards, Amlan -Original Message- From: Ioakim Perros [mailto:imper...@gmail.com] Sent: Monday, August 06, 2012 5:11 PM To: user@hbase.apache.org Subject: Re: HBase MapReduce - Using

Re: HBase MapReduce - Using mutiple tables as source

the best solution is coming in version 0.96.0). Regards, Amlan -Original Message- From: Ioakim Perros [mailto:imper...@gmail.com] Sent: Monday, August 06, 2012 5:11 PM To: user@hbase.apache.org Subject: Re: HBase MapReduce - Using mutiple tables as source Hi, Isn't that the case

RE: HBase MapReduce - Using mutiple tables as source

Subject: Re: HBase MapReduce - Using mutiple tables as source Hi, Isn't that the case that you can always initiate a scanner inside a map job (referring to another table from which had been set into the configuration of TableMapReduceUtil.initTableMapperJob(...) ) ? Hope this serves

Re: HBase MapReduce - Using mutiple tables as source

On Mon, Aug 6, 2012 at 3:22 PM, Wei Tan w...@us.ibm.com wrote: I understand that this is achievable by running multiple MR jobs, each with a different output table specified in the reduce class. What I want is to scan a source table once and generate multiple tables at one time. Thanks,

Re: HBase MapReduce - Using mutiple tables as source

Its available just as a patch on trunk for now. You wont find it in 0.92.0 ./zahoor On 06-Aug-2012, at 5:01 PM, Amlan Roy amlan@cleartrip.com wrote: https://issues.apache.org/jira/browse/HBASE-3996

A question about HBase MapReduce

Hello! I've read Lars George's blog http://www.larsgeorge.com/2009/05/hbase-mapreduce-101-part-i.html where at the end of the article, he mentioned In the next post I will show you how to import data from a raw data file into a HBase table and how you eventually process the data in the HBase

Re: A question about HBase MapReduce

florinp...@yahoo.com wrote: Hello! I've read Lars George's blog http://www.larsgeorge.com/2009/05/hbase-mapreduce-101-part-i.html where at the end of the article, he mentioned In the next post I will show you how to import data from a raw data file into a HBase table and how you eventually process

Re: HBase mapreduce sink - using a custom TableReducer to pass in Puts

My first guess would be to check if all the KVs using the same qualifier, because then it's basically the same cell 10 times. J-D On Mon, May 14, 2012 at 6:50 PM, Ben Kim benkimkim...@gmail.com wrote: Hello! I'm writing a mapreduce code to read a SequenceFile and write it to hbase table.

HBase mapreduce sink - using a custom TableReducer to pass in Puts

Hello! I'm writing a mapreduce code to read a SequenceFile and write it to hbase table. Normally, or what hbase tutorial tells us to do.. you would create a Put in TableMapper and pass it to IdentityTableReducer. This in fact work for me. But now I'm trying to separate the computations into

Re: HBase mapreduce sink - using a custom TableReducer to pass in Puts

Oops I made mistake while copy-paste The reducer initialization code should be like this TableMapReduceUtil.initTableReducerJob(rs_system, MyTableReducer, itemTableJob); On Tue, May 15, 2012 at 10:50 AM, Ben Kim benkimkim...@gmail.com wrote: Hello! I'm writing a mapreduce code to read a

HBase MapReduce Job with Multiple Scans

that has multiple scans as input. I found the following related post which points me to creating my own map reduce InputFormat type by extending HBase's TableInputFormatBase and overriding the getSplits() method: http://stackoverflow.com/questions/4821455/hbase-mapreduce-on-multiple-scan-objects So

Re: HBase MapReduce Job with Multiple Scans

by extending HBase's TableInputFormatBase and overriding the getSplits() method: http://stackoverflow.com/questions/4821455/hbase-mapreduce-on-multiple-scan-objects So, that's currently the direction I'm heading. However, before I got too far in the weeds I thought I'd ask: 1. Is this still

Re: HBase MapReduce Job with Multiple Scans

the getSplits() method: http://stackoverflow.com/questions/4821455/hbase-mapreduce-on-multiple-scan-objects So, that's currently the direction I'm heading. However, before I got too far in the weeds I thought I'd ask: 1. Is this still the best/right way

Re: hbase mapreduce running though command line

i tried to run the program from eclipse, but during that , i could not see any job running on the jobtracker/tasktracker web UI pages. i observed that on the eclipse localJobRunner is executing , so that job is not submitted to the whole cluster, but its executing on that name node machine alone.

hbase mapreduce running though command line

Hi, i want to run mapreduce program to insert data to tables in hbase. my cluster has 3 machines. If i want to run that program through command line, where can i do so..? should i do ${Hadoop_Home}/bin/hadoop jar MyJavaProg.jar java_mainclass_file source destn here MyJavaProg.jar is the jar of my

Re: hbase mapreduce running though command line

You don't need the conf dir in the jar, in fact you really don't want it there. I don't know where that alert is coming from, would be nice if you gave more details. J-D On Fri, Dec 9, 2011 at 6:45 AM, Vamshi Krishna vamshi2...@gmail.com wrote: Hi, i want to run mapreduce program to insert

Re: HBase MapReduce Zookeeper

I had the same issue. The problem for me turned out to be that the hbase.zookeeper.quorum was not set in hbase-site.xml in the server that submitted the mapreduce job. Ironically, this is also the same server that was running hbase master. This defaulted to 127.0.0.1 which was where the

Hbase Mapreduce jobs Dashboard

Hi All, When I run Hadoop mapreduce jobs, the job statistics and status is displayed in jobtracker/task tracker. But when I use HBase mapreduce it doesn't. Is there any hbase mapreduce dashboard available or am I missing something? Thanks Regards Jimson K James The Quieter You

Re: Hbase Mapreduce jobs Dashboard

. But when I use HBase mapreduce it doesn't. Is there any hbase mapreduce dashboard available or am I missing something? Thanks Regards Jimson K James The Quieter You Become The More You Are Able To Hear. * Confidentiality Statement/Disclaimer * This message and any

RE: Hbase Mapreduce jobs Dashboard

Subject: Re: Hbase Mapreduce jobs Dashboard HBase doesn't have it's own MapReduce system, it uses Hadoop's. How are you launching your jobs? On Mon, Sep 12, 2011 at 2:32 AM, Jimson K. James jimson.ja...@nestgroup.net wrote: Hi All, When I run Hadoop mapreduce jobs, the job statistics and status

Re: Hbase Mapreduce jobs Dashboard

PM To: user@hbase.apache.org Subject: Re: Hbase Mapreduce jobs Dashboard HBase doesn't have it's own MapReduce system, it uses Hadoop's. How are you launching your jobs? On Mon, Sep 12, 2011 at 2:32 AM, Jimson K. James jimson.ja...@nestgroup.net wrote: Hi All, When I run Hadoop

Fwd: HBase Mapreduce cannot find Map class

-- Forwarded message -- From: air cnwe...@gmail.com Date: 2011/7/28 Subject: HBase Mapreduce cannot find Map class To: CDH Users cdh-u...@cloudera.org import java.io.IOException; import java.text.ParseException; import java.text.SimpleDateFormat; import java.util.Date; import

Re: HBase MapReduce Zookeeper

this issue is still not resolved... unfortunatelly calling HConnectionManager.deleteConnection(conf, true); after the MR job is finished, does not close the connection to the zookeeper we have 3 zookeeper nodes by default there is a limit of 10 connections allowed from a single client so after

Re: HBase MapReduce Zookeeper

Try getting the ZooKeeperWatcher from the connection on your way out and explicitly shutdown the zk connection (see TestZooKeeper unit test for example). St.Ack On Thu, Jul 28, 2011 at 6:01 AM, Andre Reiter a.rei...@web.de wrote: this issue is still not resolved... unfortunatelly calling

Re: HBase MapReduce Zookeeper

10 connection maximum is too low. It has been recommended to go up to as many as 2000 connections in the list. This doesn't fix your problem but is something you should probably have in your configuration. ~Jeff On 7/28/2011 10:00 AM, Stack wrote: Try getting the ZooKeeperWatcher from the

Re: HBase MapReduce Zookeeper

@hbase.apache.org Sent: Thu, July 28, 2011 12:10:16 PM Subject: Re: HBase MapReduce Zookeeper 10 connection maximum is too low. It has been recommended to go up to as many as 2000 connections in the list. This doesn't fix your problem but is something you should probably have in your

Re: HBase MapReduce Zookeeper

i guess, i know the reason, why HConnectionManager.deleteConnection(conf, true); does not work for me in the MR job im using TableInputFormat, if you have a look at the source code in the method public void setConf(Configuration configuration) there is a line creating the HTable like this :

Re: HBase MapReduce Zookeeper

Yes, that's the connection leak. Use deleteAllConnections(true), and it will close all open connections. - Ruben From: Andre Reiter a.rei...@web.de To: user@hbase.apache.org Sent: Thu, July 28, 2011 4:55:52 PM Subject: Re: HBase MapReduce Zookeeper i guess

Re: HBase MapReduce Zookeeper

From: Andre Reiter a.rei...@web.de To: user@hbase.apache.org Sent: Thu, July 28, 2011 4:55:52 PM Subject: Re: HBase MapReduce Zookeeper i guess, i know the reason, why  HConnectionManager.deleteConnection(conf, true); does not work for me in the MR job im using

Re: HBase MapReduce Zookeeper

hi Ruben, St.Ack thanks a lot for your help! finally, the problem seems to be solved by an pretty sick workaround i did it like Bryan Keller described in this issue: https://issues.apache.org/jira/browse/HBASE-3792 @Ruben: thanks for the urls to that issues cheers andre

Re: HBase Mapreduce cannot find Map class

Maybe job.setJarByClass() can solve this problem. On Thu, Jul 28, 2011 at 7:06 PM, air cnwe...@gmail.com wrote: -- Forwarded message -- From: air cnwe...@gmail.com Date: 2011/7/28 Subject: HBase Mapreduce cannot find Map class To: CDH Users cdh-u...@cloudera.org import

Re: HBase MapReduce Zookeeper

Hi St.Ack, thanks for your reply but funally i miss the point, what would be the options to solve our issue? andre

Re: HBase MapReduce Zookeeper

Can you reuse Configuration instances though the configuration changes? Else in your Mapper#cleanup, call HTable.close() then try HConnectionManager.deleteConnection(table.getConfiguration()) after close (could be issue with executors used by multi* operations not completing before delete of

Re: HBase MapReduce Zookeeper

Hi Stack, just to make clear, actually the connections to the zookeeper being kept are not on our mappers (tasktrackers) but on the client, which schedules the MR job i think, the mappers are just fine, as they are andre Stack wrote: Can you reuse Configuration instances though the

Re: HBase MapReduce Zookeeper

Then similarly, can you do the deleteConnection above in your client or reuse the Configuration client-side that you use setting up the job? St.Ack On Wed, Jul 20, 2011 at 12:13 AM, Andre Reiter a.rei...@web.de wrote: Hi Stack, just to make clear, actually the connections to the zookeeper

Re: HBase MapReduce Zookeeper

Hi St.Ack, actually calling HConnectionManager.deleteConnection(conf, true); does not close the connection to the zookeeper i still can see the connection established... andre Stack wrote: Then similarly, can you do the deleteConnection above in your client or reuse the Configuration

  1   2   >