Re: issue about submit job to local ,not to cluster

please check your mapred-site.xml is available under conf. On Tue, Oct 21, 2014 at 2:47 PM, ch huang wrote: > hi,maillist: >my cluster move from one IDC to another IDC ,when all done ,i run > job ,and find the job run on local box not on cluster ,why? it is normal on > old IDC! >

issue about submit job to local ,not to cluster

hi,maillist: my cluster move from one IDC to another IDC ,when all done ,i run job ,and find the job run on local box not on cluster ,why? it is normal on old IDC!

Re: How to limit the number of containers requested by a pig script?

Hello, as far as I understand. Number of mappers you cannot drive. The number of reducers you can control via PARALEL keyword. Number of containers on a node is given by following combination of settings: yarn.nodemanager.resource.memory-mb - set on a cluster. And following properties can be "modi

Re: hadoop 2.4 using Protobuf - How does downgrade back to 2.3 works ?

New files added in 2.4.0 will not be there in the metadata of 2.3.0. You need to add once again. On 10/21/14, Manoj Samel wrote: > Is the pre-upgrade metadata also kept updated with any changes one in 2.4.0 > ? Or is it just the 2.3.0 snapshot preserved? > > Thanks, > > On Sat, Oct 18, 2014 at 1:

Re: hadoop 2.4 using Protobuf - How does downgrade back to 2.3 works ?

after upgrade, there are current an previous subdirs in the namenode metadata directory. so during rollback, just delete current directory and rename previous to current again. On Tue, Oct 21, 2014 at 9:27 AM, Manoj Samel wrote: > Is the pre-upgrade metadata also kept updated with any changes on

How to limit the number of containers requested by a pig script?

Hi Everybody, I would like to know how I can limit the number of concurrent containers requested(and used ofcourse) by my pig-script (not as a yarn queue configuration or some such stuff.. I want to limit it from outside on a per job basis. I would ideally like to set the number in my pig-scrip

Re: Spark vs Tez

Using an interpreted scripting language with something that is billing itself as being fast doesn’t sound like the best idea... B. From: Russell Jurney Sent: Saturday, October 18, 2014 7:38 AM To: user@hadoop.apache.org Subject: Re: Spark vs Tez Check out PySpark. No Scala required. On Frida

pig configuration with oozee

hi hadoopers, i am new to pig and ozzee. i have installed oozee. how to configure the pig script for the ozzee to run at patricular time of every day. please help us. thanks, Arjun.

Re: hadoop 2.4 using Protobuf - How does downgrade back to 2.3 works ?

Is the pre-upgrade metadata also kept updated with any changes one in 2.4.0 ? Or is it just the 2.3.0 snapshot preserved? Thanks, On Sat, Oct 18, 2014 at 1:46 PM, Harsh J wrote: > Until you issue a finalize command, the pre-upgrade metadata is kept > aside for rolling back. When you issue the r

YARN rack-specific, relax_locality=false container request does not respond

Hello everyone, My rack-specific(relax_locality=false) container request does not get any response from the YARN RM. Since I found no reporting on this on the web, I am assuming that my cluster setup or YARN AM code is the problem. I’d like to know if anyone else has experienced the

Re: build hadoop-2.4.1 by mvn eclipse:eclipse fail in mac os x 10.9

FYI, for these sort of issues, I highly recommend reading the BUILDING.txt that comes in the Hadoop source code. Also, if you're going to use an IDE with Hadoop, I've found IntelliJ works well for me. The instructions here

Re: build hadoop-2.4.1 by mvn eclipse:eclipse fail in mac os x 10.9

Hi Li Li, This is happening because Hadoop expects classes.jar to be in a location which no longer exists on Java 7 on Mac. Below steps will fix it, $ mkdir -p /Library/Java/JavaVirtualMachines/jdk1.7.0_65.jdk/Contents/Home/Classes/ $ cd !$ $ ln -s /Library/Java/JavaVirtualMachines/jdk1.7.0_65.j

Re: Best number of mappers and reducers when processing data to and from HBase?

Yes, it's effectively reading in the mapper and writing in the reducer. The mapper is doing more than just reading the data, but as per my initial tests the average map function time is around 1ms to 3ms so it's not a big problem. The reducer is a bit slower however but it's still pretty fast. I am

Re: Best number of mappers and reducers when processing data to and from HBase?

For number of mappers, take a look at the following in TableInputFormatBase: public List getSplits(JobContext context) throws IOException { Is reducer required in your model ? Can you write to second hbase table from the mappers ? Cheers On Mon, Oct 20, 2014 at 7:08 AM, peterm_second wrote

AW: Best number of mappers and reducers when processing data to and from HBase?

Hallo Rolf, in der letzten Oktober Woche, aber ich muss glaub ich nicht die ganze Zeit dabei sein. Mit freundlichen Grüßen Christian Kleegrewe Siemens AG Corporate Technology Research and Technology Center CT RTC BAM KMR-DE Otto-Hahn-Ring 6 81739 München, Deutschland Tel.: +49 89 636-633785 mai

Best number of mappers and reducers when processing data to and from HBase?

Hi Guys, I have a somewhat abstract question to ask. I am reading data from Hbase and I was wondering how am I to know what's the best mapper and reducer count, I mean what are the criteria that need to be taken into consideration when determining the mapper and reducer counts. My MR job is re

Re: Can add a regular check in DataNode on free disk space?

Hi Aitor, Actually I did so in my test. But the issue is that I did not find disk full info in any log. 2014-10-20 4:00 GMT-07:00 Aitor Cedres : > > Hi Sam, > > You can set the property "dfs.datanode.du.reserved" to reserve some space > for non-DFS use. By doing that, Hadoop daemons will keep wr

Re: Can add a regular check in DataNode on free disk space?

Hi Sam, You can set the property "dfs.datanode.du.reserved" to reserve some space for non-DFS use. By doing that, Hadoop daemons will keep writing to log files, and it will help you diagnose the issue. Hope it helps. Regards, Aitor On 20 October 2014 11:27, sam liu wrote: > Hi Dhiraj, > > My

Re: Can add a regular check in DataNode on free disk space?

Hi Dhiraj, My cluster only includes 1 datanode and its log does not include any warning/error messages for the out of free disk space. That wastes some of my time to find the root cause. Also I did not find any free disk checking code in DataNode.java. So it will be better if the datanode could c

Request - How to copy the file from MR after each run -

Hi Users, Need a immediate help on this. Each time my MR runs it will create below directory structure but necessary it will create all file1,file2 and file2, even sometime it will create only file1/file2 etc. And all the below folder structure will be dynamically created in MR not by manually, on

build hadoop-2.4.1 by mvn eclipse:eclipse fail in mac os x 10.9

it says can't find tools at Library/Java/JavaVirtualMachines/jdk1.7.0_65.jdk/Contents/Home/Classes/classes.jar My JAVA_HOME is /Library/Java/JavaVirtualMachines/jdk1.7.0_65.jdk/Contents/home. I can find tools.jar in $JAVA_HOME/lib directory. Why it try to find tools.jar in Library/Java/JavaVirtualM