Re: Commands not working properlry when stored in pig file

Check hdfs to see if this really exists hdfs://user/lnindrakrishna/exp/03272013/part-r-0 On Wed, Mar 27, 2013 at 2:40 PM, Mix Nin wrote: > I wrote a pig script as follows and stored it in x.pig file > > Data = LOAD '/' as ( ) > NoNullData= FILTER Data by qe is not null; > STORE (for

MiniYARNCluster logs

Hey guys, We are using the MiniYARNCluster and trying to see where the NN, RM, job logs can be found. We see the job logs are present on HDFS but not on any local dirs. Also, none of the master node logs (NN, RM) are available. Digging in a bit further (just looked at this 1 file), I see there is

Re: Reading multiple files of a directory using a Single LOAD Command in PIG

What is the error? The LoadFunc should be ignoring any filenames that begin with "_" or a period "." If you are trying to skip the _SUCCESS file, the loader you are using (PigStorage) already handles that. Also, can you double check your path is not "/Output/part-m* as opposed to backward slashes

DFS Permissions on Hadoop 2.x

Hello, We just upgraded our cluster from 0.20.2 to 2.x (with HA) and had a question around disabling dfs permissions on the latter version. For some reason, setting the following config does not seem to work dfs.permissions.enabled false Any other configs that might be needed f

Re: DFS Permissions on Hadoop 2.x

en't > seeing this stack trace in your 0.20.2 deployment? > > Chris Nauroth > Hortonworks > http://hortonworks.com/ > > > > On Tue, Jun 18, 2013 at 10:54 AM, Prashant Kommireddi > wrote: > >> Hello, >> >> We just upgraded our cluster from 0.20.2

Re: DFS Permissions on Hadoop 2.x

ission call for your usage, then you > wouldn't have seen the problem. > > I'd like to gather these details for submitting a new bug report to HDFS. > Thanks! > > Chris Nauroth > Hortonworks > http://hortonworks.com/ > > > > On Tue, Jun 18, 2013 at 12:14 PM, Le

Re: DFS Permissions on Hadoop 2.x

ationUpdater > transition > INFO: Diagnostics report from attempt_1371593763906_0001_r_00_0: > Container killed by the ApplicationMaster. > > > > On Tue, Jun 18, 2013 at 1:28 PM, Chris Nauroth > wrote: > >> Prashant, can you provide more details about what you're doing when you >>

Job end notification does not always work (Hadoop 2.x)

Hello, I came across an issue that occurs with the job notification callbacks in MR2. It works fine if the Application master has started, but does not send a callback if the initializing of AM fails. Here is the code from MRAppMaster.java . ... // set job classloader if configure

Re: Job end notification does not always work (Hadoop 2.x)

Following-up on this. Please let me know if this is expected/bug and if you would like me to file a JIRA> On Thu, Jun 20, 2013 at 9:45 PM, Prashant Kommireddi wrote: > Hello, > > I came across an issue that occurs with the job notification callbacks in > MR2. It works fine if

Re: Job end notification does not always work (Hadoop 2.x)

example when the AM OOMs), I agree with you that we can do more. If you > feel strongly about this, please create a JIRA and possibly upload a patch. > > Thanks > Ravi > > > ------ > *From:* Prashant Kommireddi > *To:* "user@hado

Re: Job end notification does not always work (Hadoop 2.x)

gt; On Sat, Jun 22, 2013 at 3:48 PM, Arun C Murthy > wrote: > > Prashanth, > > > > Please file a jira. > > > > One thing to be aware of - AMs get restarted a certain number of times > for fault-tolerance - which means we can't just ass

Yarn log directory perms

Hey guys, It looks like the default perms for app/container dirs is set to 710 and is not configurable. From DefaultContainerExecutor /** Permissions for user log dir. * $logdir/$user/$appId */ private static final short LOGDIR_PERM = (short)0710; Any reasons for not having this be a con

Re: Yarn log directory perms

, September 13, 2013, Harsh J wrote: > This is true for MRv1 too, and is done so for security reasons. > > On Sat, Sep 14, 2013 at 2:37 AM, Prashant Kommireddi > > wrote: > > Hey guys, > > > > It looks like the default perms for app/container dirs is set to 710 an

ResourceManager webapp code runs OOM

Hello, We are noticing the RM running out of memory in the webapp code. It happens in org.apache.hadoop.yarn.server.resourcemanager.webapp.AppsBlock.renderBlock(Block html). The StringBuilder object "appsTableData" grows too large in this case while appending AppInfo. Ignoring the heap size (this

Re: ResourceManager webapp code runs OOM

the Heap may also be increased. > > HTH > Ravi > > > On Monday, October 21, 2013 5:54 PM, Prashant Kommireddi < > prash1...@gmail.com> wrote: > Hello, > > We are noticing the RM running out of memory in the webapp code. It > happens in > org.apache.hadoop.

Re: Log file parsing

Take a look at Pig's HadoopJobHistoryLoader http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/piggybank/storage/HadoopJobHistoryLoader.html On Thu, Aug 16, 2012 at 9:34 PM, peter wrote: > Now, no utilities for job tracker log . > > > -- > peter > Sent with Sparrow

Re: [Hadoop-Help]About Map-Reduce implementation

Hi mayur, Flume is used for data collection. Pig is used for data processing. For eg, if you have a bunch of servers that you want to collect the logs from and push to HDFS - you would use flume. Now if you need to run some analysis on that data, you could use pig to do that. Sent from my iPhone

Re: Can anyone point me to a good Map Reduce in memory Join implementation?

Specifically, replicated join - http://pig.apache.org/docs/r0.10.0/perf.html#replicated-joins On Fri, Feb 15, 2013 at 6:22 PM, David Boyd wrote: > Use PIG it has specific directives for in memory joins of small > data sets. The whole thing might require a half a dozen lines > of code. > > > > On

Re: Rewriting Ab-Initio scripts using Hadoop MapReduce

What specific info are you looking for? On Monday, December 23, 2013, Manoj Babu wrote: > Hi All, > > Can anybody share their experience on Rewriting Ab-Initio scripts using > Hadoop MapReduce? > > > Cheers! > Manoj. >

ProcfsBasedProcessTree

What's does ProcfsBasedProcessTree do? Trying to understand a bunch of these messages in the logs of a job that is stuck forever May 25, 2014 4:01:51 AM org.apache.hadoop.yarn.util.ProcfsBasedProcessTree constructProcessInfo INFO: The process 22793 may have finished in the interim. May 25, 2014 9:

Re: max number of application master in YARN

Take a look at yarn.scheduler.capacity.maximum-am-resource-percent On Thu, Apr 30, 2015 at 11:38 AM, Shushant Arora wrote: > Is there any configuration in MR2 and YARN to limit concurrent max > applications by setting max limit on ApplicationMasters in the cluster? >

Re: Pig 0.14.0 on Hadoop 2.6.0 deprecation errors

Something that needs correction, just that no one has gotten around to doing it. Please feel free to open a JIRA, even better if you would like to contribute a fix. On Tuesday, May 12, 2015, Anand Murali wrote: > Oliver: > > Many thanks for reply. If it is not an error why is the info repeated >

Pre-tasks to redirecting to JobHistoryServer

Hello folks, Have a question - what classes can I look at to understand the the way in which application counters/logs are copied to JHS before redirecting clients to it? Thanks, Prashant