To benefit from YARN I'd suggest to use Apache Tez (application framework
that sits on top of YARN) - http://tez.apache.org/
Then you can see a big difference in how WordCount and other type jobs
could be simplified.
https://github.com/apache/tez/blob/master/tez-examples/src/main/java/org/apache/te
Yes
On Sun, May 11, 2014 at 9:57 AM, Karim Awara wrote:
> Hi,
>
> Can I open multiple files on hdfs and write data to them in parallel and
> then close them at the end?
>
> --
> Best Regards,
> Karim Ahmed Awara
>
> --
> This message and its contents, including attach
Yes there is. You can provide your own implementation of
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor and
configure it as 'yarn.nodemanager.container-executor.class' property.
There you can bypass Shell and create your own way of invoking processes.
Obviously it only makes se
from my local machine to the cluster - also is I ran multiple jobs with
> the same jar whether the jar would be copied N times (I typically chain 5
> map-reduce jobs
>
>
> On Fri, Apr 25, 2014 at 10:08 AM, Oleg Zhurakousky <
> oleg.zhurakou...@gmail.com> wrote:
>
>> Are
t;> 2) So presumably dropping core-site.xml, yarn-site into user.dir works
>>> do I need mapred-site.xml as well?
>>>
>>>
>>>
>>> On Fri, Apr 25, 2014 at 9:00 AM, Oleg Zhurakousky <
>>> oleg.zhurakou...@gmail.com> wrote:
>>>
&
Yes, if you are running MR
On Fri, Apr 25, 2014 at 12:48 PM, Steve Lewis wrote:
> Thank you for your answer
>
> 1) I am using YARN
> 2) So presumably dropping core-site.xml, yarn-site into user.dir works
> do I need mapred-site.xml as well?
>
>
>
> On Fri, Ap
What version of Hadoop you are using? (YARN or no YARN)
To answer your question; Yes its possible and simple. All you need to to is
to have Hadoop JARs on the classpath with relevant configuration files on
the same classpath pointing to the Hadoop cluster. Most often people
simply copy core-site.x
Without looking at the code its hard to say. Perhaps looking at a working
code will put you in the right direction.
For example, here is the DistributedShell from Hadoop (only few classes)
https://github.com/apache/hadoop-common/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/ha
While YARN-373 addresses a bit of a different problem the use case of reuse
of existing resources could be looked at few different ways. As I
understand YARN-373 talks about reusing resources from an old container to
spun off a new one. Another way of looking at the problem is to actually
reuse a c
Anyway, I've raised https://issues.apache.org/jira/browse/YARN-1841. This
is pretty messed up and needs to be addressed.
Oleg
On Sun, Mar 16, 2014 at 10:29 AM, Oleg Zhurakousky <
oleg.zhurakou...@gmail.com> wrote:
> Also, in your code you provide application id.
> I am t
=
RegisterApplicationMasterRequest.newInstance("", 0, "");
RegisterApplicationMasterResponse response =
applicationsManager.registerApplicationMaster(request);
so at the time I don't have application id and line 3 throws the exception.
Oleg
On Sun, Mar 16, 2014 at 9:35 AM, Oleg Zhurakousky <
oleg.
null;
>
> }
>
> });
>
> } catch (Throwable ex) {
>
> ex.printStackTrace();
>
> }
>
>
> On Sun, Mar 16, 2014 at 8:19 PM, Oleg Zhurakousky <
> oleg.zhurakou...@gmail.com> wrote:
>
>> Thanks Jeff
>>
>> Yes I a
> to use the code of hadoop-common-yarn of 2.3 instead of 2.2 This resolve
> my problem at least.
>
>
>
>
> On Sun, Mar 16, 2014 at 5:56 AM, Oleg Zhurakousky <
> oleg.zhurakou...@gmail.com> wrote:
>
>> The bug you referring to is this i think
>>
>> ht
n go in
> your configuration files and specific a specfic address to bind. You can
> also go into your hostfile and ensure localhost does not refer to an ipv6
> address. Java also has a -D switch like preferIPV4 or something like that.
>
>
> On Sat, Mar 15, 2014 at 4:18 PM, Ol
So here is my dilemma.
I am trying to register ApplicationMaster to a remote YARN cluster and I
get
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
SIMPLE authentication is not enabled. Available:[TOKEN]
at org.apache.hadoop.ipc.Client.call(C
You must be using Java 1.5 or below where @Override is not allowed on any
method that implements its counterpart from interface.
Remember, both 1.5 and 1.6 are EOL, so I would suggest upgrading to 1.7.
Oleg
On Mon, Mar 10, 2014 at 10:49 AM, Avinash Kujur wrote:
>
> hi,
>
> i downloaded the code
Wow. . . blast from the past ;)!!
How the hell are you?
Cheers
Oleg
On Wed, Mar 5, 2014 at 10:18 AM, Melissa Warnkin wrote:
> Hello Hadoop enthusiasts,
>
> As you are no doubt aware, ApacheCon North America will be held in
> Denver, Colorado starting on April 7th. Hadoop has 25 talks and
ook. So is this
> possible? E.g. getting the status of the task from somewhere an checking if
> it is finished?
>
> Am 22.01.2014 01:38, schrieb Oleg Zhurakousky:
>
> I am not sure either, you have to ask Hadoop guys, but it was giving me a
> hard time so I found a way around
On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky <
> oleg.zhurakou...@gmail.com> wrote:
>
>> No, all I do is have my own shutdown hook in the main which closes the
>> FSDataOutputStream. Before I did that it would throw an ugly exception when
>> I hit Ctrl+C, telling me
No, all I do is have my own shutdown hook in the main which closes the
FSDataOutputStream. Before I did that it would throw an ugly exception when
I hit Ctrl+C, telling me that the stream is already closed, because of this
shutdown hook (bad design on the hadoop part), so removing it keeps it open
Had the same problem recently
Here is how you can get around it.
I am using org.springframework.util.ReflectionUtils here just for
convenience (to traverse class hierarchy if necessary), but you can just
use raw reflection.
try {
Field clientFinalizerField = ReflectionUtils.findField(fileSystem
needs to help out with the documentation,
> check it out from SVN and submit patches to improve it (or at least submit a
> JIRA as Mohammad mentioned). I cleaned up much of the Hadoop Wiki as I was
> learning from it.
>
> Glen
>
> On 01/08/2013 07:13 AM, Oleg Zhurakousk
Just a little clarification
This is NOT "how open source works" by any means as there are many Open Source
projects with well written and maintained documentation.
It all comes down to the 2 Open Source models
1. ASF Open Source - which is a pure democracy or may be even anarchy without
any gov
What is the differences between the two?
It seems like MR job could be configured using one of the other (e.g, "extends
MapReduceBase implements Mapper" of "extends Mapper")
Cheers
Oleg
duser in your configs while
> submitting the job?
>
> On Tue, Dec 11, 2012 at 10:19 PM, Oleg Zhurakousky
> wrote:
>> Trying to submit a MR job from the local machine and getting the above error
>>
>> Any idea
>>
>> Thanks
>> Oleg
>
>
>
> --
> Harsh J
Trying to submit a MR job from the local machine and getting the above error
Any idea
Thanks
Oleg
I studying user logs on the two node cluster that I have setup and I was
wondering if anyone can shed some light on these "attempt*' directories
>$ ls
attempt_201212051224_0021_m_00_0 attempt_201212051224_0021_m_03_0
job-acls.xml
attempt_201212051224_0021_m_02_0 attempt_20121205122
Perfect! Thanks
On Dec 7, 2012, at 1:21 PM, Peyman Mohajerian wrote:
> I think this does it:
> http://hadoop.apache.org/docs/r0.20.1/api/org/apache/hadoop/mapreduce/lib/output/NullOutputFormat.html
>
> On Fri, Dec 7, 2012 at 10:06 AM, Oleg Zhurakousky
> wrote:
> Guys
&g
Guys
I have a simple mapper that reads a records and sends out a message as it
encounters the ones it is interested in (no reducer). So no output is ever
written, but it seems like a job can not be submitted unless Output Path is
specified. Not a big deal to specify a dummy one, but was wonderi
Not sure, haven't got that far yet ;)
On Nov 9, 2012, at 9:27 AM, "Kartashov, Andy" wrote:
> Oleg,
>
> Thanks. I had added oozie to the hadoop group. Do I also need to restart NN
> daemon?
>
>
> -----Original Message-
> From: Oleg Zhurakousky [ma
Have you tried this?
Let's say your oozie user is 'oozie' and Name node group is called hadoop
sudo adduser --ingroup hadoop oozie
Oleg
On Nov 9, 2012, at 9:20 AM, "Kartashov, Andy" wrote:
> Guys,
>
> Came across this error like many others who tried to run Ooozie examples.
> Searched and r
Have you setup passworless ssh to localhost?
In other words if when logged on as 'hdfs' can you do 'ssh localhost' and/or
'ssh ldsg-hadoop-master01'?
Oleg
On Nov 8, 2012, at 1:00 PM, "John Beaulaurier -X (jbeaulau - ADVANCED NETWORK
INFORMATION INC at Cisco)" wrote:
> Hello,
>
> Apache Had
/docs/r1.0.3/Secure_Impersonation.html
>
> Thank you,
> --Chris
>
>
> On Fri, Oct 5, 2012 at 7:42 AM, Oleg Zhurakousky <
> oleg.zhurakou...@gmail.com> wrote:
>
>> sorry clicked send too soon, but basically changing that did not produce
>> any result, still seeing the
sorry clicked send too soon, but basically changing that did not produce
any result, still seeing the same message.So I guess my question is what is
the property that is responsible for that?
Thanks
Oleg
On Fri, Oct 5, 2012 at 10:40 AM, Oleg Zhurakousky <
oleg.zhurakou...@gmail.com>
h. Any place where you have a
> Configuration/JobConf you could also set up the right properties which
> would be the location of the HDFS master (and mapred if you want to do
> something about it).
>
> Regards
>
> Bertrand
>
>
> On Fri, Oct 5, 2012 at 4:15 PM, Oleg Zhurak
35 matches
Mail list logo