To benefit from YARN I'd suggest to use Apache Tez (application framework
that sits on top of YARN) - http://tez.apache.org/
Then you can see a big difference in how WordCount and other type jobs
could be simplified.
Yes
On Sun, May 11, 2014 at 9:57 AM, Karim Awara karim.aw...@kaust.edu.sawrote:
Hi,
Can I open multiple files on hdfs and write data to them in parallel and
then close them at the end?
--
Best Regards,
Karim Ahmed Awara
--
This message and its contents,
Yes there is. You can provide your own implementation of
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor and
configure it as 'yarn.nodemanager.container-executor.class' property.
There you can bypass Shell and create your own way of invoking processes.
Obviously it only makes
Without looking at the code its hard to say. Perhaps looking at a working
code will put you in the right direction.
For example, here is the DistributedShell from Hadoop (only few classes)
What version of Hadoop you are using? (YARN or no YARN)
To answer your question; Yes its possible and simple. All you need to to is
to have Hadoop JARs on the classpath with relevant configuration files on
the same classpath pointing to the Hadoop cluster. Most often people
simply copy
, Oleg Zhurakousky
oleg.zhurakou...@gmail.com wrote:
What version of Hadoop you are using? (YARN or no YARN)
To answer your question; Yes its possible and simple. All you need to to
is to have Hadoop JARs on the classpath with relevant configuration files
on the same classpath pointing
from my local machine to the cluster - also is I ran multiple jobs with
the same jar whether the jar would be copied N times (I typically chain 5
map-reduce jobs
On Fri, Apr 25, 2014 at 10:08 AM, Oleg Zhurakousky
oleg.zhurakou...@gmail.com wrote:
Are you talking about MR or plain YARN
While YARN-373 addresses a bit of a different problem the use case of reuse
of existing resources could be looked at few different ways. As I
understand YARN-373 talks about reusing resources from an old container to
spun off a new one. Another way of looking at the problem is to actually
reuse a
to use the code of hadoop-common-yarn of 2.3 instead of 2.2 This resolve
my problem at least.
On Sun, Mar 16, 2014 at 5:56 AM, Oleg Zhurakousky
oleg.zhurakou...@gmail.com wrote:
The bug you referring to is this i think
https://issues.apache.org/jira/browse/YARN-945
, Oleg Zhurakousky
oleg.zhurakou...@gmail.com wrote:
Thanks Jeff
Yes I am using 2.3 and the issue is still there.
Oleg
On Sun, Mar 16, 2014 at 3:10 AM, Jeff Zhang zjf...@gmail.com wrote:
Hi Oleg,
I meet the same issue when I start an unmanaged AM in client side in
thread way. The issue
=
RegisterApplicationMasterRequest.newInstance(, 0, );
RegisterApplicationMasterResponse response =
applicationsManager.registerApplicationMaster(request);
so at the time I don't have application id and line 3 throws the exception.
Oleg
On Sun, Mar 16, 2014 at 9:35 AM, Oleg Zhurakousky
oleg.zhurakou...@gmail.com wrote:
Thanks
So here is my dilemma.
I am trying to register ApplicationMaster to a remote YARN cluster and I
get
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
SIMPLE authentication is not enabled. Available:[TOKEN]
at
can go in
your configuration files and specific a specfic address to bind. You can
also go into your hostfile and ensure localhost does not refer to an ipv6
address. Java also has a -D switch like preferIPV4 or something like that.
On Sat, Mar 15, 2014 at 4:18 PM, Oleg Zhurakousky
You must be using Java 1.5 or below where @Override is not allowed on any
method that implements its counterpart from interface.
Remember, both 1.5 and 1.6 are EOL, so I would suggest upgrading to 1.7.
Oleg
On Mon, Mar 10, 2014 at 10:49 AM, Avinash Kujur avin...@gmail.com wrote:
hi,
i
Wow. . . blast from the past ;)!!
How the hell are you?
Cheers
Oleg
On Wed, Mar 5, 2014 at 10:18 AM, Melissa Warnkin missywarn...@yahoo.comwrote:
Hello Hadoop enthusiasts,
As you are no doubt aware, ApacheCon North America will be held in
Denver, Colorado starting on April 7th.
from somewhere an checking if
it is finished?
Am 22.01.2014 01:38, schrieb Oleg Zhurakousky:
I am not sure either, you have to ask Hadoop guys, but it was giving me a
hard time so I found a way around it.
On Tue, Jan 21, 2014 at 6:05 PM, Jay Vyas jayunit...@gmail.com wrote:
I guess im
No, all I do is have my own shutdown hook in the main which closes the
FSDataOutputStream. Before I did that it would throw an ugly exception when
I hit Ctrl+C, telling me that the stream is already closed, because of this
shutdown hook (bad design on the hadoop part), so removing it keeps it open
.
On Tue, Jan 21, 2014 at 5:58 PM, Oleg Zhurakousky
oleg.zhurakou...@gmail.com wrote:
No, all I do is have my own shutdown hook in the main which closes the
FSDataOutputStream. Before I did that it would throw an ugly exception when
I hit Ctrl+C, telling me that the stream is already closed, because
Just a little clarification
This is NOT how open source works by any means as there are many Open Source
projects with well written and maintained documentation.
It all comes down to the 2 Open Source models
1. ASF Open Source - which is a pure democracy or may be even anarchy without
any
,
check it out from SVN and submit patches to improve it (or at least submit a
JIRA as Mohammad mentioned). I cleaned up much of the Hadoop Wiki as I was
learning from it.
Glen
On 01/08/2013 07:13 AM, Oleg Zhurakousky wrote:
Just a little clarification
This is NOT how open source works
What is the differences between the two?
It seems like MR job could be configured using one of the other (e.g, extends
MapReduceBase implements Mapper of extends Mapper)
Cheers
Oleg
Trying to submit a MR job from the local machine and getting the above error
Any idea
Thanks
Oleg
=hduser in your configs while
submitting the job?
On Tue, Dec 11, 2012 at 10:19 PM, Oleg Zhurakousky
oleg.zhurakou...@gmail.com wrote:
Trying to submit a MR job from the local machine and getting the above error
Any idea
Thanks
Oleg
--
Harsh J
I studying user logs on the two node cluster that I have setup and I was
wondering if anyone can shed some light on these attempt*' directories
$ ls
attempt_201212051224_0021_m_00_0 attempt_201212051224_0021_m_03_0
job-acls.xml
attempt_201212051224_0021_m_02_0
Guys
I have a simple mapper that reads a records and sends out a message as it
encounters the ones it is interested in (no reducer). So no output is ever
written, but it seems like a job can not be submitted unless Output Path is
specified. Not a big deal to specify a dummy one, but was
Perfect! Thanks
On Dec 7, 2012, at 1:21 PM, Peyman Mohajerian mohaj...@gmail.com wrote:
I think this does it:
http://hadoop.apache.org/docs/r0.20.1/api/org/apache/hadoop/mapreduce/lib/output/NullOutputFormat.html
On Fri, Dec 7, 2012 at 10:06 AM, Oleg Zhurakousky
oleg.zhurakou...@gmail.com
Have you tried this?
Let's say your oozie user is 'oozie' and Name node group is called hadoop
sudo adduser --ingroup hadoop oozie
Oleg
On Nov 9, 2012, at 9:20 AM, Kartashov, Andy andy.kartas...@mpac.ca wrote:
Guys,
Came across this error like many others who tried to run Ooozie examples.
Have you setup passworless ssh to localhost?
In other words if when logged on as 'hdfs' can you do 'ssh localhost' and/or
'ssh ldsg-hadoop-master01'?
Oleg
On Nov 8, 2012, at 1:00 PM, John Beaulaurier -X (jbeaulau - ADVANCED NETWORK
INFORMATION INC at Cisco) jbeau...@cisco.com wrote:
Hello,
by default from the classpath. Any place where you have a
Configuration/JobConf you could also set up the right properties which
would be the location of the HDFS master (and mapred if you want to do
something about it).
Regards
Bertrand
On Fri, Oct 5, 2012 at 4:15 PM, Oleg Zhurakousky
sorry clicked send too soon, but basically changing that did not produce
any result, still seeing the same message.So I guess my question is what is
the property that is responsible for that?
Thanks
Oleg
On Fri, Oct 5, 2012 at 10:40 AM, Oleg Zhurakousky
oleg.zhurakou...@gmail.com wrote:
Yes I
30 matches
Mail list logo
