Re: what happens to a client attempting to get a new app when the resource manager is already down
Hi All entities including NM and client etc. use the same RMProxy to talk with RM. The retry behavior is controlled by the following two configs. yarn.resourcemanager.connect.max-wait.ms controls Maximum time to wait to establish connection to ResourceManager. yarn.resourcemanager.connect.retry-interval.ms controls How often to try connecting to the ResourceManager. Jian On Fri, Jan 31, 2014 at 11:52 PM, REYANE OUKPEDJO r.oukpe...@yahoo.comwrote: Hi there, I am trying to solve a problem. My client run as a server. And was trying to make my client aware about the fact the resource manager is down but I could not figure out. The reason is that the call : yarnClient.createApplication(); never return when the resource manager is down. However it just stay in a loops and sleep after 10 iteration and continue the same loops. Below you can find the logs. Any idea how to leave this loop ? is there any parameter that control the number of seconds before giving up. Thanks Reyane OUKPEDJO logs 14/01/31 10:48:05 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:48:06 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:48:37 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:48:38 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:48:39 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:48:40 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:48:41 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:48:42 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:48:43 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:48:44 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:48:45 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:48:46 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:49:17 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:49:18 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:49:19 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:49:20 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:49:21 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 14/01/31 10:49:22 INFO ipc.Client: Retrying connect to server: isblade2/ 9.32.160.125:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under
kerberos principals per node necessary?
i s it necessary to create a kerberos principal for hdfs on every node, as in hdfs/some-host@SOME-REALM? why not use one principal hdfs@SOME-REALM? that way i could distribute the same keytab file to all nodes which makes things a lot easier. thanks! koert
Re: shifting sequenceFileOutput format to Avro format
Hi Yong,
I followed your 2nd suggestion. My data format is is nested(list of map),
So I created .avsc as below.
{namespace: test.avro,
type: record,
name: Session,
fields: [
{name:VisitCommon, type: {
type: map, values:string},
{name:events,
type: {
type: array,
items:{
name:Event,
type:map,
values:string}
}
}
]
}
And I tried creating corresponding classes by using avro tool and with
plugin, but there are few errors on generated java code. What could be the
issue?
1) Error: The method deepCopy(Schema, ListMapCharSequence,CharSequence)
is undefined for the type GenericData
2) And also observed there is some deprecated code.
@Deprecated public
java.util.Mapjava.lang.CharSequence,java.lang.CharSequence VisitCommon;
I used eclipse plugin as mentioned below.
http://avro.apache.org/docs/1.7.6/mr.html
Thanks Regards,
B Anil Kumar.
On Fri, Jan 31, 2014 at 8:27 AM, AnilKumar B akumarb2...@gmail.com wrote:
Thanks Yong.
Thanks Regards,
B Anil Kumar.
On Fri, Jan 31, 2014 at 12:44 AM, java8964 java8...@hotmail.com wrote:
In avro, you need to think about a schema to match your data. Avor's
schema is very flexible and should be able to store all kinds of data.
If you have a Json string, you have 2 options to generate the Avro schema
for it:
1) Use type: string to store the whole Json string into Avro. This will
be easiest, but you have to parse the data later when you use it.
2) Use Avro schema to match your json data, using matching structure from
avro for your data, like 'record, array, map' etc.
Yong
--
Date: Fri, 31 Jan 2014 00:13:59 +0530
Subject: shifting sequenceFileOutput format to Avro format
From: akumarb2...@gmail.com
To: user@hadoop.apache.org
Hi,
As of now in my jobs, I am using SequenceFileOutputFormat and I am
emitting custom java objects as MR output.
Now I am planning to emit it in avro format, I went through few blogs
but still have following doubts.
1) My current custom Writable objects has nested json format as
toString(), So when I shift to avro format, should I just emit json string
in avro format, instead of writable custom object?
2) If so, how can I create schema? My json string is nested and will have
random key/value pairs.
3) Or can I still emit as custom objects?
Thanks Regards,
B Anil Kumar.
commissioning and decommissioning a task tracker
Hello , I am not able to commission and decommission a tasktracker. I am using hadoop version 1.0.3 and used the property mapred.hosts property for commissioning a task tracker.. When I see the job tracker log, then it shows that it has taken the appropriate file as include file. But i am able to run the task tracker on a machine which is not there in include file. I have worked with the new property as well mapred.jobtracker.hosts.file , but it didnt worked as well. Please advise me. Regards, Som Shekhar Sharma +91-8197243810
Fwd: HDFS multi-tenancy and federation
Hi, I would like to know the following. 1) Can there be multiple namespaces in a single namenode? is it recommended? (I'm having a multi-tenant environment in mind) 2) Let's say I have a federated namespace/namenodes. There are two namenodes A /namespace A1 and namenode B/namespace B1, and have 3 datanodes. Can someone from namespace A1, access the datanode's data in anyway (hacking) belonging to namespace B1. If not how is it handled? After going through a lot of reference, my understanding on HDFS multi-tenancy and federation is that for multi-tenancy what we could do is use file/folder permissions (u,g,o) and ACL's. Or we could dedicate a namespace per tenant. The issue here is that a namenode (active namenode, passive namenode and secondary namenode) has to be assigned per tenant. Is there any other way that multi tenancy can be achieved? On federation, let's say I have a namenode for /marketing and another for /finance. Lets say that marketing bears the most load. How can we load balance this? is it possible? Appreciate any help on this. Regards, Shani.
