Fw: new message
Hello! New message, please read <http://ribbonrouge.com/grew.php?ewcxz> twinkle sachdeva
Fw: new message
Hello! New message, please read <http://pickthetopic.com/sort.php?rh> twinkle sachdeva
Fw: new message
Hello! New message, please read <http://mobile-pharma.com/any.php?zavf> twinkle sachdeva
Fw: new message
Hello! New message, please read <http://millenniumgroups.co.in/longer.php?m3> twinkle sachdeva
Fw: new message
Hello! New message, please read <http://javierclarkdiputado.com/lives.php?ai> twinkle sachdeva
Fw: new message
Hello! New message, please read <http://KathyRogilliopianotuning.com/nothing.php?0> twinkle sachdeva
Fw: new message
Hello! New message, please read <http://credemo.azurewebsites.net/year.php?g> twinkle sachdeva
Fw: new message
Hello! New message, please read <http://bannereciarj.com/idea.php?9m> twinkle sachdeva
Fw: new message
Hello! New message, please read <http://mobile-pharma.com/years.php?i60l> twinkle sachdeva
Re: MultithreadedMapper - Sharing Data Structure
Hi, We have been using the jvm reuse feature for the same reason of sharing the same structure across multiple Map Tasks. Multithreaded Map task does that partially, as within the multiple threads, same copy is used. Depending upon the hardware availability, one can get the same performance. Thanks, On Mon, Aug 24, 2015 at 1:37 PM, Harsh J wrote: > The MultiThreadedMapper won't solve your problem, as all it does is run > parallel maps within the same map task JVM as a non-MT one. Your data > structure won't be shared across the different map task JVMs on the host, > but just within the map tasks's own multiple threads running the map() > function over input records. > > Wouldn't doing reduce-side join for larger files be much faster? > > On Sun, Aug 23, 2015 at 5:08 AM Pedro Magalhaes > wrote: > >> I am developig a job that has 30B of records in the input path. (File A) >> I need to filter these records using another file that can have 30K to >> 180M of records. (File B) >> So fo each record in File A, i will make a lookup in File B. >> I am using distributed cache to share the File B. The problem is that if >> the File B is too large (for example 180 M of records), i spend too much >> time (CPU processing) allocating it in a hashmap. I make this allocation to >> each map task. >> >> In hadoop 2.X the jvm reuse was discontinued. So i am think in use >> MultithreadedMapper, >> making the hashmap thread-safe, and sharing this read-only structure across >> the mappers. >> >> Is this a good approach? >> >> >> >> >
Re: Able to connect to the same app master
Hi, Thanks Mirko. It is doing more than just monitoring the progress. It is giving some more commands from the user which affects the next execution of the code. Regards, Twinkle On Sat, Aug 1, 2015 at 9:30 PM, mirko.kaempf wrote: > Hey, > > your APPMaster could listen on a configurable Port for incomming requests > an from here you start to create your own mainenance module for this > specific APP. > > What is the client doing on the APP Master after submitting it into the > Yarn-world? Is it just about getting the progress of the APP or more > use-case specific? > > Cheers, > Mirko > > > Von Samsung Mobile gesendet > > > ---- Ursprüngliche Nachricht > Von: twinkle sachdeva > Datum:01.08.2015 07:34 (GMT-08:00) > An: USers Hadoop > Cc: > Betreff: Able to connect to the same app master > > Hi, > > Is it possible to connect the app master of an application id if the > current yarn client jvm goes down. > > To put this as a sequence of events, this is what I am looking for: > > 1. Launch an application, you get an app id in return > 2. Machine at which client was running, gets rebooted. This will lead to > jvm of yarn client to get killed. > 3. launch Yarn client in a mode, where i can specify the application id, > so that i can reconnect to the app master of that application id. > > If there is any other way of achieving the same, by some other mechanism, > please share over here. > > Thanks, > Twinkle >
Able to connect to the same app master
Hi, Is it possible to connect the app master of an application id if the current yarn client jvm goes down. To put this as a sequence of events, this is what I am looking for: 1. Launch an application, you get an app id in return 2. Machine at which client was running, gets rebooted. This will lead to jvm of yarn client to get killed. 3. launch Yarn client in a mode, where i can specify the application id, so that i can reconnect to the app master of that application id. If there is any other way of achieving the same, by some other mechanism, please share over here. Thanks, Twinkle
Re: Can't find map or reduce logs when a job ends.
Hi, Please try following: yarn logs -applicationId application_1426267324367_0005 Thanks, On Tue, Mar 17, 2015 at 8:23 AM, Ranadip Chatterjee wrote: > Is the job history server up and running on the right host and port? > Please check the job history server logs, if so? A common reason is for the > owner of job history server to not have read permission on the logs or for > the map reduce process owners to not have write permission in the job > history log location. The error should show up in the job history server > logs. > > Ranadip > On 13 Mar 2015 18:07, "xeonmailinglist-gmail" > wrote: > >> Hi, >> >> With this configuration in mapreduce (see [1] and [2]), I can’t see the >> map and reduce logs of the job when it ends. When I try to look to the >> history, I get this error Not Found: job_1426267326549_0005. But if I >> list the log dir in the hdfs (see [3]), I have some logs about the job, but >> not logs about the map or reduce tasks. >> >> Why I can’t see the map and reduce logs? Am I missing some configuration? >> >> [1] configuration in mapred-site.xml >> >> >> >> mapreduce.framework.name >> yarn >> >> >> mapreduce.jobhistory.address >> hadoop-coc-1:10020 >> >> >> >> mapreduce.jobhistory.webapp.address >> hadoop-coc-1:19888 >> >> >> mapreduce.jobhistory.max-age-ms >> 180 >> >> >> >> [2] configuration in yarn-site.xml >> >> xubuntu@hadoop-coc-1:~/Programs/hadoop$ cat etc/hadoop/yarn-site.xml >> >> >> >> >> >> yarn.nodemanager.aux-services >> mapreduce_shuffle >> >> >> >> yarn.nodemanager.aux-services.mapreduce.shuffle.class >> org.apache.hadoop.mapred.ShuffleHandler >> >> >> yarn.log-aggregation-enable >> true >> >> >> yarn.nodemanager.remote-app-log-dir >> /app-logs >> >> >> [3] List logs in hadoop >> >> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /app-logs/xeon/logs/ >> Java HotSpot(TM) Client VM warning: You have loaded library >> /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have >> disabled stack guard. The VM will try to fix the stack guard now. >> It's highly recommended that you fix the library with 'execstack -c >> ', or link it with '-z noexecstack'. >> 15/03/13 13:58:06 WARN util.NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> Found 4 items >> drwxrwx--- - xeon supergroup 0 2015-03-13 13:35 >> /app-logs/xeon/logs/application_1426267324367_0002 >> drwxrwx--- - xeon supergroup 0 2015-03-13 13:37 >> /app-logs/xeon/logs/application_1426267324367_0003 >> drwxrwx--- - xeon supergroup 0 2015-03-13 13:44 >> /app-logs/xeon/logs/application_1426267324367_0004 >> drwxrwx--- - xeon supergroup 0 2015-03-13 13:47 >> /app-logs/xeon/logs/application_1426267324367_0005 >> >> >> >> -- >> -- >> >>
Re: Node manager contributing to one queue's resources
Thanks for the sharing another view point, will look around it. But seems like, we might need something more specific. On Thu, Feb 26, 2015 at 2:43 PM, Rohith Sharma K S < rohithsharm...@huawei.com> wrote: > Hi > > > > If you are using CapacityScheduler, can you try using > DominantResourceCalculator i.e configuring below property value in > capacity-scheduler.xml file. > > > > > > yarn.scheduler.capacity.resource-calculator > > org.apache.hadoop.yarn.util.resource. > DominantResourceCalculator > > > > > > The basic Idea it works is as follows ‘if user A runs CPU-heavy tasks > and user B runs memory-heavy tasks, it attempts to equalize CPU share of > user A with Memory-share of user B’ > > > > See Java Doc > > > https://apache.googlesource.com/hadoop-common/+/60e3b885ba8344d9f448202f5f2c290b5606ff8f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/DominantResourceCalculator.java > > > > I think this may help you!!! > > > > Thanks & Regards > > Rohith Sharma K S > > > > *From:* twinkle sachdeva [mailto:twinkle.sachd...@gmail.com] > *Sent:* 26 February 2015 14:05 > *To:* USers Hadoop > *Subject:* Node manager contributing to one queue's resources > > > > Hi, > > > > I have to run two kind of applications, one requiring less cores but more > memory ( Application_High_Mem) and another application which requires more > cores but less memory ( Application_High_Core). > > > > I can use specific queues to submit them to, but that can lead to one node > contributing to only one one such application and having some part of > resources as idle. > > > > Is there a way, let's say extending concept of queues at node manager > level to do this or some other way, in which i can achieve it in YARN? > > > > Thanks, > > Twinkle >
Node manager contributing to one queue's resources
Hi, I have to run two kind of applications, one requiring less cores but more memory ( Application_High_Mem) and another application which requires more cores but less memory ( Application_High_Core). I can use specific queues to submit them to, but that can lead to one node contributing to only one one such application and having some part of resources as idle. Is there a way, let's say extending concept of queues at node manager level to do this or some other way, in which i can achieve it in YARN? Thanks, Twinkle
