Thanks Pragya. Srikanth, good point about the cluster version requirements.
It is definitely something to keep in mind. Thanks for the heads up. Venkat On 10/26/15, 3:07 AM, "pragya mittal" <[email protected]> wrote: >Typo : >Oozie version is 4.2 and RM HA testing is still in progress. > >On Sun, Oct 25, 2015 at 9:25 AM, pragya mittal <[email protected]> >wrote: > >> Hi Venkat, >> >> From past three months, Inmobi has been using Oozie in uber mode. >> Currently we are using falcon-0.7 with oozie-3.2 (along with several >> patches specific to our use case). Following use cases have been tested and >> are active in our production environment. >> >> 1. NN HA/RM HA >> 2. Oozie ubermode for launcher job >> 3. Multi cluster with prism >> 4. Sanity of falcon features with oozie(in uber mode). >> >> After falcon-0.8 release we plan to go forward with Oozie HA feature. >> >> Please let me know if you have any more concerns. >> >> Regards, >> Pragya Mittal >> >> On Sun, Oct 25, 2015 at 8:15 AM, Srikanth Sundarrajan <[email protected] >> > wrote: >> >>> Am guessing venkat you meant to refer FALCON-1555. Good check list to go >>> through for each release. >>> >>> Specifically on >>> >> 6. Multi cluster managed with Oozie (with single Falcon instance) >>> >>> Since we haven't particularly done anything by isolating the classes and >>> their dependencies for each cluster with independent class loader, I think >>> this would work only as long as all the clusters are compatible. If things >>> were to change say with hadoop-3 for ex, this may no longer work. >>> >>> Regards >>> Srikanth Sundarrajan >>> >>> > From: [email protected] >>> > Date: Sun, 25 Oct 2015 05:29:20 +0530 >>> > Subject: Re: Oozie uber mode and other modes of running the software >>> > To: [email protected] >>> > >>> > Venkat, >>> > >>> > Sounds good, let's discuss it in the next sync up. For the benefit of >>> > everyone on the mailing list, I will summarise our use cases. Yes, we >>> > (Inmobi) use Oozie's uber mode in our production environment(IIRC we are >>> > using it since Apache Falcon 0.7). We do have several thousand jobs >>> running >>> > per day(several at minutely frequencies) and a heavily utilised >>> cluster, so >>> > this makes sense for us. We do test all Oozie releases, though sometimes >>> > our Oozie releases are not exactly aligned with Oozie's general release >>> - >>> > it might contain some extra patches, might be from trunk etc. and our >>> > testing might be specific to features being used at Inmobi. In fact last >>> > time Pragya Mittal did the testing for both Apache Falcon 0.7 and Oozie. >>> > She can add more on this. >>> > >>> > List of scenarios looks comprehensive. Just curious, what storage >>> backends >>> > are other users using for titan? >>> > >>> > >>> > Cheers >>> > Ajay Yadava >>> > >>> > On Sat, Oct 24, 2015 at 9:20 PM, Venkat Ranganathan < >>> > [email protected]> wrote: >>> > >>> > > FALCON-1551 was interesting for two reasons - that we are still >>> caught by >>> > > the FS caching issue (this should have been dealt by most of the >>> projects >>> > > one time or another), but the use of Oozie uber mode - I am wondering >>> if we >>> > > have done uber mode testing generally - Is this the mode that is used >>> in >>> > > your environments – It makes sense when thousands of Oozie jobs are >>> run – >>> > > saving one container. I would like to understand more about your >>> > > experience with uber mode if you have – may be in the next Falcon >>> sync up. >>> > > >>> > > I think we have create a list of modes that we expect the product to >>> run >>> > > before we introduce changes generally – Given that there are >>> different sets >>> > > of execution modes that we want to test. Please add more on the >>> scenarios >>> > > that I am most likely missing. Let us discuss this also if we have >>> time. >>> > > >>> > > 1. Security (Kerberos and Wire encryption) >>> > > 2. NN HA/RM HA >>> > > 3. Oozie HA >>> > > 4. Oozie ubermode for launcher job >>> > > 5. Multi cluster with prism >>> > > 6. Multi cluster managed with Oozie (with single Falcon instance) >>> > > 7. Hadoop/Hive/Oozie versions and the implications (S3 and Azure >>> blob >>> > > storage went out of hadoop-common in 2.6.0+ and the implications). >>> > > >>> > > >>> > > Thanks >>> > > >>> > > Venkat >>> > > >>> > > >>> > >>> > -- >>> > _____________________________________________________________ >>> > The information contained in this communication is intended solely for >>> the >>> > use of the individual or entity to whom it is addressed and others >>> > authorized to receive it. It may contain confidential or legally >>> privileged >>> > information. If you are not the intended recipient you are hereby >>> notified >>> > that any disclosure, copying, distribution or taking any action in >>> reliance >>> > on the contents of this information is strictly prohibited and may be >>> > unlawful. If you have received this communication in error, please >>> notify >>> > us immediately by responding to this email and then delete it from your >>> > system. The firm is neither liable for the proper and complete >>> transmission >>> > of the information contained in this communication nor for any delay in >>> its >>> > receipt. >>> >>> >> >>
