YARN is not a replacement for Zookeeper. Zookeeper is mandatory to run Flink in high-availability mode and takes care of leader (JobManager) election and meta-data persistance.
With YARN, Flink can automatically start new Taskmanagers (and JobManagers) to compensate for failures. In cluster mode, you need stand-by TMs and JMs and manually take care that these are "filled-up" again in case of a failure. 2015-11-25 10:06 GMT+01:00 Welly Tambunan <if05...@gmail.com>: > Hi Fabian, > > Interesting ! > > However YARN is still tightly couple to HDFS, is that seems wasteful to > use only YARN without Hadoop ? > > Currently we are using Cassandra and CFS ( cass file system ) > > > Cheers > > On Wed, Nov 25, 2015 at 3:51 PM, Fabian Hueske <fhue...@gmail.com> wrote: > >> A strong argument for YARN mode can be the isolation of multiple users >> and jobs. You can easily start a new Flink cluster for each job or user. >> However, this comes at the price of resource (memory) fragmentation. YARN >> mode does not use memory as effective as cluster mode. >> >> 2015-11-25 9:46 GMT+01:00 Ufuk Celebi <u...@apache.org>: >> >>> > On 25 Nov 2015, at 02:35, Welly Tambunan <if05...@gmail.com> wrote: >>> > >>> > Hi All, >>> > >>> > I would like to know if there any feature differences between using >>> Standalone Cluster vs YARN ? >>> > >>> > Until now we are using Standalone cluster for our jobs. >>> > Is there any added value for using YARN ? >>> > >>> > We don't have any hadoop infrastructure in place right now but we can >>> provide that if there's some value to that. >>> >>> There are no features, which only work on YARN or in standalone >>> clusters. YARN mode is essentially starting a standalone cluster in YARN >>> containers. >>> >>> In failure cases I find YARN more convenient, because it takes care of >>> restarting failed task manager processes/containers for you. >>> >>> – Ufuk >>> >>> >> > > > -- > Welly Tambunan > Triplelands > > http://weltam.wordpress.com > http://www.triplelands.com <http://www.triplelands.com/blog/> >
