Hi Fabian, This make sense now.
I would like to avoid zookeeper if possible. Is there any way to avoid this to achieve HA ? I see that DataStax Enterprise achieve this availability for Spark Master without using Zookeeper. https://academy.datastax.com/demos/how-spark-master-high-availability-works-dse Is this possible to achieve in Flink also ? Cheers On Wed, Nov 25, 2015 at 4:11 PM, Fabian Hueske <fhue...@gmail.com> wrote: > YARN is not a replacement for Zookeeper. Zookeeper is mandatory to run > Flink in high-availability mode and takes care of leader (JobManager) > election and meta-data persistance. > > With YARN, Flink can automatically start new Taskmanagers (and > JobManagers) to compensate for failures. In cluster mode, you need stand-by > TMs and JMs and manually take care that these are "filled-up" again in case > of a failure. > > 2015-11-25 10:06 GMT+01:00 Welly Tambunan <if05...@gmail.com>: > >> Hi Fabian, >> >> Interesting ! >> >> However YARN is still tightly couple to HDFS, is that seems wasteful to >> use only YARN without Hadoop ? >> >> Currently we are using Cassandra and CFS ( cass file system ) >> >> >> Cheers >> >> On Wed, Nov 25, 2015 at 3:51 PM, Fabian Hueske <fhue...@gmail.com> wrote: >> >>> A strong argument for YARN mode can be the isolation of multiple users >>> and jobs. You can easily start a new Flink cluster for each job or user. >>> However, this comes at the price of resource (memory) fragmentation. YARN >>> mode does not use memory as effective as cluster mode. >>> >>> 2015-11-25 9:46 GMT+01:00 Ufuk Celebi <u...@apache.org>: >>> >>>> > On 25 Nov 2015, at 02:35, Welly Tambunan <if05...@gmail.com> wrote: >>>> > >>>> > Hi All, >>>> > >>>> > I would like to know if there any feature differences between using >>>> Standalone Cluster vs YARN ? >>>> > >>>> > Until now we are using Standalone cluster for our jobs. >>>> > Is there any added value for using YARN ? >>>> > >>>> > We don't have any hadoop infrastructure in place right now but we can >>>> provide that if there's some value to that. >>>> >>>> There are no features, which only work on YARN or in standalone >>>> clusters. YARN mode is essentially starting a standalone cluster in YARN >>>> containers. >>>> >>>> In failure cases I find YARN more convenient, because it takes care of >>>> restarting failed task manager processes/containers for you. >>>> >>>> – Ufuk >>>> >>>> >>> >> >> >> -- >> Welly Tambunan >> Triplelands >> >> http://weltam.wordpress.com >> http://www.triplelands.com <http://www.triplelands.com/blog/> >> > > -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>
