Hi Welly, at the moment Flink only supports HA via ZooKeeper. However, there is no limitation to use another system. The only requirement is that this system allows you to find a consensus among multiple participants and to retrieve the community decision. If this is possible, then it can be integrated into Flink to serve as an alternative HA backend.
Cheers, Till On Wed, Nov 25, 2015 at 10:30 AM, Maximilian Michels <m...@apache.org> wrote: > Hi Welly, > > > However YARN is still tightly couple to HDFS, is that seems wasteful to > use only YARN without Hadoop ? > > I wouldn't say tightly coupled. You can use YARN without HDFS. To work > with YARN properly, you would have to setup another distributed file > system like xtreemfs. Or use the one provided with the AWS or Google > Cloud Platform. You can tell Hadoop which file system to use by > modifying "fs.default.name" in the Hadoop config. > > Cheers, > Max > > On Wed, Nov 25, 2015 at 10:06 AM, Welly Tambunan <if05...@gmail.com> > wrote: > > Hi Fabian, > > > > Interesting ! > > > > However YARN is still tightly couple to HDFS, is that seems wasteful to > use > > only YARN without Hadoop ? > > > > Currently we are using Cassandra and CFS ( cass file system ) > > > > > > Cheers > > > > On Wed, Nov 25, 2015 at 3:51 PM, Fabian Hueske <fhue...@gmail.com> > wrote: > >> > >> A strong argument for YARN mode can be the isolation of multiple users > and > >> jobs. You can easily start a new Flink cluster for each job or user. > >> However, this comes at the price of resource (memory) fragmentation. > YARN > >> mode does not use memory as effective as cluster mode. > >> > >> 2015-11-25 9:46 GMT+01:00 Ufuk Celebi <u...@apache.org>: > >>> > >>> > On 25 Nov 2015, at 02:35, Welly Tambunan <if05...@gmail.com> wrote: > >>> > > >>> > Hi All, > >>> > > >>> > I would like to know if there any feature differences between using > >>> > Standalone Cluster vs YARN ? > >>> > > >>> > Until now we are using Standalone cluster for our jobs. > >>> > Is there any added value for using YARN ? > >>> > > >>> > We don't have any hadoop infrastructure in place right now but we can > >>> > provide that if there's some value to that. > >>> > >>> There are no features, which only work on YARN or in standalone > clusters. > >>> YARN mode is essentially starting a standalone cluster in YARN > containers. > >>> > >>> In failure cases I find YARN more convenient, because it takes care of > >>> restarting failed task manager processes/containers for you. > >>> > >>> – Ufuk > >>> > >> > > > > > > > > -- > > Welly Tambunan > > Triplelands > > > > http://weltam.wordpress.com > > http://www.triplelands.com >
