Hi Welly, > However YARN is still tightly couple to HDFS, is that seems wasteful to use > only YARN without Hadoop ?
I wouldn't say tightly coupled. You can use YARN without HDFS. To work with YARN properly, you would have to setup another distributed file system like xtreemfs. Or use the one provided with the AWS or Google Cloud Platform. You can tell Hadoop which file system to use by modifying "fs.default.name" in the Hadoop config. Cheers, Max On Wed, Nov 25, 2015 at 10:06 AM, Welly Tambunan <if05...@gmail.com> wrote: > Hi Fabian, > > Interesting ! > > However YARN is still tightly couple to HDFS, is that seems wasteful to use > only YARN without Hadoop ? > > Currently we are using Cassandra and CFS ( cass file system ) > > > Cheers > > On Wed, Nov 25, 2015 at 3:51 PM, Fabian Hueske <fhue...@gmail.com> wrote: >> >> A strong argument for YARN mode can be the isolation of multiple users and >> jobs. You can easily start a new Flink cluster for each job or user. >> However, this comes at the price of resource (memory) fragmentation. YARN >> mode does not use memory as effective as cluster mode. >> >> 2015-11-25 9:46 GMT+01:00 Ufuk Celebi <u...@apache.org>: >>> >>> > On 25 Nov 2015, at 02:35, Welly Tambunan <if05...@gmail.com> wrote: >>> > >>> > Hi All, >>> > >>> > I would like to know if there any feature differences between using >>> > Standalone Cluster vs YARN ? >>> > >>> > Until now we are using Standalone cluster for our jobs. >>> > Is there any added value for using YARN ? >>> > >>> > We don't have any hadoop infrastructure in place right now but we can >>> > provide that if there's some value to that. >>> >>> There are no features, which only work on YARN or in standalone clusters. >>> YARN mode is essentially starting a standalone cluster in YARN containers. >>> >>> In failure cases I find YARN more convenient, because it takes care of >>> restarting failed task manager processes/containers for you. >>> >>> – Ufuk >>> >> > > > > -- > Welly Tambunan > Triplelands > > http://weltam.wordpress.com > http://www.triplelands.com
