Terence, Thanks for the detailed answer.
What if client app does not need state recovery and other features implemented with use of Zk? It was a little surprising to hear about Zk and Kafka since they are complex systems in themselves, and Twill's primary goal is to lose complexity (of Yarn), but not acquire two new ones. Any chance Twill could make Zk dependency optional? 2016-01-27 20:41 GMT+02:00 Terence Yim <[email protected]>: > Hi Ivan, > > Twill relies on ZK for couple core functionalities, such as application > state recovery, service discovery and messaging, hence would be difficult > for the current Twill version to run without ZK. > > Twill doesn't require Kafka to run, however, each AM starts an embedded > Kafka (running in the same JVM as the AM) for the log collection purpose > for all the TwillRunnables controlled by that AM. We have a JIRA TWILL-147 > <https://issues.apache.org/jira/browse/TWILL-147> to allow using an > external Kafka/turning off log collection support. > > For the filesystem, it only needs a distributed file system which is > accessible through the HDFS API, but not necessarily HDFS implementation. > We've tested Twill can works on MapR FS and Azure FS as well. I believe the > same should go for AWS or GCS. > > Terence > > On Wed, Jan 27, 2016 at 12:22 AM, Ivan Balashov <[email protected]> > wrote: > > > Hi, > > > > Is it possible to use Twill with bare Yarn cluster, without ZK or Kafka? > > Also, does Yarn cluster need to have HDFS in order to benefit from Twill? > > Or buckets like AWS or GCS can be enough? > > > > Thanks, > > >
