Hi Ivan, Twill relies on ZK for couple core functionalities, such as application state recovery, service discovery and messaging, hence would be difficult for the current Twill version to run without ZK.
Twill doesn't require Kafka to run, however, each AM starts an embedded Kafka (running in the same JVM as the AM) for the log collection purpose for all the TwillRunnables controlled by that AM. We have a JIRA TWILL-147 <https://issues.apache.org/jira/browse/TWILL-147> to allow using an external Kafka/turning off log collection support. For the filesystem, it only needs a distributed file system which is accessible through the HDFS API, but not necessarily HDFS implementation. We've tested Twill can works on MapR FS and Azure FS as well. I believe the same should go for AWS or GCS. Terence On Wed, Jan 27, 2016 at 12:22 AM, Ivan Balashov <[email protected]> wrote: > Hi, > > Is it possible to use Twill with bare Yarn cluster, without ZK or Kafka? > Also, does Yarn cluster need to have HDFS in order to benefit from Twill? > Or buckets like AWS or GCS can be enough? > > Thanks, >
