Is zookeeper required for the tech stack? If samza does all of its checkpointing on kafka, I may be able to save us some money by eliminating the ZK cluster.
Jeremiah Adams Software Engineer www.helixeducation.com<http://www.helixeducation.com/> Blog<http://www.helixeducation.com/blog/> | Twitter<https://twitter.com/HelixEducation> | Facebook<https://www.facebook.com/HelixEducation> | LinkedIn<http://www.linkedin.com/company/3609946> From: Prateek Maheshwari <prateek...@gmail.com> Date: Monday, March 11, 2019 at 6:31 PM To: "dev@samza.apache.org" <dev@samza.apache.org>, Jeremiah Adams <jad...@helixeducation.com> Subject: Re: Backing Kafka/Yarn/Zookeeper version for Samza 1.0.0? Hi Jeremiah, We're in the process of upgrading Samza to use Kafka clients version 2.0, and YARN client version 2.9. This should be available in the next release (version 1.2). In the mean time, Kafka 0.11 and YARN 2.6 / 2.7 are the recommended versions. Can you clarify what you mean about hard requirements for Zookeeper since checkpointing? Thanks, Prateek On Fri, Mar 8, 2019 at 11:27 AM Jeremiah Adams <jad...@helixeducation.com<mailto:jad...@helixeducation.com>> wrote: I am in the process of updating our stack. I’m not seeing documentation on what versions of Kafka, Zookeeper and Yarn Samza 1.0.0 should be run on. Best I can tell is a 2.6.1 dependency on Hadoop jars and 0.11.0.2 kafka jars. If these jars are meant to match versions of the software, they are dated. Hadoop is up to version 3.2.0 Kafka is up to version 2.1.1 Any advice concerning versions of the components? Also we are planning on moving all Samza jobs to standalone and eliminating the Yarn dependencies. I prefer taking iterative steps so want to get the updates done before moving to containers. Also, does Samza have any hard requirements for Zookeeper since checkpointing was implemented? Jeremiah Adams Software Engineer www.helixeducation.com<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bnWsxM_w6KChRx8CY_UgrU5RmcwzgGL3Po63B7rJIXeNyMBLKYpptY6Rl-f5kb6p2A~~><http://www.helixeducation.com/<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~>> Blog<http://www.helixeducation.com/blog/<https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~>> | Twitter<https://twitter.com/HelixEducation<https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~>> | Facebook<https://www.facebook.com/HelixEducation<https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~>> | LinkedIn<http://www.linkedin.com/company/3609946<https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0>>
