@Jake: Yes. We removed the migration code (for 0.9 to 0.10) in the 0.11 release, I believe.
@XiaoChuan: As per Jagadish's recommendation, if you have changelog backed stores, you should upgrade from 0.9.1 to 0.10.0 before upgrading to samza 0.12.0. I checked with LinkedIn's internal release notes. The most significant change listed is adding a new configuration *job.coordinator.system*. This system can be the same as your currently configured checkpoint system (task.checkpoint.system). I am assuming you are using KafkaCheckpointManagerFactory. If you are using other custom checkpoint managers, the migration may be more involved. Please let us know and we can try to help you out. Feel free to email us if you have more questions. Cheers! Navina On Mon, Mar 27, 2017 at 10:07 AM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > Good observation Jake! > > The code for migration was removed in Samza 11. The migration would read > change-log offsets from the checkpoint topic and write them to the > coordinator stream. > > If you're using change-logged stores, I'd recommend upgrading from 0.9.1 to > 0.10.0 first. > Otherwise, you will loose offsets for change-logged stores. > > I suspect you should be okay for 0.10.0 to 0.12 upgrade. > > On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes <jacob.m...@gmail.com> wrote: > > > As I recall, samza 0.10 introduced the coordinator stream and there was > > code to do an automatic migration to use that feature. @navina, @yi, do > you > > know if that migration code is still in samza 12? > > > > If not, then it's probably better to update from 0.9.1 to 0.10.0 and then > > to 0.12.0. I don't think there were any changes requiring migration > between > > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is probably less > of > > an issue. > > > > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman < > > jagadish1...@gmail.com> wrote: > > > > > Hi Xiaochuan, > > > > > > >> Do I need to upgrade Kafka and/or YARN? > > > > > > *Yarn version:* > > > > > > - Samza 0.12 supports Yarn 2.6.1 and 2.7.1. > > > - If you already have 2.6.0 installed (as you have said), I believe > > you > > > will be fine. (but I'm not sure) > > > > > > *Kafka version: * > > > > > > - Samza 0.12 upgraded the version of Kafka to 0.10. > > > - If your Kafka brokers are on an older version of Kafka, you should > > > upgrade them to use at-least 0.10. Kafka clients are usually > > > incompatible with older versions of brokers. > > > > > > *Java version: * > > > > > > > > > > > > - Samza 0.12 binaries are compiled using Java 8. Hence, they cannot > > be > > > run on older versions of the Java run-time. > > > > > > > > > >> I'm extremely new to Samza in terms of operations aspect. I'm not > sure > > > what > > > information would be relevant in this case so please ask away. > > > > > > I'd first start by upgrading the Kafka brokers (assuming you're on Java > > 8+ > > > already). > > > Let us know how the migration goes! > > > > > > Thanks, > > > Jagadish > > > > > > > > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu <xiaochuan...@kik.com> > > > wrote: > > > > > > > Hi, > > > > > > > > What are the general steps for upgrading Samza from 0.9 to 0.12? > > > > Do I need to upgrade Kafka and/or YARN? > > > > > > > > I don't know how Samza was setup initially but we currently have the > > > > following setup: > > > > > > > > Samza version: 0.9.1 > > > > YARN version: Hadoop 2.6.0-cdh5.4.8 > > > > Kafka version: 0.9.0.1 > > > > > > > > I think installation of Kafka and YARN were managed through Puppet. > > > > I'm extremely new to Samza in terms of operations aspect. I'm not > sure > > > what > > > > information would be relevant in this case so please ask away. > > > > > > > > Thanks, > > > > Xiaochuan Yu > > > > > > > > > > > > > > > > -- > > > Jagadish V, > > > Graduate Student, > > > Department of Computer Science, > > > Stanford University > > > > > > > > > -- > Jagadish V, > Graduate Student, > Department of Computer Science, > Stanford University >
