Is there some sort of document on how to upgrade Samza through various versions like the page here for Kafka: https://kafka.apache.org/documentation/#upgrade ? Having something like this would be ideal. On Thu, Mar 30, 2017 at 1:51 PM Thomas Becker <tobec...@tivo.com> wrote:
> Thanks for the reply Yi, and I apologize if I came off a bit snarky. > I'm not sure I agree with the policy (removing migration code and > wanting people to upgrade seem at odds to me), but minimally I think we > should not assume people are upgrading to each new Samza version. We > have done so when features or fixes warrant, and even then on a per-job > basis, and I would expect this is a common practice. > > -Tommy > > On Thu, 2017-03-30 at 09:50 -0700, Yi Pan wrote: > > Hi, Thomas, > > > > Sorry to hear that you were hit by the removal of migration in Samza > > 0.11. > > The reason we removed it is following a deprecate-removal policy in > > two > > versions. We are not aware that people still using 0.9 after we > > released > > 0.11 and were not expecting a direct upgrade from 0.9 to 0.12. > > Document can > > be better to capture that. We are making changes to the design > > proposal > > s.t. it is more transparent and open to the whole community, through > > the > > newly proposed SEP process. These kind of breaking changes will go > > through > > the SEP discuss-vote process in the future and hopefully capture all > > these > > kind of concerns earlier. > > > > Best! > > > > -Yi > > > > On Thu, Mar 30, 2017 at 7:45 AM, Thomas Becker <tobec...@tivo.com> > > wrote: > > > > > > > > Yes, we were burned by this. The changelog mapping will be > > > regenerated > > > instead of migrated and the result will completely hose the job > > > (because the mapping was not generated deterministically in > > > previous > > > versions of Samza). I don't understand why the migration code was > > > removed but it was, and to the best of my knowledge the necessity > > > to > > > not skip version 0.10.0 when upgrading was not documented, let > > > alone > > > enforced. > > > > > > On Mon, 2017-03-27 at 10:07 -0700, Jagadish Venkatraman wrote: > > > > > > > > Good observation Jake! > > > > > > > > The code for migration was removed in Samza 11. The migration > > > > would > > > > read > > > > change-log offsets from the checkpoint topic and write them to > > > > the > > > > coordinator stream. > > > > > > > > If you're using change-logged stores, I'd recommend upgrading > > > > from > > > > 0.9.1 to > > > > 0.10.0 first. > > > > Otherwise, you will loose offsets for change-logged stores. > > > > > > > > I suspect you should be okay for 0.10.0 to 0.12 upgrade. > > > > > > > > On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes <jacob.m...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > As I recall, samza 0.10 introduced the coordinator stream and > > > > > there > > > > > was > > > > > code to do an automatic migration to use that feature. @navina, > > > > > @yi, do you > > > > > know if that migration code is still in samza 12? > > > > > > > > > > If not, then it's probably better to update from 0.9.1 to > > > > > 0.10.0 > > > > > and then > > > > > to 0.12.0. I don't think there were any changes requiring > > > > > migration > > > > > between > > > > > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is > > > > > probably > > > > > less of > > > > > an issue. > > > > > > > > > > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman < > > > > > jagadish1...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Xiaochuan, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do I need to upgrade Kafka and/or YARN? > > > > > > *Yarn version:* > > > > > > > > > > > > - Samza 0.12 supports Yarn 2.6.1 and 2.7.1. > > > > > > - If you already have 2.6.0 installed (as you have said), > > > > > > I > > > > > > believe > > > > > you > > > > > > > > > > > > > > > > > > will be fine. (but I'm not sure) > > > > > > > > > > > > *Kafka version: * > > > > > > > > > > > > - Samza 0.12 upgraded the version of Kafka to 0.10. > > > > > > - If your Kafka brokers are on an older version of Kafka, > > > > > > you > > > > > > should > > > > > > upgrade them to use at-least 0.10. Kafka clients are > > > > > > usually > > > > > > incompatible with older versions of brokers. > > > > > > > > > > > > *Java version: * > > > > > > > > > > > > > > > > > > > > > > > > - Samza 0.12 binaries are compiled using Java 8. Hence, > > > > > > they > > > > > > cannot > > > > > be > > > > > > > > > > > > > > > > > > run on older versions of the Java run-time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm extremely new to Samza in terms of operations aspect. > > > > > > > > I'm > > > > > > > > not sure > > > > > > what > > > > > > information would be relevant in this case so please ask > > > > > > away. > > > > > > > > > > > > I'd first start by upgrading the Kafka brokers (assuming > > > > > > you're > > > > > > on Java > > > > > 8+ > > > > > > > > > > > > > > > > > > already). > > > > > > Let us know how the migration goes! > > > > > > > > > > > > Thanks, > > > > > > Jagadish > > > > > > > > > > > > > > > > > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu <xiaochuan.yu@k > > > > > > ik.c > > > > > > om> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > What are the general steps for upgrading Samza from 0.9 to > > > > > > > 0.12? > > > > > > > Do I need to upgrade Kafka and/or YARN? > > > > > > > > > > > > > > I don't know how Samza was setup initially but we currently > > > > > > > have the > > > > > > > following setup: > > > > > > > > > > > > > > Samza version: 0.9.1 > > > > > > > YARN version: Hadoop 2.6.0-cdh5.4.8 > > > > > > > Kafka version: 0.9.0.1 > > > > > > > > > > > > > > I think installation of Kafka and YARN were managed through > > > > > > > Puppet. > > > > > > > I'm extremely new to Samza in terms of operations aspect. > > > > > > > I'm > > > > > > > not sure > > > > > > what > > > > > > > > > > > > > > > > > > > > > information would be relevant in this case so please ask > > > > > > > away. > > > > > > > > > > > > > > Thanks, > > > > > > > Xiaochuan Yu > > > > > > > > > > > > > > > > > > > -- > > > > > > Jagadish V, > > > > > > Graduate Student, > > > > > > Department of Computer Science, > > > > > > Stanford University > > > > > > > > > > > > > -- > > > > > > > > > Tommy Becker > > > > > > Senior Software Engineer > > > > > > O +1 919.460.4747 > > > > > > tivo.com > > > > > > > > > ________________________________ > > > > > > This email and any attachments may contain confidential and > > > privileged > > > material for the sole use of the intended recipient. Any review, > > > copying, > > > or distribution of this email (or any attachments) by others is > > > prohibited. > > > If you are not the intended recipient, please contact the sender > > > immediately and permanently delete this email and any attachments. > > > No > > > employee or agent of TiVo Inc. is authorized to conclude any > > > binding > > > agreement on behalf of TiVo Inc. by email. Binding agreements with > > > TiVo > > > Inc. may only be made by a signed written agreement. > > > > -- > > > Tommy Becker > > Senior Software Engineer > > O +1 919.460.4747 > > tivo.com > > > ________________________________ > > This email and any attachments may contain confidential and privileged > material for the sole use of the intended recipient. Any review, copying, > or distribution of this email (or any attachments) by others is prohibited. > If you are not the intended recipient, please contact the sender > immediately and permanently delete this email and any attachments. No > employee or agent of TiVo Inc. is authorized to conclude any binding > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo > Inc. may only be made by a signed written agreement. >
