Hi Thomas and Maxim, Thank you for bringing up this concern and pointing out the gap in the documentation. We will ensure that this is documented in a separate upgrade page going forward.
Best, Jagadish On Thu, Mar 30, 2017 at 12:02 PM, Navina Ramesh < nram...@linkedin.com.invalid> wrote: > Hi everyone, > Apologize for re-chiming in late on this issue. > > > I'm not sure I agree with the policy (removing migration code and wanting > people to upgrade seem at odds to me), but minimally I think we should not > assume people are upgrading to each new Samza version. > > I agree that we should not assume that people will upgrade by stepping > through each version of Samza. However, I don't agree that migration code > should not be removed at all. Thinking in terms of a project management and > maintenance, I think it is a common practice (at least in companies, if not > in open-source and I could be wrong too :D ) to keep migration code only > for the version it applies. It does add significant overhead to maintain > version upgrade/migration code across all future versions. > > In this case, this was the first time we tried "automatic upgrade" from one > version to the other (0.9 -> 0.10). We could have done a better job at > documenting the upgrade steps with each version. I wish we had more > outspoken voices in the community sooner than later :) > > Every project takes times to iron out issues related to release and version > upgrade. I am glad that we have so much feedback now. As Yi suggested, the > SEP process is a starting step towards documenting our changes across > versions. Additionally, we will work on adding a dedicated page for > upgrades and these will be available for all of the *upcoming* versions. > > Please let us know if you have any other concerns or ideas on how we can > improve on our process. > > @XiaoChuan: Unfortunately, we don't have proper documentation on upgrading > Samza across various versions. Like I mentioned before, we will put in > extra efforts going forward. There aren't any migration/upgrade steps > needed for versions post 0.10.*. You should be able to simply upgrade > without any issues. Upgrade from 0.9 to 0.10 is an exceptional case. Happy > to help you out in case you encounter more issues. > > Cheers! > Navina > > On Thu, Mar 30, 2017 at 11:04 AM, XiaoChuan Yu <xiaochuan...@kik.com> > wrote: > > > Is there some sort of document on how to upgrade Samza through various > > versions like the page here for Kafka: > > https://kafka.apache.org/documentation/#upgrade ? > > Having something like this would be ideal. > > On Thu, Mar 30, 2017 at 1:51 PM Thomas Becker <tobec...@tivo.com> wrote: > > > > > Thanks for the reply Yi, and I apologize if I came off a bit snarky. > > > I'm not sure I agree with the policy (removing migration code and > > > wanting people to upgrade seem at odds to me), but minimally I think we > > > should not assume people are upgrading to each new Samza version. We > > > have done so when features or fixes warrant, and even then on a per-job > > > basis, and I would expect this is a common practice. > > > > > > -Tommy > > > > > > On Thu, 2017-03-30 at 09:50 -0700, Yi Pan wrote: > > > > Hi, Thomas, > > > > > > > > Sorry to hear that you were hit by the removal of migration in Samza > > > > 0.11. > > > > The reason we removed it is following a deprecate-removal policy in > > > > two > > > > versions. We are not aware that people still using 0.9 after we > > > > released > > > > 0.11 and were not expecting a direct upgrade from 0.9 to 0.12. > > > > Document can > > > > be better to capture that. We are making changes to the design > > > > proposal > > > > s.t. it is more transparent and open to the whole community, through > > > > the > > > > newly proposed SEP process. These kind of breaking changes will go > > > > through > > > > the SEP discuss-vote process in the future and hopefully capture all > > > > these > > > > kind of concerns earlier. > > > > > > > > Best! > > > > > > > > -Yi > > > > > > > > On Thu, Mar 30, 2017 at 7:45 AM, Thomas Becker <tobec...@tivo.com> > > > > wrote: > > > > > > > > > > > > > > Yes, we were burned by this. The changelog mapping will be > > > > > regenerated > > > > > instead of migrated and the result will completely hose the job > > > > > (because the mapping was not generated deterministically in > > > > > previous > > > > > versions of Samza). I don't understand why the migration code was > > > > > removed but it was, and to the best of my knowledge the necessity > > > > > to > > > > > not skip version 0.10.0 when upgrading was not documented, let > > > > > alone > > > > > enforced. > > > > > > > > > > On Mon, 2017-03-27 at 10:07 -0700, Jagadish Venkatraman wrote: > > > > > > > > > > > > Good observation Jake! > > > > > > > > > > > > The code for migration was removed in Samza 11. The migration > > > > > > would > > > > > > read > > > > > > change-log offsets from the checkpoint topic and write them to > > > > > > the > > > > > > coordinator stream. > > > > > > > > > > > > If you're using change-logged stores, I'd recommend upgrading > > > > > > from > > > > > > 0.9.1 to > > > > > > 0.10.0 first. > > > > > > Otherwise, you will loose offsets for change-logged stores. > > > > > > > > > > > > I suspect you should be okay for 0.10.0 to 0.12 upgrade. > > > > > > > > > > > > On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes < > jacob.m...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > As I recall, samza 0.10 introduced the coordinator stream and > > > > > > > there > > > > > > > was > > > > > > > code to do an automatic migration to use that feature. @navina, > > > > > > > @yi, do you > > > > > > > know if that migration code is still in samza 12? > > > > > > > > > > > > > > If not, then it's probably better to update from 0.9.1 to > > > > > > > 0.10.0 > > > > > > > and then > > > > > > > to 0.12.0. I don't think there were any changes requiring > > > > > > > migration > > > > > > > between > > > > > > > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is > > > > > > > probably > > > > > > > less of > > > > > > > an issue. > > > > > > > > > > > > > > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman < > > > > > > > jagadish1...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Xiaochuan, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do I need to upgrade Kafka and/or YARN? > > > > > > > > *Yarn version:* > > > > > > > > > > > > > > > > - Samza 0.12 supports Yarn 2.6.1 and 2.7.1. > > > > > > > > - If you already have 2.6.0 installed (as you have said), > > > > > > > > I > > > > > > > > believe > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > will be fine. (but I'm not sure) > > > > > > > > > > > > > > > > *Kafka version: * > > > > > > > > > > > > > > > > - Samza 0.12 upgraded the version of Kafka to 0.10. > > > > > > > > - If your Kafka brokers are on an older version of Kafka, > > > > > > > > you > > > > > > > > should > > > > > > > > upgrade them to use at-least 0.10. Kafka clients are > > > > > > > > usually > > > > > > > > incompatible with older versions of brokers. > > > > > > > > > > > > > > > > *Java version: * > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Samza 0.12 binaries are compiled using Java 8. Hence, > > > > > > > > they > > > > > > > > cannot > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > run on older versions of the Java run-time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm extremely new to Samza in terms of operations aspect. > > > > > > > > > > I'm > > > > > > > > > > not sure > > > > > > > > what > > > > > > > > information would be relevant in this case so please ask > > > > > > > > away. > > > > > > > > > > > > > > > > I'd first start by upgrading the Kafka brokers (assuming > > > > > > > > you're > > > > > > > > on Java > > > > > > > 8+ > > > > > > > > > > > > > > > > > > > > > > > > already). > > > > > > > > Let us know how the migration goes! > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Jagadish > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu <xiaochuan.yu@k > > > > > > > > ik.c > > > > > > > > om> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > What are the general steps for upgrading Samza from 0.9 to > > > > > > > > > 0.12? > > > > > > > > > Do I need to upgrade Kafka and/or YARN? > > > > > > > > > > > > > > > > > > I don't know how Samza was setup initially but we currently > > > > > > > > > have the > > > > > > > > > following setup: > > > > > > > > > > > > > > > > > > Samza version: 0.9.1 > > > > > > > > > YARN version: Hadoop 2.6.0-cdh5.4.8 > > > > > > > > > Kafka version: 0.9.0.1 > > > > > > > > > > > > > > > > > > I think installation of Kafka and YARN were managed through > > > > > > > > > Puppet. > > > > > > > > > I'm extremely new to Samza in terms of operations aspect. > > > > > > > > > I'm > > > > > > > > > not sure > > > > > > > > what > > > > > > > > > > > > > > > > > > > > > > > > > > > information would be relevant in this case so please ask > > > > > > > > > away. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Xiaochuan Yu > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Jagadish V, > > > > > > > > Graduate Student, > > > > > > > > Department of Computer Science, > > > > > > > > Stanford University > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > Tommy Becker > > > > > > > > > > Senior Software Engineer > > > > > > > > > > O +1 919.460.4747 > > > > > > > > > > tivo.com > > > > > > > > > > > > > > > ________________________________ > > > > > > > > > > This email and any attachments may contain confidential and > > > > > privileged > > > > > material for the sole use of the intended recipient. Any review, > > > > > copying, > > > > > or distribution of this email (or any attachments) by others is > > > > > prohibited. > > > > > If you are not the intended recipient, please contact the sender > > > > > immediately and permanently delete this email and any attachments. > > > > > No > > > > > employee or agent of TiVo Inc. is authorized to conclude any > > > > > binding > > > > > agreement on behalf of TiVo Inc. by email. Binding agreements with > > > > > TiVo > > > > > Inc. may only be made by a signed written agreement. > > > > > > > > -- > > > > > > > > > Tommy Becker > > > > > > Senior Software Engineer > > > > > > O +1 919.460.4747 > > > > > > tivo.com > > > > > > > > > ________________________________ > > > > > > This email and any attachments may contain confidential and privileged > > > material for the sole use of the intended recipient. Any review, > copying, > > > or distribution of this email (or any attachments) by others is > > prohibited. > > > If you are not the intended recipient, please contact the sender > > > immediately and permanently delete this email and any attachments. No > > > employee or agent of TiVo Inc. is authorized to conclude any binding > > > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo > > > Inc. may only be made by a signed written agreement. > > > > > > > > > -- > Navina R. > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
