Hi everyone, Apologize for re-chiming in late on this issue. > I'm not sure I agree with the policy (removing migration code and wanting people to upgrade seem at odds to me), but minimally I think we should not assume people are upgrading to each new Samza version.
I agree that we should not assume that people will upgrade by stepping through each version of Samza. However, I don't agree that migration code should not be removed at all. Thinking in terms of a project management and maintenance, I think it is a common practice (at least in companies, if not in open-source and I could be wrong too :D ) to keep migration code only for the version it applies. It does add significant overhead to maintain version upgrade/migration code across all future versions. In this case, this was the first time we tried "automatic upgrade" from one version to the other (0.9 -> 0.10). We could have done a better job at documenting the upgrade steps with each version. I wish we had more outspoken voices in the community sooner than later :) Every project takes times to iron out issues related to release and version upgrade. I am glad that we have so much feedback now. As Yi suggested, the SEP process is a starting step towards documenting our changes across versions. Additionally, we will work on adding a dedicated page for upgrades and these will be available for all of the *upcoming* versions. Please let us know if you have any other concerns or ideas on how we can improve on our process. @XiaoChuan: Unfortunately, we don't have proper documentation on upgrading Samza across various versions. Like I mentioned before, we will put in extra efforts going forward. There aren't any migration/upgrade steps needed for versions post 0.10.*. You should be able to simply upgrade without any issues. Upgrade from 0.9 to 0.10 is an exceptional case. Happy to help you out in case you encounter more issues. Cheers! Navina On Thu, Mar 30, 2017 at 11:04 AM, XiaoChuan Yu <xiaochuan...@kik.com> wrote: > Is there some sort of document on how to upgrade Samza through various > versions like the page here for Kafka: > https://kafka.apache.org/documentation/#upgrade ? > Having something like this would be ideal. > On Thu, Mar 30, 2017 at 1:51 PM Thomas Becker <tobec...@tivo.com> wrote: > > > Thanks for the reply Yi, and I apologize if I came off a bit snarky. > > I'm not sure I agree with the policy (removing migration code and > > wanting people to upgrade seem at odds to me), but minimally I think we > > should not assume people are upgrading to each new Samza version. We > > have done so when features or fixes warrant, and even then on a per-job > > basis, and I would expect this is a common practice. > > > > -Tommy > > > > On Thu, 2017-03-30 at 09:50 -0700, Yi Pan wrote: > > > Hi, Thomas, > > > > > > Sorry to hear that you were hit by the removal of migration in Samza > > > 0.11. > > > The reason we removed it is following a deprecate-removal policy in > > > two > > > versions. We are not aware that people still using 0.9 after we > > > released > > > 0.11 and were not expecting a direct upgrade from 0.9 to 0.12. > > > Document can > > > be better to capture that. We are making changes to the design > > > proposal > > > s.t. it is more transparent and open to the whole community, through > > > the > > > newly proposed SEP process. These kind of breaking changes will go > > > through > > > the SEP discuss-vote process in the future and hopefully capture all > > > these > > > kind of concerns earlier. > > > > > > Best! > > > > > > -Yi > > > > > > On Thu, Mar 30, 2017 at 7:45 AM, Thomas Becker <tobec...@tivo.com> > > > wrote: > > > > > > > > > > > Yes, we were burned by this. The changelog mapping will be > > > > regenerated > > > > instead of migrated and the result will completely hose the job > > > > (because the mapping was not generated deterministically in > > > > previous > > > > versions of Samza). I don't understand why the migration code was > > > > removed but it was, and to the best of my knowledge the necessity > > > > to > > > > not skip version 0.10.0 when upgrading was not documented, let > > > > alone > > > > enforced. > > > > > > > > On Mon, 2017-03-27 at 10:07 -0700, Jagadish Venkatraman wrote: > > > > > > > > > > Good observation Jake! > > > > > > > > > > The code for migration was removed in Samza 11. The migration > > > > > would > > > > > read > > > > > change-log offsets from the checkpoint topic and write them to > > > > > the > > > > > coordinator stream. > > > > > > > > > > If you're using change-logged stores, I'd recommend upgrading > > > > > from > > > > > 0.9.1 to > > > > > 0.10.0 first. > > > > > Otherwise, you will loose offsets for change-logged stores. > > > > > > > > > > I suspect you should be okay for 0.10.0 to 0.12 upgrade. > > > > > > > > > > On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes <jacob.m...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > As I recall, samza 0.10 introduced the coordinator stream and > > > > > > there > > > > > > was > > > > > > code to do an automatic migration to use that feature. @navina, > > > > > > @yi, do you > > > > > > know if that migration code is still in samza 12? > > > > > > > > > > > > If not, then it's probably better to update from 0.9.1 to > > > > > > 0.10.0 > > > > > > and then > > > > > > to 0.12.0. I don't think there were any changes requiring > > > > > > migration > > > > > > between > > > > > > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is > > > > > > probably > > > > > > less of > > > > > > an issue. > > > > > > > > > > > > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman < > > > > > > jagadish1...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Xiaochuan, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do I need to upgrade Kafka and/or YARN? > > > > > > > *Yarn version:* > > > > > > > > > > > > > > - Samza 0.12 supports Yarn 2.6.1 and 2.7.1. > > > > > > > - If you already have 2.6.0 installed (as you have said), > > > > > > > I > > > > > > > believe > > > > > > you > > > > > > > > > > > > > > > > > > > > > will be fine. (but I'm not sure) > > > > > > > > > > > > > > *Kafka version: * > > > > > > > > > > > > > > - Samza 0.12 upgraded the version of Kafka to 0.10. > > > > > > > - If your Kafka brokers are on an older version of Kafka, > > > > > > > you > > > > > > > should > > > > > > > upgrade them to use at-least 0.10. Kafka clients are > > > > > > > usually > > > > > > > incompatible with older versions of brokers. > > > > > > > > > > > > > > *Java version: * > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Samza 0.12 binaries are compiled using Java 8. Hence, > > > > > > > they > > > > > > > cannot > > > > > > be > > > > > > > > > > > > > > > > > > > > > run on older versions of the Java run-time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm extremely new to Samza in terms of operations aspect. > > > > > > > > > I'm > > > > > > > > > not sure > > > > > > > what > > > > > > > information would be relevant in this case so please ask > > > > > > > away. > > > > > > > > > > > > > > I'd first start by upgrading the Kafka brokers (assuming > > > > > > > you're > > > > > > > on Java > > > > > > 8+ > > > > > > > > > > > > > > > > > > > > > already). > > > > > > > Let us know how the migration goes! > > > > > > > > > > > > > > Thanks, > > > > > > > Jagadish > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu <xiaochuan.yu@k > > > > > > > ik.c > > > > > > > om> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > What are the general steps for upgrading Samza from 0.9 to > > > > > > > > 0.12? > > > > > > > > Do I need to upgrade Kafka and/or YARN? > > > > > > > > > > > > > > > > I don't know how Samza was setup initially but we currently > > > > > > > > have the > > > > > > > > following setup: > > > > > > > > > > > > > > > > Samza version: 0.9.1 > > > > > > > > YARN version: Hadoop 2.6.0-cdh5.4.8 > > > > > > > > Kafka version: 0.9.0.1 > > > > > > > > > > > > > > > > I think installation of Kafka and YARN were managed through > > > > > > > > Puppet. > > > > > > > > I'm extremely new to Samza in terms of operations aspect. > > > > > > > > I'm > > > > > > > > not sure > > > > > > > what > > > > > > > > > > > > > > > > > > > > > > > > information would be relevant in this case so please ask > > > > > > > > away. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Xiaochuan Yu > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Jagadish V, > > > > > > > Graduate Student, > > > > > > > Department of Computer Science, > > > > > > > Stanford University > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Tommy Becker > > > > > > > > Senior Software Engineer > > > > > > > > O +1 919.460.4747 > > > > > > > > tivo.com > > > > > > > > > > > > ________________________________ > > > > > > > > This email and any attachments may contain confidential and > > > > privileged > > > > material for the sole use of the intended recipient. Any review, > > > > copying, > > > > or distribution of this email (or any attachments) by others is > > > > prohibited. > > > > If you are not the intended recipient, please contact the sender > > > > immediately and permanently delete this email and any attachments. > > > > No > > > > employee or agent of TiVo Inc. is authorized to conclude any > > > > binding > > > > agreement on behalf of TiVo Inc. by email. Binding agreements with > > > > TiVo > > > > Inc. may only be made by a signed written agreement. > > > > > > -- > > > > > > Tommy Becker > > > > Senior Software Engineer > > > > O +1 919.460.4747 > > > > tivo.com > > > > > > ________________________________ > > > > This email and any attachments may contain confidential and privileged > > material for the sole use of the intended recipient. Any review, copying, > > or distribution of this email (or any attachments) by others is > prohibited. > > If you are not the intended recipient, please contact the sender > > immediately and permanently delete this email and any attachments. No > > employee or agent of TiVo Inc. is authorized to conclude any binding > > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo > > Inc. may only be made by a signed written agreement. > > > -- Navina R.
