FYI, I created https://issues.apache.org/jira/browse/SAMZA-1177 to track this.
We will ensure that we have explicit upgrade documentation for upcoming releases. On Thu, Mar 30, 2017 at 2:21 PM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > Hi Thomas and Maxim, > > Thank you for bringing up this concern and pointing out the gap in the > documentation. We will ensure that this is documented in a separate upgrade > page going forward. > > Best, > Jagadish > > On Thu, Mar 30, 2017 at 12:02 PM, Navina Ramesh < > nram...@linkedin.com.invalid> wrote: > >> Hi everyone, >> Apologize for re-chiming in late on this issue. >> >> > I'm not sure I agree with the policy (removing migration code and >> wanting >> people to upgrade seem at odds to me), but minimally I think we should not >> assume people are upgrading to each new Samza version. >> >> I agree that we should not assume that people will upgrade by stepping >> through each version of Samza. However, I don't agree that migration code >> should not be removed at all. Thinking in terms of a project management >> and >> maintenance, I think it is a common practice (at least in companies, if >> not >> in open-source and I could be wrong too :D ) to keep migration code only >> for the version it applies. It does add significant overhead to maintain >> version upgrade/migration code across all future versions. >> >> In this case, this was the first time we tried "automatic upgrade" from >> one >> version to the other (0.9 -> 0.10). We could have done a better job at >> documenting the upgrade steps with each version. I wish we had more >> outspoken voices in the community sooner than later :) >> >> Every project takes times to iron out issues related to release and >> version >> upgrade. I am glad that we have so much feedback now. As Yi suggested, the >> SEP process is a starting step towards documenting our changes across >> versions. Additionally, we will work on adding a dedicated page for >> upgrades and these will be available for all of the *upcoming* versions. >> >> Please let us know if you have any other concerns or ideas on how we can >> improve on our process. >> >> @XiaoChuan: Unfortunately, we don't have proper documentation on upgrading >> Samza across various versions. Like I mentioned before, we will put in >> extra efforts going forward. There aren't any migration/upgrade steps >> needed for versions post 0.10.*. You should be able to simply upgrade >> without any issues. Upgrade from 0.9 to 0.10 is an exceptional case. Happy >> to help you out in case you encounter more issues. >> >> Cheers! >> Navina >> >> On Thu, Mar 30, 2017 at 11:04 AM, XiaoChuan Yu <xiaochuan...@kik.com> >> wrote: >> >> > Is there some sort of document on how to upgrade Samza through various >> > versions like the page here for Kafka: >> > https://kafka.apache.org/documentation/#upgrade ? >> > Having something like this would be ideal. >> > On Thu, Mar 30, 2017 at 1:51 PM Thomas Becker <tobec...@tivo.com> >> wrote: >> > >> > > Thanks for the reply Yi, and I apologize if I came off a bit snarky. >> > > I'm not sure I agree with the policy (removing migration code and >> > > wanting people to upgrade seem at odds to me), but minimally I think >> we >> > > should not assume people are upgrading to each new Samza version. We >> > > have done so when features or fixes warrant, and even then on a >> per-job >> > > basis, and I would expect this is a common practice. >> > > >> > > -Tommy >> > > >> > > On Thu, 2017-03-30 at 09:50 -0700, Yi Pan wrote: >> > > > Hi, Thomas, >> > > > >> > > > Sorry to hear that you were hit by the removal of migration in Samza >> > > > 0.11. >> > > > The reason we removed it is following a deprecate-removal policy in >> > > > two >> > > > versions. We are not aware that people still using 0.9 after we >> > > > released >> > > > 0.11 and were not expecting a direct upgrade from 0.9 to 0.12. >> > > > Document can >> > > > be better to capture that. We are making changes to the design >> > > > proposal >> > > > s.t. it is more transparent and open to the whole community, through >> > > > the >> > > > newly proposed SEP process. These kind of breaking changes will go >> > > > through >> > > > the SEP discuss-vote process in the future and hopefully capture all >> > > > these >> > > > kind of concerns earlier. >> > > > >> > > > Best! >> > > > >> > > > -Yi >> > > > >> > > > On Thu, Mar 30, 2017 at 7:45 AM, Thomas Becker <tobec...@tivo.com> >> > > > wrote: >> > > > >> > > > > >> > > > > Yes, we were burned by this. The changelog mapping will be >> > > > > regenerated >> > > > > instead of migrated and the result will completely hose the job >> > > > > (because the mapping was not generated deterministically in >> > > > > previous >> > > > > versions of Samza). I don't understand why the migration code was >> > > > > removed but it was, and to the best of my knowledge the necessity >> > > > > to >> > > > > not skip version 0.10.0 when upgrading was not documented, let >> > > > > alone >> > > > > enforced. >> > > > > >> > > > > On Mon, 2017-03-27 at 10:07 -0700, Jagadish Venkatraman wrote: >> > > > > > >> > > > > > Good observation Jake! >> > > > > > >> > > > > > The code for migration was removed in Samza 11. The migration >> > > > > > would >> > > > > > read >> > > > > > change-log offsets from the checkpoint topic and write them to >> > > > > > the >> > > > > > coordinator stream. >> > > > > > >> > > > > > If you're using change-logged stores, I'd recommend upgrading >> > > > > > from >> > > > > > 0.9.1 to >> > > > > > 0.10.0 first. >> > > > > > Otherwise, you will loose offsets for change-logged stores. >> > > > > > >> > > > > > I suspect you should be okay for 0.10.0 to 0.12 upgrade. >> > > > > > >> > > > > > On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes < >> jacob.m...@gmail.com >> > > > > > > >> > > > > > wrote: >> > > > > > >> > > > > > > >> > > > > > > >> > > > > > > As I recall, samza 0.10 introduced the coordinator stream and >> > > > > > > there >> > > > > > > was >> > > > > > > code to do an automatic migration to use that feature. >> @navina, >> > > > > > > @yi, do you >> > > > > > > know if that migration code is still in samza 12? >> > > > > > > >> > > > > > > If not, then it's probably better to update from 0.9.1 to >> > > > > > > 0.10.0 >> > > > > > > and then >> > > > > > > to 0.12.0. I don't think there were any changes requiring >> > > > > > > migration >> > > > > > > between >> > > > > > > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is >> > > > > > > probably >> > > > > > > less of >> > > > > > > an issue. >> > > > > > > >> > > > > > > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman < >> > > > > > > jagadish1...@gmail.com> wrote: >> > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > Hi Xiaochuan, >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > Do I need to upgrade Kafka and/or YARN? >> > > > > > > > *Yarn version:* >> > > > > > > > >> > > > > > > > - Samza 0.12 supports Yarn 2.6.1 and 2.7.1. >> > > > > > > > - If you already have 2.6.0 installed (as you have said), >> > > > > > > > I >> > > > > > > > believe >> > > > > > > you >> > > > > > > > >> > > > > > > > >> > > > > > > > will be fine. (but I'm not sure) >> > > > > > > > >> > > > > > > > *Kafka version: * >> > > > > > > > >> > > > > > > > - Samza 0.12 upgraded the version of Kafka to 0.10. >> > > > > > > > - If your Kafka brokers are on an older version of Kafka, >> > > > > > > > you >> > > > > > > > should >> > > > > > > > upgrade them to use at-least 0.10. Kafka clients are >> > > > > > > > usually >> > > > > > > > incompatible with older versions of brokers. >> > > > > > > > >> > > > > > > > *Java version: * >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > - Samza 0.12 binaries are compiled using Java 8. Hence, >> > > > > > > > they >> > > > > > > > cannot >> > > > > > > be >> > > > > > > > >> > > > > > > > >> > > > > > > > run on older versions of the Java run-time. >> > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > I'm extremely new to Samza in terms of operations >> aspect. >> > > > > > > > > > I'm >> > > > > > > > > > not sure >> > > > > > > > what >> > > > > > > > information would be relevant in this case so please ask >> > > > > > > > away. >> > > > > > > > >> > > > > > > > I'd first start by upgrading the Kafka brokers (assuming >> > > > > > > > you're >> > > > > > > > on Java >> > > > > > > 8+ >> > > > > > > > >> > > > > > > > >> > > > > > > > already). >> > > > > > > > Let us know how the migration goes! >> > > > > > > > >> > > > > > > > Thanks, >> > > > > > > > Jagadish >> > > > > > > > >> > > > > > > > >> > > > > > > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu >> <xiaochuan.yu@k >> > > > > > > > ik.c >> > > > > > > > om> >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > Hi, >> > > > > > > > > >> > > > > > > > > What are the general steps for upgrading Samza from 0.9 to >> > > > > > > > > 0.12? >> > > > > > > > > Do I need to upgrade Kafka and/or YARN? >> > > > > > > > > >> > > > > > > > > I don't know how Samza was setup initially but we >> currently >> > > > > > > > > have the >> > > > > > > > > following setup: >> > > > > > > > > >> > > > > > > > > Samza version: 0.9.1 >> > > > > > > > > YARN version: Hadoop 2.6.0-cdh5.4.8 >> > > > > > > > > Kafka version: 0.9.0.1 >> > > > > > > > > >> > > > > > > > > I think installation of Kafka and YARN were managed >> through >> > > > > > > > > Puppet. >> > > > > > > > > I'm extremely new to Samza in terms of operations aspect. >> > > > > > > > > I'm >> > > > > > > > > not sure >> > > > > > > > what >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > information would be relevant in this case so please ask >> > > > > > > > > away. >> > > > > > > > > >> > > > > > > > > Thanks, >> > > > > > > > > Xiaochuan Yu >> > > > > > > > > >> > > > > > > > >> > > > > > > > -- >> > > > > > > > Jagadish V, >> > > > > > > > Graduate Student, >> > > > > > > > Department of Computer Science, >> > > > > > > > Stanford University >> > > > > > > > >> > > > > > >> > > > > -- >> > > > > >> > > > > >> > > > > Tommy Becker >> > > > > >> > > > > Senior Software Engineer >> > > > > >> > > > > O +1 919.460.4747 <(919)%20460-4747> >> > > > > >> > > > > tivo.com >> > > > > >> > > > > >> > > > > ________________________________ >> > > > > >> > > > > This email and any attachments may contain confidential and >> > > > > privileged >> > > > > material for the sole use of the intended recipient. Any review, >> > > > > copying, >> > > > > or distribution of this email (or any attachments) by others is >> > > > > prohibited. >> > > > > If you are not the intended recipient, please contact the sender >> > > > > immediately and permanently delete this email and any attachments. >> > > > > No >> > > > > employee or agent of TiVo Inc. is authorized to conclude any >> > > > > binding >> > > > > agreement on behalf of TiVo Inc. by email. Binding agreements with >> > > > > TiVo >> > > > > Inc. may only be made by a signed written agreement. >> > > > > >> > > -- >> > > >> > > >> > > Tommy Becker >> > > >> > > Senior Software Engineer >> > > >> > > O +1 919.460.4747 >> > > >> > > tivo.com >> > > >> > > >> > > ________________________________ >> > > >> > > This email and any attachments may contain confidential and privileged >> > > material for the sole use of the intended recipient. Any review, >> copying, >> > > or distribution of this email (or any attachments) by others is >> > prohibited. >> > > If you are not the intended recipient, please contact the sender >> > > immediately and permanently delete this email and any attachments. No >> > > employee or agent of TiVo Inc. is authorized to conclude any binding >> > > agreement on behalf of TiVo Inc. by email. Binding agreements with >> TiVo >> > > Inc. may only be made by a signed written agreement. >> > > >> > >> >> >> >> -- >> Navina R. >> > > > > -- > Jagadish V, > Graduate Student, > Department of Computer Science, > Stanford University > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
