Yes, we were burned by this. The changelog mapping will be regenerated
instead of migrated and the result will completely hose the job
(because the mapping was not generated deterministically in previous
versions of Samza). I don't understand why the migration code was
removed but it was, and to the best of my knowledge the necessity to
not skip version 0.10.0 when upgrading was not documented, let alone
enforced.
On Mon, 2017-03-27 at 10:07 -0700, Jagadish Venkatraman wrote:
> Good observation Jake!
>
> The code for migration was removed in Samza 11. The migration would
> read
> change-log offsets from the checkpoint topic and write them to the
> coordinator stream.
>
> If you're using change-logged stores, I'd recommend upgrading from
> 0.9.1 to
> 0.10.0 first.
> Otherwise, you will loose offsets for change-logged stores.
>
> I suspect you should be okay for 0.10.0 to 0.12 upgrade.
>
> On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes <jacob.m...@gmail.com>
> wrote:
>
> >
> > As I recall, samza 0.10 introduced the coordinator stream and there
> > was
> > code to do an automatic migration to use that feature. @navina,
> > @yi, do you
> > know if that migration code is still in samza 12?
> >
> > If not, then it's probably better to update from 0.9.1 to 0.10.0
> > and then
> > to 0.12.0. I don't think there were any changes requiring migration
> > between
> > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is probably
> > less of
> > an issue.
> >
> > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman <
> > jagadish1...@gmail.com> wrote:
> >
> > >
> > > Hi Xiaochuan,
> > >
> > > >
> > > > >
> > > > > Do I need to upgrade Kafka and/or YARN?
> > > *Yarn version:*
> > >
> > > - Samza 0.12 supports Yarn 2.6.1 and 2.7.1.
> > > - If you already have 2.6.0 installed (as you have said), I
> > > believe
> > you
> > >
> > > will be fine. (but I'm not sure)
> > >
> > > *Kafka version: *
> > >
> > > - Samza 0.12 upgraded the version of Kafka to 0.10.
> > > - If your Kafka brokers are on an older version of Kafka, you
> > > should
> > > upgrade them to use at-least 0.10. Kafka clients are usually
> > > incompatible with older versions of brokers.
> > >
> > > *Java version: *
> > >
> > >
> > >
> > > - Samza 0.12 binaries are compiled using Java 8. Hence, they
> > > cannot
> > be
> > >
> > > run on older versions of the Java run-time.
> > >
> > >
> > > >
> > > > >
> > > > > I'm extremely new to Samza in terms of operations aspect. I'm
> > > > > not sure
> > > what
> > > information would be relevant in this case so please ask away.
> > >
> > > I'd first start by upgrading the Kafka brokers (assuming you're
> > > on Java
> > 8+
> > >
> > > already).
> > > Let us know how the migration goes!
> > >
> > > Thanks,
> > > Jagadish
> > >
> > >
> > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu <xiaochuan.yu@kik.c
> > > om>
> > > wrote:
> > >
> > > >
> > > > Hi,
> > > >
> > > > What are the general steps for upgrading Samza from 0.9 to
> > > > 0.12?
> > > > Do I need to upgrade Kafka and/or YARN?
> > > >
> > > > I don't know how Samza was setup initially but we currently
> > > > have the
> > > > following setup:
> > > >
> > > > Samza version: 0.9.1
> > > > YARN version: Hadoop 2.6.0-cdh5.4.8
> > > > Kafka version: 0.9.0.1
> > > >
> > > > I think installation of Kafka and YARN were managed through
> > > > Puppet.
> > > > I'm extremely new to Samza in terms of operations aspect. I'm
> > > > not sure
> > > what
> > > >
> > > > information would be relevant in this case so please ask away.
> > > >
> > > > Thanks,
> > > > Xiaochuan Yu
> > > >
> > >
> > >
> > > --
> > > Jagadish V,
> > > Graduate Student,
> > > Department of Computer Science,
> > > Stanford University
> > >
>
>
--
Tommy Becker
Senior Software Engineer
O +1 919.460.4747
tivo.com
________________________________
This email and any attachments may contain confidential and privileged material
for the sole use of the intended recipient. Any review, copying, or
distribution of this email (or any attachments) by others is prohibited. If you
are not the intended recipient, please contact the sender immediately and
permanently delete this email and any attachments. No employee or agent of TiVo
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by
email. Binding agreements with TiVo Inc. may only be made by a signed written
agreement.
