Re: Steps to Upgrading Samza (0.9 to 0.12)

[Jagadish Venkatraman]

FYI, I created https://issues.apache.org/jira/browse/SAMZA-1177 to track
this.

We will ensure that we have explicit upgrade documentation for upcoming
releases.

On Thu, Mar 30, 2017 at 2:21 PM, Jagadish Venkatraman <
jagadish1...@gmail.com> wrote:

> Hi Thomas and Maxim,
>
> Thank you for bringing up this concern and pointing out the gap in the
> documentation. We will ensure that this is documented in a separate upgrade
> page going forward.
>
> Best,
> Jagadish
>
> On Thu, Mar 30, 2017 at 12:02 PM, Navina Ramesh <
> nram...@linkedin.com.invalid> wrote:
>
>> Hi everyone,
>> Apologize for re-chiming in late on this issue.
>>
>> > I'm not sure I agree with the policy (removing migration code and
>> wanting
>> people to upgrade seem at odds to me), but minimally I think we should not
>> assume people are upgrading to each new Samza version.
>>
>> I agree that we should not assume that people will upgrade by stepping
>> through each version of Samza. However, I don't agree that migration code
>> should not be removed at all. Thinking in terms of a project management
>> and
>> maintenance, I think it is a common practice (at least in companies, if
>> not
>> in open-source and I could be wrong too :D ) to keep migration code only
>> for the version it applies. It does add significant overhead to maintain
>> version upgrade/migration code across all future versions.
>>
>> In this case, this was the first time we tried "automatic upgrade" from
>> one
>> version to the other (0.9 -> 0.10). We could have done a better job at
>> documenting the upgrade steps with each version. I wish we had more
>> outspoken voices in the community sooner than later :)
>>
>> Every project takes times to iron out issues related to release and
>> version
>> upgrade. I am glad that we have so much feedback now. As Yi suggested, the
>> SEP process is a starting step towards documenting our changes across
>> versions. Additionally, we will work on adding a dedicated page for
>> upgrades and these will be available for all of the *upcoming* versions.
>>
>> Please let us know if you have any other concerns or ideas on how we can
>> improve on our process.
>>
>> @XiaoChuan: Unfortunately, we don't have proper documentation on upgrading
>> Samza across various versions. Like I mentioned before, we will put in
>> extra efforts going forward. There aren't any migration/upgrade steps
>> needed for versions post 0.10.*. You should be able to simply upgrade
>> without any issues. Upgrade from 0.9 to 0.10 is an exceptional case. Happy
>> to help you out in case you encounter more issues.
>>
>> Cheers!
>> Navina
>>
>> On Thu, Mar 30, 2017 at 11:04 AM, XiaoChuan Yu 
>> wrote:
>>
>> > Is there some sort of document on how to upgrade Samza through various
>> > versions like the page here for Kafka:
>> > https://kafka.apache.org/documentation/#upgrade ?
>> > Having something like this would be ideal.
>> > On Thu, Mar 30, 2017 at 1:51 PM Thomas Becker 
>> wrote:
>> >
>> > > Thanks for the reply Yi, and I apologize if I came off a bit snarky.
>> > > I'm not sure I agree with the policy (removing migration code and
>> > > wanting people to upgrade seem at odds to me), but minimally I think
>> we
>> > > should not assume people are upgrading to each new Samza version. We
>> > > have done so when features or fixes warrant, and even then on a
>> per-job
>> > > basis, and I would expect this is a common practice.
>> > >
>> > > -Tommy
>> > >
>> > > On Thu, 2017-03-30 at 09:50 -0700, Yi Pan wrote:
>> > > > Hi, Thomas,
>> > > >
>> > > > Sorry to hear that you were hit by the removal of migration in Samza
>> > > > 0.11.
>> > > > The reason we removed it is following a deprecate-removal policy in
>> > > > two
>> > > > versions. We are not aware that people still using 0.9 after we
>> > > > released
>> > > > 0.11 and were not expecting a direct upgrade from 0.9 to 0.12.
>> > > > Document can
>> > > > be better to capture that. We are making changes to the design
>> > > > proposal
>> > > > s.t. it is more transparent and open to the whole community, through
>> > > > the
>> > > > newly proposed SEP process. These kind of breaking changes will go
>> > > > through
>> > > > the SEP discuss-vote process in the future and hopefully capture all
>> > > > these
>> > > > kind of concerns earlier.
>> > > >
>> > > > Best!
>> > > >
>> > > > -Yi
>> > > >
>> > > > On Thu, Mar 30, 2017 at 7:45 AM, Thomas Becker 
>> > > > wrote:
>> > > >
>> > > > >
>> > > > > Yes, we were burned by this. The changelog mapping will be
>> > > > > regenerated
>> > > > > instead of migrated and the result will completely hose the job
>> > > > > (because the mapping was not generated deterministically in
>> > > > > previous
>> > > > > versions of Samza). I don't understand why the migration code was
>> > > > > removed but it was, and to the best of my knowledge the necessity
>> > > > > to
>> > > > > not skip version 0.10.0 when upgrading was 

Re: Steps to Upgrading Samza (0.9 to 0.12)

[Jagadish Venkatraman]

Hi Thomas and Maxim,

Thank you for bringing up this concern and pointing out the gap in the
documentation. We will ensure that this is documented in a separate upgrade
page going forward.

Best,
Jagadish

On Thu, Mar 30, 2017 at 12:02 PM, Navina Ramesh <
nram...@linkedin.com.invalid> wrote:

> Hi everyone,
> Apologize for re-chiming in late on this issue.
>
> > I'm not sure I agree with the policy (removing migration code and wanting
> people to upgrade seem at odds to me), but minimally I think we should not
> assume people are upgrading to each new Samza version.
>
> I agree that we should not assume that people will upgrade by stepping
> through each version of Samza. However, I don't agree that migration code
> should not be removed at all. Thinking in terms of a project management and
> maintenance, I think it is a common practice (at least in companies, if not
> in open-source and I could be wrong too :D ) to keep migration code only
> for the version it applies. It does add significant overhead to maintain
> version upgrade/migration code across all future versions.
>
> In this case, this was the first time we tried "automatic upgrade" from one
> version to the other (0.9 -> 0.10). We could have done a better job at
> documenting the upgrade steps with each version. I wish we had more
> outspoken voices in the community sooner than later :)
>
> Every project takes times to iron out issues related to release and version
> upgrade. I am glad that we have so much feedback now. As Yi suggested, the
> SEP process is a starting step towards documenting our changes across
> versions. Additionally, we will work on adding a dedicated page for
> upgrades and these will be available for all of the *upcoming* versions.
>
> Please let us know if you have any other concerns or ideas on how we can
> improve on our process.
>
> @XiaoChuan: Unfortunately, we don't have proper documentation on upgrading
> Samza across various versions. Like I mentioned before, we will put in
> extra efforts going forward. There aren't any migration/upgrade steps
> needed for versions post 0.10.*. You should be able to simply upgrade
> without any issues. Upgrade from 0.9 to 0.10 is an exceptional case. Happy
> to help you out in case you encounter more issues.
>
> Cheers!
> Navina
>
> On Thu, Mar 30, 2017 at 11:04 AM, XiaoChuan Yu 
> wrote:
>
> > Is there some sort of document on how to upgrade Samza through various
> > versions like the page here for Kafka:
> > https://kafka.apache.org/documentation/#upgrade ?
> > Having something like this would be ideal.
> > On Thu, Mar 30, 2017 at 1:51 PM Thomas Becker  wrote:
> >
> > > Thanks for the reply Yi, and I apologize if I came off a bit snarky.
> > > I'm not sure I agree with the policy (removing migration code and
> > > wanting people to upgrade seem at odds to me), but minimally I think we
> > > should not assume people are upgrading to each new Samza version. We
> > > have done so when features or fixes warrant, and even then on a per-job
> > > basis, and I would expect this is a common practice.
> > >
> > > -Tommy
> > >
> > > On Thu, 2017-03-30 at 09:50 -0700, Yi Pan wrote:
> > > > Hi, Thomas,
> > > >
> > > > Sorry to hear that you were hit by the removal of migration in Samza
> > > > 0.11.
> > > > The reason we removed it is following a deprecate-removal policy in
> > > > two
> > > > versions. We are not aware that people still using 0.9 after we
> > > > released
> > > > 0.11 and were not expecting a direct upgrade from 0.9 to 0.12.
> > > > Document can
> > > > be better to capture that. We are making changes to the design
> > > > proposal
> > > > s.t. it is more transparent and open to the whole community, through
> > > > the
> > > > newly proposed SEP process. These kind of breaking changes will go
> > > > through
> > > > the SEP discuss-vote process in the future and hopefully capture all
> > > > these
> > > > kind of concerns earlier.
> > > >
> > > > Best!
> > > >
> > > > -Yi
> > > >
> > > > On Thu, Mar 30, 2017 at 7:45 AM, Thomas Becker 
> > > > wrote:
> > > >
> > > > >
> > > > > Yes, we were burned by this. The changelog mapping will be
> > > > > regenerated
> > > > > instead of migrated and the result will completely hose the job
> > > > > (because the mapping was not generated deterministically in
> > > > > previous
> > > > > versions of Samza). I don't understand why the migration code was
> > > > > removed but it was, and to the best of my knowledge the necessity
> > > > > to
> > > > > not skip version 0.10.0 when upgrading was not documented, let
> > > > > alone
> > > > > enforced.
> > > > >
> > > > > On Mon, 2017-03-27 at 10:07 -0700, Jagadish Venkatraman wrote:
> > > > > >
> > > > > > Good observation Jake!
> > > > > >
> > > > > > The code for migration was removed in Samza 11. The migration
> > > > > > would
> > > > > > read
> > > > > > change-log offsets from the checkpoint topic and write them to
> > > > 

Re: Steps to Upgrading Samza (0.9 to 0.12)

[Navina Ramesh]

Hi everyone,
Apologize for re-chiming in late on this issue.

> I'm not sure I agree with the policy (removing migration code and wanting
people to upgrade seem at odds to me), but minimally I think we should not
assume people are upgrading to each new Samza version.

I agree that we should not assume that people will upgrade by stepping
through each version of Samza. However, I don't agree that migration code
should not be removed at all. Thinking in terms of a project management and
maintenance, I think it is a common practice (at least in companies, if not
in open-source and I could be wrong too :D ) to keep migration code only
for the version it applies. It does add significant overhead to maintain
version upgrade/migration code across all future versions.

In this case, this was the first time we tried "automatic upgrade" from one
version to the other (0.9 -> 0.10). We could have done a better job at
documenting the upgrade steps with each version. I wish we had more
outspoken voices in the community sooner than later :)

Every project takes times to iron out issues related to release and version
upgrade. I am glad that we have so much feedback now. As Yi suggested, the
SEP process is a starting step towards documenting our changes across
versions. Additionally, we will work on adding a dedicated page for
upgrades and these will be available for all of the *upcoming* versions.

Please let us know if you have any other concerns or ideas on how we can
improve on our process.

@XiaoChuan: Unfortunately, we don't have proper documentation on upgrading
Samza across various versions. Like I mentioned before, we will put in
extra efforts going forward. There aren't any migration/upgrade steps
needed for versions post 0.10.*. You should be able to simply upgrade
without any issues. Upgrade from 0.9 to 0.10 is an exceptional case. Happy
to help you out in case you encounter more issues.

Cheers!
Navina

On Thu, Mar 30, 2017 at 11:04 AM, XiaoChuan Yu  wrote:

> Is there some sort of document on how to upgrade Samza through various
> versions like the page here for Kafka:
> https://kafka.apache.org/documentation/#upgrade ?
> Having something like this would be ideal.
> On Thu, Mar 30, 2017 at 1:51 PM Thomas Becker  wrote:
>
> > Thanks for the reply Yi, and I apologize if I came off a bit snarky.
> > I'm not sure I agree with the policy (removing migration code and
> > wanting people to upgrade seem at odds to me), but minimally I think we
> > should not assume people are upgrading to each new Samza version. We
> > have done so when features or fixes warrant, and even then on a per-job
> > basis, and I would expect this is a common practice.
> >
> > -Tommy
> >
> > On Thu, 2017-03-30 at 09:50 -0700, Yi Pan wrote:
> > > Hi, Thomas,
> > >
> > > Sorry to hear that you were hit by the removal of migration in Samza
> > > 0.11.
> > > The reason we removed it is following a deprecate-removal policy in
> > > two
> > > versions. We are not aware that people still using 0.9 after we
> > > released
> > > 0.11 and were not expecting a direct upgrade from 0.9 to 0.12.
> > > Document can
> > > be better to capture that. We are making changes to the design
> > > proposal
> > > s.t. it is more transparent and open to the whole community, through
> > > the
> > > newly proposed SEP process. These kind of breaking changes will go
> > > through
> > > the SEP discuss-vote process in the future and hopefully capture all
> > > these
> > > kind of concerns earlier.
> > >
> > > Best!
> > >
> > > -Yi
> > >
> > > On Thu, Mar 30, 2017 at 7:45 AM, Thomas Becker 
> > > wrote:
> > >
> > > >
> > > > Yes, we were burned by this. The changelog mapping will be
> > > > regenerated
> > > > instead of migrated and the result will completely hose the job
> > > > (because the mapping was not generated deterministically in
> > > > previous
> > > > versions of Samza). I don't understand why the migration code was
> > > > removed but it was, and to the best of my knowledge the necessity
> > > > to
> > > > not skip version 0.10.0 when upgrading was not documented, let
> > > > alone
> > > > enforced.
> > > >
> > > > On Mon, 2017-03-27 at 10:07 -0700, Jagadish Venkatraman wrote:
> > > > >
> > > > > Good observation Jake!
> > > > >
> > > > > The code for migration was removed in Samza 11. The migration
> > > > > would
> > > > > read
> > > > > change-log offsets from the checkpoint topic and write them to
> > > > > the
> > > > > coordinator stream.
> > > > >
> > > > > If you're using change-logged stores, I'd recommend upgrading
> > > > > from
> > > > > 0.9.1 to
> > > > > 0.10.0 first.
> > > > > Otherwise, you will loose offsets for change-logged stores.
> > > > >
> > > > > I suspect you should be okay for 0.10.0 to 0.12 upgrade.
> > > > >
> > > > > On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes  > > > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > As I recall, 

Re: Steps to Upgrading Samza (0.9 to 0.12)

[XiaoChuan Yu]

Is there some sort of document on how to upgrade Samza through various
versions like the page here for Kafka:
https://kafka.apache.org/documentation/#upgrade ?
Having something like this would be ideal.
On Thu, Mar 30, 2017 at 1:51 PM Thomas Becker  wrote:

> Thanks for the reply Yi, and I apologize if I came off a bit snarky.
> I'm not sure I agree with the policy (removing migration code and
> wanting people to upgrade seem at odds to me), but minimally I think we
> should not assume people are upgrading to each new Samza version. We
> have done so when features or fixes warrant, and even then on a per-job
> basis, and I would expect this is a common practice.
>
> -Tommy
>
> On Thu, 2017-03-30 at 09:50 -0700, Yi Pan wrote:
> > Hi, Thomas,
> >
> > Sorry to hear that you were hit by the removal of migration in Samza
> > 0.11.
> > The reason we removed it is following a deprecate-removal policy in
> > two
> > versions. We are not aware that people still using 0.9 after we
> > released
> > 0.11 and were not expecting a direct upgrade from 0.9 to 0.12.
> > Document can
> > be better to capture that. We are making changes to the design
> > proposal
> > s.t. it is more transparent and open to the whole community, through
> > the
> > newly proposed SEP process. These kind of breaking changes will go
> > through
> > the SEP discuss-vote process in the future and hopefully capture all
> > these
> > kind of concerns earlier.
> >
> > Best!
> >
> > -Yi
> >
> > On Thu, Mar 30, 2017 at 7:45 AM, Thomas Becker 
> > wrote:
> >
> > >
> > > Yes, we were burned by this. The changelog mapping will be
> > > regenerated
> > > instead of migrated and the result will completely hose the job
> > > (because the mapping was not generated deterministically in
> > > previous
> > > versions of Samza). I don't understand why the migration code was
> > > removed but it was, and to the best of my knowledge the necessity
> > > to
> > > not skip version 0.10.0 when upgrading was not documented, let
> > > alone
> > > enforced.
> > >
> > > On Mon, 2017-03-27 at 10:07 -0700, Jagadish Venkatraman wrote:
> > > >
> > > > Good observation Jake!
> > > >
> > > > The code for migration was removed in Samza 11. The migration
> > > > would
> > > > read
> > > > change-log offsets from the checkpoint topic and write them to
> > > > the
> > > > coordinator stream.
> > > >
> > > > If you're using change-logged stores, I'd recommend upgrading
> > > > from
> > > > 0.9.1 to
> > > > 0.10.0 first.
> > > > Otherwise, you will loose offsets for change-logged stores.
> > > >
> > > > I suspect you should be okay for 0.10.0 to 0.12 upgrade.
> > > >
> > > > On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes  > > > >
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > As I recall, samza 0.10 introduced the coordinator stream and
> > > > > there
> > > > > was
> > > > > code to do an automatic migration to use that feature. @navina,
> > > > > @yi, do you
> > > > > know if that migration code is still in samza 12?
> > > > >
> > > > > If not, then it's probably better to update from 0.9.1 to
> > > > > 0.10.0
> > > > > and then
> > > > > to 0.12.0. I don't think there were any changes requiring
> > > > > migration
> > > > > between
> > > > > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is
> > > > > probably
> > > > > less of
> > > > > an issue.
> > > > >
> > > > > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman <
> > > > > jagadish1...@gmail.com> wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > Hi Xiaochuan,
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Do I need to upgrade Kafka and/or YARN?
> > > > > > *Yarn version:*
> > > > > >
> > > > > >- Samza 0.12 supports Yarn 2.6.1 and 2.7.1.
> > > > > >- If you already have 2.6.0 installed (as you have said),
> > > > > > I
> > > > > > believe
> > > > > you
> > > > > >
> > > > > >
> > > > > >will be fine. (but I'm not sure)
> > > > > >
> > > > > > *Kafka version: *
> > > > > >
> > > > > >- Samza 0.12 upgraded the version of Kafka to 0.10.
> > > > > >- If your Kafka brokers are on an older version of Kafka,
> > > > > > you
> > > > > > should
> > > > > >upgrade them to use at-least 0.10. Kafka clients are
> > > > > > usually
> > > > > >incompatible with older versions of brokers.
> > > > > >
> > > > > > *Java version: *
> > > > > >
> > > > > >
> > > > > >
> > > > > >- Samza 0.12 binaries are compiled using Java 8.  Hence,
> > > > > > they
> > > > > > cannot
> > > > > be
> > > > > >
> > > > > >
> > > > > >run on older versions of the Java run-time.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I'm extremely new to Samza in terms of operations aspect.
> > > > > > > > I'm
> > > > > > > > not sure
> > > > > > what
> > > > > > information would be relevant in this case so please ask
> > > > > > away.
> > > > > >
> > > > > > I'd 

Re: Steps to Upgrading Samza (0.9 to 0.12)

[Thomas Becker]

Thanks for the reply Yi, and I apologize if I came off a bit snarky.
I'm not sure I agree with the policy (removing migration code and
wanting people to upgrade seem at odds to me), but minimally I think we
should not assume people are upgrading to each new Samza version. We
have done so when features or fixes warrant, and even then on a per-job
basis, and I would expect this is a common practice.

-Tommy

On Thu, 2017-03-30 at 09:50 -0700, Yi Pan wrote:
> Hi, Thomas,
>
> Sorry to hear that you were hit by the removal of migration in Samza
> 0.11.
> The reason we removed it is following a deprecate-removal policy in
> two
> versions. We are not aware that people still using 0.9 after we
> released
> 0.11 and were not expecting a direct upgrade from 0.9 to 0.12.
> Document can
> be better to capture that. We are making changes to the design
> proposal
> s.t. it is more transparent and open to the whole community, through
> the
> newly proposed SEP process. These kind of breaking changes will go
> through
> the SEP discuss-vote process in the future and hopefully capture all
> these
> kind of concerns earlier.
>
> Best!
>
> -Yi
>
> On Thu, Mar 30, 2017 at 7:45 AM, Thomas Becker 
> wrote:
>
> >
> > Yes, we were burned by this. The changelog mapping will be
> > regenerated
> > instead of migrated and the result will completely hose the job
> > (because the mapping was not generated deterministically in
> > previous
> > versions of Samza). I don't understand why the migration code was
> > removed but it was, and to the best of my knowledge the necessity
> > to
> > not skip version 0.10.0 when upgrading was not documented, let
> > alone
> > enforced.
> >
> > On Mon, 2017-03-27 at 10:07 -0700, Jagadish Venkatraman wrote:
> > >
> > > Good observation Jake!
> > >
> > > The code for migration was removed in Samza 11. The migration
> > > would
> > > read
> > > change-log offsets from the checkpoint topic and write them to
> > > the
> > > coordinator stream.
> > >
> > > If you're using change-logged stores, I'd recommend upgrading
> > > from
> > > 0.9.1 to
> > > 0.10.0 first.
> > > Otherwise, you will loose offsets for change-logged stores.
> > >
> > > I suspect you should be okay for 0.10.0 to 0.12 upgrade.
> > >
> > > On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes  > > >
> > > wrote:
> > >
> > > >
> > > >
> > > > As I recall, samza 0.10 introduced the coordinator stream and
> > > > there
> > > > was
> > > > code to do an automatic migration to use that feature. @navina,
> > > > @yi, do you
> > > > know if that migration code is still in samza 12?
> > > >
> > > > If not, then it's probably better to update from 0.9.1 to
> > > > 0.10.0
> > > > and then
> > > > to 0.12.0. I don't think there were any changes requiring
> > > > migration
> > > > between
> > > > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is
> > > > probably
> > > > less of
> > > > an issue.
> > > >
> > > > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman <
> > > > jagadish1...@gmail.com> wrote:
> > > >
> > > > >
> > > > >
> > > > > Hi Xiaochuan,
> > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Do I need to upgrade Kafka and/or YARN?
> > > > > *Yarn version:*
> > > > >
> > > > >- Samza 0.12 supports Yarn 2.6.1 and 2.7.1.
> > > > >- If you already have 2.6.0 installed (as you have said),
> > > > > I
> > > > > believe
> > > > you
> > > > >
> > > > >
> > > > >will be fine. (but I'm not sure)
> > > > >
> > > > > *Kafka version: *
> > > > >
> > > > >- Samza 0.12 upgraded the version of Kafka to 0.10.
> > > > >- If your Kafka brokers are on an older version of Kafka,
> > > > > you
> > > > > should
> > > > >upgrade them to use at-least 0.10. Kafka clients are
> > > > > usually
> > > > >incompatible with older versions of brokers.
> > > > >
> > > > > *Java version: *
> > > > >
> > > > >
> > > > >
> > > > >- Samza 0.12 binaries are compiled using Java 8.  Hence,
> > > > > they
> > > > > cannot
> > > > be
> > > > >
> > > > >
> > > > >run on older versions of the Java run-time.
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I'm extremely new to Samza in terms of operations aspect.
> > > > > > > I'm
> > > > > > > not sure
> > > > > what
> > > > > information would be relevant in this case so please ask
> > > > > away.
> > > > >
> > > > > I'd first start by upgrading the Kafka brokers (assuming
> > > > > you're
> > > > > on Java
> > > > 8+
> > > > >
> > > > >
> > > > > already).
> > > > > Let us know how the migration goes!
> > > > >
> > > > > Thanks,
> > > > > Jagadish
> > > > >
> > > > >
> > > > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu  > > > > ik.c
> > > > > om>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > What are the general steps for upgrading Samza from 0.9 to
> > > > > > 0.12?
> > > > > > Do I need to upgrade Kafka and/or YARN?
> > > > 

Re: Steps to Upgrading Samza (0.9 to 0.12)

[Yi Pan]

Hi, Thomas,

Sorry to hear that you were hit by the removal of migration in Samza 0.11.
The reason we removed it is following a deprecate-removal policy in two
versions. We are not aware that people still using 0.9 after we released
0.11 and were not expecting a direct upgrade from 0.9 to 0.12. Document can
be better to capture that. We are making changes to the design proposal
s.t. it is more transparent and open to the whole community, through the
newly proposed SEP process. These kind of breaking changes will go through
the SEP discuss-vote process in the future and hopefully capture all these
kind of concerns earlier.

Best!

-Yi

On Thu, Mar 30, 2017 at 7:45 AM, Thomas Becker  wrote:

> Yes, we were burned by this. The changelog mapping will be regenerated
> instead of migrated and the result will completely hose the job
> (because the mapping was not generated deterministically in previous
> versions of Samza). I don't understand why the migration code was
> removed but it was, and to the best of my knowledge the necessity to
> not skip version 0.10.0 when upgrading was not documented, let alone
> enforced.
>
> On Mon, 2017-03-27 at 10:07 -0700, Jagadish Venkatraman wrote:
> > Good observation Jake!
> >
> > The code for migration was removed in Samza 11. The migration would
> > read
> > change-log offsets from the checkpoint topic and write them to the
> > coordinator stream.
> >
> > If you're using change-logged stores, I'd recommend upgrading from
> > 0.9.1 to
> > 0.10.0 first.
> > Otherwise, you will loose offsets for change-logged stores.
> >
> > I suspect you should be okay for 0.10.0 to 0.12 upgrade.
> >
> > On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes 
> > wrote:
> >
> > >
> > > As I recall, samza 0.10 introduced the coordinator stream and there
> > > was
> > > code to do an automatic migration to use that feature. @navina,
> > > @yi, do you
> > > know if that migration code is still in samza 12?
> > >
> > > If not, then it's probably better to update from 0.9.1 to 0.10.0
> > > and then
> > > to 0.12.0. I don't think there were any changes requiring migration
> > > between
> > > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is probably
> > > less of
> > > an issue.
> > >
> > > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman <
> > > jagadish1...@gmail.com> wrote:
> > >
> > > >
> > > > Hi Xiaochuan,
> > > >
> > > > >
> > > > > >
> > > > > > Do I need to upgrade Kafka and/or YARN?
> > > > *Yarn version:*
> > > >
> > > >- Samza 0.12 supports Yarn 2.6.1 and 2.7.1.
> > > >- If you already have 2.6.0 installed (as you have said), I
> > > > believe
> > > you
> > > >
> > > >will be fine. (but I'm not sure)
> > > >
> > > > *Kafka version: *
> > > >
> > > >- Samza 0.12 upgraded the version of Kafka to 0.10.
> > > >- If your Kafka brokers are on an older version of Kafka, you
> > > > should
> > > >upgrade them to use at-least 0.10. Kafka clients are usually
> > > >incompatible with older versions of brokers.
> > > >
> > > > *Java version: *
> > > >
> > > >
> > > >
> > > >- Samza 0.12 binaries are compiled using Java 8.  Hence, they
> > > > cannot
> > > be
> > > >
> > > >run on older versions of the Java run-time.
> > > >
> > > >
> > > > >
> > > > > >
> > > > > > I'm extremely new to Samza in terms of operations aspect. I'm
> > > > > > not sure
> > > > what
> > > > information would be relevant in this case so please ask away.
> > > >
> > > > I'd first start by upgrading the Kafka brokers (assuming you're
> > > > on Java
> > > 8+
> > > >
> > > > already).
> > > > Let us know how the migration goes!
> > > >
> > > > Thanks,
> > > > Jagadish
> > > >
> > > >
> > > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu  > > > om>
> > > > wrote:
> > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > What are the general steps for upgrading Samza from 0.9 to
> > > > > 0.12?
> > > > > Do I need to upgrade Kafka and/or YARN?
> > > > >
> > > > > I don't know how Samza was setup initially but we currently
> > > > > have the
> > > > > following setup:
> > > > >
> > > > > Samza version: 0.9.1
> > > > > YARN version: Hadoop 2.6.0-cdh5.4.8
> > > > > Kafka version: 0.9.0.1
> > > > >
> > > > > I think installation of Kafka and YARN were managed through
> > > > > Puppet.
> > > > > I'm extremely new to Samza in terms of operations aspect. I'm
> > > > > not sure
> > > > what
> > > > >
> > > > > information would be relevant in this case so please ask away.
> > > > >
> > > > > Thanks,
> > > > > Xiaochuan Yu
> > > > >
> > > >
> > > >
> > > > --
> > > > Jagadish V,
> > > > Graduate Student,
> > > > Department of Computer Science,
> > > > Stanford University
> > > >
> >
> >
> --
>
>
> Tommy Becker
>
> Senior Software Engineer
>
> O +1 919.460.4747
>
> tivo.com
>
>
> 
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the 

Re: Steps to Upgrading Samza (0.9 to 0.12)

[Thomas Becker]

Yes, we were burned by this. The changelog mapping will be regenerated
instead of migrated and the result will completely hose the job
(because the mapping was not generated deterministically in previous
versions of Samza). I don't understand why the migration code was
removed but it was, and to the best of my knowledge the necessity to
not skip version 0.10.0 when upgrading was not documented, let alone
enforced.

On Mon, 2017-03-27 at 10:07 -0700, Jagadish Venkatraman wrote:
> Good observation Jake!
>
> The code for migration was removed in Samza 11. The migration would
> read
> change-log offsets from the checkpoint topic and write them to the
> coordinator stream.
>
> If you're using change-logged stores, I'd recommend upgrading from
> 0.9.1 to
> 0.10.0 first.
> Otherwise, you will loose offsets for change-logged stores.
>
> I suspect you should be okay for 0.10.0 to 0.12 upgrade.
>
> On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes 
> wrote:
>
> >
> > As I recall, samza 0.10 introduced the coordinator stream and there
> > was
> > code to do an automatic migration to use that feature. @navina,
> > @yi, do you
> > know if that migration code is still in samza 12?
> >
> > If not, then it's probably better to update from 0.9.1 to 0.10.0
> > and then
> > to 0.12.0. I don't think there were any changes requiring migration
> > between
> > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is probably
> > less of
> > an issue.
> >
> > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman <
> > jagadish1...@gmail.com> wrote:
> >
> > >
> > > Hi Xiaochuan,
> > >
> > > >
> > > > >
> > > > > Do I need to upgrade Kafka and/or YARN?
> > > *Yarn version:*
> > >
> > >- Samza 0.12 supports Yarn 2.6.1 and 2.7.1.
> > >- If you already have 2.6.0 installed (as you have said), I
> > > believe
> > you
> > >
> > >will be fine. (but I'm not sure)
> > >
> > > *Kafka version: *
> > >
> > >- Samza 0.12 upgraded the version of Kafka to 0.10.
> > >- If your Kafka brokers are on an older version of Kafka, you
> > > should
> > >upgrade them to use at-least 0.10. Kafka clients are usually
> > >incompatible with older versions of brokers.
> > >
> > > *Java version: *
> > >
> > >
> > >
> > >- Samza 0.12 binaries are compiled using Java 8.  Hence, they
> > > cannot
> > be
> > >
> > >run on older versions of the Java run-time.
> > >
> > >
> > > >
> > > > >
> > > > > I'm extremely new to Samza in terms of operations aspect. I'm
> > > > > not sure
> > > what
> > > information would be relevant in this case so please ask away.
> > >
> > > I'd first start by upgrading the Kafka brokers (assuming you're
> > > on Java
> > 8+
> > >
> > > already).
> > > Let us know how the migration goes!
> > >
> > > Thanks,
> > > Jagadish
> > >
> > >
> > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu  > > om>
> > > wrote:
> > >
> > > >
> > > > Hi,
> > > >
> > > > What are the general steps for upgrading Samza from 0.9 to
> > > > 0.12?
> > > > Do I need to upgrade Kafka and/or YARN?
> > > >
> > > > I don't know how Samza was setup initially but we currently
> > > > have the
> > > > following setup:
> > > >
> > > > Samza version: 0.9.1
> > > > YARN version: Hadoop 2.6.0-cdh5.4.8
> > > > Kafka version: 0.9.0.1
> > > >
> > > > I think installation of Kafka and YARN were managed through
> > > > Puppet.
> > > > I'm extremely new to Samza in terms of operations aspect. I'm
> > > > not sure
> > > what
> > > >
> > > > information would be relevant in this case so please ask away.
> > > >
> > > > Thanks,
> > > > Xiaochuan Yu
> > > >
> > >
> > >
> > > --
> > > Jagadish V,
> > > Graduate Student,
> > > Department of Computer Science,
> > > Stanford University
> > >
>
>
--


Tommy Becker

Senior Software Engineer

O +1 919.460.4747

tivo.com




This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by 
email. Binding agreements with TiVo Inc. may only be made by a signed written 
agreement.

Re: Steps to Upgrading Samza (0.9 to 0.12)

[Maksim Logvinenko]

Hi colleagues,

If I understand samza source code correctly, without migration code we
would not lose offsets but lose TaskName-to-ChangelogPartition mapping.
State restore for TaskStorage starts from ChangelogSystemStreamPartition
beginning (if we don't use data locality, offsets to start from are stored
locally in the file then)
https://github.com/apache/samza/blob/0.12.0/samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala#L161-L161
.

Hence if we lose TaskName-to-ChangelogPartition mapping, or in case of
migration if we migrate from Samza 0.9 to Samza 0.11 or 0.12 without
intermediate migration to Samza 0.10, and if we still have data in our
ChangelogSystemStream, Samza will recreate TaskName-to-ChangelogPartition
mapping and restore state from newly selected
ChangelogSystemStreamPartition. Samza 0.12 sort collection by TaskName
during re-creating of this mapping (
https://github.com/apache/samza/blob/0.12.0/samza-core/src/main/scala/org/apache/samza/coordinator/JobModelManager.scala#L259-L259),
but Samza 0.9 does not (
https://github.com/apache/samza/blob/0.9.1/samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala#L142).
Hence, in case of migration from Samza 0.9, we can end up with the wrong
state restored for TaskStorages, because there is no guarantee for
iteration order in Map.

Please, correct me if I'm wrong.

Best regards,
Maxim Logvinenko

On 27 March 2017 at 20:58:24, Navina Ramesh (Apache) (nav...@apache.org)
wrote:

@Jake: Yes. We removed the migration code (for 0.9 to 0.10) in the 0.11
release, I believe.

@XiaoChuan: As per Jagadish's recommendation, if you have changelog backed
stores, you should upgrade from 0.9.1 to 0.10.0 before upgrading to samza
0.12.0.

I checked with LinkedIn's internal release notes. The most significant
change listed is adding a new configuration *job.coordinator.system*. This
system can be the same as your currently configured checkpoint system
(task.checkpoint.system). I am assuming you are using
KafkaCheckpointManagerFactory. If you are using other custom checkpoint
managers, the migration may be more involved. Please let us know and we can
try to help you out.

Feel free to email us if you have more questions.

Cheers!
Navina

On Mon, Mar 27, 2017 at 10:07 AM, Jagadish Venkatraman <
jagadish1...@gmail.com> wrote:

> Good observation Jake!
>
> The code for migration was removed in Samza 11. The migration would read
> change-log offsets from the checkpoint topic and write them to the
> coordinator stream.
>
> If you're using change-logged stores, I'd recommend upgrading from 0.9.1
to
> 0.10.0 first.
> Otherwise, you will loose offsets for change-logged stores.
>
> I suspect you should be okay for 0.10.0 to 0.12 upgrade.
>
> On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes  wrote:
>
> > As I recall, samza 0.10 introduced the coordinator stream and there was
> > code to do an automatic migration to use that feature. @navina, @yi, do
> you
> > know if that migration code is still in samza 12?
> >
> > If not, then it's probably better to update from 0.9.1 to 0.10.0 and
then
> > to 0.12.0. I don't think there were any changes requiring migration
> between
> > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is probably less
> of
> > an issue.
> >
> > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman <
> > jagadish1...@gmail.com> wrote:
> >
> > > Hi Xiaochuan,
> > >
> > > >> Do I need to upgrade Kafka and/or YARN?
> > >
> > > *Yarn version:*
> > >
> > > - Samza 0.12 supports Yarn 2.6.1 and 2.7.1.
> > > - If you already have 2.6.0 installed (as you have said), I believe
> > you
> > > will be fine. (but I'm not sure)
> > >
> > > *Kafka version: *
> > >
> > > - Samza 0.12 upgraded the version of Kafka to 0.10.
> > > - If your Kafka brokers are on an older version of Kafka, you should
> > > upgrade them to use at-least 0.10. Kafka clients are usually
> > > incompatible with older versions of brokers.
> > >
> > > *Java version: *
> > >
> > >
> > >
> > > - Samza 0.12 binaries are compiled using Java 8. Hence, they cannot
> > be
> > > run on older versions of the Java run-time.
> > >
> > >
> > > >> I'm extremely new to Samza in terms of operations aspect. I'm not
> sure
> > > what
> > > information would be relevant in this case so please ask away.
> > >
> > > I'd first start by upgrading the Kafka brokers (assuming you're on
Java
> > 8+
> > > already).
> > > Let us know how the migration goes!
> > >
> > > Thanks,
> > > Jagadish
> > >
> > >
> > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > What are the general steps for upgrading Samza from 0.9 to 0.12?
> > > > Do I need to upgrade Kafka and/or YARN?
> > > >
> > > > I don't know how Samza was setup initially but we currently have
the
> > > > following setup:
> > > >
> > > > Samza version: 0.9.1
> > > > YARN version: Hadoop 2.6.0-cdh5.4.8
> > > > Kafka version: 0.9.0.1
> > > >
> > 

Re: Steps to Upgrading Samza (0.9 to 0.12)

[Navina Ramesh (Apache)]

@Jake: Yes. We removed the migration code (for 0.9 to 0.10) in the 0.11
release, I believe.

@XiaoChuan: As per Jagadish's recommendation, if you have changelog backed
stores, you should upgrade from 0.9.1 to 0.10.0 before upgrading to samza
0.12.0.

I checked with LinkedIn's internal release notes. The most significant
change listed is adding a new configuration *job.coordinator.system*. This
system can be the same as your currently configured checkpoint system
(task.checkpoint.system). I am assuming you are using
KafkaCheckpointManagerFactory. If you are using other custom checkpoint
managers, the migration may be more involved. Please let us know and we can
try to help you out.

Feel free to email us if you have more questions.

Cheers!
Navina

On Mon, Mar 27, 2017 at 10:07 AM, Jagadish Venkatraman <
jagadish1...@gmail.com> wrote:

> Good observation Jake!
>
> The code for migration was removed in Samza 11. The migration would read
> change-log offsets from the checkpoint topic and write them to the
> coordinator stream.
>
> If you're using change-logged stores, I'd recommend upgrading from 0.9.1 to
> 0.10.0 first.
> Otherwise, you will loose offsets for change-logged stores.
>
> I suspect you should be okay for 0.10.0 to 0.12 upgrade.
>
> On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes  wrote:
>
> > As I recall, samza 0.10 introduced the coordinator stream and there was
> > code to do an automatic migration to use that feature. @navina, @yi, do
> you
> > know if that migration code is still in samza 12?
> >
> > If not, then it's probably better to update from 0.9.1 to 0.10.0 and then
> > to 0.12.0. I don't think there were any changes requiring migration
> between
> > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is probably less
> of
> > an issue.
> >
> > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman <
> > jagadish1...@gmail.com> wrote:
> >
> > > Hi Xiaochuan,
> > >
> > > >> Do I need to upgrade Kafka and/or YARN?
> > >
> > > *Yarn version:*
> > >
> > >- Samza 0.12 supports Yarn 2.6.1 and 2.7.1.
> > >- If you already have 2.6.0 installed (as you have said), I believe
> > you
> > >will be fine. (but I'm not sure)
> > >
> > > *Kafka version: *
> > >
> > >- Samza 0.12 upgraded the version of Kafka to 0.10.
> > >- If your Kafka brokers are on an older version of Kafka, you should
> > >upgrade them to use at-least 0.10. Kafka clients are usually
> > >incompatible with older versions of brokers.
> > >
> > > *Java version: *
> > >
> > >
> > >
> > >- Samza 0.12 binaries are compiled using Java 8.  Hence, they cannot
> > be
> > >run on older versions of the Java run-time.
> > >
> > >
> > > >> I'm extremely new to Samza in terms of operations aspect. I'm not
> sure
> > > what
> > > information would be relevant in this case so please ask away.
> > >
> > > I'd first start by upgrading the Kafka brokers (assuming you're on Java
> > 8+
> > > already).
> > > Let us know how the migration goes!
> > >
> > > Thanks,
> > > Jagadish
> > >
> > >
> > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > What are the general steps for upgrading Samza from 0.9 to 0.12?
> > > > Do I need to upgrade Kafka and/or YARN?
> > > >
> > > > I don't know how Samza was setup initially but we currently have the
> > > > following setup:
> > > >
> > > > Samza version: 0.9.1
> > > > YARN version: Hadoop 2.6.0-cdh5.4.8
> > > > Kafka version: 0.9.0.1
> > > >
> > > > I think installation of Kafka and YARN were managed through Puppet.
> > > > I'm extremely new to Samza in terms of operations aspect. I'm not
> sure
> > > what
> > > > information would be relevant in this case so please ask away.
> > > >
> > > > Thanks,
> > > > Xiaochuan Yu
> > > >
> > >
> > >
> > >
> > > --
> > > Jagadish V,
> > > Graduate Student,
> > > Department of Computer Science,
> > > Stanford University
> > >
> >
>
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
>

Re: Steps to Upgrading Samza (0.9 to 0.12)

[Jagadish Venkatraman]

Good observation Jake!

The code for migration was removed in Samza 11. The migration would read
change-log offsets from the checkpoint topic and write them to the
coordinator stream.

If you're using change-logged stores, I'd recommend upgrading from 0.9.1 to
0.10.0 first.
Otherwise, you will loose offsets for change-logged stores.

I suspect you should be okay for 0.10.0 to 0.12 upgrade.

On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes  wrote:

> As I recall, samza 0.10 introduced the coordinator stream and there was
> code to do an automatic migration to use that feature. @navina, @yi, do you
> know if that migration code is still in samza 12?
>
> If not, then it's probably better to update from 0.9.1 to 0.10.0 and then
> to 0.12.0. I don't think there were any changes requiring migration between
> 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is probably less of
> an issue.
>
> On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman <
> jagadish1...@gmail.com> wrote:
>
> > Hi Xiaochuan,
> >
> > >> Do I need to upgrade Kafka and/or YARN?
> >
> > *Yarn version:*
> >
> >- Samza 0.12 supports Yarn 2.6.1 and 2.7.1.
> >- If you already have 2.6.0 installed (as you have said), I believe
> you
> >will be fine. (but I'm not sure)
> >
> > *Kafka version: *
> >
> >- Samza 0.12 upgraded the version of Kafka to 0.10.
> >- If your Kafka brokers are on an older version of Kafka, you should
> >upgrade them to use at-least 0.10. Kafka clients are usually
> >incompatible with older versions of brokers.
> >
> > *Java version: *
> >
> >
> >
> >- Samza 0.12 binaries are compiled using Java 8.  Hence, they cannot
> be
> >run on older versions of the Java run-time.
> >
> >
> > >> I'm extremely new to Samza in terms of operations aspect. I'm not sure
> > what
> > information would be relevant in this case so please ask away.
> >
> > I'd first start by upgrading the Kafka brokers (assuming you're on Java
> 8+
> > already).
> > Let us know how the migration goes!
> >
> > Thanks,
> > Jagadish
> >
> >
> > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu 
> > wrote:
> >
> > > Hi,
> > >
> > > What are the general steps for upgrading Samza from 0.9 to 0.12?
> > > Do I need to upgrade Kafka and/or YARN?
> > >
> > > I don't know how Samza was setup initially but we currently have the
> > > following setup:
> > >
> > > Samza version: 0.9.1
> > > YARN version: Hadoop 2.6.0-cdh5.4.8
> > > Kafka version: 0.9.0.1
> > >
> > > I think installation of Kafka and YARN were managed through Puppet.
> > > I'm extremely new to Samza in terms of operations aspect. I'm not sure
> > what
> > > information would be relevant in this case so please ask away.
> > >
> > > Thanks,
> > > Xiaochuan Yu
> > >
> >
> >
> >
> > --
> > Jagadish V,
> > Graduate Student,
> > Department of Computer Science,
> > Stanford University
> >
>



-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University

Re: Steps to Upgrading Samza (0.9 to 0.12)

[Jagadish Venkatraman]

Good observation Jake!

The code for migration was removed in Samza 11. The migration would read
change-log offsets from the checkpoint topic and write them to the
coordinator stream.

If you're using change-logged stores, I'd recommend upgrading from 0.9.1 to
0.10.0 first.
Otherwise, you will loose offsets for change-logged stores.

I suspect you should be okay for 0.10.0 to 0.12 upgrade.

On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes  wrote:

> As I recall, samza 0.10 introduced the coordinator stream and there was
> code to do an automatic migration to use that feature. @navina, @yi, do you
> know if that migration code is still in samza 12?
>
> If not, then it's probably better to update from 0.9.1 to 0.10.0 and then
> to 0.12.0. I don't think there were any changes requiring migration between
> 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is probably less of
> an issue.
>
> On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman <
> jagadish1...@gmail.com> wrote:
>
> > Hi Xiaochuan,
> >
> > >> Do I need to upgrade Kafka and/or YARN?
> >
> > *Yarn version:*
> >
> >- Samza 0.12 supports Yarn 2.6.1 and 2.7.1.
> >- If you already have 2.6.0 installed (as you have said), I believe
> you
> >will be fine. (but I'm not sure)
> >
> > *Kafka version: *
> >
> >- Samza 0.12 upgraded the version of Kafka to 0.10.
> >- If your Kafka brokers are on an older version of Kafka, you should
> >upgrade them to use at-least 0.10. Kafka clients are usually
> >incompatible with older versions of brokers.
> >
> > *Java version: *
> >
> >
> >
> >- Samza 0.12 binaries are compiled using Java 8.  Hence, they cannot
> be
> >run on older versions of the Java run-time.
> >
> >
> > >> I'm extremely new to Samza in terms of operations aspect. I'm not sure
> > what
> > information would be relevant in this case so please ask away.
> >
> > I'd first start by upgrading the Kafka brokers (assuming you're on Java
> 8+
> > already).
> > Let us know how the migration goes!
> >
> > Thanks,
> > Jagadish
> >
> >
> > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu 
> > wrote:
> >
> > > Hi,
> > >
> > > What are the general steps for upgrading Samza from 0.9 to 0.12?
> > > Do I need to upgrade Kafka and/or YARN?
> > >
> > > I don't know how Samza was setup initially but we currently have the
> > > following setup:
> > >
> > > Samza version: 0.9.1
> > > YARN version: Hadoop 2.6.0-cdh5.4.8
> > > Kafka version: 0.9.0.1
> > >
> > > I think installation of Kafka and YARN were managed through Puppet.
> > > I'm extremely new to Samza in terms of operations aspect. I'm not sure
> > what
> > > information would be relevant in this case so please ask away.
> > >
> > > Thanks,
> > > Xiaochuan Yu
> > >
> >
> >
> >
> > --
> > Jagadish V,
> > Graduate Student,
> > Department of Computer Science,
> > Stanford University
> >
>



-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University

Re: Steps to Upgrading Samza (0.9 to 0.12)

[Jacob Maes]

As I recall, samza 0.10 introduced the coordinator stream and there was
code to do an automatic migration to use that feature. @navina, @yi, do you
know if that migration code is still in samza 12?

If not, then it's probably better to update from 0.9.1 to 0.10.0 and then
to 0.12.0. I don't think there were any changes requiring migration between
0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is probably less of
an issue.

On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman <
jagadish1...@gmail.com> wrote:

> Hi Xiaochuan,
>
> >> Do I need to upgrade Kafka and/or YARN?
>
> *Yarn version:*
>
>- Samza 0.12 supports Yarn 2.6.1 and 2.7.1.
>- If you already have 2.6.0 installed (as you have said), I believe you
>will be fine. (but I'm not sure)
>
> *Kafka version: *
>
>- Samza 0.12 upgraded the version of Kafka to 0.10.
>- If your Kafka brokers are on an older version of Kafka, you should
>upgrade them to use at-least 0.10. Kafka clients are usually
>incompatible with older versions of brokers.
>
> *Java version: *
>
>
>
>- Samza 0.12 binaries are compiled using Java 8.  Hence, they cannot be
>run on older versions of the Java run-time.
>
>
> >> I'm extremely new to Samza in terms of operations aspect. I'm not sure
> what
> information would be relevant in this case so please ask away.
>
> I'd first start by upgrading the Kafka brokers (assuming you're on Java 8+
> already).
> Let us know how the migration goes!
>
> Thanks,
> Jagadish
>
>
> On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu 
> wrote:
>
> > Hi,
> >
> > What are the general steps for upgrading Samza from 0.9 to 0.12?
> > Do I need to upgrade Kafka and/or YARN?
> >
> > I don't know how Samza was setup initially but we currently have the
> > following setup:
> >
> > Samza version: 0.9.1
> > YARN version: Hadoop 2.6.0-cdh5.4.8
> > Kafka version: 0.9.0.1
> >
> > I think installation of Kafka and YARN were managed through Puppet.
> > I'm extremely new to Samza in terms of operations aspect. I'm not sure
> what
> > information would be relevant in this case so please ask away.
> >
> > Thanks,
> > Xiaochuan Yu
> >
>
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
>

Re: Steps to Upgrading Samza (0.9 to 0.12)

[Jagadish Venkatraman]

Hi Xiaochuan,

>> Do I need to upgrade Kafka and/or YARN?

*Yarn version:*

   - Samza 0.12 supports Yarn 2.6.1 and 2.7.1.
   - If you already have 2.6.0 installed (as you have said), I believe you
   will be fine. (but I'm not sure)

*Kafka version: *

   - Samza 0.12 upgraded the version of Kafka to 0.10.
   - If your Kafka brokers are on an older version of Kafka, you should
   upgrade them to use at-least 0.10. Kafka clients are usually
   incompatible with older versions of brokers.

*Java version: *



   - Samza 0.12 binaries are compiled using Java 8.  Hence, they cannot be
   run on older versions of the Java run-time.


>> I'm extremely new to Samza in terms of operations aspect. I'm not sure
what
information would be relevant in this case so please ask away.

I'd first start by upgrading the Kafka brokers (assuming you're on Java 8+
already).
Let us know how the migration goes!

Thanks,
Jagadish


On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu  wrote:

> Hi,
>
> What are the general steps for upgrading Samza from 0.9 to 0.12?
> Do I need to upgrade Kafka and/or YARN?
>
> I don't know how Samza was setup initially but we currently have the
> following setup:
>
> Samza version: 0.9.1
> YARN version: Hadoop 2.6.0-cdh5.4.8
> Kafka version: 0.9.0.1
>
> I think installation of Kafka and YARN were managed through Puppet.
> I'm extremely new to Samza in terms of operations aspect. I'm not sure what
> information would be relevant in this case so please ask away.
>
> Thanks,
> Xiaochuan Yu
>



-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University