Re: Samza 0.12.0 artifact for scala 2.10 in maven central
Hi Jagadish, Thanks! Best regards, Maxim Logvinenko On 18 April 2017 at 21:29:11, Jagadish Venkatraman (jagadish1...@gmail.com) wrote: Hi Maksim, I published the artifacts. Please find the artifacts here <https://repository.apache.org/#nexus-search> For instance, https://repository.apache.org/#nexus-search;quick~samza-core_2.10 <https://repository.apache.org/#nexus-search;quick~samza-core_2.10> Best, Jagadish On Mon, Apr 17, 2017 at 12:44 PM, Maksim Logvinenko wrote: > Hi Jagadish, > > Do you have any updates about publishing Samza artifacts? > > Best regards, > Maxim Logvinenko > > On 4 April 2017 at 15:00:35, Maksim Logvinenko (mlogvine...@gmail.com) > wrote: > > Hi Jagadish, > > Cool, thanks a lot! > > Best regards, > Maxim Logvinenko > > On 4 April 2017 at 03:33:57, Jagadish Venkatraman (jagadish1...@gmail.com) > wrote: > > Hi Maxim, > > I generated, and published the Scala 2.10 artifacts to the staging > artifactory at : > > https://repository.apache.org/content/repositories/orgapachesamza-1019/ > > I will send out another update when we publish the actual binaries. > > Thanks, > Jagadish > > > On Wed, Mar 22, 2017 at 7:07 PM, Jagadish Venkatraman < > jagadish1...@gmail.com> wrote: > > > Hi Maksim, > > > > >> maven central shows that samza 0.12.0 build exists for scala 2.11 only > > > > This was not a mis-build (as was stated in the release notes for 0.12). > > > > >> it would be nice to have builds for both supported scala versions > > > > +1 to this suggestion. Thanks for the feedback. We will to get to > > providing these builds for scala 2.10 ASAP. I'll provide an update on > this > > thread when it's done. > > > > Cheers, > > Jagadish > > > > > > > > > > On Wed, Mar 22, 2017 at 12:56 PM, Maksim Logvinenko < > mlogvine...@gmail.com > > > wrote: > > > >> Hi guys! > >> > >> As far as I can get from samza source code it’s scala 2.10 compatible. > But > >> maven central shows that samza 0.12.0 build exists for scala 2.11 only. > Is > >> it intended or mis-build? I think it would be nice to have builds for > both > >> supported scala versions, because scala itself is not backward > compatible. > >> > >> Thanks, > >> Maxim Logvinenko > >> > > > > > > > > -- > > Jagadish V, > > Graduate Student, > > Department of Computer Science, > > Stanford University > > > > > > -- > Jagadish V, > Graduate Student, > Department of Computer Science, > Stanford University > > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
Re: Samza 0.12.0 artifact for scala 2.10 in maven central
Hi Jagadish, Do you have any updates about publishing Samza artifacts? Best regards, Maxim Logvinenko On 4 April 2017 at 15:00:35, Maksim Logvinenko (mlogvine...@gmail.com) wrote: Hi Jagadish, Cool, thanks a lot! Best regards, Maxim Logvinenko On 4 April 2017 at 03:33:57, Jagadish Venkatraman (jagadish1...@gmail.com) wrote: Hi Maxim, I generated, and published the Scala 2.10 artifacts to the staging artifactory at : https://repository.apache.org/content/repositories/orgapachesamza-1019/ I will send out another update when we publish the actual binaries. Thanks, Jagadish On Wed, Mar 22, 2017 at 7:07 PM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > Hi Maksim, > > >> maven central shows that samza 0.12.0 build exists for scala 2.11 only > > This was not a mis-build (as was stated in the release notes for 0.12). > > >> it would be nice to have builds for both supported scala versions > > +1 to this suggestion. Thanks for the feedback. We will to get to > providing these builds for scala 2.10 ASAP. I'll provide an update on this > thread when it's done. > > Cheers, > Jagadish > > > > > On Wed, Mar 22, 2017 at 12:56 PM, Maksim Logvinenko > wrote: > >> Hi guys! >> >> As far as I can get from samza source code it’s scala 2.10 compatible. But >> maven central shows that samza 0.12.0 build exists for scala 2.11 only. Is >> it intended or mis-build? I think it would be nice to have builds for both >> supported scala versions, because scala itself is not backward compatible. >> >> Thanks, >> Maxim Logvinenko >> > > > > -- > Jagadish V, > Graduate Student, > Department of Computer Science, > Stanford University > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
Re: Samza 0.12.0 artifact for scala 2.10 in maven central
Hi Jagadish, Cool, thanks a lot! Best regards, Maxim Logvinenko On 4 April 2017 at 03:33:57, Jagadish Venkatraman (jagadish1...@gmail.com) wrote: Hi Maxim, I generated, and published the Scala 2.10 artifacts to the staging artifactory at : https://repository.apache.org/content/repositories/orgapachesamza-1019/ I will send out another update when we publish the actual binaries. Thanks, Jagadish On Wed, Mar 22, 2017 at 7:07 PM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > Hi Maksim, > > >> maven central shows that samza 0.12.0 build exists for scala 2.11 only > > This was not a mis-build (as was stated in the release notes for 0.12). > > >> it would be nice to have builds for both supported scala versions > > +1 to this suggestion. Thanks for the feedback. We will to get to > providing these builds for scala 2.10 ASAP. I'll provide an update on this > thread when it's done. > > Cheers, > Jagadish > > > > > On Wed, Mar 22, 2017 at 12:56 PM, Maksim Logvinenko > wrote: > >> Hi guys! >> >> As far as I can get from samza source code it’s scala 2.10 compatible. But >> maven central shows that samza 0.12.0 build exists for scala 2.11 only. Is >> it intended or mis-build? I think it would be nice to have builds for both >> supported scala versions, because scala itself is not backward compatible. >> >> Thanks, >> Maxim Logvinenko >> > > > > -- > Jagadish V, > Graduate Student, > Department of Computer Science, > Stanford University > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
Re: [DISCUSS] Support Scala 2.12
Hi Jagadish, I’ve fixed compile errors, could you please check again? Best regards, Maxim Logvinenko On 31 March 2017 at 10:22:01, Jagadish Venkatraman (jagadish1...@gmail.com) wrote: Maxim, I tried to merge the PR but I was getting some build errors. Perhaps, it's due to issues in my local environment. Can you please run ./bin/check-all.sh after applying your PR on current master? Let me know if it succeeds. Thanks, Jagadish On Thu, Mar 30, 2017 at 3:46 PM, Prateek Maheshwari < pmaheshw...@linkedin.com.invalid> wrote: > Hi Maksim, > > Thanks for the PR comments and updates. Looks good to me too. > > - Prateek > > On Wed, Mar 29, 2017 at 12:17 PM, Prateek Maheshwari < > pmaheshw...@linkedin.com> wrote: > > > Hi Maksim, > > > > I'm in favor of adding Scala 2.12 support as well, thanks for the PR. > > I have a few questions about the way JavaConverter APIs and some of the > > conversions in the PR work. I'll try it out locally and update the PR > with > > feedback/questions soon. > > > > Thanks, > > Prateek > > > > > > On Tue, Mar 28, 2017 at 3:01 PM, Maksim Logvinenko < > mlogvine...@gmail.com> > > wrote: > > > >> Hi guys, > >> > >> As far as I can understand nobody is against having Scala 2.12 support > in > >> Samza master. Can we merge PR then? > >> > >> Best regards, > >> Maxim Logvinenko > >> > >> On 17 March 2017 at 23:42:16, Navina Ramesh > (nram...@linkedin.com.invalid > >> ) > >> wrote: > >> > >> Thanks for creating the DISCUSS email! > >> > >> This is good. It's a good idea to update to 2.12 since it looks like we > >> are > >> fully backward compatible with older versions. +1 from me. > >> > >> Cheers! > >> Navina > >> > >> On Fri, Mar 17, 2017 at 1:34 PM, Jagadish Venkatraman < > >> jagadish1...@gmail.com> wrote: > >> > >> > Thanks for starting this discussion and the patch. +1 for supporting > >> scala > >> > 2.12. I assume the changes are fully backwards compatible with scala > >> 2.10, > >> > 2.11 (as evidenced by your check-all)? > >> > > >> > Also, another observation is that the generated Samza binaries will > have > >> > 2.12 as the suffix for the future release (I this should be totally > OK). > >> > > >> > > >> > On Fri, Mar 17, 2017 at 1:26 PM, Maksim Logvinenko < > >> mlogvine...@gmail.com> > >> > >> > wrote: > >> > > >> > > Hi guys, > >> > > > >> > > I’ve created JIRA and already submitted patch which adds support of > >> scala > >> > > 2.12. Here is the ticket: https://issues.apache.org/ > >> > jira/browse/SAMZA-1135 > >> > > . > >> > > Nothing serious: I’ve removed JavaConversions usage (because it’s > >> marked > >> > as > >> > > deprecated now) and bumped kafka and scalatest versions since > previous > >> > > versions don’t have scala 2.12 support. I run ./bin/check-all.sh on > my > >> > > laptop and it was successful for all scala versions (2.10, 2.11 and > >> 2.12) > >> > > and for both YARN versions. > >> > > > >> > > Thanks, > >> > > Maxim Logvinenko > >> > > > >> > > >> > > >> > > >> > -- > >> > Jagadish V, > >> > Graduate Student, > >> > Department of Computer Science, > >> > Stanford University > >> > > >> > >> > >> > >> -- > >> Navina R. > >> > > > > > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
Re: [DISCUSS] Support Scala 2.12
Hi guys, As far as I can understand nobody is against having Scala 2.12 support in Samza master. Can we merge PR then? Best regards, Maxim Logvinenko On 17 March 2017 at 23:42:16, Navina Ramesh (nram...@linkedin.com.invalid) wrote: Thanks for creating the DISCUSS email! This is good. It's a good idea to update to 2.12 since it looks like we are fully backward compatible with older versions. +1 from me. Cheers! Navina On Fri, Mar 17, 2017 at 1:34 PM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > Thanks for starting this discussion and the patch. +1 for supporting scala > 2.12. I assume the changes are fully backwards compatible with scala 2.10, > 2.11 (as evidenced by your check-all)? > > Also, another observation is that the generated Samza binaries will have > 2.12 as the suffix for the future release (I this should be totally OK). > > > On Fri, Mar 17, 2017 at 1:26 PM, Maksim Logvinenko > wrote: > > > Hi guys, > > > > I’ve created JIRA and already submitted patch which adds support of scala > > 2.12. Here is the ticket: https://issues.apache.org/ > jira/browse/SAMZA-1135 > > . > > Nothing serious: I’ve removed JavaConversions usage (because it’s marked > as > > deprecated now) and bumped kafka and scalatest versions since previous > > versions don’t have scala 2.12 support. I run ./bin/check-all.sh on my > > laptop and it was successful for all scala versions (2.10, 2.11 and 2.12) > > and for both YARN versions. > > > > Thanks, > > Maxim Logvinenko > > > > > > -- > Jagadish V, > Graduate Student, > Department of Computer Science, > Stanford University > -- Navina R.
Re: Steps to Upgrading Samza (0.9 to 0.12)
Hi colleagues, If I understand samza source code correctly, without migration code we would not lose offsets but lose TaskName-to-ChangelogPartition mapping. State restore for TaskStorage starts from ChangelogSystemStreamPartition beginning (if we don't use data locality, offsets to start from are stored locally in the file then) https://github.com/apache/samza/blob/0.12.0/samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala#L161-L161 . Hence if we lose TaskName-to-ChangelogPartition mapping, or in case of migration if we migrate from Samza 0.9 to Samza 0.11 or 0.12 without intermediate migration to Samza 0.10, and if we still have data in our ChangelogSystemStream, Samza will recreate TaskName-to-ChangelogPartition mapping and restore state from newly selected ChangelogSystemStreamPartition. Samza 0.12 sort collection by TaskName during re-creating of this mapping ( https://github.com/apache/samza/blob/0.12.0/samza-core/src/main/scala/org/apache/samza/coordinator/JobModelManager.scala#L259-L259), but Samza 0.9 does not ( https://github.com/apache/samza/blob/0.9.1/samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala#L142). Hence, in case of migration from Samza 0.9, we can end up with the wrong state restored for TaskStorages, because there is no guarantee for iteration order in Map. Please, correct me if I'm wrong. Best regards, Maxim Logvinenko On 27 March 2017 at 20:58:24, Navina Ramesh (Apache) (nav...@apache.org) wrote: @Jake: Yes. We removed the migration code (for 0.9 to 0.10) in the 0.11 release, I believe. @XiaoChuan: As per Jagadish's recommendation, if you have changelog backed stores, you should upgrade from 0.9.1 to 0.10.0 before upgrading to samza 0.12.0. I checked with LinkedIn's internal release notes. The most significant change listed is adding a new configuration *job.coordinator.system*. This system can be the same as your currently configured checkpoint system (task.checkpoint.system). I am assuming you are using KafkaCheckpointManagerFactory. If you are using other custom checkpoint managers, the migration may be more involved. Please let us know and we can try to help you out. Feel free to email us if you have more questions. Cheers! Navina On Mon, Mar 27, 2017 at 10:07 AM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > Good observation Jake! > > The code for migration was removed in Samza 11. The migration would read > change-log offsets from the checkpoint topic and write them to the > coordinator stream. > > If you're using change-logged stores, I'd recommend upgrading from 0.9.1 to > 0.10.0 first. > Otherwise, you will loose offsets for change-logged stores. > > I suspect you should be okay for 0.10.0 to 0.12 upgrade. > > On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes wrote: > > > As I recall, samza 0.10 introduced the coordinator stream and there was > > code to do an automatic migration to use that feature. @navina, @yi, do > you > > know if that migration code is still in samza 12? > > > > If not, then it's probably better to update from 0.9.1 to 0.10.0 and then > > to 0.12.0. I don't think there were any changes requiring migration > between > > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is probably less > of > > an issue. > > > > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman < > > jagadish1...@gmail.com> wrote: > > > > > Hi Xiaochuan, > > > > > > >> Do I need to upgrade Kafka and/or YARN? > > > > > > *Yarn version:* > > > > > > - Samza 0.12 supports Yarn 2.6.1 and 2.7.1. > > > - If you already have 2.6.0 installed (as you have said), I believe > > you > > > will be fine. (but I'm not sure) > > > > > > *Kafka version: * > > > > > > - Samza 0.12 upgraded the version of Kafka to 0.10. > > > - If your Kafka brokers are on an older version of Kafka, you should > > > upgrade them to use at-least 0.10. Kafka clients are usually > > > incompatible with older versions of brokers. > > > > > > *Java version: * > > > > > > > > > > > > - Samza 0.12 binaries are compiled using Java 8. Hence, they cannot > > be > > > run on older versions of the Java run-time. > > > > > > > > > >> I'm extremely new to Samza in terms of operations aspect. I'm not > sure > > > what > > > information would be relevant in this case so please ask away. > > > > > > I'd first start by upgrading the Kafka brokers (assuming you're on Java > > 8+ > > > already). > > > Let us know how the migration goes! > > > > > > Thanks, > > > Jagadish > > > > > > > > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu > > > wrote: > > > > > > > Hi, > > > > > > > > What are the general steps for upgrading Samza from 0.9 to 0.12? > > > > Do I need to upgrade Kafka and/or YARN? > > > > > > > > I don't know how Samza was setup initially but we currently have the > > > > following setup: > > > > > > > > Samza version: 0.9.1 > > > > YARN version: Hadoop 2.6.0-cdh5.4.8 > > > > Kafka version: 0.9.0.1 > > > > > > > > I think installation of Kafka and YARN w
Samza 0.12.0 artifact for scala 2.10 in maven central
Hi guys! As far as I can get from samza source code it’s scala 2.10 compatible. But maven central shows that samza 0.12.0 build exists for scala 2.11 only. Is it intended or mis-build? I think it would be nice to have builds for both supported scala versions, because scala itself is not backward compatible. Thanks, Maxim Logvinenko
[DISCUSS] Support Scala 2.12
Hi guys, I’ve created JIRA and already submitted patch which adds support of scala 2.12. Here is the ticket: https://issues.apache.org/jira/browse/SAMZA-1135. Nothing serious: I’ve removed JavaConversions usage (because it’s marked as deprecated now) and bumped kafka and scalatest versions since previous versions don’t have scala 2.12 support. I run ./bin/check-all.sh on my laptop and it was successful for all scala versions (2.10, 2.11 and 2.12) and for both YARN versions. Thanks, Maxim Logvinenko
