Re: [VOTE] Apache Samza 1.7.0 RC0

[Boris Shkolnik]

-1
I am getting an error when running the integ tests.
See logs attached.

On Mon, Feb 28, 2022 at 5:28 PM Daniel Chen  wrote:

> Hey all, This is a call for a vote on a release of Apache Samza 1.7.0.
> Thanks to everyone who has contributed to this release.
>
> The release candidate can be downloaded from here:
>
> https://home.apache.org/~dchen/samza-1.7.0-rc0/
>
> The release candidate is signed with pgp key 1D9ADCE059431C34, which is
> included in the repository's KEYS file:
> https://git-wip-us.apache.org/repos/asf?p=samza.git;a=blob_plain;f=KEYS;hb=c5831bfc01b2e70ba57c4bd3505c6a84a73c8a7b
> and can also be found on keyservers:
>
>
> https://keyserver.ubuntu.com/pks/lookup?search=dchen%40apache.org=on=index
>
> The git tag is release-1.7.0-rc0 and signed with the same pgp key:
>
>
> https://gitbox.apache.org/repos/asf?p=samza.git;a=tag;h=refs/tags/release-1.7.0-rc0
>
> Test binaries have been published to Maven's staging repository, and are
> available here:
>
> Scala 2.11:
> https://repository.apache.org/content/repositories/orgapachesamza-1090/
> Scala 2.12:
> https://repository.apache.org/content/repositories/orgapachesamza-1091/
>
> The vote will be open for 72 hours ( end in 5:00pm Thursday, 03/03/2022 ).
> Please download the release candidate, check the hashes/signature, build it
> and test it, and then please vote: [ ] +1 approve [ ] +0 no opinion [ ] -1
> disapprove (and reason why)
>
> I ran check-all.sh and bor...@apache.org helped ran integration tests
> (both YARN and standalone) passed.
>
> +1 from my side for the release.
> Thanks,
> Daniel
>

Apache Samza 1.2.0 is released

[Boris Shkolnik]

Documentation and Blog are published.
Thanks everyone.

Re: REMINDER. [VOTE] Apache Samza 1.2.0 RC4

[Boris Shkolnik]

Hi,
Vote passed.
Thanks.

On Sun, Jun 2, 2019 at 5:00 PM Boris Shkolnik  wrote:

> Hi,
>
> This is a call for a vote on a release of Apache Samza 1.2.0. Thanks to
> everyone who has contributed to this release.
>
>
> The release candidate can be downloaded from here:
> http://home.apache.org/~boryas/samza-1.2.0-rc
> <http://home.apache.org/~boryas/samza-1.2.0-rc0/>4
>
> (this release has a fix for standalone integration test)
>
> The release candidate is signed with pgp key 0x7D74D0CD5B5EB041, which can
> be found
> http://keyserver.ubuntu.com/pks/lookup?op=get=0x7d74d0cd5b5eb041
> <http://keyserver.ubuntu.com/pks/lookup?op=get=0xF8B95961A401BF0F>
> The git tag is release-1.2.0-rc4 and signed with the same pgp key:
>
> https://gitbox.apache.org/repos/asf?p=samza.git;a=tag;h=refs/tags/release-1.2.0-rc
> <https://gitbox.apache.org/repos/asf?p=samza.git;a=tag;h=refs/tags/release-1.1.0-rc1>
> 4
>
> Test binaries have been published to Maven's staging repository, and are
> available here:
> https://repository.apache.org/content/repositories/orgapachesamza-106
> <https://repository.apache.org/content/repositories/orgapachesamza-1065/org/>
> 9
>
> The vote will be open until 06:00 PM PST Monday, 06/03/2019.
>
>
> Please download the release candidate, check the hashes/signature, build it
> and test it, and then please vote:
>
> [ ] +1 approve
>
> [ ] +0 no opinion
>
> [ ] -1 disapprove (and reason why)
>
> I ran check-all.sh and integration tests.
>
> +1 from my side.
>
> Thanks
>

REMINDER. [VOTE] Apache Samza 1.2.0 RC4

[Boris Shkolnik]

Hi,

This is a call for a vote on a release of Apache Samza 1.2.0. Thanks to
everyone who has contributed to this release.


The release candidate can be downloaded from here:
http://home.apache.org/~boryas/samza-1.2.0-rc
4

(this release has a fix for standalone integration test)

The release candidate is signed with pgp key 0x7D74D0CD5B5EB041, which can
be found
http://keyserver.ubuntu.com/pks/lookup?op=get=0x7d74d0cd5b5eb041

The git tag is release-1.2.0-rc4 and signed with the same pgp key:
https://gitbox.apache.org/repos/asf?p=samza.git;a=tag;h=refs/tags/release-1.2.0-rc

4

Test binaries have been published to Maven's staging repository, and are
available here:
https://repository.apache.org/content/repositories/orgapachesamza-106

9

The vote will be open until 06:00 PM PST Monday, 06/03/2019.


Please download the release candidate, check the hashes/signature, build it
and test it, and then please vote:

[ ] +1 approve

[ ] +0 no opinion

[ ] -1 disapprove (and reason why)

I ran check-all.sh and integration tests.

+1 from my side.

Thanks

Canceled: [VOTE] Apache Samza 1.2.0 RC3

[Boris Shkolnik]

On Mon, May 27, 2019 at 2:48 PM Boris Shkolnik  wrote:

> Hi,
>
> This is a call for a vote on a release of Apache Samza 1.2.0. Thanks to
> everyone who has contributed to this release.
>
>
> In this release we reverted commit:
>
> "SAMZA-2155: Remove log4j log4j2 dependency from samza-test"
>
> commit f260be6be5bb184073a2187edc5898b6cba7dea1
>
>
> The release candidate can be downloaded from here:
> http://home.apache.org/~boryas/samza-1.2.0-rc3/
> <http://home.apache.org/~boryas/samza-1.2.0-rc0/>
>
> The release candidate is signed with pgp key 0x7D74D0CD5B5EB041, which can
> be found
> http://keyserver.ubuntu.com/pks/lookup?op=get=0x7d74d0cd5b5eb041
> <http://keyserver.ubuntu.com/pks/lookup?op=get=0xF8B95961A401BF0F>
> The git tag is release-1.2.0-rc3 and signed with the same pgp key:
>
> https://gitbox.apache.org/repos/asf?p=samza.git;a=tag;h=refs/tags/release-1.2.0-rc
> <https://gitbox.apache.org/repos/asf?p=samza.git;a=tag;h=refs/tags/release-1.1.0-rc1>
> 3
>
> Test binaries have been published to Maven's staging repository, and are
> available here:
> https://repository.apache.org/content/repositories/orgapachesamza-106
> <https://repository.apache.org/content/repositories/orgapachesamza-1065/org/>
> 7
>
> The vote will be open until 02:00 PM PST Thursday, 05/24/2019.
>
>
> I've ran check-all.sh and integration tests.
>
>
> Please download the release candidate, check the hashes/signature, build it
> and test it, and then please vote:
>
> [ ] +1 approve
>
> [ ] +0 no opinion
>
> [ ] -1 disapprove (and reason why)
>
> I ran check-all.sh and integration tests.
>
> +1 (non-binding) from my side.
>
> Thanks,
>

[VOTE] Apache Samza 1.2.0 RC4

[Boris Shkolnik]

Hi,

This is a call for a vote on a release of Apache Samza 1.2.0. Thanks to
everyone who has contributed to this release.


The release candidate can be downloaded from here:
http://home.apache.org/~boryas/samza-1.2.0-rc
4

(this release has a fix for standalone integration test)

The release candidate is signed with pgp key 0x7D74D0CD5B5EB041, which can
be found
http://keyserver.ubuntu.com/pks/lookup?op=get=0x7d74d0cd5b5eb041

The git tag is release-1.2.0-rc4 and signed with the same pgp key:
https://gitbox.apache.org/repos/asf?p=samza.git;a=tag;h=refs/tags/release-1.2.0-rc

4

Test binaries have been published to Maven's staging repository, and are
available here:
https://repository.apache.org/content/repositories/orgapachesamza-106

8

The vote will be open until 10:00 PM PST Friday, 05/31/2019.


Please download the release candidate, check the hashes/signature, build it
and test it, and then please vote:

[ ] +1 approve

[ ] +0 no opinion

[ ] -1 disapprove (and reason why)

I ran check-all.sh and integration tests.

+1 from my side.

Thanks,

[VOTE] Apache Samza 1.2.0 RC1

[Boris Shkolnik]

 Hi,

This is a call for a vote on a release of Apache Samza 1.2.0. Thanks to
everyone who has contributed to this release.

The release candidate can be downloaded from here:
http://home.apache.org/~boryas/samza-1.2.0-rc1/


The release candidate is signed with pgp key 0x7D74D0CD5B5EB041, which can
be found
http://keyserver.ubuntu.com/pks/lookup?op=get=0x7d74d0cd5b5eb041

The git tag is release-1.2.0-rc0 and signed with the same pgp key:
https://gitbox.apache.org/repos/asf?p=samza.git;a=tag;h=refs/tags/release-1.2.0-rc

1

Test binaries have been published to Maven's staging repository, and are
available here:
https://repository.apache.org/content/repositories/orgapachesamza-106

6

The vote will be open for 56 hours (ending at 06:00 PM PST Friday,
05/24/2019).

Please download the release candidate, check the hashes/signature, build it
and test it, and then please vote:

[ ] +1 approve

[ ] +0 no opinion

[ ] -1 disapprove (and reason why)

I ran check-all.sh and integration tests.

+1 (non-binding) from my side.

Thanks,

[VOTE] Apache Samza 1.2.0 RC0

[Boris Shkolnik]

Hi,

This is a call for a vote on a release of Apache Samza 1.2.0. Thanks to
everyone who has contributed to this release.

The release candidate can be downloaded from here:
http://home.apache.org/~boryas/samza-1.2.0-rc0/

The release candidate is signed with pgp key 0x7D74D0CD5B5EB041, which can
be found
http://keyserver.ubuntu.com/pks/lookup?op=get=0x7d74d0cd5b5eb041

The git tag is release-1.2.0-rc0 and signed with the same pgp key:
https://gitbox.apache.org/repos/asf?p=samza.git;a=tag;h=refs/tags/release-1.2.0-rc

0

Test binaries have been published to Maven's staging repository, and are
available here:
https://repository.apache.org/content/repositories/orgapachesamza-1065


The vote will be open for 56 hours (ending at 06:00 PM PST Friday,
05/24/2019).

Please download the release candidate, check the hashes/signature, build it
and test it, and then please vote:

[ ] +1 approve

[ ] +0 no opinion

[ ] -1 disapprove (and reason why)

I ran check-all.sh and integration tests.

+1 (non-binding) from my side.

Thanks,

Re: Review Request 68867: Update hell0-samza with latest code

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68867/#review209142
---


Ship it!




One comment. All the configs in this sample job are provided in code and 
removed from the configs. We need to clarify that users still can use configs 
and the values of the configs will override values given in the code (to allow 
redeployments without recompilation).

- Boris Shkolnik


On Sept. 28, 2018, 11:55 p.m., Xinyu Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68867/
> ---
> 
> (Updated Sept. 28, 2018, 11:55 p.m.)
> 
> 
> Review request for samza, Boris Shkolnik and Prateek Maheshwari.
> 
> 
> Repository: samza-hello-samza
> 
> 
> Description
> ---
> 
> Update hell0-samza with latest code
> 
> 
> Diffs
> -
> 
>   build.gradle 9d1f5433afc548ee19b00c0e0e2e73e7963d25ef 
>   src/main/config/pageview-adclick-joiner.properties 
> eba7b0b6efc86869dbdf9402ae069550ed4c1723 
>   src/main/config/pageview-filter.properties 
> 331ee1a1f1315c13aa955353795853eca792d669 
>   src/main/config/pageview-profile-table-joiner.properties 
> 7cec6013e744042295568cef17928321abf95b35 
>   src/main/config/pageview-sessionizer.properties 
> 420cdde0d2d82f22daafc49b7428040a7dcd1eef 
>   src/main/config/stock-price-table-joiner.properties 
> f9bd684ed3601ba887b48c87c070f31c8b137f8e 
>   src/main/config/tumbling-pageview-counter.properties 
> b58dbe9a951a1ec1978a745efdd43e0f9fe87483 
>   src/main/config/wikipedia-application-local-runner.properties 
> b770f1317dfd61b33122c71127ba8163135a26e5 
>   src/main/config/wikipedia-application.properties 
> 841fcc5a3ca6b5e52c82e717fb7baa1e380d134f 
>   src/main/java/samza/examples/azure/AzureApplication.java 
> 9f565fe47373c5bb054cfcc1ccc00e54111d786e 
>   src/main/java/samza/examples/azure/AzureZKLocalApplication.java 
> 3d4f8b05cd1405070448c7598afa2712bb2db13d 
>   src/main/java/samza/examples/cookbook/PageViewAdClickJoiner.java 
> f6c3810ad59532563994ff344952490369c87350 
>   src/main/java/samza/examples/cookbook/PageViewFilterApp.java 
> a2accfdd8fee6033133e77e74fa8c6fd8de186c5 
>   src/main/java/samza/examples/cookbook/PageViewProfileTableJoiner.java 
> 86deb614cc7bf9720df86e9d9f49bb4613ce1384 
>   src/main/java/samza/examples/cookbook/PageViewSessionizerApp.java 
> 2bcd9f5b8f8e14e451451a12b627e189ad673288 
>   src/main/java/samza/examples/cookbook/StockPriceTableJoiner.java 
> cb735d284f82b67f01f797c2ede2c130b80eb047 
>   src/main/java/samza/examples/cookbook/TumblingPageViewCounterApp.java 
> acf1411af7d951081ad988c92822fde2b140608d 
>   
> src/main/java/samza/examples/wikipedia/application/WikipediaApplication.java 
> 032608f4f1f2525c0d5598ea80b84ed4d40f6f7e 
>   
> src/main/java/samza/examples/wikipedia/application/WikipediaZkLocalApplication.java
>  51dd28f69b146962b2853d2426e153e13a6ab1e6 
>   src/main/java/samza/examples/wikipedia/model/WikipediaParser.java 
> 93479626afa0fb9f8a230fa755e8f30fdbc57563 
>   src/main/java/samza/examples/wikipedia/system/WikipediaInputDescriptor.java 
> PRE-CREATION 
>   
> src/main/java/samza/examples/wikipedia/system/WikipediaSystemDescriptor.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/68867/diff/2/
> 
> 
> Testing
> ---
> 
> Only tested the high-level wikipedia example for now. I will keep testing the 
> rest of it in the next few days. Commit the change for now to unblock others 
> to test.
> 
> 
> Thanks,
> 
> Xinyu Liu
> 
>

Re: Review Request 58866: fixed SAMZA-1248. use processor id for stand alone barrier

[Boris Shkolnik]

> On April 29, 2017, 12:23 a.m., Navina Ramesh wrote:
> > samza-core/src/main/java/org/apache/samza/zk/ZkJobCoordinator.java
> > Line 65 (original), 65 (patched)
> > <https://reviews.apache.org/r/58866/diff/1/?file=1703683#file1703683line65>
> >
> > Why is newJobModel useful? Please add some comments as it is not very 
> > obvious.

Sure. It is needed for implementation of JobCoordinator.getJobModel(), to avoid 
extra calles to ZK.


- Boris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58866/#review173409
---


On April 28, 2017, 11:54 p.m., Boris Shkolnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58866/
> ---
> 
> (Updated April 28, 2017, 11:54 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-1248
> https://issues.apache.org/jira/browse/SAMZA-1248
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> use processor id for stand alone barrier
> 
> 
> Diffs
> -
> 
>   samza-core/src/main/java/org/apache/samza/zk/ZkJobCoordinator.java 
> 2535654cee37feeb472517b8673a7bb12b3cc1fc 
>   samza-core/src/main/java/org/apache/samza/zk/ZkUtils.java 
> fee840511fbc19da2e19525a97fcfb5812a70a53 
>   samza-core/src/test/java/org/apache/samza/zk/TestZkUtils.java 
> b8dc2953ead2fb11fa22db5ec30b19a74a779830 
> 
> 
> Diff: https://reviews.apache.org/r/58866/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Boris Shkolnik
> 
>

Re: Review Request 58866: fixed SAMZA-1248. use processor id for stand alone barrier

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58866/
---

(Updated April 28, 2017, 11:54 p.m.)


Review request for samza.


Bugs: SAMZA-1248
https://issues.apache.org/jira/browse/SAMZA-1248


Repository: samza


Description (updated)
---

use processor id for stand alone barrier


Diffs
-

  samza-core/src/main/java/org/apache/samza/zk/ZkJobCoordinator.java 
2535654cee37feeb472517b8673a7bb12b3cc1fc 
  samza-core/src/main/java/org/apache/samza/zk/ZkUtils.java 
fee840511fbc19da2e19525a97fcfb5812a70a53 
  samza-core/src/test/java/org/apache/samza/zk/TestZkUtils.java 
b8dc2953ead2fb11fa22db5ec30b19a74a779830 


Diff: https://reviews.apache.org/r/58866/diff/1/


Testing
---


Thanks,

Boris Shkolnik

Review Request 58866: fixed SAMZA-1248. use processor id for stand alone barrier

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58866/
---

Review request for samza.


Bugs: SAMZA-1248
https://issues.apache.org/jira/browse/SAMZA-1248


Repository: samza


Description
---

removed unused variable


Diffs
-

  samza-core/src/main/java/org/apache/samza/zk/ZkJobCoordinator.java 
2535654cee37feeb472517b8673a7bb12b3cc1fc 
  samza-core/src/main/java/org/apache/samza/zk/ZkUtils.java 
fee840511fbc19da2e19525a97fcfb5812a70a53 
  samza-core/src/test/java/org/apache/samza/zk/TestZkUtils.java 
b8dc2953ead2fb11fa22db5ec30b19a74a779830 


Diff: https://reviews.apache.org/r/58866/diff/1/


Testing
---


Thanks,

Boris Shkolnik

Re: [DISCUSS] Samza 0.12.0 release

[Boris Shkolnik]

Looks good. +1 to cut the new release.

On Fri, Dec 23, 2016 at 10:44 AM, santhosh venkat <
santhoshvenkat1...@gmail.com> wrote:

> Hi All,
>
> There have been quite a lot of new features added to master since 0.11
> release to warrant a new major release. At LinkedIn, we've done functional
> and performance testing against master in the past weeks, and deployed jobs
> with the latest build in production. We will continue to test for stability
> in the next few weeks.
>
> Here are the list of JIRA patches that are available to be added as a part
> of 0.12.0 release.
>
> Potential breaking changes :
>
>* SAMZA-469 : Support Scala 2.11 (not binary compatible with 2.10.x
> versions)
>* SAMZA-855 : Upgrade Samza's Kafka client version to 0.10.0.0 (
> https://kafka.apache.org/documentation#upgrade_10_breaking)
>* SAMZA-1031 : Use Java 1.8 source compatibility for Samza (Java 7 is no
> longer supported)
>
> Here are the JIRAs of other main features that will be included in this
> release (sorted in chronological order):
>
>* SAMZA-967 : Add HDFS system consumer to Samza
>* SAMZA-974 : Build an end-of-stream concept into Samza
>* SAMZA-1012 : Generated changelog mappings are not consistent
>* SAMZA-1013 : Add YARN Node label support
>* SAMZA-1014 : Add property to set YARN AM cpu cores
>* SAMZA-1015 : Support lambdas, streams in checkStyle checks
>* SAMZA-1017 : Disk Quotas - Add throttling support for AsyncRunLoop
>* SAMZA-1033 : Remove import-control from checkstyle
>* SAMZA-1040 : Revert the ClassLoaderHelper change in SamzaContainer
>* SAMZA-1042 : Allow offset notifications for input systems
>* SAMZA-1043 : Samza performance improvements
>* SAMZA-1047 : testEndOfStreamWithOutOfOrderProcess is flaky
>* SAMZA-1048 : upgrade jetty dependency to Jetty 9 from Jetty 8
>* SAMZA-1055 : Disable broken tests in SamzaRest due to Jetty version
> upgrade
>* SAMZA-1058 : Fix check-all.sh to remove JDK7 build
>* SAMZA-1060 : Allow to specify a changelog system separately
>* SAMZA-1065 : Change the commit order when deduping with local state
> store
>* SAMZA-1066 : Update JavaStorageConfig
>* SAMZA-1069 : Deadlock between KafkaSystemProducer and KafkaProducer
> from kafka-clients lib
>
> Here's what I purpose:
>
> 1. Cut an 0.12.0 release branch.
> 2. Work on getting as many of the pending JIRAs done as possible.
> 3. Target a release vote on the first week of Jan'17.
>
>
> Thoughts?
> Shanthoosh
>

Re: [DISCUSS] Samza 0.12.0 release

[Boris Shkolnik]

Please add SAMZA-1061  (it
is docs for 1060)

On Fri, Dec 23, 2016 at 10:44 AM, santhosh venkat <
santhoshvenkat1...@gmail.com> wrote:

> Hi All,
>
> There have been quite a lot of new features added to master since 0.11
> release to warrant a new major release. At LinkedIn, we've done functional
> and performance testing against master in the past weeks, and deployed jobs
> with the latest build in production. We will continue to test for stability
> in the next few weeks.
>
> Here are the list of JIRA patches that are available to be added as a part
> of 0.12.0 release.
>
> Potential breaking changes :
>
>* SAMZA-469 : Support Scala 2.11 (not binary compatible with 2.10.x
> versions)
>* SAMZA-855 : Upgrade Samza's Kafka client version to 0.10.0.0 (
> https://kafka.apache.org/documentation#upgrade_10_breaking)
>* SAMZA-1031 : Use Java 1.8 source compatibility for Samza (Java 7 is no
> longer supported)
>
> Here are the JIRAs of other main features that will be included in this
> release (sorted in chronological order):
>
>* SAMZA-967 : Add HDFS system consumer to Samza
>* SAMZA-974 : Build an end-of-stream concept into Samza
>* SAMZA-1012 : Generated changelog mappings are not consistent
>* SAMZA-1013 : Add YARN Node label support
>* SAMZA-1014 : Add property to set YARN AM cpu cores
>* SAMZA-1015 : Support lambdas, streams in checkStyle checks
>* SAMZA-1017 : Disk Quotas - Add throttling support for AsyncRunLoop
>* SAMZA-1033 : Remove import-control from checkstyle
>* SAMZA-1040 : Revert the ClassLoaderHelper change in SamzaContainer
>* SAMZA-1042 : Allow offset notifications for input systems
>* SAMZA-1043 : Samza performance improvements
>* SAMZA-1047 : testEndOfStreamWithOutOfOrderProcess is flaky
>* SAMZA-1048 : upgrade jetty dependency to Jetty 9 from Jetty 8
>* SAMZA-1055 : Disable broken tests in SamzaRest due to Jetty version
> upgrade
>* SAMZA-1058 : Fix check-all.sh to remove JDK7 build
>* SAMZA-1060 : Allow to specify a changelog system separately
>* SAMZA-1065 : Change the commit order when deduping with local state
> store
>* SAMZA-1066 : Update JavaStorageConfig
>* SAMZA-1069 : Deadlock between KafkaSystemProducer and KafkaProducer
> from kafka-clients lib
>
> Here's what I purpose:
>
> 1. Cut an 0.12.0 release branch.
> 2. Work on getting as many of the pending JIRAs done as possible.
> 3. Target a release vote on the first week of Jan'17.
>
>
> Thoughts?
> Shanthoosh
>

Re: Review Request 54563: added new config for job.changelog.system

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54563/
---

(Updated Dec. 9, 2016, 1:33 a.m.)


Review request for samza.


Bugs: SAMZA-1060
https://issues.apache.org/jira/browse/SAMZA-1060


Repository: samza


Description
---

added new config for job.changelog.system.


Diffs (updated)
-

  samza-core/src/main/scala/org/apache/samza/config/StorageConfig.scala 
be3f1068f7921c9c8f8697577efea6580672813e 
  samza-core/src/test/scala/org/apache/samza/config/TestStorageConfig.scala 
81a35ec719975052f50af249dfcc17249e8342d1 

Diff: https://reviews.apache.org/r/54563/diff/


Testing
---

added unit test.


Thanks,

Boris Shkolnik

Review Request 54563: added new config for job.changelog.system

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54563/
---

Review request for samza.


Bugs: SAMZA-1060
https://issues.apache.org/jira/browse/SAMZA-1060


Repository: samza


Description
---

added new config for job.changelog.system.


Diffs
-

  samza-core/src/main/scala/org/apache/samza/config/StorageConfig.scala 
be3f1068f7921c9c8f8697577efea6580672813e 
  samza-core/src/test/scala/org/apache/samza/config/TestStorageConfig.scala 
81a35ec719975052f50af249dfcc17249e8342d1 

Diff: https://reviews.apache.org/r/54563/diff/


Testing
---

added unit test.


Thanks,

Boris Shkolnik

Re: Review Request 53826: added user document for Checkpoint callbacks

[Boris Shkolnik]

> On Nov. 22, 2016, 6:51 p.m., Prateek Maheshwari wrote:
> > docs/learn/documentation/versioned/container/checkpointing.md, line 129
> > <https://reviews.apache.org/r/53826/diff/2/?file=1567297#file1567297line129>
> >
> > What does the "push system" mean? Maybe clarify/remove?

done.


> On Nov. 22, 2016, 6:51 p.m., Prateek Maheshwari wrote:
> > docs/learn/documentation/versioned/container/checkpointing.md, line 138
> > <https://reviews.apache.org/r/53826/diff/2/?file=1567297#file1567297line138>
> >
> > Multiple "and"s in the sentence. Maybe "... for a task. These are the 
> > ..."?

done.


- Boris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53826/#review156607
---


On Nov. 19, 2016, 12:52 a.m., Boris Shkolnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53826/
> ---
> 
> (Updated Nov. 19, 2016, 12:52 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-1046
> https://issues.apache.org/jira/browse/SAMZA-1046
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> added user documentation for Checkpoint callbacks
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/container/checkpointing.md 
> 6f8c6d694be92f973af456ddd518d70540abe5c3 
> 
> Diff: https://reviews.apache.org/r/53826/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Boris Shkolnik
> 
>

Re: Review Request 53826: added user document for Checkpoint callbacks

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53826/
---

(Updated Nov. 19, 2016, 12:52 a.m.)


Review request for samza.


Bugs: SAMZA-1046
https://issues.apache.org/jira/browse/SAMZA-1046


Repository: samza


Description
---

added user documentation for Checkpoint callbacks


Diffs (updated)
-

  docs/learn/documentation/versioned/container/checkpointing.md 
6f8c6d694be92f973af456ddd518d70540abe5c3 

Diff: https://reviews.apache.org/r/53826/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 53826: added user document for Checkpoint callbacks

[Boris Shkolnik]

> On Nov. 18, 2016, 2:36 a.m., Navina Ramesh wrote:
> > docs/learn/documentation/versioned/container/checkpointing.md, line 130
> > <https://reviews.apache.org/r/53826/diff/1/?file=1565544#file1565544line130>
> >
> > Should we mention this since I don't know of a real world use-case that 
> > does this? :P Not a big deal.. Just wondering.. if you think it is useful, 
> > we can leave it here

Kafka may use it in the future, so I'd rather leave it in.


> On Nov. 18, 2016, 2:36 a.m., Navina Ramesh wrote:
> > docs/learn/documentation/versioned/container/checkpointing.md, line 138
> > <https://reviews.apache.org/r/53826/diff/1/?file=1565544#file1565544line138>
> >
> > suggestion: Highlight "not" in "not atomic" -> **not** atomic

good suggestion. done.


- Boris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53826/#review156276
---


On Nov. 16, 2016, 11:28 p.m., Boris Shkolnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53826/
> ---
> 
> (Updated Nov. 16, 2016, 11:28 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-1046
> https://issues.apache.org/jira/browse/SAMZA-1046
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> added user documentation for Checkpoint callbacks
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/container/checkpointing.md 
> 6f8c6d694be92f973af456ddd518d70540abe5c3 
> 
> Diff: https://reviews.apache.org/r/53826/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Boris Shkolnik
> 
>

Re: Review Request 53826: added user document for Checkpoint callbacks

[Boris Shkolnik]

> On Nov. 18, 2016, 3:17 a.m., Prateek Maheshwari wrote:
> >

Thanks!. Done.


- Boris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53826/#review156279
---


On Nov. 16, 2016, 11:28 p.m., Boris Shkolnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53826/
> ---
> 
> (Updated Nov. 16, 2016, 11:28 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-1046
> https://issues.apache.org/jira/browse/SAMZA-1046
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> added user documentation for Checkpoint callbacks
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/container/checkpointing.md 
> 6f8c6d694be92f973af456ddd518d70540abe5c3 
> 
> Diff: https://reviews.apache.org/r/53826/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Boris Shkolnik
> 
>

Review Request 53826: added user document for Checkpoint callbacks

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53826/
---

Review request for samza.


Bugs: SAMZA-1046
https://issues.apache.org/jira/browse/SAMZA-1046


Repository: samza


Description
---

added user documentation for Checkpoint callbacks


Diffs
-

  docs/learn/documentation/versioned/container/checkpointing.md 
6f8c6d694be92f973af456ddd518d70540abe5c3 

Diff: https://reviews.apache.org/r/53826/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 53453: Add optional interface for SystemConsumer checkpontListener() for checkpoint notifications

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53453/
---

(Updated Nov. 10, 2016, 6:58 p.m.)


Review request for samza.


Bugs: SAMZA-1042
https://issues.apache.org/jira/browse/SAMZA-1042


Repository: samza


Description
---

Add optional interface for SystemConsumer checkpontListener() for checkpoint 
notifications.


Diffs (updated)
-

  samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointListener.java 
PRE-CREATION 
  samza-api/src/main/java/org/apache/samza/system/SystemConsumer.java 
8dfcc7499659442aabd3085a8787475fe38f29db 
  samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala 
c41eadb70f4675816245f7ac40f0db2fc16335f0 
  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
e0468ee89c89fd720834461771ebb36475475bcb 
  samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala 
cb78223f1b59a78bbeb1e42b5974670a53def504 

Diff: https://reviews.apache.org/r/53453/diff/


Testing
---

gradlew test.
manual testing with LiKafkaConsumer.


Thanks,

Boris Shkolnik

Re: Review Request 53453: Add optional interface for SystemConsumer checkpontListener() for checkpoint notifications

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53453/
---

(Updated Nov. 10, 2016, 6:54 p.m.)


Review request for samza.


Bugs: SAMZA-1042
https://issues.apache.org/jira/browse/SAMZA-1042


Repository: samza


Description
---

Add optional interface for SystemConsumer checkpontListener() for checkpoint 
notifications.


Diffs
-

  samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointListener.java 
PRE-CREATION 
  samza-api/src/main/java/org/apache/samza/system/SystemConsumer.java 
8dfcc7499659442aabd3085a8787475fe38f29db 
  samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala 
c41eadb70f4675816245f7ac40f0db2fc16335f0 
  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
e0468ee89c89fd720834461771ebb36475475bcb 
  samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala 
cb78223f1b59a78bbeb1e42b5974670a53def504 

Diff: https://reviews.apache.org/r/53453/diff/


Testing (updated)
---

gradlew test.
manual testing with LiKafkaConsumer.


Thanks,

Boris Shkolnik

Re: Review Request 53453: Add optional interface for SystemConsumer checkpontListener() for checkpoint notifications

[Boris Shkolnik]

> On Nov. 9, 2016, 8:48 p.m., Navina Ramesh wrote:
> > Is there any practical implementation of this interface . eg. adding to 
> > kafkasystemconsumer? Or is it expected in the future?

I have used it with LiKafka for manual testing, but nothing to check in..


- Boris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53453/#review155495
---


On Nov. 4, 2016, 11:23 p.m., Boris Shkolnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53453/
> ---
> 
> (Updated Nov. 4, 2016, 11:23 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-1042
> https://issues.apache.org/jira/browse/SAMZA-1042
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Add optional interface for SystemConsumer checkpontListener() for checkpoint 
> notifications.
> 
> 
> Diffs
> -
> 
>   samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointListener.java 
> PRE-CREATION 
>   samza-api/src/main/java/org/apache/samza/system/SystemConsumer.java 
> 8dfcc7499659442aabd3085a8787475fe38f29db 
>   samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala 
> c41eadb70f4675816245f7ac40f0db2fc16335f0 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
> e0468ee89c89fd720834461771ebb36475475bcb 
>   
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala 
> cb78223f1b59a78bbeb1e42b5974670a53def504 
> 
> Diff: https://reviews.apache.org/r/53453/diff/
> 
> 
> Testing
> ---
> 
> gradlew test.
> manual testing.
> 
> 
> Thanks,
> 
> Boris Shkolnik
> 
>

Re: Review Request 53453: Add optional interface for SystemConsumer checkpontListener() for checkpoint notifications

[Boris Shkolnik]

> On Nov. 8, 2016, 11:52 p.m., Prateek Maheshwari wrote:
> > samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala, 
> > line 239
> > <https://reviews.apache.org/r/53453/diff/2/?file=1554966#file1554966line239>
> >
> > Are you missing a foreach here?
> > 
> > I think you need something like:
> > lastProcessedOffsets.get(taskName)
> >   .foreach { sspToOffsets => sspToOffsets.foreach { case (ssp, 
> > checkpoint) => 
> > offsetManagerMetrics.checkpointedOffsets(ssp).set(checkpoint) } }
> > 
> > If the version above is correct, can we add a test for this? Its easy 
> > to miss.

Interesting, Scala was doing the right thing, although it is not clear why. I 
had some tests for it and they passed. 
So I rewrote this part to make it clearer.


> On Nov. 8, 2016, 11:52 p.m., Prateek Maheshwari wrote:
> > samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala, 
> > line 77
> > <https://reviews.apache.org/r/53453/diff/2/?file=1554966#file1554966line77>
> >
> > Strongly prefer not adding empty Map() as default value here (and maybe 
> > clean up the other one too). See my comment on Xinyu's HDFS performance RB 
> > for explanation.

In general I agree with you, but in this case it is kind of hard to do it, 
because having a parameter without a default between parameters with default 
will confuse compiler when only some of the arguments are provided.


- Boris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53453/#review155377
---


On Nov. 4, 2016, 11:23 p.m., Boris Shkolnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53453/
> ---
> 
> (Updated Nov. 4, 2016, 11:23 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-1042
> https://issues.apache.org/jira/browse/SAMZA-1042
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Add optional interface for SystemConsumer checkpontListener() for checkpoint 
> notifications.
> 
> 
> Diffs
> -
> 
>   samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointListener.java 
> PRE-CREATION 
>   samza-api/src/main/java/org/apache/samza/system/SystemConsumer.java 
> 8dfcc7499659442aabd3085a8787475fe38f29db 
>   samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala 
> c41eadb70f4675816245f7ac40f0db2fc16335f0 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
> e0468ee89c89fd720834461771ebb36475475bcb 
>   
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala 
> cb78223f1b59a78bbeb1e42b5974670a53def504 
> 
> Diff: https://reviews.apache.org/r/53453/diff/
> 
> 
> Testing
> ---
> 
> gradlew test.
> manual testing.
> 
> 
> Thanks,
> 
> Boris Shkolnik
> 
>

Re: Producer stopped during leader switch

[Boris Shkolnik]

Just a follow up question - was a container failing with this error?
And if it was - when it was restarted somewhere else, was it still getting
the error?

On Fri, Oct 28, 2016 at 10:12 AM, David Yu  wrote:

> Hi,
>
> We recently experienced a Kafka broker crash. When a new broker was brought
> up, we started seeing the following errors in Samza (0.10.1):
>
> WARN  o.a.k.c.producer.internals.Sender - Got error produce response with
> correlation id 5199601 on topic-partition
> session_key_partitioned_sessions-39, retrying (2147483646 attempts left).
> Error: NOT_LEADER_FOR_PARTITION
>
> Is the Producer not able to detect the new broker/leader for that
> partition?
>
> Thanks,
> David
>

Re: Review Request 53453: Add optional interface for SystemConsumer checkpontListener() for checkpoint notifications

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53453/
---

(Updated Nov. 4, 2016, 11:23 p.m.)


Review request for samza.


Bugs: SAMZA-1042
https://issues.apache.org/jira/browse/SAMZA-1042


Repository: samza


Description
---

Add optional interface for SystemConsumer checkpontListener() for checkpoint 
notifications.


Diffs (updated)
-

  samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointListener.java 
PRE-CREATION 
  samza-api/src/main/java/org/apache/samza/system/SystemConsumer.java 
8dfcc7499659442aabd3085a8787475fe38f29db 
  samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala 
c41eadb70f4675816245f7ac40f0db2fc16335f0 
  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
e0468ee89c89fd720834461771ebb36475475bcb 
  samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala 
cb78223f1b59a78bbeb1e42b5974670a53def504 

Diff: https://reviews.apache.org/r/53453/diff/


Testing (updated)
---

gradlew test.
manual testing.


Thanks,

Boris Shkolnik

Review Request 53453: Add optional interface for SystemConsumer checkpontListener() for checkpoint notifications

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53453/
---

Review request for samza.


Bugs: SAMZA-1042
https://issues.apache.org/jira/browse/SAMZA-1042


Repository: samza


Description
---

Add optional interface for SystemConsumer checkpontListener() for checkpoint 
notifications.


Diffs
-

  samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointListener.java 
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala 
c41eadb70f4675816245f7ac40f0db2fc16335f0 
  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
e0468ee89c89fd720834461771ebb36475475bcb 
  samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala 
cb78223f1b59a78bbeb1e42b5974670a53def504 

Diff: https://reviews.apache.org/r/53453/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 52492: Adding monitor to clean up stale local stores of jobs/tasks.

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52492/#review153821
---




samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java (line 
72)
<https://reviews.apache.org/r/52492/#comment223302>

nit. It is beeter to be more precise what the exception is thrown.



samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java (line 
73)
<https://reviews.apache.org/r/52492/#comment223293>

I am not sure if we need guava for anything else, but is it worth bringing 
it in for this check ? IMHO, a regulare if() throw Exception, should suffice.



samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java (line 
92)
<https://reviews.apache.org/r/52492/#comment223304>

nit. should we check the return value here?



samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java (line 
95)
<https://reviews.apache.org/r/52492/#comment223305>

nit. I think we usually use name LOG.



samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java (line 
97)
<https://reviews.apache.org/r/52492/#comment223306>

'.'->':'



samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java (line 
104)
<https://reviews.apache.org/r/52492/#comment223307>

InetAddress.getLocalHost().getHostName() should be run outside of the loops.



samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java (line 
107)
<https://reviews.apache.org/r/52492/#comment223308>

this should be INFO level.



samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java (line 
118)
<https://reviews.apache.org/r/52492/#comment223312>

'an' -> 'this'



samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java (line 
124)
<https://reviews.apache.org/r/52492/#comment223311>

please add comment on the format of the name.



samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java (line 
158)
<https://reviews.apache.org/r/52492/#comment223690>

nit. Do you need to handle any exceptions here?



samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java (line 
166)
<https://reviews.apache.org/r/52492/#comment223704>

Seems like this class may be usefull elsewhere. Is it worth moving it to a 
top-level class?



samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitorConfig.java 
(line 40)
<https://reviews.apache.org/r/52492/#comment223697>

can we add a check for it and give a warning?


- Boris Shkolnik


On Oct. 26, 2016, 12:41 a.m., Shanthoosh Venkataraman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52492/
> ---
> 
> (Updated Oct. 26, 2016, 12:41 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> This patch contains the samza-rest monitor that periodically cleans up the 
> stale local stores of dead jobs/tasks. It performs the store deletion in two 
> phases. Initially it deletes the offset file in the local task stores if the 
> following condition is true. ((jobIsNotRunning || preferedHost != nmHost) && 
> offsetFilelastModifiedTime is greater than deleteRetention). During the 
> subsequent run, it deletes the local task stores if it does not contain 
> offset file. Please refer to the design doc of SAMZA-656 
> (https://issues.apache.org/jira/secure/attachment/12828083/DESIGN-SAMZA-656.pdf)
>  for more details.
> 
> 
> Diffs
> -
> 
>   build.gradle 2bea27b75288d3103178bc3762b9556f6e69cdd1 
>   samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java 
> PRE-CREATION 
>   
> samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitorConfig.java
>  PRE-CREATION 
>   
> samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitorFactory.java
>  PRE-CREATION 
>   
> samza-rest/src/main/java/org/apache/samza/rest/resources/ResourceConstants.java
>  PRE-CREATION 
>   
> samza-rest/src/test/java/org/apache/samza/monitor/TestLocalStoreMonitor.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/52492/diff/
> 
> 
> Testing
> ---
> 
> Unit testing and manual testing are done to verify the functionality.
> 
> 
> Thanks,
> 
> Shanthoosh Venkataraman
> 
>

[DISCUSS] Offset checkpoint notification to the System Consumer

[Boris Shkolnik]

Currently we keep track of the progress of each consumer using
checkpointing mechanism. These checkpoints are controlled by Samza, and the
system consumer is not aware of it.
But some systems may require knowledge of the current checkpointed position
in the stream.
For this purpose we propose an optional notification mechanism for input
systems.
See the doc attached to SAMZA-1042 for details.

Re: Review Request 52476: Do not load task store which are older than delete tombstones.

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52476/#review153468
---




samza-core/src/main/scala/org/apache/samza/config/StorageConfig.scala (line 44)
<https://reviews.apache.org/r/52476/#comment222787>

nit. can we call it storeName.



samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 
(line 132)
<https://reviews.apache.org/r/52476/#comment222789>

nit. using System.currentTimeMillis directly makes it difficult to unit 
test.



samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala 
(line 323)
<https://reviews.apache.org/r/52476/#comment222800>

isn't it easier just to make method package private?


- Boris Shkolnik


On Oct. 19, 2016, 10:04 p.m., Shanthoosh Venkataraman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52476/
> ---
> 
> (Updated Oct. 19, 2016, 10:04 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Every local task store is backed up by a kafka changelog topic. Due to log 
> compaction, delete tombstones of the changelog topic have a ttl of 
> delete.retention.ms. Replaying the events from the changelog that has missing 
> delete tombstones, would result in creation of an inconsistent local 
> store(due to the missing of some delete events). This patch deletes the local 
> stores in which difference between current time and last modified time of the 
> offset file is greater than delete.retention.ms during the container startup.
> 
> 
> Diffs
> -
> 
>   samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java 
> 9329edf7d724f3a0d9235354bb77936f713e3b5f 
>   samza-core/src/main/scala/org/apache/samza/config/StorageConfig.scala 
> be3f1068f7921c9c8f8697577efea6580672813e 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
> 05a996c98075ea8ed3767af666b9beeb1933f2a6 
>   samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 
> 0b7bcdda1639eea8239a69c31bdf42558e9077d2 
>   
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala
>  4d40f520e54beb643acd8410c772b75e2f6a9162 
>   samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 
> 973ab8cfb3d248bec7efe5e338f5e667f097556d 
> 
> Diff: https://reviews.apache.org/r/52476/diff/
> 
> 
> Testing
> ---
> 
> Unit testing and manual testing has been done to verify the functionality.
> 
> 
> Thanks,
> 
> Shanthoosh Venkataraman
> 
>

Re: Review Request 52476: Do not load task store which are older than delete tombstones.

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52476/#review153476
---


Ship it!




- Boris Shkolnik


On Oct. 19, 2016, 10:04 p.m., Shanthoosh Venkataraman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52476/
> ---
> 
> (Updated Oct. 19, 2016, 10:04 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Every local task store is backed up by a kafka changelog topic. Due to log 
> compaction, delete tombstones of the changelog topic have a ttl of 
> delete.retention.ms. Replaying the events from the changelog that has missing 
> delete tombstones, would result in creation of an inconsistent local 
> store(due to the missing of some delete events). This patch deletes the local 
> stores in which difference between current time and last modified time of the 
> offset file is greater than delete.retention.ms during the container startup.
> 
> 
> Diffs
> -
> 
>   samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java 
> 9329edf7d724f3a0d9235354bb77936f713e3b5f 
>   samza-core/src/main/scala/org/apache/samza/config/StorageConfig.scala 
> be3f1068f7921c9c8f8697577efea6580672813e 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
> 05a996c98075ea8ed3767af666b9beeb1933f2a6 
>   samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 
> 0b7bcdda1639eea8239a69c31bdf42558e9077d2 
>   
> samza-core/src/test/scala/org/apache/samza/storage/TestTaskStorageManager.scala
>  4d40f520e54beb643acd8410c772b75e2f6a9162 
>   samza-kafka/src/main/scala/org/apache/samza/config/KafkaConfig.scala 
> 973ab8cfb3d248bec7efe5e338f5e667f097556d 
> 
> Diff: https://reviews.apache.org/r/52476/diff/
> 
> 
> Testing
> ---
> 
> Unit testing and manual testing has been done to verify the functionality.
> 
> 
> Thanks,
> 
> Shanthoosh Venkataraman
> 
>

Re: Review Request 53028: SAMZA-1040: Revert the ClassLoaderHelper change in SamzaContainer

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53028/#review153466
---


Ship it!




Ship It!

- Boris Shkolnik


On Oct. 19, 2016, 6:30 p.m., Xinyu Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53028/
> ---
> 
> (Updated Oct. 19, 2016, 6:30 p.m.)
> 
> 
> Review request for samza, Jake Maes and Jagadish Venkatraman.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> The change introduced by ClassLoaderHelpler does not work for 
> AsyncStreamTask, so the patch has been reverted in 0.11.0 branch. This is the 
> fix forward for the master branch.
> 
> 
> Diffs
> -
> 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
> 05a996c98075ea8ed3767af666b9beeb1933f2a6 
> 
> Diff: https://reviews.apache.org/r/53028/diff/
> 
> 
> Testing
> ---
> 
> Tested by AsyncStreamTask jobs.
> 
> 
> Thanks,
> 
> Xinyu Liu
> 
>

Re: Review Request 53000: SAMZA-1038: Update hello-samza master to use Samza 0.11.0

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53000/#review153462
---


Ship it!




Ship It!

- Boris Shkolnik


On Oct. 18, 2016, 10:17 p.m., Xinyu Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53000/
> ---
> 
> (Updated Oct. 18, 2016, 10:17 p.m.)
> 
> 
> Review request for samza, Navina Ramesh and Jagadish Venkatraman.
> 
> 
> Repository: samza-hello-samza
> 
> 
> Description
> ---
> 
> Changes for the lastest branch
> 
> 
> Diffs
> -
> 
>   gradle.properties fe71c660ab38146108f9a12ec4ce4cd2777879a3 
>   pom.xml 9cc44deb117833e0af8181697a016bce4e85498e 
> 
> Diff: https://reviews.apache.org/r/53000/diff/
> 
> 
> Testing
> ---
> 
> Tested locally by publish samza snapshot build to mvn local and build 
> hello-samza and runs fine.
> 
> 
> Thanks,
> 
> Xinyu Liu
> 
>

Re: Review Request 52962: SAMZA-1029: Prepare release candidate for 0.11.0

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52962/#review153105
---


Ship it!




- Boris Shkolnik


On Oct. 18, 2016, midnight, Xinyu Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52962/
> ---
> 
> (Updated Oct. 18, 2016, midnight)
> 
> 
> Review request for samza and Navina Ramesh.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> docs update for 0.11.0 release branch
> 
> 
> Diffs
> -
> 
>   docs/_config.yml dc1a66fa743d464c70d92406540fd7122c45272c 
>   docs/learn/tutorials/versioned/deploy-samza-job-from-hdfs.md 
> ca7b5f1a59724bbae9c46c7abd0d68cb3f019e3b 
>   docs/learn/tutorials/versioned/deploy-samza-to-CDH.md 
> daf762bc9f536520cceb503c5053283a80488bb1 
>   docs/learn/tutorials/versioned/remote-debugging-samza.md 
> 40db31a8152a999b549fde8f9155f4541d03147d 
>   docs/learn/tutorials/versioned/run-in-multi-node-yarn.md 
> bf2b59e3f4e0c6a3bfde0187db0b799f76797afb 
>   docs/learn/tutorials/versioned/samza-rest-getting-started.md 
> 942329e968bb02886df44b680bea8f75a221a289 
>   docs/startup/download/index.md 6a0c670bca01e01b9d8a73482af35cc144f1d524 
>   docs/startup/hello-samza/versioned/index.md 
> 8baacd390d41c5c87a426d63eec9ce5028de0cc2 
> 
> Diff: https://reviews.apache.org/r/52962/diff/
> 
> 
> Testing
> ---
> 
> local website.
> 
> 
> Thanks,
> 
> Xinyu Liu
> 
>

Re: [VOTE] Apache Samza 0.11.0 RC2

[Boris Shkolnik]

+1 (non-binding)
built, ran tests, verified MD5 and the signature.

Boris.

On Mon, Oct 10, 2016 at 4:07 PM, xinyu liu  wrote:

> Hey all,
>
> This is a call for a vote on a release of Apache Samza 0.11.0. Thanks to
> everyone who has contributed to this release. We are very glad to see some
> new contributors in this release.
>
> Note: this release candidate reverted the changes that caused the
> AsyncStreamTask to break in 0.11.0 RC1. All tests are verified for this
> candidate.
>
> The release candidate can be downloaded from here:
> http://home.apache.org/~xinyu/samza-0.11.0-rc2/
>
> The release candidate is signed with pgp key C31D7061, which can be found
> on keyservers:
> http://pgp.mit.edu/pks/lookup?op=get=0xC31D7061
>
> The git tag is release-0.11.0-rc2 and signed with the same pgp key:
> https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tag;h=
> refs/tags/release-0.11.0-rc2
>
> Test binaries have been published to Maven's staging repository, and are
> available here:
> https://repository.apache.org/content/repositories/orgapachesamza-1015
>
> Note that the binaries were built with JDK7 without incident.
>
> 38 issues were resolved for this release:
> https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20SAMZA%20AND%20fixVersion%20in%20(0.11%2C%200.11.0)%
> 20AND%20status%20in%20(Resolved%2C%20Closed)
>
> The vote will be open for 72 hours ( end in 12:00pm Thursday, 10/13/2016 ).
>
> Please download the release candidate, check the hashes/signature, build it
> and test it, and then please vote:
>
>
> [ ] +1 approve
>
> [ ] +0 no opinion
>
> [ ] -1 disapprove (and reason why)
>
>
> +1 from my side for the release.
>
> Cheers!
>
> Xinyu
>

Re: [DISCUSS] [VOTE] Apache Samza 0.11.0 RC0

[Boris Shkolnik]

+1

On Fri, Sep 30, 2016 at 1:39 PM, xinyu liu  wrote:

> Subject correction: [VOTE] Apache Samza 0.11.0 RC0.
>
> Thanks,
> Xinyu
>
> On Fri, Sep 30, 2016 at 12:00 PM, xinyu liu  wrote:
>
> > Hey all,
> >
> > This is a call for a vote on a release of Apache Samza 0.11.0. Thanks to
> > everyone who has contributed to this release. We are very glad to see
> > some new contributors in this release.
> >
> > The release candidate can be downloaded from here:
> > http://home.apache.org/~xinyu/samza-0.11.0-rc0/
> >
> > The release candidate is signed with pgp key C31D7061, which can be
> found on
> > keyservers:
> > http://pgp.mit.edu/pks/lookup?op=get=0xC31D7061
> >
> > The git tag is release-0.11.0-rc0 and signed with the same pgp key:
> > https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tag;h=
> > refs/tags/release-0.11.0-rc0
> >
> > Test binaries have been published to Maven's staging repository, and are
> > available here:
> > https://repository.apache.org/content/repositories/orgapachesamza-1013
> >
> > Note that the binaries were built with JDK7 without incident.
> >
> > 38 issues were resolved for this release:
> > https://issues.apache.org/jira/issues/?jql=project%20%3D%
> > 20SAMZA%20AND%20fixVersion%20in%20(0.11%2C%200.11.0)%20AND%
> > 20status%20in%20(Resolved%2C%20Closed)
> >
> > The vote will be open for 72 hours ( end in 12:00pm Wednesday, 10/05/2016
> > ).
> >
> > Please download the release candidate, check the hashes/signature, build
> > it and test it, and then please vote:
> >
> >
> > [ ] +1 approve
> >
> > [ ] +0 no opinion
> >
> > [ ] -1 disapprove (and reason why)
> >
> >
> > +1 from my side for the release.
> >
> > Cheers!
> > Xinyu Liu
> >
>

Re: [Discuss] Moving Samza to Java 1.8 source compatibility.

[Boris Shkolnik]

+1 for moving to 1.8.

On Wed, Apr 27, 2016 at 6:27 PM, Jacob Maes  wrote:

> Hey everyone,
>
> I wanted to start a discussion to see what folks think about moving to Java
> 1.8 source compatibility at some point after the 10.1 release.
>
> Java 8 has a number of nice features that can help us build more concise,
> maintainable, and robust software. A few notable features that would
> benefit Samza:
> 1. Stream API - provide a compact syntax for expressing transformations on
> collections. These may be foundational for future API work including
> Operators (SAMZA-914)
> 2. Default Methods - enable us to evolve interfaces without breaking
> compatibility
> 3. Concurrent package enhancements - generally make concurrent programming
> easier, which will be more important with features like multithreading
> support (SAMZA-863)
> 4. Lambdas - love them or hate them, they do reduce the amount of
> boilerplate code, especially when used in place of anonymous classes.
>
> It certainly would be nice to leverage some of the features above. However,
> we have historically supported Java versions N and N-1 and it doesn't look
> like Java 9 is coming until next year. So, discontinuing support for Java
> 1.7 at this point would be a departure from our normal support matrix for a
> significant period of time. Thoughts on the pros and cons?
>
> I know some folks in this community are still on Java 1.7. How many of you
> stay up to date with the latest Samza? Do you have a roadmap to move to
> Java 1.8?
>
> Thanks,
> Jake
>

Re: Review Request 52140: added docs for split deployment

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52140/
---

(Updated Sept. 27, 2016, 12:58 a.m.)


Review request for samza.


Changes
---

converted document to be more end-user oriented.


Bugs: SAMZA-927
https://issues.apache.org/jira/browse/SAMZA-927


Repository: samza


Description
---

added docs for split deployment


Diffs (updated)
-

  docs/learn/documentation/versioned/index.html 
a997445db2af40e3ac87dc01d510a5e9045d73f4 
  docs/learn/documentation/versioned/jobs/split-deployment.md PRE-CREATION 

Diff: https://reviews.apache.org/r/52140/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 52140: added docs for split deployment

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52140/
---

(Updated Sept. 23, 2016, 5:24 p.m.)


Review request for samza.


Bugs: SAMZA-927
https://issues.apache.org/jira/browse/SAMZA-927


Repository: samza


Description
---

added docs for split deployment


Diffs (updated)
-

  docs/learn/documentation/versioned/operations/split-deployment.md 
PRE-CREATION 

Diff: https://reviews.apache.org/r/52140/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 52140: added docs for split deployment

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52140/
---

(Updated Sept. 22, 2016, 11:38 p.m.)


Review request for samza.


Repository: samza


Description
---

added docs for split deployment


Diffs (updated)
-

  docs/learn/documentation/versioned/operations/split-deployment.md 
PRE-CREATION 

Diff: https://reviews.apache.org/r/52140/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 52140: added docs for split deployment

[Boris Shkolnik]

> On Sept. 22, 2016, 1:07 a.m., Xinyu Liu wrote:
> > docs/learn/documentation/versioned/operations/split-deployment.md, line 48
> > <https://reviews.apache.org/r/52140/diff/1/?file=1507623#file1507623line48>
> >
> > seems here are two "split deployment" bullets here. Do we want to 
> > distinguish the second and third type of deployments by naming?

well, they both are split deployment, and the difference is in the description. 
Otherwise we'll endup repeating parts of the description. I can add '1' and '2' 
but that is kind of useless.


> On Sept. 22, 2016, 1:07 a.m., Xinyu Liu wrote:
> > docs/learn/documentation/versioned/operations/split-deployment.md, line 65
> > <https://reviews.apache.org/r/52140/diff/1/?file=1507623#file1507623line65>
> >
> > 1. => 2.

this is md format. It will convert second '1' into '2', and third '1' into 3.


> On Sept. 22, 2016, 1:07 a.m., Xinyu Liu wrote:
> > docs/learn/documentation/versioned/operations/split-deployment.md, line 90
> > <https://reviews.apache.org/r/52140/diff/1/?file=1507623#file1507623line90>
> >
> > Since this is 0.11 release, can we use 0.11.0 acrose all examples here?

Sure.


> On Sept. 22, 2016, 1:07 a.m., Xinyu Liu wrote:
> > docs/learn/documentation/versioned/operations/split-deployment.md, line 113
> > <https://reviews.apache.org/r/52140/diff/1/?file=1507623#file1507623line113>
> >
> > I think we need to mention the framework libraries needed to be placed 
> > to each node in the cluster.

we are talking about 206 jars! how do you mention it here? I guess the user 
should figure it out by themselves (for example copy from a current deployment).


- Boris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52140/#review149931
---


On Sept. 22, 2016, 12:48 a.m., Boris Shkolnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52140/
> ---
> 
> (Updated Sept. 22, 2016, 12:48 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> added docs for split deployment
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/operations/split-deployment.md 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/52140/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Boris Shkolnik
> 
>

Re: Review Request 52133: reopen a Kafka system producer if send called again

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52133/
---

(Updated Sept. 22, 2016, 12:56 a.m.)


Review request for samza.


Changes
---

added comment.


Repository: samza


Description
---

reopen a Kafka system producer if send called again


Diffs (updated)
-

  
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala
 5ff6d3caf54ed148aa40c7c752c587e556a4f34a 

Diff: https://reviews.apache.org/r/52133/diff/


Testing
---

manual.


Thanks,

Boris Shkolnik

Review Request 52140: added docs for split deployment

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52140/
---

Review request for samza.


Repository: samza


Description
---

added docs for split deployment


Diffs
-

  docs/learn/documentation/versioned/operations/split-deployment.md 
PRE-CREATION 

Diff: https://reviews.apache.org/r/52140/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 51934: reopen a Kafka system producer if send called again

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51934/
---

(Updated Sept. 21, 2016, 8:37 p.m.)


Review request for samza.


Bugs: SAMZA-1019
https://issues.apache.org/jira/browse/SAMZA-1019


Repository: samza


Description (updated)
---

reopen a Kafka system producer if send called again

Closing this patch. Replacing it with 52133.


Diffs
-

  
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala
 5ff6d3caf54ed148aa40c7c752c587e556a4f34a 

Diff: https://reviews.apache.org/r/51934/diff/


Testing
---

manual testing.


Thanks,

Boris Shkolnik

Review Request 52133: reopen a Kafka system producer if send called again

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52133/
---

Review request for samza.


Repository: samza


Description
---

reopen a Kafka system producer if send called again


Diffs
-

  
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala
 5ff6d3caf54ed148aa40c7c752c587e556a4f34a 

Diff: https://reviews.apache.org/r/52133/diff/


Testing
---

manual.


Thanks,

Boris Shkolnik

Review Request 51934: reopen a Kafka system producer if send called again

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51934/
---

Review request for samza.


Bugs: SAMZA-1019
https://issues.apache.org/jira/browse/SAMZA-1019


Repository: samza


Description
---

reopen a Kafka system producer if send called again


Diffs
-

  
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemProducer.scala
 5ff6d3caf54ed148aa40c7c752c587e556a4f34a 

Diff: https://reviews.apache.org/r/51934/diff/


Testing
---

manual testing.


Thanks,

Boris Shkolnik

Re: Review Request 50583: SAMZA-954 Improve logging for Samza

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50583/#review146007
---


Ship it!





samza-core/src/main/scala/org/apache/samza/util/Logging.scala (line 31)
<https://reviews.apache.org/r/50583/#comment212368>

Suggestion: what if we need a message at a WARN level. May be you can have 
another parameter 'level' with the default set to INFO.


- Boris Shkolnik


On July 28, 2016, 8:41 p.m., Jake Maes wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50583/
> ---
> 
> (Updated July 28, 2016, 8:41 p.m.)
> 
> 
> Review request for samza, Boris Shkolnik, Chris Pettitt, Fred Ji, Jake Maes, 
> Navina Ramesh, Jagadish Venkatraman, Xinyu Liu, and Yi Pan (Data 
> Infrastructure).
> 
> 
> Bugs: SAMZA-954
> https://issues.apache.org/jira/browse/SAMZA-954
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-954 Improve logging for Samza
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/jobs/logging.md 
> 0726d37affa06f85e20fbd3bc2395c28f30159a8 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
> a37f3536c45fdcb6d5410328f031b0263b71a82d 
>   samza-core/src/main/scala/org/apache/samza/metrics/JmxServer.scala 
> ad00ca00f918df4d71d1c920b77027401a55c80b 
>   samza-core/src/main/scala/org/apache/samza/util/Logging.scala 
> 250de1e2fa103be1a426d9da31187c12dbff8678 
>   samza-test/src/main/resources/log4j.xml 
> 29345f39ecef6f9ec769bf9d8eaab239f34e5d1e 
> 
> Diff: https://reviews.apache.org/r/50583/diff/
> 
> 
> Testing
> ---
> 
> Tested using hello-samza with the following log4j.xml
> 
> ```
> 
> http://jakarta.apache.org/log4j/;>
>   
> 
>class="org.apache.log4j.RollingFileAppender">
>   />
>  
>  
>  
>   
>  
>   
>class="org.apache.log4j.RollingFileAppender">
>   value="${samza.log.dir}/${samza.container.name}-startup.log" />
>  
>  
>  
>   
>  
>   
>   
> 
> 
> 
>   
>   
> 
> 
>   
> 
> 
> 
> ```
> 
> 
> Thanks,
> 
> Jake Maes
> 
>

Re: [DISCUSS] [VOTE] Apache Samza 0.10.1 RC0

[Boris Shkolnik]

+1 (non-binding).

Boris.

On Mon, Aug 1, 2016 at 11:39 AM, Navina Ramesh  wrote:

> Hey all,
>
> This is a call for a vote on a release of Apache Samza 0.10.0. Thanks
> to everyone
> who has contributed to this release. We are very glad to see some new
> contributors in this release.
>
> **NOTE**: This release is primarily a bug-fix release with no major changes
> to the public api. Since we are behind schedule in terms of releases, we
> are combining the DISCUSS and VOTE email into a single VOTE email. If
> anyone has objections, please do raise them. We plan to shortly follow-up
> with a DISCUSS email for 0.11.0 release, that brings in new features in
> multi-threading, REST api etc.
>
>
> The release candidate can be downloaded from here:
> http://home.apache.org/~navina/samza-0.10.1-rc0/
>
>
> The release candidate is signed with pgp key 331C8F69, which can be found
> on keyservers:
> http://pgp.mit.edu/pks/lookup?op=get=0x331C8F69
>
>
> The git tag is release-0.10.1-rc0 and signed with the same pgp key:
> *
> https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tag;h=33eff8a3e1eb2dc1dbffc661f910b1272b233640
> <
> https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tag;h=33eff8a3e1eb2dc1dbffc661f910b1272b233640
> >*
>
>
> Test binaries have been published to Maven's staging repository, and
> are available
> here:
> https://repository.apache.org/content/repositories/orgapachesamza-1012/
>
>
> Note that the binaries were built with JDK7 without incident.
>
> 69 issues were resolved for this release:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20fixVersion%20in%20(0.10.1)%20AND%20status%20in%20(Resolved%2C%20Closed)
>
>
> The vote will be open for 72 hours ( end in 12:00pm Thursday, 08/05/2016 ).
>
>
> Please download the release candidate, check the hashes/signature, build
> it and
> test it, and then please vote:
>
>
> [ ] +1 approve
>
> [ ] +0 no opinion
>
> [ ] -1 disapprove (and reason why)
>
>
> +1 from my side for the release.
>
> Cheers!
> --
> Navina R.
> nav...@apache.org
>

Re: Review Request 50318: SAMZA-979: Remove KafkaCheckpointMigration

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50318/#review143632
---


Ship it!




Are we sure that noone will try to upgraded from a version before migration to 
version 11 directly?

- Boris Shkolnik


On July 25, 2016, 9:29 p.m., Xinyu Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50318/
> ---
> 
> (Updated July 25, 2016, 9:29 p.m.)
> 
> 
> Review request for samza, Navina Ramesh and Yi Pan (Data Infrastructure).
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> KafkaCheckpointMigration is not needed anymore for ver 11. remove the code.
> 
> Also find an extra logging introduced by the change. Remove it during this 
> fix.
> 
> 
> Diffs
> -
> 
>   samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala 
> 7245902c69c751a4e8853745de46adf5553d45f5 
>   samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 
> a3613ff601131ec8643e407dd89a5b496aa686ea 
>   
> samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala 
> f38b87ac4a5d3f62e419e0c866e15b5a7ddad26d 
>   samza-core/src/test/scala/org/apache/samza/job/TestJobRunner.scala 
> e97656aeac270bddcd16248b37312c146d0a7d1b 
>   
> samza-kafka/src/main/scala/org/apache/samza/migration/KafkaCheckpointMigration.scala
>  5d2641ab1e54d49f7b983bc526762cfb50f2911b 
>   
> samza-kafka/src/test/scala/org/apache/samza/migration/TestKafkaCheckpointMigration.scala
>  504fc89133702952dcaacb01d06449134b6d8921 
> 
> Diff: https://reviews.apache.org/r/50318/diff/
> 
> 
> Testing
> ---
> 
> Tested by locally deployed jobs.
> 
> Passed gradle tests.
> 
> 
> Thanks,
> 
> Xinyu Liu
> 
>

Re: Review Request 48862: allow empty serde for SystemStream

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48862/
---

(Updated June 17, 2016, 6:20 p.m.)


Review request for samza and Yi Pan (Data Infrastructure).


Changes
---

add and use getNonEmptyOption() for serde


Bugs: SAMZA-969
https://issues.apache.org/jira/browse/SAMZA-969


Repository: samza


Description (updated)
---

allow empty serde for SystemStream and System
by using getNonEmptyOption() for serde


Diffs (updated)
-

  samza-core/src/main/scala/org/apache/samza/config/ScalaMapConfig.scala 
5fb1f52e813672162a8a12ef82231294b05d0c0a 
  samza-core/src/main/scala/org/apache/samza/config/StreamConfig.scala 
e172589e075852c0163a40e65e26038f682ecc17 
  samza-core/src/main/scala/org/apache/samza/config/SystemConfig.scala 
bf974df349b0d8dfef8f6d8977ef95318af28583 
  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
086531e4d72e88d21735db6a5afd1c7e84b6c2e5 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 
fc3d085d7fff9f7dcec766ba48e550eb0052e99d 

Diff: https://reviews.apache.org/r/48862/diff/


Testing
---

Manual.


Thanks,

Boris Shkolnik

Re: Review Request 48862: allow empty serde for SystemStream

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48862/
---

(Updated June 17, 2016, 5:17 p.m.)


Review request for samza and Yi Pan (Data Infrastructure).


Bugs: SAMZA-969
https://issues.apache.org/jira/browse/SAMZA-969


Repository: samza


Description
---

allow empty serde for SystemStream


Diffs
-

  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
086531e4d72e88d21735db6a5afd1c7e84b6c2e5 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 
fc3d085d7fff9f7dcec766ba48e550eb0052e99d 

Diff: https://reviews.apache.org/r/48862/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 48862: allow empty serde for SystemStream

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48862/
---

(Updated June 17, 2016, 5:15 p.m.)


Review request for samza and Yi Pan (Data Infrastructure).


Bugs: SAMZA-959
https://issues.apache.org/jira/browse/SAMZA-959


Repository: samza


Description
---

allow empty serde for SystemStream


Diffs
-

  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
086531e4d72e88d21735db6a5afd1c7e84b6c2e5 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 
fc3d085d7fff9f7dcec766ba48e550eb0052e99d 

Diff: https://reviews.apache.org/r/48862/diff/


Testing
---


Thanks,

Boris Shkolnik

Review Request 48862: allow empty serde for SystemStream

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48862/
---

Review request for samza and Yi Pan (Data Infrastructure).


Repository: samza


Description
---

allow empty serde for SystemStream


Diffs
-

  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
086531e4d72e88d21735db6a5afd1c7e84b6c2e5 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 
fc3d085d7fff9f7dcec766ba48e550eb0052e99d 

Diff: https://reviews.apache.org/r/48862/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 48352: Added java friendly api to KafkaMetrics

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48352/
---

(Updated June 7, 2016, 11:59 p.m.)


Review request for samza and Navina Ramesh.


Bugs: SAMZA-962
https://issues.apache.org/jira/browse/SAMZA-962


Repository: samza


Description
---

Added java friendly api to KafkaSystemConsumerMetrics.
KafkaSystemConsumer metrics is done in scala and accessing this api in Java is 
not very intuitive.
This patch adds more straighforward Java api to this class.


Diffs (updated)
-

  
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemConsumerMetrics.scala
 741a176259b9a1cf2e5f7d64b0be98497f1c9242 

Diff: https://reviews.apache.org/r/48352/diff/


Testing
---

manual testing.


Thanks,

Boris Shkolnik

Review Request 48352: Added java friendly api to KafkaMetrics

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48352/
---

Review request for samza and Navina Ramesh.


Bugs: SAMZA-962
https://issues.apache.org/jira/browse/SAMZA-962


Repository: samza


Description
---

Added java friendly api to KafkaSystemConsumerMetrics.
KafkaSystemConsumer metrics is done in scala and accessing this api in Java is 
not very intuitive.
This patch adds more straighforward Java api to this class.


Diffs
-

  
samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemConsumerMetrics.scala
 741a176259b9a1cf2e5f7d64b0be98497f1c9242 

Diff: https://reviews.apache.org/r/48352/diff/


Testing
---

manual testing.


Thanks,

Boris Shkolnik

Re: Review Request 47251: SAMZA-852 - Better logging when system can not be created Always log the exception when we cant instantiate a producer or consumer

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47251/#review133835
---


Ship it!




- Boris Shkolnik


On May 11, 2016, 6:53 p.m., Jake Maes wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47251/
> ---
> 
> (Updated May 11, 2016, 6:53 p.m.)
> 
> 
> Review request for samza, Boris Shkolnik, Chris Pettitt, Jake Maes, Navina 
> Ramesh, Jagadish Venkatraman, Xinyu Liu, and Yi Pan (Data Infrastructure).
> 
> 
> Bugs: SAMZA-852
> https://issues.apache.org/jira/browse/SAMZA-852
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-852 - Better logging when system can not be created Always log the 
> exception when we cant instantiate a producer or consumer
> 
> 
> Diffs
> -
> 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
> 5462208c08cddbfd30d886daffa8c02c82447059 
> 
> Diff: https://reviews.apache.org/r/47251/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew build
> 
> 
> Thanks,
> 
> Jake Maes
> 
>

Re: Review Request 47029: SAMZA-932 port collisions in JmxServer

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47029/#review132060
---




samza-core/src/main/scala/org/apache/samza/metrics/JmxServer.scala (line 92)
<https://reviews.apache.org/r/47029/#comment196122>

Can you please add a log line, so we can know what port was chosen.


- Boris Shkolnik


On May 5, 2016, 4:50 p.m., Jake Maes wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47029/
> ---
> 
> (Updated May 5, 2016, 4:50 p.m.)
> 
> 
> Review request for samza, Boris Shkolnik, Chris Pettitt, Jake Maes, Navina 
> Ramesh, Jagadish Venkatraman, Xinyu Liu, and Yi Pan (Data Infrastructure).
> 
> 
> Bugs: SAMZA-932
> https://issues.apache.org/jira/browse/SAMZA-932
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Experimented with 2 solutions:
> 1. Simplify the JMXServiceUrl to use the same port for the registry and 
> server. This works with tunneling, but from what I read online, will not work 
> if/ SSL is enabled on the server.
> 2. Just get another ephemeral port. This still has a small(er) risk of 
> port collision because of a race condition where the port is stolen by 
> another process before we bind to it. However, it works with tunneling and 
> should work with SSL because the registry and server ports are different.
> 
> Went with option 2 because it'll work with SSL if we enable it in the future.
> 
> 
> Diffs
> -
> 
>   samza-core/src/main/scala/org/apache/samza/metrics/JmxServer.scala 
> e6204c10878589d34096378e6000709266a9b4a5 
> 
> Diff: https://reviews.apache.org/r/47029/diff/
> 
> 
> Testing
> ---
> 
> Tested both options in a test job behind a firewall.
> 
> 
> Thanks,
> 
> Jake Maes
> 
>

Re: Review Request 47029: SAMZA-932 port collisions in JmxServer

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47029/#review132059
---


Ship it!




For my sake, can you explain why the first solution doesn't work with SSL?

- Boris Shkolnik


On May 5, 2016, 4:50 p.m., Jake Maes wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47029/
> ---
> 
> (Updated May 5, 2016, 4:50 p.m.)
> 
> 
> Review request for samza, Boris Shkolnik, Chris Pettitt, Jake Maes, Navina 
> Ramesh, Jagadish Venkatraman, Xinyu Liu, and Yi Pan (Data Infrastructure).
> 
> 
> Bugs: SAMZA-932
> https://issues.apache.org/jira/browse/SAMZA-932
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Experimented with 2 solutions:
> 1. Simplify the JMXServiceUrl to use the same port for the registry and 
> server. This works with tunneling, but from what I read online, will not work 
> if/ SSL is enabled on the server.
> 2. Just get another ephemeral port. This still has a small(er) risk of 
> port collision because of a race condition where the port is stolen by 
> another process before we bind to it. However, it works with tunneling and 
> should work with SSL because the registry and server ports are different.
> 
> Went with option 2 because it'll work with SSL if we enable it in the future.
> 
> 
> Diffs
> -
> 
>   samza-core/src/main/scala/org/apache/samza/metrics/JmxServer.scala 
> e6204c10878589d34096378e6000709266a9b4a5 
> 
> Diff: https://reviews.apache.org/r/47029/diff/
> 
> 
> Testing
> ---
> 
> Tested both options in a test job behind a firewall.
> 
> 
> Thanks,
> 
> Jake Maes
> 
>

Re: How to implement broadcasted KeyValueStorage with 0.9

[Boris Shkolnik]

I may be missing something - but doesn't that sounds exactly as broadcast
stream? You can make all the tasks read from all the partitions.
(
https://samza.apache.org/learn/documentation/0.10/container/samza-container.html,
search for broadcast)

On Tue, May 3, 2016 at 12:46 PM, Jacob Maes  wrote:

> This sounds like SAMZA-402 (
> https://issues.apache.org/jira/browse/SAMZA-402)
> which, unfortunately, hasn't been implemented yet.
>
>
>
> On Tue, May 3, 2016 at 7:48 AM, Bae, Jae Hyeon  wrote:
>
> > Can we make a kafka topic as KV store globally to all tasks, not
> > partitioned? For example, we have a kafka topic "cache" and it has a 4
> > partitions. If we launch a container with 4 tasks, each task will have
> one
> > partition of "cache" kafka topic and create a KV store instance. I want
> to
> > make all 4 partitions in "cache" topic available in all tasks.
> >
> > On Mon, May 2, 2016 at 11:26 PM, Jagadish Venkatraman <
> > jagadish1...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > What do you mean by broadcasted KV storage? Currently a KV store
> instance
> > > is specific to a task. Are you referring to broadcast streams?
> > >
> > >
> > >
> > > On Mon, May 2, 2016 at 7:58 PM, Bae, Jae Hyeon 
> > wrote:
> > >
> > > > Hi Samza devs and users
> > > >
> > > > Is it possible to implement broadcasted KeyValueStorage with 0.9? I
> > know
> > > > 0.10 supports broadcast but we're not yet to 0.10.
> > > >
> > > > Thank you
> > > > Best, Jae
> > > >
> > >
> > >
> > >
> > > --
> > > Jagadish V,
> > > Graduate Student,
> > > Department of Computer Science,
> > > Stanford University
> > >
> >
>

Re: Review Request 46732: SAMZA-930 fix issue with json deserialisation in YarnUtil

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46732/#review131526
---



Can you please clarify in the descriptions - what was the issue and the fix.

- Boris Shkolnik


On April 29, 2016, 11:13 a.m., Alex Buck wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46732/
> ---
> 
> (Updated April 29, 2016, 11:13 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> fix issue with json deserialisation in YarnUtil
> 
> 
> Diffs
> -
> 
>   checkstyle/import-control.xml c15b8e74de8e5aac5ac83278c52ab3dba1630e50 
>   
> samza-autoscaling/src/main/java/org/apache/samza/autoscaling/utils/YarnUtil.java
>  376c549ededaa04401513752b510a4e6c3a6e386 
>   
> samza-autoscaling/src/test/java/org/apache/samza/autoscaling/utils/YarnUtilTest.java
>  PRE-CREATION 
>   samza-autoscaling/src/test/resources/exampleResourceManagerOutput.json 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/46732/diff/
> 
> 
> Testing
> ---
> 
> Very open to any feedback especially as it's my first review request for 
> Samza. Thanks
> 
> https://issues.apache.org/jira/browse/SAMZA-930
> 
> 
> Thanks,
> 
> Alex Buck
> 
>

Re: Review Request 46129: SAMZA 910 fix the container allocator tests

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46129/#review128698
---



test ship it - don't!

- Boris Shkolnik


On April 13, 2016, 1:51 a.m., Jake Maes wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46129/
> ---
> 
> (Updated April 13, 2016, 1:51 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> test commit
> 
> 
> Diffs
> -
> 
>   build.gradle 16facbbf4dff378c561461786ff186bd9eed 
> 
> Diff: https://reviews.apache.org/r/46129/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jake Maes
> 
>

Re: Review Request 46129: SAMZA 910 fix the container allocator tests

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46129/#review128697
---




build.gradle (line 20)
<https://reviews.apache.org/r/46129/#comment192165>

Test comment!


- Boris Shkolnik


On April 13, 2016, 1:51 a.m., Jake Maes wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46129/
> ---
> 
> (Updated April 13, 2016, 1:51 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> test commit
> 
> 
> Diffs
> -
> 
>   build.gradle 16facbbf4dff378c561461786ff186bd9eed 
> 
> Diff: https://reviews.apache.org/r/46129/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jake Maes
> 
>

Re: Review Request 45464: SAMZA-922 Host Affinity - Bug in SamzaContainerRequest causes (recoverable) exceptions in YARN

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45464/#review128508
---




samza-yarn/src/test/java/org/apache/samza/job/yarn/TestSamzaContainerRequest.java
 (line 45)
<https://reviews.apache.org/r/45464/#comment191932>

But it is passed!? Or the name of the test is confusing..


- Boris Shkolnik


On March 30, 2016, 12:30 a.m., Jake Maes wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45464/
> ---
> 
> (Updated March 30, 2016, 12:30 a.m.)
> 
> 
> Review request for samza, Navina Ramesh, Jagadish Venkatraman, and Yi Pan 
> (Data Infrastructure).
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-922 Host Affinity - Bug in SamzaContainerRequest causes (recoverable) 
> exceptions in YARN
> 
> 
> Diffs
> -
> 
>   
> samza-yarn/src/main/java/org/apache/samza/job/yarn/SamzaContainerRequest.java 
> a84e53fc2babe9a932cba2f758cf52103abb4407 
>   
> samza-yarn/src/test/java/org/apache/samza/job/yarn/TestSamzaContainerRequest.java
>  aefae65df5f333d7e9ea405d9d3417dc27718a27 
> 
> Diff: https://reviews.apache.org/r/45464/diff/
> 
> 
> Testing
> ---
> 
> Added a unit test and verified the change in the test job I'm using for 
> SAMZA-886. When relaxed locality is false, these exceptions are much more 
> problematic.
> 
> 
> Thanks,
> 
> Jake Maes
> 
>

Re: Review Request 45504: SAMZA-924: Add disk space monitoring

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45504/#review127660
---


Ship it!




Please see the comment. Up to you to decide to act on it or not.


samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala (line 
398)
<https://reviews.apache.org/r/45504/#comment191047>

Totally agree with your comment - it feels strange to hard code this stuff 
(although that what we do in TaskStorageManager). 
May be it would be cleaner to add a getStoreLocation() method to the 
TaskStorageManager and call it for each taskInstance?


- Boris Shkolnik


On March 30, 2016, 7:43 p.m., Chris Pettitt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45504/
> ---
> 
> (Updated March 30, 2016, 7:43 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> This change introduces the measurement of disk usage for selected
> directories (currently those used by stores only). The feature is off by
> default, but can be enabled by setting "container.disk.poll.interval.ms"
> to a non-zero value.
> 
> 
> Diffs
> -
> 
>   
> samza-core/src/main/java/org/apache/samza/container/disk/DiskSpaceMonitor.java
>  PRE-CREATION 
>   
> samza-core/src/main/java/org/apache/samza/container/disk/PollingScanDiskSpaceMonitor.java
>  PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
> bcbc90a0a460f8733e6d3a50dbc33f3720cad7d0 
>   
> samza-core/src/main/scala/org/apache/samza/container/SamzaContainerMetrics.scala
>  6fae6509d177cc3a54dac9ad1d3e5cc479f4a4f5 
>   
> samza-core/src/test/java/org/apache/samza/container/disk/TestPollingScanDiskSpaceMonitor.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/45504/diff/
> 
> 
> Testing
> ---
> 
> - Added tests for our disk space monitoring implementation.
> - Verified metrics are correctly reported and updated when enabling the 
> feature
> - Verified metrics are not attached to the container when the feature is 
> disabled
> 
> Perf testing for this feature is still pending and is a requirement for
> this to be committed.
> 
> 
> Thanks,
> 
> Chris Pettitt
> 
>

Re: Review Request 44604: split deployment logic

[Boris Shkolnik]

> On March 25, 2016, 5:14 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala,
> >  line 17
> > <https://reviews.apache.org/r/44604/diff/2/?file=127#file127line17>
> >
> > One question on this: how about ThreadJobFactory? Is it the current 
> > limitation that split deployment only works for ProcessJobFactory and 
> > YarnJobFactory? Is it possible to set fwkPath for ThreadJobFactory as well? 
> > If yes, I would recommend to open a JIRA to track that, and document it.

ThreadJobFactory is the only one that doesn't need it, because it is using the 
same phisical process of the JobRunner, which was run from the correct location 
(of the split deployment). ProcessJobFactory, OTOH, needs to build the path to 
the executable, and thus needs the setting.


> On March 25, 2016, 5:14 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerUtil.java, line 
> > 109
> > <https://reviews.apache.org/r/44604/diff/2/?file=131#file131line109>
> >
> > This segment of code is almost dup from the ProcessJobFactory. Can we 
> > fold this in the config.get()? Or some utility class?

created a common method.


- Boris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44604/#review125429
---


On March 15, 2016, 5:33 p.m., Boris Shkolnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44604/
> ---
> 
> (Updated March 15, 2016, 5:33 p.m.)
> 
> 
> Review request for samza and Yi Pan (Data Infrastructure).
> 
> 
> Bugs: SAMZA-849
> https://issues.apache.org/jira/browse/SAMZA-849
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> samza split deployment. See SAMZA-849 for details and design.
> 
> 
> Diffs
> -
> 
>   samza-api/src/main/java/org/apache/samza/job/CommandBuilder.java 
> 97bb05afe7ccd17a9a2443cd1cdf96dd8d1cc740 
>   samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 
> 4f3e9a297ce2c0df0f5f25e0aad62f7bed774cd6 
>   samza-core/src/main/scala/org/apache/samza/job/ShellCommandBuilder.scala 
> c3f60acf9ca88326ef7f2cfbbff2289f70cdd46e 
>   
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala 
> 0792a59cb7b042220c8dbfc0713c5ef42e93ab25 
>   
> samza-core/src/test/scala/org/apache/samza/job/TestShellCommandBuilder.scala 
> fd014b326c8cca914513fa3ac9d913ea78deca23 
>   samza-shell/src/main/bash/run-class.sh 
> 46603c50f1fe02758aae5a568f06b1a65dff9734 
>   samza-shell/src/main/bash/run-container.sh 
> 069279c88c88bd7d8b4a37510b38094446bb034b 
>   samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerUtil.java 
> 6580b9ab2cb293c98f31fee2766db7471e5d99ee 
>   samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 
> 1aa26bba00d2c3d42ef3b8951fa72314d410 
>   
> samza-yarn/src/test/java/org/apache/samza/job/yarn/TestSamzaTaskManager.java 
> 9da1edf6ff165ef0306de8730853ad30551a9831 
> 
> Diff: https://reviews.apache.org/r/44604/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Boris Shkolnik
> 
>

Re: Review Request 44418: SAMZA-887: Use cached JobModel everywhere in the Samza AM container

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44418/#review124688
---


Ship it!




Ship It!

- Boris Shkolnik


On March 5, 2016, 7:08 a.m., Tao Feng wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44418/
> ---
> 
> (Updated March 5, 2016, 7:08 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-877: Use cached JobModel everywhere in the Samza AM container
> 
> 
> Diffs
> -
> 
>   
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java 
> 0c6329ede9b3df4dc05125729b5b44ba2c98803a 
> 
> Diff: https://reviews.apache.org/r/44418/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew clean build && ./gradlew checkstyleMain checkstyleTest
> 
> 
> Thanks,
> 
> Tao Feng
> 
>

Re: Review Request 44604: split deployment logic

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44604/
---

(Updated March 15, 2016, 5:33 p.m.)


Review request for samza and Yi Pan (Data Infrastructure).


Changes
---

merged with the master branch.


Bugs: SAMZA-849
https://issues.apache.org/jira/browse/SAMZA-849


Repository: samza


Description
---

samza split deployment. See SAMZA-849 for details and design.


Diffs (updated)
-

  samza-api/src/main/java/org/apache/samza/job/CommandBuilder.java 
97bb05afe7ccd17a9a2443cd1cdf96dd8d1cc740 
  samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 
4f3e9a297ce2c0df0f5f25e0aad62f7bed774cd6 
  samza-core/src/main/scala/org/apache/samza/job/ShellCommandBuilder.scala 
c3f60acf9ca88326ef7f2cfbbff2289f70cdd46e 
  samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala 
0792a59cb7b042220c8dbfc0713c5ef42e93ab25 
  samza-core/src/test/scala/org/apache/samza/job/TestShellCommandBuilder.scala 
fd014b326c8cca914513fa3ac9d913ea78deca23 
  samza-shell/src/main/bash/run-class.sh 
46603c50f1fe02758aae5a568f06b1a65dff9734 
  samza-shell/src/main/bash/run-container.sh 
069279c88c88bd7d8b4a37510b38094446bb034b 
  samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerUtil.java 
6580b9ab2cb293c98f31fee2766db7471e5d99ee 
  samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 
1aa26bba00d2c3d42ef3b8951fa72314d410 
  samza-yarn/src/test/java/org/apache/samza/job/yarn/TestSamzaTaskManager.java 
9da1edf6ff165ef0306de8730853ad30551a9831 

Diff: https://reviews.apache.org/r/44604/diff/


Testing
---


Thanks,

Boris Shkolnik

Review Request 44604: split deployment logic

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44604/
---

Review request for samza and Yi Pan (Data Infrastructure).


Bugs: SAMZA-849
https://issues.apache.org/jira/browse/SAMZA-849


Repository: samza


Description
---

samza split deployment. See SAMZA-849 for details and design.


Diffs
-

  samza-api/src/main/java/org/apache/samza/job/CommandBuilder.java 
97bb05afe7ccd17a9a2443cd1cdf96dd8d1cc740 
  samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 
4f3e9a297ce2c0df0f5f25e0aad62f7bed774cd6 
  samza-core/src/main/scala/org/apache/samza/job/ShellCommandBuilder.scala 
c3f60acf9ca88326ef7f2cfbbff2289f70cdd46e 
  samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala 
0792a59cb7b042220c8dbfc0713c5ef42e93ab25 
  samza-core/src/test/scala/org/apache/samza/job/TestShellCommandBuilder.scala 
fd014b326c8cca914513fa3ac9d913ea78deca23 
  samza-shell/src/main/bash/run-class.sh 
46603c50f1fe02758aae5a568f06b1a65dff9734 
  samza-shell/src/main/bash/run-container.sh 
069279c88c88bd7d8b4a37510b38094446bb034b 
  samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerUtil.java 
91fae98c074e1648e7168fb8e76d6e1e656816fc 
  samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 
1aa26bba00d2c3d42ef3b8951fa72314d410 
  samza-yarn/src/test/java/org/apache/samza/job/yarn/TestSamzaTaskManager.java 
88d9f24d16fc3d9842b387cfc22edaf1dfa6fd06 

Diff: https://reviews.apache.org/r/44604/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 44241: SAMZA-883 Improve logging for container handling and kafka refresh

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44241/#review121736
---


Ship it!




Looks like this debug->info conversions are mainly for container allocations. 
So it should not add too much noise and should be quite helpfull.

- Boris Shkolnik


On March 2, 2016, 2:24 a.m., Jake Maes wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44241/
> ---
> 
> (Updated March 2, 2016, 2:24 a.m.)
> 
> 
> Review request for samza, Jagadish Venkatraman and Yi Pan (Data 
> Infrastructure).
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-883 Improve logging for container handling and kafka refresh
> 
> 
> Diffs
> -
> 
>   samza-kafka/src/main/scala/org/apache/samza/system/kafka/BrokerProxy.scala 
> 9aa98184f8ab186eff226482f894e8842b9525e5 
>   
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemConsumer.scala
>  b37375360bcad8e089d95e8e3da63a1b9aab 
>   
> samza-kafka/src/main/scala/org/apache/samza/util/ClientUtilTopicMetadataStore.scala
>  0f91622c0c4c270a376f8b91b3e785cca57bc9dd 
>   
> samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerRequestState.java 
> 54db5e5b2b1b109d202e814809adbfd2bc84fb4b 
>   samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerUtil.java 
> 91fae98c074e1648e7168fb8e76d6e1e656816fc 
> 
> Diff: https://reviews.apache.org/r/44241/diff/
> 
> 
> Testing
> ---
> 
> Manual testing and log verification (made sure they weren't verbose)
> 
> 
> In case there is some concern over the messages I switched from debug to 
> info. I let a job run for 4 hours:
> NUM_OCCURRENCES | Message | Notes
> 2 | "ClientUtilTopicMetadataStore [INFO] Fetching topic metadata." | Same 
> occurrences as the ClientUtil INFO log message from Kafka
> 33 | "ContainerRequestState [INFO] Got a new container: ..." | Approx equal 
> to the number of containers in the normal case. All other logs updated in 
> ContainerRequestState will be of similar or lower frequency.
> 
> 
> Thanks,
> 
> Jake Maes
> 
>

Re: checkpoint example?

[Boris Shkolnik]

To add to Jacob's and Jagadish's answers. If you want to to read from 24
hours before (not from the beginning or the end of the stream) you can set
the checkpoint interval (see Jagadish's comment) to 24 hours. It is kind of
unusual, but should work :).

On Tue, Mar 1, 2016 at 4:32 PM, Jacob Maes  wrote:

> A couple notes that may be helpful:
>
> 1. When you have a stateful processor that dies, the changelog is the
> default means by which the state is restored. Change logging is enabled
> with this config:
> stores.store-name.changelog
>
> 2. If, when the job comes back up, it needs to reprocess historical
> messages, it sounds like you actually don't want checkpoints, but you want
> to rewind to the beginning of the topic. You can achieve this with the
> following configs
> systems.system-name.streams.stream-name.samza.reset.offset = true
> systems.system-name.streams.stream-name.samza.offset.default = oldest
> and possibly
> systems.system-name.streams.stream-name.samza.bootstrap = true   // read
> the doc on this one to decide if you need it
>
>
> http://samza.apache.org/learn/documentation/0.10/jobs/configuration-table.html
>
> On Tue, Mar 1, 2016 at 2:57 PM, Jagadish Venkatraman <
> jagadish1...@gmail.com
> > wrote:
>
> > Users need not worry about checkpointing. Samza will automatically commit
> > offsets every 60s. You can choose to commit more often by either
> > 1. Setting task.commit.ms to a smaller value (or)
> > 2. Doing manual commit yourself by setting task.commit.ms = -1. and
> > calling
> > taskCoordinator.commit();
> >
> > I'm curious as to Why processing from the exact previous offset is
> > unacceptable in your usecase?
> >
> > Let's say you process till offfset 100, and crash. Should you not want to
> > resume from 100?
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Mar 1, 2016 at 1:41 PM, Jeff Ramin 
> > wrote:
> >
> > >
> > >
> > > On 03/01/2016 03:10 PM, Jagadish Venkatraman wrote:
> > >
> > >> You don't have to implement any state checkpoint. Samza automatically
> > >> checkpoints state for you. When you recover from a failure/restart you
> > >> will
> > >> resume processing from the previous checkpoint.
> > >>
> > > So, it's merely a configuration issue?
> > >
> > >   What's your usecase?
> > >>
> > >
> > > Pretty standard: have a consumer processing messages, which dies. When
> it
> > > comes back up,
> > > it needs to process messages not just from when it died, but perhaps 24
> > > hours prior to that time.
> > >
> > >
> > > --
> > > Jeff Ramin
> > > Software Engineer
> > > Singlewire Software
> > > 2601 W Beltline Hwy #510
> > > Madison, WI 53713
> > >
> > > Phone Direct - 608.661.1172
> > > www.singlewire.com
> > >
> > >
> >
> >
> > --
> > Jagadish V,
> > Graduate Student,
> > Department of Computer Science,
> > Stanford University
> >
>

Re: Review Request 43550: SAMZA-836: fix unit test failure w/ FlushOptions() in rocksdbjni-3.13.1

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43550/#review121371
---


Ship it!




I wish we understood the reason for it.

- Boris Shkolnik


On Feb. 13, 2016, 12:13 a.m., Yi Pan (Data Infrastructure) wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43550/
> ---
> 
> (Updated Feb. 13, 2016, 12:13 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-836
> https://issues.apache.org/jira/browse/SAMZA-836
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-836: fix unit test failure w/ FlushOptions() in rocksdbjni-3.13.1
> 
> 
> Diffs
> -
> 
>   
> samza-kv-rocksdb/src/test/scala/org/apache/samza/storage/kv/TestRocksDbKeyValueStore.scala
>  0c86a5a1550894645c2ab0010b70b124ad3dbd98 
> 
> Diff: https://reviews.apache.org/r/43550/diff/
> 
> 
> Testing
> ---
> 
> Ran the full set of unit tests both on Mac and Linux and both passed.
> 
> 
> Thanks,
> 
> Yi Pan (Data Infrastructure)
> 
>

Re: Review Request 43766: SAMZA-872: little change in Logging docs

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43766/#review120637
---


Ship it!




after addressing Jagadish's comment.

- Boris Shkolnik


On Feb. 19, 2016, 2:48 p.m., Branislav Cogic wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43766/
> ---
> 
> (Updated Feb. 19, 2016, 2:48 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-872
> https://issues.apache.org/jira/browse/SAMZA-872
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Changed docs for Logging: DailyRollingFileAppender removed with 
> RollingFileAppender
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/jobs/logging.md 
> caca43f9c3f4c630365decd425e375afe8b09355 
> 
> Diff: https://reviews.apache.org/r/43766/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Branislav Cogic
> 
>

Re: Review Request 43053: allow warning instead of fail in case of invalid num of partitions in the checkpoint partition

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43053/
---

(Updated Feb. 18, 2016, 8:04 p.m.)


Review request for samza.


Changes
---

updated docs.


Bugs: SAMZA-864
https://issues.apache.org/jira/browse/SAMZA-864


Repository: samza


Description
---

We have a validation code that verifies that checkpoint topic has the right 
number of partitions (1).
But, in some environments, it is difficult to repair or delete the invalid 
topic. 
This config will allow to by pass this validation (it will issue a warning 
only) and to continue with a checkpoint topic with an incorrect number of 
partitions. 
The checkpoints are written into partion 0.


Diffs (updated)
-

  docs/learn/documentation/versioned/jobs/configuration-table.html 
67055307329c81c9347f1de352420a48a63634b4 
  samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 
1a8adae4d30fa198c90e8c177c7f17269c5953cd 
  
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala
 787de1f62479a098bf251f072fca03bbf92f7c6d 
  
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala
 7db894091284794b7f5fac164eb55b5d78184a36 
  
samza-kafka/src/main/scala/org/apache/samza/migration/KafkaCheckpointMigration.scala
 c6b1fe4bf3c3601502e014d582d86f8ea0850b20 
  samza-kafka/src/main/scala/org/apache/samza/util/KafkaUtil.scala 
f4311d1cda7c66c66544c5a3ac94a17cae62863a 
  
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala
 af4051b28df5eeaeaee527a24907a8e66441f743 

Diff: https://reviews.apache.org/r/43053/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 43053: allow warning instead of fail in case of invalid num of partitions in the checkpoint partition

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43053/
---

(Updated Feb. 18, 2016, 7:21 p.m.)


Review request for samza.


Changes
---

fixed the test.


Bugs: SAMZA-864
https://issues.apache.org/jira/browse/SAMZA-864


Repository: samza


Description
---

We have a validation code that verifies that checkpoint topic has the right 
number of partitions (1).
But, in some environments, it is difficult to repair or delete the invalid 
topic. 
This config will allow to by pass this validation (it will issue a warning 
only) and to continue with a checkpoint topic with an incorrect number of 
partitions. 
The checkpoints are written into partion 0.


Diffs (updated)
-

  samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 
1a8adae4d30fa198c90e8c177c7f17269c5953cd 
  
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala
 787de1f62479a098bf251f072fca03bbf92f7c6d 
  
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala
 7db894091284794b7f5fac164eb55b5d78184a36 
  
samza-kafka/src/main/scala/org/apache/samza/migration/KafkaCheckpointMigration.scala
 c6b1fe4bf3c3601502e014d582d86f8ea0850b20 
  samza-kafka/src/main/scala/org/apache/samza/util/KafkaUtil.scala 
f4311d1cda7c66c66544c5a3ac94a17cae62863a 
  
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala
 af4051b28df5eeaeaee527a24907a8e66441f743 

Diff: https://reviews.apache.org/r/43053/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 43648: SAMZA-872: removed unsafe log4j DaillyRollingFileAppender from hello-samza

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43648/#review119545
---


Ship it!




Ship It!

- Boris Shkolnik


On Feb. 17, 2016, 11:01 a.m., Branislav Cogic wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43648/
> ---
> 
> (Updated Feb. 17, 2016, 11:01 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-872
> https://issues.apache.org/jira/browse/SAMZA-872
> 
> 
> Repository: samza-hello-samza
> 
> 
> Description
> ---
> 
> Changed log4j configuration in hello-samza project to reference 
> RolingFileAppender instead of DailyRollingFileAppender
> 
> 
> Diffs
> -
> 
>   src/main/resources/log4j.xml f0de7658c76fe1b3925b0d73dd3236ff8a297590 
> 
> Diff: https://reviews.apache.org/r/43648/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Branislav Cogic
> 
>

Re: Review Request 39032: SAMZA-787: task.log4j.system should not be guessed if not configured

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39032/#review118471
---


Ship it!




- Boris Shkolnik


On Oct. 6, 2015, 12:44 a.m., Navina Ramesh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39032/
> ---
> 
> (Updated Oct. 6, 2015, 12:44 a.m.)
> 
> 
> Review request for samza, Yan Fang, Chinmay Soman, Chris Riccomini, and Yi 
> Pan (Data Infrastructure).
> 
> 
> Bugs: SAMZA-787
> https://issues.apache.org/jira/browse/SAMZA-787
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-787: task.log4j.system should not be guessed if not configured
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/jobs/configuration-table.html 
> b42c34cf55096467ab20507d82af7d002abd8e1e 
>   docs/learn/documentation/versioned/jobs/logging.md 
> a3bb054c0febabf2d7b6e3e18fa710f93382d744 
>   samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 
> 209296dd48d64159d14bf20a286d971b54fb710e 
>   
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java 
> 894845332f2a08b984c587884ed13f2854b4d492 
>   
> samza-log4j/src/test/java/org/apache/samza/config/TestLog4jSystemConfig.java 
> f7d3cbecfb8901f5fe1a290ce9b08446ce48f4df 
>   
> samza-log4j/src/test/java/org/apache/samza/logging/log4j/TestStreamAppender.java
>  3e812405edf1ad4fca43784b1668ab3ae6ff6475 
> 
> Diff: https://reviews.apache.org/r/39032/diff/
> 
> 
> Testing
> ---
> 
> ./bin/check-all.sh passed
> 
> 
> Thanks,
> 
> Navina Ramesh
> 
>

Re: Review Request 43053: allow warning instead of fail in case of invalid num of partitions in the checkpoint partition

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43053/
---

(Updated Feb. 5, 2016, 9:48 p.m.)


Review request for samza.


Bugs: SAMZA-864
https://issues.apache.org/jira/browse/SAMZA-864


Repository: samza


Description
---

We have a validation code that verifies that checkpoint topic has the right 
number of partitions (1).
But, in some environments, it is difficult to repair or delete the invalid 
topic. 
This config will allow to by pass this validation (it will issue a warning 
only) and to continue with a checkpoint topic with an incorrect number of 
partitions. 
The checkpoints are written into partion 0.


Diffs
-

  samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 
1a8adae4d30fa198c90e8c177c7f17269c5953cd 
  
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala
 787de1f62479a098bf251f072fca03bbf92f7c6d 
  
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala
 7db894091284794b7f5fac164eb55b5d78184a36 
  
samza-kafka/src/main/scala/org/apache/samza/migration/KafkaCheckpointMigration.scala
 c6b1fe4bf3c3601502e014d582d86f8ea0850b20 
  samza-kafka/src/main/scala/org/apache/samza/util/KafkaUtil.scala 
f4311d1cda7c66c66544c5a3ac94a17cae62863a 
  
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala
 af4051b28df5eeaeaee527a24907a8e66441f743 

Diff: https://reviews.apache.org/r/43053/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 43053: allow warning instead of fail in case of invalid num of partitions in the checkpoint partition

[Boris Shkolnik]

> On Feb. 3, 2016, 2:36 a.m., Jagadish Venkatraman wrote:
> > samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala,
> >  line 84
> > <https://reviews.apache.org/r/43053/diff/2/?file=1230302#file1230302line84>
> >
> > nit: failOnTopicValidation.

tnx, fixed.


- Boris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43053/#review117554
-------


On Feb. 3, 2016, 1:41 a.m., Boris Shkolnik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43053/
> ---
> 
> (Updated Feb. 3, 2016, 1:41 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> We have a validation code that verifies that checkpoint topic has the right 
> number of partitions (1).
> But, in some environments, it is difficult to repair or delete the invalid 
> topic. 
> This config will allow to by pass this validation (it will issue a warning 
> only) and to continue with a checkpoint topic with an incorrect number of 
> partitions. 
> The checkpoints are written into partion 0.
> 
> 
> Diffs
> -
> 
>   samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 
> 1a8adae4d30fa198c90e8c177c7f17269c5953cd 
>   
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala
>  787de1f62479a098bf251f072fca03bbf92f7c6d 
>   
> samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala
>  7db894091284794b7f5fac164eb55b5d78184a36 
>   
> samza-kafka/src/main/scala/org/apache/samza/migration/KafkaCheckpointMigration.scala
>  c6b1fe4bf3c3601502e014d582d86f8ea0850b20 
>   samza-kafka/src/main/scala/org/apache/samza/util/KafkaUtil.scala 
> f4311d1cda7c66c66544c5a3ac94a17cae62863a 
>   
> samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala
>  af4051b28df5eeaeaee527a24907a8e66441f743 
> 
> Diff: https://reviews.apache.org/r/43053/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Boris Shkolnik
> 
>

Re: ChangeLog Question for TTL rocksDB stores

[Boris Shkolnik]

As Jacob mentioned there is not direct relationship between the rocksdb tts
(internal to rocksdb) and changelog (done by Samza).
The problem may arise if the store is restored from the changelog, since
the log will have the expired entries, and they will be entered with the
NEW date (and as Yi mentioned, there is no TTL on kafka based changelogs
now).
But since it is not an error per se, SAMZA-862
 has changed this message
to be a warning instead of an error.

On Thu, Jan 28, 2016 at 11:51 AM, Yi Pan  wrote:

> Hi, David,
>
> The "compaction" referred to together w/ TTL is referring to RocksDb's
> compaction, not the Kafka-based changelog topic. Currently, TTL is not
> applied to Kafka-based changelog topic. SAMZA-677 is opened for this.
>
> -Yi
>
> On Thu, Jan 28, 2016 at 11:36 AM, David Garcia
>  > wrote:
>
> > Ok, that makes sense.  I had assumed that the changelog was supported
> > because the docs mention that TTL is enforced upon ³compaction² (I had
> > assumed compaction of the DB changelog).  Which topic does the TTL policy
> > listen for the compaction of (since compaction policies of topics can
> > differ)?
> >
> > -David
> >
> > On 1/27/16, 8:46 PM, "Jacob Maes"  wrote:
> >
> > >Here's my understanding. The others can correct me if I'm mistaken.
> > >
> > >Samza provides the changelog functionality by intercepting RocksDB "put"
> > >and "delete" operations. However, TTL is managed by RocksDB internally
> and
> > >there aren't any hooks exposed in the RocksDB JNI. So there are 2
> problems
> > >that arise with TTL and change logging:
> > >1. Samza doesn't know when an entry expires, so it can't delete the
> > >expired
> > >entry from the changelog.
> > >2. The changelog currently has no concept of entry age/timestamp, so
> when
> > >the changelog is restored, it's unknown whether some subset (or all) of
> > >the
> > >entries should be immediately expired.
> > >
> > >These issues aren't insurmountable, but they weren't pursued for the
> > >initial implementation. Perhaps because there was a shortage of use
> cases
> > >that needed both TTL and changelogging, but I'm not sure.
> > >
> > >-Jake
> > >
> > >On Wed, Jan 27, 2016 at 6:19 PM, David Garcia
> > >
> > >wrote:
> > >
> > >> So, I saw this very scary message:
> > >>
> > >>
> > >> ERROR - e.kv.RocksDbKeyValueStore$ - sessionJoinStore is a TTL based
> > >> store, changelog is not supported for TTL based stores, use at your
> own
> > >> discretion
> > >>
> > >>
> > >>
> > >>
> > >> A few of questions:
> > >>
> > >> 1.) Does this mean that this store is NOT backed by the changelog?
> > >>
> > >> 2.) Provided that the store IS backed by a change log, do the TTL
> > >> expirations commit removals from the changelog (I.e.
> Nulls)...presumably
> > >> upon compaction
> > >>
> > >> 3.) Can I please get a bit more detail on how TTL affects a changelog
> > >> store?
> > >>
> > >>
> > >> -David
> > >>
> >
> >
>

Re: Review Request 43053: allow warning instead of fail in case of invalid num of partitions in the checkpoint partition

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43053/
---

(Updated Feb. 3, 2016, 1:41 a.m.)


Review request for samza.


Changes
---

addressed review comments.


Repository: samza


Description
---

We have a validation code that verifies that checkpoint topic has the right 
number of partitions (1).
But, in some environments, it is difficult to repair or delete the invalid 
topic. 
This config will allow to by pass this validation (it will issue a warning 
only) and to continue with a checkpoint topic with an incorrect number of 
partitions. 
The checkpoints are written into partion 0.


Diffs (updated)
-

  samza-core/src/main/scala/org/apache/samza/config/JobConfig.scala 
1a8adae4d30fa198c90e8c177c7f17269c5953cd 
  
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManager.scala
 787de1f62479a098bf251f072fca03bbf92f7c6d 
  
samza-kafka/src/main/scala/org/apache/samza/checkpoint/kafka/KafkaCheckpointManagerFactory.scala
 7db894091284794b7f5fac164eb55b5d78184a36 
  
samza-kafka/src/main/scala/org/apache/samza/migration/KafkaCheckpointMigration.scala
 c6b1fe4bf3c3601502e014d582d86f8ea0850b20 
  samza-kafka/src/main/scala/org/apache/samza/util/KafkaUtil.scala 
f4311d1cda7c66c66544c5a3ac94a17cae62863a 
  
samza-kafka/src/test/scala/org/apache/samza/checkpoint/kafka/TestKafkaCheckpointManager.scala
 af4051b28df5eeaeaee527a24907a8e66441f743 

Diff: https://reviews.apache.org/r/43053/diff/


Testing
---


Thanks,

Boris Shkolnik

Review Request 42815: issue a warn() instead of error() in case of rocksdb store with both TTL and changelog enabled

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42815/
---

Review request for samza.


Bugs: SAMZA-862
https://issues.apache.org/jira/browse/SAMZA-862


Repository: samza


Description
---

issue a warn() instead of error() in case of rocksdb store with both TTL and 
changelog enabled


Diffs
-

  
samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDbKeyValueStore.scala
 211fc3be1e168f1f92812406785b39b5a3fd9174 

Diff: https://reviews.apache.org/r/42815/diff/


Testing
---


Thanks,

Boris Shkolnik

Review Request 42619: move public kv api from samza-kv to samza-api package

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42619/
---

Review request for samza.


Bugs: SAMZA-861
https://issues.apache.org/jira/browse/SAMZA-861


Repository: samza


Description
---

move public kv api from samza-kv to samza-api package


Diffs
-

  samza-kv/src/main/java/org/apache/samza/storage/kv/Entry.java  
  samza-kv/src/main/java/org/apache/samza/storage/kv/KeyValueIterator.java  
  samza-kv/src/main/java/org/apache/samza/storage/kv/KeyValueStore.java  

Diff: https://reviews.apache.org/r/42619/diff/


Testing
---


Thanks,

Boris Shkolnik

Re: Review Request 41884: SAMZA-802: KafkaSystemAdmin needs to handle empty topic offsets

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41884/#review112954
---

Ship it!


Logic seems correct.

- Boris Shkolnik


On Jan. 4, 2016, 12:59 p.m., Aleksandar Pejakovic wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41884/
> ---
> 
> (Updated Jan. 4, 2016, 12:59 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Quick fix to prevent NullPointerException.
> 
> 
> Diffs
> -
> 
>   
> samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala
>  9dc436a 
> 
> Diff: https://reviews.apache.org/r/41884/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aleksandar Pejakovic
> 
>

Re: Review Request 41663: SAMZA-843 : Slow start of Samza jobs with large number of containers

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41663/#review112678
---



samza-autoscaling/src/main/java/org/apache/samza/autoscaling/deployer/ConfigManager.java
 (line 328)
<https://reviews.apache.org/r/41663/#comment173182>

Why do we need defaultReadJobModelDelayMS?
If it is not specified - will still use Rand(100)?



samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala 
(line 117)
<https://reviews.apache.org/r/41663/#comment173183>

Where is the change to JobCoordinator constructor? Did I miss it?


- Boris Shkolnik


On Jan. 4, 2016, 10:40 p.m., Navina Ramesh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41663/
> ---
> 
> (Updated Jan. 4, 2016, 10:40 p.m.)
> 
> 
> Review request for samza, Boris Shkolnik, Yan Fang, Chris Riccomini, Jake 
> Maes, Jagadish Venkatraman, Xinyu Liu, and Yi Pan (Data Infrastructure).
> 
> 
> Bugs: SAMZA-843
> https://issues.apache.org/jira/browse/SAMZA-843
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-843 : Slow start of Samza jobs with large number of containers
> 
> 
> Diffs
> -
> 
>   
> samza-autoscaling/src/main/java/org/apache/samza/autoscaling/deployer/ConfigManager.java
>  87346bc9f3d92e3ae86ed264c82d12a4d27f3188 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
> ddce1481cee41dfb997103dbe3d5df44a123564e 
>   samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala 
> 112ec1c2935fb6c2ebe9b5d9090263cea94c0cf9 
>   
> samza-core/src/main/scala/org/apache/samza/coordinator/server/JobServlet.scala
>  a3baddbe81c96f58e718e062e964485eaa2dc701 
>   samza-core/src/main/scala/org/apache/samza/util/Util.scala 
> 58fbb8f8177a109d35659a29ef6660e239334de2 
>   
> samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 
> 365ff0a8ece808a8143ee3580f1f42238dd292d1 
>   
> samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala
>  80cccf3104e6e0ce7457303f40baa2f4fa807782 
> 
> Diff: https://reviews.apache.org/r/41663/diff/
> 
> 
> Testing
> ---
> 
> Added some additional unit tests
> ./gradlew clean build
> 
> 
> Thanks,
> 
> Navina Ramesh
> 
>

Re: Review Request 41874: SAMZA-837: Support Gradle 2.9

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41874/#review112707
---

Ship it!


Ship It!

- Boris Shkolnik


On Jan. 4, 2016, 9:59 a.m., Aleksandar Bircakovic wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41874/
> ---
> 
> (Updated Jan. 4, 2016, 9:59 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Formatted code in some Java classes so now gradle clean check runs with no 
> checkstyle errors with new versions of Gradle.
> 
> 
> Diffs
> -
> 
>   samza-api/src/main/java/org/apache/samza/storage/StorageEngineFactory.java 
> 963ccf2 
>   
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java
>  429573b 
>   
> samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemConsumer.java
>  0e73e18 
> 
> Diff: https://reviews.apache.org/r/41874/diff/
> 
> 
> Testing
> ---
> 
> gradle clean check - BUILD SUCCESSFUL
> ./gradlew clean build - BUILD SUCCESSFUL
> 
> 
> Thanks,
> 
> Aleksandar Bircakovic
> 
>

Re: Review Request 41106: SAMZA-833: ProcessJob mishandling containers

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41106/#review109619
---

Ship it!


- Boris Shkolnik


On Dec. 9, 2015, 6:41 a.m., Tao Feng wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41106/
> ---
> 
> (Updated Dec. 9, 2015, 6:41 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> 1. throws runtime exception if user sets yarn.container.count or 
> job.container.count larger than 1 for ProcessJobFactory; 
> 2. update per Yi's comment
> 
> 
> Diffs
> -
> 
>   
> samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala 
> 0792a59cb7b042220c8dbfc0713c5ef42e93ab25 
> 
> Diff: https://reviews.apache.org/r/41106/diff/
> 
> 
> Testing
> ---
> 
> 1. ./gradlew clean build
> 2. ./gradlew checkstyleMain checkstyleTest
> 3. manual test open source hello samza and verify that by giving 
> yarn.container.count or job.container.count with value greater than 1 for 
> ProcessJobFactory, the test will throw the desired exception.
> 
> 
> Thanks,
> 
> Tao Feng
> 
>

Re: Passing Java system properties to YARN containers of Samza Job

[Boris Shkolnik]

I am not sure about AWS credentials, but, in general, you can use task.opts
config value to pass any java level settings, which will be passed to each
container jvm.

On Tue, Dec 8, 2015 at 9:57 AM, Gordon Tai  wrote:

> Hi,
>
> Is there any convenient way to pass custom Java system properties to Samza
> jobs?
> Specifically, since the system properties I wish to pass are
> aws.accessKeyId and aws.secretKey for AWS credentials  (
>
> http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html
> ),
> writing
> them into the Samza job config file doesn't seem like a good idea.
> Any help is great, thanks!
>
> BR,
> Gordon
>
> --
> *Gordon Tai* | Data Engineer | gor...@vm5.com | http://www.vm5.com
>
>
>
> *VMFIVE CONFIDENTIAL*
> This message, including any attachments, may contain information that is
> confidential, proprietary, privileged or otherwise protected by law, and is
> intended only for the parties involved in the subject discussed therein. If
> you have received this e-mail by mistake, please immediately delete the
> message and notify the sender of such incident. Please be noted that any
> unauthorized use, dissemination, distribution or copying of this email is
> strictly prohibited.
>

Re: Review Request 40933: SAMZA-829: Add YARN configure doc to allow clean shutdown for large state jobs

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40933/#review109673
---

Ship it!


Ship It!

- Boris Shkolnik


On Dec. 3, 2015, 11:10 p.m., Yi Pan (Data Infrastructure) wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40933/
> ---
> 
> (Updated Dec. 3, 2015, 11:10 p.m.)
> 
> 
> Review request for samza, Jake Maes and Jon Bringhurst.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-829: Add YARN configure doc to allow clean shutdown for large state jobs
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/yarn/yarn-host-affinity.md 
> 108dfbc914d572334878b7e3718de5e541b8c6a6 
> 
> Diff: https://reviews.apache.org/r/40933/diff/
> 
> 
> Testing
> ---
> 
> Locally tested. Configure variables verified in LinkedIn tests.
> 
> 
> Thanks,
> 
> Yi Pan (Data Infrastructure)
> 
>

Re: [VOTE] Samza 0.10.0 Release Candidate 2

[Boris Shkolnik]

I've opened a ticket to track this rocksdb:test failure:

   1. SAMZA-836 <https://issues.apache.org/jira/browse/SAMZA-836>


On Wed, Dec 9, 2015 at 7:11 PM, Boris Shkolnik <bor...@gmail.com> wrote:

> +1 (non-binding).
> One note. I failed for me couple of times with the test below, but it
> might've been related to some old versions of rocksdbjni, but I am not
> sure. I've run it couple of more times and it ran fine.
>
>
> On Wed, Dec 9, 2015 at 12:15 PM, xinyu liu <xinyuliu...@gmail.com> wrote:
>
>> +1 on my side. I also ran the gradle build and unit tests without failure.
>>
>> Thanks,
>> Xinyu
>>
>> On Wed, Dec 9, 2015 at 9:54 AM, Tao Feng <fengta...@gmail.com> wrote:
>>
>> > +1 from my side(non-binding). I download the package and successfully
>> run
>> > all the unit tests without failure.
>> >
>> > On Tue, Dec 8, 2015 at 3:38 PM, Yi Pan <nickpa...@gmail.com> wrote:
>> >
>> > > Hey all,
>> > >
>> > >
>> > > This is a call for a vote on a release of Apache Samza 0.10.0. Thanks
>> to
>> > > everyone who has contributed to this release. We are very glad to see
>> > some
>> > > new contributors in this release.
>> > >
>> > >
>> > > The release candidate can be downloaded from here:
>> > >
>> > >
>> > > http://home.apache.org/~nickpan47/samza-0.10.0-rc2/
>> > >
>> > >
>> > > The release candidate is signed with pgp key 911402D8, which is
>> > >
>> > > included in the repository's KEYS file:
>> > >
>> > >
>> > >
>> > >
>> >
>> https://git-wip-us.apache.org/repos/asf?p=samza.git;a=blob;f=KEYS;h=66cbd15cddbbd798c3529e9a8b7f052aab0037a7
>> > >
>> > >
>> > > and can also be found on keyservers:
>> > >
>> > > http://pgp.mit.edu/pks/lookup?op=get=0x911402D8
>> > >
>> > >
>> > > The git tag is release-0.10.0-rc2 and signed with the same pgp key:
>> > >
>> > >
>> > >
>> > >
>> >
>> https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tag;h=4a37a3c4754b94805646522fc6644f2dd998e828
>> > >
>> > >
>> > > Test binaries have been published to Maven's staging repository, and
>> are
>> > >
>> > > available here:
>> > >
>> > >
>> > >
>> https://repository.apache.org/content/repositories/orgapachesamza-1011/
>> > >
>> > >
>> > > Note that the binaries were built with JDK7 without incident. This is
>> the
>> > > first version of Samza that does not support JDK6 any more.
>> > >
>> > >
>> > > 128 issues were resolved for this release:
>> > >
>> > >
>> > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SAMZA%20AND%20fixVersion%20%3D%200.10.0%20AND%20status%20in%20(Resolved%2C%20Closed)
>> > >
>> > >
>> > > The vote will be open for 72 hours ( end in 4:00pm Friday, 12/11/2015
>> ).
>> > >
>> > > Please download the release candidate, check the hashes/signature,
>> build
>> > it
>> > >
>> > > and test it, and then please vote:
>> > >
>> > >
>> > > [ ] +1 approve
>> > >
>> > > [ ] +0 no opinion
>> > >
>> > > [ ] -1 disapprove (and reason why)
>> > >
>> > >
>> > > +1 from my side for the release.
>> > >
>> > >
>> > > Yi Pan
>> > >
>> > > nickpa...@gmail.com
>> > >
>> >
>>
>
>

Re: Review Request 40857: SAMZA 826 Fix string format issue with log message in ContainerUtil

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40857/#review109674
---

Ship it!


Ship It!

- Boris Shkolnik


On Dec. 2, 2015, 7:53 a.m., Aleksandar Bircakovic wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40857/
> ---
> 
> (Updated Dec. 2, 2015, 7:53 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> The '%s' replaced with '{}'
> 
> 
> Diffs
> -
> 
>   samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerUtil.java 
> 1fb6a5f 
> 
> Diff: https://reviews.apache.org/r/40857/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aleksandar Bircakovic
> 
>

Re: Review Request 41071: SAMZA-843: fix heap usage increase caused by container timer change

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41071/#review109423
---

Ship it!


We should consider more permanent fix later - may be using preallocated storage 
for timer, to avoid allocation and deallocation of the map values.

- Boris Shkolnik


On Dec. 8, 2015, 2:30 a.m., Xinyu Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41071/
> ---
> 
> (Updated Dec. 8, 2015, 2:30 a.m.)
> 
> 
> Review request for samza and Yi Pan (Data Infrastructure).
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> After the change of container timer metrics (chooseNs, windowNs, processNs, 
> and commitNs) from millisecond to nanosecond, we noticed a dramatic increase 
> of memory heap usage in one of our production job. After investigation we 
> found that the SlidingTimeWindowReservoir.update(duration) will be called 
> much more frequently due to the duration is non-zero after the nanosecond 
> change (In contrast, it is often zero when using millisecond). Within the 
> 5-minute window, the storage inside SlidingTimeWindowReservoir increases a 
> lot for a high qps job (for our job with around 10K qps, it increases the 
> heap from <5M to 100M). It causes long GCs and degrades the job performance. 
> The fix will make the SlidingTimeWindowReservoir collision buffer default to 
> be 1 and configurable from the constructor.
> 
> 
> Diffs
> -
> 
>   
> samza-api/src/main/java/org/apache/samza/metrics/SlidingTimeWindowReservoir.java
>  df543599771b7cda3be6ea702a85091cf61883d7 
>   samza-api/src/main/java/org/apache/samza/metrics/Timer.java 
> b49d14758116660ac6b26cc7e2459b293343e47e 
>   
> samza-api/src/test/java/org/apache/samza/metrics/TestSlidingTimeWindowReservoir.java
>  d392b3258b7aca4323d663159c4e545e113277bb 
>   samza-api/src/test/java/org/apache/samza/metrics/TestTimer.java 
> 63c183f9283c58006de23a342e61717e062030c3 
>   samza-core/src/test/scala/org/apache/samza/container/TestRunLoop.scala 
> b4d6f351f97a7c3a29c133b4ba3876ce4b6baca2 
> 
> Diff: https://reviews.apache.org/r/41071/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Xinyu Liu
> 
>

Re: Review Request 40485: SAMZA-767 yarn.queue option is not used anywhere

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40485/#review107232
---

Ship it!



samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala (line 76)
<https://reviews.apache.org/r/40485/#comment166262>

nit. add a new line here.


- Boris Shkolnik


On Nov. 19, 2015, 2:56 p.m., Aleksandar Pejakovic wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40485/
> ---
> 
> (Updated Nov. 19, 2015, 2:56 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Added missing code from 
> [SAMZA-491](https://issues.apache.org/jira/browse/SAMZA-491)
> 
> 
> Diffs
> -
> 
>   samza-yarn/src/main/java/org/apache/samza/config/YarnConfig.java a572aa2 
>   samza-yarn/src/main/scala/org/apache/samza/job/yarn/ClientHelper.scala 
> a2b9279 
>   samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 02f46a1 
> 
> Diff: https://reviews.apache.org/r/40485/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aleksandar Pejakovic
> 
>

Re: Review Request 40313: SAMZA-785

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40313/#review106697
---

Ship it!


- Boris Shkolnik


On Nov. 13, 2015, 10:51 p.m., VENKATA KRISHNA NANNAPANENI wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40313/
> ---
> 
> (Updated Nov. 13, 2015, 10:51 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> check for system serde name for stream appender.
> 
> 
> Diffs
> -
> 
>   samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 
> 59015a90acaf822469b0b24164e2d4c4a8000bce 
>   
> samza-log4j/src/main/java/org/apache/samza/logging/log4j/StreamAppender.java 
> 0c6329ede9b3df4dc05125729b5b44ba2c98803a 
>   
> samza-log4j/src/test/java/org/apache/samza/logging/log4j/TestStreamAppender.java
>  e2e17a06159a35234270f342e8ae1c81031d971b 
> 
> Diff: https://reviews.apache.org/r/40313/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> VENKATA KRISHNA NANNAPANENI
> 
>

Re: Monitoring consumer lag

[Boris Shkolnik]

Just to add to Jagasish's suggestion - you can configure
MetricsSnapshotRecorder to output the metrics into a stream and read them
from there if it works better for you.

Boris.

On Mon, Nov 16, 2015 at 12:32 PM, Jordan Shaw  wrote:

> Michael,
> I should have added if your using Burrow in the context of samza consumers
> it probably won't work because samza does it's own offset tracking (see
> checkpoint topics). The messages-behind-high-watermark is probably your
> best bet if you just want something out of the box and don't care about lag
> time only message count behind.
> -Jordan
>
> On Mon, Nov 16, 2015 at 11:40 AM, Michael Ravits 
> wrote:
>
> > Thanks Jagadish! I'll look further into this.
> >
> > Jordan, I tested Burrow with 0.8.3-SNAPSHOT and set it to read consumer
> > offsets from zookeeper because I assumed that it's the default Kafka
> config
> > for commiting offsets. Will try again with Burrow set to read from
> > __consumer_offsets.
> >
> > Thanks
> >
> > On Mon, Nov 16, 2015 at 8:04 PM, Jordan Shaw  wrote:
> >
> > > Michael,
> > > It depends on how you define lag.
> > >
> > > 1) If you define lag as the total number of messages behind then burrow
> > is
> > > a good tool as long as all your infrastructure is on 0.8.2, it
> basically
> > > works by inspecting the __consumer_offsets topic which was introduced
> in
> > > 0.8.2 (they said they were going to support <0.8.2 but i don't think
> > that's
> > > a thing yet).
> > >
> > > 2. If you define lag in time units I would recommend sending a
> timestamp
> > > with the msg and doing some "manual" inspecting on the consumer end.
> > > -Jordan
> > >
> > > On Mon, Nov 16, 2015 at 9:16 AM, Michael Ravits  >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'd like to monitor consumer's lag.
> > > > Found this tool https://github.com/linkedin/Burrow.
> > > > But now realized that Samza is using it's own checkpointing
> mechanism.
> > > >
> > > > So question is what's the best way to monitor whether and how much
> the
> > > > consumer is lagging?
> > > >
> > > > On a related subject, I'd also like to monitor throughput per topic
> in
> > > > terms of messages per second and bytes per second. Should I query
> > brokers
> > > > periodically, or maybe there is a better way?
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > >
> > >
> > >
> > > --
> > > Jordan Shaw
> > > Full Stack Software Engineer
> > > PubNub Inc
> > > 1045 17th St
> > > San Francisco, CA 94107
> > >
> >
>
>
>
> --
> Jordan Shaw
> Full Stack Software Engineer
> PubNub Inc
> 1045 17th St
> San Francisco, CA 94107
>

Re: API for containers?

[Boris Shkolnik]

Here is couple of ideas (not sure if that exactly what you want).
You can read JobModel of a job (form JobCoordinator server) and this job
model will contain container to tasks (and to partitions) mapping.

In order to find containers and the nodes - you can use yarn web api.
http://RM/cluster, then go to ApplicationMaster and list all the containers.

But remember this mapping can change, if a container fails, it will be
restarted on the same or different node.

On Wed, Nov 4, 2015 at 11:42 AM, Rick Mangi  wrote:

> Hello,
>
> I’m wondering if there’s currently any API (or other way) to find out
> which container would handle a theoretical key and what node that container
> is running on.
>
> The use-case I’m thinking of is if I wanted to run a rest service on each
> node which could handle queries against the current state of a job by
> talking to a shared kv store. I saw that there is an elastic search kv
> implementation in the codebase but didn’t see any way to direct a query to
> a specific instance of elastic search running on the cluster. To do this
> you would want to look at a key and determine which container would handle
> that key in the stream, then hit a service on that node (potentially there
> could be more than 1 container on the node of course…).
>
> Thanks,
>
> Rick
>
>
>

Re: Review Request 39806: SAMZA-798 : Performance and stability issue after combining checkpoint and coordinator stream

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39806/#review104818
---



samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala 
(line 30)
<https://reviews.apache.org/r/39806/#comment163068>

a nit.
I think the name of this setting is somewhat confusing.
Either it should be skip-auto-migration or skip-kafka-migration, or we 
should check its value in case of any migration (even for kafka migration).
Or am I missing something?


- Boris Shkolnik


On Nov. 2, 2015, 9:56 p.m., Navina Ramesh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39806/
> ---
> 
> (Updated Nov. 2, 2015, 9:56 p.m.)
> 
> 
> Review request for samza, Chris Riccomini, Jake Maes, Jagadish Venkatraman, 
> and Yi Pan (Data Infrastructure).
> 
> 
> Bugs: SAMZA-798
> https://issues.apache.org/jira/browse/SAMZA-798
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Adding interfaces for CheckpointManager, CheckpointManagerFactory and moving 
> Checkpoint to api
> 
> 
> Adding KafkaCheckpointLogKey, KafkaCheckpointManager and 
> KafkaCheckpointManagerFactory back from 0.9.1
> 
> 
> Changed SamzaContainer and OffsetManager
> 
> 
> Removed checkpointmanager in JC and modified TaskModel to remove 
> offsetMapping. Container will continue to use offsetmanager for fetching 
> offsets
> 
> 
> Fixed OffsetManager bugs
> 
> 
> Got rid of all compile errors during build with -x test
> 
> 
> Fixing Jackson object mapper for TaskModel
> 
> 
> Commented tests in checkpoint manager and fixed other failing tests
> 
> 
> Refactored KCM and moved generic functions like createTopic & validateTopic 
> to kafkaUtil.scala
> 
> 
> KCM unit tests work
> 
> 
> Got rid of old migration code and its test. Got rid of redundant KCM
> 
> 
> Commented out migration related tests in jobrunner
> 
> 
> Moved migration code from old.checkpoint package
> 
> 
> Fixed 1 migration test
> 
> 
> Fixed checkpoint migration and its unit tests
> 
> 
> Removed migration related tests from TestKafkaCheckpointManager
> 
> 
> Removed some commented lines and fixed a test in TestJobCoordinator
> 
> 
> Deleted CheckpointManager and SetCheckpoint
> 
> 
> Diffs
> -
> 
>   docs/learn/documentation/versioned/jobs/configuration-table.html 
> 4adac09305cbdb07b0d2cd9f8b189df1c290 
>   samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 
> PRE-CREATION 
>   
> samza-api/src/main/java/org/apache/samza/checkpoint/CheckpointManagerFactory.java
>  PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/checkpoint/Checkpoint.java  
>   samza-core/src/main/java/org/apache/samza/checkpoint/CheckpointManager.java 
> 0185751c28979e50b1bddc28c90339defd94200b 
>   
> samza-core/src/main/java/org/apache/samza/coordinator/stream/messages/SetCheckpoint.java
>  21afa8569801150e81b4c14ee21a9077dfa1895f 
>   samza-core/src/main/java/org/apache/samza/job/model/TaskModel.java 
> e00c49d5255c0af6d44e251aed4e8360cd3026c5 
>   
> samza-core/src/main/java/org/apache/samza/serializers/model/JsonTaskModelMixIn.java
>  172358a5428c9789e0883fc0e5ad3e5c3398478a 
>   samza-core/src/main/scala/org/apache/samza/checkpoint/CheckpointTool.scala 
> 2e3aeb8fd5a86aa39464adff9e75aca96622ebad 
>   samza-core/src/main/scala/org/apache/samza/checkpoint/OffsetManager.scala 
> 1464acc7ec6592a21c3cdf96f34847e094e9e5e3 
>   
> samza-core/src/main/scala/org/apache/samza/checkpoint/file/FileSystemCheckpointManager.scala
>  PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
> 0b73403018b895879ed2c0538a5cd495813d2eae 
>   samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala 
> 03299cb7cb93d43165a74206113497462d8119e9 
>   
> samza-core/src/main/scala/org/apache/samza/migration/JobRunnerMigration.scala 
> 374e27e8233a27132019d429f6fa1f131db3fe15 
>   
> samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamWrappedConsumer.java
>  dd04d28e54e7afe0cc6d6c2aa508911a14e668bf 
>   
> samza-core/src/test/java/org/apache/samza/serializers/model/TestSamzaObjectMapper.java
>  ad1fbc597802078c1a1b7d8f1dbafbd5adf610ae 
>   
> samza-core/src/test/scala/org/apache/samza/checkpoint/TestCheckpointTool.scala
>  00b89773ad00b8f445bb1320121ab8af56870327 
>   
> samza-core/s

Re: Review Request 35397: Fix SAMZA-697

[Boris Shkolnik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35397/#review88455
---


This is partial review (I didn't go thru the test).


samza-core/src/main/java/org/apache/samza/task/TaskClassLoader.java (line 50)
https://reviews.apache.org/r/35397/#comment140992

Do we have test for this case?



samza-core/src/main/java/org/apache/samza/task/TaskClassLoader.java (line 63)
https://reviews.apache.org/r/35397/#comment140989

Do we need a LOG.warn in case the file doesn't exist.



samza-core/src/main/java/org/apache/samza/task/TaskClassLoader.java (line 72)
https://reviews.apache.org/r/35397/#comment140993

nit. may be if (blacklistClassnames == null) return false.



samza-core/src/main/java/org/apache/samza/task/TaskClassLoader.java (line 103)
https://reviews.apache.org/r/35397/#comment140995

should we have little bit more validation here (checking for empty strings 
for example)



samza-core/src/main/java/org/apache/samza/task/TaskClassLoader.java (line 109)
https://reviews.apache.org/r/35397/#comment140996

nit. should be checked at the beginning of the method.



samza-core/src/main/java/org/apache/samza/task/TaskClassLoader.java (line 129)
https://reviews.apache.org/r/35397/#comment140997

nit. 'blacklisted'



samza-core/src/main/java/org/apache/samza/task/TaskClassLoader.java (line 140)
https://reviews.apache.org/r/35397/#comment140998

else log an error?



samza-core/src/main/java/org/apache/samza/task/TaskClassLoader.java (line 146)
https://reviews.apache.org/r/35397/#comment141000

do we need to override it if it is the same a default implementation.



samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala (line 
435)
https://reviews.apache.org/r/35397/#comment141001

Do we want to create this taskClassLoader if the taskClassLoaderPath is not 
configured? If this is the case we are creating classLoader with 'null' list of 
URLs. Is it safe?


This

- Boris Shkolnik


On June 18, 2015, 6:42 p.m., Guozhang Wang wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/35397/
 ---
 
 (Updated June 18, 2015, 6:42 p.m.)
 
 
 Review request for samza.
 
 
 Bugs: SAMZA-697
 https://issues.apache.org/jira/browse/SAMZA-697
 
 
 Repository: samza
 
 
 Description
 ---
 
 Address Yan's comments
 
 
 Diffs
 -
 
   checkstyle/import-control.xml 3374f0c432e61ac4cda275377604cfd481f0cddf 
   docs/learn/documentation/versioned/jobs/configuration-table.html 
 405e2cea4fd1d037cc26b3537f6bb406eded202b 
   samza-core/src/main/java/org/apache/samza/task/TaskClassLoader.java 
 PRE-CREATION 
   samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala 
 0b3a235b5ab1d6bd60669bfe6023f6b0b4e943d3 
   samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
 cbacd183420e9d1d72b05693b55a8f0a62d59fc5 
   samza-core/src/main/scala/org/apache/samza/container/TaskInstance.scala 
 c5a5ea5dea9a950fc741625238f5bf8b1f362180 
   samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 
 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 
   
 samza-core/src/main/scala/org/apache/samza/job/local/ProcessJobFactory.scala 
 4fac154709d72ab594485dad93c912b55fb1617e 
   samza-core/src/test/java/org/apache/samza/task/TestTaskClassLoader.java 
 PRE-CREATION 
   
 samza-core/src/test/scala/org/apache/samza/container/TestSamzaContainer.scala 
 9fb1aa98fcd14397e8a4cb00c67537482e95fa53 
   samza-core/src/test/scala/org/apache/samza/container/TestTaskInstance.scala 
 7caad28c9298485753ab861da76793cf925953ed 
 
 Diff: https://reviews.apache.org/r/35397/diff/
 
 
 Testing
 ---
 
 unit tests
 
 
 Thanks,
 
 Guozhang Wang