Re: Producer stopped during leader switch

[Jagadish Venkatraman]

Hey David,

Apologies for the delayed response. There are a bunch of producer *send*
issues that were fixed in samza-11 (SAMZA-1028, SAMZA-1003). With Samza-11,
there's improved resiliency and if retries fail, the exception will be
propagated to your *StreamTask*, and will end up failing the container (if
you don't swallow it).

Thanks,
Jagadish



On Fri, Oct 28, 2016 at 10:12 AM, David Yu  wrote:

> Hi,
>
> We recently experienced a Kafka broker crash. When a new broker was brought
> up, we started seeing the following errors in Samza (0.10.1):
>
> WARN  o.a.k.c.producer.internals.Sender - Got error produce response with
> correlation id 5199601 on topic-partition
> session_key_partitioned_sessions-39, retrying (2147483646 attempts left).
> Error: NOT_LEADER_FOR_PARTITION
>
> Is the Producer not able to detect the new broker/leader for that
> partition?
>
> Thanks,
> David
>



-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University

[GitHub] samza pull request #17: SAMZA-1033: Remove import-control from checkstyle

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/17


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Review Request 53326: SAMZA-1045 Move classes from samza-operator/api into samza-api.

[Fred Ji]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53326/#review154347
---




build.gradle (line 126)


I would recommend keeping the checkstyle on and fixing the incompatibility 
issue so that checkstyle plugin can caputre the style issue in the build 
process. If we disable it, some of the style issues (although minor) may be 
hidden and be merged into master.


- Fred Ji


On Oct. 31, 2016, 11:27 p.m., Jagadish Venkatraman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53326/
> ---
> 
> (Updated Oct. 31, 2016, 11:27 p.m.)
> 
> 
> Review request for samza, Jake Maes, Yi Pan (Data Infrastructure), Prateek 
> Maheshwari, and Xinyu Liu.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Refactor API classes from Samza-Operator/operator into samza-api. 
> 
> - Programmers write code against classes on samza-api, and it would be nice 
> to not have a dependency on samza-operator (when defining the transform logic 
> for the operator). 
> - For split-deployment to be effective, it'd be nice to independently evolve 
> samza-operator components.
> 
> Note: There are a couple of bugs that I found in the operator logic 
> (SAMZA-914/915). They are intentionally kept out of the scope this RB
> 
> 
> Diffs
> -
> 
>   build.gradle aeefd1cfd049efa37d20e12e34232600ae6bd127 
>   checkstyle/checkstyle.xml 770b5e7f7a091198bbf53b3908600f9ac0caa4c7 
>   gradle.properties 16e1f5d43f0415c511689480f8cb67d84e2baadf 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/MessageStream.java
>  b5e1028662a67e6248722ef7c842c565fef7a458 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/MessageStreams.java
>  59dd91c2f537828640b1ede6b6366e37f6b5c63b 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/TriggerBuilder.java
>  fc3ea37563ded2fcceeb332b0edfef45208bb9bb 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/WindowState.java 
> 402cc42f63bad0c8745c6a480b03769f92053622 
>   samza-operator/src/main/java/org/apache/samza/operators/api/Windows.java 
> e557b34c25ff425385aedae24bad6bb89ee03a30 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/IncomingSystemMessage.java
>  ba74618961d3ea53ad455de3263c59c0ca2a0fa7 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/InputSystemMessage.java
>  c7860254e38a897e42e00b60111526db0f7510ea 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/LongOffset.java
>  f059b337e299aab07c85af9866371c87fcf59786 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/Message.java 
> 9b53b4582ed12fa7d9948ff756ed9d0a4b38e280 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/Offset.java 
> 0fac2c0123b133566630bdf11e9b1c00a207bd2e 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/internal/Operators.java
>  e9bfe0b38c1048ecc7fe634ed7b0d71fee6f2ac6 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/internal/Trigger.java
>  33a0134e88b966d8f08658f18ca73564702305fa 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/internal/WindowFn.java
>  1fd88e76119960d1e6b83189bb571e64ac329489 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/internal/WindowOutput.java
>  e202c20cb0af6e53268e0c80820bfb1d7fc80892 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/ChainedOperators.java
>  59de16bd11fee34b23baac14d1d32cf9df555058 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorFactory.java
>  f16cbc6bf405851488c22b943a2256864b7a9f07 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorImpl.java
>  3ca8bdeb7d73c72f469fa33a152ed7518f9e8ac4 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/ProcessorContext.java
>  5a375bc7bb3b00fd252c6b1bbe211de8feddd942 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/SimpleOperatorImpl.java
>  b29d9c8c412c70820ac8355137df447d373af1f6 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/SinkOperatorImpl.java
>  5d25cfa106424f45660a45434c4abffd242b8672 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/StateStoreImpl.java
>  f573fd04e1aaf827d2ca52a5edca8f7060723aa0 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/join/PartialJoinOpImpl.java
>  bbe08a404b306d07d93c75ce21d00aebc38b42c3 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/window/SessionWindowImpl.java
>  59e2dec217bb7999ea3f42f106b55dd5563addcc 
>   
> samza-operator/src/main/java/org/apache/samza/task/StreamOperatorAdaptorTask.java
> 

Re: Review Request 53326: SAMZA-1045 Move classes from samza-operator/api into samza-api.

[Xinyu Liu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53326/#review154345
---


Ship it!




Discussed offline and we are going to move files first. Please take a look at 
my comments after it. Thanks.

- Xinyu Liu


On Oct. 31, 2016, 11:27 p.m., Jagadish Venkatraman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53326/
> ---
> 
> (Updated Oct. 31, 2016, 11:27 p.m.)
> 
> 
> Review request for samza, Jake Maes, Yi Pan (Data Infrastructure), Prateek 
> Maheshwari, and Xinyu Liu.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Refactor API classes from Samza-Operator/operator into samza-api. 
> 
> - Programmers write code against classes on samza-api, and it would be nice 
> to not have a dependency on samza-operator (when defining the transform logic 
> for the operator). 
> - For split-deployment to be effective, it'd be nice to independently evolve 
> samza-operator components.
> 
> Note: There are a couple of bugs that I found in the operator logic 
> (SAMZA-914/915). They are intentionally kept out of the scope this RB
> 
> 
> Diffs
> -
> 
>   build.gradle aeefd1cfd049efa37d20e12e34232600ae6bd127 
>   checkstyle/checkstyle.xml 770b5e7f7a091198bbf53b3908600f9ac0caa4c7 
>   gradle.properties 16e1f5d43f0415c511689480f8cb67d84e2baadf 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/MessageStream.java
>  b5e1028662a67e6248722ef7c842c565fef7a458 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/MessageStreams.java
>  59dd91c2f537828640b1ede6b6366e37f6b5c63b 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/TriggerBuilder.java
>  fc3ea37563ded2fcceeb332b0edfef45208bb9bb 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/WindowState.java 
> 402cc42f63bad0c8745c6a480b03769f92053622 
>   samza-operator/src/main/java/org/apache/samza/operators/api/Windows.java 
> e557b34c25ff425385aedae24bad6bb89ee03a30 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/IncomingSystemMessage.java
>  ba74618961d3ea53ad455de3263c59c0ca2a0fa7 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/InputSystemMessage.java
>  c7860254e38a897e42e00b60111526db0f7510ea 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/LongOffset.java
>  f059b337e299aab07c85af9866371c87fcf59786 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/Message.java 
> 9b53b4582ed12fa7d9948ff756ed9d0a4b38e280 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/Offset.java 
> 0fac2c0123b133566630bdf11e9b1c00a207bd2e 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/internal/Operators.java
>  e9bfe0b38c1048ecc7fe634ed7b0d71fee6f2ac6 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/internal/Trigger.java
>  33a0134e88b966d8f08658f18ca73564702305fa 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/internal/WindowFn.java
>  1fd88e76119960d1e6b83189bb571e64ac329489 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/internal/WindowOutput.java
>  e202c20cb0af6e53268e0c80820bfb1d7fc80892 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/ChainedOperators.java
>  59de16bd11fee34b23baac14d1d32cf9df555058 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorFactory.java
>  f16cbc6bf405851488c22b943a2256864b7a9f07 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorImpl.java
>  3ca8bdeb7d73c72f469fa33a152ed7518f9e8ac4 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/ProcessorContext.java
>  5a375bc7bb3b00fd252c6b1bbe211de8feddd942 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/SimpleOperatorImpl.java
>  b29d9c8c412c70820ac8355137df447d373af1f6 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/SinkOperatorImpl.java
>  5d25cfa106424f45660a45434c4abffd242b8672 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/StateStoreImpl.java
>  f573fd04e1aaf827d2ca52a5edca8f7060723aa0 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/join/PartialJoinOpImpl.java
>  bbe08a404b306d07d93c75ce21d00aebc38b42c3 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/window/SessionWindowImpl.java
>  59e2dec217bb7999ea3f42f106b55dd5563addcc 
>   
> samza-operator/src/main/java/org/apache/samza/task/StreamOperatorAdaptorTask.java
>  e340fe82109f8b59796fa2042dec2614671136b6 
>   samza-operator/src/main/java/org/apache/samza/task/StreamOperatorTask.java 
> cfdb694e337a663ce6fa179f0f3e91b05da51fa8 
>   
> 

Re: Review Request 53326: SAMZA-1045 Move classes from samza-operator/api into samza-api.

[Xinyu Liu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53326/#review154337
---




samza-operator/src/main/java/org/apache/samza/operators/api/MessageStream.java 
(line 56)


We shouldn't expose this function as a publich interface.



samza-operator/src/main/java/org/apache/samza/operators/api/WindowState.java 
(line 27)


Do we need to expose this to the user?



samza-operator/src/main/java/org/apache/samza/operators/api/data/InputSystemMessage.java
 (line 28)


I don't see the value of this interface since it's not used anywhere and 
IncomingSystemMessage is the one used in all operator tasks.


- Xinyu Liu


On Oct. 31, 2016, 11:27 p.m., Jagadish Venkatraman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53326/
> ---
> 
> (Updated Oct. 31, 2016, 11:27 p.m.)
> 
> 
> Review request for samza, Jake Maes, Yi Pan (Data Infrastructure), Prateek 
> Maheshwari, and Xinyu Liu.
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Refactor API classes from Samza-Operator/operator into samza-api. 
> 
> - Programmers write code against classes on samza-api, and it would be nice 
> to not have a dependency on samza-operator (when defining the transform logic 
> for the operator). 
> - For split-deployment to be effective, it'd be nice to independently evolve 
> samza-operator components.
> 
> Note: There are a couple of bugs that I found in the operator logic 
> (SAMZA-914/915). They are intentionally kept out of the scope this RB
> 
> 
> Diffs
> -
> 
>   build.gradle aeefd1cfd049efa37d20e12e34232600ae6bd127 
>   checkstyle/checkstyle.xml 770b5e7f7a091198bbf53b3908600f9ac0caa4c7 
>   gradle.properties 16e1f5d43f0415c511689480f8cb67d84e2baadf 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/MessageStream.java
>  b5e1028662a67e6248722ef7c842c565fef7a458 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/MessageStreams.java
>  59dd91c2f537828640b1ede6b6366e37f6b5c63b 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/TriggerBuilder.java
>  fc3ea37563ded2fcceeb332b0edfef45208bb9bb 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/WindowState.java 
> 402cc42f63bad0c8745c6a480b03769f92053622 
>   samza-operator/src/main/java/org/apache/samza/operators/api/Windows.java 
> e557b34c25ff425385aedae24bad6bb89ee03a30 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/IncomingSystemMessage.java
>  ba74618961d3ea53ad455de3263c59c0ca2a0fa7 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/InputSystemMessage.java
>  c7860254e38a897e42e00b60111526db0f7510ea 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/LongOffset.java
>  f059b337e299aab07c85af9866371c87fcf59786 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/Message.java 
> 9b53b4582ed12fa7d9948ff756ed9d0a4b38e280 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/data/Offset.java 
> 0fac2c0123b133566630bdf11e9b1c00a207bd2e 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/internal/Operators.java
>  e9bfe0b38c1048ecc7fe634ed7b0d71fee6f2ac6 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/internal/Trigger.java
>  33a0134e88b966d8f08658f18ca73564702305fa 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/internal/WindowFn.java
>  1fd88e76119960d1e6b83189bb571e64ac329489 
>   
> samza-operator/src/main/java/org/apache/samza/operators/api/internal/WindowOutput.java
>  e202c20cb0af6e53268e0c80820bfb1d7fc80892 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/ChainedOperators.java
>  59de16bd11fee34b23baac14d1d32cf9df555058 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorFactory.java
>  f16cbc6bf405851488c22b943a2256864b7a9f07 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorImpl.java
>  3ca8bdeb7d73c72f469fa33a152ed7518f9e8ac4 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/ProcessorContext.java
>  5a375bc7bb3b00fd252c6b1bbe211de8feddd942 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/SimpleOperatorImpl.java
>  b29d9c8c412c70820ac8355137df447d373af1f6 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/SinkOperatorImpl.java
>  5d25cfa106424f45660a45434c4abffd242b8672 
>   
> samza-operator/src/main/java/org/apache/samza/operators/impl/StateStoreImpl.java
>  f573fd04e1aaf827d2ca52a5edca8f7060723aa0 
>   
> 

Re: Review Request 53326: SAMZA-1045 Move classes from samza-operator/api into samza-api.

[Jagadish Venkatraman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53326/
---

(Updated Oct. 31, 2016, 11:27 p.m.)


Review request for samza, Jake Maes, Yi Pan (Data Infrastructure), Prateek 
Maheshwari, and Xinyu Liu.


Repository: samza


Description (updated)
---

Refactor API classes from Samza-Operator/operator into samza-api. 

- Programmers write code against classes on samza-api, and it would be nice to 
not have a dependency on samza-operator (when defining the transform logic for 
the operator). 
- For split-deployment to be effective, it'd be nice to independently evolve 
samza-operator components.

Note: There are a couple of bugs that I found in the operator logic 
(SAMZA-914/915). They are intentionally kept out of the scope this RB


Diffs
-

  build.gradle aeefd1cfd049efa37d20e12e34232600ae6bd127 
  checkstyle/checkstyle.xml 770b5e7f7a091198bbf53b3908600f9ac0caa4c7 
  gradle.properties 16e1f5d43f0415c511689480f8cb67d84e2baadf 
  
samza-operator/src/main/java/org/apache/samza/operators/api/MessageStream.java 
b5e1028662a67e6248722ef7c842c565fef7a458 
  
samza-operator/src/main/java/org/apache/samza/operators/api/MessageStreams.java 
59dd91c2f537828640b1ede6b6366e37f6b5c63b 
  
samza-operator/src/main/java/org/apache/samza/operators/api/TriggerBuilder.java 
fc3ea37563ded2fcceeb332b0edfef45208bb9bb 
  samza-operator/src/main/java/org/apache/samza/operators/api/WindowState.java 
402cc42f63bad0c8745c6a480b03769f92053622 
  samza-operator/src/main/java/org/apache/samza/operators/api/Windows.java 
e557b34c25ff425385aedae24bad6bb89ee03a30 
  
samza-operator/src/main/java/org/apache/samza/operators/api/data/IncomingSystemMessage.java
 ba74618961d3ea53ad455de3263c59c0ca2a0fa7 
  
samza-operator/src/main/java/org/apache/samza/operators/api/data/InputSystemMessage.java
 c7860254e38a897e42e00b60111526db0f7510ea 
  
samza-operator/src/main/java/org/apache/samza/operators/api/data/LongOffset.java
 f059b337e299aab07c85af9866371c87fcf59786 
  samza-operator/src/main/java/org/apache/samza/operators/api/data/Message.java 
9b53b4582ed12fa7d9948ff756ed9d0a4b38e280 
  samza-operator/src/main/java/org/apache/samza/operators/api/data/Offset.java 
0fac2c0123b133566630bdf11e9b1c00a207bd2e 
  
samza-operator/src/main/java/org/apache/samza/operators/api/internal/Operators.java
 e9bfe0b38c1048ecc7fe634ed7b0d71fee6f2ac6 
  
samza-operator/src/main/java/org/apache/samza/operators/api/internal/Trigger.java
 33a0134e88b966d8f08658f18ca73564702305fa 
  
samza-operator/src/main/java/org/apache/samza/operators/api/internal/WindowFn.java
 1fd88e76119960d1e6b83189bb571e64ac329489 
  
samza-operator/src/main/java/org/apache/samza/operators/api/internal/WindowOutput.java
 e202c20cb0af6e53268e0c80820bfb1d7fc80892 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/ChainedOperators.java
 59de16bd11fee34b23baac14d1d32cf9df555058 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorFactory.java
 f16cbc6bf405851488c22b943a2256864b7a9f07 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorImpl.java 
3ca8bdeb7d73c72f469fa33a152ed7518f9e8ac4 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/ProcessorContext.java
 5a375bc7bb3b00fd252c6b1bbe211de8feddd942 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/SimpleOperatorImpl.java
 b29d9c8c412c70820ac8355137df447d373af1f6 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/SinkOperatorImpl.java
 5d25cfa106424f45660a45434c4abffd242b8672 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/StateStoreImpl.java
 f573fd04e1aaf827d2ca52a5edca8f7060723aa0 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/join/PartialJoinOpImpl.java
 bbe08a404b306d07d93c75ce21d00aebc38b42c3 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/window/SessionWindowImpl.java
 59e2dec217bb7999ea3f42f106b55dd5563addcc 
  
samza-operator/src/main/java/org/apache/samza/task/StreamOperatorAdaptorTask.java
 e340fe82109f8b59796fa2042dec2614671136b6 
  samza-operator/src/main/java/org/apache/samza/task/StreamOperatorTask.java 
cfdb694e337a663ce6fa179f0f3e91b05da51fa8 
  samza-operator/src/test/java/org/apache/samza/operators/api/TestMessage.java 
0f00fdbe7e2a2ddb4a4beb3cc77290221688cc97 
  
samza-operator/src/test/java/org/apache/samza/operators/api/TestMessageStream.java
 9f9ad6bde2471f6c29e908cc8d3041e5e0deaf98 
  
samza-operator/src/test/java/org/apache/samza/operators/api/TestMessageStreams.java
 e6aa6923a9536651bc976f218bff94bc80e85e57 
  
samza-operator/src/test/java/org/apache/samza/operators/api/TestOutputMessage.java
 225e77f04ede6803977559879ba30b976a6a962a 
  
samza-operator/src/test/java/org/apache/samza/operators/api/TestTriggerBuilder.java
 8faa92c08814cf19ef6e3e72dc21797db4ce11a1 
  

Re: Review Request 52492: Adding monitor to clean up stale local stores of jobs/tasks.

[Shanthoosh Venkataraman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52492/
---

(Updated Oct. 31, 2016, 11:04 p.m.)


Review request for samza.


Repository: samza


Description
---

This patch contains the samza-rest monitor that periodically cleans up the 
stale local stores of dead jobs/tasks. It performs the store deletion in two 
phases. Initially it deletes the offset file in the local task stores if the 
following condition is true. ((jobIsNotRunning || preferedHost != nmHost) && 
offsetFilelastModifiedTime is greater than deleteRetention). During the 
subsequent run, it deletes the local task stores if it does not contain offset 
file. Please refer to the design doc of SAMZA-656 
(https://issues.apache.org/jira/secure/attachment/12828083/DESIGN-SAMZA-656.pdf)
 for more details.


Diffs (updated)
-

  build.gradle 2bea27b75288d3103178bc3762b9556f6e69cdd1 
  samza-rest/src/main/java/org/apache/samza/monitor/JobsClient.java 
PRE-CREATION 
  samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitor.java 
PRE-CREATION 
  
samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitorConfig.java 
PRE-CREATION 
  
samza-rest/src/main/java/org/apache/samza/monitor/LocalStoreMonitorFactory.java 
PRE-CREATION 
  
samza-rest/src/main/java/org/apache/samza/rest/resources/ResourceConstants.java 
PRE-CREATION 
  samza-rest/src/test/java/org/apache/samza/monitor/TestLocalStoreMonitor.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/52492/diff/


Testing
---

Unit testing and manual testing are done to verify the functionality.


Thanks,

Shanthoosh Venkataraman

Review Request 53326: SAMZA-1045 Move classes from samza-operator/api into samza-api.

[Jagadish Venkatraman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53326/
---

Review request for samza, Jake Maes, Yi Pan (Data Infrastructure), Prateek 
Maheshwari, and Xinyu Liu.


Summary (updated)
-

SAMZA-1045 Move classes from samza-operator/api into samza-api.


Repository: samza


Description (updated)
---

Refactor API classes from Samza-Operator/operator into samza-api. Programmers 
write code against classes on samza-api, and it would be nice to not have a 
dependency on samza-operator (when defining the transform logic for the 
operator).

Note: There are a couple of bugs that I found in the operator logic 
(SAMZA-914/915). They are intentionally kept out of the scope this RB


Diffs (updated)
-

  build.gradle aeefd1cfd049efa37d20e12e34232600ae6bd127 
  checkstyle/checkstyle.xml 770b5e7f7a091198bbf53b3908600f9ac0caa4c7 
  gradle.properties 16e1f5d43f0415c511689480f8cb67d84e2baadf 
  
samza-operator/src/main/java/org/apache/samza/operators/api/MessageStream.java 
b5e1028662a67e6248722ef7c842c565fef7a458 
  
samza-operator/src/main/java/org/apache/samza/operators/api/MessageStreams.java 
59dd91c2f537828640b1ede6b6366e37f6b5c63b 
  
samza-operator/src/main/java/org/apache/samza/operators/api/TriggerBuilder.java 
fc3ea37563ded2fcceeb332b0edfef45208bb9bb 
  samza-operator/src/main/java/org/apache/samza/operators/api/WindowState.java 
402cc42f63bad0c8745c6a480b03769f92053622 
  samza-operator/src/main/java/org/apache/samza/operators/api/Windows.java 
e557b34c25ff425385aedae24bad6bb89ee03a30 
  
samza-operator/src/main/java/org/apache/samza/operators/api/data/IncomingSystemMessage.java
 ba74618961d3ea53ad455de3263c59c0ca2a0fa7 
  
samza-operator/src/main/java/org/apache/samza/operators/api/data/InputSystemMessage.java
 c7860254e38a897e42e00b60111526db0f7510ea 
  
samza-operator/src/main/java/org/apache/samza/operators/api/data/LongOffset.java
 f059b337e299aab07c85af9866371c87fcf59786 
  samza-operator/src/main/java/org/apache/samza/operators/api/data/Message.java 
9b53b4582ed12fa7d9948ff756ed9d0a4b38e280 
  samza-operator/src/main/java/org/apache/samza/operators/api/data/Offset.java 
0fac2c0123b133566630bdf11e9b1c00a207bd2e 
  
samza-operator/src/main/java/org/apache/samza/operators/api/internal/Operators.java
 e9bfe0b38c1048ecc7fe634ed7b0d71fee6f2ac6 
  
samza-operator/src/main/java/org/apache/samza/operators/api/internal/Trigger.java
 33a0134e88b966d8f08658f18ca73564702305fa 
  
samza-operator/src/main/java/org/apache/samza/operators/api/internal/WindowFn.java
 1fd88e76119960d1e6b83189bb571e64ac329489 
  
samza-operator/src/main/java/org/apache/samza/operators/api/internal/WindowOutput.java
 e202c20cb0af6e53268e0c80820bfb1d7fc80892 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/ChainedOperators.java
 59de16bd11fee34b23baac14d1d32cf9df555058 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorFactory.java
 f16cbc6bf405851488c22b943a2256864b7a9f07 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorImpl.java 
3ca8bdeb7d73c72f469fa33a152ed7518f9e8ac4 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/ProcessorContext.java
 5a375bc7bb3b00fd252c6b1bbe211de8feddd942 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/SimpleOperatorImpl.java
 b29d9c8c412c70820ac8355137df447d373af1f6 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/SinkOperatorImpl.java
 5d25cfa106424f45660a45434c4abffd242b8672 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/StateStoreImpl.java
 f573fd04e1aaf827d2ca52a5edca8f7060723aa0 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/join/PartialJoinOpImpl.java
 bbe08a404b306d07d93c75ce21d00aebc38b42c3 
  
samza-operator/src/main/java/org/apache/samza/operators/impl/window/SessionWindowImpl.java
 59e2dec217bb7999ea3f42f106b55dd5563addcc 
  
samza-operator/src/main/java/org/apache/samza/task/StreamOperatorAdaptorTask.java
 e340fe82109f8b59796fa2042dec2614671136b6 
  samza-operator/src/main/java/org/apache/samza/task/StreamOperatorTask.java 
cfdb694e337a663ce6fa179f0f3e91b05da51fa8 
  samza-operator/src/test/java/org/apache/samza/operators/api/TestMessage.java 
0f00fdbe7e2a2ddb4a4beb3cc77290221688cc97 
  
samza-operator/src/test/java/org/apache/samza/operators/api/TestMessageStream.java
 9f9ad6bde2471f6c29e908cc8d3041e5e0deaf98 
  
samza-operator/src/test/java/org/apache/samza/operators/api/TestMessageStreams.java
 e6aa6923a9536651bc976f218bff94bc80e85e57 
  
samza-operator/src/test/java/org/apache/samza/operators/api/TestOutputMessage.java
 225e77f04ede6803977559879ba30b976a6a962a 
  
samza-operator/src/test/java/org/apache/samza/operators/api/TestTriggerBuilder.java
 8faa92c08814cf19ef6e3e72dc21797db4ce11a1 
  samza-operator/src/test/java/org/apache/samza/operators/api/TestWindows.java 

Review Request 53297: Initial version of adding metrics into samza rest.

[Shanthoosh Venkataraman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53297/
---

Review request for samza.


Repository: samza


Description
---

This patch aims at enabling users to define custom reporters to send metrics 
from the monitors. Configurations required for the definition of the metrics 
reporters follows the same convention as of the samza jobs.


Diffs
-

  docs/learn/documentation/versioned/rest/monitors.md 
46678bbe5fed99f767c3324dc9578ee1a64cec66 
  samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 
e0468ee89c89fd720834461771ebb36475475bcb 
  
samza-core/src/main/scala/org/apache/samza/metrics/ContainerProcessManagerMetrics.scala
 f24beb1e099dd44b15b475e0a4a7f70560c6965e 
  samza-core/src/main/scala/org/apache/samza/util/MetricsReporterLoader.scala 
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 
c4836f202f7eda1d4e71eac94fd48e46207b0316 
  samza-rest/src/main/java/org/apache/samza/rest/SamzaRestConfig.java 
47b0663637f6db187d86961377ee3ee203b73fdb 
  samza-rest/src/main/java/org/apache/samza/rest/SamzaRestService.java 
2a3e83a24a5343bb53b93fc9d0a647c1b253714b 
  
samza-rest/src/test/java/org/apache/samza/rest/resources/TestSamzaRestService.java
 PRE-CREATION 
  
samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterMetrics.scala 
8a5b4aaea6e11a5af999f12d50e5b6135dbc70ca 

Diff: https://reviews.apache.org/r/53297/diff/


Testing
---

Unit tests are done to verify the intended functionality.


Thanks,

Shanthoosh Venkataraman

Re: Review Request 53163: SAMZA-901: SamzaAppState re-design for thread safety

[Jagadish Venkatraman]

> On Oct. 26, 2016, 2:05 a.m., Jagadish Venkatraman wrote:
> > samza-core/src/main/java/org/apache/samza/clustermanager/SamzaApplicationState.java,
> >  line 56
> > 
> >
> > Prefer `volatile` here.
> > 
> > There are 2 aspects to concurrency in Java:
> > 
> > 1. Atomicity:
> > In Java, reads and writes are atomic for references and for primitives 
> > (with the exception of double/long).
> > 
> > 2. Visibility:
> > Volatile guarantees visibility of updates, and defines a happens-before 
> > ordering between updates.
> > 
> > For this case, we can use a `volatile` instead of using a per-object 
> > lock. (Using a non-volatile variable would still provide guarantees on 
> > atomicity but not visibility. Using a lock, will provide you both - except 
> > that it appears to be an overkill for this particular scenario).
> 
> Branislav Cogic wrote:
> So basicly, all that it was needed regarding SamzaApplicationState thread 
> safety is to put locks or volatile on these three variables. You have stated 
> in the javadocs that the scope of this change is larger. Also, making a bunch 
> of accessor methods seems to be needless. Am I getting something wrong?

>> You have stated in the javadocs that the scope of this change is larger. 

The RB for the Job coordinator refactor has been through a lot of changes and 
churn. (So, it's possible that the TODO was not accurate). The original intent 
then, was to move the `JobModelManager` out of `SamzaAppState`. 

>> making a bunch of accessor methods seems to be needless 

Yes, Agreed (on the accessors). 

I think for now, we can simply stick to making the 4 variables `volatile`. 

We can certainly, revisit this when we have integration with other resource 
managers like Mesos.


- Jagadish


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53163/#review153840
---


On Oct. 27, 2016, 9:25 a.m., Branislav Cogic wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53163/
> ---
> 
> (Updated Oct. 27, 2016, 9:25 a.m.)
> 
> 
> Review request for samza and Jagadish Venkatraman.
> 
> 
> Bugs: SAMZA-901
> https://issues.apache.org/jira/browse/SAMZA-901
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> SAMZA-901 SamzaAppState re-design for thread safety
> 
> 
> Diffs
> -
> 
>   
> samza-core/src/main/java/org/apache/samza/clustermanager/AbstractContainerAllocator.java
>  d47f217 
>   
> samza-core/src/main/java/org/apache/samza/clustermanager/ClusterBasedJobCoordinator.java
>  d0d4e34 
>   
> samza-core/src/main/java/org/apache/samza/clustermanager/ContainerProcessManager.java
>  b4309d9 
>   
> samza-core/src/main/java/org/apache/samza/clustermanager/HostAwareContainerAllocator.java
>  da73049 
>   
> samza-core/src/main/java/org/apache/samza/clustermanager/SamzaApplicationState.java
>  cf91044 
>   
> samza-core/src/main/scala/org/apache/samza/metrics/ContainerProcessManagerMetrics.scala
>  f24beb1 
>   
> samza-core/src/test/java/org/apache/samza/clustermanager/TestContainerAllocator.java
>  5351bc3 
>   
> samza-core/src/test/java/org/apache/samza/clustermanager/TestContainerProcessManager.java
>  0d61814 
>   
> samza-core/src/test/java/org/apache/samza/clustermanager/TestHostAwareContainerAllocator.java
>  b6651f2 
>   
> samza-yarn/src/main/java/org/apache/samza/job/yarn/YarnResourceManagerFactory.java
>  988a8e8 
>   samza-yarn/src/main/resources/scalate/WEB-INF/views/index.scaml 93176ff 
>   
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterMetrics.scala
>  8a5b4aa 
>   
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaYarnAppMasterLifecycle.scala
>  c9c1e18 
>   
> samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaYarnAppMasterService.scala
>  5f2bfc5 
>   
> samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterRestServlet.scala
>  cdd389c 
> 
> Diff: https://reviews.apache.org/r/53163/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew clean build
> ./gradlew checkstyleMain checkstyleTest
> 
> Checked that app state is shown correctly on web servlet index page.
> 
> 
> Thanks,
> 
> Branislav Cogic
> 
>