Re: Review Request 33419: SAMZA-625: Provide tool to consume changelog and materialize a state store

2015-07-15 Thread Navina Ramesh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33419/#review91865
---

Ship it!


Ship It!

- Navina Ramesh


On July 16, 2015, 12:56 a.m., Yan Fang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33419/
> ---
> 
> (Updated July 16, 2015, 12:56 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-625
> https://issues.apache.org/jira/browse/SAMZA-625
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Implemented in Java.
> 
> * modified build.gradle to have the gradle compile scala first. Because some 
> jave code has dependencies to Scala code
> * change the state store name by removing the space ( in TaskManager )
> * add scala java conversion method in Util because some classes only accept 
> scala map
> * add java version of some configs 
> * remove duplicated config in samza-log4j
> * add StorageRevoery class, which does most of the recoverying job. The logic 
> mimics what happens in SamzaContainer.
> * add StateStorageTool, for the commandline usage
> * unit tests
> * docs
> 
> 
> Diffs
> -
> 
>   checkstyle/import-control.xml 3374f0c 
>   docs/learn/documentation/versioned/container/state-management.md 79067bb 
>   samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java 
> PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java 
> PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/storage/StateStorageTool.java 
> PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java 
> PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 
> aeba61a 
>   samza-core/src/main/scala/org/apache/samza/util/Util.scala 2feb65b 
>   samza-core/src/test/java/org/apache/samza/config/TestJavaStorageConfig.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/config/TestJavaSystemConfig.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/MockStorageEngine.java 
> PRE-CREATION 
>   
> samza-core/src/test/java/org/apache/samza/storage/MockStorageEngineFactory.java
>  PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/MockSystemConsumer.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/MockSystemFactory.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/TestStorageRecovery.java 
> PRE-CREATION 
>   samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 
> d5e24f2 
>   samza-shell/src/main/bash/state-storage-tool.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33419/diff/
> 
> 
> Testing
> ---
> 
> tested with multiple partitions and multiple stores recovery.
> 
> 
> Thanks,
> 
> Yan Fang
> 
>

Re: Review Request 33419: SAMZA-625: Provide tool to consume changelog and materialize a state store

2015-07-15 Thread Yan Fang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33419/
---

(Updated July 16, 2015, 12:56 a.m.)


Review request for samza.


Changes
---

fix nits.
get rid of writing the latest offset to the file


Bugs: SAMZA-625
https://issues.apache.org/jira/browse/SAMZA-625


Repository: samza


Description
---

Implemented in Java.

* modified build.gradle to have the gradle compile scala first. Because some 
jave code has dependencies to Scala code
* change the state store name by removing the space ( in TaskManager )
* add scala java conversion method in Util because some classes only accept 
scala map
* add java version of some configs 
* remove duplicated config in samza-log4j
* add StorageRevoery class, which does most of the recoverying job. The logic 
mimics what happens in SamzaContainer.
* add StateStorageTool, for the commandline usage
* unit tests
* docs


Diffs (updated)
-

  checkstyle/import-control.xml 3374f0c 
  docs/learn/documentation/versioned/container/state-management.md 79067bb 
  samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/storage/StateStorageTool.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java 
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 
aeba61a 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 2feb65b 
  samza-core/src/test/java/org/apache/samza/config/TestJavaStorageConfig.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/config/TestJavaSystemConfig.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockStorageEngine.java 
PRE-CREATION 
  
samza-core/src/test/java/org/apache/samza/storage/MockStorageEngineFactory.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockSystemConsumer.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockSystemFactory.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/TestStorageRecovery.java 
PRE-CREATION 
  samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 
d5e24f2 
  samza-shell/src/main/bash/state-storage-tool.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/33419/diff/


Testing
---

tested with multiple partitions and multiple stores recovery.


Thanks,

Yan Fang

Re: Review Request 33419: SAMZA-625: Provide tool to consume changelog and materialize a state store

2015-07-15 Thread Navina Ramesh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33419/#review91785
---

Ship it!


Looks good to me. Very useful tool, Yan. Nice work!


samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java (line 
35)


nit: use static strings , esp. when it is repeated. (like ".factory")



samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java (line 42)


nit: SYSTEM_FACTORY can be defined in this class rather than from the scala 
class. Perhaps a future work of cleanup can do this as well.


- Navina Ramesh


On June 21, 2015, 6:10 a.m., Yan Fang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33419/
> ---
> 
> (Updated June 21, 2015, 6:10 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-625
> https://issues.apache.org/jira/browse/SAMZA-625
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Implemented in Java.
> 
> * modified build.gradle to have the gradle compile scala first. Because some 
> jave code has dependencies to Scala code
> * change the state store name by removing the space ( in TaskManager )
> * add scala java conversion method in Util because some classes only accept 
> scala map
> * add java version of some configs 
> * remove duplicated config in samza-log4j
> * add StorageRevoery class, which does most of the recoverying job. The logic 
> mimics what happens in SamzaContainer.
> * add StateStorageTool, for the commandline usage
> * unit tests
> * docs
> 
> 
> Diffs
> -
> 
>   checkstyle/import-control.xml 3374f0c 
>   docs/learn/documentation/versioned/container/state-management.md 79067bb 
>   samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java 
> PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java 
> PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/storage/StateStorageTool.java 
> PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java 
> PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 
> aeba61a 
>   samza-core/src/main/scala/org/apache/samza/util/Util.scala 2feb65b 
>   samza-core/src/test/java/org/apache/samza/config/TestJavaStorageConfig.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/config/TestJavaSystemConfig.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/MockStorageEngine.java 
> PRE-CREATION 
>   
> samza-core/src/test/java/org/apache/samza/storage/MockStorageEngineFactory.java
>  PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/MockSystemConsumer.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/MockSystemFactory.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/TestStorageRecovery.java 
> PRE-CREATION 
>   samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 
> d5e24f2 
>   samza-shell/src/main/bash/state-storage-tool.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33419/diff/
> 
> 
> Testing
> ---
> 
> tested with multiple partitions and multiple stores recovery.
> 
> 
> Thanks,
> 
> Yan Fang
> 
>

Re: Review Request 33419: SAMZA-625: Provide tool to consume changelog and materialize a state store

2015-06-20 Thread Yan Fang 


> On May 21, 2015, 6:45 p.m., Navina Ramesh wrote:
> > samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java, 
> > line 28
> > 
> >
> > Is "moving away from scala based configs" a motivation for all the 
> > "Java" prefixed classes?
> > 
> > If the plan is to refactor configs everywhere with Java based configs, 
> > then we should probably extend these classes from 
> > org.apache.samza.config.MapConfig. It provides some convenient methods 
> > because it is backed by a Map. Just curious if you explored that option.

This makes sense. I omitted this option. Thanks. Using MapConfig now. Also 
added unit test to verify that.


> On May 21, 2015, 6:45 p.m., Navina Ramesh wrote:
> > samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java, 
> > line 33
> > 
> >
> > Why is Config "protected" here and "private" in JavaStorageConfig?

fixed in above change.


> On May 21, 2015, 6:45 p.m., Navina Ramesh wrote:
> > samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala,
> >  line 39
> > 
> >
> > why not sanitize the entire file name, instead of just the task name?
> > Personally, I think we should follow a standard convention for 
> > sanitizing names, irrespective of whether it is storename or taskname. Just 
> > my two cents.

aggreed.


- Yan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33419/#review84612
---


On June 21, 2015, 6:10 a.m., Yan Fang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33419/
> ---
> 
> (Updated June 21, 2015, 6:10 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-625
> https://issues.apache.org/jira/browse/SAMZA-625
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Implemented in Java.
> 
> * modified build.gradle to have the gradle compile scala first. Because some 
> jave code has dependencies to Scala code
> * change the state store name by removing the space ( in TaskManager )
> * add scala java conversion method in Util because some classes only accept 
> scala map
> * add java version of some configs 
> * remove duplicated config in samza-log4j
> * add StorageRevoery class, which does most of the recoverying job. The logic 
> mimics what happens in SamzaContainer.
> * add StateStorageTool, for the commandline usage
> * unit tests
> * docs
> 
> 
> Diffs
> -
> 
>   checkstyle/import-control.xml 3374f0c 
>   docs/learn/documentation/versioned/container/state-management.md 79067bb 
>   samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java 
> PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java 
> PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/storage/StateStorageTool.java 
> PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java 
> PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 
> aeba61a 
>   samza-core/src/main/scala/org/apache/samza/util/Util.scala 2feb65b 
>   samza-core/src/test/java/org/apache/samza/config/TestJavaStorageConfig.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/config/TestJavaSystemConfig.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/MockStorageEngine.java 
> PRE-CREATION 
>   
> samza-core/src/test/java/org/apache/samza/storage/MockStorageEngineFactory.java
>  PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/MockSystemConsumer.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/MockSystemFactory.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/TestStorageRecovery.java 
> PRE-CREATION 
>   samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 
> d5e24f2 
>   samza-shell/src/main/bash/state-storage-tool.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33419/diff/
> 
> 
> Testing
> ---
> 
> tested with multiple partitions and multiple stores recovery.
> 
> 
> Thanks,
> 
> Yan Fang
> 
>

Re: Review Request 33419: SAMZA-625: Provide tool to consume changelog and materialize a state store

2015-06-20 Thread Yan Fang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33419/
---

(Updated June 21, 2015, 6:10 a.m.)


Review request for samza.


Changes
---

update based on Navina's comment.


Bugs: SAMZA-625
https://issues.apache.org/jira/browse/SAMZA-625


Repository: samza


Description
---

Implemented in Java.

* modified build.gradle to have the gradle compile scala first. Because some 
jave code has dependencies to Scala code
* change the state store name by removing the space ( in TaskManager )
* add scala java conversion method in Util because some classes only accept 
scala map
* add java version of some configs 
* remove duplicated config in samza-log4j
* add StorageRevoery class, which does most of the recoverying job. The logic 
mimics what happens in SamzaContainer.
* add StateStorageTool, for the commandline usage
* unit tests
* docs


Diffs (updated)
-

  checkstyle/import-control.xml 3374f0c 
  docs/learn/documentation/versioned/container/state-management.md 79067bb 
  samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/storage/StateStorageTool.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java 
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 
aeba61a 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 2feb65b 
  samza-core/src/test/java/org/apache/samza/config/TestJavaStorageConfig.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/config/TestJavaSystemConfig.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockStorageEngine.java 
PRE-CREATION 
  
samza-core/src/test/java/org/apache/samza/storage/MockStorageEngineFactory.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockSystemConsumer.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockSystemFactory.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/TestStorageRecovery.java 
PRE-CREATION 
  samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 
d5e24f2 
  samza-shell/src/main/bash/state-storage-tool.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/33419/diff/


Testing
---

tested with multiple partitions and multiple stores recovery.


Thanks,

Yan Fang

Re: Review Request 33419: SAMZA-625: Provide tool to consume changelog and materialize a state store

2015-05-21 Thread Navina Ramesh 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33419/#review84612
---


Apart from some questions, everything looks good to me!


docs/learn/documentation/versioned/container/state-management.md


Thanks for proactively adding docs :)



samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java


Is "moving away from scala based configs" a motivation for all the "Java" 
prefixed classes?

If the plan is to refactor configs everywhere with Java based configs, then 
we should probably extend these classes from org.apache.samza.config.MapConfig. 
It provides some convenient methods because it is backed by a Map. Just curious 
if you explored that option.



samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java


Why is Config "protected" here and "private" in JavaStorageConfig?



samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala


why not sanitize the entire file name, instead of just the task name?
Personally, I think we should follow a standard convention for sanitizing 
names, irrespective of whether it is storename or taskname. Just my two cents.


- Navina Ramesh


On May 20, 2015, 10:04 p.m., Yan Fang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33419/
> ---
> 
> (Updated May 20, 2015, 10:04 p.m.)
> 
> 
> Review request for samza.
> 
> 
> Bugs: SAMZA-625
> https://issues.apache.org/jira/browse/SAMZA-625
> 
> 
> Repository: samza
> 
> 
> Description
> ---
> 
> Implemented in Java.
> 
> * modified build.gradle to have the gradle compile scala first. Because some 
> jave code has dependencies to Scala code
> * change the state store name by removing the space ( in TaskManager )
> * add scala java conversion method in Util because some classes only accept 
> scala map
> * add java version of some configs 
> * remove duplicated config in samza-log4j
> * add StorageRevoery class, which does most of the recoverying job. The logic 
> mimics what happens in SamzaContainer.
> * add StateStorageTool, for the commandline usage
> * unit tests
> * docs
> 
> 
> Diffs
> -
> 
>   checkstyle/import-control.xml 5f8e103 
>   docs/learn/documentation/versioned/container/state-management.md 79067bb 
>   samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java 
> PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java 
> PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/storage/StateStorageTool.java 
> PRE-CREATION 
>   samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java 
> PRE-CREATION 
>   samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 
> aeba61a 
>   samza-core/src/main/scala/org/apache/samza/util/Util.scala 2feb65b 
>   samza-core/src/test/java/org/apache/samza/config/TestJavaSystemConfig.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/MockStorageEngine.java 
> PRE-CREATION 
>   
> samza-core/src/test/java/org/apache/samza/storage/MockStorageEngineFactory.java
>  PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/MockSystemConsumer.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/MockSystemFactory.java 
> PRE-CREATION 
>   samza-core/src/test/java/org/apache/samza/storage/TestStorageRecovery.java 
> PRE-CREATION 
>   samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 
> d5e24f2 
>   samza-shell/src/main/bash/state-storage-tool.sh PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33419/diff/
> 
> 
> Testing
> ---
> 
> tested with multiple partitions and multiple stores recovery.
> 
> 
> Thanks,
> 
> Yan Fang
> 
>

Re: Review Request 33419: SAMZA-625: Provide tool to consume changelog and materialize a state store

2015-05-20 Thread Yan Fang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33419/
---

(Updated May 20, 2015, 10:04 p.m.)


Review request for samza.


Changes
---

rebased. also fixed some formatting issues.


Bugs: SAMZA-625
https://issues.apache.org/jira/browse/SAMZA-625


Repository: samza


Description
---

Implemented in Java.

* modified build.gradle to have the gradle compile scala first. Because some 
jave code has dependencies to Scala code
* change the state store name by removing the space ( in TaskManager )
* add scala java conversion method in Util because some classes only accept 
scala map
* add java version of some configs 
* remove duplicated config in samza-log4j
* add StorageRevoery class, which does most of the recoverying job. The logic 
mimics what happens in SamzaContainer.
* add StateStorageTool, for the commandline usage
* unit tests
* docs


Diffs (updated)
-

  checkstyle/import-control.xml 5f8e103 
  docs/learn/documentation/versioned/container/state-management.md 79067bb 
  samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/storage/StateStorageTool.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java 
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 
aeba61a 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 2feb65b 
  samza-core/src/test/java/org/apache/samza/config/TestJavaSystemConfig.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockStorageEngine.java 
PRE-CREATION 
  
samza-core/src/test/java/org/apache/samza/storage/MockStorageEngineFactory.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockSystemConsumer.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockSystemFactory.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/TestStorageRecovery.java 
PRE-CREATION 
  samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 
d5e24f2 
  samza-shell/src/main/bash/state-storage-tool.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/33419/diff/


Testing
---

tested with multiple partitions and multiple stores recovery.


Thanks,

Yan Fang

Re: Review Request 33419: SAMZA-625: Provide tool to consume changelog and materialize a state store

2015-05-05 Thread Yan Fang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33419/
---

(Updated May 5, 2015, 9:56 p.m.)


Review request for samza.


Changes
---

rebased from master branch


Bugs: SAMZA-625
https://issues.apache.org/jira/browse/SAMZA-625


Repository: samza


Description
---

Implemented in Java.

* modified build.gradle to have the gradle compile scala first. Because some 
jave code has dependencies to Scala code
* change the state store name by removing the space ( in TaskManager )
* add scala java conversion method in Util because some classes only accept 
scala map
* add java version of some configs 
* remove duplicated config in samza-log4j
* add StorageRevoery class, which does most of the recoverying job. The logic 
mimics what happens in SamzaContainer.
* add StateStorageTool, for the commandline usage
* unit tests
* docs


Diffs (updated)
-

  docs/learn/documentation/versioned/container/state-management.md 79067bb 
  samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/config/JavaTaskConfig.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/storage/StateStorageTool.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java 
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 
f68a7fe 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 8a83566 
  samza-core/src/test/java/org/apache/samza/config/TestJavaSystemConfig.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockStorageEngine.java 
PRE-CREATION 
  
samza-core/src/test/java/org/apache/samza/storage/MockStorageEngineFactory.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockSystemConsumer.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockSystemFactory.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/TestStorageRecovery.java 
PRE-CREATION 
  samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 
d5e24f2 
  samza-log4j/src/test/java/org/apache/samza/config/TestLog4jSystemConfig.java 
6314a3e 
  samza-shell/src/main/bash/state-storage-tool.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/33419/diff/


Testing
---

tested with multiple partitions and multiple stores recovery.


Thanks,

Yan Fang

Review Request 33419: SAMZA-625: Provide tool to consume changelog and materialize a state store

2015-04-21 Thread Yan Fang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33419/
---

Review request for samza.


Bugs: SAMZA-625
https://issues.apache.org/jira/browse/SAMZA-625


Repository: samza


Description
---

Implemented in Java.

* modified build.gradle to have the gradle compile scala first. Because some 
jave code has dependencies to Scala code
* change the state store name by removing the space ( in TaskManager )
* add scala java conversion method in Util because some classes only accept 
scala map
* add java version of some configs 
* remove duplicated config in samza-log4j
* add StorageRevoery class, which does most of the recoverying job. The logic 
mimics what happens in SamzaContainer.
* add StateStorageTool, for the commandline usage
* unit tests
* docs


Diffs
-

  build.gradle 97de3a2 
  docs/learn/documentation/versioned/container/state-management.md 79067bb 
  samza-core/src/main/java/org/apache/samza/config/JavaStorageConfig.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/config/JavaSystemConfig.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/config/JavaTaskConfig.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/storage/StateStorageTool.java 
PRE-CREATION 
  samza-core/src/main/java/org/apache/samza/storage/StorageRecovery.java 
PRE-CREATION 
  samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala 
f68a7fe 
  samza-core/src/main/scala/org/apache/samza/util/Util.scala 1a67586 
  samza-core/src/test/java/org/apache/samza/config/TestJavaSystemConfig.java 
PRE-CREATION 
  
samza-core/src/test/java/org/apache/samza/storage/MockCheckpointManagerFactory.java
 PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockStorageEngine.java 
PRE-CREATION 
  
samza-core/src/test/java/org/apache/samza/storage/MockStorageEngineFactory.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockSystemConsumer.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/MockSystemFactory.java 
PRE-CREATION 
  samza-core/src/test/java/org/apache/samza/storage/TestStorageRecovery.java 
PRE-CREATION 
  samza-log4j/src/main/java/org/apache/samza/config/Log4jSystemConfig.java 
d5e24f2 
  samza-log4j/src/test/java/org/apache/samza/config/TestLog4jSystemConfig.java 
6314a3e 
  samza-shell/src/main/bash/state-storage-tool.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/33419/diff/


Testing
---

tested with multiple partitions and multiple stores recovery.


Thanks,

Yan Fang