----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37817/ -----------------------------------------------------------
(Updated Sept. 9, 2015, 12:28 a.m.) Review request for samza, Yan Fang, Chinmay Soman, Chris Riccomini, and Yi Pan (Data Infrastructure). Changes ------- Just updated some javadoc comments Bugs: SAMZA-619 https://issues.apache.org/jira/browse/SAMZA-619 Repository: samza Description ------- This a major change to how we request and assign containers to resources. It uses a threaded model for request and allocation. More comments in the javadoc. Major changes to note: 1. Changed yarn dependency to 2.6 2. Moved YarnConfig to java config class 2. Removed SamzaAppMasterTaskManager 3. SamzaAppState replaces SamzaAppMasterState 4. SamzaTaskManager is very similar to SamzaAppMasterTaskManager, except that it delegates onContainerAllocated and requestContainers to the thread in ContainerAllocator 5. Removed state.unclaimedContainers 6. Allocator thread sleep time and request timeout is configurable 7. host-affinity can be enabled by using the config "yarn.samza.host-affinity.enabled". Host affinity is guaranteed to work with FairScheduler that has continuous scheduling enabled. Details on this config can be found at SAMZA-617 [design](https://issues.apache.org/jira/secure/attachment/12726945/DESIGN-SAMZA-617-2.pdf). 8. Added unit tests for TaskManager & ContainerAllocator 9. Updated config documentation Pending items: 1. Update web-site with info on this feature (SAMZA-668) Diffs (updated) ----- build.gradle 3a7fabcd4f4e5c8db950c3a33bba33618c5565b4 docs/learn/documentation/versioned/jobs/configuration-table.html c23d7d37f151a0dbdd71f52588773bc67edf88c8 gradle/dependency-versions.gradle 36d564b6ca895f042ee4802643e49180f4947b62 samza-core/src/main/java/org/apache/samza/container/LocalityManager.java c567bf4a1747a7a6aff23be74cf25b1a44e57b42 samza-core/src/main/java/org/apache/samza/coordinator/stream/messages/SetContainerHostMapping.java 5455881e2a4a9c865137f2708f655641b93c2bfb samza-core/src/main/java/org/apache/samza/job/model/JobModel.java 7b592740c082225fc217a24188bc874f883db63b samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala 61e228b1ff8619da4ad2e11b15a5afb7999c4996 samza-core/src/main/scala/org/apache/samza/util/Util.scala c8438bd6ed01a95860867505d84c04c493a5a197 samza-yarn/src/main/java/org/apache/samza/config/YarnConfig.java PRE-CREATION samza-yarn/src/main/java/org/apache/samza/job/yarn/AbstractContainerAllocator.java PRE-CREATION samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerAllocator.java PRE-CREATION samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerFailure.java PRE-CREATION samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerRequestState.java PRE-CREATION samza-yarn/src/main/java/org/apache/samza/job/yarn/ContainerUtil.java PRE-CREATION samza-yarn/src/main/java/org/apache/samza/job/yarn/HostAwareContainerAllocator.java PRE-CREATION samza-yarn/src/main/java/org/apache/samza/job/yarn/SamzaAppState.java PRE-CREATION samza-yarn/src/main/java/org/apache/samza/job/yarn/SamzaContainerRequest.java PRE-CREATION samza-yarn/src/main/java/org/apache/samza/job/yarn/SamzaTaskManager.java PRE-CREATION samza-yarn/src/main/resources/scalate/WEB-INF/views/index.scaml ce2145adde09b0e8097a1c711bc296ee20392e63 samza-yarn/src/main/scala/org/apache/samza/config/YarnConfig.scala 1f51551893c42bb13d7941c8b0e5594ac7f42585 samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala af42c6a6636953a95f79837fe372e0dbd735df70 samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterLifecycle.scala d475c4c566f17f88a04db5fbc84cc2e27eb333d2 samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterMetrics.scala 03acfe1bbbabf8f54be9f36fdae785476da45135 samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterService.scala 060538623e4d67b986bc635518e7fe8ebdde9e24 samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterState.scala f667c83a7fff43a5efdc64cde019b2cf35f38cb9 samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterTaskManager.scala 1743c8611a94fa1c7f7dafd8ff8c713039f3df8c samza-yarn/src/main/scala/org/apache/samza/job/yarn/YarnJob.scala 8dd70c9977473e083e463d01b049c40e15b21f4a samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterRestServlet.scala 09f4dc32a4b18aeb3accb856c360ea2f95c82673 samza-yarn/src/main/scala/org/apache/samza/webapp/ApplicationMasterWebServlet.scala 7fd51221149ba59e7ea1bc0b51d58726ec17a7d7 samza-yarn/src/test/java/org/apache/samza/job/yarn/TestContainerAllocator.java PRE-CREATION samza-yarn/src/test/java/org/apache/samza/job/yarn/TestContainerRequestState.java PRE-CREATION samza-yarn/src/test/java/org/apache/samza/job/yarn/TestHostAwareContainerAllocator.java PRE-CREATION samza-yarn/src/test/java/org/apache/samza/job/yarn/TestSamzaContainerRequest.java PRE-CREATION samza-yarn/src/test/java/org/apache/samza/job/yarn/TestSamzaTaskManager.java PRE-CREATION samza-yarn/src/test/java/org/apache/samza/job/yarn/util/MockContainerAllocator.java PRE-CREATION samza-yarn/src/test/java/org/apache/samza/job/yarn/util/MockContainerUtil.java PRE-CREATION samza-yarn/src/test/java/org/apache/samza/job/yarn/util/MockHttpServer.java PRE-CREATION samza-yarn/src/test/java/org/apache/samza/job/yarn/util/MockNMClient.java PRE-CREATION samza-yarn/src/test/java/org/apache/samza/job/yarn/util/TestAMRMClientImpl.java PRE-CREATION samza-yarn/src/test/java/org/apache/samza/job/yarn/util/TestUtil.java PRE-CREATION samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterLifecycle.scala df5992e659302d2918c4e2c30b6122ed51ab9fe8 samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterService.scala 6f4bfaf916d687426f746fd3b09cd56490bb500e samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala 2eec65f02826de40493925c08ff344a8cc4feecb Diff: https://reviews.apache.org/r/37817/diff/ Testing ------- Tested with hello-samza locally. Tested with a sample job on a 3 node Yarn cluster. Works as expected. Thanks, Navina Ramesh