GitHub user shanthoosh opened a pull request:
https://github.com/apache/samza/pull/583
SAMZA-1786: Introduce metadata store abstraction.
As a part of SEP-11, this patch adds MetadataStore interface to store task
and container locality in both yarn and standalone deployment models. Please
refer to SEP-11 for more details.
Few important points to note:
1. As a part of this changes, LocalityManager/TaskAsssignmentManager alone
will be updated to use this interface(subsequently in upcoming future RB's
other util classes will be moved to use this interface as well).
2. In an immediate followup RB, ZkMetadataStore(storing metadata
information in zookeeper) will be added. It will be used in standalone to
read/write locality(through LocalityManager & other standard util classes).
3. In future, ExecutionPlan, streamGraph and other job related metadata can
be stored in any custom store through the same abstraction.
Testing:
1. Added unit tests for new classes introduced in the patch(Fixed the
existing unit tests in LocalityManager/TaskAssignmentManager).
2. All the changes in the patch were validated with test jobs in
samza-hello-samza(https://github.com/apache/samza-hello-samza).
3. LinkedIn testing job(maes-tests-host-affinity) was verified with these
changes to validate if things work end-to-end.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shanthoosh/samza metadata_store_iface
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/583.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #583
----
commit fad4fc09238dda276661a5a201e398980b05caeb
Author: Shanthoosh Venkataraman <spvenkat@...>
Date: 2018-07-24T18:37:47Z
Initial version of MetadataStore interface.
----
---
