Hi, Robert, Thanks for digging into this. I am embedding my answers below:
On Thu, Jun 25, 2015 at 7:40 AM, Robert Zuljevic <r.zulje...@levi9.com> wrote: > 1. Is checkpoint topic referred to in the description coordinator > stream/topic? > In the master branch, checkpoint topic is deprecated (except for migration purpose). The checkpoints will be sent to coordinator stream. Hence, there is no need to change the creation of checkpoint topic any more. > 2. Changelog topic creation is handled by TaskStorageManager class > which is required by SamzaContainer. Would it be preferable to: > > a. Create TaskStoreManager instance(s) in JobRunner and pass them > to SamzaContainer > > b. Create changelog stream in JobRunner/JobCoordinator and skip it > in TaskStoreManager > I prefer option b in your above proposal, w/ a slight modification: 1. JobCoordinator now has a changelogManager which reads the changelog partition to task mapping from the coordinator stream 2. JobCoordinator also has access to the job config and will be able to figure out what are the changelog topics needed in the job 3. JobCoordinator start should have an additional step to create all the changelog topics needed in the job. If exists, validate the partition numbers 4. In TaskStoreManager, we should just get the changelog topic metadata and validate the partitions are correct. Does that sound reasonable? -Yi > > > > > Met vriendelijke groet / Kind regards, > > Robert Žuljević > > Software Developer > > [image: Title: Levi9 IT Services] > ------------------------------ > > Address: Trifkovicev trg 6, 21000 Novi Sad, Serbia > > Tel.: +31 20 6701 947 | +381 21 2155 500 > > Mobile: +381 64 428 28 46 > > Skype: robert.zuljevic > > Internet: www.levi9.com > > > > Chamber of commerce Levi9 Holding: 34221951 > > Chamber of commerce Levi9 IT Services BV: 34224746 > ------------------------------ > > This e-mail may contain confidential or privileged information. If you are > not (one of) the intended recipient(s), please notify the sender > immediately by reply e-mail and delete this message and any attachments > permanently without retaining a copy. Any review, disclosure, copying, > distribution or taking any action in reliance on the contents of this > e-mail by persons or entities other than the intended recipient(s) is > strictly prohibited and may be unlawful. > > The services of Levi9 are exclusively subject to its general terms and > conditions. These general terms and conditions can be found on > www.levi9.com and a copy will be promptly submitted to you on your > request and free of charge. > > >
