Re: Kafka client.id collision

[Navina Ramesh (Apache)]

Hi David,

I think this is expected to occur as a warning since we spin up all kafka
clients with the same client-id, which is $job.name + $job.id.

As Jagadish mentioned, it will be great if you can provide us the entire
log so that we can take a look.

As a side note for the samza contributors, I do believe the container spins
up kafka clients for each kafka systems defined, even if it is not used.
Iirc, we use `KafkaUtil.getClientId` for generating the client id. Perhaps
it makes sense to append another identifier with the client id (such as
system name or component name). That way, we won't lose the kafka-client
related metrics and there will be no overlap between the client ids.
Thoughts?

Thanks!
Navina

On Thu, Jul 20, 2017 at 9:13 AM, Jagadish Venkatraman <
jagadish1...@gmail.com> wrote:

> Can you share the entire log file if that's okay? The warning should be a
> red-herring IMHO.
>
> On Thu, Jul 20, 2017 at 7:50 AM Davide Simoncelli 
> wrote:
>
> > Hi,
> >
> > Thanks for the reply.
> >
> > It is a warning, but the application fails. Here is the logging:
> >
> >
> > 017-07-20 10:43:06.349 [main] AppInfoParser [INFO] Kafka version :
> 0.10.1.1
> > 2017-07-20 10:43:06.349 [main] AppInfoParser [INFO] Kafka commitId :
> > f10ef2720b03b247
> > 2017-07-20 10:43:06.351 [main] AppInfoParser [WARN] Error registering
> > AppInfo mbean
> > javax.management.InstanceAlreadyExistsException:
> > kafka.producer:type=app-info,id=samza_producer-wikipedia_feed-1
> > at com.sun.jmx.mbeanserver.Repository.addMBean(
> Repository.java:437)
> > at
> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.
> registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> > at
> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.
> registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> > at
> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(
> DefaultMBeanServerInterceptor.java:900)
> > at
> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(
> DefaultMBeanServerInterceptor.java:324)
> > at
> > com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(
> JmxMBeanServer.java:522)
> > at
> > org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(
> AppInfoParser.java:58)
> > at
> > org.apache.kafka.clients.producer.KafkaProducer.(
> KafkaProducer.java:331)
> > at
> > org.apache.kafka.clients.producer.KafkaProducer.(
> KafkaProducer.java:163)
> > at
> > org.apache.samza.system.kafka.KafkaSystemFactory$$anonfun$3.
> apply(KafkaSystemFactory.scala:89)
> > at
> > org.apache.samza.system.kafka.KafkaSystemFactory$$anonfun$3.
> apply(KafkaSystemFactory.scala:89)
> > at
> > org.apache.samza.system.kafka.KafkaSystemProducer.send(
> KafkaSystemProducer.scala:144)
> > at
> > org.apache.samza.coordinator.stream.CoordinatorStreamSystemProduce
> r.send(CoordinatorStreamSystemProducer.java:113)
> > at
> > org.apache.samza.coordinator.stream.CoordinatorStreamWriter.
> sendSetConfigMessage(CoordinatorStreamWriter.java:98)
> > at
> > org.apache.samza.coordinator.stream.CoordinatorStreamWriter.sendMessage(
> CoordinatorStreamWriter.java:82)
> > at
> > org.apache.samza.job.yarn.SamzaYarnAppMasterService.onInit(
> SamzaYarnAppMasterService.scala:68)
> > at
> > org.apache.samza.job.yarn.YarnClusterResourceManager.start(
> YarnClusterResourceManager.java:180)
> > at
> > org.apache.samza.clustermanager.ContainerProcessManager.start(
> ContainerProcessManager.java:167)
> > at
> > org.apache.samza.clustermanager.ClusterBasedJobCoordinator.run(
> ClusterBasedJobCoordinator.java:154)
> > at
> > org.apache.samza.clustermanager.ClusterBasedJobCoordinator.main(
> ClusterBasedJobCoordinator.java:222)
> > 2017-07-20 10:43:06.549 [main] CoordinatorStreamWriter [INFO] Stopping
> the
> > coordinator stream producer.
> > 2017-07-20 10:43:06.549 [main] CoordinatorStreamSystemProducer [INFO]
> > Stopping coordinator stream producer.
> > 2017-07-20 10:43:06.549 [main] KafkaProducer [INFO] Closing the Kafka
> > producer with timeoutMillis = 9223372036854775807 ms.
> >
> >
> > > On 20 Jul 2017, at 3:16 pm, Jagadish Venkatraman <
> jagadish1...@gmail.com>
> > wrote:
> > >
> > > Hi Davide,
> > >
> > > Is this logged as an error or as a warning?
> > >
> > > IIUC, this warning should not fail the job. It may not cause some Mbean
> > > sensors / metrics emitted from Kafka to be correctly reported (since,
> > those
> > > are reported per-clientId).
> > >
> > > The job should still continue to run.
> > >
> > > The entire log file will be helpful for further debugging!
> > >
> > > On Thu, Jul 20, 2017 at 3:32 AM, Davide Simoncelli <
> > netcelli@gmail.com >
> > > wrote:
> > >
> > >> Hello,
> > >>
> > >> We are running Kafka 0.10.1.1 in production. Unfortunately the Samza
> app
> > >> fails to start because of this bug: https:/

Re: Samza Meetup

[Navina Ramesh (Apache)]

No worries. We would love to meet you in person too. Keep an eye out on the
mailing list for the Meetup link.

Cheers!
Navina

On Jul 20, 2017 08:37, "Renato Marroquín Mogrovejo" <
renatoj.marroq...@gmail.com> wrote:

> Thanks Jagadish and Navina!
> I am really interested in attending as I am in the area, it'd be my first
> in-person Samza meetup :D
> But unfortunately I don't have anything to present this time :(
>
>
> Renato M.
>
> 2017-07-18 23:46 GMT-07:00 Navina Ramesh (Apache) :
>
> > Hi Renato,
> >
> > We are planning for mid-August as a tentative target for the next meetup.
> >
> > If you are interested in participating or speaking at the meetup, please
> > let us know.
> >
> > Thanks!
> > Navina
> >
> >
> >
> > On Tue, Jul 18, 2017 at 10:36 AM, Renato Marroquín Mogrovejo <
> > renatoj.marroq...@gmail.com> wrote:
> >
> > > Hi Samza experts and users,
> > >
> > > I was wondering if there is going to be a meetup this summer or when
> the
> > > next one is.
> > > Thanks!
> > >
> > >
> > > Best,
> > >
> > > Renato M.
> > >
> >
>

Re: Samza Meetup

[Navina Ramesh (Apache)]

Hi Renato,

We are planning for mid-August as a tentative target for the next meetup.

If you are interested in participating or speaking at the meetup, please
let us know.

Thanks!
Navina



On Tue, Jul 18, 2017 at 10:36 AM, Renato Marroquín Mogrovejo <
renatoj.marroq...@gmail.com> wrote:

> Hi Samza experts and users,
>
> I was wondering if there is going to be a meetup this summer or when the
> next one is.
> Thanks!
>
>
> Best,
>
> Renato M.
>

Re: [VOTE] SEP-5: Enable partition expansion of input streams

[Navina Ramesh (Apache)]

After a lot of Q&A, let's get this done :)

+1 (binding)

Thanks!
Navina

On Tue, Jun 20, 2017 at 10:31 AM, xinyu liu  wrote:

> +1 (non-binding) on this design.
>
> To me the task-count based groupers should work well in practice for
> partition expansion of system using hash for partitions, e.g. Kafka. It
> will not cause any state transfer between hosts so the runtime cost will be
> minimal. In the future when we support dynamically re-balancing the tasks,
> we can further scale the task count if needed.
>
> Thanks,
> Xinyu
>
> On Mon, Jun 19, 2017 at 9:27 AM, Dong Lin  wrote:
>
> > Hi everyone,
> >
> > Can you please vote for SEP-5? The wiki can be found at
> > *https://cwiki.apache.org/confluence/display/SAMZA/SEP-
> > 5%3A+Enable+partition+expansion+of+input+streams
> >  > 5%3A+Enable+partition+expansion+of+input+streams>.*
> >
> > Thanks,
> > Dong
> >
>

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

[Navina Ramesh (Apache)]

> But IMO it is the best available solution towards the support of
partition expansion in comparison to alternative, no?

At this time, relative to the other alternatives you have listed, this is a
path of least effort to solving this problem. I agree to that. :)

> I can merge those two sections or update the statement if the current 
> statement
has not clearly explained the reason of partition expansion in Kafka.

Given the significance of what you are actually trying to solve, I think it
will be better to have it in points. Let me come find you and we can update
it.

> I have updated wiki and added the task expansion to the Future Work section.
On the other hand I still keep it in the Rejected Alternative section to
explain why this future work does not replace the existing proposal in
SEP-5. Does this sound reasonable?

It is very confusing to me how the same point can be under "Future Work"
and "Rejected Alternative". There is no question about the future work
*replacing* SEP-5. Iiuc, this SEP is a subset for the partition expansion
solution. So, I don't think increasing task count should be a rejected
alternative.

> I am also not sure why a feature needs to be "utmost priority" in order
to be accepted. Can you explain a bit on that?

I don't think I ever claimed that the feature needs to be of "utmost
priority" to be accepted. I was just stating my opinion.


Thanks!
Navina

On Wed, Jun 21, 2017 at 3:52 PM, Dong Lin  wrote:

> Thanks much for the reply Navina. Please see my reply inline.
>
> On Wed, Jun 21, 2017 at 2:57 PM, Navina Ramesh (Apache)  >
> wrote:
>
> > Thanks to Jake, Dong and Kartik for keeping the discussion going.
> >
> > > Here are the pros and cons of the extra re-partitioning stage in
> > comparison
> > to SEP-5.
> >
> > I think that is good summarization of pros/cons for the repartitioning
> > stage based solution. Can you please include it in your SEP? It seems
> like
> > you already have access. If you are still unable to access the wiki page,
> > feel free to walk over to Samza area and find me!
> >
>
> Sure. I have added this summary to the Alternative Section.
>
>
> >
> > > I think there is always a way for user to mess up their job if they
> > configure the Samza job incorrectly.
> >
> > I don't think Jake or anyone is arguing about an "incorrectly" configured
> > Samza job. The question was towards how easy/difficult it is for users to
> > *not mess* up their job with incorrect configurations.
> >
> > > I also think the assumption made in this SEP is not particularly harder
> > to understand than other existing configs in Samza.
> >
> > I disagree here. Other configs don't require you understand more than one
> > assumption.
> >
> > There is already an overload of configs in Samza and I think we are
> trying
> > to shield it as much as possible from the users (esp. with fluent api).
> > More specifically, we don't want the user to know about the internals of
> > Samza such ssp grouper, taskname grouper etc. Since the proposed solution
> > makes the configuration more complex to understand, it *is a* burden on
> the
> > user.
> >
> > Just because configs are the way it is, it doesn't mean we increase the
> > complexity of it and push the burden on users to manage it correctly. My
> > two cents.
> >
>
> Sure, I agree the proposal requires user to understand the assumption in
> order to expand the partition of the topic. But it is very subjective as to
> whether the added complexity is acceptable or not. If there is better way
> to allow user to expand partition of the input stream without making
> assumption, then we can just do that. The current solution is not perfect.
> But IMO it is the best available solution towards the support of partition
> expansion in comparison to alternative, no?
>
>
> > Here are a few things that I believe are needed for wrapping up the SEP:
> >
> > 1. For the longest time, I thought partition expansion happens in Kafka
> > only when the volume of messages across partitions is too high. Based on
> > this assumption, I would only assume that re-mapping expanded partitions
> to
> > the same task will have adverse effect on the throughput/resource
> > utilization of the processor/container in Samza (for example, disk
> > utilization may increase significantly. With disk quota throttling, it
> > could cause the processor to drop.). However, after speaking with Xinyu,
> it
> > turns out that partition expansion also happens when there is a
> > per-partition data retention limit imposed by Kafka (

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

[Navina Ramesh (Apache)]

Thanks to Jake, Dong and Kartik for keeping the discussion going.

> Here are the pros and cons of the extra re-partitioning stage in
comparison
to SEP-5.

I think that is good summarization of pros/cons for the repartitioning
stage based solution. Can you please include it in your SEP? It seems like
you already have access. If you are still unable to access the wiki page,
feel free to walk over to Samza area and find me!

> I think there is always a way for user to mess up their job if they
configure the Samza job incorrectly.

I don't think Jake or anyone is arguing about an "incorrectly" configured
Samza job. The question was towards how easy/difficult it is for users to
*not mess* up their job with incorrect configurations.

> I also think the assumption made in this SEP is not particularly harder
to understand than other existing configs in Samza.

I disagree here. Other configs don't require you understand more than one
assumption.

There is already an overload of configs in Samza and I think we are trying
to shield it as much as possible from the users (esp. with fluent api).
More specifically, we don't want the user to know about the internals of
Samza such ssp grouper, taskname grouper etc. Since the proposed solution
makes the configuration more complex to understand, it *is a* burden on the
user.

Just because configs are the way it is, it doesn't mean we increase the
complexity of it and push the burden on users to manage it correctly. My
two cents.

Here are a few things that I believe are needed for wrapping up the SEP:

1. For the longest time, I thought partition expansion happens in Kafka
only when the volume of messages across partitions is too high. Based on
this assumption, I would only assume that re-mapping expanded partitions to
the same task will have adverse effect on the throughput/resource
utilization of the processor/container in Samza (for example, disk
utilization may increase significantly. With disk quota throttling, it
could cause the processor to drop.). However, after speaking with Xinyu, it
turns out that partition expansion also happens when there is a
per-partition data retention limit imposed by Kafka (not sure if it is only
in LinkedIn or in Kafka open-source as well). Imo, this is the primary
use-case that we are trying to solve for in Samza and it is not very
obvious from the SEP.
@Dong, can you please explain *the circumstances under which partition
expansion can happen*, under "Motivation" section?  I disagree to the
current motivation described as -> "This design doc provides a solution to
increase partition number of the input streams of a stateful Samza job
while still ensuring the correctness of Samze job output. "
This is a solution, albeit not fully done through this SEP alone.

2. I think we are in consensus about the fact that increasing the task
number and handling the state correctly is a good solution for Samza in the
long-run. In your rejected alternatives, you mention "However, this feature
alone does not solve the problem of allowing partition expansion.". What
else is required to allow partition expansion? Can you please elaborate on
that in point #1 of the rejected alternatives? If there is still more work
to be done to support partition expansion in Samza, it is worthwhile to
mention it under *Future Work*, instead of under "Rejected Alternatives".
Perhaps you were waiting for edit permissions to the wiki. Please make this
change so it is well-tracked.

I am still not totally crazy about the proposed solution because it is not
clear for open-source, who or which use-cases stand to benefit. I am not
convinced that this problem is of utmost priority for the Samza community
*at this point of time*.

I am on the same page as Jake on this one. Not a +1, just a 0 (if that even
matters).

Thanks!
Navina

On Sun, Jun 18, 2017 at 12:04 AM, Dong Lin  wrote:

> BTW, I will update the SEP-5 wiki with our latest discussion after I have
> got the wiki edit access.
>
> On Sat, Jun 17, 2017 at 11:36 PM, Dong Lin  wrote:
>
> > Thanks everyone for the comment!
> >
> > I am currently leaning towards the current approach. I think Kartik
> raised
> > a good point that the extra repartitoning stage will also incur
> additional
> > throughput on Kafka in addition to the potential storage cost. Any other
> > Samza developers also chime in and provide your opinions on this
> proposal?
> >
> > Since this discussion thread has been open for three weeks, I will
> > initiate voting thread on Monday if there is no major revision
> suggestion.
> >
> > Thanks,
> > Dong
> >
> >
> > On Thu, Jun 15, 2017 at 6:32 PM, Kartik Paramasivam <
> > kparamasi...@linkedin.com.invalid> wrote:
> >
> >> Great discussion !
> >>
> >> Here are some more thoughts
> >>
> >> The point that repartitioning is a more general purpose solution is
> surely
> >> spot on.  For many source systems (Kinesis, Google Pub-Sub, any of the
> >> older queuing systems (rabbitMQ etc. etc.), repartitioning is anyways

Re: [VOTE] Apache Samza 0.13.0 RC6

[Navina Ramesh (Apache)]

+1 (binding)

Thanks to everyone for diligently testing out the RCs and getting this
release out!

Cheers!
Navina

On Thu, Jun 8, 2017 at 9:09 AM, Chris Riccomini 
wrote:

> +1 (binding)
>
> On Wed, Jun 7, 2017 at 8:55 AM, Yi Pan  wrote:
>
> > +1 (binding)
> > build and ran all local integration tests on Linux.
> >
> > On Tue, Jun 6, 2017 at 4:01 PM, Boris S  wrote:
> >
> > > +1 (non-binding)
> > > build and tested on Linux (with python 2.7; 2.4 and 3.5 - didn't work)
> > >
> > > On Tue, Jun 6, 2017 at 2:49 PM, Jacob Maes 
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Built and tested on both OSX and RHEL with gradle 2.0 and 2.2
> > > respectively.
> > > >
> > > > Also verified the high level API + YARN host affinity on a test job
> > with
> > > 32
> > > > containers.
> > > >
> > > >
> > > >
> > > > On Tue, Jun 6, 2017 at 9:14 AM, xinyu liu 
> > wrote:
> > > >
> > > > > +1 (non-binding).
> > > > >
> > > > > Downloaded the source tar, built it and run check-all.sh on REHL6
> > with
> > > > > gradle 2.8. All passed.
> > > > >
> > > > > As a side note to Jagadish's comments, the build doesn't work on a
> > > higher
> > > > > gradle version either (gradle 3.5). Seems
> > > "-language:implicitConversions
> > > > > -language:reflectiveCalls" is not a valid build option anymore.
> > > > >
> > > > > Thanks,
> > > > > Xinyu
> > > > >
> > > > > On Mon, Jun 5, 2017 at 10:06 AM, Jagadish Venkatraman <
> > > > > jagadish1...@gmail.com> wrote:
> > > > >
> > > > > > Checked out, ran tests, and all of them pass.
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > I did get an error when running with gradle 2.4:
> > > > > > >>Could not resolve all dependencies for configuration
> > > > > > ':samza-kafka_2.11:compile'. > java.lang.
> > > UnsupportedOperationException
> > > > > (no
> > > > > > error message)
> > > > > >
> > > > > > However, when I used gradle 2.8, it was resolved.
> > > > > >
> > > > > > *gradle wrapper --gradle-version 2.8*
> > > > > >
> > > > > > Best,
> > > > > > Jagadish
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 5, 2017 at 8:37 AM, Jake Maes 
> > wrote:
> > > > > >
> > > > > > > This is a call for a vote on a release of Apache Samza 0.13.0.
> > > Thanks
> > > > > to
> > > > > > > everyone who has contributed to this release. We are very glad
> to
> > > see
> > > > > > some
> > > > > > > new contributors and features in this release.
> > > > > > >
> > > > > > > The release candidate can be downloaded from here:
> > > > > > > http://home.apache.org/~jmakes/samza-0.13.0-rc6/
> > > > > > >
> > > > > > > The release candidate is signed with pgp key 940AFC5A, which
> can
> > be
> > > > > found
> > > > > > > on keyservers:
> > > > > > > *http://pgp.mit.edu/pks/lookup?op=get&search=0x940AFC5A
> > > > > > > *
> > > > > > >
> > > > > > > The git tag is release-0.13.0-rc6 and signed with the same pgp
> > key:
> > > > > > > https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tag;h=
> > > > > > > refs/tags/release-0.13.0-rc6
> > > > > > >
> > > > > > > Test binaries have been published to Maven's staging
> repository,
> > > and
> > > > > are
> > > > > > > available here:
> > > > > > > https://repository.apache.org/content/repositories/
> > > > orgapachesamza-1026
> > > > > > >
> > > > > > > 144 issues were resolved for this release:
> > > > > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%
> > > > > > > 20SAMZA%20AND%20fixVersion%20in%20(0.13%2C%200.13.0)%
> > > > > > > 20AND%20status%20in%20(
> > > > > > > Resolved%2C%20Closed)
> > > > > > >
> > > > > > > The vote will be open for 72 hours (ending at 9:00AM Thursday,
> > > > > > 06/08/2017).
> > > > > > >
> > > > > > > Please download the release candidate, check the
> > hashes/signature,
> > > > > build
> > > > > > it
> > > > > > > and test it, and then please vote:
> > > > > > >
> > > > > > >
> > > > > > > [ ] +1 approve
> > > > > > >
> > > > > > > [ ] +0 no opinion
> > > > > > >
> > > > > > > [ ] -1 disapprove (and reason why)
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Jagadish V,
> > > > > > Graduate Student,
> > > > > > Department of Computer Science,
> > > > > > Stanford University
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

[Navina Ramesh (Apache)]

on section that "*The feature of
> > task expansion is out of the scope of this proposal and will be addressed
> > in a future SEP*". The second paragraph in the Motivation section is
> mainly
> > used to explain the thinking process that we have gone through, what
> other
> > alternative we have considered, and we plan to do in Samza in the nex
> step.
> >
> > To answer your question why increasing the partition number will increase
> > the throughput of the kafka consumer in the container, Kafka consumer can
> > potentially fetch more data in one FetchResponse with more partitions in
> > the FetchRequest. This is because we limit the maximum amount of data
> that
> > can be fetch for a given partition in the FetchResponse. This by default
> is
> > set to 1 MB. And there is reason that we can not arbitrarily bump up this
> > limit.
> >
> > To answer your question how partition expansion in Kafka impacts the
> > clients, Kafka consumer is able to automatically detect new partition of
> > the topic and reassign all (both old and new) partitions across consumers
> > in the consumer group IF you tell consumer the topic to be subscribed.
> But
> > consumer in Samza's container uses another way of subscription. Instead
> of
> > subscribing to the topic, the consumer in Samza's container subscribes to
> > the specific partitions of the topic. In this case, if new partitions
> have
> > been added, Samza will need to explicitly subscribe to the new partitions
> > of the topic. The "Handle partition expansion while tasks are running"
> > section in the SEP addresses this issue in Samza -- it recalculates the
> job
> > model and restart container so that consumer can subscribe to the new
> > partitions.
> >
> > I will ask other dev to take a look at the proposal. I will start the
> > voting thread tomorrow if there is no further concern with the SEP.
> >
> > Thanks!
> > Dong
> >
> >
> > On Wed, May 31, 2017 at 12:01 AM, Navina Ramesh (Apache) <
> > nav...@apache.org>
> > wrote:
> >
> > > Hey Dong,
> > >
> > > >  I have updated the motivation section to clarify this.
> > >
> > > Thanks for updating the motivation. Couple of notes here:
> > >
> > > 1.
> > > > "The motivation of increasing partition number of Kafka topic
> includes
> > 1)
> > > limit the maximum size of a partition in order to improve broker
> > > performance and 2) increase throughput of Kafka consumer in the Samza
> > > container."
> > >
> > > It's unclear to me how increasing the partition number will increase
> the
> > > throughput of the kafka consumer in the container? Theoretically, you
> > will
> > > still be consuming the same amount of data in the container,
> irrespective
> > > of whether it is coming from one partition or more than one expanded
> > > partitions. Can you please explain it for me here, what you mean by
> that?
> > >
> > > 2. I believe the second paragraph under motivation is simply talking
> > about
> > > the scope of the current SEP. It will be easier to read if what
> solution
> > is
> > > included in this SEP and what is left out as not in scope. (for
> example,
> > > expansions for stateful jobs is supported or not).
> > >
> > > > We need to persist the task-to-sspList mapping in the
> > > coordinator stream so that the job can derive the original number of
> > > partitions of each input stream regardless of how many times the
> > partition
> > > has expanded. Does this make sense?
> > >
> > > Yes. It does!
> > >
> > > > I am not sure how this is related to the locality though. Can you
> > clarify
> > > your question if I haven't answered your question?
> > >
> > > It's not related. I just meant to give an example of yet another
> > > coordinator message that is persisted. Your ssp-to-task mapping is
> > > following a similar pattern for persisting. Just wanted to clarify
> that.
> > >
> > > > Can you let me know if this, together with the answers in the
> previous
> > > email, addresses all your questions?
> > >
> > > Yes. I believe you have addressed most of my questions. Thanks for
> taking
> > > time to do that.
> > >
> > > > Is there specific question you have regarding partition
> > > expansion in Kafka?
> > >
> > > 

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

[Navina Ramesh (Apache)]

gt;
>
> On Wed, May 24, 2017 at 11:15 PM, Dong Lin  wrote:
>
> > Hey Navina,
> >
> > Thanks much for your comments. Please see my reply inline.
> >
> > On Wed, May 24, 2017 at 10:22 AM, Navina Ramesh (Apache) <
> > nav...@apache.org> wrote:
> >
> >> Thanks for the SEP, Dong. I have a couple of questions to understand
> your
> >> proposal better:
> >>
> >> * Under motivation, you mention that "_We expect this solution to work
> >> similarly with other input system as well._", yet I don't see any
> >> discussion on how it will work with other input systems. That is, what
> >> kind
> >> of contract does samza expect from other input systems ? If we are not
> >> planning to provide a generic solution, it might be worth calling it out
> >> in
> >> the SEP.
> >>
> >
> > I think the contract we expect from other systems are exactly the
> > operational requirement mentioned in the SEP, i.e. partitions should
> always
> > be doubled and the hash algorithm should module the number of partitions.
> > SEP-5 should also allow partition expansion of all input systems that
> meet
> > these two requirements. I have updated the motivation section to clarify
> > this.
> >
> >
> >>
> >> * I understand the partition mapping logic you have proposed. But I
> think
> >> the example explanation doesn't match the diagram. In the diagram, after
> >> expansion, partiion-0 and partition-1 are pointing to bucket 0 and
> >> partition-3 and partition-4 are pointing to bucket 1. I think the former
> >> has to be partition-0 and partition-2 and the latter, is partition-1 and
> >> partition-3. If I am wrong, please help me understand the logic :)
> >>
> >
> > Good catch. I will update the figure to fix this problem.
> >
> >
> >>
> >> * I don't know how partition expansion in Kafka works. I am familiar
> with
> >> how shard splitting happens in Kinesis - there is hierarchical relation
> >> between the parent and child shards. This way, it will also allow the
> >> shards to be merged back. Iiuc, Kafka only supports partition
> "expansion",
> >> as opposed to "splits". Can you provide some context or link related to
> >> how
> >> partition expansion works in Kafka?
> >>
> >
> > I couldn't find any wiki on partition expansion in Kafka. The partition
> > expansion logic in Kafka is very simply -- it simply adds new partition
> to
> > the existing topic. Is there specific question you have regarding
> partition
> > expansion in Kafka?
> >
> >
> >>
> >> * Are you only recommending that expansion can be supported for samza
> jobs
> >> that use Kafka as input systems **and** configure the SSPGrouper as
> >> GroupByPartitionFixedTaskNum? Sounds to me like this only applies for
> >> GroupByPartition. Please correct me if I am wrong. What is the
> expectation
> >> for custom SSP Groupers?
> >>
> >
> > The expansion can be supported for Samza jobs if the input system meets
> > the operational requirement mentioned above. It doesn't have to use Kafka
> > as input system.
> >
> > The current proposal provided solution for jobs that currently use
> > GroupByPartition. The proposal can be extended to support jobs that use
> > other grouper that are pre-defined in Samza. The custom SSP grouper needs
> > to handle partition expansion similar to how GroupByPartitionFixedTaskNum
> > handles it and it is users' responsibility to update their custom grouper
> > implementation.
> >
> >
> >>
> >> * Regarding storing SSP-to-Task assignment to coordinator stream: Today,
> >> the JobModel encapsulates the data model in samza which also includes
> >> **TaskModels**. TaskModel, typically shows the task-to-sspList mapping.
> >> What is the reason for using a separate coordinator stream message
> >> *SetSSPTaskMapping*? Is it because the JobModel itself is not persisted
> in
> >> the coordinator stream today?  The reason locality exists outside of the
> >> jobmodel is because *locality* information is written by each container,
> >> where as it is consumed only by the leader jobcoordinator/AM. In this
> >> case,
> >> the writer of the mapping information and the reader is still the leader
> >> jobcoordinator/AM. So, I want to understand the motivation for this
> >> choice.
> >>
> 

[VOTE] Apache Samza 0.13.0 RC1

[Navina Ramesh (Apache)]

Hi everyone,

This is a call for a vote on a release of Apache Samza 0.13.0. Thanks to
everyone who has contributed to this release. We are very glad to see some
new contributors and features in this release.

The release candidate can be downloaded from here:
*http://home.apache.org/~navina/samza-0.13.0-rc1/
*

The release candidate is signed with pgp key 331C8F69 , which can be found
on keyservers:
http://pgp.mit.edu/pks/lookup?op=get&search=0x331C8F69

The git tag is release-0.13.0-rc1 and signed with the same pgp key:
https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tag;h=refs/tags/release-0.13.0-rc1

Test binaries have been published to Maven's staging repository, and are
available here:
https://repository.apache.org/content/repositories/orgapachesamza-1021

137 issues were resolved for this release:
https://issues.apache.org/jira/issues/?jql=project%20%
3D%20SAMZA%20AND%20fixVersion%20in%20(0.13%2C%200.13.0)%
20AND%20status%20in%20(Resolved%2C%20Closed)

The vote will be open for 3 *working* days (ending at 8:00PM Monday,
05/13/2017). We have an extended deadline this time as it is too close to a
long weekend.

Please download the release candidate, check the hashes/signature, build it
and test it, and then please vote:


[ ] +1 approve

[ ] +0 no opinion

[ ] -1 disapprove (and reason why)

Cheers!
Navina

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

[Navina Ramesh (Apache)]

Thanks for the SEP, Dong. I have a couple of questions to understand your
proposal better:

* Under motivation, you mention that "_We expect this solution to work
similarly with other input system as well._", yet I don't see any
discussion on how it will work with other input systems. That is, what kind
of contract does samza expect from other input systems ? If we are not
planning to provide a generic solution, it might be worth calling it out in
the SEP.

* I understand the partition mapping logic you have proposed. But I think
the example explanation doesn't match the diagram. In the diagram, after
expansion, partiion-0 and partition-1 are pointing to bucket 0 and
partition-3 and partition-4 are pointing to bucket 1. I think the former
has to be partition-0 and partition-2 and the latter, is partition-1 and
partition-3. If I am wrong, please help me understand the logic :)

* I don't know how partition expansion in Kafka works. I am familiar with
how shard splitting happens in Kinesis - there is hierarchical relation
between the parent and child shards. This way, it will also allow the
shards to be merged back. Iiuc, Kafka only supports partition "expansion",
as opposed to "splits". Can you provide some context or link related to how
partition expansion works in Kafka?

* Are you only recommending that expansion can be supported for samza jobs
that use Kafka as input systems **and** configure the SSPGrouper as
GroupByPartitionFixedTaskNum? Sounds to me like this only applies for
GroupByPartition. Please correct me if I am wrong. What is the expectation
for custom SSP Groupers?

* Regarding storing SSP-to-Task assignment to coordinator stream: Today,
the JobModel encapsulates the data model in samza which also includes
**TaskModels**. TaskModel, typically shows the task-to-sspList mapping.
What is the reason for using a separate coordinator stream message
*SetSSPTaskMapping*? Is it because the JobModel itself is not persisted in
the coordinator stream today?  The reason locality exists outside of the
jobmodel is because *locality* information is written by each container,
where as it is consumed only by the leader jobcoordinator/AM. In this case,
the writer of the mapping information and the reader is still the leader
jobcoordinator/AM. So, I want to understand the motivation for this choice.

Cheers!
Navina

On Tue, May 23, 2017 at 5:45 PM, Dong Lin  wrote:

> Hi all,
>
> We created SEP-5: Enable partition expansion of input streams. Please find
> the SEP wiki in the link
> https://cwiki.apache.org/confluence/display/SAMZA/SEP-
> 5%3A+Enable+partition+expansion+of+input+streams
> .
>
> You feedback is appreciated!
>
> Thanks,
> Dong
>

Re: [VOTE] Apache Samza 0.13.0 RC0

[Navina Ramesh (Apache)]

Prateek told me that he sent out a cancel email. It didn't reach the
mail-archive I think. Lately, we have this kind of issues where the emails
are not reaching our dev list.

On Wed, May 17, 2017 at 2:06 PM, Yi Pan  wrote:

> Hi, all,
>
> Based on the conversation above, can we officially cancel this vote?
>
> Thanks!
>
> -Yi
>
> On Mon, May 15, 2017 at 9:31 AM, Ignacio Solis  wrote:
>
> > Thanks!
> >
> > On Mon, May 15, 2017 at 8:00 AM, Navina Ramesh
> >  wrote:
> > > I will try to get the patch out today. Work doesn't look trivial. I am
> on
> > > it.
> > >
> > > Navina
> > >
> > > On May 14, 2017 23:10, "Ignacio Solis"  wrote:
> > >
> > >> We should hold off until it is solved.  How long will it take to fix
> > this?
> > >>
> > >> On Sun, May 14, 2017 at 10:13 PM, Navina Ramesh (Apache)
> > >>  wrote:
> > >> > I just changed the status of this JIRA to "BLOCKER" -
> > >> > https://issues.apache.org/jira/browse/SAMZA-1128
> > >> >
> > >> > This causes a bug in standalone deployment where any failure in the
> > >> barrier
> > >> > protocol stops the scheduled executorservice. Unfortunately,
> > >> > CoordinationUtils creates its own scheduled executorservice, which
> is
> > >> > incorrect. Scheduled ExecutorService is meant to be the working
> queue
> > for
> > >> > the ZkJobCoordinator. This needs to be fixed. Bharath already ran
> into
> > >> this
> > >> > bug during testing on Friday.
> > >> >
> > >> > veto for this release candidate.
> > >> >
> > >> > @Prateek/Jagadish:
> > >> > I recommend sending a "non-vote, testing release candidate" for this
> > >> > release until we complete all pending tasks (includes docs, tests
> > etc).
> > >> It
> > >> > will also be useful to share the pending tasks and their progress.
> In
> > >> case
> > >> > you have already shared it, I might have missed it since some emails
> > are
> > >> > bouncing off my inbox.
> > >> >
> > >> > Thanks!
> > >> > Navina
> > >> >
> > >> > On Sun, May 14, 2017 at 1:30 PM, Boris S  wrote:
> > >> >
> > >> >> I think we need to add SAMZA-1286 and
> > >> >> SAMZA-1279 to the release .
> > >> >>
> > >> >> On Wed, May 10, 2017 at 7:51 PM, Jagadish Venkatraman <
> > >> jagad...@apache.org
> > >> >> >
> > >> >> wrote:
> > >> >>
> > >> >> > This is a call for a vote on a release of Apache Samza 0.13.0.
> > Thanks
> > >> to
> > >> >> > everyone who has contributed to this release. We are very glad to
> > see
> > >> >> some
> > >> >> > new contributors and features in this release.
> > >> >> >
> > >> >> > The release candidate can be downloaded from here:
> > >> >> > http://home.apache.org/~jagadish/samza-0.13.0-rc0/
> > >> >> >
> > >> >> > The release candidate is signed with pgp key AF81FFBF, which can
> be
> > >> found
> > >> >> > on keyservers:
> > >> >> > http://pgp.mit.edu/pks/lookup?op=get&search=0xAF81FFBF
> > >> >> >
> > >> >> > The git tag is release-0.13.0-rc0 and signed with the same pgp
> key:
> > >> >> > https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tag;h=
> > >> >> > refs/tags/release-0.13.0-rc0
> > >> >> >
> > >> >> > Test binaries have been published to Maven's staging repository,
> > and
> > >> are
> > >> >> > available here:
> > >> >> > https://repository.apache.org/content/repositories/
> > >> orgapachesamza-1020
> > >> >> >
> > >> >> > 127 issues were resolved for this release:
> > >> >> > https://issues.apache.org/jira/issues/?jql=project%20%
> > >> >> > 3D%20SAMZA%20AND%20fixVersion%20in%20(0.13%2C%200.13.0)%
> > >> >> > 20AND%20status%20in%20(Resolved%2C%20Closed)
> > >> >> >
> > >> >> > The vote will be open for 72 hours (ending at 8:00PM Saturday,
> > >> >> 05/13/2017).
> > >> >> >
> > >> >> > Please download the release candidate, check the
> hashes/signature,
> > >> build
> > >> >> it
> > >> >> > and test it, and then please vote:
> > >> >> >
> > >> >> >
> > >> >> > [ ] +1 approve
> > >> >> >
> > >> >> > [ ] +0 no opinion
> > >> >> >
> > >> >> > [ ] -1 disapprove (and reason why)
> > >> >> >
> > >> >> >
> > >> >> > +1 from my side for the release.
> > >> >> >
> > >> >> > Cheers!
> > >> >> >
> > >> >>
> > >>
> > >>
> > >>
> > >> --
> > >> Nacho - Ignacio Solis - iso...@igso.net
> > >>
> >
> >
> >
> > --
> > Nacho - Ignacio Solis - iso...@igso.net
> >
>

Re: [DISCUSS] SEP-4: Adjunct Data Store for Unbounded DataSets

[Navina Ramesh (Apache)]

Thanks for trying 3 times, Wei. Sorry about the trouble. Not sure where the
problem lies. Looking forward to review your design.

Navina

On Tue, May 16, 2017 at 8:56 AM, Wei Song  wrote:

> Hey everyone,
>
> I created a proposal for SAMZA-1278
> , Adjunct Data Store
> for Unbounded DataSets, which introduces an automatic mechanism to store
> adjunct data for stream tasks.
>
> https://cwiki.apache.org/confluence/display/SAMZA/Adjunct+Da
> ta+Store+for+Unbounded+DataSets
>
> Please review and comments are welcome!
>
> For those who are not actively following the master branch, you may have
> more questions than others. Feel free to ask them here.
>
> P.S. this is the 3rd try, sent this last week, but apparently no one at
> Linkedin has received, including samza-dev here just to be sure.
>
> --
> Thanks,
> -Wei
>

Re: [VOTE] Apache Samza 0.13.0 RC0

[Navina Ramesh (Apache)]

I just changed the status of this JIRA to "BLOCKER" -
https://issues.apache.org/jira/browse/SAMZA-1128

This causes a bug in standalone deployment where any failure in the barrier
protocol stops the scheduled executorservice. Unfortunately,
CoordinationUtils creates its own scheduled executorservice, which is
incorrect. Scheduled ExecutorService is meant to be the working queue for
the ZkJobCoordinator. This needs to be fixed. Bharath already ran into this
bug during testing on Friday.

veto for this release candidate.

@Prateek/Jagadish:
I recommend sending a "non-vote, testing release candidate" for this
release until we complete all pending tasks (includes docs, tests etc). It
will also be useful to share the pending tasks and their progress. In case
you have already shared it, I might have missed it since some emails are
bouncing off my inbox.

Thanks!
Navina

On Sun, May 14, 2017 at 1:30 PM, Boris S  wrote:

> I think we need to add SAMZA-1286 and
> SAMZA-1279 to the release .
>
> On Wed, May 10, 2017 at 7:51 PM, Jagadish Venkatraman  >
> wrote:
>
> > This is a call for a vote on a release of Apache Samza 0.13.0. Thanks to
> > everyone who has contributed to this release. We are very glad to see
> some
> > new contributors and features in this release.
> >
> > The release candidate can be downloaded from here:
> > http://home.apache.org/~jagadish/samza-0.13.0-rc0/
> >
> > The release candidate is signed with pgp key AF81FFBF, which can be found
> > on keyservers:
> > http://pgp.mit.edu/pks/lookup?op=get&search=0xAF81FFBF
> >
> > The git tag is release-0.13.0-rc0 and signed with the same pgp key:
> > https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tag;h=
> > refs/tags/release-0.13.0-rc0
> >
> > Test binaries have been published to Maven's staging repository, and are
> > available here:
> > https://repository.apache.org/content/repositories/orgapachesamza-1020
> >
> > 127 issues were resolved for this release:
> > https://issues.apache.org/jira/issues/?jql=project%20%
> > 3D%20SAMZA%20AND%20fixVersion%20in%20(0.13%2C%200.13.0)%
> > 20AND%20status%20in%20(Resolved%2C%20Closed)
> >
> > The vote will be open for 72 hours (ending at 8:00PM Saturday,
> 05/13/2017).
> >
> > Please download the release candidate, check the hashes/signature, build
> it
> > and test it, and then please vote:
> >
> >
> > [ ] +1 approve
> >
> > [ ] +0 no opinion
> >
> > [ ] -1 disapprove (and reason why)
> >
> >
> > +1 from my side for the release.
> >
> > Cheers!
> >
>

Re: [DISCUSS] SEP-3: Heart-beat mechanism between JobCoordinator and all running containers

[Navina Ramesh (Apache)]

Abhishek,
Thanks for clarifying and updating the SEP.

Cheers!
Navina

On Wed, May 3, 2017 at 8:20 PM, Jagadish Venkatraman  wrote:

> Navina,
>
>
> >> The ContainerHeartbeatMonitor and the ContainerHeartbeatClient are both
> internal
> APIs and have a concrete implementations.
>
> More specifically, both of these are purely internal implementation classes
> (and have nothing to do with any pluggable public API that we expose)
>
> Best,
> Jagadish
>
> On Wed, May 3, 2017 at 7:34 PM, Abhishek Shivanna 
> wrote:
>
> > Hey Navina,
> >
> > Thank you for reviewing the SEP.
> >
> > > Are you planning on exposing this monitor class as a public api? What
> is
> > the significance of doing so?
> >
> > Sorry for the confusion of having implementation details under "public
> > interfaces".
> > The ContainerHeartbeatMonitor and the ContainerHeartbeatClient are both
> > internal APIs
> > and have a concrete implementations.
> >
> > > Is "Execution Container ID" the name of the environmental variable? I
> > don't
> > think environmental variables can contain whitespace??
> >
> > Again, confusion that stemmed from my initial draft. I have fixed the SEP
> > with the actual name in the implementation.
> >
> > > I think the first sentence corresponds to your design. The second one
> is
> > more of an implementation detail. You may want to split it up or just
> > discard one of them. I got confused reading them together because one
> talks
> > about adding to container and the other about the ContainerRunner.
> >
> > Fixed the SEP to make it more clear.
> >
> > Thanks,
> > Abhishek
> >
> >
> > On Wed, May 3, 2017 at 2:08 PM, Navina Ramesh (Apache) <
> nav...@apache.org>
> > wrote:
> >
> > > Hi Abhishek,
> > > I checked your latest proposal in SEP and it looks good to me.
> > >
> > > QQ:
> > > > A new ContainerHeartbeatMonitor class that accepts a
> > > ContainerHeartbeatClient (which has the business logic to make
> heartbeat
> > > checks on the JC endpoint) and a callback.
> > >
> > > Are you planning on exposing this monitor class as a public api? What
> is
> > > the significance of doing so?
> > >
> > > > set an environment variable with the "Execution Container ID" during
> > > container launch. This can be read from the container to make requests
> to
> > > the above endpoint.
> > >
> > > Is "Execution Container ID" the name of the environmental variable? I
> > don't
> > > think environmental variables can contain whitespace??
> > >
> > > > On the container side we start a new thread that periodically polls
> > this
> > > endpoint described above to check if the container is valid. If its
> not,
> > we
> > > shutdown the run loop and raise an error (so that the exit code is non
> 0
> > so
> > > that YARN reschedules the container)
> > > The plan is to setup a monitor in the LocalContainerRunner class that
> > > schedules a thread to check the above endpoint at regular intervals. On
> > > failure the thread modifies state on the LocalContainerRunner to denote
> > > that there was an error. This state is checked during exit in the
> > > LocalContainerRunner to exit with a non-zero code.
> > >
> > > I think the first sentence corresponds to your design. The second one
> is
> > > more of an implementation detail. You may want to split it up or just
> > > discard one of them. I got confused reading them together because one
> > talks
> > > about adding to container and the other about the ContainerRunner.
> > >
> > > Design looks pretty elegant and easily portable.
> > >
> > > Thanks!
> > > Navina
> > >
> > >
> > > On Wed, May 3, 2017 at 9:52 AM, Abhishek Shivanna 
> > > wrote:
> > >
> > > > Hey Jagadish,
> > > >
> > > > Thank you for taking the time to review the design.
> > > > I agree with moving the heartbeat into the the LocalContainerRunner
> > > instead
> > > > of fitting it into the SamzaContainer. I will update the SEP with the
> > new
> > > > design changes.
> > > > Also agree with the changes to the configuration and choosing
> suitable
> > > > defaults should be good enough.
> > > >
> > > > Thanks,
> > > >

Re: [DISCUSS] SEP-3: Heart-beat mechanism between JobCoordinator and all running containers

[Navina Ramesh (Apache)]

Hi Abhishek,
I checked your latest proposal in SEP and it looks good to me.

QQ:
> A new ContainerHeartbeatMonitor class that accepts a
ContainerHeartbeatClient (which has the business logic to make heartbeat
checks on the JC endpoint) and a callback.

Are you planning on exposing this monitor class as a public api? What is
the significance of doing so?

> set an environment variable with the "Execution Container ID" during
container launch. This can be read from the container to make requests to
the above endpoint.

Is "Execution Container ID" the name of the environmental variable? I don't
think environmental variables can contain whitespace??

> On the container side we start a new thread that periodically polls this
endpoint described above to check if the container is valid. If its not, we
shutdown the run loop and raise an error (so that the exit code is non 0 so
that YARN reschedules the container)
The plan is to setup a monitor in the LocalContainerRunner class that
schedules a thread to check the above endpoint at regular intervals. On
failure the thread modifies state on the LocalContainerRunner to denote
that there was an error. This state is checked during exit in the
LocalContainerRunner to exit with a non-zero code.

I think the first sentence corresponds to your design. The second one is
more of an implementation detail. You may want to split it up or just
discard one of them. I got confused reading them together because one talks
about adding to container and the other about the ContainerRunner.

Design looks pretty elegant and easily portable.

Thanks!
Navina


On Wed, May 3, 2017 at 9:52 AM, Abhishek Shivanna  wrote:

> Hey Jagadish,
>
> Thank you for taking the time to review the design.
> I agree with moving the heartbeat into the the LocalContainerRunner instead
> of fitting it into the SamzaContainer. I will update the SEP with the new
> design changes.
> Also agree with the changes to the configuration and choosing suitable
> defaults should be good enough.
>
> Thanks,
> Abhishek
>
>
>
> On Wed, Apr 26, 2017 at 3:23 PM, Jagadish Venkatraman <
> jagadish1...@gmail.com> wrote:
>
> > Hi Abhishek,
> >
> > Heartbeat between the AM and container has been a long awaited Samza
> > feature. It will go a long way in ensuring our reliability! +1 for this
> > SEP.
> >
> > *High level comments:*
> >
> > Currently, the only use-case for the heartbeat mechanism seems to be when
> > running Samza on Yarn. IMHO, it makes sense to pull the heart beat logic
> > into the *LocalContainerRunner* instead of baking it into the
> > *SamzaContainer* class. Long term, we can re-visit this when we have a
> > pluggable liveness detection mechanism.
> >
> > I'm thinking of a flow like this:
> >
> > There is a separate component (or a thread) inside LocalContainerRunner
> > that periodically polls the coordinator, and determines if it should
> > continue running. If the coordinator determines that the container should
> > not run, the *LocalContainerRunner* cleanly shuts-down the container and
> > the process exits with a non-zero exit status.
> >
> > The following nice properties fall out:
> >
> >- We can remove the proposed config *job.container.validator.enabled.
> *
> >- We can also remove the proposed *Killable* interface since
> >*SamzaContainer* (and runLoops) don't have to implement *Killable *
> >anymore. The life-cycle is managed by the *LocalContainerRunner* that
> >started it.
> >
> > *On the proposed public interfaces:*
> >
> > *job.container.validator.enabled:  *I am not in favor of adding this as
> a
> > new public config. IIUC, When running Samza jobs on Yarn, we always want
> > the validator/heartbeats to be enabled. OTOH, when running Samza jobs in
> > standalone mode, we currently do not have a pluggable mechanism for
> > heartbeat.
> >
> > *job.container.schedule.ms : *It does
> > seem that we can pick a sensible default, and be done with it (instead of
> > adding a new config)? Is there a reason this needs to be configurable?
> >
> > *On proposed Killable interface: *
> >
> > Not entirely sure we need this new "*Killable"* interface (esp. given
> that
> > there's currently only one implementation - *SamzaContainer*).
> >
> >- The *LocalContainerRunner* can instead directly invoke shut-down on
> >the *SamzaContainer* when its heart-beat expires. The extra level of
> >indirection (making *SamzaContainer* to implement *Killable*) is
> >probably unnecessary IMHO.
> >
> >
> >- Since, the *LocalContainerRunner* invokes *start/run* on the
> >*SamzaContainer*, it seems simpler also have it invoke *shutdown* on
> the
> >*SamzaContainer. *
> >
> > *Minor Comments:*
> >
> > >> Expose a REST endpoint (eg: /isContainerValid) who's purpose is to get
> > requests from the Samza container periodically and respond back weather
> the
> > container is in the Job Coordinator's current list of valid containers.
> >
> > Wondering if i

[RESULT] [VOTE] SEP-1: Semantics of ProcessorId in Samza

[Navina Ramesh (Apache)]

Hi everyone,

The vote on SEP-1 passes with 7 +1 votes (3 binding) and no -1.

Votes are as follows:
+1 (binding) - Navina Ramesh, Yi Pan, Yan Fang
+1 (non-binding) - Boris Shkolnik, Xinyu Liu, Renato Marroquin Mogrovejo,
Ignacio Solis

The following are the discuss and vote mail threads:
DISCUSS mail thread -
http://mail-archives.apache.org/mod_mbox/samza-dev/201703.mbox/%3CCANazzuuHiO%3DvZQyFbTiYU-0Sfh3riK%3Dz4j_AdCicQ8rBO%3DXuYQ%40mail.gmail.com%3E

VOTE mail thread -
http://mail-archives.apache.org/mod_mbox/samza-dev/201703.mbox/%3CCANazzutAX23PYv3%2BN%2BGkXbDTrF0kvRG5aHRDifX5rJ%3Din0VtzA%40mail.gmail.com%3E

Thanks to everyone who participated.

Cheers!
Navina

Re: [VOTE] SEP-1: Semantics of ProcessorId in Samza

[Navina Ramesh (Apache)]

you have a different perspective on this.
> >> > > >
> >> > > > Cheers!
> >> > > > Navina
> >> > > >
> >> > > > On Thu, Mar 30, 2017 at 9:42 AM, Yi Pan 
> wrote:
> >> > > >
> >> > > > > @Navina,
> >> > > > >
> >> > > > > Sorry to chime in late. One question:
> >> > > > > 1. Why is it in JobCoordinator, and why not in StreamProcessor
> >> class?
> >> > > > > Because JobCoordinator provides coordination service across many
> >> > > > > processors, an interface getProcessorId() in JobCoordinator is
> >> > > confusing
> >> > > > > regarding to which processorId we are getting.
> >> > > > >
> >> > > > > Otherwise, the proposal looks good.
> >> > > > >
> >> > > > > -Yi
> >> > > > >
> >> > > > > On Wed, Mar 29, 2017 at 7:57 PM, Navina Ramesh
> >> > > > >  >> > > > > > wrote:
> >> > > > >
> >> > > > > > Good to hear from you, Yan. Thanks! :)
> >> > > > > >
> >> > > > > > On Wed, Mar 29, 2017 at 7:48 PM, Yan Fang <
> yanfang...@gmail.com>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > +1 . Thanks for the proposal, Navina. :)
> >> > > > > > >
> >> > > > > > > Fang, Yan
> >> > > > > > > yanfang...@gmail.com
> >> > > > > > >
> >> > > > > > > On Thu, Mar 30, 2017 at 4:24 AM, Prateek Maheshwari <
> >> > > > > > > pmaheshw...@linkedin.com.invalid> wrote:
> >> > > > > > >
> >> > > > > > > > +1 (non binding) from me.
> >> > > > > > > >
> >> > > > > > > > - Prateek
> >> > > > > > > >
> >> > > > > > > > On Tue, Mar 28, 2017 at 2:17 PM, Boris S <
> bor...@gmail.com>
> >> > > wrote:
> >> > > > > > > >
> >> > > > > > > > > +1 Looks good to me.
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Mar 28, 2017 at 2:00 PM, xinyu liu <
> >> > > > xinyuliu...@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > +1 on my side. Very happy to see this proposal. This
> is a
> >> > > > blocker
> >> > > > > > for
> >> > > > > > > > > > integrating fluent API with StreamProcessor, and
> >> hopefully
> >> > we
> >> > > > can
> >> > > > > > get
> >> > > > > > > > it
> >> > > > > > > > > > resolved soon :).
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Xinyu
> >> > > > > > > > > >
> >> > > > > > > > > > On Tue, Mar 28, 2017 at 11:28 AM, Navina Ramesh
> (Apache)
> >> <
> >> > > > > > > > > > nav...@apache.org>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi everyone,
> >> > > > > > > > > > >
> >> > > > > > > > > > > This is a voting thread for SEP-1: Semantics of
> >> > ProcessorId
> >> > > > in
> >> > > > > > > Samza.
> >> > > > > > > > > > > For reference, here is the wiki link:
> >> > > > > > > > > > > https://cwiki.apache.org/
> confluence/display/SAMZA/SEP-
> >> > > > > > > > > > > 1%3A+Semantics+of+ProcessorId+in+Samza
> >> > > > > > > > > > >
> >> > > > > > > > > > > Link to discussion mail thread:
> >> > > > > > > > > > > http://mail-archives.apache.
> >> > org/mod_mbox/samza-dev/201703.
> >> > > > > > > > > > > mbox/%3CCANazzuuHiO%3DvZQyFbTiYU-0Sfh3riK%3Dz4j_
> >> > > > > > > > > > AdCicQ8rBO%3DXuYQ%40mail.
> >> > > > > > > > > > > gmail.com%3E
> >> > > > > > > > > > >
> >> > > > > > > > > > > Please vote on this SEP asap. :)
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks!
> >> > > > > > > > > > > Navina
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Navina R.
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Navina R.
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Navina R.
> >> >
> >>
> >
> >
> >
> > --
> > We are hiring in Streams Infra (Kafka/Samza/Datastream) !!
>
>
>
> --
> Nacho - Ignacio Solis - iso...@igso.net
>

[VOTE] SEP-1: Semantics of ProcessorId in Samza

[Navina Ramesh (Apache)]

Hi everyone,

This is a voting thread for SEP-1: Semantics of ProcessorId in Samza.
For reference, here is the wiki link:
https://cwiki.apache.org/confluence/display/SAMZA/SEP-1%3A+Semantics+of+ProcessorId+in+Samza

Link to discussion mail thread:
http://mail-archives.apache.org/mod_mbox/samza-dev/201703.mbox/%3CCANazzuuHiO%3DvZQyFbTiYU-0Sfh3riK%3Dz4j_AdCicQ8rBO%3DXuYQ%40mail.gmail.com%3E

Please vote on this SEP asap. :)

Thanks!
Navina

Re: Steps to Upgrading Samza (0.9 to 0.12)

[Navina Ramesh (Apache)]

@Jake: Yes. We removed the migration code (for 0.9 to 0.10) in the 0.11
release, I believe.

@XiaoChuan: As per Jagadish's recommendation, if you have changelog backed
stores, you should upgrade from 0.9.1 to 0.10.0 before upgrading to samza
0.12.0.

I checked with LinkedIn's internal release notes. The most significant
change listed is adding a new configuration *job.coordinator.system*. This
system can be the same as your currently configured checkpoint system
(task.checkpoint.system). I am assuming you are using
KafkaCheckpointManagerFactory. If you are using other custom checkpoint
managers, the migration may be more involved. Please let us know and we can
try to help you out.

Feel free to email us if you have more questions.

Cheers!
Navina

On Mon, Mar 27, 2017 at 10:07 AM, Jagadish Venkatraman <
jagadish1...@gmail.com> wrote:

> Good observation Jake!
>
> The code for migration was removed in Samza 11. The migration would read
> change-log offsets from the checkpoint topic and write them to the
> coordinator stream.
>
> If you're using change-logged stores, I'd recommend upgrading from 0.9.1 to
> 0.10.0 first.
> Otherwise, you will loose offsets for change-logged stores.
>
> I suspect you should be okay for 0.10.0 to 0.12 upgrade.
>
> On Mon, Mar 27, 2017 at 9:30 AM, Jacob Maes  wrote:
>
> > As I recall, samza 0.10 introduced the coordinator stream and there was
> > code to do an automatic migration to use that feature. @navina, @yi, do
> you
> > know if that migration code is still in samza 12?
> >
> > If not, then it's probably better to update from 0.9.1 to 0.10.0 and then
> > to 0.12.0. I don't think there were any changes requiring migration
> between
> > 0.10.and 0.12, so upgrading directly from 0.10 to 0.12 is probably less
> of
> > an issue.
> >
> > On Fri, Mar 24, 2017 at 11:05 PM, Jagadish Venkatraman <
> > jagadish1...@gmail.com> wrote:
> >
> > > Hi Xiaochuan,
> > >
> > > >> Do I need to upgrade Kafka and/or YARN?
> > >
> > > *Yarn version:*
> > >
> > >- Samza 0.12 supports Yarn 2.6.1 and 2.7.1.
> > >- If you already have 2.6.0 installed (as you have said), I believe
> > you
> > >will be fine. (but I'm not sure)
> > >
> > > *Kafka version: *
> > >
> > >- Samza 0.12 upgraded the version of Kafka to 0.10.
> > >- If your Kafka brokers are on an older version of Kafka, you should
> > >upgrade them to use at-least 0.10. Kafka clients are usually
> > >incompatible with older versions of brokers.
> > >
> > > *Java version: *
> > >
> > >
> > >
> > >- Samza 0.12 binaries are compiled using Java 8.  Hence, they cannot
> > be
> > >run on older versions of the Java run-time.
> > >
> > >
> > > >> I'm extremely new to Samza in terms of operations aspect. I'm not
> sure
> > > what
> > > information would be relevant in this case so please ask away.
> > >
> > > I'd first start by upgrading the Kafka brokers (assuming you're on Java
> > 8+
> > > already).
> > > Let us know how the migration goes!
> > >
> > > Thanks,
> > > Jagadish
> > >
> > >
> > > On Fri, Mar 24, 2017 at 8:23 PM, XiaoChuan Yu 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > What are the general steps for upgrading Samza from 0.9 to 0.12?
> > > > Do I need to upgrade Kafka and/or YARN?
> > > >
> > > > I don't know how Samza was setup initially but we currently have the
> > > > following setup:
> > > >
> > > > Samza version: 0.9.1
> > > > YARN version: Hadoop 2.6.0-cdh5.4.8
> > > > Kafka version: 0.9.0.1
> > > >
> > > > I think installation of Kafka and YARN were managed through Puppet.
> > > > I'm extremely new to Samza in terms of operations aspect. I'm not
> sure
> > > what
> > > > information would be relevant in this case so please ask away.
> > > >
> > > > Thanks,
> > > > Xiaochuan Yu
> > > >
> > >
> > >
> > >
> > > --
> > > Jagadish V,
> > > Graduate Student,
> > > Department of Computer Science,
> > > Stanford University
> > >
> >
>
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
>

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

[Navina Ramesh (Apache)]

Hi everyone,
I have updated the SEP
<https://cwiki.apache.org/confluence/display/SAMZA/SEP-1%3A+Semantics+of+ProcessorId+in+Samza>
based
on all the feedback. Feel free to comment.

I will start the [vote] mail thread, if there are no further questions
within the next 24 hours.

Thanks!
Navina

On Tue, Mar 21, 2017 at 10:33 AM, Navina Ramesh (Apache) 
wrote:

> Hi Jagadish,
> Thanks for the suggestion. You are right in that it should be the
> responsibility of the JobCoordinator to assign identifiers.
>
> > 'm only wondering if this logic could instead reside inside the
> Job Coordinator (which is internal to the StreamProcessor) instead of
> relying on something external to it?
>
> I think this is a consequence of our initial StandaloneJobCoordinator,
> which is pretty much a pass-through. I didn't see any usage for
> getProcessorId() and was wondering why we put it in the JobCoordinator
> interface. I think I should keep your design proposal from last year handy
> :) Thanks for pitching in!
>
>
> @All:
> Yesterday, there was a discussion on naming of the configuration used in
> this SEP - whether it should be within the "job" scope or "app" scope
> (introduced by SAMZA-1041
> <https://issues.apache.org/jira/browse/SAMZA-1041>).  Multi-stage feature
> and fluent-api for Samza introduces the notion of "application". Since the
> processorId generator config applies to all jobs within a Samza
> application, we decided to add the config for generator under "app" scope.
> Further details on config scope changes can be found in SAMZA-1120.
> <https://issues.apache.org/jira/browse/SAMZA-1120>
>
> I will send out an update once I change the SEP based on yesterday's
> meeting and Jagadish's idea.
>
> Thanks!
> Navina
>
> On Mon, Mar 20, 2017 at 5:22 PM, Jagadish Venkatraman <
> jagadish1...@gmail.com> wrote:
>
>> Thanks for writing this SEP!
>>
>> Here's an alternate approach instead of taking the "String processorId" as
>> a parameter in the constructor. In my view, the "processorId" could be
>> generated by the StreamProcessor internally (instead of being generated
>> up-stream and passed in). The Job Coordinator API could be as follows:
>>
>>
>> public interface JobCoordinator {
>>
>>  ProcessorIdGenerator getProcessorIDGenerator();
>>
>> // could be String getProcessorID()
>>
>>  JobModel getJobModel();
>>
>> }
>>
>> public interface ProcessorIDGenerator {
>>
>>  String getProcessorID();
>> }
>>
>>
>> For instance, an Yarn job coordinator can merely parse the ID from config,
>> and return it. A Zk backed implementation of the Job coordinator can agree
>> on IDs using coordination leveraging Zk. One nice property with this
>> approach is that it keeps all logic related to coordination, agreement on
>> the Job Model, leader election (with potentially pluggable components for
>> each) inside the JobCoordinator.
>>
>> To be clear, I'm all for pluggability for ID generation logic that this
>> SEP
>> advocates. I'm only wondering if this logic could instead reside inside
>> the
>> Job Coordinator (which is internal to the StreamProcessor) instead of
>> relying on something external to it?
>>
>> Of course, there may be other considerations around the way the current
>> code is structured that may prevent this. Let me know if you agree with
>> this change.
>>
>> Thanks,
>> Jag
>>
>>
>> On Thu, Mar 16, 2017 at 5:21 PM, Navina Ramesh
>> > > wrote:
>>
>> > > I am working on the ApplicationRunner SEP right now. Will send out the
>> > discussion email once I am done.
>> >
>> > Perfect! :)
>> >
>> > On Thu, Mar 16, 2017 at 5:13 PM, xinyu liu 
>> wrote:
>> >
>> > > Right, the static factory is very simple as you said. It's pretty
>> > > convenient for the client to use.
>> > >
>> > > I am working on the ApplicationRunner SEP right now. Will send out the
>> > > discussion email once I am done.
>> > >
>> > > Thanks,
>> > > Xinyu
>> > >
>> > > On Thu, Mar 16, 2017 at 4:50 PM, Navina Ramesh (Apache) <
>> > nav...@apache.org
>> > > >
>> > > wrote:
>> > >
>> > > > > One minor thing I found is that the name of the config is camel
>> case
>> > > > (*processor.idGenerator.class*). Seems Samza's practice is to use
>> all
>

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

[Navina Ramesh (Apache)]

Hi Jagadish,
Thanks for the suggestion. You are right in that it should be the
responsibility of the JobCoordinator to assign identifiers.

> 'm only wondering if this logic could instead reside inside the
Job Coordinator (which is internal to the StreamProcessor) instead of
relying on something external to it?

I think this is a consequence of our initial StandaloneJobCoordinator,
which is pretty much a pass-through. I didn't see any usage for
getProcessorId() and was wondering why we put it in the JobCoordinator
interface. I think I should keep your design proposal from last year handy
:) Thanks for pitching in!


@All:
Yesterday, there was a discussion on naming of the configuration used in
this SEP - whether it should be within the "job" scope or "app" scope
(introduced by SAMZA-1041 <https://issues.apache.org/jira/browse/SAMZA-1041>).
Multi-stage feature and fluent-api for Samza introduces the notion of
"application". Since the processorId generator config applies to all jobs
within a Samza application, we decided to add the config for generator
under "app" scope. Further details on config scope changes can be found in
SAMZA-1120. <https://issues.apache.org/jira/browse/SAMZA-1120>

I will send out an update once I change the SEP based on yesterday's
meeting and Jagadish's idea.

Thanks!
Navina

On Mon, Mar 20, 2017 at 5:22 PM, Jagadish Venkatraman <
jagadish1...@gmail.com> wrote:

> Thanks for writing this SEP!
>
> Here's an alternate approach instead of taking the "String processorId" as
> a parameter in the constructor. In my view, the "processorId" could be
> generated by the StreamProcessor internally (instead of being generated
> up-stream and passed in). The Job Coordinator API could be as follows:
>
>
> public interface JobCoordinator {
>
>  ProcessorIdGenerator getProcessorIDGenerator();
>
> // could be String getProcessorID()
>
>  JobModel getJobModel();
>
> }
>
> public interface ProcessorIDGenerator {
>
>  String getProcessorID();
> }
>
>
> For instance, an Yarn job coordinator can merely parse the ID from config,
> and return it. A Zk backed implementation of the Job coordinator can agree
> on IDs using coordination leveraging Zk. One nice property with this
> approach is that it keeps all logic related to coordination, agreement on
> the Job Model, leader election (with potentially pluggable components for
> each) inside the JobCoordinator.
>
> To be clear, I'm all for pluggability for ID generation logic that this SEP
> advocates. I'm only wondering if this logic could instead reside inside the
> Job Coordinator (which is internal to the StreamProcessor) instead of
> relying on something external to it?
>
> Of course, there may be other considerations around the way the current
> code is structured that may prevent this. Let me know if you agree with
> this change.
>
> Thanks,
> Jag
>
>
> On Thu, Mar 16, 2017 at 5:21 PM, Navina Ramesh
>  > wrote:
>
> > > I am working on the ApplicationRunner SEP right now. Will send out the
> > discussion email once I am done.
> >
> > Perfect! :)
> >
> > On Thu, Mar 16, 2017 at 5:13 PM, xinyu liu 
> wrote:
> >
> > > Right, the static factory is very simple as you said. It's pretty
> > > convenient for the client to use.
> > >
> > > I am working on the ApplicationRunner SEP right now. Will send out the
> > > discussion email once I am done.
> > >
> > > Thanks,
> > > Xinyu
> > >
> > > On Thu, Mar 16, 2017 at 4:50 PM, Navina Ramesh (Apache) <
> > nav...@apache.org
> > > >
> > > wrote:
> > >
> > > > > One minor thing I found is that the name of the config is camel
> case
> > > > (*processor.idGenerator.class*). Seems Samza's practice is to use
> all
> > > > lower
> > > > case configs with "." delimiter. Do you think we should stick to this
> > > > convention?
> > > >
> > > > I am always torn between the "convention" we have and the better way
> of
> > > > doing things. But I don't have strong opinions about it. I can change
> > it.
> > > >
> > > > > One more suggestion is to have a static factory method in the
> > > > ProcessorIdGenerator (Like what we have in ApplicationRunner):
> > > >
> > > > I couldn't grasp these requirements from the ApplicationRunner
> design.
> > It
> > > > will be great if you can put it out in an SEP :)
> > > >
> > > > I can add the static fac

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

[Navina Ramesh (Apache)]

> One minor thing I found is that the name of the config is camel case
(*processor.idGenerator.class*). Seems Samza's practice is to use all lower
case configs with "." delimiter. Do you think we should stick to this
convention?

I am always torn between the "convention" we have and the better way of
doing things. But I don't have strong opinions about it. I can change it.

> One more suggestion is to have a static factory method in the
ProcessorIdGenerator (Like what we have in ApplicationRunner):

I couldn't grasp these requirements from the ApplicationRunner design. It
will be great if you can put it out in an SEP :)

I can add the static factory method for it. Just to clarify, the static
method simply class loads the ProcessorIdGenerator ? It uses reflection to
create the instance ?

Thanks!
Navina



On Thu, Mar 16, 2017 at 4:31 PM, xinyu liu  wrote:

> The proposal looks great to me! Changing the id type to string will make
> sure this can work with other types of cluster which doesn't support
> integer id. The interface and config provides a pluggable way to have
> different id generators for different use cases. One minor thing I found is
> that the name of the config is camel case (*processor.idGenerator.class*).
> Seems Samza's practice is to use all lower case configs with "." delimiter.
> Do you think we should stick to this convention?
>
> One more suggestion is to have a static factory method in
> the ProcessorIdGenerator (Like what we have in ApplicationRunner):
>
> static ProcessIdGenerator fromConfig(Config config) { ... }.
>
> With this, It will be more convenient for the ApplicationRunner to
> construct the generator. What do you think?
>
> Thanks,
> Xinyu
>
> On Wed, Mar 15, 2017 at 10:59 PM, Navina Ramesh (Apache) <
> nav...@apache.org>
> wrote:
>
> > Hi everyone,
> > I created a proposal for SAMZA-1126, which addresses the semantics of
> > ProcessorId in Samza. For most purposes, ProcessorId is same as the
> logical
> > id that Samza assigns for each Yarn container. It is primarily used in
> > JobModel as a key for the corresponding ContainerModel and also, in
> > container-level metrics. We are expanding the applicability of
> processorId
> > to be beyond a fixed set of processors.
> >
> > Please review and comment on this SEP.
> >
> > For those who are not actively following the master branch, you may have
> > more questions than others. Feel free to ask them here.
> >
> > @Xinyu: Since you are working on SAMZA-1067 and other related integration
> > APIs, can you please add an SEP for SAMZA-1067 ? This will help others
> (adn
> > me as well) get on the same page with your design/code. Let me know if
> > SEP-1 will work per your design for ApplicationRunner.
> >
> > Thanks!
> > Navina
> >
>

[DISCUSS] SEP-1: Semantics of ProcessorId in Samza

[Navina Ramesh (Apache)]

Hi everyone,
I created a proposal for SAMZA-1126, which addresses the semantics of
ProcessorId in Samza. For most purposes, ProcessorId is same as the logical
id that Samza assigns for each Yarn container. It is primarily used in
JobModel as a key for the corresponding ContainerModel and also, in
container-level metrics. We are expanding the applicability of processorId
to be beyond a fixed set of processors.

Please review and comment on this SEP.

For those who are not actively following the master branch, you may have
more questions than others. Feel free to ask them here.

@Xinyu: Since you are working on SAMZA-1067 and other related integration
APIs, can you please add an SEP for SAMZA-1067 ? This will help others (adn
me as well) get on the same page with your design/code. Let me know if
SEP-1 will work per your design for ApplicationRunner.

Thanks!
Navina

[DISCUSS] SAMZA-1141 - Apache Samza Development Process Improvements

[Navina Ramesh (Apache)]

Hi everyone,

We switched to using Pull Requests for code reviews a few months back.
Clearly, there are some drawbacks to that model and we are trying to
address the shortcomings. I have gathered input from some of the committers
regarding what is missing the code review process and what can be improved.
Please take a look and provide feedback.

Additionally, we are considering moving to a KIP/FLIP-like model for
submitting design proposals (major changes to samza). Lately, there have
been some major feature discussions that are not documented consistently in
a centralized location. The proposal in SAMZA-1141
 address the design
review process as well. Please review it too. I have already created a wiki
page

describing the Samza Enhancement Proposal (SEP) process and an SEP
template. Going forward, let's start adding all major change proposals to
the wiki and discuss the design on the mailing list.

Your cooperation is highly appreciated during this period of transition in
the process :)

Feedbacks welcome!

Thanks!
-- 
Navina R

PS: Alternatives name suggestions for "SEP" are welcome !

Standalone Samza

[Navina Ramesh (Apache)]

Hi everyone,
I have uploaded the first draft of the Standalone Samza design.
SAMZA-1063  contains an
umbrella document describing the long-term goals of the standalone project.
SAMZA-1064  contains the
draft of the standalone design using zookeeper for coordination.

The approach we have take in the design differs quite a bit from the
previous approach to standalone SAMZA-516. We are targeting a more broader
spectrum of operating environment and features.

Appreciate any feedback on the overall goals and/or zookeeper-based design
for Samza.

Thanks!
Navina

Re: [DISCUSS] [VOTE] Apache Samza 0.11.0 RC0

[Navina Ramesh (Apache)]

Verified MD5, signature. Ran bin/check-all.sh on MacOS. My RHEL box is
broken. I think others have tested it on RHEL.

Hence, +1 (binding) from me!

**On a side note** I think we need to upgrade the gradle version used by
the bootstrap script to 2.6 or higher. At least, make sure that Samza
doesn't throw checkstyle errors when used with a higher version of gradle.
This should be followed-up after release. Hopefully, someone can volunteer
:)

Thanks for driving this release, Xinyu!

Cheers!
Navina

On Tue, Oct 4, 2016 at 12:45 PM, Jakob Homan  wrote:

> +1 binding.
>
> Verified MD5, checked code, built and tested.
>
> Good job, everyone.
> -Jakob
>
>
> On 4 October 2016 at 09:23, Jacob Maes  wrote:
> > +1 (non-binding)
> >
> > Downloaded and bin/check-all.sh on OSX
> > Downloaded, built and ran unit tests on RHEL
> >
> > I got a checkstyle error when I tried to run bin/check-all.sh on RHEL,
> but
> > I think it's something environmental.
> >
> > Looks good.
> >
> > On Mon, Oct 3, 2016 at 4:19 PM, Jagadish Venkatraman <
> jagadish1...@gmail.com
> >> wrote:
> >
> >> +1 from my side for the release (non-binding)
> >>
> >> On Mon, Oct 3, 2016 at 12:36 PM, Boris Shkolnik 
> wrote:
> >>
> >> > +1
> >> >
> >> > On Fri, Sep 30, 2016 at 1:39 PM, xinyu liu 
> >> wrote:
> >> >
> >> > > Subject correction: [VOTE] Apache Samza 0.11.0 RC0.
> >> > >
> >> > > Thanks,
> >> > > Xinyu
> >> > >
> >> > > On Fri, Sep 30, 2016 at 12:00 PM, xinyu liu 
> >> > wrote:
> >> > >
> >> > > > Hey all,
> >> > > >
> >> > > > This is a call for a vote on a release of Apache Samza 0.11.0.
> Thanks
> >> > to
> >> > > > everyone who has contributed to this release. We are very glad to
> see
> >> > > > some new contributors in this release.
> >> > > >
> >> > > > The release candidate can be downloaded from here:
> >> > > > http://home.apache.org/~xinyu/samza-0.11.0-rc0/
> >> > > >
> >> > > > The release candidate is signed with pgp key C31D7061, which can
> be
> >> > > found on
> >> > > > keyservers:
> >> > > > http://pgp.mit.edu/pks/lookup?op=get&search=0xC31D7061
> >> > > >
> >> > > > The git tag is release-0.11.0-rc0 and signed with the same pgp
> key:
> >> > > > https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tag;h=
> >> > > > refs/tags/release-0.11.0-rc0
> >> > > >
> >> > > > Test binaries have been published to Maven's staging repository,
> and
> >> > are
> >> > > > available here:
> >> > > > https://repository.apache.org/content/repositories/
> >> orgapachesamza-1013
> >> > > >
> >> > > > Note that the binaries were built with JDK7 without incident.
> >> > > >
> >> > > > 38 issues were resolved for this release:
> >> > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%
> >> > > > 20SAMZA%20AND%20fixVersion%20in%20(0.11%2C%200.11.0)%20AND%
> >> > > > 20status%20in%20(Resolved%2C%20Closed)
> >> > > >
> >> > > > The vote will be open for 72 hours ( end in 12:00pm Wednesday,
> >> > 10/05/2016
> >> > > > ).
> >> > > >
> >> > > > Please download the release candidate, check the hashes/signature,
> >> > build
> >> > > > it and test it, and then please vote:
> >> > > >
> >> > > >
> >> > > > [ ] +1 approve
> >> > > >
> >> > > > [ ] +0 no opinion
> >> > > >
> >> > > > [ ] -1 disapprove (and reason why)
> >> > > >
> >> > > >
> >> > > > +1 from my side for the release.
> >> > > >
> >> > > > Cheers!
> >> > > > Xinyu Liu
> >> > > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Jagadish V,
> >> Graduate Student,
> >> Department of Computer Science,
> >> Stanford University
> >>
>

Re: Samza kinesis implementation

[Navina Ramesh (Apache)]

Hi Shekar,

Last year, we had one of the Samza committers, Yan Fang, mentor a PhD
student - Renato ,as a part of the GSoC program, where they worked on
integrating Samza with Amazon Kinesis. I have cc'd the two of them so they
can provide you more context.
Here is the presentation from ApacheCon 2015 as result of their work -
http://www.slideshare.net/RenatoJavierMarroqun/apachecon-bigdata-europe-2015

We have an implementation at LinkedIn, although it has not been submitted
to open-source. I have cc'd Ryanne and Jason who work on the Kinesis
integration at LinkedIn.

I am afraid that's all the guidance I can offer for now. Please let us know
if you have any questions. We would to be happy to review your design and
code contribution!

Thanks!
Navina

On Tue, Sep 13, 2016 at 5:17 PM, Shekar Tippur  wrote:

> Hello,
>
> I am looking for direction on implementing samza over Kinesis.
> I see that jira ticket is in unresolved state.
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/SAMZA-489
>
> I also saw that with Samza release 0.10 this implementation is in place.
>
> Appreciate any pointers on this.
>
> Sent from my iPhone
>