Re: Kafka client.id collision

Hi David, I think this is expected to occur as a warning since we spin up all kafka clients with the same client-id, which is $job.name + $job.id. As Jagadish mentioned, it will be great if you can provide us the entire log so that we can take a look. As a side note for the samza contributors, I

Re: Samza Meetup

n attending as I am in the area, it'd be my first > in-person Samza meetup :D > But unfortunately I don't have anything to present this time :( > > > Renato M. > > 2017-07-18 23:46 GMT-07:00 Navina Ramesh (Apache) : > > > Hi Renato, > > > >

Re: Samza Meetup

Hi Renato, We are planning for mid-August as a tentative target for the next meetup. If you are interested in participating or speaking at the meetup, please let us know. Thanks! Navina On Tue, Jul 18, 2017 at 10:36 AM, Renato Marroquín Mogrovejo < renatoj.marroq...@gmail.com> wrote: > Hi Sa

Re: [VOTE] SEP-5: Enable partition expansion of input streams

After a lot of Q&A, let's get this done :) +1 (binding) Thanks! Navina On Tue, Jun 20, 2017 at 10:31 AM, xinyu liu wrote: > +1 (non-binding) on this design. > > To me the task-count based groupers should work well in practice for > partition expansion of system using hash for partitions, e.g.

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

at 3:52 PM, Dong Lin wrote: > Thanks much for the reply Navina. Please see my reply inline. > > On Wed, Jun 21, 2017 at 2:57 PM, Navina Ramesh (Apache) > > wrote: > > > Thanks to Jake, Dong and Kartik for keeping the discussion going. > > > > > Here are the pros

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

Thanks to Jake, Dong and Kartik for keeping the discussion going. > Here are the pros and cons of the extra re-partitioning stage in comparison to SEP-5. I think that is good summarization of pros/cons for the repartitioning stage based solution. Can you please include it in your SEP? It seems li

Re: [VOTE] Apache Samza 0.13.0 RC6

+1 (binding) Thanks to everyone for diligently testing out the RCs and getting this release out! Cheers! Navina On Thu, Jun 8, 2017 at 9:09 AM, Chris Riccomini wrote: > +1 (binding) > > On Wed, Jun 7, 2017 at 8:55 AM, Yi Pan wrote: > > > +1 (binding) > > build and ran all local integration te

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

consumer in Samza's container uses another way of subscription. Instead > of > > subscribing to the topic, the consumer in Samza's container subscribes to > > the specific partitions of the topic. In this case, if new partitions > have > > been added, Samza

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

if this, together with the answers in the previous > email, addresses all your questions? Thanks for taking time to review the > proposal. > > Regards, > Dong > > > > > > > > On Wed, May 24, 2017 at 11:15 PM, Dong Lin wrote: > > > Hey Navina, > >

[VOTE] Apache Samza 0.13.0 RC1

Hi everyone, This is a call for a vote on a release of Apache Samza 0.13.0. Thanks to everyone who has contributed to this release. We are very glad to see some new contributors and features in this release. The release candidate can be downloaded from here: *http://home.apache.org/~navina/samza-

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

Thanks for the SEP, Dong. I have a couple of questions to understand your proposal better: * Under motivation, you mention that "_We expect this solution to work similarly with other input system as well._", yet I don't see any discussion on how it will work with other input systems. That is, what

Re: [VOTE] Apache Samza 0.13.0 RC0

t look trivial. I am > on > > > it. > > > > > > Navina > > > > > > On May 14, 2017 23:10, "Ignacio Solis" wrote: > > > > > >> We should hold off until it is solved. How long will it take to fix > > this? >

Re: [DISCUSS] SEP-4: Adjunct Data Store for Unbounded DataSets

Thanks for trying 3 times, Wei. Sorry about the trouble. Not sure where the problem lies. Looking forward to review your design. Navina On Tue, May 16, 2017 at 8:56 AM, Wei Song wrote: > Hey everyone, > > I created a proposal for SAMZA-1278 > ,

Re: [VOTE] Apache Samza 0.13.0 RC0

I just changed the status of this JIRA to "BLOCKER" - https://issues.apache.org/jira/browse/SAMZA-1128 This causes a bug in standalone deployment where any failure in the barrier protocol stops the scheduled executorservice. Unfortunately, CoordinationUtils creates its own scheduled executorservic

Re: [DISCUSS] SEP-3: Heart-beat mechanism between JobCoordinator and all running containers

ne > is > > more of an implementation detail. You may want to split it up or just > > discard one of them. I got confused reading them together because one > talks > > about adding to container and the other about the ContainerRunner. > > > > Fixed the SEP to mak

Re: [DISCUSS] SEP-3: Heart-beat mechanism between JobCoordinator and all running containers

Hi Abhishek, I checked your latest proposal in SEP and it looks good to me. QQ: > A new ContainerHeartbeatMonitor class that accepts a ContainerHeartbeatClient (which has the business logic to make heartbeat checks on the JC endpoint) and a callback. Are you planning on exposing this monitor clas

[RESULT] [VOTE] SEP-1: Semantics of ProcessorId in Samza

Hi everyone, The vote on SEP-1 passes with 7 +1 votes (3 binding) and no -1. Votes are as follows: +1 (binding) - Navina Ramesh, Yi Pan, Yan Fang +1 (non-binding) - Boris Shkolnik, Xinyu Liu, Renato Marroquin Mogrovejo, Ignacio Solis The following are the discuss and vote mail threads: DISCUSS m

Re: [VOTE] SEP-1: Semantics of ProcessorId in Samza

an interface getProcessorId() in JobCoordinator is > >> > > confusing > >> > > > > regarding to which processorId we are getting. > >> > > > > > >> > > > > Otherwise, the proposal looks good. > >> > >

[VOTE] SEP-1: Semantics of ProcessorId in Samza

Hi everyone, This is a voting thread for SEP-1: Semantics of ProcessorId in Samza. For reference, here is the wiki link: https://cwiki.apache.org/confluence/display/SAMZA/SEP-1%3A+Semantics+of+ProcessorId+in+Samza Link to discussion mail thread: http://mail-archives.apache.org/mod_mbox/samza-dev/

Re: Steps to Upgrading Samza (0.9 to 0.12)

@Jake: Yes. We removed the migration code (for 0.9 to 0.10) in the 0.11 release, I believe. @XiaoChuan: As per Jagadish's recommendation, if you have changelog backed stores, you should upgrade from 0.9.1 to 0.10.0 before upgrading to samza 0.12.0. I checked with LinkedIn's internal release notes

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

na On Tue, Mar 21, 2017 at 10:33 AM, Navina Ramesh (Apache) wrote: > Hi Jagadish, > Thanks for the suggestion. You are right in that it should be the > responsibility of the JobCoordinator to assign identifiers. > > > 'm only wondering if this logic could instead reside

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

nce I am done. > > > > Perfect! :) > > > > On Thu, Mar 16, 2017 at 5:13 PM, xinyu liu > wrote: > > > > > Right, the static factory is very simple as you said. It's pretty > > > convenient for the client to use. > > > > &g

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

n is to have a static factory method in > the ProcessorIdGenerator (Like what we have in ApplicationRunner): > > static ProcessIdGenerator fromConfig(Config config) { ... }. > > With this, It will be more convenient for the ApplicationRunner to > construct the generator. What do you think? >

[DISCUSS] SEP-1: Semantics of ProcessorId in Samza

Hi everyone, I created a proposal for SAMZA-1126, which addresses the semantics of ProcessorId in Samza. For most purposes, ProcessorId is same as the logical id that Samza assigns for each Yarn container. It is primarily used in JobModel as a key for the corresponding ContainerModel and also, in c

[DISCUSS] SAMZA-1141 - Apache Samza Development Process Improvements

Hi everyone, We switched to using Pull Requests for code reviews a few months back. Clearly, there are some drawbacks to that model and we are trying to address the shortcomings. I have gathered input from some of the committers regarding what is missing the code review process and what can be imp

Standalone Samza

Hi everyone, I have uploaded the first draft of the Standalone Samza design. SAMZA-1063 contains an umbrella document describing the long-term goals of the standalone project. SAMZA-1064 contains t

Re: [DISCUSS] [VOTE] Apache Samza 0.11.0 RC0

Verified MD5, signature. Ran bin/check-all.sh on MacOS. My RHEL box is broken. I think others have tested it on RHEL. Hence, +1 (binding) from me! **On a side note** I think we need to upgrade the gradle version used by the bootstrap script to 2.6 or higher. At least, make sure that Samza doesn't

Re: Samza kinesis implementation

Hi Shekar, Last year, we had one of the Samza committers, Yan Fang, mentor a PhD student - Renato ,as a part of the GSoC program, where they worked on integrating Samza with Amazon Kinesis. I have cc'd the two of them so they can provide you more context. Here is the presentation from ApacheCon 20