Re: Samza 0.14.1 not correctly handling OffsetOutOfRangeException exception?

2018-08-24 Thread Yi Pan
Hi, Gaurav, Thanks for working on the patch for the problem. Could you open a ticket and PR for the change? The dev mailing list stripping off all attachments and is hard to follow if the change is not embedded as text in the email. And to Debraj's comment, yes, we are aware of it and are

Re: Samza 0.14.1 not correctly handling OffsetOutOfRangeException exception?

2018-08-27 Thread Yi Pan
; >> On Sat, Aug 25, 2018 at 5:56 AM Yi Pan wrote: >> >>> Hi, Gaurav, >>> >>> Thanks for working on the patch for the problem. Could you open a ticket >>> and PR for the change? The dev mailing list stripping off all attachment

Re: Samza/Yarn cluster having issue with OffsetOutOfRangeException

2018-08-27 Thread Yi Pan
Hi, Will, Can you check the description in SAMZA-1822 to see whether this is exactly the problem you encountered? We just submitted the fix today. Thanks! On Tue, Aug 21, 2018 at 9:12 AM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > Hi Will, > > Is the topic in question your

Welcome Wei Song and Srinivasulu Punuru as committers to Samza!

2018-07-24 Thread Yi Pan
Hi, all, We are happy to announce that Wei Song and Srinivasulu Punuru are officially Samza committers now! Welcome, Wei and Srini! -Yi

Apache Samza - April 2018

2018-04-11 Thread Yi Pan
## Description: - Apache Samza is a distributed stream processing engine that are highly configurable to process events from various data sources, including real-time messaging system (e.g. Kafka) and distributed file systems (e.g. HDFS). ## Issues: - No issues requires board

Stream Processing Meetup @LinkedIn tomorrow (3/21)

2018-03-20 Thread Yi Pan
Hi, all, Just a reminder that we will host another stream processing meetup @LinkedIn tomorrow at 6pm: https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/248309045/ Expect to see you there! -Yi

[VOTE] SEP-13: unified ApplicationDescriptor and ApplicationRunner APIs for high and low- level APIs in YARN and standalone deployment

2018-10-12 Thread Yi Pan
Hi, all, Given SAMZA-1789 has been reviewed and implemented, SEP-13 has been updated to the latest API classes as well. Please vote on whether there is further breaking changes needed in the API, or we can accept this proposal and seal it for 1.0.

Re: [VOTE] SEP-14: System and Stream Descriptors

2018-10-12 Thread Yi Pan
+1 (binding) from me. Thanks! On Fri, Oct 12, 2018 at 12:30 PM Prateek Maheshwari wrote: > Hi folks, > > Now that SAMZA-1804 has been implemented and reviewed, we've updated > SEP-14 with the latest APIs and design decisions. > > Please vote for accepting SEP-14 in its current form for the

Re: [VOTE] SEP-15: New Runtime Context API

2018-10-16 Thread Yi Pan
+1 (binding) This has been long-waited feature to allow us to have better control and access to shared object in different scope of context! -Yi On Mon, Oct 15, 2018 at 10:47 AM Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > +1 (binding) from my side. > > LGTM > > On Mon, Oct 15, 2018

[RESULT] [VOTE] SEP-13: unified ApplicationDescriptor and ApplicationRunner APIs for high and low- level APIs in YARN and standalone deployment

2018-10-16 Thread Yi Pan
te: > > > > > +1 (non-binding) from me. Thanks for making the changes and updating > the > > > SEP! > > > > > > - Prateek > > > > > > On Fri, Oct 12, 2018 at 12:15 PM Yi Pan wrote: > > > > > > > > Hi, all, > &g

Re: [VOTE] Apache Samza 1.0.0 RC2

2018-10-24 Thread Yi Pan
Ran check-all and deployed locally with the test jobs. All tests passed. +1 (binding) from my end. Thanks for push the release! -Yi On Wed, Oct 24, 2018 at 8:53 AM Prateek Maheshwari wrote: > Hi Jagadish, > > PR 755 is mis-titled. Its only adding back the tests for the old > consumer. The

Re: [VOTE] Apache Samza 1.0.0 RC4

2018-11-05 Thread Yi Pan
Ran check-all and deployed locally with the test jobs. All tests passed. +1 (binding) from my end. Thank you all! -Yi On Fri, Nov 2, 2018 at 6:02 PM Jagadish Venkatraman wrote: > Extending the vote till Monday 4PM so that more folks can try it out. > > Thanks, > Jagadish > > On Fri, Nov 2,

Welcome Prateek Maheshwari to Apache Samza PMC

2018-11-01 Thread Yi Pan
Hi, all, Please join me to welcome Prateek Maheshwari to Apache Samza PMC! Great to have you in the PMC, Prateek! Cheers! -Yi

Welcome Hai Lu and Aditya Toomla as committers to Apache Samza!

2018-11-06 Thread Yi Pan
Hi, all, All official steps are completed and please join me to welcome Hai and Aditya to Apache Samza community as committers! They have been making significant contribution to many important projects in Samza such as SQL, Samza-on-Hadoop, Kinesis connector, etc. Welcome Hai and Aditya! -Yi

Re: [VOTE] [RESULT] Apache Samza 1.0.0 RC4

2018-11-07 Thread Yi Pan
Awesome! Jagadish, let me know when the blog is ready. Cheers! -Yi On Tue, Nov 6, 2018 at 9:33 PM Jagadish Venkatraman wrote: > The vote of 1.0.0 has been more than 72 hours. We got +1 (binding)x 3 and > +1 (non-binding) x 3 and no vetos. > > *Binding +1: Prateek M, Jagadish V, Yi

Re: Open JDK 11 Early Results

2018-10-02 Thread Yi Pan
. > > > > Jeremiah Adams > Software Engineer > www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > > From: Yi Pan > Sent: Tuesday, September 25, 2018 9:09 AM > To: dev@samza.apache.org > Subject: Re: Java 11,

Re: Upcoming Streams Processing Meetup @ LinkedIn on 10/23

2018-10-01 Thread Yi Pan
Sorry. correction on the link: https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/255016589/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJDcwMTNkNTNjLTRkNjUtNDEyMi1hM2UzLWU5ZjExNzFmMTFjMg On Mon, Oct 1, 2018 at 3:31 PM Yi Pan wrote: > Hi, all, > > We are happy to announce an

Upcoming Streams Processing Meetup @ LinkedIn on 10/23

2018-10-01 Thread Yi Pan
Hi, all, We are happy to announce another Streams Processing Meetup @ LinkedIn with a very exciting agenda, including CruiseControl for streams, Streams Analytics, and Operation of Samza in LinekdIn: https://mail.google.com/mail/u/0/#search/stream+meetup/FMfcgxvzKktTnTBsbwSHfxlTHcVmPhxK . Please

Re: Samza 1.0 Release

2018-09-24 Thread Yi Pan
+1 for 1.0! Thanks Rayman for sending out this email! P.S. @Tom, we are in the path to move closer to Kafka 1.0 by removing all deprecated SimpleConsumer client usage in 1.0. We will definitely keeping that target in mind after we stabilize the major features and APIs in this release. Thanks for

Re: Java 11, Oracle Java and OpenJDK Release Cadence

2018-09-25 Thread Yi Pan
Hey, Jeremiah, Sorry to reply late. The team is pretty busy for Samza 1.0 preparation these days. Regarding to Java 11 and the steps forward: 1. I don't have objection to adopt OpenJDK, as long as Samza code and compile and run on it. It might be an interesting issue since Samza also depends on

Re: [DISCUSS] Mandatory migration of Samza git repo to gitbox.apache.org

2019-01-15 Thread Yi Pan
+1 (binding) for the move as well. @Jake Maes I think our merge script requires setting up a local alias 'apache-samza' for the remote repo. Hence, it should be a simple step to point the 'apache-samza' repo to the new repo on gitbox, hopefully. -Yi On Tue, Jan 15, 2019 at 12:12 PM Jake Maes

[REPORT] Samza - January 2019

2019-01-10 Thread Yi Pan
## Description: - Apache Samza is a distributed stream processing engine that are highly configurable to process events from various data sources, including real-time messaging system (e.g. Kafka) and distributed file systems (e.g. HDFS). ## Issues: - No issues requires board attention

Re: Draft report to board - Jan 2019

2019-01-09 Thread Yi Pan
ce this was an intern project and is not checked in to > master. Otherwise LGTM. > > Thanks, > Prateek > > On Wed, Jan 9, 2019 at 12:28 PM Yi Pan wrote: > > > > Hi, all, > > > > Our quarterly report is due this Wed (1/9). The following is the draft > > repo

Draft report to board - Jan 2019

2019-01-09 Thread Yi Pan
Hi, all, Our quarterly report is due this Wed (1/9). The following is the draft report. Please let me know by the end of the day if I missed anything. Thanks! ## Description: - Apache Samza is a distributed stream processing engine that are highly configurable to process events from

Re: Change log level in samza application

2018-09-17 Thread Yi Pan
Hi, Debraj, Samza application's log level is controlled by the log4j.xml configure files packaged together with binary tgz file. In YARN, it is usually under ${YARN_INSTALL_DIR}/usercache/appcache/${APP_ATTEMPT_NAME}/${YARN_CONTAINER_ID}/__package/lib. Changing log level usually would mean that

Re: Change log level in samza application

2018-09-17 Thread Yi Pan
is to change the log level via JMX, since Samza provides JmxServer on containers already. -Yi On Mon, Sep 17, 2018 at 9:57 AM Debraj Manna wrote: > Thanks for replying. > > Is there a way to dynamically change this for running containers in YARN? > > On Mon 17 Sep, 2018, 10:19 PM Yi Pan,

Re: problem building samza

2018-12-09 Thread Yi Pan
HI, blitzerr, I don't think that Samza's official build supports jdk11 yet (please see some earlier discussion in the dev list regarding to Openjdk11. You would probably still need to set your compilation jdk to Java8 and run: gradle -b bootstrap.gradle to setup the gradlew wrapper for Samza.

Re: [VOTE] Migration of Samza git repo to gitbox.apache.org

2019-01-24 Thread Yi Pan
+1 (binding). Thanks a lot! On Thu, Jan 24, 2019 at 6:02 PM Boris S wrote: > +1. Thanks! > > On Thu, Jan 24, 2019 at 8:53 AM santhosh venkat < > santhoshvenkat1...@gmail.com> wrote: > > > +1 (non-binding). > > > > On Thu, Jan 24, 2019 at 7:10 AM Jake Maes wrote: > > > > > +1 (binding) > > > >

Re: Context when Converting from Yarn Task to Standalone LocalApplicationRunner

2019-04-03 Thread Yi Pan
Hi, Jeremiah, In the new apis, you should be using ApplicationContainerContextFactory and ApplicationTaskContextFactory to instantiate context objects used in the whole container or in a task instance, respectively. The context factories should be implemented as dependencies injected to your

Re: Trouble running samza 0.14 in standalone mode

2019-03-06 Thread Yi Pan
Hi, Anoop, 1. Please provide the full log file if possible. Just listing out a single log line reporting the failure does not help. 2. Is there any reason that you still stay with Samza 0.14? I would highly recommend to upgrade to 1.0 since there are tons of API and standalone related

Re: [DISCUSS] Samza 1.1.0 release

2019-03-06 Thread Yi Pan
+1 (binding) On Wed, Mar 6, 2019 at 10:08 PM Daniel Chen wrote: > Hello everyone, > > We have added couple of major features to master since 1.0.0 that warrants > a major release. > > Within LinkedIn, some of these features have already been tested as part of > our test suites. We plan to

Re: Trouble running samza 0.14 in standalone mode

2019-03-07 Thread Yi Pan
ring logs and code snippets. > > Issue is resolved. I was using the default task.name.grouper.factory > instead of GroupByContainerIdsFactory. > Appreciate and thanks for the response. I will consider using 1.0 version. > > -anoop > > > On Thu, 7 Mar 2019 at 01:41, Yi P

Re: code cleanups

2019-02-04 Thread Yi Pan
Hi, Andrey, Thanks for bring this up. See my response below: On Mon, Feb 4, 2019 at 4:54 AM Andrey Paykin wrote: > > 1. Is this idea make sense (code cleanup) > Yes, totally make sense. > 2. Do I need JIRA ticket(s)? May be there is some example ticket or > additional requirements? > I

Fwd: [REPORT] Samza - April 2019

2019-04-11 Thread Yi Pan
-- Forwarded message - From: Yi Pan (Data Infrastructure) Date: Thu, Apr 11, 2019 at 9:16 PM Subject: [REPORT] Samza - April 2019 To: Cc: ## Description: - Apache Samza is a distributed stream processing engine that are highly configurable to process events from various data

Re: [ANNOUNCE] Please welcome Boris Shkolnik to the Samza PMC

2019-06-09 Thread Yi Pan
Welcome and well deserved, Boris! -Yi On Sat, Jun 8, 2019 at 3:28 PM Hai Lu wrote: > Congratulations, Boris! > > On Fri, Jun 7, 2019 at 6:13 PM Aditya wrote: > > > Congrats Boris! > > > > > On Jun 7, 2019, at 4:58 PM, Weiqing Yang > > wrote: > > > > > > Congrats, Boris! > > > > > > On Fri,

Re: Tracing the Samza+YARN startup process

2019-06-19 Thread Yi Pan
Great and detailed report! Really appreciate it! -Yi On Tue, Jun 18, 2019 at 2:37 PM Malcolm McFarland wrote: > Just want to follow up on this, for anybody that might be trying to do > something similar. > > There are two things that were getting in the way of us using YARN+Samza on > ECS: 1)

Re: REMINDER. [VOTE] Apache Samza 1.2.0 RC4

2019-06-10 Thread Yi Pan
+1 (binding), verified signature and built successfully. One more: can we make sure that we address SAMZA-2064? The Infra team has mentioned this issue multiple times in the feedbacks to our report. Thanks! -Yi On Thu, Jun 6, 2019 at 5:59 PM Jagadish Venkatraman wrote: > +1 (binding) > >

Re: Apache Samza 1.2.0 is released

2019-06-12 Thread Yi Pan
Thanks, Boris! -Yi On Wed, Jun 12, 2019 at 1:17 PM Boris Shkolnik wrote: > Documentation and Blog are published. > Thanks everyone. >

Re: [DISCUSS] Hygene for merging PRs

2019-05-19 Thread Yi Pan
Hi, Cameron, That's generally the case. Thanks for Xinyu to bring this to our attention! +1 to the stated guidelines. -Yi On Fri, May 17, 2019 at 11:10 AM Cameron Lee wrote: > Thanks Xinyu for starting this thread. > I support the guidelines that you mentioned, with a couple clarifications >

Draft July report for Samza

2019-07-11 Thread Yi Pan
Hi, all, Here is the draft report I had for Samza this quarter. Please let me know if I miss anything. Thanks! ## Description: - Apache Samza is a distributed stream processing engine that are highly configurable to process events from various data sources, including real-time messaging

Re: [ANNOUNCE] New committer announcement: Cameron Lee

2019-04-17 Thread Yi Pan
Congrats Cameron!!! On Wed, Apr 17, 2019 at 6:12 AM Daniel Nishimura wrote: > Congrats! > > Sent from my iPhone > > > On Apr 16, 2019, at 9:22 PM, Jagadish Venkatraman < > jagadish1...@gmail.com> wrote: > > > > Awesome addition! Congrats Cameron. well deserved. > > > >> On Tuesday, April 16,

Re: Questions about using custom groupers

2019-09-30 Thread Yi Pan
HI, Malcolm, The configuration should be in *_coordinator_* topic. If your configuration of cleanup policy for this topic is compact only, you should not lose the configuration. If your configuration on this topic is a combination of compact + time retention (i.e. newer Kafka version on the

Re: [ANNOUNCE] Welcoming Rayman and Bharath as new Samza committers!

2019-07-09 Thread Yi Pan
Big congrats to Ray and Bharath! Looking forward to more PRs from you guys! -Yi On Tue, Jul 9, 2019 at 9:54 AM Weiqing Yang wrote: > Congrats Rayman and Bharath! > > - Weiqing > > On Mon, Jul 8, 2019 at 10:55 PM Yang Zhang wrote: > > > Congratulations! > > > > Best, > > Yang > > > > On Mon,

Re: [VOTE] SEP-20: Samza on Kubernetes

2019-11-08 Thread Yi Pan
+1 (binding) On Thu, Nov 7, 2019 at 7:38 PM Jagadish Venkatraman wrote: > +1 binding. > > Thanks Weiqing for driving this! > > On Thursday, November 7, 2019, Xinyu Liu wrote: > > > +1 (binding). > > > > Thanks, > > Xinyu > > > > On Thu, Nov 7, 2019 at 10:50 AM Weiqing Yang > > wrote: > > > >

Re: [DISCUSS] SEP-20: Samza on Kubernetes

2019-11-05 Thread Yi Pan
+1! Great to see this coming through! -Yi On Mon, Nov 4, 2019 at 9:20 PM Jagadish Venkatraman wrote: > +1, look forward to Samza K8s integration :) > > On Monday, November 4, 2019, Xinyu Liu wrote: > > > +1 on the design. This is a great feature to allow Samza to expand its > > deployment to

[REPORT] Samza - November 2019

2019-11-10 Thread Yi Pan
## Description: - Apache Samza is a distributed stream processing engine that are highly configurable to process events from various data sources, including real-time messaging system (e.g. Kafka) and distributed file systems (e.g. HDFS). ## Issues: - No issues requires board attention ##

Re: [VOTE] SEP-23: Simplify Job Runner

2019-12-11 Thread Yi Pan
+1(binding). Thanks! -Yi On Wed, Dec 11, 2019 at 12:13 PM Prateek Maheshwari wrote: > +1 (binding). > > Thanks, > Prateek > > On Wed, Dec 11, 2019 at 11:50 AM Xinyu Liu wrote: > > > +1 (binding). This proposal will help future split deployment as well as > > make the deployment simple.

Re: [DISCUSS] SEP 25: PR Title and Description Guidelines

2019-12-16 Thread Yi Pan
+1 (binding). lgtm. Thanks! -Yi On Mon, Dec 16, 2019 at 8:08 AM Daniel Nishimura wrote: > +1. Thank Prateek for standardizing the PR process better. > > On Sun, Dec 15, 2019 at 10:55 PM Bharath Kumara Subramanian < > codin.mart...@gmail.com> wrote: > > > +1. Template looks good to me. > > It

Re: [VOTE] Apache Samza 1.3.0 RC2

2019-12-01 Thread Yi Pan
+1 (binding), verified the signature, built and local integration tests passed. Thanks! -Yi On Wed, Nov 27, 2019 at 2:49 PM Hai Lu wrote: > Hi, > > This is a call for a vote on a release of Apache Samza 1.3.0. Thanks to > everyone who has contributed to this release. > > The release candidate

Re: [VOTE] SEP 25: PR Title and Description Guidelines

2019-12-18 Thread Yi Pan
+1 (binding) On Wed, Dec 18, 2019 at 10:49 AM Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > +1 (non-binding). > > Thanks, > Bharath > > On Wed, Dec 18, 2019 at 10:42 AM Prateek Maheshwari > wrote: > > > Hi folks, > > > > This is a call for a vote on SEP 25: PR Title and

Re: [DISCUSS] 1.4 release

2020-02-28 Thread Yi Pan
sending out the VOTE thread today, and then release by > the end of next week. > Cameron > > On Thu, Feb 27, 2020 at 9:08 PM Yi Pan wrote: > > > Hey, Cameron, > > > > Briefly browsed through the list and there are total 24 tickets tagged > with > > 1.4

Re: Samza - New Relic integration

2020-02-28 Thread Yi Pan
> > > Yes. That's exactly what I am looking for. > > > > Let me try it out and I will get back to you if I face any issues. > > > > Thanks again > > > > Vaibhav Garg > > +91-9505020924 > > vaibhavgar...@gmail.com > > LinkedIn &

Re: Samza - New Relic integration

2020-02-27 Thread Yi Pan
Hi, Vaibhav, Check the description of task.opts in the configuration doc here: http://samza.apache.org/learn/documentation/latest/jobs/samza-configurations.html Is this what you are looking for? On Thu, Feb 27, 2020 at 9:54 PM Vaibhav Garg wrote: > Hi, > > Any help here would be great. > >

Re: [DISCUSS] 1.4 release

2020-02-27 Thread Yi Pan
Hey, Cameron, Briefly browsed through the list and there are total 24 tickets tagged with 1.4 and 11 are assigned/in-progress/done. Are we targeting to finish all 24 for 1.4? And what's the proposed timeline for 1.4? Thanks! -Yi On Wed, Feb 26, 2020 at 1:25 PM Xinyu Liu wrote: > This is

Re: Samza consumer without Zookeeper

2020-02-05 Thread Yi Pan
Hi, Robert, Which version of Samza are you using? -Yi On Tue, Feb 4, 2020 at 9:17 AM Robert Wigginton wrote: > Hello, > > We are currently evaluating moving from an on prem Kafka deployment to > Confluent Cloud. Confluent Cloud does not expose Zookeeper to user. > Is it possible to setup a

Re: Samza consumer without Zookeeper

2020-02-05 Thread Yi Pan
the systems.x.consumer.zookeeper.connect from your config? -Yi On Wed, Feb 5, 2020 at 3:12 PM Robert Wigginton wrote: > Currently 1.0.0 but are in the process of upgrading to 1.3.0. > > On Wed, 2020-02-05 at 15:01 -0800, Yi Pan wrote: > > Hi, Robert, > > > > Which version of Samza are you using? > &

Re: [VOTE] Apache Samza 1.3.1 RC0

2020-02-18 Thread Yi Pan
Ran check-all and integration tests successfully. +1 (binding) On Thu, Feb 13, 2020 at 12:02 PM Hai Lu wrote: > Hi, > > This is a call for a vote on a release of Apache Samza 1.3.1 to redress > certain issues found in 1.3.0 > > The release candidate can be downloaded from here: >

Re: [ANNOUNCE] Please welcome Bharath Kumarasubramanian to the Samza PMC

2020-02-13 Thread Yi Pan
Congrats! Bharath, well deserved! -Yi On Thu, Feb 13, 2020 at 10:17 PM Wei Song wrote: > Congrats, Bharath, well deserved !!! > > > On 2/13/20, 8:51 PM, "Jagadish Venkatraman" > wrote: > > Congrats Bharath. Great work! Looking forward to continued > contributions! > > On Thursday,

Re: job.coordinator.replication.factor configuration

2020-02-15 Thread Yi Pan
Hi, Robert, Thanks for pointing this out. This is an error on the documentation side.The meaning of job.coordinator.replication.factor should not change post 1.0. I will file a bug against the doc site. Thanks! -Yi On Fri, Feb 14, 2020 at 7:52 AM Robert Wigginton wrote: > Hello, > > > >

Re: job.coordinator.replication.factor configuration

2020-02-15 Thread Yi Pan
JIRA opened: SAMZA-2460. Thanks! -Yi On Sat, Feb 15, 2020 at 9:42 AM Yi Pan wrote: > Hi, Robert, > > Thanks for pointing this out. This is an error on the documentation > side.The meaning of job.coordinator.replication.factor should not change > post 1.0. I will file a bug

Re: [Draft] Samza quarterly report

2020-01-09 Thread Yi Pan
ache Samza 1.0: Recent Advances and our plans for future in > Stream Processing > Seattle - Scalable Stream Processing with Apache Samza > > On Thu, Jan 9, 2020 at 1:23 AM Yi Pan wrote: > > > Hi, all, > > > > Another time to report our project status. I have a draft

Re: [Draft] Samza quarterly report

2020-01-09 Thread Yi Pan
: Recent Advances and our plans for future in Stream Processing Seattle - Scalable Stream Processing with Apache Samza P.S. just fixing one typo. -Yi On Thu, Jan 9, 2020 at 1:42 PM Yi Pan wrote: > ## Description: > - Apache Samza is a distributed stream processing engine that are

Re: [VOTE] SEP-26: Add SystemProducer for Azure Blob Storage

2020-01-09 Thread Yi Pan
+1 (binding). Good to see more cloud native integrations in Samza. -Yi On Wed, Jan 8, 2020 at 10:31 AM Prateek Maheshwari wrote: > +1 (binding). Thanks for the contribution. > > - Prateek > > On Tue, Jan 7, 2020 at 7:59 PM Jagadish Venkatraman < > jagadish1...@gmail.com> > wrote: > > > +1

Re: [VOTE] SEP-24: Cluster-based Job Coordinator Dependency Isolation

2020-03-11 Thread Yi Pan
ple, what if some > execution environment does not use URLs to specify resource locations > (although maybe this isn't a reasonable concern to worry about?)? > > Thanks, > Cameron > > On Wed, Mar 11, 2020 at 4:43 PM Yi Pan wrote: > > > Hi, Cameron, > > > > T

Re: [VOTE] Apache Samza 1.4.0 RC1

2020-03-06 Thread Yi Pan
Have downloaded the files, build with check-all.sh, and ran both YARN and standalone integration tests. All passed. +1 (binding). Thanks! -Yi On Tue, Mar 3, 2020 at 3:03 PM Cameron Lee wrote: > Hi all, > > This is a call for a vote on a release of Apache Samza 1.4.0. Thanks to > everyone who

Re: [VOTE] SEP-24: Cluster-based Job Coordinator Dependency Isolation

2020-03-11 Thread Yi Pan
run some tests regarding this classloading pattern, but we will do > further verification for general split deployment. > d) Yes, you are correct. Good catch. It should be "described above at > Application classloader". > > Thanks for all of your questions. I will clarify s

Re: Got Error Produce Respons with Correlation Id.

2020-03-11 Thread Yi Pan
Hi, Jeremiah, Sorry to reply late. This WARN message indicates that producer failed to flush to checkpoint topic and would retry. Do you see the Samza job hanging after that? Is the checkpointed offset metrics incrementing in this case? Not clear on your claiming: "logs stop at that point". No

[REPORT] Samza - April 2020

2020-04-08 Thread Yi Pan
Apache Samza is a distributed stream processing engine that are highly configurable to process events from various data sources, including real-time messaging system (e.g. Kafka) and distributed file systems (e.g. HDFS). ## Issues: - No issues require board attention ## Membership Data: Apache

Re: Streamtasks not instantiating consumers on startup

2020-04-10 Thread Yi Pan
Hi, Malcolm, Samza 0.14.1 is pretty old and if you have already upgraded Kafka to 2.2.1, I would highly recommend you to migrate to latest Samza version, which has Kafka 2.0 client. Depending on how you configure your broker, a 2.0 Kafka broker can be incompatible with older client like 0.8.6,

Re: Got Error Produce Respons with Correlation Id.

2020-03-13 Thread Yi Pan
after testing Confluent's Cloud kafka offering. > We had no issues when running our own kafka clusters in aws. > > > Jeremiah Adams > Software Engineer > www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > > From: Yi Pa

Re: Got Error Produce Respons with Correlation Id.

2020-03-13 Thread Yi Pan
h Adams > Software Engineer > www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > ________ > From: Yi Pan > Sent: Friday, March 13, 2020 1:27 PM > To: dev@samza.apache.org > Subject: Re: Got Error Produce Respons with Correlation

Re: [VOTE] SEP-24: Cluster-based Job Coordinator Dependency Isolation

2020-03-09 Thread Yi Pan
Hi, Cameron, Sorry to chime in late. Overall, looks great! I do have a few suggestions/questions before I can cast my vote here: a) for the configuration variable names, why are we limiting ourselves to yarn.resource.*? We have changed some of the configuration variables from yarn specific to

Re: [VOTE] SEP-24: Cluster-based Job Coordinator Dependency Isolation

2020-03-16 Thread Yi Pan
in the code to make the file > system location configurable (or specified through an environment > variable). The configuration or environment variable for the file system > location would not be YARN-specific, and it would be applicable to any > execution environment. > > On Wed, Ma

Re: Problem regarding the Monitoring of Samza Applications

2020-09-02 Thread Yi Pan
Hi, Jan, Thanks for reporting this metrics issue. We will take a further look and get back to you. Thanks a lot! -Yi On Tue, Sep 1, 2020 at 8:01 AM Jan Bensien wrote: > Hello, > > I hope this is the right place to ask. > > I am having problems monitoring my Samza application. Using the >

Re: Samza jdk11 support

2020-10-16 Thread Yi Pan
ound the below doc but that talks about samza 0.10. > > https://samza.apache.org/contribute/tests.html > > Thanks, > > On Thu, Jun 4, 2020 at 12:18 AM Yi Pan wrote: > > > Hi, Debraj and Jordan, > > > > Thanks a lot for the ping. I dug a bit deeper in the past em

Re: Issues with Samza-Kafka compatibility while upgrading to latest versions

2020-10-10 Thread Yi Pan
Hi, Choudhary, Thanks for reporting your issues. Samza manages its dependencies in a few gradle files under ${ROOT}/gradle/dependency-versions-*.gradle. Specifically, the Kafka dependency is defined in the file ${ROOT}/gradle/dependency-versions.gradle. I quickly checked the versions listed in

Draft board report for Samza

2020-10-10 Thread Yi Pan
Hi, all, Here is the draft report for Samza. Please let me know if I missed anything. Thanks! == ## Description: The mission of Samza is the creation and maintenance of software related to distributed stream processing framework ##

Re: [DISCUSS] Samza 1.5.1 release

2020-08-19 Thread Yi Pan
+1 to this release as well! On Wed, Aug 19, 2020 at 9:21 AM Prateek Maheshwari wrote: > +1, this is a critical bug and we should release the fix ASAP. > > Thanks, > Prateek > > On Tue, Aug 18, 2020 at 9:02 PM Bharath Kumara Subramanian < > codin.mart...@gmail.com> wrote: > > > Hi all, > > > >

Re: [VOTE] Apache Samza 1.5.1 RC0

2020-08-27 Thread Yi Pan
+1 (binding), ran ./bin/check-all.sh and all integration tests. Verified signatures. On Sat, Aug 22, 2020 at 8:09 PM Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > Hi all, > > This is a call for a vote on a release of Apache Samza 1.5.1. We are > releasing 1.5.1 to address a

Re: [DISCUSS] Samza 1.5 release

2020-06-01 Thread Yi Pan
lgtm. Thanks for kicking off the discussion! -Yi On Tue, May 26, 2020 at 8:55 PM Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > Hi all, > > We have accumulated few features/improvements since the last release and > would like to make Samza 1.5 release. > > I wanted to kick off

Re: Samza jdk11 support

2020-05-26 Thread Yi Pan
Hi, Debraj, Thanks for the reminder. We did discussed about JDK11 before. Unfortunately, I don't know whether JDK11 is up to the roadmap as of now. Let me sync up with the team and get back to you. -Yi On Mon, May 25, 2020 at 9:53 AM Debraj Manna wrote: > Hi > > I have seen a few earlier

Re: [VOTE] Apache Samza 1.5.0 RC1

2020-06-10 Thread Yi Pan
+1 (binding). Ran check-all, verified sha1 and signature, ran both standalone and YARN integration tests. LGTM. On Mon, Jun 8, 2020 at 5:16 PM Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > Hi all, > > This is a call for a vote on a release of Apache Samza 1.5.0. We are > excited

Re: Samza jdk11 support

2020-06-03 Thread Yi Pan
: > This is a pressing issue for our team as well. > > Jordan > > > On May 31, 2020, at 7:15 PM, Debraj Manna > wrote: > > > > Thanks Yi. > > > > Did you get any update on the JDK 11 roadmap? > > > > > > On Wed, May 27, 2020 at 1:04 AM Yi Pan wrote:

Re: [VOTE] Apache Samza 1.5.0 RC1

2020-06-12 Thread Yi Pan
Hey, Yang, Can you open a JIRA for this failure? This is again related to a test code using sleep() trying to make sure that the test is completed before verification. -Yi On Fri, Jun 12, 2020 at 8:01 AM Yang Zhang wrote: > Being late to the party. Verified apache-samza-1.5.0-src.tgz in both

Re: Zookeeper and Client upgrade

2021-01-11 Thread Yi Pan
Hi, Stuart, Please feel free to raise tickets for update requests like these. Thanks for reporting! -Yi On Fri, Jan 8, 2021 at 9:58 PM Stuart Perks wrote: > The Zookeeper version sits at 3.4.6 with 3.6.2 now available bringing > security enhancements. > > >

Re: SAMZA-2612: Kafka topic naming not supported fully

2020-12-28 Thread Yi Pan
Hey, Stuart, Sounds great that you have found the way around it! Thanks! -Yi On Sat, Dec 19, 2020 at 12:28 PM Stuart Perks wrote: > This can be done using withPhysicalName > > Closed the JIRA > > On 2020/12/17 12:19:27, Stuart Perks wrote: > >

Re: Zookeeper and Client upgrade

2021-01-14 Thread Yi Pan
/jira/browse/SAMZA-2617 > > Happy to start looking onto one of these. > > On 2021/01/11 23:04:08, Yi Pan wrote: > > Hi, Stuart,> > > > > Please feel free to raise tickets for update requests like these.> > > > > Thanks for reporting!> > > >

Re: [VOTE] Apache Samza 1.6.0 RC1

2021-01-13 Thread Yi Pan
Verified signature and sha1. Ran check-all and integration tests. All passed. +1 (binding). -Yi On Mon, Jan 11, 2021 at 4:08 PM Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > Thanks for driving this release Boris! Verified signatures and ran all the > tests. > check-all and

[REPORT] Apache Samza - Jan 2021

2021-01-13 Thread Yi Pan
## Description Apache Samza is a distributed stream processing engine that is highly configurable to process events from various data sources, including real-time messaging system (e.g. Kafka) and distributed file systems (e.g. HDFS). ## Issues: - There are no issues requiring board attention.

Re: [VOTE] SEP-28: Samza State Backend Interface and Checkpointing Improvement

2021-06-22 Thread Yi Pan
+1 (binding) this is going to improve our state recovery story significantly! -Yi On Mon, Jun 21, 2021 at 1:03 PM Daniel Chen wrote: > Hi all, > > This is a call for a vote on SEP-28: Samza State Backend Interface and > Checkpointing Improvements. Thanks to everyone who was involved with the >

Re: [VOTE] SEP-29: Blob Store Based State Backup And Restore

2021-06-22 Thread Yi Pan
+1 (binding). Thanks for rolling out this big feature! -Yi On Tue, Jun 22, 2021 at 1:42 PM Sanil Jain wrote: > +1 (non-binding) Thanks for this contribution! > > -Sanil > > On Tue, 22 Jun 2021 at 13:13, Daniel Chen wrote: > > > +1 (non-binding), thanks! > > > > On Tue, Jun 22, 2021 at 1:10 PM

Reminder: LinkedIn Stream Processing Meetup on Jun 24

2021-06-15 Thread Yi Pan
The meet up is coming in less than 10 days! Please remember to sign up and join us! https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/278266182/ Short abstracts for the contents:

Upcoming streams meetup event

2021-05-25 Thread Yi Pan
& Yuhong Cheng, LinkedIn Scaling Kafka Audit Service - Abhishek Mendhekar, LinkedIn Please visit our Meetup site for access to more details & the live event link (TBD), and to let us know you're coming! https://lnkd.in/g7Fq-3K Hope to see you there! Best, - Yi Pan

Welcome Ke Wu as Apache Samza committer

2021-02-23 Thread Yi Pan
Hi, everyone, I am glad to announce that Ke Wu has officially accepted our invitation and become an Apache Samza committer now. Please join me to give him a warm welcome! Cheers! -Yi

Re: How are samza.container.id generated in yarn?

2021-04-22 Thread Yi Pan
Hi, Debraj, In YARN environment, Samza uses YARN generated containerIds as environmental variables to set each container process's samza.container.id. i.e. when containers are requested by Samza AM process in YARN, YARN RM will reply with a set of allocated container objects, which is of class

Re: Malformed URL issue while deploying Samza on a pure IPv6 VM

2021-08-26 Thread Yi Pan
Hi Vishal, Could you open a JIRA to track this one? I will circle back to our internal team to do a quick assessment on IPv6 related issues. Meanwhile, I strongly encourage you to submit the patch and continue your test, since that's the best way to discover any hidden issues on that front.

Re: [ANNOUNCE] Welcome Daniel Chen as Samza Committer

2021-09-17 Thread Yi Pan
Congrats, Daniel, well deserved!!! -Yi On Fri, Sep 17, 2021 at 11:23 AM Xinyu Liu wrote: > Hi, all, > > I am glad to announce that Daniel Chen has officially accepted our > invitation and become an Apache Samza Committer now. > > Daniel has contributed to many areas of Samza, from his early

[Draft] [REPORT] Samza Report July 2021

2021-07-14 Thread Yi Pan
## Description: The mission of Samza is the creation and maintenance of software related to distributed stream processing framework ## Issues: - There are no issues requiring board attention. ## Membership Data: Apache Samza was founded 2015-01-22 (6 years ago) There are currently 28 committers

[Draft][REPORT] Apache Samza Oct 2021

2021-10-11 Thread Yi Pan
Hi, team, Please read the following draft report for Oct 2021. Let me know if I missed anything. Thanks a lot! ## Description: The mission of Samza is the creation and maintenance of software related to distributed

[Draft] [REPORT] Apache Samza Jan 2022

2022-01-12 Thread Yi Pan
Hi, team, Please read the draft report below and let me know if I missed anything. Thanks! - Yi == ## Description: The mission of Samza is the creation and maintenance of software related to distributed stream processing framework ##

<    1   2   3   4   5   6   7   8   >