Re: Question about stop in SystemProducer

Hi Jagadish, I resolved the problem. Samza must close the FileSystem somewhere when “shutdown” is triggered. Now I don’t get FileSystem instance from cache and everything works fine. Thanks! Qi Shu > 在 2017年8月24日，15:06，舒琦 写道： > > Yes, in the same thread.

Re: Question about stop in SystemProducer

re the “closeFiles” method executed completely. > > It's unlikely that the container quit without returning from *close*. Are > you sure there were filesList is not empty? I'd suggest adding more logging. > > I'm assuming you are calling *closeFiles* in the same t

Re: Question about stop in SystemProducer

年8月24日，14:44，Jagadish Venkatraman 写道： > > Is "Begin to close file" printed? Where exactly is your application stalled > ? I'd suggest you take a stack dump. > > On Wed, Aug 23, 2017 at 11:32 PM, 舒琦 wrote: > >> Hi Jagadish, >> >>Belo

Re: Question about stop in SystemProducer

cache data to be flushed) in > stop method is not executed completely > > Are you seeing the *close()* method hang? SystemProducer.*close* is a > synchronous operation, and will block for the *close* method to finish. > > Best, > Jagadish > > On Wed, Aug 23, 2017 at 11:17

Question about stop in SystemProducer

Hi, I write a SystemProducer for HDFS and everything is fine. When samza job is shutdown, the stop method in SystemProducer is called, but the close files operation(may need some time, there may be cache data to be flushed) in stop method is not executed completely. How can I r

Re: Can't start container in newly added hosts

t-affinity.html > for more information. > > > On Mon, Jul 17, 2017 at 3:07 AM, 舒琦 wrote: > >> Hi Guys, >> >>After add 3 hosts to yarn cluster, some job can’t startup, then I >> found some informations as shown below: >> >>2

Can't start container in newly added hosts

Hi Guys, After add 3 hosts to yarn cluster, some job can’t startup, then I found some informations as shown below: 2017-07-17 17:37:29.928 [AMRM Callback Handler Thread] YarnClusterResourceManager [INFO] Container allocated from RM on rflow79.antfact.com 2017-

Re: A Question About multi-output

Hi Jagadish, Thank you very much! QiShu > 在 2017年7月12日，10:45，Jagadish Venkatraman 写道： > > Yes, before the completion of a checkpoint/commit, all pending messages > (across all streams) are guaranteed to be flushed. > > On Tue, Jul 11, 2017 at 7:33 PM 舒

Re: A Question About multi-output

thogonal. If the docs are misleading, please let > us know. We'll fix them > > On Mon, Jul 10, 2017 at 11:56 PM, 舒琦 wrote: > >> Hi guys, >> >>How to write one message to two different output stream, let’s >> say two different kafka cluster? >>

A Question About multi-output

Hi guys, How to write one message to two different output stream, let’s say two different kafka cluster? For what I understand now, MessageCollector.send can only be called once in process, because it relates to Acks, checkpoint and processing loop. Thanks for

Re: A Question about Release

es, new > features, improvements, JIRAs resolved, community developments etc. > > Please check out : https://blogs.apache.org/samza/ > > Best, > Jagadish > > > > On Wed, Jun 14, 2017 at 10:06 PM, 舒琦 wrote: > >> Hi, >> >>I want to know where I ca

A Question about Release

Hi, I want to know where I can find an overview of the major features and improvements for each release. Thanks. Qi Shu

Re: A question about checkpointing

be thrown out and the container will fail. Hence, when > restarted, the container will repeat from the previous checkpoint (i.e. at > least once delivery still holds and no data loss). > > Hope the above answers your question. > > Thanks! > > -Yi > > On Thu, May

Re: A question about checkpointing

erstand your specific question about checkpointing? > > > On Thu, May 11, 2017 at 12:28 AM, 舒琦 wrote: > >> Hi, >> >>Below is the description about checkpointing. >> >>『Checkpointing is guaranteed to only cover events that are

A question about checkpointing

Hi, Below is the description about checkpointing. 『Checkpointing is guaranteed to only cover events that are fully processed. It happens only when there are no pending process()/processAsync() or WindowableTask.window() invocations. All preceding invocations happen-before check

Re: About reconnect times?

Hi Jagadish, Thanks for your help, I’ll check our Kafka first. 舒琦 地址：长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F 网址：http://www.eefung.com 微博：http://weibo.com/eefung 邮编：410013 电话：400-677-0986 传真：0731-88519609 > 在 2017年4月25日，11:36，Jagadish Venkatraman 写道： > > Hi ShuQi, > >>> There

Re: About reconnect times?

- The default behavior is to attempt a re-connect, and then re-create the > Consumer instance. The number of reconnect attempts is unbounded (and not > configurable). > > > Best, > Jagadish > > > On Tue, Apr 18, 2017 at 10:51 PM, 舒琦 wrote: > >> Hi Guys, >

About reconnect times?

Hi Guys, One of brokers in Kafka cluster is going down, the samza got the following exception: 2017-04-19 10:42:36.751 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at 172.19.105.20:9096 for client samza_consumer-canal_status_content_distinct-1] DefaultFetchSimpleConsumer [INFO] Recon

What happened when start a samza job?

Hi, I’m using samza 0.12 and kafka 0.10. after running for several days, one of our job was restarted and it seems stuck in the “Get latest offset”. And after almost 30 minutes, it still stuck there. 2017-03-23 08:24:04.503 [main] KafkaUtil [INFO] Successfully validated topic __

Re: Exceed max.request.size of Kafka

tions and fail the container. Default is false. > task.drop.serialization.errors This property is to define how the system > deals with serialization failure situation. If set to true, the system will > drop the error messages and keep running. If set to false, the system with > throw

Exceed max.request.size of Kafka

Hi, Sometimes there are huge size of data will occur in our flow, like 2MB, now samza will catch exception and shutdown like belowing.But what I want is I can handle such specific exception and just discard such data and the flow continues. 2017-02-23 16:17:01.949 [main] SamzaContainer

Re: Can't fetch messages from kafka

Hi, Thanks for your replying. I did not make a thread dump, if it happen again I’ll dump the thread stack. 舒琦 地址：长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F 网址：http://www.eefung.com 微博：http://weibo.com/eefung 邮编：410013 电话：400-677-0986 传真：0731-88519609 > 在 2017年2月7日，23:52，Jagadish Venkatra

Can't fetch messages from kafka

Hi, During the running of Samza, one of brokers in our Kafka cluster went down, the states of relative topics are shown as pictures below. Then all the task can’t fetch any messages any more, even after restarted the samza job. But at the same time , another app consuming the same topic

Re: How to gracefully stop samza job

this a problem? Thanks. 舒琦 地址：长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F 网址：http://www.eefung.com 微博：http://weibo.com/eefung 邮编：410013 电话：400-677-0986 传真：0731-88519609 > 在 2017年1月18日，01:59，Yi Pan 写道： > > You probably should return a valid SystemAdmin object, but returning null > for Sy

Re: How to gracefully stop samza job

Hi, My SystemFactory implementation return null for both 『getConsumer』 and 『getAdmin』, is this the cause of the problem? Thanks. 舒琦 地址：长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F 网址：http://www.eefung.com 微博：http://weibo.com/eefung 邮编：410013 电话：400-677-0986 传真：0731-88519609 > 在 2017年1月17日，17:18，Yi

Re: How to gracefully stop samza job

Sorry, forget the log file. 舒琦 地址：长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F 网址：http://www.eefung.com 微博：http://weibo.com/eefung 邮编：410013 电话：400-677-0986 传真：0731-88519609 > 在 2017年1月17日，10:40，舒琦 写道： > > Hi, > > Actually I check the log by using“tail" on the yarn local data

Re: How to gracefully stop samza job

use is 0.10.1.0， samza version is 0.11.0, kafka client version samza using is 0.8.2.1. Thanks. 舒琦 地址：长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F 网址：http://www.eefung.com 微博：http://weibo.com/eefung 邮编：410013 电话：400-677-0986 传真：0731-88519609 > 在 2017年1月17日，10:40，舒琦 写道： > > Hi, > > Actua

Re: How to gracefully stop samza job

Hi, Actually I check the log by using“tail" on the yarn local data dir on which the container is running, the container log I found in hdfs already, but can’t tell the format for log. ———— 舒琦 地址：长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F 网址：http://www.eefung.com 微博：http://weibo.com/eefung 邮编：410013 电

How to gracefully stop samza job

Hi Guys, How can I stop running samza job gracefully except killing it? Because when samza job was killed, the close and stop method in BaseMessageChooser and SystemProducer will not be called and the container log will be removed automatically, how can resolve this? Th

Question about Samza Metrics

Hi, I found most metrics of Samza are accumulative, is there a way to get metrics in a certain time frame. Thanks ShuQi

Re: Question about Samza Metrcis

Hi Navina, Thanks for your clear description, I understand more now. ShuQi 舒琦 地址：长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F 网址：http://www.eefung.com 微博：http://weibo.com/eefung 邮编：410013 电话：400-677-0986 传真：0731-88519609 > 在 2016年9月9日，04:19，Navina Ramesh 写道： > > Hi ShuQi, > > A

Re: Question about Samza Metrcis

; Can you run your program in DEBUG log-level? Does sending the metric to the > producer fail? Is the metric reporter thread showing an exception? (check > the stderr file too) > > Producing to a kafka topic should usually auto-create it. > > On Wed, Sep 7, 2016 at 2:09 AM, 舒琦

Question about Samza Metrcis

48576 send.buffer.bytes = 131072 acks = 1 reconnect.backoff.ms = 10 linger.ms = 0 metrics.num.samples = 2 metadata.fetch.timeout.ms = 6 Thanks. ———— 舒琦 地址：长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F 网址：http://www.eefung.com 微博：http://weibo.com/eefung 邮编：410013 电