Re: is there any detrimental side-effect if i set the max parallelism as 32768

Hi Hangxiang, David, Thank you for your replies. Your responses are very helpful. Best regards, Tony Wei David Anderson 於 2023年3月14日 週二 下午12:12寫道： > I believe there is some noticeable overhead if you are using the > heap-based state backend, but with RocksDB I think the differe

is there any detrimental side-effect if i set the max parallelism as 32768

much between each subtask. I'm wondering if this is a good practice, because based on the official document it is not recommended actually. If possible, I would like to know the detail about this side-effect. Which state backend will have this issue? and Why? Please give me an advice. Thanks in

Re: confused about `TO_TIMESTAMP` document description

Hi Leonard, Thanks for confirmation. I have created the jira ticket [1]. The pull request will be submitted later. best regards, [1] https://issues.apache.org/jira/browse/FLINK-22970 Leonard Xu 於 2021年6月10日 週四 下午8:58寫道： > Hi，Tony > > > I found this code snippet [2] might be related to `TO_TIM

confused about `TO_TIMESTAMP` document description

Hi Expert, this document [1] said `TO_TIMESTAMP` will use the session time zone to convert date time string into a timestamp. If I understand correctly, when I set session time zone to `Asia/Shanghai` and query `SELECT TO_TIMESTAMP('1970-01-01 08:00:00');`, the result should be epoch timestamp `0`

Re: when should `FlinkYarnSessionCli` be included for parsing CLI arguments?

Hi Till, I have created the ticket to extend the description of `execution.targe`. https://issues.apache.org/jira/browse/FLINK-22476 best regards, Tony Wei 於 2021年4月26日 週一 下午5:24寫道： > Hi Till, Yangze, > > I think FLINK-15852 should solve my problem. > It is my fault that my flin

Re: when should `FlinkYarnSessionCli` be included for parsing CLI arguments?

t;> > Cheers, >> > Till >> > >> > On Mon, Apr 26, 2021 at 8:37 AM Yangze Guo wrote: >> >> >> >> Hi, Tony. >> >> >> >> What is the version of your flink-dist. AFAIK, this issue should be >> >> addressed in F

when should `FlinkYarnSessionCli` be included for parsing CLI arguments?

Hi Experts, I recently tried to run yarn-application mode on my yarn cluster, and I had a problem related to configuring `execution.target`. After reading the source code and doing some experiments, I found that there should be some room of improvement for `FlinkYarnSessionCli` or `AbstractYarnCli

Re: ***UNCHECKED*** Error while confirming Checkpoint

Hi Piotrek, There was already an issue [1] and PR for this thread. Should we mark it as duplicated or related issue? Best, Tony Wei [1] https://issues.apache.org/jira/browse/FLINK-10377 Piotr Nowojski 於 2019年11月28日 週四 上午12:17寫道： > Hi Tony, > > Thanks for the explanation. Assumi

Re: ***UNCHECKED*** Error while confirming Checkpoint

esney said. The following checkpoint succeeded before the previous savepoint, handling both of their pending transaction, but savepoint still succeeded and sent the notification to each TaskManager. That led to this exception. Could you double check if this is the case? Thank you. Best, Tony Wei

Re: ***UNCHECKED*** Error while confirming Checkpoint

Hi, As the follow up, it seem that savepoint can't be subsumed, so that its notification could still be send to each TMs. Is this a bug that need to be fixed in TwoPhaseCommitSinkFunction? Best, Tony Wei Tony Wei 於 2019年11月27日 週三 下午3:43寫道： > Hi, > > I want to raise this questio

Re: ***UNCHECKED*** Error while confirming Checkpoint

ier and led to this exception happened, because there was no pending transaction in queue. Does anyone know the details about subsumed notifications mechanism and how checkpoint coordinator handle this situation? Please correct me if I'm wrong. Thanks. Best, Tony Wei Stefan Richter 於 2018年10

Questions about how to use State Processor API

Thanks in advance. Best, Tony Wei

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

action` list is empty before executing `enqueueNewPartitions` function. Am I right? If it can be confirmed as a bug, I would like to submit my patch to fix it. Thanks for your help. Best, Tony Wei [1] https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L20

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

x27;s coordinator, since the empty transaction won't make any request to server. The attachments are my simple producer code. Please help to verify what I thought is correct. Thanks. Best, Tony Wei [1] https://github.com/apache/kafka/blob/c0019e653891182d7a95464175c9b4ef63f8bae1/clients/src/mai

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

? Is there any expert who is familiar with both kafka and flink's kafka connector could help me solve this? Thanks very much. The attachment is my code to reproduce this problem. The cluster's versions are the same as I mentioned in my first email. Best, Tony Wei *flink taskmanager:*

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

r's behavior. I tried to use kafka java producer to reproduce the exception, but it worked well. Maybe my observation is not correct, but the experiment result seems like that. Do you have any thoughts on this? Best, Tony Wei Tony Wei 於 2019年9月19日 週四 上午11:08寫道： > Hi Becket, > > On

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

Hi Becket, One more thing, I have tried to restart other brokers without active controller, but this exception might happen as well. So it should be independent of the active controller like you said. Best, Tony Wei Tony Wei 於 2019年9月18日 週三 下午6:14寫道： > Hi Becket, > > I have reprod

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

8 07:13:42,779] DEBUG [TransactionCoordinator id=2] Returning > NOT_COORDINATOR error code to client for blacklist -> Sink: > kafka-sink--eba862242e60de7e4744f3307058f865-7's AddPartitions request > (kafka.coordinator.transaction.TransactionCoordinator) > [2019-09-18 07:13:43,633] DEBUG

Re: Kafka producer failed with InvalidTxnStateException when performing commit transaction

to find out what's wrong about my kafka producer. Could someone help me please? Best, Tony Wei Fabian Hueske 於 2019年8月16日 週五 下午4:10寫道： > Hi Tony, > > I'm sorry I cannot help you with this issue, but Becket (in CC) might have > an idea what went wrong here. > > Best

Kafka producer failed with InvalidTxnStateException when performing commit transaction

nt checkpoint: 1 max checkpoint duration before and after the exception occurred: < 2 minutes Best, Tony Wei

Re: Kafka ProducerFencedException after checkpointing

Hi Piotr, Thanks a lot. I need exactly once in my use case, but instead of having the risk of losing data, at least once is more acceptable when error occurred. Best, Tony Wei Piotr Nowojski 於 2019年8月12日 週一 下午3:27寫道： > Hi, > > Yes, if it’s due to transaction timeout you will lose

Re: Kafka ProducerFencedException after checkpointing

Hi, I had the same exception recently. I want to confirm that if it is due to transaction timeout, then I will lose those data. Am I right? Can I make it fall back to at least once semantic in this situation? Best, Tony Wei Piotr Nowojski 於 2018年3月21日 週三 下午10:28寫道： > Hi, > > B

Re: How to make two SQLs use the same KafkaTableSource?

is fine with me. My original question just focused on reused nodes in SQL api, and seems Blink planner is what I need. Thanks for your help again. Best, Tony Wei Zhenghua Gao 於 2019年8月9日 週五 下午1:54寫道： > Blink planner support lazy translation for multiple SQLs, and the common > nodes will

Re: How to make two SQLs use the same KafkaTableSource?

forgot to send to user mailing list. Tony Wei 於 2019年8月9日 週五 下午12:36寫道： > Hi Zhenghua, > > I didn't get your point. It seems that `isEagerOperationTranslation` is > always return false. Is that > means even I used Blink planner, the sql translation is still in a lazy >

How to make two SQLs use the same KafkaTableSource?

ka. A workaround might be registering the same kafka topic twice with different name, group_id for two SQLs. But I would still like to know if there is any way to make two SQLs just read from the same KafkaTableSource? Thanks in advance. Best, Tony Wei

Re: Is it possible to decide the order of where conditions in Flink SQL

true > END Best regards, Tony Wei Hequn Cheng 於 2019年7月28日 週日 下午3:30寫道： > Hi Tony, > > There is no order guarantee for filter conditions. The conditions would be > pushed down or merged during query optimization. > > However, you can use the case when[1] to achieve what you want

Re: Is it possible to decide the order of where conditions in Flink SQL

operator of physical plan have any meaning to represent the execution order of `where` conditions? Best, Tony Wei sri hari kali charan Tummala 於 2019年7月27日 週六 上午3:02寫道： > try cte common table expressions if it supports or sql subquery. > > On Fri, Jul 26, 2019 at 1:00 PM Fanbin Bu w

Is it possible to decide the order of where conditions in Flink SQL

`!user.is_robot` first then executing UDF, it will reduce the number of database access. Those records with `true` in `user.is_robot` will be dropped earlier and don't need to access database. select * from users where !user.is_robot and UDF_NEED_TO_QUERY_DB(user) Thanks, Tony Wei

Re: left join failing with FlinkLogicalJoinConverter NPE

Hi, I also found the similar issue here [1]. Best, Tony Wei [1] https://issues.apache.org/jira/browse/FLINK-11433 Tony Wei 於 2019年7月19日 週五 下午5:38寫道： > Hi, > > Is there any update for this issue? I have had the same problem just like > Karl's. > After I remove query like

Re: left join failing with FlinkLogicalJoinConverter NPE

Hi, Is there any update for this issue? I have had the same problem just like Karl's. After I remove query like "select collect(data) ..." from one of the joined tables, the sql can be executed correctly without throwing any NPE. Best regards, Tony Wei Xingcan Cui 於 2019年2月2

Question about counting top k on streaming data

n't work here in streaming mode, is there any other optimization can apply in this case? It is not necessary to focus on SQL only. Any improvement on DataStream is also welcome. Thank you. Best Regards, Tony Wei

Re: What does "Continuous incremental cleanup" mean in Flink 1.8 release notes

red states should be clean up as well based on Flink 1.6's implementation. Am I right? Best, Tony Wei Konstantin Knauf 於 2019年3月9日 週六 上午7:00寫道： > Hi Tony, > > before Flink 1.8 expired state is only cleaned up, when you try to access > it after expiration, i.e. when user code tries

What does "Continuous incremental cleanup" mean in Flink 1.8 release notes

ecuting TTL mechanism. Could you give me more references to learn about it? A simple example to illustrate it is more appreciated. Thank you. Best, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-master/release-notes/flink-1.8.html#state

Re: how to override s3 key config in flink job

Hi Andrey, Thanks for your detailed answer, and I have created a JIRA issue to discuss it [1]. Please check the description and help me to fill the details, like component/s, since I'm not sure where it should be put. Thank you very much. Best, Tony Wei [1] https://issues.apache.org/jira/b

Re: how to override s3 key config in flink job

Hi yinhua, Our flink version is 1.6.0. Best, Tony Wei yinhua.dai 於 2018年11月27日 週二 下午2:32寫道： > Which flink version are you using. > I know how it works in yarn, but not very clear with standalone mode. > > > > -- > Sent from: > http://apache-flink-user-mailing

Re: how to override s3 key config in flink job

to submit different flink applications with different s3 key for flink presto s3 filesystem. Any other suggestions are also welcome. Thank you. Best, Tony Wei yinhua.dai 於 2018年11月27日 週二 上午11:37寫道： > Did you try "-Dkey=value"? > > > > -- > Sent from: > http

Re: Reset kafka offets to latest on restart

write your own implementation with kafka client and always seek to the latest position when the job begin to run. Best, Tony Wei Vishal Santoshi 於 2018年11月25日 週日 上午4:51寫道： > I think I can set . a new uuid but it seems `allowNonRestoreState` is a > CLI hint. I need the "automatic" restart

Re: how to override s3 key config in flink job

Hi, Is there anyone can answer me? Thanks, Tony Wei Tony Wei 於 2018年11月20日 週二 下午7:39寫道： > Hi, > > Is there any way to provide s3.access-key and s3.secret-key in flink > application, instead of setting > them in flink-conf.yaml? > > In our use case, my team provide a fli

Re: Reset kafka offets to latest on restart

the way that Rong provided to set the initial start position. cc. Gordon who know more about the details of kafka source. Best, Tony Wei Rong Rong 於 2018年11月22日 週四 上午8:23寫道： > Hi Vishal, > > You can probably try using similar offset configuration as a service > consumer. > May

how to override s3 key config in flink job

store checkpoints. So, we want to know if is it feasible to let users provide their checkpoint path and corresponding aws key to access their own s3 bucket? If not, could you show me why it doesn't work currently? And, is it possible to become a new feature? Thanks in advance for your help. Best, Tony Wei

sys.exist(1) led to standalonesession daemon closed

he.flink.runtime.blob.TransientBlobCache - Shutting > down BLOB cache > 2018-11-05 13:28:04,761 INFO org.apache.flink.runtime.blob.BlobServer > - Stopped BLOB server at 0.0.0.0:42075 Best, Tony Wei. [1] https://github.com/scallop/scallop

Re: How to get Cluster metrics in FLIP-6 mode

Hi Gary, Thanks for your information. Best, Tony Wei 2018-09-11 20:26 GMT+08:00 Gary Yao : > Hi Tony, > > You are right that these metrics are missing. There is already a ticket for > that [1]. At the moment you can obtain these information from the REST API > (/overview) [2]. &

How to get Cluster metrics in FLIP-6 mode

these metrics? Or did I miss something? Thank you. Best Regards, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html#cluster

Question about akka configuration for FLIP-6

that FLIP-6 tried to get rid of akka and use its own rpc interface. Please correct me if I misunderstood. Thanks. akka.watch.heartbeat.interval akka.watch.heartbeat.pause taskmanager.exit-on-fatal-akka-error Best Regards, Tony Wei

Re: checkpoint failed due to s3 exception: request timeout

Hi Andrey, Cool! I will add it in my flink-conf.yaml. However, I'm still wondering if anyone is familiar with this problem or has any idea to find the root cause. Thanks. Best, Tony Wei 2018-08-29 16:20 GMT+08:00 Andrey Zagrebin : > Hi, > > the current Flink 1.6.0 version uses

Re: checkpoint failed due to s3 exception: request timeout

Hi Vino, I thought this config is for aws s3 client, but this client is inner flink-s3-fs-presto. So, I guessed I should find a way to pass this config to this library. Best, Tony Wei 2018-08-29 14:13 GMT+08:00 vino yang : > Hi Tony, > > Sorry, I just saw the timeout, I thought

Re: checkpoint failed due to s3 exception: request timeout

filesystem and I thought it might have a simple way to support this setting like other s3.xxx config. Very much appreciate for your answer and help. Best, Tony Wei 2018-08-29 11:51 GMT+08:00 vino yang : > Hi Tony, > > A while ago, I have answered a similar question.[1] > > You

Re: Need a clarification about removing a stateful operator

Hi Stefan, Thanks for your detailed explanation. Best, Tony Wei 2018-08-17 15:56 GMT+08:00 Stefan Richter : > Hi, > > it will not be transported. The JM does the state assignment to create the > deployment information for all tasks. If will just exclude the state for > operato

Re: Need a clarification about removing a stateful operator

Hi Chesnay, Thanks for your quick reply. I have another question. Will the state, which is ignored, be transported to TMs from DFS? Or will it be detected by JM's checkpoint coordinator and only those states reuired by operators be transported to each TM? Best, Tony Wei 2018-08-17 14:38 G

Need a clarification about removing a stateful operator

ator? And could this behavior differ between different state backend (Memory, FS, RocksDB) ? Many thanks, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#application-topology

Re: flink_taskmanager_job_task_operator_records_lag_max == -Inf on Flink 1.4.2

`records_lag_max` is the maximum lag in terms of number of records for any partition in this "window". I'm not sure what this "window" means and if it is configurable. If it is configurable, then you can directly pass the config argument to Flink-Kafka-Connector to set kafka consumer.

Re: Conceptual question

generated by another state descriptor. Please correct me if I misunderstood. Thank you. Best Regards, Tony Wei 2018-06-09 9:45 GMT+08:00 TechnoMage : > Thank you all. This discussion is very helpful. It sounds like I can > wait for 1.6 though given our development status. > >

Re: Conceptual question

ut getKeyes you still can implement lazy conversion) you would have > to write your own operator. > > Piotrek > > > On 7 Jun 2018, at 15:26, Tony Wei wrote: > > Hi Piotrek, > > I used `ProcessFunction` to implement it, but it seems that I can't call > `getKey

Re: Conceptual question

7;t look up all keys and migrate the entire previous states to the new states in `ProcessFunction#open()`? As I said, do I need to port `ProcessFunction` to `KeyedProcessOperator` to migration state like the manner showed in `WindowOperator`? Best Regards, Tony Wei 2018-06-07 20:28 GMT+08:00

Re: Conceptual question

was implemented state operator by `ProcessFunction` API, is it possible to port it to `KeyedProcessOperator` and do the state migration that you mentioned? And are there something concerned and difficulties that will leads to restored state failed or other problems? Thank you! Best Regards, Tony Wei

Re: Question about the behavior of TM when it lost the zookeeper client session in HA mode

st ZK connection. So, as Piotr said, it looks like an error in Kafka producer and I will pay more attention on it to see if there is something unexpected happens again. Best Regards, Tony Wei 2018-05-15 19:56 GMT+08:00 Piotr Nowojski : > Hi, > > It looks like there was an error in asynchronous

Question about the behavior of TM when it lost the zookeeper client session in HA mode

ny times during the last weekend and made my kafka log delay grew up. Please help me. Thank you very much! Best Regards, Tony Wei

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

and enabled incremental checkpoint mechanism as well. Our job has run healthily for more than two weeks. Thank you all for helping me to investigate and solve this issue. Best Regards, Tony Wei [1] EBS: I/O Credits and Burst Performance <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVo

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

Hi Stefan, We prepared to run it on local SSDs yesterday. However, as I said, the problem just disappeared. Of course we will replace it to local SSDs, but I'm afraid that I might be able to reproduce the same situation for both environments to compare the difference. Best Regards, Ton

Re: Fwd: checkpoint stuck with rocksdb statebackend and s3 filesystem

root cause or stop the investigation and make the conclusion in this mailing thread. What do you think? Best Regards, Tony Wei 2018-03-06 15:13 GMT+08:00 周思华 : > Hi Tony, > > Sorry for missing the factor of cpu, I found that the "bad tm"'s cpu load > is so much high

Fwd: checkpoint stuck with rocksdb statebackend and s3 filesystem

Sent to the wrong mailing list. Forward it to the correct one. -- Forwarded message -- From: Tony Wei Date: 2018-03-06 14:43 GMT+08:00 Subject: Re: checkpoint stuck with rocksdb statebackend and s3 filesystem To: 周思华 , Stefan Richter Cc: "user-subscr...@flink.apache.org&q

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

and it is far from my timeout configuration. I believe "incremental checkpoint" will reduce the latency and make this issue might occur seldom, but I can't promise it won't happen again if I have bigger states growth in the future. Am I right? Best Regards, Tony Wei 2018-03-

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

another s3 filesystem package, because it is the last thing I want to see that the checkpoint timeout happened very often. Thank you very much for all your advices. Best Regards, Tony Wei 2018-03-06 1:07 GMT+08:00 Stefan Richter : > Hi, > > thanks for all the info. I had a look into the problem

Re: checkpoint stuck with rocksdb statebackend and s3 filesystem

Hi Stefan, Here is my checkpointing configuration. Checkpointing Mode Exactly Once Interval 20m 0s Timeout 10m 0s Minimum Pause Between Checkpoints 0ms Maximum Concurrent Checkpoints 1 Persist Checkpoints Externally Enabled (delete on cancellation) Best Regards, Tony Wei 2018-03-05 21:30 GMT+08

Re: Will that be a problem if POJO key type didn't override equals?

Hi Timo, Thanks a lot. I will try it out. Best Regards, Tony Wei 2018-02-06 17:25 GMT+08:00 Timo Walther : > With heap-based state I meant state that is stored using the > MemoryStateBackend or FsStateBackend [1]. In general, even if you are just > using ValueState, the key migh

Re: Will that be a problem if POJO key type didn't override equals?

ValueState as key state. BTW, could you please give more explanations about what heap-based state is? Since I'm not familiar with the details below the state implementations, it will be great if you can share more technical details or some references to me. Thank you! Best Regards, Tony Wei 20

Will that be a problem if POJO key type didn't override equals?

state because I didn't override Object#equals. Is it necessary that POJO key type overrides Object#equals? Or PojoTypeInfo didn't rely on MyClass#equals? Or keyBy() didn't rely on equals? Thank you very much. Best Regards, Tony Wei

Re: JobManager doesn't recover in HA mode

HA's responsibility. One possible way might be using supervisord [1] to launch your JM instance, it can help you monitor your process and automatically restart when the process accidentally failed. Or you can use YARN cluster, the YARN cluster will be responsible for recovering the dead JM. Best,

Re: does the flink sink only support bio?

Hi Stefan, Your reply really helps me a lot. Thank you. 2018-01-08 19:38 GMT+08:00 Stefan Richter : > Hi, > > 1. If `snapshotState` failed at the first checkpoint, does it mean there > is no state and no transaction can be aborted by default? > > > This is a general problem and not only limited

Re: does the flink sink only support bio?

transaction id pool, which has multiple ids to be reused by producer, and it aborts all ids in this pool in the `initializeState`. Is this pool designed for the situation in the first problem or something I haven't noticed? Thank you. Best Regards, Tony Wei 2018-01-04 22:15 GMT+08:00 Stefan Ri

Re: Using latency markers

Hi, I think FLINK-7692 won't solve this problem. It is just a new feature to provide more functionality for user to use metric system. Regards, Tony Wei 2017-12-26 18:23 GMT+08:00 Marvin777 : > Hi, > > I just want to say we're having the same issues. Using latency marke

Re: user driven stream processing

Hi KZ, https://data-artisans.com/blog/real-time-fraud-detection-ing-bank-apache-flink This article seems to be a good example to trigger a new calculation on a running job. Maybe you can get some help from it. Best Regards, Tony Wei 2017-11-29 4:53 GMT+08:00 zanqing zhang : > Hi All, >

kafka consumer client seems not auto commit offset

. My Kafka cluster version is 0.10.2.1. My Zookeeper version for Kafka is 3.4.6-1569965, built on 02/20/2014 09:09 GMT. My Kafka connector library is "org.apache.flink" % "flink-connector-kafka-0.10_2.10" % "1.3.2" Thanks for your help in advance. Best Regards, Tony Wei

Re: Broadcast to all the other operators

and be calculated by the models. I think this approach might be close to the solution in that article, but it could have some difference according to different use cases. Best Regards, Tony Wei 2017-11-09 17:27 GMT+08:00 Ladhari Sadok : > > -- Forwarded message -- > From

Re: Broadcast to all the other operators

. update some rules or 2. refetch all rules from Redis. Is that what you want? Best Regards, Tony Wei 2017-11-09 15:52 GMT+08:00 Ladhari Sadok : > Thank you for the answer, I know that solution, but I don't want to stream > the rules all time. > In my case I have the rules in Redis

Re: Broadcast to all the other operators

.flatMap(...); dataWithRule.keyBy(...).process(...); Hope this will make sense to you. Best Regards, Tony Wei 2017-11-09 6:25 GMT+08:00 Ladhari Sadok : > Hello, > > I'm working on Rules Engine project with Flink 1.3, in this project I want > to update some keyed operator state whe

Re: Use keyBy to deterministically hash each record to a processor/task/slot

which subtask that operator will go to. I'm not sure if this is a good solution and I am also wondering if it can be fulfilled. Best Regards, Tony Wei [1] https://github.com/apache/flink/blob/0182141d41be52ae0cf4a3563d4b8c6f3daca02e/flink-runtime/src/main/java/org/apache/flink/runtime/

Re: count and window question with kafka

Hi, I think ProcessFunction[1] is what you want. You can add it after keyBy and emit the result to sink after timeout or buffer filled. The reference has a good example that show you how to use it. Best Regards, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream

Re: StreamTransformation object

other slotSharing group. Best Regards, Tony Wei 2017-10-27 17:18 GMT+08:00 AndreaKinn : > I'm trying to create an API to get results from build() but anyway I'm > stuck > on the StreamTransformation which is essential to implement > slotSharingGroup(...). I have to pro

Re: StreamTransformation object

Hi Andrea, The `learn` operator is defined in this method [1]. If you need to set its slotSharing group, you should add `slotSharingGroup(...)` behind line 97 [2] or a new API to get the result from `inferenceStreamBuilder.build()`. Best Regards, Tony Wei [1] https://github.com/htm-community

Re: StreamTransformation object

Hi Andrea, In this way, you will only set a slotSharing group on select operator and learn operator will remain in the default group. If you want to set lean operator as well, I am afraid that you need to refactor `HTMStream` to expose `InferenceStreamBuilder.build()`. Best Regards, Tony Wei

Re: StreamTransformation object

.select(new InferenceSelectFunction>() {...}; ((SingleOutputStreamOperator) LCxAccResult).slotSharingGroup("..."); Best Regards, Tony Wei [1] https://github.com/htm-community/flink-htm/blob/master/flink-htm-streaming-java/src/main/java/org/numenta/nupic/flink/streaming/api/HT

Re: StreamTransformation object

`SingleOutputStreamOperator` [1], I think it is easier to just change the return type of that function. And you can leverage the functionality of the `SingleOutputStreamOperator`. Hope this will help you. Best Regards, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/api/java/org/apache/flink

Re: Get EOF from PrometheusReporter in JM

Hi Max, Good to know. Thanks very much. Best Regards, Tony Wei 2017-10-24 13:52 GMT+08:00 Maximilian Bode : > Hi Tony, > > thanks for troubleshooting this. I have added a commit to > https://github.com/apache/flink/pull/4586 that should enable you to use > the reporter wit

Re: Questions about checkpoints/savepoints

Hi, Did you enable externalized checkpoints? [1] Best, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/checkpoints.html#externalized-checkpoints 2017-10-24 13:07 GMT+08:00 vipul singh : > Thanks Aljoscha for the answer above. > > I am experimen

Re: Balancing operation across machines

the parallelism as the number of TMs. Hope this will help you. Best, Tony Wei [1] https://issues.apache.org/jira/browse/FLINK-7851 2017-10-22 10:35 GMT+08:00 Le Xu : > Hello! > > Just wondering if there is a way to load-balancing all executors on one > operator across the machi

Re: Doubts about parallelism

, shuffle, or global, so that the "learn" and "select" can be applied different parallelism setting. Hope this will help you. Best Regards, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/config.html#common-options [2] https://ci.apache.org/projec

Re: Problems with window function

Regards, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/api/java/org/apache/flink/streaming/api/datastream/DataStream.html#keyBy-org.apache.flink.api.java.functions.KeySelector- 2017-10-15 7:00 GMT+08:00 AndreaKinn : > Hi all, > I'm trying to implement a ti

Re: Question about configuring Rich Functions

Hi Steve, I think the discussion in this thread [1] could answer your questions. Best Regards, Tony Wei [1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/RichMapFunction-parameters-in-the-Streaming-API-td16121.html Steve Jerman 於 2017年10月14日 週六，上午12:41寫道： > This docum

Re: Windowing isn't applied per key

t all keyed stream use the same strategy. That is the definition of sliding window in Streaming API. And the definition of yours is not supported by Flink intuitively. One way is to implement it by yourself with ProcessFunction. Best Regards, Tony Wei mclendenin 於 2017年10月11日 週三，下午8:52寫道： > Hi Tony

Re: Windowing isn't applied per key

trigger each window, and all these will be done on each key. [2] Best Regards, Tony Wei [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html#sliding-windows [2] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/process_function.html#example 2017-10

Re: Stream Task seems to be blocked after checkpoint timeout

Hi Stefan, Thank you very much. I will try to investigate what's the problem on my cluster and S3. BTW, Is there any Jira issue associated with your improvement, so that I can track it? Best Regards, Tony Wei 2017-10-03 16:01 GMT+08:00 Stefan Richter : > Hi, > > from the stack t

Re: Stream Task seems to be blocked after checkpoint timeout

asynchronous checkpoints both machine's resources and s3 connection. BTW, s3 serves both operator and checkpointing and I used aws java api to access s3 in upload operator in order to control where the files go. Best Regards, Tony Wei Stefan Richter 於 2017年9月28日 週四，下午7:43寫道： > Hi, > > the g

Re: Stream Task seems to be blocked after checkpoint timeout

Hi, Sorry. This is the correct one. Best Regards, Tony Wei 2017-09-28 18:55 GMT+08:00 Tony Wei : > Hi Stefan, > > Sorry for providing partial information. The attachment is the full logs > for checkpoint #1577. > > Why I would say it seems that asynchronous part was not exec

Re: Stream Task seems to be blocked after checkpoint timeout

had already arrived at the operator and started as soon as when the JM triggered the checkpoint? Best Regards, Tony Wei 2017-09-28 18:22 GMT+08:00 Stefan Richter : > Hi, > > I agree that the memory consumption looks good. If there is only one TM, > it will run inside one JVM.

Re: Stream Task seems to be blocked after checkpoint timeout

o resource from thread pool. It is better to measure the time between creation time and processing time, and log it and checkpoint id with the original log that showed what time the asynchronous part took. Best Regards, Tony Wei 2017-09-28 16:25 GMT+08:00 Stefan Richter : > Hi, > > when t

Re: Stream Task seems to be blocked after checkpoint timeout

oints/7c039572b13346f1b17dcc0ace2b72c2/chk-1245/eedd7ca5-ee34-45a5-bf0b-11cc1fc67ab8 2017-09-26 18:13:34 37419 tony-dev/flink-checkpoints/7c039572b13346f1b17dcc0ace2b72c2/chk-1246/9aa5c6c4-8c74-465d-8509-5fea4ed25af6 Hope these informations are helpful. Thank you. Best Regards, Tony Wei 2017-09-27 16:14 GMT+

Re: Stream Task seems to be blocked after checkpoint timeout

picture I sent in the last email. I will keep my eye on taking a thread dump from that JVM if this happens again. Best Regards, Tony Wei 2017-09-26 23:46 GMT+08:00 Stefan Richter : > Hi, > > that is very strange indeed. I had a look at the logs and there is no > error or exception

Re: Question about Flink Metrics

round for now. Thank you! Best Regards, Tony Wei 2017-09-26 19:13 GMT+08:00 Hai Zhou : > Hi Tony, > > you can consider implementing a reporter, use a trick to convert the > flink's metrics to the structure that suits your needs. > > This is just my personal practice, ho

Question about Flink Metrics

porters work, I am afraid that it is not a good design to just fulfill the requirement on particular reporter. Please guide and thanks for your help. Best Regards, Tony Wei [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#scope [2]: https://ci.apache.org

  1   2   >