Re: Migrating Kafka Sources (major version change)

2021-07-25 Thread JING ZHANG
Hi Dan, Do you plan to continue to read a new Kafka topic after finished read current Kafka topic? If yes, Your plan could works. BTW, if the schema of data in the new Kafka topic and the current topic are same with each other, however their topic name are different with each other, maybe you coul

Re: Finding matched state

2021-07-25 Thread JING ZHANG
Hi Abu, Would you please explain more about your case? What input information in your case? Type about a given state? In what scope do you need to search for? A given job or a given operator? For example, input is: a state with Long value state, try to find out all states with Long value state sear

Re: FlinkKinesis consumer

2021-07-25 Thread Caizhi Weng
Hi! It's stated on the line just below that in the document. It is recommended to monitor the shard distribution and adjust assignment > appropriately. A custom assigner implementation can be set via > setShardAssigner(KinesisShardAssigner) to optimize the hash function or use > static overrides

Re: How to keep flink batch job running continously on local

2021-07-25 Thread Caizhi Weng
Hi! Flink 1.7.2 is a very dated version. I suppose by batch job you mean DataSet API? A socket source might be what you need. However I'm not sure if the provided socket source in 1.7 supports DataSet API. If not you might need to write your own socket source. Samir Vasani 于2021年7月24日周六 下午9:31写

Re: Move already processed file from one folder to another folder in flink

2021-07-25 Thread Caizhi Weng
Hi! For the UDF solution, you can add a "file name" column to your csv file like this: id,value,filename 1,100, 2,200, 3,300,test.csv Only the filename of the last record of the csv file is filled, so that this indicates the end of file. Then write a UDF like this: public class MyUDF extends Sc

Finding matched state

2021-07-25 Thread Abu Bakar Siddiqur Rahman Rocky
Hello, is there any library or tools to get the matched state (if the current state is similar to the previous state, in that case)? Thank you -- Regards, Abu Bakar Siddiqur Rahman