Re: Live updating Serialization Schemas in Flink

Hi, Unfortunately I don't have a nice solution for you. I would also generally discourage such a pattern. Usually how multiple/dynamic schemas are used is with a help of schema registry. In that case you have some sort of an id serialized along with records which you can use to look up the schema.

Re: why we need keyed state and operate state when we already have checkpoint?

I think there is some misunderstanding here: a checkpoint IS (a snapshot of) the keyed state and operator state (among a few more things). [1] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/learn-flink/fault_tolerance.html#definitions On Wed, Oct 7, 2020 at 6:51 AM 大森林 wrote:

?????? why we need keyed state and operate state when we already have checkpoint?

when the job is killed,state is also misssing. so why we need keyed state?Is keyed state useful when we try to resuming the killed job? --  -- ??:

Re: why we need keyed state and operate state when we already have checkpoint?

The checkpoint is a snapshot for the job and we can resume the job if the job is killed unexpectedly. The state is another thing to memorize the intermediate result of calculation. I don't think the checkpoint can replace state. 大森林 于2020年10月7日周三 下午12:26写道： > Could you tell me: > > why we need k

why we need keyed state and operate state when we already have checkpoint?

Could you tell me: why we need keyed state and operator state when we already have checkpoint? when a running jar crash,we can resume from the checkpoint automatically/manually. So why did we still need keyed state and operator state. Thanks

S3 StreamingFileSink issues

First, let me say, Flink is super cool - thanks everyone for making my life easier in a lot of ways! Wish I had this 10 years ago Onto the fun stuff: I am attempting to use the StreamingFileSink with S3. Note that Flink is embedded in my app, not running as a standalone cluster. I am having a

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

@Aljoscha - Thanks! That setup lets fixing the hacky absolute path reference. However, the actual log calls are not printing to the console. Only errors appear in my terminal window and the test logs. Maybe console logger does not work for this junit setup. I'll see if the file version works.

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

What Aljoscha suggested is what works for us! On Tue, Oct 6, 2020 at 6:58 PM Aljoscha Krettek wrote: > Hi Dan, > > to make the log properties file work this should do it: assuming the > log4j.properties is in //src/main/resources. You will need a BUILD.bazel > in that directory that has only the

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

Hi Dan, to make the log properties file work this should do it: assuming the log4j.properties is in //src/main/resources. You will need a BUILD.bazel in that directory that has only the line "exports_files(["log4j.properties"]). Then you can reference it in your test via "resources = ["//src/

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

I'm trying to use Table API for my job. I'll soon try to get a test working for my stream job. - I'll parameterize so I can have different sources and sink for tests. How should I mock out a Kafka source? For my test, I was planning on changing the input to be from a temp file (instead of Kafka).

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

Thanks! Great to know. I copied this junit5-jupiter-starter-bazel rule into my repository (I don't think junit5 is supported directly with java_test yet). I tried a few ways of bundling `log4j.properties` into

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

Oops, this is actually the JOIN issue thread [1]. Guess I should revise my previous "haven't had issues" statement hah. Sorry for the spam! [1]: apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Streaming-SQL-Job-Switches-to-FINISHED-before-all-records-processed-td38382.html On Tue, Oc

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

Unless it's related to this issue[1], which was w/ my JOIN and time characteristics, though not sure that applies for batch. Best, Austin [1]: apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SQL-Streaming-Join-Creates-Duplicates-td37764.html On Tue, Oct 6, 2020 at 6:20 PM Aus

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

Hey Dan, We use Junit5 and Bazel to run Flink SQL tests on a mini cluster and haven’t had issues, though we’re only testing on streaming jobs. Happy to help setting up logging with that if you’d like. Best, Austin On Tue, Oct 6, 2020 at 6:02 PM Dan Hill wrote: > I don't think any of the gotch

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

I don't think any of the gotchas apply to me (at the bottom of this link). https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#junit-rule-miniclusterwithclientresource I'm assuming for a batch job that I don't have to do anything for: "You can implement a custom parallel

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

I've tried to enable additional logging for a few hours today. I think something with junit5 is swallowing the logs. I'm using Bazel and junit5. I setup MiniClusterResourceConfiguration using a custom extension. Are there any known issues with Flink and junit5? I can try switching to junit4. W

Re: List of Running Flink Jobs from Long Running Flink Yarn Cluster (AWS EMR)

Never mind. I was trying to use EMR API, but i realized that i could use the Flink REST API Directly and achieve the same. Thanks Sateesh -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

?????? what's the example for datastream data generator?

My java is poor I have tried like this,but it's wrong,Could you tell me how to fix this... import org.apache.flink.streaming.api.datastream.DataStream; import org.apache.flink.streaming.api.environment.LocalStreamEnvironment; import org.apache.flink.streaming.api.environment.StreamExecutionEnviro

Re: what's the example for datastream data generator?

Hi, I think you can take a look at *org.apache.flink.table.planner.plan.nodes.common.CommonPhysicalTableSourceScan#createSourceTransformation*, which will tell you how to get transformations by source function and stream execution environment. In datastream api, we also have a DataGen that is *org.

The file STDOUT does not exist on the TaskExecutor

Hi, I am running dockerized flink:1.11.0-scala_2.11 container in ecs. I am getting the following error after the job runs: ERROR org.apache.flink.runtime.rest.handler.taskmanager. TaskManagerStdoutFileHandler [] - Unhandled exception. org.apache.flink.util.FlinkException: The file STDOUT does not

how to simulate the scene "back pressure" in flink?Thanks~!

I want to learn the concept "back pressure". but I can not find the datastream generator example to generate a lot of data. besides, is there any example on how to simulate the scene "back pressure"  in WEB UI? Thanks for your help~ 

Is it possible that late events are processed before the window?

I have a job with event-time session window of 30 minutes. I output late events to side output, where I have a tumbling processing time window of 30 minutes. I observe that the late events are written to storage before the "main" events. I wanted to know if it's normal before digging into the co

windowsState() and globalState()

Dear all:       How do I use the windowsState() method and the globalState() method in process Windows Function? Can I give a demo? Thanks, Jiazhi

Re: Reading from Multiple Directories with StreamExecutionEnvironment

Hi Jason, Your analysis seems correct. As an alternative, you could: 1) either call readFile multiple times on the StreamExecutionEnvironment (once for each dir you want to monitor) and then union the streams, or 2) you could put all the dirs you want to monitor under a common parent dir and spec

?????? need help about "incremental checkpoint",Thanks

I have solved it. I resumed from the wrong class in the jar. while the jar contains two different experiment. Thanks for your help. Please close my issue. MUCH MUCH THANKS! --  -- ??:

Re: 回复： need help about "incremental checkpoint",Thanks

I'm forwarding my comment from the Jira Issue [1]: In https://github.com/appleyuchi/wrongcheckpoint/blob/master/src/main/scala/wordcount_increstate.scala you set the RocksDBStateBackend, in https://github.com/appleyuchi/wrongcheckpoint/blob/master/src/main/scala/StateWordCount.scala you set t

?????? need help about "incremental checkpoint",Thanks

I don't  know where I did change the state backends. There are two meaning of "restarting": ??Restarting automatically(success in my experiment) ??Restarting manually(failure in my experiment) The whole experiment(just a wordcount case) and steps are listed in my github: https://github.com/appl

Re: need help about "incremental checkpoint",Thanks

This error comes because you changed state backends. The checkpoint was written by a different state backend. This is not supported. To use incremental checkpoints, you must only use the RocksDbStateBackend: first, when running the job and writing the checkpoint, and again later when restarting.

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

Hi Dan, there were some bugs and quirks in the MiniCluster that we recently fixed: - https://issues.apache.org/jira/browse/FLINK-19123 - https://issues.apache.org/jira/browse/FLINK-19264 But I think they are probably unrelated to your case. Could you enable logging and see from the logs whet

Re: ConnectionPool to DB and parallelism of operator question

Hi, since I don't know the implementation of the Sink I can only guess. I would say you get 82 * 300 connections because you will get 82 instances of a sink operator and each of those would then have a connection pool of 300 connections. The individual sink instances will (potentially) run on

Re: Guide on writing Flink plugins

I would recommend looking at one of the existing reporter modules, like flink-metrics-slf4j. The important bit plugin-wise is to have a factory class and the META-INF/services entry. On 10/6/2020 5:23 AM, Kien Truong wrote: Hi all, We want to write a Flink plugins to integrate Flink jobs wi