Re: Map Of DataStream getting NullPointer Exception

Hi Khachatryan, This is the use case to create multiple streams: I have a use case where multiple types of Avro records are coming in single Kafka topic as we are suing TopicRecordNameStrategy for the subject in the schema registry. Now I have written a consumer to read that topic and build a Dat

Re: How JobManager and TaskManager find each other?

/So do you mean the ip address changes during running or the taskmanager failed and relaunched with a same hostname, but the ip address is different?/ Well that also but actually I was thinking about running FLink on PaaS platforms where process can be re-spawned during runtime on different machin

Re: Emit message at start and end of event time session window

Hi Rafi and Till, Thank you for pointing out that edge case, Rafi. Till, I am trying to get this example working with the BroadcastState pattern upstream to the window operator[1]. The problem is that introducing the BroadcastState makes the onEventTime() *never* fire. Is the BroadcastState someho

Has the "Stop" Button next to the "Cancel" Button been removed in Flink's 1.9 web interface?

Hello, before Flink 1.9 I was able to "Stop" a streaming pipeline - after clicking that button in the webinterface it performed a clean shutdown. Now with Flink 1.9 I just see the option to cancel it. However, using the commandline flink stop -d 266c5b38cf9d8e61a398a0bef4a1b350 still does the tri

Re: Does Flink 1.9 support create or replace views syntax in raw sql?

Hi, 1) Where does the hive catalog persist view definitions? in mysql? or HDFS? 2) If the views are not persisted what happens if the application crashes and restarted? will it create the view again and safely read the data from where it left off? Thanks! On Wed, Feb 26, 2020 at 6:47 AM godfrey

How can I programmatically set RocksDBStateBackend?

Hi All, How can I programmatically set RocksDBStateBackend? I did the following [image: Screen Shot 2020-02-27 at 4.53.38 AM.png] env.setStateBackend always shows deprecated. so what is the right way to do this in flink 1.10? Thanks!

Re: How can I programmatically set RocksDBStateBackend?

Hi, I had same case but with FsStateBackend. Cast it to StateBackend type. env.setStateBackend((StateBackend) new FsStateBackend("some/Path")); I think this is some inconsistency in API. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink on Kubernetes - Session vs Job cluster mode and storage

Hi Jin Yi, For standalone per-job cluster, it is a little different about the recovery. Just as you say, the user jar has built in the image, when the JobManager failed and relaunched by the K8s, the user `main()` will be executed again to get the job graph, not like session cluster to get the job

Re: Map Of DataStream getting NullPointer Exception

Hi Anuj, is that piece of code in your first mail in the same main? Then at this point, nothing has been executed in Flink. So we are looking at a normal Java programming error that you can easily debug and unit test. Most likely, there is no event config for *SEARCH_LIST_KEYLESS* or there is a s

Re: Does Flink 1.9 support create or replace views syntax in raw sql?

https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_catalog.html#hivecatalog Can I use the hive catalog to store view definitions in HDFS? I am assuming the metastore can be anything or does it have to be have MySQL? On Thu, Feb 27, 2020 at 4:46 AM kant kodali wrote:

Re: Does Flink 1.9 support create or replace views syntax in raw sql?

HI kant, > 1) Where does the hive catalog persist view definitions? in mysql? or HDFS? hive catalog stores all metadata in derby or mysql [1] > 2) If the views are not persisted what happens if the application crashes and restarted? will it create the view again and safely read the data from wher

Re: Colocating Sub-Tasks across jobs / Sharing Task Slots across jobs

Hi Ben, I think at the moment, it is not possible because of current scheduling design which Xintong has already mentioned. The jobs are completely isolated and there is no synchronisation between their deployment. Alignment of tasks by e.g. key groups in general is difficult as it is up to the sc

Re: Async Datastream Checkpointing

Hi, We tried to use ExecutionContext.fromExecutor(Executors.newFixedThreadPool(1)) as executor and we still have this problem. We even tried even other executors, but still the same. Only once we succeded to have a checkpoint when we tried without the web requests. Do you have any other ideas? T

How are jobs balanced between nodes?

Hi, running 1.10 I have 3 nodes with 12 slots per node. So 36 task slots total. I deployed 9 jobs. The scheduler put all jobs on 2 nodes and left the 3rd node completely empty. Should it not try to fairly distribute the task over all nodes?

Re: Async Datastream Checkpointing

Hi Alexandru, I cannot follow what you posted, so let me point you to one example [1]. The executor needs to be used inside the async function. [1] https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/AsyncIOEx

Re: How are jobs balanced between nodes?

Please see the 1.10 release notes on the matter. On 27/02/2020 19:15, John Smith wrote: Hi, running 1.10 I have 3 nodes with 12 slot

invoking REST call at job startup

At job startup, I need to make a single REST client call. Is there a suggested REST client to use for this?

[Native Kubernetes] [Ceph S3] Unable to load credentials from service endpoint.

Hi, I have a problem with accessing my own S3 system from within Flink when running on Kubernetes. *TL;DR* I have my own S3 (Ceph), Locally my application works, when running in K8s it fails with Caused by: com.amazonaws.SdkClientException: Unable to load credentials from service endpoint Caused

Re: How are jobs balanced between nodes?

Thanks! On Thu, 27 Feb 2020 at 15:24, Chesnay Schepler wrote: > Please see the 1.10 release notes > > on the matter. > > On 27/02/202

The main method caused an error: Unable to instantiate java compiler in Flink 1.10

Hi All, My sample program works in Flink 1.9 but in 1.0 I get the following error when I am submitting the job. otherwords it fails to submit a job. any idea? Thanks! org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Unable to instantiate java c

Re: The main method caused an error: Unable to instantiate java compiler in Flink 1.10

Fixed the typo. Hi All, My sample program works in Flink 1.9 but in 1.10 I get the following error when I am submitting the job. otherwords it fails to submit a job. any idea? Thanks! On Thu, Feb 27, 2020 at 2:19 PM kant kodali wrote: > Hi All, > > > My sample program works in Flink 1.9 bu

Re: The main method caused an error: Unable to instantiate java compiler in Flink 1.10

Below is the sample code using Flink 1.10 public class Test { public static void main(String... args) throws Exception { EnvironmentSettings bsSettings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build(); final StreamExecutionEnvironment env = Stre

Re: The main method caused an error: Unable to instantiate java compiler in Flink 1.10

It works within IDE but not when I submit using command using flink run myApp.jar On Thu, Feb 27, 2020 at 3:32 PM kant kodali wrote: > Below is the sample code using Flink 1.10 > > public class Test { > > public static void main(String... args) throws Exception { > > EnvironmentSet

Re: Flink on Kubernetes - Session vs Job cluster mode and storage

Hi Hao Sun, I just post the explanation to the user ML so that others could also have the same problem. Gven the job graph is fetched from the jar, do we still need Zookeeper for > HA? Maybe we still need it for checkpoint locations? Yes, we still need the zookeeper(maybe in the future we will

Re: The main method caused an error: Unable to instantiate java compiler in Flink 1.10

Hi Kant, Are you depending a custom janino or something like hive-exec in your pom.xml? Best, Jark On Fri, 28 Feb 2020 at 10:13, kant kodali wrote: > It works within IDE but not when I submit using command using flink run > myApp.jar > > > > On Thu, Feb 27, 2020 at 3:32 PM kant kodali wrote:

Re: [Native Kubernetes] [Ceph S3] Unable to load credentials from service endpoint.

Hi Niels, Glad to hear that you are trying Flink native K8s integration and share you feedback. What is causing the differences in behavior between local and in k8s? It > works locally but not in the cluster. In your case, the job could be executed successfully local. That means S3 endpoint cou

Re: How JobManager and TaskManager find each other?

Well i think the biggest challenge when running Flink on PaaS platforms is about the NAT network. Since the jobmanager/taskmanagers are running in a NAT network. So they could not talk directly. What we need to do is make the JM/TM to bind the local machine hostname and use a bind-port for the comm

Re: The main method caused an error: Unable to instantiate java compiler in Flink 1.10

Hi kant, As Jark said, Your user jar should not contains " org.codehaus.commons.compiler.ICompilerFactory" dependencies. This will make calcite can not work. In 1.10, have made Flink client respect classloading policy that default policy is child first [1]. More details can find in [2]. [1] http

Re: Emit message at start and end of event time session window

Hi, This problem is solved[1]. The issue was that the BroadcastStream did not contain any watermark, which prevented watermarks for any downstream operators from advancing. I appreciate all the help. [1] https://stackoverflow.com/questions/60430520/how-do-i-fire-downstream-oneventtime-method-when-u

Flink 1.10 exception : Unable to instantiate java compiler

Hi community, now I am using the flink 1.10 to run the flink task ,cluster type is yarn . I use commandline to submit my flink job , the commandline just like this : flink run -m yarn-cluster --allowNonRestoredState -c xxx.xxx.xx flink-stream-xxx.jar Bug there is a exception to

Re: The main method caused an error: Unable to instantiate java compiler in Flink 1.10

Here is my build.gradle and I am not sure which jar uses org.codehaus.commons.compiler.ICompilerFactory buildscript { repositories { jcenter() // this applies only to the Gradle 'Shadow' plugin } dependencies { classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.4

Re: Flink 1.10 exception : Unable to instantiate java compiler

Same problem! On Thu, Feb 27, 2020 at 11:10 PM LakeShen wrote: > Hi community, > now I am using the flink 1.10 to run the flink task > ,cluster type is yarn . I use commandline to submit my flink job , the > commandline just like this : > > flink run -m yarn-cluster --allowNonRe