Re: HA setting for per-job YARN session

Thanks Gary, I have tested those ha settings and it works well. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: How to submit a job with dependency jars by flink cli in Flink 1.4.2?

Hi, I’m glad that you have found a solution to your problem :) To shorten feedback you can/should test as much logic as possible using smaller unit tests and some small scale integration tests: https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/testing.html

Re: Event Time Session Window does not trigger..

Hi, By setting the time characteristic to EventTime, you enable the internal handling of record timestamps and watermarks. In contrast to EventTime, ProcessingTime does not require any additional data. You can use both, EventTime and ProcessingTime in the same application and StreamExecutionEnvir

connection failed when running flink in a cluster

Hello everyone, I am trying to run Flink on Raspberry Pis. My first test for word count in a single node worked. I just have to decrease the Heap memory of the jobmanager.heap.mb and taskmanager.heap.mb to 512. My second test is to add 2 slave nodes I got the error: "Java HotSpot(TM) Client VM war

Re: connection failed when running flink in a cluster

Did you start job manager and task manager on the same resbery pi ? On Mon, 6 Aug 2018, 12:01 Felipe Gutierrez, wrote: > Hello everyone, > > I am trying to run Flink on Raspberry Pis. My first test for word count in > a single node worked. I just have to decrease the Heap memory of the > jobmana

Re: connection failed when running flink in a cluster

yes. when I execute the jps command on the master node I see TaskManagerRunner and StandaloneSessionClusterEntrypoint (which I believe it is the jobManager). On the slave nodes I see TaskManagerRunner when I run jps command *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https:

Re: connection failed when running flink in a cluster

Hi Felipe, >From the exception information, it seems that you did not start the socket server, the socket source needs to connect to the socket server. Please make sure the socket server has started and is available. Thanks, vino. 2018-08-06 18:45 GMT+08:00 Felipe Gutierrez : > yes. > > when I

Re: Flink Forwards 2018 videos

Hi Elias, If you are using an ad blocker, can you turn it off and try it again? I have forwarded your email internally at data Artisans, and we are working on a proper solution. Best, Gary On Sun, Aug 5, 2018 at 8:30 PM, Elias Levy wrote: > It appears the Flink Forwards 2018 videos are FUBAR.

Re: connection failed when running flink in a cluster

do you mean "nc -l 9000"? If so, I did start before. the task manager running on the master can connect to the job manager. but the task manager on the slave node cannot. The second time that I start the WordCount task it recognizes only one task manager (from the master) and runs my task. But the

Re: connection failed when running flink in a cluster

Hi, nc exits after the first connection is closed. Are you re-running the nc command every time the job finishes? The stacktrace you copied does not indicate that a TaskManager cannot connect to the JobManager. I can only see that the SocketTextStreamFunction (from the SocketWindowWordCount job?)

Re: connection failed when running flink in a cluster

yes. with this example (examples/streaming/WordCount.jar) my cluster worked. the file log/*out from the master is still empty and the file log/*out from the slave node has my result. The dashboard also shows that the job is completed. So, like you said there are some external dependencies that I

Re: Delay in REST/UI readiness during JM recovery

Hi Joey, Did you create these JIRA issues based on Till's suggestion? If you didn't create them or you don't know how to do it, I can do it for you. But I won't do it right away, I will wait for a while. Thanks, vino. 2018-08-03 17:23 GMT+08:00 Till Rohrmann : > Hi Joey, > > your analysis is c

Re: connection failed when running flink in a cluster

Hi Felipe, You got the result? And the web UI shown the job is completed? If it throws the exception you provided, the job's status should be failed. Thanks, vino. 2018-08-06 23:42 GMT+08:00 Felipe Gutierrez : > yes. with this example (examples/streaming/WordCount.jar) my cluster > worked. > >

Re: connection failed when running flink in a cluster

Hi Vino, the UI shows the job as completed. I had run "./bin/flink run examples/streaming/WordCount.jar" and I get no error. When I start netcat "nc -l 9000" and in other terminal I run "./bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000" I have this exception. Starting exe

Re: Event Time Session Window does not trigger..

Hi Hequn and Fabian, Thanks. Appreciate your help On Mon, Aug 6, 2018 at 1:32 AM, Fabian Hueske wrote: > Hi, > > By setting the time characteristic to EventTime, you enable the internal > handling of record timestamps and watermarks. > In contrast to EventTime, ProcessingTime does not require an

Re: Converting a DataStream into a Table throws error

Hi, I resolved this issue of java.lang.NoClassDefFoundError: org/apache/flink/table/api/TableEnvironment By adding the jar file flink-table_2.11-1.5.0.jar To $FLINK_HOME/lib It compiles and run OK now. Rather strange as I had this dependency in my SBT libraryDependencies += "org.apache.flin

Re: Converting a DataStream into a Table throws error

The problem is that you declared it as provided. This means the build tool assumes it will be there and therefore does not include it in the Jar file. By adding it to the lib folder you are providing the dependency. Best, Fabian 2018-08-06 18:58 GMT+02:00 Mich Talebzadeh : > Hi, > > I resolved t

Re: Converting a DataStream into a Table throws error

Thanks Fabian, I looked at the maven and this is what it says *provided* [image: image.png] However, this jar file is not shipped with Flink? Is this deliberate? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Accessing source table data from hive/Presto

Hi all, I read the Flink documentation and came across the connectors supported https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/index.html#bundled-connectors We have some data that resides in Hive/Presto that needs to be made available to the flink job. The data in the

Re: connection failed when running flink in a cluster

Hi, Can you try submitting with: ./bin/flink run examples/streaming/SocketWindowWordCount.jar --hostname --port 9000 where IP is the IP of the node where you started nc? If not specified, the default hostname is localhost. This problematic is if the source operator is scheduled on a differe

Running SQL to print to Std Out

Hi, This is the streaming program I have for trade prices following the doc for result set for tables https://flink.apache.org/news/2017/03/29/table-sql-api-update.html val streamExecEnv = StreamExecutionEnvironment.getExecutionEnvironment val ds = streamExecEnv .addSource(new F

flink 1.4.2 NPE after job startup when using managed ValueState

Dear Flink Users, I have two environments, one old, one new. I'm trying to migrate my flink job. The exact same application code that runs in one environment, fails in the second with a very confusing NPE (listed below). I can garuntee the application code only calls ValueState.get() inside of a

Using a custom DeserializationSchema with Kafka and Python

Hi, I'm trying to write a pipeline using the new Python streaming API, which reads from Kafka using FlinkKafkaConsumer010. This works fine when using an existing deserializer like the SimpleStringSchema but I need to define my own deserializer to process a custom format. I've written a class whic

Re: Delay in REST/UI readiness during JM recovery

Thanks for the ping Vino. I created two JIRAs for the first two items: 1) https://issues.apache.org/jira/browse/FLINK-10077 2) https://issues.apache.org/jira/browse/FLINK-10078 Regarding (3) we’re doing some testing with different options for the state storage. I’ll report back if we find anyth

Re: Converting a DataStream into a Table throws error

Hi Mich, I think this is the behavior of the compiler. When run your job in local, you have to remove the provided or add jar to the lib path. But if run on cluster, you have to add the provided to ignore flink classes, since these classes are already exist in your installation version. Best, Heq

Re: Accessing source table data from hive/Presto

Hi srimugunthan, I found a related link[1]. Hope it helps. [1] https://stackoverflow.com/questions/41683108/flink-1-1-3-interact-with-hive-2-1-0 On Tue, Aug 7, 2018 at 2:35 AM, srimugunthan dhandapani < srimugunthan.dhandap...@gmail.com> wrote: > Hi all, > I read the Flink documentation and ca

Re: Running SQL to print to Std Out

Hi Mich, When you print to stdout on cluster, you have to look at the taskmanager .out file (also available in the UI). Best, Hequn On Tue, Aug 7, 2018 at 4:48 AM, Mich Talebzadeh wrote: > Hi, > > This is the streaming program I have for trade prices following the doc > for result set for tabl

Re: Delay in REST/UI readiness during JM recovery

Hi Joey, Thank you for finding these issues and creating them. Thanks, vino. 2018-08-07 8:18 GMT+08:00 Joey Echeverria : > Thanks for the ping Vino. > > I created two JIRAs for the first two items: > > 1) https://issues.apache.org/jira/browse/FLINK-10077 > 2) https://issues.apache.org/jira/brow

Re: Running SQL to print to Std Out

Hi Mich, Hequn is correct, click on the “Task Managers” menu located on the left side of the Flink web UI, select a TM from the TM list, and then click on the “Stdout” option in the TM Details tab. Thanks, vino. 2018-08-07 9:42 GMT+08:00 Hequn Cheng : > Hi Mich, > > When you print to stdout on

Re: Converting a DataStream into a Table throws error

Ok gents thanks for clarification. Regards, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaimer:*

Could not build the program from JAR file.

Hi all, I'm trying to run the wordCount example on my YARN cluster and this is not working.. I get the error message specified in title: Could not build the program from JAR file. > $ ./bin/flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 > ./examples/batch/WordCount.jar > Setting HADOOP

Re: Could not build the program from JAR file.

Hi Florian, You write that Flink 1.4.2 works but what version is not working for you? Best, Gary On Tue, Aug 7, 2018 at 8:25 AM, Florian Simond wrote: > Hi all, > > > I'm trying to run the wordCount example on my YARN cluster and this is not > working.. I get the error message specified in ti