Re: Flink 1.5.0 no more allow multiple program argument?

Thanks Chesnay! I guess it is related to StringJoiner(",") used in Flink 1.5 :( From: Chesnay Schepler Sent: Tuesday, June 19, 2018 1:44 PM To: user@flink.apache.org Subject: Re: Flink 1.5.0 no more allow multiple program argument? For now you'll have to replace

Re: Flink 1.5.0 no more allow multiple program argument?

For now you'll have to replace all commas with some other delimiter. On 19.06.2018 04:15, Siew Wai Yow wrote: Hi, I get the following error when upgrade flink from 1.3.2 to 1.5.0 when using REST api to upload and run a jar. {"errors":["Expected only one value [--KAFKA_IN PREBLN_O@192.168

Flink 1.5.0 no more allow multiple program argument?

Hi, I get the following error when upgrade flink from 1.3.2 to 1.5.0 when using REST api to upload and run a jar. {"errors":["Expected only one value [--KAFKA_IN PREBLN_O@192.168.56.120:9092, 192.168.56.121:9092, 192.168.56.122:9092/BHARTI_FL_PREBLN_O_124 --KAFKA_OUT FX_AGGR_ASCII@192.168.56

Re: [DISCUSS] Flink 1.6 features

Hi, Sagar Thank your for your review. I will fix it when available. > 2. Will you be able to add more unit tests in the commit ? Eg. Writing some > example data with simple schema which will initialize OrcWriter object and > sinking it to local hdfs node ? Ans: Yes. I will add more unit test

Re: [DISCUSS] Flink 1.6 features

Thanks @zhangminglei for replying. I agree, hive on Flink would be a big project. By the way, i looked at the Jira ticket related to ORC format which you shared. Couple of comments/requests about the pull request in th ticket: 1. Sorry for nitpicking but meatSchema is mispelled. I think it shou

Flink 1.5 Yarn Connection unexpectedly closed

Hey all, My jobs that I am trying to write in Flink 1.5 are failing after a few minutes. I think its because the idle task managers are shutting down, which seems to kill the client and the running job. The running job itself was still going on one of the other task managers. I get: org.apache

Control insert database with dataset

Hello, I'm trying catch a BatchUpdateException when insert DataSet using a method output. Because, I need control if insert a duplicate key. How I can do this? [2018-06-18 22:18:56,419] INFO DataSink (org.apache.flink.api.java.io.jdbc.JDBCOutputFormat@64aad6db) (1/1) (00a77c9e18f893cde9c62a3

Test Bounce

Test

Metrics from detached execution mode

Hi! I am trying to fetch metrics provided by Beam SDK via Flink runner in detached mode, but looks like it is not supported yet. I understand from class DetachedJobExecutionResult that metrics are not supported to be extracted in detached mode job execution. Is this a limitation of Flink as a runn

Re: Exception while submitting jobs through Yarn

Yes, it is. On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann wrote: > Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to ` > /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`? > > On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma wrote: > >> I don't think I can access core-default

Re: Memory Leak in ProcessingTimeSessionWindow

Right, thats where I am headed now but was wondering there are any “gochas” I am missing before I try and dig into a few gigs of heap dump.  Thanks, Ashish Sent from Yahoo Mail for iPhone On Monday, June 18, 2018, 3:37 AM, Stefan Richter wrote: Hi, can you take a heap dump from a JVM that r

Passing records between two jobs

Hello, We are planning a system that will be comprised of 3 different jobs: 1. Getting a stream of events, adding some metadata to the events, and outputting them to a temporary message queue. 2. Performing some calculations on the events we got from job 1, as required for product A. 3.

Re: Flink application does not scale as expected, please help!

Thanks @Fabian for your confirmation about the better performance when scaling happened at same TM machine. But it is so funny that it give impression "the more I scale the less I get" when the performance drop with more TM in play. @Ovidiu question is interesting to know too. @Till do you mind

Re: Exception while submitting jobs through Yarn

Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to ` /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`? On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma wrote: > I don't think I can access core-default as it comes with Hadoop jar > > On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrman

Debug job execution from savepoint

Hi all, I would like to test my checkpointing implementation doing a step-by-step debugging under an IDE. Is there a way to restore a job from a local savepoint in a local stream environnement, like a command "flink run -s :savepontpath" would do ? Thanks, Manuel

Re: Exception while submitting jobs through Yarn

I don't think I can access core-default as it comes with Hadoop jar On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann wrote: > Hmm, could you check whether core-default.xml contains any suspicious > entries? Apparently xerces:2.9.1 cannot read it. > > On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma w

Re: Exception while submitting jobs through Yarn

Hmm, could you check whether core-default.xml contains any suspicious entries? Apparently xerces:2.9.1 cannot read it. On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma wrote: > Hi, > > After putting the following log in my code, I can see that the Xerces > version is - Xerces version : Xerces-J 2.9

Re: Exception while submitting jobs through Yarn

Hi, After putting the following log in my code, I can see that the Xerces version is - Xerces version : Xerces-J 2.9.1 log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion()); Also, following is the response of *$* *locate xerces* command on the server - /usr/hdp/2.6.1.0-1

Re: Exception while submitting jobs through Yarn

Could you check which xerces version you have on your classpath? Apparently, it cannot read core-default.xml as Ted pointed out. This might be the root cause for the failure. Cheers, Till On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma wrote: > Hi, > > Sorry for the confusion, but the yarn is run

Re: flink and akka HTTP

Try adding akka-remote:2.5.11 to see whether it resolves the issues. The problem is that you have akka-actor as a dependency which is also required by Flink. If Flink loads akka-actor:2.5.11 and then tries to use akka-remote:2.4.x, this might cause the problem. Cheers, Till On Mon, Jun 18, 2018 a

[ANNOUNCE] Weekly community update #25.

Dear community, this is the weekly community update thread #25. Please post any news and updates you want to share with the community to this thread. # Making Flink Table Scala free The community started discussing whether and how to make flink-table Scala free [1]. This change would be an incre

RE: flink and akka HTTP

Here are the dependencies: runtimeClasspath - Runtime classpath of source set 'main'. +--- org.apache.flink:flink-connector-kafka-0.11_2.11:1.4.2 |+--- org.apache.flink:flink-connector-kafka-0.10_2.11:1.4.2 ||+--- org.apache.flink:flink-connector-kafka-0.9_2.11:1.4.2 |||+--

Re: Exception while submitting jobs through Yarn

Hi, Sorry for the confusion, but the yarn is running on Hadoop version 2.7 only and hence I am using Flink 1.5 Hadoop 2.7 binary. Below are the details provided by Yarn version command : Hadoop 2.7.3.2.6.3.0-235 Subversion g...@github.com:hortonworks/hadoop.git -r 45bfd33bba8acadfa0e6024c80981c0

Re: Exception while submitting jobs through Yarn

Which Hadoop version have you installed? It looks as if Flink has been build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class path. If you want to run Flink on Hadoop 2.6, then try to use the Hadoop free Flink binaries or the one built for Hadoop 2.6. Cheers, Till On Mon, Jun 18, 2018

Re: flink and akka HTTP

Just for clarification: The user jar contains the required Akka dependency (including akka-remote and akka-actor)? Which version of Akka do you require? On Mon, Jun 18, 2018 at 1:07 PM Gäckler Martin < martin.gaeck...@esolutions.de> wrote: > No, without including the flink runtime to my own jar,

RE: flink and akka HTTP

No, without including the flink runtime to my own jar, nothing works. Regards Martin -- Martin Gäckler Entwicklung Schwarmdienste Im Auftrag der e.solutions GmbH Despagstr. 4a 85055 Ingolstadt Germany Registered

Re: flink and akka HTTP

Is it then working or not? On Mon, Jun 18, 2018, 11:52 Gäckler Martin wrote: > Good morning, > > > > Thanks for the hint. AKKA streams and http are added to our user jar. If > not, our HTTP client won't work. According to the flink documentation here: > https://flink.apache.org/news/2017/12/12/r

Re: Flink application does not scale as expected, please help!

Hi all, Allow me to add some comments/questions on this issue that is very interesting. According to documentation [1] the pipeline example assumes the source is running with the same parallelism as successive map operator and the workflow optimizes to collocate source and map tasks if possible.

Re: A question about Kryo and Window State

Any more insight? On Wed, Jun 13, 2018, 3:34 PM Vishal Santoshi wrote: > Any ideas on the standard way ( or any roundabout way ) of doing a version > upgrade that looks back ward compatible. > The @FieldSerializer.Optional("0") actually does ignore the field ( > even if reset ) giving it the d

RE: flink and akka HTTP

Good morning, Thanks for the hint. AKKA streams and http are added to our user jar. If not, our HTTP client won't work. According to the flink documentation here: https://flink.apache.org/news/2017/12/12/release-1.4.0.html#changes-to-dynamic-class-loading-of-user-code child first class loading

Re: Stream Join With Early firings

Hi Fabian, thanks for the hints, though I somehow got the feeling that I am on the wrong track given how much code I would need to write for implementing a "blueprint" usecase. Would a join be more simple using the Table API? In the end it's the classical Order & OrderPosition example, where the

Re: Exception while submitting jobs through Yarn

Ok, I have attached the log file. Please check and let me know. Thanks, On Mon, Jun 18, 2018 at 2:07 PM Amit Jain wrote: > Hi Gravit, > > I think Till is interested to know about classpath details present at the > start of JM and TM logs e.g. following logs provide classpath details used > by

Re: Exception while submitting jobs through Yarn

Hi Gravit, I think Till is interested to know about classpath details present at the start of JM and TM logs e.g. following logs provide classpath details used by TM in our case. 2018-06-17 19:01:30,656 INFO org.apache.flink.yarn.YarnTaskExecutorRunner -

Re: Exception while submitting jobs through Yarn

Hi, Please refer to my previous mail for complete logs. Thanks, On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann wrote: > Could you also please share the complete log file with us. > > Cheers, > Till > > On Sat, Jun 16, 2018 at 5:22 PM Ted Yu wrote: > >> The error for core-default.xml is interes

Re: Flink application does not scale as expected, please help!

Not sure if TM local assignment is explicitly designed in 1.5.0, but it might be an artifact of how slots are registered in the resource manager. Till (in CC) should know how that works. Anyway, tasks that run in the same TM exchange data via in-memory channels which is of course much faster than

Re: Flink application does not scale as expected, please help!

*additional info in bold. From: Siew Wai Yow Sent: Monday, June 18, 2018 3:57 PM To: Fabian Hueske Cc: Jörn Franke; user@flink.apache.org Subject: Re: Flink application does not scale as expected, please help! Hi Fabian, We are using Flink 1.5.0. Any different

Re: Stream Join With Early firings

Hi Johannes, EventTimeSessionWindows [1] use the EventTimeTrigger [2] as default trigger (see EventTimeSessionWindows.getDefaultTrigger()). I would take the EventTimeTrigger and extend it with early firing functionality. However, there are a few things to consider * you need to be aware that sess

Re: Flink application does not scale as expected, please help!

Hi Fabian, We are using Flink 1.5.0. Any different in scheduler in Flink 1.5.0? "Hence, applications might scale better until tasks are scheduled to different machines." This seems the case. We have 32 vCPU 16 slots in one TM machine. So the scaling work perfectly 1-2-4-8-16 because all happe

Re: [BucketingSink] incorrect indexing of part files, when part suffix is specified

I’ve created a JIRA issue https://issues.apache.org/jira/browse/FLINK-9603 and added a proposal with PR. Thx > On 16 Jun 2018, at 17:21, Rinat wrote: > > Hi mates, since 1.5 release, BucketingSink has ability to configure suffix of > the part

Re: Flink application does not scale as expected, please help!

Hi, Which Flink version are you using? Did you try to analyze the bottleneck of the application, i.e., is it CPU, disk IO, or network bound? Regarding the task scheduling. AFAIK, before version 1.5.0, Flink tried to schedule tasks on the same machine to reduce the amount of network transfer. Henc

Re: Exception while submitting jobs through Yarn

Could you also please share the complete log file with us. Cheers, Till On Sat, Jun 16, 2018 at 5:22 PM Ted Yu wrote: > The error for core-default.xml is interesting. > > Flink doesn't have this file. Probably it came with Yarn. Please check the > hadoop version Flink was built with versus the

Re: Memory Leak in ProcessingTimeSessionWindow

Hi, can you take a heap dump from a JVM that runs into the problem and share it with us? That would make finding the cause a lot easier. Best, Stefan > Am 15.06.2018 um 23:01 schrieb ashish pok : > > All, > > I have another slow Memory Leak situation using basic TimeSession Window > (earlier

Re: IoT Use Case, Problem and Thoughts

Hi Ashish, the atomic savepoint with savepoint is going to be implemented for all state backends. Cheers, Till On Sat, Jun 16, 2018 at 4:29 AM Ashish Pokharel wrote: > Hi Till, Fabian, > > Thanks for your responses again. > > Till, you have nailed it. I will comment on them individually. But f

Re: flink and akka HTTP

Hi, I assume that you have an Akka dependency conflict. By adding the Akka dependency version to your user jar and enabling child first class loading you should be able to control which Akka version is loaded. The only thing you have to check is whether Flink works with a newer version of Akka. C