flink build error

Hi, I am trying to build flink 1.6 but cannot build it to run also the tests. Any ideas of why the surefire error to run junits tests? [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (default-test) on project flink-test-utils-junit:

FW: What's the advantage of using BroadcastState?

what you think.. Dr. Radu Tudoran Staff Research Engineer - Big Data Expert IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH German Research Center Munich Office Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com<mailto:radu.tudo...@huawei.

RE: What's the advantage of using BroadcastState?

atterns", Types.VOID, Types.POJO(Pattern.class))); // storing in MapState with null as VOID default value bcState.put(null, pattern); If (test) { bcState.put(null, new Pattern(test) ); } } } Dr. Radu Tudoran Staff Research Engineer - Big Data Expert IT R Division [cid:image

RE: concurrency?

Hi, Yes it does – thanks a lot Knowing that this is the order time = 2, onTimer(2) -> access state with key t=2-1, get A, B time = 2, pocessElement(C) -> put C in state keyed to t=2, registerProcTimer(3) is useful! Dr. Radu Tudoran Senior Research Engineer - Big Data Expert IT R Di

RE: concurrency?

: (processElement – executed second) ev3-arrives state=ev3 onTime(executed first): out = ev2 Best regards, Dr. Radu Tudoran Senior Research Engineer - Big Data Expert IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH German Research Center

concurrency?

Hi, I would like to use a processFunction to accumulate elements. Therefore in the processElement function I will accumulate this element into a state. However, I would like to emit the output only 1ms later. Therefore I would register a timer to trigger one second later and read the state and

RE: Dummy DataStream

Hi Duck, I am not 100% sure I understand your exact scenario but I will try to give you some pointers, maybe it will help. Typically when you do the split you have some knowledge about the criterion to do the split. For example if you follow the example from the website

RE: [DISCUSS] "Who's hiring on Flink" monthly thread in the mailing lists?

Hi, I think the idea of having such a monthly thread is very good and it might even help to further attract new people in the community. In the same time I do not think that 1 extra mail per month is necessary a spam ☺ In the same time – we can also consider a jobs@flink mailing list Dr. Radu

RE: TIMESTAMP TypeInformation

TypeInfos. Can you open a JIRA for that? Thanks, Fabian 2016-10-27 9:31 GMT+02:00 Radu Tudoran <radu.tudo...@huawei.com<mailto:radu.tudo...@huawei.com>>: Hi, I dig meanwhile more through this and I think I found a bug actually. The scenario that I was trying to describe was somet

RE: TIMESTAMP TypeInformation

u rely on the type extraction mechanism called after the returns to recognize TIMESTAMP of type SqlTimeTypeInfo it will not happen but instead a GenericType will be created. It looks like the type parsers were not extended to consider this types Dr. Radu Tudoran Senior Research Engineer - Big Data

RE: TIMESTAMP TypeInformation

ill be converted to SqlTimeTypeInfo...but it is converted to GenericType. Anythoughts how I could force the type to be recognize as a SqlTimeType? From: Radu Tudoran Sent: Tuesday, October 25, 2016 4:46 PM To: 'user@flink.apache.org' Subject: TIMESTAMP TypeInformation Hi, I would like to cr

TIMESTAMP TypeInformation

Hi, I would like to create a TIMESTAMP type from the data schema. I would need this to match against the FlinkTypeFactory (toTypeInfo()) def toTypeInfo(relDataType: RelDataType): TypeInformation[_] = relDataType.getSqlTypeName match { case BOOLEAN => BOOLEAN_TYPE_INFO case TINYINT =>

question for generic streams

Hi, I am trying to read the types of an input stream from a file and then generate the datastream resulting from parsing a source accordingly (e.g. DataStream>). Finally this will be registered as a table. What would be the best way to do this? I am trying currently to

RE: org.apache.flink.core.fs.Path error?

Path, but directly generates a URI from it. On 20.10.2016 15:48, Fabian Hueske wrote: The error message suggests that Flink tries to resolve "D:" as a file system schema such as "file:" or "hdfs:". Can you try to use specify your path as "file:/D:/dir/my

RE: org.apache.flink.core.fs.Path error?

ld suggest the usual: check that the path is indeed correct, that you have sufficient permissions to access the file. And yes, you can report problems here ;) Regards, Chesnay On 20.10.2016 13:17, Radu Tudoran wrote: Hi, I am running a program that is suppose to read a CSV file from the local d

emit watermarks

Hi, Is there some way to emit a watermark in the trigger? I see that in the evictor there is the option to check the StreamRecord if it is a watermark..so I would hope that there is some option also to create them

RE: SQL / Tuple question

this? TypeInformation<Tuple2<String, Long>> tpinf = new TypeHint<Tuple2<String, Long>>(){}.getTypeInfo(); .toDataStream( , tpinf ) Best, Fabian 2016-09-19 17:53 GMT+02:00 Radu Tudoran <radu.tudo...@huawei.com<mailto:radu.tudo...@huawei.com>>: Hi, I

SQL for Flink

(DataStreamScanRule.INSTANCE , CUSTOM_TABLE_SCAN_Rule.INSTANCE) - or if the selection of the new translation rule can be handled from the cost than simply configuration.addRule( CUSTOM_TABLE_SCAN_Rule.INSTANCE) What do you think? Dr. Radu Tudoran Senior Research Engineer - Big Data Expert IT R Division

RE: error for building flink-runtime from source

Hi, I am building the 1.1 snapshot (should be the latest release). I will try to build the whole project to check if it works Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center

error for building flink-runtime from source

org.apache.flink.api.common.restartstrategy.RestartStrategies.FixedDelayRestartStrategyConfiguration [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Best regards, Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Divis

Flink and Calcite

Hi, Can someone point me to the repository where the integration of Calcite with Flink is available? Does this come with the master branch (as indicated by the link in the blog post)? https://github.com/apache/flink/tree/master Thanks Dr. Radu Tudoran Research Engineer - Big Data Expert IT R

RE: lost connection

. org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Communication with JobManager failed: Lost connection to the JobManager. I do not understand what could be the root cause of this... the IPs look ok and there is not firewall to block things... Dr. Radu Tudoran Research Engineer - Big

RE: Custom time window in Flink

the public int evict class. Same story with the triggers https://github.com/apache/flink/tree/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI

RE: Custom time window in Flink

would be that you have a field that marks whether a window was created or not (e.g., a Boolean field in the evictor class), which once a window is created you can set it to false and from there one you operate on 5 minutes windows Dr. Radu Tudoran Research Engineer - Big Data Expert IT R

RE: ClasNotFound when submitting job from command line

Hi, In my case the root cause for this was mainly that I was using eclipse to package the jar. Try using mvn instead. Additioanlly you can copy the dependency jars in the lib of the task managers and restart them Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Division [cid:image007

jar dependency in the cluster

or submit the job to add extra dependencies. Many thanks Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49

RE: operators

. The best thing usually is location transparency and local affinity (as a performance optimization). Is the file large, i.e., would it hurt to do it on a DFS? Or actually use a Kafka Queue between the operators? Stephan On Wed, Mar 9, 2016 at 5:38 PM, Radu Tudoran <radu.tudo.

operators

ew stream .=> the red items should go on the same machine Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49

RE: ype of TypeVariable could not be determined

I can pass it using the “.returns()” method as Timo suggested Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com

ype of TypeVariable could not be determined

) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1089) ... 1 more Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo

RE: [ANNOUNCE] Flink 1.0.0 has been released

Hi, Do you have also a linkedin post that I could share - or should I make a blogpost in which I take this announcement? Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Division HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail

RE: events eviction

016, at 18:07, Radu Tudoran <radu.tudo...@huawei.com> wrote: > > Hi, > > Thanks Aljoscha for the details! > > The warning about performance and evictors is useful, but I am not sure how > it can be put in practice always. Take for example a GlobalWindow that you >

events eviction

e some way of creating a state in the evictor that will be check pointed and restore in case of failure. I would be interested if something like an operator state is possible in the evictor. Regards, Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Division [cid:image007.jpg@01CD52E

RE: Flink writeAsCsv

Hi Radu, It is indeed interesting to know how each window could be registered separately - I am not sure it any of the existing mechanisms in Flink support this. I think you need to create your own output sink. It is a bit tricky to pass the window sequence number (actually I do not think

RE: release of task slot

like the input flow is faster than the GC collector Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49

release of task slot

) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E

maxtime / watermark for GlobaWindow

w arg0, Iterable< Tuple1, > arg1, Collector<Tuple1> arg2) throws Exception { // - get the even timestamp } }) Dr. Radu Tudoran Research Engineer - Big Data Ex

continous time triger

) is that timeWindow will end up generating multiple windows and duplicating data, while having the option from the trigger to actually fire the processing at certain times, independent of when the events arrived) would enable to operate with a single window. Regards, Dr. Radu Tudoran Research

RE: global function over partitions

}).setParallelism(1) .writeAsText("/tmp/testoutput", WriteMode.OVERWRITE); env.execute("main stream application"); } Regards, Dr. Radu Tudoran Research Engineer - Big Data Expert IT R Division

global function over partitions

Hi, I am trying to compute some final statistics over a stream topology. For this I would like to gather all data from all windows and parallel partitions into a single/global window. Could you suggest a solution for this. I saw that the map function has a ".global()" but I end up with the

RE: Behaviour of CountWindowAll

Hi, I believe this question might have been asked before - so sorry for repeating it (I just did not find the discussion on the mailing list). Is it possible somehow to create a new DataStream from the elements that are evicted from a window? A simple use case for this is: We have data

RE: Question about DataStream serialization

Dataset -> use ...is empty } @Override public void open( { dataset -> load } }) Dr. Radu Tudoran Research Engineer IT R Division HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49 15209084330 Telephone: +49

RE: Question about DataStream serialization

dataset.update(dataset.value()+1); } //System.out.println("dataset value "+dataset.value()); } }); stream.print(); env.ex

RE: Question about DataStream serialization

Hi, The state that is being loaded can very well be partitioned by keys. Assuming this scenario and that you would now that the keys go from 0 to N, is there some possibility to load and partitioned the initial data in the open function? Dr. Radu Tudoran Research Engineer IT R Division

RE: Question about DataStream serialization

Hi, Is the partitioned functioned used by the ".keyBy(Object)" of the form: Object.hash % getNumberOfParallelSubtasks() ? Dr. Radu Tudoran Research Engineer IT R Division HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail:

question about DataStream serialization

event to it and apply an operation Regards, Dr. Radu Tudoran Research Engineer IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49 15209084330 Telephone: +49

RE: flink connectors

Hi, Is there any alternative to avoiding maven? That is why I was curious if there is a binary distribution of this available for download directly Dr. Radu Tudoran Research Engineer IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research

RE: flink connectors

Hi, Thank you for the tips! For future references in case someone else wants to search for the binaries for this, I would like to share the link to the maven repository http://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka Dr. Radu Tudoran Research Engineer IT R Division

flink connectors

distribution that contains these connectors? Regards, Dr. Radu Tudoran Research Engineer IT R Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49 15209084330 Telephone: +49

RE: output writer

file Might be that other solutions exist Dr. Radu Tudoran Research Engineer IT RD Division HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49 15209084330 Telephone: +49 891588344173 HUAWEI TECHNOLOGIES

RE: output writer

I will double check and try to commit this in the next days Dr. Radu Tudoran Research Engineer IT RD Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf GmbH European Research Center Riesstrasse 25, 80992 München E-mail: radu.tudo...@huawei.com Mobile: +49 15209084330

RE: output writer

that could be contributed. We should identify a couple of common formats. The first one that comes in my mind is to have something for writing to memory (e.g. memory buffer) Dr. Radu Tudoran Research Engineer IT RD Division [cid:image007.jpg@01CD52EB.AD060EE0] HUAWEI TECHNOLOGIES Duesseldorf