Hi,
I am trying to build flink 1.6 but cannot build it to run also the tests. Any
ideas of why the surefire error to run junits tests?
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (default-test) on
project flink-test-utils-junit: ExecutionException:
what you think..
Dr. Radu Tudoran
Staff Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
German Research Center
Munich Office
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com<mailto:radu.tudo...@hua
atterns", Types.VOID,
Types.POJO(Pattern.class)));
// storing in MapState with null as VOID default value
bcState.put(null, pattern);
If (test) {
bcState.put(null, new Pattern(test) );
}
}
}
Dr. Radu Tudoran
Staff Research Engineer - Big Data Expert
IT R&
Hi,
Yes it does – thanks a lot
Knowing that this is the order
time = 2, onTimer(2) -> access state with key t=2-1, get A, B
time = 2, pocessElement(C) -> put C in state keyed to t=2, registerProcTimer(3)
is useful!
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&
: (processElement – executed second) ev3-arrives state=ev3
onTime(executed first): out = ev2
Best regards,
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
German Research Ce
Hi,
I would like to use a processFunction to accumulate elements. Therefore in the
processElement function I will accumulate this element into a state. However, I
would like to emit the output only 1ms later. Therefore I would register a
timer to trigger one second later and read the state and
Hi Duck,
I am not 100% sure I understand your exact scenario but I will try to give you
some pointers, maybe it will help.
Typically when you do the split you have some knowledge about the criterion to
do the split.
For example if you follow the example from the website
https://ci.apache.org/pr
Hi,
I think the idea of having such a monthly thread is very good and it might even
help to further attract new people in the community.
In the same time I do not think that 1 extra mail per month is necessary a spam
☺
In the same time – we can also consider a jobs@flink mailing list
Dr. Radu
TypeInfos.
Can you open a JIRA for that?
Thanks, Fabian
2016-10-27 9:31 GMT+02:00 Radu Tudoran
mailto:radu.tudo...@huawei.com>>:
Hi,
I dig meanwhile more through this and I think I found a bug actually.
The scenario that I was trying to describe was something like
1. You have a g
mechanism called after
the returns to recognize TIMESTAMP of type SqlTimeTypeInfo it will not happen
but instead a GenericType will be created.
It looks like the type parsers were not extended to consider this types
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Divi
ill be converted to
SqlTimeTypeInfo...but it is converted to GenericType. Anythoughts how I could
force the type to be recognize as a SqlTimeType?
From: Radu Tudoran
Sent: Tuesday, October 25, 2016 4:46 PM
To: 'user@flink.apache.org'
Subject: TIMESTAMP TypeInformation
Hi,
I would
Hi,
I would like to create a TIMESTAMP type from the data schema. I would need this
to match against the FlinkTypeFactory (toTypeInfo())
def toTypeInfo(relDataType: RelDataType): TypeInformation[_] =
relDataType.getSqlTypeName match {
case BOOLEAN => BOOLEAN_TYPE_INFO
case TINYINT => BY
Hi,
I am trying to read the types of an input stream from a file and then generate
the datastream resulting from parsing a source accordingly (e.g.
DataStream>). Finally this will be registered as a table.
What would be the best way to do this?
I am trying currently to use the generic type of T
Path, but directly generates a URI from it.
On 20.10.2016 15:48, Fabian Hueske wrote:
The error message suggests that Flink tries to resolve "D:" as a file system
schema such as "file:" or "hdfs:".
Can you try to use specify your path as
"file:/D:/dir/my
right now i would suggest the usual: check that the path is indeed correct,
that you have sufficient permissions to access the file.
And yes, you can report problems here ;)
Regards,
Chesnay
On 20.10.2016 13:17, Radu Tudoran wrote:
Hi,
I am running a program that is suppose to read a CSV file f
/dir/myfile.csv"". The problem is that this is never changed
when the file is actually open and accessed which leads to an error.
...not sure if signaling this error here is the best place or if I should have
used some other media..
Best regards,
Dr. Radu Tudoran
Senior Research En
Hi,
Is there some way to emit a watermark in the trigger?
I see that in the evictor there is the option to check the StreamRecord
if it is a watermark..so I would hope that there is some option also to create
them
this?
TypeInformation> tpinf = new TypeHint>(){}.getTypeInfo();
.toDataStream( , tpinf )
Best, Fabian
2016-09-19 17:53 GMT+02:00 Radu Tudoran
mailto:radu.tudo...@huawei.com>>:
Hi,
I am trying to create an sql statement that is suppose to return a string and
and integer
Mytable
Hi,
I am trying to create an sql statement that is suppose to return a string and
and integer
Mytable.sql("select mystringfield,myintfield )
I am trying to give the typefinormation as a class to the
TypeInformation> tpinf = new TypeHint>(){}.getTypeInfo();
.toDataStream( , tpinf.getTypeC
(DataStreamScanRule.INSTANCE ,
CUSTOM_TABLE_SCAN_Rule.INSTANCE) - or if the selection of the new translation
rule can be handled from the cost than simply configuration.addRule(
CUSTOM_TABLE_SCAN_Rule.INSTANCE)
What do you think?
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Divi
Hi,
I am building the 1.1 snapshot (should be the latest release).
I will try to build the whole project to check if it works
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Ce
org.apache.flink.api.common.restartstrategy.RestartStrategies.FixedDelayRestartStrategyConfiguration
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
Best regards,
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D D
Hi,
Can someone point me to the repository where the integration of Calcite with
Flink is available?
Does this come with the master branch (as indicated by the link in the blog
post)?
https://github.com/apache/flink/tree/master
Thanks
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R
Yes - it suddenly occurred on something that used to work. I am restarting the
deployment to see if this solves the problem
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Ce
.
org.apache.flink.client.program.ProgramInvocationException: The program
execution failed: Communication with JobManager failed: Lost connection to the
JobManager.
I do not understand what could be the root cause of this... the IPs look ok and
there is not firewall to block things...
Dr. Radu Tudoran
Research Engineer - Big
Hi,
I am trying to submit a jar via the console (flink run my.jar). The result is
that I get an error saying that the communication with the jobmanager failed:
Lost connection to the jobmanager.
Can you give me some hints/ recommendations about approaching this issue.
Thanks
Dr. Radu Tudoran
public int evict class.
Same story with the triggers
https://github.com/apache/flink/tree/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HU
would be that you have a field that marks whether a window was
created or not (e.g., a Boolean field in the evictor class), which once a
window is created you can set it to false and from there one you operate on 5
minutes windows
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&a
Hi,
In my case the root cause for this was mainly that I was using eclipse to
package the jar. Try using mvn instead. Additioanlly you can copy the
dependency jars in the lib of the task managers and restart them
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Divi
submit the job
to add extra dependencies.
Many thanks
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile:
.
The best thing usually is location transparency and local affinity (as a
performance optimization).
Is the file large, i.e., would it hurt to do it on a DFS? Or actually use a
Kafka Queue between the operators?
Stephan
On Wed, Mar 9, 2016 at 5:38 PM, Radu Tudoran
mailto:radu.tudo...@huawei.c
.=> the red items should go on the same machine
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile:
I can pass
it using the “.returns()” method as Timo suggested
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei
xecutionEnvironment.java:1107)
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1089)
... 1 more
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Ries
Hi,
Do you have also a linkedin post that I could share - or should I make a
blogpost in which I take this announcement?
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-
.."
Stephan
On Mon, Mar 7, 2016 at 7:36 PM, Radu Tudoran
mailto:radu.tudo...@huawei.com>> wrote:
Hi,
Can someone explain how and where a stream operator is mapped to a stream task.
I am particularly interested in the way the stream outputs are created and
attached to the op
Hi,
Can someone explain how and where a stream operator is mapped to a stream task.
I am particularly interested in the way the stream outputs are created and
attached to the operators. I saw that this happen in OperatorChain functions
but I do not have the picture of the lifecycle of an stream
Feb 2016, at 18:07, Radu Tudoran wrote:
>
> Hi,
>
> Thanks Aljoscha for the details!
>
> The warning about performance and evictors is useful, but I am not sure how
> it can be put in practice always. Take for example a GlobalWindow that you
> would use to aggregate
. And then to ensure resiliency, the state should be recoverable if a crash
happens. Could you approach this without an evictor state?
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 Mün
e some way of creating a state in the
evictor that will be check pointed and restore in case of failure. I would be
interested if something like an operator state is possible in the evictor.
Regards,
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01C
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile: +49 15209084330
Telephone: +49 891588344173
HUAWEI TEC
like the input flow is faster than the GC collector
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile:
Hi Radu,
It is indeed interesting to know how each window could be registered separately
- I am not sure it any of the existing mechanisms in Flink support this.
I think you need to create your own output sink. It is a bit tricky to pass the
window sequence number (actually I do not think such
)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 Münche
Hi,
Thank you for sharing your experience and also to Till for the advice.
What I would like to do is to be able to fire the window potentially multiple
times, even if an event did not arrive. I will look more about how dealing with
the processing time could help in this
Dr. Radu Tudoran
that
timeWindow will end up generating multiple windows and duplicating data, while
having the option from the trigger to actually fire the processing at certain
times, independent of when the events arrived) would enable to operate with a
single window.
Regards,
Dr. Radu Tudoran
Research
rg0,
Iterable< Tuple1, > arg1,
Collector> arg2)
throws Exception {
// - get the even timestamp
}
})
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
[
rentTimeMillis());
arg2.collect(new Tuple1(count));
}
}).setParallelism(1)
.writeAsText("/tmp/testoutput", WriteMode.OVERWRITE);
env.execute(&quo
Hi,
I am trying to compute some final statistics over a stream topology. For this I
would like to gather all data from all windows and parallel partitions into a
single/global window. Could you suggest a solution for this. I saw that the map
function has a ".global()" but I end up with the same
Hi,
I believe this question might have been asked before - so sorry for repeating
it (I just did not find the discussion on the mailing list).
Is it possible somehow to create a new DataStream from the elements that are
evicted from a window?
A simple use case for this is:
We have data
Hi,
I believe i found 2 small inconsistencies in the documentation for the
description of Window Apply
https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html#windows-on-unkeyed-data-streams
1) in the example provided I believe it should be corrected to remove
Hi,
Is the partitioned functioned used by the ".keyBy(Object)" of the form:
Object.hash % getNumberOfParallelSubtasks()
?
Dr. Radu Tudoran
Research Engineer
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-m
Hi,
The state that is being loaded can very well be partitioned by keys. Assuming
this scenario and that you would now that the keys go from 0 to N, is there
some possibility to load and partitioned the initial data in the open function?
Dr. Radu Tudoran
Research Engineer
IT R&D Divi
//System.out.println("dataset
value "+dataset.value());
}
});
stream.print();
env.execute("test open
use ...is empty
}
@Override
public void open(
{
dataset -> load
}
})
Dr. Radu Tudoran
Research Engineer
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile: +49 15209084330
Telephone: +4
an empty dataset in the
flatmap function while when this finish executing the dataset gets loaded. Is
this an error or I am doing something wrong?
Dr. Radu Tudoran
Research Engineer
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research
thinking that I could keep the fixed set as a
DataSet and as each new event comes, transform it into a dataset and then join
with reference set and apply an operation
Regards,
Dr. Radu Tudoran
Research Engineer
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOG
et and
launch a batch processing while operating in stream mode- in case this is
possible, can you give me a reference for it, because it might be the good
solution to use in case I could treat the fixed set as a DataSet and just
add/remove the incoming event to it and apply an operation
Regard
expected
to go wrong somewhere else without having it?
@Marton – the answer was that I need 2 of these fields to be used to do the
keyBy (long, string)
Dr. Radu Tudoran
Research Engineer
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Rese
(GenericType) cannot be used as key.
Can you give me a hint about a solution to this?
Thanks
Dr. Radu Tudoran
Research Engineer
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.t
Hi,
Thank you for the tips!
For future references in case someone else wants to search for the binaries for
this, I would like to share the link to the maven repository
http://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka
Dr. Radu Tudoran
Research Engineer
IT R&a
Hi,
Is there any alternative to avoiding maven?
That is why I was curious if there is a binary distribution of this available
for download directly
Dr. Radu Tudoran
Research Engineer
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Rese
distribution that contains these connectors?
Regards,
Dr. Radu Tudoran
Research Engineer
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile: +49 15209084330
Telephone:
that could
be contributed. We should identify a couple of common formats. The first one
that comes in my mind is to have something for writing to memory (e.g. memory
buffer)
Dr. Radu Tudoran
Research Engineer
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duessel
I will double check and try to commit this in the next days
Dr. Radu Tudoran
Research Engineer
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile: +49 1520908
ach to the appropriate file
Might be that other solutions exist
Dr. Radu Tudoran
Research Engineer
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile: +49 15209084330
Telephone: +49 891588344173
HUAWE
66 matches
Mail list logo