Hi,
This is the user subscription request.
Thanks & Regards,
Samir Vasani
Hi,
I am a new bee to flink and facing some challenges to solve below use case
Use Case description:
I will receive a csv file with a timestamp on every single day in some
folder say *input*.The file format would be
*file_name_dd-mm-yy-hh-mm-ss.csv*.
Now my flink pipeline will read this csv fil
Could this be related to https://issues.apache.org/jira/browse/FLINK-22414?
On Thu, Jul 22, 2021 at 3:53 PM Timo Walther wrote:
> Thanks, this should definitely work with the pre-packaged connectors of
> Ververica platform.
>
> I guess we have to investigate what is going on. Until then, a
> wor
Hi!
JobListener#onJobExecuted might help, if your job is not a forever-running
streaming job. See
https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/core/execution/JobListener.html
Samir Vasani 于2021年7月23日周五 下午3:22写道：
> Hi,
>
> I am a new bee to flink and facing so
Hi Caizhi Weng,
Thanks for your input.
I would explain the requirement in little more detail.
Flink pipeline will be running forever (until some issue happens and we
would need to restart) so It will continuously monitor if a new file comes
to the *input *folder or not.
In this case will your sugg
Hi!
In this case it won't work, as JobListener#onJobExecuted will only be
called when the job finishes, successfully or unsuccessfully.
For a forever-running job I would suggest adding a UDF right after the
source and adding a special "EOF" record in each of the csv file. This UDF
monitors the da
Finally I found the mistake. I put the „—host 10.1.2.3“ param as one
argument. I think the savepoint argument was not interpreted correctly or
ignored. Might be that the „-s“ param was used as value for „—host
10.1.2.3“ and „s3p://…“ as new param and because these are not valid
arguments they were
Hi,
Can you elaborate more on UDF as I did not understand it.
Thanks & Regards,
Samir Vasani
On Fri, Jul 23, 2021 at 1:22 PM Caizhi Weng wrote:
> Hi!
>
> In this case it won't work, as JobListener#onJobExecuted will only be
> called when the job finishes, successfully or unsuccessfully.
>
>
Hi Samir,
to unsubscribe please send an empty-body / empty-subject email to
user-unsubscr...@flink.apache.org. You can see a community page [1] in docs
for more details.
[1] https://flink.apache.org/community.html
Best,
D.
On Fri, Jul 23, 2021 at 9:01 AM Samir Vasani wrote:
> Hi,
>
> This is
Hello,
It’s hard to say what caused the timeout to trigger – I agree with you that it
should not have stopped the heartbeat thread, but it did. The easy fix was to
increase it until we no longer see our app self-killed. The task was using a
CPU-intensive computation (with a few threads created
Hi Dan,
1) If the key doesn’t change in the downstream operators and you want to avoid
shuffling, maybe the DataStreamUtils#reinterpretAsKeyedStream would be helpful.
2) I am not sure that if you are saying that the data are already partitioned
in the Kafka and you want to avoid shuffling in th
in my app pom.xml
org.apache.flink
flink-connector-kafka_2.11
1.13.1
provided
and I have copy flink-connector-kafka_2.11-1.13.1.jar to FLINK_HOME/lib/
but I also get a error
ClassNotFoundException: org.apache.kafka.clients.consumer.ConsumerRecord
so
How can fix it?
igyu
Great, glad it was an easy fix :) Thanks for following up!
On Fri, Jul 23, 2021 at 3:54 AM Thms Hmm wrote:
> Finally I found the mistake. I put the „—host 10.1.2.3“ param as one
> argument. I think the savepoint argument was not interpreted correctly or
> ignored. Might be that the „-s“ param wa
Hello,
With the default memory settings, after about 5000 records in my
KafkaFlinkConsumer, and some operators in my pipeline, I get the below error:
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Unknown Source) ~[?:?]
at java.nio.Dire
Hi Team,
https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.html
*" There is no perfect generic default assignment function. Default shard
to subtask assignment, which is based on hash code, may result in skew,
15 matches
Mail list logo
