Thanks again for the explanation.
On Fri, 6 Aug 2021, 15:39 Arvid Heise, wrote:
> User function from the point of view of Flink is any function that you as
> a user write. In this case, MetadataEnrichment.AsyncFlowLookup is a
> user-function of the Async Operator.
>
> I don't recommend harness
Hi Joe,
there was a follow-up issue, FLINK-23450, which was only fixed for 1.13.3
(not yet released) which I think is what you're seeing?
Best
Ingo
On Fri, Aug 6, 2021, 21:17 Joseph Lorenzini wrote:
> Hi all,
>
>
>
> I am on flink 1.13.2. I set up create table like so:
>
>
>
> CREATE TABLE
Hi all,
I am on flink 1.13.2. I set up create table like so:
CREATE TABLE lead_buffer (
`id` INT NOT NULL,
`state` STRING NOT NULL,
PRIMARY KEY (`id`) NOT ENFORCED
) WITH (
'connector'= 'kafka',
'topic' = 'buffer',
'format'= 'debezium-avro-confluent',
Hey guys,
I wrote a simple DataStream that counts up some numbers into a SideOutput which
I am trying to sink into a StreamFileSink so that I can write the results to
disk and read them from there.
I'm running my little test locally and I can see that the data is being written
to hidden
Hi all,
Does the sample processing also sample threads that do not belong to the
Flink framework? For example, a background thread that is created by and
managed by the user?
Best,
Mason
Hi Niklas,
We are currently using the Lyft operator for Flink in production (
https://github.com/lyft/flinkk8soperator), which is additional alternative.
The project itself is pretty much in Zombie state, but commits happen every
now and then.
1. Native Kubernetes could definitely work with
Hi Niklas,
We had the same problem one year ago and we choose Ververica Platform
Community Edttion.
Pros:
- support for jobs on Session Clusters
- good support for restoring jobs from checkpoints and savepoints
- support for even hundreds of jobs
Cons:
- state in SQLite (we've already corrupted db
Sorry to hear that. Did I got that right that there is no way to save and
recover state in batch mode?
Tobi
On 6. Aug 2021, 13:20 +0200, Chesnay Schepler , wrote:
> All checkpointing-related features do not work in batch mode.
>
>
finally, I work out how to build a custom flink image, the Dockerfile just
as:
>
> FROM flink:1.13.1-scala_2.11
> ADD ./flink-s3-fs-hadoop-1.13.1.jar /opt/flink/plugins
> ADD ./flink-s3-fs-presto-1.13.1.jar /opt/flink/plugins
>
the wrong Docker file is :
> FROM apache/flink:1.13.1-scala_2.11
Hi Flink Community,
I'm currently assessing the situation about how to properly deploy Flink on
Kubernetes via GitOps. There are some options available to deploy Flink on
Kubernetes, which I would like to discuss. In general we are looking for an
open source or at least unpaid solution, but I
The reason this doesn't work is that your application works directly
against Hadoop.
The filesystems in the plugins directory are only loaded via specific
code-paths, specifically when the Flink FileSystem class is used.
Since you are using Hadoop directly you are side-stepping the plugin
All checkpointing-related features do not work in batch mode.
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/execution_mode/#important-considerations
On 04/08/2021 21:23, tobias.schu...@xing.com wrote:
He folks,
This is a crosspost of a stack overflow
it is not the same.
Kafka's 'auto.offset.reset' is used when the configured consumer group
id does not have offset info stored in kafka. not exist. If you want
to consume from latest no matter whether there is group offset info in
kafka, you should use flink's setStartFromLatest.
suman shil
User function from the point of view of Flink is any function that you as a
user write. In this case, MetadataEnrichment.AsyncFlowLookup is a
user-function of the Async Operator.
I don't recommend harness for several reasons:
- It's not Public API, we will adjust it and we will introduce breaking
I am a heavy user of rocksdb, and my application works well on flink-1.10,but I
am interst in the new features.I want to know if I can increase the manage
memory to eliminate the impact of the performance regression In a certain
degree。
Can you show me the params about rocksdb of your task?
Thanks it worked.
Can you please let me know what are you referring to as user function and
why it is not recommended to not use harness with it?
On Wed, 4 Aug 2021, 22:43 Arvid Heise, wrote:
> I would strongly recommend not using the harness for testing user
> functions.
>
> Instead I'd just
Hey All,
I am running flink in docker containers (image Tag
:flink:scala_2.11-java11) on EC2 and getting exception as I am trying to
submit a job through the local ./opt/flink/bin
*org.apache.flink.client.program.ProgramInvocationException: The main
method caused an error: No FileSystem for
17 matches
Mail list logo
