Using Flink

Hi, I have few questions on how I need to model my use case in flink. Please advise. Thanks for the help. - I'm currently running my flink program on 1.2 SNAPSHOT with kafka source and I have checkpoint enabled. When I look at the consumer offsets in kafka it appears to be stagnant and there

Re: Using Flink

Hi! - I'm currently running my flink program on 1.2 SNAPSHOT with kafka source and I have checkpoint enabled. When I look at the consumer offsets in kafka it appears to be stagnant and there is a huge lag. But I can see my flink program is in pace with kafka source in JMX metrics and outputs. I

Re: Using Flink

Hi Gordon, - I'm currently running my flink program on 1.2 SNAPSHOT with kafka source and I have checkpoint enabled. When I look at the consumer offsets in kafka it appears to be stagnant and there is a huge lag. But I can see my flink program is in pace with kafka source in JMX metrics and output

Re: Using Flink

Hi Govindarajan, you can broadcast the stream with debug logger information by calling `stream.broadcast`. Then every stream record should be send to all sub-tasks of the downstream operator. Cheers, Till On Mon, Oct 3, 2016 at 5:13 PM, Govindarajan Srinivasaraghavan < govindragh...@gmail.com> w

Re: Using Flink

Hi Govindarajan, Regarding the stagnant Kakfa offsets, it’ll be helpful if you can supply more information for the following to help us identify the cause: 1. What is your checkpointing interval set to? 2. Did you happen to have set the “max.partition.fetch.bytes” property in the properties give

Using Flink with Accumulo

Hello, I'm using Flink with Accumulo and wanted to read data from the database by using the createHadoopInput function. Therefore I configure an AccumuloInputFormat. The source code you can find here: https://github.com/ OSwoboda/masterthesis/blob/master/aggregation.flink/src/ main/ja

Exception using flink-connector-elasticsearch

Hi, We are using the sink for ElasticSearch and when we try to run our job we get the following exception: java.lang.ExceptionInInitializerError Caused by: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene410' does not exist. You need to ad

Accessing RDF triples using Flink

Hi, I need some suggestions regarding accessing RDF triples from flink. I'm trying to integrate flink in a pipeline where the input for flink comes from SPARQL query on a Jena model. And after modification of triples using flink, I will be performing SPARQL update using Jena to save my ch

Re: Using Flink with Accumulo

might be able to try to use the CleanUp[1] utility to close out the thread pools/connections when your Flink "task" is done. Second problem I have: I want to compare aggregations on time series data with Accumulo (with Iterators) and with flink. Unfortunately, the results vary inexplicable wh

Re: Using Flink with Accumulo

the Flink side can help > give context where necessary :) > > Oliver Swoboda wrote: > >> Hello, >> >> I'm using Flink with Accumulo and wanted to read data from the database >> by using the createHadoopInput function. Therefore I configure an >> Ac

Re: Using Flink with Accumulo

uggest something outright wrong. Hopefully someone from the Flink side can help give context where necessary :) Oliver Swoboda wrote: Hello, I'm using Flink with Accumulo and wanted to read data from the database by using the createHadoopInput function.

Custom timer implementation using Flink

I have a use case which I am trying to solve using Flink. Need an advice to decide on the correct approach. Use case: -- I have a stream of events partitioned by a key. For some events, I need to start a timer (conside this as a SLA i.e. if something is not done in x secs/minutes do

Using Flink with Redis question

Hello, So I am trying to use jedis (redis java client) with Flink streaming api, but I get an exception: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452) at or

[Stateful Functions] Using Flink CEP

Hi there, I was wondering if I could somehow use CEP within a Function. Have you experimented with this before? Or, do you have any suggestions to do CEP within a Function? I am looking for a standalone library now. Oytun -- [image: MotaWord] Oytun Tez M O T A W O R D | CTO & Co-Founder oy.

Docker using flink socketwordcount example

I had a question about a topic. but I could not find a solution. Could you help? My question: https://stackoverflow.com/questions/54806830/docker-using-flink-socketwordcount-example-apache-flink -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Challenges using Flink REST API

Hey all! I'm looking for some advice on the following; I'm working on an abstraction on top of Apache Flink to 'pipeline' Flink applications using Kafka. For deployment this means that all these Flink jobs are embedded into one jar and each job is started using an program argument (e.g. "--stage '

Issue using Flink on EMR

Hello, We have a Flink on EMR setup following this guide . YARN, apparently changes the io.tmp.dirs property to /mnt/yarn & /mnt1/yarn. When using these directories, the flink job gets the following error. 2019-05-22 12:23:12,515 INF

Oracle data streaming using Flink

Hi Users, I am new to the flink world. Our requirement is to stream the data from Oracle DB to Oracle DB real time (more like data replication) and we wanted to understand if Flink is the choice to achieve this. If Flink is a choice : 1. Is there any documents I can follow to perform this a

Using Flink Ml with DataStream

Hi, Is there a way to use Stochastic Outlier Selection (SOS) and/or SVM using CoCoA with streaming data. Please suggest and give pointers. Regards, Adarsh ‌

Reading parquet files using Flink

Hi team, Is there any defined way to read Parquet files for flink 1.17.1? I did some search, and found this for

Re: Exception using flink-connector-elasticsearch

Hi Javier, it seems as if you either are missing the lucene-codecs jar in your classpath or that you have a wrong version (not 4.10.4). Could you check in your job jar whether it includes lucence-codecs? If so, could you run mvn dependency:tree in the root directory of your project. There you shou

Re: Exception using flink-connector-elasticsearch

Hi, could you please try adding the lucene-core-4.10.4.jar file to your lib folder of Flink. (https://repo1.maven.org/maven2/org/apache/lucene/lucene-core/4.10.4/) Elasticsearch uses dependency injection to resolve the classes and maven is not really aware of this. Also you could try adding lu

Re: Exception using flink-connector-elasticsearch

Hi, Thanks Aljoscha, the libraries solved the problem. It worked perfectly!. On 12 January 2016 at 14:03, Aljoscha Krettek wrote: > Hi, > could you please try adding the lucene-core-4.10.4.jar file to your lib > folder of Flink. ( > https://repo1.maven.org/maven2/org/apache/lucene/lucene-core/4

Re: Accessing RDF triples using Flink

wrote: > Hi, > > I need some suggestions regarding accessing RDF triples from flink. I'm > trying to integrate flink in a pipeline where the input for flink comes > from SPARQL query on a Jena model. And after modification of triples using > flink, I will be performing SPARQL

Re: Accessing RDF triples using Flink

16 00:51, "Ritesh Kumar Singh" > wrote: > >> Hi, >> >> I need some suggestions regarding accessing RDF triples from flink. I'm >> trying to integrate flink in a pipeline where the input for flink comes >> from SPARQL query on a Jena model. And afte

Re: Accessing RDF triples using Flink

tion. >> On 6 Apr 2016 00:51, "Ritesh Kumar Singh" >> wrote: >> >>> Hi, >>> >>> I need some suggestions regarding accessing RDF triples from flink. I'm >>> trying to integrate flink in a pipeline where the input for flink comes &g

general design questions when using flink

1. Suppose I have stream of different events(A,B,C). Each event will need it's own processing pipeline. what is recommended approach of splitting pipelines per each event? I can do some filter operator at the beginning. I can setup different jobs per each event. I can hold every such event in diffe

Using Flink and Cassandra with Scala

Hi, Does the Cassandra sink support Scala and case classes? It looks like using Java is at the moment best practice. Cheers, Sanne

Streaming data from MongoDB using Flink

Good morning, I am trying to get data from MongoDB to be analysed in Flink. I would like to know if it is possible to stream data from MongoDB into Flink. I have looked into Flink's source function to add in the addSource method of the StreamExecutionEnvironment but I had no luck. Can anyone help

Re: Custom timer implementation using Flink

/flink-docs-release-1.2/dev/stream/process_function.html Cheers, Gordon On April 11, 2017 at 3:25:39 AM, jaxbihani (jagad...@helpshift.com) wrote: I have a use case which I am trying to solve using Flink. Need an advice to decide on the correct approach. Use case: -- I have a

Re: Custom timer implementation using Flink

-mailing-list-archive.2336050.n4.nabble.com/Custom-timer-implementation-using-Flink-tp12581p12592.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Using Flink with Redis question

Maybe wrapping Jedis with a serializable class will do the trick? But in general is there a way to reference jar classes in flink apps without serializable them? > On Sep 4, 2015, at 1:36 PM, Jerry Peng wrote: > > Hello, > > So I am trying to use jedis (redis java client) with Flink streami

Re: Using Flink with Redis question

Hey Jerry, Jay is on the right track, this issue has to do with the Flink operator life cycle. When you hit execute all your user defined classes get serialized, so that they can be shipped to the workers on the cluster. To execute some code before your FlatMapFunction starts processing the data y

Re: [Stateful Functions] Using Flink CEP

I am leaning towards using Siddhi as a library, but I would really love to stick with Flink CEP, or at least the specific CEP mechanism that Flink CEP uses. Exploring the codebase of Flink CEP wasn't much promising on this end. -- [image: MotaWord] Oytun Tez M O T A W O R D | CTO & Co-Founder o

Re: [Stateful Functions] Using Flink CEP

Hi! It isn't possible to use Flink CEP within Stateful Functions. That could be an interesting primitive, to add CEP-based function constructs. Could your briefly describe what you are trying to achieve? On the other hand, there are plans to integrate Stateful Functions more closely with the Fli

Re: [Stateful Functions] Using Flink CEP

Hi, I'm not familiar with the other library that you have mentioned, and indeed using Flink CEP from within a stateful function is not possible within a single Flink job, as Gordon mentioned. I'm wondering what aspects of CEP are you interested in? Because essentially a stateful funct

Re: [Stateful Functions] Using Flink CEP

Hi Gordon, Getting a little closer to Flink API could be helpful here with integration. DataStreams as ingress/egress would be AMAZING. Deploying regular Flink API code and statefun together as a single job is also something I will explore soon. With CEP, I simply want to keep a Function-specific

Re: [Stateful Functions] Using Flink CEP

41 AM Igal Shilman wrote: > Hi, > > I'm not familiar with the other library that you have mentioned, and > indeed using Flink CEP from within a stateful function is not possible > within a single Flink job, as Gordon mentioned. > > I'm wondering what aspects of CEP

Developing Beam applications using Flink checkpoints

Hi, we are starting to use Beam with Flink as runner on our applications, and recently we would like to get advantages that Flink checkpoiting provides, but it seems we are not understanding it clearly. Simplifying, our application does the following: - Read meesages from a couple of Kafka topic

Using Flink in an university course

Hi all, I'm working on a setup to use Apache Flink in an assignment for a Big Data (bachelor) university course and I'm interested in your view on this. To sketch the situation: - > 200 students follow this course - students have to write some (simple) Flink applications using the DataStream API;

Re: Challenges using Flink REST API

You should get the full stacktrace if you upgrade to 1.7.2 . On 13.03.2019 09:55, Wouter Zorgdrager wrote: Hey all! I'm looking for some advice on the following; I'm working on an abstraction on top of Apache Flink to 'pipeline' Flink applications using Kafka. For deployment this means that a

Re: Challenges using Flink REST API

Hi Chesnay, Unfortunately this is not true when I run the Flink 1.7.2 docker images. The response is still: { "errors": [ "org.apache.flink.client.program.ProgramInvocationException: The main method caused an error." ] } Regards, Wouter Zorgdrager Op wo 13 mrt. 2019 om 10:42 schr

Re: Challenges using Flink REST API

Can you give me the stacktrace that is logged in the JobManager logs? On 13.03.2019 10:57, Wouter Zorgdrager wrote: Hi Chesnay, Unfortunately this is not true when I run the Flink 1.7.2 docker images. The response is still: { "errors": [ "org.apache.flink.client.program.ProgramInvocationE

Re: Challenges using Flink REST API

Hey Chesnay, Actually I was mistaken by stating that in the JobManager logs I got the full stacktrace because I actually got the following there: 2019-03-13 11:55:13,906 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler- Exception occurred in REST handler: org.apache.flink.clien

Re: Challenges using Flink REST API

My bad, I was looking at the wrong code path. The linked issue isn't helpful, as it only slightly extends the exception message. You cannot get the stacktrace in 1.7.X nor in the current RC for 1.8.0 . I've filed https://issues.apache.org/jira/browse/FLINK-11902 to change this. The 1.8.0 RC j

Re: Using Flink Ml with DataStream

I don't believe this to be possible. The ML library works exclusively with the Batch API. On 30.10.2017 12:52, Adarsh Jain wrote: Hi, Is there a way to use Stochastic Outlier Selection (SOS) and/or SVM using CoCoA with streaming data. Please suggest and give pointers. Regards, Adarsh ‌

Re: Using Flink Ml with DataStream

Hi Chesnay, Thanks for the reply, do you know how to serve using the trained model? Where is the model saved? Regards, Adarsh ‌ On Wed, Nov 1, 2017 at 4:46 PM, Chesnay Schepler wrote: > I don't believe this to be possible. The ML library works exclusively with > the Batch API. > > > On 30.

Re: Reading parquet files using Flink

Hi, Are you using the DataStream API to read parquet file? Why not use Flink SQL to read the it? The ParquetRowInputFormat has been removed, you can use ParquetColumnarRowInputFormat in 1.17.1. Best, Ron Hou, Lijuan via user 于2023年9月12日周二 05:49写道： > Hi team, > > > > Is there any defined way t

Using Flink k8s operator on OKD

Hi community, I was wondering if anyone tried to deploy Flink using Flink k8s operator on machine where OKD [1] is installed? We have tried to install Flink k8s operator version 1.6 which seems to succeed, however when we try to deploy simple Flink deployment we are getting an error. 2023-09-19

Testing Kafka interface using Flink interactive shell

Hi, IN Spark shell I can load Kafka jar file through spark-shell option --jar spark-shell --master spark://50.140.197.217:7077 --jars ,/home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar This works fine. In Flink I have added the jar file /home/hduser/jars/flink-connector-kafka-0.10

Re: general design questions when using flink

Hi, if it is a fixed number of event types and logical pipelines I would probably split them into several jobs to achieve good isolation. There are, however people who go a different way and integrate everything into a general-purpose job that can be dynamically modified and also deals with errors

Re: Using Flink and Cassandra with Scala

the cassandra sink only supports java tuples and POJO's. On 29.09.2016 16:33, Sanne de Roever wrote: Hi, Does the Cassandra sink support Scala and case classes? It looks like using Java is at the moment best practice. Cheers, Sanne

Re: Using Flink and Cassandra with Scala

Thanks Chesnay. Have a good weekend. On Thu, Sep 29, 2016 at 5:03 PM, Chesnay Schepler wrote: > the cassandra sink only supports java tuples and POJO's. > > > On 29.09.2016 16:33, Sanne de Roever wrote: > >> Hi, >> >> Does the Cassandra sink support Scala and case classes? It looks like >> using

Re: Using Flink and Cassandra with Scala

How hard would it be to add case class support? Internally, tuples and case classes are treated quite similar, so I think it could be a quite simple extension... On Fri, Sep 30, 2016 at 4:22 PM, Sanne de Roever wrote: > Thanks Chesnay. Have a good weekend. > > On Thu, Sep 29, 2016 at 5:03 PM, C

Re: Using Flink and Cassandra with Scala

FYI: FLINK-4497 [1] requests Scala tuple and case class support for the Cassandra sink and was opened about a month ago. [1] https://issues.apache.org/jira/browse/FLINK-4497 2016-09-30 23:14 GMT+02:00 Stephan Ewen : > How hard would it be to add case class support? > > Internally, tuples and cas

Re: Streaming data from MongoDB using Flink

Hi Pedro! This is definitely possible, by simply writing a Flink `SourceFunction` that uses MongoDB clients to fetch the data. It should be straightforward and works well with MongoDB’s cursor APIs. Could you explain a bit which part in particular you were stuck with? Cheers, Gordon On Februa

Re: Streaming data from MongoDB using Flink

Dear Tzu-Li, Thank you so much for your prompt response. Lets assume I have a variable, in Java, env which is my StreamExecutionEnvironment. When I go ahead and attempt to do: > ​env.addSource(); > ​It requests an implementation of a Source Function interface: ​ > env.addSource(new SourceFunct

Re: Streaming data from MongoDB using Flink

I would recommend checking out the Flink RabbitMQ Source for examples: https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-rabbitmq/src/main/java/org/apache/flink/streaming/connectors/rabbitmq/RMQSource.java For your case, you should extend the `RichSourceFunction` which p

Re: Streaming data from MongoDB using Flink

Thank you again for your prompt response. I will give it a try and will come back to you. *Pedro Lima Monteiro* On 16 February 2017 at 10:20, Tzu-Li (Gordon) Tai wrote: > I would recommend checking out the Flink RabbitMQ Source for examples: > https://github.com/apache/flink/blob/master/flink-

Re: Streaming data from MongoDB using Flink

Dear Gordon, Thanks for your help, I think I am on the right track as of now. On the other hand, I have another question: is it possible to add sources to environments that are already executing? In what I am currently developing, I need to add new sources as they arrive to my system. I will wai

Re: Streaming data from MongoDB using Flink

Good to know! On February 16, 2017 at 10:13:28 PM, Pedro Monteiro (pedro.mlmonte...@gmail.com) wrote: Dear Gordon, Thanks for your help, I think I am on the right track as of now. On the other hand, I have another question: is it possible to add sources to environments that are already execu

Re: Streaming data from MongoDB using Flink

Hi Pedro, in order to add new sources you have to first stop the job (maybe taking a savepoint if you want to resume later on) and then restart the job with the changed topology. Cheers, Till On Thu, Feb 16, 2017 at 4:06 PM, Tzu-Li (Gordon) Tai wrote: > Good to know! > > > On February 16, 2017

Re: Streaming data from MongoDB using Flink

Sorry, I just realized I didn’t notice the second part question of your last email when replying. Thanks Till for answering it! On February 17, 2017 at 6:05:58 PM, Till Rohrmann (trohrm...@apache.org) wrote: Dear Gordon, Thanks for your help, I think I am on the right track as of now. On the

Re: Streaming data from MongoDB using Flink

Dear Gordon, Till Thank you so much for your helpful answers. I managed to solve my problem with your guidelines. Much appreciated, keep up the good work! Cheers Cumprimentos, *Pedro Lima Monteiro* On 17 February 2017 at 10:10, Tzu-Li (Gordon) Tai wrote: > Sorry, I just realized I didn’t no

Doubt Regarding producing to kafka using flink

Hi I am using flink-connector-kafka-0.10_2.10 while producing i am getting error as java.lang.IllegalArgumentException:* Invalid timestamp -9223372036854775808* at org.apache.kafka.clients.producer.ProducerRecord.(ProducerRecord.java:60) ~[kafka-clients-0.10.0.1.jar:na] at

Re: Developing Beam applications using Flink checkpoints

Hi Ivan, First let's address the issue with idle partitions. The solution is to use a watermark assigner that also emits a watermark with some idle timeout [1]. Now the second question, on why Kafka commits are committed for in-flight, checkpointed data. The basic idea is that you are not losing

Re: Developing Beam applications using Flink checkpoints

Thanks for your complete answer Arvid, we will try to approach all things you mentioned, but take into account we are using Beam on top of Flink, so, to be honest, I don't know how could we implement the custom serialization thing ( https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/s

Re: Developing Beam applications using Flink checkpoints

Hi Ivan, The easiest way is to use some implementation that's already there [1]. I already mentioned Avro and would strongly recommend giving it a go. If you make sure to provide a default value for as many fields as possible, you can always remove them later giving you great flexibility. I can gi

Re: Developing Beam applications using Flink checkpoints

Yep, sorry if I'm bothering you but I think I'm still not getting this, how could I tell Beam to tell Flink to use that serializer instead of Java standard one, because I think Beam is abstracting us from Flink checkpointing mechanism, so I'm afraid that if we use Flink API directly we might break

Re: Developing Beam applications using Flink checkpoints

Actually I'm also thinking about how Beam coders are related with runner's serialization... I mean, on Beam you specify a coder per each Java type in order to Beam be able to serialize/deserialize that type, but then, is the runner used under the hood serializing/deserializing again the result, so

Re: Developing Beam applications using Flink checkpoints

Hi Ivan, I'm fearing that only a few mailing list users have actually deeper Beam experience. I only used it briefly 3 years ago. Most users here are using Flink directly to avoid these kinds of double-abstraction issues. It might be better to switch to the Beam mailing list if you have

Re: Developing Beam applications using Flink checkpoints

only used it briefly 3 years ago. Most users here > are using Flink directly to avoid these kinds of double-abstraction > issues. > > It might be better to switch to the Beam mailing list if you have > Beam-specific questions including how the Flink runner actually > translates th

Re: Developing Beam applications using Flink checkpoints

link, but it seems is not so popular. > > On Tue, 2020-05-19 at 12:46 +0200, Arvid Heise wrote: > > Hi Ivan, > > > > I'm fearing that only a few mailing list users have actually deeper > > Beam experience. I only used it briefly 3 years ago. Most users here >

UnsatisfiedLinkError when using flink-s3-fs-hadoop

Hi, I have experience UnsatisfiedLinkError when I tried to use flink-s3-fs-hadoop to sink to s3 in my local Windows machine. I googled and tried several solutions like download hadoop.dll and winutils.exe, set up HADOOP_HOME and PATH environment variables, copy hadoop.dll to C:\Windows\System32,

Re: Using Flink in an university course

It would help to understand the current issues that you have with this approach? I used a similar approach (not with Flink, but a similar big data technology) some years ago > Am 04.03.2019 um 11:32 schrieb Wouter Zorgdrager : > > Hi all, > > I'm working on a setup to use Apache Flink in an as

Re: Using Flink in an university course

Hi Wouter, We are using Docker Compose (Flink JM, Flink TM, Kafka, Zookeeper) setups for our trainings and it is working very well. We have an additional container that feeds a Kafka topic via the commandline producer to simulate a somewhat realistic behavior. Of course, you can do it without Kafk

Re: Using Flink in an university course

Hey all, Thanks for the replies. The issues we were running into (which are not specific to Docker): - Students changing the template wrongly failed the container. - We give full points if the output matches our solutions (and none otherwise), but it would be nice if we could give partial grades p

Re: Using Flink in an university course

Hi there, As far as a runtime for students, it seems like docker is your best bet. However, you could have them instead package a jar using some interface (for example, see https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/packaging.html, which details the `Program` interface) and th

Re: Using Flink in an university course

Hi all, Thanks for the input. Much appreciated. Regards, Wouter Op ma 4 mrt. 2019 om 20:40 schreef Addison Higham : > Hi there, > > As far as a runtime for students, it seems like docker is your best bet. > However, you could have them instead package a jar using some interface > (for example,

multiple pipeline deployment using flink k8s operator

Hi all, we just started using the flink k8s operator to deploy our flink cluster. >From what we understand we are only able to start a flink cluster per job. So in our case when we have 2 jobs we have to create 2 different clusters. obviously we would prefer to deploy these 2 job which relate to th

Unable to start job using Flink Operator

Greetings all, I am attempting to start a flink job using the Flink oeprator (version 1.1) however am running into a problem. While attempting to create the deployment i receive the following error: Resource: "flink.apache.org/v1beta1, Resource=flinkdeployments", GroupVersionKind: "flink.apa

Cannot run pyflink example using Flink CLI

Hi, I'm new to PyFlink, and I couldn't run a basic example that shipped with Flink. This is the command I tried: ./bin/flink run -py examples/python/datastream/word_count.py Here below are the results I got with different setups: 1. On AWS EMR 6.8.0 (Flink 1.15.1): *Error: No module named 'goog

configMap value error when using flink-operator?

hi:    I'm trying to deploy a flink job with flink-operaotor. The flink-operator's version is 1.2.0. And the yaml i use is here: apiVersion: flink.apache.org/v1beta1 kind: FlinkDeployment metadata: name: basic-example spec: image: flink:1.15 flinkVersion: v1_15 flinkConfiguration:

Re: Using Flink k8s operator on OKD

want to have Flink deployments in. kubectl apply -f - < wrote: > Hi community, > I was wondering if anyone tried to deploy Flink using Flink k8s operator > on machine where OKD [1] is installed? > > We have tried to install Flink k8s operator version 1.6 which seems to > succeed

Re: Using Flink k8s operator on OKD

; name: flink-role-binding > roleRef: > apiGroup: rbac.authorization.k8s.io > kind: Role > name: flink > subjects: > - kind: ServiceAccount > name: flink > EOF > > Hopefully that helps. > > > On Tue, Sep 19, 2023 at 5:40 PM Krzysztof Chmielewski < >

Re: Using Flink k8s operator on OKD

/finalizers >> verbs: >> - '*' >> --- >> apiVersion: rbac.authorization.k8s.io/v1 >> kind: RoleBinding >> metadata: >> labels: >> app.kubernetes.io/name: flink-kubernetes-operator >> app.kubernetes.io/version: 1.5.0 >> name

Re: Using Flink k8s operator on OKD

t;>> resources: >>> - deployments >>> - deployments/finalizers >>> verbs: >>> - '*' >>> --- >>> apiVersion: rbac.authorization.k8s.io/v1 >>> kind: RoleBinding >>> metadata: >>> labels: >>&g

Using Flink with Scala 2.11 and Java 8

Is it possible to use Scala 2.11 and Java 8? I'm able to get our project to compile correctly, however there are runtime errors with the Reflectasm library (I'm guessing due to Kyro). I looked into the error and it seems Spark had the same issue ( https://issues.apache.org/jira/browse/SPARK-6152,

protobuf messages from Kafka to elasticsearch using flink

Friends, Can someone guide me or share an example on how to consume protobuf message from kafka and index into Elasticsearch using flink?

Re: Testing Kafka interface using Flink interactive shell

Hi Mich, You can add external dependencies to Scala shell using `--addclasspath` option. There is more detail description in documentation [1]. [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/scala_shell.html#adding-external-dependencies Regards, Chiwan Park > On Apr 17,

Re: Testing Kafka interface using Flink interactive shell

Thanks Chiwan. It worked. Now I have this simple streaming program in Spark Scala that gets streaming data via Kafka. It is pretty simple. Please see attached. I am trying to make it work with Flink + Kafka Any hints will be appreciated. Thanks Dr Mich Talebzadeh LinkedIn * https://www.l

Re: Doubt Regarding producing to kafka using flink

://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/event_timestamps_watermarks.html [2]  https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/event_timestamp_extractors.html On April 2, 2017 at 6:38:13 PM, Archit Mittal (marchi...@gmail.com) wrote: Hi  I am using flink-connector-kafka

Re: Doubt Regarding producing to kafka using flink

> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/event_timestamps_watermarks.html> > [2] https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/event_ > timestamp_extractors.html > > On April 2, 2017 at 6:38:13 PM, Archit Mittal (marchi...@gmail.com) wrote: &

Re: Doubt Regarding producing to kafka using flink

Cheers, Gordon [1]  https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/event_timestamps_watermarks.html [2]  https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/event_timestamp_extractors.html On April 2, 2017 at 6:38:13 PM, Archit Mittal (marchi...@gmail.com) wrote: Hi  I

Read data from Oracle using Flink SQL API

Hi to all, I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you shou

Stream aggregation using Flink Table API (Blink plan)

Hi community, I am testing the "Split Distinct Aggregation" [1] consuming the taxi ride data set. My sql query from the table environment is the one below: Table tableCountDistinct = tableEnv.sqlQuery("SELECT startDate, COUNT(driverId) FROM TaxiRide GROUP BY startDate"); and I am enableing: conf

Getting "ProducerFenced" exception while using flink kafka producer

Hi, I have two flink jobs, both the jobs are using Flink Kafka Producer and Flink Kafka Consumer running in Exactly-Once mode. Parallelism of both the jobs is one. Both the jobs are same in number of operators and type of operators. When we start one job then that job runs fine. But as soon as

Conflicting Cassandra versions while using flink-connector-cassandra

I'm having trouble using CassandraSink while also using datastax's cassandra-driver. While digging trough the flink-connector-cassandra.jar I realized that it bundles *cassandra-driver-core 3.0.0* while internally we use version *3.1.4* to read data from Cassandra. I couldn't find the reason why t

Skewed Data when joining tables using Flink SQL

Hi, I have a Flink batch job that needs to join a large skewed table with a smaller table, and because records are not evenly distributed to each subtask, it always fails with a "too much data in partition" error. I explored using DataStream API to broadcast the smaller tables as a broadcast state

Distribute crawling of a URL list using Flink

Hi all, I am fairly new to Flink. I have this project where I have a list of URLs (In one node) which need to be crawled distributedly. Then for each URL, I need the serialized crawled result to be written to a single text file. I want to know if there are similar projects which I can look into o

  1   2   3   4   >