Re: issues with Flink kinesis connector

Thank you for your help! On Thu, May 23, 2024 at 1:40 PM Aleksandr Pilipenko wrote: > Hi Nick, > > You need to use another method to add sink to your job - sinkTo. > KinesisStreamsSink implements newer Sink interface, while addSink expect > old SinkFunction. You can see this by looking at

Re: issues with Flink kinesis connector

Hi Nick, You need to use another method to add sink to your job - sinkTo. KinesisStreamsSink implements newer Sink interface, while addSink expect old SinkFunction. You can see this by looking at method signatures[1] and in usage examples in documentation[2] [1]

issues with Flink kinesis connector

Hello, I am currently having issues trying to use the python flink 1.18 Datastream api with the Amazon Kinesis Data Streams Connector. >From the documentation https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/kinesis/ I have downloaded the

Re: Flink kinesis connector 4.3.0 release estimated date

That’s great news. Thanks. From: Leonard Xu Date: Thursday, 23 May 2024 at 04:42 To: Vararu, Vadim Cc: user , Danny Cranmer Subject: Re: Flink kinesis connector 4.3.0 release estimated date Hey, Vararu The kinesis connector 4.3.0 release is under vote phase and we hope to finalize

Re: Flink kinesis connector 4.3.0 release estimated date

Hey, Vararu The kinesis connector 4.3.0 release is under vote phase and we hope to finalize the release work in this week if everything goes well. Best, Leonard > 2024年5月22日 下午11:51，Vararu, Vadim 写道： > > Hi guys, > > Any idea when the 4.3.0 kinesis connector is estimated

Flink kinesis connector 4.3.0 release estimated date

Hi guys, Any idea when the 4.3.0 kinesis connector is estimated to be released? Cheers, Vadim.

Kinesis connector writes wrong sequence number at stop with savepoint

ritten in the savepoint _metadata file. The problem is that Kinesis connector does not write to the savepoint file the sequence number corresponding to the last event written into Kafka! Instead, it writes the same sequence number as the one from the last checkpoint. That way, after resuming the jo

Re: Kinesis Connector does not work

11/2022 11:49, Matt Fysh wrote: Ok thanks, will give that a try. Is that something that should be added to the Kinesis connector docs page? There are existing instructions there for adding the flink-connector-kinesis jar as a dependency, but no instructions for adding commo

Re: Kinesis Connector does not work

Ok thanks, will give that a try. Is that something that should be added to the Kinesis connector docs page? There are existing instructions there for adding the flink-connector-kinesis jar as a dependency, but no instructions for adding commons-logging Or if this is something more general

Re: Kinesis Connector does not work

Said dependency (on commons-logging) is not meant to be provided by the docker image, but bundled in your user-jar (along with the connector). On 08/11/2022 02:14, Matt Fysh wrote: Hi, I'm following the kinesis connector instructions as documented here: https://nightlies.apache.org/flink

Kinesis Connector does not work

Hi, I'm following the kinesis connector instructions as documented here: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/kinesis/ I'm also running Flink in standalone session mode using docker compose and the Python images, as described in the Flink docs

Flink Kinesis Connector EFO not working for cross-accounts

Hey there, Checking if Flink kinesis connector (using 1.15.2 version) EFO can do cross-accounts? My configuration looks like this: if (kinesisIamRole != null && !kinesisIamRole.isEmpty()) {  kinesisConsumerProps.put(AWSConfigConstants.AWS_ROLE_ARN, kinesisIamRole);  kinesisConsumerP

Re: Flink Kinesis connector - EFO connection error with http proxy settings

ntribution process. > > > > Thanks > > -Saravan > > > > *From: *Danny Cranmer > *Date: *Wednesday, January 19, 2022 at 3:10 AM > *To: *Gnanamoorthy, Saravanan > *Cc: *user@flink.apache.org > *Subject: *Re: Flink Kinesis connector - EFO connection erro

Re: Flink Kinesis connector - EFO connection error with http proxy settings

Hi Danny, I should be able to make the contribution to add proxy support. Please let me know the contribution process. Thanks -Saravan From: Danny Cranmer Date: Wednesday, January 19, 2022 at 3:10 AM To: Gnanamoorthy, Saravanan Cc: user@flink.apache.org Subject: Re: Flink Kinesis connector

Re: Flink Kinesis connector - EFO connection error with http proxy settings

/org/apache/flink/streaming/connectors/kinesis/util/AwsV2Util.java#L113 Thanks, Danny Cranmer. On Tue, Jan 18, 2022 at 12:52 AM Gnanamoorthy, Saravanan < saravanan.gnanamoor...@fmr.com> wrote: > Hello, > > We are using Flink kinesis connector for processing the streaming data >

Flink Kinesis connector - EFO connection error with http proxy settings

Hello, We are using Flink kinesis connector for processing the streaming data from kinesis. We are running the application behind the proxy. After the proxyhost and proxyport settings, the Connector works with default publisher type(Polling) but it doesn’t work when we enable the publisher type

Re: Kinesis Connector and Savepoint/Checkpoint restore.

r of shards which results >>>>> into >>>>> scheduling those shards with earliest stream position. >>>>> I am suspecting that somehow the checkpoint is missing state for some >>>>> of the shards. But if this is the case then that checkpoint s

Re: Kinesis Connector and Savepoint/Checkpoint restore.

n that checkpoint should had >>>> failed. >>>> >>>> Any further information to resolve this issue would be highly >>>> appreciated... >>>> >>>> Regards, >>>> Ravi >>>> >>>> On Wed, Oct 16, 20

Re: Kinesis Connector and Savepoint/Checkpoint restore.

>> appreciated... >>> >>> Regards, >>> Ravi >>> >>> On Wed, Oct 16, 2019 at 5:57 AM Yun Tang wrote: >>> >>>> Hi Steven >>>> >>>> If you restore savepoint/checkpoint successfully, I think this might >>&

Re: Kinesis Connector and Savepoint/Checkpoint restore.

gt; to the shard wasn't discovered in the previous run, therefore it would be >>> consumed from the beginning. Please refer to the implementation here: [1] >>> >>> [1] >>> https://github.com/apache/flink/blob/2c411686d23f456cdc502abf1c6b97a61070a17d/flink-connect

Re: Kinesis Connector and Savepoint/Checkpoint restore.

the beginning. Please refer to the implementation here: [1] >> >> [1] >> https://github.com/apache/flink/blob/2c411686d23f456cdc502abf1c6b97a61070a17d/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L307 >

Re: Kinesis Connector and Savepoint/Checkpoint restore.

est > Yun Tang > -- > *From:* Steven Nelson > *Sent:* Wednesday, October 16, 2019 4:31 > *To:* user > *Subject:* Kinesis Connector and Savepoint/Checkpoint restore. > > Hello, we currently use Flink 1.9.0 with Kinesis to process data. > > We ha

Re: Kinesis Connector and Savepoint/Checkpoint restore.

/2c411686d23f456cdc502abf1c6b97a61070a17d/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L307 Best Yun Tang From: Steven Nelson Sent: Wednesday, October 16, 2019 4:31 To: user Subject: Kinesis Connector

Kinesis Connector and Savepoint/Checkpoint restore.

Hello, we currently use Flink 1.9.0 with Kinesis to process data. We have extended data retention on the Kinesis stream, which gives us 7 days of data. We have found that when a savepoint/checkpoint is restored that it appears to be restarting the Kinesis Consumer from the start of the stream.

Re: [ANNOUNCE] Kinesis connector becomes part of Flink releases

Good news! Thanks for your efforts, Bowen! Best, Vino Yu Li 于2019年9月2日周一 上午6:04写道： > Great to know, thanks for the efforts Bowen! > > And I believe it worth a release note in the original JIRA, wdyt? Thanks. > > Best Regards, > Yu > > > On Sat, 31 Aug 2019 at 11:01, Bowen Li wrote: > >> Hi

Re: [ANNOUNCE] Kinesis connector becomes part of Flink releases

Great to know, thanks for the efforts Bowen! And I believe it worth a release note in the original JIRA, wdyt? Thanks. Best Regards, Yu On Sat, 31 Aug 2019 at 11:01, Bowen Li wrote: > Hi all, > > I'm glad to announce that, as #9494 > was merged

[ANNOUNCE] Kinesis connector becomes part of Flink releases

Hi all, I'm glad to announce that, as #9494 was merged today, flink-connector-kinesis is officially of Apache 2.0 license now in master branch and its artifact will be deployed to Maven central as part of Flink releases starting from Flink 1.10.0. Users

Re: Connectors (specifically Kinesis Connector)

Hi Steven, I assume you are referring to the problem that we don't publish the Kinesis connector artifacts to Maven, due to the licensing issue with KCL? I didn't manage to find any JIRAs that were addressing this issue directly, but the most related one would be this: https://issues.apache.org

Connectors (specifically Kinesis Connector)

Hello! We keep having difficulties with the Kinesis connector. We have to publish our own version, and we understand why. What I am curious about is the plan to make this better in the future. Is there an issue/FLIP that I can reference when talking internally about this? -Steve

Flink kinesis connector crashes with daemon exception

We are using flink 1.7.2 on debian slim, and kubernetes as the resource manager. But when we deploy it, it works for an hour or so without any issues and then starts failing with the following error: java.lang.RuntimeException: Exception occurred while processing valve output watermark: at

Re: Kinesis Connector - NoClassDefFoundError

Hey, Have you updated the versions both on the environment and the dependency on the job? >From my personal experience, 95 % of such issues is due to the mismatch between Flink versions on the cluster you are using and Your job. Best Regards, Dom. wt., 20 lis 2018 o 07:41 Steve Bistline

Kinesis Connector - NoClassDefFoundError

Hey all... upgrade from Flink 1.5.0 to 1.6.2 and for some reason cannot figure out what I missed in setting up the new environment. I am gettin this error: java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.kinesis.shaded.com.amazonaws.partitions.PartitionsLoader

Re: FLINK Kinesis Connector Error - ProvisionedThroughputExceededException

Hi all, I think Steve's occurrence of the warning was from the consumer side. For the Flink Kinesis Consumer, this could most likely occur due to excessive ListShard API calls on the target Kinesis stream. The consumer uses this API to discover shards, at a fixed interval. The problem with the

Re: FLINK Kinesis Connector Error - ProvisionedThroughputExceededException

Hi Steve, We've encountered this also. We have way more than enough shards, but were still getting exceptions. We think we know what is the reason, we would love for someone to approve/reject. What we suspect is happening is as follows: The KPL's RateLimit parameter is tracking the amount of

Re: FLINK Kinesis Connector Error - ProvisionedThroughputExceededException

If it's running in parallel aren't you just adding readers which maxes out your provisioned throughput? probably doesn't belong in here but rather a Kinesis thing, but i suggest increasing your number of shards? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

FLINK Kinesis Connector Error - ProvisionedThroughputExceededException

I am getting this error from the Flink Kinesis Connector. I have a native KCL app running in parallel with no problems. Any help would be appreciated Thanks so much!! Steve flink-sbistl919-taskexecutor-0-CACSVML-15736.log:2018-11-09 07:46:11,579 WARN

Re: Kinesis Connector

Hi Steve, I think we can check the following things step by step: 1. Confirm if the data source has data. 2. Take a look at the log of Taskmanager or Jobmanager and check if there are exceptions. 3. Take a thread dump to see what was doing in the TaskManager. Best, Hequn On Fri, Nov 2, 2018 at

Kinesis Connector

I have tried just about everything to get a simple Flink application to consume from Kinesis. The application appears to connect ( I think ), no complaints... never receives any data. Even a very simple JAVA app see attached. Any help would be very much appreciated. Thanks -- Sent from:

Re: Kinesis Connector SequenceNumber deserialisation on restart from Savepoint/checkpoint

nevermind :) PEBKAC error. using the CLI and specifying the switches in the wrong location. Hopefully this will not be followed by an 'actually... there's still this other issue' post On 2018/02/28 15:38:29, dyana.r...@salecycle.com wrote: > 2018-02-28

Re: Kinesis Connector SequenceNumber deserialisation on restart from Savepoint/checkpoint

should note this is under 1.4.1 On 2018/02/28 15:38:29, dyana.r...@salecycle.com wrote: > 2018-02-28 15:16:36,675 INFO > org.apache.flink.api.java.typeutils.TypeExtractor - class > org.apache.flink.streaming.connectors.kinesis.model.SequenceNumber does

Kinesis Connector SequenceNumber deserialisation on restart from Savepoint/checkpoint

2018-02-28 15:16:36,675 INFO org.apache.flink.api.java.typeutils.TypeExtractor - class org.apache.flink.streaming.connectors.kinesis.model.SequenceNumber does not contain a setter for field sequenceNumber 2018-02-28 15:16:36,675 INFO

Re: Kinesis connector - Jackson issue

Hi guys, I was able to solve the issue. I deleted all my Flink distributions and followed these steps: 1) Clone Flink source (because I'm building Flink with Kinesis connector) 2) Checkout to release-1.3.1 (that's the version of Flink on EMR) 3) mvn clean install -Pinclude-kinesis -DskipTests

Re: Kinesis connector - Jackson issue

masz Dobrzycki > (dobrzycki.tom...@gmail.com) wrote: > > Hi Gordon, > > Thanks for your answer. > - I've built it with Maven 3.2.5 > - I am using Jackson in my application (version 2.7.4) > > Something that I have noticed when building Kinesis connector is that > it exclu

Re: Kinesis connector - Jackson issue

." On 26 September 2017 at 5:27:47 PM, Tomasz Dobrzycki (dobrzycki.tom...@gmail.com) wrote: Hi Gordon, Thanks for your answer. - I've built it with Maven 3.2.5 - I am using Jackson in my application (version 2.7.4) Something that I have noticed when building Kinesis conn

Re: Kinesis connector - Jackson issue

Hi Gordon, Thanks for your answer. - I've built it with Maven 3.2.5 - I am using Jackson in my application (version 2.7.4) Something that I have noticed when building Kinesis connector is that it excludes jackson: [INFO] Excluding com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar

Re: Kinesis connector - Jackson issue

Hi Tomasz, Yes, dependency clashes may surface when executing actual job runs on clusters. A few things to probably check first: - Have you built Flink or the Kinesis connector with Maven version 3.3 or above? If yes, try using a lower version, as 3.3+ results in some shading issues when used

Kinesis connector - Jackson issue

Hi guys, I'm working with Kinesis connector and currently trying to solve a bizarre issue. I had problems with Kinesis and httpcomponents which I was able to solve using steps shown in: https://github.com/apache/flink/pull/4150/commits/9b539470ac308d7af9df9a70792aa1fa8c6995fc That did the trick

Re: Kinesis connector SHARD_GETRECORDS_MAX default value

.apache.org/jira/browse/FLINK-6365. >> >> Really appreciate your contributions for the Kinesis connector! >> >> Cheers, >> Steffen >> >> On 22/03/2017 20:21, Tzu-Li (Gordon) Tai wrote: >> > Hi Steffan, >> > >> &

Re: Flink Kinesis connector in 1.3.0

n. *From: *Chesnay Schepler <ches...@apache.org> *Date: *Tuesday, June 13, 2017 at 1:44 PM *To: *"Foster, Craig" <foscr...@amazon.com>, "user@flink.apache.org" <user@flink.apache.org>, Robert Metzger <rmetz...@apache.org> *Subject: *Re: Flink Kinesi

Re: Flink Kinesis connector in 1.3.0

foscr...@amazon.com>, "user@flink.apache.org" <user@flink.apache.org>, Robert Metzger <rmetz...@apache.org> Subject: Re: Flink Kinesis connector in 1.3.0 Here's the relevant JIRA: https://issues.apache.org/jira/browse/FLINK-6812 Apologies if I was unclear, i meant

Re: Flink Kinesis connector in 1.3.0

Here's the relevant JIRA: https://issues.apache.org/jira/browse/FLINK-6812 Apologies if I was unclear, i meant that you could use the 1.3-SNAPSHOT version of the kinesis connector, as it is compatible with 1.3.0. Alternatively you can take the 1.3.0 sources and build the connector manually

Re: Flink Kinesis connector in 1.3.0

er@flink.apache.org" <user@flink.apache.org> Subject: Re: Flink Kinesis connector in 1.3.0 Thanks! Does this also explain why commons HttpClient is not included in flink-dist-*? From: Chesnay Schepler <ches...@apache.org> Date: Tuesday, June 13, 2017 at 8:53 AM To: "user@flink.apach

Re: Flink Kinesis connector in 1.3.0

Thanks! Does this also explain why commons HttpClient is not included in flink-dist-*? From: Chesnay Schepler <ches...@apache.org> Date: Tuesday, June 13, 2017 at 8:53 AM To: "user@flink.apache.org" <user@flink.apache.org> Subject: Re: Flink Kinesis connector in 1.3.0

Re: Flink Kinesis connector in 1.3.0

the Flink Kinesis Connector in 1.3.0. However, I don’t see that resolving anymore. It resolved with 1.2.x but doesn’t with 1.3.0. Is there something I need to now do differently than described here? https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/kinesis.html Thanks

Re: Kinesis connector SHARD_GETRECORDS_MAX default value

to file the issue: https://issues.apache.org/jira/browse/FLINK-6365. Really appreciate your contributions for the Kinesis connector! Cheers, Steffen On 22/03/2017 20:21, Tzu-Li (Gordon) Tai wrote: > Hi Steffan, > > I have to admit that I didn’t put too much thoughts in th

Re: Kinesis connector SHARD_GETRECORDS_MAX default value

Hi Gordon, thanks for looking into this and sorry it took me so long to file the issue: https://issues.apache.org/jira/browse/FLINK-6365. Really appreciate your contributions for the Kinesis connector! Cheers, Steffen On 22/03/2017 20:21, Tzu-Li (Gordon) Tai wrote: Hi Steffan, I have

Re: Kinesis connector SHARD_GETRECORDS_MAX default value

Hi Steffan, I have to admit that I didn’t put too much thoughts in the default values for the Kinesis consumer. I’d say it would be reasonable to change the default values to follow KCL’s settings. Could you file a JIRA for this? In general, we might want to reconsider all the default values

Kinesis connector SHARD_GETRECORDS_MAX default value

Hi there, I recently ran into problems with a Flink job running on an EMR cluster consuming events from a Kinesis stream receiving roughly 15k event/second. Although the EMR cluster was substantially scaled and CPU utilization and system load were well below any alarming threshold, the

Re: Fw: Flink Kinesis Connector

Fw: Flink Kinesis Connector Hi Matt! As mentioned in the docs, due to the ASL license, we do not deploy the artifact to the Maven central repository on Flink releases. You will need to build the Kinesis connector by yourself (the instructions to do so are also in the Flink Kinesis conne

Re: Fw: Flink Kinesis Connector

Hi Matt! As mentioned in the docs, due to the ASL license, we do not deploy the artifact to the Maven central repository on Flink releases. You will need to build the Kinesis connector by yourself (the instructions to do so are also in the Flink Kinesis connector docs :)), and install

Re: Kinesis Connector Dependency Problems

ncies. > >You have to use Maven 3.0.3 (see [1]). > > > >Best, Fabian > > > >[1] > >https://ci.apache.org/projects/flink/flink-docs- > release-1.1/setup/building.html > > > >2016-11-08 11:05 GMT+01:00 Till Rohrmann <trohr

Re: Kinesis Connector Dependency Problems

;: > >> Yes this definitely looks like a similar issue. Once we shade the aws >> dependencies in the Kinesis connector, the problem should be >(hopefully) >> resolved. I've added your problem description to the JIRA. Thanks for >> reporting it. >> >>

Re: Kinesis Connector Dependency Problems

Yes this definitely looks like a similar issue. Once we shade the aws dependencies in the Kinesis connector, the problem should be (hopefully) resolved. I've added your problem description to the JIRA. Thanks for reporting it. Cheers, Till On Mon, Nov 7, 2016 at 8:01 PM, Foster, Craig <fo

Re: Kinesis Connector Dependency Problems

;> /usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar >>> /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar >>> /usr/lib/hadoop-yarn/lib/activation-1.1.jar >>> /usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar >>> /usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar >>> /usr/lib/had

Re: Kinesis Connector Dependency Problems

mr/emrfs/lib/emrfs-hadoop-2.4.0.jar >> /usr/share/aws/emr/emrfs/lib/protobuf-java-2.5.0.jar >> /usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.16.jar >> /usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar >> /usr/share/aws/emr/emrfs/lib/commons-logging-1.1.3.jar >> /usr/share/aws/emr/emrfs

Re: Kinesis Connector Dependency Problems

gt; /usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar > > On Tue, Nov 1, 2016 at 3:57 AM, Till Rohrmann <trohrm...@apache.org> > wrote: > >> Hi Justin, >> >> I think this might be a problem in Flink's Kinesis consumer. The Flink >> Kinesis consumer

Re: Kinesis Connector Dependency Problems

> Hi all - first time on the mailing list, so my apologies if I break >> protocol on anything. Really excited to be using Flink, and hoping to be >> active here in the future! Also, apologies for the length of this email - >> I tried to include details but may have gone overboard. &g

Re: Kinesis Connector Dependency Problems

gies if I break > protocol on anything. Really excited to be using Flink, and hoping to be > active here in the future! Also, apologies for the length of this email - > I tried to include details but may have gone overboard. > > The gist of my problem is an issue with packaging the F

Kinesis Connector Dependency Problems

is an issue with packaging the Flink Kinesis Connector into my user code for execution on a YARN cluster in EMR - there's some dependency trouble happening, but after about 48 hours of attempts, I'm not sure how to make progress, and I'd really appreciate any ideas or assistance. Thank you in advance

Re: Kinesis connector - Iterator expired exception

57 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: Hi Josh, Thank you for reporting this, I’m looking into it. There was some major changes to the Kinesis connector after mid June, but the changes don’t seem to be related to the iterator timeout, so it may be a bug that had always

Re: Kinesis connector - Iterator expired exception

is, I’m looking into it. There was some major > changes to the Kinesis connector after mid June, but the changes don’t seem > to be related to the iterator timeout, so it may be a bug that had always > been there. > > I’m not sure yet if it may be related, but may I ask how long wa

Re: Kinesis connector - Iterator expired exception

Hi Josh, Thank you for reporting this, I’m looking into it. There was some major changes to the Kinesis connector after mid June, but the changes don’t seem to be related to the iterator timeout, so it may be a bug that had always been there. I’m not sure yet if it may be related, but may I

Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

ilesystem.FsStateBackend.(FsStateBackend.java:175) >>>>> at >>>>> org.apache.flink.runtime.state.filesystem.FsStateBackend.(FsStateBackend.java:144) >>>>> at >>>>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.(RocksDBS

Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

eBackend.(FsStateBackend.java:144) >>>> at >>>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.(RocksDBStateBackend.java:205) >>>> >>>> This is strange because I used an s3:// checkpoint directory when >>>> running Flink 1

Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

hould be needed to use S3 when running on EMR). >> >> Anyway I tried executing /etc/hadoop/conf/hadoop-env.sh before running my >> job, as this sets up the HADOOP_CLASSPATH env var. The exception then >> changed to: >> java.lang.NoClassDefFoundError: org/apache/hadoo

Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

r. The exception then > changed to: > java.lang.NoClassDefFoundError: org/apache/hadoop/fs/common/Abortable > > I found that this class is related to a jar called s3-dist-cp, so then I > tried copying that jar to Flink's lib directory from > /usr/share/aws/emr/s3-dist-

Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

ectionParams.class").toString` > > Also, on which commit was your kinesis connector built? > > Regards, > Gordon > > > On June 17, 2016 at 1:08:37 AM, Josh (jof...@gmail.com) wrote: > > Hey, > > I've been running the Kinesis connector successfully now for a couple

Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

Hey, I've been running the Kinesis connector successfully now for a couple of weeks, on a Flink cluster running Flink 1.0.3 on EMR 2.7.1/YARN. Today I've been trying to get it working on a cluster running the current Flink master (1.1-SNAPSHOT) but am running into a classpath issue when starting

Re: Kinesis Connector

Hi Giancarlo, Since it has been a while since the last post and there hasn't been a JIRA ticket opened for Kinesis connector yet, I'm wondering how you are doing on the Kinesis connector and hope to help out with this feature :) I've opened a JIRA (https://issues.apache.org/jira/browse/FLINK

Re: Kinesis Connector

<gianca...@beamly.com> wrote: > Hi, > > Is there any project already working on a Kinesis connector for Flink or > any plan to add a Kinesis connector to the main Flink distribution in the > future? > > Thanks, > Giancarlo

Re: Kinesis Connector

Hi Giancarlo! I am not aware of any existing Kinesis connector. Would be definitely something to put onto the roadmap for the near future. This is a stream source we should support similarly to Kafka. I am not super familiar with Kinesis, but it looks a bit like offering a similar abstraction

Kinesis Connector

Hi, Is there any project already working on a Kinesis connector for Flink or any plan to add a Kinesis connector to the main Flink distribution in the future? Thanks, Giancarlo