Hi,
The issue is that the KCL inside the Spark Streaming connector does not provide
a way to pass KCL configuration in, which means we can’t supply configuration
to disable SSL cert checks. In a typical (non-Spark Streaming) KCL app, we can
instantiate the KCL via:
Worker worker = new Worker.Builder()
.config(kclConfig) , //we can specify to disable SSL certs here.
.kinesisClient(kinesisClient)
.recordProcessorFactory(processorFactory)
.build();
However with the Kinesis Spark Streaming consumer, we do not have the ability
to set this. Our only interface with Kinesis is:
KinesisUtils.createStream(context, appName, streamName, serviceEndpoint,
regionName,
InitialPositionInStream.TRIM_HORIZON, checkpoint,
StorageLevel.MEMORY_AND_DISK_2())
and so we can’t pass config anywhere for the KCL to read.
Regards,
Ben
Shashikant Bangera | DevOps Engineer
Payment Services DevOps Engineering
Email: shashikantbang...@discover.com<mailto:shashikantbang...@discover.com>
Group email: eppdev...@discover.com<mailto:eppdev...@discover.com>
Tel: +44 (0)
Mob: +44 (0) 7440783885
From: Ben Watson
Sent: 07 January 2019 12:32
To: yeikel valdes ; Shashikant Bangera
Cc: user@spark.apache.org
Subject: RE: [EXTERNAL] RE: Re: Spark Kinesis Connector SSL issue
Hi,
The issue is that the KCL inside the Spark Streaming connector does not provide
a way to pass KCL configuration in, which means we can’t supply configuration
to disable SSL cert checks. In a typical (non-Spark Streaming) KCL app, we can
instantiate the KCL via:
Worker worker = new Worker.Builder()
.config(kclConfig) , //we can specify to disable SSL certs here.
.kinesisClient(kinesisClient)
.recordProcessorFactory(processorFactory)
.build();
However with the Kinesis Spark Streaming consumer, we do not have the ability
to set this. Our only interface with Kinesis is:
KinesisUtils.createStream(context, appName, streamName, serviceEndpoint,
regionName,
InitialPositionInStream.TRIM_HORIZON, checkpoint,
StorageLevel.MEMORY_AND_DISK_2())
and so we can’t pass config anywhere for the KCL to read.
Regards,
Ben
From: yeikel valdes [mailto:em...@yeikel.com]
Sent: 07 January 2019 12:21
To: Shashikant Bangera
Cc: user@spark.apache.org; Ben Watson
Subject: [EXTERNAL] RE: Re: Spark Kinesis Connector SSL issue
CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected
emails.
Any chance you can share a minimum example to replicate the issue?
On Mon, 07 Jan 2019 04:17:44 -0800 shashikantbang...@discover.com wrote
Hi Valdes,
Thank you for your response, to answer to your question. yes I can
@ben : correct me if I am wrong.
Cheers,
Shashi
Shashikant Bangera | DevOps Engineer
Payment Services DevOps Engineering
Email: shashikantbang...@discover.com<mailto:shashikantbang...@discover.com>
Group email: eppdev...@discover.com<mailto:eppdev...@discover.com>
Tel: +44 (0)
Mob: +44 (0) 7440783885
From: yeikel valdes [mailto:em...@yeikel.com<mailto:em...@yeikel.com>]
Sent: 07 January 2019 12:15
To: Shashikant Bangera
mailto:shashikantbang...@discover.com>>
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: [EXTERNAL] Re: Spark Kinesis Connector SSL issue
CAUTION EXTERNAL EMAIL
DO NOT open attachments or click on links from unknown senders or unexpected
emails.
Can you call this service with regular code(No Spark)?
On Mon, 07 Jan 2019 02:42:48 -0800
shashikantbang...@discover.com<mailto:shashikantbang...@discover.com> wrote
Hi team,
please help , we are kind of blocked here.
Cheers,
Shashi
--
Sent from:
http://apache-spark-user-list.1001560.n3.nabble.com/<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Duser-2Dlist.1001560.n3.nabble.com_=DwMFaQ=aGHIJeafFG1OvdAmHquqHk8LwduOIVr390d9RxG_Ii0=5iagzyAhXr_lW4Qru-HH2EiLU8DXPfEhqvrYe8cl44A=bruN0LP0xvrh5SboalWPOu1Iws5Ia1RONxxCEzzYQ9g=h1CVgmdCzA6yS9NqRH9XMVXrD4aeTZsKJMxauzIFDq4=>
-
To unsubscribe e-mail:
user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org>
