Hi, The issue is that the KCL inside the Spark Streaming connector does not provide a way to pass KCL configuration in, which means we can’t supply configuration to disable SSL cert checks. In a typical (non-Spark Streaming) KCL app, we can instantiate the KCL via:
Worker worker = new Worker.Builder() .config(kclConfig) , //we can specify to disable SSL certs here. .kinesisClient(kinesisClient) .recordProcessorFactory(processorFactory) .build(); However with the Kinesis Spark Streaming consumer, we do not have the ability to set this. Our only interface with Kinesis is: KinesisUtils.createStream(context, appName, streamName, serviceEndpoint, regionName, InitialPositionInStream.TRIM_HORIZON, checkpoint, StorageLevel.MEMORY_AND_DISK_2()) and so we can’t pass config anywhere for the KCL to read. Regards, Ben Shashikant Bangera | DevOps Engineer Payment Services DevOps Engineering Email: shashikantbang...@discover.com<mailto:shashikantbang...@discover.com> Group email: eppdev...@discover.com<mailto:eppdev...@discover.com> Tel: +44 (0) Mob: +44 (0) 7440783885 From: Ben Watson Sent: 07 January 2019 12:32 To: yeikel valdes <em...@yeikel.com>; Shashikant Bangera <shashikantbang...@discover.com> Cc: user@spark.apache.org Subject: RE: [EXTERNAL] RE: Re: Spark Kinesis Connector SSL issue Hi, The issue is that the KCL inside the Spark Streaming connector does not provide a way to pass KCL configuration in, which means we can’t supply configuration to disable SSL cert checks. In a typical (non-Spark Streaming) KCL app, we can instantiate the KCL via: Worker worker = new Worker.Builder() .config(kclConfig) , //we can specify to disable SSL certs here. .kinesisClient(kinesisClient) .recordProcessorFactory(processorFactory) .build(); However with the Kinesis Spark Streaming consumer, we do not have the ability to set this. Our only interface with Kinesis is: KinesisUtils.createStream(context, appName, streamName, serviceEndpoint, regionName, InitialPositionInStream.TRIM_HORIZON, checkpoint, StorageLevel.MEMORY_AND_DISK_2()) and so we can’t pass config anywhere for the KCL to read. Regards, Ben From: yeikel valdes [mailto:em...@yeikel.com] Sent: 07 January 2019 12:21 To: Shashikant Bangera <shashikantbang...@discover.com> Cc: user@spark.apache.org; Ben Watson <benwat...@discover.com> Subject: [EXTERNAL] RE: Re: Spark Kinesis Connector SSL issue CAUTION EXTERNAL EMAIL DO NOT open attachments or click on links from unknown senders or unexpected emails. Any chance you can share a minimum example to replicate the issue? ---- On Mon, 07 Jan 2019 04:17:44 -0800 shashikantbang...@discover.com wrote ---- Hi Valdes, Thank you for your response, to answer to your question. yes I can @ben : correct me if I am wrong. Cheers, Shashi Shashikant Bangera | DevOps Engineer Payment Services DevOps Engineering Email: shashikantbang...@discover.com<mailto:shashikantbang...@discover.com> Group email: eppdev...@discover.com<mailto:eppdev...@discover.com> Tel: +44 (0) Mob: +44 (0) 7440783885 From: yeikel valdes [mailto:em...@yeikel.com<mailto:em...@yeikel.com>] Sent: 07 January 2019 12:15 To: Shashikant Bangera <shashikantbang...@discover.com<mailto:shashikantbang...@discover.com>> Cc: user@spark.apache.org<mailto:user@spark.apache.org> Subject: [EXTERNAL] Re: Spark Kinesis Connector SSL issue CAUTION EXTERNAL EMAIL DO NOT open attachments or click on links from unknown senders or unexpected emails. Can you call this service with regular code(No Spark)? ---- On Mon, 07 Jan 2019 02:42:48 -0800 shashikantbang...@discover.com<mailto:shashikantbang...@discover.com> wrote ---- Hi team, please help , we are kind of blocked here. Cheers, Shashi -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/<https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Duser-2Dlist.1001560.n3.nabble.com_&d=DwMFaQ&c=aGHIJeafFG1OvdAmHquqHk8LwduOIVr390d9RxG_Ii0&r=5iagzyAhXr_lW4Qru-HH2EiLU8DXPfEhqvrYe8cl44A&m=bruN0LP0xvrh5SboalWPOu1Iws5Ia1RONxxCEzzYQ9g&s=h1CVgmdCzA6yS9NqRH9XMVXrD4aeTZsKJMxauzIFDq4&e=> --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org>