[ https://issues.apache.org/jira/browse/SPARK-44285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739751#comment-17739751 ]
Nihal Potdar commented on SPARK-44285: -------------------------------------- Working on it in [https://github.com/apache/spark/pull/41791] > Add support for MSK IAM to the kafka connector > ---------------------------------------------- > > Key: SPARK-44285 > URL: https://issues.apache.org/jira/browse/SPARK-44285 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming > Affects Versions: 3.4.1 > Reporter: Nihal Potdar > Priority: Major > > Managed Streaming for Kafka (MSK) is a fully managed service offered by > Amazon Web Services (AWS) that simplifies the deployment, management, and > scaling of Apache Kafka clusters. Kafka is an open-source streaming platform > widely used for building real-time data pipelines and streaming applications. > MSK provides a managed environment for running Kafka clusters, reducing the > operational overhead associated with self-managing Kafka infrastructure. > Currently when we connect to MSK using spark, users would have to install an > external library and then try to connect with MSK IAM auth. To simplify this > for users, we can add this as a dependency directly to Spark and then users > can connect to it by simply specifying the class paths directly. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org