Dear spark dev
I am trying to run IPython notebook with Kafka structured streaming
support, I couldn't find a way to load Kafka package by adding "--packages
org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0"
to PYSPARK_DRIVER_PYTHON_OPTS or even I changed my local pyspark script to
"exec "${SPARK_HOME}"/bin/spark-submit --packages
org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0 pyspark-shell-main --name
"PySparkShell" "$@"". Could anyone here point out the correct way to add
support for Kafka streaming for python notebook.
Much appreciated.
Haibo
