Re: [Spark Kafka Structured Streaming] Adding partition and topic to the kafka dynamically

2020-08-28 Thread Amit Joshi
Hi Jungtaek, Thanks for the input. I did tried and it worked. I got confused earlier after reading some blogs. Regards Amit On Friday, August 28, 2020, Jungtaek Lim wrote: > Hi Amit, > > if I remember correctly, you don't need to restart the query to reflect > the newly added topic and

Re: Connecting to Oracle Autonomous Data warehouse (ADW) from Spark via JDBC

2020-08-28 Thread Mich Talebzadeh
Ok I sorted this one out. In file $SPARK_HOME/bin/conf/spark-defaults.conf Set the parameter *spark.driver.extraClassPath* to the additional jar files that you need --> ojdbc8.jar","oraclepki.jar","osdt_cert.jar","osdt_core.jar . spark.driver.extraClassPath

Re: [Spark Kafka Structured Streaming] Adding partition and topic to the kafka dynamically

2020-08-28 Thread Gabor Somogyi
Hi Amit, The answer is no. G On Fri, Aug 28, 2020 at 9:16 AM Jungtaek Lim wrote: > Hi Amit, > > if I remember correctly, you don't need to restart the query to reflect > the newly added topic and partition, if your subscription covers the topic > (like subscribe pattern). Please try it out.

Kotlin for Apache Spark 1.0.0-preview released

2020-08-28 Thread Maria Khalusova
Hi folks! Happy to share with you that we have released the first preview of Kotlin for Apache Spark. You can learn about the features, supported versions, and roadmap in the blog post: https://blog.jetbrains.com/kotlin/2020/08/introducing-kotlin-for-apache-spark-preview/ Have a great day!

Re: [Spark Kafka Structured Streaming] Adding partition and topic to the kafka dynamically

2020-08-28 Thread Jungtaek Lim
Hi Amit, if I remember correctly, you don't need to restart the query to reflect the newly added topic and partition, if your subscription covers the topic (like subscribe pattern). Please try it out. Hope this helps. Thanks, Jungtaek Lim (HeartSaVioR) On Fri, Aug 28, 2020 at 1:56 PM Amit