Anonymous functions cannot be found

Hi I'm trying to use sparkContext.addJar method for adding new jar files like TomCat. But in some cases, it does not work well. The error message says an Executor can not load an anonymous function. Why anonymous functions cannot be loaded in spite of adding a jar to all Executors? This is

Spark Fair Scheduler does not work correctly

Hello, I tested Spark Fair Scheduler and found that the scheduler did not work well. According to Spark doc, Fair Scheduler assigns tasks in a round-robin fashion. https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application In my understanding, if there are 2

Re: How to update structured streaming apps gracefully

pache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing, and some prototyping. Have you faced any missed events? On Mon, Dec 17, 2018 at 6:56 PM Yuta Morisawa mailto:yu-moris...@kddi-research.jp>>

How to update structured streaming apps gracefully

Hi Now I'm trying to update my structured streaming application. But I have no idea how to update it gracefully. Should I stop it, replace a jar file then restart it? In my understanding, in that case, all the state will be recovered if I use checkpoints. Is this correct? Thank you, --

Re: Continuous Processing mode behaves differently from Batch mode

.com/> <https://databricks.com/sparkaisummit/north-america> On Tue, May 15, 2018 at 5:38 PM, Yuta Morisawa <yu-moris...@kddi-research.jp <mailto:yu-moris...@kddi-research.jp>> wrote: Hi all Now I am using Structured Streaming in Continuous Processing mode and

Continuous Processing mode behaves differently from Batch mode

Hi all Now I am using Structured Streaming in Continuous Processing mode and I faced a odd problem. My code is so simple that it is similar to the sample code on the documentation. https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#continuous-processing When

Re: Spark 2.3.0 Structured Streaming Kafka Timestamp

ble = true) |-- value: binary (nullable = true) |-- topic: string (nullable = true) |-- partition: integer (nullable = true) |-- offset: long (nullable = true) |-- timestamp: timestamp (nullable = true) |-- timestampType: integer (nullable = true) Regards, Yuta On 2018/05/09 16:14, Yuta Moris

Spark 2.3.0 Structured Streaming Kafka Timestamp

Hi All I'm trying to extract Kafka-timestamp from Kafka topics. The timestamp does not contain milli-seconds information, but it should contain because ConsumerRecord class of Kafka 0.10 supports milli-second timestamp. How can I get milli-second timestamp from Kafka topics? These are