Re: share datasets across multiple spark-streaming applications for lookup

Any info on the below will be really appreciated. I read about Alluxio and Ignite. Has anybody used any of them? Do they work well with multiple Apps doing lookups simultaneously? Are there better options? Thank you. From: roshan joe Date: Monday, October 30, 2017 at

Structured Streaming Questions

I am using Structured Streaming with Spark 2.1 and have some basic questions. * Is there a way to automatically refresh the Hive Partitions when using Parquet Sink with Partition? My query looks like below val queryCount = windowedCount

RE: Spark SQL DataFrame to Kafka Topic

Thanks Michael, that worked, appreciate your help. From: Michael Armbrust [mailto:mich...@databricks.com] Sent: Monday, May 15, 2017 11:45 AM To: Revin Chalil <rcha...@expedia.com> Cc: User <user@spark.apache.org> Subject: Re: Spark SQL DataFrame to Kafka Topic The foreach sink fr

RE: Spark SQL DataFrame to Kafka Topic

I couldn’t get this working yet.. If anyone has successfully used forEach Sink for kafka with structured streaming, plz share... Thanks. From: Revin Chalil [mailto:rcha...@expedia.com] Sent: Sunday, May 14, 2017 9:32 AM To: Tathagata Das <tathagata.das1...@gmail.com>; mich...@databricks.

RE: Spark SQL DataFrame to Kafka Topic

Hi TD / Michael, I am trying to use the foreach sink to write to Kafka and followed this from DBricks blog by Sunil

RE: Reading Avro messages from Kafka using Structured Streaming in Spark 2.1

Just following up on this; would appreciate any responses on this. Thanks. From: Revin Chalil [mailto:rcha...@expedia.com] Sent: Wednesday, May 10, 2017 11:21 PM To: user@spark.apache.org Subject: Reading Avro messages from Kafka using Structured Streaming in Spark 2.1 I am trying to convert

Reading Avro messages from Kafka using Structured Streaming in Spark 2.1

I am trying to convert avro records with field type = bytes to json string using Structured Streaming in Spark 2.1. Please see below. object AvroConvert { case class KafkaMessage( payload: String ) val schemaString ="""{