Hi,
I have a spark streaming application running on yarn that consumes from a jms
source. I have the checkpointing and WAL enabled to ensure zero data loss.
However, When I suddenly kill my application and restarts it, sometimes it
recovers the data from the WAL but sometimes it doesn’t !! In a
bal Temporary View. Temporary views in Spark SQL are session-scoped and
> will disappear if the session that creates it terminates. If you want to have
> a temporary view that is shared among all sessions and keep alive until the
> Spark application terminates, you can create a global tem
Hi,
I have a parquet on S3 partitioned by day. I have 2 years of data (-> about
1000 partitions). With spark, when I just want to know the schema of this
parquet without even asking for a single row of data, spark tries to list
all the partitions and the nested partitions of the parquet. Which mak
gt;> On Sat, Apr 2, 2016 at 6:48 AM, Walid Lezzar wrote:
>> Hi,
>>
>> I looked into the spark code at how spark report metrics using the
>> MetricsSystem class. I've seen that the spark MetricsSystem class when
>> instantiated parses the metrics.properties
Hi,
I looked into the spark code at how spark report metrics using the
MetricsSystem class. I've seen that the spark MetricsSystem class when
instantiated parses the metrics.properties file, tries to find the sinks class
name and load them dinamically. It would be great to implement my own sink
Hi,
I'm running a Spark Streaming job that pulls data from Kafka (using the
direct approach method - without receiver) and pushes it into
elasticsearch. The job is running fine but I was suprised once I opened
jconsole to monitor it : I noticed that the heap memory is constantly
increasing until t
Hi,
We have been using spark streaming for a little while now.
Until now, we were running our spark streaming jobs in spark 1.5.1 and it
was working well. Yesterday, we upgraded to spark 1.6.0 without any changes
in the code. But our streaming jobs are not working any more. We are
getting an "Abs
