Re: Spark 2.0.2 - JdbcRelationProvider does not allow create table as select

Hi all, Bumping it again! Please let me know if anyone has faced this in 2.0.x versions. I am using spark 2.0.2 for runtime. Based on the comments, I will open a bug if necessary. Thanks! On Thu, Jul 6, 2017 at 4:00 PM, Kanagha Kumar wrote: > Hi, > > I'm running spark

Re: Union of 2 streaming data frames

Great! Even, val dfAllEvents = sparkSession.table("oldEvents").union(sparkSession.table("newEvents")) doesn’t work. Will this be addressed in 2.2? From: Michael Armbrust Date: Friday, July 7, 2017 at 5:42 PM To: "Lalwani, Jayesh" Cc:

Re: Union of 2 streaming data frames

Ah, looks like you are hitting SPARK-20441 . Should be fixed in 2.2. On Fri, Jul 7, 2017 at 2:37 PM, Lalwani, Jayesh < jayesh.lalw...@capitalone.com> wrote: > I created a small sample code to verify this. It looks like union using > Spark SQL

Re: Union of 2 streaming data frames

I created a small sample code to verify this. It looks like union using Spark SQL doesn’t work. Calling union on dataframe works. https://gist.github.com/GaalDornick/8920577ca92842f44d7bfd3a277c7545. I’m on 2.1.0 I get the following exception. If I change val dfAllEvents =

Iterate over grouped df to create new rows/df

Hi everyone, I am kind of stuck in a problem and was hoping for some pointers or help :) have tried different things but couldn't achieve the desired results. I want to *create single row from multiple rows if those rows are continuous* (based on time i.e if next row's time is within 2 minutes

Re: Union of 2 streaming data frames

df.union(df2) should be supported when both DataFrames are created from a streaming source. What error are you seeing? On Fri, Jul 7, 2017 at 11:27 AM, Lalwani, Jayesh < jayesh.lalw...@capitalone.com> wrote: > In structured streaming, Is there a way to Union 2 streaming data frames? > Are there

Union of 2 streaming data frames

In structured streaming, Is there a way to Union 2 streaming data frames? Are there any plans to support Union of 2 streaming dataframes soon? I can understand the inherent complexity in joining 2 streaming data frames. But, Union is just concatenating 2 microbatches, innit? The problem that

Integrating Kafka 0.10 or higher with Spark 2.1.1 -- required jars

Hi, After finding that support for streaming api in python is not present for kafka brokers version > 0.10 . I was trying to run a kafka example in scala. I used following spark submit command : /usr/local/spark/bin/spark-submit --jars

Integrating Kafka 0.10 or higher with Spark 2.1.1 -- required jars

Hi, After finding that support for streaming api in python is not present for kafka brokers version > 0.10 . I was trying to run a kafka example in scala. I used following spark submit command : /usr/local/spark/bin/spark-submit --jars /home/mahendra/spark- >

Integrating Kafka 0.10 or higher with Spark 2.1.1 -- required jars

Hi, After finding that support for streaming api in python is not present for kafka brokers version > 0.10 . I was trying to run a kafka example in scala. I used following spark submit command : /usr/local/spark/bin/spark-submit --jars >

Re: Using Spark as a simulator

On 7 Jul 2017, at 08:37, Esa Heikkinen > wrote: I only want to simulate very huge "network" with even millions parallel time syncronized actors (state machines). There are also communication between actors via some (key-value

Unsubscribe

VS: VS: Using Spark as a simulator

I only want to simulate very huge "network" with even millions parallel time syncronized actors (state machines). There are also communication between actors via some (key-value pairs) database. I also want the simulation should work in the real time. I don't know what would be the best

Re: VS: Using Spark as a simulator

Spark dropped Akka some time ago... I think the main issue he will face is a library for simulating the state machines (randomly), storing a huge amount of files (HDFS is probably the way to go if you want it fast) and distributing the work (here you can select different options). Are you

VS: Using Spark as a simulator

Would it be better to use Akka as simulator rather than Spark ? http://akka.io/ Akka akka.io Build powerful reactive, concurrent & distributed applications more easily. Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient ... The spark