Re: Spark Shell issue on HDInsight

Oh ok Denny, great!!! Also, thanks for your effort in resolving my issue. can I ask one more (more open ended) question? We have a requirement where we want to read data from either Blob storage or Hive table, and upsert few records in CosmosDB. One option is run a C# activity on a windows batch

Re: Spark Shell issue on HDInsight

Sorry for the delay, you just did as I'm with the Azure CosmosDB (formerly DocumentDB) team. If you'd like to make it official, why not add an issue to the GitHub repo at https://github.com/Azure/azure-documentdb-spark/issues. HTH! On Thu, May 11, 2017 at 9:08 PM ayan guha

[PYTHON] PySpark typing hints

Hi everyone, For the last few months I've been working on static type annotations for PySpark. For those of you, who are not familiar with the idea, typing hints have been introduced by PEP 484 (https://www.python.org/dev/peps/pep-0484/) and further extended with PEP 526

Re: Spark <--> S3 flakiness

Are you running EMR? On Sun, May 14, 2017 at 4:59 AM, Miguel Morales wrote: > Some things just didn't work as i had first expected it. For example, > when writing from a spark collection to an alluxio destination didn't > persist them to s3 automatically. > > I

Cassandra Simple Insert Statement using Spark SQL Fails with org.apache.spark.sql.catalyst.parser.ParseException

I'm trying to insert data into Cassandra table with Spark SQL as follows: String query = "CREATE TEMPORARY TABLE my_table USING org.apache.spark.sql.cassandra OPTIONS (table \"my_table\",keyspace \"my_keyspace\", pushdown \"true\")"; spark.sparkSession.sql(query);

RE: Spark SQL DataFrame to Kafka Topic

Hi TD / Michael, I am trying to use the foreach sink to write to Kafka and followed this from DBricks blog by Sunil

Cassandra Simple Insert Statement using Spark SQL Fails with org.apache.spark.sql.catalyst.parser.ParseException

I'm trying to insert data into Cassandra table with Spark SQL as follows: String query = "CREATE TEMPORARY TABLE my_table USING org.apache.spark.sql.cassandra OPTIONS (table \"my_table\",keyspace \"my_keyspace\", pushdown \"true\")"; spark.sparkSession.sql(query);

Cassandra Simple Insert Statement using Spark SQL Fails with org.apache.spark.sql.catalyst.parser.ParseException

I'm trying to insert data into Cassandra table with Spark SQL as follows: String query = "CREATE TEMPORARY TABLE my_table USING org.apache.spark.sql.cassandra OPTIONS (table \"my_table\",keyspace \"my_keyspace\", pushdown \"true\")"; spark.sparkSession.sql(query);