High number of tasks when ran on a hybrid cluster

Hi , I recently created a spark cluster on AWS-EMR using a fleet configuration with hybrid instance types. The instance types on this cluster vary depending on the availability of the type. While running the same spark applications that were running on homogenous cluster(some pyspark apps doing

Re: Spark program not receiving messages from Cloud Pubsub

I was able to get it working. It needed a SparkSession to be instantiated and wait for termination signal from the user. In my case I used a StreamingContext - https://spark.apache.org/docs/2.2.0/api/java/org/apache/spark/streaming/StreamingContext.html Pramod Biligiri On Sun, Aug 7, 2022 at

Re: [Spark SQL] Omit Create Table Statement in Spark Sql

you have to saveAsTable or view to make a SQL query. As the title, does Spark Sql have a feature like Flink Catalog to omit `Create Table` statement, and write sql query directly ? - To unsubscribe e-mail:

[Spark SQL] Omit Create Table Statement in Spark Sql

As the title, does Spark Sql have a feature like Flink Catalog to omit `Create Table` statement, and write sql query directly ? | | comeonyfzhu | | comeonyf...@163.com |