Re: pyspark connect to spark thrift server port

2022-10-21 Thread Artemis User
I guess there are some confusions here between the metastore and the actual Hive database.  Spark (as well as Apache Hive) requires two databases for Hive DB operations.  Metastore is used for storing metadata only (e.g., schema info), whereas the actual Hive database, accessible through

[PySpark, Spark Streaming] Bug in timestamp handling in Structured Streaming?

2022-10-21 Thread kai-michael.roes...@sap.com.INVALID
Hi, I suspect I may have come across a bug in the handling of data containing timestamps in PySpark "Structured Streaming" using the foreach option. I'm "just" a user of PySpark, no Spark community member, so I don't know how to properly address the issue. I have posted a

Prometheus with spark

2022-10-21 Thread Raj ks
Hi Team, We wanted to query Prometheus data with spark. Any suggestions will be appreciated Searched for documents but did not got any prompt one