Re: Spark ACID compatibility

I think we are hitting an old bug. tried it with Hadoop 3.1.1 Hive 3.1.1 Spark 3.1.1 Try to create an ORC transactional table in Hive (PySpark) CREATE TABLE if not exists test.randomDataDelta( ID INT , CLUSTERED INT , SCATTERED INT , RANDOMISED INT , RANDOM_STRING

Re: Spark ACID compatibility

No this also does not work. Steps I followed. spark-sql: CREATE TABLE students (id int, name string, marks int) STORED AS ORC TBLPROPERTIES ('transactional' = 'true'); hive-cli: created a students_copy table and inserted some values in it and did "INSERT OVERWRITE TABLE students select * from

Re: Spark ACID compatibility

Ok there were issues in the past with the ORC table read through Spark. If the ORC table is created through Spark I believe it will work Do a test. Create the ORC table through Spark first. Then do insert overwrite into that table through Hive cli from your Hive created ORC table and see if you

Re: Spark ACID compatibility

Table was created by hive (hive-cli) , format is orc. I am able to get data from hive-cli (hive return rows). But spark-sql/spark-shell does not return any rows. On Mon, 14 Jun 2021 at 19:26, Mich Talebzadeh wrote: > How the table was created in the first place, spark or Hive? > > Is this table

Re: Spark ACID compatibility

How the table was created in the first place, spark or Hive? Is this table an ORC table and does Spark or Hive return rows? HTH view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any

Spark ACID compatibility

Hi Does spark support querying hive tables which are transactional? I am using spark 3.0.2 / hive metastore 3.1.2 and trying to query the table but I am not able to see the data from the table , although *show tables *does list the table from hive metastore and desc table works fine but *select *

Fwd: CRAN package SparkR

It looks like they would not allow caching the Spark Distribution. I’m not sure what can be done about this. If I recall, the package should remove this during test. Or maybe spark.install() ie optional (hence getting user confirmation?) -- Forwarded message - Date: Sun, Jun