RushabhK commented on PR #9844:
URL:
https://github.com/apache/incubator-gluten/pull/9844#issuecomment-2948628851
> > @RushabhK I followed the steps you provided using the main branch code,
and the test code is as follows:
> > ```
> > val store_sales =
spark.read.format("parquet").load("/user/root/tpcds_parquet_partition__100/store_sales")
> >
> >
store_sales.write.mode("append").format("parquet").save("/user/root/parquet-write")
> >
> > val velox_store_sales =
spark.read.format("parquet").load("/user/root/parquet-write/")
> > velox_store_sales.count()
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Here is the Spark history UI for reference.
> >
> > However, I was unable to reproduce the issue you mentioned. Am I missing
something?
>
> @JkSelf I am running this on Ubuntu, what OS are you running it on? Can
you share the setup for your spark image? The setup looks right, although I am
applying this path on v1.3.0 branch and not on main.. Are there any fixes on
main branch related to this?
Also @JkSelf I am writing the parquet in overwrite mode. I am using the
following code for write:
```
df.coalesce(300).write.partitionBy("date_key", "hour").
format("parquet").mode("overwrite").option("compression", "zstd").
option("partitionOverwriteMode", "dynamic").
save("table_name")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
