Filipe Souza created SPARK-40430:
------------------------------------
Summary: Spark session does not update number of files for
partition
Key: SPARK-40430
URL: https://issues.apache.org/jira/browse/SPARK-40430
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.1.2
Environment: I'm using spark 3.1.2 on AWS EMR and AWS Glue as catalog.
Reporter: Filipe Souza
When a spark session has already queried data from a table and partition and
new files are inserted into the partition externally, the spark session keeps
the outdated number of files and does not return the new records.
If the data is inserted into a new partition, the problem will not occur.
Steps to reproduce the behavior:
Open a Spark session
Query a count in a table
Open another spark session
insert data into an existing partition
Check the count again in the first session
I expect to see the inserted records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
