[
https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380170#comment-17380170
]
Shardul Mahadik commented on SPARK-28266:
-
I would like to propose another angle to look at the
[
https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380093#comment-17380093
]
Apache Spark commented on SPARK-28266:
--
User 'shardulm94' has created a pull request for this
[
https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014681#comment-17014681
]
Dongjoon Hyun commented on SPARK-28266:
---
Oh, I see. Thank you for reply.
> data duplication when
[
https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014467#comment-17014467
]
Ruslan Dautkhanov commented on SPARK-28266:
---
Thank you for checking [~dongjoon]
That may have
[
https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014053#comment-17014053
]
Dongjoon Hyun commented on SPARK-28266:
---
For 2.3.4 and 2.2.3, it's the same. I cannot reproduce
[
https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014048#comment-17014048
]
Dongjoon Hyun commented on SPARK-28266:
---
In Apache Spark 2.4.4, I also cannot reproduce this
[
https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014046#comment-17014046
]
Dongjoon Hyun commented on SPARK-28266:
---
BTW, SPARK-22158 is irrelevant to this one because it
[
https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014042#comment-17014042
]
Dongjoon Hyun commented on SPARK-28266:
---
So, the observation was that `InMemoryFileIndex` has
[
https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014041#comment-17014041
]
Dongjoon Hyun commented on SPARK-28266:
---
Hi, [~Tagar].
For now, I cannot reproduce this in
