[jira] [Commented] (SPARK-27589) Spark file source V2
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273161#comment-17273161 ] Chao Sun commented on SPARK-27589: -- [~xkrogen] FWIW I'm working on a POC for SPARK-32935 at the moment. There is also a design doc under working. Hopefully we'll be able to share it soon. cc [~rdblue] too. > Spark file source V2 > > > Key: SPARK-27589 > URL: https://issues.apache.org/jira/browse/SPARK-27589 > Project: Spark > Issue Type: Umbrella > Components: SQL >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Priority: Major > > Re-implement file sources with data source V2 API -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27589) Spark file source V2
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273154#comment-17273154 ] Erik Krogen commented on SPARK-27589: - [~Gengliang.Wang] are you or anyone else planning to work on SPARK-32935 or SPARK-30628? IIUC we are very close to being able to turn on V2 by default, it's a shame we are stuck due to these last two issues. > Spark file source V2 > > > Key: SPARK-27589 > URL: https://issues.apache.org/jira/browse/SPARK-27589 > Project: Spark > Issue Type: Umbrella > Components: SQL >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Priority: Major > > Re-implement file sources with data source V2 API -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27589) Spark file source V2
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198339#comment-17198339 ] Thomas Graves commented on SPARK-27589: --- thanks for confirming and filing the jira, wanted to make sure I wasn't missing something. > Spark file source V2 > > > Key: SPARK-27589 > URL: https://issues.apache.org/jira/browse/SPARK-27589 > Project: Spark > Issue Type: Umbrella > Components: SQL >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Priority: Major > > Re-implement file sources with data source V2 API -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27589) Spark file source V2
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198203#comment-17198203 ] Gengliang Wang commented on SPARK-27589: [~tgraves] I am really sorry that I missed your question. Yes bucketing is not supported yet. I have just created https://issues.apache.org/jira/browse/SPARK-32935 > Spark file source V2 > > > Key: SPARK-27589 > URL: https://issues.apache.org/jira/browse/SPARK-27589 > Project: Spark > Issue Type: Umbrella > Components: SQL >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Priority: Major > > Re-implement file sources with data source V2 API -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27589) Spark file source V2
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197669#comment-17197669 ] Thomas Graves commented on SPARK-27589: --- I'm guessing my question got missed - does it currently support bucketing or do we have a Jira for it? > Spark file source V2 > > > Key: SPARK-27589 > URL: https://issues.apache.org/jira/browse/SPARK-27589 > Project: Spark > Issue Type: Umbrella > Components: SQL >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Priority: Major > > Re-implement file sources with data source V2 API -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27589) Spark file source V2
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197377#comment-17197377 ] Gengliang Wang commented on SPARK-27589: [~dongjoon] Thanks for reminder. I will revisit this part recently and create a proper JIRA ticket for it. > Spark file source V2 > > > Key: SPARK-27589 > URL: https://issues.apache.org/jira/browse/SPARK-27589 > Project: Spark > Issue Type: Umbrella > Components: SQL >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Priority: Major > > Re-implement file sources with data source V2 API -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27589) Spark file source V2
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197228#comment-17197228 ] Dongjoon Hyun commented on SPARK-27589: --- [~Gengliang.Wang]. What is the new JIRA issue for tracking `1. Make the File source V2 writer working` because SPARK-28396 is closed as 'Won't Fix'? > Spark file source V2 > > > Key: SPARK-27589 > URL: https://issues.apache.org/jira/browse/SPARK-27589 > Project: Spark > Issue Type: Umbrella > Components: SQL >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Priority: Major > > Re-implement file sources with data source V2 API -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27589) Spark file source V2
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197143#comment-17197143 ] Thomas Graves commented on SPARK-27589: --- somewhat related, I was looking through the v2 code for parquet and I don't see anything for bucketing, is bucketing supported with the V2 api? > Spark file source V2 > > > Key: SPARK-27589 > URL: https://issues.apache.org/jira/browse/SPARK-27589 > Project: Spark > Issue Type: Umbrella > Components: SQL >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Priority: Major > > Re-implement file sources with data source V2 API -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27589) Spark file source V2
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168414#comment-17168414 ] Gengliang Wang commented on SPARK-27589: [~tgraves] we still need: 1. Make the File source V2 writer working 2. Support partition pruning with subqueries. > Spark file source V2 > > > Key: SPARK-27589 > URL: https://issues.apache.org/jira/browse/SPARK-27589 > Project: Spark > Issue Type: Umbrella > Components: SQL >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Priority: Major > > Re-implement file sources with data source V2 API -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27589) Spark file source V2
[ https://issues.apache.org/jira/browse/SPARK-27589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168133#comment-17168133 ] Thomas Graves commented on SPARK-27589: --- Hey, I see most of the sources are still in the useV1SourceList, what is left to make the v2 on by default? Is it just the remaining Jira here or other things? > Spark file source V2 > > > Key: SPARK-27589 > URL: https://issues.apache.org/jira/browse/SPARK-27589 > Project: Spark > Issue Type: Umbrella > Components: SQL >Affects Versions: 3.0.0 >Reporter: Gengliang Wang >Priority: Major > > Re-implement file sources with data source V2 API -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
