[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py

2020-10-19 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217339#comment-17217339 ] Maciej Szymkiewicz commented on SPARK-32084: [~hyukjin.kwon] I can work on t

[jira] [Assigned] (SPARK-33190) Set upperbound of PyArrow version in GitHub Actions

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33190: Assignee: Apache Spark > Set upperbound of PyArrow version in GitHub Actions > --

[jira] [Assigned] (SPARK-33190) Set upperbound of PyArrow version in GitHub Actions

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33190: Assignee: (was: Apache Spark) > Set upperbound of PyArrow version in GitHub Actions >

[jira] [Assigned] (SPARK-33190) Set upperbound of PyArrow version in GitHub Actions

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33190: Assignee: Apache Spark > Set upperbound of PyArrow version in GitHub Actions > --

[jira] [Commented] (SPARK-33190) Set upperbound of PyArrow version in GitHub Actions

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217328#comment-17217328 ] Apache Spark commented on SPARK-33190: -- User 'HyukjinKwon' has created a pull reque

[jira] [Created] (SPARK-33190) Set upperbound of PyArrow version in GitHub Actions

2020-10-19 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-33190: Summary: Set upperbound of PyArrow version in GitHub Actions Key: SPARK-33190 URL: https://issues.apache.org/jira/browse/SPARK-33190 Project: Spark Issue Typ

[jira] [Updated] (SPARK-33189) Support PyArrow 2.0.0+

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-33189: - Affects Version/s: (was: 3.0.1) 3.1.0 > Support PyArrow 2.0.0+ >

[jira] [Created] (SPARK-33189) Support PyArrow 2.0.0+

2020-10-19 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-33189: Summary: Support PyArrow 2.0.0+ Key: SPARK-33189 URL: https://issues.apache.org/jira/browse/SPARK-33189 Project: Spark Issue Type: Improvement Comp

[jira] [Resolved] (SPARK-32229) Application entry parsing fails because DriverWrapper registered instead of the normal driver

2020-10-19 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32229. -- Fix Version/s: 3.1.0 Assignee: Gabor Somogyi Resolution: Fixed Resolve

[jira] [Assigned] (SPARK-33160) Allow saving/loading INT96 in parquet w/o rebasing

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-33160: Assignee: Maxim Gekk > Allow saving/loading INT96 in parquet w/o rebasing > -

[jira] [Resolved] (SPARK-33160) Allow saving/loading INT96 in parquet w/o rebasing

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33160. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30056 [https://gi

[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217263#comment-17217263 ] Hyukjin Kwon commented on SPARK-32084: -- [~zero323] are you interested in this? > R

[jira] [Resolved] (SPARK-33182) Using Delegation Tokens to access HBase through Spark (Java)

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33182. -- Resolution: Incomplete > Using Delegation Tokens to access HBase through Spark (Java) > --

[jira] [Updated] (SPARK-33184) spark doesn't read data source column if it is used as an index to an array under a struct

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-33184: - Component/s: (was: Spark Core) SQL > spark doesn't read data source column

[jira] [Commented] (SPARK-33182) Using Delegation Tokens to access HBase through Spark (Java)

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217262#comment-17217262 ] Hyukjin Kwon commented on SPARK-33182: -- Spark 2.3.x is EOL. Mind testing in higher

[jira] [Updated] (SPARK-33188) PipelineModel load resulting in error

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-33188: - Component/s: ML > PipelineModel load resulting in error > -

[jira] [Assigned] (SPARK-33140) make Analyzer rules using SQLConf.get

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33140: Assignee: Apache Spark > make Analyzer rules using SQLConf.get >

[jira] [Commented] (SPARK-33140) make Analyzer rules using SQLConf.get

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217257#comment-17217257 ] Apache Spark commented on SPARK-33140: -- User 'leanken' has created a pull request f

[jira] [Assigned] (SPARK-33140) make Analyzer rules using SQLConf.get

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33140: Assignee: (was: Apache Spark) > make Analyzer rules using SQLConf.get > -

[jira] [Created] (SPARK-33188) PipelineModel load resulting in error

2020-10-19 Thread rohit agarwal (Jira)
rohit agarwal created SPARK-33188: - Summary: PipelineModel load resulting in error Key: SPARK-33188 URL: https://issues.apache.org/jira/browse/SPARK-33188 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-33187) Add a check on the number of returned partitions in the HiveShim#getPartitionsByFilter method

2020-10-19 Thread jinhai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinhai updated SPARK-33187: --- Description: In the method HiveShim#getPartitionsByFilter, when filter is empty or when the hive table has

[jira] [Assigned] (SPARK-32351) Partially pushed partition filters are not explained

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32351: Assignee: Nan Zhu (was: pavithra ramachandran) > Partially pushed partition filters are

[jira] [Commented] (SPARK-32351) Partially pushed partition filters are not explained

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217246#comment-17217246 ] Hyukjin Kwon commented on SPARK-32351: -- Oops, sure. > Partially pushed partition f

[jira] [Created] (SPARK-33187) Add a check on the number of returned partitions in the HiveShim#getPartitionsByFilter method

2020-10-19 Thread jinhai (Jira)
jinhai created SPARK-33187: -- Summary: Add a check on the number of returned partitions in the HiveShim#getPartitionsByFilter method Key: SPARK-33187 URL: https://issues.apache.org/jira/browse/SPARK-33187 Pro

[jira] [Updated] (SPARK-33186) Respect ANSI standard and change FIRST_VALUE as pure window function.

2020-10-19 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-33186: --- Description: FIRST_VALUE is an ANSI window function. But the current implement of FIRST_VALUE in Spa

[jira] [Commented] (SPARK-33186) Respect ANSI standard and change FIRST_VALUE as pure window function.

2020-10-19 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217240#comment-17217240 ] jiaan.geng commented on SPARK-33186: I'm working on. > Respect ANSI standard and ch

[jira] [Created] (SPARK-33186) Respect ANSI standard and change FIRST_VALUE as pure window function.

2020-10-19 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-33186: -- Summary: Respect ANSI standard and change FIRST_VALUE as pure window function. Key: SPARK-33186 URL: https://issues.apache.org/jira/browse/SPARK-33186 Project: Spark

[jira] [Commented] (SPARK-32351) Partially pushed partition filters are not explained

2020-10-19 Thread Nan Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217238#comment-17217238 ] Nan Zhu commented on SPARK-32351: - [~hyukjin.kwon] nit: could you reassign this to me? [

[jira] [Updated] (SPARK-32863) Full outer stream-stream join

2020-10-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-32863: - Priority: Major (was: Trivial) > Full outer stream-stream join > -

[jira] [Updated] (SPARK-32883) Stream-stream join improvement

2020-10-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-32883: - Priority: Major (was: Minor) > Stream-stream join improvement > --

[jira] [Updated] (SPARK-32862) Left semi stream-stream join

2020-10-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-32862: - Priority: Major (was: Minor) > Left semi stream-stream join > > >

[jira] [Assigned] (SPARK-32351) Partially pushed partition filters are not explained

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32351: Assignee: pavithra ramachandran > Partially pushed partition filters are not explained >

[jira] [Resolved] (SPARK-32351) Partially pushed partition filters are not explained

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32351. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29831 [https://gi

[jira] [Resolved] (SPARK-33181) SQL Reference: Run SQL on files directly

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33181. -- Fix Version/s: 3.0.2 3.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-33181) SQL Reference: Run SQL on files directly

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-33181: Assignee: Aoyuan Liao > SQL Reference: Run SQL on files directly > --

[jira] [Commented] (SPARK-31760) Simplification Based on Containment

2020-10-19 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217180#comment-17217180 ] Aoyuan Liao commented on SPARK-31760: - [~yumwang] To implement the containment rule:

[jira] [Updated] (SPARK-31760) Simplification Based on Containment

2020-10-19 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aoyuan Liao updated SPARK-31760: Attachment: image-2020-10-19-17-18-31-849.png > Simplification Based on Containment >

[jira] [Commented] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217169#comment-17217169 ] Apache Spark commented on SPARK-33185: -- User 'xkrogen' has created a pull request f

[jira] [Assigned] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33185: Assignee: (was: Apache Spark) > YARN: Print direct links to driver logs alongside app

[jira] [Commented] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217167#comment-17217167 ] Apache Spark commented on SPARK-33185: -- User 'xkrogen' has created a pull request f

[jira] [Assigned] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33185: Assignee: Apache Spark > YARN: Print direct links to driver logs alongside application re

[jira] [Updated] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

2020-10-19 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-33185: Description: Currently when run in {{cluster}} mode on YARN, the Spark {{yarn.Client}} will print

[jira] [Created] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode

2020-10-19 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-33185: --- Summary: YARN: Print direct links to driver logs alongside application report in cluster mode Key: SPARK-33185 URL: https://issues.apache.org/jira/browse/SPARK-33185 Pr

[jira] [Resolved] (SPARK-33069) Skip test result report if no JUnit XML files are found

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33069. -- Resolution: Not A Problem > Skip test result report if no JUnit XML files are found >

[jira] [Assigned] (SPARK-33181) SQL Reference: Run SQL on files directly

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33181: Assignee: Apache Spark > SQL Reference: Run SQL on files directly > -

[jira] [Assigned] (SPARK-33181) SQL Reference: Run SQL on files directly

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33181: Assignee: (was: Apache Spark) > SQL Reference: Run SQL on files directly > --

[jira] [Commented] (SPARK-33003) Add type hints guideliness to the documentation

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217137#comment-17217137 ] Apache Spark commented on SPARK-33003: -- User 'zero323' has created a pull request f

[jira] [Assigned] (SPARK-33003) Add type hints guideliness to the documentation

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33003: Assignee: (was: Apache Spark) > Add type hints guideliness to the documentation > ---

[jira] [Commented] (SPARK-33181) SQL Reference: Run SQL on files directly

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217139#comment-17217139 ] Apache Spark commented on SPARK-33181: -- User 'liaoaoyuan97' has created a pull requ

[jira] [Assigned] (SPARK-33003) Add type hints guideliness to the documentation

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33003: Assignee: Apache Spark > Add type hints guideliness to the documentation > --

[jira] [Commented] (SPARK-33044) Add a Jenkins build and test job for Scala 2.13

2020-10-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217075#comment-17217075 ] Dongjoon Hyun commented on SPARK-33044: --- Gentle ping, [~shaneknapp]. Could you set

[jira] [Commented] (SPARK-33146) Encountering an invalid rolling event log folder prevents loading other applications in SHS

2020-10-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217064#comment-17217064 ] Dongjoon Hyun commented on SPARK-33146: --- Hi, [~kabhwan] I don't think the revert

[jira] [Assigned] (SPARK-17333) Make pyspark interface friendly with mypy static analysis

2020-10-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-17333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-17333: - Assignee: Fokko Driesprong > Make pyspark interface friendly with mypy static analysis

[jira] [Resolved] (SPARK-17333) Make pyspark interface friendly with mypy static analysis

2020-10-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-17333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-17333. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30088 [https://

[jira] [Commented] (SPARK-33181) SQL Reference: Run SQL on files directly

2020-10-19 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217023#comment-17217023 ] Aoyuan Liao commented on SPARK-33181: - [~smilegator] If this one hasn't been done, c

[jira] [Commented] (SPARK-33183) Bug in optimizer rule EliminateSorts

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217008#comment-17217008 ] Apache Spark commented on SPARK-33183: -- User 'allisonwang-db' has created a pull re

[jira] [Assigned] (SPARK-33183) Bug in optimizer rule EliminateSorts

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33183: Assignee: Apache Spark > Bug in optimizer rule EliminateSorts > -

[jira] [Assigned] (SPARK-33183) Bug in optimizer rule EliminateSorts

2020-10-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33183: Assignee: (was: Apache Spark) > Bug in optimizer rule EliminateSorts > --

[jira] [Commented] (SPARK-33184) spark doesn't read data source column if it is used as an index to an array under a struct

2020-10-19 Thread colin fang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217001#comment-17217001 ] colin fang commented on SPARK-33184: I notice there is a quotation mark before `Proj

[jira] [Updated] (SPARK-33184) spark doesn't read data source column if it is used as an index to an array under a struct

2020-10-19 Thread colin fang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] colin fang updated SPARK-33184: --- Issue Type: Bug (was: Improvement) > spark doesn't read data source column if it is used as an inde

[jira] [Updated] (SPARK-33184) spark doesn't read data source column if it is used as an index to an array under a struct

2020-10-19 Thread colin fang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] colin fang updated SPARK-33184: --- Summary: spark doesn't read data source column if it is used as an index to an array under a struct

[jira] [Updated] (SPARK-33184) spark doesn't read data source column if it is needed as an index to an array in a nested struct

2020-10-19 Thread colin fang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] colin fang updated SPARK-33184: --- Description: {code:python} df = spark.createDataFrame([[1, [[1, 2, schema='x:int,y:struct>') df

[jira] [Updated] (SPARK-33184) spark doesn't read data source column if it is needed as an index to an array in a nested struct

2020-10-19 Thread colin fang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] colin fang updated SPARK-33184: --- Description: {code:python} df = spark.createDataFrame([[1, [[1, 2, schema='x:int,y:struct>') df

[jira] [Created] (SPARK-33184) spark doesn't read data source column if it is needed as an index to an array in a nested struct

2020-10-19 Thread colin fang (Jira)
colin fang created SPARK-33184: -- Summary: spark doesn't read data source column if it is needed as an index to an array in a nested struct Key: SPARK-33184 URL: https://issues.apache.org/jira/browse/SPARK-33184

[jira] [Created] (SPARK-33183) Bug in optimizer rule EliminateSorts

2020-10-19 Thread Allison Wang (Jira)
Allison Wang created SPARK-33183: Summary: Bug in optimizer rule EliminateSorts Key: SPARK-33183 URL: https://issues.apache.org/jira/browse/SPARK-33183 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-33069) Skip test result report if no JUnit XML files are found

2020-10-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33069: -- Fix Version/s: (was: 3.0.2) (was: 2.4.8) (was

[jira] [Reopened] (SPARK-33069) Skip test result report if no JUnit XML files are found

2020-10-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-33069: --- Assignee: (was: Hyukjin Kwon) > Skip test result report if no JUnit XML files are foun

[jira] [Resolved] (SPARK-32941) Optimize WithFields expression chain

2020-10-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32941. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29812 [https://

[jira] [Updated] (SPARK-33182) Using Delegation Tokens to access HBase through Spark (Java)

2020-10-19 Thread Rishi S Balajis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rishi S Balajis updated SPARK-33182: Description: I have a requirement to access a kerberized HBase through delegation tokens

[jira] [Updated] (SPARK-33182) Using Delegation Tokens to access HBase through Spark (Java)

2020-10-19 Thread Rishi S Balajis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rishi S Balajis updated SPARK-33182: Description: I have a requirement to access a kerberized HBase through delegation tokens

[jira] [Updated] (SPARK-33182) Using Delegation Tokens to access HBase through Spark (Java)

2020-10-19 Thread Rishi S Balajis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rishi S Balajis updated SPARK-33182: Affects Version/s: (was: 2.4.0) 2.3.0 > Using Delegation Tokens

[jira] [Created] (SPARK-33182) Using Delegation Tokens to access HBase through Spark (Java)

2020-10-19 Thread Rishi S Balajis (Jira)
Rishi S Balajis created SPARK-33182: --- Summary: Using Delegation Tokens to access HBase through Spark (Java) Key: SPARK-33182 URL: https://issues.apache.org/jira/browse/SPARK-33182 Project: Spark

[jira] [Commented] (SPARK-24266) Spark client terminates while driver is still running

2020-10-19 Thread Martin Tapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216759#comment-17216759 ] Martin Tapp commented on SPARK-24266: - Can we update Fix version to 3.0.2 then? Than

[jira] [Commented] (SPARK-24266) Spark client terminates while driver is still running

2020-10-19 Thread Nicola Bova (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216749#comment-17216749 ] Nicola Bova commented on SPARK-24266: - It would be great if the fix could be target

[jira] [Commented] (SPARK-33168) spark REST API Unable to get JobDescription

2020-10-19 Thread zhaoyachao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216712#comment-17216712 ] zhaoyachao commented on SPARK-33168: I don't know what I did wrong, this result is n

[jira] [Updated] (SPARK-33168) spark REST API Unable to get JobDescription

2020-10-19 Thread zhaoyachao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyachao updated SPARK-33168: --- Attachment: (was: image-2020-10-19-21-08-45-123.png) > spark REST API Unable to get JobDescripti

[jira] [Updated] (SPARK-33168) spark REST API Unable to get JobDescription

2020-10-19 Thread zhaoyachao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyachao updated SPARK-33168: --- Attachment: (was: image-2020-10-19-21-07-01-514.png) > spark REST API Unable to get JobDescripti

[jira] [Updated] (SPARK-33168) spark REST API Unable to get JobDescription

2020-10-19 Thread zhaoyachao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyachao updated SPARK-33168: --- Attachment: image-2020-10-19-21-13-09-960.png > spark REST API Unable to get JobDescription > --

[jira] [Updated] (SPARK-33168) spark REST API Unable to get JobDescription

2020-10-19 Thread zhaoyachao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyachao updated SPARK-33168: --- Attachment: image-2020-10-19-21-12-41-145.png > spark REST API Unable to get JobDescription > --

[jira] [Updated] (SPARK-33168) spark REST API Unable to get JobDescription

2020-10-19 Thread zhaoyachao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyachao updated SPARK-33168: --- Attachment: image-2020-10-19-21-08-45-123.png > spark REST API Unable to get JobDescription > --

[jira] [Updated] (SPARK-33168) spark REST API Unable to get JobDescription

2020-10-19 Thread zhaoyachao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyachao updated SPARK-33168: --- Attachment: image-2020-10-19-21-07-01-514.png > spark REST API Unable to get JobDescription > --

[jira] [Resolved] (SPARK-33169) Check propagation of datasource options to underlying file system for built-in file-based datasources

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33169. -- Fix Version/s: 3.1.0 Resolution: Fixed Fixed in https://github.com/apache/spark/pull/30

[jira] [Commented] (SPARK-33128) mismatched input since Spark 3.0

2020-10-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216539#comment-17216539 ] Yuming Wang commented on SPARK-33128: - Thank you [~LuciferYang]. > mismatched input

[jira] [Resolved] (SPARK-33069) Skip test result report if no JUnit XML files are found

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33069. -- Resolution: Invalid We don't need this fix anymore. Now it fails to download when there are no

[jira] [Reopened] (SPARK-33069) Skip test result report if no JUnit XML files are found

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-33069: -- Reverted at https://github.com/apache/spark/commit/a7a8dae4836f455a26ba6cb3c7d733775b6af0f6 > Sk

[jira] [Resolved] (SPARK-33180) Enables 'fail_if_no_tests' when reporting test results

2020-10-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33180. -- Resolution: Invalid Now it aborts at downloading when there are no artifacts to download, see

[jira] [Created] (SPARK-33181) SQL Reference: Run SQL on files directly

2020-10-19 Thread Xiao Li (Jira)
Xiao Li created SPARK-33181: --- Summary: SQL Reference: Run SQL on files directly Key: SPARK-33181 URL: https://issues.apache.org/jira/browse/SPARK-33181 Project: Spark Issue Type: Documentation

[jira] [Updated] (SPARK-33181) SQL Reference: Run SQL on files directly

2020-10-19 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-33181: Labels: starter (was: ) > SQL Reference: Run SQL on files directly >