[jira] [Commented] (SPARK-24554) Add MapType Support for Arrow in PySpark

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233307#comment-17233307 ] Apache Spark commented on SPARK-24554: -- User 'BryanCutler' has created a pull request for this

[jira] [Assigned] (SPARK-24554) Add MapType Support for Arrow in PySpark

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24554: Assignee: Apache Spark > Add MapType Support for Arrow in PySpark >

[jira] [Assigned] (SPARK-24554) Add MapType Support for Arrow in PySpark

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24554: Assignee: (was: Apache Spark) > Add MapType Support for Arrow in PySpark >

[jira] [Created] (SPARK-33465) RDD.takeOrdered should get rid of usage of reduce or use treeReduce instead

2020-11-16 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-33465: --- Summary: RDD.takeOrdered should get rid of usage of reduce or use treeReduce instead Key: SPARK-33465 URL: https://issues.apache.org/jira/browse/SPARK-33465 Project:

[jira] [Assigned] (SPARK-33465) RDD.takeOrdered should get rid of usage of reduce or use treeReduce instead

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33465: Assignee: L. C. Hsieh (was: Apache Spark) > RDD.takeOrdered should get rid of usage of

[jira] [Commented] (SPARK-33465) RDD.takeOrdered should get rid of usage of reduce or use treeReduce instead

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233285#comment-17233285 ] Apache Spark commented on SPARK-33465: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33465) RDD.takeOrdered should get rid of usage of reduce or use treeReduce instead

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33465: Assignee: Apache Spark (was: L. C. Hsieh) > RDD.takeOrdered should get rid of usage of

[jira] [Commented] (SPARK-33465) RDD.takeOrdered should get rid of usage of reduce or use treeReduce instead

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233284#comment-17233284 ] Apache Spark commented on SPARK-33465: -- User 'viirya' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-33395) Spark reading data in scientific notation

2020-11-16 Thread Nilesh Patil (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233279#comment-17233279 ] Nilesh Patil edited comment on SPARK-33395 at 11/17/20, 5:57 AM: - In

[jira] [Commented] (SPARK-33395) Spark reading data in scientific notation

2020-11-16 Thread Nilesh Patil (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233279#comment-17233279 ] Nilesh Patil commented on SPARK-33395: -- In decimal type scale and precision will be constant for

[jira] [Commented] (SPARK-33395) Spark reading data in scientific notation

2020-11-16 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233276#comment-17233276 ] Takeshi Yamamuro commented on SPARK-33395: -- How about using a decimal type instead?  > Spark

[jira] [Commented] (SPARK-33395) Spark reading data in scientific notation

2020-11-16 Thread Nilesh Patil (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233271#comment-17233271 ] Nilesh Patil commented on SPARK-33395: -- Can we have any method implementation which will persist

[jira] [Updated] (SPARK-33395) Spark reading data in scientific notation

2020-11-16 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-33395: - Issue Type: Improvement (was: Bug) > Spark reading data in scientific notation >

[jira] [Commented] (SPARK-33395) Spark reading data in scientific notation

2020-11-16 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233267#comment-17233267 ] Takeshi Yamamuro commented on SPARK-33395: -- hm, but we cannot avoid the rounding in this case,

[jira] [Updated] (SPARK-33395) Spark reading data in scientific notation

2020-11-16 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-33395: - Priority: Major (was: Critical) > Spark reading data in scientific notation >

[jira] [Commented] (SPARK-33395) Spark reading data in scientific notation

2020-11-16 Thread Nilesh Patil (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233264#comment-17233264 ] Nilesh Patil commented on SPARK-33395: -- Yes inferred type is double, but we want the data as it is

[jira] [Updated] (SPARK-33395) Spark reading data in scientific notation

2020-11-16 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-33395: - Affects Version/s: 3.1.0 2.4.8 > Spark reading data in

[jira] [Updated] (SPARK-33395) Spark reading data in scientific notation

2020-11-16 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-33395: - Component/s: (was: Spark Core) SQL > Spark reading data in

[jira] [Commented] (SPARK-33395) Spark reading data in scientific notation

2020-11-16 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233263#comment-17233263 ] Takeshi Yamamuro commented on SPARK-33395: -- The inferred type is double, so they are

[jira] [Resolved] (SPARK-33407) Simplify the exception message from Python UDFs

2020-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33407. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30309

[jira] [Assigned] (SPARK-33407) Simplify the exception message from Python UDFs

2020-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-33407: Assignee: Hyukjin Kwon > Simplify the exception message from Python UDFs >

[jira] [Assigned] (SPARK-33464) Add/remove (un)necessary cache and restructure GitHub Actions yaml

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33464: Assignee: (was: Apache Spark) > Add/remove (un)necessary cache and restructure

[jira] [Commented] (SPARK-33464) Add/remove (un)necessary cache and restructure GitHub Actions yaml

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233254#comment-17233254 ] Apache Spark commented on SPARK-33464: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-33464) Add/remove (un)necessary cache and restructure GitHub Actions yaml

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33464: Assignee: Apache Spark > Add/remove (un)necessary cache and restructure GitHub Actions

[jira] [Updated] (SPARK-33464) Add/remove (un)necessary cache and restructure GitHub Actions yaml

2020-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-33464: - Description: Currently, GitHub Actions build has some unnecessary cache/commands. For example,

[jira] [Updated] (SPARK-33464) Add/remove (un)necessary cache and restructure GitHub Actions yaml

2020-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-33464: - Summary: Add/remove (un)necessary cache and restructure GitHub Actions yaml (was: Remove

[jira] [Commented] (SPARK-33395) Spark reading data in scientific notation

2020-11-16 Thread Nilesh Patil (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233249#comment-17233249 ] Nilesh Patil commented on SPARK-33395: -- [~zhangway],  Hi, I am expecting out like below.  DAta

[jira] [Created] (SPARK-33464) Remove unnecessary cache and restructure

2020-11-16 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-33464: Summary: Remove unnecessary cache and restructure Key: SPARK-33464 URL: https://issues.apache.org/jira/browse/SPARK-33464 Project: Spark Issue Type:

[jira] [Updated] (SPARK-33454) Add GitHub Action job for Hadoop 2

2020-11-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-33454: - Parent: SPARK-32244 Issue Type: Sub-task (was: New Feature) > Add GitHub Action job

[jira] [Commented] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-11-16 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233227#comment-17233227 ] Prashant Sharma commented on SPARK-30985: - Thanks [~dongjoon], you have resolved the confusion I

[jira] [Resolved] (SPARK-33379) The link address of ‘’this page’’ in docs/pyspark-migration-guide.md is incorrect

2020-11-16 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-33379. -- Resolution: Not A Problem > The link address of ‘’this page’’ in

[jira] [Assigned] (SPARK-33209) Clean up unit test file UnsupportedOperationsSuite.scala

2020-11-16 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-33209: Assignee: Cheng Su > Clean up unit test file UnsupportedOperationsSuite.scala >

[jira] [Resolved] (SPARK-33209) Clean up unit test file UnsupportedOperationsSuite.scala

2020-11-16 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33209. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30347

[jira] [Resolved] (SPARK-33445) Can't parse decimal type from csv file

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33445. --- Resolution: Cannot Reproduce I'll close this for now, [~bullsoverbears] . However, you are

[jira] [Commented] (SPARK-33445) Can't parse decimal type from csv file

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233191#comment-17233191 ] Dongjoon Hyun commented on SPARK-33445: --- PySpark is also working. {code:java} >>> mydf2 =

[jira] [Commented] (SPARK-33445) Can't parse decimal type from csv file

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233189#comment-17233189 ] Dongjoon Hyun commented on SPARK-33445: --- [~bullsoverbears]. For me, I don't see `ValueError`. Are

[jira] [Updated] (SPARK-33445) Can't parse decimal type from csv file

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33445: -- Component/s: (was: PySpark) > Can't parse decimal type from csv file >

[jira] [Commented] (SPARK-33449) Add cache for Parquet Metadata

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233184#comment-17233184 ] Dongjoon Hyun commented on SPARK-33449: --- Is this targeting at Apache Spark 3.1, [~yumwang]? > Add

[jira] [Resolved] (SPARK-33399) Normalize output partitioning and sortorder with respect to aliases to avoid unneeded exchange/sort nodes

2020-11-16 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-33399. -- Fix Version/s: 3.1.0 Assignee: Prakhar Jain Resolution: Fixed

[jira] [Assigned] (SPARK-23499) Mesos Cluster Dispatcher should support priority queues to submit drivers

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-23499: - Assignee: Pascal GILLET > Mesos Cluster Dispatcher should support priority queues to

[jira] [Resolved] (SPARK-23499) Mesos Cluster Dispatcher should support priority queues to submit drivers

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-23499. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30352

[jira] [Commented] (SPARK-33463) Spark Thrift Server, keep Job Id when using incremental collect

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233126#comment-17233126 ] Apache Spark commented on SPARK-33463: -- User 'gumartinm' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33463) Spark Thrift Server, keep Job Id when using incremental collect

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33463: Assignee: Apache Spark > Spark Thrift Server, keep Job Id when using incremental collect

[jira] [Assigned] (SPARK-33463) Spark Thrift Server, keep Job Id when using incremental collect

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33463: Assignee: (was: Apache Spark) > Spark Thrift Server, keep Job Id when using

[jira] [Updated] (SPARK-33463) Spark Thrift Server, keep Job Id when using incremental collect

2020-11-16 Thread Gustavo Martin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gustavo Martin updated SPARK-33463: --- Fix Version/s: (was: 3.0.1) 3.1.0 > Spark Thrift Server, keep Job Id

[jira] [Commented] (SPARK-33463) Spark Thrift Server, keep Job Id when using incremental collect

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233125#comment-17233125 ] Apache Spark commented on SPARK-33463: -- User 'gumartinm' has created a pull request for this issue:

[jira] [Commented] (SPARK-33463) Spark Thrift Server, keep Job Id when using incremental collect

2020-11-16 Thread Gustavo Martin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233124#comment-17233124 ] Gustavo Martin commented on SPARK-33463: PR: https://github.com/apache/spark/pull/30390 > Spark

[jira] [Created] (SPARK-33463) Spark Thrift Server, keep Job Id when using incremental collect

2020-11-16 Thread Gustavo Martin (Jira)
Gustavo Martin created SPARK-33463: -- Summary: Spark Thrift Server, keep Job Id when using incremental collect Key: SPARK-33463 URL: https://issues.apache.org/jira/browse/SPARK-33463 Project: Spark

[jira] [Comment Edited] (SPARK-33183) Bug in optimizer rule EliminateSorts

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233058#comment-17233058 ] Dongjoon Hyun edited comment on SPARK-33183 at 11/16/20, 8:23 PM: -- I

[jira] [Comment Edited] (SPARK-33183) Bug in optimizer rule EliminateSorts

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233058#comment-17233058 ] Dongjoon Hyun edited comment on SPARK-33183 at 11/16/20, 8:23 PM: -- I

[jira] [Commented] (SPARK-33183) Bug in optimizer rule EliminateSorts

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233058#comment-17233058 ] Dongjoon Hyun commented on SPARK-33183: --- I added SPARK-23973 as "is caused by". > Bug in

[jira] [Updated] (SPARK-33183) Bug in optimizer rule EliminateSorts

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33183: -- Affects Version/s: 2.4.0 2.4.1 2.4.2

[jira] [Commented] (SPARK-31450) Make ExpressionEncoder thread safe

2020-11-16 Thread Navin Viswanath (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233054#comment-17233054 ] Navin Viswanath commented on SPARK-31450: - [~hvanhovell]  [~dongjoon] I was in the process of

[jira] [Updated] (SPARK-33435) DSv2: REFRESH TABLE should invalidate caches

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33435: -- Labels: DSv2 correctness (was: DSv2) > DSv2: REFRESH TABLE should invalidate caches >

[jira] [Created] (SPARK-33462) ResourceProfile use Int for memory in ExecutorResourcesOrDefaults

2020-11-16 Thread Thomas Graves (Jira)
Thomas Graves created SPARK-33462: - Summary: ResourceProfile use Int for memory in ExecutorResourcesOrDefaults Key: SPARK-33462 URL: https://issues.apache.org/jira/browse/SPARK-33462 Project: Spark

[jira] [Resolved] (SPARK-33460) Accessing map values should fail if key is not found.

2020-11-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33460. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30386

[jira] [Assigned] (SPARK-33460) Accessing map values should fail if key is not found.

2020-11-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-33460: --- Assignee: Leanken.Lin > Accessing map values should fail if key is not found. >

[jira] [Updated] (SPARK-32222) Add K8s IT for conf propagation

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-3: -- Summary: Add K8s IT for conf propagation (was: Add integration tests) > Add K8s IT for conf

[jira] [Resolved] (SPARK-33453) Unify v1 and v2 SHOW PARTITIONS tests

2020-11-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33453. - Fix Version/s: 3.1.0 Assignee: Maxim Gekk Resolution: Fixed > Unify v1 and v2

[jira] [Assigned] (SPARK-32222) Add integration tests

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-3: - Assignee: Prashant Sharma > Add integration tests > - > >

[jira] [Commented] (SPARK-33461) Propagating SPARK_CONF_DIR in K8s and tests

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232864#comment-17232864 ] Dongjoon Hyun commented on SPARK-33461: --- I reorganized this for you as an example, [~prashant]. I

[jira] [Updated] (SPARK-33461) Propagating SPARK_CONF_DIR in K8s and tests

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33461: -- Fix Version/s: (was: 3.1.0) > Propagating SPARK_CONF_DIR in K8s and tests >

[jira] [Updated] (SPARK-33461) Propagating SPARK_CONF_DIR in K8s and tests

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33461: -- Summary: Propagating SPARK_CONF_DIR in K8s and tests (was: Foundational work for propagating

[jira] [Updated] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30985: -- Parent: SPARK-33461 Issue Type: Sub-task (was: Improvement) > Propagate

[jira] [Updated] (SPARK-32223) Support adding a user provided config map.

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32223: -- Parent: SPARK-33461 (was: SPARK-30985) > Support adding a user provided config map. >

[jira] [Updated] (SPARK-33461) Foundational work for propagating SPARK_CONF_DIR

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33461: -- Parent: (was: SPARK-30985) Issue Type: Improvement (was: Sub-task) >

[jira] [Updated] (SPARK-32221) Avoid possible errors due to incorrect file size or type supplied in spark conf.

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32221: -- Parent: SPARK-33461 (was: SPARK-30985) > Avoid possible errors due to incorrect file size or

[jira] [Updated] (SPARK-32222) Add integration tests

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-3: -- Parent: SPARK-33461 (was: SPARK-30985) > Add integration tests > - > >

[jira] [Commented] (SPARK-33461) Foundational work for propagating SPARK_CONF_DIR

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232860#comment-17232860 ] Dongjoon Hyun commented on SPARK-33461: --- The PR is merged by SPARK-30985, not SPARK-33461. The

[jira] [Reopened] (SPARK-33461) Foundational work for propagating SPARK_CONF_DIR

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-33461: --- > Foundational work for propagating SPARK_CONF_DIR >

[jira] [Commented] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232852#comment-17232852 ] Dongjoon Hyun commented on SPARK-30985: --- Hi, [~prashant]. You should not reopen this. We cannot

[jira] [Resolved] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30985. --- Assignee: Prashant Sharma Resolution: Fixed > Propagate SPARK_CONF_DIR files to

[jira] [Resolved] (SPARK-33389) make internal classes of SparkSession always using active SQLConf

2020-11-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33389. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30299

[jira] [Assigned] (SPARK-33389) make internal classes of SparkSession always using active SQLConf

2020-11-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-33389: --- Assignee: Lu Lu > make internal classes of SparkSession always using active SQLConf >

[jira] [Commented] (SPARK-33143) Make SocketAuthServer socket timeout configurable

2020-11-16 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232824#comment-17232824 ] Gabor Somogyi commented on SPARK-33143: --- [~mszurap] the OS and network guys are still working on

[jira] [Commented] (SPARK-33143) Make SocketAuthServer socket timeout configurable

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232818#comment-17232818 ] Apache Spark commented on SPARK-33143: -- User 'gaborgsomogyi' has created a pull request for this

[jira] [Assigned] (SPARK-33143) Make SocketAuthServer socket timeout configurable

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33143: Assignee: (was: Apache Spark) > Make SocketAuthServer socket timeout configurable >

[jira] [Assigned] (SPARK-33143) Make SocketAuthServer socket timeout configurable

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33143: Assignee: Apache Spark > Make SocketAuthServer socket timeout configurable >

[jira] [Assigned] (SPARK-32222) Add integration tests

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-3: Assignee: (was: Apache Spark) > Add integration tests > - > >

[jira] [Commented] (SPARK-32222) Add integration tests

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232762#comment-17232762 ] Apache Spark commented on SPARK-3: -- User 'ScrapCodes' has created a pull request for this

[jira] [Assigned] (SPARK-32222) Add integration tests

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-3: Assignee: Apache Spark > Add integration tests > - > >

[jira] [Commented] (SPARK-17914) Spark SQL casting to TimestampType with nanosecond results in incorrect timestamp

2020-11-16 Thread Pablo Cocko (Jira)
[ https://issues.apache.org/jira/browse/SPARK-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232671#comment-17232671 ] Pablo Cocko commented on SPARK-17914: - I think that the solution only apply if there are more than 6

[jira] [Assigned] (SPARK-33443) LEAD/LAG should support [ IGNORE NULLS | RESPECT NULLS ]

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33443: Assignee: (was: Apache Spark) > LEAD/LAG should support [ IGNORE NULLS | RESPECT

[jira] [Commented] (SPARK-33443) LEAD/LAG should support [ IGNORE NULLS | RESPECT NULLS ]

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232664#comment-17232664 ] Apache Spark commented on SPARK-33443: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33443) LEAD/LAG should support [ IGNORE NULLS | RESPECT NULLS ]

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33443: Assignee: Apache Spark > LEAD/LAG should support [ IGNORE NULLS | RESPECT NULLS ] >

[jira] [Commented] (SPARK-33443) LEAD/LAG should support [ IGNORE NULLS | RESPECT NULLS ]

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232665#comment-17232665 ] Apache Spark commented on SPARK-33443: -- User 'beliefer' has created a pull request for this issue:

[jira] [Updated] (SPARK-33443) LEAD/LAG should support [ IGNORE NULLS | RESPECT NULLS ]

2020-11-16 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-33443: --- Description: The current implement of LEAD/LAG don't support IGNORE/RESPECT NULLS, but the

[jira] [Assigned] (SPARK-33460) Accessing map values should fail if key is not found.

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33460: Assignee: Apache Spark > Accessing map values should fail if key is not found. >

[jira] [Commented] (SPARK-33460) Accessing map values should fail if key is not found.

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232658#comment-17232658 ] Apache Spark commented on SPARK-33460: -- User 'leanken' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33460) Accessing map values should fail if key is not found.

2020-11-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33460: Assignee: (was: Apache Spark) > Accessing map values should fail if key is not

[jira] [Comment Edited] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-11-16 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232631#comment-17232631 ] Prashant Sharma edited comment on SPARK-30985 at 11/16/20, 9:21 AM:

[jira] [Assigned] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-11-16 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma reassigned SPARK-30985: --- Assignee: (was: Prashant Sharma) > Propagate SPARK_CONF_DIR files to driver

[jira] [Assigned] (SPARK-33461) Foundational work for propagating SPARK_CONF_DIR

2020-11-16 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma reassigned SPARK-33461: --- Assignee: Prashant Sharma > Foundational work for propagating SPARK_CONF_DIR >

[jira] [Resolved] (SPARK-33461) Foundational work for propagating SPARK_CONF_DIR

2020-11-16 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma resolved SPARK-33461. - Fix Version/s: 3.1.0 Resolution: Fixed > Foundational work for propagating

[jira] [Created] (SPARK-33461) Foundational work for propagating SPARK_CONF_DIR

2020-11-16 Thread Prashant Sharma (Jira)
Prashant Sharma created SPARK-33461: --- Summary: Foundational work for propagating SPARK_CONF_DIR Key: SPARK-33461 URL: https://issues.apache.org/jira/browse/SPARK-33461 Project: Spark Issue

[jira] [Reopened] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-11-16 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma reopened SPARK-30985: - Reopening this JIRA as, this is Umbrella jira and I created a PR incorrectly targeting the

[jira] [Commented] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-11-16 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232628#comment-17232628 ] Prashant Sharma commented on SPARK-30985: - [~dongjoon] Hm.. my mistake. As you said, I added

[jira] [Commented] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232601#comment-17232601 ] Dongjoon Hyun commented on SPARK-30985: --- [~prashant]. You should not make a PR with the umbrella

[jira] [Comment Edited] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232601#comment-17232601 ] Dongjoon Hyun edited comment on SPARK-30985 at 11/16/20, 8:14 AM: --

[jira] [Comment Edited] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-11-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17232600#comment-17232600 ] Dongjoon Hyun edited comment on SPARK-30985 at 11/16/20, 8:08 AM: --

  1   2   >