[jira] [Commented] (SPARK-28836) Improve canonicalize API

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913973#comment-16913973 ] Dongjoon Hyun commented on SPARK-28836: --- This issue content is switched with SPARK

[jira] [Resolved] (SPARK-28835) Introduce TPCDSSchema

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28835. --- Fix Version/s: 3.0.0 Assignee: Ali Afroozeh Resolution: Fixed This is resolv

[jira] [Updated] (SPARK-28836) Improve canonicalize API

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28836: -- Description: This PR improves the `canonicalize` API by removing the method `def canonicalize(

[jira] [Updated] (SPARK-28835) Introduce TPCDSSchema

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28835: -- Description: This PR extracts the schema information of TPCDS tables into a separate class cal

[jira] [Updated] (SPARK-28836) Improve canonicalize API

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28836: -- Summary: Improve canonicalize API (was: Introduce TPCDSSchema) > Improve canonicalize API > -

[jira] [Updated] (SPARK-28835) Introduce TPCDSSchema

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28835: -- Summary: Introduce TPCDSSchema (was: Improve canonicalize API) > Introduce TPCDSSchema >

[jira] [Reopened] (SPARK-28836) Introduce TPCDSSchema

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-28836: --- > Introduce TPCDSSchema > - > > Key: SPARK-28836 >

[jira] [Commented] (SPARK-28836) Introduce TPCDSSchema

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913971#comment-16913971 ] Dongjoon Hyun commented on SPARK-28836: --- Oops. Sorry, [~hyukjin.kwon]. I merged th

[jira] [Resolved] (SPARK-28319) DataSourceV2: Support SHOW TABLES

2019-08-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28319. - Fix Version/s: 3.0.0 Assignee: Terry Kim Resolution: Fixed > DataSourceV2: Suppo

[jira] [Assigned] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-08-22 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-28025: Assignee: Jungtaek Lim > HDFSBackedStateStoreProvider should not leak .crc files > -

[jira] [Resolved] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

2019-08-22 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-28025. -- Fix Version/s: 3.0.0 Resolution: Fixed > HDFSBackedStateStoreProvider should not leak .

[jira] [Commented] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

2019-08-22 Thread hemanth meka (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913965#comment-16913965 ] hemanth meka commented on SPARK-23519: -- I have a fix for this. checkColumnNameDupli

[jira] [Resolved] (SPARK-28730) Configurable type coercion policy for table insertion

2019-08-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28730. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25453 [https://gith

[jira] [Assigned] (SPARK-28730) Configurable type coercion policy for table insertion

2019-08-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28730: --- Assignee: Gengliang Wang > Configurable type coercion policy for table insertion >

[jira] [Created] (SPARK-28857) Clean up the comments of PR template during merging

2019-08-22 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-28857: - Summary: Clean up the comments of PR template during merging Key: SPARK-28857 URL: https://issues.apache.org/jira/browse/SPARK-28857 Project: Spark Issue T

[jira] [Assigned] (SPARK-28702) Display useful error message (instead of NPE) for invalid Dataset operations (e.g. calling actions inside of transformations)

2019-08-22 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-28702: -- Assignee: Shivu Sondur > Display useful error message (instead of NPE) for invalid Dataset op

[jira] [Resolved] (SPARK-28702) Display useful error message (instead of NPE) for invalid Dataset operations (e.g. calling actions inside of transformations)

2019-08-22 Thread Josh Rosen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-28702. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25503 [https://github

[jira] [Resolved] (SPARK-28832) Document SHOW SCHEMAS statement in SQL Reference.

2019-08-22 Thread jobit mathew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jobit mathew resolved SPARK-28832. -- Resolution: Duplicate Closing the JIRA as it will cover as a part of SHOW DATABASES statement

[jira] [Commented] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

2019-08-22 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913915#comment-16913915 ] Liang-Chi Hsieh commented on SPARK-23519: - Thanks for pinging me. I am going on

[jira] [Commented] (SPARK-28827) Document SELECT CURRENT_DATABASE in SQL Reference

2019-08-22 Thread Shivu Sondur (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913903#comment-16913903 ] Shivu Sondur commented on SPARK-28827: -- i will work on this > Document SELECT CURR

[jira] [Commented] (SPARK-28823) Document CREATE ROLE Statement

2019-08-22 Thread Shivu Sondur (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913902#comment-16913902 ] Shivu Sondur commented on SPARK-28823: -- i will work on this > Document CREATE ROLE

[jira] [Commented] (SPARK-28482) Data incomplete when using pandas udf in Python 3

2019-08-22 Thread jiangyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913879#comment-16913879 ] jiangyu commented on SPARK-28482: - hi, [~bryanc] , i have tested toPandas(), it is okay.

[jira] [Commented] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

2019-08-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913855#comment-16913855 ] Wenchen Fan commented on SPARK-23519: - I think this is a bug and should be fixed. cc

[jira] [Commented] (SPARK-27594) spark.sql.orc.enableVectorizedReader causes milliseconds in Timestamp to be read incorrectly

2019-08-22 Thread Owen O'Malley (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913665#comment-16913665 ] Owen O'Malley commented on SPARK-27594: --- This is being caused by an ORC bug that w

[jira] [Assigned] (SPARK-28769) Improve warning message in Barrier Execution Mode in case required slots > maximum slots

2019-08-22 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28769: - Assignee: Kousuke Saruta > Improve warning message in Barrier Execution Mode in case required s

[jira] [Resolved] (SPARK-28769) Improve warning message in Barrier Execution Mode in case required slots > maximum slots

2019-08-22 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28769. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25487 [https://github.c

[jira] [Commented] (SPARK-28482) Data incomplete when using pandas udf in Python 3

2019-08-22 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913611#comment-16913611 ] Bryan Cutler commented on SPARK-28482: -- I'm not really sure what you are doing abov

[jira] [Commented] (SPARK-28832) Document SHOW SCHEMAS statement in SQL Reference.

2019-08-22 Thread Dilip Biswal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913560#comment-16913560 ] Dilip Biswal commented on SPARK-28832: -- [~jobitmathew] Thanks .. Yeah it will be do

[jira] [Closed] (SPARK-28846) Set OMP_NUM_THREADS to executor cores for python

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-28846. - > Set OMP_NUM_THREADS to executor cores for python > ---

[jira] [Resolved] (SPARK-28846) Set OMP_NUM_THREADS to executor cores for python

2019-08-22 Thread Ryan Blue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved SPARK-28846. --- Resolution: Duplicate > Set OMP_NUM_THREADS to executor cores for python > -

[jira] [Commented] (SPARK-28856) DataSourceV2: Support SHOW DATABASES

2019-08-22 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913426#comment-16913426 ] Terry Kim commented on SPARK-28856: --- I will work on this. > DataSourceV2: Support SHO

[jira] [Created] (SPARK-28856) DataSourceV2: Support SHOW DATABASES

2019-08-22 Thread Terry Kim (Jira)
Terry Kim created SPARK-28856: - Summary: DataSourceV2: Support SHOW DATABASES Key: SPARK-28856 URL: https://issues.apache.org/jira/browse/SPARK-28856 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-28577) Ensure executorMemoryHead requested value not less than MEMORY_OFFHEAP_SIZE when MEMORY_OFFHEAP_ENABLED is true

2019-08-22 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-28577: -- Docs Text: On YARN, The off heap memory size is now separately included for the user in the co

[jira] [Updated] (SPARK-28577) Ensure executorMemoryHead requested value not less than MEMORY_OFFHEAP_SIZE when MEMORY_OFFHEAP_ENABLED is true

2019-08-22 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-28577: -- Labels: release-notes (was: ) > Ensure executorMemoryHead requested value not less than MEMOR

[jira] [Updated] (SPARK-28854) Zipping iterators in mapPartitions will fail

2019-08-22 Thread Hao Yang Ang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Yang Ang updated SPARK-28854: - Description: scala> sc.parallelize(Seq(1, 2, 3)).mapPartitions(xs => xs.map(2*).zip(xs)).collec

[jira] [Resolved] (SPARK-13677) Support Tree-Based Feature Transformation for ML

2019-08-22 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-13677. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25383 [https://github.c

[jira] [Assigned] (SPARK-13677) Support Tree-Based Feature Transformation for ML

2019-08-22 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-13677: - Assignee: zhengruifeng > Support Tree-Based Feature Transformation for ML > ---

[jira] [Resolved] (SPARK-28776) SparkML MLWriter gets hadoop conf from spark context instead of session

2019-08-22 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28776. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25505 [https://github.c

[jira] [Assigned] (SPARK-28776) SparkML MLWriter gets hadoop conf from spark context instead of session

2019-08-22 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28776: - Assignee: Helen Yu > SparkML MLWriter gets hadoop conf from spark context instead of session >

[jira] [Created] (SPARK-28855) Remove outdated Experimental, Evolving annotations

2019-08-22 Thread Sean Owen (Jira)
Sean Owen created SPARK-28855: - Summary: Remove outdated Experimental, Evolving annotations Key: SPARK-28855 URL: https://issues.apache.org/jira/browse/SPARK-28855 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-28854) Zipping iterators in mapPartitions will fail

2019-08-22 Thread Hao Yang Ang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Yang Ang updated SPARK-28854: - Description: scala> sc.parallelize(Seq(1, 2, 3)).mapPartitions(xs => xs.map(2*).zip(xs)).foreac

[jira] [Created] (SPARK-28854) Zipping iterators in mapPartitions will fail

2019-08-22 Thread Hao Yang Ang (Jira)
Hao Yang Ang created SPARK-28854: Summary: Zipping iterators in mapPartitions will fail Key: SPARK-28854 URL: https://issues.apache.org/jira/browse/SPARK-28854 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-28512) New optional mode: throw runtime exceptions on casting failures

2019-08-22 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28512: --- Description: In popular DBMS like MySQL/PostgreSQL/Oracle, runtime exceptions are thrown on

[jira] [Created] (SPARK-28853) Support conf to organize filePartitions by file path

2019-08-22 Thread ZhangYao (Jira)
ZhangYao created SPARK-28853: Summary: Support conf to organize filePartitions by file path Key: SPARK-28853 URL: https://issues.apache.org/jira/browse/SPARK-28853 Project: Spark Issue Type: New

[jira] [Updated] (SPARK-28741) New optional mode: Throw exceptions when casting to integers causes overflow

2019-08-22 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28741: --- Summary: New optional mode: Throw exceptions when casting to integers causes overflow (was:

[jira] [Resolved] (SPARK-28503) Return null result on cast an out-of-range value to a integral type

2019-08-22 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-28503. Resolution: Won't Fix After consideration, I decide to close this one and open https://is

[jira] [Updated] (SPARK-28741) Throw exceptions when casting to integers causes overflow

2019-08-22 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28741: --- Parent: SPARK-28589 Issue Type: Sub-task (was: New Feature) > Throw exceptions when

[jira] [Updated] (SPARK-28741) Throw exceptions when casting to integers causes overflow

2019-08-22 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28741: --- Parent: (was: SPARK-26217) Issue Type: New Feature (was: Sub-task) > Throw exce

[jira] [Created] (SPARK-28852) Implement GetCatalogsOperation

2019-08-22 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-28852: --- Summary: Implement GetCatalogsOperation Key: SPARK-28852 URL: https://issues.apache.org/jira/browse/SPARK-28852 Project: Spark Issue Type: Sub-task C

[jira] [Updated] (SPARK-28852) Implement GetCatalogsOperation for Thrift Server

2019-08-22 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28852: Summary: Implement GetCatalogsOperation for Thrift Server (was: Implement GetCatalogsOperation)

[jira] [Resolved] (SPARK-28848) insert overwrite local directory stored as parquet does not creates snappy.parquet data file at local directory path

2019-08-22 Thread Ajith S (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajith S resolved SPARK-28848. - Resolution: Duplicate Will be fixed as part of SPARK-28659 > insert overwrite local directory stored a

[jira] [Updated] (SPARK-28495) Introduce ANSI store assignment policy for table insertion

2019-08-22 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28495: --- Description: In Spark version 2.4 and earlier, when inserting into a table, Spark will cast

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

2019-08-22 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-28849: Description: Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling {{transferTo

[jira] [Commented] (SPARK-28797) Document DROP FUNCTION statement in SQL Reference.

2019-08-22 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913219#comment-16913219 ] Sandeep Katta commented on SPARK-28797: --- PR is created https://github.com/apache/

[jira] [Comment Edited] (SPARK-28482) Data incomplete when using pandas udf in Python 3

2019-08-22 Thread jiangyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913183#comment-16913183 ] jiangyu edited comment on SPARK-28482 at 8/22/19 9:37 AM: -- hi,

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

2019-08-22 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-28849: Description: Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling {{transferTo

[jira] [Commented] (SPARK-28482) Data incomplete when using pandas udf in Python 3

2019-08-22 Thread jiangyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913183#comment-16913183 ] jiangyu commented on SPARK-28482: - hi, [~bryanc] , maybe you should produce more data, l

[jira] [Updated] (SPARK-28495) Introduce ANSI store assignment policy for table insertion

2019-08-22 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28495: --- Summary: Introduce ANSI store assignment policy for table insertion (was: Follow ANSI SQL o

[jira] [Comment Edited] (SPARK-22876) spark.yarn.am.attemptFailuresValidityInterval does not work correctly

2019-08-22 Thread Nikita Gorbachevski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913181#comment-16913181 ] Nikita Gorbachevski edited comment on SPARK-22876 at 8/22/19 9:22 AM:

[jira] [Commented] (SPARK-22876) spark.yarn.am.attemptFailuresValidityInterval does not work correctly

2019-08-22 Thread Nikita Gorbachevski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913181#comment-16913181 ] Nikita Gorbachevski commented on SPARK-22876: - Hi [~praveentallapudi], these

[jira] [Updated] (SPARK-28730) Configurable type coercion policy for table insertion

2019-08-22 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28730: --- Description: After all the discussions in the dev list: http://apache-spark-developers-list

[jira] [Updated] (SPARK-28851) Connect HBase using Spark SQL in Spark 2.x

2019-08-22 Thread ARUN KINDRA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ARUN KINDRA updated SPARK-28851: Description: Hi,   I am basically trying a sample Spark SQL Code which actually read data from

[jira] [Updated] (SPARK-28730) Configurable type coercion policy for table insertion

2019-08-22 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28730: --- Parent: SPARK-28589 Issue Type: Sub-task (was: Improvement) > Configurable type coe

[jira] [Created] (SPARK-28851) Connect HBase using Spark SQL in Spark 2.x

2019-08-22 Thread ARUN KINDRA (Jira)
ARUN KINDRA created SPARK-28851: --- Summary: Connect HBase using Spark SQL in Spark 2.x Key: SPARK-28851 URL: https://issues.apache.org/jira/browse/SPARK-28851 Project: Spark Issue Type: Question

[jira] [Updated] (SPARK-28850) Binary Files RDD allocates false number of threads

2019-08-22 Thread Marco Lotz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Lotz updated SPARK-28850: --- Description: When making a call to: {code:java} sc.binaryFiles(somePath){code}   It creates a Binar

[jira] [Updated] (SPARK-28850) Binary Files RDD allocates false number of threads

2019-08-22 Thread Marco Lotz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Lotz updated SPARK-28850: --- Description: When making a call to: {code:java} sc.binaryFiles(somePath){code}   It creates a Binar

[jira] [Updated] (SPARK-28850) Binary Files RDD allocates false number of threads

2019-08-22 Thread Marco Lotz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Lotz updated SPARK-28850: --- Description: When making a call to: {code:java} sc.binaryFiles(somePath){code}   It creates a Binar

[jira] [Updated] (SPARK-28850) Binary Files RDD allocates false number of threads

2019-08-22 Thread Marco Lotz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Lotz updated SPARK-28850: --- Description: When making a call to: {code:java} sc.binaryFiles(somePath){code}   It creates a Binar

[jira] [Created] (SPARK-28850) Binary Files RDD allocates false number of threads

2019-08-22 Thread Marco Lotz (Jira)
Marco Lotz created SPARK-28850: -- Summary: Binary Files RDD allocates false number of threads Key: SPARK-28850 URL: https://issues.apache.org/jira/browse/SPARK-28850 Project: Spark Issue Type: Bu

[jira] [Commented] (SPARK-28779) CSV writer doesn't handle older Mac line endings

2019-08-22 Thread nicolas paris (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913151#comment-16913151 ] nicolas paris commented on SPARK-28779: --- good to know thanks > CSV writer doesn't

[jira] [Commented] (SPARK-28832) Document SHOW SCHEMAS statement in SQL Reference.

2019-08-22 Thread jobit mathew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913149#comment-16913149 ] jobit mathew commented on SPARK-28832: -- [~dkbiswal],But commands are different righ

[jira] [Resolved] (SPARK-28836) Introduce TPCDSSchema

2019-08-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28836. -- Resolution: Won't Fix See {{TPCDSQueryBenchmark}}. We already have the ways in Spark code base

[jira] [Resolved] (SPARK-28844) Fix typo in SQLConf FILE_COMRESSION_FACTOR

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28844. --- Fix Version/s: 2.4.4 2.3.4 3.0.0 Resolution: Fix

[jira] [Updated] (SPARK-26895) When running spark 2.3 as a proxy user (--proxy-user), SparkSubmit fails to resolve globs owned by target user

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26895: -- Fix Version/s: 2.3.4 > When running spark 2.3 as a proxy user (--proxy-user), SparkSubmit fail

[jira] [Assigned] (SPARK-28847) Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28847: - Assignee: Dongjoon Hyun > Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTe

[jira] [Updated] (SPARK-28780) Delete the incorrect setWeightCol method in LinearSVCModel

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28780: -- Fix Version/s: 2.3.4 > Delete the incorrect setWeightCol method in LinearSVCModel > --

[jira] [Updated] (SPARK-28780) Delete the incorrect setWeightCol method in LinearSVCModel

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28780: -- Fix Version/s: 2.4.4 > Delete the incorrect setWeightCol method in LinearSVCModel > --

[jira] [Resolved] (SPARK-28847) Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest

2019-08-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28847. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25550 [https://

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

2019-08-22 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-28849: Description: Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling {{transferTo

[jira] [Resolved] (SPARK-28821) Document COMPUTE STAT in SQL Reference

2019-08-22 Thread ABHISHEK KUMAR GUPTA (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ABHISHEK KUMAR GUPTA resolved SPARK-28821. -- Resolution: Duplicate Will be covered in ANALYZE TABLE JIRA for document > Do

[jira] [Updated] (SPARK-28848) insert overwrite local directory stored as parquet does not creates snappy.parquet data file at local directory path

2019-08-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28848: - Description: {code} 0: jdbc:hive2://10.18.18.214:23040/func> insert overwrite local directory '

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

2019-08-22 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-28849: Description: Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling {{transferTo

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

2019-08-22 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-28849: Description: Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling {{transferTo

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

2019-08-22 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-28849: Description: Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling {{transferTo

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

2019-08-22 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-28849: Attachment: D18F4.png 95330.png 91ADA.png > Spark's UnsafeShuffleW

[jira] [Updated] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

2019-08-22 Thread Saisai Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-28849: Description: Spark's {{UnsafeShuffleWriter}} may run into infinite loop when calling {{transferTo

[jira] [Created] (SPARK-28849) Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally

2019-08-22 Thread Saisai Shao (Jira)
Saisai Shao created SPARK-28849: --- Summary: Spark's UnsafeShuffleWriter may run into infinite loop in transferTo occasionally Key: SPARK-28849 URL: https://issues.apache.org/jira/browse/SPARK-28849 Proje