[jira] [Updated] (SPARK-23619) Document the column names created by explode and posexplode functions

[ https://issues.apache.org/jira/browse/SPARK-23619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23619: -- Affects Version/s: (was: 2.3.0) 3.0.0 > Document the column names

[jira] [Commented] (SPARK-26855) SparkSubmitSuite fails on a clean build

[ https://issues.apache.org/jira/browse/SPARK-26855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765759#comment-16765759 ] Felix Cheung commented on SPARK-26855: -- IMO we have two options: # document tests only pass after

[jira] [Resolved] (SPARK-26853) Enhance expression descriptions for commonly used aggregate function functions.

[ https://issues.apache.org/jira/browse/SPARK-26853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-26853. --- Resolution: Fixed Assignee: Dilip Biswal Fix Version/s: 3.0.0 This is

[jira] [Updated] (SPARK-26853) Add example and version for commonly used aggregate function descriptions

[ https://issues.apache.org/jira/browse/SPARK-26853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26853: -- Summary: Add example and version for commonly used aggregate function descriptions (was:

[jira] [Updated] (SPARK-26853) Enhance expression descriptions for commonly used aggregate function functions.

[ https://issues.apache.org/jira/browse/SPARK-26853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26853: -- Affects Version/s: (was: 2.4.0) 3.0.0 > Enhance expression

[jira] [Assigned] (SPARK-26857) Return UnsafeArrayData for date/timestamp type in ColumnarArray.copy()

[ https://issues.apache.org/jira/browse/SPARK-26857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26857: Assignee: Apache Spark > Return UnsafeArrayData for date/timestamp type in

[jira] [Assigned] (SPARK-26857) Return UnsafeArrayData for date/timestamp type in ColumnarArray.copy()

[ https://issues.apache.org/jira/browse/SPARK-26857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26857: Assignee: (was: Apache Spark) > Return UnsafeArrayData for date/timestamp type in

[jira] [Created] (SPARK-26857) Return UnsafeArrayData for date/timestamp type in ColumnarArray.copy()

Gengliang Wang created SPARK-26857: -- Summary: Return UnsafeArrayData for date/timestamp type in ColumnarArray.copy() Key: SPARK-26857 URL: https://issues.apache.org/jira/browse/SPARK-26857 Project:

[jira] [Commented] (SPARK-26509) Parquet DELTA_BYTE_ARRAY is not supported in Spark 2.x's Vectorized Reader

[ https://issues.apache.org/jira/browse/SPARK-26509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765719#comment-16765719 ] Jialin Qiao commented on SPARK-26509: - You can add this conf. It works for me  

[jira] [Comment Edited] (SPARK-26509) Parquet DELTA_BYTE_ARRAY is not supported in Spark 2.x's Vectorized Reader

[ https://issues.apache.org/jira/browse/SPARK-26509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765719#comment-16765719 ] Jialin Qiao edited comment on SPARK-26509 at 2/12/19 6:24 AM: -- You can  try

[jira] [Commented] (SPARK-24374) SPIP: Support Barrier Execution Mode in Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765694#comment-16765694 ] luzengxiang commented on SPARK-24374: - Hi [~mengxr], I am using Scala API.  > SPIP: Support Barrier

[jira] [Assigned] (SPARK-26762) Arrow optimization for conversion from Spark DataFrame to R DataFrame

[ https://issues.apache.org/jira/browse/SPARK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-26762: Assignee: Hyukjin Kwon > Arrow optimization for conversion from Spark DataFrame to R

[jira] [Commented] (SPARK-26762) Arrow optimization for conversion from Spark DataFrame to R DataFrame

[ https://issues.apache.org/jira/browse/SPARK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765668#comment-16765668 ] Hyukjin Kwon commented on SPARK-26762: -- I happened to make a PR for this one first .. :). dapply

[jira] [Assigned] (SPARK-25158) Executor accidentally exit because ScriptTransformationWriterThread throws TaskKilledException.

[ https://issues.apache.org/jira/browse/SPARK-25158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25158: --- Assignee: Yang Jie > Executor accidentally exit because ScriptTransformationWriterThread

[jira] [Resolved] (SPARK-25158) Executor accidentally exit because ScriptTransformationWriterThread throws TaskKilledException.

[ https://issues.apache.org/jira/browse/SPARK-25158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25158. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22149

[jira] [Assigned] (SPARK-26762) Arrow optimization for conversion from Spark DataFrame to R DataFrame

[ https://issues.apache.org/jira/browse/SPARK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26762: Assignee: Apache Spark > Arrow optimization for conversion from Spark DataFrame to R

[jira] [Assigned] (SPARK-26762) Arrow optimization for conversion from Spark DataFrame to R DataFrame

[ https://issues.apache.org/jira/browse/SPARK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26762: Assignee: (was: Apache Spark) > Arrow optimization for conversion from Spark

[jira] [Closed] (SPARK-25823) map_filter can generate incorrect data

[ https://issues.apache.org/jira/browse/SPARK-25823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-25823. - > map_filter can generate incorrect data > -- > >

[jira] [Resolved] (SPARK-25823) map_filter can generate incorrect data

[ https://issues.apache.org/jira/browse/SPARK-25823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25823. --- Resolution: Duplicate Since this is resolved by SPARK-25829, I close this as a `Duplicate`.

[jira] [Assigned] (SPARK-26696) Dataset encoder should be publicly accessible

[ https://issues.apache.org/jira/browse/SPARK-26696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26696: --- Assignee: Simeon Simeonov > Dataset encoder should be publicly accessible >

[jira] [Resolved] (SPARK-26696) Dataset encoder should be publicly accessible

[ https://issues.apache.org/jira/browse/SPARK-26696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26696. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23620

[jira] [Assigned] (SPARK-26654) Use Timestamp/DateFormatter in CatalogColumnStat

[ https://issues.apache.org/jira/browse/SPARK-26654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26654: --- Assignee: Maxim Gekk > Use Timestamp/DateFormatter in CatalogColumnStat >

[jira] [Resolved] (SPARK-26654) Use Timestamp/DateFormatter in CatalogColumnStat

[ https://issues.apache.org/jira/browse/SPARK-26654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26654. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23662

[jira] [Assigned] (SPARK-26740) Statistics for date and timestamp columns depend on system time zone

[ https://issues.apache.org/jira/browse/SPARK-26740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-26740: --- Assignee: Maxim Gekk > Statistics for date and timestamp columns depend on system time

[jira] [Resolved] (SPARK-26740) Statistics for date and timestamp columns depend on system time zone

[ https://issues.apache.org/jira/browse/SPARK-26740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-26740. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23662

[jira] [Resolved] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang resolved SPARK-26795. - Resolution: Not A Problem > Retry remote fileSegmentManagedBuffer when creating inputStream failed

[jira] [Closed] (SPARK-26795) Retry remote fileSegmentManagedBuffer when creating inputStream failed during shuffle read phase

[ https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang closed SPARK-26795. --- > Retry remote fileSegmentManagedBuffer when creating inputStream failed during > shuffle read phase >

[jira] [Commented] (SPARK-24437) Memory leak in UnsafeHashedRelation

[ https://issues.apache.org/jira/browse/SPARK-24437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765487#comment-16765487 ] t oo commented on SPARK-24437: -- [~dvogelbacher] any luck on the test? > Memory leak in

[jira] [Commented] (SPARK-22860) Spark workers log ssl passwords passed to the executors

[ https://issues.apache.org/jira/browse/SPARK-22860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765482#comment-16765482 ] t oo commented on SPARK-22860: -- gentle ping, fix waiting to be committed > Spark workers log ssl passwords

[jira] [Commented] (SPARK-8659) Spark SQL Thrift Server does NOT honour hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory

[ https://issues.apache.org/jira/browse/SPARK-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765481#comment-16765481 ] t oo commented on SPARK-8659: - bump > Spark SQL Thrift Server does NOT honour >

[jira] [Commented] (SPARK-10892) Join with Data Frame returns wrong results

[ https://issues.apache.org/jira/browse/SPARK-10892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765392#comment-16765392 ] Jungtaek Lim commented on SPARK-10892: -- [~jashgala] Just to clarify, did you also try applying

[jira] [Commented] (SPARK-21492) Memory leak in SortMergeJoin

[ https://issues.apache.org/jira/browse/SPARK-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765378#comment-16765378 ] Tao Luo commented on SPARK-21492: - I'll take a stab at this jira, should have something to review today

[jira] [Commented] (SPARK-26045) Error in the spark 2.4 release package with the spark-avro_2.11 depdency

[ https://issues.apache.org/jira/browse/SPARK-26045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765102#comment-16765102 ] Pushpendra Jaiswal commented on SPARK-26045: Its happening with spark 2.3.1 , spark 2.4.0

[jira] [Updated] (SPARK-20597) KafkaSourceProvider falls back on path as synonym for topic

[ https://issues.apache.org/jira/browse/SPARK-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valeria Vasylieva updated SPARK-20597: -- Attachment: Jacek Laskowski.url > KafkaSourceProvider falls back on path as synonym

[jira] [Commented] (SPARK-20597) KafkaSourceProvider falls back on path as synonym for topic

[ https://issues.apache.org/jira/browse/SPARK-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765060#comment-16765060 ] Valeria Vasylieva commented on SPARK-20597: --- Hi! [~Satyajit] are you still working on this

[jira] [Commented] (SPARK-22826) [SQL] findWiderTypeForTwo Fails over StructField of Array

[ https://issues.apache.org/jira/browse/SPARK-22826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765055#comment-16765055 ] Aleksander Eskilson commented on SPARK-22826: - Yeah, I believe I saw in source code this was 

[jira] [Updated] (SPARK-20597) KafkaSourceProvider falls back on path as synonym for topic

[ https://issues.apache.org/jira/browse/SPARK-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valeria Vasylieva updated SPARK-20597: -- Attachment: (was: Jacek Laskowski.url) > KafkaSourceProvider falls back on path

[jira] [Commented] (SPARK-26836) Columns get switched in Spark SQL using Avro backed Hive table if schema evolves

[ https://issues.apache.org/jira/browse/SPARK-26836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764965#comment-16764965 ] Tamas Nemeth commented on SPARK-26836: -- And one more thing. I also got this warning which I did

[jira] [Commented] (SPARK-26836) Columns get switched in Spark SQL using Avro backed Hive table if schema evolves

[ https://issues.apache.org/jira/browse/SPARK-26836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764933#comment-16764933 ] Tamas Nemeth commented on SPARK-26836: -- On hive directly the query returns with the correct

[jira] [Commented] (SPARK-26856) Python support for "from_avro" and "to_avro" APIs

[ https://issues.apache.org/jira/browse/SPARK-26856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764819#comment-16764819 ] Gabor Somogyi commented on SPARK-26856: --- Until now nothing fancy just added a direct call to the

[jira] [Commented] (SPARK-26856) Python support for "from_avro" and "to_avro" APIs

[ https://issues.apache.org/jira/browse/SPARK-26856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764839#comment-16764839 ] Gabor Somogyi commented on SPARK-26856: --- What can enhanced here is to add a check whether the JVM

[jira] [Commented] (SPARK-26760) [Spark Incorrect display in SPARK UI Executor Tab when number of cores is 4 and Active Task display as 5 in Executor Tab of SPARK UI]

[ https://issues.apache.org/jira/browse/SPARK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764803#comment-16764803 ] shahid commented on SPARK-26760: Yes. Writing store too frequently maybe a costly operation. So, after a

[jira] [Updated] (SPARK-26760) [Spark Incorrect display in SPARK UI Executor Tab when number of cores is 4 and Active Task display as 5 in Executor Tab of SPARK UI]

[ https://issues.apache.org/jira/browse/SPARK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shahid updated SPARK-26760: --- Attachment: Screenshot from 2019-02-11 15-09-09.png > [Spark Incorrect display in SPARK UI Executor Tab

[jira] [Commented] (SPARK-26856) Python support for "from_avro" and "to_avro" APIs

[ https://issues.apache.org/jira/browse/SPARK-26856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764801#comment-16764801 ] Hyukjin Kwon commented on SPARK-26856: -- Yea, actually I was thinking about that. I guess it's good

[jira] [Commented] (SPARK-26856) Python support for "from_avro" and "to_avro" APIs

[ https://issues.apache.org/jira/browse/SPARK-26856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764790#comment-16764790 ] Gabor Somogyi commented on SPARK-26856: --- I've created a patch but hesitant to create a PR because

[jira] [Created] (SPARK-26856) Python support for "from_avro" and "to_avro" APIs

Gabor Somogyi created SPARK-26856: - Summary: Python support for "from_avro" and "to_avro" APIs Key: SPARK-26856 URL: https://issues.apache.org/jira/browse/SPARK-26856 Project: Spark Issue

[jira] [Commented] (SPARK-26760) [Spark Incorrect display in SPARK UI Executor Tab when number of cores is 4 and Active Task display as 5 in Executor Tab of SPARK UI]

[ https://issues.apache.org/jira/browse/SPARK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764777#comment-16764777 ] ABHISHEK KUMAR GUPTA commented on SPARK-26760: -- So u mean this is the issue with small

[jira] [Commented] (SPARK-26783) Kafka parameter documentation doesn't match with the reality (upper/lowercase)

[ https://issues.apache.org/jira/browse/SPARK-26783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764772#comment-16764772 ] Gabor Somogyi commented on SPARK-26783: --- [~dongjoon] at the moment waiting on [~sindiri] to report

[jira] [Comment Edited] (SPARK-26845) Avro to_avro from_avro roundtrip fails if data type is string

[ https://issues.apache.org/jira/browse/SPARK-26845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764769#comment-16764769 ] Gabor Somogyi edited comment on SPARK-26845 at 2/11/19 8:50 AM:

[jira] [Commented] (SPARK-26845) Avro to_avro from_avro roundtrip fails if data type is string

[ https://issues.apache.org/jira/browse/SPARK-26845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764769#comment-16764769 ] Gabor Somogyi commented on SPARK-26845: --- [~Gengliang.Wang] Thanks for the confirmation! Hope

[jira] [Commented] (SPARK-26845) Avro to_avro from_avro roundtrip fails if data type is string

[ https://issues.apache.org/jira/browse/SPARK-26845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764747#comment-16764747 ] Gengliang Wang commented on SPARK-26845: [~attilapiros]Thanks for the help! [~gsomogyi] Sorry