[jira] [Commented] (SPARK-27555) cannot create table by using the hive default fileformat in both hive-site.xml and spark-defaults.conf

2019-05-03 Thread Sandeep Katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832999#comment-16832999 ] Sandeep Katta commented on SPARK-27555: --- [~hyukjin.kwon] I think by mistake assignee name is

[jira] [Assigned] (SPARK-27631) Avoid repeating calculate table statistics when AUTO_SIZE_UPDATE_ENABLED is enabled

2019-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27631: Assignee: (was: Apache Spark) > Avoid repeating calculate table statistics when

[jira] [Assigned] (SPARK-27631) Avoid repeating calculate table statistics when AUTO_SIZE_UPDATE_ENABLED is enabled

2019-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27631: Assignee: Apache Spark > Avoid repeating calculate table statistics when

[jira] [Updated] (SPARK-27631) Avoid repeating calculate table statistics when AUTO_SIZE_UPDATE_ENABLED is enabled

2019-05-03 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-27631: Description: How to reproduce: {code:java} build/sbt clean package -Phive -Phadoop-3.2 export

[jira] [Created] (SPARK-27631) Avoid repeating calculate table statistics when AUTO_SIZE_UPDATE_ENABLED is enabled

2019-05-03 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-27631: --- Summary: Avoid repeating calculate table statistics when AUTO_SIZE_UPDATE_ENABLED is enabled Key: SPARK-27631 URL: https://issues.apache.org/jira/browse/SPARK-27631

[jira] [Assigned] (SPARK-27629) Prevent Unpickler from intervening each unpickling

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27629: Assignee: Liang-Chi Hsieh > Prevent Unpickler from intervening each unpickling >

[jira] [Resolved] (SPARK-27629) Prevent Unpickler from intervening each unpickling

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27629. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24521

[jira] [Updated] (SPARK-27629) Prevent Unpickler from intervening each unpickling

2019-05-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-27629: Description: In SPARK-27612, one correctness issue was reported. When protocol 4 is used

[jira] [Updated] (SPARK-27629) Prevent Unpickler from intervening each unpickling

2019-05-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-27629: Description: In SPARK-27612, one correctness issue was reported. When protocol 4 is used

[jira] [Updated] (SPARK-27577) Wrong thresholds selected by BinaryClassificationMetrics when downsampling

2019-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27577: -- Affects Version/s: (was: 2.2.3) (was: 2.2.2)

[jira] [Assigned] (SPARK-27555) cannot create table by using the hive default fileformat in both hive-site.xml and spark-defaults.conf

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27555: Assignee: Hyukjin Kwon > cannot create table by using the hive default fileformat in

[jira] [Resolved] (SPARK-27555) cannot create table by using the hive default fileformat in both hive-site.xml and spark-defaults.conf

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27555. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24489

[jira] [Commented] (SPARK-27609) from_json expects values of options dictionary to be

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832933#comment-16832933 ] Hyukjin Kwon commented on SPARK-27609: -- It's fine. Take a look at 

[jira] [Assigned] (SPARK-27621) Calling transform() method on a LinearRegressionModel throws NoSuchElementException

2019-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27621: - Assignee: Anca Sarb > Calling transform() method on a LinearRegressionModel throws >

[jira] [Resolved] (SPARK-27621) Calling transform() method on a LinearRegressionModel throws NoSuchElementException

2019-05-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27621. --- Resolution: Fixed Fix Version/s: 2.4.4 2.3.4 3.0.0

[jira] [Resolved] (SPARK-27510) Master fall into dead loop while launching executor failed in Worker

2019-05-03 Thread Xingbo Jiang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingbo Jiang resolved SPARK-27510. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24408

[jira] [Updated] (SPARK-12312) JDBC connection to Kerberos secured databases fails on remote executors

2019-05-03 Thread shanyu zhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated SPARK-12312: Affects Version/s: 2.4.2 > JDBC connection to Kerberos secured databases fails on remote

[jira] [Commented] (SPARK-26130) Change Event Timeline Display Functionality on the Stages Page to use either REST API or data from other tables

2019-05-03 Thread Parth Gandhi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832846#comment-16832846 ] Parth Gandhi commented on SPARK-26130: -- Will put up a PR soon. > Change Event Timeline Display

[jira] [Updated] (SPARK-27625) ScalaReflection.serializerFor fails for annotated types

2019-05-03 Thread Patrick Grandjean (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Grandjean updated SPARK-27625: -- Affects Version/s: 2.4.2 > ScalaReflection.serializerFor fails for annotated types >

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

2019-05-03 Thread Robert Joseph Evans (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832776#comment-16832776 ] Robert Joseph Evans commented on SPARK-27396: - [~bryanc] The nice to have arrow formatting

[jira] [Comment Edited] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-03 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832671#comment-16832671 ] Sujith Chacko edited comment on SPARK-27617 at 5/3/19 6:28 PM: --- CREATE

[jira] [Comment Edited] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-03 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832671#comment-16832671 ] Sujith Chacko edited comment on SPARK-27617 at 5/3/19 6:27 PM: --- CREATE

[jira] [Comment Edited] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-03 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832718#comment-16832718 ] Sujith Chacko edited comment on SPARK-27617 at 5/3/19 6:20 PM: --- HIVE does

[jira] [Comment Edited] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-03 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832718#comment-16832718 ] Sujith Chacko edited comment on SPARK-27617 at 5/3/19 6:18 PM: --- HIVE does

[jira] [Commented] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-03 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832718#comment-16832718 ] Sujith Chacko commented on SPARK-27617: --- HIVE does not "externalise" table  if LOCATION specified,

[jira] [Issue Comment Deleted] (SPARK-27612) Creating a DataFrame in PySpark with ArrayType produces some Rows with Arrays of None

2019-05-03 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-27612: - Comment: was deleted (was: Thanks for checking this out [~viirya] and [~hyukjin.kwon]. I agree

[jira] [Commented] (SPARK-27612) Creating a DataFrame in PySpark with ArrayType produces some Rows with Arrays of None

2019-05-03 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832712#comment-16832712 ] Bryan Cutler commented on SPARK-27612: -- Thanks for checking this out [~viirya] and [~hyukjin.kwon].

[jira] [Assigned] (SPARK-27630) Stage retry causes totalRunningTasks calculation to be negative

2019-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27630: Assignee: (was: Apache Spark) > Stage retry causes totalRunningTasks calculation to

[jira] [Commented] (SPARK-27630) Stage retry causes totalRunningTasks calculation to be negative

2019-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832699#comment-16832699 ] Apache Spark commented on SPARK-27630: -- User 'cxzl25' has created a pull request for this issue:

[jira] [Commented] (SPARK-27630) Stage retry causes totalRunningTasks calculation to be negative

2019-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832698#comment-16832698 ] Apache Spark commented on SPARK-27630: -- User 'cxzl25' has created a pull request for this issue:

[jira] [Assigned] (SPARK-27630) Stage retry causes totalRunningTasks calculation to be negative

2019-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27630: Assignee: Apache Spark > Stage retry causes totalRunningTasks calculation to be negative

[jira] [Created] (SPARK-27630) Stage retry causes totalRunningTasks calculation to be negative

2019-05-03 Thread dzcxzl (JIRA)
dzcxzl created SPARK-27630: -- Summary: Stage retry causes totalRunningTasks calculation to be negative Key: SPARK-27630 URL: https://issues.apache.org/jira/browse/SPARK-27630 Project: Spark Issue

[jira] [Commented] (SPARK-22974) CountVectorModel does not attach attributes to output column

2019-05-03 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832691#comment-16832691 ] yuhao yang commented on SPARK-22974: On a business trip from April 29th to May 3rd . Please expect

[jira] [Updated] (SPARK-22974) CountVectorModel does not attach attributes to output column

2019-05-03 Thread William Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Zhang updated SPARK-22974: -- Description: If CountVectorModel transforms columns, the output column will not have

[jira] [Updated] (SPARK-22974) CountVectorModel does not attach attributes to output column

2019-05-03 Thread William Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Zhang updated SPARK-22974: -- Description: If CountVectorModel transforms columns, the output column will not have

[jira] [Updated] (SPARK-22974) CountVectorModel does not attach attributes to output column

2019-05-03 Thread William Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Zhang updated SPARK-22974: -- Description: If CountVectorModel transforms columns, the output column will not have

[jira] [Comment Edited] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-03 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832671#comment-16832671 ] Sujith Chacko edited comment on SPARK-27617 at 5/3/19 5:32 PM: --- CREATE

[jira] [Resolved] (SPARK-27626) Fix `docker-image-tool.sh` to be robust in non-bash shell env

2019-05-03 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27626. --- Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 3.0.0

[jira] [Comment Edited] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-03 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832671#comment-16832671 ] Sujith Chacko edited comment on SPARK-27617 at 5/3/19 5:15 PM: --- CREATE

[jira] [Commented] (SPARK-8682) Range Join for Spark SQL

2019-05-03 Thread Matthew Porter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832678#comment-16832678 ] Matthew Porter commented on SPARK-8682: --- Are there any plans to improve range joins in Spark 3.0? 

[jira] [Comment Edited] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-03 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832671#comment-16832671 ] Sujith Chacko edited comment on SPARK-27617 at 5/3/19 5:06 PM: --- CREATE

[jira] [Commented] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-03 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832671#comment-16832671 ] Sujith Chacko commented on SPARK-27617: --- CREATE TABLE IF NOT EXISTS ext2 (name STRING) LOCATION

[jira] [Updated] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-03 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujith Chacko updated SPARK-27617: -- Description: In Spark whenever user specifies location uri in create table without external

[jira] [Commented] (SPARK-27609) from_json expects values of options dictionary to be

2019-05-03 Thread Zachary Jablons (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832629#comment-16832629 ] Zachary Jablons commented on SPARK-27609: - I can try, but I have exactly 0 experience developing

[jira] [Assigned] (SPARK-27629) Prevent Unpickler from intervening each unpickling

2019-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27629: Assignee: Apache Spark > Prevent Unpickler from intervening each unpickling >

[jira] [Assigned] (SPARK-27629) Prevent Unpickler from intervening each unpickling

2019-05-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27629: Assignee: (was: Apache Spark) > Prevent Unpickler from intervening each unpickling >

[jira] [Updated] (SPARK-27629) Prevent Unpickler from intervening each unpickling

2019-05-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-27629: Component/s: (was: SQL) > Prevent Unpickler from intervening each unpickling >

[jira] [Created] (SPARK-27629) Prevent Unpickler from intervening each unpickling

2019-05-03 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-27629: --- Summary: Prevent Unpickler from intervening each unpickling Key: SPARK-27629 URL: https://issues.apache.org/jira/browse/SPARK-27629 Project: Spark

[jira] [Updated] (SPARK-27628) SortMergeJoin on a low-cardinality column results in heavy skew and large partitions

2019-05-03 Thread Michael Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Wu updated SPARK-27628: --- Description: Let's say we have a dataframe *a* that looks like this: {code:java} | temp | active |

[jira] [Created] (SPARK-27628) SortMergeJoin on a low-cardinality column results in heavy skew and large partitions

2019-05-03 Thread Michael Wu (JIRA)
Michael Wu created SPARK-27628: -- Summary: SortMergeJoin on a low-cardinality column results in heavy skew and large partitions Key: SPARK-27628 URL: https://issues.apache.org/jira/browse/SPARK-27628

[jira] [Assigned] (SPARK-27618) Unnecessary access to externalCatalog

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27618: Assignee: Oliver Caballero Alvarez > Unnecessary access to externalCatalog >

[jira] [Commented] (SPARK-22814) JDBC support date/timestamp type as partitionColumn

2019-05-03 Thread Shyama (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832420#comment-16832420 ] Shyama commented on SPARK-22814: I am trying the same thing but getting error , so how does

[jira] [Commented] (SPARK-27609) from_json expects values of options dictionary to be

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832352#comment-16832352 ] Hyukjin Kwon commented on SPARK-27609: -- We can do from_json, from_csv too while we're here. Wanna

[jira] [Commented] (SPARK-27602) SparkSQL CBO can't get true size of partition table after partition pruning

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832354#comment-16832354 ] Hyukjin Kwon commented on SPARK-27602: -- {quote}  I am tring to build a framework to estimate cost

[jira] [Commented] (SPARK-27609) from_json expects values of options dictionary to be

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832350#comment-16832350 ] Hyukjin Kwon commented on SPARK-27609: -- Okay, we can match to \{read.option(...)}. > from_json

[jira] [Updated] (SPARK-27609) from_json expects values of options dictionary to be

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27609: - Description: When reading a column of a DataFrame that consists of serialized JSON, one of the

[jira] [Updated] (SPARK-27609) from_json expects values of options dictionary to be

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27609: - Summary: from_json expects values of options dictionary to be (was: [Documentation Issue?]

[jira] [Updated] (SPARK-27609) [Documentation Issue?] from_json expects values of options dictionary to be

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27609: - Description: When reading a column of a DataFrame that consists of serialized JSON, one of the

[jira] [Commented] (SPARK-27613) Caching an RDD composed of Row Objects produces some kind of key recombination

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832342#comment-16832342 ] Hyukjin Kwon commented on SPARK-27613: -- Can you show expected input/output? It's hard to read and

[jira] [Updated] (SPARK-27614) Executor shuffle fetch hang

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27614: - Description: Most of the Tasks have been completed, and individual Tasks have a particularly

[jira] [Commented] (SPARK-27615) Merge small files in the read stage

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832341#comment-16832341 ] Hyukjin Kwon commented on SPARK-27615: -- You can just use DataFrame/Dataset APIs or

[jira] [Resolved] (SPARK-27615) Merge small files in the read stage

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27615. -- Resolution: Not A Problem > Merge small files in the read stage >

[jira] [Commented] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832339#comment-16832339 ] Hyukjin Kwon commented on SPARK-27617: -- Can you clarify expected input and output and current input

[jira] [Updated] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27617: - Description: In Spark whenever user specifies location uri in create table without external

[jira] [Updated] (SPARK-27623) Provider org.apache.spark.sql.avro.AvroFileFormat could not be instantiated

2019-05-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27623: - Priority: Major (was: Critical) > Provider org.apache.spark.sql.avro.AvroFileFormat could not

[jira] [Commented] (SPARK-20193) Selecting empty struct causes ExpressionEncoder error.

2019-05-03 Thread Terry Moschou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832314#comment-16832314 ] Terry Moschou commented on SPARK-20193: --- I am in agreement with [~a.ionescu], and have a need for