[jira] [Commented] (SPARK-23040) BlockStoreShuffleReader's return Iterator isn't interruptible if aggregator or ordering is specified

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421170#comment-16421170 ] Apache Spark commented on SPARK-23040: -- User 'jiangxb1987' has created a pull request for this

[jira] [Resolved] (SPARK-23827) StreamingJoinExec should ensure that input data is partitioned into specific number of partitions

2018-03-30 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-23827. --- Resolution: Fixed Fix Version/s: 2.3.1 2.4.0

[jira] [Assigned] (SPARK-23822) Improve error message for Parquet schema mismatches

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23822: Assignee: (was: Apache Spark) > Improve error message for Parquet schema mismatches >

[jira] [Assigned] (SPARK-23822) Improve error message for Parquet schema mismatches

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23822: Assignee: Apache Spark > Improve error message for Parquet schema mismatches >

[jira] [Commented] (SPARK-23822) Improve error message for Parquet schema mismatches

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421051#comment-16421051 ] Apache Spark commented on SPARK-23822: -- User 'yuchenhuo' has created a pull request for this issue:

[jira] [Commented] (SPARK-23836) Support returning StructType & MapType in Arrow's "scalar" UDFS (or similar)

2018-03-30 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421044#comment-16421044 ] holdenk commented on SPARK-23836: - cc [~bryanc] > Support returning StructType & MapType in Arrow's

[jira] [Created] (SPARK-23836) Support returning StructType & MapType in Arrow's "scalar" UDFS (or similar)

2018-03-30 Thread holdenk (JIRA)
holdenk created SPARK-23836: --- Summary: Support returning StructType & MapType in Arrow's "scalar" UDFS (or similar) Key: SPARK-23836 URL: https://issues.apache.org/jira/browse/SPARK-23836 Project: Spark

[jira] [Assigned] (SPARK-6951) History server slow startup if the event log directory is large

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6951: --- Assignee: (was: Apache Spark) > History server slow startup if the event log directory

[jira] [Assigned] (SPARK-6951) History server slow startup if the event log directory is large

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6951: --- Assignee: Apache Spark > History server slow startup if the event log directory is large >

[jira] [Commented] (SPARK-6951) History server slow startup if the event log directory is large

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420951#comment-16420951 ] Apache Spark commented on SPARK-6951: - User 'vanzin' has created a pull request for this issue:

[jira] [Commented] (SPARK-23828) PySpark StringIndexerModel should have constructor from labels

2018-03-30 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420938#comment-16420938 ] Huaxin Gao commented on SPARK-23828: [~bryanc] Are you going to work on this yourself? If not, can I

[jira] [Resolved] (SPARK-23640) Hadoop config may override spark config

2018-03-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23640. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20785

[jira] [Assigned] (SPARK-23640) Hadoop config may override spark config

2018-03-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23640: -- Assignee: Yuming Wang > Hadoop config may override spark config >

[jira] [Commented] (SPARK-23099) Migrate foreach sink

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420927#comment-16420927 ] Apache Spark commented on SPARK-23099: -- User 'jose-torres' has created a pull request for this

[jira] [Assigned] (SPARK-23834) Flaky test: LauncherServerSuite.testAppHandleDisconnect

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23834: Assignee: Apache Spark > Flaky test: LauncherServerSuite.testAppHandleDisconnect >

[jira] [Commented] (SPARK-23834) Flaky test: LauncherServerSuite.testAppHandleDisconnect

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420887#comment-16420887 ] Apache Spark commented on SPARK-23834: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23834) Flaky test: LauncherServerSuite.testAppHandleDisconnect

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23834: Assignee: (was: Apache Spark) > Flaky test:

[jira] [Commented] (SPARK-19018) spark csv writer charset support

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420871#comment-16420871 ] Apache Spark commented on SPARK-19018: -- User 'crafty-coder' has created a pull request for this

[jira] [Updated] (SPARK-23827) StreamingJoinExec should ensure that input data is partitioned into specific number of partitions

2018-03-30 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23827: -- Target Version/s: 2.3.1, 2.4.0, 3.0.0 (was: 2.3.1, 3.0.0) > StreamingJoinExec should ensure

[jira] [Resolved] (SPARK-14044) Allow configuration of DynamicPartitionWriterContainer#writeRows to bypass sort step

2018-03-30 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-14044. -- Resolution: Duplicate I'm resolving this as a duplicate of SPARK-19563 -- please re-open if

[jira] [Commented] (SPARK-23835) When Dataset.as converts column from nullable to non-nullable type, null Doubles are converted silently to -1

2018-03-30 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420809#comment-16420809 ] Michael Armbrust commented on SPARK-23835: -- /cc [~cloud_fan] > When Dataset.as converts column

[jira] [Created] (SPARK-23835) When Dataset.as converts column from nullable to non-nullable type, null Doubles are converted silently to -1

2018-03-30 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-23835: - Summary: When Dataset.as converts column from nullable to non-nullable type, null Doubles are converted silently to -1 Key: SPARK-23835 URL:

[jira] [Created] (SPARK-23834) Flaky test: LauncherServerSuite.testAppHandleDisconnect

2018-03-30 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-23834: -- Summary: Flaky test: LauncherServerSuite.testAppHandleDisconnect Key: SPARK-23834 URL: https://issues.apache.org/jira/browse/SPARK-23834 Project: Spark

[jira] [Updated] (SPARK-23785) LauncherBackend doesn't check state of connection before setting state

2018-03-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-23785: --- Fix Version/s: (was: 2.3.1) > LauncherBackend doesn't check state of connection before

[jira] [Updated] (SPARK-23833) Incorrect primitive type check for input arguments of udf

2018-03-30 Thread Valentin Nikotin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valentin Nikotin updated SPARK-23833: - Description: There is claimed behavior for scala UDFs with primitive type arguments:

[jira] [Created] (SPARK-23833) Incorrect primitive type check for input arguments of udf

2018-03-30 Thread Valentin Nikotin (JIRA)
Valentin Nikotin created SPARK-23833: Summary: Incorrect primitive type check for input arguments of udf Key: SPARK-23833 URL: https://issues.apache.org/jira/browse/SPARK-23833 Project: Spark

[jira] [Commented] (SPARK-23705) dataframe.groupBy() may inadvertently receive sequence of non-distinct strings

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420640#comment-16420640 ] Apache Spark commented on SPARK-23705: -- User 'vinodkc' has created a pull request for this issue:

[jira] [Resolved] (SPARK-23789) Shouldn't set hive.metastore.uris before invoking HiveDelegationTokenProvider

2018-03-30 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-23789. - Resolution: Won't Fix > Shouldn't set hive.metastore.uris before invoking

[jira] [Assigned] (SPARK-23565) Improved error message for when the number of sources for a query changes

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23565: Assignee: (was: Apache Spark) > Improved error message for when the number of sources

[jira] [Commented] (SPARK-23565) Improved error message for when the number of sources for a query changes

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420548#comment-16420548 ] Apache Spark commented on SPARK-23565: -- User 'patrickmcgloin' has created a pull request for this

[jira] [Assigned] (SPARK-23565) Improved error message for when the number of sources for a query changes

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23565: Assignee: Apache Spark > Improved error message for when the number of sources for a

[jira] [Created] (SPARK-23832) Adding possibility to set timestamp into KafkaRowWriter

2018-03-30 Thread Alexey (JIRA)
Alexey created SPARK-23832: -- Summary: Adding possibility to set timestamp into KafkaRowWriter Key: SPARK-23832 URL: https://issues.apache.org/jira/browse/SPARK-23832 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-23790) proxy-user failed connecting to a kerberos configured metastore

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23790: Assignee: Apache Spark > proxy-user failed connecting to a kerberos configured metastore

[jira] [Commented] (SPARK-23790) proxy-user failed connecting to a kerberos configured metastore

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420472#comment-16420472 ] Apache Spark commented on SPARK-23790: -- User 'skonto' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23790) proxy-user failed connecting to a kerberos configured metastore

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23790: Assignee: (was: Apache Spark) > proxy-user failed connecting to a kerberos configured

[jira] [Commented] (SPARK-22371) dag-scheduler-event-loop thread stopped with error Attempted to access garbage collected accumulator 5605982

2018-03-30 Thread Marek Byszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420397#comment-16420397 ] Marek Byszewski commented on SPARK-22371: - We were hit by this issue on 2.3.0 too while running

[jira] [Updated] (SPARK-23831) Add org.apache.derby to IsolatedClientLoader

2018-03-30 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-23831: Description: Add org.apache.derby to IsolatedClientLoader,otherwise it may throw an exception:

[jira] [Updated] (SPARK-23831) Add org.apache.derby to IsolatedClientLoader

2018-03-30 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-23831: Description: Add org.apache.derby to IsolatedClientLoader,otherwise it may throw an exception:

[jira] [Assigned] (SPARK-23831) Add org.apache.derby to IsolatedClientLoader

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23831: Assignee: Apache Spark > Add org.apache.derby to IsolatedClientLoader >

[jira] [Commented] (SPARK-23831) Add org.apache.derby to IsolatedClientLoader

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420377#comment-16420377 ] Apache Spark commented on SPARK-23831: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23831) Add org.apache.derby to IsolatedClientLoader

2018-03-30 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23831: Assignee: (was: Apache Spark) > Add org.apache.derby to IsolatedClientLoader >

[jira] [Updated] (SPARK-23831) Add org.apache.derby to IsolatedClientLoader

2018-03-30 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-23831: Description: How to reproduce: {noformat} sed

[jira] [Created] (SPARK-23831) Add org.apache.derby to IsolatedClientLoader

2018-03-30 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-23831: --- Summary: Add org.apache.derby to IsolatedClientLoader Key: SPARK-23831 URL: https://issues.apache.org/jira/browse/SPARK-23831 Project: Spark Issue Type:

[jira] [Created] (SPARK-23830) Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object

2018-03-30 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-23830: --- Summary: Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object Key: SPARK-23830 URL:

[jira] [Commented] (SPARK-23761) Dataframe filter(udf) followed by groupby in pyspark throws a casting error

2018-03-30 Thread Dhaniram Kshirsagar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420339#comment-16420339 ] Dhaniram Kshirsagar commented on SPARK-23761: - Sure, will try it with latest version of

[jira] [Commented] (SPARK-23791) Sub-optimal generated code for sum aggregating

2018-03-30 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420335#comment-16420335 ] Marco Gaido commented on SPARK-23791: - Hi [~rednikotin]. Thanks for reporting this. The error you

[jira] [Resolved] (SPARK-23767) DirectStream is producing the incorrect type of message

2018-03-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23767. - Resolution: Not A Problem > DirectStream is producing the incorrect type of message >

[jira] [Commented] (SPARK-23767) DirectStream is producing the incorrect type of message

2018-03-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420291#comment-16420291 ] Saisai Shao commented on SPARK-23767: - This seems not a Spark issue, more like a microsoft event-hub

[jira] [Updated] (SPARK-21337) SQL which has large ‘case when’ expressions may cause code generation beyond 64KB

2018-03-30 Thread fengchaoge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengchaoge updated SPARK-21337: --- Attachment: HmsClient.bak > SQL which has large ‘case when’ expressions may cause code generation

[jira] [Updated] (SPARK-21337) SQL which has large ‘case when’ expressions may cause code generation beyond 64KB

2018-03-30 Thread fengchaoge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengchaoge updated SPARK-21337: --- Attachment: (was: HmsClient.bak) > SQL which has large ‘case when’ expressions may cause code

[jira] [Commented] (SPARK-23814) Couldn't read file with colon in name and new line character in one of the field.

2018-03-30 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420241#comment-16420241 ] Hyukjin Kwon commented on SPARK-23814: -- If I remember this correctly, it was fixed in 2.3.0. Can you

[jira] [Resolved] (SPARK-23727) Support DATE predict push down in parquet

2018-03-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23727. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20851

[jira] [Assigned] (SPARK-23727) Support DATE predict push down in parquet

2018-03-30 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23727: --- Assignee: yucai > Support DATE predict push down in parquet >

[jira] [Updated] (SPARK-23811) FetchFailed comes before Success of same task will cause child stage never succeed

2018-03-30 Thread Li Yuanjian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Yuanjian updated SPARK-23811: Description: This is a bug caused by abnormal scenario describe below: # ShuffleMapTask 1.0

[jira] [Commented] (SPARK-22968) java.lang.IllegalStateException: No current assignment for partition kssh-2

2018-03-30 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420215#comment-16420215 ] Jepson commented on SPARK-22968: Adjust these parameters,keep monitor. “request.timeout.ms“ -> (21:

[jira] [Assigned] (SPARK-23743) IsolatedClientLoader.isSharedClass returns an unindented result against `slf4j` keyword

2018-03-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23743: --- Assignee: Jongyoul Lee > IsolatedClientLoader.isSharedClass returns an unindented result

[jira] [Resolved] (SPARK-23743) IsolatedClientLoader.isSharedClass returns an unindented result against `slf4j` keyword

2018-03-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23743. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20860