[jira] [Comment Edited] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

2018-08-04 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569363#comment-16569363 ] Mridul Muralidharan edited comment on SPARK-24375 at 8/5/18 4:42 AM: -

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

2018-08-04 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569363#comment-16569363 ] Mridul Muralidharan commented on SPARK-24375: - {quote} We've thought hard on the issue and

[jira] [Resolved] (SPARK-23911) High-order function: aggregate(array, initialState S, inputFunction, outputFunction) → R

2018-08-04 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-23911. --- Resolution: Fixed Assignee: Takuya Ueshin (was: Herman van Hovell) Fix

[jira] [Commented] (SPARK-25025) Remove the default value of isAll in INTERSECT/EXCEPT

2018-08-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569307#comment-16569307 ] Apache Spark commented on SPARK-25025: -- User 'dilipbiswal' has created a pull request for this

[jira] [Assigned] (SPARK-25025) Remove the default value of isAll in INTERSECT/EXCEPT

2018-08-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25025: Assignee: Dilip Biswal (was: Apache Spark) > Remove the default value of isAll in

[jira] [Assigned] (SPARK-25025) Remove the default value of isAll in INTERSECT/EXCEPT

2018-08-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25025: Assignee: Apache Spark (was: Dilip Biswal) > Remove the default value of isAll in

[jira] [Created] (SPARK-25027) LegacyAccumulatorWrapper test fails in Scala 2.12

2018-08-04 Thread Sean Owen (JIRA)
Sean Owen created SPARK-25027: - Summary: LegacyAccumulatorWrapper test fails in Scala 2.12 Key: SPARK-25027 URL: https://issues.apache.org/jira/browse/SPARK-25027 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-25026) Binary releases don't contain Kafka integration modules

2018-08-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25026: -- Affects Version/s: (was: 2.3.1) (was: 2.2.2) Target Version/s:

[jira] [Updated] (SPARK-25026) Binary releases don't contain Kafka integration modules

2018-08-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25026: -- Target Version/s: 2.4.0 (was: 2.3.2, 2.4.0) Priority: Major (was: Blocker) Yeah, I felt

[jira] [Commented] (SPARK-25026) Binary releases don't contain Kafka integration modules

2018-08-04 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569283#comment-16569283 ] Cody Koeninger commented on SPARK-25026: I don't think distributions have ever included the

[jira] [Resolved] (SPARK-25015) Update Hadoop 2.7 to 2.7.7

2018-08-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25015. --- Resolution: Fixed Fix Version/s: 2.3.2 2.4.0 Issue resolved by pull

[jira] [Created] (SPARK-25026) Binary releases don't contain Kafka integration modules

2018-08-04 Thread Sean Owen (JIRA)
Sean Owen created SPARK-25026: - Summary: Binary releases don't contain Kafka integration modules Key: SPARK-25026 URL: https://issues.apache.org/jira/browse/SPARK-25026 Project: Spark Issue

[jira] [Updated] (SPARK-24940) Coalesce and Repartition Hint for SQL Queries

2018-08-04 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-24940: --- Description: Many Spark SQL users in my company have asked for a way to control the number of

[jira] [Updated] (SPARK-24940) Coalesce and Repartition Hint for SQL Queries

2018-08-04 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-24940: --- Description: Many Spark SQL users in my company have asked for a way to control the number of

[jira] [Updated] (SPARK-24940) Coalesce and Repartition Hint for SQL Queries

2018-08-04 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-24940: --- Summary: Coalesce and Repartition Hint for SQL Queries (was: Coalesce Hint for SQL Queries) >

[jira] [Resolved] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

2018-08-04 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger resolved SPARK-24987. Resolution: Fixed Fix Version/s: 2.4.0 > Kafka Cached Consumer Leaking File

[jira] [Assigned] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

2018-08-04 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger reassigned SPARK-24987: -- Assignee: Yuval Itzchakov > Kafka Cached Consumer Leaking File Descriptors >

[jira] [Commented] (SPARK-24940) Coalesce Hint for SQL Queries

2018-08-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569267#comment-16569267 ] Apache Spark commented on SPARK-24940: -- User 'jzhuge' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25001) Fix build miscellaneous warnings

2018-08-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25001: - Assignee: Hyukjin Kwon > Fix build miscellaneous warnings > >

[jira] [Resolved] (SPARK-25001) Fix build miscellaneous warnings

2018-08-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25001. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21975

[jira] [Updated] (SPARK-24926) Ensure numCores is used consistently in all netty configuration (driver and executors)

2018-08-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24926: -- Priority: Minor (was: Major) > Ensure numCores is used consistently in all netty configuration

[jira] [Resolved] (SPARK-24926) Ensure numCores is used consistently in all netty configuration (driver and executors)

2018-08-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24926. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21885

[jira] [Assigned] (SPARK-24926) Ensure numCores is used consistently in all netty configuration (driver and executors)

2018-08-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-24926: - Assignee: Nihar Sheth (was: Imran Rashid) > Ensure numCores is used consistently in all netty

[jira] [Assigned] (SPARK-24926) Ensure numCores is used consistently in all netty configuration (driver and executors)

2018-08-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-24926: - Assignee: Imran Rashid > Ensure numCores is used consistently in all netty configuration

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

2018-08-04 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569202#comment-16569202 ] Jiang Xingbo commented on SPARK-24375: -- [~mridulm80] You are right that now we are not able to

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

2018-08-04 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569136#comment-16569136 ] Stavros Kontopoulos commented on SPARK-25024: - Several security features require a secret

[jira] [Created] (SPARK-25025) Remove the default value of isAll in INTERSECT/EXCEPT

2018-08-04 Thread Xiao Li (JIRA)
Xiao Li created SPARK-25025: --- Summary: Remove the default value of isAll in INTERSECT/EXCEPT Key: SPARK-25025 URL: https://issues.apache.org/jira/browse/SPARK-25025 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24987) Kafka Cached Consumer Leaking File Descriptors

2018-08-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569112#comment-16569112 ] Apache Spark commented on SPARK-24987: -- User 'YuvalItzchakov' has created a pull request for this

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-04 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569109#comment-16569109 ] Felix Cheung commented on SPARK-24924: --   I tend to agree that we shouldn't "magically" remap

[jira] [Commented] (SPARK-23911) High-order function: aggregate(array, initialState S, inputFunction, outputFunction) → R

2018-08-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569108#comment-16569108 ] Xiao Li commented on SPARK-23911: - Sounds good to me. Just changed reduce to aggregate > High-order

[jira] [Updated] (SPARK-23911) High-order function: aggregate(array, initialState S, inputFunction, outputFunction) → R

2018-08-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23911: Description: Ref: https://prestodb.io/docs/current/functions/array.html Returns a single value reduced

[jira] [Updated] (SPARK-23911) High-order function: aggregate(array, initialState S, inputFunction, outputFunction) → R

2018-08-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23911: Summary: High-order function: aggregate(array, initialState S, inputFunction, outputFunction) → R (was:

[jira] [Resolved] (SPARK-23909) High-order function: filter(array, function) → array

2018-08-04 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-23909. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21965

[jira] [Assigned] (SPARK-23909) High-order function: filter(array, function) → array

2018-08-04 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-23909: - Assignee: Takuya Ueshin (was: Herman van Hovell) > High-order function: filter(array,

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569104#comment-16569104 ] Hyukjin Kwon commented on SPARK-24924: -- For fully qualifed path, we already could specify like

[jira] [Commented] (SPARK-14220) Build and test Spark against Scala 2.12

2018-08-04 Thread Rifaqat Shah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569088#comment-16569088 ] Rifaqat Shah commented on SPARK-14220: -- great! congratulations and thanks.. > Build and test Spark

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-04 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569085#comment-16569085 ] Wenchen Fan commented on SPARK-24924: - when the short name conflicts, I feel it's better to pick the

[jira] [Resolved] (SPARK-24940) Coalesce Hint for SQL Queries

2018-08-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24940. - Resolution: Fixed Assignee: John Zhuge Fix Version/s: 2.4.0 > Coalesce Hint for SQL

[jira] [Resolved] (SPARK-24722) Column-based API for pivoting

2018-08-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24722. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21699

[jira] [Assigned] (SPARK-24722) Column-based API for pivoting

2018-08-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24722: Assignee: Maxim Gekk > Column-based API for pivoting > - > >