[jira] [Assigned] (SPARK-24691) Add new API `supportDataType` in FileFormat

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24691: Assignee: Apache Spark > Add new API `supportDataType` in FileFormat > --

[jira] [Assigned] (SPARK-24691) Add new API `supportDataType` in FileFormat

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24691: Assignee: (was: Apache Spark) > Add new API `supportDataType` in FileFormat > ---

[jira] [Commented] (SPARK-24691) Add new API `supportDataType` in FileFormat

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527260#comment-16527260 ] Apache Spark commented on SPARK-24691: -- User 'gengliangwang' has created a pull req

[jira] [Created] (SPARK-24691) Add new API `supportDataType` in FileFormat

2018-06-28 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-24691: -- Summary: Add new API `supportDataType` in FileFormat Key: SPARK-24691 URL: https://issues.apache.org/jira/browse/SPARK-24691 Project: Spark Issue Type: I

[jira] [Created] (SPARK-24690) Add a new config to control plan stats computation in LogicalRelation

2018-06-28 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-24690: Summary: Add a new config to control plan stats computation in LogicalRelation Key: SPARK-24690 URL: https://issues.apache.org/jira/browse/SPARK-24690 Project

[jira] [Assigned] (SPARK-24535) Fix java version parsing in SparkR

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24535: Assignee: Apache Spark (was: Felix Cheung) > Fix java version parsing in SparkR > --

[jira] [Commented] (SPARK-24535) Fix java version parsing in SparkR

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527234#comment-16527234 ] Apache Spark commented on SPARK-24535: -- User 'felixcheung' has created a pull reque

[jira] [Assigned] (SPARK-24535) Fix java version parsing in SparkR

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24535: Assignee: Felix Cheung (was: Apache Spark) > Fix java version parsing in SparkR > --

[jira] [Updated] (SPARK-24689) java.io.NotSerializableException: org.apache.spark.mllib.clustering.DistributedLDAModel

2018-06-28 Thread konglingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] konglingbo updated SPARK-24689: --- Attachment: @CLZ98635A644[_edx...@e.png > java.io.NotSerializableException: > org.apache.spark.mlli

[jira] [Updated] (SPARK-24689) java.io.NotSerializableException: org.apache.spark.mllib.clustering.DistributedLDAModel

2018-06-28 Thread konglingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] konglingbo updated SPARK-24689: --- Description: scala> val predictionAndLabels=testing.map{case LabeledPoint(label,features)=> | val p

[jira] [Created] (SPARK-24689) java.io.NotSerializableException: org.apache.spark.mllib.clustering.DistributedLDAModel

2018-06-28 Thread konglingbo (JIRA)
konglingbo created SPARK-24689: -- Summary: java.io.NotSerializableException: org.apache.spark.mllib.clustering.DistributedLDAModel Key: SPARK-24689 URL: https://issues.apache.org/jira/browse/SPARK-24689 P

[jira] [Commented] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-06-28 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527204#comment-16527204 ] Ted Yu commented on SPARK-23636: It seems in KafkaDataConsumer#close : {code} def close

[jira] [Updated] (SPARK-24535) Fix java version parsing in SparkR

2018-06-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24535: - Target Version/s: 2.3.2 > Fix java version parsing in SparkR > -

[jira] [Updated] (SPARK-24535) Fix java version parsing in SparkR

2018-06-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24535: - Priority: Blocker (was: Major) > Fix java version parsing in SparkR > -

[jira] [Assigned] (SPARK-24535) Fix java version parsing in SparkR

2018-06-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-24535: Assignee: Felix Cheung > Fix java version parsing in SparkR > ---

[jira] [Assigned] (SPARK-24688) Comments of the example code have some typos

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24688: Assignee: Apache Spark > Comments of the example code have some typos > -

[jira] [Assigned] (SPARK-24688) Comments of the example code have some typos

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24688: Assignee: (was: Apache Spark) > Comments of the example code have some typos > --

[jira] [Issue Comment Deleted] (SPARK-24144) monotonically_increasing_id on streaming dataFrames

2018-06-28 Thread Richard Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Yu updated SPARK-24144: --- Comment: was deleted (was: So do you propose to send the information regarding monotonically_increa

[jira] [Commented] (SPARK-24688) Comments of the example code have some typos

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527134#comment-16527134 ] Apache Spark commented on SPARK-24688: -- User 'uzmijnlm' has created a pull request

[jira] [Reopened] (SPARK-24672) No job is running but there are active tasks

2018-06-28 Thread Deegue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue reopened SPARK-24672: Coditions to this issue: When the amount of data I selected is larger than spark.driver.maxResultSize , It r

[jira] [Updated] (SPARK-24672) No job is running but there are active tasks

2018-06-28 Thread Deegue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-24672: --- Attachment: image4.png > No job is running but there are active tasks >

[jira] [Commented] (SPARK-8659) Spark SQL Thrift Server does NOT honour hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527087#comment-16527087 ] Takeshi Yamamuro commented on SPARK-8659: - I think Spark doesn't support GRANT/RE

[jira] [Commented] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527084#comment-16527084 ] Apache Spark commented on SPARK-24678: -- User 'caneGuy' has created a pull request f

[jira] [Created] (SPARK-24688) Comments of the example code have some typos

2018-06-28 Thread Weizhe Huang (JIRA)
Weizhe Huang created SPARK-24688: Summary: Comments of the example code have some typos Key: SPARK-24688 URL: https://issues.apache.org/jira/browse/SPARK-24688 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-24687) When NoClassDefError thrown during task serialization will cause job hang

2018-06-28 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-24687: - Description: When below exception thrown: {code:java} Exception in thread "dag-scheduler-event-loop" ja

[jira] [Created] (SPARK-24687) When NoClassDefError thrown during task serialization will cause job hang

2018-06-28 Thread zhoukang (JIRA)
zhoukang created SPARK-24687: Summary: When NoClassDefError thrown during task serialization will cause job hang Key: SPARK-24687 URL: https://issues.apache.org/jira/browse/SPARK-24687 Project: Spark

[jira] [Updated] (SPARK-24687) When NoClassDefError thrown during task serialization will cause job hang

2018-06-28 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-24687: - Description: When below exception thrown: {code:java} Exception in thread "dag-scheduler-event-loop" ja

[jira] [Updated] (SPARK-24687) When NoClassDefError thrown during task serialization will cause job hang

2018-06-28 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-24687: - Attachment: hanging-960.png > When NoClassDefError thrown during task serialization will cause job hang

[jira] [Assigned] (SPARK-24680) spark.executorEnv.JAVA_HOME does not take effect in Standalone mode

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24680: Assignee: (was: Apache Spark) > spark.executorEnv.JAVA_HOME does not take effect in S

[jira] [Commented] (SPARK-24680) spark.executorEnv.JAVA_HOME does not take effect in Standalone mode

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527044#comment-16527044 ] Apache Spark commented on SPARK-24680: -- User 'stanzhai' has created a pull request

[jira] [Assigned] (SPARK-24680) spark.executorEnv.JAVA_HOME does not take effect in Standalone mode

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24680: Assignee: Apache Spark > spark.executorEnv.JAVA_HOME does not take effect in Standalone m

[jira] [Commented] (SPARK-24498) Add JDK compiler for runtime codegen

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527036#comment-16527036 ] Takeshi Yamamuro commented on SPARK-24498: -- I added some benchmark results on d

[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527020#comment-16527020 ] Takeshi Yamamuro commented on SPARK-24673: -- I'm not 100% sure though, probably

[jira] [Commented] (SPARK-24643) from_json should accept an aggregate function as schema

2018-06-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527013#comment-16527013 ] Hyukjin Kwon commented on SPARK-24643: -- [~maxgekk] shall we leave this closed for n

[jira] [Created] (SPARK-24686) Provide spark distributions for hadoop-2.8 rather than hadoop-2.7 as releases on apache mirrors

2018-06-28 Thread t oo (JIRA)
t oo created SPARK-24686: Summary: Provide spark distributions for hadoop-2.8 rather than hadoop-2.7 as releases on apache mirrors Key: SPARK-24686 URL: https://issues.apache.org/jira/browse/SPARK-24686 Proje

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

2018-06-28 Thread Richard Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526936#comment-16526936 ] Richard Yu commented on SPARK-18258: {quote} * We need agreement on whether it is wo

[jira] [Resolved] (SPARK-24386) implement continuous processing coalesce(1)

2018-06-28 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24386. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21560 [https://

[jira] [Assigned] (SPARK-24386) implement continuous processing coalesce(1)

2018-06-28 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24386: - Assignee: Jose Torres > implement continuous processing coalesce(1) > -

[jira] [Commented] (SPARK-24662) Structured Streaming should support LIMIT

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526859#comment-16526859 ] Apache Spark commented on SPARK-24662: -- User 'mukulmurthy' has created a pull reque

[jira] [Assigned] (SPARK-24662) Structured Streaming should support LIMIT

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24662: Assignee: (was: Apache Spark) > Structured Streaming should support LIMIT > -

[jira] [Assigned] (SPARK-24662) Structured Streaming should support LIMIT

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24662: Assignee: Apache Spark > Structured Streaming should support LIMIT >

[jira] [Assigned] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

2018-06-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24530: - Assignee: Hyukjin Kwon > Sphinx doesn't render autodoc_docstring_signature correctly (w

[jira] [Commented] (SPARK-24662) Structured Streaming should support LIMIT

2018-06-28 Thread Mukul Murthy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526848#comment-16526848 ] Mukul Murthy commented on SPARK-24662: -- Calling .limit(n) on a DataFrame (or in SQL

[jira] [Commented] (SPARK-24684) DAGScheduler reports the wrong attempt number to the commit coordinator

2018-06-28 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526834#comment-16526834 ] Ryan Blue commented on SPARK-24684: --- Yeah, I just backported this wrong and moved to u

[jira] [Resolved] (SPARK-24679) Download page should not link to unreleased code

2018-06-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24679. Resolution: Fixed Assignee: Luciano Resende > Download page should not link to unrel

[jira] [Commented] (SPARK-23940) High-order function: transform_values(map, function) → map

2018-06-28 Thread Neha Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526808#comment-16526808 ] Neha Patil commented on SPARK-23940: I can work on this one. > High-order function:

[jira] [Resolved] (SPARK-24439) Add distanceMeasure to BisectingKMeans in PySpark

2018-06-28 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-24439. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21557 [https://gi

[jira] [Assigned] (SPARK-24439) Add distanceMeasure to BisectingKMeans in PySpark

2018-06-28 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-24439: Assignee: Huaxin Gao > Add distanceMeasure to BisectingKMeans in PySpark > --

[jira] [Assigned] (SPARK-24685) Adjust release scripts to build all versions for older releases

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24685: Assignee: (was: Apache Spark) > Adjust release scripts to build all versions for olde

[jira] [Commented] (SPARK-24685) Adjust release scripts to build all versions for older releases

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526764#comment-16526764 ] Apache Spark commented on SPARK-24685: -- User 'vanzin' has created a pull request fo

[jira] [Assigned] (SPARK-24685) Adjust release scripts to build all versions for older releases

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24685: Assignee: Apache Spark > Adjust release scripts to build all versions for older releases

[jira] [Created] (SPARK-24685) Adjust release scripts to build all versions for older releases

2018-06-28 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-24685: -- Summary: Adjust release scripts to build all versions for older releases Key: SPARK-24685 URL: https://issues.apache.org/jira/browse/SPARK-24685 Project: Spark

[jira] [Resolved] (SPARK-24670) How to stream only newer files from a folder in Apache Spark?

2018-06-28 Thread Mahbub Murshed (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahbub Murshed resolved SPARK-24670. Resolution: Fixed The problem with count difference was ultimately solved by setting maxF

[jira] [Resolved] (SPARK-24408) Move abs function to math_funcs group

2018-06-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-24408. - Resolution: Fixed Fix Version/s: 2.4.0 Thanks for helping improve our docs :) > Move abs functio

[jira] [Assigned] (SPARK-24408) Move abs function to math_funcs group

2018-06-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-24408: --- Assignee: Jacek Laskowski > Move abs function to math_funcs group > ---

[jira] [Updated] (SPARK-24408) Move abs function to math_funcs group

2018-06-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-24408: Description: A few math function ( {{abs}} )  are is in {{math_funcs}}  group. It should really be. (was:

[jira] [Updated] (SPARK-24408) Move abs function to math_funcs group

2018-06-28 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-24408: Summary: Move abs function to math_funcs group (was: Move abs, bitwiseNOT, isnan, nanvl f

[jira] [Resolved] (SPARK-23120) Add PMML pipeline export support to PySpark

2018-06-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-23120. - Resolution: Fixed Fix Version/s: 2.4.0 > Add PMML pipeline export support to PySpark > --

[jira] [Commented] (SPARK-24684) DAGScheduler reports the wrong attempt number to the commit coordinator

2018-06-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526714#comment-16526714 ] Marcelo Vanzin commented on SPARK-24684: The code still uses the attempt number

[jira] [Assigned] (SPARK-14712) spark.ml LogisticRegressionModel.toString should summarize model

2018-06-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-14712: --- Assignee: Bravo Zhang > spark.ml LogisticRegressionModel.toString should summarize model >

[jira] [Resolved] (SPARK-14712) spark.ml LogisticRegressionModel.toString should summarize model

2018-06-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-14712. - Resolution: Fixed Fix Version/s: 2.4.0 > spark.ml LogisticRegressionModel.toString should summari

[jira] [Commented] (SPARK-24682) from_json / to_json do not handle java.sql.Date inside Maps correctly

2018-06-28 Thread Patrick McGloin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526687#comment-16526687 ] Patrick McGloin commented on SPARK-24682: - I would like to work on this. > from

[jira] [Resolved] (SPARK-24684) DAGScheduler reports the wrong attempt number to the commit coordinator

2018-06-28 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved SPARK-24684. --- Resolution: Not A Problem Closing this. In master, the attempt number is still used. Looks like this

[jira] [Created] (SPARK-24684) DAGScheduler reports the wrong attempt number to the commit coordinator

2018-06-28 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-24684: - Summary: DAGScheduler reports the wrong attempt number to the commit coordinator Key: SPARK-24684 URL: https://issues.apache.org/jira/browse/SPARK-24684 Project: Spark

[jira] [Commented] (SPARK-24683) SparkLauncher.NO_RESOURCE doesn't work with Java applications

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526628#comment-16526628 ] Apache Spark commented on SPARK-24683: -- User 'mccheah' has created a pull request f

[jira] [Assigned] (SPARK-24683) SparkLauncher.NO_RESOURCE doesn't work with Java applications

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24683: Assignee: Apache Spark > SparkLauncher.NO_RESOURCE doesn't work with Java applications >

[jira] [Assigned] (SPARK-24683) SparkLauncher.NO_RESOURCE doesn't work with Java applications

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24683: Assignee: (was: Apache Spark) > SparkLauncher.NO_RESOURCE doesn't work with Java appl

[jira] [Created] (SPARK-24683) SparkLauncher.NO_RESOURCE doesn't work with Java applications

2018-06-28 Thread Matt Cheah (JIRA)
Matt Cheah created SPARK-24683: -- Summary: SparkLauncher.NO_RESOURCE doesn't work with Java applications Key: SPARK-24683 URL: https://issues.apache.org/jira/browse/SPARK-24683 Project: Spark Is

[jira] [Assigned] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24530: Assignee: (was: Apache Spark) > Sphinx doesn't render autodoc_docstring_signature cor

[jira] [Commented] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526560#comment-16526560 ] Apache Spark commented on SPARK-24530: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24530: Assignee: Apache Spark > Sphinx doesn't render autodoc_docstring_signature correctly (wit

[jira] [Commented] (SPARK-24642) Add a function which infers schema from a JSON column

2018-06-28 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526547#comment-16526547 ] Maxim Gekk commented on SPARK-24642: > I think this is too complicated and unpredict

[jira] [Created] (SPARK-24682) from_json / to_json do not handle java.sql.Date inside Maps correctly

2018-06-28 Thread Patrick McGloin (JIRA)
Patrick McGloin created SPARK-24682: --- Summary: from_json / to_json do not handle java.sql.Date inside Maps correctly Key: SPARK-24682 URL: https://issues.apache.org/jira/browse/SPARK-24682 Project:

[jira] [Updated] (SPARK-24681) Cannot create a view from a table when a nested column name contains ':'

2018-06-28 Thread Adrian Ionescu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Ionescu updated SPARK-24681: --- Description: Here's a patch that reproduces the issue:  {code:java} diff --git a/sql/hive/s

[jira] [Created] (SPARK-24681) Cannot create a view from a table when a nested column name contains ':'

2018-06-28 Thread Adrian Ionescu (JIRA)
Adrian Ionescu created SPARK-24681: -- Summary: Cannot create a view from a table when a nested column name contains ':' Key: SPARK-24681 URL: https://issues.apache.org/jira/browse/SPARK-24681 Project:

[jira] [Updated] (SPARK-24020) Sort-merge join inner range optimization

2018-06-28 Thread Petar Zecevic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Petar Zecevic updated SPARK-24020: -- Description: The problem we are solving is the case where you have two big tables partitioned

[jira] [Updated] (SPARK-24020) Sort-merge join inner range optimization

2018-06-28 Thread Petar Zecevic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Petar Zecevic updated SPARK-24020: -- Description: The problem we are solving is the case where you have two big tables partitioned

[jira] [Updated] (SPARK-24020) Sort-merge join inner range optimization

2018-06-28 Thread Petar Zecevic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Petar Zecevic updated SPARK-24020: -- Description: The problem we are solving is the case where you have two big tables partitioned

[jira] [Updated] (SPARK-24020) Sort-merge join inner range optimization

2018-06-28 Thread Petar Zecevic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Petar Zecevic updated SPARK-24020: -- Description: The problem we are solving is the case where you have two big tables partitioned

[jira] [Updated] (SPARK-24020) Sort-merge join inner range optimization

2018-06-28 Thread Petar Zecevic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Petar Zecevic updated SPARK-24020: -- Attachment: SMJ-innerRange-PR24020-designDoc.pdf > Sort-merge join inner range optimization >

[jira] [Created] (SPARK-24680) spark.executorEnv.JAVA_HOME does not take effect in Standalone mode

2018-06-28 Thread StanZhai (JIRA)
StanZhai created SPARK-24680: Summary: spark.executorEnv.JAVA_HOME does not take effect in Standalone mode Key: SPARK-24680 URL: https://issues.apache.org/jira/browse/SPARK-24680 Project: Spark

[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String

2018-06-28 Thread Antonio Murgia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526307#comment-16526307 ] Antonio Murgia commented on SPARK-24673: Looks doable. Should I go with a method

[jira] [Created] (SPARK-24679) Download page should not link to unreleased code

2018-06-28 Thread Luciano Resende (JIRA)
Luciano Resende created SPARK-24679: --- Summary: Download page should not link to unreleased code Key: SPARK-24679 URL: https://issues.apache.org/jira/browse/SPARK-24679 Project: Spark Issue

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-06-28 Thread Jackey Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526264#comment-16526264 ] Jackey Lee commented on SPARK-24630: Main Goal: * SQL API for StructStreaming Benef

[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526118#comment-16526118 ] Takeshi Yamamuro commented on SPARK-24673: -- It makes sense. Can you make a pr?

[jira] [Commented] (SPARK-14834) Force adding doc for new api in pyspark with @since annotation

2018-06-28 Thread Alexander Gorokhov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526102#comment-16526102 ] Alexander Gorokhov commented on SPARK-14834: So, basically, this is about to

[jira] [Resolved] (SPARK-24674) Spark on Kubernetes BLAS performance

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-24674. -- Resolution: Invalid > Spark on Kubernetes BLAS performance > -

[jira] [Commented] (SPARK-24674) Spark on Kubernetes BLAS performance

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526101#comment-16526101 ] Takeshi Yamamuro commented on SPARK-24674: -- You should first ask in the spark-u

[jira] [Assigned] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24678: Assignee: (was: Apache Spark) > We should use 'PROCESS_LOCAL' first for Spark-Streami

[jira] [Assigned] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24678: Assignee: Apache Spark > We should use 'PROCESS_LOCAL' first for Spark-Streaming > --

[jira] [Commented] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526078#comment-16526078 ] Apache Spark commented on SPARK-24678: -- User 'sharkdtu' has created a pull request

[jira] [Updated] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

2018-06-28 Thread sharkd tu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sharkd tu updated SPARK-24678: -- Description: Currently, `BlockRDD.getPreferredLocations`  only get hosts info of blocks, which result

[jira] [Commented] (SPARK-24144) monotonically_increasing_id on streaming dataFrames

2018-06-28 Thread Richard Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526075#comment-16526075 ] Richard Yu commented on SPARK-24144: So do you propose to send the information regar

[jira] [Resolved] (SPARK-24672) No job is running but there are active tasks

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-24672. -- Resolution: Invalid > No job is running but there are active tasks > -

[jira] [Commented] (SPARK-24672) No job is running but there are active tasks

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526069#comment-16526069 ] Takeshi Yamamuro commented on SPARK-24672: -- You should first ask in the spark-u

[jira] [Commented] (SPARK-24662) Structured Streaming should support LIMIT

2018-06-28 Thread Richard Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526065#comment-16526065 ] Richard Yu commented on SPARK-24662: Just to be clear on the function of the limit o

[jira] [Created] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

2018-06-28 Thread sharkd tu (JIRA)
sharkd tu created SPARK-24678: - Summary: We should use 'PROCESS_LOCAL' first for Spark-Streaming Key: SPARK-24678 URL: https://issues.apache.org/jira/browse/SPARK-24678 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-24676) Project required data from parsed data when csvColumnPruning disabled

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24676: Assignee: (was: Apache Spark) > Project required data from parsed data when csvColumn

[jira] [Assigned] (SPARK-24676) Project required data from parsed data when csvColumnPruning disabled

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24676: Assignee: Apache Spark > Project required data from parsed data when csvColumnPruning dis

[jira] [Commented] (SPARK-24676) Project required data from parsed data when csvColumnPruning disabled

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16526057#comment-16526057 ] Apache Spark commented on SPARK-24676: -- User 'maropu' has created a pull request fo

  1   2   >