[jira] [Updated] (SPARK-24689) java.io.NotSerializableException: org.apache.spark.mllib.clustering.DistributedLDAModel

2018-06-28 Thread konglingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] konglingbo updated SPARK-24689: --- Attachment: @CLZ98635A644[_edx...@e.png > java.io.NotSerializableException: >

[jira] [Updated] (SPARK-24689) java.io.NotSerializableException: org.apache.spark.mllib.clustering.DistributedLDAModel

2018-06-28 Thread konglingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] konglingbo updated SPARK-24689: --- Description: scala> val predictionAndLabels=testing.map{case LabeledPoint(label,features)=> | val

[jira] [Created] (SPARK-24689) java.io.NotSerializableException: org.apache.spark.mllib.clustering.DistributedLDAModel

2018-06-28 Thread konglingbo (JIRA)
konglingbo created SPARK-24689: -- Summary: java.io.NotSerializableException: org.apache.spark.mllib.clustering.DistributedLDAModel Key: SPARK-24689 URL: https://issues.apache.org/jira/browse/SPARK-24689

[jira] [Commented] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2018-06-28 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527204#comment-16527204 ] Ted Yu commented on SPARK-23636: It seems in KafkaDataConsumer#close : {code} def close(): Unit =

[jira] [Updated] (SPARK-24535) Fix java version parsing in SparkR

2018-06-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24535: - Target Version/s: 2.3.2 > Fix java version parsing in SparkR >

[jira] [Updated] (SPARK-24535) Fix java version parsing in SparkR

2018-06-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-24535: - Priority: Blocker (was: Major) > Fix java version parsing in SparkR >

[jira] [Assigned] (SPARK-24535) Fix java version parsing in SparkR

2018-06-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-24535: Assignee: Felix Cheung > Fix java version parsing in SparkR >

[jira] [Assigned] (SPARK-24688) Comments of the example code have some typos

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24688: Assignee: Apache Spark > Comments of the example code have some typos >

[jira] [Assigned] (SPARK-24688) Comments of the example code have some typos

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24688: Assignee: (was: Apache Spark) > Comments of the example code have some typos >

[jira] [Issue Comment Deleted] (SPARK-24144) monotonically_increasing_id on streaming dataFrames

2018-06-28 Thread Richard Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Yu updated SPARK-24144: --- Comment: was deleted (was: So do you propose to send the information regarding

[jira] [Commented] (SPARK-24688) Comments of the example code have some typos

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527134#comment-16527134 ] Apache Spark commented on SPARK-24688: -- User 'uzmijnlm' has created a pull request for this issue:

[jira] [Reopened] (SPARK-24672) No job is running but there are active tasks

2018-06-28 Thread Deegue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue reopened SPARK-24672: Coditions to this issue: When the amount of data I selected is larger than spark.driver.maxResultSize , It

[jira] [Updated] (SPARK-24672) No job is running but there are active tasks

2018-06-28 Thread Deegue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deegue updated SPARK-24672: --- Attachment: image4.png > No job is running but there are active tasks >

[jira] [Commented] (SPARK-8659) Spark SQL Thrift Server does NOT honour hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527087#comment-16527087 ] Takeshi Yamamuro commented on SPARK-8659: - I think Spark doesn't support GRANT/REVOKE now. >

[jira] [Commented] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527084#comment-16527084 ] Apache Spark commented on SPARK-24678: -- User 'caneGuy' has created a pull request for this issue:

[jira] [Created] (SPARK-24688) Comments of the example code have some typos

2018-06-28 Thread Weizhe Huang (JIRA)
Weizhe Huang created SPARK-24688: Summary: Comments of the example code have some typos Key: SPARK-24688 URL: https://issues.apache.org/jira/browse/SPARK-24688 Project: Spark Issue Type:

[jira] [Updated] (SPARK-24687) When NoClassDefError thrown during task serialization will cause job hang

2018-06-28 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-24687: - Description: When below exception thrown: {code:java} Exception in thread "dag-scheduler-event-loop"

[jira] [Created] (SPARK-24687) When NoClassDefError thrown during task serialization will cause job hang

2018-06-28 Thread zhoukang (JIRA)
zhoukang created SPARK-24687: Summary: When NoClassDefError thrown during task serialization will cause job hang Key: SPARK-24687 URL: https://issues.apache.org/jira/browse/SPARK-24687 Project: Spark

[jira] [Updated] (SPARK-24687) When NoClassDefError thrown during task serialization will cause job hang

2018-06-28 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-24687: - Description: When below exception thrown: {code:java} Exception in thread "dag-scheduler-event-loop"

[jira] [Updated] (SPARK-24687) When NoClassDefError thrown during task serialization will cause job hang

2018-06-28 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-24687: - Attachment: hanging-960.png > When NoClassDefError thrown during task serialization will cause job hang

[jira] [Assigned] (SPARK-24680) spark.executorEnv.JAVA_HOME does not take effect in Standalone mode

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24680: Assignee: (was: Apache Spark) > spark.executorEnv.JAVA_HOME does not take effect in

[jira] [Commented] (SPARK-24680) spark.executorEnv.JAVA_HOME does not take effect in Standalone mode

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527044#comment-16527044 ] Apache Spark commented on SPARK-24680: -- User 'stanzhai' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24680) spark.executorEnv.JAVA_HOME does not take effect in Standalone mode

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24680: Assignee: Apache Spark > spark.executorEnv.JAVA_HOME does not take effect in Standalone

[jira] [Commented] (SPARK-24498) Add JDK compiler for runtime codegen

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527036#comment-16527036 ] Takeshi Yamamuro commented on SPARK-24498: -- I added some benchmark results on different

[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527020#comment-16527020 ] Takeshi Yamamuro commented on SPARK-24673: -- I'm not 100% sure though, probably I think we

[jira] [Commented] (SPARK-24643) from_json should accept an aggregate function as schema

2018-06-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527013#comment-16527013 ] Hyukjin Kwon commented on SPARK-24643: -- [~maxgekk] shall we leave this closed for now? > from_json

[jira] [Created] (SPARK-24686) Provide spark distributions for hadoop-2.8 rather than hadoop-2.7 as releases on apache mirrors

2018-06-28 Thread t oo (JIRA)
t oo created SPARK-24686: Summary: Provide spark distributions for hadoop-2.8 rather than hadoop-2.7 as releases on apache mirrors Key: SPARK-24686 URL: https://issues.apache.org/jira/browse/SPARK-24686

[jira] [Commented] (SPARK-18258) Sinks need access to offset representation

2018-06-28 Thread Richard Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526936#comment-16526936 ] Richard Yu commented on SPARK-18258: {quote} * We need agreement on whether it is worth making a

[jira] [Resolved] (SPARK-24386) implement continuous processing coalesce(1)

2018-06-28 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-24386. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21560

[jira] [Assigned] (SPARK-24386) implement continuous processing coalesce(1)

2018-06-28 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24386: - Assignee: Jose Torres > implement continuous processing coalesce(1) >

[jira] [Commented] (SPARK-24662) Structured Streaming should support LIMIT

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526859#comment-16526859 ] Apache Spark commented on SPARK-24662: -- User 'mukulmurthy' has created a pull request for this

[jira] [Assigned] (SPARK-24662) Structured Streaming should support LIMIT

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24662: Assignee: (was: Apache Spark) > Structured Streaming should support LIMIT >

[jira] [Assigned] (SPARK-24662) Structured Streaming should support LIMIT

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24662: Assignee: Apache Spark > Structured Streaming should support LIMIT >

[jira] [Assigned] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

2018-06-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24530: - Assignee: Hyukjin Kwon > Sphinx doesn't render autodoc_docstring_signature correctly

[jira] [Commented] (SPARK-24662) Structured Streaming should support LIMIT

2018-06-28 Thread Mukul Murthy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526848#comment-16526848 ] Mukul Murthy commented on SPARK-24662: -- Calling .limit(n) on a DataFrame (or in SQL, SELECT ...

[jira] [Commented] (SPARK-24684) DAGScheduler reports the wrong attempt number to the commit coordinator

2018-06-28 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526834#comment-16526834 ] Ryan Blue commented on SPARK-24684: --- Yeah, I just backported this wrong and moved to using unique ids

[jira] [Resolved] (SPARK-24679) Download page should not link to unreleased code

2018-06-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24679. Resolution: Fixed Assignee: Luciano Resende > Download page should not link to

[jira] [Commented] (SPARK-23940) High-order function: transform_values(map, function) → map

2018-06-28 Thread Neha Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526808#comment-16526808 ] Neha Patil commented on SPARK-23940: I can work on this one. > High-order function:

[jira] [Resolved] (SPARK-24439) Add distanceMeasure to BisectingKMeans in PySpark

2018-06-28 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-24439. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21557

[jira] [Assigned] (SPARK-24439) Add distanceMeasure to BisectingKMeans in PySpark

2018-06-28 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-24439: Assignee: Huaxin Gao > Add distanceMeasure to BisectingKMeans in PySpark >

[jira] [Assigned] (SPARK-24685) Adjust release scripts to build all versions for older releases

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24685: Assignee: (was: Apache Spark) > Adjust release scripts to build all versions for

[jira] [Commented] (SPARK-24685) Adjust release scripts to build all versions for older releases

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526764#comment-16526764 ] Apache Spark commented on SPARK-24685: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24685) Adjust release scripts to build all versions for older releases

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24685: Assignee: Apache Spark > Adjust release scripts to build all versions for older releases

[jira] [Created] (SPARK-24685) Adjust release scripts to build all versions for older releases

2018-06-28 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-24685: -- Summary: Adjust release scripts to build all versions for older releases Key: SPARK-24685 URL: https://issues.apache.org/jira/browse/SPARK-24685 Project: Spark

[jira] [Resolved] (SPARK-24670) How to stream only newer files from a folder in Apache Spark?

2018-06-28 Thread Mahbub Murshed (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahbub Murshed resolved SPARK-24670. Resolution: Fixed The problem with count difference was ultimately solved by setting

[jira] [Resolved] (SPARK-24408) Move abs function to math_funcs group

2018-06-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-24408. - Resolution: Fixed Fix Version/s: 2.4.0 Thanks for helping improve our docs :) > Move abs

[jira] [Assigned] (SPARK-24408) Move abs function to math_funcs group

2018-06-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-24408: --- Assignee: Jacek Laskowski > Move abs function to math_funcs group >

[jira] [Updated] (SPARK-24408) Move abs function to math_funcs group

2018-06-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-24408: Description: A few math function ( {{abs}} )  are is in {{math_funcs}}  group. It should really be. (was:

[jira] [Updated] (SPARK-24408) Move abs function to math_funcs group

2018-06-28 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-24408: Summary: Move abs function to math_funcs group (was: Move abs, bitwiseNOT, isnan, nanvl

[jira] [Resolved] (SPARK-23120) Add PMML pipeline export support to PySpark

2018-06-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-23120. - Resolution: Fixed Fix Version/s: 2.4.0 > Add PMML pipeline export support to PySpark >

[jira] [Commented] (SPARK-24684) DAGScheduler reports the wrong attempt number to the commit coordinator

2018-06-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526714#comment-16526714 ] Marcelo Vanzin commented on SPARK-24684: The code still uses the attempt number currently (and

[jira] [Assigned] (SPARK-14712) spark.ml LogisticRegressionModel.toString should summarize model

2018-06-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-14712: --- Assignee: Bravo Zhang > spark.ml LogisticRegressionModel.toString should summarize model >

[jira] [Resolved] (SPARK-14712) spark.ml LogisticRegressionModel.toString should summarize model

2018-06-28 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-14712. - Resolution: Fixed Fix Version/s: 2.4.0 > spark.ml LogisticRegressionModel.toString should

[jira] [Commented] (SPARK-24682) from_json / to_json do not handle java.sql.Date inside Maps correctly

2018-06-28 Thread Patrick McGloin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526687#comment-16526687 ] Patrick McGloin commented on SPARK-24682: - I would like to work on this. > from_json / to_json

[jira] [Resolved] (SPARK-24684) DAGScheduler reports the wrong attempt number to the commit coordinator

2018-06-28 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved SPARK-24684. --- Resolution: Not A Problem Closing this. In master, the attempt number is still used. Looks like

[jira] [Created] (SPARK-24684) DAGScheduler reports the wrong attempt number to the commit coordinator

2018-06-28 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-24684: - Summary: DAGScheduler reports the wrong attempt number to the commit coordinator Key: SPARK-24684 URL: https://issues.apache.org/jira/browse/SPARK-24684 Project: Spark

[jira] [Commented] (SPARK-24683) SparkLauncher.NO_RESOURCE doesn't work with Java applications

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526628#comment-16526628 ] Apache Spark commented on SPARK-24683: -- User 'mccheah' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24683) SparkLauncher.NO_RESOURCE doesn't work with Java applications

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24683: Assignee: Apache Spark > SparkLauncher.NO_RESOURCE doesn't work with Java applications >

[jira] [Assigned] (SPARK-24683) SparkLauncher.NO_RESOURCE doesn't work with Java applications

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24683: Assignee: (was: Apache Spark) > SparkLauncher.NO_RESOURCE doesn't work with Java

[jira] [Created] (SPARK-24683) SparkLauncher.NO_RESOURCE doesn't work with Java applications

2018-06-28 Thread Matt Cheah (JIRA)
Matt Cheah created SPARK-24683: -- Summary: SparkLauncher.NO_RESOURCE doesn't work with Java applications Key: SPARK-24683 URL: https://issues.apache.org/jira/browse/SPARK-24683 Project: Spark

[jira] [Assigned] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24530: Assignee: (was: Apache Spark) > Sphinx doesn't render autodoc_docstring_signature

[jira] [Commented] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526560#comment-16526560 ] Apache Spark commented on SPARK-24530: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24530: Assignee: Apache Spark > Sphinx doesn't render autodoc_docstring_signature correctly

[jira] [Commented] (SPARK-24642) Add a function which infers schema from a JSON column

2018-06-28 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526547#comment-16526547 ] Maxim Gekk commented on SPARK-24642: > I think this is too complicated and unpredictable. ok. I

[jira] [Created] (SPARK-24682) from_json / to_json do not handle java.sql.Date inside Maps correctly

2018-06-28 Thread Patrick McGloin (JIRA)
Patrick McGloin created SPARK-24682: --- Summary: from_json / to_json do not handle java.sql.Date inside Maps correctly Key: SPARK-24682 URL: https://issues.apache.org/jira/browse/SPARK-24682 Project:

[jira] [Updated] (SPARK-24681) Cannot create a view from a table when a nested column name contains ':'

2018-06-28 Thread Adrian Ionescu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Ionescu updated SPARK-24681: --- Description: Here's a patch that reproduces the issue:  {code:java} diff --git

[jira] [Created] (SPARK-24681) Cannot create a view from a table when a nested column name contains ':'

2018-06-28 Thread Adrian Ionescu (JIRA)
Adrian Ionescu created SPARK-24681: -- Summary: Cannot create a view from a table when a nested column name contains ':' Key: SPARK-24681 URL: https://issues.apache.org/jira/browse/SPARK-24681

[jira] [Updated] (SPARK-24020) Sort-merge join inner range optimization

2018-06-28 Thread Petar Zecevic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Petar Zecevic updated SPARK-24020: -- Description: The problem we are solving is the case where you have two big tables

[jira] [Updated] (SPARK-24020) Sort-merge join inner range optimization

2018-06-28 Thread Petar Zecevic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Petar Zecevic updated SPARK-24020: -- Description: The problem we are solving is the case where you have two big tables

[jira] [Updated] (SPARK-24020) Sort-merge join inner range optimization

2018-06-28 Thread Petar Zecevic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Petar Zecevic updated SPARK-24020: -- Description: The problem we are solving is the case where you have two big tables

[jira] [Updated] (SPARK-24020) Sort-merge join inner range optimization

2018-06-28 Thread Petar Zecevic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Petar Zecevic updated SPARK-24020: -- Description: The problem we are solving is the case where you have two big tables

[jira] [Updated] (SPARK-24020) Sort-merge join inner range optimization

2018-06-28 Thread Petar Zecevic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Petar Zecevic updated SPARK-24020: -- Attachment: SMJ-innerRange-PR24020-designDoc.pdf > Sort-merge join inner range optimization >

[jira] [Created] (SPARK-24680) spark.executorEnv.JAVA_HOME does not take effect in Standalone mode

2018-06-28 Thread StanZhai (JIRA)
StanZhai created SPARK-24680: Summary: spark.executorEnv.JAVA_HOME does not take effect in Standalone mode Key: SPARK-24680 URL: https://issues.apache.org/jira/browse/SPARK-24680 Project: Spark

[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String

2018-06-28 Thread Antonio Murgia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526307#comment-16526307 ] Antonio Murgia commented on SPARK-24673: Looks doable. Should I go with a method overload,

[jira] [Created] (SPARK-24679) Download page should not link to unreleased code

2018-06-28 Thread Luciano Resende (JIRA)
Luciano Resende created SPARK-24679: --- Summary: Download page should not link to unreleased code Key: SPARK-24679 URL: https://issues.apache.org/jira/browse/SPARK-24679 Project: Spark Issue

[jira] [Commented] (SPARK-24630) SPIP: Support SQLStreaming in Spark

2018-06-28 Thread Jackey Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526264#comment-16526264 ] Jackey Lee commented on SPARK-24630: Main Goal: * SQL API for StructStreaming Benefits: * Users,

[jira] [Commented] (SPARK-24673) scala sql function from_utc_timestamp second argument could be Column instead of String

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526118#comment-16526118 ] Takeshi Yamamuro commented on SPARK-24673: -- It makes sense. Can you make a pr? > scala sql

[jira] [Commented] (SPARK-14834) Force adding doc for new api in pyspark with @since annotation

2018-06-28 Thread Alexander Gorokhov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526102#comment-16526102 ] Alexander Gorokhov commented on SPARK-14834: So, basically, this is about to make "since"

[jira] [Resolved] (SPARK-24674) Spark on Kubernetes BLAS performance

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-24674. -- Resolution: Invalid > Spark on Kubernetes BLAS performance >

[jira] [Commented] (SPARK-24674) Spark on Kubernetes BLAS performance

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526101#comment-16526101 ] Takeshi Yamamuro commented on SPARK-24674: -- You should first ask in the spark-user mailing

[jira] [Assigned] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24678: Assignee: (was: Apache Spark) > We should use 'PROCESS_LOCAL' first for

[jira] [Assigned] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24678: Assignee: Apache Spark > We should use 'PROCESS_LOCAL' first for Spark-Streaming >

[jira] [Commented] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526078#comment-16526078 ] Apache Spark commented on SPARK-24678: -- User 'sharkdtu' has created a pull request for this issue:

[jira] [Updated] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

2018-06-28 Thread sharkd tu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sharkd tu updated SPARK-24678: -- Description: Currently, `BlockRDD.getPreferredLocations`  only get hosts info of blocks, which

[jira] [Commented] (SPARK-24144) monotonically_increasing_id on streaming dataFrames

2018-06-28 Thread Richard Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526075#comment-16526075 ] Richard Yu commented on SPARK-24144: So do you propose to send the information regarding

[jira] [Resolved] (SPARK-24672) No job is running but there are active tasks

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-24672. -- Resolution: Invalid > No job is running but there are active tasks >

[jira] [Commented] (SPARK-24672) No job is running but there are active tasks

2018-06-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526069#comment-16526069 ] Takeshi Yamamuro commented on SPARK-24672: -- You should first ask in the spark-user mailing

[jira] [Commented] (SPARK-24662) Structured Streaming should support LIMIT

2018-06-28 Thread Richard Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526065#comment-16526065 ] Richard Yu commented on SPARK-24662: Just to be clear on the function of the limit operator, could

[jira] [Created] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

2018-06-28 Thread sharkd tu (JIRA)
sharkd tu created SPARK-24678: - Summary: We should use 'PROCESS_LOCAL' first for Spark-Streaming Key: SPARK-24678 URL: https://issues.apache.org/jira/browse/SPARK-24678 Project: Spark Issue

[jira] [Assigned] (SPARK-24676) Project required data from parsed data when csvColumnPruning disabled

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24676: Assignee: (was: Apache Spark) > Project required data from parsed data when

[jira] [Assigned] (SPARK-24676) Project required data from parsed data when csvColumnPruning disabled

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24676: Assignee: Apache Spark > Project required data from parsed data when csvColumnPruning

[jira] [Commented] (SPARK-24676) Project required data from parsed data when csvColumnPruning disabled

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526057#comment-16526057 ] Apache Spark commented on SPARK-24676: -- User 'maropu' has created a pull request for this issue:

[jira] [Commented] (SPARK-24677) MedianHeap is empty when speculation is enabled, causing the SparkContext to stop

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526051#comment-16526051 ] Apache Spark commented on SPARK-24677: -- User 'cxzl25' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24677) MedianHeap is empty when speculation is enabled, causing the SparkContext to stop

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24677: Assignee: (was: Apache Spark) > MedianHeap is empty when speculation is enabled,

[jira] [Assigned] (SPARK-24677) MedianHeap is empty when speculation is enabled, causing the SparkContext to stop

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24677: Assignee: Apache Spark > MedianHeap is empty when speculation is enabled, causing the

[jira] [Assigned] (SPARK-24675) Rename table: validate existence of new location

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24675: Assignee: (was: Apache Spark) > Rename table: validate existence of new location >

[jira] [Assigned] (SPARK-24675) Rename table: validate existence of new location

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24675: Assignee: Apache Spark > Rename table: validate existence of new location >

[jira] [Commented] (SPARK-24675) Rename table: validate existence of new location

2018-06-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526048#comment-16526048 ] Apache Spark commented on SPARK-24675: -- User 'gengliangwang' has created a pull request for this

[jira] [Created] (SPARK-24677) MedianHeap is empty when speculation is enabled, causing the SparkContext to stop

2018-06-28 Thread dzcxzl (JIRA)
dzcxzl created SPARK-24677: -- Summary: MedianHeap is empty when speculation is enabled, causing the SparkContext to stop Key: SPARK-24677 URL: https://issues.apache.org/jira/browse/SPARK-24677 Project: Spark

[jira] [Created] (SPARK-24676) Project required data from parsed data when csvColumnPruning disabled

2018-06-28 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-24676: Summary: Project required data from parsed data when csvColumnPruning disabled Key: SPARK-24676 URL: https://issues.apache.org/jira/browse/SPARK-24676

  1   2   >