[jira] [Updated] (SPARK-19779) structured streaming exist needless tmp file

2017-03-01 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19779: - Affects Version/s: 2.1.1 2.0.3 > structured streaming exist needless tmp f

[jira] [Assigned] (SPARK-19792) In the Master Page,the column named “Memory per Node” ,I think it is not all right

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19792: Assignee: Apache Spark > In the Master Page,the column named “Memory per Node” ,I think i

[jira] [Assigned] (SPARK-19792) In the Master Page,the column named “Memory per Node” ,I think it is not all right

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19792: Assignee: (was: Apache Spark) > In the Master Page,the column named “Memory per Node”

[jira] [Commented] (SPARK-19792) In the Master Page,the column named “Memory per Node” ,I think it is not all right

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891731#comment-15891731 ] Apache Spark commented on SPARK-19792: -- User '10110346' has created a pull request f

[jira] [Updated] (SPARK-19779) structured streaming exist needless tmp file

2017-03-01 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19779: - Affects Version/s: (was: 2.1.0) 2.2.0 > structured streaming exist nee

[jira] [Comment Edited] (SPARK-19788) DataStreamReader/DataStreamWriter.option shall accept user-defined type

2017-03-01 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891722#comment-15891722 ] Shixiong Zhu edited comment on SPARK-19788 at 3/2/17 7:04 AM: -

[jira] [Commented] (SPARK-19788) DataStreamReader/DataStreamWriter.option shall accept user-defined type

2017-03-01 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891722#comment-15891722 ] Shixiong Zhu commented on SPARK-19788: -- I remember that we want to support both Scal

[jira] [Resolved] (SPARK-19734) OneHotEncoder __init__ uses dropLast but doc strings all say includeFirst

2017-03-01 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang resolved SPARK-19734. - Resolution: Fixed Assignee: Mark Grover Fix Version/s: 2.2.0 > OneHotEncoder __in

[jira] [Assigned] (SPARK-19583) CTAS for data source tables with an created location does not work

2017-03-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-19583: --- Assignee: Song Jun > CTAS for data source tables with an created location does not work > --

[jira] [Resolved] (SPARK-19583) CTAS for data source tables with an created location does not work

2017-03-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19583. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16938 [https://githu

[jira] [Assigned] (SPARK-19745) SVCAggregator serializes coefficients

2017-03-01 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang reassigned SPARK-19745: --- Shepherd: Yanbo Liang Assignee: Seth Hendrickson > SVCAggregator serializes coefficients

[jira] [Created] (SPARK-19792) In the Master Page,the column named “Memory per Node” ,I think it is not all right

2017-03-01 Thread liuxian (JIRA)
liuxian created SPARK-19792: --- Summary: In the Master Page,the column named “Memory per Node” ,I think it is not all right Key: SPARK-19792 URL: https://issues.apache.org/jira/browse/SPARK-19792 Project: Sp

[jira] [Commented] (SPARK-19503) Execution Plan Optimizer: avoid sort or shuffle when it does not change end result such as df.sort(...).count()

2017-03-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891699#comment-15891699 ] Kazuaki Ishizaki commented on SPARK-19503: -- If it is good to leave sort intact f

[jira] [Commented] (SPARK-19766) INNER JOIN on constant alias columns return incorrect results

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891692#comment-15891692 ] Apache Spark commented on SPARK-19766: -- User 'stanzhai' has created a pull request f

[jira] [Resolved] (SPARK-13931) Resolve stage hanging up problem in a particular case

2017-03-01 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout resolved SPARK-13931. Resolution: Fixed Fix Version/s: 2.2.0 > Resolve stage hanging up problem in a parti

[jira] [Commented] (SPARK-19468) Dataset slow because of unnecessary shuffles

2017-03-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891658#comment-15891658 ] Kazuaki Ishizaki commented on SPARK-19468: -- Interesting. For {{val joined1 = ds

[jira] [Resolved] (SPARK-19777) Scan runningTasksSet when check speculatable tasks in TaskSetManager.

2017-03-01 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout resolved SPARK-19777. Resolution: Fixed Assignee: jin xing Fix Version/s: 2.2.0 > Scan runningTas

[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled

2017-03-01 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891610#comment-15891610 ] Xuefu Zhang commented on SPARK-18769: - Just as fyi, the problem is real and happens w

[jira] [Commented] (SPARK-18389) Disallow cyclic view reference

2017-03-01 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891575#comment-15891575 ] Jiang Xingbo commented on SPARK-18389: -- I‘ve just figure out a way to work this out,

[jira] [Comment Edited] (SPARK-19741) ClassCastException when using Dataset with type containing value types

2017-03-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891548#comment-15891548 ] Kazuaki Ishizaki edited comment on SPARK-19741 at 3/2/17 3:17 AM: -

[jira] [Commented] (SPARK-19741) ClassCastException when using Dataset with type containing value types

2017-03-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891548#comment-15891548 ] Kazuaki Ishizaki commented on SPARK-19741: -- I am afraid whether my sample progra

[jira] [Comment Edited] (SPARK-19741) ClassCastException when using Dataset with type containing value types

2017-03-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891463#comment-15891463 ] Hyukjin Kwon edited comment on SPARK-19741 at 3/2/17 2:00 AM: -

[jira] [Commented] (SPARK-19741) ClassCastException when using Dataset with type containing value types

2017-03-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891463#comment-15891463 ] Hyukjin Kwon commented on SPARK-19741: -- I just tried the code above in the current m

[jira] [Commented] (SPARK-19754) Casting to int from a JSON-parsed float rounds instead of truncating

2017-03-01 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891422#comment-15891422 ] Takeshi Yamamuro commented on SPARK-19754: -- cc: [~cloud_fan] > Casting to int f

[jira] [Commented] (SPARK-19741) ClassCastException when using Dataset with type containing value types

2017-03-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891348#comment-15891348 ] Sean Owen commented on SPARK-19741: --- Duplicate of SPARK-17368 ? > ClassCastException w

[jira] [Assigned] (SPARK-19791) Add doc and example for fpgrowth

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19791: Assignee: (was: Apache Spark) > Add doc and example for fpgrowth > ---

[jira] [Assigned] (SPARK-19791) Add doc and example for fpgrowth

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19791: Assignee: Apache Spark > Add doc and example for fpgrowth > --

[jira] [Commented] (SPARK-19791) Add doc and example for fpgrowth

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891337#comment-15891337 ] Apache Spark commented on SPARK-19791: -- User 'hhbyyh' has created a pull request for

[jira] [Created] (SPARK-19791) Add doc and example for fpgrowth

2017-03-01 Thread yuhao yang (JIRA)
yuhao yang created SPARK-19791: -- Summary: Add doc and example for fpgrowth Key: SPARK-19791 URL: https://issues.apache.org/jira/browse/SPARK-19791 Project: Spark Issue Type: Sub-task C

[jira] [Commented] (SPARK-19373) Mesos implementation of spark.scheduler.minRegisteredResourcesRatio looks at acquired cores rather than registerd cores

2017-03-01 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891332#comment-15891332 ] Michael Gummelt commented on SPARK-19373: - [~skonto] Either decline or hoard. >

[jira] [Resolved] (SPARK-19775) Remove an obsolete `partitionBy().insertInto()` test case

2017-03-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19775. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17106 [https://github.co

[jira] [Assigned] (SPARK-19775) Remove an obsolete `partitionBy().insertInto()` test case

2017-03-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-19775: - Assignee: Dongjoon Hyun > Remove an obsolete `partitionBy().insertInto()` test case > --

[jira] [Updated] (SPARK-19373) Mesos implementation of spark.scheduler.minRegisteredResourcesRatio looks at acquired cores rather than registerd cores

2017-03-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19373: -- Fix Version/s: 2.1.1 > Mesos implementation of spark.scheduler.minRegisteredResourcesRatio looks at >

[jira] [Commented] (SPARK-19776) Is the JavaKafkaWordCount example correct for Spark version 2.1?

2017-03-01 Thread Russell Abedin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891280#comment-15891280 ] Russell Abedin commented on SPARK-19776: Great - thanks for the answer [~srowen]

[jira] [Commented] (SPARK-19373) Mesos implementation of spark.scheduler.minRegisteredResourcesRatio looks at acquired cores rather than registerd cores

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891244#comment-15891244 ] Apache Spark commented on SPARK-19373: -- User 'mgummelt' has created a pull request f

[jira] [Commented] (SPARK-15848) Spark unable to read partitioned table in avro format and column name in upper case

2017-03-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891180#comment-15891180 ] Dongjoon Hyun commented on SPARK-15848: --- Hi, [~pratik.shah2462]. It doesn't happen

[jira] [Comment Edited] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current

2017-03-01 Thread Nick Afshartous (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888572#comment-15888572 ] Nick Afshartous edited comment on SPARK-19767 at 3/1/17 10:15 PM: -

[jira] [Commented] (SPARK-19754) Casting to int from a JSON-parsed float rounds instead of truncating

2017-03-01 Thread Juan Pumarino (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891069#comment-15891069 ] Juan Pumarino commented on SPARK-19754: --- Thank you both for your replies and for lo

[jira] [Commented] (SPARK-19578) Poor pyspark performance + incorrect UI input-size metrics

2017-03-01 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890930#comment-15890930 ] Nicholas Chammas commented on SPARK-19578: -- Makes sense to me. I suppose the Apa

[jira] [Commented] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890907#comment-15890907 ] Apache Spark commented on SPARK-18352: -- User 'felixcheung' has created a pull reques

[jira] [Commented] (SPARK-19734) OneHotEncoder __init__ uses dropLast but doc strings all say includeFirst

2017-03-01 Thread Corey (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890849#comment-15890849 ] Corey commented on SPARK-19734: --- Not at all, thanks for doing it. > OneHotEncoder __init__

[jira] [Commented] (SPARK-19578) Poor pyspark performance + incorrect UI input-size metrics

2017-03-01 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890846#comment-15890846 ] holdenk commented on SPARK-19578: - [~nchammas] It's an interesting idea but I don't think

[jira] [Commented] (SPARK-19734) OneHotEncoder __init__ uses dropLast but doc strings all say includeFirst

2017-03-01 Thread Mark Grover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890825#comment-15890825 ] Mark Grover commented on SPARK-19734: - Don't mean to step on any toes but since there

[jira] [Assigned] (SPARK-19734) OneHotEncoder __init__ uses dropLast but doc strings all say includeFirst

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19734: Assignee: (was: Apache Spark) > OneHotEncoder __init__ uses dropLast but doc strings a

[jira] [Assigned] (SPARK-19734) OneHotEncoder __init__ uses dropLast but doc strings all say includeFirst

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19734: Assignee: Apache Spark > OneHotEncoder __init__ uses dropLast but doc strings all say incl

[jira] [Commented] (SPARK-19734) OneHotEncoder __init__ uses dropLast but doc strings all say includeFirst

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890823#comment-15890823 ] Apache Spark commented on SPARK-19734: -- User 'markgrover' has created a pull request

[jira] [Resolved] (SPARK-19787) Different default regParam values in ALS

2017-03-01 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-19787. Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17121 [https:/

[jira] [Closed] (SPARK-19773) SparkDataFrame should not allow duplicate names

2017-03-01 Thread Wayne Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wayne Zhang closed SPARK-19773. --- Resolution: Not A Problem > SparkDataFrame should not allow duplicate names > ---

[jira] [Commented] (SPARK-17931) taskScheduler has some unneeded serialization

2017-03-01 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890749#comment-15890749 ] Imran Rashid commented on SPARK-17931: -- [~gbloisi] thanks for reporting the issue.

[jira] [Commented] (SPARK-19773) SparkDataFrame should not allow duplicate names

2017-03-01 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890740#comment-15890740 ] Felix Cheung commented on SPARK-19773: -- Let's close this unless you want to look int

[jira] [Commented] (SPARK-19741) ClassCastException when using Dataset with type containing value types

2017-03-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890712#comment-15890712 ] Kazuaki Ishizaki commented on SPARK-19741: -- The following program causes an exce

[jira] [Commented] (SPARK-19790) OutputCommitCoordinator should not allow another task to commit after an ExecutorFailure

2017-03-01 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890654#comment-15890654 ] Imran Rashid commented on SPARK-19790: -- cc [~kayousterhout] [~markhamstra] [~mridulm

[jira] [Created] (SPARK-19790) OutputCommitCoordinator should not allow another task to commit after an ExecutorFailure

2017-03-01 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-19790: Summary: OutputCommitCoordinator should not allow another task to commit after an ExecutorFailure Key: SPARK-19790 URL: https://issues.apache.org/jira/browse/SPARK-19790

[jira] [Assigned] (SPARK-19789) Add the shortcut of .format("parquet").option("path", "/hdfs/path").partitionBy("col1", "col2").start()

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19789: Assignee: (was: Apache Spark) > Add the shortcut of .format("parquet").option("path",

[jira] [Assigned] (SPARK-19789) Add the shortcut of .format("parquet").option("path", "/hdfs/path").partitionBy("col1", "col2").start()

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19789: Assignee: Apache Spark > Add the shortcut of .format("parquet").option("path", > "/hdfs/p

[jira] [Commented] (SPARK-19789) Add the shortcut of .format("parquet").option("path", "/hdfs/path").partitionBy("col1", "col2").start()

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890647#comment-15890647 ] Apache Spark commented on SPARK-19789: -- User 'CodingCat' has created a pull request

[jira] [Created] (SPARK-19789) Add the shortcut of .format("parquet").option("path", "/hdfs/path").partitionBy("col1", "col2").start()

2017-03-01 Thread Nan Zhu (JIRA)
Nan Zhu created SPARK-19789: --- Summary: Add the shortcut of .format("parquet").option("path", "/hdfs/path").partitionBy("col1", "col2").start() Key: SPARK-19789 URL: https://issues.apache.org/jira/browse/SPARK-19789

[jira] [Commented] (SPARK-15474) ORC data source fails to write and read back empty dataframe

2017-03-01 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890639#comment-15890639 ] Nicholas Chammas commented on SPARK-15474: -- There is a related discussion on ORC

[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled

2017-03-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890629#comment-15890629 ] Thomas Graves commented on SPARK-18769: --- [~yuming] I already made a comment on that

[jira] [Commented] (SPARK-19578) Poor pyspark performance + incorrect UI input-size metrics

2017-03-01 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890588#comment-15890588 ] Nicholas Chammas commented on SPARK-19578: -- [~holdenk] - Would it make sense to

[jira] [Commented] (SPARK-19211) Explicitly prevent Insert into View or Create View As Insert

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890545#comment-15890545 ] Apache Spark commented on SPARK-19211: -- User 'jiangxb1987' has created a pull reques

[jira] [Commented] (SPARK-19768) Error for both aggregate and non-aggregate queries in Structured Streaming - "This query does not support recovering from checkpoint location"

2017-03-01 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890546#comment-15890546 ] Shixiong Zhu commented on SPARK-19768: -- Yeah, just recalled that I fixed the error m

[jira] [Assigned] (SPARK-19211) Explicitly prevent Insert into View or Create View As Insert

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19211: Assignee: (was: Apache Spark) > Explicitly prevent Insert into View or Create View As

[jira] [Assigned] (SPARK-19211) Explicitly prevent Insert into View or Create View As Insert

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19211: Assignee: Apache Spark > Explicitly prevent Insert into View or Create View As Insert > --

[jira] [Assigned] (SPARK-19779) structured streaming exist needless tmp file

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19779: Assignee: Apache Spark > structured streaming exist needless tmp file > -

[jira] [Assigned] (SPARK-19779) structured streaming exist needless tmp file

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19779: Assignee: (was: Apache Spark) > structured streaming exist needless tmp file > --

[jira] [Updated] (SPARK-19788) DataStreamReader/DataStreamWriter.option shall accept user-defined type

2017-03-01 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nan Zhu updated SPARK-19788: Description: There are many other data sources/sinks which has very different configuration ways than Kafk

[jira] [Commented] (SPARK-19779) structured streaming exist needless tmp file

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890531#comment-15890531 ] Apache Spark commented on SPARK-19779: -- User 'gf53520' has created a pull request fo

[jira] [Commented] (SPARK-19779) structured streaming exist needless tmp file

2017-03-01 Thread Feng Gui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890528#comment-15890528 ] Feng Gui commented on SPARK-19779: -- [~srowen] The `Background maintenance` don't clean f

[jira] [Updated] (SPARK-19788) DataStreamReader/DataStreamWriter.option shall accept user-defined type

2017-03-01 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nan Zhu updated SPARK-19788: Description: There are many other data sources/sinks which has very different configuration ways than Kafk

[jira] [Comment Edited] (SPARK-19788) DataStreamReader/DataStreamWriter.option shall accept user-defined type

2017-03-01 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890522#comment-15890522 ] Nan Zhu edited comment on SPARK-19788 at 3/1/17 4:45 PM: - another

[jira] [Updated] (SPARK-19788) DataStreamReader/DataStreamWriter.option shall accept user-defined type

2017-03-01 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nan Zhu updated SPARK-19788: Summary: DataStreamReader/DataStreamWriter.option shall accept user-defined type (was: DataStreamReader.op

[jira] [Commented] (SPARK-19788) DataStreamReader.option shall accept user-defined type

2017-03-01 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890522#comment-15890522 ] Nan Zhu commented on SPARK-19788: - another drawback is that it might look like incompatib

[jira] [Created] (SPARK-19788) DataStreamReader.option shall accept user-defined type

2017-03-01 Thread Nan Zhu (JIRA)
Nan Zhu created SPARK-19788: --- Summary: DataStreamReader.option shall accept user-defined type Key: SPARK-19788 URL: https://issues.apache.org/jira/browse/SPARK-19788 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-19779) structured streaming exist needless tmp file

2017-03-01 Thread Feng Gui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feng Gui updated SPARK-19779: - Summary: structured streaming exist needless tmp file (was: structured streaming exist residual tmp fil

[jira] [Assigned] (SPARK-19766) INNER JOIN on constant alias columns return incorrect results

2017-03-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-19766: --- Assignee: StanZhai > INNER JOIN on constant alias columns return incorrect results > ---

[jira] [Resolved] (SPARK-19766) INNER JOIN on constant alias columns return incorrect results

2017-03-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19766. - Resolution: Fixed > INNER JOIN on constant alias columns return incorrect results > -

[jira] [Updated] (SPARK-19766) INNER JOIN on constant alias columns return incorrect results

2017-03-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19766: Fix Version/s: 2.2.0 2.1.1 > INNER JOIN on constant alias columns return incorrect resul

[jira] [Updated] (SPARK-19766) INNER JOIN on constant alias columns return incorrect results

2017-03-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19766: Labels: Correctness (was: ) > INNER JOIN on constant alias columns return incorrect results >

[jira] [Commented] (SPARK-19781) Bucketizer's handleInvalid leave null values untouched unlike the NaNs

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890373#comment-15890373 ] Apache Spark commented on SPARK-19781: -- User 'crackcell' has created a pull request

[jira] [Assigned] (SPARK-19781) Bucketizer's handleInvalid leave null values untouched unlike the NaNs

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19781: Assignee: Apache Spark > Bucketizer's handleInvalid leave null values untouched unlike the

[jira] [Assigned] (SPARK-19781) Bucketizer's handleInvalid leave null values untouched unlike the NaNs

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19781: Assignee: (was: Apache Spark) > Bucketizer's handleInvalid leave null values untouched

[jira] [Assigned] (SPARK-19786) Facilitate loop optimizations in a JIT compiler regarding range()

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19786: Assignee: (was: Apache Spark) > Facilitate loop optimizations in a JIT compiler regard

[jira] [Commented] (SPARK-19786) Facilitate loop optimizations in a JIT compiler regarding range()

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890271#comment-15890271 ] Apache Spark commented on SPARK-19786: -- User 'kiszk' has created a pull request for

[jira] [Commented] (SPARK-19754) Casting to int from a JSON-parsed float rounds instead of truncating

2017-03-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890272#comment-15890272 ] Hyukjin Kwon commented on SPARK-19754: -- I see. It'd be great if anyone can identify

[jira] [Commented] (SPARK-19754) Casting to int from a JSON-parsed float rounds instead of truncating

2017-03-01 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890287#comment-15890287 ] Takeshi Yamamuro commented on SPARK-19754: -- Good. > Casting to int from a JSON-

[jira] [Commented] (SPARK-19754) Casting to int from a JSON-parsed float rounds instead of truncating

2017-03-01 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890261#comment-15890261 ] Takeshi Yamamuro commented on SPARK-19754: -- Aha, I see. I also checked this in v

[jira] [Assigned] (SPARK-19786) Facilitate loop optimizations in a JIT compiler regarding range()

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19786: Assignee: Apache Spark > Facilitate loop optimizations in a JIT compiler regarding range()

[jira] [Resolved] (SPARK-19754) Casting to int from a JSON-parsed float rounds instead of truncating

2017-03-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-19754. -- Resolution: Cannot Reproduce > Casting to int from a JSON-parsed float rounds instead of trunca

[jira] [Commented] (SPARK-19767) API Doc pages for Streaming with Kafka 0.10 not current

2017-03-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890250#comment-15890250 ] Sean Owen commented on SPARK-19767: --- Sounds fine to me, you can add a note and link to

[jira] [Commented] (SPARK-19754) Casting to int from a JSON-parsed float rounds instead of truncating

2017-03-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890249#comment-15890249 ] Hyukjin Kwon commented on SPARK-19754: -- Thank you for cc'ing me. It seems it returns

[jira] [Assigned] (SPARK-19787) Different default regParam values in ALS

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19787: Assignee: (was: Apache Spark) > Different default regParam values in ALS > ---

[jira] [Commented] (SPARK-19787) Different default regParam values in ALS

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890239#comment-15890239 ] Apache Spark commented on SPARK-19787: -- User 'datumbox' has created a pull request f

[jira] [Assigned] (SPARK-19787) Different default regParam values in ALS

2017-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19787: Assignee: Apache Spark > Different default regParam values in ALS > --

[jira] [Commented] (SPARK-17931) taskScheduler has some unneeded serialization

2017-03-01 Thread Giambattista (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890226#comment-15890226 ] Giambattista commented on SPARK-17931: -- I just wanted to report that after this chan

[jira] [Comment Edited] (SPARK-17931) taskScheduler has some unneeded serialization

2017-03-01 Thread Giambattista (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890226#comment-15890226 ] Giambattista edited comment on SPARK-17931 at 3/1/17 2:06 PM: -

[jira] [Updated] (SPARK-19787) Different default regParam values in ALS

2017-03-01 Thread Vasilis Vryniotis (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasilis Vryniotis updated SPARK-19787: -- Priority: Trivial (was: Major) > Different default regParam values in ALS > --

[jira] [Updated] (SPARK-19787) Different default regParam values in ALS

2017-03-01 Thread Vasilis Vryniotis (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasilis Vryniotis updated SPARK-19787: -- Description: In the ALS method the default values of regParam do not match within the s

[jira] [Updated] (SPARK-19787) Different default regParam values in ALS

2017-03-01 Thread Vasilis Vryniotis (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasilis Vryniotis updated SPARK-19787: -- Issue Type: Improvement (was: Bug) > Different default regParam values in ALS > --

[jira] [Updated] (SPARK-19786) Facilitate loop optimizations in a JIT compiler regarding range()

2017-03-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-19786: - Summary: Facilitate loop optimizations in a JIT compiler regarding range() (was: Facilit

  1   2   >