[jira] [Commented] (SPARK-15086) Update Java API once the Scala one is finalized

2016-06-09 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322074#comment-15322074 ] Weichen Xu commented on SPARK-15086: If do so, only rename the java API in this type

[jira] [Commented] (SPARK-15086) Update Java API once the Scala one is finalized

2016-06-09 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322075#comment-15322075 ] Reynold Xin commented on SPARK-15086: - I was suggesting renaming both, so the two wou

[jira] [Commented] (SPARK-15086) Update Java API once the Scala one is finalized

2016-06-09 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322082#comment-15322082 ] Weichen Xu commented on SPARK-15086: OK. [~srowen] What do you think about it? > Upd

[jira] [Created] (SPARK-15839) Maven doc JAR generation fails when JAVA_7_HOME is set

2016-06-09 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-15839: -- Summary: Maven doc JAR generation fails when JAVA_7_HOME is set Key: SPARK-15839 URL: https://issues.apache.org/jira/browse/SPARK-15839 Project: Spark Issue Type

[jira] [Commented] (SPARK-15839) Maven doc JAR generation fails when JAVA_7_HOME is set

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322103#comment-15322103 ] Apache Spark commented on SPARK-15839: -- User 'JoshRosen' has created a pull request

[jira] [Resolved] (SPARK-12712) test-dependencies.sh script fails when run against empty .m2 cache

2016-06-09 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-12712. Resolution: Fixed Fix Version/s: 1.6.2 2.0.0 Issue resolved by pull reque

[jira] [Created] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
Ernst Sjöstrand created SPARK-15840: --- Summary: New csv reader does not "determine the input schema" Key: SPARK-15840 URL: https://issues.apache.org/jira/browse/SPARK-15840 Project: Spark Is

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322127#comment-15322127 ] Ernst Sjöstrand commented on SPARK-15840: - The old databricks csv had an option c

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322137#comment-15322137 ] Ernst Sjöstrand commented on SPARK-15840: - Perhaps related to SPARK-13667 ? > Ne

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322140#comment-15322140 ] Hyukjin Kwon commented on SPARK-15840: -- There is {{inferSchema}} option but it seems

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322142#comment-15322142 ] Ernst Sjöstrand commented on SPARK-15840: - Also, the documentation implies that a

[jira] [Comment Edited] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322140#comment-15322140 ] Hyukjin Kwon edited comment on SPARK-15840 at 6/9/16 8:24 AM: -

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322143#comment-15322143 ] Ernst Sjöstrand commented on SPARK-15840: - I have only tested this for Python, no

[jira] [Updated] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ernst Sjöstrand updated SPARK-15840: Description: When testing the new csv reader I found that it would not determine the input

[jira] [Updated] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ernst Sjöstrand updated SPARK-15840: Description: When testing the new csv reader I found that it would not determine the input

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322147#comment-15322147 ] Hyukjin Kwon commented on SPARK-15840: -- For custom dateFormat, here there are, http

[jira] [Commented] (SPARK-11765) Avoid assign UI port between browser unsafe ports (or just 4045: lockd)

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322180#comment-15322180 ] Sean Owen commented on SPARK-11765: --- That's how it works now. > Avoid assign UI port b

[jira] [Updated] (SPARK-15841) [SPARK REPL] REPLSuite has in correct env set for a couple of tests.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15841: Component/s: Spark Shell > [SPARK REPL] REPLSuite has in correct env set for a couple of te

[jira] [Created] (SPARK-15841) [SPARK REPL] REPLSuite has in correct env set for a couple of tests.

2016-06-09 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-15841: --- Summary: [SPARK REPL] REPLSuite has in correct env set for a couple of tests. Key: SPARK-15841 URL: https://issues.apache.org/jira/browse/SPARK-15841 Project: S

[jira] [Commented] (SPARK-15837) PySpark ML Word2Vec should support maxSentenceLength

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322195#comment-15322195 ] Sean Owen commented on SPARK-15837: --- Yeah, ideally we would have suggested and done thi

[jira] [Updated] (SPARK-15841) [SPARK REPL] REPLSuite has incorrect env set for a couple of tests.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15841: Summary: [SPARK REPL] REPLSuite has incorrect env set for a couple of tests. (was: [SPARK

[jira] [Resolved] (SPARK-15836) Spark 2.0/master maven snapshots are broken

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15836. --- Resolution: Duplicate Target Version/s: (was: 2.0.0) > Spark 2.0/master maven snapshots a

[jira] [Assigned] (SPARK-15841) [SPARK REPL] REPLSuite has incorrect env set for a couple of tests.

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15841: Assignee: Apache Spark > [SPARK REPL] REPLSuite has incorrect env set for a couple of test

[jira] [Commented] (SPARK-15841) [SPARK REPL] REPLSuite has incorrect env set for a couple of tests.

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322199#comment-15322199 ] Apache Spark commented on SPARK-15841: -- User 'ScrapCodes' has created a pull request

[jira] [Assigned] (SPARK-15841) [SPARK REPL] REPLSuite has incorrect env set for a couple of tests.

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15841: Assignee: (was: Apache Spark) > [SPARK REPL] REPLSuite has incorrect env set for a cou

[jira] [Updated] (SPARK-15697) [SPARK REPL] unblock some of the useful repl commands.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15697: Description: "implicits", "javap", "power", "type", "kind" commands in repl are blocked. H

[jira] [Updated] (SPARK-15697) [SPARK REPL] unblock some of the useful repl commands.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15697: Description: "implicits", "javap", "power", "type", "kind" commands in repl are blocked. H

[jira] [Updated] (SPARK-15697) [SPARK REPL] unblock some of the useful repl commands.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15697: Description: "implicits", "javap", "power", "type", "kind" commands in repl are blocked. H

[jira] [Updated] (SPARK-15818) Upgrade to Hadoop 2.7.2

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15818: -- Assignee: Adam Roberts > Upgrade to Hadoop 2.7.2 > --- > > Key: SPA

[jira] [Resolved] (SPARK-15818) Upgrade to Hadoop 2.7.2

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15818. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13556 [https://github.co

[jira] [Updated] (SPARK-15781) Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15781: -- Summary: Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config (was:

[jira] [Resolved] (SPARK-15802) SparkSQL connection fail using shell command "bin/beeline -u "jdbc:hive2://*.*.*.*:10000/default""

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15802. --- Resolution: Not A Problem It looks like you show the answer in your question, not sure what you're l

[jira] [Resolved] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15716. --- Resolution: Not A Problem > Memory usage of driver keeps growing up in Spark Streaming >

[jira] [Resolved] (SPARK-15801) spark-submit --num-executors switch also works without YARN

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15801. --- Resolution: Not A Problem This much seems to be not a problem. > spark-submit --num-executors switch

[jira] [Updated] (SPARK-15823) Add @property for 'accuracy' in MulticlassMetrics

2016-06-09 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-15823: - Summary: Add @property for 'accuracy' in MulticlassMetrics (was: Add @property for 'property' in

[jira] [Updated] (SPARK-15831) Kryo 2.21 TreeMap serialization bug causes random job failures with RDDs of HBase puts

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15831: -- Affects Version/s: 1.5.2 1.6.1 Target Version/s: (was: 1.5.0) > Kryo 2.21

[jira] [Commented] (SPARK-15823) Add @property for 'accuracy' in MulticlassMetrics

2016-06-09 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322308#comment-15322308 ] zhengruifeng commented on SPARK-15823: -- {MulticlassMetrics.confusionMatrix} may need

[jira] [Commented] (SPARK-15823) Add @property for 'accuracy' in MulticlassMetrics

2016-06-09 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322309#comment-15322309 ] zhengruifeng commented on SPARK-15823: -- {MulticlassMetrics.confusionMatrix} may need

[jira] [Comment Edited] (SPARK-15823) Add @property for 'accuracy' in MulticlassMetrics

2016-06-09 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322308#comment-15322308 ] zhengruifeng edited comment on SPARK-15823 at 6/9/16 10:20 AM:

[jira] [Issue Comment Deleted] (SPARK-15823) Add @property for 'accuracy' in MulticlassMetrics

2016-06-09 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-15823: - Comment: was deleted (was: {MulticlassMetrics.confusionMatrix} may need {@property} too, but I am

[jira] [Commented] (SPARK-15801) spark-submit --num-executors switch also works without YARN

2016-06-09 Thread Jonathan Taws (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322322#comment-15322322 ] Jonathan Taws commented on SPARK-15801: --- I don't think it is a problem, but it migh

[jira] [Commented] (SPARK-15781) Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

2016-06-09 Thread Jonathan Taws (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322324#comment-15322324 ] Jonathan Taws commented on SPARK-15781: --- By launching a session with {{SPARK_WORKER

[jira] [Commented] (SPARK-15472) Add partitioned `csv`, `json`, `text` format support for FileStreamSink

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322325#comment-15322325 ] Apache Spark commented on SPARK-15472: -- User 'lw-lin' has created a pull request for

[jira] [Updated] (SPARK-15472) Add support for writing in `csv`, `json`, `text` formats in Structured Streaming

2016-06-09 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-15472: -- Summary: Add support for writing in `csv`, `json`, `text` formats in Structured Streaming (was: Add pa

[jira] [Commented] (SPARK-1882) Support dynamic memory sharing in Mesos

2016-06-09 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322364#comment-15322364 ] Stavros Kontopoulos commented on SPARK-1882: Does dynamic allocation help with

[jira] [Commented] (SPARK-15781) Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322425#comment-15322425 ] Sean Owen commented on SPARK-15781: --- Oh I'm sorry, I put this comment on entirely the w

[jira] [Issue Comment Deleted] (SPARK-15781) Misleading deprecated property in standalone cluster configuration documentation

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15781: -- Comment: was deleted (was: PS [~JonathanTaws] do you have some output from -verbose:gc that might conf

[jira] [Updated] (SPARK-15781) Misleading deprecated property in standalone cluster configuration documentation

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15781: -- Summary: Misleading deprecated property in standalone cluster configuration documentation (was: Reduce

[jira] [Updated] (SPARK-15796) Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15796: -- Summary: Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config (was:

[jira] [Commented] (SPARK-15796) Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322426#comment-15322426 ] Sean Owen commented on SPARK-15796: --- PS [~gfeher] do you have some output from -verbose

[jira] [Created] (SPARK-15842) Add support for socket stream.

2016-06-09 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-15842: --- Summary: Add support for socket stream. Key: SPARK-15842 URL: https://issues.apache.org/jira/browse/SPARK-15842 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-15842) Add support for socket stream.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15842: Description: Streaming so far has an offset based sources with all the available sources l

[jira] [Assigned] (SPARK-15842) Add support for socket stream.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma reassigned SPARK-15842: --- Assignee: Prashant Sharma > Add support for socket stream. > ---

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322440#comment-15322440 ] Apache Spark commented on SPARK-15840: -- User 'HyukjinKwon' has created a pull reques

[jira] [Assigned] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15840: Assignee: (was: Apache Spark) > New csv reader does not "determine the input schema" >

[jira] [Assigned] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15840: Assignee: Apache Spark > New csv reader does not "determine the input schema" > --

[jira] [Commented] (SPARK-11765) Avoid assign UI port between browser unsafe ports (or just 4045: lockd)

2016-06-09 Thread Willy Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322442#comment-15322442 ] Willy Lee commented on SPARK-11765: --- As of what version? I'll want to have our team upg

[jira] [Commented] (SPARK-11765) Avoid assign UI port between browser unsafe ports (or just 4045: lockd)

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322446#comment-15322446 ] Sean Owen commented on SPARK-11765: --- For as long as I can remember it has iterated thro

[jira] [Updated] (SPARK-15842) Add support for socket stream.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15842: Description: Streaming so far has offset based sources with all the available sources like

[jira] [Commented] (SPARK-11765) Avoid assign UI port between browser unsafe ports (or just 4045: lockd)

2016-06-09 Thread Willy Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322453#comment-15322453 ] Willy Lee commented on SPARK-11765: --- I'm sorry, I must not have been clear. I think it'

[jira] [Created] (SPARK-15843) Spark RAM issue

2016-06-09 Thread Sreetej Lakkam (JIRA)
Sreetej Lakkam created SPARK-15843: -- Summary: Spark RAM issue Key: SPARK-15843 URL: https://issues.apache.org/jira/browse/SPARK-15843 Project: Spark Issue Type: Question Components

[jira] [Comment Edited] (SPARK-15801) spark-submit --num-executors switch also works without YARN

2016-06-09 Thread Jonathan Taws (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322322#comment-15322322 ] Jonathan Taws edited comment on SPARK-15801 at 6/9/16 1:08 PM:

[jira] [Comment Edited] (SPARK-15801) spark-submit --num-executors switch also works without YARN

2016-06-09 Thread Jonathan Taws (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322322#comment-15322322 ] Jonathan Taws edited comment on SPARK-15801 at 6/9/16 1:08 PM:

[jira] [Resolved] (SPARK-15843) Spark RAM issue

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15843. --- Resolution: Invalid Target Version/s: (was: 1.6.1) Please read https://cwiki.apache.org/

[jira] [Commented] (SPARK-11765) Avoid assign UI port between browser unsafe ports (or just 4045: lockd)

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322494#comment-15322494 ] Sean Owen commented on SPARK-11765: --- You can do two things -- pick another starting por

[jira] [Commented] (SPARK-11765) Avoid assign UI port between browser unsafe ports (or just 4045: lockd)

2016-06-09 Thread Willy Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322503#comment-15322503 ] Willy Lee commented on SPARK-11765: --- The problem as I see it is that 4045 is an unused

[jira] [Commented] (SPARK-15585) Don't use null in data source options to indicate default value

2016-06-09 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322670#comment-15322670 ] Takeshi Yamamuro commented on SPARK-15585: -- I'm afraid the `sep` option for `csv

[jira] [Commented] (SPARK-15772) Improve Scala API docs

2016-06-09 Thread nirav patel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322709#comment-15322709 ] nirav patel commented on SPARK-15772: - I can't point you to every individual function

[jira] [Commented] (SPARK-15837) PySpark ML Word2Vec should support maxSentenceLength

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322720#comment-15322720 ] Apache Spark commented on SPARK-15837: -- User 'WeichenXu123' has created a pull reque

[jira] [Assigned] (SPARK-15837) PySpark ML Word2Vec should support maxSentenceLength

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15837: Assignee: Apache Spark > PySpark ML Word2Vec should support maxSentenceLength > --

[jira] [Assigned] (SPARK-15837) PySpark ML Word2Vec should support maxSentenceLength

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15837: Assignee: (was: Apache Spark) > PySpark ML Word2Vec should support maxSentenceLength >

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

2016-06-09 Thread Sandeep (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322739#comment-15322739 ] Sandeep commented on SPARK-2984: I tried with spark.speculation=false as well and it still

[jira] [Created] (SPARK-15844) HistoryServer doesn't come up if spark.authenticate = true

2016-06-09 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-15844: -- Summary: HistoryServer doesn't come up if spark.authenticate = true Key: SPARK-15844 URL: https://issues.apache.org/jira/browse/SPARK-15844 Project: Spark

[jira] [Commented] (SPARK-15844) HistoryServer doesn't come up if spark.authenticate = true

2016-06-09 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322774#comment-15322774 ] Steve Loughran commented on SPARK-15844: Stack. {code} 16/05/31 22:46:25 INFO Sec

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

2016-06-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322788#comment-15322788 ] Saisai Shao commented on SPARK-15828: - I think this issue is not related to dynamic a

[jira] [Assigned] (SPARK-15844) HistoryServer doesn't come up if spark.authenticate = true

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15844: Assignee: Apache Spark > HistoryServer doesn't come up if spark.authenticate = true >

[jira] [Commented] (SPARK-15844) HistoryServer doesn't come up if spark.authenticate = true

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322793#comment-15322793 ] Apache Spark commented on SPARK-15844: -- User 'steveloughran' has created a pull requ

[jira] [Assigned] (SPARK-15844) HistoryServer doesn't come up if spark.authenticate = true

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15844: Assignee: (was: Apache Spark) > HistoryServer doesn't come up if spark.authenticate =

[jira] [Commented] (SPARK-15801) spark-submit --num-executors switch also works without YARN

2016-06-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322800#comment-15322800 ] Saisai Shao commented on SPARK-15801: - It has already been mentioned in {{spark-submi

[jira] [Commented] (SPARK-15800) Accessing kerberised hdfs from Spark running with Resource Manager

2016-06-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322804#comment-15322804 ] Saisai Shao commented on SPARK-15800: - {quote} Spark is currently running using the R

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

2016-06-09 Thread Miles Crawford (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322827#comment-15322827 ] Miles Crawford commented on SPARK-15828: Possibly could happen without dynamic al

[jira] [Updated] (SPARK-15845) Expose metrics for sub-stage transformations and action

2016-06-09 Thread nirav patel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nirav patel updated SPARK-15845: Summary: Expose metrics for sub-stage transformations and action (was: Expose metrics for sub-tas

[jira] [Created] (SPARK-15845) Expose metrics for sub-task steps

2016-06-09 Thread nirav patel (JIRA)
nirav patel created SPARK-15845: --- Summary: Expose metrics for sub-task steps Key: SPARK-15845 URL: https://issues.apache.org/jira/browse/SPARK-15845 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-15845) Expose metrics for sub-task transformations and action

2016-06-09 Thread nirav patel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nirav patel updated SPARK-15845: Summary: Expose metrics for sub-task transformations and action (was: Expose metrics for sub-task

[jira] [Updated] (SPARK-15845) Expose metrics for sub-stage transformations and action

2016-06-09 Thread nirav patel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nirav patel updated SPARK-15845: Description: Spark optimizes DAG processing by efficiently selecting stage boundaries. This makes

[jira] [Resolved] (SPARK-15804) Manually added metadata not saving with parquet

2016-06-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-15804. - Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13555 [https://githu

[jira] [Updated] (SPARK-15804) Manually added metadata not saving with parquet

2016-06-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-15804: Assignee: kevin yu > Manually added metadata not saving with parquet >

[jira] [Resolved] (SPARK-15788) PySpark IDFModel missing "idf" property

2016-06-09 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-15788. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13540 [https:/

[jira] [Commented] (SPARK-14560) Cooperative Memory Management for Spillables

2016-06-09 Thread Peter Halliday (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322857#comment-15322857 ] Peter Halliday commented on SPARK-14560: Is this going to be in 1.6.2 too? Is th

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

2016-06-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322865#comment-15322865 ] Saisai Shao commented on SPARK-15828: - I see, but I don't clearly understand your sce

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

2016-06-09 Thread Miles Crawford (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322876#comment-15322876 ] Miles Crawford commented on SPARK-15828: This is Hadoop's standard means of resiz

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

2016-06-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322892#comment-15322892 ] Saisai Shao commented on SPARK-15828: - OK, I guess you're running on AWS or similar c

[jira] [Commented] (SPARK-14485) Task finished cause fetch failure when its executor has already been removed by driver

2016-06-09 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322893#comment-15322893 ] Kay Ousterhout commented on SPARK-14485: I don't think (a) is especially rare: th

[jira] [Commented] (SPARK-14485) Task finished cause fetch failure when its executor has already been removed by driver

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322894#comment-15322894 ] Apache Spark commented on SPARK-14485: -- User 'kayousterhout' has created a pull requ

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322912#comment-15322912 ] Yan Chen commented on SPARK-15716: -- Original problem comes from Hortonworks. We also tri

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322920#comment-15322920 ] Yan Chen commented on SPARK-15716: -- [~srowen] Could I know why this issue is closed? >

[jira] [Comment Edited] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322912#comment-15322912 ] Yan Chen edited comment on SPARK-15716 at 6/9/16 5:34 PM: -- Origi

[jira] [Comment Edited] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322920#comment-15322920 ] Yan Chen edited comment on SPARK-15716 at 6/9/16 5:34 PM: -- [~sro

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322930#comment-15322930 ] Yan Chen commented on SPARK-15716: -- Why it is "not a problem" even if it crashes the str

[jira] [Issue Comment Deleted] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15716: -- Comment: was deleted (was: [~yani.chen] the problem is you've described several different problems, an

  1   2   3   >