[jira] [Commented] (SPARK-12141) Use Jackson to serialize all events when writing event log

2015-12-03 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-12141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15039776#comment-15039776 ] Jean-Baptiste Onofré commented on SPARK-12141: -- [~vanzin] Do you mind if I take a look on

[jira] [Commented] (SPARK-12140) Support Streaming UI in HistoryServer

2015-12-03 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15039778#comment-15039778 ] Jean-Baptiste Onofré commented on SPARK-12140: -- [~vanzin] Do you mind if I try something

[jira] [Commented] (SPARK-12089) java.lang.NegativeArraySizeException when growing BufferHolder

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15040771#comment-15040771 ] Apache Spark commented on SPARK-12089: -- User 'nongli' has created a pull request for this issue:

[jira] [Commented] (SPARK-12144) Implement DataFrameReader and DataFrameWriter API in SparkR

2015-12-03 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15040773#comment-15040773 ] Shivaram Venkataraman commented on SPARK-12144: --- Personally I don't think this API is a

[jira] [Commented] (SPARK-12117) Column Aliases are Ignored in callUDF while using struct()

2015-12-03 Thread sachin aggarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15039767#comment-15039767 ] sachin aggarwal commented on SPARK-12117: - Hi, can u suggest me some work around to make this

[jira] [Commented] (SPARK-5264) Support `drop temporary table [if exists]` DDL command

2015-12-03 Thread niranda perera (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15039713#comment-15039713 ] niranda perera commented on SPARK-5264: --- any update on this? > Support `drop temporary table [if

[jira] [Created] (SPARK-12144) Implement DataFrameReader and DataFrameWriter API in SparkR

2015-12-03 Thread Sun Rui (JIRA)
Sun Rui created SPARK-12144: --- Summary: Implement DataFrameReader and DataFrameWriter API in SparkR Key: SPARK-12144 URL: https://issues.apache.org/jira/browse/SPARK-12144 Project: Spark Issue

[jira] [Updated] (SPARK-12139) REGEX Column Specification for Hive Queries

2015-12-03 Thread Derek Sabry (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Sabry updated SPARK-12139: Description: When executing a query of the form Select `(a)?+.+` from A, Hive would interprets

[jira] [Commented] (SPARK-12144) Implement DataFrameReader and DataFrameWriter API in SparkR

2015-12-03 Thread Sun Rui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041177#comment-15041177 ] Sun Rui commented on SPARK-12144: - @shivaram, your opinion is reasonable. It seems that read.df and

[jira] [Resolved] (SPARK-12104) collect() does not handle multiple columns with same name

2015-12-03 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-12104. --- Resolution: Fixed Fix Version/s: 1.6.1 2.0.0 Issue

[jira] [Updated] (SPARK-12139) REGEX Column Specification for Hive Queries

2015-12-03 Thread Derek Sabry (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Sabry updated SPARK-12139: Description: When executing a query of the form Select `(a)?\+.\+` from A, Hive would interprets

[jira] [Updated] (SPARK-12139) REGEX Column Specification for Hive Queries

2015-12-03 Thread Derek Sabry (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Sabry updated SPARK-12139: Description: When executing a query of the form Select `(a)?\+.\+` from A, Hive would interpret

[jira] [Commented] (SPARK-12117) Column Aliases are Ignored in callUDF while using struct()

2015-12-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041232#comment-15041232 ] Liang-Chi Hsieh commented on SPARK-12117: - Hmm, because the problem is due to the field names are

[jira] [Assigned] (SPARK-12122) Recovered streaming context can sometimes run a batch twice

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12122: Assignee: Apache Spark (was: Tathagata Das) > Recovered streaming context can sometimes

[jira] [Commented] (SPARK-5836) Highlight in Spark documentation that by default Spark does not delete its temporary files

2015-12-03 Thread Ewan Higgs (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037593#comment-15037593 ] Ewan Higgs commented on SPARK-5836: --- [~tdas] {quote} The only case there may be issues is when the

[jira] [Created] (SPARK-12125) pull out nondeterministic expressions from Join

2015-12-03 Thread iward (JIRA)
iward created SPARK-12125: - Summary: pull out nondeterministic expressions from Join Key: SPARK-12125 URL: https://issues.apache.org/jira/browse/SPARK-12125 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-7379) pickle.loads expects a string instead of bytes in Python 3.

2015-12-03 Thread Xusen Yin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037678#comment-15037678 ] Xusen Yin commented on SPARK-7379: -- OK, I fixed my code in that PR. My case is getSplits returns an

[jira] [Created] (SPARK-12127) Spark sql Support Cross join with Mongo DB

2015-12-03 Thread himanshu singhal (JIRA)
himanshu singhal created SPARK-12127: Summary: Spark sql Support Cross join with Mongo DB Key: SPARK-12127 URL: https://issues.apache.org/jira/browse/SPARK-12127 Project: Spark Issue

[jira] [Commented] (SPARK-12124) Spark Sql MongoDB Cross Join Not Working

2015-12-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037684#comment-15037684 ] Hyukjin Kwon commented on SPARK-12124: -- I think this might be asked to

[jira] [Created] (SPARK-12126) JDBC datasource processes filters only commonly pushed down.

2015-12-03 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-12126: Summary: JDBC datasource processes filters only commonly pushed down. Key: SPARK-12126 URL: https://issues.apache.org/jira/browse/SPARK-12126 Project: Spark

[jira] [Updated] (SPARK-12125) pull out nondeterministic expressions from Join

2015-12-03 Thread iward (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] iward updated SPARK-12125: -- Fix Version/s: 1.6.0 > pull out nondeterministic expressions from Join >

[jira] [Commented] (SPARK-9182) filter and groupBy on DataFrames are not passed through to jdbc source

2015-12-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037670#comment-15037670 ] Hyukjin Kwon commented on SPARK-9182: - Filed here https://issues.apache.org/jira/browse/SPARK-12126.

[jira] [Comment Edited] (SPARK-7379) pickle.loads expects a string instead of bytes in Python 3.

2015-12-03 Thread Xusen Yin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037678#comment-15037678 ] Xusen Yin edited comment on SPARK-7379 at 12/3/15 11:19 AM: OK, I fixed my

[jira] [Updated] (SPARK-12126) JDBC datasource processes filters only commonly pushed down.

2015-12-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-12126: - Description: As suggested

[jira] [Commented] (SPARK-11238) SparkR: Documentation change for merge function

2015-12-03 Thread RaviShankar KS (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037661#comment-15037661 ] RaviShankar KS commented on SPARK-11238: Hi , I would like to take this up. Can you tell me

[jira] [Assigned] (SPARK-12125) pull out nondeterministic expressions from Join

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12125: Assignee: (was: Apache Spark) > pull out nondeterministic expressions from Join >

[jira] [Assigned] (SPARK-12125) pull out nondeterministic expressions from Join

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12125: Assignee: Apache Spark > pull out nondeterministic expressions from Join >

[jira] [Assigned] (SPARK-12120) Improve exception message when failing to initialize HiveContext in PySpark

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12120: Assignee: (was: Apache Spark) > Improve exception message when failing to initialize

[jira] [Commented] (SPARK-12120) Improve exception message when failing to initialize HiveContext in PySpark

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037538#comment-15037538 ] Apache Spark commented on SPARK-12120: -- User 'zjffdu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12120) Improve exception message when failing to initialize HiveContext in PySpark

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12120: Assignee: Apache Spark > Improve exception message when failing to initialize HiveContext

[jira] [Commented] (SPARK-12122) Recovered streaming context can sometimes run a batch twice

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037560#comment-15037560 ] Apache Spark commented on SPARK-12122: -- User 'tdas' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12122) Recovered streaming context can sometimes run a batch twice

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12122: Assignee: Tathagata Das (was: Apache Spark) > Recovered streaming context can sometimes

[jira] [Resolved] (SPARK-12088) check connection.isClose before connection.getAutoCommit in JDBCRDD.close

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12088. --- Resolution: Fixed Assignee: Huaxin Gao Fix Version/s: 2.0.0 1.6.1

[jira] [Commented] (SPARK-12118) SparkR: Documentation change for isNaN

2015-12-03 Thread RaviShankar KS (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037660#comment-15037660 ] RaviShankar KS commented on SPARK-12118: Hi , I would like to take this up. Can you tell me

[jira] [Commented] (SPARK-9697) Project Tungsten (Spark 1.6)

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037458#comment-15037458 ] Sean Owen commented on SPARK-9697: -- [~rxin] or should this at least be titled "... (Phase 2)"? > Project

[jira] [Created] (SPARK-12119) Support compression in PySpark

2015-12-03 Thread Jeff Zhang (JIRA)
Jeff Zhang created SPARK-12119: -- Summary: Support compression in PySpark Key: SPARK-12119 URL: https://issues.apache.org/jira/browse/SPARK-12119 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-4117) Spark on Yarn handle AM being told command from RM

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-4117: --- Assignee: Apache Spark > Spark on Yarn handle AM being told command from RM >

[jira] [Created] (SPARK-12124) Spark Sql MongoDB Cross Join Not Working

2015-12-03 Thread himanshu singhal (JIRA)
himanshu singhal created SPARK-12124: Summary: Spark Sql MongoDB Cross Join Not Working Key: SPARK-12124 URL: https://issues.apache.org/jira/browse/SPARK-12124 Project: Spark Issue Type:

[jira] [Commented] (SPARK-4117) Spark on Yarn handle AM being told command from RM

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037617#comment-15037617 ] Apache Spark commented on SPARK-4117: - User 'devaraj-kavali' has created a pull request for this

[jira] [Created] (SPARK-12120) Improve exception message when failing to initialize HiveContext in PySpark

2015-12-03 Thread Jeff Zhang (JIRA)
Jeff Zhang created SPARK-12120: -- Summary: Improve exception message when failing to initialize HiveContext in PySpark Key: SPARK-12120 URL: https://issues.apache.org/jira/browse/SPARK-12120 Project:

[jira] [Created] (SPARK-12121) Remote Spark init.sh called from spark_ec2.py points to incorrect prebuilt image URL

2015-12-03 Thread Andre Schumacher (JIRA)
Andre Schumacher created SPARK-12121: Summary: Remote Spark init.sh called from spark_ec2.py points to incorrect prebuilt image URL Key: SPARK-12121 URL: https://issues.apache.org/jira/browse/SPARK-12121

[jira] [Updated] (SPARK-12120) Improve exception message when failing to initialize HiveContext in PySpark

2015-12-03 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated SPARK-12120: --- Description: I get the following exception message when failing to initialize HiveContext. This is

[jira] [Created] (SPARK-12122) Recovered streaming context can sometimes run a batch twice

2015-12-03 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-12122: - Summary: Recovered streaming context can sometimes run a batch twice Key: SPARK-12122 URL: https://issues.apache.org/jira/browse/SPARK-12122 Project: Spark

[jira] [Resolved] (SPARK-12123) Spark ava.lang.NullPointerException

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12123. --- Resolution: Invalid [~michael_han] This should be a question on user@, not a JIRA. Have a look at

[jira] [Resolved] (SPARK-12111) need upgrade instruction

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12111. --- Resolution: Not A Problem [~aedwip] I am someone who knows the details of Spark gets built and

[jira] [Closed] (SPARK-12111) need upgrade instruction

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen closed SPARK-12111. - > need upgrade instruction > > > Key: SPARK-12111 >

[jira] [Created] (SPARK-12123) Spark ava.lang.NullPointerException

2015-12-03 Thread Michael Han (JIRA)
Michael Han created SPARK-12123: --- Summary: Spark ava.lang.NullPointerException Key: SPARK-12123 URL: https://issues.apache.org/jira/browse/SPARK-12123 Project: Spark Issue Type: Question

[jira] [Assigned] (SPARK-4117) Spark on Yarn handle AM being told command from RM

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-4117: --- Assignee: (was: Apache Spark) > Spark on Yarn handle AM being told command from RM >

[jira] [Assigned] (SPARK-12117) Column Aliases are Ignored in callUDF while using struct()

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12117: Assignee: Apache Spark > Column Aliases are Ignored in callUDF while using struct() >

[jira] [Updated] (SPARK-12104) collect() does not handle multiple columns with same name

2015-12-03 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-12104: -- Assignee: Sun Rui > collect() does not handle multiple columns with same name

[jira] [Assigned] (SPARK-12117) Column Aliases are Ignored in callUDF while using struct()

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12117: Assignee: (was: Apache Spark) > Column Aliases are Ignored in callUDF while using

[jira] [Commented] (SPARK-12117) Column Aliases are Ignored in callUDF while using struct()

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15039761#comment-15039761 ] Apache Spark commented on SPARK-12117: -- User 'viirya' has created a pull request for this issue:

[jira] [Resolved] (SPARK-2688) Need a way to run multiple data pipeline concurrently

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-2688. -- Resolution: Won't Fix > Need a way to run multiple data pipeline concurrently >

[jira] [Commented] (SPARK-5312) Use sbt to detect new or changed public classes in PRs

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037808#comment-15037808 ] Sean Owen commented on SPARK-5312: -- [~boyork] are you going to do more with this? > Use sbt to detect

[jira] [Commented] (SPARK-11801) Notify driver when OOM is thrown before executor JVM is killed

2015-12-03 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037869#comment-15037869 ] Mridul Muralidharan commented on SPARK-11801: - I have preference for (a) because I am not

[jira] [Assigned] (SPARK-12105) Add a DataFrame.show() with argument for output PrintStream

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12105: Assignee: Apache Spark > Add a DataFrame.show() with argument for output PrintStream >

[jira] [Commented] (SPARK-12128) Multiplication on decimals in dataframe returns null

2015-12-03 Thread Philip Dodds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037839#comment-15037839 ] Philip Dodds commented on SPARK-12128: -- I tried to add a testcase, but when I do it using a

[jira] [Commented] (SPARK-12105) Add a DataFrame.show() with argument for output PrintStream

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037795#comment-15037795 ] Apache Spark commented on SPARK-12105: -- User 'jbonofre' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12105) Add a DataFrame.show() with argument for output PrintStream

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12105: Assignee: (was: Apache Spark) > Add a DataFrame.show() with argument for output

[jira] [Resolved] (SPARK-5261) In some cases ,The value of word's vector representation is too big

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-5261. -- Resolution: Duplicate Re-resolving as a duplicate of several subsequent issues > In some cases ,The

[jira] [Created] (SPARK-12128) Multiplication on decimals in dataframe returns null

2015-12-03 Thread Philip Dodds (JIRA)
Philip Dodds created SPARK-12128: Summary: Multiplication on decimals in dataframe returns null Key: SPARK-12128 URL: https://issues.apache.org/jira/browse/SPARK-12128 Project: Spark Issue

[jira] [Commented] (SPARK-7893) Complex Operators between Graphs

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037813#comment-15037813 ] Sean Owen commented on SPARK-7893: -- [~andyyehoo] this and its subtasks don't seem to have progressed; I'm

[jira] [Commented] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-03 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037773#comment-15037773 ] Cody Koeninger commented on SPARK-12103: A cursory review of the Kafka project documentation

[jira] [Resolved] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12103. --- Resolution: Not A Problem > KafkaUtils createStream with multiple topics -- does not work as

[jira] [Created] (SPARK-12129) spark.submit.deployMode not being picked up in SparkSubmitArguments

2015-12-03 Thread Doug Balog (JIRA)
Doug Balog created SPARK-12129: -- Summary: spark.submit.deployMode not being picked up in SparkSubmitArguments Key: SPARK-12129 URL: https://issues.apache.org/jira/browse/SPARK-12129 Project: Spark

[jira] [Updated] (SPARK-12130) Replace shuffleManagerClass with shortShuffleMgrNames in ExternalShuffleBlockResolver

2015-12-03 Thread Lianhui Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lianhui Wang updated SPARK-12130: - Component/s: YARN Shuffle > Replace shuffleManagerClass with

[jira] [Created] (SPARK-12130) Replace shuffleManagerClass with shortShuffleMgrNames in ExternalShuffleBlockResolver

2015-12-03 Thread Lianhui Wang (JIRA)
Lianhui Wang created SPARK-12130: Summary: Replace shuffleManagerClass with shortShuffleMgrNames in ExternalShuffleBlockResolver Key: SPARK-12130 URL: https://issues.apache.org/jira/browse/SPARK-12130

[jira] [Commented] (SPARK-10854) MesosExecutorBackend: Received launchTask but executor was null

2015-12-03 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038058#comment-15038058 ] Matei Zaharia commented on SPARK-10854: --- Just a note, I saw a log where this happened, and the

[jira] [Commented] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-03 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038084#comment-15038084 ] Cody Koeninger commented on SPARK-12103: On the off chance you're acting in good faith, actually

[jira] [Commented] (SPARK-12128) Multiplication on decimals in dataframe returns null

2015-12-03 Thread Philip Dodds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038104#comment-15038104 ] Philip Dodds commented on SPARK-12128: -- I took a look at the plan {code} Results do not match for

[jira] [Commented] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-03 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038106#comment-15038106 ] Dan Dutrow commented on SPARK-12103: Thanks. I don't mean to be a pain. I did make an honest attempt

[jira] [Assigned] (SPARK-12131) Cannot create ExpressionEncoder for Array[T] where T is a nested class

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12131: Assignee: Cheng Lian (was: Apache Spark) > Cannot create ExpressionEncoder for Array[T]

[jira] [Commented] (SPARK-11827) Support java.math.BigInteger in Type-Inference utilities for POJOs

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038038#comment-15038038 ] Apache Spark commented on SPARK-11827: -- User 'kevinyu98' has created a pull request for this issue:

[jira] [Assigned] (SPARK-11827) Support java.math.BigInteger in Type-Inference utilities for POJOs

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11827: Assignee: Apache Spark > Support java.math.BigInteger in Type-Inference utilities for

[jira] [Commented] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038115#comment-15038115 ] Apache Spark commented on SPARK-12103: -- User 'koeninger' has created a pull request for this issue:

[jira] [Reopened] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-03 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger reopened SPARK-12103: PR for doc change: https://github.com/apache/spark/pull/10132 > KafkaUtils createStream with

[jira] [Commented] (SPARK-12129) spark.submit.deployMode not being picked up in SparkSubmitArguments

2015-12-03 Thread Doug Balog (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038148#comment-15038148 ] Doug Balog commented on SPARK-12129: Ok, Thanks. > spark.submit.deployMode not being picked up in

[jira] [Commented] (SPARK-9039) Jobs page shows nonsensical task-progress-bar numbers when speculation occurs

2015-12-03 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038027#comment-15038027 ] Ryan Williams commented on SPARK-9039: -- Just saw this in Spark 1.5.2

[jira] [Resolved] (SPARK-12129) spark.submit.deployMode not being picked up in SparkSubmitArguments

2015-12-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-12129. Resolution: Invalid That's an internal property, it's not documented, and users are not

[jira] [Commented] (SPARK-12128) Multiplication on decimals in dataframe returns null

2015-12-03 Thread Philip Dodds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038068#comment-15038068 ] Philip Dodds commented on SPARK-12128: -- Actually that was a little too simple, I went back to the

[jira] [Closed] (SPARK-12128) Multiplication on decimals in dataframe returns null

2015-12-03 Thread Philip Dodds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Dodds closed SPARK-12128. Resolution: Invalid The issue is the promotion of the scale, since the default BigDecimal is a

[jira] [Assigned] (SPARK-11827) Support java.math.BigInteger in Type-Inference utilities for POJOs

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11827: Assignee: (was: Apache Spark) > Support java.math.BigInteger in Type-Inference

[jira] [Created] (SPARK-12131) Cannot create ExpressionEncoder for Array[T] where T is a nested class

2015-12-03 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-12131: -- Summary: Cannot create ExpressionEncoder for Array[T] where T is a nested class Key: SPARK-12131 URL: https://issues.apache.org/jira/browse/SPARK-12131 Project: Spark

[jira] [Commented] (SPARK-12131) Cannot create ExpressionEncoder for Array[T] where T is a nested class

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038121#comment-15038121 ] Apache Spark commented on SPARK-12131: -- User 'liancheng' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12131) Cannot create ExpressionEncoder for Array[T] where T is a nested class

2015-12-03 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12131: Assignee: Apache Spark (was: Cheng Lian) > Cannot create ExpressionEncoder for Array[T]

[jira] [Resolved] (SPARK-12116) Document workaround when method conflicts with another R package, like dplyr

2015-12-03 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-12116. --- Resolution: Fixed Assignee: Felix Cheung Fix Version/s: 1.6.0

[jira] [Commented] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-03 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037986#comment-15037986 ] Cody Koeninger commented on SPARK-12103: Knowing that kafka messages have a key and value isn't

[jira] [Resolved] (SPARK-3200) Class defined with reference to external variables crashes in REPL.

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-3200. -- Resolution: Cannot Reproduce It works as of 1.6, so must have gotten fixed along the way somewhere, but

[jira] [Commented] (SPARK-12128) Multiplication on decimals in dataframe returns null

2015-12-03 Thread Philip Dodds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037933#comment-15037933 ] Philip Dodds commented on SPARK-12128: -- The problem is with the Decimal constructor I believe, if

[jira] [Resolved] (SPARK-8796) make sure SparkPlan is only executed at driver side

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-8796. -- Resolution: Won't Fix > make sure SparkPlan is only executed at driver side >

[jira] [Comment Edited] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-03 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037955#comment-15037955 ] Dan Dutrow edited comment on SPARK-12103 at 12/3/15 4:16 PM: - I'm sure all of

[jira] [Comment Edited] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-03 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037955#comment-15037955 ] Dan Dutrow edited comment on SPARK-12103 at 12/3/15 4:15 PM: - I'm sure all of

[jira] [Commented] (SPARK-12129) spark.submit.deployMode not being picked up in SparkSubmitArguments

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037915#comment-15037915 ] Sean Owen commented on SPARK-12129: --- I'm not sure you're intended to set {{spark.submit.deployMode}}

[jira] [Closed] (SPARK-7337) FPGrowth algo throwing OutOfMemoryError

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen closed SPARK-7337. > FPGrowth algo throwing OutOfMemoryError > --- > > Key:

[jira] [Resolved] (SPARK-7337) FPGrowth algo throwing OutOfMemoryError

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-7337. -- Resolution: Invalid > FPGrowth algo throwing OutOfMemoryError > ---

[jira] [Reopened] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-03 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Dutrow reopened SPARK-12103: Please update the Kafka word count examples to comment on what the key field is. The current examples

[jira] [Commented] (SPARK-12129) spark.submit.deployMode not being picked up in SparkSubmitArguments

2015-12-03 Thread Doug Balog (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037969#comment-15037969 ] Doug Balog commented on SPARK-12129: I thought using {{--conf spark.submit.deployMode=cluster}} with

[jira] [Comment Edited] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-03 Thread Dan Dutrow (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037955#comment-15037955 ] Dan Dutrow edited comment on SPARK-12103 at 12/3/15 3:53 PM: - I'm sure all of

[jira] [Commented] (SPARK-12129) spark.submit.deployMode not being picked up in SparkSubmitArguments

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037980#comment-15037980 ] Sean Owen commented on SPARK-12129: --- Hm I see. Let's see if others chime in about what is supposed to

[jira] [Commented] (SPARK-5081) Shuffle write increases

2015-12-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037932#comment-15037932 ] Sean Owen commented on SPARK-5081: -- Is this still an issue? I'm trying to figure out whether we believe

  1   2   >