[jira] [Commented] (SPARK-4985) Parquet support for date type

2014-12-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259922#comment-14259922 ] Apache Spark commented on SPARK-4985: - User 'adrian-wang' has created a pull request

[jira] [Commented] (SPARK-4988) Create table ..as select ..from..order by .. limit 10 report error when one col is a Decimal

2014-12-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259923#comment-14259923 ] Apache Spark commented on SPARK-4988: - User 'guowei2' has created a pull request for

[jira] [Created] (SPARK-4989) wrong application configuration cause cluster down in standalone mode

2014-12-29 Thread Zhang, Liye (JIRA)
Zhang, Liye created SPARK-4989: -- Summary: wrong application configuration cause cluster down in standalone mode Key: SPARK-4989 URL: https://issues.apache.org/jira/browse/SPARK-4989 Project: Spark

[jira] [Updated] (SPARK-4989) wrong application configuration cause cluster down in standalone mode

2014-12-29 Thread Zhang, Liye (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhang, Liye updated SPARK-4989: --- Description: when enabling eventlog in standalone mode, if give the wrong configuration, the

[jira] [Created] (SPARK-4990) Search SPARK_CONF_DIR first when --properties-file is not specified

2014-12-29 Thread WangTaoTheTonic (JIRA)
WangTaoTheTonic created SPARK-4990: -- Summary: Search SPARK_CONF_DIR first when --properties-file is not specified Key: SPARK-4990 URL: https://issues.apache.org/jira/browse/SPARK-4990 Project: Spark

[jira] [Created] (SPARK-4991) Worker should reconnect to Master when Master actor restart

2014-12-29 Thread Zhang, Liye (JIRA)
Zhang, Liye created SPARK-4991: -- Summary: Worker should reconnect to Master when Master actor restart Key: SPARK-4991 URL: https://issues.apache.org/jira/browse/SPARK-4991 Project: Spark Issue

[jira] [Commented] (SPARK-4990) Search SPARK_CONF_DIR first when --properties-file is not specified

2014-12-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259930#comment-14259930 ] Apache Spark commented on SPARK-4990: - User 'WangTaoTheTonic' has created a pull

[jira] [Commented] (SPARK-4989) wrong application configuration cause cluster down in standalone mode

2014-12-29 Thread Zhang, Liye (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259938#comment-14259938 ] Zhang, Liye commented on SPARK-4989: An following JIRA is opened for resolving the

[jira] [Commented] (SPARK-4991) Worker should reconnect to Master when Master actor restart

2014-12-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259952#comment-14259952 ] Apache Spark commented on SPARK-4991: - User 'liyezhang556520' has created a pull

[jira] [Commented] (SPARK-4990) Search SPARK_CONF_DIR first when --properties-file is not specified

2014-12-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259962#comment-14259962 ] Sean Owen commented on SPARK-4990: -- As you say though, this case is already handled. Why

[jira] [Created] (SPARK-4992) Prominent Python example has bad, beginner style

2014-12-29 Thread Eric O. LEBIGOT (EOL) (JIRA)
Eric O. LEBIGOT (EOL) created SPARK-4992: Summary: Prominent Python example has bad, beginner style Key: SPARK-4992 URL: https://issues.apache.org/jira/browse/SPARK-4992 Project: Spark

[jira] [Commented] (SPARK-4960) Interceptor pattern in receivers

2014-12-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259972#comment-14259972 ] Saisai Shao commented on SPARK-4960: Hi [~tdas] and [~hshreedharan], these days I

[jira] [Commented] (SPARK-4960) Interceptor pattern in receivers

2014-12-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259976#comment-14259976 ] Saisai Shao commented on SPARK-4960: Hi Hari, I think ReliableKafkaReciever only

[jira] [Commented] (SPARK-4963) SchemaRDD.sample may return wrong results

2014-12-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259977#comment-14259977 ] Apache Spark commented on SPARK-4963: - User 'yanbohappy' has created a pull request

[jira] [Created] (SPARK-4993) dd.count failed when storage level is OFF_HEAP

2014-12-29 Thread pengyanhong (JIRA)
pengyanhong created SPARK-4993: -- Summary: dd.count failed when storage level is OFF_HEAP Key: SPARK-4993 URL: https://issues.apache.org/jira/browse/SPARK-4993 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-4963) SchemaRDD.sample may return wrong results

2014-12-29 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259986#comment-14259986 ] Yanbo Liang commented on SPARK-4963: SchemaRDD.sample() return wrong results due to

[jira] [Updated] (SPARK-4993) execute rdd.count failed when storage level is OFF_HEAP

2014-12-29 Thread pengyanhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengyanhong updated SPARK-4993: --- Summary: execute rdd.count failed when storage level is OFF_HEAP (was: dd.count failed when storage

[jira] [Commented] (SPARK-4992) Prominent Python example has bad, beginner style

2014-12-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260028#comment-14260028 ] Sean Owen commented on SPARK-4992: -- Seems reasonable to me, to avoid anything that could

[jira] [Updated] (SPARK-4992) Prominent Python example has bad, beginner style

2014-12-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-4992: - Attachment: SPARK-4992.patch Example, that changes all of these vars to textFile Prominent Python

[jira] [Commented] (SPARK-4992) Prominent Python example has bad, beginner style

2014-12-29 Thread Eric O. LEBIGOT (EOL) (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260069#comment-14260069 ] Eric O. LEBIGOT (EOL) commented on SPARK-4992: -- Thank you for your response!

[jira] [Comment Edited] (SPARK-4992) Prominent Python example has bad, beginner style

2014-12-29 Thread Eric O. LEBIGOT (EOL) (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260069#comment-14260069 ] Eric O. LEBIGOT (EOL) edited comment on SPARK-4992 at 12/29/14 11:57 AM:

[jira] [Created] (SPARK-4994) Cleanup removed executors' ShuffleInfo in yarn shuffle service

2014-12-29 Thread Lianhui Wang (JIRA)
Lianhui Wang created SPARK-4994: --- Summary: Cleanup removed executors' ShuffleInfo in yarn shuffle service Key: SPARK-4994 URL: https://issues.apache.org/jira/browse/SPARK-4994 Project: Spark

[jira] [Updated] (SPARK-4994) Cleanup removed executors' ShuffleInfo in yarn shuffle service

2014-12-29 Thread Lianhui Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lianhui Wang updated SPARK-4994: Description: when the application is completed, yarn's nodemanager can remove application's

[jira] [Created] (SPARK-4995) Replace Vector.toBreeze.activeIterator with foreachActive

2014-12-29 Thread Jakub Dubovsky (JIRA)
Jakub Dubovsky created SPARK-4995: - Summary: Replace Vector.toBreeze.activeIterator with foreachActive Key: SPARK-4995 URL: https://issues.apache.org/jira/browse/SPARK-4995 Project: Spark

[jira] [Created] (SPARK-4996) Memory leak?

2014-12-29 Thread uncleGen (JIRA)
uncleGen created SPARK-4996: --- Summary: Memory leak? Key: SPARK-4996 URL: https://issues.apache.org/jira/browse/SPARK-4996 Project: Spark Issue Type: Bug Affects Versions: 1.2.0

[jira] [Updated] (SPARK-4996) Memory leak?

2014-12-29 Thread uncleGen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] uncleGen updated SPARK-4996: Description: When I migrate my job from spark 1.1.1 to spark 1.2, it failed. However, everything is OK In

[jira] [Updated] (SPARK-4996) Memory leak?

2014-12-29 Thread uncleGen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] uncleGen updated SPARK-4996: Description: When I migrate my job from spark 1.1.1 to spark 1.2, it failed. However, everything is OK In

[jira] [Commented] (SPARK-4994) Cleanup removed executors' ShuffleInfo in yarn shuffle service

2014-12-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260082#comment-14260082 ] Apache Spark commented on SPARK-4994: - User 'lianhuiwang' has created a pull request

[jira] [Resolved] (SPARK-4966) The MemoryOverhead value is not correct

2014-12-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-4966. -- Resolution: Fixed Fix Version/s: 1.2.1 1.3.0 Assignee:

[jira] [Commented] (SPARK-4992) Prominent Python example has bad, beginner style

2014-12-29 Thread Eric O. LEBIGOT (EOL) (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260123#comment-14260123 ] Eric O. LEBIGOT (EOL) commented on SPARK-4992: -- Now I see that the problem

[jira] [Comment Edited] (SPARK-4992) Prominent Python example has bad, beginner style

2014-12-29 Thread Eric O. LEBIGOT (EOL) (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260123#comment-14260123 ] Eric O. LEBIGOT (EOL) edited comment on SPARK-4992 at 12/29/14 2:24 PM:

[jira] [Comment Edited] (SPARK-4989) wrong application configuration cause cluster down in standalone mode

2014-12-29 Thread Zhang, Liye (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259938#comment-14259938 ] Zhang, Liye edited comment on SPARK-4989 at 12/29/14 2:39 PM: --

[jira] [Comment Edited] (SPARK-4989) wrong application configuration cause cluster down in standalone mode

2014-12-29 Thread Zhang, Liye (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259938#comment-14259938 ] Zhang, Liye edited comment on SPARK-4989 at 12/29/14 2:39 PM: --

[jira] [Commented] (SPARK-4960) Interceptor pattern in receivers

2014-12-29 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260144#comment-14260144 ] Cody Koeninger commented on SPARK-4960: --- You're saying for an implementation that

[jira] [Commented] (SPARK-4995) Replace Vector.toBreeze.activeIterator with foreachActive

2014-12-29 Thread Jakub Dubovsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260177#comment-14260177 ] Jakub Dubovsky commented on SPARK-4995: --- I started to work on this. Replace

[jira] [Commented] (SPARK-4996) Memory leak?

2014-12-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260186#comment-14260186 ] Sean Owen commented on SPARK-4996: -- What this error really says is that Spark thought

[jira] [Updated] (SPARK-4992) Prominent Python example has bad, beginner style

2014-12-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-4992: - Attachment: (was: SPARK-4992.patch) Prominent Python example has bad, beginner style

[jira] [Updated] (SPARK-4992) Prominent Python example has bad, beginner style

2014-12-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-4992: - Attachment: SPARK-4992.patch OK, right I can update the patch to use foo_bar style for Python. The patch

[jira] [Commented] (SPARK-2757) Add Mima test for Spark Sink after 1.10 is released

2014-12-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260237#comment-14260237 ] Sean Owen commented on SPARK-2757: -- [~hshreedharan] Is this still an issue? Exclusions

[jira] [Commented] (SPARK-1272) Don't fail job if some local directories are buggy

2014-12-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260238#comment-14260238 ] Sean Owen commented on SPARK-1272: -- [~qqsun8819] I'd also like to see this implemented,

[jira] [Commented] (SPARK-3955) Different versions between jackson-mapper-asl and jackson-core-asl

2014-12-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260291#comment-14260291 ] Apache Spark commented on SPARK-3955: - User 'srowen' has created a pull request for

[jira] [Resolved] (SPARK-4982) The `spark.ui.retainedJobs` meaning is wrong in `Spark UI` configuration

2014-12-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4982. --- Resolution: Fixed Assignee: XiaoJing wang The `spark.ui.retainedJobs` meaning is wrong in

[jira] [Commented] (SPARK-4779) PySpark Shuffle Fails Looking for Files that Don't Exist when low on Memory

2014-12-29 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260311#comment-14260311 ] Davies Liu commented on SPARK-4779: --- [~joshrosen] maybe, but I still can not reproduce

[jira] [Updated] (SPARK-4982) The `spark.ui.retainedJobs` meaning is wrong in `Spark UI` configuration

2014-12-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4982: -- Fix Version/s: (was: 1.2.0) The `spark.ui.retainedJobs` meaning is wrong in `Spark UI`

[jira] [Commented] (SPARK-4989) wrong application configuration cause cluster down in standalone mode

2014-12-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260323#comment-14260323 ] Josh Rosen commented on SPARK-4989: --- If you have a full stacktrace, can you past it into

[jira] [Commented] (SPARK-4921) Performance issue caused by TaskSetManager returning PROCESS_LOCAL for NO_PREF tasks

2014-12-29 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260322#comment-14260322 ] Sandy Ryza commented on SPARK-4921: --- Offline [~xuefuz] and [~lirui] mentioned to me that

[jira] [Commented] (SPARK-4921) Performance issue caused by TaskSetManager returning PROCESS_LOCAL for NO_PREF tasks

2014-12-29 Thread Sandy Ryza (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260325#comment-14260325 ] Sandy Ryza commented on SPARK-4921: --- [~xuefuz] [~lirui] was that query against data not

[jira] [Updated] (SPARK-4959) Attributes are case sensitive when using a select query from a projection

2014-12-29 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-4959: Priority: Critical (was: Major) Target Version/s: 1.3.0 Attributes are case

[jira] [Commented] (SPARK-4968) [SparkSQL] java.lang.UnsupportedOperationException when hive partition doesn't exist and order by and limit are used

2014-12-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260343#comment-14260343 ] Apache Spark commented on SPARK-4968: - User 'saucam' has created a pull request for

[jira] [Updated] (SPARK-4963) SchemaRDD.sample may return wrong results

2014-12-29 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-4963: - Assignee: Yanbo Liang SchemaRDD.sample may return wrong results

[jira] [Resolved] (SPARK-4946) Using AkkaUtils.askWithReply in MapOutputTracker.askTracker to reduce the chance of the communicating problem

2014-12-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4946. --- Resolution: Fixed Fix Version/s: 1.3.0 Issue resolved by pull request 3785

[jira] [Updated] (SPARK-4946) Using AkkaUtils.askWithReply in MapOutputTracker.askTracker to reduce the chance of the communicating problem

2014-12-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4946: -- Assignee: YanTang Zhai Using AkkaUtils.askWithReply in MapOutputTracker.askTracker to reduce the

[jira] [Commented] (SPARK-4963) SchemaRDD.sample may return wrong results

2014-12-29 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260355#comment-14260355 ] Xiangrui Meng commented on SPARK-4963: -- [~yanboliang] Thanks for looking into this

[jira] [Updated] (SPARK-4968) [SparkSQL] java.lang.UnsupportedOperationException when hive partition doesn't exist and order by and limit are used

2014-12-29 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-4968: --- Description: Create table with partitions run query for partition which doesn't exist and contains

[jira] [Commented] (SPARK-4882) PySpark broadcast breaks when using KryoSerializer

2014-12-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260415#comment-14260415 ] Apache Spark commented on SPARK-4882: - User 'JoshRosen' has created a pull request for

[jira] [Resolved] (SPARK-4409) Additional (but limited) Linear Algebra Utils

2014-12-29 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-4409. -- Resolution: Fixed Fix Version/s: 1.3.0 Issue resolved by pull request 3319

[jira] [Resolved] (SPARK-2638) Improve concurrency of fetching Map outputs

2014-12-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2638. --- Resolution: Not a Problem I'm resolving this as Not a Problem since this doesn't seem like an actual

[jira] [Updated] (SPARK-4879) Missing output partitions after job completes with speculative execution

2014-12-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4879: -- Description: When speculative execution is enabled ({{spark.speculation=true}}), jobs that save output

[jira] [Commented] (SPARK-4983) Tag EC2 instances in the same call that launches them

2014-12-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260479#comment-14260479 ] Josh Rosen commented on SPARK-4983: --- Did the original code tag them in a separate call

[jira] [Updated] (SPARK-4694) Long-run user thread(such as HiveThriftServer2) causes the 'process leak' in yarn-client mode

2014-12-29 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-4694: Target Version/s: 1.3.0, 1.2.1 Long-run user thread(such as HiveThriftServer2) causes the

[jira] [Commented] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries

2014-12-29 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260490#comment-14260490 ] Michael Armbrust commented on SPARK-4908: - You don't even need the JDBC server to

[jira] [Updated] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries

2014-12-29 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4908: --- Target Version/s: 1.2.1 Spark SQL built for Hive 13 fails under concurrent metadata queries

[jira] [Updated] (SPARK-4766) ML Estimator Params should subclass Transformer Params

2014-12-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-4766: - Assignee: (was: Joseph K. Bradley) ML Estimator Params should subclass Transformer

[jira] [Updated] (SPARK-4766) ML Estimator Params should subclass Transformer Params

2014-12-29 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-4766: - Description: Currently, in spark.ml, both Transformers and Estimators extend the same

[jira] [Commented] (SPARK-4987) Parquet support for timestamp type

2014-12-29 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260523#comment-14260523 ] Michael Armbrust commented on SPARK-4987: - Hey, thanks for working on this.

[jira] [Commented] (SPARK-2757) Add Mima test for Spark Sink after 1.10 is released

2014-12-29 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260531#comment-14260531 ] Hari Shreedharan commented on SPARK-2757: - Actually yes, the SparkSink and the

[jira] [Resolved] (SPARK-4156) Add expectation maximization for Gaussian mixture models to MLLib clustering

2014-12-29 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-4156. -- Resolution: Fixed Fix Version/s: 1.3.0 Issue resolved by pull request 3022

[jira] [Commented] (SPARK-4660) JavaSerializer uses wrong classloader

2014-12-29 Thread Matei Zaharia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260544#comment-14260544 ] Matei Zaharia commented on SPARK-4660: -- [~pkolaczk] mind sending a pull request

[jira] [Updated] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries

2014-12-29 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-4908: Priority: Blocker (was: Critical) Spark SQL built for Hive 13 fails under concurrent

[jira] [Updated] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries

2014-12-29 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-4908: Assignee: Cheng Lian Spark SQL built for Hive 13 fails under concurrent metadata queries

[jira] [Resolved] (SPARK-4972) Updated the scala doc for lasso and ridge regression for the change of LeastSquaresGradient

2014-12-29 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-4972. -- Resolution: Fixed Fix Version/s: 1.3.0 Issue resolved by pull request 3808

[jira] [Updated] (SPARK-4972) Updated the scala doc for lasso and ridge regression for the change of LeastSquaresGradient

2014-12-29 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-4972: - Assignee: DB Tsai Updated the scala doc for lasso and ridge regression for the change of

[jira] [Updated] (SPARK-4993) execute rdd.count failed when storage level is OFF_HEAP

2014-12-29 Thread pengyanhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengyanhong updated SPARK-4993: --- Description: in the file [ WARN] [2014-12-29 17:47:52 187] org.apache.spark.scheduler.TaskSetManager

[jira] [Commented] (SPARK-4835) Streaming saveAs*HadoopFiles() methods may throw FileAlreadyExistsException during checkpoint recovery

2014-12-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260654#comment-14260654 ] Apache Spark commented on SPARK-4835: - User 'JoshRosen' has created a pull request for

[jira] [Commented] (SPARK-4960) Interceptor pattern in receivers

2014-12-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260655#comment-14260655 ] Saisai Shao commented on SPARK-4960: Hi Cody, please see the inline comments. {quote}

[jira] [Comment Edited] (SPARK-4960) Interceptor pattern in receivers

2014-12-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260655#comment-14260655 ] Saisai Shao edited comment on SPARK-4960 at 12/30/14 1:34 AM: --

[jira] [Commented] (SPARK-4835) Streaming saveAs*HadoopFiles() methods may throw FileAlreadyExistsException during checkpoint recovery

2014-12-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260660#comment-14260660 ] Josh Rosen commented on SPARK-4835: --- [~tdas], After some more thought, I think that the

[jira] [Updated] (SPARK-4993) execute rdd.count failed when storage level is OFF_HEAP

2014-12-29 Thread pengyanhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengyanhong updated SPARK-4993: --- Description: There have been config for Tachyon in the file [ WARN] [2014-12-29 17:47:52 187]

[jira] [Commented] (SPARK-4989) wrong application configuration cause cluster down in standalone mode

2014-12-29 Thread Zhang, Liye (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260667#comment-14260667 ] Zhang, Liye commented on SPARK-4989: Assume that we set

[jira] [Commented] (SPARK-4983) Tag EC2 instances in the same call that launches them

2014-12-29 Thread Nicholas Chammas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260668#comment-14260668 ] Nicholas Chammas commented on SPARK-4983: - It's something we overlooked. I'm not

[jira] [Commented] (SPARK-4987) Parquet support for timestamp type

2014-12-29 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260709#comment-14260709 ] Adrian Wang commented on SPARK-4987: Yeah, using INT96 would be an approach to resolve

[jira] [Commented] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries

2014-12-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260714#comment-14260714 ] Apache Spark commented on SPARK-4908: - User 'marmbrus' has created a pull request for

[jira] [Updated] (SPARK-4998) train methods in object DecisionTree cannot work when using java reflection

2014-12-29 Thread Liu Jiongzhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Jiongzhou updated SPARK-4998: - Description: When using the Java reflection, the several train methods defined in object

[jira] [Commented] (SPARK-4963) SchemaRDD.sample may return wrong results

2014-12-29 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260766#comment-14260766 ] Cheng Lian commented on SPARK-4963: --- [~yanboliang] Making {{HiveTableScan}} return

[jira] [Commented] (SPARK-3299) [SQL] Public API in SQLContext to list tables

2014-12-29 Thread Bill Bejeck (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260779#comment-14260779 ] Bill Bejeck commented on SPARK-3299: Implementation done. Just working on unit tests

[jira] [Commented] (SPARK-4963) SchemaRDD.sample may return wrong results

2014-12-29 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260782#comment-14260782 ] Michael Armbrust commented on SPARK-4963: - We could create a new operator, but the

[jira] [Commented] (SPARK-4998) train methods in object DecisionTree cannot work when using java reflection

2014-12-29 Thread Liu Jiongzhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260789#comment-14260789 ] Liu Jiongzhou commented on SPARK-4998: -- I have make a PR to solve this issue.

[jira] [Commented] (SPARK-4963) SchemaRDD.sample may return wrong results

2014-12-29 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260790#comment-14260790 ] Cheng Lian commented on SPARK-4963: --- OK I agree. However the {{_.copy()}} call should be

[jira] [Updated] (SPARK-4999) No need to put WAL-backed block into block manager by default

2014-12-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-4999: --- Description: Currently WAL-backed block is read out from HDFS and put into BlockManger with storage

[jira] [Commented] (SPARK-4999) No need to put WAL-backed block into block manager by default

2014-12-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260812#comment-14260812 ] Saisai Shao commented on SPARK-4999: {code} // Since storeInBlockManager =

[jira] [Commented] (SPARK-4986) Graceful shutdown for Spark Streaming does not work in Standalone cluster mode

2014-12-29 Thread Jesper Lundgren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260815#comment-14260815 ] Jesper Lundgren commented on SPARK-4986: I am currently using this patch as a work

[jira] [Created] (SPARK-5000) Alias support string literal in spark sql parser

2014-12-29 Thread wangfei (JIRA)
wangfei created SPARK-5000: -- Summary: Alias support string literal in spark sql parser Key: SPARK-5000 URL: https://issues.apache.org/jira/browse/SPARK-5000 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-5000) Alias support string literal in spark sql parser

2014-12-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260852#comment-14260852 ] Apache Spark commented on SPARK-5000: - User 'scwf' has created a pull request for this

[jira] [Updated] (SPARK-5001) BlockRDD removed unreasonablly in streaming

2014-12-29 Thread hanhonggen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hanhonggen updated SPARK-5001: -- Attachment: fix_bug_BlockRDD_removed_not_reasonablly_in_streaming.patch BlockRDD removed unreasonablly

[jira] [Updated] (SPARK-5000) Alias support string literal in spark sql

2014-12-29 Thread wangfei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangfei updated SPARK-5000: --- Summary: Alias support string literal in spark sql (was: Alias support string literal in spark sql parser)

[jira] [Updated] (SPARK-4992) Prominent Python example has bad, beginner style

2014-12-29 Thread Eric O. LEBIGOT (EOL) (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric O. LEBIGOT (EOL) updated SPARK-4992: - Attachment: SPARK-4992-EOL.patch.txt Same as SPARK-4492.patch.txt, but with

[jira] [Commented] (SPARK-4992) Prominent Python example has bad, beginner style

2014-12-29 Thread Eric O. LEBIGOT (EOL) (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260885#comment-14260885 ] Eric O. LEBIGOT (EOL) commented on SPARK-4992: -- Thanks. I attached a version

[jira] [Commented] (SPARK-4999) No need to put WAL-backed block into block manager by default

2014-12-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260890#comment-14260890 ] Saisai Shao commented on SPARK-4999: cc [~tdas], what is your opinion ? No need to