[jira] [Resolved] (SPARK-12198) SparkR support read.parquet and deprecate parquetFile

2015-12-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-12198. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request

[jira] [Commented] (SPARK-12172) Consider removing SparkR internal RDD APIs

2015-12-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051320#comment-15051320 ] Shivaram Venkataraman commented on SPARK-12172: --- My opinion is that we should introduce the

[jira] [Updated] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Severin Thaler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Severin Thaler updated SPARK-12266: --- Description: on server postgis running with a table that has a column of type raster on

[jira] [Updated] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Severin Thaler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Severin Thaler updated SPARK-12266: --- Description: on server postgis server running with a table that has a column of type raster

[jira] [Updated] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Severin Thaler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Severin Thaler updated SPARK-12266: --- Description: on server postgis running with a table that has a column of type raster on

[jira] [Resolved] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12266. --- Resolution: Not A Problem That sounds like a custom type, according to JDBC. I don't think you can

[jira] [Updated] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Severin Thaler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Severin Thaler updated SPARK-12266: --- Description: on server postgis running with a table that has a column of type raster on

[jira] [Comment Edited] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Severin Thaler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051475#comment-15051475 ] Severin Thaler edited comment on SPARK-12266 at 12/10/15 7:08 PM: -- yes,

[jira] [Created] (SPARK-12267) Standalone master keeps references to disassociated workers until they sent no heartbeats

2015-12-10 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-12267: --- Summary: Standalone master keeps references to disassociated workers until they sent no heartbeats Key: SPARK-12267 URL: https://issues.apache.org/jira/browse/SPARK-12267

[jira] [Commented] (SPARK-12267) Standalone master keeps references to disassociated workers until they sent no heartbeats

2015-12-10 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051502#comment-15051502 ] Shixiong Zhu commented on SPARK-12267: -- cc [~vanzin] > Standalone master keeps references to

[jira] [Updated] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Severin Thaler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Severin Thaler updated SPARK-12266: --- Description: on server postgis server running with a table that has a column of type raster

[jira] [Updated] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Severin Thaler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Severin Thaler updated SPARK-12266: --- Description: on server postgis running with a table that has a column of type raster on

[jira] [Commented] (SPARK-12237) Unsupported message RpcMessage causes message retries

2015-12-10 Thread Nan Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051499#comment-15051499 ] Nan Zhu commented on SPARK-12237: - if that's the case, I don't think it would happen in the real world

[jira] [Commented] (SPARK-12237) Unsupported message RpcMessage causes message retries

2015-12-10 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051500#comment-15051500 ] Jacek Laskowski commented on SPARK-12237: - Sure, but that's not the issue who talks to whom, but

[jira] [Assigned] (SPARK-12250) Allow users to define a UDAF without providing details of its inputSchema

2015-12-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai reassigned SPARK-12250: Assignee: Yin Huai > Allow users to define a UDAF without providing details of its inputSchema >

[jira] [Updated] (SPARK-11959) Document normal equation solver for ordinary least squares in user guide

2015-12-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-11959: -- Assignee: Yanbo Liang (was: Xiangrui Meng) > Document normal equation solver for ordinary

[jira] [Updated] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Severin Thaler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Severin Thaler updated SPARK-12266: --- Description: on server postgis server running with a table that has a column of type raster

[jira] [Updated] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Severin Thaler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Severin Thaler updated SPARK-12266: --- Description: on server postgis server running with a table that has a column of type raster

[jira] [Commented] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Severin Thaler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051475#comment-15051475 ] Severin Thaler commented on SPARK-12266: yes, im going to simplify the example and just use a

[jira] [Commented] (SPARK-2791) Fix committing, reverting and state tracking in shuffle file consolidation

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051530#comment-15051530 ] Apache Spark commented on SPARK-2791: - User 'aarondav' has created a pull request for this issue:

[jira] [Resolved] (SPARK-12250) Allow users to define a UDAF without providing details of its inputSchema

2015-12-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-12250. -- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 10236

[jira] [Resolved] (SPARK-12228) Use in-memory for execution hive's derby metastore

2015-12-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-12228. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 10204

[jira] [Created] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Severin Thaler (JIRA)
Severin Thaler created SPARK-12266: -- Summary: cannot handle postgis raster type Key: SPARK-12266 URL: https://issues.apache.org/jira/browse/SPARK-12266 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-12266) cannot handle postgis raster type

2015-12-10 Thread Severin Thaler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Severin Thaler updated SPARK-12266: --- Description: on server postgis server running with a table that has a column of type raster

[jira] [Issue Comment Deleted] (SPARK-12258) Hive Timestamp UDF is binded with '1969-12-31 15:59:59.999999' for null value

2015-12-10 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12258: Comment: was deleted (was: [~cloud_fan] It sounds like it is related to the PR

[jira] [Commented] (SPARK-8162) Run spark-shell cause NullPointerException

2015-12-10 Thread Michael Han (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050327#comment-15050327 ] Michael Han commented on SPARK-8162: Same with Aliaksei, Just got the same problem with

[jira] [Commented] (SPARK-12231) Failed to generate predicate Error when using dropna

2015-12-10 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050357#comment-15050357 ] Liang-Chi Hsieh commented on SPARK-12231: - I have opened a PR

[jira] [Commented] (SPARK-12247) Documentation for spark.ml's ALS and collaborative filtering in general

2015-12-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050420#comment-15050420 ] Sean Owen commented on SPARK-12247: --- [~timhunter] no thank you, go ahead > Documentation for

[jira] [Created] (SPARK-12261) pyspark crash for large dataset

2015-12-10 Thread zihao (JIRA)
zihao created SPARK-12261: - Summary: pyspark crash for large dataset Key: SPARK-12261 URL: https://issues.apache.org/jira/browse/SPARK-12261 Project: Spark Issue Type: Bug Affects Versions:

[jira] [Updated] (SPARK-10364) Support Parquet logical type TIMESTAMP_MILLIS

2015-12-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-10364: --- Description: The {{TimestampType}} in Spark SQL is of microsecond precision. Ideally, we should

[jira] [Commented] (SPARK-8743) Deregister Codahale metrics for streaming when StreamingContext is closed

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050296#comment-15050296 ] Apache Spark commented on SPARK-8743: - User 'nssalian' has created a pull request for this issue:

[jira] [Commented] (SPARK-8657) Fail to upload conf archive to viewfs

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050295#comment-15050295 ] Apache Spark commented on SPARK-8657: - User 'litao-buptsse' has created a pull request for this issue:

[jira] [Created] (SPARK-12262) describe extended doesn't return table on detail info tabled stored as PARQUET format

2015-12-10 Thread pin_zhang (JIRA)
pin_zhang created SPARK-12262: - Summary: describe extended doesn't return table on detail info tabled stored as PARQUET format Key: SPARK-12262 URL: https://issues.apache.org/jira/browse/SPARK-12262

[jira] [Created] (SPARK-12260) Graceful Shutdown with In-Memory State

2015-12-10 Thread Mao, Wei (JIRA)
Mao, Wei created SPARK-12260: Summary: Graceful Shutdown with In-Memory State Key: SPARK-12260 URL: https://issues.apache.org/jira/browse/SPARK-12260 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-7751) Add @Since annotation to stable and experimental methods in MLlib

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050289#comment-15050289 ] Apache Spark commented on SPARK-7751: - User 'petz2000' has created a pull request for this issue:

[jira] [Commented] (SPARK-12258) Hive Timestamp UDF is binded with '1969-12-31 15:59:59.999999' for null value

2015-12-10 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050398#comment-15050398 ] Xiao Li commented on SPARK-12258: - A PR has been submitted. Thanks > Hive Timestamp UDF is binded with

[jira] [Commented] (SPARK-12260) Graceful Shutdown with In-Memory State

2015-12-10 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050282#comment-15050282 ] Mao, Wei commented on SPARK-12260: -- Here is the design doc:

[jira] [Resolved] (SPARK-10366) Support Parquet logical type DATE

2015-12-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-10366. Resolution: Fixed Actually this has already been implemented since at least 1.4. > Support

[jira] [Commented] (SPARK-8657) Fail to upload conf archive to viewfs

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050473#comment-15050473 ] Apache Spark commented on SPARK-8657: - User 'litao-buptsse' has created a pull request for this issue:

[jira] [Commented] (SPARK-8657) Fail to upload conf archive to viewfs

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050471#comment-15050471 ] Apache Spark commented on SPARK-8657: - User 'litao-buptsse' has created a pull request for this issue:

[jira] [Commented] (SPARK-8657) Fail to upload conf archive to viewfs

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050470#comment-15050470 ] Apache Spark commented on SPARK-8657: - User 'litao-buptsse' has created a pull request for this issue:

[jira] [Commented] (SPARK-12257) Non partitioned insert into a partitioned Hive table doesn't fail

2015-12-10 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050319#comment-15050319 ] Dilip Biswal commented on SPARK-12257: -- Was able to reproduce this issue. Looking into it. > Non

[jira] [Commented] (SPARK-12148) SparkR: rename DataFrame to SparkDataFrame

2015-12-10 Thread Michael Lawrence (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051720#comment-15051720 ] Michael Lawrence commented on SPARK-12148: -- I agree. A deprecation cycle is good idea. >

[jira] [Commented] (SPARK-12269) Update aws-java-sdk version

2015-12-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051741#comment-15051741 ] Sean Owen commented on SPARK-12269: --- Ah, I realize Spark is on Jackson 2.4, not 2.5. If that difference

[jira] [Commented] (SPARK-6270) Standalone Master hangs when streaming job completes and event logging is enabled

2015-12-10 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051753#comment-15051753 ] Steve Loughran commented on SPARK-6270: --- [~shivaram] : can you have a look @ the logs and see what

[jira] [Commented] (SPARK-12267) Standalone master keeps references to disassociated workers until they sent no heartbeats

2015-12-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051683#comment-15051683 ] Marcelo Vanzin commented on SPARK-12267: Pasting Shixiong's comments from github

[jira] [Resolved] (SPARK-11713) Initial RDD for updateStateByKey for pyspark

2015-12-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11713. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 10082

[jira] [Updated] (SPARK-12212) Clarify the distinction between spark.mllib and spark.ml

2015-12-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-12212: -- Target Version/s: 1.6.0 > Clarify the distinction between spark.mllib and spark.ml >

[jira] [Commented] (SPARK-12217) Document invalid handling for StringIndexer

2015-12-10 Thread Benjamin Fradet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051698#comment-15051698 ] Benjamin Fradet commented on SPARK-12217: - Sorry [~srowen], my bad, I wanted to duplicate the

[jira] [Commented] (SPARK-4816) Maven profile netlib-lgpl does not work

2015-12-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051699#comment-15051699 ] Sean Owen commented on SPARK-4816: -- I just tried building the 1.4.1 tarball with -Pnetlib-lgpl and I see

[jira] [Commented] (SPARK-12148) SparkR: rename DataFrame to SparkDataFrame

2015-12-10 Thread Dan Putler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051716#comment-15051716 ] Dan Putler commented on SPARK-12148: Michael Lawrence's arguments are very valid. The S4Vector

[jira] [Commented] (SPARK-9578) Stemmer feature transformer

2015-12-10 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051775#comment-15051775 ] holdenk commented on SPARK-9578: [~yuhaoyan] Are you working on this? > Stemmer feature transformer >

[jira] [Resolved] (SPARK-12212) Clarify the distinction between spark.mllib and spark.ml

2015-12-10 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-12212. --- Resolution: Fixed Fix Version/s: 1.6.1 2.0.0 Issue

[jira] [Created] (SPARK-12269) Update aws-java-sdk version

2015-12-10 Thread Brian London (JIRA)
Brian London created SPARK-12269: Summary: Update aws-java-sdk version Key: SPARK-12269 URL: https://issues.apache.org/jira/browse/SPARK-12269 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-2870) Thorough schema inference directly on RDDs of Python dictionaries

2015-12-10 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051696#comment-15051696 ] holdenk commented on SPARK-2870: I can take a crack at implementing this if no one else is planning too.

[jira] [Comment Edited] (SPARK-12267) Standalone master keeps references to disassociated workers until they sent no heartbeats

2015-12-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051700#comment-15051700 ] Marcelo Vanzin edited comment on SPARK-12267 at 12/10/15 9:59 PM: --

[jira] [Created] (SPARK-12268) pyspark shell uses execfile which breaks python3 compatibility

2015-12-10 Thread Erik Selin (JIRA)
Erik Selin created SPARK-12268: -- Summary: pyspark shell uses execfile which breaks python3 compatibility Key: SPARK-12268 URL: https://issues.apache.org/jira/browse/SPARK-12268 Project: Spark

[jira] [Commented] (SPARK-12267) Standalone master keeps references to disassociated workers until they sent no heartbeats

2015-12-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051700#comment-15051700 ] Marcelo Vanzin commented on SPARK-12267: [~shixi...@databricks.com] you're right but that's the

[jira] [Issue Comment Deleted] (SPARK-12042) Python API for mllib.stat.test.StreamingTest

2015-12-10 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-12042: Comment: was deleted (was: I could take a crack at this since I've been giving some thought recently to

[jira] [Commented] (SPARK-12072) python dataframe ._jdf.schema().json() breaks on large metadata dataframes

2015-12-10 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051705#comment-15051705 ] holdenk commented on SPARK-12072: - Ok cool, let me take a look and see if there is another way to fix

[jira] [Commented] (SPARK-12267) Standalone master keeps references to disassociated workers until they sent no heartbeats

2015-12-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051732#comment-15051732 ] Marcelo Vanzin commented on SPARK-12267: I think the following would work. The problem right now

[jira] [Commented] (SPARK-12269) Update aws-java-sdk version

2015-12-10 Thread Brian London (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051733#comment-15051733 ] Brian London commented on SPARK-12269: -- The jackson 2.4 and 2.5 incompatibility has appeared

[jira] [Commented] (SPARK-12267) Standalone master keeps references to disassociated workers until they sent no heartbeats

2015-12-10 Thread dont_ping_this_account (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051757#comment-15051757 ] dont_ping_this_account commented on SPARK-12267: Sounds correct. But looks a lot changes

[jira] [Updated] (SPARK-12269) Update aws-java-sdk version

2015-12-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12269: -- Priority: Minor (was: Major) Makes some sense, but the question is always: are there any incompatible

[jira] [Resolved] (SPARK-11563) Use RpcEnv to transfer generated classes in spark-shell

2015-12-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-11563. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.0.0 > Use

[jira] [Updated] (SPARK-12258) Hive Timestamp UDF is binded with '1969-12-31 15:59:59.999999' for null value

2015-12-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-12258: - Target Version/s: 1.6.0 > Hive Timestamp UDF is binded with '1969-12-31 15:59:59.99' for null value

[jira] [Assigned] (SPARK-12213) Query with only one distinct should not having on expand

2015-12-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-12213: -- Assignee: Davies Liu > Query with only one distinct should not having on expand >

[jira] [Commented] (SPARK-12267) Standalone master keeps references to disassociated workers until they sent no heartbeats

2015-12-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051767#comment-15051767 ] Marcelo Vanzin commented on SPARK-12267: bq. I think we can just handle it in `NettyRpcHandler`

[jira] [Commented] (SPARK-12032) Filter can't be pushed down to correct Join because of bad order of Join

2015-12-10 Thread Min Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051829#comment-15051829 ] Min Qiu commented on SPARK-12032: - We had run into the same problem in our product development and we had

[jira] [Commented] (SPARK-11796) Docker JDBC integration tests fail in Maven build due to dependency issue

2015-12-10 Thread Mark Grover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051937#comment-15051937 ] Mark Grover commented on SPARK-11796: - Excellent, thank you! > Docker JDBC integration tests fail in

[jira] [Resolved] (SPARK-12258) Hive Timestamp UDF is binded with '1969-12-31 15:59:59.999999' for null value

2015-12-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-12258. -- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 10259

[jira] [Updated] (SPARK-12258) Hive Timestamp UDF is binded with '1969-12-31 15:59:59.999999' for null value

2015-12-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-12258: - Assignee: Davies Liu > Hive Timestamp UDF is binded with '1969-12-31 15:59:59.99' for null value >

[jira] [Assigned] (SPARK-12269) Update aws-java-sdk version

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12269: Assignee: Apache Spark > Update aws-java-sdk version > --- > >

[jira] [Commented] (SPARK-12217) Document invalid handling for StringIndexer

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051832#comment-15051832 ] Apache Spark commented on SPARK-12217: -- User 'BenFradet' has created a pull request for this issue:

[jira] [Commented] (SPARK-9578) Stemmer feature transformer

2015-12-10 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051915#comment-15051915 ] yuhao yang commented on SPARK-9578: --- Oh, I got a porter implementation now. I'll send it today or

[jira] [Commented] (SPARK-12267) Standalone master keeps references to disassociated workers until they sent no heartbeats

2015-12-10 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051952#comment-15051952 ] Shixiong Zhu commented on SPARK-12267: -- Could you send a PR quickly so that we can get the fix into

[jira] [Commented] (SPARK-12267) Standalone master keeps references to disassociated workers until they sent no heartbeats

2015-12-10 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051953#comment-15051953 ] Shixiong Zhu commented on SPARK-12267: -- If you don't have time, I can do it. > Standalone master

[jira] [Assigned] (SPARK-12268) pyspark shell uses execfile which breaks python3 compatibility

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12268: Assignee: Apache Spark > pyspark shell uses execfile which breaks python3 compatibility >

[jira] [Resolved] (SPARK-12251) Document Spark 1.6's off-heap memory configurations and add config validation

2015-12-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12251. --- Resolution: Fixed > Document Spark 1.6's off-heap memory configurations and add config validation >

[jira] [Updated] (SPARK-12251) Document Spark 1.6's off-heap memory configurations and add config validation

2015-12-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-12251: -- Fix Version/s: 1.6.0 > Document Spark 1.6's off-heap memory configurations and add config validation >

[jira] [Commented] (SPARK-11796) Docker JDBC integration tests fail in Maven build due to dependency issue

2015-12-10 Thread Mark Grover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051931#comment-15051931 ] Mark Grover commented on SPARK-11796: - Hey [~joshrosen], just checking if you have removed the

[jira] [Assigned] (SPARK-12155) Execution OOM after a relative large dataset cached in the cluster.

2015-12-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or reassigned SPARK-12155: - Assignee: Andrew Or (was: Josh Rosen) > Execution OOM after a relative large dataset cached in

[jira] [Resolved] (SPARK-12155) Execution OOM after a relative large dataset cached in the cluster.

2015-12-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12155. --- Resolution: Fixed Fix Version/s: 1.6.0 > Execution OOM after a relative large dataset cached

[jira] [Resolved] (SPARK-12253) UnifiedMemoryManager race condition: storage can starve new tasks

2015-12-10 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12253. --- Resolution: Fixed Fix Version/s: 1.6.0 > UnifiedMemoryManager race condition: storage can

[jira] [Commented] (SPARK-9578) Stemmer feature transformer

2015-12-10 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051917#comment-15051917 ] holdenk commented on SPARK-9578: Cool :) I'll keep an eye out for the PR :) > Stemmer feature transformer

[jira] [Commented] (SPARK-11796) Docker JDBC integration tests fail in Maven build due to dependency issue

2015-12-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051935#comment-15051935 ] Josh Rosen commented on SPARK-11796: Yep, I removed it from the Master and 1.6 Maven builds right

[jira] [Commented] (SPARK-12260) Graceful Shutdown with In-Memory State

2015-12-10 Thread Mao, Wei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051949#comment-15051949 ] Mao, Wei commented on SPARK-12260: -- In order to recover from process restarting, there are several

[jira] [Commented] (SPARK-11962) Add getAsOpt[T] functions to org.apache.spark.sql.Row

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050493#comment-15050493 ] Apache Spark commented on SPARK-11962: -- User 'aa8y' has created a pull request for this issue:

[jira] [Commented] (SPARK-12196) Store blocks in storage devices with hierarchy way

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050563#comment-15050563 ] Apache Spark commented on SPARK-12196: -- User 'yucai' has created a pull request for this issue:

[jira] [Resolved] (SPARK-12255) why "cache table a as select * from b" will do shuffle,and create 2 stages

2015-12-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12255. --- Resolution: Invalid Questions go to u...@spark.apache.org not JIRA > why "cache table a as select *

[jira] [Resolved] (SPARK-3106) Fix the race condition issue about Connection and ConnectionManager

2015-12-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-3106. -- Resolution: Incomplete > Fix the race condition issue about Connection and ConnectionManager >

[jira] [Commented] (SPARK-3461) Support external groupByKey using repartitionAndSortWithinPartitions

2015-12-10 Thread Pere Ferrera Bertran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050499#comment-15050499 ] Pere Ferrera Bertran commented on SPARK-3461: - Hi [~rxin], does this mean that the current

[jira] [Commented] (SPARK-3461) Support external groupByKey using repartitionAndSortWithinPartitions

2015-12-10 Thread Pere Ferrera Bertran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050537#comment-15050537 ] Pere Ferrera Bertran commented on SPARK-3461: - Made a question before, but I think it made not

[jira] [Commented] (SPARK-12260) Graceful Shutdown with In-Memory State

2015-12-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050906#comment-15050906 ] Sean Owen commented on SPARK-12260: --- Isn't this what updateStateByKey and similar methods already

[jira] [Assigned] (SPARK-12257) Non partitioned insert into a partitioned Hive table doesn't fail

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12257: Assignee: (was: Apache Spark) > Non partitioned insert into a partitioned Hive table

[jira] [Commented] (SPARK-12257) Non partitioned insert into a partitioned Hive table doesn't fail

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050587#comment-15050587 ] Apache Spark commented on SPARK-12257: -- User 'dilipbiswal' has created a pull request for this

[jira] [Commented] (SPARK-8498) Fix NullPointerException in error-handling path in UnsafeShuffleWriter

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050613#comment-15050613 ] Apache Spark commented on SPARK-8498: - User 'andrewor14' has created a pull request for this issue:

[jira] [Resolved] (SPARK-12261) pyspark crash for large dataset

2015-12-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12261. --- Resolution: Not A Problem I think it's pretty clear: you pulled a lot of data to your driver and it

[jira] [Issue Comment Deleted] (SPARK-3461) Support external groupByKey using repartitionAndSortWithinPartitions

2015-12-10 Thread Pere Ferrera Bertran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pere Ferrera Bertran updated SPARK-3461: Comment: was deleted (was: Hi [~rxin], does this mean that the current DataFrames

[jira] [Assigned] (SPARK-12257) Non partitioned insert into a partitioned Hive table doesn't fail

2015-12-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12257: Assignee: Apache Spark > Non partitioned insert into a partitioned Hive table doesn't

  1   2   3   >