[jira] [Assigned] (SPARK-12197) Kryo's Avro Serializer add support for dynamic schemas using SchemaRepository

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12197: Assignee: (was: Apache Spark) > Kryo's Avro Serializer add support for dynamic

[jira] [Commented] (SPARK-12199) Follow-up: Refine example code in ml-features.md

2015-12-07 Thread Xusen Yin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046527#comment-15046527 ] Xusen Yin commented on SPARK-12199: --- I am working on it > Follow-up: Refine example code in

[jira] [Updated] (SPARK-12144) Support more external data source API in SparkR

2015-12-07 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-12144: Summary: Support more external data source API in SparkR (was: Implement DataFrameReader and

[jira] [Updated] (SPARK-12198) SparkR support read.parquet and deprecate parquetFile

2015-12-07 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-12198: Issue Type: Sub-task (was: Improvement) Parent: SPARK-12144 > SparkR support read.parquet

[jira] [Assigned] (SPARK-12198) SparkR support read.parquet and deprecate parquetFile

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12198: Assignee: Apache Spark > SparkR support read.parquet and deprecate parquetFile >

[jira] [Assigned] (SPARK-12198) SparkR support read.parquet and deprecate parquetFile

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12198: Assignee: (was: Apache Spark) > SparkR support read.parquet and deprecate parquetFile

[jira] [Updated] (SPARK-12160) spark.mllib should use SQLContext.getOrCreate

2015-12-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-12160: -- Fix Version/s: 1.6.1 > spark.mllib should use SQLContext.getOrCreate >

[jira] [Assigned] (SPARK-12197) Kryo's Avro Serializer add support for dynamic schemas using SchemaRepository

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12197: Assignee: Apache Spark > Kryo's Avro Serializer add support for dynamic schemas using

[jira] [Commented] (SPARK-12197) Kryo's Avro Serializer add support for dynamic schemas using SchemaRepository

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046528#comment-15046528 ] Apache Spark commented on SPARK-12197: -- User 'RotemShaul' has created a pull request for this issue:

[jira] [Created] (SPARK-12197) Kryo's Avro Serializer add support for dynamic schemas using SchemaRepository

2015-12-07 Thread Rotem Shaul (JIRA)
Rotem Shaul created SPARK-12197: --- Summary: Kryo's Avro Serializer add support for dynamic schemas using SchemaRepository Key: SPARK-12197 URL: https://issues.apache.org/jira/browse/SPARK-12197 Project:

[jira] [Updated] (SPARK-12197) Kryo's Avro Serializer add support for dynamic schemas using SchemaRepository

2015-12-07 Thread Rotem Shaul (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rotem Shaul updated SPARK-12197: Description: The original problem: Serializing GenericRecords in Spark Core results in a very

[jira] [Resolved] (SPARK-11551) Replace example code in ml-features.md using include_example

2015-12-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-11551. --- Resolution: Fixed Fix Version/s: 1.6.1 Issue resolved by pull request 10002

[jira] [Updated] (SPARK-12144) Support more external data source API in SparkR

2015-12-07 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-12144: Description: Currently we support read.df, write.df, jsonFile, parquetFile in SQLContext, we

[jira] [Assigned] (SPARK-12196) Store blocks in storage devices with hierarchy way

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12196: Assignee: Apache Spark > Store blocks in storage devices with hierarchy way >

[jira] [Commented] (SPARK-12196) Store blocks in storage devices with hierarchy way

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046544#comment-15046544 ] Apache Spark commented on SPARK-12196: -- User 'yucai' has created a pull request for this issue:

[jira] [Commented] (SPARK-12197) Kryo's Avro Serializer add support for dynamic schemas using SchemaRepository

2015-12-07 Thread Rotem Shaul (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046530#comment-15046530 ] Rotem Shaul commented on SPARK-12197: - Implemented solution as described above. Added new spark

[jira] [Resolved] (SPARK-10259) Add @Since annotation to ml.classification

2015-12-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-10259. --- Resolution: Fixed Fix Version/s: 1.6.1 Issue resolved by pull request 8534

[jira] [Updated] (SPARK-12196) Store blocks in storage devices with hierarchy way

2015-12-07 Thread yucai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yucai updated SPARK-12196: -- Description: Problem: Nowadays, users have both SSDs and HDDs. SSDs have great performance, but

[jira] [Closed] (SPARK-10290) Spark can register temp table and hive table with the same table name

2015-12-07 Thread pin_zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pin_zhang closed SPARK-10290. - not a bug > Spark can register temp table and hive table with the same table name >

[jira] [Assigned] (SPARK-12170) Deprecate the JAVA-specific deserialized storage levels

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12170: Assignee: Apache Spark > Deprecate the JAVA-specific deserialized storage levels >

[jira] [Assigned] (SPARK-12170) Deprecate the JAVA-specific deserialized storage levels

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12170: Assignee: (was: Apache Spark) > Deprecate the JAVA-specific deserialized storage

[jira] [Commented] (SPARK-12170) Deprecate the JAVA-specific deserialized storage levels

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046429#comment-15046429 ] Apache Spark commented on SPARK-12170: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Updated] (SPARK-11964) Create user guide section explaining export/import

2015-12-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-11964: -- Shepherd: Joseph K. Bradley > Create user guide section explaining export/import >

[jira] [Created] (SPARK-12198) SparkR support read.parquet and deprecate parquetFile

2015-12-07 Thread Yanbo Liang (JIRA)
Yanbo Liang created SPARK-12198: --- Summary: SparkR support read.parquet and deprecate parquetFile Key: SPARK-12198 URL: https://issues.apache.org/jira/browse/SPARK-12198 Project: Spark Issue

[jira] [Resolved] (SPARK-11957) SQLTransformer docs are unclear about generality of SQL statements

2015-12-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-11957. --- Resolution: Fixed Fix Version/s: 1.6.1 Issue resolved by pull request 10006

[jira] [Resolved] (SPARK-11958) Create user guide section for SQLTransformer

2015-12-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-11958. --- Resolution: Fixed Fix Version/s: 1.6.1 Issue resolved by pull request 10006

[jira] [Updated] (SPARK-11551) Replace example code in ml-features.md using include_example

2015-12-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-11551: -- Assignee: somil deshmukh > Replace example code in ml-features.md using include_example >

[jira] [Resolved] (SPARK-12160) spark.mllib should use SQLContext.getOrCreate

2015-12-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-12160. --- Resolution: Fixed Fix Version/s: (was: 1.6.1) 1.5.3 Issue

[jira] [Updated] (SPARK-12160) spark.mllib should use SQLContext.getOrCreate

2015-12-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-12160: -- Target Version/s: 1.5.3, 1.6.1 (was: 1.4.2, 1.5.3, 1.6.1) > spark.mllib should use

[jira] [Assigned] (SPARK-12196) Store blocks in storage devices with hierarchy way

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12196: Assignee: (was: Apache Spark) > Store blocks in storage devices with hierarchy way >

[jira] [Updated] (SPARK-12144) Support more external data source API in SparkR

2015-12-07 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-12144: Description: Currently we support read.df, write.df, jsonFile, parquetFile in SQLContext, we

[jira] [Commented] (SPARK-11487) Spark Master shutdown automatically after some applications execution

2015-12-07 Thread Xu Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046438#comment-15046438 ] Xu Chen commented on SPARK-11487: - {code} java.lang.OutOfMemoryError: Java heap space {code} Increment

[jira] [Created] (SPARK-12196) Store blocks in storage devices with hierarchy way

2015-12-07 Thread yucai (JIRA)
yucai created SPARK-12196: - Summary: Store blocks in storage devices with hierarchy way Key: SPARK-12196 URL: https://issues.apache.org/jira/browse/SPARK-12196 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-12199) Follow-up: Refine example code in ml-features.md

2015-12-07 Thread Xusen Yin (JIRA)
Xusen Yin created SPARK-12199: - Summary: Follow-up: Refine example code in ml-features.md Key: SPARK-12199 URL: https://issues.apache.org/jira/browse/SPARK-12199 Project: Spark Issue Type:

[jira] [Updated] (SPARK-12199) Follow-up: Refine example code in ml-features.md

2015-12-07 Thread Xusen Yin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xusen Yin updated SPARK-12199: -- Fix Version/s: (was: 1.6.1) > Follow-up: Refine example code in ml-features.md >

[jira] [Commented] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045028#comment-15045028 ] Sean Owen commented on SPARK-12179: --- Yes, that's the critical information. I think it's hard to parse

[jira] [Commented] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045014#comment-15045014 ] Tao Li commented on SPARK-12179: [~sowen] My sql and command line param is like this: DATE=$1

[jira] [Commented] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045053#comment-15045053 ] Tao Li commented on SPARK-12179: The query is on a hive table and the hive data is not changing. I think

[jira] [Commented] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045012#comment-15045012 ] Tao Li commented on SPARK-12179: In the same stage with the same shuffle read, I think it should get the

[jira] [Comment Edited] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045053#comment-15045053 ] Tao Li edited comment on SPARK-12179 at 12/7/15 3:21 PM: - The query is on a hive

[jira] [Commented] (SPARK-7727) Avoid inner classes in RuleExecutor

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044996#comment-15044996 ] Apache Spark commented on SPARK-7727: - User 'stephankessler' has created a pull request for this

[jira] [Commented] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045015#comment-15045015 ] Sean Owen commented on SPARK-12179: --- I don't see that you've shown results at all. What do you expect

[jira] [Created] (SPARK-12180) DataFrame.join() in PySpark gives misleading exception when column name exists on both side

2015-12-07 Thread Daniel Thomas (JIRA)
Daniel Thomas created SPARK-12180: - Summary: DataFrame.join() in PySpark gives misleading exception when column name exists on both side Key: SPARK-12180 URL: https://issues.apache.org/jira/browse/SPARK-12180

[jira] [Commented] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045010#comment-15045010 ] Tao Li commented on SPARK-12179: [~srowen] In this case, there is no task failures and I set

[jira] [Commented] (SPARK-12124) Spark Sql MongoDB Cross Join Not Working

2015-12-07 Thread himanshu singhal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045008#comment-15045008 ] himanshu singhal commented on SPARK-12124: -- I am using the hadoop mongo dependency for the

[jira] [Commented] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045024#comment-15045024 ] Tao Li commented on SPARK-12179: For example: 1. First Time Run, I got: 20151204maingoogle

[jira] [Commented] (SPARK-5095) Support launching multiple mesos executors in coarse grained mesos mode

2015-12-07 Thread YONG FENG (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045281#comment-15045281 ] YONG FENG commented on SPARK-5095: -- My team is now working on [MESOS-938]. After that, we actually could

[jira] [Commented] (SPARK-8118) Turn off noisy log output produced by Parquet 1.7.0

2015-12-07 Thread Justin Bailey (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045325#comment-15045325 ] Justin Bailey commented on SPARK-8118: -- Did some research and Parquet uses java.util.logging. You can

[jira] [Comment Edited] (SPARK-8855) Python API for Association Rules

2015-12-07 Thread Caique Rodrigues Marques (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037211#comment-15037211 ] Caique Rodrigues Marques edited comment on SPARK-8855 at 12/7/15 4:10 PM:

[jira] [Commented] (SPARK-7101) Spark SQL should support java.sql.Time

2015-12-07 Thread kevin yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045318#comment-15045318 ] kevin yu commented on SPARK-7101: - I will work on this. > Spark SQL should support java.sql.Time >

[jira] [Created] (SPARK-12181) Check Cached unaligned-access capability before using Unsafe

2015-12-07 Thread Ted Yu (JIRA)
Ted Yu created SPARK-12181: -- Summary: Check Cached unaligned-access capability before using Unsafe Key: SPARK-12181 URL: https://issues.apache.org/jira/browse/SPARK-12181 Project: Spark Issue

[jira] [Commented] (SPARK-12181) Check Cached unaligned-access capability before using Unsafe

2015-12-07 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045309#comment-15045309 ] Ted Yu commented on SPARK-12181: We can add method to Platform which performs check on the architecture.

[jira] [Commented] (SPARK-12124) Spark Sql MongoDB Cross Join Not Working

2015-12-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044653#comment-15044653 ] Sean Owen commented on SPARK-12124: --- [~himanshu.techsk...@gmail.com] are you using this 3rd party

[jira] [Comment Edited] (SPARK-4036) Add Conditional Random Fields (CRF) algorithm to Spark MLlib

2015-12-07 Thread hujiayin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010353#comment-15010353 ] hujiayin edited comment on SPARK-4036 at 12/7/15 8:10 AM: -- Hi Sasaki, I'm not

[jira] [Updated] (SPARK-12176) SparkLauncher's setConf() does not support configs containing spaces

2015-12-07 Thread Yuhang Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuhang Chen updated SPARK-12176: Flags: Important > SparkLauncher's setConf() does not support configs containing spaces >

[jira] [Commented] (SPARK-12175) Add new flag to Spark that identify if the driver run on application servers.

2015-12-07 Thread Sasi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044644#comment-15044644 ] Sasi commented on SPARK-12175: -- Yes my driver runs on JBoss. I search same issues, and the fix was to remove

[jira] [Commented] (SPARK-11938) Expose numFeatures in all ML PredictionModel for PySpark

2015-12-07 Thread Kai Sasaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044673#comment-15044673 ] Kai Sasaki commented on SPARK-11938: [~mengxr] [~yanboliang] Sorry for bothering you, but could you

[jira] [Resolved] (SPARK-12106) Flaky Test: BatchedWriteAheadLog - name log with aggregated entries with the timestamp of last entry

2015-12-07 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-12106. --- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 1.6.0 > Flaky Test:

[jira] [Created] (SPARK-12175) Add new flag to Spark that identify if the driver run on application servers.

2015-12-07 Thread Sasi (JIRA)
Sasi created SPARK-12175: Summary: Add new flag to Spark that identify if the driver run on application servers. Key: SPARK-12175 URL: https://issues.apache.org/jira/browse/SPARK-12175 Project: Spark

[jira] [Resolved] (SPARK-12175) Add new flag to Spark that identify if the driver run on application servers.

2015-12-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12175. --- Resolution: Duplicate [~sasi2103] this is the same issue that you already opened. Please just keep

[jira] [Updated] (SPARK-12176) SparkLauncher's setConf() does not support configs containing spaces

2015-12-07 Thread Yuhang Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuhang Chen updated SPARK-12176: Description: The spark-submit uses '--conf K=V' pattern for setting configs. According to the

[jira] [Issue Comment Deleted] (SPARK-12048) JDBCRDD calls close() twice - SQLite then throws an exception

2015-12-07 Thread R. H. (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] R. H. updated SPARK-12048: -- Comment: was deleted (was: Thanks for including that fix so fast. I have just tested to fix in the master

[jira] [Updated] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

2015-12-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12103: -- Fix Version/s: (was: 1.4.2) > KafkaUtils createStream with multiple topics -- does not work as

[jira] [Created] (SPARK-12177) Update KafkaDStreams to new Kafka 0.9 Consumer API

2015-12-07 Thread Nikita Tarasenko (JIRA)
Nikita Tarasenko created SPARK-12177: Summary: Update KafkaDStreams to new Kafka 0.9 Consumer API Key: SPARK-12177 URL: https://issues.apache.org/jira/browse/SPARK-12177 Project: Spark

[jira] [Commented] (SPARK-12162) Embedded Spark on JBoss server cause crashing due system.exit when SparkUncaughtExceptionHandler called

2015-12-07 Thread Sasi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044726#comment-15044726 ] Sasi commented on SPARK-12162: -- Yes my driver runs on JBoss. > Embedded Spark on JBoss server cause

[jira] [Reopened] (SPARK-12177) Update KafkaDStreams to new Kafka 0.9 Consumer API

2015-12-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-12177: --- This can stay open as an item targeted for 2.0.0. Your PR can stay closed for now but can be revived

[jira] [Updated] (SPARK-12177) Update KafkaDStreams to new Kafka 0.9 Consumer API

2015-12-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12177: -- Target Version/s: 2.0.0 > Update KafkaDStreams to new Kafka 0.9 Consumer API >

[jira] [Commented] (SPARK-12178) Expose reporting of StreamInputInfo for custom made streams

2015-12-07 Thread Rodrigo Boavida (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044846#comment-15044846 ] Rodrigo Boavida commented on SPARK-12178: - For any new implementation of a custom stream. For

[jira] [Assigned] (SPARK-12177) Update KafkaDStreams to new Kafka 0.9 Consumer API

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12177: Assignee: (was: Apache Spark) > Update KafkaDStreams to new Kafka 0.9 Consumer API >

[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.9 Consumer API

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044799#comment-15044799 ] Apache Spark commented on SPARK-12177: -- User 'ntarasenko' has created a pull request for this issue:

[jira] [Created] (SPARK-12178) Expose reporting of StreamInputInfo for custom made streams

2015-12-07 Thread Rodrigo Boavida (JIRA)
Rodrigo Boavida created SPARK-12178: --- Summary: Expose reporting of StreamInputInfo for custom made streams Key: SPARK-12178 URL: https://issues.apache.org/jira/browse/SPARK-12178 Project: Spark

[jira] [Commented] (SPARK-12178) Expose reporting of StreamInputInfo for custom made streams

2015-12-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044829#comment-15044829 ] Sean Owen commented on SPARK-12178: --- What is the use case for this? > Expose reporting of

[jira] [Assigned] (SPARK-12177) Update KafkaDStreams to new Kafka 0.9 Consumer API

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12177: Assignee: Apache Spark > Update KafkaDStreams to new Kafka 0.9 Consumer API >

[jira] [Closed] (SPARK-12177) Update KafkaDStreams to new Kafka 0.9 Consumer API

2015-12-07 Thread Nikita Tarasenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Tarasenko closed SPARK-12177. Resolution: Later This kind of change would have to wait for Spark 2.x. > Update

[jira] [Commented] (SPARK-12178) Expose reporting of StreamInputInfo for custom made streams

2015-12-07 Thread Rodrigo Boavida (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044850#comment-15044850 ] Rodrigo Boavida commented on SPARK-12178: - This is probably the first step onto the effort of

[jira] [Commented] (SPARK-9184) spark.task.cpus under Mesos should accept floating point numbers (partial CPU)

2015-12-07 Thread Brandon Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044851#comment-15044851 ] Brandon Bradley commented on SPARK-9184: Hello! Can we have some discussion? We can use something

[jira] [Created] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
Tao Li created SPARK-12179: -- Summary: Spark SQL get different result with the same code Key: SPARK-12179 URL: https://issues.apache.org/jira/browse/SPARK-12179 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-9184) spark.task.cpus under Mesos should accept floating point numbers (partial CPU)

2015-12-07 Thread Brandon Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Bradley updated SPARK-9184: --- Priority: Major (was: Minor) > spark.task.cpus under Mesos should accept floating point

[jira] [Updated] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated SPARK-12179: --- Description: I run the sql in yarn-client mode, but get different result each time. As you can see the

[jira] [Updated] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated SPARK-12179: --- Description: I run the sql in yarn-client mode, but get different result each time. As you can see the

[jira] [Issue Comment Deleted] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated SPARK-12179: --- Comment: was deleted (was: [~liancheng] [~sowen ][~andrewor14] [~rxin]) > Spark SQL get different result

[jira] [Updated] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12179: -- Priority: Minor (was: Major) What's the problem here? you're not actually getting different *query*

[jira] [Updated] (SPARK-12176) SparkLauncher's setConf() does not support configs containing spaces

2015-12-07 Thread Yuhang Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuhang Chen updated SPARK-12176: Description: The spark-submit uses '--conf K=V' pattern for setting configs. According to the

[jira] [Created] (SPARK-12188) [SQL] Code refactoring and comment correction in Dataset APIs

2015-12-07 Thread Xiao Li (JIRA)
Xiao Li created SPARK-12188: --- Summary: [SQL] Code refactoring and comment correction in Dataset APIs Key: SPARK-12188 URL: https://issues.apache.org/jira/browse/SPARK-12188 Project: Spark Issue

[jira] [Assigned] (SPARK-12188) [SQL] Code refactoring and comment correction in Dataset APIs

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12188: Assignee: (was: Apache Spark) > [SQL] Code refactoring and comment correction in

[jira] [Commented] (SPARK-12188) [SQL] Code refactoring and comment correction in Dataset APIs

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046196#comment-15046196 ] Apache Spark commented on SPARK-12188: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Updated] (SPARK-12176) SparkLauncher's setConf() does not support configs containing spaces

2015-12-07 Thread Yuhang Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuhang Chen updated SPARK-12176: Description: The spark-submit uses '--conf K=V' pattern for setting configs. According to the

[jira] [Assigned] (SPARK-12188) [SQL] Code refactoring and comment correction in Dataset APIs

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12188: Assignee: Apache Spark > [SQL] Code refactoring and comment correction in Dataset APIs >

[jira] [Updated] (SPARK-12176) SparkLauncher's setConf() does not support configs containing spaces

2015-12-07 Thread Yuhang Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuhang Chen updated SPARK-12176: Description: The spark-submit uses '--conf K=V' pattern for setting configs. According to the

[jira] [Commented] (SPARK-12176) SparkLauncher's setConf() does not support configs containing spaces

2015-12-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046205#comment-15046205 ] Marcelo Vanzin commented on SPARK-12176: As I tried to explain, you need to show (i) code and

[jira] [Created] (SPARK-12189) UnifiedMemoryManager double counts storage memory freed

2015-12-07 Thread Andrew Or (JIRA)
Andrew Or created SPARK-12189: - Summary: UnifiedMemoryManager double counts storage memory freed Key: SPARK-12189 URL: https://issues.apache.org/jira/browse/SPARK-12189 Project: Spark Issue

[jira] [Assigned] (SPARK-12189) UnifiedMemoryManager double counts storage memory freed

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12189: Assignee: Apache Spark (was: Josh Rosen) > UnifiedMemoryManager double counts storage

[jira] [Commented] (SPARK-12189) UnifiedMemoryManager double counts storage memory freed

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046236#comment-15046236 ] Apache Spark commented on SPARK-12189: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12189) UnifiedMemoryManager double counts storage memory freed

2015-12-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12189: Assignee: Josh Rosen (was: Apache Spark) > UnifiedMemoryManager double counts storage

[jira] [Commented] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046240#comment-15046240 ] Tao Li commented on SPARK-12179: yes, my spark version is 1.5.3-SNAPSHOT,and the issue SPARK-11009 is

[jira] [Issue Comment Deleted] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated SPARK-12179: --- Comment: was deleted (was: yes, my spark version is 1.5.3-SNAPSHOT,and the issue SPARK-11009 is already

[jira] [Issue Comment Deleted] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated SPARK-12179: --- Comment: was deleted (was: yes, my spark version is 1.5.3-SNAPSHOT,and the issue SPARK-11009 is already

[jira] [Commented] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046238#comment-15046238 ] Tao Li commented on SPARK-12179: yes, my spark version is 1.5.3-SNAPSHOT,and the issue SPARK-11009 is

[jira] [Commented] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046239#comment-15046239 ] Tao Li commented on SPARK-12179: yes, my spark version is 1.5.3-SNAPSHOT,and the issue SPARK-11009 is

[jira] [Commented] (SPARK-12179) Spark SQL get different result with the same code

2015-12-07 Thread Tao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046243#comment-15046243 ] Tao Li commented on SPARK-12179: I see there was some exceptions in my executors stderr log: ```

  1   2   3   >