[jira] [Comment Edited] (SPARK-19068) Large number of executors causing a ton of ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdat

2017-01-04 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800637#comment-15800637 ] Liang-Chi Hsieh edited comment on SPARK-19068 at 1/5/17 7:38 AM: - Does it

[jira] [Commented] (SPARK-19068) Large number of executors causing a ton of ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(41,

2017-01-04 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800637#comment-15800637 ] Liang-Chi Hsieh commented on SPARK-19068: - Does it affect the correctness of the results of the

[jira] [Commented] (SPARK-19081) spark sql use HIVE UDF throw exception when return a Map value

2017-01-04 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800602#comment-15800602 ] Liang-Chi Hsieh commented on SPARK-19081: - I want to make sure that this issue is happened on

[jira] [Resolved] (SPARK-19058) fix partition related behaviors with DataFrameWriter.saveAsTable

2017-01-04 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19058. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16460

[jira] [Updated] (SPARK-18877) Unable to read given csv data. Excepion: java.lang.IllegalArgumentException: requirement failed: Decimal precision 28 exceeds max precision 20

2017-01-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-18877: Fix Version/s: 2.0.3 > Unable to read given csv data. Excepion: java.lang.IllegalArgumentException: >

[jira] [Commented] (SPARK-19082) The config ignoreCorruptFiles doesn't work for Parquet

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800342#comment-15800342 ] Apache Spark commented on SPARK-19082: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19082) The config ignoreCorruptFiles doesn't work for Parquet

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19082: Assignee: (was: Apache Spark) > The config ignoreCorruptFiles doesn't work for

[jira] [Assigned] (SPARK-19082) The config ignoreCorruptFiles doesn't work for Parquet

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19082: Assignee: Apache Spark > The config ignoreCorruptFiles doesn't work for Parquet >

[jira] [Commented] (SPARK-18934) Writing to dynamic partitions does not preserve sort order if spill occurs

2017-01-04 Thread Charles Pritchard (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800323#comment-15800323 ] Charles Pritchard commented on SPARK-18934: --- Use case and other issues may be related to

[jira] [Created] (SPARK-19082) The config ignoreCorruptFiles doesn't work for Parquet

2017-01-04 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-19082: --- Summary: The config ignoreCorruptFiles doesn't work for Parquet Key: SPARK-19082 URL: https://issues.apache.org/jira/browse/SPARK-19082 Project: Spark

[jira] [Commented] (SPARK-5493) Support proxy users under kerberos

2017-01-04 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800304#comment-15800304 ] Ruslan Dautkhanov commented on SPARK-5493: -- Did you figure this out? Is this possible to use

[jira] [Commented] (SPARK-5158) Allow for keytab-based HDFS security in Standalone mode

2017-01-04 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800298#comment-15800298 ] Ruslan Dautkhanov commented on SPARK-5158: -- I think one reason for that could be that one user

[jira] [Comment Edited] (SPARK-19017) NOT IN subquery with more than one column may return incorrect results

2017-01-04 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799388#comment-15799388 ] Nattavut Sutyanyong edited comment on SPARK-19017 at 1/5/17 3:05 AM: -

[jira] [Comment Edited] (SPARK-19017) NOT IN subquery with more than one column may return incorrect results

2017-01-04 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799388#comment-15799388 ] Nattavut Sutyanyong edited comment on SPARK-19017 at 1/5/17 3:04 AM: -

[jira] [Comment Edited] (SPARK-19017) NOT IN subquery with more than one column may return incorrect results

2017-01-04 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799388#comment-15799388 ] Nattavut Sutyanyong edited comment on SPARK-19017 at 1/5/17 2:59 AM: -

[jira] [Created] (SPARK-19081) spark sql use HIVE UDF throw exception when return a Map value

2017-01-04 Thread Davy Song (JIRA)
Davy Song created SPARK-19081: - Summary: spark sql use HIVE UDF throw exception when return a Map value Key: SPARK-19081 URL: https://issues.apache.org/jira/browse/SPARK-19081 Project: Spark

[jira] [Updated] (SPARK-13385) Enable AssociationRules to generate consequents with user-defined lengths

2017-01-04 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-13385: - Affects Version/s: (was: 1.6.0) > Enable AssociationRules to generate consequents with

[jira] [Updated] (SPARK-13385) Enable AssociationRules to generate consequents with user-defined lengths

2017-01-04 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-13385: - Priority: Minor (was: Major) > Enable AssociationRules to generate consequents with

[jira] [Closed] (SPARK-11585) AssociationRules should generates all association rules with consequents of arbitrary length

2017-01-04 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng closed SPARK-11585. Resolution: Duplicate > AssociationRules should generates all association rules with consequents

[jira] [Updated] (SPARK-13385) Enable AssociationRules to generate consequents with user-defined lengths

2017-01-04 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-13385: - Component/s: (was: MLlib) ML > Enable AssociationRules to generate

[jira] [Commented] (SPARK-13385) Enable AssociationRules to generate consequents with user-defined lengths

2017-01-04 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800052#comment-15800052 ] zhengruifeng commented on SPARK-13385: -- Since mllib is in a maintainence mode, I close the PR. After

[jira] [Commented] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

2017-01-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799931#comment-15799931 ] Shixiong Zhu commented on SPARK-19013: -- Thanks, [~zzztimbo] That must be caused by the negative

[jira] [Commented] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

2017-01-04 Thread Tim Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799881#comment-15799881 ] Tim Chan commented on SPARK-19013: -- [~zsxwing] {code} Error: java.util.ConcurrentModificationException:

[jira] [Commented] (SPARK-14819) Improve the "SET" and "SET -v" command

2017-01-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799788#comment-15799788 ] Dongjoon Hyun commented on SPARK-14819: --- Hi, [~bomeng]. I hit the same issue in these days. Could

[jira] [Resolved] (SPARK-19009) Add doc for Streaming Rest API

2017-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-19009. Resolution: Fixed Assignee: Genmao Yu Fix Version/s: 2.2.0

[jira] [Updated] (SPARK-19009) Add doc for Streaming Rest API

2017-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-19009: --- Affects Version/s: (was: 2.0.2) (was: 2.1.0)

[jira] [Commented] (SPARK-19069) Expose task 'status' and 'duration' in spark history server REST API.

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799553#comment-15799553 ] Apache Spark commented on SPARK-19069: -- User 'paragpc' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19069) Expose task 'status' and 'duration' in spark history server REST API.

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19069: Assignee: (was: Apache Spark) > Expose task 'status' and 'duration' in spark history

[jira] [Assigned] (SPARK-19069) Expose task 'status' and 'duration' in spark history server REST API.

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19069: Assignee: Apache Spark > Expose task 'status' and 'duration' in spark history server REST

[jira] [Reopened] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

2017-01-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-19013: -- > java.util.ConcurrentModificationException when using s3 path as > checkpointLocation >

[jira] [Commented] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

2017-01-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799501#comment-15799501 ] Shixiong Zhu commented on SPARK-19013: -- [~zzztimbo] Could you provide the full stack trace so that I

[jira] [Comment Edited] (SPARK-19017) NOT IN subquery with more than one column may return incorrect results

2017-01-04 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799388#comment-15799388 ] Nattavut Sutyanyong edited comment on SPARK-19017 at 1/4/17 9:25 PM: -

[jira] [Commented] (SPARK-19017) NOT IN subquery with more than one column may return incorrect results

2017-01-04 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799388#comment-15799388 ] Nattavut Sutyanyong commented on SPARK-19017: - On c1: (1 <> 2) or (2 <> null) => true or

[jira] [Comment Edited] (SPARK-19017) NOT IN subquery with more than one column may return incorrect results

2017-01-04 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799374#comment-15799374 ] Herman van Hovell edited comment on SPARK-19017 at 1/4/17 9:16 PM: ---

[jira] [Commented] (SPARK-19017) NOT IN subquery with more than one column may return incorrect results

2017-01-04 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799374#comment-15799374 ] Herman van Hovell commented on SPARK-19017: --- Thanks! > NOT IN subquery with more than one

[jira] [Commented] (SPARK-19017) NOT IN subquery with more than one column may return incorrect results

2017-01-04 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799369#comment-15799369 ] Herman van Hovell commented on SPARK-19017: --- I agree that they are equal. It just seems weird

[jira] [Commented] (SPARK-19017) NOT IN subquery with more than one column may return incorrect results

2017-01-04 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799335#comment-15799335 ] Nattavut Sutyanyong commented on SPARK-19017: - I also have the output from a MySQL system.

[jira] [Comment Edited] (SPARK-17975) EMLDAOptimizer fails with ClassCastException on YARN

2017-01-04 Thread Jeff Stein (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799156#comment-15799156 ] Jeff Stein edited comment on SPARK-17975 at 1/4/17 7:47 PM: Attaching

[jira] [Updated] (SPARK-17975) EMLDAOptimizer fails with ClassCastException on YARN

2017-01-04 Thread Jeff Stein (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Stein updated SPARK-17975: --- Attachment: docs.txt Attaching vertical bar delimited documents (one per line). > EMLDAOptimizer

[jira] [Commented] (SPARK-18178) Importing Pandas Tables with Missing Values

2017-01-04 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799122#comment-15799122 ] Bryan Cutler commented on SPARK-18178: -- The error I get is {noformat} TypeError: Can not merge type

[jira] [Updated] (SPARK-19062) Utils.writeByteBuffer should not modify buffer position

2017-01-04 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout updated SPARK-19062: --- Affects Version/s: (was: 1.2.1) 2.1.0 > Utils.writeByteBuffer

[jira] [Resolved] (SPARK-19062) Utils.writeByteBuffer should not modify buffer position

2017-01-04 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout resolved SPARK-19062. Resolution: Fixed Fix Version/s: 2.2.0 > Utils.writeByteBuffer should not modify

[jira] [Commented] (SPARK-14804) Graph vertexRDD/EdgeRDD checkpoint results ClassCastException:

2017-01-04 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799094#comment-15799094 ] Joseph K. Bradley commented on SPARK-14804: --- I think this is a separate issue from local

[jira] [Commented] (SPARK-17265) EdgeRDD Difference throws an exception

2017-01-04 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799091#comment-15799091 ] Joseph K. Bradley commented on SPARK-17265: --- [~shishir167] Could you please give more info

[jira] [Commented] (SPARK-15984) WARN message "o.a.h.y.s.resourcemanager.rmapp.RMAppImpl: The specific max attempts: 0 for application: 8 is invalid" when starting application on YARN

2017-01-04 Thread Henry Kim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799087#comment-15799087 ] Henry Kim commented on SPARK-15984: --- the workaround that Saisai stated is to add the following config

[jira] [Commented] (SPARK-19017) NOT IN subquery with more than one column may return incorrect results

2017-01-04 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799055#comment-15799055 ] Nattavut Sutyanyong commented on SPARK-19017: - I think we both agree that the result of the

[jira] [Updated] (SPARK-17747) WeightCol support non-double datatypes

2017-01-04 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-17747: -- Priority: Minor (was: Major) > WeightCol support non-double datatypes >

[jira] [Updated] (SPARK-18206) Log instrumentation in MPC, NB, LDA, AFT, GLR, Isotonic, LinReg

2017-01-04 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18206: -- Shepherd: Joseph K. Bradley > Log instrumentation in MPC, NB, LDA, AFT, GLR, Isotonic,

[jira] [Commented] (SPARK-18194) Log instrumentation in OneVsRest, CrossValidator, TrainValidationSplit

2017-01-04 Thread Sue Ann Hong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799007#comment-15799007 ] Sue Ann Hong commented on SPARK-18194: -- I'll work on this :-) > Log instrumentation in OneVsRest,

[jira] [Commented] (SPARK-17169) To use scala macros to update code when SharedParamsCodeGen.scala changed

2017-01-04 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798989#comment-15798989 ] Joseph K. Bradley commented on SPARK-17169: --- Is this worthwhile? A lot of developers aren't

[jira] [Commented] (SPARK-18877) Unable to read given csv data. Excepion: java.lang.IllegalArgumentException: requirement failed: Decimal precision 28 exceeds max precision 20

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798945#comment-15798945 ] Apache Spark commented on SPARK-18877: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Updated] (SPARK-18877) Unable to read given csv data. Excepion: java.lang.IllegalArgumentException: requirement failed: Decimal precision 28 exceeds max precision 20

2017-01-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-18877: Fix Version/s: 2.1.1 > Unable to read given csv data. Excepion: java.lang.IllegalArgumentException: >

[jira] [Resolved] (SPARK-19079) Spark 1.6.1 SASL Error with Yarn

2017-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-19079. Resolution: Not A Problem This has all the symptoms of you using an external shuffle

[jira] [Commented] (SPARK-19017) NOT IN subquery with more than one column may return incorrect results

2017-01-04 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798758#comment-15798758 ] Herman van Hovell commented on SPARK-19017: --- Ok, my bad. Lets try this again. If I follow the

[jira] [Commented] (SPARK-18863) Output non-aggregate expressions without GROUP BY in a subquery does not yield an error

2017-01-04 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798578#comment-15798578 ] Nattavut Sutyanyong commented on SPARK-18863: - I made the conclusion too soon. It turns out

[jira] [Resolved] (SPARK-19047) Invalid correlated column may not be reported as an error

2017-01-04 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nattavut Sutyanyong resolved SPARK-19047. - Resolution: Duplicate This is just a variation of SPARK-18863. The fix will be

[jira] [Resolved] (SPARK-19070) Clean-up dataset actions

2017-01-04 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19070. - Resolution: Fixed Fix Version/s: 2.2.0 > Clean-up dataset actions >

[jira] [Commented] (SPARK-19080) simplify data source analysis

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798446#comment-15798446 ] Apache Spark commented on SPARK-19080: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19080) simplify data source analysis

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19080: Assignee: Wenchen Fan (was: Apache Spark) > simplify data source analysis >

[jira] [Assigned] (SPARK-19080) simplify data source analysis

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19080: Assignee: Apache Spark (was: Wenchen Fan) > simplify data source analysis >

[jira] [Created] (SPARK-19080) simplify data source analysis

2017-01-04 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-19080: --- Summary: simplify data source analysis Key: SPARK-19080 URL: https://issues.apache.org/jira/browse/SPARK-19080 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-19079) Spark 1.6.1 SASL Error with Yarn

2017-01-04 Thread Derek M Miller (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek M Miller updated SPARK-19079: --- Description: Currently, there seems to be an issue when using SASL in Spark with yarn with

[jira] [Created] (SPARK-19079) Spark 1.6.1 SASL Error with Yarn

2017-01-04 Thread Derek M Miller (JIRA)
Derek M Miller created SPARK-19079: -- Summary: Spark 1.6.1 SASL Error with Yarn Key: SPARK-19079 URL: https://issues.apache.org/jira/browse/SPARK-19079 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-19078) PCAModel.transform avoid extra vector conversion

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19078: Assignee: Apache Spark > PCAModel.transform avoid extra vector conversion >

[jira] [Assigned] (SPARK-19078) PCAModel.transform avoid extra vector conversion

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19078: Assignee: (was: Apache Spark) > PCAModel.transform avoid extra vector conversion >

[jira] [Commented] (SPARK-19078) PCAModel.transform avoid extra vector conversion

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798120#comment-15798120 ] Apache Spark commented on SPARK-19078: -- User 'zhengruifeng' has created a pull request for this

[jira] [Created] (SPARK-19078) PCAModel.transform avoid extra vector conversion

2017-01-04 Thread zhengruifeng (JIRA)
zhengruifeng created SPARK-19078: Summary: PCAModel.transform avoid extra vector conversion Key: SPARK-19078 URL: https://issues.apache.org/jira/browse/SPARK-19078 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-19054) Eliminate extra pass in NB

2017-01-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19054. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16453

[jira] [Updated] (SPARK-19054) Eliminate extra pass in NB

2017-01-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19054: -- Assignee: zhengruifeng > Eliminate extra pass in NB > -- > >

[jira] [Resolved] (SPARK-19077) no way to prevent spark streaming's eventLog inprogress file becoming too big?

2017-01-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19077. --- Resolution: Invalid Questions go to u...@spark.apache.org http://spark.apache.org/contributing.html

[jira] [Created] (SPARK-19077) no way to prevent spark streaming's eventLog inprogress file becoming too big?

2017-01-04 Thread Zhong Chunlai (JIRA)
Zhong Chunlai created SPARK-19077: - Summary: no way to prevent spark streaming's eventLog inprogress file becoming too big? Key: SPARK-19077 URL: https://issues.apache.org/jira/browse/SPARK-19077

[jira] [Resolved] (SPARK-19060) remove the supportsPartial flag in AggregateFunction

2017-01-04 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-19060. --- Resolution: Fixed Fix Version/s: 2.2.0 > remove the supportsPartial flag in

[jira] [Updated] (SPARK-19073) LauncherState should be only set to SUBMITTED after the application is submitted

2017-01-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19073: -- Assignee: shimingfei > LauncherState should be only set to SUBMITTED after the application is >

[jira] [Resolved] (SPARK-19073) LauncherState should be only set to SUBMITTED after the application is submitted

2017-01-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19073. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16459

[jira] [Resolved] (SPARK-19075) Plz make MinMaxScaler can work with a Number type field

2017-01-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19075. --- Resolution: Not A Problem (Please, edit your title and description. It's very sloppy.) Most

[jira] [Comment Edited] (SPARK-18948) Add Mean Percentile Rank metric for ranking algorithms

2017-01-04 Thread Danilo Ascione (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797848#comment-15797848 ] Danilo Ascione edited comment on SPARK-18948 at 1/4/17 10:16 AM: - Since

[jira] [Commented] (SPARK-18948) Add Mean Percentile Rank metric for ranking algorithms

2017-01-04 Thread Danilo Ascione (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797848#comment-15797848 ] Danilo Ascione commented on SPARK-18948: I am already working on it, I'll open a PR asap. Thank

[jira] [Commented] (SPARK-19053) Supporting multiple evaluation metrics in DataFrame-based API: discussion

2017-01-04 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797559#comment-15797559 ] zhengruifeng commented on SPARK-19053: -- I perfer Evaluator to Summary, in many cases we do not have

[jira] [Commented] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797548#comment-15797548 ] Apache Spark commented on SPARK-19033: -- User 'jerryshao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19033: Assignee: Apache Spark > HistoryServer still uses old ACLs even if ACLs are updated >

[jira] [Assigned] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated

2017-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19033: Assignee: (was: Apache Spark) > HistoryServer still uses old ACLs even if ACLs are