[jira] [Commented] (SPARK-21306) OneVsRest Conceals Columns That May Be Relevant To Underlying Classifier

2017-07-05 Thread 颜发才
[ https://issues.apache.org/jira/browse/SPARK-21306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075974#comment-16075974 ] Yan Facai (颜发才) commented on SPARK-21306: - [~cathalgarvey] By the way, since LogisticRegression

[jira] [Comment Edited] (SPARK-21306) OneVsRest Conceals Columns That May Be Relevant To Underlying Classifier

2017-07-05 Thread 颜发才
[ https://issues.apache.org/jira/browse/SPARK-21306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075970#comment-16075970 ] Yan Facai (颜发才) edited comment on SPARK-21306 at 7/6/17 5:40 AM: - I agree

[jira] [Commented] (SPARK-21306) OneVsRest Conceals Columns That May Be Relevant To Underlying Classifier

2017-07-05 Thread 颜发才
[ https://issues.apache.org/jira/browse/SPARK-21306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075970#comment-16075970 ] Yan Facai (颜发才) commented on SPARK-21306: - I agree with [~n...@svana.org]. It seems that we will

[jira] [Commented] (SPARK-12157) Support numpy types as return values of Python UDFs

2017-07-05 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075886#comment-16075886 ] Li Jin commented on SPARK-12157: [~zero323], do you have any updates for this issue? > Support numpy

[jira] [Updated] (SPARK-21100) Add summary method as alternative to describe that gives quartiles similar to Pandas

2017-07-05 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ray updated SPARK-21100: --- Summary: Add summary method as alternative to describe that gives quartiles similar to Pandas (was:

[jira] [Resolved] (SPARK-21311) Not able to fetch the data from RDBMS when two spark jobs are accessing the same Database table

2017-07-05 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21311. -- Resolution: Cannot Reproduce I am resolving this. It sounds almost impossible for me to

[jira] [Commented] (SPARK-21277) Spark is invoking an incorrect serializer after UDAF completion

2017-07-05 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075818#comment-16075818 ] Erik Erlandson commented on SPARK-21277: It would be ideal to document the requirement that all

[jira] [Assigned] (SPARK-21324) Improve statistics test suites

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21324: Assignee: (was: Apache Spark) > Improve statistics test suites >

[jira] [Commented] (SPARK-21324) Improve statistics test suites

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075794#comment-16075794 ] Apache Spark commented on SPARK-21324: -- User 'wzhfy' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21324) Improve statistics test suites

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21324: Assignee: Apache Spark > Improve statistics test suites > --

[jira] [Resolved] (SPARK-21248) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite.assign from specific offsets (failOnDataLoss: true)

2017-07-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-21248. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 2.3.0 > Flaky test:

[jira] [Created] (SPARK-21324) Improve statistics test suites

2017-07-05 Thread Zhenhua Wang (JIRA)
Zhenhua Wang created SPARK-21324: Summary: Improve statistics test suites Key: SPARK-21324 URL: https://issues.apache.org/jira/browse/SPARK-21324 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-20383) SparkSQL unsupports to create function with the key word 'OR REPLACE' and 'IF NOT EXISTS'

2017-07-05 Thread Xiaochen Ouyang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaochen Ouyang updated SPARK-20383: Summary: SparkSQL unsupports to create function with the key word 'OR REPLACE' and 'IF NOT

[jira] [Updated] (SPARK-20383) SparkSQL unsupports to create function with the keyword 'OR REPLACE' and 'IF NOT EXISTS'

2017-07-05 Thread Xiaochen Ouyang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaochen Ouyang updated SPARK-20383: Summary: SparkSQL unsupports to create function with the keyword 'OR REPLACE' and 'IF NOT

[jira] [Closed] (SPARK-21298) The class "StorageListener" method "onStageSubmitted" don't need to change name when don't exist corresponding key

2017-07-05 Thread he.qiao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] he.qiao closed SPARK-21298. --- Resolution: Not A Problem > The class "StorageListener" method "onStageSubmitted" don't need to change >

[jira] [Commented] (SPARK-21323) Rename sql.catalyst.plans.logical.statsEstimation.Range to ValueInterval

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075673#comment-16075673 ] Apache Spark commented on SPARK-21323: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-21323) Rename sql.catalyst.plans.logical.statsEstimation.Range to ValueInterval

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21323: Assignee: (was: Apache Spark) > Rename

[jira] [Assigned] (SPARK-21323) Rename sql.catalyst.plans.logical.statsEstimation.Range to ValueInterval

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21323: Assignee: Apache Spark > Rename sql.catalyst.plans.logical.statsEstimation.Range to

[jira] [Created] (SPARK-21323) Rename sql.catalyst.plans.logical.statsEstimation.Range to ValueInterval

2017-07-05 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-21323: -- Summary: Rename sql.catalyst.plans.logical.statsEstimation.Range to ValueInterval Key: SPARK-21323 URL: https://issues.apache.org/jira/browse/SPARK-21323

[jira] [Resolved] (SPARK-21278) Upgrade to Py4J 0.10.6

2017-07-05 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-21278. - Resolution: Fixed Issue resolved by pull request 18546 [https://github.com/apache/spark/pull/18546] >

[jira] [Assigned] (SPARK-21278) Upgrade to Py4J 0.10.6

2017-07-05 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-21278: --- Assignee: Dongjoon Hyun > Upgrade to Py4J 0.10.6 > -- > > Key:

[jira] [Updated] (SPARK-21278) Upgrade to Py4J 0.10.6

2017-07-05 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-21278: Fix Version/s: 2.3.0 > Upgrade to Py4J 0.10.6 > -- > > Key:

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

2017-07-05 Thread Anthony Truchet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075600#comment-16075600 ] Anthony Truchet commented on SPARK-18838: - Hello, We (Criteo Predictive Search team with

[jira] [Commented] (SPARK-21273) Decouple stats propagation from logical plan

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075504#comment-16075504 ] Apache Spark commented on SPARK-21273: -- User 'gengliangwang' has created a pull request for this

[jira] [Commented] (SPARK-15491) JSON serialization fails for JDBC DataFrames

2017-07-05 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075491#comment-16075491 ] Huaxin Gao commented on SPARK-15491: Hi Marc, Is there a use case you need to convert JDBCRelation to

[jira] [Commented] (SPARK-18859) Catalyst codegen does not mark column as nullable when it should. Causes NPE

2017-07-05 Thread Anton Okolnychyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075432#comment-16075432 ] Anton Okolnychyi commented on SPARK-18859: -- I took a look at this issue and seems this cannot be

[jira] [Created] (SPARK-21322) support histogram in filter cardinality estimation

2017-07-05 Thread Ron Hu (JIRA)
Ron Hu created SPARK-21322: -- Summary: support histogram in filter cardinality estimation Key: SPARK-21322 URL: https://issues.apache.org/jira/browse/SPARK-21322 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-21321) Spark very verbose on shutdown confusing users

2017-07-05 Thread Jong Yoon Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jong Yoon Lee updated SPARK-21321: -- Description: on shutdown spark can be very verbose and spit out errors that cause the user to

[jira] [Assigned] (SPARK-21321) Spark very verbose on shutdown confusing users

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21321: Assignee: (was: Apache Spark) > Spark very verbose on shutdown confusing users >

[jira] [Assigned] (SPARK-21321) Spark very verbose on shutdown confusing users

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21321: Assignee: Apache Spark > Spark very verbose on shutdown confusing users >

[jira] [Commented] (SPARK-21321) Spark very verbose on shutdown confusing users

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075203#comment-16075203 ] Apache Spark commented on SPARK-21321: -- User 'yoonlee95' has created a pull request for this issue:

[jira] [Commented] (SPARK-21278) Upgrade to Py4J 0.10.6

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075192#comment-16075192 ] Apache Spark commented on SPARK-21278: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-16803) SaveAsTable does not work when source DataFrame is built on a Hive Table

2017-07-05 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075178#comment-16075178 ] Ruslan Dautkhanov commented on SPARK-16803: --- Any chance `saveAsTable` can be reverted to use

[jira] [Resolved] (SPARK-19439) PySpark's registerJavaFunction Should Support UDAFs

2017-07-05 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19439. - Resolution: Fixed Fix Version/s: 2.3.0 > PySpark's registerJavaFunction Should Support UDAFs >

[jira] [Assigned] (SPARK-19439) PySpark's registerJavaFunction Should Support UDAFs

2017-07-05 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-19439: --- Assignee: Jeff Zhang > PySpark's registerJavaFunction Should Support UDAFs >

[jira] [Commented] (SPARK-21122) Address starvation issues when dynamic allocation is enabled

2017-07-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075147#comment-16075147 ] Thomas Graves commented on SPARK-21122: --- I agree with Sean on this. if you are aiming this at

[jira] [Commented] (SPARK-21316) Dataset Union output is not consistent with the column sequence

2017-07-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075133#comment-16075133 ] Dongjoon Hyun commented on SPARK-21316: --- For your case, could you try the following? {code} -

[jira] [Updated] (SPARK-21316) Dataset Union output is not consistent with the column sequence

2017-07-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21316: -- Priority: Major (was: Critical) > Dataset Union output is not consistent with the column

[jira] [Commented] (SPARK-21316) Dataset Union output is not consistent with the column sequence

2017-07-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075129#comment-16075129 ] Dongjoon Hyun commented on SPARK-21316: --- Union assumes the schema ordering are the same for both

[jira] [Updated] (SPARK-21258) Window result incorrect using complex object with spilling

2017-07-05 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-21258: - Fix Version/s: (was: 2.1.2) > Window result incorrect using complex object with spilling >

[jira] [Commented] (SPARK-21258) Window result incorrect using complex object with spilling

2017-07-05 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075055#comment-16075055 ] Yin Huai commented on SPARK-21258: -- Since this change is not in branch-2.1, I am removing 2.1.2 from the

[jira] [Commented] (SPARK-21321) Spark very verbose on shutdown confusing users

2017-07-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075008#comment-16075008 ] Sean Owen commented on SPARK-21321: --- There is no detail here about what you're suggesting to change. >

[jira] [Created] (SPARK-21321) Spark very verbose on shutdown confusing users

2017-07-05 Thread Jong Yoon Lee (JIRA)
Jong Yoon Lee created SPARK-21321: - Summary: Spark very verbose on shutdown confusing users Key: SPARK-21321 URL: https://issues.apache.org/jira/browse/SPARK-21321 Project: Spark Issue Type:

[jira] [Commented] (SPARK-21280) org.apache.spark.util.sketch.BloomFilter not bean compliant

2017-07-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074928#comment-16074928 ] Sean Owen commented on SPARK-21280: --- Not sure what you mean here. Serialization can mean serializing an

[jira] [Created] (SPARK-21320) Make sure all expressions support interpreted evaluation

2017-07-05 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-21320: --- Summary: Make sure all expressions support interpreted evaluation Key: SPARK-21320 URL: https://issues.apache.org/jira/browse/SPARK-21320 Project: Spark Issue

[jira] [Commented] (SPARK-21320) Make sure all expressions support interpreted evaluation

2017-07-05 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074924#comment-16074924 ] Wenchen Fan commented on SPARK-21320: - cc [~rednaxelafx] > Make sure all expressions support

[jira] [Commented] (SPARK-21280) org.apache.spark.util.sketch.BloomFilter not bean compliant

2017-07-05 Thread Eran Moscovici (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074912#comment-16074912 ] Eran Moscovici commented on SPARK-21280: Unfortunately the Encoders.javaSerialization serializes

[jira] [Assigned] (SPARK-21318) The exception message thrown by `lookupFunction` is ambiguous.

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21318: Assignee: Apache Spark > The exception message thrown by `lookupFunction` is ambiguous. >

[jira] [Commented] (SPARK-21318) The exception message thrown by `lookupFunction` is ambiguous.

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074888#comment-16074888 ] Apache Spark commented on SPARK-21318: -- User 'stanzhai' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21318) The exception message thrown by `lookupFunction` is ambiguous.

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21318: Assignee: (was: Apache Spark) > The exception message thrown by `lookupFunction` is

[jira] [Commented] (SPARK-11069) Add RegexTokenizer option to convert to lowercase

2017-07-05 Thread Levente Torok (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074864#comment-16074864 ] Levente Torok commented on SPARK-11069: --- PySpark interface doesn't have this function implemented,

[jira] [Comment Edited] (SPARK-11069) Add RegexTokenizer option to convert to lowercase

2017-07-05 Thread Levente Torok (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074864#comment-16074864 ] Levente Torok edited comment on SPARK-11069 at 7/5/17 2:39 PM: --- PySpark

[jira] [Assigned] (SPARK-21319) UnsafeExternalRowSorter.RowComparator memory leak

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21319: Assignee: Apache Spark > UnsafeExternalRowSorter.RowComparator memory leak >

[jira] [Assigned] (SPARK-21319) UnsafeExternalRowSorter.RowComparator memory leak

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21319: Assignee: (was: Apache Spark) > UnsafeExternalRowSorter.RowComparator memory leak >

[jira] [Commented] (SPARK-21319) UnsafeExternalRowSorter.RowComparator memory leak

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074860#comment-16074860 ] Apache Spark commented on SPARK-21319: -- User 'j-baker' has created a pull request for this issue:

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-05 Thread Leif Walsh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074857#comment-16074857 ] Leif Walsh commented on SPARK-21190: If the user specifies an int return type but produces floats in

[jira] [Updated] (SPARK-21319) UnsafeExternalRowSorter.RowComparator memory leak

2017-07-05 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kruszewski updated SPARK-21319: -- Labels: (was: patch) > UnsafeExternalRowSorter.RowComparator memory leak >

[jira] [Updated] (SPARK-21319) UnsafeExternalRowSorter.RowComparator memory leak

2017-07-05 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kruszewski updated SPARK-21319: -- Flags: (was: Patch) > UnsafeExternalRowSorter.RowComparator memory leak >

[jira] [Updated] (SPARK-21319) UnsafeExternalRowSorter.RowComparator memory leak

2017-07-05 Thread James Baker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Baker updated SPARK-21319: Flags: Patch Labels: patch (was: ) > UnsafeExternalRowSorter.RowComparator memory leak >

[jira] [Updated] (SPARK-21319) UnsafeExternalRowSorter.RowComparator memory leak

2017-07-05 Thread James Baker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Baker updated SPARK-21319: Attachment: 0001-SPARK-21319-Fix-memory-leak-in-UnsafeExternalRowSort.patch This is my proposed

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-05 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074848#comment-16074848 ] Li Jin commented on SPARK-21190: > I have 2 thoughts: > 1. How should we handle null values? IIRC,

[jira] [Updated] (SPARK-21319) UnsafeExternalRowSorter.RowComparator memory leak

2017-07-05 Thread James Baker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Baker updated SPARK-21319: Description: When we wish to sort within partitions, we produce an UnsafeExternalRowSorter. This

[jira] [Updated] (SPARK-21319) UnsafeExternalRowSorter.RowComparator memory leak

2017-07-05 Thread James Baker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Baker updated SPARK-21319: Description: When we wish to sort within partitions, we produce an UnsafeExternalRowSorter. This

[jira] [Created] (SPARK-21319) UnsafeExternalRowSorter.RowComparator memory leak

2017-07-05 Thread James Baker (JIRA)
James Baker created SPARK-21319: --- Summary: UnsafeExternalRowSorter.RowComparator memory leak Key: SPARK-21319 URL: https://issues.apache.org/jira/browse/SPARK-21319 Project: Spark Issue Type:

[jira] [Updated] (SPARK-21319) UnsafeExternalRowSorter.RowComparator memory leak

2017-07-05 Thread James Baker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Baker updated SPARK-21319: Attachment: hprof.png > UnsafeExternalRowSorter.RowComparator memory leak >

[jira] [Commented] (SPARK-21318) The exception message thrown by `lookupFunction` is ambiguous.

2017-07-05 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074840#comment-16074840 ] StanZhai commented on SPARK-21318: -- yes. It has been registered into the `functionRegistry`, but it's

[jira] [Updated] (SPARK-21318) The exception message thrown by `lookupFunction` is ambiguous.

2017-07-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21318: -- Issue Type: Improvement (was: Bug) (not a bug) Internally, can it distinguish between something that

[jira] [Assigned] (SPARK-21285) VectorAssembler should report the column name when data type used is not supported

2017-07-05 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang reassigned SPARK-21285: --- Assignee: Yan Facai (颜发才) > VectorAssembler should report the column name when data type

[jira] [Updated] (SPARK-21318) The exception message thrown by `lookupFunction` is ambiguous.

2017-07-05 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StanZhai updated SPARK-21318: - Description: The function actually exists in current selected database, but the exception message is:

[jira] [Updated] (SPARK-21318) The exception message thrown by `lookupFunction` is ambiguous.

2017-07-05 Thread StanZhai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StanZhai updated SPARK-21318: - Description: The function actually exists, but the exception message is: {code} This function is

[jira] [Assigned] (SPARK-21317) Avoid unnecessary sort in FileFormatWriter if data is already bucketed

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21317: Assignee: Apache Spark > Avoid unnecessary sort in FileFormatWriter if data is already

[jira] [Commented] (SPARK-21317) Avoid unnecessary sort in FileFormatWriter if data is already bucketed

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074813#comment-16074813 ] Apache Spark commented on SPARK-21317: -- User 'pwoody' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21317) Avoid unnecessary sort in FileFormatWriter if data is already bucketed

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21317: Assignee: (was: Apache Spark) > Avoid unnecessary sort in FileFormatWriter if data is

[jira] [Created] (SPARK-21318) The exception message thrown by `lookupFunction` is ambiguous.

2017-07-05 Thread StanZhai (JIRA)
StanZhai created SPARK-21318: Summary: The exception message thrown by `lookupFunction` is ambiguous. Key: SPARK-21318 URL: https://issues.apache.org/jira/browse/SPARK-21318 Project: Spark

[jira] [Created] (SPARK-21317) Avoid unnecessary sort in FileFormatWriter if data is already bucketed

2017-07-05 Thread Patrick Woody (JIRA)
Patrick Woody created SPARK-21317: - Summary: Avoid unnecessary sort in FileFormatWriter if data is already bucketed Key: SPARK-21317 URL: https://issues.apache.org/jira/browse/SPARK-21317 Project:

[jira] [Resolved] (SPARK-20858) Document ListenerBus event queue size property

2017-07-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20858. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18476

[jira] [Created] (SPARK-21316) Dataset Union output is not consistent with the column sequence

2017-07-05 Thread Kaushal Prajapati (JIRA)
Kaushal Prajapati created SPARK-21316: - Summary: Dataset Union output is not consistent with the column sequence Key: SPARK-21316 URL: https://issues.apache.org/jira/browse/SPARK-21316 Project:

[jira] [Assigned] (SPARK-21286) [spark core UT]Modify a error for unit test

2017-07-05 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21286: --- Assignee: he.qiao > [spark core UT]Modify a error for unit test >

[jira] [Resolved] (SPARK-21286) [spark core UT]Modify a error for unit test

2017-07-05 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21286. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18511

[jira] [Assigned] (SPARK-20383) SparkSQL unsupports to create function with key word 'if not exists'

2017-07-05 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20383: --- Assignee: Xiaochen Ouyang > SparkSQL unsupports to create function with key word 'if not

[jira] [Resolved] (SPARK-20383) SparkSQL unsupports to create function with key word 'if not exists'

2017-07-05 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20383. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 17681

[jira] [Resolved] (SPARK-16167) RowEncoder should preserve array/map type nullability.

2017-07-05 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-16167. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 13873

[jira] [Assigned] (SPARK-16167) RowEncoder should preserve array/map type nullability.

2017-07-05 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-16167: --- Assignee: Takuya Ueshin > RowEncoder should preserve array/map type nullability. >

[jira] [Comment Edited] (SPARK-21303) Web-UI shows some Jobs get stuck randomly and stays like that. Neither able to kill

2017-07-05 Thread Arun Achuthan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074296#comment-16074296 ] Arun Achuthan edited comment on SPARK-21303 at 7/5/17 11:44 AM: Thank

[jira] [Commented] (SPARK-21299) except is throwing the fallowing exception after perform dropDuplicates on the Dataset object

2017-07-05 Thread jalendhar Baddam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074598#comment-16074598 ] jalendhar Baddam commented on SPARK-21299: -- [~hyukjin.kwon] Thanks issue is resolved > except

[jira] [Resolved] (SPARK-21310) Add offset to PySpark GLM

2017-07-05 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang resolved SPARK-21310. - Resolution: Fixed Assignee: Wayne Zhang Fix Version/s: 2.3.0 > Add offset to

[jira] [Updated] (SPARK-21315) Skip some spill files when generateIterator(startIndex) in ExternalAppendOnlyUnsafeRowArray.

2017-07-05 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-21315: - Description: In current code, it is expensive to use {{UnboundedFollowingWindowFunctionFrame}}, because

[jira] [Assigned] (SPARK-21315) Skip some spill files when generateIterator(startIndex) in ExternalAppendOnlyUnsafeRowArray.

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21315: Assignee: (was: Apache Spark) > Skip some spill files when

[jira] [Assigned] (SPARK-21315) Skip some spill files when generateIterator(startIndex) in ExternalAppendOnlyUnsafeRowArray.

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21315: Assignee: Apache Spark > Skip some spill files when generateIterator(startIndex) in >

[jira] [Commented] (SPARK-21315) Skip some spill files when generateIterator(startIndex) in ExternalAppendOnlyUnsafeRowArray.

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074549#comment-16074549 ] Apache Spark commented on SPARK-21315: -- User 'jinxing64' has created a pull request for this issue:

[jira] [Created] (SPARK-21315) Skip some spill files when generateIterator(startIndex) in ExternalAppendOnlyUnsafeRowArray.

2017-07-05 Thread jin xing (JIRA)
jin xing created SPARK-21315: Summary: Skip some spill files when generateIterator(startIndex) in ExternalAppendOnlyUnsafeRowArray. Key: SPARK-21315 URL: https://issues.apache.org/jira/browse/SPARK-21315

[jira] [Updated] (SPARK-21314) ByteArrayMethods.arrayEquals could use some optimizations

2017-07-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21314: -- Issue Type: Improvement (was: Bug) (not a bug) I would be surprised if it made much difference,

[jira] [Created] (SPARK-21314) ByteArrayMethods.arrayEquals could use some optimizations

2017-07-05 Thread Sumedh Wale (JIRA)
Sumedh Wale created SPARK-21314: --- Summary: ByteArrayMethods.arrayEquals could use some optimizations Key: SPARK-21314 URL: https://issues.apache.org/jira/browse/SPARK-21314 Project: Spark

[jira] [Commented] (SPARK-19451) rangeBetween method should accept Long value as boundary

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074468#comment-16074468 ] Apache Spark commented on SPARK-19451: -- User 'jiangxb1987' has created a pull request for this

[jira] [Updated] (SPARK-19451) rangeBetween method should accept Long value as boundary

2017-07-05 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xingbo updated SPARK-19451: - Summary: rangeBetween method should accept Long value as boundary (was: Long values in Window

[jira] [Commented] (SPARK-21306) OneVsRest Conceals Columns That May Be Relevant To Underlying Classifier

2017-07-05 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074404#comment-16074404 ] Nick Pentreath commented on SPARK-21306: This is definitely an issue. I don't think it is an

[jira] [Commented] (SPARK-21313) ConsoleSink's string representation

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074402#comment-16074402 ] Apache Spark commented on SPARK-21313: -- User 'jaceklaskowski' has created a pull request for this

[jira] [Assigned] (SPARK-21313) ConsoleSink's string representation

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21313: Assignee: Apache Spark > ConsoleSink's string representation >

[jira] [Assigned] (SPARK-21313) ConsoleSink's string representation

2017-07-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21313: Assignee: (was: Apache Spark) > ConsoleSink's string representation >

[jira] [Created] (SPARK-21313) ConsoleSink's string representation

2017-07-05 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-21313: --- Summary: ConsoleSink's string representation Key: SPARK-21313 URL: https://issues.apache.org/jira/browse/SPARK-21313 Project: Spark Issue Type:

  1   2   >