[jira] [Assigned] (SPARK-21137) Spark reads many small files slowly

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21137: Assignee: (was: Apache Spark) > Spark reads many small files slowly >

[jira] [Commented] (SPARK-21137) Spark reads many small files slowly

2017-06-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065515#comment-16065515 ] Steve Loughran commented on SPARK-21137: bq. so it is something that could be opt

[jira] [Created] (SPARK-21231) Conda install of packages during Jenkins testing is causing intermittent failure

2017-06-27 Thread holdenk (JIRA)
holdenk created SPARK-21231: --- Summary: Conda install of packages during Jenkins testing is causing intermittent failure Key: SPARK-21231 URL: https://issues.apache.org/jira/browse/SPARK-21231 Project: Spark

[jira] [Assigned] (SPARK-21231) Conda install of packages during Jenkins testing is causing intermittent failure

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21231: Assignee: (was: Apache Spark) > Conda install of packages during Jenkins testing is ca

[jira] [Commented] (SPARK-21231) Conda install of packages during Jenkins testing is causing intermittent failure

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065621#comment-16065621 ] Apache Spark commented on SPARK-21231: -- User 'holdenk' has created a pull request fo

[jira] [Assigned] (SPARK-21231) Conda install of packages during Jenkins testing is causing intermittent failure

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21231: Assignee: Apache Spark > Conda install of packages during Jenkins testing is causing inter

[jira] [Commented] (SPARK-21152) Use level 3 BLAS operations in LogisticAggregator

2017-06-27 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065694#comment-16065694 ] yuhao yang commented on SPARK-21152: This is something that we should investigate any

[jira] [Commented] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe

2017-06-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065768#comment-16065768 ] Hyukjin Kwon commented on SPARK-21227: -- I tested both as below on Python 3.6.0 and 2

[jira] [Issue Comment Deleted] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe

2017-06-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-21227: - Comment: was deleted (was: I tested both as below on Python 3.6.0 and 2.7.10 as below but I could

[jira] [Commented] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe

2017-06-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065769#comment-16065769 ] Hyukjin Kwon commented on SPARK-21227: -- I can reproduce this in both Python 2.7 and

[jira] [Resolved] (SPARK-21155) Add (? running tasks) into Spark UI progress

2017-06-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21155. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18369 [https://githu

[jira] [Assigned] (SPARK-21155) Add (? running tasks) into Spark UI progress

2017-06-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21155: --- Assignee: Eric Vandenberg > Add (? running tasks) into Spark UI progress > -

[jira] [Commented] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe

2017-06-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065775#comment-16065775 ] Hyukjin Kwon commented on SPARK-21227: -- In scala too: {code} val jsons = Seq( """

[jira] [Commented] (SPARK-10915) Add support for UDAFs in Python

2017-06-27 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065787#comment-16065787 ] Erik Erlandson commented on SPARK-10915: This would be great for exposing {{TDige

[jira] [Commented] (SPARK-16542) bugs about types that result an array of null when creating dataframe using python

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065789#comment-16065789 ] Apache Spark commented on SPARK-16542: -- User 'zasdfgbnm' has created a pull request

[jira] [Commented] (SPARK-10915) Add support for UDAFs in Python

2017-06-27 Thread Han Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065790#comment-16065790 ] Han Xu commented on SPARK-10915: I'm currently traveling without access to my email. To

[jira] [Updated] (SPARK-21222) Move elimination of Distinct clause from analyzer to optimizer

2017-06-27 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-21222: --- Description: Move elimination of Distinct clause from analyzer to optimizer Distinct clause

[jira] [Commented] (SPARK-21222) Move elimination of Distinct clause from analyzer to optimizer

2017-06-27 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065791#comment-16065791 ] Gengliang Wang commented on SPARK-21222: [~srowen] thanks! I have corrected the s

[jira] [Commented] (SPARK-19726) Faild to insert null timestamp value to mysql using spark jdbc

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065869#comment-16065869 ] Apache Spark commented on SPARK-19726: -- User 'shuangshuangwang' has created a pull r

[jira] [Assigned] (SPARK-19726) Faild to insert null timestamp value to mysql using spark jdbc

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19726: Assignee: (was: Apache Spark) > Faild to insert null timestamp value to mysql using sp

[jira] [Assigned] (SPARK-19726) Faild to insert null timestamp value to mysql using spark jdbc

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19726: Assignee: Apache Spark > Faild to insert null timestamp value to mysql using spark jdbc >

[jira] [Commented] (SPARK-21182) Structured streaming on Spark-shell on windows

2017-06-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065876#comment-16065876 ] Hyukjin Kwon commented on SPARK-21182: -- Looks I can't reproduce this on Windows at t

[jira] [Commented] (SPARK-21076) R dapply doesn't return array or raw columns when array have different length

2017-06-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065902#comment-16065902 ] Hyukjin Kwon commented on SPARK-21076: -- I believe this produces the similar error de

[jira] [Resolved] (SPARK-21053) Number overflow on agg function of Dataframe

2017-06-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21053. -- Resolution: Cannot Reproduce I tried to follow what's written in this JIRA but I could not repr

[jira] [Resolved] (SPARK-21019) read orc when some of the columns are missing in some files

2017-06-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21019. -- Resolution: Duplicate I believe this should be a duplicate of SPARK-11412. > read orc when so

[jira] [Resolved] (SPARK-14486) For partition table, the dag occurs oom because of too many same rdds

2017-06-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-14486. -- Resolution: Invalid I am resolving this.I also agree with the opinion above. > For partition t

[jira] [Commented] (SPARK-21182) Structured streaming on Spark-shell on windows

2017-06-27 Thread Vijay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065921#comment-16065921 ] Vijay commented on SPARK-21182: --- I'm still facing the same issue. Actually I have configure

[jira] [Commented] (SPARK-21182) Structured streaming on Spark-shell on windows

2017-06-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065929#comment-16065929 ] Hyukjin Kwon commented on SPARK-21182: -- Ah, could you maybe try to explicitly specif

[jira] [Updated] (SPARK-21232) New built-in SQL function - Data_Type

2017-06-27 Thread Mario Molina (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mario Molina updated SPARK-21232: - Description: This function returns the data type of a given column. {code:java} data_type("a") /

[jira] [Created] (SPARK-21232) New built-in SQL function - Data_Type

2017-06-27 Thread Mario Molina (JIRA)
Mario Molina created SPARK-21232: Summary: New built-in SQL function - Data_Type Key: SPARK-21232 URL: https://issues.apache.org/jira/browse/SPARK-21232 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-21233) Support pluggable offset storage

2017-06-27 Thread darion yaphet (JIRA)
darion yaphet created SPARK-21233: - Summary: Support pluggable offset storage Key: SPARK-21233 URL: https://issues.apache.org/jira/browse/SPARK-21233 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-21234) When the function returns Option[Iterator[_]] is None,then get on None will cause java.util.NoSuchElementException: None.get

2017-06-27 Thread wangjiaochun (JIRA)
wangjiaochun created SPARK-21234: Summary: When the function returns Option[Iterator[_]] is None,then get on None will cause java.util.NoSuchElementException: None.get Key: SPARK-21234 URL: https://issues.apache.o

[jira] [Created] (SPARK-21235) UTest should clear temp results when run case

2017-06-27 Thread wangjiaochun (JIRA)
wangjiaochun created SPARK-21235: Summary: UTest should clear temp results when run case Key: SPARK-21235 URL: https://issues.apache.org/jira/browse/SPARK-21235 Project: Spark Issue Type: Te

[jira] [Created] (SPARK-21236) Make the threshold of using HighlyCompressedStatus configurable.

2017-06-27 Thread jin xing (JIRA)
jin xing created SPARK-21236: Summary: Make the threshold of using HighlyCompressedStatus configurable. Key: SPARK-21236 URL: https://issues.apache.org/jira/browse/SPARK-21236 Project: Spark Iss

[jira] [Commented] (SPARK-21232) New built-in SQL function - Data_Type

2017-06-27 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065972#comment-16065972 ] Felix Cheung commented on SPARK-21232: -- I don't see this in Scala - are you proposin

[jira] [Commented] (SPARK-21236) Make the threshold of using HighlyCompressedStatus configurable.

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065973#comment-16065973 ] Apache Spark commented on SPARK-21236: -- User 'jinxing64' has created a pull request

[jira] [Assigned] (SPARK-21236) Make the threshold of using HighlyCompressedStatus configurable.

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21236: Assignee: Apache Spark > Make the threshold of using HighlyCompressedStatus configurable.

[jira] [Assigned] (SPARK-21236) Make the threshold of using HighlyCompressedStatus configurable.

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21236: Assignee: (was: Apache Spark) > Make the threshold of using HighlyCompressedStatus con

[jira] [Assigned] (SPARK-21232) New built-in SQL function - Data_Type

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21232: Assignee: Apache Spark > New built-in SQL function - Data_Type > -

[jira] [Assigned] (SPARK-21232) New built-in SQL function - Data_Type

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21232: Assignee: (was: Apache Spark) > New built-in SQL function - Data_Type > --

[jira] [Commented] (SPARK-21232) New built-in SQL function - Data_Type

2017-06-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065980#comment-16065980 ] Apache Spark commented on SPARK-21232: -- User 'mmolimar' has created a pull request f

[jira] [Commented] (SPARK-21232) New built-in SQL function - Data_Type

2017-06-27 Thread Mario Molina (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065982#comment-16065982 ] Mario Molina commented on SPARK-21232: -- I just created a PR for this. > New built-i

[jira] [Commented] (SPARK-18502) Spark does not handle columns that contain backquote (`)

2017-06-27 Thread Sudeshna Bora (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065990#comment-16065990 ] Sudeshna Bora commented on SPARK-18502: --- What is the expected time for resolution o

<    1   2