[jira] [Commented] (SPARK-10877) Assertions fail straightforward DataFrame job due to word alignment

2015-10-17 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14962093#comment-14962093 ] Davies Liu commented on SPARK-10877: Yes > Assertions fail straightforward DataFrame

[jira] [Resolved] (SPARK-10185) Spark SQL does not handle comma separates paths on Hadoop FileSystem

2015-10-17 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10185. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8416 [https://github.c

[jira] [Resolved] (SPARK-10877) Assertions fail straightforward DataFrame job due to word alignment

2015-10-16 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10877. Resolution: Fixed Target Version/s: 1.5.2, 1.6.0 > Assertions fail straightforward DataFr

[jira] [Commented] (SPARK-10877) Assertions fail straightforward DataFrame job due to word alignment

2015-10-16 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961505#comment-14961505 ] Davies Liu commented on SPARK-10877: This is already fixed in master and 1.5 branch.

[jira] [Created] (SPARK-11151) Use Long internally for DecimalType with precision <= 18

2015-10-16 Thread Davies Liu (JIRA)
Davies Liu created SPARK-11151: -- Summary: Use Long internally for DecimalType with precision <= 18 Key: SPARK-11151 URL: https://issues.apache.org/jira/browse/SPARK-11151 Project: Spark Issue Ty

[jira] [Created] (SPARK-11149) Improve performance of primitive types in columnar cache

2015-10-16 Thread Davies Liu (JIRA)
Davies Liu created SPARK-11149: -- Summary: Improve performance of primitive types in columnar cache Key: SPARK-11149 URL: https://issues.apache.org/jira/browse/SPARK-11149 Project: Spark Issue Ty

[jira] [Resolved] (SPARK-11076) Decimal Support for Ceil/Floor

2015-10-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11076. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9086 [https://github.c

[jira] [Resolved] (SPARK-10104) Consolidate different forms of table identifiers

2015-10-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10104. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8453 [https://github.c

[jira] [Created] (SPARK-11114) Add getOrCreate for SparkContext/SQLContext for Python

2015-10-14 Thread Davies Liu (JIRA)
Davies Liu created SPARK-4: -- Summary: Add getOrCreate for SparkContext/SQLContext for Python Key: SPARK-4 URL: https://issues.apache.org/jira/browse/SPARK-4 Project: Spark Issue Type

[jira] [Created] (SPARK-11111) Fast null-safe join

2015-10-14 Thread Davies Liu (JIRA)
Davies Liu created SPARK-1: -- Summary: Fast null-safe join Key: SPARK-1 URL: https://issues.apache.org/jira/browse/SPARK-1 Project: Spark Issue Type: Improvement Reporter:

[jira] [Commented] (SPARK-11083) insert overwrite table failed when beeline reconnect

2015-10-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957281#comment-14957281 ] Davies Liu commented on SPARK-11083: Maybe this one: https://github.com/apache/spark/

[jira] [Resolved] (SPARK-11083) insert overwrite table failed when beeline reconnect

2015-10-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11083. Resolution: Fixed Assignee: Davies Liu Fix Version/s: 1.6.0 > insert overwrite tabl

[jira] [Resolved] (SPARK-11068) Add callback to query execution

2015-10-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11068. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9078 [https://github.c

[jira] [Comment Edited] (SPARK-10617) Leap year miscalculated in sql query

2015-10-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955691#comment-14955691 ] Davies Liu edited comment on SPARK-10617 at 10/13/15 9:15 PM: -

[jira] [Resolved] (SPARK-10617) Leap year miscalculated in sql query

2015-10-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10617. Resolution: Not A Problem Assignee: Davies Liu > Leap year miscalculated in sql query > -

[jira] [Commented] (SPARK-10617) Leap year miscalculated in sql query

2015-10-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955691#comment-14955691 ] Davies Liu commented on SPARK-10617: If you really want to get 2016-02-28 back, you c

[jira] [Commented] (SPARK-9182) filter and groupBy on DataFrames are not passed through to jdbc source

2015-10-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955646#comment-14955646 ] Davies Liu commented on SPARK-9182: --- For JDBC, I think we could push more stuff (for exa

[jira] [Commented] (SPARK-10617) Leap year miscalculated in sql query

2015-10-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955486#comment-14955486 ] Davies Liu commented on SPARK-10617: [~shao.lo] I still do not get it, if the add_mon

[jira] [Commented] (SPARK-5391) SparkSQL fails to create tables with custom JSON SerDe

2015-10-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955427#comment-14955427 ] Davies Liu commented on SPARK-5391: --- I tried this, it worked in master, will close this

[jira] [Resolved] (SPARK-5391) SparkSQL fails to create tables with custom JSON SerDe

2015-10-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-5391. --- Resolution: Fixed Fix Version/s: 1.6.0 > SparkSQL fails to create tables with custom JSON SerDe

[jira] [Commented] (SPARK-11067) Spark SQL thrift server fails to handle decimal value

2015-10-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955416#comment-14955416 ] Davies Liu commented on SPARK-11067: [~navis] Could you send out a PR for this patch?

[jira] [Commented] (SPARK-11083) insert overwrite table failed when beeline reconnect

2015-10-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955409#comment-14955409 ] Davies Liu commented on SPARK-11083: What's the version you are using? Does this stil

[jira] [Created] (SPARK-11089) Add a option for thrift-server to share a single session across all connections

2015-10-13 Thread Davies Liu (JIRA)
Davies Liu created SPARK-11089: -- Summary: Add a option for thrift-server to share a single session across all connections Key: SPARK-11089 URL: https://issues.apache.org/jira/browse/SPARK-11089 Project:

[jira] [Resolved] (SPARK-11036) AttributeReference should not be created outside driver

2015-10-13 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11036. Resolution: Duplicate > AttributeReference should not be created outside driver > -

[jira] [Resolved] (SPARK-10617) Leap year miscalculated in sql query

2015-10-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10617. Resolution: Not A Problem This is expected behavior, see https://docs.oracle.com/cd/B28359_01/serv

[jira] [Resolved] (SPARK-11042) Introduce a mechanism to ban creating new root SQLContexts in a JVM

2015-10-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11042. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9058 [https://github.c

[jira] [Resolved] (SPARK-8170) Ctrl-C in pyspark shell doesn't kill running job

2015-10-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8170. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9033 [https://github.com/

[jira] [Resolved] (SPARK-11007) Add dictionary support for CatalystDecimalConverter

2015-10-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11007. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9040 [https://github.c

[jira] [Updated] (SPARK-11038) Consolidate the format of UnsafeArrayData and UnsafeMapData

2015-10-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-11038: --- Component/s: (was: Spark Core) SQL > Consolidate the format of UnsafeArrayData a

[jira] [Resolved] (SPARK-9280) New HiveContext object unexpectedly loads configuration settings from history

2015-10-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9280. --- Resolution: Fixed Assignee: Davies Liu Fix Version/s: 1.6.0 > New HiveContext object u

[jira] [Created] (SPARK-11038) Consolidate the format of UnsafeArrayData and UnsafeMapData

2015-10-09 Thread Davies Liu (JIRA)
Davies Liu created SPARK-11038: -- Summary: Consolidate the format of UnsafeArrayData and UnsafeMapData Key: SPARK-11038 URL: https://issues.apache.org/jira/browse/SPARK-11038 Project: Spark Issu

[jira] [Resolved] (SPARK-10535) Support for recommendUsersForProducts and recommendProductsForUsers in matrix factorization model for PySpark

2015-10-09 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10535. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8700 [https://github.c

[jira] [Assigned] (SPARK-11009) RowNumber in HiveContext returns negative values in cluster mode

2015-10-09 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-11009: -- Assignee: Davies Liu > RowNumber in HiveContext returns negative values in cluster mode >

[jira] [Created] (SPARK-11036) AttributeReference should not be created outside driver

2015-10-09 Thread Davies Liu (JIRA)
Davies Liu created SPARK-11036: -- Summary: AttributeReference should not be created outside driver Key: SPARK-11036 URL: https://issues.apache.org/jira/browse/SPARK-11036 Project: Spark Issue Typ

[jira] [Created] (SPARK-11030) SQLTab should be shared by across sessions

2015-10-09 Thread Davies Liu (JIRA)
Davies Liu created SPARK-11030: -- Summary: SQLTab should be shared by across sessions Key: SPARK-11030 URL: https://issues.apache.org/jira/browse/SPARK-11030 Project: Spark Issue Type: Improvemen

[jira] [Commented] (SPARK-9280) New HiveContext object unexpectedly loads configuration settings from history

2015-10-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949706#comment-14949706 ] Davies Liu commented on SPARK-9280: --- [~tien-dung.le] I check this with latest master (wi

[jira] [Commented] (SPARK-6540) Spark SQL thrift server fails to pass settings to following query in the same session

2015-10-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949700#comment-14949700 ] Davies Liu commented on SPARK-6540: --- [~alexliu68] Could you check that this is fixed in

[jira] [Resolved] (SPARK-8552) Using incorrect database in multiple sessions

2015-10-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8552. --- Resolution: Fixed Assignee: Davies Liu Fix Version/s: 1.6.0 Resolved by https://github

[jira] [Resolved] (SPARK-10810) Improve session management for SQL

2015-10-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10810. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8909 [https://github.c

[jira] [Resolved] (SPARK-10902) Hive UDF current_database() does not work

2015-10-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10902. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8909 [https://github.c

[jira] [Created] (SPARK-11018) Support UDT in codegen and unsafe projection

2015-10-08 Thread Davies Liu (JIRA)
Davies Liu created SPARK-11018: -- Summary: Support UDT in codegen and unsafe projection Key: SPARK-11018 URL: https://issues.apache.org/jira/browse/SPARK-11018 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-11006) Rename NullColumnAccess as NullColumnAccessor

2015-10-08 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11006. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9028 [https://github.c

[jira] [Created] (SPARK-10990) Avoid the serialization multiple times during unrolling of complex types

2015-10-07 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10990: -- Summary: Avoid the serialization multiple times during unrolling of complex types Key: SPARK-10990 URL: https://issues.apache.org/jira/browse/SPARK-10990 Project: Spark

[jira] [Resolved] (SPARK-10917) Improve performance of complex types in columnar cache

2015-10-07 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10917. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8971 [https://github.c

[jira] [Resolved] (SPARK-10980) Create wrong decimal if unscaled > 1e18 and scale > 0

2015-10-07 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10980. Resolution: Fixed Fix Version/s: 1.2.3 1.5.2 1.3.2

[jira] [Updated] (SPARK-10980) Create wrong decimal if unscaled > 1e18 and scale > 0

2015-10-07 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10980: --- Description: Decimal(100L, 20, 2) will become 100 instead of 1000

[jira] [Updated] (SPARK-10980) Create wrong decimal if unscaled > 1e18 and scale > 0

2015-10-07 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10980: --- Summary: Create wrong decimal if unscaled > 1e18 and scale > 0 (was: Create wrong decimal with unsca

[jira] [Updated] (SPARK-10980) Create wrong decimal with unscaled value and precision > 18

2015-10-07 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10980: --- Affects Version/s: 1.2.2 1.3.1 1.4.1 > Create wrong dec

[jira] [Created] (SPARK-10980) Create wrong decimal with unscaled value and precision > 18

2015-10-07 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10980: -- Summary: Create wrong decimal with unscaled value and precision > 18 Key: SPARK-10980 URL: https://issues.apache.org/jira/browse/SPARK-10980 Project: Spark Issu

[jira] [Resolved] (SPARK-10752) Implement corr() and cov in DataFrameStatFunctions

2015-10-07 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10752. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8869 [https://github.c

[jira] [Updated] (SPARK-10810) Improve session management for SQL

2015-10-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10810: --- Attachment: Session management in Spark SQL 1.6.pdf Design doc > Improve session management for SQL

[jira] [Updated] (SPARK-10585) only copy data once when generate unsafe projection

2015-10-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10585: --- Assignee: Wenchen Fan > only copy data once when generate unsafe projection > ---

[jira] [Closed] (SPARK-9158) PyLint should only fail on error

2015-10-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-9158. - Resolution: Won't Fix > PyLint should only fail on error > > >

[jira] [Commented] (SPARK-9158) PyLint should only fail on error

2015-10-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945359#comment-14945359 ] Davies Liu commented on SPARK-9158: --- Just realized that PyLint is not that useful, right

[jira] [Resolved] (SPARK-10938) Remove typeId in columnar cache

2015-10-06 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10938. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8989 [https://github.c

[jira] [Updated] (SPARK-10934) hashCode of unsafe array may crush

2015-10-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10934: --- Assignee: Wenchen Fan > hashCode of unsafe array may crush > -- > >

[jira] [Updated] (SPARK-10934) hashCode of unsafe array may crush

2015-10-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10934: --- Fix Version/s: 1.5.2 > hashCode of unsafe array may crush > -- > >

[jira] [Resolved] (SPARK-10934) hashCode of unsafe array may crush

2015-10-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10934. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8987 [https://github.c

[jira] [Created] (SPARK-10938) Remove typeId in columnar cache

2015-10-05 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10938: -- Summary: Remove typeId in columnar cache Key: SPARK-10938 URL: https://issues.apache.org/jira/browse/SPARK-10938 Project: Spark Issue Type: Task Repo

[jira] [Commented] (SPARK-10685) Misaligned data with RDD.zip and DataFrame.withColumn after repartition

2015-10-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944112#comment-14944112 ] Davies Liu commented on SPARK-10685: [~jdanbrown] The zip after repartition (or shuff

[jira] [Resolved] (SPARK-10585) only copy data once when generate unsafe projection

2015-10-05 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10585. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8747 [https://github.c

[jira] [Created] (SPARK-10917) Improve performance of complex types in columnar cache

2015-10-02 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10917: -- Summary: Improve performance of complex types in columnar cache Key: SPARK-10917 URL: https://issues.apache.org/jira/browse/SPARK-10917 Project: Spark Issue Type

[jira] [Commented] (SPARK-10342) Cooperative memory management

2015-10-01 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940524#comment-14940524 ] Davies Liu commented on SPARK-10342: This will be used internal for SQL. For example,

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-01 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940508#comment-14940508 ] Davies Liu commented on SPARK-10903: LGTM. Another question is that can we have diffe

[jira] [Resolved] (SPARK-10866) [Spark SQL] [UDF] the floor function got wrong return value type

2015-10-01 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10866. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8933 [https://github.c

[jira] [Resolved] (SPARK-10865) [Spark SQL] [UDF] the ceil/ceiling function got wrong return value type

2015-10-01 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10865. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8933 [https://github.c

[jira] [Created] (SPARK-10902) Hive UDF current_database() does not work

2015-10-01 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10902: -- Summary: Hive UDF current_database() does not work Key: SPARK-10902 URL: https://issues.apache.org/jira/browse/SPARK-10902 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-9741) approx count distinct function

2015-09-30 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9741. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8362 [https://github.com/

[jira] [Resolved] (SPARK-6919) Add .asDict method to StatCounter

2015-09-29 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-6919. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 5516 [https://github.com/

[jira] [Resolved] (SPARK-10415) Enhance Navigation Sidebar in PySpark API

2015-09-29 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10415. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8571 [https://github.c

[jira] [Resolved] (SPARK-10395) Simplify CatalystReadSupport

2015-09-28 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10395. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8553 [https://github.c

[jira] [Created] (SPARK-10859) Predicates pushed to InmemoryColumnarTableScan are not evaluated correctly

2015-09-28 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10859: -- Summary: Predicates pushed to InmemoryColumnarTableScan are not evaluated correctly Key: SPARK-10859 URL: https://issues.apache.org/jira/browse/SPARK-10859 Project: Spark

[jira] [Created] (SPARK-10810) Improve session management for SQL

2015-09-24 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10810: -- Summary: Improve session management for SQL Key: SPARK-10810 URL: https://issues.apache.org/jira/browse/SPARK-10810 Project: Spark Issue Type: Bug Comp

[jira] [Resolved] (SPARK-9821) pyspark reduceByKey should allow a custom partitioner

2015-09-21 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9821. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8569 [https://github.com/

[jira] [Closed] (SPARK-10559) DataFrame schema ArrayType should accept ResultIterable

2015-09-18 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10559. -- Resolution: Won't Fix > DataFrame schema ArrayType should accept ResultIterable > -

[jira] [Commented] (SPARK-10559) DataFrame schema ArrayType should accept ResultIterable

2015-09-18 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876377#comment-14876377 ] Davies Liu commented on SPARK-10559: It's easy to turn the ResultIterable into a list

[jira] [Assigned] (SPARK-10538) java.lang.NegativeArraySizeException during join

2015-09-18 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-10538: -- Assignee: Davies Liu > java.lang.NegativeArraySizeException during join >

[jira] [Commented] (SPARK-10538) java.lang.NegativeArraySizeException during join

2015-09-18 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876338#comment-14876338 ] Davies Liu commented on SPARK-10538: Do you know which operator (join or other) the e

[jira] [Commented] (SPARK-10635) pyspark - running on a different host

2015-09-18 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875966#comment-14875966 ] Davies Liu commented on SPARK-10635: We have the assumption that python and JVM are i

[jira] [Updated] (SPARK-10685) Misaligned data with RDD.zip and DataFrame.withColumn after repartition

2015-09-17 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10685: --- Target Version/s: 1.5.1 Priority: Blocker (was: Major) Component/s: SQL

[jira] [Commented] (SPARK-10685) Misaligned data with RDD.zip and DataFrame.withColumn after repartition

2015-09-17 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804924#comment-14804924 ] Davies Liu commented on SPARK-10685: Internally, Python UDF use RDD.zip(), and comput

[jira] [Resolved] (SPARK-10642) Crash in rdd.lookup() with "java.lang.Long cannot be cast to java.lang.Integer"

2015-09-17 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10642. Resolution: Fixed Fix Version/s: 1.2.3 1.3.2 1.4.2

[jira] [Resolved] (SPARK-10459) PythonUDF could process UnsafeRow

2015-09-17 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10459. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8616 [https://github.c

[jira] [Resolved] (SPARK-10542) The PySpark 1.5 closure serializer can't serialize a namedtuple instance.

2015-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10542. Resolution: Fixed Fix Version/s: 1.5.1 1.6.0 Target Version/s

[jira] [Resolved] (SPARK-10522) Nanoseconds part of Timestamp should be positive in parquet

2015-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10522. Resolution: Fixed Fix Version/s: 1.5.1 1.6.0 Issue resolved by pull reque

[jira] [Commented] (SPARK-9325) Support `collect` on DataFrame columns

2015-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744254#comment-14744254 ] Davies Liu commented on SPARK-9325: --- I would -1 on this. I'm worried that once we have

[jira] [Assigned] (SPARK-10593) sql lateral view same name gives wrong value

2015-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-10593: -- Assignee: Davies Liu > sql lateral view same name gives wrong value >

[jira] [Updated] (SPARK-10593) sql lateral view same name gives wrong value

2015-09-14 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10593: --- Description: This query will return wrong result: {code} select insideLayer1.json as json_insideLaye

[jira] [Created] (SPARK-10593) sql lateral view same name gives wrong value

2015-09-14 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10593: -- Summary: sql lateral view same name gives wrong value Key: SPARK-10593 URL: https://issues.apache.org/jira/browse/SPARK-10593 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-6548) stddev_pop and stddev_samp aggregate functions

2015-09-12 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-6548. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 6297 [https://github.com/

[jira] [Resolved] (SPARK-9014) Allow Python spark API to use built-in exponential operator

2015-09-11 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9014. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8658 [https://github.com/

[jira] [Created] (SPARK-10572) Investigate the contentions bewteen tasks in the same executor

2015-09-11 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10572: -- Summary: Investigate the contentions bewteen tasks in the same executor Key: SPARK-10572 URL: https://issues.apache.org/jira/browse/SPARK-10572 Project: Spark I

[jira] [Closed] (SPARK-10553) Allow Ctrl-C in pyspark shell to kill running job

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10553. -- Resolution: Duplicate > Allow Ctrl-C in pyspark shell to kill running job > ---

[jira] [Created] (SPARK-10553) Allow Ctrl-C in pyspark shell to kill running job

2015-09-10 Thread Davies Liu (JIRA)
Davies Liu created SPARK-10553: -- Summary: Allow Ctrl-C in pyspark shell to kill running job Key: SPARK-10553 URL: https://issues.apache.org/jira/browse/SPARK-10553 Project: Spark Issue Type: New

[jira] [Closed] (SPARK-10397) Make Python's SparkContext self-descriptive on "print sc"

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10397. -- Resolution: Won't Fix > Make Python's SparkContext self-descriptive on "print sc" > ---

[jira] [Commented] (SPARK-10397) Make Python's SparkContext self-descriptive on "print sc"

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739603#comment-14739603 ] Davies Liu commented on SPARK-10397: I'd like to stick with current approach, that's

[jira] [Resolved] (SPARK-10056) PySpark Row - Support for row["columnName"] syntax

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10056. Resolution: Fixed Assignee: Yanbo Liang Fix Version/s: 1.6.0 > PySpark Row - Suppor

[jira] [Resolved] (SPARK-7544) pyspark.sql.types.Row should implement __getitem__

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-7544. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8333 [https://github.com/

[jira] [Resolved] (SPARK-10443) Refactor SortMergeOuterJoin to reduce duplication

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10443. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8596 [https://github.c

[jira] [Closed] (SPARK-10544) Serialization of Python namedtuple subclasses in functions / closures is broken

2015-09-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10544. -- Resolution: Duplicate Fix Version/s: (was: 1.5.1) Target Version/s: 1.5.1 > Serial

<    8   9   10   11   12   13   14   15   16   17   >