[jira] [Resolved] (SPARK-11076) Decimal Support for Ceil/Floor

[ https://issues.apache.org/jira/browse/SPARK-11076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11076. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9086

[jira] [Commented] (SPARK-11083) insert overwrite table failed when beeline reconnect

[ https://issues.apache.org/jira/browse/SPARK-11083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957281#comment-14957281 ] Davies Liu commented on SPARK-11083: Maybe this one: https://github.com/apache/spark/pull/8909, it

[jira] [Resolved] (SPARK-11083) insert overwrite table failed when beeline reconnect

[ https://issues.apache.org/jira/browse/SPARK-11083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11083. Resolution: Fixed Assignee: Davies Liu Fix Version/s: 1.6.0 > insert overwrite

[jira] [Resolved] (SPARK-10104) Consolidate different forms of table identifiers

[ https://issues.apache.org/jira/browse/SPARK-10104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10104. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8453

[jira] [Resolved] (SPARK-11036) AttributeReference should not be created outside driver

[ https://issues.apache.org/jira/browse/SPARK-11036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11036. Resolution: Duplicate > AttributeReference should not be created outside driver >

[jira] [Commented] (SPARK-10617) Leap year miscalculated in sql query

[ https://issues.apache.org/jira/browse/SPARK-10617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955486#comment-14955486 ] Davies Liu commented on SPARK-10617: [~shao.lo] I still do not get it, if the add_months is right,

[jira] [Commented] (SPARK-11083) insert overwrite table failed when beeline reconnect

[ https://issues.apache.org/jira/browse/SPARK-11083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955409#comment-14955409 ] Davies Liu commented on SPARK-11083: What's the version you are using? Does this still happen on

[jira] [Commented] (SPARK-5391) SparkSQL fails to create tables with custom JSON SerDe

[ https://issues.apache.org/jira/browse/SPARK-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955427#comment-14955427 ] Davies Liu commented on SPARK-5391: --- I tried this, it worked in master, will close this as resolved. >

[jira] [Created] (SPARK-11089) Add a option for thrift-server to share a single session across all connections

Davies Liu created SPARK-11089: -- Summary: Add a option for thrift-server to share a single session across all connections Key: SPARK-11089 URL: https://issues.apache.org/jira/browse/SPARK-11089 Project:

[jira] [Commented] (SPARK-11067) Spark SQL thrift server fails to handle decimal value

[ https://issues.apache.org/jira/browse/SPARK-11067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955416#comment-14955416 ] Davies Liu commented on SPARK-11067: [~navis] Could you send out a PR for this patch? > Spark SQL

[jira] [Resolved] (SPARK-5391) SparkSQL fails to create tables with custom JSON SerDe

[ https://issues.apache.org/jira/browse/SPARK-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-5391. --- Resolution: Fixed Fix Version/s: 1.6.0 > SparkSQL fails to create tables with custom JSON

[jira] [Comment Edited] (SPARK-10617) Leap year miscalculated in sql query

[ https://issues.apache.org/jira/browse/SPARK-10617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955691#comment-14955691 ] Davies Liu edited comment on SPARK-10617 at 10/13/15 9:15 PM: -- If you really

[jira] [Resolved] (SPARK-10617) Leap year miscalculated in sql query

[ https://issues.apache.org/jira/browse/SPARK-10617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10617. Resolution: Not A Problem Assignee: Davies Liu > Leap year miscalculated in sql query >

[jira] [Commented] (SPARK-10617) Leap year miscalculated in sql query

[ https://issues.apache.org/jira/browse/SPARK-10617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955691#comment-14955691 ] Davies Liu commented on SPARK-10617: If you really want to get 2016-02-28 back, you could do

[jira] [Commented] (SPARK-9182) filter and groupBy on DataFrames are not passed through to jdbc source

[ https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955646#comment-14955646 ] Davies Liu commented on SPARK-9182: --- For JDBC, I think we could push more stuff (for example, a + b > 3)

[jira] [Resolved] (SPARK-11068) Add callback to query execution

[ https://issues.apache.org/jira/browse/SPARK-11068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11068. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9078

[jira] [Resolved] (SPARK-8170) Ctrl-C in pyspark shell doesn't kill running job

[ https://issues.apache.org/jira/browse/SPARK-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8170. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9033

[jira] [Updated] (SPARK-11038) Consolidate the format of UnsafeArrayData and UnsafeMapData

[ https://issues.apache.org/jira/browse/SPARK-11038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-11038: --- Component/s: (was: Spark Core) SQL > Consolidate the format of UnsafeArrayData

[jira] [Resolved] (SPARK-9280) New HiveContext object unexpectedly loads configuration settings from history

[ https://issues.apache.org/jira/browse/SPARK-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9280. --- Resolution: Fixed Assignee: Davies Liu Fix Version/s: 1.6.0 > New HiveContext object

[jira] [Resolved] (SPARK-11007) Add dictionary support for CatalystDecimalConverter

[ https://issues.apache.org/jira/browse/SPARK-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11007. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9040

[jira] [Resolved] (SPARK-11042) Introduce a mechanism to ban creating new root SQLContexts in a JVM

[ https://issues.apache.org/jira/browse/SPARK-11042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11042. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9058

[jira] [Resolved] (SPARK-10617) Leap year miscalculated in sql query

[ https://issues.apache.org/jira/browse/SPARK-10617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10617. Resolution: Not A Problem This is expected behavior, see

[jira] [Created] (SPARK-11038) Consolidate the format of UnsafeArrayData and UnsafeMapData

Davies Liu created SPARK-11038: -- Summary: Consolidate the format of UnsafeArrayData and UnsafeMapData Key: SPARK-11038 URL: https://issues.apache.org/jira/browse/SPARK-11038 Project: Spark

[jira] [Resolved] (SPARK-10535) Support for recommendUsersForProducts and recommendProductsForUsers in matrix factorization model for PySpark

[ https://issues.apache.org/jira/browse/SPARK-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10535. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8700

[jira] [Assigned] (SPARK-11009) RowNumber in HiveContext returns negative values in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-11009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-11009: -- Assignee: Davies Liu > RowNumber in HiveContext returns negative values in cluster mode >

[jira] [Created] (SPARK-11036) AttributeReference should not be created outside driver

Davies Liu created SPARK-11036: -- Summary: AttributeReference should not be created outside driver Key: SPARK-11036 URL: https://issues.apache.org/jira/browse/SPARK-11036 Project: Spark Issue

[jira] [Created] (SPARK-11030) SQLTab should be shared by across sessions

Davies Liu created SPARK-11030: -- Summary: SQLTab should be shared by across sessions Key: SPARK-11030 URL: https://issues.apache.org/jira/browse/SPARK-11030 Project: Spark Issue Type:

[jira] [Commented] (SPARK-9280) New HiveContext object unexpectedly loads configuration settings from history

[ https://issues.apache.org/jira/browse/SPARK-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949706#comment-14949706 ] Davies Liu commented on SPARK-9280: --- [~tien-dung.le] I check this with latest master (with SQLContext,

[jira] [Resolved] (SPARK-10902) Hive UDF current_database() does not work

[ https://issues.apache.org/jira/browse/SPARK-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10902. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8909

[jira] [Resolved] (SPARK-10810) Improve session management for SQL

[ https://issues.apache.org/jira/browse/SPARK-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10810. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8909

[jira] [Created] (SPARK-11018) Support UDT in codegen and unsafe projection

Davies Liu created SPARK-11018: -- Summary: Support UDT in codegen and unsafe projection Key: SPARK-11018 URL: https://issues.apache.org/jira/browse/SPARK-11018 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-8552) Using incorrect database in multiple sessions

[ https://issues.apache.org/jira/browse/SPARK-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8552. --- Resolution: Fixed Assignee: Davies Liu Fix Version/s: 1.6.0 Resolved by

[jira] [Commented] (SPARK-6540) Spark SQL thrift server fails to pass settings to following query in the same session

[ https://issues.apache.org/jira/browse/SPARK-6540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949700#comment-14949700 ] Davies Liu commented on SPARK-6540: --- [~alexliu68] Could you check that this is fixed in master or not?

[jira] [Resolved] (SPARK-11006) Rename NullColumnAccess as NullColumnAccessor

[ https://issues.apache.org/jira/browse/SPARK-11006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-11006. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9028

[jira] [Updated] (SPARK-10980) Create wrong decimal if unscaled > 1e18 and scale > 0

[ https://issues.apache.org/jira/browse/SPARK-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10980: --- Description: Decimal(100L, 20, 2) will become 100 instead of

[jira] [Resolved] (SPARK-10917) Improve performance of complex types in columnar cache

[ https://issues.apache.org/jira/browse/SPARK-10917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10917. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8971

[jira] [Created] (SPARK-10980) Create wrong decimal with unscaled value and precision > 18

Davies Liu created SPARK-10980: -- Summary: Create wrong decimal with unscaled value and precision > 18 Key: SPARK-10980 URL: https://issues.apache.org/jira/browse/SPARK-10980 Project: Spark

[jira] [Updated] (SPARK-10980) Create wrong decimal with unscaled value and precision > 18

[ https://issues.apache.org/jira/browse/SPARK-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10980: --- Affects Version/s: 1.2.2 1.3.1 1.4.1 > Create wrong

[jira] [Updated] (SPARK-10980) Create wrong decimal if unscaled > 1e18 and scale > 0

[ https://issues.apache.org/jira/browse/SPARK-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10980: --- Summary: Create wrong decimal if unscaled > 1e18 and scale > 0 (was: Create wrong decimal with

[jira] [Resolved] (SPARK-10980) Create wrong decimal if unscaled > 1e18 and scale > 0

[ https://issues.apache.org/jira/browse/SPARK-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10980. Resolution: Fixed Fix Version/s: 1.2.3 1.5.2 1.3.2

[jira] [Created] (SPARK-10990) Avoid the serialization multiple times during unrolling of complex types

Davies Liu created SPARK-10990: -- Summary: Avoid the serialization multiple times during unrolling of complex types Key: SPARK-10990 URL: https://issues.apache.org/jira/browse/SPARK-10990 Project: Spark

[jira] [Resolved] (SPARK-10752) Implement corr() and cov in DataFrameStatFunctions

[ https://issues.apache.org/jira/browse/SPARK-10752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10752. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8869

[jira] [Resolved] (SPARK-10938) Remove typeId in columnar cache

[ https://issues.apache.org/jira/browse/SPARK-10938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10938. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8989

[jira] [Updated] (SPARK-10810) Improve session management for SQL

[ https://issues.apache.org/jira/browse/SPARK-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10810: --- Attachment: Session management in Spark SQL 1.6.pdf Design doc > Improve session management for SQL

[jira] [Updated] (SPARK-10585) only copy data once when generate unsafe projection

[ https://issues.apache.org/jira/browse/SPARK-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10585: --- Assignee: Wenchen Fan > only copy data once when generate unsafe projection >

[jira] [Commented] (SPARK-9158) PyLint should only fail on error

[ https://issues.apache.org/jira/browse/SPARK-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945359#comment-14945359 ] Davies Liu commented on SPARK-9158: --- Just realized that PyLint is not that useful, right now it's

[jira] [Resolved] (SPARK-10585) only copy data once when generate unsafe projection

[ https://issues.apache.org/jira/browse/SPARK-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10585. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8747

[jira] [Commented] (SPARK-10685) Misaligned data with RDD.zip and DataFrame.withColumn after repartition

[ https://issues.apache.org/jira/browse/SPARK-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944112#comment-14944112 ] Davies Liu commented on SPARK-10685: [~jdanbrown] The zip after repartition (or shuffling) is another

[jira] [Resolved] (SPARK-10934) hashCode of unsafe array may crush

[ https://issues.apache.org/jira/browse/SPARK-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10934. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8987

[jira] [Updated] (SPARK-10934) hashCode of unsafe array may crush

[ https://issues.apache.org/jira/browse/SPARK-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10934: --- Fix Version/s: 1.5.2 > hashCode of unsafe array may crush > -- > >

[jira] [Updated] (SPARK-10934) hashCode of unsafe array may crush

[ https://issues.apache.org/jira/browse/SPARK-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10934: --- Assignee: Wenchen Fan > hashCode of unsafe array may crush > -- > >

[jira] [Resolved] (SPARK-10866) [Spark SQL] [UDF] the floor function got wrong return value type

[ https://issues.apache.org/jira/browse/SPARK-10866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10866. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8933

[jira] [Resolved] (SPARK-10865) [Spark SQL] [UDF] the ceil/ceiling function got wrong return value type

[ https://issues.apache.org/jira/browse/SPARK-10865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10865. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8933

[jira] [Commented] (SPARK-10342) Cooperative memory management

[ https://issues.apache.org/jira/browse/SPARK-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940524#comment-14940524 ] Davies Liu commented on SPARK-10342: This will be used internal for SQL. For example, aggregation and

[jira] [Commented] (SPARK-10903) Make sqlContext global

[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940508#comment-14940508 ] Davies Liu commented on SPARK-10903: LGTM. Another question is that can we have different SQLContext

[jira] [Resolved] (SPARK-9741) approx count distinct function

[ https://issues.apache.org/jira/browse/SPARK-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9741. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8362

[jira] [Resolved] (SPARK-6919) Add .asDict method to StatCounter

[ https://issues.apache.org/jira/browse/SPARK-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-6919. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 5516

[jira] [Resolved] (SPARK-10415) Enhance Navigation Sidebar in PySpark API

[ https://issues.apache.org/jira/browse/SPARK-10415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10415. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8571

[jira] [Created] (SPARK-10859) Predicates pushed to InmemoryColumnarTableScan are not evaluated correctly

Davies Liu created SPARK-10859: -- Summary: Predicates pushed to InmemoryColumnarTableScan are not evaluated correctly Key: SPARK-10859 URL: https://issues.apache.org/jira/browse/SPARK-10859 Project:

[jira] [Resolved] (SPARK-10395) Simplify CatalystReadSupport

[ https://issues.apache.org/jira/browse/SPARK-10395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10395. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8553

[jira] [Created] (SPARK-10810) Improve session management for SQL

Davies Liu created SPARK-10810: -- Summary: Improve session management for SQL Key: SPARK-10810 URL: https://issues.apache.org/jira/browse/SPARK-10810 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-9821) pyspark reduceByKey should allow a custom partitioner

[ https://issues.apache.org/jira/browse/SPARK-9821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9821. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8569

[jira] [Commented] (SPARK-10538) java.lang.NegativeArraySizeException during join

[ https://issues.apache.org/jira/browse/SPARK-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876338#comment-14876338 ] Davies Liu commented on SPARK-10538: Do you know which operator (join or other) the exception came

[jira] [Assigned] (SPARK-10538) java.lang.NegativeArraySizeException during join

[ https://issues.apache.org/jira/browse/SPARK-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-10538: -- Assignee: Davies Liu > java.lang.NegativeArraySizeException during join >

[jira] [Closed] (SPARK-10559) DataFrame schema ArrayType should accept ResultIterable

[ https://issues.apache.org/jira/browse/SPARK-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10559. -- Resolution: Won't Fix > DataFrame schema ArrayType should accept ResultIterable >

[jira] [Commented] (SPARK-10559) DataFrame schema ArrayType should accept ResultIterable

[ https://issues.apache.org/jira/browse/SPARK-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876377#comment-14876377 ] Davies Liu commented on SPARK-10559: It's easy to turn the ResultIterable into a list, so I'd like to

[jira] [Resolved] (SPARK-10459) PythonUDF could process UnsafeRow

[ https://issues.apache.org/jira/browse/SPARK-10459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10459. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8616

[jira] [Resolved] (SPARK-10642) Crash in rdd.lookup() with "java.lang.Long cannot be cast to java.lang.Integer"

[ https://issues.apache.org/jira/browse/SPARK-10642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10642. Resolution: Fixed Fix Version/s: 1.2.3 1.3.2 1.4.2

[jira] [Commented] (SPARK-10685) Misaligned data with RDD.zip and DataFrame.withColumn after repartition

[ https://issues.apache.org/jira/browse/SPARK-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804924#comment-14804924 ] Davies Liu commented on SPARK-10685: Internally, Python UDF use RDD.zip(), and compute the upstream

[jira] [Updated] (SPARK-10685) Misaligned data with RDD.zip and DataFrame.withColumn after repartition

[ https://issues.apache.org/jira/browse/SPARK-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10685: --- Target Version/s: 1.5.1 Priority: Blocker (was: Major) Component/s: SQL

[jira] [Assigned] (SPARK-10593) sql lateral view same name gives wrong value

[ https://issues.apache.org/jira/browse/SPARK-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-10593: -- Assignee: Davies Liu > sql lateral view same name gives wrong value >

[jira] [Resolved] (SPARK-10522) Nanoseconds part of Timestamp should be positive in parquet

[ https://issues.apache.org/jira/browse/SPARK-10522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10522. Resolution: Fixed Fix Version/s: 1.5.1 1.6.0 Issue resolved by pull

[jira] [Commented] (SPARK-9325) Support `collect` on DataFrame columns

[ https://issues.apache.org/jira/browse/SPARK-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744254#comment-14744254 ] Davies Liu commented on SPARK-9325: --- I would -1 on this. I'm worried that once we have

[jira] [Resolved] (SPARK-10542) The PySpark 1.5 closure serializer can't serialize a namedtuple instance.

[ https://issues.apache.org/jira/browse/SPARK-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10542. Resolution: Fixed Fix Version/s: 1.5.1 1.6.0 Target

[jira] [Created] (SPARK-10593) sql lateral view same name gives wrong value

Davies Liu created SPARK-10593: -- Summary: sql lateral view same name gives wrong value Key: SPARK-10593 URL: https://issues.apache.org/jira/browse/SPARK-10593 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-10593) sql lateral view same name gives wrong value

[ https://issues.apache.org/jira/browse/SPARK-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10593: --- Description: This query will return wrong result: {code} select insideLayer1.json as

[jira] [Resolved] (SPARK-6548) stddev_pop and stddev_samp aggregate functions

[ https://issues.apache.org/jira/browse/SPARK-6548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-6548. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 6297

[jira] [Created] (SPARK-10572) Investigate the contentions bewteen tasks in the same executor

Davies Liu created SPARK-10572: -- Summary: Investigate the contentions bewteen tasks in the same executor Key: SPARK-10572 URL: https://issues.apache.org/jira/browse/SPARK-10572 Project: Spark

[jira] [Resolved] (SPARK-9014) Allow Python spark API to use built-in exponential operator

[ https://issues.apache.org/jira/browse/SPARK-9014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9014. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8658

[jira] [Resolved] (SPARK-10065) Avoid triple copy of var-length objects in Array in tungsten projection

[ https://issues.apache.org/jira/browse/SPARK-10065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10065. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8496

[jira] [Created] (SPARK-10542) The PySpark 1.5 closure serializer can't serialize a namedtuple instance.

Davies Liu created SPARK-10542: -- Summary: The PySpark 1.5 closure serializer can't serialize a namedtuple instance. Key: SPARK-10542 URL: https://issues.apache.org/jira/browse/SPARK-10542 Project:

[jira] [Resolved] (SPARK-6931) python: struct.pack('!q', value) in write_long(value, stream) in serializers.py require int(but doesn't raise exceptions in common cases)

[ https://issues.apache.org/jira/browse/SPARK-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-6931. --- Resolution: Fixed Fix Version/s: 1.2.3 1.3.2 Issue resolved by pull request

[jira] [Created] (SPARK-10553) Allow Ctrl-C in pyspark shell to kill running job

Davies Liu created SPARK-10553: -- Summary: Allow Ctrl-C in pyspark shell to kill running job Key: SPARK-10553 URL: https://issues.apache.org/jira/browse/SPARK-10553 Project: Spark Issue Type:

[jira] [Closed] (SPARK-10553) Allow Ctrl-C in pyspark shell to kill running job

[ https://issues.apache.org/jira/browse/SPARK-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10553. -- Resolution: Duplicate > Allow Ctrl-C in pyspark shell to kill running job >

[jira] [Closed] (SPARK-10544) Serialization of Python namedtuple subclasses in functions / closures is broken

[ https://issues.apache.org/jira/browse/SPARK-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10544. -- Resolution: Duplicate Fix Version/s: (was: 1.5.1) Target Version/s: 1.5.1 >

[jira] [Resolved] (SPARK-10443) Refactor SortMergeOuterJoin to reduce duplication

[ https://issues.apache.org/jira/browse/SPARK-10443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10443. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8596

[jira] [Resolved] (SPARK-10056) PySpark Row - Support for row["columnName"] syntax

[ https://issues.apache.org/jira/browse/SPARK-10056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10056. Resolution: Fixed Assignee: Yanbo Liang Fix Version/s: 1.6.0 > PySpark Row -

[jira] [Closed] (SPARK-10397) Make Python's SparkContext self-descriptive on "print sc"

[ https://issues.apache.org/jira/browse/SPARK-10397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10397. -- Resolution: Won't Fix > Make Python's SparkContext self-descriptive on "print sc" >

[jira] [Commented] (SPARK-10512) Fix @since when a function doesn't have doc

[ https://issues.apache.org/jira/browse/SPARK-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736961#comment-14736961 ] Davies Liu commented on SPARK-10512: As we discussed here

[jira] [Closed] (SPARK-10512) Fix @since when a function doesn't have doc

[ https://issues.apache.org/jira/browse/SPARK-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10512. -- Resolution: Won't Fix > Fix @since when a function doesn't have doc >

[jira] [Resolved] (SPARK-9730) Sort Merge Join for Full Outer Join

[ https://issues.apache.org/jira/browse/SPARK-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9730. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8579

[jira] [Created] (SPARK-10522) Nanoseconds part of Timestamp should be positive in parquet

Davies Liu created SPARK-10522: -- Summary: Nanoseconds part of Timestamp should be positive in parquet Key: SPARK-10522 URL: https://issues.apache.org/jira/browse/SPARK-10522 Project: Spark

[jira] [Updated] (SPARK-10474) Aggregation failed with unable to acquire memory

[ https://issues.apache.org/jira/browse/SPARK-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10474: --- Target Version/s: 1.6.0, 1.5.1 Priority: Blocker (was: Critical) > Aggregation failed

[jira] [Resolved] (SPARK-10461) make sure `input.primitive` is always variable name not code at GenerateUnsafeProjection

[ https://issues.apache.org/jira/browse/SPARK-10461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10461. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8613

[jira] [Commented] (SPARK-10519) Investigate if we should encode timezone information to a timestamp value stored in JSON

[ https://issues.apache.org/jira/browse/SPARK-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737375#comment-14737375 ] Davies Liu commented on SPARK-10519: +1 for 3, user have the ability to control timezone, it's also

[jira] [Comment Edited] (SPARK-10309) Some tasks failed with Unable to acquire memory

[ https://issues.apache.org/jira/browse/SPARK-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737185#comment-14737185 ] Davies Liu edited comment on SPARK-10309 at 9/9/15 4:53 PM: [~nadenf] Thanks

[jira] [Commented] (SPARK-10309) Some tasks failed with Unable to acquire memory

[ https://issues.apache.org/jira/browse/SPARK-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737185#comment-14737185 ] Davies Liu commented on SPARK-10309: [~nadenf] Thanks for letting us know, just realized that your

[jira] [Commented] (SPARK-10439) Catalyst should check for overflow / underflow of date and timestamp values

[ https://issues.apache.org/jira/browse/SPARK-10439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737644#comment-14737644 ] Davies Liu commented on SPARK-10439: There are many places there could be overflow, even for A + B,

[jira] [Commented] (SPARK-8632) Poor Python UDF performance because of RDD caching

[ https://issues.apache.org/jira/browse/SPARK-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735392#comment-14735392 ] Davies Liu commented on SPARK-8632: --- [~rxin] As [~justin.uang] suggested before, the batch mode will

[jira] [Commented] (SPARK-10309) Some tasks failed with Unable to acquire memory

[ https://issues.apache.org/jira/browse/SPARK-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735401#comment-14735401 ] Davies Liu commented on SPARK-10309: [~nadenf] In my case, the job finally finished (after retry), so

<    8   9   10   11   12   13   14   15   16   17   >