[jira] [Created] (SPARK-10542) The PySpark 1.5 closure serializer can't serialize a namedtuple instance.

Davies Liu created SPARK-10542: -- Summary: The PySpark 1.5 closure serializer can't serialize a namedtuple instance. Key: SPARK-10542 URL: https://issues.apache.org/jira/browse/SPARK-10542 Project: Spark

[jira] [Resolved] (SPARK-6931) python: struct.pack('!q', value) in write_long(value, stream) in serializers.py require int(but doesn't raise exceptions in common cases)

[ https://issues.apache.org/jira/browse/SPARK-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-6931. --- Resolution: Fixed Fix Version/s: 1.2.3 1.3.2 Issue resolved by pull request

[jira] [Resolved] (SPARK-10065) Avoid triple copy of var-length objects in Array in tungsten projection

[ https://issues.apache.org/jira/browse/SPARK-10065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10065. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8496 [https://github.c

[jira] [Resolved] (SPARK-9730) Sort Merge Join for Full Outer Join

[ https://issues.apache.org/jira/browse/SPARK-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9730. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8579 [https://github.com/

[jira] [Created] (SPARK-10522) Nanoseconds part of Timestamp should be positive in parquet

Davies Liu created SPARK-10522: -- Summary: Nanoseconds part of Timestamp should be positive in parquet Key: SPARK-10522 URL: https://issues.apache.org/jira/browse/SPARK-10522 Project: Spark Issu

[jira] [Commented] (SPARK-10439) Catalyst should check for overflow / underflow of date and timestamp values

[ https://issues.apache.org/jira/browse/SPARK-10439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737644#comment-14737644 ] Davies Liu commented on SPARK-10439: There are many places there could be overflow, e

[jira] [Updated] (SPARK-10474) Aggregation failed with unable to acquire memory

[ https://issues.apache.org/jira/browse/SPARK-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10474: --- Target Version/s: 1.6.0, 1.5.1 Priority: Blocker (was: Critical) > Aggregation failed wi

[jira] [Commented] (SPARK-10519) Investigate if we should encode timezone information to a timestamp value stored in JSON

[ https://issues.apache.org/jira/browse/SPARK-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737375#comment-14737375 ] Davies Liu commented on SPARK-10519: +1 for 3, user have the ability to control timez

[jira] [Resolved] (SPARK-10461) make sure `input.primitive` is always variable name not code at GenerateUnsafeProjection

[ https://issues.apache.org/jira/browse/SPARK-10461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10461. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8613 [https://github.c

[jira] [Commented] (SPARK-10309) Some tasks failed with Unable to acquire memory

[ https://issues.apache.org/jira/browse/SPARK-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737185#comment-14737185 ] Davies Liu commented on SPARK-10309: [~nadenf] Thanks for letting us know, just reali

[jira] [Comment Edited] (SPARK-10309) Some tasks failed with Unable to acquire memory

[ https://issues.apache.org/jira/browse/SPARK-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737185#comment-14737185 ] Davies Liu edited comment on SPARK-10309 at 9/9/15 4:53 PM: [

[jira] [Closed] (SPARK-10512) Fix @since when a function doesn't have doc

[ https://issues.apache.org/jira/browse/SPARK-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10512. -- Resolution: Won't Fix > Fix @since when a function doesn't have doc > -

[jira] [Commented] (SPARK-10512) Fix @since when a function doesn't have doc

[ https://issues.apache.org/jira/browse/SPARK-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736961#comment-14736961 ] Davies Liu commented on SPARK-10512: As we discussed here https://github.com/apache/

[jira] [Commented] (SPARK-10309) Some tasks failed with Unable to acquire memory

[ https://issues.apache.org/jira/browse/SPARK-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735908#comment-14735908 ] Davies Liu commented on SPARK-10309: This also could be related to https://issues.ap

[jira] [Commented] (SPARK-10309) Some tasks failed with Unable to acquire memory

[ https://issues.apache.org/jira/browse/SPARK-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735903#comment-14735903 ] Davies Liu commented on SPARK-10309: [~nadenf] Could you post the physical plan here?

[jira] [Commented] (SPARK-10466) UnsafeRow exception in Sort-Based Shuffle with data spill

[ https://issues.apache.org/jira/browse/SPARK-10466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735776#comment-14735776 ] Davies Liu commented on SPARK-10466: [~chenghao] I tried your test case, it passed in

[jira] [Created] (SPARK-10494) Multiple Python UDFs together with aggregation or sort merge join may cause OOM (failed to acquire memory)

Davies Liu created SPARK-10494: -- Summary: Multiple Python UDFs together with aggregation or sort merge join may cause OOM (failed to acquire memory) Key: SPARK-10494 URL: https://issues.apache.org/jira/browse/SPARK-1

[jira] [Commented] (SPARK-8632) Poor Python UDF performance because of RDD caching

[ https://issues.apache.org/jira/browse/SPARK-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735644#comment-14735644 ] Davies Liu commented on SPARK-8632: --- The upstream means child of current SparkPlan, coul

[jira] [Commented] (SPARK-10309) Some tasks failed with Unable to acquire memory

[ https://issues.apache.org/jira/browse/SPARK-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735401#comment-14735401 ] Davies Liu commented on SPARK-10309: [~nadenf] In my case, the job finally finished (

[jira] [Commented] (SPARK-8632) Poor Python UDF performance because of RDD caching

[ https://issues.apache.org/jira/browse/SPARK-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735392#comment-14735392 ] Davies Liu commented on SPARK-8632: --- [~rxin] As [~justin.uang] suggested before, the bat

[jira] [Assigned] (SPARK-8632) Poor Python UDF performance because of RDD caching

[ https://issues.apache.org/jira/browse/SPARK-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-8632: - Assignee: Davies Liu > Poor Python UDF performance because of RDD caching > -

[jira] [Created] (SPARK-10459) PythonUDF could process UnsafeRow

Davies Liu created SPARK-10459: -- Summary: PythonUDF could process UnsafeRow Key: SPARK-10459 URL: https://issues.apache.org/jira/browse/SPARK-10459 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-10436) spark-submit overwrites spark.files defaults with the job script filename

[ https://issues.apache.org/jira/browse/SPARK-10436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10436: --- Target Version/s: 1.6.0 > spark-submit overwrites spark.files defaults with the job script filename >

[jira] [Updated] (SPARK-10434) Parquet compatibility with 1.4 is broken when writing arrays that may contain nulls

[ https://issues.apache.org/jira/browse/SPARK-10434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10434: --- Priority: Minor (was: Critical) > Parquet compatibility with 1.4 is broken when writing arrays that

[jira] [Commented] (SPARK-10434) Parquet compatibility with 1.4 is broken when writing arrays that may contain nulls

[ https://issues.apache.org/jira/browse/SPARK-10434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729445#comment-14729445 ] Davies Liu commented on SPARK-10434: [~lian cheng] I think it's hard to guarantee for

[jira] [Commented] (SPARK-10425) Add a regression test for SPARK-10379

[ https://issues.apache.org/jira/browse/SPARK-10425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729421#comment-14729421 ] Davies Liu commented on SPARK-10425: [~sowen] Thanks for your comment, The reason tha

[jira] [Resolved] (SPARK-10422) String column in InMemoryColumnarCache needs to override clone method

[ https://issues.apache.org/jira/browse/SPARK-10422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10422. Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8578 [https://github.c

[jira] [Created] (SPARK-10425) Add a regression test for SPARK-10379

Davies Liu created SPARK-10425: -- Summary: Add a regression test for SPARK-10379 Key: SPARK-10425 URL: https://issues.apache.org/jira/browse/SPARK-10425 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-10424) ShuffleHashOuterJoin should consider condition

[ https://issues.apache.org/jira/browse/SPARK-10424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10424: --- Priority: Blocker (was: Major) > ShuffleHashOuterJoin should consider condition > --

[jira] [Created] (SPARK-10424) ShuffleHashOuterJoin should consider condition

Davies Liu created SPARK-10424: -- Summary: ShuffleHashOuterJoin should consider condition Key: SPARK-10424 URL: https://issues.apache.org/jira/browse/SPARK-10424 Project: Spark Issue Type: New Fe

[jira] [Resolved] (SPARK-10417) Iterating through Column results in infinite loop

[ https://issues.apache.org/jira/browse/SPARK-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10417. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8574 [https://github.c

[jira] [Updated] (SPARK-10392) Pyspark - Wrong DateType support on JDBC connection

[ https://issues.apache.org/jira/browse/SPARK-10392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10392: --- Fix Version/s: 1.5.1 > Pyspark - Wrong DateType support on JDBC connection >

[jira] [Resolved] (SPARK-10392) Pyspark - Wrong DateType support on JDBC connection

[ https://issues.apache.org/jira/browse/SPARK-10392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10392. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8556 [https://github.c

[jira] [Resolved] (SPARK-10162) PySpark filters with datetimes mess up when datetimes have timezones.

[ https://issues.apache.org/jira/browse/SPARK-10162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10162. Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8555 [https://github.c

[jira] [Created] (SPARK-10404) Worker should terminate previous executor before launch new one

Davies Liu created SPARK-10404: -- Summary: Worker should terminate previous executor before launch new one Key: SPARK-10404 URL: https://issues.apache.org/jira/browse/SPARK-10404 Project: Spark

[jira] [Updated] (SPARK-10379) UnsafeShuffleExternalSorter should preserve first page

[ https://issues.apache.org/jira/browse/SPARK-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10379: --- Target Version/s: 1.6.0, 1.5.1 (was: 1.5.0) > UnsafeShuffleExternalSorter should preserve first page

[jira] [Created] (SPARK-10403) UnsafeRowSerializer can't work with UnsafeShuffleManager (tungsten-sort)

Davies Liu created SPARK-10403: -- Summary: UnsafeRowSerializer can't work with UnsafeShuffleManager (tungsten-sort) Key: SPARK-10403 URL: https://issues.apache.org/jira/browse/SPARK-10403 Project: Spark

[jira] [Created] (SPARK-10379) UnsafeShuffleExternalSorter should preserve first page

Davies Liu created SPARK-10379: -- Summary: UnsafeShuffleExternalSorter should preserve first page Key: SPARK-10379 URL: https://issues.apache.org/jira/browse/SPARK-10379 Project: Spark Issue Type

[jira] [Commented] (SPARK-10373) Move @since annotator to pyspark to be shared by all components

[ https://issues.apache.org/jira/browse/SPARK-10373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724009#comment-14724009 ] Davies Liu commented on SPARK-10373: [~mengxr] Do we want to add @since for the MLLib

[jira] [Created] (SPARK-10345) Flaky test: HiveCompatibilitySuite.nonblock_op_deduplicate

Davies Liu created SPARK-10345: -- Summary: Flaky test: HiveCompatibilitySuite.nonblock_op_deduplicate Key: SPARK-10345 URL: https://issues.apache.org/jira/browse/SPARK-10345 Project: Spark Issue

[jira] [Updated] (SPARK-10341) SMJ fail with unable to acquire memory

[ https://issues.apache.org/jira/browse/SPARK-10341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10341: --- Target Version/s: 1.5.0 (was: 1.5.1) > SMJ fail with unable to acquire memory >

[jira] [Resolved] (SPARK-10323) NPE in code-gened In expression

[ https://issues.apache.org/jira/browse/SPARK-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10323. Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8492 [https://github.c

[jira] [Created] (SPARK-10343) Consider nullability of expression in codegen

Davies Liu created SPARK-10343: -- Summary: Consider nullability of expression in codegen Key: SPARK-10343 URL: https://issues.apache.org/jira/browse/SPARK-10343 Project: Spark Issue Type: Improve

[jira] [Updated] (SPARK-10342) Cooperative memory management

[ https://issues.apache.org/jira/browse/SPARK-10342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-10342: --- Issue Type: Improvement (was: Story) > Cooperative memory management > -

[jira] [Created] (SPARK-10342) Cooperative memory management

Davies Liu created SPARK-10342: -- Summary: Cooperative memory management Key: SPARK-10342 URL: https://issues.apache.org/jira/browse/SPARK-10342 Project: Spark Issue Type: Bug Component

[jira] [Created] (SPARK-10341) SMJ fail with unable to acquire memory

Davies Liu created SPARK-10341: -- Summary: SMJ fail with unable to acquire memory Key: SPARK-10341 URL: https://issues.apache.org/jira/browse/SPARK-10341 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-10321) OrcRelation doesn't override sizeInBytes

[ https://issues.apache.org/jira/browse/SPARK-10321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-10321: -- Assignee: Davies Liu > OrcRelation doesn't override sizeInBytes >

[jira] [Created] (SPARK-10309) Some tasks failed with Unable to acquire memory

Davies Liu created SPARK-10309: -- Summary: Some tasks failed with Unable to acquire memory Key: SPARK-10309 URL: https://issues.apache.org/jira/browse/SPARK-10309 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-10305) PySpark createDataFrame on list of LabeledPoints fails (regression)

[ https://issues.apache.org/jira/browse/SPARK-10305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10305. Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8470 [https://github.c

[jira] [Closed] (SPARK-10302) NPE while save a DataFrame as ORC

[ https://issues.apache.org/jira/browse/SPARK-10302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-10302. -- Resolution: Duplicate Fix Version/s: 1.5.0 > NPE while save a DataFrame as ORC > ---

[jira] [Created] (SPARK-10302) NPE while save a DataFrame as ORC

Davies Liu created SPARK-10302: -- Summary: NPE while save a DataFrame as ORC Key: SPARK-10302 URL: https://issues.apache.org/jira/browse/SPARK-10302 Project: Spark Issue Type: Bug Compo

[jira] [Commented] (SPARK-9228) Combine unsafe and codegen into a single option

[ https://issues.apache.org/jira/browse/SPARK-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715512#comment-14715512 ] Davies Liu commented on SPARK-9228: --- [~jameszhouyi] unsafe.offHeap is another option tha

[jira] [Created] (SPARK-10245) SQLContext can't parse literal less than 0.1 ( 0.01)

Davies Liu created SPARK-10245: -- Summary: SQLContext can't parse literal less than 0.1 ( 0.01) Key: SPARK-10245 URL: https://issues.apache.org/jira/browse/SPARK-10245 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-10196) Failed to save json data with a decimal type in the schema

[ https://issues.apache.org/jira/browse/SPARK-10196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10196. Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8408 [https://github.c

[jira] [Commented] (SPARK-10215) Div of Decimal returns null

[ https://issues.apache.org/jira/browse/SPARK-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14710696#comment-14710696 ] Davies Liu commented on SPARK-10215: I think we have not enough time to figure out th

[jira] [Resolved] (SPARK-8580) Test Parquet interoperability and compatibility with other libraries/systems

[ https://issues.apache.org/jira/browse/SPARK-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8580. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8392 [https://github.com/

[jira] [Closed] (SPARK-7506) pyspark.sql.types.StructType.fromJson() is incorrectly named

[ https://issues.apache.org/jira/browse/SPARK-7506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-7506. - Resolution: Won't Fix Assignee: (was: Davies Liu) These functions could be used by users (even w

[jira] [Commented] (SPARK-10177) Parquet support interprets timestamp values differently from Hive 0.14.0+

[ https://issues.apache.org/jira/browse/SPARK-10177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709878#comment-14709878 ] Davies Liu commented on SPARK-10177: [~lian cheng] After some investigation, just rea

[jira] [Resolved] (SPARK-9401) Fully implement code generation for ConcatWs

[ https://issues.apache.org/jira/browse/SPARK-9401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9401. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8353 [https://github.com/

[jira] [Commented] (SPARK-9228) Combine unsafe and codegen into a single option

[ https://issues.apache.org/jira/browse/SPARK-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706289#comment-14706289 ] Davies Liu commented on SPARK-9228: --- Right now, it's an internal configuration (could be

[jira] [Resolved] (SPARK-9400) Implement code generation for StringLocate

[ https://issues.apache.org/jira/browse/SPARK-9400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9400. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8330 [https://github.com/

[jira] [Reopened] (SPARK-3533) Add saveAsTextFileByKey() method to RDDs

[ https://issues.apache.org/jira/browse/SPARK-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reopened SPARK-3533: --- Re-open this for discussion. > Add saveAsTextFileByKey() method to RDDs > ---

[jira] [Created] (SPARK-10129) math function: stddev_samp

Davies Liu created SPARK-10129: -- Summary: math function: stddev_samp Key: SPARK-10129 URL: https://issues.apache.org/jira/browse/SPARK-10129 Project: Spark Issue Type: New Feature Comp

[jira] [Commented] (SPARK-7379) pickle.loads expects a string instead of bytes in Python 3.

[ https://issues.apache.org/jira/browse/SPARK-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703583#comment-14703583 ] Davies Liu commented on SPARK-7379: --- [~mengxr] This is a known issue that how to unpickl

[jira] [Commented] (SPARK-9644) Support update DecimalType with precision > 18 in UnsafeRow

[ https://issues.apache.org/jira/browse/SPARK-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703474#comment-14703474 ] Davies Liu commented on SPARK-9644: --- [~robbinspg] It's fixed by https://issues.apache.or

[jira] [Commented] (SPARK-6798) Fix Date serialization in SparkR

[ https://issues.apache.org/jira/browse/SPARK-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702625#comment-14702625 ] Davies Liu commented on SPARK-6798: --- Not a bug, it's just not efficient. > Fix Date se

[jira] [Updated] (SPARK-6798) Fix Date serialization in SparkR

[ https://issues.apache.org/jira/browse/SPARK-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-6798: -- Issue Type: Improvement (was: Bug) > Fix Date serialization in SparkR > ---

[jira] [Created] (SPARK-10107) NPE in format_number

Davies Liu created SPARK-10107: -- Summary: NPE in format_number Key: SPARK-10107 URL: https://issues.apache.org/jira/browse/SPARK-10107 Project: Spark Issue Type: Bug Components: SQL

[jira] [Resolved] (SPARK-10095) Should not use the private field of BigInteger

[ https://issues.apache.org/jira/browse/SPARK-10095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-10095. Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8286 [https://github.c

[jira] [Commented] (SPARK-9627) SQL job failed if the dataframe with string columns is cached

[ https://issues.apache.org/jira/browse/SPARK-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702390#comment-14702390 ] Davies Liu commented on SPARK-9627: --- I can reproduce it with latest master. > SQL job f

[jira] [Commented] (SPARK-9627) SQL job failed if the dataframe with string columns is cached

[ https://issues.apache.org/jira/browse/SPARK-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702387#comment-14702387 ] Davies Liu commented on SPARK-9627: --- The `df.show()` will succeed, but `df.groupBy(df.da

[jira] [Created] (SPARK-10095) Should not use the private field of BigInteger

Davies Liu created SPARK-10095: -- Summary: Should not use the private field of BigInteger Key: SPARK-10095 URL: https://issues.apache.org/jira/browse/SPARK-10095 Project: Spark Issue Type: Improv

[jira] [Commented] (SPARK-9644) Support update DecimalType with precision > 18 in UnsafeRow

[ https://issues.apache.org/jira/browse/SPARK-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701974#comment-14701974 ] Davies Liu commented on SPARK-9644: --- In a benchmark of doing aggregation on a decimal co

[jira] [Commented] (SPARK-9644) Support update DecimalType with precision > 18 in UnsafeRow

[ https://issues.apache.org/jira/browse/SPARK-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701891#comment-14701891 ] Davies Liu commented on SPARK-9644: --- [~robbinspg] Thanks for point this out, would is be

[jira] [Commented] (SPARK-10056) PySpark Row - Support for row["columnName"] syntax

[ https://issues.apache.org/jira/browse/SPARK-10056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701883#comment-14701883 ] Davies Liu commented on SPARK-10056: [~maver1ck], Yes, changes in Row class and unit

[jira] [Commented] (SPARK-10079) Make `column` and `col` functions be S4 functions

[ https://issues.apache.org/jira/browse/SPARK-10079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701877#comment-14701877 ] Davies Liu commented on SPARK-10079: There are two public col/column function in Sca

[jira] [Created] (SPARK-10090) After division, Decimal may have longer precision than expected

Davies Liu created SPARK-10090: -- Summary: After division, Decimal may have longer precision than expected Key: SPARK-10090 URL: https://issues.apache.org/jira/browse/SPARK-10090 Project: Spark

[jira] [Closed] (SPARK-5901) [PySpark] pickle classes in main module

[ https://issues.apache.org/jira/browse/SPARK-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-5901. - Resolution: Invalid Target Version/s: (was: 1.5.0) couldpickle does support to serialize class

[jira] [Created] (SPARK-10065) Avoid triple copy of var-length objects in Array in tungsten projection

Davies Liu created SPARK-10065: -- Summary: Avoid triple copy of var-length objects in Array in tungsten projection Key: SPARK-10065 URL: https://issues.apache.org/jira/browse/SPARK-10065 Project: Spark

[jira] [Commented] (SPARK-9427) Add expression functions in SparkR

[ https://issues.apache.org/jira/browse/SPARK-9427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700044#comment-14700044 ] Davies Liu commented on SPARK-9427: --- Should we target this for 1.6? > Add expression f

[jira] [Assigned] (SPARK-10038) TungstenProject code generation fails when applied to array

[ https://issues.apache.org/jira/browse/SPARK-10038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-10038: -- Assignee: Davies Liu > TungstenProject code generation fails when applied to array > -

[jira] [Commented] (SPARK-10056) PySpark Row - Support for row["columnName"] syntax

[ https://issues.apache.org/jira/browse/SPARK-10056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700028#comment-14700028 ] Davies Liu commented on SPARK-10056: We support accessing nested column by `df['a.b']

[jira] [Commented] (SPARK-9705) outdated Python 3 and IPython information

[ https://issues.apache.org/jira/browse/SPARK-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699983#comment-14699983 ] Davies Liu commented on SPARK-9705: --- The PyLab thing is already fixed by https://github.

[jira] [Commented] (SPARK-9705) outdated Python 3 and IPython information

[ https://issues.apache.org/jira/browse/SPARK-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699973#comment-14699973 ] Davies Liu commented on SPARK-9705: --- As the exception said, PySpark cannot run with diff

[jira] [Closed] (SPARK-9822) Update doc about supported Python versions

[ https://issues.apache.org/jira/browse/SPARK-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-9822. - Resolution: Duplicate > Update doc about supported Python versions > -

[jira] [Created] (SPARK-10059) Broken test: YarnClusterSuite

Davies Liu created SPARK-10059: -- Summary: Broken test: YarnClusterSuite Key: SPARK-10059 URL: https://issues.apache.org/jira/browse/SPARK-10059 Project: Spark Issue Type: Test Report

[jira] [Created] (SPARK-10058) Flaky test: HeartbeatReceiverSuite: normal heartbeat

Davies Liu created SPARK-10058: -- Summary: Flaky test: HeartbeatReceiverSuite: normal heartbeat Key: SPARK-10058 URL: https://issues.apache.org/jira/browse/SPARK-10058 Project: Spark Issue Type:

[jira] [Commented] (SPARK-9982) SparkR DataFrame fail to return data of Decimal type

[ https://issues.apache.org/jira/browse/SPARK-9982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699861#comment-14699861 ] Davies Liu commented on SPARK-9982: --- R has numeric (similar to double), no decimal, is i

[jira] [Comment Edited] (SPARK-9982) SparkR DataFrame fail to return data of Decimal type

[ https://issues.apache.org/jira/browse/SPARK-9982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699861#comment-14699861 ] Davies Liu edited comment on SPARK-9982 at 8/17/15 5:21 PM: R

[jira] [Commented] (SPARK-9427) Add expression functions in SparkR

[ https://issues.apache.org/jira/browse/SPARK-9427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699843#comment-14699843 ] Davies Liu commented on SPARK-9427: --- [~yu_ishikawa] `rand` does work in PySpark (Python

[jira] [Created] (SPARK-10057) Faill to load class org.slf4j.impl.StaticLoggerBinder

Davies Liu created SPARK-10057: -- Summary: Faill to load class org.slf4j.impl.StaticLoggerBinder Key: SPARK-10057 URL: https://issues.apache.org/jira/browse/SPARK-10057 Project: Spark Issue Type:

[jira] [Commented] (SPARK-9734) java.lang.IllegalArgumentException: Don't know how to save StructField(sal,DecimalType(7,2),true) to JDBC

[ https://issues.apache.org/jira/browse/SPARK-9734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698121#comment-14698121 ] Davies Liu commented on SPARK-9734: --- [~rmullapudi] It's fixed by https://github.com/apac

[jira] [Resolved] (SPARK-9725) spark sql query string field return empty/garbled string

[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9725. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8210 [https://github.com/

[jira] [Commented] (SPARK-9734) java.lang.IllegalArgumentException: Don't know how to save StructField(sal,DecimalType(7,2),true) to JDBC

[ https://issues.apache.org/jira/browse/SPARK-9734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14697998#comment-14697998 ] Davies Liu commented on SPARK-9734: --- [~rmullapudi] Good catch! I will sending out a quic

[jira] [Commented] (SPARK-9971) MaxFunction not working correctly with columns containing Double.NaN

[ https://issues.apache.org/jira/browse/SPARK-9971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14697868#comment-14697868 ] Davies Liu commented on SPARK-9971: --- We had a long discussion about how to support NaN,

[jira] [Assigned] (SPARK-9978) Window functions require partitionBy to work as expected

[ https://issues.apache.org/jira/browse/SPARK-9978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-9978: - Assignee: Davies Liu > Window functions require partitionBy to work as expected > ---

[jira] [Resolved] (SPARK-9589) Flaky test: HiveCompatibilitySuite.groupby8

[ https://issues.apache.org/jira/browse/SPARK-9589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9589. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8177 [https://github.com/

[jira] [Resolved] (SPARK-9946) NPE in TaskMemoryManager

[ https://issues.apache.org/jira/browse/SPARK-9946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-9946. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 8177 [https://github.com/

[jira] [Assigned] (SPARK-9589) Flaky test: HiveCompatibilitySuite.groupby8

[ https://issues.apache.org/jira/browse/SPARK-9589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-9589: - Assignee: Davies Liu (was: Josh Rosen) > Flaky test: HiveCompatibilitySuite.groupby8 > -

[jira] [Comment Edited] (SPARK-9725) spark sql query string field return empty/garbled string

[ https://issues.apache.org/jira/browse/SPARK-9725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14697426#comment-14697426 ] Davies Liu edited comment on SPARK-9725 at 8/14/15 5:36 PM: I

<    9   10   11   12   13   14   15   16   17   18   >