[jira] [Updated] (SPARK-6743) Join with empty projection on one side produces invalid results

2019-05-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-6743: -- Labels: correctness (was: ) > Join with empty projection on one side produces invalid results >

[jira] [Updated] (SPARK-18504) Scalar subquery with extra group by columns returning incorrect result

2019-05-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18504: --- Labels: correctness (was: ) > Scalar subquery with extra group by columns returning incorrect

[jira] [Updated] (SPARK-18473) Correctness issue in INNER join result with window functions

2019-05-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18473: --- Labels: correctness (was: ) > Correctness issue in INNER join result with window functions >

[jira] [Updated] (SPARK-19017) NOT IN subquery with more than one column may return incorrect results

2019-05-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-19017: --- Labels: correctness (was: ) > NOT IN subquery with more than one column may return incorrect

[jira] [Updated] (SPARK-18578) Full outer join in correlated subquery returns incorrect results

2019-05-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-18578: --- Labels: correctness (was: ) > Full outer join in correlated subquery returns incorrect results >

[jira] [Updated] (SPARK-20356) Spark sql group by returns incorrect results after join + distinct transformations

2019-05-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-20356: --- Labels: correctness (was: ) > Spark sql group by returns incorrect results after join + distinct

[jira] [Updated] (SPARK-27685) `union` doesn't promote non-nullable columns of struct to nullable

2019-05-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27685: --- Labels: correctness (was: ) > `union` doesn't promote non-nullable columns of struct to nullable >

[jira] [Updated] (SPARK-27684) Reduce ScalaUDF conversion overheads for primitives

2019-05-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27684: --- Description: I believe that we can reduce ScalaUDF overheads when operating over primitive types.

[jira] [Updated] (SPARK-27684) Reduce ScalaUDF conversion overheads for primitives

2019-05-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27684: --- Description: I believe that we can reduce ScalaUDF overheads when operating over primitive types.

[jira] [Updated] (SPARK-27684) Reduce ScalaUDF conversion overheads for primitives

2019-05-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27684: --- Description: I believe that we can reduce ScalaUDF overheads when operating over primitive types.

[jira] [Updated] (SPARK-27684) Reduce ScalaUDF conversion overheads for primitives

2019-05-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27684: --- Description: I believe that we can reduce ScalaUDF overheads when operating over primitive types.

[jira] [Created] (SPARK-27684) Reduce ScalaUDF conversion overheads for primitives

2019-05-12 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27684: -- Summary: Reduce ScalaUDF conversion overheads for primitives Key: SPARK-27684 URL: https://issues.apache.org/jira/browse/SPARK-27684 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-8351) Umbella for improving Spark documentation CSS + JS

2019-05-11 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-8351. --- Resolution: Done > Umbella for improving Spark documentation CSS + JS >

[jira] [Resolved] (SPARK-8352) Affixed table of contents, similar to Bootstrap 3 docs

2019-05-11 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-8352. --- Resolution: Fixed The new docs have a sidebar TOC, so marking as done. > Affixed table of contents,

[jira] [Resolved] (SPARK-3289) Avoid job failures due to rescheduling of failing tasks on buggy machines

2019-05-11 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3289. --- Resolution: Fixed As part of a cleanup of old tickets filed by me, I'm resolving this as "Fixed"

[jira] [Updated] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27676: --- Description: Spark's {{InMemoryFileIndex}} contains two places where {{FileNotFound}} exceptions

[jira] [Updated] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27676: --- Description: Spark's {{InMemoryFileIndex}} contains two places where {{FileNotFound}} exceptions

[jira] [Updated] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27676: --- Description: Spark's {{InMemoryFileIndex}} contains two places where {{FileNotFound}} exceptions

[jira] [Created] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-05-10 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27676: -- Summary: InMemoryFileIndex should hard-fail on missing files instead of logging and continuing Key: SPARK-27676 URL: https://issues.apache.org/jira/browse/SPARK-27676

[jira] [Created] (SPARK-27653) Add max_by() / min_by() SQL aggregate functions

2019-05-07 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27653: -- Summary: Add max_by() / min_by() SQL aggregate functions Key: SPARK-27653 URL: https://issues.apache.org/jira/browse/SPARK-27653 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-26555) Thread safety issue causes createDataset to fail with misleading errors

2019-05-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834203#comment-16834203 ] Josh Rosen commented on SPARK-26555: I won't be able to tackle a backport for at least a week, so

[jira] [Commented] (SPARK-26555) Thread safety issue causes createDataset to fail with misleading errors

2019-05-06 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834025#comment-16834025 ] Josh Rosen commented on SPARK-26555: [~cloud_fan] [~srowen], could we backport this to the 2.4.x

[jira] [Updated] (SPARK-27619) MapType should be prohibited in hash expressions

2019-05-02 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27619: --- Description: Spark currently allows MapType expressions to be used as input to hash expressions,

[jira] [Updated] (SPARK-27619) MapType should be prohibited in hash expressions

2019-05-02 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27619: --- Description: Spark currently allows MapType expressions to be used as input to hash expressions,

[jira] [Created] (SPARK-27619) MapType should be prohibited in hash expressions

2019-05-01 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27619: -- Summary: MapType should be prohibited in hash expressions Key: SPARK-27619 URL: https://issues.apache.org/jira/browse/SPARK-27619 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-27619) MapType should be prohibited in hash expressions

2019-05-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27619: --- Description: Spark currently allows MapType expressions to be used as input to hash expressions,

[jira] [Updated] (SPARK-27619) MapType should be prohibited in hash expressions

2019-05-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27619: --- Description: Spark currently allows MapType expressions to be used as input to hash expressions,

[jira] [Commented] (SPARK-17637) Packed scheduling for Spark tasks across executors

2019-05-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831014#comment-16831014 ] Josh Rosen commented on SPARK-17637: I think this old feature suggestion is still very relevant and

[jira] [Commented] (SPARK-27607) Improve performance of Row.toString()

2019-05-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830997#comment-16830997 ] Josh Rosen commented on SPARK-27607: Feel free to take this. > Improve performance of

[jira] [Created] (SPARK-27607) Improve performance of Row.toString()

2019-04-30 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27607: -- Summary: Improve performance of Row.toString() Key: SPARK-27607 URL: https://issues.apache.org/jira/browse/SPARK-27607 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-27213) Unexpected results when filter is used after distinct

2019-04-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829309#comment-16829309 ] Josh Rosen commented on SPARK-27213: Hmm, this must have been fixed relatively recently. For now, I

[jira] [Commented] (SPARK-27586) Improve binary comparison: replace Scala's for-comprehension if statements with while loop

2019-04-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829303#comment-16829303 ] Josh Rosen commented on SPARK-27586: Good find! This sounds pretty straightforward to fix; want to

[jira] [Updated] (SPARK-27581) DataFrame countDistinct("*") fails with AnalysisException: "Invalid usage of '*' in expression 'count'"

2019-04-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27581: --- Description: If I have a DataFrame then I can use {{count("*")}} as an expression, e.g.:

[jira] [Commented] (SPARK-27213) Unexpected results when filter is used after distinct

2019-04-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827453#comment-16827453 ] Josh Rosen commented on SPARK-27213: SPARK-26767 sounds like a similar, possibly-duplicated issue.

[jira] [Comment Edited] (SPARK-27213) Unexpected results when filter is used after distinct

2019-04-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827449#comment-16827449 ] Josh Rosen edited comment on SPARK-27213 at 4/27/19 5:13 AM: - Thank you for

[jira] [Commented] (SPARK-27213) Unexpected results when filter is used after distinct

2019-04-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827449#comment-16827449 ] Josh Rosen commented on SPARK-27213: Since this sounds like a legitimate query correctness bug, I'm

[jira] [Updated] (SPARK-27213) Unexpected results when filter is used after distinct

2019-04-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27213: --- Labels: correctness distinct filter (was: distinct filter) > Unexpected results when filter is

[jira] [Commented] (SPARK-27290) remove unneed sort under Aggregate

2019-04-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827448#comment-16827448 ] Josh Rosen commented on SPARK-27290: Regarding that test case, my best guess is that SPARK-23375 was

[jira] [Updated] (SPARK-27290) remove unneed sort under Aggregate

2019-04-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27290: --- Description: I saw some tickets to remove unneeded sort in plan while I think there's another case

[jira] [Updated] (SPARK-27290) remove unneed sort under Aggregate

2019-04-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27290: --- Description: I saw some tickets to remove unneeded sort in plan while I think there's another case

[jira] [Updated] (SPARK-27581) DataFrame countDistinct("*") fails with AnalysisException: "Invalid usage of '*' in expression 'count'"

2019-04-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27581: --- Issue Type: Bug (was: New Feature) > DataFrame countDistinct("*") fails with AnalysisException:

[jira] [Updated] (SPARK-27581) DataFrame countDistinct("*") fails with AnalysisException: "Invalid usage of '*' in expression 'count'"

2019-04-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27581: --- Description: If I have a DataFrame then I can use {{count("*")}} as an expression, e.g.: {code}

[jira] [Created] (SPARK-27581) DataFrame countDistinct("*") fails with AnalysisException: "Invalid usage of '*' in expression 'count'"

2019-04-26 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27581: -- Summary: DataFrame countDistinct("*") fails with AnalysisException: "Invalid usage of '*' in expression 'count'" Key: SPARK-27581 URL:

[jira] [Updated] (SPARK-27573) Skip partial aggregation when data is already partitioned (or collapse adjacent partial and final aggregates)

2019-04-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27573: --- Description: When an aggregation requires a shuffle, Spark SQL performs separate partial and final

[jira] [Updated] (SPARK-27573) Skip partial aggregation when data is already partitioned (or collapse adjacent partial and final aggregates)

2019-04-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27573: --- Summary: Skip partial aggregation when data is already partitioned (or collapse adjacent partial

[jira] [Updated] (SPARK-27573) Collapse adjacent physical aggregate operators when possible

2019-04-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27573: --- Description: When an aggregation requires a shuffle, Spark SQL performs separate partial and final

[jira] [Updated] (SPARK-27573) Collapse adjacent physical aggregate operators when possible

2019-04-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27573: --- Description: When an aggregation requires a shuffle, Spark SQL performs separate partial and final

[jira] [Updated] (SPARK-27573) Collapse adjacent physical aggregate operators when possible

2019-04-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27573: --- Summary: Collapse adjacent physical aggregate operators when possible (was: Collapse adjacent

[jira] [Created] (SPARK-27573) Collapse adjacent aggregate physical operators when possible

2019-04-25 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27573: -- Summary: Collapse adjacent aggregate physical operators when possible Key: SPARK-27573 URL: https://issues.apache.org/jira/browse/SPARK-27573 Project: Spark

[jira] [Commented] (SPARK-23178) Kryo Unsafe problems with count distinct from cache

2019-04-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825783#comment-16825783 ] Josh Rosen commented on SPARK-23178: This might be fixed by SPARK-27216 > Kryo Unsafe problems with

[jira] [Comment Edited] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

2019-04-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825782#comment-16825782 ] Josh Rosen edited comment on SPARK-27216 at 4/25/19 6:38 AM: - I've added the

[jira] [Updated] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

2019-04-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27216: --- Labels: correctness (was: ) > Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue >

[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

2019-04-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825782#comment-16825782 ] Josh Rosen commented on SPARK-27216: I've added the {{correctness}} label to this ticket because it

[jira] [Comment Edited] (SPARK-27530) FetchFailedException: Received a zero-size buffer for block shuffle

2019-04-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825770#comment-16825770 ] Josh Rosen edited comment on SPARK-27530 at 4/25/19 6:32 AM: - This specific

[jira] [Comment Edited] (SPARK-27530) FetchFailedException: Received a zero-size buffer for block shuffle

2019-04-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825770#comment-16825770 ] Josh Rosen edited comment on SPARK-27530 at 4/25/19 6:32 AM: - This specific

[jira] [Commented] (SPARK-27530) FetchFailedException: Received a zero-size buffer for block shuffle

2019-04-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825770#comment-16825770 ] Josh Rosen commented on SPARK-27530: This specific error message was added in SPARK-24160. As

[jira] [Updated] (SPARK-27561) Support "lateral column alias references" to allow column aliases to be used within SELECT clauses

2019-04-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27561: --- Description: Amazon Redshift has a feature called "lateral column alias references":

[jira] [Updated] (SPARK-27561) Support "lateral column alias references" to allow column aliases to be used within SELECT clauses

2019-04-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27561: --- Description: Amazon Redshift has a feature called "lateral column alias references:

[jira] [Created] (SPARK-27561) Support "lateral column alias references" to allow column aliases to be used within SELECT clauses

2019-04-24 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27561: -- Summary: Support "lateral column alias references" to allow column aliases to be used within SELECT clauses Key: SPARK-27561 URL: https://issues.apache.org/jira/browse/SPARK-27561

[jira] [Commented] (SPARK-27542) SparkHadoopWriter doesn't set call setWorkOutputPath, causing NPEs when using certain legacy OutputFormats

2019-04-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825560#comment-16825560 ] Josh Rosen commented on SPARK-27542: [~shivuson...@gmail.com], unfortunately I don't have a

[jira] [Created] (SPARK-27542) SparkHadoopWriter doesn't set call setWorkOutputPath, causing NPEs for some legacy OutputFormats

2019-04-22 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27542: -- Summary: SparkHadoopWriter doesn't set call setWorkOutputPath, causing NPEs for some legacy OutputFormats Key: SPARK-27542 URL: https://issues.apache.org/jira/browse/SPARK-27542

[jira] [Updated] (SPARK-27542) SparkHadoopWriter doesn't set call setWorkOutputPath, causing NPEs when using certain legacy OutputFormats

2019-04-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27542: --- Summary: SparkHadoopWriter doesn't set call setWorkOutputPath, causing NPEs when using certain

[jira] [Comment Edited] (SPARK-24107) ChunkedByteBuffer.writeFully method has not reset the limit value

2018-05-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475350#comment-16475350 ] Josh Rosen edited comment on SPARK-24107 at 5/15/18 4:25 PM: - To work around

[jira] [Commented] (SPARK-24107) ChunkedByteBuffer.writeFully method has not reset the limit value

2018-05-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475350#comment-16475350 ] Josh Rosen commented on SPARK-24107: To work around this bug on unpatched / unhotfixed Spark 2.3.x

[jira] [Updated] (SPARK-24107) ChunkedByteBuffer.writeFully method has not reset the limit value

2018-05-14 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-24107: --- Labels: correctness (was: ) Priority: Blocker (was: Major) This bug was originally

[jira] [Created] (SPARK-24160) ShuffleBlockFetcherIterator should fail if it receives zero-size blocks

2018-05-02 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-24160: -- Summary: ShuffleBlockFetcherIterator should fail if it receives zero-size blocks Key: SPARK-24160 URL: https://issues.apache.org/jira/browse/SPARK-24160 Project: Spark

[jira] [Updated] (SPARK-23274) ReplaceExceptWithFilter fails on dataframes filtered on same column

2018-01-30 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-23274: --- Labels: (was: correctness) > ReplaceExceptWithFilter fails on dataframes filtered on same column >

[jira] [Updated] (SPARK-23274) ReplaceExceptWithFilter fails on dataframes filtered on same column

2018-01-30 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-23274: --- Labels: correctness (was: ) > ReplaceExceptWithFilter fails on dataframes filtered on same column >

[jira] [Commented] (SPARK-22982) Remove unsafe asynchronous close() call from FileDownloadChannel

2018-01-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320664#comment-16320664 ] Josh Rosen commented on SPARK-22982: In theory this affects all 1.6.0+ versions. It's going to be

[jira] [Resolved] (SPARK-22997) Add additional defenses against use of freed MemoryBlocks

2018-01-10 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-22997. Resolution: Fixed Fix Version/s: 2.3.0 Fixed for 2.3.0. > Add additional defenses against

[jira] [Created] (SPARK-22997) Add additional defenses against use of freed MemoryBlocks

2018-01-08 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-22997: -- Summary: Add additional defenses against use of freed MemoryBlocks Key: SPARK-22997 URL: https://issues.apache.org/jira/browse/SPARK-22997 Project: Spark Issue

[jira] [Updated] (SPARK-22985) Fix argument escaping bug in from_utc_timestamp / to_utc_timestamp codegen

2018-01-08 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-22985: --- Priority: Blocker (was: Major) > Fix argument escaping bug in from_utc_timestamp / to_utc_timestamp

[jira] [Created] (SPARK-22985) Fix argument escaping bug in from_utc_timestamp / to_utc_timestamp codegen

2018-01-07 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-22985: -- Summary: Fix argument escaping bug in from_utc_timestamp / to_utc_timestamp codegen Key: SPARK-22985 URL: https://issues.apache.org/jira/browse/SPARK-22985 Project:

[jira] [Created] (SPARK-22984) Fix incorrect bitmap copying and offset shifting in GenerateUnsafeRowJoiner

2018-01-07 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-22984: -- Summary: Fix incorrect bitmap copying and offset shifting in GenerateUnsafeRowJoiner Key: SPARK-22984 URL: https://issues.apache.org/jira/browse/SPARK-22984 Project:

[jira] [Created] (SPARK-22983) Don't push filters beneath aggregates with empty grouping expressions

2018-01-07 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-22983: -- Summary: Don't push filters beneath aggregates with empty grouping expressions Key: SPARK-22983 URL: https://issues.apache.org/jira/browse/SPARK-22983 Project: Spark

[jira] [Created] (SPARK-22982) Remove unsafe asynchronous close() call from FileDownloadChannel

2018-01-07 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-22982: -- Summary: Remove unsafe asynchronous close() call from FileDownloadChannel Key: SPARK-22982 URL: https://issues.apache.org/jira/browse/SPARK-22982 Project: Spark

[jira] [Commented] (SPARK-14643) Remove overloaded methods which become ambiguous in Scala 2.12

2017-07-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097169#comment-16097169 ] Josh Rosen commented on SPARK-14643: [~srowen], I just posted a comment about this over at

[jira] [Resolved] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-21444. Resolution: Fixed Fix Version/s: 2.3.0 > Fetch failure due to node reboot causes job

[jira] [Assigned] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-21444: -- Assignee: Josh Rosen > Fetch failure due to node reboot causes job failure >

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090815#comment-16090815 ] Josh Rosen commented on SPARK-21444: I spot the problem: in the old code, we removed the broadcast

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090809#comment-16090809 ] Josh Rosen commented on SPARK-21444: I'm going to adjust the "affects versions" on this because it

[jira] [Updated] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-21444: --- Affects Version/s: (was: 2.0.2) 2.3.0 > Fetch failure due to node reboot

[jira] [Assigned] (SPARK-14280) Update change-version.sh and pom.xml to add Scala 2.12 profiles

2017-07-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-14280: -- Assignee: Josh Rosen > Update change-version.sh and pom.xml to add Scala 2.12 profiles >

[jira] [Assigned] (SPARK-14280) Update change-version.sh and pom.xml to add Scala 2.12 profiles

2017-07-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-14280: -- Assignee: (was: Josh Rosen) > Update change-version.sh and pom.xml to add Scala 2.12

[jira] [Assigned] (SPARK-14650) Compile Spark REPL for Scala 2.12

2017-07-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-14650: -- Assignee: (was: Josh Rosen) > Compile Spark REPL for Scala 2.12 >

[jira] [Resolved] (SPARK-14438) Cross-publish Breeze for Scala 2.12

2017-07-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-14438. Resolution: Fixed > Cross-publish Breeze for Scala 2.12 > --- > >

[jira] [Assigned] (SPARK-14280) Update change-version.sh and pom.xml to add Scala 2.12 profiles

2017-07-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-14280: -- Assignee: (was: Josh Rosen) > Update change-version.sh and pom.xml to add Scala 2.12

[jira] [Resolved] (SPARK-14519) Cross-publish Kafka for Scala 2.12

2017-07-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-14519. Resolution: Fixed > Cross-publish Kafka for Scala 2.12 > -- > >

[jira] [Resolved] (SPARK-20715) MapStatuses shouldn't be redundantly stored in both ShuffleMapStage and MapOutputTracker

2017-06-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-20715. Resolution: Fixed Fix Version/s: 2.3.0 Fixed for 2.3.0. > MapStatuses shouldn't be

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

2017-06-08 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043622#comment-16043622 ] Josh Rosen commented on SPARK-20178: Update: I commented over on

[jira] [Updated] (SPARK-20945) NoSuchElementException key not found in TaskSchedulerImpl

2017-05-31 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-20945: --- Labels: (was: scheduler) > NoSuchElementException key not found in TaskSchedulerImpl >

[jira] [Updated] (SPARK-20945) NoSuchElementException key not found in TaskSchedulerImpl

2017-05-31 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-20945: --- Component/s: (was: Spark Core) Scheduler > NoSuchElementException key not found

[jira] [Commented] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory

2017-05-30 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029962#comment-16029962 ] Josh Rosen commented on SPARK-20923: It doesn't seem to be used, as far as I can tell from a quick

[jira] [Updated] (SPARK-20916) Improve error message for unaliased subqueries in FROM clause

2017-05-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-20916: --- Priority: Major (was: Blocker) > Improve error message for unaliased subqueries in FROM clause >

[jira] [Updated] (SPARK-20916) Improve error message for unaliased subqueries in FROM clause

2017-05-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-20916: --- Description: The following query parses in branch-2.2, but doesn't parse correctly as of today's

[jira] [Updated] (SPARK-20916) Improve error message for unaliased subqueries in FROM clause

2017-05-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-20916: --- Summary: Improve error message for unaliased subqueries in FROM clause (was: Regression in parsing

[jira] [Commented] (SPARK-20916) Regression in parsing of anonymous subqueries in FROM clause

2017-05-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028685#comment-16028685 ] Josh Rosen commented on SPARK-20916: It looks like this was caused by SPARK-20690 which changed the

[jira] [Updated] (SPARK-20916) Regression in parsing of anonymous subqueries in FROM clause

2017-05-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-20916: --- Description: The following query parses in branch-2.2, but doesn't parse correctly as of today's

[jira] [Updated] (SPARK-20916) Regression in parsing of anonymous subqueries in FROM clause

2017-05-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-20916: --- Summary: Regression in parsing of anonymous subqueries in FROM clause (was: Regression in parsing

[jira] [Created] (SPARK-20916) Regression in parsing of anonymous subqueries in SELECT clause

2017-05-29 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-20916: -- Summary: Regression in parsing of anonymous subqueries in SELECT clause Key: SPARK-20916 URL: https://issues.apache.org/jira/browse/SPARK-20916 Project: Spark

<    1   2   3   4   5   6   7   8   9   10   >