[jira] [Commented] (SPARK-23908) High-order function: transform(array, function) → array

[ https://issues.apache.org/jira/browse/SPARK-23908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549628#comment-16549628 ] Frederick Reiss commented on SPARK-23908: - Thanks Herman, looking forward to seeing this

[jira] [Commented] (SPARK-23908) High-order function: transform(array, function) → array

[ https://issues.apache.org/jira/browse/SPARK-23908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548421#comment-16548421 ] Frederick Reiss commented on SPARK-23908: - This Jira is marked as "in progress" with the target

[jira] [Commented] (SPARK-20964) Make some keywords reserved along with the ANSI/SQL standard

[ https://issues.apache.org/jira/browse/SPARK-20964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408578#comment-16408578 ] Frederick Reiss commented on SPARK-20964: - SQL2016 prohibits the use of reserved words in

[jira] [Updated] (SPARK-21084) Improvements to dynamic allocation for notebook use cases

[ https://issues.apache.org/jira/browse/SPARK-21084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-21084: Description: One important application of Spark is to support many notebook users with a

[jira] [Updated] (SPARK-21084) Improvements to dynamic allocation for notebook use cases

[ https://issues.apache.org/jira/browse/SPARK-21084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-21084: Description: One important application of Spark is to support many notebook users with a

[jira] [Commented] (SPARK-21084) Improvements to dynamic allocation for notebook use cases

[ https://issues.apache.org/jira/browse/SPARK-21084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049669#comment-16049669 ] Frederick Reiss commented on SPARK-21084: - [~sowen] thanks for having a look at this JIRA and

[jira] [Updated] (SPARK-21084) Improvements to dynamic allocation for notebook use cases

[ https://issues.apache.org/jira/browse/SPARK-21084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-21084: Description: One important application of Spark is to support many notebook users with a

[jira] [Created] (SPARK-21084) Improvements to dynamic allocation for notebook use cases

Frederick Reiss created SPARK-21084: --- Summary: Improvements to dynamic allocation for notebook use cases Key: SPARK-21084 URL: https://issues.apache.org/jira/browse/SPARK-21084 Project: Spark

[jira] [Commented] (SPARK-18127) Add hooks and extension points to Spark

[ https://issues.apache.org/jira/browse/SPARK-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983872#comment-15983872 ] Frederick Reiss commented on SPARK-18127: - Is there a design document or a public design and

[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas

[ https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15596756#comment-15596756 ] Frederick Reiss commented on SPARK-13534: - We ([~bryanc], [~holdenk], [~yinxusen], and myself)

[jira] [Commented] (SPARK-15777) Catalog federation

[ https://issues.apache.org/jira/browse/SPARK-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15542995#comment-15542995 ] Frederick Reiss commented on SPARK-15777: - Thanks for posting this design doc. Some comments from

[jira] [Commented] (SPARK-10815) API design: data sources and sinks

[ https://issues.apache.org/jira/browse/SPARK-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530313#comment-15530313 ] Frederick Reiss commented on SPARK-10815: - Thanks for the clarification. Do you have a design to

[jira] [Commented] (SPARK-17631) Structured Streaming - Do we need to output results through http API?

[ https://issues.apache.org/jira/browse/SPARK-17631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528403#comment-15528403 ] Frederick Reiss commented on SPARK-17631: - I can think of two use cases where one might want to

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508260#comment-15508260 ] Frederick Reiss commented on SPARK-16407: - It looks like there is a general agreement here that

[jira] [Commented] (SPARK-10815) API design: data sources and sinks

[ https://issues.apache.org/jira/browse/SPARK-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505102#comment-15505102 ] Frederick Reiss commented on SPARK-10815: - I'm confused by the current description of this task.

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494795#comment-15494795 ] Frederick Reiss commented on SPARK-16407: - With respect, I'm not seeing a whole lot of flux in

[jira] [Created] (SPARK-17543) Missing log4j config file for tests in common/network-shuffle

Frederick Reiss created SPARK-17543: --- Summary: Missing log4j config file for tests in common/network-shuffle Key: SPARK-17543 URL: https://issues.apache.org/jira/browse/SPARK-17543 Project: Spark

[jira] [Created] (SPARK-17542) Compiler warning in UnsafeInMemorySorter class

Frederick Reiss created SPARK-17542: --- Summary: Compiler warning in UnsafeInMemorySorter class Key: SPARK-17542 URL: https://issues.apache.org/jira/browse/SPARK-17542 Project: Spark Issue

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15487780#comment-15487780 ] Frederick Reiss commented on SPARK-15406: - +1 for taking the simple route in the short term. I'm

[jira] [Created] (SPARK-17513) StreamExecution should discard unneeded metadata

Frederick Reiss created SPARK-17513: --- Summary: StreamExecution should discard unneeded metadata Key: SPARK-17513 URL: https://issues.apache.org/jira/browse/SPARK-17513 Project: Spark Issue

[jira] [Updated] (SPARK-17475) HDFSMetadataLog should not leak CRC files

[ https://issues.apache.org/jira/browse/SPARK-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-17475: Description: When HDFSMetadataLog uses a log directory on a filesystem other than HDFS

[jira] [Commented] (SPARK-17421) Warnings about "MaxPermSize" parameter when building with Maven and Java 8

[ https://issues.apache.org/jira/browse/SPARK-17421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477864#comment-15477864 ] Frederick Reiss commented on SPARK-17421: - Committer feedback on first PR was that the necessary

[jira] [Created] (SPARK-17475) HDFSMetadataLog should not leak CRC files

Frederick Reiss created SPARK-17475: --- Summary: HDFSMetadataLog should not leak CRC files Key: SPARK-17475 URL: https://issues.apache.org/jira/browse/SPARK-17475 Project: Spark Issue Type:

[jira] [Updated] (SPARK-17386) Default polling and trigger intervals cause excessive RPC calls

[ https://issues.apache.org/jira/browse/SPARK-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-17386: Description: The default trigger interval for a Structured Streaming query is

[jira] [Updated] (SPARK-17386) Default polling and trigger intervals cause excessive RPC calls

[ https://issues.apache.org/jira/browse/SPARK-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-17386: Description: The default trigger interval for a Structured Streaming query is

[jira] [Updated] (SPARK-17386) Default polling and trigger intervals cause excessive RPC calls

[ https://issues.apache.org/jira/browse/SPARK-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-17386: Description: The default trigger interval for a Structured Streaming query is

[jira] [Updated] (SPARK-17386) Default polling and trigger intervals cause excessive RPC calls

[ https://issues.apache.org/jira/browse/SPARK-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-17386: Description: The default trigger interval for a Structured Streaming query is

[jira] [Updated] (SPARK-17386) Default polling and trigger intervals cause excessive RPC calls

[ https://issues.apache.org/jira/browse/SPARK-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-17386: Priority: Minor (was: Major) > Default polling and trigger intervals cause excessive RPC

[jira] [Updated] (SPARK-17386) Default polling and trigger intervals cause excessive RPC calls

[ https://issues.apache.org/jira/browse/SPARK-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-17386: Summary: Default polling and trigger intervals cause excessive RPC calls (was: Default

[jira] [Updated] (SPARK-17386) Default trigger interval causes excessive RPC calls

[ https://issues.apache.org/jira/browse/SPARK-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-17386: Description: The default trigger interval for a Structured Streaming query is

[jira] [Created] (SPARK-17421) Warnings about "MaxPermSize" parameter when building with Maven and Java 8

Frederick Reiss created SPARK-17421: --- Summary: Warnings about "MaxPermSize" parameter when building with Maven and Java 8 Key: SPARK-17421 URL: https://issues.apache.org/jira/browse/SPARK-17421

[jira] [Updated] (SPARK-17386) Default trigger interval causes excessive RPC calls

[ https://issues.apache.org/jira/browse/SPARK-17386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-17386: Description: The default trigger interval for a Structured Streaming query is

[jira] [Created] (SPARK-17386) Default trigger interval causes excessive RPC calls

Frederick Reiss created SPARK-17386: --- Summary: Default trigger interval causes excessive RPC calls Key: SPARK-17386 URL: https://issues.apache.org/jira/browse/SPARK-17386 Project: Spark

[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas

[ https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456990#comment-15456990 ] Frederick Reiss commented on SPARK-13534: - [~wesmckinn], are you planning to work on this issue

[jira] [Closed] (SPARK-17343) Prerequisites for Kafka 0.8 support in Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-17343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss closed SPARK-17343. --- Resolution: Won't Fix There will be no Kafka 0.8 connectors for Structured Streaming. See

[jira] [Closed] (SPARK-17344) Kafka 0.8 support for Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss closed SPARK-17344. --- Resolution: Won't Fix There will be no Kafka 0.8 connectors for Structured Streaming. See

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454320#comment-15454320 ] Frederick Reiss commented on SPARK-15406: - Thanks for clearing that up. Marking SPARK-17343 and

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453749#comment-15453749 ] Frederick Reiss commented on SPARK-15406: - WRT Kafka 0.8: I'm under the impression that there is

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453059#comment-15453059 ] Frederick Reiss commented on SPARK-15406: - I've taken the liberty of splitting this issue into

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453047#comment-15453047 ] Frederick Reiss commented on SPARK-15406: - There has been some discussion of this JIRA on the

[jira] [Created] (SPARK-17346) Kafka 0.10 support in Structured Streaming

Frederick Reiss created SPARK-17346: --- Summary: Kafka 0.10 support in Structured Streaming Key: SPARK-17346 URL: https://issues.apache.org/jira/browse/SPARK-17346 Project: Spark Issue Type:

[jira] [Created] (SPARK-17345) Prerequisites for Kafka 0.10 support in Structured Streaming

Frederick Reiss created SPARK-17345: --- Summary: Prerequisites for Kafka 0.10 support in Structured Streaming Key: SPARK-17345 URL: https://issues.apache.org/jira/browse/SPARK-17345 Project: Spark

[jira] [Created] (SPARK-17344) Kafka 0.8 support for Structured Streaming

Frederick Reiss created SPARK-17344: --- Summary: Kafka 0.8 support for Structured Streaming Key: SPARK-17344 URL: https://issues.apache.org/jira/browse/SPARK-17344 Project: Spark Issue Type:

[jira] [Created] (SPARK-17343) Prerequisites for Kafka 0.8 support in Structured Streaming

Frederick Reiss created SPARK-17343: --- Summary: Prerequisites for Kafka 0.8 support in Structured Streaming Key: SPARK-17343 URL: https://issues.apache.org/jira/browse/SPARK-17343 Project: Spark

[jira] [Commented] (SPARK-17267) Long running structured streaming requirements

[ https://issues.apache.org/jira/browse/SPARK-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452993#comment-15452993 ] Frederick Reiss commented on SPARK-17267: - Converted SPARK-16963 to a subtask of this JIRA. >

[jira] [Updated] (SPARK-16963) Change Source API so that sources do not need to keep unbounded state

[ https://issues.apache.org/jira/browse/SPARK-16963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frederick Reiss updated SPARK-16963: Issue Type: Sub-task (was: Improvement) Parent: SPARK-17267 > Change Source API

[jira] [Created] (SPARK-17303) dev/run-tests fails if spark-warehouse directory exists

Frederick Reiss created SPARK-17303: --- Summary: dev/run-tests fails if spark-warehouse directory exists Key: SPARK-17303 URL: https://issues.apache.org/jira/browse/SPARK-17303 Project: Spark

[jira] [Commented] (SPARK-16963) Change Source API so that sources do not need to keep unbounded state

[ https://issues.apache.org/jira/browse/SPARK-16963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440598#comment-15440598 ] Frederick Reiss commented on SPARK-16963: - Updated the pull request to address some conflicting

[jira] [Commented] (SPARK-17165) FileStreamSource should not track the list of seen files indefinitely

[ https://issues.apache.org/jira/browse/SPARK-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439391#comment-15439391 ] Frederick Reiss commented on SPARK-17165: - This problem is actually deeper than just

[jira] [Commented] (SPARK-16963) Change Source API so that sources do not need to keep unbounded state

[ https://issues.apache.org/jira/browse/SPARK-16963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431779#comment-15431779 ] Frederick Reiss commented on SPARK-16963: - The proposed changes in the attached PR are now ready

[jira] [Created] (SPARK-16963) Change Source API so that sources do not need to keep unbounded state

Frederick Reiss created SPARK-16963: --- Summary: Change Source API so that sources do not need to keep unbounded state Key: SPARK-16963 URL: https://issues.apache.org/jira/browse/SPARK-16963 Project:

[jira] [Commented] (SPARK-15370) Some correlated subqueries return incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293496#comment-15293496 ] Frederick Reiss commented on SPARK-15370: - [~hvanhovell]'s comments on the first PR pointed out

[jira] [Commented] (SPARK-15372) TPC-DS Qury 84 returns wrong results against TPC official

[ https://issues.apache.org/jira/browse/SPARK-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288004#comment-15288004 ] Frederick Reiss commented on SPARK-15372: - There is no CONCAT function in the SQL standard. The

[jira] [Created] (SPARK-15370) Some correlated subqueries return incorrect answers

Frederick Reiss created SPARK-15370: --- Summary: Some correlated subqueries return incorrect answers Key: SPARK-15370 URL: https://issues.apache.org/jira/browse/SPARK-15370 Project: Spark

[jira] [Commented] (SPARK-15122) TPC-DS Qury 41 fails with The correlated scalar subquery can only contain equality predicates

[ https://issues.apache.org/jira/browse/SPARK-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271874#comment-15271874 ] Frederick Reiss commented on SPARK-15122: - In the official version of the query, the expression

[jira] [Commented] (SPARK-14781) Support subquery in nested predicates

[ https://issues.apache.org/jira/browse/SPARK-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265000#comment-15265000 ] Frederick Reiss commented on SPARK-14781: - Yeah, Distinct will impact performance for the

[jira] [Commented] (SPARK-14781) Support subquery in nested predicates

[ https://issues.apache.org/jira/browse/SPARK-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264789#comment-15264789 ] Frederick Reiss commented on SPARK-14781: - [~davies]: I think I have a minimally-invasive plan

[jira] [Commented] (SPARK-14781) Support subquery in nested predicates

[ https://issues.apache.org/jira/browse/SPARK-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261474#comment-15261474 ] Frederick Reiss commented on SPARK-14781: - Sure, I'd be happy to put something together to cover

[jira] [Commented] (SPARK-14785) Support correlated scalar subquery

[ https://issues.apache.org/jira/browse/SPARK-14785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261157#comment-15261157 ] Frederick Reiss commented on SPARK-14785: - Note that the rewritten query in the example above

[jira] [Commented] (SPARK-14781) Support subquery in nested predicates

[ https://issues.apache.org/jira/browse/SPARK-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261032#comment-15261032 ] Frederick Reiss commented on SPARK-14781: - [~davies] where is the definition of the SemiPlus

[jira] [Commented] (SPARK-14781) Support subquery in nested predicates

[ https://issues.apache.org/jira/browse/SPARK-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261028#comment-15261028 ] Frederick Reiss commented on SPARK-14781: - I'm not so sure about Q45. Here's the template for

[jira] [Commented] (SPARK-10936) UDAF "mode" for categorical variables

[ https://issues.apache.org/jira/browse/SPARK-10936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949306#comment-14949306 ] Frederick Reiss commented on SPARK-10936: - Mode is not an algebraic aggregate. To find the mode

[jira] [Commented] (SPARK-6649) DataFrame created through SQLContext.jdbc() failed if columns table must be quoted

[ https://issues.apache.org/jira/browse/SPARK-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14540967#comment-14540967 ] Frederick Reiss commented on SPARK-6649: Did some more digging through the code

[jira] [Commented] (SPARK-6649) DataFrame created through SQLContext.jdbc() failed if columns table must be quoted

[ https://issues.apache.org/jira/browse/SPARK-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527738#comment-14527738 ] Frederick Reiss commented on SPARK-6649: I was able to reproduce this problem on

[jira] [Commented] (SPARK-7043) KryoSerializer cannot be used with REPL to interpret code in which case class definition and its shipping are in the same line

[ https://issues.apache.org/jira/browse/SPARK-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527683#comment-14527683 ] Frederick Reiss commented on SPARK-7043: Both issues appear to have the same root

[jira] [Commented] (SPARK-7043) KryoSerializer cannot be used with REPL to interpret code in which case class definition and its shipping are in the same line

[ https://issues.apache.org/jira/browse/SPARK-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527575#comment-14527575 ] Frederick Reiss commented on SPARK-7043: I did some looking into this issue.

[jira] [Commented] (SPARK-7273) The SQLContext.jsonFile() api has a problem when load a format json file?

[ https://issues.apache.org/jira/browse/SPARK-7273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523384#comment-14523384 ] Frederick Reiss commented on SPARK-7273: The error in the description indicates