[jira] [Updated] (SPARK-17815) Report committed offsets

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17815: - Component/s: (was: SQL) Structured Streaming > Report committed

[jira] [Updated] (SPARK-17812) More granular control of starting offsets (assign)

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17812: - Component/s: (was: SQL) Structured Streaming > More granular

[jira] [Updated] (SPARK-17813) Maximum data per trigger

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17813: - Component/s: (was: SQL) Structured Streaming > Maximum data per

[jira] [Updated] (SPARK-17344) Kafka 0.8 support for Structured Streaming

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17344: - Component/s: (was: DStreams) Structured Streaming > Kafka 0.8

[jira] [Closed] (SPARK-17345) Prerequisites for Kafka 0.10 support in Structured Streaming

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust closed SPARK-17345. Resolution: Fixed > Prerequisites for Kafka 0.10 support in Structured Streaming >

[jira] [Updated] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-15406: - Component/s: Structured Streaming > Structured streaming support for consuming from

[jira] [Resolved] (SPARK-18025) Port streaming to use the commit protocol API

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-18025. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15710

[jira] [Updated] (SPARK-18025) Port streaming to use the commit protocol API

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-18025: - Component/s: Structured Streaming > Port streaming to use the commit protocol API >

[jira] [Resolved] (SPARK-16411) Add textFile API to structured streaming.

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-16411. -- Resolution: Fixed Assignee: Prashant Sharma Fix Version/s: 2.1.0 > Add

[jira] [Commented] (SPARK-16738) Queryable state for Spark State Store

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627148#comment-15627148 ] Michael Armbrust commented on SPARK-16738: -- You can already query the state store today, if you

[jira] [Updated] (SPARK-18187) CompactibleFileStreamLog should not rely on "compactInterval" to detect a compaction batch

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-18187: - Priority: Critical (was: Major) > CompactibleFileStreamLog should not rely on

[jira] [Commented] (SPARK-18187) CompactibleFileStreamLog should not rely on "compactInterval" to detect a compaction batch

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627134#comment-15627134 ] Michael Armbrust commented on SPARK-18187: -- I think the configuration should only be used when

[jira] [Commented] (SPARK-16454) Consider adding a per-batch transform for structured streaming

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627125#comment-15627125 ] Michael Armbrust commented on SPARK-16454: -- What specifically is missing from the {{foreach}}

[jira] [Updated] (SPARK-17879) Don't compact metadata logs constantly into a single compacted file

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17879: - Target Version/s: 2.1.0 > Don't compact metadata logs constantly into a single compacted

[jira] [Commented] (SPARK-17879) Don't compact metadata logs constantly into a single compacted file

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627116#comment-15627116 ] Michael Armbrust commented on SPARK-17879: -- What is the status here? From my perspective I

[jira] [Updated] (SPARK-17475) HDFSMetadataLog should not leak CRC files

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17475: - Affects Version/s: 2.0.1 Target Version/s: 2.1.0 > HDFSMetadataLog should not leak

[jira] [Updated] (SPARK-17475) HDFSMetadataLog should not leak CRC files

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17475: - Assignee: Frederick Reiss > HDFSMetadataLog should not leak CRC files >

[jira] [Updated] (SPARK-18124) Observed delay based event time watermarks

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-18124: - Target Version/s: 2.1.0 > Observed delay based event time watermarks >

[jira] [Updated] (SPARK-18124) Observed delay based event time watermarks

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-18124: - Summary: Observed delay based event time watermarks (was: Implement watermarking for

[jira] [Resolved] (SPARK-8360) Structured Streaming (aka Streaming DataFrames)

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-8360. - Resolution: Implemented Assignee: Michael Armbrust Fix Version/s: 2.1.0

[jira] [Closed] (SPARK-10823) API design: external state management

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust closed SPARK-10823. Resolution: Later We have a state store implemented today. I think we should revisit

[jira] [Updated] (SPARK-10815) Public API: Streaming Sources and Sinks

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-10815: - Priority: Critical (was: Major) Summary: Public API: Streaming Sources and Sinks

[jira] [Updated] (SPARK-10815) API design: data sources and sinks

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-10815: - Issue Type: New Feature (was: Sub-task) Parent: (was: SPARK-8360) > API

[jira] [Commented] (SPARK-10816) EventTime based sessionization

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627072#comment-15627072 ] Michael Armbrust commented on SPARK-10816: -- Windows were added in [SPARK-14160], so I'm going to

[jira] [Updated] (SPARK-10816) API design: window and session specification

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-10816: - Issue Type: New Feature (was: Sub-task) Parent: (was: SPARK-8360) > API

[jira] [Updated] (SPARK-10816) EventTime based sessionization

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-10816: - Target Version/s: 2.2.0 > EventTime based sessionization >

[jira] [Updated] (SPARK-10816) EventTime based sessionization

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-10816: - Summary: EventTime based sessionization (was: API design: window and session

[jira] [Updated] (SPARK-15472) Add support for writing partitioned `csv`, `json`, `text` formats in Structured Streaming

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-15472: - Issue Type: New Feature (was: Sub-task) Parent: (was: SPARK-8360) > Add

[jira] [Updated] (SPARK-17829) Stable format for offset log

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17829: - Issue Type: Improvement (was: Sub-task) Parent: (was: SPARK-8360) > Stable

[jira] [Updated] (SPARK-18124) Implement watermarking for handling late data

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-18124: - Issue Type: New Feature (was: Sub-task) Parent: (was: SPARK-8360) >

[jira] [Updated] (SPARK-18187) CompactibleFileStreamLog should not rely on "compactInterval" to detect a compaction batch

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-18187: - Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-8360) >

[jira] [Updated] (SPARK-18144) StreamingQueryListener.QueryStartedEvent is not written to event log

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-18144: - Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-8360) >

[jira] [Commented] (SPARK-18212) Flaky test: org.apache.spark.sql.kafka010.KafkaSourceSuite.assign from specific offsets

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627050#comment-15627050 ] Michael Armbrust commented on SPARK-18212: -- /cc [~c...@koeninger.org] > Flaky test:

[jira] [Updated] (SPARK-17267) Long running structured streaming requirements

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17267: - Component/s: (was: DStreams) (was: SQL)

[jira] [Updated] (SPARK-8360) Structured Streaming (aka Streaming DataFrames)

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-8360: Component/s: (was: DStreams) (was: SQL) Structured

[jira] [Updated] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17604: - Issue Type: New Feature (was: Sub-task) Parent: (was: SPARK-17267) >

[jira] [Commented] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626803#comment-15626803 ] Michael Armbrust commented on SPARK-17604: -- I think it would be good for us to support data

[jira] [Commented] (SPARK-17935) Add KafkaForeachWriter in external kafka-0.8.0 for structured streaming module

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626800#comment-15626800 ] Michael Armbrust commented on SPARK-17935: -- You should be able to use casts (which I'd expect

[jira] [Resolved] (SPARK-17764) to_json function for parsing Structs to json Strings

2016-11-01 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-17764. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15354

[jira] [Assigned] (SPARK-18124) Implement watermarking for handling late data

2016-10-28 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust reassigned SPARK-18124: Assignee: Michael Armbrust (was: Tathagata Das) > Implement watermarking for

[jira] [Updated] (SPARK-18147) Broken Spark SQL Codegen

2016-10-27 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-18147: - Target Version/s: 2.1.0 Priority: Critical (was: Minor) > Broken Spark SQL

[jira] [Commented] (SPARK-18147) Broken Spark SQL Codegen

2016-10-27 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15613555#comment-15613555 ] Michael Armbrust commented on SPARK-18147: -- /cc [~cloud_fan] > Broken Spark SQL Codegen >

[jira] [Resolved] (SPARK-17770) Make ObjectType SQL Type Public

2016-10-26 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-17770. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15453

[jira] [Commented] (SPARK-18084) write.partitionBy() does not recognize nested columns that select() can access

2016-10-26 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610097#comment-15610097 ] Michael Armbrust commented on SPARK-18084: -- I'd put this in the general class of issues where it

[jira] [Updated] (SPARK-18127) Add hooks and extension points to Spark

2016-10-26 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-18127: - Assignee: Herman van Hovell > Add hooks and extension points to Spark >

[jira] [Created] (SPARK-18122) Fallback to Kryo for unknown classes in ExpressionEncoder

2016-10-26 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-18122: Summary: Fallback to Kryo for unknown classes in ExpressionEncoder Key: SPARK-18122 URL: https://issues.apache.org/jira/browse/SPARK-18122 Project: Spark

[jira] [Commented] (SPARK-17829) Stable format for offset log

2016-10-21 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595716#comment-15595716 ] Michael Armbrust commented on SPARK-17829: -- Yeah, I agree. I think it could be really simple.

[jira] [Updated] (SPARK-17829) Stable format for offset log

2016-10-20 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17829: - Assignee: Tyson Condie > Stable format for offset log > > >

[jira] [Resolved] (SPARK-17900) Mark the following Spark SQL APIs as stable

2016-10-14 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-17900. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15469

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-14 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576840#comment-15576840 ] Michael Armbrust commented on SPARK-17812: -- That sounds pretty good to me, with one question:

[jira] [Commented] (SPARK-17813) Maximum data per trigger

2016-10-14 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576826#comment-15576826 ] Michael Armbrust commented on SPARK-17813: -- I think its okay to ignore compacted topics, at

[jira] [Resolved] (SPARK-11775) Allow PySpark to register Java UDF

2016-10-14 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-11775. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 9766

[jira] [Commented] (SPARK-12776) Implement Python API for Datasets

2016-10-14 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576739#comment-15576739 ] Michael Armbrust commented on SPARK-12776: -- I would love to see better support here, but I don't

[jira] [Resolved] (SPARK-16063) Add storageLevel to Dataset

2016-10-14 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-16063. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 13780

[jira] [Assigned] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust reassigned SPARK-17812: Assignee: Cody Koeninger > More granular control of starting offsets (assign) >

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573786#comment-15573786 ] Michael Armbrust commented on SPARK-17812: -- Please do work on it. It might be good to update

[jira] [Resolved] (SPARK-17368) Scala value classes create encoder problems and break at runtime

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-17368. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15284

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573677#comment-15573677 ] Michael Armbrust commented on SPARK-17812: -- bq. with your proposed interface, what, as a user,

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573526#comment-15573526 ] Michael Armbrust commented on SPARK-17812: -- As far as I understand it, {{auto.offset.reset}} is

[jira] [Comment Edited] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573459#comment-15573459 ] Michael Armbrust edited comment on SPARK-17812 at 10/13/16 10:53 PM: -

[jira] [Comment Edited] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573459#comment-15573459 ] Michael Armbrust edited comment on SPARK-17812 at 10/13/16 10:53 PM: -

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573459#comment-15573459 ] Michael Armbrust commented on SPARK-17812: -- +1 to the suggested was of subscribing, and for

[jira] [Commented] (SPARK-17344) Kafka 0.8 support for Structured Streaming

2016-10-11 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567321#comment-15567321 ] Michael Armbrust commented on SPARK-17344: -- These are good questions. A few thoughts: bq. How

[jira] [Updated] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-11 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17812: - Summary: More granular control of starting offsets (assign) (was: More granular control

[jira] [Commented] (SPARK-17812) More granular control of starting offsets

2016-10-11 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566392#comment-15566392 ] Michael Armbrust commented on SPARK-17812: -- For the seeking back {{X}} offsets use case, I was

[jira] [Updated] (SPARK-17812) More granular control of starting offsets

2016-10-11 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17812: - Description: Right now you can only run a Streaming Query starting from either the

[jira] [Commented] (SPARK-17344) Kafka 0.8 support for Structured Streaming

2016-10-11 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566191#comment-15566191 ] Michael Armbrust commented on SPARK-17344: -- I think the fact that CDH is still distributing 0.9

[jira] [Resolved] (SPARK-17830) Annotate Spark SQL public APIs with InterfaceStability

2016-10-10 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-17830. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15392

[jira] [Commented] (SPARK-8957) Backport Hive 1.X support to Branch 1.4

2016-10-07 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557139#comment-15557139 ] Michael Armbrust commented on SPARK-8957: - Yeah, close it. > Backport Hive 1.X support to Branch

[jira] [Reopened] (SPARK-17344) Kafka 0.8 support for Structured Streaming

2016-10-07 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust reopened SPARK-17344: -- I'm going to reopen for now since there is interests and you can't vote on closed issues.

[jira] [Created] (SPARK-17829) Stable format for offset log

2016-10-07 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-17829: Summary: Stable format for offset log Key: SPARK-17829 URL: https://issues.apache.org/jira/browse/SPARK-17829 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-10-06 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554150#comment-15554150 ] Michael Armbrust commented on SPARK-15406: -- The SQL offsets are stored in a WAL at

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-10-06 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554027#comment-15554027 ] Michael Armbrust commented on SPARK-15406: -- It would be helpful if you could enumerate problems

[jira] [Commented] (SPARK-17815) Report committed offsets

2016-10-06 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553900#comment-15553900 ] Michael Armbrust commented on SPARK-17815: -- The only reason its blocked is because setting it

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-10-06 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553852#comment-15553852 ] Michael Armbrust commented on SPARK-15406: -- Restarting a query is just running that query again,

[jira] [Created] (SPARK-17815) Report committed offsets

2016-10-06 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-17815: Summary: Report committed offsets Key: SPARK-17815 URL: https://issues.apache.org/jira/browse/SPARK-17815 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-10-06 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553405#comment-15553405 ] Michael Armbrust commented on SPARK-15406: -- The programming guide has a pretty good description

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-10-06 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553328#comment-15553328 ] Michael Armbrust commented on SPARK-15406: -- This missed 2.0.1, but I'm hopeful it will be in

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-10-06 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15553212#comment-15553212 ] Michael Armbrust commented on SPARK-15406: -- We are definitly not done! I added a few extra

[jira] [Created] (SPARK-17813) Maximum data per trigger

2016-10-06 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-17813: Summary: Maximum data per trigger Key: SPARK-17813 URL: https://issues.apache.org/jira/browse/SPARK-17813 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-17812) More granular control of starting offsets

2016-10-06 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-17812: Summary: More granular control of starting offsets Key: SPARK-17812 URL: https://issues.apache.org/jira/browse/SPARK-17812 Project: Spark Issue

[jira] [Resolved] (SPARK-17780) Report NoClassDefFoundError in StreamExecution

2016-10-06 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-17780. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.2 Issue resolved

[jira] [Commented] (SPARK-17344) Kafka 0.8 support for Structured Streaming

2016-10-06 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552915#comment-15552915 ] Michael Armbrust commented on SPARK-17344: -- BTW, people should still comment here if this is

[jira] [Commented] (SPARK-17764) to_json function for parsing Structs to json Strings

2016-10-03 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15543229#comment-15543229 ] Michael Armbrust commented on SPARK-17764: -- I would start by looking at the implementation of

[jira] [Updated] (SPARK-17764) to_json function for parsing Structs to json Strings

2016-10-03 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17764: - Target Version/s: 2.1.0 > to_json function for parsing Structs to json Strings >

[jira] [Created] (SPARK-17699) from_json function for parsing json Strings into Structs

2016-09-27 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-17699: Summary: from_json function for parsing json Strings into Structs Key: SPARK-17699 URL: https://issues.apache.org/jira/browse/SPARK-17699 Project: Spark

[jira] [Resolved] (SPARK-17153) [Structured streams] readStream ignores partition columns

2016-09-26 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-17153. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14803

[jira] [Created] (SPARK-17643) Remove comparable requirement from Offset

2016-09-22 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-17643: Summary: Remove comparable requirement from Offset Key: SPARK-17643 URL: https://issues.apache.org/jira/browse/SPARK-17643 Project: Spark Issue

[jira] [Assigned] (SPARK-17643) Remove comparable requirement from Offset

2016-09-22 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust reassigned SPARK-17643: Assignee: Michael Armbrust > Remove comparable requirement from Offset >

[jira] [Created] (SPARK-17627) Streaming Providers should be labeled Experimental

2016-09-21 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-17627: Summary: Streaming Providers should be labeled Experimental Key: SPARK-17627 URL: https://issues.apache.org/jira/browse/SPARK-17627 Project: Spark

[jira] [Updated] (SPARK-17627) Streaming Providers should be labeled Experimental

2016-09-21 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17627: - Component/s: SQL > Streaming Providers should be labeled Experimental >

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-21 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510787#comment-15510787 ] Michael Armbrust commented on SPARK-16407: -- I'm still a little unclear on the use cases we are

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-19 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504301#comment-15504301 ] Michael Armbrust commented on SPARK-16407: -- You are taking an experimental interface, which

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-19 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504303#comment-15504303 ] Michael Armbrust commented on SPARK-16407: -- You are taking an experimental interface, which

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494475#comment-15494475 ] Michael Armbrust commented on SPARK-16407: -- Sure, but the bar for compatibility is different for

[jira] [Comment Edited] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-15 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494475#comment-15494475 ] Michael Armbrust edited comment on SPARK-16407 at 9/15/16 8:42 PM: ---

[jira] [Commented] (SPARK-16407) Allow users to supply custom StreamSinkProviders

2016-09-14 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491819#comment-15491819 ] Michael Armbrust commented on SPARK-16407: -- I think it is likely that we will want to make it

[jira] [Updated] (SPARK-17445) Reference an ASF page as the main place to find third-party packages

2016-09-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17445: - Target Version/s: 2.0.1, 2.1.0 > Reference an ASF page as the main place to find

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-09-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488496#comment-15488496 ] Michael Armbrust commented on SPARK-15406: -- For the types that are coming out, the SQL way would

[jira] [Commented] (SPARK-15406) Structured streaming support for consuming from Kafka

2016-09-13 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488477#comment-15488477 ] Michael Armbrust commented on SPARK-15406: -- Streaming is labeled experimental, we can continue

<    1   2   3   4   5   6   7   8   9   10   >