[jira] [Created] (CARBONDATA-1072) Streaming Ingestion Feature
Aniket Adnaik created CARBONDATA-1072: - Summary: Streaming Ingestion Feature Key: CARBONDATA-1072 URL: https://issues.apache.org/jira/browse/CARBONDATA-1072 Project: CarbonData Issue Type: New Feature Components: core, data-load, data-query, examples, file-format, spark-integration, sql Affects Versions: 1.1.0 Reporter: Aniket Adnaik Fix For: 1.2.0 High level break down of work Items/Implementation phases: Design document will be attached soon. Phase – 1 – Spark Structured Streaming with regular Carbondata Format This phase will mainly focus on supporting Streaming ingestion using Spark Structured streaming 1. Write Path Implementation - Integration with Spark’s Structured Streaming framework (FileStreamSink etc) - StreamingOutputWriter (StreamingOuputWriterFactory) - Prepare Write (Schema Validation, Segment creation, Streaming file creation etc) - StreamingRecordWriter ( Data conversion from Catalyst InternalRow to Carbondata compatible format , make use of new load path) 2. Read Path Implementation (some overlap with phase-2) - Modify getsplits() to read from Streaming Segment - Read commited info from meta data to get correct offsets - Make use of Min-Max index if available - Use sequential scan - data is unsorted , cannot use Btree index 3. Compaction - Minor Compaction - Major Compaction 4. Metadata Management - Streaming metadata store (e.g. Offsets, timestamps etc.) 5. Failure Recovery - Rollback on failure - Handle asynchronous writes to CarbonData (using hflush) Phase – 2 : Spark Structured Streaming with Appendable CarbonData format 1.Streaming File Format - Writers use V3 file format for appending Columnar unsorted data blockets - Modify Readers to read from appendable streaming file format - Phase -3 : 1. Inter-opertability Support - Functionality with other features/Components - Concurrent queries with streaming ingestion - Concurrent operations with Streaming Ingestion (e.g. Compaction, Alter table, Secondary Index etc. 2. Kafka Connect Ingestion / Carbondata connector - Direct ingestion from Kafka Connect without Spark Structured Streaming - Separate Kafka Connector to receive data through network port - Data commit and Offset management - Phase-4 : Support for other streaming engines - Analysis of Streaming APIs/interface with other streaming engines - Implementation of connectors for different streaming engines storm, flink , flume, etc. - Phase -5 : In-memory Streaming table (probable feature) - 1. In-memory Cache for Streaming data - Fault tolerant in-memory buffering / checkpoint with WAL - Readers read from in-memory tables if available - Background threads for writing streaming data ,etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CARBONDATA-1072) Streaming Ingestion Feature
[ https://issues.apache.org/jira/browse/CARBONDATA-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aniket Adnaik updated CARBONDATA-1072: -- Request participants: (was: ) Description: High level break down of work Items/Implementation phases: Design document will be attached soon. Phase – 1 – Spark Structured Streaming with regular Carbondata Format This phase will mainly focus on supporting Streaming ingestion using Spark Structured streaming 1. Write Path Implementation - Integration with Spark’s Structured Streaming framework (FileStreamSink etc) - StreamingOutputWriter (StreamingOuputWriterFactory) - Prepare Write (Schema Validation, Segment creation, Streaming file creation etc) - StreamingRecordWriter ( Data conversion from Catalyst InternalRow to Carbondata compatible format , make use of new load path) 2. Read Path Implementation (some overlap with phase-2) - Modify getsplits() to read from Streaming Segment - Read commited info from meta data to get correct offsets - Make use of Min-Max index if available - Use sequential scan - data is unsorted , cannot use Btree index 3. Compaction - Minor Compaction - Major Compaction 4. Metadata Management - Streaming metadata store (e.g. Offsets, timestamps etc.) 5. Failure Recovery - Rollback on failure - Handle asynchronous writes to CarbonData (using hflush) - Phase – 2 : Spark Structured Streaming with Appendable CarbonData format 1.Streaming File Format - Writers use V3 file format for appending Columnar unsorted data blockets - Modify Readers to read from appendable streaming file format - Phase -3 : 1. Inter-opertability Support - Functionality with other features/Components - Concurrent queries with streaming ingestion - Concurrent operations with Streaming Ingestion (e.g. Compaction, Alter table, Secondary Index etc.) 2. Kafka Connect Ingestion / Carbondata connector - Direct ingestion from Kafka Connect without Spark Structured Streaming - Separate Kafka Connector to receive data through network port - Data commit and Offset management - Phase-4 : Support for other streaming engines - Analysis of Streaming APIs/interface with other streaming engines - Implementation of connectors for different streaming engines storm, flink , flume, etc. Phase -5 : In-memory Streaming table (probable feature) 1. In-memory Cache for Streaming data - Fault tolerant in-memory buffering / checkpoint with WAL - Readers read from in-memory tables if available - Background threads for writing streaming data ,etc. was: High level break down of work Items/Implementation phases: Design document will be attached soon. Phase – 1 – Spark Structured Streaming with regular Carbondata Format This phase will mainly focus on supporting Streaming ingestion using Spark Structured streaming 1. Write Path Implementation - Integration with Spark’s Structured Streaming framework (FileStreamSink etc) - StreamingOutputWriter (StreamingOuputWriterFactory) - Prepare Write (Schema Validation, Segment creation, Streaming file creation etc) - StreamingRecordWriter ( Data conversion from Catalyst InternalRow to Carbondata compatible format , make use of new load path) 2. Read Path Implementation (some overlap with phase-2) - Modify getsplits() to read from Streaming Segment - Read commited info from meta data to get correct offsets - Make use of Min-Max index if available - Use sequential scan - data is unsorted , cannot use Btree index 3. Compaction - Minor Compaction - Major Compaction 4. Metadata Management - Streaming metadata store (e.g. Offsets, timestamps etc.) 5. Failure Recovery - Rollback on failure - Handle asynchronous writes to CarbonData (using hflush) Phase – 2 : Spark Structured Streaming with Appendable CarbonData format 1.Streaming File Format - Writers use V3 file format for appending Columnar unsorted data blockets - Modify Readers to read from appendable streaming file format - Phase -3 : 1. Inter-opertability Support - Functionality with other features/Components - Concurrent queries with streaming ingestion - Concurrent operations with Streaming Ingestion (e.g. Compaction, Alter table, Secondary Index etc. 2. Kafka Connect Ingestion / Carbon
[jira] [Created] (CARBONDATA-1173) Streaming Ingest: Write path framework implementation
Aniket Adnaik created CARBONDATA-1173: - Summary: Streaming Ingest: Write path framework implementation Key: CARBONDATA-1173 URL: https://issues.apache.org/jira/browse/CARBONDATA-1173 Project: CarbonData Issue Type: Sub-task Components: core, data-load, hadoop-integration, spark-integration Affects Versions: 1.2.0 Reporter: Aniket Adnaik Assignee: Aniket Adnaik Fix For: 1.2.0 Carbondata with Spark Structured streaming write path framework - Carbondata StreamingOutputWriter, StreamingRecordWriter, metadata writer classes, etc - initial framework for streaming ingest feature -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1174) Streaming Ingest: Write path schema validation/inference
Aniket Adnaik created CARBONDATA-1174: - Summary: Streaming Ingest: Write path schema validation/inference Key: CARBONDATA-1174 URL: https://issues.apache.org/jira/browse/CARBONDATA-1174 Project: CarbonData Issue Type: Sub-task Components: core, spark-integration Affects Versions: 1.2.0 Reporter: Aniket Adnaik Fix For: 1.2.0 Streaming Ingest: Write path - schema validation / schema inference from existing carbondata table - streaming ingest allowed to existing tables only -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1175) Streaming Ingest: Write path data conversion/transformation
Aniket Adnaik created CARBONDATA-1175: - Summary: Streaming Ingest: Write path data conversion/transformation Key: CARBONDATA-1175 URL: https://issues.apache.org/jira/browse/CARBONDATA-1175 Project: CarbonData Issue Type: Sub-task Affects Versions: 1.2.0 Reporter: Aniket Adnaik Fix For: 1.2.0 Streaming Ingest: Write path Data conversion/Transformation - input data is a byte stream in catalyst InternalRow format (row major), which needs to be converted to column format - column converter and corresponding iterators needs to be created before invoking carbon layer load path - various carbon properties needs to be set, for example SORT_SCOPE (to skip sorting), blocket size, skip global dictionary creation etc -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1176) Streaming Ingest: Write path streaming segment/file creation
Aniket Adnaik created CARBONDATA-1176: - Summary: Streaming Ingest: Write path streaming segment/file creation Key: CARBONDATA-1176 URL: https://issues.apache.org/jira/browse/CARBONDATA-1176 Project: CarbonData Issue Type: Sub-task Components: core, data-load, hadoop-integration Affects Versions: 1.2.0 Reporter: Aniket Adnaik Fix For: 1.2.0 Streaming Ingest: Write path : segment /streaming file - Streaming segment creation and streaming file creation - resolve conflict with spark structured streaming file names. Spark structured streaming names streaming files with unique batch id to avoid overwriting - maintain spark structured streaming recover-ability as streaming file names generated by spark structured streaming are unique and recorded in spark structured streaming metadata. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1072) Streaming Ingestion Feature
[ https://issues.apache.org/jira/browse/CARBONDATA-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aniket Adnaik updated CARBONDATA-1072: -- Attachment: StreamingIngestionSupportInCarbonData.pdf > Streaming Ingestion Feature > > > Key: CARBONDATA-1072 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1072 > Project: CarbonData > Issue Type: New Feature > Components: core, data-load, data-query, examples, file-format, > spark-integration, sql >Affects Versions: NONE >Reporter: Aniket Adnaik > Fix For: NONE > > Attachments: StreamingIngestionSupportInCarbonData.pdf > > > High level break down of work Items/Implementation phases: > Design document will be attached soon. > > Phase – 1 – Spark Structured Streaming with regular Carbondata Format > > This phase will mainly focus on supporting Streaming ingestion using > Spark Structured streaming > 1.Write Path Implementation >- Integration with Spark’s Structured Streaming framework >(FileStreamSink etc) >- StreamingOutputWriter (StreamingOuputWriterFactory) >- Prepare Write (Schema Validation, Segment creation, > Streaming file creation etc) >- StreamingRecordWriter ( Data conversion from Catalyst InternalRow > to Carbondata compatible format , make use of new load path) > 2. Read Path Implementation (some overlap with phase-2) > - Modify getsplits() to read from Streaming Segment > - Read commited info from meta data to get correct offsets > - Make use of Min-Max index if available > - Use sequential scan - data is unsorted , cannot use Btree index > 3.Compaction > -Minor Compaction > -Major Compaction >4. Metadata Management > - Streaming metadata store (e.g. Offsets, timestamps etc.) > >5. Failure Recovery > - Rollback on failure > - Handle asynchronous writes to CarbonData (using hflush) > - > Phase – 2 : Spark Structured Streaming with Appendable CarbonData format > 1.Streaming File Format > - Writers use V3 file format for appending Columnar unsorted >data blockets > - Modify Readers to read from appendable streaming file format > - > Phase -3 : > 1. Inter-opertability Support > - Functionality with other features/Components > - Concurrent queries with streaming ingestion > - Concurrent operations with Streaming Ingestion (e.g. Compaction, > Alter table, Secondary Index etc.) > 2. Kafka Connect Ingestion / Carbondata connector > - Direct ingestion from Kafka Connect without Spark Structured > Streaming > - Separate Kafka Connector to receive data through network port > - Data commit and Offset management > - > Phase-4 : Support for other streaming engines > - Analysis of Streaming APIs/interface with other streaming engines > - Implementation of connectors for different streaming engines storm, >flink , flume, etc. > > Phase -5 : In-memory Streaming table (probable feature) >1. In-memory Cache for Streaming data > -Fault tolerant in-memory buffering / checkpoint with WAL > -Readers read from in-memory tables if available > -Background threads for writing streaming data ,etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-1174) Streaming Ingest: Write path schema validation/inference
[ https://issues.apache.org/jira/browse/CARBONDATA-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aniket Adnaik reassigned CARBONDATA-1174: - Assignee: Aniket Adnaik > Streaming Ingest: Write path schema validation/inference > > > Key: CARBONDATA-1174 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1174 > Project: CarbonData > Issue Type: Sub-task > Components: core, spark-integration >Affects Versions: 1.2.0 >Reporter: Aniket Adnaik >Assignee: Aniket Adnaik > Labels: features > Fix For: 1.2.0 > > > Streaming Ingest: Write path > - schema validation / schema inference from existing carbondata table > - streaming ingest allowed to existing tables only -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1173) Streaming Ingest: Write path framework implementation
[ https://issues.apache.org/jira/browse/CARBONDATA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aniket Adnaik updated CARBONDATA-1173: -- Fix Version/s: (was: NONE) 1.3.0 > Streaming Ingest: Write path framework implementation > - > > Key: CARBONDATA-1173 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1173 > Project: CarbonData > Issue Type: Sub-task > Components: core, data-load, hadoop-integration, spark-integration >Affects Versions: NONE >Reporter: Aniket Adnaik >Assignee: Aniket Adnaik > Fix For: 1.3.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Carbondata with Spark Structured streaming write path framework > - Carbondata StreamingOutputWriter, StreamingRecordWriter, metadata writer >classes, etc > - initial framework for streaming ingest feature -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1173) Streaming Ingest: Write path framework implementation
[ https://issues.apache.org/jira/browse/CARBONDATA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aniket Adnaik updated CARBONDATA-1173: -- Affects Version/s: 1.3.0 > Streaming Ingest: Write path framework implementation > - > > Key: CARBONDATA-1173 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1173 > Project: CarbonData > Issue Type: Sub-task > Components: core, data-load, hadoop-integration, spark-integration >Affects Versions: NONE >Reporter: Aniket Adnaik >Assignee: Aniket Adnaik > Fix For: 1.3.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Carbondata with Spark Structured streaming write path framework > - Carbondata StreamingOutputWriter, StreamingRecordWriter, metadata writer >classes, etc > - initial framework for streaming ingest feature -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1173) Streaming Ingest: Write path framework implementation
[ https://issues.apache.org/jira/browse/CARBONDATA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aniket Adnaik updated CARBONDATA-1173: -- Affects Version/s: (was: 1.3.0) NONE > Streaming Ingest: Write path framework implementation > - > > Key: CARBONDATA-1173 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1173 > Project: CarbonData > Issue Type: Sub-task > Components: core, data-load, hadoop-integration, spark-integration >Affects Versions: NONE >Reporter: Aniket Adnaik >Assignee: Aniket Adnaik > Fix For: 1.3.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Carbondata with Spark Structured streaming write path framework > - Carbondata StreamingOutputWriter, StreamingRecordWriter, metadata writer >classes, etc > - initial framework for streaming ingest feature -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1173) Streaming Ingest: Write path framework implementation
[ https://issues.apache.org/jira/browse/CARBONDATA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aniket Adnaik updated CARBONDATA-1173: -- Affects Version/s: (was: NONE) 1.2.0 > Streaming Ingest: Write path framework implementation > - > > Key: CARBONDATA-1173 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1173 > Project: CarbonData > Issue Type: Sub-task > Components: core, data-load, hadoop-integration, spark-integration >Affects Versions: 1.2.0 >Reporter: Aniket Adnaik >Assignee: Aniket Adnaik > Fix For: 1.3.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Carbondata with Spark Structured streaming write path framework > - Carbondata StreamingOutputWriter, StreamingRecordWriter, metadata writer >classes, etc > - initial framework for streaming ingest feature -- This message was sent by Atlassian JIRA (v6.4.14#64029)