[jira] [Commented] (FLINK-12149) Support Proto for Streaming File Sink
[ https://issues.apache.org/jira/browse/FLINK-12149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17333962#comment-17333962 ] Flink Jira Bot commented on FLINK-12149: This issue was marked "stale-assigned" and has not received an update in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Support Proto for Streaming File Sink > -- > > Key: FLINK-12149 > URL: https://issues.apache.org/jira/browse/FLINK-12149 > Project: Flink > Issue Type: New Feature > Components: Connectors / FileSystem >Reporter: Kailash Hassan Dayanand >Assignee: vinoyang >Priority: Major > Labels: stale-assigned > > Currently we are supporting AuroParquetWriters in flink > here:[https://ci.apache.org/projects/flink/flink-docs-release-1.7/api/java/org/apache/flink/formats/parquet/avro/ParquetAvroWriters.html]. > Supporting an protoParquetWriter within Flink will be a good addition as > well. > Currently Parquet already have support for ProtoParquetWriters here: > [https://github.com/apache/parquet-mr/blob/master/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoParquetWriter.java]. > We can extend the classes from here and make it available in the appropriate > format to support ProtoParquet writing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-12149) Support Proto for Streaming File Sink
[ https://issues.apache.org/jira/browse/FLINK-12149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17323985#comment-17323985 ] J Y commented on FLINK-12149: - does [https://github.com/apache/flink/blob/master/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/protobuf/ParquetProtoWriters.java] address proto parquet writing for flink? i tried backporting this to 1.11.1, and i am generating a parquet file that has the right schema, but command line parquet tools or aws athena (pinot) can't read any actual rows. looking at the binary of the file, it looks like it has the row data, but again, doesn't seem to be working properly. > Support Proto for Streaming File Sink > -- > > Key: FLINK-12149 > URL: https://issues.apache.org/jira/browse/FLINK-12149 > Project: Flink > Issue Type: New Feature > Components: Connectors / FileSystem >Reporter: Kailash Hassan Dayanand >Assignee: vinoyang >Priority: Major > Labels: stale-assigned > > Currently we are supporting AuroParquetWriters in flink > here:[https://ci.apache.org/projects/flink/flink-docs-release-1.7/api/java/org/apache/flink/formats/parquet/avro/ParquetAvroWriters.html]. > Supporting an protoParquetWriter within Flink will be a good addition as > well. > Currently Parquet already have support for ProtoParquetWriters here: > [https://github.com/apache/parquet-mr/blob/master/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoParquetWriter.java]. > We can extend the classes from here and make it available in the appropriate > format to support ProtoParquet writing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-12149) Support Proto for Streaming File Sink
[ https://issues.apache.org/jira/browse/FLINK-12149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17323313#comment-17323313 ] Flink Jira Bot commented on FLINK-12149: This issue is assigned but has not received an update in 7 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Support Proto for Streaming File Sink > -- > > Key: FLINK-12149 > URL: https://issues.apache.org/jira/browse/FLINK-12149 > Project: Flink > Issue Type: New Feature > Components: Connectors / FileSystem >Reporter: Kailash Hassan Dayanand >Assignee: vinoyang >Priority: Major > Labels: stale-assigned > > Currently we are supporting AuroParquetWriters in flink > here:[https://ci.apache.org/projects/flink/flink-docs-release-1.7/api/java/org/apache/flink/formats/parquet/avro/ParquetAvroWriters.html]. > Supporting an protoParquetWriter within Flink will be a good addition as > well. > Currently Parquet already have support for ProtoParquetWriters here: > [https://github.com/apache/parquet-mr/blob/master/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoParquetWriter.java]. > We can extend the classes from here and make it available in the appropriate > format to support ProtoParquet writing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-12149) Support Proto for Streaming File Sink
[ https://issues.apache.org/jira/browse/FLINK-12149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838775#comment-16838775 ] Kailash Hassan Dayanand commented on FLINK-12149: - [~yanghua] I have some work which I put into this here: [https://github.com/kailashhd/flink/commit/10fcbd41dbaa49959d61d3ab08c0ac29da554ab2]. I can share my clean PR and send it across for review in a day or so. > Support Proto for Streaming File Sink > -- > > Key: FLINK-12149 > URL: https://issues.apache.org/jira/browse/FLINK-12149 > Project: Flink > Issue Type: New Feature > Components: Connectors / FileSystem >Reporter: Kailash Hassan Dayanand >Assignee: vinoyang >Priority: Major > > Currently we are supporting AuroParquetWriters in flink > here:[https://ci.apache.org/projects/flink/flink-docs-release-1.7/api/java/org/apache/flink/formats/parquet/avro/ParquetAvroWriters.html]. > Supporting an protoParquetWriter within Flink will be a good addition as > well. > Currently Parquet already have support for ProtoParquetWriters here: > [https://github.com/apache/parquet-mr/blob/master/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoParquetWriter.java]. > We can extend the classes from here and make it available in the appropriate > format to support ProtoParquet writing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-12149) Support Proto for Streaming File Sink
[ https://issues.apache.org/jira/browse/FLINK-12149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838377#comment-16838377 ] vinoyang commented on FLINK-12149: -- [~StephanEwen] What do you think about this issue? > Support Proto for Streaming File Sink > -- > > Key: FLINK-12149 > URL: https://issues.apache.org/jira/browse/FLINK-12149 > Project: Flink > Issue Type: New Feature > Components: Connectors / FileSystem >Reporter: Kailash Hassan Dayanand >Assignee: vinoyang >Priority: Major > > Currently we are supporting AuroParquetWriters in flink > here:[https://ci.apache.org/projects/flink/flink-docs-release-1.7/api/java/org/apache/flink/formats/parquet/avro/ParquetAvroWriters.html]. > Supporting an protoParquetWriter within Flink will be a good addition as > well. > Currently Parquet already have support for ProtoParquetWriters here: > [https://github.com/apache/parquet-mr/blob/master/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoParquetWriter.java]. > We can extend the classes from here and make it available in the appropriate > format to support ProtoParquet writing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
