Compression Streamingfilesink ROW-encoded format
Hi Team,
Is there a way to enable compression in StreamingFileSink API for
Row-encoded formats ?.
val sink: StreamingFileSink[String] = StreamingFileSink
.forRowFormat(new Path(outputPath), new
SimpleStringEncoder[String]("UTF-8"))
Regards,
Vijay
Re: Compression Streamingfilesink ROW-encoded format
Hi Vijayendra,
You could achieve row encoded with like this as well
codecName = "org.apache.hadoop.io.compress.GzipCodec"
val streamingFileSink:StreamingFileSink[String] =
StreamingFileSink.forBulkFormat(new
Path(outputPath),CompressWriters.forExtractor(new
DefaultExtractor[String]).withHadoopCompression(codecName)).build()
Regards,
Ravi
On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav
wrote:
> Hi Team,
>
> Is there a way to enable compression in StreamingFileSink API for
> Row-encoded formats ?.
>
> val sink: StreamingFileSink[String] = StreamingFileSink
> .forRowFormat(new Path(outputPath), new
> SimpleStringEncoder[String]("UTF-8"))
>
>
> Regards,
> Vijay
>
Re: Compression Streamingfilesink ROW-encoded format
Hi Ravi,
Thanks for your response. But your example is for *forBulkForma**t*. How
about *forRowFormat* ?.
Regards,
Vijay
On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <
ravibhushanratna...@gmail.com> wrote:
> Hi Vijayendra,
>
> You could achieve row encoded with like this as well
>
> codecName = "org.apache.hadoop.io.compress.GzipCodec"
>
> val streamingFileSink:StreamingFileSink[String] =
> StreamingFileSink.forBulkFormat(new
> Path(outputPath),CompressWriters.forExtractor(new
> DefaultExtractor[String]).withHadoopCompression(codecName)).build()
>
> Regards,
> Ravi
>
> On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav
> wrote:
>
>> Hi Team,
>>
>> Is there a way to enable compression in StreamingFileSink API for
>> Row-encoded formats ?.
>>
>> val sink: StreamingFileSink[String] = StreamingFileSink
>> .forRowFormat(new Path(outputPath), new
>> SimpleStringEncoder[String]("UTF-8"))
>>
>>
>> Regards,
>> Vijay
>>
>
Re: Compression Streamingfilesink ROW-encoded format
Hi Vijayendra,
As far as rowFormat is concerned, it doesn't support compression.
Regards,
Ravi
On Tue 28 Jul, 2020, 22:08 Vijayendra Yadav, wrote:
> Hi Ravi,
>
> Thanks for your response. But your example is for *forBulkForma**t*. How
> about *forRowFormat* ?.
>
> Regards,
> Vijay
>
> On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <
> ravibhushanratna...@gmail.com> wrote:
>
>> Hi Vijayendra,
>>
>> You could achieve row encoded with like this as well
>>
>> codecName = "org.apache.hadoop.io.compress.GzipCodec"
>>
>> val streamingFileSink:StreamingFileSink[String] =
>> StreamingFileSink.forBulkFormat(new
>> Path(outputPath),CompressWriters.forExtractor(new
>> DefaultExtractor[String]).withHadoopCompression(codecName)).build()
>>
>> Regards,
>> Ravi
>>
>> On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav
>> wrote:
>>
>>> Hi Team,
>>>
>>> Is there a way to enable compression in StreamingFileSink API for
>>> Row-encoded formats ?.
>>>
>>> val sink: StreamingFileSink[String] = StreamingFileSink
>>> .forRowFormat(new Path(outputPath), new
>>> SimpleStringEncoder[String]("UTF-8"))
>>>
>>>
>>> Regards,
>>> Vijay
>>>
>>
Re: Compression Streamingfilesink ROW-encoded format
Thank You Ravi for Quick help. One Last Question is this compression
supported with Flink Version 1.10.0 ?
Regards,
Vijay
On Tue, Jul 28, 2020 at 1:20 PM Ravi Bhushan Ratnakar <
ravibhushanratna...@gmail.com> wrote:
> Hi Vijayendra,
>
> As far as rowFormat is concerned, it doesn't support compression.
>
>
> Regards,
> Ravi
>
> On Tue 28 Jul, 2020, 22:08 Vijayendra Yadav,
> wrote:
>
>> Hi Ravi,
>>
>> Thanks for your response. But your example is for *forBulkForma**t*. How
>> about *forRowFormat* ?.
>>
>> Regards,
>> Vijay
>>
>> On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <
>> ravibhushanratna...@gmail.com> wrote:
>>
>>> Hi Vijayendra,
>>>
>>> You could achieve row encoded with like this as well
>>>
>>> codecName = "org.apache.hadoop.io.compress.GzipCodec"
>>>
>>> val streamingFileSink:StreamingFileSink[String] =
>>> StreamingFileSink.forBulkFormat(new
>>> Path(outputPath),CompressWriters.forExtractor(new
>>> DefaultExtractor[String]).withHadoopCompression(codecName)).build()
>>>
>>> Regards,
>>> Ravi
>>>
>>> On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav
>>> wrote:
>>>
Hi Team,
Is there a way to enable compression in StreamingFileSink API for
Row-encoded formats ?.
val sink: StreamingFileSink[String] = StreamingFileSink
.forRowFormat(new Path(outputPath), new
SimpleStringEncoder[String]("UTF-8"))
Regards,
Vijay
>>>
Re: Compression Streamingfilesink ROW-encoded format
Yes, flink-compress module is supported from 1.10.0 and onward.
Regards,
Ravi
On Tue 28 Jul, 2020, 23:11 Vijayendra Yadav, wrote:
> Thank You Ravi for Quick help. One Last Question is this compression
> supported with Flink Version 1.10.0 ?
>
> Regards,
> Vijay
>
> On Tue, Jul 28, 2020 at 1:20 PM Ravi Bhushan Ratnakar <
> ravibhushanratna...@gmail.com> wrote:
>
>> Hi Vijayendra,
>>
>> As far as rowFormat is concerned, it doesn't support compression.
>>
>>
>> Regards,
>> Ravi
>>
>> On Tue 28 Jul, 2020, 22:08 Vijayendra Yadav,
>> wrote:
>>
>>> Hi Ravi,
>>>
>>> Thanks for your response. But your example is for *forBulkForma**t*.
>>> How about *forRowFormat* ?.
>>>
>>> Regards,
>>> Vijay
>>>
>>> On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <
>>> ravibhushanratna...@gmail.com> wrote:
>>>
Hi Vijayendra,
You could achieve row encoded with like this as well
codecName = "org.apache.hadoop.io.compress.GzipCodec"
val streamingFileSink:StreamingFileSink[String] =
StreamingFileSink.forBulkFormat(new
Path(outputPath),CompressWriters.forExtractor(new
DefaultExtractor[String]).withHadoopCompression(codecName)).build()
Regards,
Ravi
On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav
wrote:
> Hi Team,
>
> Is there a way to enable compression in StreamingFileSink API for
> Row-encoded formats ?.
>
> val sink: StreamingFileSink[String] = StreamingFileSink
> .forRowFormat(new Path(outputPath), new
> SimpleStringEncoder[String]("UTF-8"))
>
>
> Regards,
> Vijay
>
Re: Compression Streamingfilesink ROW-encoded format
Thank You.
On Wed, Jul 29, 2020 at 2:07 AM Ravi Bhushan Ratnakar <
ravibhushanratna...@gmail.com> wrote:
> Yes, flink-compress module is supported from 1.10.0 and onward.
>
> Regards,
> Ravi
>
> On Tue 28 Jul, 2020, 23:11 Vijayendra Yadav,
> wrote:
>
>> Thank You Ravi for Quick help. One Last Question is this compression
>> supported with Flink Version 1.10.0 ?
>>
>> Regards,
>> Vijay
>>
>> On Tue, Jul 28, 2020 at 1:20 PM Ravi Bhushan Ratnakar <
>> ravibhushanratna...@gmail.com> wrote:
>>
>>> Hi Vijayendra,
>>>
>>> As far as rowFormat is concerned, it doesn't support compression.
>>>
>>>
>>> Regards,
>>> Ravi
>>>
>>> On Tue 28 Jul, 2020, 22:08 Vijayendra Yadav,
>>> wrote:
>>>
Hi Ravi,
Thanks for your response. But your example is for *forBulkForma**t*.
How about *forRowFormat* ?.
Regards,
Vijay
On Tue, Jul 28, 2020 at 11:28 AM Ravi Bhushan Ratnakar <
ravibhushanratna...@gmail.com> wrote:
> Hi Vijayendra,
>
> You could achieve row encoded with like this as well
>
> codecName = "org.apache.hadoop.io.compress.GzipCodec"
>
> val streamingFileSink:StreamingFileSink[String] =
> StreamingFileSink.forBulkFormat(new
> Path(outputPath),CompressWriters.forExtractor(new
> DefaultExtractor[String]).withHadoopCompression(codecName)).build()
>
> Regards,
> Ravi
>
> On Tue, Jul 28, 2020 at 8:03 PM Vijayendra Yadav <
> contact@gmail.com> wrote:
>
>> Hi Team,
>>
>> Is there a way to enable compression in StreamingFileSink API for
>> Row-encoded formats ?.
>>
>> val sink: StreamingFileSink[String] = StreamingFileSink
>> .forRowFormat(new Path(outputPath), new
>> SimpleStringEncoder[String]("UTF-8"))
>>
>>
>> Regards,
>> Vijay
>>
>
