Re: override file in flink

[Chesnay Schepler]

by using DataStream#writeAsCsv(String path, WriteMode writeMode)

On 22.03.2016 12:18, subash basnet wrote:

Hello all,

I am trying to write the streaming data to file and update it 
recurrently with the streaming data. I get the following unable to 
override exception error:


*Caused by: java.io.IOException: File or directory already exists. 
Existing files and directories are not overwritten in NO_OVERWRITE 
mode. Use OVERWRITE mode to overwrite existing files and directories.*
at 
org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:594)
at 
org.apache.flink.api.common.io.FileOutputFormat.open(FileOutputFormat.java:227)
at 
org.apache.flink.api.java.io.CsvOutputFormat.open(CsvOutputFormat.java:160)
at 
org.apache.flink.streaming.api.functions.sink.OutputFormatSinkFunction.open(OutputFormatSinkFunction.java:60)
at 
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)
at 
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:314)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:214)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
at java.lang.Thread.run(Thread.java:745)



How to set the necessary override property?

Best Regards,
Subash Basnet

Re: override file in flink

[Chiwan Park]

Hi subash,

You can pass WriteMode in second parameter of write* method. For example:

```
DataStream<…> myStream = …;
myStream.writeAsCsv(“path of output”, FileSystem.WriteMode.OVERWRITE);
```

I hope this helps.

Regards,
Chiwan Park

> On Mar 22, 2016, at 8:18 PM, subash basnet  wrote:
> 
> Hello all,
> 
> I am trying to write the streaming data to file and update it recurrently 
> with the streaming data. I get the following unable to override exception 
> error:
> 
> Caused by: java.io.IOException: File or directory already exists. Existing 
> files and directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE 
> mode to overwrite existing files and directories.
>   at 
> org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:594)
>   at 
> org.apache.flink.api.common.io.FileOutputFormat.open(FileOutputFormat.java:227)
>   at 
> org.apache.flink.api.java.io.CsvOutputFormat.open(CsvOutputFormat.java:160)
>   at 
> org.apache.flink.streaming.api.functions.sink.OutputFormatSinkFunction.open(OutputFormatSinkFunction.java:60)
>   at 
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)
>   at 
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:314)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:214)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
>   at java.lang.Thread.run(Thread.java:745)
> 
> 
> 
> How to set the necessary override property?
> 
> Best Regards,
> Subash Basnet
>