Mariappan Asokan created PARQUET-134:
----------------------------------------
Summary: Enhance ParquetWriter with file creation flag
Key: PARQUET-134
URL: https://issues.apache.org/jira/browse/PARQUET-134
Project: Parquet
Issue Type: Improvement
Reporter: Mariappan Asokan
Currently, Parquet files can be created only with "create in exclusive mode"
(like {{O_CREAT|O_EXCL}} in UNIX {{open}} system call.) This may be
inconvenient in a situation where user knows that overwriting an existing file
is okay. This is especially true for Parquet files created on a local disk.
For example, a user might want to specify a named pipe (on UNIX) as the target,
encrypt the bytes on the fly by reading from the named pipe in another process,
and write to disk. The named pipe file has to exist first.
I am thinking that we can have OVERWRITE and CREATE modes for now. If
appending to Parquet files is supported in the future, we can add APPEND mode
later. These mode flags can be defined as constants in {{ParquetWriter.java.}}
There are 7 constructors in {{ParquetWriter}} class. I am thinking of adding
one more that takes the mode flag (and all the arguments of the constructor
with the most arguments today.) Also, a new constructor of
{{ParquetFileWriter}} will take the mode flag as an argument and the following
statement
{code}
this.out = fs.create(file, false);
{code}
will be modified so that for OVERWRITE mode, the second argument to
{{create()}} will be set to {{true.}}
I can submit a patch with the above changes and a test. Committers, please
give your feedback with suggestions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
