[ https://issues.apache.org/jira/browse/ARROW-10695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320130#comment-17320130 ]
Antoine Pitrou commented on ARROW-10695: ---------------------------------------- Side note: perhaps we could still implement an option to raise an error instead of overwriting existing files. > [C++][Dataset] Allow to use a UUID in the basename_template when writing a > dataset > ---------------------------------------------------------------------------------- > > Key: ARROW-10695 > URL: https://issues.apache.org/jira/browse/ARROW-10695 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ > Reporter: Joris Van den Bossche > Priority: Minor > Labels: dataset, dataset-parquet-write > Fix For: 5.0.0 > > > Currently we allow the user to specify a {{basename_template}}, and this can > include a {{"\{i\}"}} part to replace it with an automatically incremented > integer (so each generated file written to a single partition is unique): > https://github.com/apache/arrow/blob/master/python/pyarrow/dataset.py#L713-L717 > It _might_ be useful to also have the ability to use a UUID, to ensure the > file is unique in general (not only for a single write) and to mimic the > behaviour of the old {{write_to_dataset}} implementation. > For example, we could look for a {{"\{uuid\}"}} in the template string, and > if present replace it for each file with a new UUID. -- This message was sent by Atlassian Jira (v8.3.4#803005)