[
https://issues.apache.org/jira/browse/BEAM-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144944#comment-16144944
]
Etienne Chauchot commented on BEAM-2802:
----------------------------------------
I agree the {{\002\n}} example seems odd. But IMHO we should allow the user to
add whatever he wants as a delimiter, we should not limit him as long as this
liberty has a limited performance / maintenance cost.
For the text file definition: I think that as long as a file does not contain
binary and is not in a common format like XML or JSON, it can be considered as
a pure text file no matter if it can be easily read by a human or if it has new
line delimiters.
IMHO, I also think that files with custom delimiters are quite a common user
problem (we have several client use cases and the other related PR tends to
also prove its need). IMHO, I think it merits being included in the Beam SDK.
I would be reluctant to chip a specific TextIO to clients for maintenance
reasons.
I could create another IO but I find it's a pity to duplicate all the structure
of an IO whereas the change is 15 lines of code in
TextIO#findSeparatorBounds(). Maybe I can submit the PR that updates the TextIO
and we can discuss it towards maintenance, performance and code location
aspects in the PR.
> TextIO should allow specifying a custom delimiter
> -------------------------------------------------
>
> Key: BEAM-2802
> URL: https://issues.apache.org/jira/browse/BEAM-2802
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-extensions
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: Minor
>
> Currently TextIO use {{\r}} {{\n}} or {{\r\n}} or a mix of the two to split a
> text file into PCollection elements. It might happen that a record is spread
> across more than one line. In that case we should be able to specify a custom
> record delimiter to be used in place of the default ones.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
