The conversion from object to string will have uses outside of just
TextIO.Write so it seems logical that we would want to have a ParDo do the
conversion.
Text file formats have a lot of variance, even if you consider the subset
of CSV like formats where it could have fixed width fields, or escaping and
quoting around other fields, or headers that should be placed at the top.
Having all these format conversions within TextIO.Write seems like a lot of
logic to contain in that transform which should just focus on writing to
files.
On Tue, Nov 8, 2016 at 8:15 AM, Jesse Anderson <je...@smokinghand.com>
wrote:
> This is a thread moved over from the user mailing list.
>
> I think there needs to be a way to convert a PCollection<KV> to
> PCollection<String> Conversion.
>
> To do a minimal WordCount, you have to manually convert the KV to a String:
> p
> .apply(TextIO.Read.from("playing_cards.tsv"))
> .apply(Regex.split("\\W+"))
> .apply(Count.perElement())
> * .apply(MapElements.via((KV<String, Long> count) ->*
> * count.getKey() + ":" + count.getValue()*
> * ).withOutputType(TypeDescriptors.strings()))*
> .apply(TextIO.Write.to("output/stringcounts"));
>
> This code really should be something like:
> p
> .apply(TextIO.Read.from("playing_cards.tsv"))
> .apply(Regex.split("\\W+"))
> .apply(Count.perElement())
> * .apply(ToString.stringify())*
> .apply(TextIO.Write.to("output/stringcounts"));
>
> To summarize the discussion:
>
> - JA: Add a method to StringDelegateCoder to output any KV or list
> - JA and DH: Add a SimpleFunction that takes an type and runs toString()
> on it:
> class ToStringFn<InputT> extends SimpleFunction<InputT, String> {
> public static String apply(InputT input) {
> return input.toString();
> }
> }
> - JB: Add a general purpose type converter like in Apache Camel.
> - JA: Add Object support to TextIO.Write that would write out the
> toString of any Object.
>
> My thoughts:
>
> Is converting to a PCollection<String> mostly needed when you're using
> TextIO.Write? Will a general purpose transform only work in certain cases
> and you'll normally have to write custom code format the strings the way
> you want them?
>
> IMHO, it's yes to both. I'd prefer to add Object support to TextIO.Write or
> a SimpleFunction that takes a delimiter as an argument. Making a
> SimpleFunction that's able to specify a delimiter (and perhaps a prefix and
> suffix) should cover the majority of formats and cases.
>
> Thanks,
>
> Jesse
>
