Brandon Inman created CRUNCH-258:
------------------------------------
Summary: Multiple output channels from DoFn
Key: CRUNCH-258
URL: https://issues.apache.org/jira/browse/CRUNCH-258
Project: Crunch
Issue Type: New Feature
Components: Core
Reporter: Brandon Inman
Assignee: Josh Wills
As discussed on the mailing list[1], add support for multiple output channels
from a DoFn.
Summarized outcome of the thread:
A DoFn that has multiples outputs emits a Pair<T,U>. For example, this might
be to separate objects representing an error condition from those representing
expected outputs. In a typical case, one element in the pair will be null and
the other nonnull, although both may be nonnull (and both may be null but this
pair would be effectively discarded without error)
A utility method is provided that takes a PCollection<Pair<T,U>> and returns a
Pair<PCollection<T>, PCollection<U>>. The caller will write, call parallelDo
on, and/or materialize these collections as needed.
Psuedocode for the utility method-
public static <T, U> Pair<PCollection<T>,PCollection<U>> filterChannels(final
PCollection<Pair<T,U>> pCollection, final PType<T>firstPType, final PType<U>
secondPType) {
final PCollection<T> stdout = collection.parallelDo(new FirstEmittingDoFn<T>,
firstPType);
final PCollection<U> stderr = collection.parallelDo(new
SecondEmittingDoFn<U>, secondPType);
return Pair.of(stdout,stderr);
}
Psuedocode for the FirstEmttitingDoFn (SecondEmittingDoFn follows similar
pattern)-
public static class FirstEmittingDoFn<T extends Pair<U, ?>, U> extends
DoFn<Pair<U, ?>, U> {
@Override
public void process(Pair<U, ?> input, Emitter<U> emitter) {
final U first = input.first();
if (first != null) {
emitter.emit(first);
}
}
}
}
[1]
https://mail-archives.apache.org/mod_mbox/crunch-user/201308.mbox/%3CCAH29n6NK3bypEc9f2KADfkFFYL-b8%2B-HP3tXNW1_yF6f3TA65w%40mail.gmail.com%3E
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
