By the way, for the "Integration" DSL, in addition of explicit debug
transform, it would make sense to have an implicit "Tracer". It's
something that I planned: it would allow us to have sampling on
PCollection if the pipeline tracer is enabled (like we do in a Camel
route with the tracer).
Regards
JB
On 03/20/2016 10:14 AM, Ismaël Mejía wrote:
Hello,
I just started playing with Beam and I wanted to debug what happens
between transforms in pipelines. I wrote a simple 'Debug' transform for
this.
The idea is to apply a function based on a predicate to any element in a
collection without changing the collection, or in other words, a
transform that
does not transform but produces side effects.
The idea is better illustrated with this simple example:
.apply(FlatMapElements.via((String text) ->
Arrays.asList(text.split(" ")))
.withOutputType(new TypeDescriptor<String>() {
}))
.apply(Debug
.when((String s) -> s.startsWith("A"))
.with((String s) -> {
System.out.println(s);
return null;
}));
.apply(Filter.byPredicate((String text) -> text.length() > 5))
.apply(Debug.print()); // sugared method, same as above
I think this can be useful (at least for debugging purposes), is there
something
like this already in the SDK ? If this is not the case, can you please
give me some
feedback/ideas to improve my transform.
Thanks,
-Ismael
ps. You can find the code of the first version of the transform here:
https://github.com/iemejia/beam-playground/blob/master/src/main/java/org/apache/beam/transforms/Debug.java
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
