Hi Ismaël,
I guess you expect it will work with any runner ?
So, it means that we will have to modify the runners (in the Runner API).
From an user perspective, you can always extend a PTransform as you
need (I guess it's what you did for Debug).
My only concern is that the print could depend of the runner (do we use
System.out, slf4j, etc).
Do you already have the Debug PTransform code ? I would like to take a look.
Regards
JB
On 03/20/2016 10:14 AM, Ismaël Mejía wrote:
Hello,
I just started playing with Beam and I wanted to debug what happens
between transforms in pipelines. I wrote a simple 'Debug' transform for
this.
The idea is to apply a function based on a predicate to any element in a
collection without changing the collection, or in other words, a
transform that
does not transform but produces side effects.
The idea is better illustrated with this simple example:
.apply(FlatMapElements.via((String text) ->
Arrays.asList(text.split(" ")))
.withOutputType(new TypeDescriptor<String>() {
}))
.apply(Debug
.when((String s) -> s.startsWith("A"))
.with((String s) -> {
System.out.println(s);
return null;
}));
.apply(Filter.byPredicate((String text) -> text.length() > 5))
.apply(Debug.print()); // sugared method, same as above
I think this can be useful (at least for debugging purposes), is there
something
like this already in the SDK ? If this is not the case, can you please
give me some
feedback/ideas to improve my transform.
Thanks,
-Ismael
ps. You can find the code of the first version of the transform here:
https://github.com/iemejia/beam-playground/blob/master/src/main/java/org/apache/beam/transforms/Debug.java
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
