Dear Hoppers, The other day I had this idea which is just too cool not to act upon. The idea is to provide methods of reflection for Hop. In particular, reflection on what's going on during the execution of pipelines and workflows. So to act upon these urges of mine I started working on a new "Reflection" plugin for Hop.
The idea is to have a new type of metadata "Pipeline Log" which allows you to specify to which pipeline you want pipeline logging information to be sent at the start of any pipeline, at the end and periodically. In the specified pipeline you'll have a transform which will provide the actual logging information in the form of fields. We can then do the same for workflows of-course but also for data probes: capture the data output of a specific transform, type of transform, all transforms, ... and deliver it to another pipeline in a normalized fashion. You can do all sorts of cool things with it this way, I believe in the realms of continuous data profiling, data quality, machine learning and what not. I guess it's one of these ideas that have always been around and sound so incredibly simple but for some reason elude us. Cheers, Matt
