Hi Hop users, We're evaluating whether HOP is the right tool to solve a common problem for our business.
We encounter hundreds of different file formats containing similar layers of one-to-many hierarchy (simplified example below). Getting this to work using out-of-box inputs/outputs and transform components results in a complex/convoluted set of workflows & pipelines. Since we run into this so often, we would like to develop a plugin with a custom "input" component that reads the input file, inserts some ID fields for relationships, and exposes multiple output rowsets (one for each schema/row type) that can be mapped to separate downstream transformations. Eventually we'd like to make another custom "output" component that can accept multiple inputs to load them where we need them with hierarchy preserved (JSON, relational DB, etc.). After reviewing the plugin documentation and samples, I'm still not sure whether this is possible. It seems that the relevant plugin base classes assume there will always be a single schema (IRowMeta) and single rowset shared by all input and output connections/hops. I believe we would require a single "transform" to have multiple IRowMeta and multiple rowsets and the ability to select a specific one for any given hop to a downstream transform/component. Is there a good path to accomplishing this with a HOP plugin? Or perhaps a better approach to the problem with existing Hop features? Thanks! Example file: REC|Jane Smith|03-20-2003 ADDR|123 Main Street|Apartment 321|Anytown|US|55555 ACT|987654321|$4321.56|02-01-2023|03-02-2023 DTL|debit|$23.45|02-05-2023 DTL|debit|$143.20|02-13-2023 DTL|credit|$652.02|02-14-2023 DTL|debit|$8.78|02-28-2023 ACT|56789123|$7894.56|02-01-2023|03-02-2023 DTL|credit|$0.28|02-14-2023 REC|John Jacobs|03-20-2003 ADDR|876 Big Avenue||Anywhere|US|55556 ACT|5632178|$2256.79|02-01-2023|03-02-2023 DTL|credit|$0.02|02-14-2023
