Jackie-Jiang commented on pull request #6845:
URL: https://github.com/apache/incubator-pinot/pull/6845#issuecomment-828082724
For the complex type column, I can see 4 different operations:
- SKIP (not include in the output)
- FLATTEN (into key-value pairs, can generate multiple records for list)
- RETAIN (as Map or List)
- TO_JSON_STRING
We can have a default operation, then have an override field list for each
operation.
We should probably separate the map config from the list config because I
can see a common case where user only want to flatten map but not the list, so
that list can be stored as MV column.
Also, I think we should make delimiter configurable. I remember there are
some issues storing column name with dot in presto connector (@xiangfu0 to
confirm).
So the overall config would look like:
```
complexTypeConfig: {
"map": {
"default": "FLATTEN",
"SKIP": ["colA"],
"TO_JSON_STRING": ["colB"],
"delimiter": "_"
},
"list": {
"default": "RETAIN",
"FLATTEN": ["colC_listField"],
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
