adutra commented on PR #3217: URL: https://github.com/apache/polaris/pull/3217#issuecomment-3626926537
The approach is very interesting imho. Thanks for the proposal! A few general remarks: 1. Redacting fields from JSON is tricky, we must be sure to replace the field with a valid value for the field type. Granted, most fields we'll redact will be of `string` type. `null` is probably a safer choice but even then, you could break the JSON schema if the field is non-nullable. 2. I wonder if the redaction rules should be made configurable. Some users may want more stuff redacted, some others may not want anything redacted (e.g. some use cases may require access to the table location). We should imo strive to be the least prescriptive possible, providing sane defaults for everyone to start with. 3. We may explore the possibility to use dedicated libraries for redaction, e.g. Phileas – although, probably overkill as a first implementation. 4. The redaction system may be considerably simpler to implement if flattened event types are implemented: https://lists.apache.org/thread/swn73976xj1lx8tkmcxpfvc2gv6znn2h. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
