[ https://issues.apache.org/jira/browse/AVRO-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267690#comment-17267690 ]
Oscar Westra van Holthe - Kind edited comment on AVRO-3026 at 1/19/21, 6:54 AM: -------------------------------------------------------------------------------- Nice idea! But to be honest, I like the current implementation beter. This is how it's implemented now: {code} @tags(["PI", "Non PI"]) record Employee { @tags(["Name"]) string fullName; boolean active = true; long salary; @tags(["Phone"]) string phone; } {code} was (Author: opwvhk): What you propose can already be done: {code} @tags(["PI", "Non PI"]) record Employee { @tags(["Name"]) string fullName; boolean active = true; long salary; @tags(["Phone"]) string phone; } {code} > Allow custom annotations in IDL files and support translating them to AVSC > Avro. > -------------------------------------------------------------------------------- > > Key: AVRO-3026 > URL: https://issues.apache.org/jira/browse/AVRO-3026 > Project: Apache Avro > Issue Type: New Feature > Components: spec > Affects Versions: 1.9.0, 1.9.1, 1.9.2, 1.10.1 > Reporter: Feroze Daud > Priority: Major > > h2. Introduction > Our company has standardized on Avro schemas for all Data intestion and > storage. As part of this, and to satisfy CCPA, we need to be able to tag the > records and fields appropriately if they have PI, or Non PI information, etc. > Avro AVSC files, being valid json, can easily be modified to add tags that > will be used by downstream processors, and also wont interfere with Avro > itself ( to generate POJO, serialization, deserialization etc) > One such key we chose is simply called *tags*. It's example usage is shown > below. > {code:java} > { > "type": "record", > "name": "PropertyOwner", > "namespace": "com.acme.Property", > "tags": ["PI", "PII" ], > "fields": [ > { > "name": "FullName", > "type": "string", > "tags": ["Name"] > }, > { > "name": "PhoneNumber", > "type": "string", > "tags": ["Phone"] > }], > }{code} > > These tags can be processed by downstream processors and the data landing in > datalake, or database can be tagged appropriately. > > h2. Problem Description > While tagging will work fine for AVSC because adding extra fields doesnt make > it invalid, we will have a problem when using IDL to author schemas. IDL spec > does not allow a way to add extra tags that are copied over to the Avro > schema. > > h2. Proposal > I propose that we allow a special *@annotation* tag . And, this tag can be > applied to records and fields. Whatever is in this annotation should be > copied verbatim to the output AVSC. > For eg: > {code:java} > @annotation("tags", "[\"PI\", \"Non PI\"]" > record Employee { > @annotation("tags", "[\"Name\"]" > string fullName; > boolean active = true; > long salary; > @annotation("tags", "[\"Phone\"]" > string phone; > } {code} > > would generate an avro schema as folllows: > > {code:java} > { > "type": "record", > "name": "Employee", > "tags": ["PI", "PII" ], > "fields": [ > { > "name": "FullName", > "type": "string", > "tags": ["Name"] > }, > { > "name": "PhoneNumber", > "type": "string", > "tags": ["Phone"] > }], > }{code} > > As you can see, we dont need to support any wellformed JSONness in the > *@annotation* . It just takes a string and we just render it into the output > json. > @annotation("foo", "[\"bar\"]") -> "tags": ["bar"] > @annotation("foo", "\{\"bar\": \"jar\"}") -> "tags": {"bar": "jar"} > -- This message was sent by Atlassian Jira (v8.3.4#803005)