[ https://issues.apache.org/jira/browse/PHOENIX-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Jasani updated PHOENIX-5521: ---------------------------------- Description: An HBase coprocessor Endpoint hook that takes in a request from a remote cluster (containing both the WALEdit's data and the WALKey's annotated metadata telling the remote cluster what tenant_id, logical tablename, and timestamp the data is associated with). Ideally the API's message format should be configurable / pluggable, and could be either a protobuf or an Avro schema similar to the WALEdit-like one described by PHOENIX-5443. Endpoints in HBase are structured to work with protobufs, so some conversion may be necessary in an Avro-compatible version. Future work may also extend this to any conforming schema given by a schema service such as the one in PHOENIX-5443, which would be useful in allowing PHOENIX-5442's CDC service to be used as a backup / migration tool. The endpoint hook would take the metadata + data and regenerate a complete set of Phoenix mutations, both data and indexes, just as the phoenix client did for the original SQL statement that generated the source-side edits. These mutations would be written to the remote cluster by the normal Phoenix write path. HBASE-27529 provides regionserver coproc hook to attach WAL extended attributes to mutations at replication sink. We can utilize this hook and provide end-to-end flow for Phoenix metadata attributes (tenant id, schema name, logical table name, table type etc). The source cluster can attach the metadata attributes to source mutations. By using "phoenix.append.metadata.to.wal", the attributes can be appended to WAL in the form of extended attributes. By using a new regionserver coproc in Phoenix, we can utilize HBASE-27529 and allow the sink cluster to attach the WAL extended attributes to Mutations. This way, IndexRegionObserver and other coproc endpoints would be able to get Phoenix metadata attributes in both source and sink clusters. The changes required to enable replication sink coproc, and allow it to attach phoenix metadata as Mutation attributes at the Sink cluster: # Add "org.apache.phoenix.coprocessor.ReplicationSinkEndpoint" to hbase.coprocessor.regionserver.classes config # phoenix.append.metadata.to.wal = true # Use "CHANGE_DETECTION_ENABLED = true" for the given table was: An HBase coprocessor Endpoint hook that takes in a request from a remote cluster (containing both the WALEdit's data and the WALKey's annotated metadata telling the remote cluster what tenant_id, logical tablename, and timestamp the data is associated with). Ideally the API's message format should be configurable / pluggable, and could be either a protobuf or an Avro schema similar to the WALEdit-like one described by PHOENIX-5443. Endpoints in HBase are structured to work with protobufs, so some conversion may be necessary in an Avro-compatible version. Future work may also extend this to any conforming schema given by a schema service such as the one in PHOENIX-5443, which would be useful in allowing PHOENIX-5442's CDC service to be used as a backup / migration tool. The endpoint hook would take the metadata + data and regenerate a complete set of Phoenix mutations, both data and indexes, just as the phoenix client did for the original SQL statement that generated the source-side edits. These mutations would be written to the remote cluster by the normal Phoenix write path. HBASE-27529 provides regionserver coproc hook to attach WAL extended attributes to mutations at replication sink. We can utilize this hook and provide end-to-end flow for Phoenix metadata attributes (tenant id, schema name, logical table name, table type etc). The source cluster can attach the metadata attributes to source mutations. By using "phoenix.append.metadata.to.wal", the attributes can be appended to WAL in the form of extended attributes. By using a new regionserver coproc in Phoenix, we can utilize HBASE-27529 and allow the sink cluster to attach the WAL extended attributes to Mutations. This way, IndexRegionObserver and other coproc endpoints would be able to get Phoenix metadata attributes in both source and sink clusters. > Phoenix-level HBase Replication sink (Endpoint coproc) > ------------------------------------------------------ > > Key: PHOENIX-5521 > URL: https://issues.apache.org/jira/browse/PHOENIX-5521 > Project: Phoenix > Issue Type: Sub-task > Reporter: Geoffrey Jacoby > Assignee: Viraj Jasani > Priority: Major > > An HBase coprocessor Endpoint hook that takes in a request from a remote > cluster (containing both the WALEdit's data and the WALKey's annotated > metadata telling the remote cluster what tenant_id, logical tablename, and > timestamp the data is associated with). > Ideally the API's message format should be configurable / pluggable, and > could be either a protobuf or an Avro schema similar to the WALEdit-like one > described by PHOENIX-5443. Endpoints in HBase are structured to work with > protobufs, so some conversion may be necessary in an Avro-compatible version. > Future work may also extend this to any conforming schema given by a schema > service such as the one in PHOENIX-5443, which would be useful in allowing > PHOENIX-5442's CDC service to be used as a backup / migration tool. > The endpoint hook would take the metadata + data and regenerate a complete > set of Phoenix mutations, both data and indexes, just as the phoenix client > did for the original SQL statement that generated the source-side edits. > These mutations would be written to the remote cluster by the normal Phoenix > write path. > > HBASE-27529 provides regionserver coproc hook to attach WAL extended > attributes to mutations at replication sink. We can utilize this hook and > provide end-to-end flow for Phoenix metadata attributes (tenant id, schema > name, logical table name, table type etc). The source cluster can attach the > metadata attributes to source mutations. By using > "phoenix.append.metadata.to.wal", the attributes can be appended to WAL in > the form of extended attributes. By using a new regionserver coproc in > Phoenix, we can utilize HBASE-27529 and allow the sink cluster to attach the > WAL extended attributes to Mutations. This way, IndexRegionObserver and other > coproc endpoints would be able to get Phoenix metadata attributes in both > source and sink clusters. > > The changes required to enable replication sink coproc, and allow it to > attach phoenix metadata as Mutation attributes at the Sink cluster: > # Add "org.apache.phoenix.coprocessor.ReplicationSinkEndpoint" to > hbase.coprocessor.regionserver.classes config > # phoenix.append.metadata.to.wal = true > # Use "CHANGE_DETECTION_ENABLED = true" for the given table -- This message was sent by Atlassian Jira (v8.20.10#820010)