hmh-the-one opened a new issue, #28967:
URL: https://github.com/apache/shardingsphere/issues/28967
## Question
In most recent version (5.4.0), queryWithCipherColumn & plainColumn were
removed, this change is apply according to this issue:
[25448](https://github.com/apache/shardingsphere/pull/25448).
Then in a massive records situation (e.g. more than 100 millions), how to
migrate smoothly from non-encrypted to encrypted? Following
[shardingsphere_docs_en.pdf](https://shardingsphere.apache.org/document/5.0.0-alpha/en/shardingsphere_docs_en.pdf),
chapter 3.6 Encryption, "Before system migration" & "During system migration"
parts, transparent data encryption process can be taken with only configuration
changes and self-designed and simple history plaintext data initialization.
Without queryWithCipherColumn, the solution I can figure out is like the
follow:
> * db logs listening, and logs are pushed to MQ, e.g. kafka, only
plaintext add/modify logs reserve
> * MQ consumption, plaintext data is retrieved, encrypted and written
back to coresponding record
> * initialzing history data, according the quantity this will take some
time. At the same time, new add/modify logs are continuously generated by the
application and encypted by MQ consumption
> * at a suitable moment, deep night, when MQ consumption lag is really
low or 0, proceed new version application release, switch to encrypted data
reading & writing
But at the last step, new and old version of application instances are
running at the same time, if a record is ADD/MODIFY in old version and QUERY in
new version at once, the QUERY function may not work properly.
Is there any better solution for data migration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
