Hello, I'm working on the SingleStore IO connector and would like to discuss it with Beam developers. It would be great if the connector can use SingleStore parallel read <https://docs.singlestore.com/managed-service/en/query-data/query-procedures/read-query-results-in-parallel.html>. In the ideal case, the connector should use Single-read mode as it is faster than Multiple-read and consumes much less memory.
One of the problems is that in Single-read mode, each reader must initiate its read query before any readers will receive data. Is it possible to somehow configure Beam to start all DoFns at the same time? Or to get the numbers of started DoFns at the runtime? The other problem is that Single-read allows reading data from partition only once, so if one reading thread failed - all others should be restarted to retry. Is it possible to achieve this behavior? Or to at least gracefully fail without additional retries? Here are the first drafts of the design documentation <https://docs.google.com/document/d/1WU-hkoZ93SaGXyOz_UtX0jXzIRl194hCId_IdmEV9jw/edit?usp=sharing> . I would appreciate any help with this stuff :) -- Adalbert Makarovych Software Engineer at SingleStore <https://www.singlestore.com/customers/?utm_source=singlestore&utm_medium=email&utm_campaign=1-on-trustradius>
