Hi users, We use arrow flight to serve data between tech stacks. The server is written in python. When serving data we stream it in batches. For large batches, I have noticed that it will make a custom split of the data serving some columns first followed by others. The client has the final schema so can glue them together. We are implementing the orchestrator-worker architecture suggested in the arrow flight documentation. Is there a supported method for the orchestrator to break down queries into custom splits – like the batching process? So the orchestrator can return multiple tickets to be served by different workers with a joint schema – so the client knows how to reassemble them. It isn’t too hard to implement this in a custom way – though as the arrow flight clients are written in different tech stacks, any change to the custom API would need to be reimplemented by each team. So I was looking to see if there is a supported way in the arrow flight API.
Thanks for any help, Alex This email has been sent by a member of the Man group (“Man”). Man's parent company, Man Group plc, is registered in Jersey (company number 127570) with its registered office at 22 Grenville Street, St Helier, Jersey, JE4 8PX. The contents of this email are for the named addressee(s) only. It contains information which may be confidential and privileged. If you are not the intended recipient, please notify the sender immediately, destroy this email and any attachments and do not otherwise disclose or use them. Email transmission is not a secure method of communication and Man cannot accept responsibility for the completeness or accuracy of this email or any attachments. Whilst Man makes every effort to keep its network free from viruses, it does not accept responsibility for any computer virus which might be transferred by way of this email or any attachments. This email does not constitute a request, offer, recommendation or solicitation of any kind to buy, subscribe, sell or redeem any investment instruments or to perform other such transactions of any kind. Man reserves the right to monitor, record and retain all electronic and telephone communications through its network in accordance with applicable laws and regulations. During the course of our business relationship with you, we may process your personal data, including through the monitoring of electronic communications. We will only process your personal data to the extent permitted by laws and regulations; for the purposes of ensuring compliance with our legal and regulatory obligations and internal policies; and for managing client relationships. For further information please see our Privacy Notice: https://www.man.com/privacy-policy
