Re: In-memory sorting of plasma objects

2019-08-23 Thread Tanveer Ahmad - EWI
Thank you Wes. I see. Regards, Tanveer Ahmad From: Wes McKinney Sent: Thursday, August 22, 2019 5:12:06 PM To: dev@arrow.apache.org Subject: Re: In-memory sorting of plasma objects hi Tanveer, IIUC there is logic for moving data that's managed by P

Re: In-memory sorting of plasma objects

2019-08-22 Thread Wes McKinney
hi Tanveer, IIUC there is logic for moving data that's managed by Plasma servers between nodes in the Ray project (https://github.com/ray-project/ray) --if you need to move the bytes from one node to another you need to use some kind of messaging / RPC tool. The Ray developers might have some advi

In-memory sorting of plasma objects

2019-08-22 Thread Tanveer Ahmad - EWI
Hi, I need some help regarding data exchange between Arrow based plasma shared memory objects on cluster nodes. I have two Plasma shared memory objects each contains a RecordBatch on different nodes of a cluster. I want to use pandas dataframes or something like that (dask) on a single node