On Tue, Jan 26, 2016 at 6:03 PM Joseph Winston <[email protected]> wrote:
> I am looking for a document or hints on best practices when using PDX. > The two specific use cases that I’m interested in understanding are: > 1. Large arrays — Currently these data types are kept in a shared memory > segments that are organized using the most common access pattern (For > example: z fastest, then y, then x). When using PDX, should a single large > array that normally is on the order of 100s of GB be broken into smaller > objects, say z slices to help with loading the data? Are there better ways > to use PDX for these 3D and higher dimension arrays? > Like most network based storage you should break your objects into smaller chunks. You cannot fetch portions of the object so if you only need one entry in the array you will have to fetch the entire array (100s GB). You should consider logical chunks of data that fit the common operations you will perform. For example if you operations are on individual entries in this array I would split the data there, each array entry is and entry in the cache. These chunks should be in the order of MBs for best performance. -Jake
