pimdh commented on PR #38008: URL: https://github.com/apache/arrow/pull/38008#issuecomment-2800814128
Thanks a lot! I'm currently using a combination of Arrow with Ray (Data) for data transformation pipelines, and Lance for storage. Now I'm using a custom VariableShapeTensor, awaiting this official type to land. This is all on host memory. I copy things to numpy first, and then to GPU. I plan to experiment with Arrow+DLPack in the future. As I understand it, the FixedShapeTensorArray has less need for a Large version, because the FixedSizeListType does not hold an offset int32 array like the ListType does. So 1 array can hold many tensors, as long as each is less than 2^31 elements. The VariableShapeTensorArray as it's currently documented can hold at most 2^31 elements in all tensors in the array combined, so that limit is hit earlier. Also, there doesn't seem to be a LargeFixedSizeListType, which has an int64 size? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
