thomasw21 commented on code in PR #33925: URL: https://github.com/apache/arrow/pull/33925#discussion_r1095639660
########## docs/source/format/CanonicalExtensions.rst: ########## @@ -72,4 +72,30 @@ same rules as laid out above, and provide backwards compatibility guarantees. Official List ============= -No canonical extension types have been standardized yet. +Fixed shape tensor +================== + +* Extension name: `arrow.fixed_shape_tensor`. + +* The storage type of the extension: ``FixedSizeList`` where: + + * **value_type** is the data type of individual tensors and + is an instance of ``pyarrow.DataType`` or ``pyarrow.Field``. + * **list_size** is the product of all the elements in tensor shape. + +* Extension type parameters: + + * **value_type** = Arrow DataType of the tensor elements + * **shape** = shape of the contained tensors as a tuple + * **is_row_major** = boolean indicating the order of elements Review Comment: > [@jorisvandenbossche ] @thomasw21 the page you link mentions the difference between "physical order" and "logical order". So the actual, physical layout in memory that pytorch uses can either be channels first (NCHW) or channels last (NHWC). But in both cases, this physical order is row-major. When interacting with those tensors as a user, pytorch however always shows the data as if it's NCHW, independent of the physical layout, i.e. what they call the "logical order". And it's when viewing the data as NCHW while the physical layout is NHWC that you get custom strides / non-C-contiguous layout for that view. But when considering the physical layout NHWC, the data is still row-major. I think I have a hard time understanding this point. To me, torch has the ability to change the underlying "physical" ordering depending on how you interpret data. See Figure 1 of that blogpost <img width="664" alt="image" src="https://user-images.githubusercontent.com/24695242/216576039-dfc056f5-b9cc-4c65-8ed1-d6ab9e2efe7d.png";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
![https://user-images.githubusercontent.com/24695242/216576039-dfc056f5-b9cc-4c65-8ed1-d6ab9e2efe7d.png"](https://user-images.githubusercontent.com/24695242/216576039-dfc056f5-b9cc-4c65-8ed1-d6ab9e2efe7d.png"){kind=link}