lidavidm commented on code in PR #41823:
URL: https://github.com/apache/arrow/pull/41823#discussion_r1685922783
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -283,6 +283,148 @@ UUID
A specific UUID version is not required or guaranteed. This extension
represents
UUIDs as FixedSizeBinary(16) with big-endian notation and does not
interpret the bytes in any way.
+Opaque
+=======
+
+Opaque represents a type or array that an Arrow-based system received from an
+external (often non-Arrow) system, which it cannot interpret or did not have
+support for in advance. In this case, it can pass on Opaque to its clients to
+show that a field exists, but that it cannot interpret the field or data.
+
+Extension parameters:
+
+* Extension name: ``arrow.opaque``.
+
+* The storage type of this extension is any type. If there is no underlying
+ data, the storage type should be Null.
+
+* Extension type parameters:
+
+ * **type_name** = the name of the unknown type in the external system.
+ * **vendor_name** = the name of the external system.
+
+* Description of the serialization:
+
+ A valid JSON object containing the parameters as fields. In the future,
+ additional fields may be added, but all fields current and future are never
+ required to interpret the array.
+
+Rationale
+---------
+
+Arrow systems often wrap non-Arrow systems, and so they must be prepared to
+handle data types and data that don't have an equivalent Arrow type. A client
+may still want to know of the existence of a field, or the types of other,
+supported fields. So returning an error because of an unrecognized type in
+one column, or dropping unsupported fields/columns, are poor solutions.
+
+Of course, the Arrow system can use extension types. But it will not have an
+extension type prepared for every possible type in advance; for example, the
+non-Arrow system may have its own extension mechanisms. It could "make up" an
+extension type on the fly. But this misleads clients who cannot tell if the
+type is truly supported or not by the intermediate Arrow application.
+
+The Opaque type can be used instead. Because it explicitly means that the
+*intermediate* system does not support a type, it can be used to declare an
+unsupported field or column without silently losing data or erroring. In
+other words: if an Arrow system encounters a non-Arrow type it was not
+prepared to handle, it can use Opaque to still pass the type on to a client.
+
+Applications **should not** make conventions around vendor_name and type_name.
Review Comment:
Thanks, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
