sdf-jkl commented on issue #50622: URL: https://github.com/apache/arrow/issues/50622#issuecomment-5073509393
> I am not sure how important this case is, though, because if you had data that was originally highly typed and structured you probably wouldn't be using Variant 🤔 @alamb That's the point I was trying to make. Variant's physical types only include signed integers, same goes for the `typed_value` type support in the Shredding spec. I am proposing to drop unsigned integers from the `typed_value` allowed types for Arrow since they always have to be written as signed. > > Null -> Null: the shredding table has no null row > > This is a good argument. Parquet indeed has a UNKNOWN logical type to represent a column with no known type. It would be useful when user-hinted column cannot be shredded. So it seems like a missing piece in the Parquet shredding spec? @wgtmac Same goes for the Null type, I'm proposing to drop it from the Arrow table rather than add a null row to the Parquet spec. A null in `typed_value` already carries meaning: the row's value couldn't be shredded into the column type (or the field is missing, for object fields). An UNKNOWN column is always null, so it could never mark a row as shredded, and a Variant null would still have to be encoded in `value` as `00`. Worse, for object fields `typed_value` null + `value` null means the field is missing, so shredding null into `typed_value` would make "field present and null" indistinguishable from "field missing". I don't think the null row is a missing piece of the shredding spec, that state is reserved by the null-signaling protocol itself. > Can you file this as an issue on arrow-go so we can track and fix this? We should definitely make sure that arrow-go is spec conformant @zeroshade before filing I want to pin the correct behavior. I think Arrow should not support unsigned int `typed_value` at all, which would make the fix different. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
