JigaoLuo commented on PR #624: URL: https://github.com/apache/parquet-format/pull/624#issuecomment-6562447325
Thanks for driving this! I’ve also been following the Iceberg vector proposal, and I’d like to raise two points about vector's metadata for discussion. I agree that min/max statistics don’t make sense for vectors. However, there is other metadata that would make vector data more self-describing. It could live in Parquet or in Iceberg: 1. Distance function. See the reasoning from Harsha and the related discussion on the Iceberg side: https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?disco=AAACDqu3oWI&pli=1 2. Lineage: which embedding model or API produced the vectors. Just raising for discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
