rok commented on PR #624: URL: https://github.com/apache/parquet-format/pull/624#issuecomment-6564308362
Hey @JigaoLuo thanks for reviewing! Discussion on points you raise would better fit the discusson on the ML ([1](https://lists.apache.org/thread/kgmgn2vmrz49thbfhv8b66lglsyd81xt), [2](https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3)). I'll still try to reply here, but if you want to push this further the dev@parquet ML is the platform you should use. > there is other metadata that would make vector data more self-describing. It could live in Parquet or in Iceberg: > > 1. Distance function. See the reasoning from Harsha and the related discussion on the Iceberg side: https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?disco=AAACDqu3oWI&pli=1 I am not an embeddings or vector DB expert, but wouldn't distance metric always an operation acting on pairs of vectors? It feels more like an application level concern and parquet already has a file-level metadata key-value metadata where such information could be stored. Or am I missing something here that could not be captured there? > 2. Lineage: which embedding model or API produced the vectors. Agreed that this is important information, but I'm not sure it's something that should live in the type. It would make parquet more self-contained, but it would require agreement on model identity between embedding producers which is out of reach for parquet. If you feel strongly about this, please sketch out how you imagine this would look like. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
