Neiko2002 commented on issue #1528: URL: https://github.com/apache/incubator-fury/issues/1528#issuecomment-2061017027
It is just a hassle, right now the meta information of all images are stored in row-based styles. Then the embeddings information of all the images are in column based. I see three options with the current fury implementation: 1. Store meta information and embeddings in seperate files, and split the embeddings file in smaller files to cirumvent the 2GB limit 2. Keep everything in one file, but creates additional files if its breaks the 2GB barrier 3. Store everything in row-based format and split the files For 2 and 3 we would need to keep track how big the file already is to make a sensable split (not splitting the meta data or embedding of an image into two files). Finding a good splitting point for option 1 is more straight forwards since the number of embeddings fitting into one file can be calculated in advanced. Ideal memory allocation and ordering of data would need more consideration, if data is stored in multiple files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
