Re: [I] Avro adapter - Read and write Avro container files [arrow-java]
lidavidm commented on issue #794: URL: https://github.com/apache/arrow-java/issues/794#issuecomment-3048848089 I think if the need comes up we can add it - I don't think we have to do it until there's a need though. (The batch API is just a general Java API thing that I think is kind of missing, not something we have to tackle here.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Avro adapter - Read and write Avro container files [arrow-java]
martin-traverse commented on issue #794: URL: https://github.com/apache/arrow-java/issues/794#issuecomment-3048800196 Ok great :) If you'd like a batch level reader / writer as separate classes in the tree I can do that fairly easily - anyway each batch has to be decoded from a single ByteBuffer because of the encryption codecs. Would that be helpful or did I misunderstand? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Avro adapter - Read and write Avro container files [arrow-java]
lidavidm commented on issue #794: URL: https://github.com/apache/arrow-java/issues/794#issuecomment-3048488638 Ok, we can keep the proposed API. I agree with many of the problems you have - I prefer explicit control too. I have this same problem with some other Arrow implementations, unfortunately (actually in general I think file readers shouldn't read files, but that's neither here nor there). (Also, it's probably worth considering if it's possible to have a 'RecordBatchReader' API that's simpler - it would be useful for the C Data Interface - but this is getting off topic.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Avro adapter - Read and write Avro container files [arrow-java]
martin-traverse commented on issue #794: URL: https://github.com/apache/arrow-java/issues/794#issuecomment-3047867486 Hm - I have found ArrowReader / ArrowWriter to be a bit opinionated. Particularly an issue with reader because you have less control (when bytes are going to arrive) but also writer makes a lot of assumptions e.g. dictionaries are static, no deltas etc. Neither gives you much control of what is happening at the file level. I ended up overriding and shading some of the supporting classes in order to use them (I have patches to submit which could reduce the need for this). I think the goal with ArrowReader / ArrowWriter was to abstract away the different formats (file and stream), so perhaps that abstraction is partly what causes the loss of control. My preference would be for a more explicit API that directly maps to the file structure, it is always possible to layer generalisation on top for a specific pattern. Here is a quick summary of what is the same / different: Writer: /// These can be the same as ArrowWriter void writeBatch(); long bytesWritten(); void close;(); // These are different void writeHeader(); // Explicit control instead of start() / end(), which may or may not trigger writes void resetBatch(VectorSchemaRoot batch); // Allow streaming pattern Reader: // These are the same as ArrowReader: VectorSchemaRoot getVectorSchemaRoot(); long bytesRead(); void close(); // These are also the same, from DictionaryProvider Set getDictionaryIds(); Dictionary lookup(long id); // These are different - Arrow has readSchema() and initialize(), which may or may not trigger reads void readHeader(); // Explicit read control Schema getSchema(); // Explicit get - does not trigger reading // These are aloo different - explicit control over reading boolean readBatch(); boolean hasNextBatch(); long nextBatchPosition(); long nextBatchSize(); One minor detail on naming - ArrowReader has `loadNaextBatch()` which is possibly more descriptive, but I think there is value in sticking with just one scheme - Load / save, read / write, get / put etc. otherwise it gets confusing. Happy to take a steer if you feel differently on any of this. Otherwise if you are happy lmk and I'll make a start :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Avro adapter - Read and write Avro container files [arrow-java]
lidavidm commented on issue #794: URL: https://github.com/apache/arrow-java/issues/794#issuecomment-3047008954 Is it possible to at least fit the ArrowReader/ArrowWriter 'interfaces' or do those base classes carry too much baggage? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
