Gustav created AVRO-3020:
----------------------------
Summary: [C#] Alternate AppendToFile Interface
Key: AVRO-3020
URL: https://issues.apache.org/jira/browse/AVRO-3020
Project: Apache Avro
Issue Type: Improvement
Components: csharp
Reporter: Gustav
I think it's wonderful that the csharp library finally has some functionality
to append to files. However we're writing to files located in Azure Block
Blobs. You can read more information about what/why we do that
[here|[https://www.gustavwengel.dk/azure-blob-block-avro-storage].]
But the gist is - we'd like to be able to "append" to a file that's not
actually located on the machine that the program is running, but in the cloud.
Currently we do this by caching the sync-marker, writing new rows and uploading
them to the cloud.
We unfortunately cannot use any of the new overloads. Here's why none of them
are fit for this use-case.
{noformat}
// Does not work, as I do not have a file path to write to
// the file is not located on the hard drive of the user
static IFileWriter<T> Avro.File.DataFileWriter< T >.OpenAppendWriter (
DatumWriter< T >
writer,
string path)
// Does not work as I do not have the full file lying on the computer, so I
have no proper
// inputStream to put into this overload.
static IFileWriter<T> Avro.File.DataFileWriter< T >.OpenAppendWriter (
DatumWriter< T > writer,
Stream inStream,
Stream outStream
)
{noformat}
I propose another overload (which we have internally at my company) that looks
like this:
{noformat}
static IFileWriter<T> Avro.File.DataFileWriter< T >.OpenAppendWriter (
DatumWriter< T > writer,
Stream outStream,
Codec codec,
byte[] syncBlock
)
{noformat}
Or alternatively an API that takes in an {{Avro.File.Header}}, an {{outStream}}
and a {{DatumWriter<T>}} then the user can construct their header on their own.
Let me know what you think.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
