Re: Load a csv or a avro?

On 7/6/24 13:09, sud wrote: On Fri, Jul 5, 2024 at 8:24 PM Adrian Klaver > wrote: On 7/5/24 02:08, sud wrote: > Hello all, > > Its postgres database. We have option of getting files in csv and/or in > avro format messages from another

Re: Load a csv or a avro?

On Sat, Jul 6, 2024 at 4:10 PM sud wrote: > On Fri, Jul 5, 2024 at 8:24 PM Adrian Klaver > wrote: > >> On 7/5/24 02:08, sud wrote: >> > Hello all, >> > >> > Its postgres database. We have option of getting files in csv and/or in >> > avro format messages from another system to load it into our p

Re: Load a csv or a avro?

On Fri, Jul 5, 2024 at 8:24 PM Adrian Klaver wrote: > On 7/5/24 02:08, sud wrote: > > Hello all, > > > > Its postgres database. We have option of getting files in csv and/or in > > avro format messages from another system to load it into our postgres > > database. The volume will be 300million me

Re: Load a csv or a avro?

On Fri, Jul 5, 2024 at 3:27 PM Kashif Zeeshan wrote: > Hi > > There are different data formats available, following are few points for > there performance implications > > 1. CSV : It's easy to use and widely supported but it can be slower due to > parsing overload. > 2. Binary : Its faster to lo

Re: Load a csv or a avro?

On 7/5/24 02:08, sud wrote: Hello all, Its postgres database. We have option of getting files in csv and/or in avro format messages from another system to load it into our postgres database. The volume will be 300million messages per day across many files in batches. Are dumping the entire

Re: Load a csv or a avro?

On Fri, Jul 5, 2024 at 5:08 AM sud wrote: > Hello all, > > Its postgres database. We have option of getting files in csv and/or in > avro format messages from another system to load it into our postgres > database. The volume will be 300million messages per day across many files > in batches. > >

Re: Load a csv or a avro?

Hi, Performance Considerations Avro files are smaller due to compression so needing less I/O time. whereas CSV files are simpler but larger in size so read/write will need more time. COPY command works very well with CSV files whereas ETL process is required for handling Avro. Regards, Mu

Re: Load a csv or a avro?

pá 5. 7. 2024 v 11:08 odesílatel sud napsal: > > Hello all, > > Its postgres database. We have option of getting files in csv and/or in avro > format messages from another system to load it into our postgres database. > The volume will be 300million messages per day across many files in batches.

Re: Load a csv or a avro?

Hi There are different data formats available, following are few points for there performance implications 1. CSV : It's easy to use and widely supported but it can be slower due to parsing overload. 2. Binary : Its faster to load but not human understandable. Hope this helps. Regards Kashif Ze

Load a csv or a avro?

Hello all, Its postgres database. We have option of getting files in csv and/or in avro format messages from another system to load it into our postgres database. The volume will be 300million messages per day across many files in batches. My question was, which format should we chose in regards