[openstack-dev] [MagnetoDB] Best practices for uploading large amounts of data

2014-03-27 Thread Illia Khudoshyn
Hi, Openstackers, I'm currently working on adding bulk data load functionality to MagnetoDB. This functionality implies inserting huge amounts of data (billions of rows, gigabytes of data). The data being uploaded is a set of JSON's (for now). The question I'm interested in is a way of data transp

Re: [openstack-dev] [MagnetoDB] Best practices for uploading large amounts of data

2014-03-28 Thread Serge Kovaleff
Hi Iliia, I would take a look into BSON http://bsonspec.org/ Cheers, Serge Kovaleff On Thu, Mar 27, 2014 at 8:23 PM, Illia Khudoshyn wrote: > Hi, Openstackers, > > I'm currently working on adding bulk data load functionality to MagnetoDB. > This functionality implies inserting huge amounts of d

Re: [openstack-dev] [MagnetoDB] Best practices for uploading large amounts of data

2014-03-28 Thread Maksym Iarmak
2014-03-28 11:29 GMT+02:00 Serge Kovaleff : > Hi Iliia, > > I would take a look into BSON http://bsonspec.org/ > > Cheers, > Serge Kovaleff > > On Thu, Mar 27, 2014 at 8:23 PM, Illia Khudoshyn > wrote: > >> Hi, Openstackers, >> >> I'm currently working on adding bulk data load functionality to >>

Re: [openstack-dev] [MagnetoDB] Best practices for uploading large amounts of data

2014-03-28 Thread Maksym Iarmak
Hi guys, I suggest taking a look, how Swift and Ceph do such things. 2014-03-28 12:33 GMT+02:00 Maksym Iarmak : > > > > 2014-03-28 11:29 GMT+02:00 Serge Kovaleff : > > Hi Iliia, >> >> I would take a look into BSON http://bsonspec.org/ >> >> Cheers, >> Serge Kovaleff >> >> On Thu, Mar 27, 2014 a

Re: [openstack-dev] [MagnetoDB] Best practices for uploading large amounts of data

2014-03-28 Thread Chmouel Boudjnah
Maksym Iarmak wrote: > I suggest taking a look, how Swift and Ceph do such things. under swift (and CEPH via the radosgw which implement swift API) we are using POST and PUT which has been working relatively well Chmouel ___ OpenStack-dev mailing list O

Re: [openstack-dev] [MagnetoDB] Best practices for uploading large amounts of data

2014-03-28 Thread Dmitriy Ukhlov
On 03/28/2014 11:29 AM, Serge Kovaleff wrote: Hi Iliia, I would take a look into BSON http://bsonspec.org/ Cheers, Serge Kovaleff On Thu, Mar 27, 2014 at 8:23 PM, Illia Khudoshyn mailto:ikhudos...@mirantis.com>> wrote: Hi, Openstackers, I'm currently working on adding bulk data loa

Re: [openstack-dev] [MagnetoDB] Best practices for uploading large amounts of data

2014-03-28 Thread Aleksandr Chudnovets
Dmitriy Ukhlov wrote: > > I guess if we a talking about cassandra batch loading the fastest way > is to generate sstables locally and load it into Cassandra via JMX or > sstableloader > http://www.datastax.com/dev/blog/bulk-loading > > Good idea, Dmitriy. IMHO bulk load is back-end specific task

Re: [openstack-dev] [MagnetoDB] Best practices for uploading large amounts of data

2014-03-28 Thread Romain Hardouin
Bulk loading with sstableloader is blazingly fast (the price to pay is that's not portable of course).  Also it's network efficient thanks to SSTable compression. If the network is not a limiting factor then LZ4 will be great. Le Vendredi 28 mars 2014 13h46, Aleksandr Chudnovets a écrit :