David, I'm importing into 2.26, postgresql is tuned as far as possible (the test was on my laptop), and 12GB RAM. I was using CSV, put there should not be any significant difference between JSON and CSV.
I will import the 2.5 mill events directly and see what time that takes, comparatively. Regards Calle On 30 June 2017 at 18:05, David Huser <dhu...@baosystems.com> wrote: > Hi Calle > > When some conditions are met, the event import can be really speedy > (imports of ~100 000 in <30 minutes). > > - Recent version (2.25+) > - PostgreSQL properly tuned (this is important) > - Enough RAM (8 GB+) > > I usually use JSON files and post them against /api/events. > > David > > On Fri, Jun 30, 2017 at 12:58 PM, Calle Hedberg <calle.hedb...@gmail.com> > wrote: > >> Ime, >> >> I was using the UI - have not looked at the API endpoint. >> >> For now I will use sql scripts as I've done before - I'm just trying to >> find out if anybody is working on improving what is (now) an import >> function that in reality only caters for small data sets. There is no >> question about the core developers ability to speed this up - I saw that >> with the import of OU meta-data - but not sure if anybody has done anything. >> >> If no answers from core developers, the I will write a JIRA issue for it. >> >> Regards >> Calle >> >> On 29 June 2017 at 02:25, Ime Asangansi <imeasanga...@gmail.com> wrote: >> >>> Hi Calle, >>> >>> Thanks for your question. >>> Just curious (I don't have answer). As we are planning to do this soon. >>> >>> Are you doing this through UI or API? >>> If API, which endpoint enables this? >>> >>> Thanks >>> >>> Ime >>> >>> >>> >>> On Jun 28, 2017 19:15, "Calle Hedberg" <calle.hedb...@gmail.com> wrote: >>> >>>> Hi >>>> >>>> I started importing around 700,000 events 2 days ago (about 5 mill >>>> individual values) - and the import is still running, 48 hours later. >>>> Import is slowing - it seemed to be importing around 7-8 values per second, >>>> now down to 2-3 per second. It looks like that import might take around 200 >>>> hours - which makes no sense from a usability point of view. Importing >>>> OrgUnits used to be equally slow, but a revamp recently change a 10,000 >>>> OrgUnit import&update from hours to something like 5 seconds. >>>> >>>> It seems obvious that event imports have been designed to import values >>>> one by one, which is equivalent to asking people to fetch water using a >>>> leaking tea-spoon. >>>> >>>> I've searched JIRA but cannot find any issue related to this, thus this >>>> post: Are there any plans underway to make import of Event data usable for >>>> larger record sets? >>>> >>>> Regards >>>> Calle >>>> >>>> ******************************************* >>>> >>>> Calle Hedberg >>>> >>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA >>>> >>>> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472> >>>> >>>> Cell: +27-82-853-5352 <+27%2082%20853%205352> >>>> >>>> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119> >>>> >>>> Email: calle.hedb...@gmail.com >>>> >>>> Skype: calle_hedberg >>>> >>>> ******************************************* >>>> >>>> >>>> _______________________________________________ >>>> Mailing list: https://launchpad.net/~dhis2-devs >>>> Post to : dhis2-devs@lists.launchpad.net >>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>> More help : https://help.launchpad.net/ListHelp >>>> >>>> >> >> >> -- >> >> ******************************************* >> >> Calle Hedberg >> >> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA >> >> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472> >> >> Cell: +27-82-853-5352 <+27%2082%20853%205352> >> >> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119> >> >> Email: calle.hedb...@gmail.com >> >> Skype: calle_hedberg >> >> ******************************************* >> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~dhis2-devs >> Post to : dhis2-devs@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~dhis2-devs >> More help : https://help.launchpad.net/ListHelp >> >> > > > -- > > > *David Huser* > DHIS2 Support Specialist > dhu...@baosystems.com | https://baosystems.com | Skype: dafhus | 2900 K > Street, Suite 406, Washington D.C. 20007 > -- ******************************************* Calle Hedberg 46D Alma Road, 7700 Rosebank, SOUTH AFRICA Tel/fax (home): +27-21-685-6472 Cell: +27-82-853-5352 Iridium SatPhone: +8816-315-19119 Email: calle.hedb...@gmail.com Skype: calle_hedberg *******************************************
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp