David,
I'm importing into 2.26, postgresql is tuned as far as possible (the test
was on my laptop), and 12GB RAM. I was using CSV, put there should not be
any significant difference between JSON and CSV.
I will import the 2.5 mill events directly and see what time that takes,
comparatively.
Regards
Calle
On 30 June 2017 at 18:05, David Huser wrote:
> Hi Calle
>
> When some conditions are met, the event import can be really speedy
> (imports of ~100 000 in <30 minutes).
>
> - Recent version (2.25+)
> - PostgreSQL properly tuned (this is important)
> - Enough RAM (8 GB+)
>
> I usually use JSON files and post them against /api/events.
>
> David
>
> On Fri, Jun 30, 2017 at 12:58 PM, Calle Hedberg
> wrote:
>
>> Ime,
>>
>> I was using the UI - have not looked at the API endpoint.
>>
>> For now I will use sql scripts as I've done before - I'm just trying to
>> find out if anybody is working on improving what is (now) an import
>> function that in reality only caters for small data sets. There is no
>> question about the core developers ability to speed this up - I saw that
>> with the import of OU meta-data - but not sure if anybody has done anything.
>>
>> If no answers from core developers, the I will write a JIRA issue for it.
>>
>> Regards
>> Calle
>>
>> On 29 June 2017 at 02:25, Ime Asangansi wrote:
>>
>>> Hi Calle,
>>>
>>> Thanks for your question.
>>> Just curious (I don't have answer). As we are planning to do this soon.
>>>
>>> Are you doing this through UI or API?
>>> If API, which endpoint enables this?
>>>
>>> Thanks
>>>
>>> Ime
>>>
>>>
>>>
>>> On Jun 28, 2017 19:15, "Calle Hedberg" wrote:
>>>
Hi
I started importing around 700,000 events 2 days ago (about 5 mill
individual values) - and the import is still running, 48 hours later.
Import is slowing - it seemed to be importing around 7-8 values per second,
now down to 2-3 per second. It looks like that import might take around 200
hours - which makes no sense from a usability point of view. Importing
OrgUnits used to be equally slow, but a revamp recently change a 10,000
OrgUnit import&update from hours to something like 5 seconds.
It seems obvious that event imports have been designed to import values
one by one, which is equivalent to asking people to fetch water using a
leaking tea-spoon.
I've searched JIRA but cannot find any issue related to this, thus this
post: Are there any plans underway to make import of Event data usable for
larger record sets?
Regards
Calle
***
Calle Hedberg
46D Alma Road, 7700 Rosebank, SOUTH AFRICA
Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>
Cell: +27-82-853-5352 <+27%2082%20853%205352>
Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>
Email: calle.hedb...@gmail.com
Skype: calle_hedberg
***
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
>>
>>
>> --
>>
>> ***
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>
>>
>> Cell: +27-82-853-5352 <+27%2082%20853%205352>
>>
>> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>
>>
>> Email: calle.hedb...@gmail.com
>>
>> Skype: calle_hedberg
>>
>> ***
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to : dhis2-devs@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
>
>
> *David Huser*
> DHIS2 Support Specialist
> dhu...@baosystems.com | https://baosystems.com | Skype: dafhus | 2900 K
> Street, Suite 406, Washington D.C. 20007
>
--
***
Calle Hedberg
46D Alma Road, 7700 Rosebank, SOUTH AFRICA
Tel/fax (home): +27-21-685-6472
Cell: +27-82-853-5352
Iridium SatPhone: +8816-315-19119
Email: calle.hedb...@gmail.com
Skype: calle_hedberg
***
___
Mailing list: https://launchpad.net/~dhis2-devs
Post to : dhis2-devs@lists.launchpad.net
Unsubscribe : https://launchpad.net/~dhis2-devs
More help : https://help.launchpad.net/ListHelp
