Hi Lars, thanks for the hint. Currently I am just running on standards. I'll do a bit of monitoring first before adjusting those values. I'll post the difference.
Regards, Uwe --- > Lars Helge Øverland <larshe...@gmail.com> hat am 2. Februar 2016 um 23:43 > geschrieben: > > > Hi Uwe, > > make sure that you have tuned Postgres properly through postgresql.conf, > especially the last 5 settings are crucial for getting good write > performance. > > http://dhis2.github.io/dhis2-docs/master/en/implementer/html/ch08s03.html#d5e464 > > checkpoint_segments = 32 > > PostgreSQL writes new transactions to a log file called WAL segments which > are 16MB in size. When a number of segments have been written a checkpoint > occurs. Setting this number to a larger value will thus improve performance > for write-heavy systems such as DHIS 2. > > checkpoint_completion_target = 0.8 > > Determines the percentage of segment completion before a checkpoint occurs. > Setting this to a high value will thus spread the writes out and lower the > average write overhead. > > wal_buffers = 16MB > > Sets the memory used for buffering during the WAL write process. Increasing > this value might improve throughput in write-heavy systems. > > synchronous_commit = off > > Specifies whether transaction commits will wait for WAL records to be > written to the disk before returning to the client or not. Setting this to > off will improve performance considerably. It also implies that there is a > slight delay between the transaction is reported successful to the client > and it actually being safe, but the database state cannot be corrupted and > this is a good alternative for performance-intensive and write-heavy > systems like DHIS 2. > > wal_writer_delay = 10000ms > > Specifies the delay between WAL write operations. Setting this to a high > value will improve performance on write-heavy systems since potentially > many write operations can be executed within a single flush to disk. > > > > > > > > On Tue, Feb 2, 2016 at 9:18 PM, Bob Jolliffe <bobjolli...@gmail.com> wrote: > > > Lars is right that ADX won't be faster than dxf. Both because it > > internally converts to dxf on import and because it abstracts away the > > categoryoptioncombo. The first isn't really very costly but the other > > is. > > > > This means that that the two systems only have to match categories and > > categoryoptions which is a much easier mapping to maintain. > > > > But if you need raw speed it is going to be faster to produce dxf > > style categoryoptioncombos as that is closest to the way the data gets > > stored. I am going to speed up the adx import code, but will still > > always be slower :-) > > > > On 2 February 2016 at 20:07, uwe wahser <u...@wahser.de> wrote: > > > Ok, that sounds like ADX might even be a bit slower eventually, if the > > > transformation process outweighs a potentially reduced datavolume. I > > might > > > just stick with the json. > > > > > > @Jason: I also thought about SQL-Injection shortly, but I am fearing > > > internal changes of the data-model, which I'd have to understand fully in > > > the first place. Of course the api's also change more than I expected, > > but > > > at least that is announced :-D > > > > > > Uwe > > > > > > --- > > > > > > > > > > > > Am 02.02.2016 um 19:49 schrieb Lars Helge Øverland: > > > > > > Hi Uwe, > > > > > > ADX will not be faster than DXF, as for ADX, the stream is first > > converted > > > into DXF and then passed on to the regular importer. > > > > > > Lars > > > > > > On Tue, Feb 2, 2016 at 5:33 PM, Jason Pickering > > > <jason.p.picker...@gmail.com> wrote: > > >> > > >> This was a very trivial lab test,so not really conclusive at all. I > > would > > >> just give it a try and see. If you see differences, please let the devs > > >> know. > > >> > > >> Given the scale of what you are attempting, have you considered using > > >> direct SQL injection? Not that I am recommending that route as there are > > >> many pitfalls, but it might be an option if implemented properly, > > especially > > >> considering your reported architecture. > > >> > > >> Regards > > >> Jason > > >> > > >> > > >> On Tue, Feb 2, 2016, 17:04 Uwe Wahser <u...@wahser.de> wrote: > > >>> > > >>> Hi Jason, > > >>> > > >>> thanks for sharing the links. As I can see on a quick glance, you are > > >>> also > > >>> experimenting with the ADX-api - did you observe any significant > > >>> performance > > >>> differences between ADX and dataValueSets apis? > > >>> > > >>> Regards, > > >>> > > >>> Uwe > > >>> > > >>> > Jason Pickering <jason.p.picker...@gmail.com> hat am 2. Februar > > 2016 um > > >>> > 18:21 > > >>> > geschrieben: > > >>> > > > >>> > > > >>> > Hi Olav, > > >>> > I have not worked with the DHS API per se, but have imported lots of > > >>> > data > > >>> > using the same approach which they outline here ( > > >>> > http://api.dhsprogram.com/#/samples-r.cfm) > > >>> > > > >>> > I have written up a walkthrough of getting data out of one DHIS > > >>> > instance > > >>> > and into another one, and I think the basic principles would be the > > >>> > same ( > > >>> > http://rpubs.com/jason_p_pickering/139589) > > >>> > > > >>> > Metadata needs to be mapped (or created), the data needs to be > > >>> > reshaped, > > >>> > and correctly formatted. > > >>> > > > >>> > It should not be too difficult. I used R, but there are other > > examples > > >>> > with > > >>> > Python and JavaScript on their examples page. > > >>> > > > >>> > Regards, > > >>> > Jason > > >>> > > > >>> > > > >>> > On Tue, Feb 2, 2016 at 3:31 PM, Alex Tumwesigye < > > atumwesi...@gmail.com> > > >>> > wrote: > > >>> > > > >>> > > Dear Uwe, > > >>> > > > > >>> > > Have you tried to send data via the endpoint api/dataValueSets, it > > >>> > > may be > > >>> > > faster. Just stage your data and push it once. > > >>> > > > > >>> > > > > >>> > > > > http://dhis2.github.io/dhis2-docs/master/en/developer/html/ch01s13.html#d5e1372 > > >>> > > > > >>> > > Also to note, is how you send it, I have seen curl taking ages to > > >>> > > submit > > >>> > > individual values via the api. You need to send it as once file via > > >>> > > once > > >>> > > request or implement concurrency. > > >>> > > > > >>> > > Alex > > >>> > > > > >>> > > On Tue, Feb 2, 2016 at 5:13 PM, Olav Poppe <olav.po...@me.com> > > wrote: > > >>> > > > > >>> > >> Hi Randy and Uwe, > > >>> > >> thanks, interesting to hear you experiences. Uwe, what you are > > >>> > >> working on > > >>> > >> sounds quite a bit more complicated, and not least with far more > > >>> > >> data. I > > >>> > >> image that with household surveys, it would be a matter of < 100 > > >>> > >> indicators > > >>> > >> for < 200 orgunits for 2-3 periods, i.e. a fraction of what you > > are > > >>> > >> dealing > > >>> > >> with! > > >>> > >> > > >>> > >> Olav > > >>> > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> > >> 31. jan. 2016 kl. 09.29 skrev uwe wahser <u...@wahser.de>: > > >>> > >> > > >>> > >> Hi Olav & Randy, > > >>> > >> > > >>> > >> I am currently banging on kettle (aka Pentaho DI) to extract data > > >>> > >> from a > > >>> > >> source-system (SQL-ERP in our case) into DHIS2 dataSets in json > > >>> > >> format. In > > >>> > >> our current test-scenario (2 dataElements in a dataSet with a > > >>> > >> categoryCombination of 5 categories) we are currently updating > > ca. 4 > > >>> > >> mio > > >>> > >> dataValues every night in a pseudo-delta mode (reading all data > > from > > >>> > >> source, comparing to what is there in DHIS2 already, then only > > >>> > >> pushing > > >>> > >> records for creating, updating or deleting dataValues into the > > api: > > >>> > >> ca. > > >>> > >> 150k per night in 1 hour, initial load was 7hrs). We still have to > > >>> > >> prove, > > >>> > >> that this is feasible when setting up the first real life dataSet > > >>> > >> where > > >>> > >> there will be more categories and more dataElements, thus > > exploding > > >>> > >> the > > >>> > >> number of dataValues. > > >>> > >> > > >>> > >> Getting there was a bit painful, but now it seems to work. I chose > > >>> > >> kettle > > >>> > >> instead of Talend ETL (both open source) as it seemed to be easier > > >>> > >> to get > > >>> > >> used to. However, from a data warehouse perspective I'd prefer to > > >>> > >> have > > >>> > >> DHIS2 offering some sort of an integrated ETL landscape on the > > long > > >>> > >> run, > > >>> > >> which would also allow to aggregate data from tracker into > > dataSets, > > >>> > >> tracker to tracker, dataSets to dataSets etc. > > >>> > >> > > >>> > >> Our current version of the kettle transformations and jobs were > > >>> > >> designed > > >>> > >> to be generic (not for a specific dataSet, but you have to design > > >>> > >> your own > > >>> > >> extractor which could be a simple csv-reader or maybe a DHS > > >>> > >> api-call). If > > >>> > >> you are interested, I will share them. Just be aware that they are > > >>> > >> currently in a very early and rough state and not documented. > > You'd > > >>> > >> have to > > >>> > >> bring along the willingness to dig yourself into kettle and be > > pain > > >>> > >> resistant to a certain degree :-) > > >>> > >> > > >>> > >> I'd be interested to hear from other experiences ... > > >>> > >> > > >>> > >> Have a nice sunday, > > >>> > >> > > >>> > >> Uwe > > >>> > >> > > >>> > >> --- > > >>> > >> > > >>> > >> Am 29.01.2016 um 17:31 schrieb Wilson, Randy: > > >>> > >> > > >>> > >> Not here unfortunately...just doing csv imports from DHS Excel > > >>> > >> files. > > >>> > >> Would be useful for our data warehouse. > > >>> > >> Randy > > >>> > >> On Jan 29, 2016 2:59 PM, "Olav Poppe" <olav.po...@me.com> wrote: > > >>> > >> > > >>> > >>> Hi all, > > >>> > >>> I wanted to hear if anyone has any experience with the DHS API ( > > >>> > >>> http://api.dhsprogram.com/#/index.html), and using it to import > > >>> > >>> survey > > >>> > >>> results into DHIS? > > >>> > >>> > > >>> > >>> Olav > > >>> > >>> > > >>> > >>> _______________________________________________ > > >>> > >>> Mailing list: https://launchpad.net/~dhis2-users > > >>> > >>> Post to : dhis2-users@lists.launchpad.net > > >>> > >>> Unsubscribe : https://launchpad.net/~dhis2-users > > >>> > >>> More help : https://help.launchpad.net/ListHelp > > >>> > >>> > > >>> > >>> > > >>> > >> *This message and its attachments are confidential and solely for > > >>> > >> the > > >>> > >> intended recipients. If received in error, please delete them and > > >>> > >> notify > > >>> > >> the sender via reply e-mail immediately.* > > >>> > >> > > >>> > >> _______________________________________________ > > >>> > >> Mailing list: https://launchpad.net/~dhis2-users > > >>> > >> Post to : dhis2-users@lists.launchpad.net > > >>> > >> Unsubscribe : https://launchpad.net/~dhis2-users > > >>> > >> More help : https://help.launchpad.net/ListHelp > > >>> > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> > >> _______________________________________________ > > >>> > >> Mailing list: https://launchpad.net/~dhis2-users > > >>> > >> Post to : dhis2-users@lists.launchpad.net > > >>> > >> Unsubscribe : https://launchpad.net/~dhis2-users > > >>> > >> More help : https://help.launchpad.net/ListHelp > > >>> > >> > > >>> > >> > > >>> > > > > >>> > > > > >>> > > -- > > >>> > > Alex Tumwesigye > > >>> > > > > >>> > > Technical Advisor - DHIS2 (Consultant), > > >>> > > Ministry of Health/AFENET > > >>> > > Kampala > > >>> > > Uganda > > >>> > > > > >>> > > IT Consultant - BarefootPower Uganda Ltd, SmartSolar, Kenya > > >>> > > > > >>> > > IT Specialist (Servers, Networks and Security, Health Information > > >>> > > Systems > > >>> > > - DHIS2 ) & Solar Consultant > > >>> > > > > >>> > > +256 774149 775, + 256 759 800161 > > >>> > > > > >>> > > "I don't want to be anything other than what I have been - one tree > > >>> > > hill " > > >>> > > > > >>> > > _______________________________________________ > > >>> > > Mailing list: https://launchpad.net/~dhis2-users > > >>> > > Post to : dhis2-users@lists.launchpad.net > > >>> > > Unsubscribe : https://launchpad.net/~dhis2-users > > >>> > > More help : https://help.launchpad.net/ListHelp > > >>> > > > > >>> > > > > >>> > > > >>> > > > >>> > -- > > >>> > Jason P. Pickering > > >>> > email: jason.p.picker...@gmail.com > > >>> > tel:+46764147049 > > >>> > _______________________________________________ > > >>> > Mailing list: https://launchpad.net/~dhis2-users > > >>> > Post to : dhis2-users@lists.launchpad.net > > >>> > Unsubscribe : https://launchpad.net/~dhis2-users > > >>> > More help : https://help.launchpad.net/ListHelp > > >> > > >> > > >> _______________________________________________ > > >> Mailing list: https://launchpad.net/~dhis2-users > > >> Post to : dhis2-users@lists.launchpad.net > > >> Unsubscribe : https://launchpad.net/~dhis2-users > > >> More help : https://help.launchpad.net/ListHelp > > >> > > > > > > > > > > > > -- > > > Lars Helge Øverland > > > Lead developer, DHIS 2 > > > University of Oslo > > > Skype: larshelgeoverland > > > http://www.dhis2.org > > > > > > > > > > > > _______________________________________________ > > > Mailing list: https://launchpad.net/~dhis2-users > > > Post to : dhis2-users@lists.launchpad.net > > > Unsubscribe : https://launchpad.net/~dhis2-users > > > More help : https://help.launchpad.net/ListHelp > > > > > > > > > -- > Lars Helge Øverland > Lead developer, DHIS 2 > University of Oslo > Skype: larshelgeoverland > http://www.dhis2.org <https://www.dhis2.org> _______________________________________________ Mailing list: https://launchpad.net/~dhis2-users Post to : dhis2-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-users More help : https://help.launchpad.net/ListHelp
