On Thu, May 27, 2010 at 6:39 PM, Bob Jolliffe <bobjolli...@gmail.com> wrote: > On 27 May 2010 17:27, Knut Staring <knu...@gmail.com> wrote: >> DHIS already has the concept of datamart and report tables, which do >> provide some separation of the transactional from the analytical, >> though we also have plans for improving this. >> >> R-node with Jquery /GeoExt looks interesting: >> http://www.r-statistics.com/2010/04/r-node-a-web-front-end-to-r-with-protovis/ > > Slight aside: protoviz looks like a pretty powerful visualization library ..
Yes! And I like the very nice overview of BI tools and visualizations in this presentation: http://squirelove.net/r-node-extra/r-node-plug-10-04-14.html (from the R-node homepage: http://www.squirelove.net/r-node/doku.php) k >> >> On Thu, May 27, 2010 at 5:10 PM, Jason Pickering >> <jason.p.picker...@gmail.com> wrote: >>> Hi Roger, >>> >>> Valid concerns, but I would assume that the typical use case for R >>> would be that the user would typically only be looking at 1) a view >>> that has been prepared for them or 2) be provided with read only >>> access to selected tables. I would not expect that locks would be a >>> problem in this case. >>> >>> However, your point is well taken. It is therefore I have been >>> tinkering around with luciddb, a column-oriented database, that may be >>> more appropriate for analysis. Which database that should be used is >>> probably a discussion, but the point is that a separation between the >>> "transactional" database that is being used for data entry, and the >>> "analysis" database is a good idea. I would regard this to probably >>> outside the scope of what DHIS is really intended to do. If people >>> need to use tools like R, they will likely as well of being capable of >>> coming up with their own solution. >>> >>> However, this does not exclude that certain simple examples could be >>> built into DHIS. Obviously performance is a concern, but of course, it >>> depends on what you are trying to do. R is incredibly powerful when it >>> comes to producing graphics as I am sure that you are aware, and >>> lightyears ahead of the other components we are using (jPlot I think). >>> So, I would think that the typical use case would be to leverage R, >>> possibly as an extension to DHIS2 for those that need it, for the >>> generation of analysis tables and graphics, that would be beyond the >>> scope of the "basic" package, which is really limited to aggregation. >>> >>> Anyway, just a few more thoughts. >>> >>> Regards, >>> Jason >>> >>> >>> On Thu, May 27, 2010 at 3:39 PM, Friedman, Roger (CDC/OID/NCHHSTP) >>> (CTR) <r...@cdc.gov> wrote: >>>> My concern would be DB performance, there's no telling what kind of locks >>>> R or any other product using odbc/jdbc is going to use. I'm already >>>> worried about simultaneous transactional and reporting use. Have there >>>> been any large-volume performance tests? Has any thought been given to >>>> splitting reporting and data entry between different DB servers? I know >>>> everyone has been focused on getting the distributed DB aspects right, but >>>> assuming universal availability of internet, how would DHIS2 perform on a >>>> single (possibly clustered) national DB server? >>>> >>>> -----Original Message----- >>>> From: dhis2-users-bounces+rdf4=cdc....@lists.launchpad.net >>>> [mailto:dhis2-users-bounces+rdf4=cdc....@lists.launchpad.net] On Behalf Of >>>> Jason Pickering >>>> Sent: Thursday, May 27, 2010 7:03 AM >>>> To: Bob Jolliffe >>>> Cc: dhis2-us...@lists.launchpad.net; dhis2-devs >>>> Subject: Re: [Dhis2-users] [Dhis2-devs] DHIS2 with R >>>> >>>> Yeah, this I guess comes back time and time again, with my some what >>>> uncomfortable relationship with Hibernate and Java. Clearly, we need >>>> to think about how to make certain procedures crossplatform compatible >>>> (cross platform in the sense of working between Postgres/MySQL and >>>> other DBs) with the need to offer advanced analysis capabilities, with >>>> acceptable performance. >>>> >>>> There could be multiple ways of doing it, but in the absense of having >>>> R integrated into DHIS2, I think the most likely shorterm use case >>>> would be just some documentation on how to use the R client with the >>>> DHIS2 database. Perhaps those users that use R over time with DHIS2 >>>> could contribute their procedures, which should be able to be >>>> generalized either with PL/R. >>>> >>>> Of course the difference with using Postgres, is that R procedures can >>>> be embedded as a new language inside the DB. I am not really sure this >>>> is possible with MySQL. This of course reduces the internal overhead >>>> of getting the data out of Postgres, through Java, and into the R >>>> interpreter, but I am not sure really what the impact of this might be >>>> without testing it. >>>> >>>> >>>> >>>> On Thu, May 27, 2010 at 12:26 PM, Bob Jolliffe <bobjolli...@gmail.com> >>>> wrote: >>>>> On 27 May 2010 11:15, Jason Pickering <jason.p.picker...@gmail.com> wrote: >>>>>> Hi Bob, >>>>>> >>>>>> Yes, I suspect that most R users would probably want to do things >>>>>> their own way. It has a rather steep learning curve. :) >>>>>> >>>>>> As for canned R scripts, the best way would probably with with PL/R, a >>>>>> procedural Postgresql language which utilizes R. >>>>>> >>>>>> http://www.joeconway.com/plr/doc/index.html >>>>>> >>>>>> I have done some very basic testing and it seems to work just fine on >>>>>> the server side. >>>>> >>>>> Swings and roundabouts to a certain extent. The main thing is that >>>>> the r scripts are evaluated using the r c library. If they were >>>>> invoked from within java/dhis then I guess data access would be slower >>>>> than from pl/r (we'd need to have a way to get the data to the r >>>>> interpreter), but number crunching would be similar and would also >>>>> work with mysql and friends. Not sure which of these are bigger >>>>> problems in typical/possible scenarios. >>>>> >>>>>> >>>>>> I think they are two separate problems really, but I totally agree, C >>>>>> is likely going to be faster than Java for big operations. However, I >>>>>> do think (as all of you know) that the use of stored procedures (with >>>>>> the wrapper facade type of approach) for certain functions (like >>>>>> aggregation and heavy cross tab operations) would be much better to be >>>>>> executed on the database server as a native stored procedure. >>>>>> >>>>>> Regards, >>>>>> Jason >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, May 27, 2010 at 11:45 AM, Bob Jolliffe <bobjolli...@gmail.com> >>>>>> wrote: >>>>>>> We've talked before about integrating scripting engine (such as R) >>>>>>> into dhis : http://www.rforge.net/rscript/ >>>>>>> >>>>>>> But my guess is that most R users are going to be of a level of >>>>>>> sophistication that they would be most comfortable doing the kind of >>>>>>> thing you describe - conecting directly to db with r client and doing >>>>>>> their stuff. >>>>>>> >>>>>>> OTOH if there were sufficiently useful "canned" dhis R scripts which >>>>>>> could take some number crunching load off the jvm and produce canned >>>>>>> useful analysis then that would be different. >>>>>>> >>>>>>> Sadly I don't know sufficient about R to know. But I sense it ... >>>>>>> >>>>>>> Regards >>>>>>> Bob >>>>>>> >>>>>>> On 27 May 2010 10:08, Jason Pickering <jason.p.picker...@gmail.com> >>>>>>> wrote: >>>>>>>> Hi everyone. I have had a recent question from a user about how DHIS2 >>>>>>>> can be used with R. I am including a trivial example here about how to >>>>>>>> use R as as a client to access data and produce a graph in DHIS2. >>>>>>>> >>>>>>>> Just get a copy of R and install the DBI and RPostregSQL packages with >>>>>>>> >>>>>>>>>install.packages() >>>>>>>> >>>>>>>> >>>>>>>> After that, just connect to the DB, retrieve your data (in this case >>>>>>>> from a report table) and produce a graph. >>>>>>>> >>>>>>>>>library(DBI) >>>>>>>> >>>>>>>>>library(RPostgreSQL) >>>>>>>> >>>>>>>>>drv <- dbDriver("PostgreSQL") >>>>>>>> >>>>>>>>>con <- dbConnect(drv, dbname="dhis2_zm_prod2", user="postgres", >>>>>>>>>password="postgres") >>>>>>>> >>>>>>>>>rs <- dbSendQuery(con, "SELECT * FROM >>>>>>>>>_report_malaria_indicators_district where >>>>>>>> organisationunitid = 3904") >>>>>>>> >>>>>>>>>data <- fetch(rs,n=-1) >>>>>>>> >>>>>>>>>barplot(data$malaria_confirm_incidence, >>>>>>>>>names.arg=as.character(data$periodname), >>>>>>>>>main=as.character(data$organisationunitname[1]),las=2) >>>>>>>> >>>>>>>>>dev.print(png, file="/home/jason/test.png") >>>>>>>> >>>>>>>> Regards, >>>>>>>> Jason >>>>>>>> >>>>>>>> --- >>>>>>>> Jason P. Pickering >>>>>>>> email: jason.p.picker...@gmail.com >>>>>>>> tel:+260968395190 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs >>>>>>>> Post to : dhis2-devs@lists.launchpad.net >>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>>>>>> More help : https://help.launchpad.net/ListHelp >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -- >>>>>> Jason P. Pickering >>>>>> email: jason.p.picker...@gmail.com >>>>>> tel:+260968395190 >>>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> -- >>>> Jason P. Pickering >>>> email: jason.p.picker...@gmail.com >>>> tel:+260968395190 >>>> >>>> _______________________________________________ >>>> Mailing list: https://launchpad.net/~dhis2-users >>>> Post to : dhis2-us...@lists.launchpad.net >>>> Unsubscribe : https://launchpad.net/~dhis2-users >>>> More help : https://help.launchpad.net/ListHelp >>>> >>>> >>>> >>>> _______________________________________________ >>>> Mailing list: https://launchpad.net/~dhis2-users >>>> Post to : dhis2-us...@lists.launchpad.net >>>> Unsubscribe : https://launchpad.net/~dhis2-users >>>> More help : https://help.launchpad.net/ListHelp >>>> >>> >>> >>> >>> -- >>> -- >>> Jason P. Pickering >>> email: jason.p.picker...@gmail.com >>> tel:+260968395190 >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~dhis2-devs >>> Post to : dhis2-devs@lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~dhis2-devs >>> More help : https://help.launchpad.net/ListHelp >>> >> >> >> >> -- >> Cheers, >> Knut Staring >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~dhis2-devs >> Post to : dhis2-devs@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~dhis2-devs >> More help : https://help.launchpad.net/ListHelp >> > -- Cheers, Knut Staring _______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp