[R] What does KalmanRun$states really return?
For Kalman Filter Durbin/Koopman give at p67 equations 4.13: v = y - Z a, F = Z P Z' + H, K = T P Z' / F + H, a[t+1] = T a + K v, P[t+1] = T P L' + R Q R' for P1 = 0, Q=0, T=Z=R=1 that reduces to: v = y - a, F = H, K = H, a[t+1] = a + K v, P[t+1] = 0 (equivalent to exponential moving average, Durbin/Koopman p49) for vector y = c(1,2,3,4,5), H = 0.66 manual calculations using the equations above give a = c(1,1.66,2.55,3.51,4.50). KalmanRun with these parameters gives res$states = (1,1,1,1,1). Looking into the code of arima.c we have at line 109 an equivalent of a[t+1] = anew + Pnew * resid0 / gain where gain = mod$h = H (by line 97), resid0 = y-a = v (by lines 94-96) Since Pnew = 0, then a[t+1] = a, which explains why the computation returns res$states = c(1,1,1,1,1). The help file says "'states', the contemporaneous state estimates", which I assumed to mean 'a' in the equations above. But that assumption does not agree with the numerical results. It also does not agree with the coding(?) as a[t+1] = a + K v differs substantially from a[t+1] = anew + Pnew * resid0 / gain. So, what does 'states' contain? Where did I go wrong? - [[alternative HTML version deleted]] __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
[R] unexpected result from KalmanRun (KalmanLike, StructTS)
(re-formulate, re-send, without html) for vector y = c(1,2,3,4,5), H = 0.66 manual calculations using the equations below give a = c(1,1.66,2.55,3.51,4.50). KalmanRun with these parameters gives res$states = (1,1,1,1,1)! for Kalman Filter Durbin/Koopman give at p67 eqs 4.13: v = y - Z a, F = Z P Z' + H, K = T P Z' / F + H, a[t+1] = T a + K v, P[t+1] = T P L' + R Q R' for P1 = 0, Q=0, T=Z=R=1 that reduces to: v = y - a, F = H, K = H, a[t+1] = a + K v, P[t+1] = 0 (also equivalent to exponential moving average, Durbin/Koopman p49) So I am getting a serious discrepancy between manual and KalmanRun computations. To make things more interesting, looking into the code of arima.c we have at line 109 an equivalent of a[t+1] = anew + Pnew * resid0 / gain where gain = mod$h = H (by line 97), resid0 = y-a = v (by lines 94-96) Since Pnew = 0, then a[t+1] = a, which explains why the computation returns res$states = c(1,1,1,1,1). The help file says "'states', the contemporaneous state estimates", which I assumed to mean 'a' in the equations above. But that assumption does not agree with the numerical results. It also does not agree with the coding(?) as a[t+1] = a + K v differs substantially from a[t+1] = anew + Pnew * resid0 / gain. (all the previous lines of coding follow the kalman filter equations, but this one does not seem to - do we have some strong reformulation of the equations in the basic form?). So, what does 'states' contain? __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
[R] R and databases - a comment
1. That was a part of a private email exchange. It has been suggested that more people may be interested. 2. I did use various databases (significant part of my job) for the last 15 years. Some with R for the last 3 years as a hobby. Some comments on the ones used below. Sorry, no links - I am time-constrained at the moment - please google if interested in details. The remarks are from the point of view of R user, not that of 'general database user'. 3. SQLITE. www.sqlite.org - probably the best datase to use with R. No setup, no administration, embedded - so less connection overhead. All data in one file - so easy to transfer. Solid. Very functional SQL, fast if you play it right (almost as fast as SQLServer on Windows...) . Some limitations - no stored procedures. Some preprocessing/parsing can be done using TCL - well integrated with sqlite if you need that. Due to the implementation quirk you can even compute recursive functions (like exponential moving average or Fibonacci numbers) with SQL :-). Easy import/export of data to text files. After trying few other dbs I settled down on this one. Even considered writing a tutorial on SQLite use with R (like how to process gigabytes of data on a 128mb computer :-) ) - but time constraints stopped me. [Personally I think that SQLite should come bundled with the standard R installation. Could even be used to keep a lot of R's internal stuff, would probably simplify overall coding. But that is for others to decide] All other databases (including mysql) require typical setup - installation, administration, user rights, keeping track of ports, services/daemons, directories, backups etc - so some db administrative skills are required.I am not sure how many R users are willing to go through that. The ones who may be interested in the stuff below 4. www.postgres.org Postgres. Free. As complete as one can wish, small download, great functionality. Interfaces well to other languages, so you can do numerics in C++ and store that in the database (though why not do numerics in R?). Current version 8.1, much improved. 5. Firebird. open source verion of Interbase. Easy setup and can have all data in one file. But... slow development - not many developers there. SQL full but somewhat quirky (when porting from other dialects). 6. Mysql. the inheritance from the original ISAM system still shows. Nice user interface, but... if you need real db why not use postgres? if you need something simpler, without administration, why not use SQLITE? No doubt mysql is fine for many simple websites etc - this is mysql's niche. 7. derby and hsqldb. both are written in Java, open source. HSQLDB (used now by OpenOffice) allows creation of in-memory tables and it's fast there - but it's usage from inside R is tricky - there is no easily available, installable and current ODBC driver. Similar for derby - the ODBC driver is there, but installation can be tricky to non-professionals. May be in the future... There are three 'express' versions of commercial databases. They all share some restrictions, like max disc data size 2-4gb, max mem size 1-2gb and usage of single processor only. Plus various licensing restrictions, so be careful how you use them. - Microsoft - in beta now, over 100mb download (windows only) (the old version, MSDE, is also available) - Oracle - 150mb download, if i remember correctly even free to distribute, but check the license - DB2 - 500mb download, currently 90 day version, IBM strong rumour is that early next year the new version will be free. Each commercial DB has some OLAP capability, but I am not sure how much of it is/will be available in the Express version. __ Just $16.99/mo. or less. dsl.yahoo.com __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] Reasons to Use R (no memory limitations :-))
This thread discussed R memory limitations, compared handling with S and SAS. Since I routinely use R to process multi-gigababyte sets on computers with sometimes 256mb of memory - here are some comments on that. Most memory limitations vanish if R is used with any relational database. [My personal preference is SQLite (RSQLite packaga) because of speed and no-admin (used in embedded mode)]. The comments below apply to any relational database, unless otherwise stated. Most people appear to think about database tables as dataframes - that is to store and load the _whole_ dataframe in one go - probably because appropriate function names are suggesting this approach. Also, it is a natural mapping. This is convenient if the data set can fit fully in memory - but limits the size of the data set the same way as without using the database. However, using SQL language directly one can expand the size of the data set R is capable of operating on - we just have to stop treating database tables as 'atomic'. For example, assume we have a set of several million patients and want to analyze some specific subset - the following SQL statement SELECT * FROM patients WHERE gender='M" AND AGE BETWEEN 30 AND 35 will result in bringing to R much smaller dataframe than selection of the whole table. [Also, such subset selection may take _less_time_ then selecting from the total dataframe - assuming the table is properly indexed]. Also, direct SQL statements can be used to pre-compute some characteristics internally in the database and bring only the summaries to R: SELECT AVG(age) FROM patients GROUP BY gender will bring a data frame of two rows only. Admittedly, if the data set is really large and we cannot operate on its subsets, the above does not help. Though I do not believe that this would the the majority of the situations. Naturally, going for a 64bit system with enough memory will solve some problems without using the database - but not all of them. Relational databases can be very efficient at selecting subsets as they do not have to do linear scans [when the tables are indexed] - while R has to do a linear scan every time(??? I did not look up the source code of R - please correct me if I am wrong). Two other areas where a database is better than R, especially for large data sets: - verification of data correctness for individual points [a frequent problem with large data sets] - combining data from several different types of tables into one dataframe In summary: using SQL from R allows to process extremely large data sets in a limited memory, sometimes even faster then if we had a large memory and kept our data set fully in it. Relational database perfectly complements R capabilities. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] how-to: installing R on Knoppix
Installing R on Knoppix Knoppix is a small LINUX distribution (http://www.knoppix.net/) runnable from a CD (without an installation) It can be also installed on a hard drive from a single CD. The idea was to reuse an old laptop with 6gb hard drive and 128mb memory. I had few days experience with Linux and few years experience with R. The latter was not required :-) The Knoppix installation went on flawlessly - using defaults. Partitioning required rerun, as I preferred to have a separate swap partition and a separate boot partition. Putting everything in one big partition would be less fussy, but I wanted to experiment. Important: I found several description on how to install Knoppix onto a hard drive, but all of them were too long - so I used http://www.aims.ac.za/resources/tutorials/install/knoppix.php. The longer descriptions may be useful when you try to install Knoppix on a computer to run in parallel with other systems or for older Knoppix versions. After installing Knoppix looked very impressively - neat graphics, a lot of apps (including OpenOffice) KDE windowing system. All of that from one CD - version 4.0.2. This is a really neat-looking system - even OpenOffice 2.0 (beta) was there. The laptop was not connectable to the Internet, so I had to transfer the packages through an USB drive. To use the usb drive I created first a mount point mkdir /mnt/usbhd then I mounted the usb drive mount /dev/sda1 /mnt/usbhd On my uSB drive I had a package downloaded from CRAN, debian distribution (Knoppix is 'related' to Debian so I assumed there will be less problems when using Debian's packages - I did not try rpms). As per my mount point I tried to install it as dpkg -i /mnt/usbhd/r-base-core_2.1.1-0woody1_i386.deb which, naturally, resulted in multiple dependency errors, because many of the libraries were missing. After some trial and error I found that following packages need to be installed: dpkg -i /mnt/usbhd/gcc-3.4-base_3.4.5-3_i386.deb dpkg -i /mnt/usbhd/libg2c0_3.4.5-3_i386.deb dpkg -i /mnt/usbhd/f2c_20020621-3.4_i386.deb dpkg -i /mnt/usbhd/libpng2_1.0.18-1_all.deb dpkg -i /mnt/usbhd/atlas2-base_3.2.1ln-15_i386.deb dpkg -i /mnt/usbhd/tcl8.3_8.3.5-5_i386.deb dpkg -i /mnt/usbhd/tk8.3_8.3.5-4_i386.deb dpkg -i /mnt/usbhd/zlib-bin_1.2.3-9_i386.deb after that I rerun the original dpkg -i /mnt/usbhd/r-base-core_2.1.1-0woody1_i386.deb which went through without problems then I changed to directory cd /usr/bin and executed R no problems. I did not have time to test installation of other R packages - that was just the core. Observations: 128mb memory is not sufficient - KDE uses much more. Most likely for a computer like that much better approach would be to use Damn Small Linux - a 50mb distribution, descending from Knoppix I did not try to install R there, but DSL - when booted - takes only 30mb of memory, leaving plenty for R computations. I must try it one weekend... That's all. I am no LINUX expert - last 10 years I did spend on Windows (though earlier I did work on UNIX). Installation was rather easy - except for hunting for all these packages. While all the dependencies are stated, it is rather tedious to hunt them down. P.S. Tried to install Fedora Core 3 on that laptop - took forever and was very slow afterwards. Did not look as well as Knoppix either. I did not even try to install FC4, as it requires twice more memory than the laptop has. P.P.S. Naturally, with larger memories and computers Knoppix may not be the best solution - large distributions like RedHat, FC, Ubuntu, Debian, or Scientific Linux may be preferable and the installation of R would be trouble-free. This was an attempt to revitalize and old and tired computer. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Re: [R] how-to: installing R on Knoppix
After some sleep and with clearer head I'd better explain the background of my posting. Going back to UNIX from Win I spent whole day trying out various distros from available CDs (no time to dowload the stuff). Knoppix was the last one and the cutest. So I decided to put R on it. The other part was trying to establish (my hobby) how an old laptop with nonexpandable memory and limited disk space can be revitilized. This is why the R part of the test was so ... contradictory in nature... I grabbed whatever was available and tried to check out how did UNIX change in the last 10 years :-) - and it did a lot. This is why I did not looked through R mailings before embarking on that - apologies for posting something already known. That mail was just a summary of a long evening of testing. Still - I am not totally repentant, because I got the Quantian thing out of your answers - didn't run into it before - so many thanks for that. [I use R to process computer performance logs in batch mode, stats are simple there and all are in the R core]. Some other things I noticed - W2K I had to remove Win2K from that laptop. Before that I managed to drop memory usage there to 43mb after startup by disabling unused services, startups etc (this laptop is not networked nor internettable). DSL used (displayed) 30mb after startup. That was rather surprising to me - always thought of W2K as a big operating system, yet difference in memory with DSL was only 12 mb. Naturally, some of that could have been caused by different default size of the system cache for both OSes, but I did not analyse it. Debian. We just got the info that my paid job moves me to RedHat (EL/AS) (because of hosting support) so it looks like I won't be continuing with that experimenting for a while. It was fun and I did learn something about package managers on LINUX. Having said that - something tells me that I may try this weekend to try to install R on DSL 2.2. Any pointers welcomed :-) P.S. Can I use apt-get from Windows? Only Win at home. Need to buy another computer next week... P.P.S. I had a look at your site - amazing work to put all that stuff in one big release. And on Knoppix :-) too! --- Dirk Eddelbuettel <[EMAIL PROTECTED]> wrote: > > Hi Charles, > > Nice iniative for Linux newbie :) Some of us may > have done a few things > differently. I'll just point out one or two below. > If you want to continue > this discussion, why don't we carry it over to the > r-sig-debian list that is > more appropriate? I'll set the reply-to header > accordingly. > > On 5 March 2006 at 03:47, charles loboz wrote: > | On my uSB drive I had a package downloaded from > CRAN, > | debian distribution (Knoppix is 'related' to > Debian so > | I assumed there will be less problems when using > | Debian's packages - I did not try rpms). > | As per my mount point I tried to install it as > |dpkg -i > | /mnt/usbhd/r-base-core_2.1.1-0woody1_i386.deb > > Well, 2.1.1. is one release behind -- and we even > offer you the current one, > i.e. R 2.2.1, as a backport as is explained in the R > FAQ itself (with thanks to > Christian for building the backport of my Debian R > packages). > > And in this particular case, I think you could also > have installed R 2.2.1 > straight from Debian testing as well as the Knoppix > 4.0.2 libraries should be > current. > > | which, naturally, resulted in multiple dependency > | errors, because many of the libraries were > missing. > | After some trial and error I found that following > > Now that is a feasible strategy, but unfortunately > the most laborious one. > Debian has truly excellent package management and > dependency resolution: the > apt-get program is designed to do just that for you. > You said you're new at > Linux --- but I think you will find time spend on > reading the documentation [ > you'll have to google for suitable intros ] to be a > most excellent > investment. > > | Observations: 128mb memory is not sufficient - KDE > | uses much more. Most likely for a computer like > that > | much better approach would be to use Damn Small > Linux > | - a 50mb distribution, descending from Knoppix I > did > > Gabor and I had some off-line chats a few weeks back > and he had success with > Damn Small Linux running _in parallel_ to his > Windows installation using > coLinux. I like running Quantian [ my Knoppix > derivative that adds a few gb > of software to Knoppix, see > dirk.eddelbuettel.com/quantian ] using VMware > Player running alongside my normal Linux (home) -- > or Windows (work) -- > workstation. In either case, more memory is always > better and 128mb is good > for very few things these days. > > | insta
[R] Installing Damn Small Linux and R
Installing Damn Small Linux and R Following my experiences with Knoppix (described in another mail) I did similar test on Damn Small Linux (dsl -http://www.damnsmalllinux.org/ ). DSL was interesting, because it is the smallest yet fully functional version of Linux, hence more suited to older and smaller computers than Knoppix. Importantly, it uses lightweight windowing system (no KDE or GNOME) so its memory requirements are much smaller. Also, it seems to have a lively user community, which saved me a lot of work. DSL can be used as 'live CD' but can also be installed on a hard disk - I followed the instructions on http://www.damnsmalllinux.org/dsl-hd-install.html , using the latest DSL 2.2b version . (I installed it to an empty machine, so no dual-boot considerations were important). Upon startup the memory usage reported was 20mb (not 30 as I mentioned in my Knoppix article). Trying to follow the same way of installing R as with Knoppix I discovered that many more libraries are missing or are older in DSL than in Knoppix. There is a list of packages already prepared for installation on DSL at http://distro.ibiblio.org/pub/linux/distributions/damnsmall/mydsl/ and to my joy one of them (under 'applications') was R.tar.gz (version 2.0.1) Using identical procedure as described in the Knoppix article I transferred that through an USB drive to /opt directory, then executed tar -xvzf R.tar.gz It created /opt/R directory. In bin directory I executed R shell script and it simply worked. After that I decided to get adventerous and, from the same place (under 'system'), I downloaded tcltk8.3.dsl . That is a special packaging for DSL extensions and http://www.damnsmalllinux.org/wiki/index.php/Installing_MyDSL_Extensions describes how to install them. Actually it is as simple as opening the file manager EmelFM , higlighting the tcltk8.3 file and pressing the button 'MyDSL'. After that I could run Rtcltk script, which started the R GUI. Memory usage was 40mb at that stage. The whole process took less than one hour and at the end of it I had fully functional R installation, including GUI. And enough memory spare (80mb) for a lot of calculations. P.S. I tested also OpenOffice.uci - a mountable version of OpenOffice 2.0 and it started and worked without problems. The memory usage after starting Writer was around 60mb (R was not running at that time - just the Writer). I expected more, following dire warnings on the DSL site. P.P.S. Judging by these numbers there is a possiblity that with DSL, any old computer with 64mb memory and 1gb hard drive can comfortably run R and OpenOffice. The interface is not so cute as in Knoppix, but for a working computer it quite comfortable and rather fast. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
[R] KalmanXXXX and deJong-Penzer statistic?
A question about: Kalman in R, time series and deJong-Penzer statistic - how to compute it using available artefacts of KalmanX? Background. in the paper http://www.lse.ac.uk/collections/statistics/documents/researchreport34.pdf 'Diagnosing Shocks in TIme Series', de Jong and Penzer construct a statistic (tau) which can be used to locate potential shocks. [p15, Theorem 6.1 and below]. They also state that all the components of that statistic (v_i, F_i, r_i, N_i) 'are computed with Kalman Filter Smoother applied to the null model'. Also, as I understand, that part has been implemented in one of the S packages , SsfPack, as the book on that states on p 531 'the standardized smoothed disturbances may be interpreted as t-statistics for impulse intervention variable in the transition and measurement equations.' and equations for the statistic are: eta_t / sqrt(var(eta_t), where eta_t and var have hats over them. The second equation is identical, with 'eta' replaced by 'epsilon'. On page 524 we also have: "the smoothed disturbance estimates are the estimates of the measurement equation innovations epsilon and transition equation innovations eta based on all available information Y. ... the computation of hat(eta) and hat(epsilon) from the Kalman smoother algorithm is described in Durbin and Koopman chapter 7, 'Time series analysis by state space methods', OUP (2001) " Local libraries do not have this book and it will take several weeks to get it. Assuming I will get the book: does the KalmanXXX set of functions produce all the necessary artefacts to compute this statistic either as per deJong-Penzer or as per SsfPack? Reading carefully through the manual I see that we have artefacts of states and normalized residuals (presumably of states - but how can I unscale them if I need them)? What about other stats? How to compute smoothed disturbance estimates? I am rather confused - that's my first approach to Kalman filter and state models. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
[R] How can I change SQLite cache size for R session?
How can I change SQLite cache size for R session? SQLite cache size can be set by a pragma, for the duration of the session - or by default. .pragma cache_size .pragma default_cache_size my questions are about RSQLite, version 0.4, running on Windows: - what is the cache size set to when SQLite is invoked from R? - if the page_size is set to 4096 what will be the cache size in bytes when cache_size 1000 is specified? (in other words: does R know about the page size setting?) P.S. Very impressed with SQLite and its (embedded) integration with R - using it fully really simplifies a lot of programming as well as moving computations between computers. It is also, for my current needs, much faster than postgress and much less cumbersome in operation (permissions, administration, mobility). The price to pay - need to be careful with crafting SQL - no optimizer here. However, controlling cache size is an important thing for any database and I would like to know how to do it for R. Documentation does not mention it and quick scan of the source code of RSQLite did not show any obvious comments. __ R-help@stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
[R] string syntactic sugar in R? - long post
Currently in R, constructing a string containing
values of variables is done using 'paste' and can be
an error-prone and traumatic experience. For example,
when constructing a db query we have to write,
paste("SELECT " value " FROM table where
date ='",cdate,"'")
we are getting null result from it, because without
(forgotten...) sep="" we get
SELECT value FROM table where date='
2005-05-05 '
instead of
SELECT value FROM table where date='2005-05-05'
Adding sep="" as a habit results in other errors, like
column names joined with keywords - because of
forgotten spaces. Not to mention mixing up or
unbalancing quote marks etc. The approach used by
paste is similar to that of many other languages (like
early Java, VB etc) and is inherently error-prone
because of poor visualization. There is a way to
improve it.
In the Java world gstrings were introduced
specifically for this purpose. A gstring is a string
with variable names embedded and replaced by values
(converted to strings, lazy eval) before use. An
example in R-syntax would be:
>alpha <- 8; beta="xyz"
>gstr <- "the result is ${alpha} with the comment
${beta}"
>cat(gstr)
the result is 8 with the comment xyz
This syntactic sugar reduces significantly the number
of mistakes made with normal string concatenations.
Gstrings are used in ant and groovy - (for details see
http://groovy.codehaus.org/Strings, jump to GStrings).
They are particularly useful for creating readable and
error-free SQL statements, but obviously the simplify
'normal' string+value handling in all situations. [ps:
gstrings are not nestable]
I was wondering how difficult it would be to add such
syntactic sugar to R and would that create some
language problems? May be it is possible that it could
be done as some gpaste function, parsing the argument
for ${var}, extracting variables from the environment,
evaluating them and producing the final string?
I admit my bias - using ant for years and groovy for
months and having to do a lot of SQL queries does not
put me in the mainstream of R users - so it may be
that this idea is not usable to a wider group of
users.
__
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
