I did try the log version as well prior to posting but although it would seem to exaggerate the difference to me the insights from plotting the raw data with loess (i.e. constancy of the total, piecewise constant growth of R) come through best.
On Wed, Jan 7, 2009 at 6:53 PM, Spencer Graves <spencer.gra...@prodsyse.com> wrote: > Thanks, Gabor, Marc, Max: > The image is even more striking (and more accurately reflects reality, I > believe) if you add "log='y'" to "ts.plot". > Best Wishes, > Spencer > > Gabor Grothendieck wrote: >> >> Here is the same number of messages/posts data >> for each of S, SAS, R: >> - reworked into a 3 column ts class time series >> - with Jan 2009 removed since its not complete >> - leading and trailing NA rows removed >> >> At end we plot the raw data as well as the time >> series of totals and show loess smooths for each. >> >> By running the code below we see that the: >> - sum of the three seems to be rising at a constant rate >> - S is declining >> - SAS and R are rising >> - R is rising the fastest through its completed its phase >> of highest growth which ended around 2004 >> >> tt3 <- structure(c(15, 458, 330, 219, 472, 517, 546, 511, 658, 681, >> 712, 751, 763, 975, 703, 805, 752, 666, 548, 734, 963, 792, 945, >> 1002, 775, 969, 745, 691, 773, 765, 853, 1024, 805, 1052, 1163, >> 999, 1184, 1053, 1176, 1197, 911, 844, 1007, 1150, 1108, 1315, >> 1212, 1127, 1074, 692, 947, 900, 853, 677, 894, 1068, 945, 784, >> 448, 813, 896, 823, 894, 1129, 733, 492, 514, 493, 659, 1077, >> 778, 540, 476, 612, 1351, 1708, 1720, 1595, 1720, 1519, 1177, >> 1163, 1963, 1615, 1572, 1696, 1491, 1669, 1490, 1298, 1826, 1537, >> 1915, 1467, 1735, 1905, 2027, 1976, 1439, 1592, 1636, 1424, 1941, >> 1845, 2010, 2199, 2373, 2133, 2445, 1492, 1864, 2133, 1663, 1520, >> 1832, 1846, 1755, 1757, 1863, 1701, 1926, 1689, 1646, 1832, 1545, >> 1445, 1636, 1652, 2188, 1826, 1836, 2606, 1843, 2143, 1784, 1712, >> 1786, 2148, 2122, 1960, 629, 2169, 2283, 2407, 2061, 1793, 1365, >> 1427, 1518, 1524, 2722, 1645, 1711, 2796, 3147, 2723, 761, 2027, >> 2714, 2983, 2848, 2374, 2750, 926, 1728, 2766, 2974, 2691, 2435, >> 2592, 1868, 2320, 2112, 1948, 2305, 2255, 2712, 2789, 2025, 2368, >> 2607, 2584, 2554, 2434, 1984, 1921, NA, NA, NA, NA, NA, NA, NA, >> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, >> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, >> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, >> 273, 378, 293, 330, 243, 219, 209, 191, 241, 181, 141, 210, 173, >> 313, 300, 334, 254, 284, 270, 300, 253, 300, 194, 264, 313, 285, >> 264, 306, 247, 245, 302, 204, 251, 261, 176, 246, 232, 252, 300, >> 331, 282, 258, 260, 260, 229, 232, 194, 230, 255, 242, 228, 219, >> 248, 230, 207, 221, 280, 228, 177, 189, 179, 218, 196, 189, 217, >> 221, 187, 186, 295, 197, 142, 197, 230, 257, 151, 164, 175, 154, >> 187, 195, 150, 176, 176, 174, 161, 193, 182, 174, 109, 159, 144, >> 107, 98, 82, 84, 109, 87, 99, 123, 107, 96, 84, 97, 68, 73, 53, >> 20, 51, 59, 74, 48, 46, 34, 47, 39, 35, 70, 56, 41, 48, 63, 58, >> 47, 31, 27, 40, 28, 41, 30, 27, 36, NA, NA, NA, NA, NA, NA, NA, >> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, >> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, >> NA, NA, NA, NA, NA, NA, 92, 36, 47, 41, 37, 40, 76, 61, 57, 135, >> 79, 114, 101, 90, 105, 110, 64, 94, 96, 184, 105, 226, 145, 195, >> 189, 161, 186, 184, 148, 203, 231, 318, 221, 205, 355, 377, 377, >> 504, 418, 293, 356, 434, 418, 433, 422, 558, 583, 651, 470, 552, >> 550, 615, 562, 678, 657, 825, 530, 884, 697, 880, 965, 1057, >> 926, 918, 824, 705, 1055, 1038, 742, 1017, 1137, 1203, 1488, >> 1268, 1319, 1344, 1210, 1443, 1567, 1605, 1158, 1116, 1580, 1946, >> 1657, 1561, 1714, 1618, 1493, 1534, 1712, 1895, 1481, 1746, 1724, >> 1703, 2057, 1887, 2056, 1872, 1777, 1709, 1810, 1907, 1508, 2075, >> 1920, 2270, 1818, 2029, 1811, 1785, 1898, 1902, 2328, 2127, 1450, >> 1714, 1907, 2191, 2145, 2210, 2307, 2138, 2241, 2028, 2708, 2594, >> 2028, 2490, 2583, 2740, 2487, 2517, 2774, 3268, 2813, 2990, 3037, >> 2730, 2399), .Dim = c(186L, 3L), .Dimnames = list(NULL, c("SAS", >> "S", "R")), .Tsp = c(1993.5, 2008.91666666667, 12), class = c("mts", >> "ts")) >> >> tt4 <- cbind(tt3, rowSums(tt3)) >> colnames(tt4) <- c(colnames(tt3), "Sum") >> ts.plot(tt4, col = 1:4) >> grid() >> legend("topleft", colnames(tt4), lty = 1, col = 1:4) >> >> library(dyn) >> for(i in 1:4) lines(fitted(dyn$loess(tt4[, i] ~ time(tt4))), col = i) >> >> >> On Wed, Jan 7, 2009 at 3:07 PM, Marc Schwartz <marc_schwa...@comcast.net> >> wrote: >> >>> >>> on 01/07/2009 09:29 AM Max Kuhn wrote: >>> >>>>> >>>>> "You can look on the SAS message boards and see there is a proportional >>>>> downturn in traffic." >>>>> >>>> >>>> I think that I actually made this statement about both the SAS and >>>> Splus traffic... >>>> >>>> I wasn't really trying to be critical of SAS. I was trying to get >>>> across that SAS focused their resources on features that had nothing >>>> to do with *statistical analysis* (e.g. data warehousing etc.) >>>> >>> >>> Presuming that the Google Groups archive of SAS-L is reasonably complete: >>> >>> http://groups.google.com/group/comp.soft-sys.sas/about >>> >>> The monthly posting frequency data since 1993 is: >>> >>> Posts <- structure(list(Jan = c(NA, 546L, 548L, 853L, 1007L, 894L, 514L, >>> 1720L, 1826L, 1941L, 1832L, 1636L, 2122L, 2722L, 2750L, 2305L, >>> 357L), Feb = c(NA, 511L, 734L, 1024L, 1150L, 1068L, 493L, 1519L, >>> 1537L, 1845L, 1846L, 1652L, 1960L, 1645L, 926L, 2255L, NA), Mar = c(NA, >>> 658L, 963L, 805L, 1108L, 945L, 659L, 1177L, 1915L, 2010L, 1755L, >>> 2188L, 629L, 1711L, 1728L, 2712L, NA), Apr = c(NA, 681L, 792L, >>> 1052L, 1315L, 784L, 1077L, 1163L, 1467L, 2199L, 1757L, 1826L, >>> 2169L, 2796L, 2766L, 2789L, NA), May = c(NA, 712L, 945L, 1163L, >>> 1212L, 448L, 778L, 1963L, 1735L, 2373L, 1863L, 1836L, 2283L, >>> 3147L, 2974L, 2025L, NA), Jun = c(NA, 751L, 1002L, 999L, 1127L, >>> 813L, 540L, 1615L, 1905L, 2133L, 1701L, 2606L, 2407L, 2723L, >>> 2691L, 2368L, NA), Jul = c(15L, 763L, 775L, 1184L, 1074L, 896L, >>> 476L, 1572L, 2027L, 2445L, 1926L, 1843L, 2061L, 761L, 2435L, >>> 2607L, NA), Aug = c(458L, 975L, 969L, 1053L, 692L, 823L, 612L, >>> 1696L, 1976L, 1492L, 1689L, 2143L, 1793L, 2027L, 2592L, 2584L, >>> NA), Sep = c(330L, 703L, 745L, 1176L, 947L, 894L, 1351L, 1491L, >>> 1439L, 1864L, 1646L, 1784L, 1365L, 2714L, 1868L, 2554L, NA), >>> Oct = c(219L, 805L, 691L, 1197L, 900L, 1129L, 1708L, 1669L, >>> 1592L, 2133L, 1832L, 1712L, 1427L, 2983L, 2320L, 2434L, NA >>> ), Nov = c(472L, 752L, 773L, 911L, 853L, 733L, 1720L, 1490L, >>> 1636L, 1663L, 1545L, 1786L, 1518L, 2848L, 2112L, 1984L, NA >>> ), Dec = c(517L, 666L, 765L, 844L, 677L, 492L, 1595L, 1298L, >>> 1424L, 1520L, 1445L, 2148L, 1524L, 2374L, 1948L, 1921L, NA >>> )), .Names = c("Jan", "Feb", "Mar", "Apr", "May", "Jun", >>> "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"), class = "data.frame", >>> row.names = c("1993", >>> "1994", "1995", "1996", "1997", "1998", "1999", "2000", "2001", >>> "2002", "2003", "2004", "2005", "2006", "2007", "2008", "2009" >>> )) >>> >>> >>> >>> >>>> >>>> Posts >>>> >>> >>> Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec >>> 1993 NA NA NA NA NA NA 15 458 330 219 472 517 >>> 1994 546 511 658 681 712 751 763 975 703 805 752 666 >>> 1995 548 734 963 792 945 1002 775 969 745 691 773 765 >>> 1996 853 1024 805 1052 1163 999 1184 1053 1176 1197 911 844 >>> 1997 1007 1150 1108 1315 1212 1127 1074 692 947 900 853 677 >>> 1998 894 1068 945 784 448 813 896 823 894 1129 733 492 >>> 1999 514 493 659 1077 778 540 476 612 1351 1708 1720 1595 >>> 2000 1720 1519 1177 1163 1963 1615 1572 1696 1491 1669 1490 1298 >>> 2001 1826 1537 1915 1467 1735 1905 2027 1976 1439 1592 1636 1424 >>> 2002 1941 1845 2010 2199 2373 2133 2445 1492 1864 2133 1663 1520 >>> 2003 1832 1846 1755 1757 1863 1701 1926 1689 1646 1832 1545 1445 >>> 2004 1636 1652 2188 1826 1836 2606 1843 2143 1784 1712 1786 2148 >>> 2005 2122 1960 629 2169 2283 2407 2061 1793 1365 1427 1518 1524 >>> 2006 2722 1645 1711 2796 3147 2723 761 2027 2714 2983 2848 2374 >>> 2007 2750 926 1728 2766 2974 2691 2435 2592 1868 2320 2112 1948 >>> 2008 2305 2255 2712 2789 2025 2368 2607 2584 2554 2434 1984 1921 >>> 2009 357 NA NA NA NA NA NA NA NA NA NA NA >>> >>> >>> One can then review the annual posting frequency via: >>> >>> pdf("SAS-L.pdf", height = 4, width = 7) >>> >>> mp <- barplot(rowSums(Posts, na.rm = TRUE), >>> beside = TRUE, >>> cex.names = 0.6, main = "SAS-L Traffic", >>> cex.axis = 0.75, las = 1) >>> >>> mtext(text = rowSums(Posts, na.rm = TRUE), at = mp, side = 1, >>> line = 2, cex = 0.5) >>> >>> dev.off() >>> >>> >>> There would appear to be marked increases in 2000 and again in 2006. >>> However, it has been flat for the past 3 calendar years. No decline yet, >>> but it will happen in due course... >>> >>> >>> >>> No comparable posting data table exists for S-News as far as I can find, >>> so I wrote a quick program to read the S-News archive pages here: >>> >>> http://www.biostat.wustl.edu/archives/html/s-news/ >>> >>> and get monthly posting counts, using the 'Thread' based html pages, >>> where each monthly embedded post link has a URL of the form: >>> >>> http://www.biostat.wustl.edu/archives/html/s-news/YYYY-MM/msgXXXXX.html >>> >>> >>> Thus, the program I used is: >>> >>> TD <- paste(rep(1998:2009, each = 12), sprintf("%02d", 1:12), sep = "-") >>> Posts <- numeric(length(TD)) >>> >>> for (i in seq(along = TD)) >>> { >>> URL <- paste("http://www.biostat.wustl.edu/archives/html/s-news/", >>> TD[i], "/threads.html", sep = "") >>> >>> cat(URL, "\n") >>> >>> if (!inherits(try(con <- readLines(URL)), "try-error")) >>> { >>> Posts[i] <- length(grep("msg.*\\.html", con)) >>> rm(con) >>> } else { >>> Posts[i] <- NA >>> } >>> } >>> >>> >>> Posts <- matrix(Posts, ncol = 12, byrow = TRUE) >>> rownames(Posts) <- 1998:2009 >>> colnames(Posts) <- month.abb >>> >>> That gives you: >>> >>> Posts <- structure(c(NA, 210, 264, 246, 230, 189, 197, 174, 109, 51, 48, >>> 5, 273, 173, 313, 232, 255, 179, 230, 161, 87, 59, 63, NA, 378, >>> 313, 285, 252, 242, 218, 257, 193, 99, 74, 58, NA, 293, 300, >>> 264, 300, 228, 196, 151, 182, 123, 48, 47, NA, 330, 334, 306, >>> 331, 219, 189, 164, 174, 107, 46, 31, NA, 243, 254, 247, 282, >>> 248, 217, 175, 109, 96, 34, 27, NA, 219, 284, 245, 258, 230, >>> 221, 154, 159, 84, 47, 40, NA, 209, 270, 302, 260, 207, 187, >>> 187, 144, 97, 39, 28, NA, 191, 300, 204, 260, 221, 186, 195, >>> 107, 68, 35, 41, NA, 241, 253, 251, 229, 280, 295, 150, 98, 73, >>> 70, 30, NA, 181, 300, 261, 232, 228, 197, 176, 82, 53, 56, 27, >>> NA, 141, 194, 176, 194, 177, 142, 176, 84, 20, 41, 36, NA), .Dim = c(12L, >>> 12L), .Dimnames = list(c("1998", "1999", "2000", "2001", "2002", >>> "2003", "2004", "2005", "2006", "2007", "2008", "2009"), c("Jan", >>> "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", >>> "Nov", "Dec"))) >>> >>> >>> >>>> >>>> Posts >>>> >>> >>> Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec >>> 1998 NA 273 378 293 330 243 219 209 191 241 181 141 >>> 1999 210 173 313 300 334 254 284 270 300 253 300 194 >>> 2000 264 313 285 264 306 247 245 302 204 251 261 176 >>> 2001 246 232 252 300 331 282 258 260 260 229 232 194 >>> 2002 230 255 242 228 219 248 230 207 221 280 228 177 >>> 2003 189 179 218 196 189 217 221 187 186 295 197 142 >>> 2004 197 230 257 151 164 175 154 187 195 150 176 176 >>> 2005 174 161 193 182 174 109 159 144 107 98 82 84 >>> 2006 109 87 99 123 107 96 84 97 68 73 53 20 >>> 2007 51 59 74 48 46 34 47 39 35 70 56 41 >>> 2008 48 63 58 47 31 27 40 28 41 30 27 36 >>> 2009 5 NA NA NA NA NA NA NA NA NA NA NA >>> >>> >>> Which can then be graphed by: >>> >>> pdf("S-News.pdf", height = 4, width = 7) >>> >>> mp <- barplot(rowSums(Posts, na.rm = TRUE), >>> beside = TRUE, >>> cex.names = 0.6, main = "S-News Traffic", >>> cex.axis = 0.75, las = 1) >>> >>> mtext(text = rowSums(Posts, na.rm = TRUE), at = mp, side = 1, >>> line = 2, cex = 0.5) >>> >>> dev.off() >>> >>> >>> >>> The consistent decline in posting frequency since 1999 is notable. The >>> temporal association with the introduction of R is perhaps profound. >>> >>> >>> >>> As long as I am on the subject, I figured that I would do the same for >>> R-Help. The downside is that readLines() (really url() ) does not >>> support https:, so I took a somewhat different approach, using wget: >>> >>> >>> TD <- paste(rep(1997:2009, each = 12), month.name, sep = "-") >>> Posts <- numeric(length(TD)) >>> >>> for (i in seq(along = TD)) >>> { >>> URL <- paste("https://stat.ethz.ch/pipermail/r-help/", >>> TD[i], "/thread.html", sep = "") >>> >>> cat(URL, "\n") >>> >>> CMD <- paste("wget", URL) >>> system(CMD) >>> >>> if (file.exists("thread.html")) >>> { >>> con <- readLines("thread.html") >>> Posts[i] <- length(grep("[0-9]+\\.html", con)) >>> rm(con) >>> unlink("thread.html") >>> } else { >>> Posts[i] <- NA >>> } >>> } >>> >>> Posts <- matrix(Posts, ncol = 12, byrow = TRUE) >>> rownames(Posts) <- 1997:2009 >>> colnames(Posts) <- month.abb >>> >>> >>> This gives you: >>> >>> Posts <- structure(c(NA, 135, 226, 205, 558, 884, 1017, 1116, 1746, >>> 2075, 1714, 2490, 462, NA, 79, 145, 355, 583, 697, 1137, 1580, 1724, >>> 1920, 1907, 2583, NA, NA, 114, 195, 377, 651, 880, 1203, 1946, >>> 1703, 2270, 2191, 2740, NA, 92, 101, 189, 377, 470, 965, 1488, >>> 1657, 2057, 1818, 2145, 2487, NA, 36, 90, 161, 504, 552, 1057, >>> 1268, 1561, 1887, 2029, 2210, 2517, NA, 47, 105, 186, 418, 550, >>> 926, 1319, 1714, 2056, 1811, 2307, 2774, NA, 41, 110, 184, 293, >>> 615, 918, 1344, 1618, 1872, 1785, 2138, 3268, NA, 37, 64, 148, >>> 356, 562, 824, 1210, 1493, 1777, 1898, 2241, 2813, NA, 40, 94, >>> 203, 434, 678, 705, 1443, 1534, 1709, 1902, 2028, 2990, NA, 76, >>> 96, 231, 418, 657, 1055, 1567, 1712, 1810, 2328, 2708, 3037, >>> NA, 61, 184, 318, 433, 825, 1038, 1605, 1895, 1907, 2127, 2594, >>> 2730, NA, 57, 105, 221, 422, 530, 742, 1158, 1481, 1508, 1450, >>> 2028, 2399, NA), .Dim = c(13L, 12L), .Dimnames = list(c("1997", >>> "1998", "1999", "2000", "2001", "2002", "2003", "2004", "2005", >>> "2006", "2007", "2008", "2009"), c("Jan", "Feb", "Mar", "Apr", >>> "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"))) >>> >>> >>> >>>> >>>> Posts >>>> >>> >>> Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec >>> 1997 NA NA NA 92 36 47 41 37 40 76 61 57 >>> 1998 135 79 114 101 90 105 110 64 94 96 184 105 >>> 1999 226 145 195 189 161 186 184 148 203 231 318 221 >>> 2000 205 355 377 377 504 418 293 356 434 418 433 422 >>> 2001 558 583 651 470 552 550 615 562 678 657 825 530 >>> 2002 884 697 880 965 1057 926 918 824 705 1055 1038 742 >>> 2003 1017 1137 1203 1488 1268 1319 1344 1210 1443 1567 1605 1158 >>> 2004 1116 1580 1946 1657 1561 1714 1618 1493 1534 1712 1895 1481 >>> 2005 1746 1724 1703 2057 1887 2056 1872 1777 1709 1810 1907 1508 >>> 2006 2075 1920 2270 1818 2029 1811 1785 1898 1902 2328 2127 1450 >>> 2007 1714 1907 2191 2145 2210 2307 2138 2241 2028 2708 2594 2028 >>> 2008 2490 2583 2740 2487 2517 2774 3268 2813 2990 3037 2730 2399 >>> 2009 462 NA NA NA NA NA NA NA NA NA NA NA >>> >>> >>> Which again can be graphed as: >>> >>> pdf("R-Help.pdf", height = 4, width = 7) >>> >>> mp <- barplot(rowSums(Posts, na.rm = TRUE), >>> beside = TRUE, >>> cex.names = 0.6, main = "R-Help Traffic", >>> cex.axis = 0.75, las = 1) >>> >>> mtext(text = rowSums(Posts, na.rm = TRUE), at = mp, side = 1, >>> line = 2, cex = 0.5) >>> >>> dev.off() >>> >>> >>> Now....there's a healthy growth curve.... :-) >>> >>> Note that the annual traffic volume for 2008 on R-Help exceeds that on >>> SAS-L. >>> >>> For convenience, I am attaching each of the 3 plots. >>> >>> Regards, >>> >>> Marc Schwartz >>> >>> >>> ______________________________________________ >>> R-help@r-project.org mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-help >>> PLEASE do read the posting guide >>> http://www.R-project.org/posting-guide.html >>> and provide commented, minimal, self-contained, reproducible code. >>> >>> >>> >> >> ______________________________________________ >> R-help@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> >> > ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.