Re: [R] R help - Web Scraping of Google News using R

[boB Rudis]

What you are doing wrong is both trying yourself and asking others to
violate Google's Terms of Service and (amongst other things) get your
IP banned along with anyone who aids you (or worse). Please don't.
Just because something can be done does not mean it should be done.

On Tue, May 24, 2016 at 11:21 AM, Kumar Gauraw  wrote:
> Hello Experts,
>
> I am trying to scrap data from Google news for a particular topic using XML
> and Curl Package of R. I am able to extract the summary part of the news
> through *XPath* but in a similar way, I am trying to extract title and
> Links of news which is not working.Please note this work is just for POC
> purpose and I would make maximum of 500 requests per day so that Google TOS
> remains intact.
>
>
> library(XML)
>
> library(RCurl)
>
> getGoogleURL <- function(search.term, domain = '.co.in', quotes=TRUE)
>
> {
>
>   search.term <- gsub(' ', '%20', search.term)
>
>   if(quotes) search.term <- paste('%22', search.term, '%22', sep='')
>
>   getGoogleURL <- paste('http://www.google', domain,
> '/search?hl=en&gl=in&tbm=nws&authuser=0&q=',search.term, sep='')
>
> }
>
> search.term <- "IPL 2016"
>
> quotes <- "FALSE"
>
> search.url <- getGoogleURL(search.term=search.term, quotes=quotes)
>
> getGoogleSummary <- function(google.url) {
>
>   doc <- getURL(google.url, httpheader = c("User-Agent" = "R(2.10.0)"))
>
>   html <- htmlTreeParse(doc, useInternalNodes = TRUE, error=function(...){})
>
>   nodes <- getNodeSet(html, "//div[@class='st']")
>
>   return(sapply(nodes, function(x) x <- xmlValue(x)))
>
> }
>
> *#Problem is with this part of code*
>
> getGoogleTitle <- function(google.url) {
>
>   doc <- getURL(google.url, httpheader = c("User-Agent" = "R(2.10.0)"))
>
>   html <- htmlTreeParse(doc, useInternalNodes = TRUE, error=function(...){})
>
>  * nodes <- getNodeSet(html, "//a[@class='l _HId']")*
>
>   return(sapply(nodes, function(x) x <- xmlValue(x)))
>
> }
>
> Kindly help me to understand where I am getting wrong so that I can rectify
> the code and get the correct output.
>
> Thank you.
>
> With Regards,
> Kumar Gauraw
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Mixed model analysis

[Bert Gunter]

This has nothing to do with R, per se. This is a statistical issue. You
need to work with a statistician, as your statistical background is
inadequate (google "mixed effects models") if you really need this.

Cheers,
Bert

On Tue, May 24, 2016 at 7:27 PM Neny Sitorus 
wrote:

> Hi,
>
> what is exactly mixed model analysis in R?
> could someone give me a better description.
>
>
> Thank you,
> Neny
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] Mixed model analysis

[Neny Sitorus]

Hi,

what is exactly mixed model analysis in R?
could someone give me a better description.


Thank you,
Neny

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Sprintf to call data frame from environment

[Jim Lemon]

Hi Beatriz,
I'll guess that you have a number of files with names like this:

Samples_1.txt
Samples_2.txt
...

Each one can be read with a function like read.table and will return a
data frame with default names (V1, V2, ...). You then want to extract
the first element (column) of the data frame. If I'm correct, try
this:

V1s<-list()
# nfiles is the number of files you want to read
for(i in 1:nfiles) {
 filename<-paste("Samples_",i,".txt,sep="")
 # you will probably have to add the appropriate arguments to read.table
 V1s[[i]]<-read.table(filename)$V1
}

The list V1s should contain the first columns of all the data frames read.

Jim


On Wed, May 25, 2016 at 7:01 AM, Beatriz  wrote:
>
> In my environment I have a data frame called Samples_1.txt.
> From this data frame I need to get variable V1.  My code doesn't work.
> Thanks!
>
> sprintf("Samples_%s.txt", 1)$V1
>
> Note: I need to do it in this way because I have the code into a for loop.
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Sprintf to call data frame from environment

[David Winsemius]

> On May 24, 2016, at 2:01 PM, Beatriz  wrote:
> 
> 
> In my environment I have a data frame called Samples_1.txt.
> From this data frame I need to get variable V1.  My code doesn't work. Thanks!
> 
>   $V1
> 
> Note: I need to do it in this way because I have the code into a for loop.

You are treating this as if R were a macro processor, which it's not. The only 
function that lets you pull in a data-object from the store of named objects 
using a character vector is `get`, so perhaps:

get( sprintf("Samples_%s.txt", 1) )$V1

And if you were considering the next step of hoping to pass a computed item to 
`$`, then forget that as well, and learn to use `[`.

-- 

David Winsemius
Alameda, CA, USA

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Sprintf to call data frame from environment

[Nordlund, Dan (DSHS/RDA)]

It is not clear (at least to me) what your actual task is.  But, if 
Samples_1.txt is the actual name of a data frame that exists in memory (and not 
a filename), then you need to wrap the sprintf() in a get() function.

get(sprintf("Samples_%s.txt", 1))$V1

I am no expert  on "computing on the language" in R, but I can't help but think 
you are going about your task in the wrong way.  If you provide more detail 
about what you are trying to do, someone will probably be able to provide you a 
solution where you don't need to do it this way.


Hope this is helpful,

Dan

Daniel Nordlund, PhD
Research and Data Analysis Division
Services & Enterprise Support Administration
Washington State Department of Social and Health Services


> -Original Message-
> From: R-help [mailto:r-help-boun...@r-project.org] On Behalf Of Beatriz
> Sent: Tuesday, May 24, 2016 2:01 PM
> To: r-help@r-project.org
> Subject: [R] Sprintf to call data frame from environment
> 
> 
> In my environment I have a data frame called Samples_1.txt.
>  From this data frame I need to get variable V1.  My code doesn't work.
> Thanks!
> 
>   sprintf("Samples_%s.txt", 1)$V1
> 
> Note: I need to do it in this way because I have the code into a for loop.
> 
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-
> guide.html
> and provide commented, minimal, self-contained, reproducible code.

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] Sprintf to call data frame from environment

[Beatriz]

In my environment I have a data frame called Samples_1.txt.
From this data frame I need to get variable V1.  My code doesn't work. Thanks!

sprintf("Samples_%s.txt", 1)$V1

Note: I need to do it in this way because I have the code into a for loop.

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] Sprintf to call data frame from environment

[Beatriz]

In my environment I have a data frame called Samples_1.txt.
From this data frame I need to get variable V1.  My code doesn't work.
Note: I need to do it in this way because I have the code into a for loop.

sprintf("Samples_%s.txt", 1)$V1

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] identify duplicate entries in data frame and calculate mean

[Matthew]

Thank you very much, Dan.

These work great. Two more great answers to my question.

Matthew

On 5/24/2016 4:15 PM, Nordlund, Dan (DSHS/RDA) wrote:

You have several  options.

1.  You could use the aggregate function.  If your data frame is called DF, you 
could do something like

with(DF, aggregate(Length, list(Identifier), mean))

2.  You could use the dplyr package like this

library(dplyr)
summarize(group_by(DF, Identifier), mean(Length))


Hope this is helpful,

Dan

Daniel Nordlund, PhD
Research and Data Analysis Division
Services & Enterprise Support Administration
Washington State Department of Social and Health Services



-Original Message-
From: R-help [mailto:r-help-boun...@r-project.org] On Behalf Of Matthew
Sent: Tuesday, May 24, 2016 12:47 PM
To: r-help@r-project.org
Subject: [R] identify duplicate entries in data frame and calculate mean

I have a data frame with 10 columns.
In the last column is an alphaneumaric identifier.
For most rows, this alphaneumaric identifier is unique to the file, however
some of these alphanemeric idenitifiers occur in duplicate, triplicate or more.
When they do occur more than once they are in consecutive rows, so when
there is a duplicate or triplicate or quadruplicate (let's call them 
multiplicates),
they are in consecutive rows.

In column 7 there is an integer number (may or may not be unique. does not
matter).

I want to identify each multiple entries (multiplicates) occurring in column 10
and then for each multiplicate calculate the mean of the integers column 7.

As an example, I will show just two columns:
Length  Identifier
321 A234
350 A234
340 A234
180 B123
198 B225

What I want to do (in the above example) is collapse all the A234's and report
the mean to get this:
Length  Identifier
337 A234
180 B123
198 B225


Matthew

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-
guide.html
and provide commented, minimal, self-contained, reproducible code.


__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] identify duplicate entries in data frame and calculate mean

[Matthew]

Thanks, Tom.  I was making a mistake looking at your example and that's 
what my problem was.

Cool answer, works great. Thank you very much.

Matthew

On 5/24/2016 4:23 PM, Tom Wright wrote:
> Don't see that as being a big problem. If your data grows then dplyr 
> supports connections to external databases. Alternately if you just 
> want a mean, most databases can do that directly in SQL.
>
> On Tue, May 24, 2016 at 4:17 PM, Matthew 
>  > wrote:
>
> Thank you very much, Tom.
> This gets me thinking in the right direction.
> One thing I should have mentioned that I did not is that the
> number of rows in the data frame will be a little over 40,000 rows.
>
>
> On 5/24/2016 4:08 PM, Tom Wright wrote:
>> Using dplyr
>>
>> $ library(dplyr)
>> $ x<-data.frame(Length=c(321,350,340,180,198),
>> ID=c(rep('A234',3),'B123','B225') )
>> $ x %>% group_by(ID) %>% summarise(m=mean(Length))
>>
>>
>>
>> On Tue, May 24, 2016 at 3:46 PM, Matthew
>> > > wrote:
>>
>> I have a data frame with 10 columns.
>> In the last column is an alphaneumaric identifier.
>> For most rows, this alphaneumaric identifier is unique to the
>> file, however some of these alphanemeric idenitifiers occur
>> in duplicate, triplicate or more. When they do occur more
>> than once they are in consecutive rows, so when there is a
>> duplicate or triplicate or quadruplicate (let's call them
>> multiplicates), they are in consecutive rows.
>>
>> In column 7 there is an integer number (may or may not be
>> unique. does not matter).
>>
>> I want to identify each multiple entries (multiplicates)
>> occurring in column 10 and then for each multiplicate
>> calculate the mean of the integers column 7.
>>
>> As an example, I will show just two columns:
>> Length  Identifier
>> 321 A234
>> 350 A234
>> 340 A234
>> 180 B123
>> 198 B225
>>
>> What I want to do (in the above example) is collapse all the
>> A234's and report the mean to get this:
>> Length  Identifier
>> 337 A234
>> 180 B123
>> 198 B225
>>
>>
>> Matthew
>>
>> __
>> R-help@r-project.org  mailing
>> list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible
>> code.
>>
>>
>
>


[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] identify duplicate entries in data frame and calculate mean

[Tom Wright]

Don't see that as being a big problem. If your data grows then dplyr
supports connections to external databases. Alternately if you just want a
mean, most databases can do that directly in SQL.

On Tue, May 24, 2016 at 4:17 PM, Matthew 
wrote:

> Thank you very much, Tom.
> This gets me thinking in the right direction.
> One thing I should have mentioned that I did not is that the number of
> rows in the data frame will be a little over 40,000 rows.
>
>
> On 5/24/2016 4:08 PM, Tom Wright wrote:
>
> Using dplyr
>
> $ library(dplyr)
> $ x<-data.frame(Length=c(321,350,340,180,198),
> ID=c(rep('A234',3),'B123','B225') )
> $ x %>% group_by(ID) %>% summarise(m=mean(Length))
>
>
>
> On Tue, May 24, 2016 at 3:46 PM, Matthew  > wrote:
>
>> I have a data frame with 10 columns.
>> In the last column is an alphaneumaric identifier.
>> For most rows, this alphaneumaric identifier is unique to the file,
>> however some of these alphanemeric idenitifiers occur in duplicate,
>> triplicate or more. When they do occur more than once they are in
>> consecutive rows, so when there is a duplicate or triplicate or
>> quadruplicate (let's call them multiplicates), they are in consecutive rows.
>>
>> In column 7 there is an integer number (may or may not be unique. does
>> not matter).
>>
>> I want to identify each multiple entries (multiplicates) occurring in
>> column 10 and then for each multiplicate calculate the mean of the integers
>> column 7.
>>
>> As an example, I will show just two columns:
>> Length  Identifier
>> 321 A234
>> 350 A234
>> 340 A234
>> 180 B123
>> 198 B225
>>
>> What I want to do (in the above example) is collapse all the A234's and
>> report the mean to get this:
>> Length  Identifier
>> 337 A234
>> 180 B123
>> 198 B225
>>
>>
>> Matthew
>>
>> __
>> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
>
>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] identify duplicate entries in data frame and calculate mean

[Matthew]

Thank you very much, Tom.
This gets me thinking in the right direction.
One thing I should have mentioned that I did not is that the number of 
rows in the data frame will be a little over 40,000 rows.

On 5/24/2016 4:08 PM, Tom Wright wrote:
> Using dplyr
>
> $ library(dplyr)
> $ x<-data.frame(Length=c(321,350,340,180,198),
> ID=c(rep('A234',3),'B123','B225') )
> $ x %>% group_by(ID) %>% summarise(m=mean(Length))
>
>
>
> On Tue, May 24, 2016 at 3:46 PM, Matthew 
>  > wrote:
>
> I have a data frame with 10 columns.
> In the last column is an alphaneumaric identifier.
> For most rows, this alphaneumaric identifier is unique to the
> file, however some of these alphanemeric idenitifiers occur in
> duplicate, triplicate or more. When they do occur more than once
> they are in consecutive rows, so when there is a duplicate or
> triplicate or quadruplicate (let's call them multiplicates), they
> are in consecutive rows.
>
> In column 7 there is an integer number (may or may not be unique.
> does not matter).
>
> I want to identify each multiple entries (multiplicates) occurring
> in column 10 and then for each multiplicate calculate the mean of
> the integers column 7.
>
> As an example, I will show just two columns:
> Length  Identifier
> 321 A234
> 350 A234
> 340 A234
> 180 B123
> 198 B225
>
> What I want to do (in the above example) is collapse all the
> A234's and report the mean to get this:
> Length  Identifier
> 337 A234
> 180 B123
> 198 B225
>
>
> Matthew
>
> __
> R-help@r-project.org  mailing list --
> To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
>


[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] identify duplicate entries in data frame and calculate mean

[Nordlund, Dan (DSHS/RDA)]

You have several  options.  

1.  You could use the aggregate function.  If your data frame is called DF, you 
could do something like

with(DF, aggregate(Length, list(Identifier), mean))

2.  You could use the dplyr package like this

library(dplyr)
summarize(group_by(DF, Identifier), mean(Length))


Hope this is helpful,

Dan

Daniel Nordlund, PhD
Research and Data Analysis Division
Services & Enterprise Support Administration
Washington State Department of Social and Health Services


> -Original Message-
> From: R-help [mailto:r-help-boun...@r-project.org] On Behalf Of Matthew
> Sent: Tuesday, May 24, 2016 12:47 PM
> To: r-help@r-project.org
> Subject: [R] identify duplicate entries in data frame and calculate mean
> 
> I have a data frame with 10 columns.
> In the last column is an alphaneumaric identifier.
> For most rows, this alphaneumaric identifier is unique to the file, however
> some of these alphanemeric idenitifiers occur in duplicate, triplicate or 
> more.
> When they do occur more than once they are in consecutive rows, so when
> there is a duplicate or triplicate or quadruplicate (let's call them 
> multiplicates),
> they are in consecutive rows.
> 
> In column 7 there is an integer number (may or may not be unique. does not
> matter).
> 
> I want to identify each multiple entries (multiplicates) occurring in column 
> 10
> and then for each multiplicate calculate the mean of the integers column 7.
> 
> As an example, I will show just two columns:
> Length  Identifier
> 321 A234
> 350 A234
> 340 A234
> 180 B123
> 198 B225
> 
> What I want to do (in the above example) is collapse all the A234's and report
> the mean to get this:
> Length  Identifier
> 337 A234
> 180 B123
> 198 B225
> 
> 
> Matthew
> 
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-
> guide.html
> and provide commented, minimal, self-contained, reproducible code.

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] identify duplicate entries in data frame and calculate mean

[Tom Wright]

Using dplyr

$ library(dplyr)
$ x<-data.frame(Length=c(321,350,340,180,198),
ID=c(rep('A234',3),'B123','B225') )
$ x %>% group_by(ID) %>% summarise(m=mean(Length))



On Tue, May 24, 2016 at 3:46 PM, Matthew 
wrote:

> I have a data frame with 10 columns.
> In the last column is an alphaneumaric identifier.
> For most rows, this alphaneumaric identifier is unique to the file,
> however some of these alphanemeric idenitifiers occur in duplicate,
> triplicate or more. When they do occur more than once they are in
> consecutive rows, so when there is a duplicate or triplicate or
> quadruplicate (let's call them multiplicates), they are in consecutive rows.
>
> In column 7 there is an integer number (may or may not be unique. does not
> matter).
>
> I want to identify each multiple entries (multiplicates) occurring in
> column 10 and then for each multiplicate calculate the mean of the integers
> column 7.
>
> As an example, I will show just two columns:
> Length  Identifier
> 321 A234
> 350 A234
> 340 A234
> 180 B123
> 198 B225
>
> What I want to do (in the above example) is collapse all the A234's and
> report the mean to get this:
> Length  Identifier
> 337 A234
> 180 B123
> 198 B225
>
>
> Matthew
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] identify duplicate entries in data frame and calculate mean

[Matthew]

I have a data frame with 10 columns.
In the last column is an alphaneumaric identifier.
For most rows, this alphaneumaric identifier is unique to the file, 
however some of these alphanemeric idenitifiers occur in duplicate, 
triplicate or more. When they do occur more than once they are in 
consecutive rows, so when there is a duplicate or triplicate or 
quadruplicate (let's call them multiplicates), they are in consecutive rows.


In column 7 there is an integer number (may or may not be unique. does 
not matter).


I want to identify each multiple entries (multiplicates) occurring in 
column 10 and then for each multiplicate calculate the mean of the 
integers column 7.


As an example, I will show just two columns:
Length  Identifier
321 A234
350 A234
340 A234
180 B123
198 B225

What I want to do (in the above example) is collapse all the A234's and 
report the mean to get this:

Length  Identifier
337 A234
180 B123
198 B225


Matthew

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Creating a data frame from scratch

[Nordlund, Dan (DSHS/RDA)]

I would probably write the function something like this:


t_count_na <- function(dataset,
   variables = "all") {
  if (identical(variables, "all")) {
variable_list <- names(dataset)
  }  else {
variable_list <- variables
  }  
  apply(dataset[,variable_list], 1, function(x) sum(is.na(x)))
}


Hope this is helpful,

Dan

Daniel Nordlund, PhD
Research and Data Analysis Division
Services & Enterprise Support Administration
Washington State Department of Social and Health Services


> -Original Message-
> From: R-help [mailto:r-help-boun...@r-project.org] On Behalf Of
> g.maub...@gmx.de
> Sent: Tuesday, May 24, 2016 11:55 AM
> To: r-help@r-project.org
> Subject: [R] Creating a data frame from scratch
> 
> Hi All,
> 
> I need to create a data frame from scratch and fill variables created on the 
> fly
> with values. What I have so far:
> 
> -- schnipp --
> 
> # Example dataset
> gene <-
> c("ENSG0208234","ENSG0199674","ENSG0221622","ENSG0
> 207604",
> 
> "ENSG0207431","ENSG0221312","ENSG00134940305","ENSG0039403
> 9490",
>   "ENSG09943004048")
> hsap <- c(0,0,0, 0, 0, 0, 1,1, 1)
> mmul <- c(NA,2 ,3, NA, 2, 1 , NA,2, NA)
> mmus <- c(NA,2 ,NA, NA, NA, 2 , NA,3, 1) rnor <- c(NA,2 ,NA, 1 , NA, 3 ,
> NA,NA, 2) cfam <- c(NA,2,NA, 2, 1, 2, 2,NA, NA)
> 
> ds_example <- data.frame(gene, hsap, mmul, mmus, rnor, cfam)
> ds_example$gene <- as.character(ds_example$gene)
> 
> t_count_na <- function(dataset,
>variables = "all")
>   # credit: http://stackoverflow.com/questions/4862178/remove-rows-with-
> nas-in-data-frame
>   {
>   ds_na <- data.frame()
>   # if variables = "all" create character vector of variable names
>   if (variables == "all") {
> variable_list <- dimnames(dataset)[[ 2 ]]
>   }
>   # if a character vector with variable names is given
>   # to run the function on a defined set of selected variables
>   else {
> variable_list <- variables
>   }
> 
>   for (var in variable_list) {
> new_name <- paste0("na_", var)
> ds_na[[ new_name ]] <- as.data.frame(is.na(dataset[[ var ]]))
>   }
> 
>   ds_na[[ "na_count" ]] <- rowSums(ds_na)
>   return(ds_na)
> }
> 
> test <- t_count_na(dataset = ds_example, variables = c("mmul", "mmus"))
> 
> -- schnipp --
> 
> gives:
> 
>  Error in `[[<-.data.frame`(`*tmp*`, new_name, value =
> list(`is.na(dataset[[var]])` = c(TRUE,  :
>   replacement has 9 rows, data has 0 In addition: Warning message:
> In if (variables == "all") { :
>   the condition has length > 1 and only the first element will be used
> 
> My goal is to create a dataset from scratch on the fly which has the same
> amount of variables as the dataset ds_example plus a single variable storing
> the amount of NA's in a row for the given variables. This is the basis for a
> decious which cases to keep and which to drop.
> 
> I do not want to alter the base dataset like ds_example in the first place nor
> do I want to make a copy of the existing dataset due to memory allocation.
> The function shall also work with big data, e. g. datasets with more than 1 GB
> memory consumption.
> 
> I also do not want the newly created variables to be stored in the original
> data frame. They shall be separate.
> 
> A former similar solution worked:
> http://r.789695.n4.nabble.com/Creating-variables-on-the-fly-td4720034.html
> 
> Why doesn't this one?
> 
> How do I create the variables within the data frame if the data frame is
> empty?
> 
> Kind regards
> 
> Georg Maubach
> 
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-
> guide.html
> and provide commented, minimal, self-contained, reproducible code.

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] Creating a data frame from scratch

[G . Maubach]

Hi All,

I need to create a data frame from scratch and fill variables created on the 
fly with values. What I have so far:

-- schnipp --

# Example dataset
gene <- 
c("ENSG0208234","ENSG0199674","ENSG0221622","ENSG0207604", 
  "ENSG0207431","ENSG0221312","ENSG00134940305","ENSG00394039490",
  "ENSG09943004048")
hsap <- c(0,0,0, 0, 0, 0, 1,1, 1)
mmul <- c(NA,2 ,3, NA, 2, 1 , NA,2, NA)
mmus <- c(NA,2 ,NA, NA, NA, 2 , NA,3, 1)
rnor <- c(NA,2 ,NA, 1 , NA, 3 , NA,NA, 2)
cfam <- c(NA,2,NA, 2, 1, 2, 2,NA, NA)

ds_example <- data.frame(gene, hsap, mmul, mmus, rnor, cfam)
ds_example$gene <- as.character(ds_example$gene)

t_count_na <- function(dataset,
   variables = "all")
  # credit: 
http://stackoverflow.com/questions/4862178/remove-rows-with-nas-in-data-frame
  {
  ds_na <- data.frame()
  # if variables = "all" create character vector of variable names
  if (variables == "all") {
variable_list <- dimnames(dataset)[[ 2 ]] 
  }
  # if a character vector with variable names is given
  # to run the function on a defined set of selected variables
  else {
variable_list <- variables
  }
  
  for (var in variable_list) {
new_name <- paste0("na_", var)
ds_na[[ new_name ]] <- as.data.frame(is.na(dataset[[ var ]]))
  }
  
  ds_na[[ "na_count" ]] <- rowSums(ds_na)
  return(ds_na)
}

test <- t_count_na(dataset = ds_example, variables = c("mmul", "mmus"))

-- schnipp --

gives:

 Error in `[[<-.data.frame`(`*tmp*`, new_name, value = 
list(`is.na(dataset[[var]])` = c(TRUE,  : 
  replacement has 9 rows, data has 0 In addition: Warning message:
In if (variables == "all") { :
  the condition has length > 1 and only the first element will be used

My goal is to create a dataset from scratch on the fly which has the same 
amount of variables as the dataset ds_example plus a single variable storing 
the amount of NA's in a row for the given variables. This is the basis for a 
decious which cases to keep and which to drop.

I do not want to alter the base dataset like ds_example in the first place nor 
do I want to make a copy of the existing dataset due to memory allocation. The 
function shall also work with big data, e. g. datasets with more than 1 GB 
memory consumption.

I also do not want the newly created variables to be stored in the original 
data frame. They shall be separate.

A former similar solution worked:
http://r.789695.n4.nabble.com/Creating-variables-on-the-fly-td4720034.html

Why doesn't this one?

How do I create the variables within the data frame if the data frame is empty?

Kind regards

Georg Maubach

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] R help - Web Scraping of Google News using R

[Kumar Gauraw]

Hello Experts,

I am trying to scrap data from Google news for a particular topic using XML
and Curl Package of R. I am able to extract the summary part of the news
through *XPath* but in a similar way, I am trying to extract title and
Links of news which is not working.Please note this work is just for POC
purpose and I would make maximum of 500 requests per day so that Google TOS
remains intact.


library(XML)

library(RCurl)

getGoogleURL <- function(search.term, domain = '.co.in', quotes=TRUE)

{

  search.term <- gsub(' ', '%20', search.term)

  if(quotes) search.term <- paste('%22', search.term, '%22', sep='')

  getGoogleURL <- paste('http://www.google', domain,
'/search?hl=en&gl=in&tbm=nws&authuser=0&q=',search.term, sep='')

}

search.term <- "IPL 2016"

quotes <- "FALSE"

search.url <- getGoogleURL(search.term=search.term, quotes=quotes)

getGoogleSummary <- function(google.url) {

  doc <- getURL(google.url, httpheader = c("User-Agent" = "R(2.10.0)"))

  html <- htmlTreeParse(doc, useInternalNodes = TRUE, error=function(...){})

  nodes <- getNodeSet(html, "//div[@class='st']")

  return(sapply(nodes, function(x) x <- xmlValue(x)))

}

*#Problem is with this part of code*

getGoogleTitle <- function(google.url) {

  doc <- getURL(google.url, httpheader = c("User-Agent" = "R(2.10.0)"))

  html <- htmlTreeParse(doc, useInternalNodes = TRUE, error=function(...){})

 * nodes <- getNodeSet(html, "//a[@class='l _HId']")*

  return(sapply(nodes, function(x) x <- xmlValue(x)))

}

Kindly help me to understand where I am getting wrong so that I can rectify
the code and get the correct output.

Thank you.

With Regards,
Kumar Gauraw

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] Downloading attachment from gmail

[Christofer Bogaso]

Hi folks,

I am wondering if it is really possible via some R code which shall do
the following

1. Login to a Gmail account (account name and password will be provided to R)
2. Search for all mails which has a word "ABCD" in the mail body
3. Download all the attachments (if available) which will hit the criteria #2


Am not sure if above is too ambitious, however will really appreciate
if R can do this.

Thanks for your time

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] numeric inputs to sweep produce NaN...

[David Winsemius]

> On May 24, 2016, at 8:49 AM, Witold E Wolski  wrote:
> 
> I have two inputs to sweep which are numeric (with a few NA's) but the
> output is NaN. How Why?
> 
> 
>> sum(!is.numeric(unlist(protquant)))
> [1] 0
>> sum(!is.numeric(normalize))
> [1] 0
>> normprotquant <- sweep(protquant, 2, normalize, "-" )
>> sum(is.nan(unlist(normprotquant)))
> [1] 31
> 

Post output of dput(head(protquant)) and dput(head(normalize))

> 
> version R 3.3.0
> 
> best regards Witold
> -- 
> Witold Eryk Wolski
> 
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

David Winsemius
Alameda, CA, USA

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Factor Variable frequency

[ruipbarradas]

Hello,

Maybe the following (untested).

table(df$Protocol[df$Speed == "SLOW"])

Hope this helps,

Rui Barradas
 

Citando ch.elahe via R-help :

> Hi all,
> I have the following df:
>
>    $ Protocol       : Factor w/ 48 levels "DP FS QTSE SAG",..: 2 3  
> 43 42 31 36 37 30 28 5 ...
>
>    $ Speed         : chr  "SLOW" "SLOW" "SLOW" "VerySLOW" ...
> How can I get the most frequent Protocol when Speed is "SLOW"?
> Thanks for any help!
> Elahe
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide  
> http://www.R-project.org/posting-guide.htmland provide commented,  
> minimal, self-contained, reproducible code.

 

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] file connection when using parallel

[Arnaud Mosnier]

Dear UserRs,

I have a little problem creating a file connection when working in parallel
(see the reproducable script below).
I am sure this is something obvious,
Can you enlighten me ?

Thanks,

Arnaud



# This part works
#
cat("This is a test file" , file={f <- tempfile()})
con <- file(f, "rt")


# Doing what I think is the same thing gives an error message when executed
in parallel
#--

library(parallel)
cl <- makeCluster(2)

## Exporting the object f into the cluster

  clusterExport(cl, "f")
  clusterEvalQ(cl[1], con <- file(f[[1]], "rt"))
   #Error in checkForRemoteErrors(lapply(cl, recvResult)) :
   # one node produced an error: cannot open the connection


## Creating the object f into the cluster

  clusterEvalQ(cl[1],cat("This is a test file" , file={f <- tempfile()}))
  clusterEvalQ(cl[1],con <- file(f, "rt"))
   #Error in checkForRemoteErrors(lapply(cl, recvResult)) :
   # one node produced an error: cannot open the connection

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] Factor Variable frequency

[ch.elahe via R-help]

Hi all,
I have the following df:


$ Protocol   : Factor w/ 48 levels "DP FS QTSE SAG",..: 2 3 43 42 31 36 
37 30 28 5 ...

$ Speed : chr  "SLOW" "SLOW" "SLOW" "VerySLOW" ...
How can I get the most frequent Protocol when Speed is "SLOW"?
Thanks for any help!
Elahe

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] numeric inputs to sweep produce NaN...

[Witold E Wolski]

I have two inputs to sweep which are numeric (with a few NA's) but the
output is NaN. How Why?


> sum(!is.numeric(unlist(protquant)))
[1] 0
> sum(!is.numeric(normalize))
[1] 0
> normprotquant <- sweep(protquant, 2, normalize, "-" )
> sum(is.nan(unlist(normprotquant)))
[1] 31


version R 3.3.0

best regards Witold
-- 
Witold Eryk Wolski

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] mgcv::gam(): NA parametric coefficient in a model with two categorical variables + model interpretation

[Fotis Fotiadis]

Dear Prof. Wood

Thank you, again, for your immediate response.

Best,
Fotis

On Mon, May 23, 2016 at 4:32 PM, Simon Wood  wrote:

> Q1: It looks like the model is not fully identifiably given the data and
> as a result igcCAT.ideo has been set to zero - there is no sensible test to
> conduct with such a term, hence the NAs in the test stat an p-value fields.
>
> Q2: A separate (centred) smooth is estimated for each level of igc. If you
> want a baseline (igcCAT.pseudo) smooth, and difference smooths for the rest
> of the levels of igc then you need to set igc to be an ordered factor, and
> use something like...
> ~ igc + s(ctrial) + s(ctrial,by=igc)
> - see section on `by' variables in ?gam.models.
>
> best,
> Simon
>
>
> On 22/05/16 23:29, Fotis Fotiadis wrote:
>
>> Hallo all
>>
>> I am using a gam model for my data.
>>
>> m2.4<-bam(acc~ 1 + igc + s(ctrial, by=igc) + shape + s(ctrial, by=shape) +
>> s(ctrial, sbj, bs = "fs", m = 1) , data=data, family=binomial)
>>
>> igc codes condition and there are four levels (CAT.pseudo,
>> CAT.ideo,PA.pseudo, PA.ideo), and shape is a factor (that cannot be
>> considered random effect) with four levels too (rand21, rand22, rand23,
>> rand30).
>>
>> Here is the summary of the model
>>
>>> summary(m2.4)
>>>
>> Family: binomial
>> Link function: logit
>>
>> Formula:
>> acc ~ 1 + igc + s(ctrial, by = igc) + shape + s(ctrial, by = shape) +
>>  s(ctrial, sbj, bs = "fs", m = 1)
>>
>> Parametric coefficients:
>>   Estimate Std. Error z value Pr(>|z|)
>> (Intercept)3.5321 0.1930  18.302  < 2e-16 ***
>> igcCAT.ideo0. 0.  NA   NA
>> igcPA.ideo-0.3650 0.2441  -1.495   0.1348
>> igcPA.pseudo  -0.2708 0.2574  -1.052   0.2928
>> shaperand22   -0.1390 0.1548  -0.898   0.3693
>> shaperand230.3046 0.1670   1.823   0.0682 .
>> shaperand30   -0.5839 0.1163  -5.020 5.16e-07 ***
>> ---
>> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
>>
>> Approximate significance of smooth terms:
>>  edf  Ref.df   Chi.sq  p-value
>> s(ctrial):igcCAT.pseudo   3.902   4.853   74.787 1.07e-14 ***
>> s(ctrial):igcCAT.ideo 2.293   2.702   13.794 0.001750 **
>> s(ctrial):igcPA.ideo  1.000   1.000   11.391 0.000738 ***
>> s(ctrial):igcPA.pseudo3.158   3.815   20.411 0.000413 ***
>> s(ctrial):shaperand21 2.556   3.316   31.387 1.46e-06 ***
>> s(ctrial):shaperand22 1.000   1.0000.898 0.343381
>> s(ctrial):shaperand23 2.304   2.8506.144 0.118531
>> s(ctrial):shaperand30 4.952   5.947   27.806 0.000144 ***
>> s(ctrial,sbj)   221.476 574.000 1502.779  < 2e-16 ***
>> ---
>> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
>>
>> Rank: 652/655
>> R-sq.(adj) =  0.405   Deviance explained = 43.9%
>> fREML =  24003  Scale est. = 1 n = 18417
>>
>>
>> I am not sure how this model works, but I guess it creates four smooths
>> for
>> each level of condition, and four smooths for each level of shape.
>>
>> There is also the intercept of the model, set at the reference level of
>> condition (CAT.pseudo) and at the reference level of shape (rand21). Each
>> parametric term represents the difference of each level of each of the two
>> factors from the intercept.
>>
>> I have two questions
>>
>> Q1:
>> Does anyone now why I get NA results in the second line of the parametric
>> terms?
>>
>> Q2:
>> The term igcCAT.ideo denotes the difference in the intercept between
>> (A): condition=igcCAT.ideo,  and
>> (B): (condition=igcCATpseudo ) &(shape=rand21).
>> But what is the value (level) of shape for (A)?
>> Is it the reference level? Or is it, perhaps, the "grand mean" of the
>> shape
>> variable?
>>
>>
>> Thank you in advance for your time,
>> Fotis
>>
>>
>>
>
> --
> Simon Wood, School of Mathematics, University of Bristol BS8 1TW UK
> +44 (0)117 33 18273 http://www.maths.bris.ac.uk/~sw15190
>
>


-- 
PhD Candidate
Department of Philosophy and History of Science
University of Athens, Greece.
http://users.uoa.gr/~aprotopapas/LLL/en/members.html#fotisfotiadis

Notice: Please do not use this account for social networks invitations, for
sending chain-mails to me, or as it were a facebook account. Thank you for
respecting my privacy.

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] R Course in Dublin (July 20th-22nd, 2016) Intoductory -> Modern

[Antony Unwin]

An R course from introductory to modern will be given by

Louis Aslett (Oxford University, author of the packages PhaseType and 
ReliabilityTheory)
and
Antony Unwin (author of the book “Graphical Data Analysis with R” CRC Press 
2015  http://www.gradaanwr.net).

The course will be offered again on September 7th-9th, 2016 in Dublin.

Details at  

http://insightsc.ie/training/r-statistical-software/ 



Antony Unwin
Insight Statistical Consulting, Dublin, Ireland
University of Augsburg, Germany




[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] Course: Introduction to Zero Inflated Models

[Highland Statistics Ltd]

There are places available on the following course:


Course: Introduction to Zero Inflated Models (Bayesian and frequentist 
approaches)

When: 13-17 June 2016

Where: Australian Institute of Marine Science, Perth, Australia

Course website: http://highstat.com/statscourse.htm

Course flyer: http://highstat.com/Courses/Flyers/Flyer2016_06Perth_ZI_V2.pdf

Keywords: Zero inflated count data. Zero inflated continuous data. 
Dependency. ZIP and ZAP models. Zero inflated GLMMs with random effects. 
Bayesian statistics, MCMC and JAGS. lme4, glmmADMB, JAGS. Overdispersion 
and solutions. Bayesian model selection.

Description: Suppose you want to study hippos and the effect of habitat 
variables on their distribution. When sampling, you may count zero 
hippos at many sites, potentially resulting in overdispersed Poisson 
GLMs.  In such cases zero inflated models can be applied. During the 
course several case studies are presented, in which the statistical 
theory for zero inflated models is integrated with applied analyses in a 
clear and understandable manner. Zero inflated models consist of two 
integrated GLMs and therefore we will start with a revision of GLM. Zero 
inflated GLMMs for nested data (repeated measurements, short time 
series, clustered data, etc.) are discussed in the second part of the 
course. We will focus on zero inflated count data, and zero inflated 
continuous data.




-- 
Dr. Alain F. Zuur

First author of:
1. Beginner's Guide to GAMM with R (2014).
2. Beginner's Guide to GLM and GLMM with R (2013).
3. Beginner's Guide to GAM with R (2012).
4. Zero Inflated Models and GLMM with R (2012).
5. A Beginner's Guide to R (2009).
6. Mixed effects models and extensions in ecology with R (2009).
7. Analysing Ecological Data (2007).

Highland Statistics Ltd.
9 St Clair Wynd
UK - AB41 6DZ Newburgh
Tel:   0044 1358 788177
Email: highs...@highstat.com
URL:   www.highstat.com


[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.