Re: [R] detect and replace outliers by the average
I think this discussion has gone off the rails to matters lying out of
the purview of this list.
Bert
On Fri, Apr 21, 2023 at 6:16 PM Ebert,Timothy Aaron wrote:
>
> Sometimes outliers happen. No matter the sample size there is always the
> possibility that one or more values are correct though highly improbable.
>
> -Original Message-
> From: R-help On Behalf Of Richard O'Keefe
> Sent: Friday, April 21, 2023 7:31 PM
> To: AbouEl-Makarim Aboueissa
> Cc: R mailing list
> Subject: Re: [R] detect and replace outliers by the average
>
> [External Email]
>
> This can be seen as three steps:
> (1) identify outliers
> (2) replace them with NA (trivial)
> (3) impute missing values.
> There are packages for imputing missing data.
> See
> https://www.analyticsvidhya.com/blog/2016/03/tutorial-powerful-packages-imputing-missing-values/
>
> Here I just want to address the first step.
> An observation is only an outlier relative to some model.
> Outliers can indicate
> - data that are just wrong (data entry error, failing battery in measurement
> device, all sorts of stuff). In this case, deletion + imputation makes
> sense.
> - data that are generated by a mixture of two or more processes,
> not the single process you thought was there. In this case,
> deletion + imputation is dangerous. The world is trying to tell
> you something and you are ignoring it.
> - the model is wrong. Here again, deletion + imputation is
> dangerous. You need a better model.
>
> "Detecting outliers in R" as a web query turned up
> https://statsandr.com/blog/outliers-detection-in-r/
> on the first page of results. There's plenty of material about finding
> outliers.
>
> But please give very VERY serious consideration to the possibility that some
> or even all of your outliers are actually GOOD data telling you something you
> need to know.
>
>
> On Fri, 21 Apr 2023 at 06:38, AbouEl-Makarim Aboueissa <
> abouelmakarim1...@gmail.com> wrote:
>
> > Dear All:
> >
> >
> >
> > *Re:* detect and replace outliers by the average
> >
> >
> >
> > The dataset, please see attached, contains a group factoring column "
> > *factor*" and two columns of data "x1" and "x2" with some NA values. I
> > need some help to detect the outliers and replace it and the NAs with
> > the average within each level (0,1,2) for each variable "x1" and "x2".
> >
> >
> >
> > I tried the below code, but it did not accomplish what I want to do.
> >
> >
> >
> >
> >
> > data<-read.csv("G:/20-Spring_2023/Outliers/data.csv", header=TRUE)
> >
> > data
> >
> > replace_outlier_with_mean <- function(x) {
> >
> > replace(x, x %in% boxplot.stats(x)$out, mean(x, na.rm=TRUE)) ,
> > na.rm=TRUE NOT working
> >
> > }
> >
> > data[] <- lapply(data, replace_outlier_with_mean)
> >
> >
> >
> >
> >
> > Thank you all very much for your help in advance.
> >
> >
> >
> >
> >
> > with many thanks
> >
> > abou
> >
> >
> > __
> >
> >
> > *AbouEl-Makarim Aboueissa, PhD*
> >
> > *Professor, Mathematics and Statistics* *Graduate Coordinator*
> >
> > *Department of Mathematics and Statistics* *University of Southern
> > Maine* __
> > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat/
> > .ethz.ch%2Fmailman%2Flistinfo%2Fr-help=05%7C01%7Ctebert%40ufl.edu
> > %7C1b625ca69ad442654a3e08db42c07f15%7C0d4da0f84a314d76ace60a62331e1b84
> > %7C0%7C0%7C638177166777282433%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
> > MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C
> > ta=TkZ0pb02TnNHZz94QtR5j%2BcYHwVJLLZRVqnMhmdxpz8%3D=0
> > PLEASE do read the posting guide
> > http://www.r/
> > -project.org%2Fposting-guide.html=05%7C01%7Ctebert%40ufl.edu%7C1b
> > 625ca69ad442654a3e08db42c07f15%7C0d4da0f84a314d76ace60a62331e1b84%7C0%
> > 7C0%7C638177166777282433%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> > CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Rw
> > %2F3iEOV%2Fu2bF16LPt8y8xt8aA9a0P8DsaeXYpo%2F97k%3D=0
> > and provide commented, minimal, self-contained, reproducible code.
> >
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.r-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
Re: [R] detect and replace outliers by the average
Sometimes outliers happen. No matter the sample size there is always the
possibility that one or more values are correct though highly improbable.
-Original Message-
From: R-help On Behalf Of Richard O'Keefe
Sent: Friday, April 21, 2023 7:31 PM
To: AbouEl-Makarim Aboueissa
Cc: R mailing list
Subject: Re: [R] detect and replace outliers by the average
[External Email]
This can be seen as three steps:
(1) identify outliers
(2) replace them with NA (trivial)
(3) impute missing values.
There are packages for imputing missing data.
See
https://www.analyticsvidhya.com/blog/2016/03/tutorial-powerful-packages-imputing-missing-values/
Here I just want to address the first step.
An observation is only an outlier relative to some model.
Outliers can indicate
- data that are just wrong (data entry error, failing battery in measurement
device, all sorts of stuff). In this case, deletion + imputation makes
sense.
- data that are generated by a mixture of two or more processes,
not the single process you thought was there. In this case,
deletion + imputation is dangerous. The world is trying to tell
you something and you are ignoring it.
- the model is wrong. Here again, deletion + imputation is
dangerous. You need a better model.
"Detecting outliers in R" as a web query turned up
https://statsandr.com/blog/outliers-detection-in-r/
on the first page of results. There's plenty of material about finding
outliers.
But please give very VERY serious consideration to the possibility that some or
even all of your outliers are actually GOOD data telling you something you need
to know.
On Fri, 21 Apr 2023 at 06:38, AbouEl-Makarim Aboueissa <
abouelmakarim1...@gmail.com> wrote:
> Dear All:
>
>
>
> *Re:* detect and replace outliers by the average
>
>
>
> The dataset, please see attached, contains a group factoring column "
> *factor*" and two columns of data "x1" and "x2" with some NA values. I
> need some help to detect the outliers and replace it and the NAs with
> the average within each level (0,1,2) for each variable "x1" and "x2".
>
>
>
> I tried the below code, but it did not accomplish what I want to do.
>
>
>
>
>
> data<-read.csv("G:/20-Spring_2023/Outliers/data.csv", header=TRUE)
>
> data
>
> replace_outlier_with_mean <- function(x) {
>
> replace(x, x %in% boxplot.stats(x)$out, mean(x, na.rm=TRUE)) ,
> na.rm=TRUE NOT working
>
> }
>
> data[] <- lapply(data, replace_outlier_with_mean)
>
>
>
>
>
> Thank you all very much for your help in advance.
>
>
>
>
>
> with many thanks
>
> abou
>
>
> __
>
>
> *AbouEl-Makarim Aboueissa, PhD*
>
> *Professor, Mathematics and Statistics* *Graduate Coordinator*
>
> *Department of Mathematics and Statistics* *University of Southern
> Maine* __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat/
> .ethz.ch%2Fmailman%2Flistinfo%2Fr-help=05%7C01%7Ctebert%40ufl.edu
> %7C1b625ca69ad442654a3e08db42c07f15%7C0d4da0f84a314d76ace60a62331e1b84
> %7C0%7C0%7C638177166777282433%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
> MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C
> ta=TkZ0pb02TnNHZz94QtR5j%2BcYHwVJLLZRVqnMhmdxpz8%3D=0
> PLEASE do read the posting guide
> http://www.r/
> -project.org%2Fposting-guide.html=05%7C01%7Ctebert%40ufl.edu%7C1b
> 625ca69ad442654a3e08db42c07f15%7C0d4da0f84a314d76ace60a62331e1b84%7C0%
> 7C0%7C638177166777282433%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=Rw
> %2F3iEOV%2Fu2bF16LPt8y8xt8aA9a0P8DsaeXYpo%2F97k%3D=0
> and provide commented, minimal, self-contained, reproducible code.
>
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.r-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] detect and replace outliers by the average
What does it mean when one column is just blank, neither a
number nor NA, just nothing?
On Fri, 21 Apr 2023 at 07:08, AbouEl-Makarim Aboueissa <
abouelmakarim1...@gmail.com> wrote:
> Dear All: the attached file in the .txt format
>
>
>
> *Re:* detect and replace outliers by the average
>
>
>
> The dataset, please see attached, contains a group factoring column “
> *factor*” and two columns of data “x1” and “x2” with some NA values. I need
> some help to detect the outliers and replace it and the NAs with the
> average within each level (0,1,2) for each variable “x1” and “x2”.
>
>
>
> I tried the below code, but it did not accomplish what I want to do.
>
>
>
> The average within each level should be computed after discard the
> outliers.
>
>
>
> data<-read.csv("G:/20-Spring_2023/Outliers/data.csv", header=TRUE)
>
> data
>
> replace_outlier_with_mean <- function(x) {
>
> replace(x, x %in% boxplot.stats(x)$out, mean(x, na.rm=TRUE)) ,
> na.rm=TRUE NOT working
>
> }
>
> data[] <- lapply(data, replace_outlier_with_mean)
>
>
>
>
>
> Thank you all very much for your help in advance.
>
>
>
>
>
> with many thanks
>
> abou
> __
>
>
> *AbouEl-Makarim Aboueissa, PhD*
>
> *Professor, Mathematics and Statistics*
> *Graduate Coordinator*
>
> *Department of Mathematics and Statistics*
> *University of Southern Maine*
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] detect and replace outliers by the average
This can be seen as three steps:
(1) identify outliers
(2) replace them with NA (trivial)
(3) impute missing values.
There are packages for imputing missing data.
See
https://www.analyticsvidhya.com/blog/2016/03/tutorial-powerful-packages-imputing-missing-values/
Here I just want to address the first step.
An observation is only an outlier relative to some model.
Outliers can indicate
- data that are just wrong (data entry error, failing battery in measurement
device, all sorts of stuff). In this case, deletion + imputation makes
sense.
- data that are generated by a mixture of two or more processes,
not the single process you thought was there. In this case,
deletion + imputation is dangerous. The world is trying to tell
you something and you are ignoring it.
- the model is wrong. Here again, deletion + imputation is
dangerous. You need a better model.
"Detecting outliers in R" as a web query turned up
https://statsandr.com/blog/outliers-detection-in-r/
on the first page of results. There's plenty of material
about finding outliers.
But please give very VERY serious consideration to the
possibility that some or even all of your outliers are
actually GOOD data telling you something you need to know.
On Fri, 21 Apr 2023 at 06:38, AbouEl-Makarim Aboueissa <
abouelmakarim1...@gmail.com> wrote:
> Dear All:
>
>
>
> *Re:* detect and replace outliers by the average
>
>
>
> The dataset, please see attached, contains a group factoring column “
> *factor*” and two columns of data “x1” and “x2” with some NA values. I need
> some help to detect the outliers and replace it and the NAs with the
> average within each level (0,1,2) for each variable “x1” and “x2”.
>
>
>
> I tried the below code, but it did not accomplish what I want to do.
>
>
>
>
>
> data<-read.csv("G:/20-Spring_2023/Outliers/data.csv", header=TRUE)
>
> data
>
> replace_outlier_with_mean <- function(x) {
>
> replace(x, x %in% boxplot.stats(x)$out, mean(x, na.rm=TRUE)) ,
> na.rm=TRUE NOT working
>
> }
>
> data[] <- lapply(data, replace_outlier_with_mean)
>
>
>
>
>
> Thank you all very much for your help in advance.
>
>
>
>
>
> with many thanks
>
> abou
>
>
> __
>
>
> *AbouEl-Makarim Aboueissa, PhD*
>
> *Professor, Mathematics and Statistics*
> *Graduate Coordinator*
>
> *Department of Mathematics and Statistics*
> *University of Southern Maine*
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Query about code
"Perhaps you could supply a reference to the work you are using?" ... in which case they should simply email the author directly, no? -- Bert On Fri, Apr 21, 2023 at 3:22 PM Jim Lemon wrote: > > Hi Yeswanth, > You seem to be referring to a specific publication by a specific > author. Unless someone in R-help knows who and what you are referring > to, it seems very difficult. Perhaps you could supply a reference to > the work you are using? > > JIm > > On Sat, Apr 22, 2023 at 7:03 AM ADIGARLA YESWANTH NAIDU > <102213...@smail.iitpkd.ac.in> wrote: > > > > I have been trying to write the code for the CCM analysis that you used in > > your study, but unfortunately, I haven't been able to write it. I was > > wondering if you would be willing to share the code with me. I understand > > that the code may be your intellectual property, but I assure you that I > > will use it solely for academic and non-commercial purposes. > > > > If you are able to share the code, I would greatly appreciate it. The code > > will be a valuable resource for me to understand the implementation details > > and reproduce the results of your study. Alternatively, if you are unable > > to share the code, I would be grateful for any guidance or suggestions you > > can provide in implementing the CCM analysis using the rEDM and > > multispatialCCM packages. > > > > Thank you very much for your consideration. I look forward to your positive > > response. > > > > Best regards, > > Yeswanth. > > > > [[alternative HTML version deleted]] > > > > __ > > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > > https://stat.ethz.ch/mailman/listinfo/r-help > > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > > and provide commented, minimal, self-contained, reproducible code. > > __ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Query about code
Hi Yeswanth, You seem to be referring to a specific publication by a specific author. Unless someone in R-help knows who and what you are referring to, it seems very difficult. Perhaps you could supply a reference to the work you are using? JIm On Sat, Apr 22, 2023 at 7:03 AM ADIGARLA YESWANTH NAIDU <102213...@smail.iitpkd.ac.in> wrote: > > I have been trying to write the code for the CCM analysis that you used in > your study, but unfortunately, I haven't been able to write it. I was > wondering if you would be willing to share the code with me. I understand > that the code may be your intellectual property, but I assure you that I > will use it solely for academic and non-commercial purposes. > > If you are able to share the code, I would greatly appreciate it. The code > will be a valuable resource for me to understand the implementation details > and reproduce the results of your study. Alternatively, if you are unable > to share the code, I would be grateful for any guidance or suggestions you > can provide in implementing the CCM analysis using the rEDM and > multispatialCCM packages. > > Thank you very much for your consideration. I look forward to your positive > response. > > Best regards, > Yeswanth. > > [[alternative HTML version deleted]] > > __ > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Query about code
I have been trying to write the code for the CCM analysis that you used in your study, but unfortunately, I haven't been able to write it. I was wondering if you would be willing to share the code with me. I understand that the code may be your intellectual property, but I assure you that I will use it solely for academic and non-commercial purposes. If you are able to share the code, I would greatly appreciate it. The code will be a valuable resource for me to understand the implementation details and reproduce the results of your study. Alternatively, if you are unable to share the code, I would be grateful for any guidance or suggestions you can provide in implementing the CCM analysis using the rEDM and multispatialCCM packages. Thank you very much for your consideration. I look forward to your positive response. Best regards, Yeswanth. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] [Rd] R 4.3.0 is released
The build system rolled up R-4.3.0.tar.gz (codename "Already Tomorrow") this morning. This is a major update, containing many new features and bug fixes. The list below details the changes in this release. You can get the source code from https://cran.r-project.org/src/base/R-4/R-4.3.0.tar.gz or wait for it to be mirrored at a CRAN site nearer to you. Binaries for various platforms will appear in due course. For the R Core Team, Peter Dalgaard These are the checksums (md5 and SHA-256) for the freshly created files, in case you wish to check that they are uncorrupted: MD5 (AUTHORS) = 320967884b547734d6279dedbc739dd4 MD5 (COPYING) = eb723b61539feef013de476e68b5c50a MD5 (COPYING.LIB) = a6f89e2100d9b6cdffcea4f398e37343 MD5 (FAQ) = 3f5d6539d3c4878cbbb74dcbee74447f MD5 (INSTALL) = 7893f754308ca31f1ccf62055090ad7b MD5 (NEWS) = a63858aea2a450900edf8c965cac1efa MD5 (NEWS.0) = bfcd7c147251b5474d96848c6f57e5a8 MD5 (NEWS.1) = 4108ab429e768e29b1c3b418c224246e MD5 (NEWS.2) = b38d94569700664205a76a7de836ba83 MD5 (NEWS.3) = e55ed2c8a547b827b46e08eb7137ba23 MD5 (R-latest.tar.gz) = f8f409fa66b82dbc79ddc5706f19638f MD5 (README) = f468f281c919665e276a1b691decbbe6 MD5 (RESOURCES) = a79b9b338cab09bd665f6b62ac6f455b MD5 (THANKS) = 45b6d2e88a6ecb5b24fa33a781351cd5 MD5 (VERSION-INFO.dcf) = cf5eba51ce402edaaa26eacdd4af55b5 MD5 (R-4/R-4.3.0.tar.gz) = f8f409fa66b82dbc79ddc5706f19638f 60a0d150e6fc1f424be76ad7b645d236b56e747692a4679f81ce6536c550e949 AUTHORS e6d6a009505e345fe949e1310334fcb0747f28dae2856759de102ab66b722cb4 COPYING 6095e9ffa777dd22839f7801aa845b31c9ed07f3d6bf8a26dc5d2dec8ccc0ef3 COPYING.LIB 8c396b0db6b1b072e66ea78deab8c89f71150744420ad9f74de6505aacee9a80 FAQ f87461be6cbaecc4dce44ac58e5bd52364b0491ccdadaf846cb9b452e9550f31 INSTALL be75619066603438a01d75f6f9ed741bc4fe15981eac900dd6dd0b1a4861dbb2 NEWS 4e21b62f515b749f80997063fceab626d7258c7d650e81a662ba8e0640f12f62 NEWS.0 5de7657c5e58e481403c0dd1a74a5c090b3ef481ce75a91dfe05d4b03f63163f NEWS.1 cde079b6beab7d700d3d4ecda494e2681ad3b7f8fab13b68be090f949393ec62 NEWS.2 1910a2405300b9bc7c76beeb0753a5249cf799afe175ce28f8d782fab723e012 NEWS.3 45dcc48b6cf27d361020f77fde1a39209e997b81402b3663ca1c010056a6a609 R-latest.tar.gz 2fdd3e90f23f32692d4b3a0c0452f2c219a10882033d1774f8cadf25886c3ddc README 8b7d3856100220f4555d4d57140829f2e81c27eccec5b441f5dce616e9ec9061 RESOURCES 8319c5415de58ee10d4bc058d79c370fd8e6b2ad09e25d7a1e04b74ca5f380a6 THANKS 3281640b70d461113ecb20020f3ed760bd30411c51657354bc3b322fa457fc49 VERSION-INFO.dcf 45dcc48b6cf27d361020f77fde1a39209e997b81402b3663ca1c010056a6a609 R-4/R-4.3.0.tar.gz This is the relevant part of the NEWS file CHANGES IN R 4.3.0: SIGNIFICANT USER-VISIBLE CHANGES: * Calling && or || with LHS or (if evaluated) RHS of length greater than one is now always an error, with a report of the form 'length = 4' in coercion to 'logical(1)' Environment variable _R_CHECK_LENGTH_1_LOGIC2_ no longer has any effect. NEW FEATURES: * The included BLAS sources have been updated to those shipped with LAPACK version 3.10.1. (This caused some platform-dependent changes to package check output.) And then to the sources from LAPACK version 3.11.0 (with changes only to double complex subroutines). * The included LAPACK sources have been updated to include the four Fortran 90 routines rather than their Fortran 77 predecessors. This may give some different signs in SVDs or eigendecompositions.. (This completes the transition to LAPACK 3.10.x begun in R 4.2.0.) * The LAPACK sources have been updated to version 3.11.0. (No new subroutines have been added, so this almost entirely bug fixes: Those fixes do affect some computations with NaNs, including R's NA.) * The parser now signals _classed_ errors, notably in case of the pipe operator |>. The error object and message now give line and column numbers, mostly as proposed and provided by Duncan Murdoch in PR#18328. * toeplitz() is now generalized for asymmetric cases, with a toeplitz2() variant. * xy.coords() and xyz.coords() and consequently, e.g., plot(x,y, log = "y") now signal a _classed_ warning about negative values of y (where log(.) is NA). Such a warning can be specifically suppressed or caught otherwise. * Regular expression functions now check more thoroughly whether their inputs are valid strings (in their encoding, e.g. in UTF-8). * The performance of grep(), sub(), gsub() and strsplit() has been improved, particularly with perl = TRUE and fixed = TRUE. Use of useBytes = TRUE for performance reasons should no longer be needed and is discouraged: it may lead to incorrect results. * apropos() gains an argument dot_internals which is used by the completion (help(rcompgen)) engine to also see base internals such as .POSIXct(). *
Re: [ESS] Emacs, ESS and Rmarkdown: Is this the way compile is supposed to work?
Hi Kevin,
If you have an older version of `poly-R`, the function `poly-gfm+r-mode` might
not be present. This was added to `poly-R` dev in July 2021, I'm not sure
exactly when it would have been added to the released version.
If that's the case, updating to the current version of `poly-R` should fix the
problem.
Alternatively, you need the following code evaluated *after* `(require
'poly-R)` and *before* `(add-to-list ...)`:
```
(define-polymode poly-gfm+r-mode poly-markdown+r-mode
:lighter " PM-Rmd(gfm)"
:hostmode 'poly-gfm-hostmode)
```
But that really shouldn't be necessary.
Best,
Tyler
--
plantarum.ca
On Fri, Apr 21, 2023, at 10:58 AM, Kevin Zembower via ESS-help wrote:
> Tyler, thanks for your response. I had found your website while
> searching for a solution to my problem.
>
> Currently, I have this in my .emacs.d/init.el file:
> (setq load-path (append '("~/.emacs.d/polymode/"
> "~/.emacs.d/polymode/modes") load-path))
> (require 'poly-R)
> (require 'poly-markdown)
> (add-to-list 'auto-mode-alist '("\\.Rmd" . poly-markdown+r-mode))
>
> This has been working for me without error so far.
>
> I commented out the last line and added this one:
> (add-to-list 'auto-mode-alist
>'("\\.[rR]md\\'" . poly-gfm+r-mode))
>
> When I save my init.el file and restart Emacs with a .Rmd file, I get
> this error message:
>
> File mode specification error: (void-function poly-gfm+r-mode)
> Package cl is deprecated
>
> I'm using GNU Emacs 27.1, with rmarkdown 2.21. I'm not able to install
> or find a poly-R or polymode package for this version of R
>
> Any ideas what I'm doing wrong? Thanks for any help and guidance.
>
> -Kevin
>
>
> On 4/21/23 06:00, ess-help-requ...@r-project.org wrote:
>> Message: 2
>> Date: Thu, 20 Apr 2023 13:55:19 -0400
>> From: "Tyler Smith"
>> To: ESS-help
>> Subject: Re: [ESS] Fwd: Emacs, ESS and Rmarkdown: Is this the way
>> compile is supposed to work?
>> Message-ID:<98fe4222-3e4f-408a-8773-bde2cf87b...@app.fastmail.com>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> Hi Kevin,
>>
>> You should be able to do what you want directly from the `.Rmd` buffer. I
>> posted a tutorial here:
>>
>> https://plantarum.ca/2021/10/03/emacs-tutorial-rmarkdown/
>>
>> With the right packages installed, from the .Rmd buffer you can export your
>> file with `M-n e`, select `markdown` and `html` when prompted, and the html
>> will be generated and opened in a web browser.
>>
>> The tutorial walks you through the steps in more detail.
>>
>> Best,
>>
>> Tyler
>>
>> -- plantarum.ca
>
>
> __
> ESS-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/ess-help
__
ESS-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/ess-help
[ESS] Emacs, ESS and Rmarkdown: Is this the way compile is supposed to work?
Tyler, thanks for your response. I had found your website while
searching for a solution to my problem.
Currently, I have this in my .emacs.d/init.el file:
(setq load-path (append '("~/.emacs.d/polymode/"
"~/.emacs.d/polymode/modes") load-path))
(require 'poly-R)
(require 'poly-markdown)
(add-to-list 'auto-mode-alist '("\\.Rmd" . poly-markdown+r-mode))
This has been working for me without error so far.
I commented out the last line and added this one:
(add-to-list 'auto-mode-alist
'("\\.[rR]md\\'" . poly-gfm+r-mode))
When I save my init.el file and restart Emacs with a .Rmd file, I get
this error message:
File mode specification error: (void-function poly-gfm+r-mode)
Package cl is deprecated
I'm using GNU Emacs 27.1, with rmarkdown 2.21. I'm not able to install
or find a poly-R or polymode package for this version of R
Any ideas what I'm doing wrong? Thanks for any help and guidance.
-Kevin
On 4/21/23 06:00, ess-help-requ...@r-project.org wrote:
> Message: 2
> Date: Thu, 20 Apr 2023 13:55:19 -0400
> From: "Tyler Smith"
> To: ESS-help
> Subject: Re: [ESS] Fwd: Emacs, ESS and Rmarkdown: Is this the way
> compile is supposed to work?
> Message-ID:<98fe4222-3e4f-408a-8773-bde2cf87b...@app.fastmail.com>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi Kevin,
>
> You should be able to do what you want directly from the `.Rmd` buffer. I
> posted a tutorial here:
>
> https://plantarum.ca/2021/10/03/emacs-tutorial-rmarkdown/
>
> With the right packages installed, from the .Rmd buffer you can export your
> file with `M-n e`, select `markdown` and `html` when prompted, and the html
> will be generated and opened in a web browser.
>
> The tutorial walks you through the steps in more detail.
>
> Best,
>
> Tyler
>
> -- plantarum.ca
__
ESS-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/ess-help
Re: [R] detect and replace outliers by the averaged
Hello,
Às 09:42 de 21/04/2023, Jeff Newmiller escreveu:
0
Somewhat cryptic...
Rui Barradas
On April 21, 2023 4:08:08 AM GMT+09:00, Dr Eberhard W Lisse
wrote:
There is at least one outliers package on CRAN.
el
On 20/04/2023 20:43, AbouEl-Makarim Aboueissa wrote:
Dear All: *please discard my previous email*
*Re:* detect and replace outliers by the average
The dataset, please see attached, contains a group factoring column “
*factor*” and two columns of data “x1” and “x2” with some NA values. I need
some help to detect the outliers and replace it and the NAs with the
average within each level (0,1,2) for each variable “x1” and “x2”.
I tried the below code, but it did not accomplish what I want to do.
data<-read.csv("G:/20-Spring_2023/Outliers/data.csv", header=TRUE)
data
replace_outlier_with_mean <- function(x) {
replace(x, x %in% boxplot.stats(x)$out, mean(x, na.rm=TRUE)) ,
na.rm=TRUE NOT working
}
data[] <- lapply(data, replace_outlier_with_mean)
Thank you all very much for your help in advance.
with many thanks
abou
__
*AbouEl-Makarim Aboueissa, PhD*
*Professor, Mathematics and Statistics*
*Graduate Coordinator*
*Department of Mathematics and Statistics*
*University of Southern Maine*
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Warning message: In length(barlist) == 1 && as.character(barlist[[1]][[2]]) == "1"
On Thu, 20 Apr 2023 10:11:03 +0200
Giozz Lau wrote:
> I am writing to seek assistance regarding a warning message that I
> encountered while running a CLMM with two main factors and one random
> factor ("participants"). One of the main factors consists of six
> levels that can be either categorical or numerical.
Debugging, like the CLMM modelling that you're trying to perform here,
can be described as a kind of inverse problem: given experimental data
('Some kind of warning is raised by length(barlist) == 1 &&
as.character(barlist[[1]][[2]]) == "1"'), one intends to model the
state of the program at the time when the problem happens (since it's
the whole expression that raises the warning and not any of its parts,
perhaps barlist[[1]][[2]] is of length() > 1? But that's an error in
recent versions of R, and used to be a warning in a few older
versions...) and go about solving the problem from there.
Sometimes, there's not enough experimental data, and the problem
becomes ill-posed: strictly speaking, given the information above,
there can be too many possible explanations for what you're
experiencing. I see you've provided some information, but it's better
to start with the full error message, the code you're running and a
small piece of data that would reproduce the error on a different
computer.
Sometimes, it's possible to apply one's psychic debugging skills (read:
make educated guesses and hope they are right) to regularise the
ill-posed problem and solve it anyway. Searching CRAN for "clmm" yields
the "ordinal" package, which has a clmm() function, which calls another
function to operate on a variable called "barlist" and which also has
been recently (November 2022) changed to read:
>> length(barlist) == 1 && as.character(barlist[[1]][[2]])[1] == "1"
(Note the extra [1] compared to the expression in your warning message.)
Have you tried upgrading to ordinal version 2022.11-16?
If that doesn't help, please provide the full warning, the code and the
data to reproduce it, and the sessionInfo() output.
--
Best regards,
Ivan
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] Warning message: In length(barlist) == 1 && as.character(barlist[[1]][[2]]) == "1"
Hello,
I am writing to seek assistance regarding a warning message that I
encountered while running a CLMM with two main factors and one random
factor ("participants"). One of the main factors consists of six
levels that can be either categorical or numerical.
As this is my first time running this type of model, I would greatly
appreciate your expertise in understanding and resolving this issue.
Thank you in advance for your time and support.
Sincerely,
Gio.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] detect and replace outliers by the averaged
0
On April 21, 2023 4:08:08 AM GMT+09:00, Dr Eberhard W Lisse
wrote:
>There is at least one outliers package on CRAN.
>
>el
>On 20/04/2023 20:43, AbouEl-Makarim Aboueissa wrote:
>> Dear All: *please discard my previous email*
>>
>>
>>
>> *Re:* detect and replace outliers by the average
>>
>>
>>
>> The dataset, please see attached, contains a group factoring column “
>> *factor*” and two columns of data “x1” and “x2” with some NA values. I need
>> some help to detect the outliers and replace it and the NAs with the
>> average within each level (0,1,2) for each variable “x1” and “x2”.
>>
>>
>>
>> I tried the below code, but it did not accomplish what I want to do.
>>
>>
>>
>>
>>
>> data<-read.csv("G:/20-Spring_2023/Outliers/data.csv", header=TRUE)
>>
>> data
>>
>> replace_outlier_with_mean <- function(x) {
>>
>> replace(x, x %in% boxplot.stats(x)$out, mean(x, na.rm=TRUE)) ,
>> na.rm=TRUE NOT working
>>
>> }
>>
>> data[] <- lapply(data, replace_outlier_with_mean)
>>
>>
>>
>>
>>
>> Thank you all very much for your help in advance.
>>
>>
>>
>>
>>
>> with many thanks
>>
>> abou
>>
>>
>> __
>>
>>
>> *AbouEl-Makarim Aboueissa, PhD*
>>
>> *Professor, Mathematics and Statistics*
>> *Graduate Coordinator*
>>
>> *Department of Mathematics and Statistics*
>> *University of Southern Maine*
>
>__
>R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.
--
Sent from my phone. Please excuse my brevity.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] glm.nb and Error in x[good, , drop = FALSE] * w : non-conformable arrays
Many thanks Ivan ! This is fairly clear to me, now... When I dumped the data.frame, I found strange to have a "table" declaration for deg, but was not able to judge if it was a problem or not (I would have expected something as "numeric") Your workaround is fine to me (I do not need to carry on with table attributes). Best, Patrick Le 21/04/2023 à 10:08, Ivan Krylov a écrit : > On Fri, 21 Apr 2023 09:02:37 +0200 > Patrick Giraudoux wrote: > >> I meet an error with glm.nb that I cannot explain the origin (and >> find a fix). The model I want to fit is the following: >> >> library(MASS) >> >> glm.nb(deg~offset(log(durobs))+zone,data=db) >> >> and the data.frame is dumped below. > Thank you for providing both the code and a small piece of data that > reproduces the error! > > (It almost worked. Your mailer automatically generated a plain text > version of the e-mail and put Unicode non-breaking spaces in there. R > considers it a syntax error to encounter any of the various Unicode > space-like characters outside string literals.) > >> deg = structure(c(0, 1, 0, 3, 0, 1, 0, 2, 1, 0, 3, 0, 0, 0, 4, 1, 0, >> 0, 0, 0, 4, 0, 0, 0, 4, 3, 2, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, >> 0, 3, 2, 3, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 2, >> 0, 0, 0, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 2, >> 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 1, 0, 0, 1, 3, 2, >> 1, 2, 0, 0, 0, 0, 0, 1, 0, 0, 2, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 2, >> 1, 1, 0), dim = 135L, class = "table") > The problem is that `deg` is a table, which ends up making the > effective weights a table too. tables are arrays, and element-wise > product rules are stricter for them than for plain matrices. The code > makes use of the ability to take an element-wise product between a > matrix and a vector of the same length as the number of rows in the > matrix: > > matrix(1:12, 4) * 1:4 # works > matrix(1:12, 4) * as.array(1:4) # results in the same error > > # the right way to take products with an array is to make sure that the > # shapes match exactly > matrix(1:12, 4) * as.array(cbind(1:4, 1:4, 1:4)) > > One possible solution is to to remove all attributes from db$deg: > > db$deg <- as.vector(db$deg) > > This way the values of the expressions involved in glm.fit end up being > of the expected type, and the function completes successfully. > [[alternative HTML version deleted]] __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] glm.nb and Error in x[good, , drop = FALSE] * w : non-conformable arrays
On Fri, 21 Apr 2023 09:02:37 +0200 Patrick Giraudoux wrote: > I meet an error with glm.nb that I cannot explain the origin (and > find a fix). The model I want to fit is the following: > > library(MASS) > > glm.nb(deg~offset(log(durobs))+zone,data=db) > > and the data.frame is dumped below. Thank you for providing both the code and a small piece of data that reproduces the error! (It almost worked. Your mailer automatically generated a plain text version of the e-mail and put Unicode non-breaking spaces in there. R considers it a syntax error to encounter any of the various Unicode space-like characters outside string literals.) > deg = structure(c(0, 1, 0, 3, 0, 1, 0, 2, 1, 0, 3, 0, 0, 0, 4, 1, 0, > 0, 0, 0, 4, 0, 0, 0, 4, 3, 2, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, > 0, 3, 2, 3, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 2, > 0, 0, 0, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 2, > 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 1, 0, 0, 1, 3, 2, > 1, 2, 0, 0, 0, 0, 0, 1, 0, 0, 2, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 2, > 1, 1, 0), dim = 135L, class = "table") The problem is that `deg` is a table, which ends up making the effective weights a table too. tables are arrays, and element-wise product rules are stricter for them than for plain matrices. The code makes use of the ability to take an element-wise product between a matrix and a vector of the same length as the number of rows in the matrix: matrix(1:12, 4) * 1:4 # works matrix(1:12, 4) * as.array(1:4) # results in the same error # the right way to take products with an array is to make sure that the # shapes match exactly matrix(1:12, 4) * as.array(cbind(1:4, 1:4, 1:4)) One possible solution is to to remove all attributes from db$deg: db$deg <- as.vector(db$deg) This way the values of the expressions involved in glm.fit end up being of the expected type, and the function completes successfully. -- Best regards, Ivan __ R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] glm.nb and Error in x[good, , drop = FALSE] * w : non-conformable arrays
Dear Listers,
I meet an error with glm.nb that I cannot explain the origin (and find a
fix). The model I want to fit is the following:
library(MASS)
glm.nb(deg~offset(log(durobs))+zone,data=db)
and the data.frame is dumped below.
Has anyone an idea about what the trouble comes from ? (except computing
leads to a non-conformable array somewhere... the question is why;
fitting goes through without any problem eg with a Poisson link)
Best,
Patrick
db <-
structure(list(deg = structure(c(0, 1, 0, 3, 0, 1, 0, 2, 1, 0,
3, 0, 0, 0, 4, 1, 0, 0, 0, 0, 4, 0, 0, 0, 4, 3, 2, 0, 0, 0, 0,
0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 3, 2, 3, 0, 1, 1, 0, 0, 0, 0, 1,
0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 2, 0, 0, 0, 2, 1, 0, 2, 0, 1, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 2, 1, 0, 1, 0, 0, 0, 0, 0,
0, 0, 0, 0, 1, 0, 0, 0, 2, 1, 0, 0, 1, 3, 2, 1, 2, 0, 0, 0, 0,
0, 1, 0, 0, 2, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 2, 1, 1, 0), dim = 135L,
class = "table"),
durobs = c(371, 371, 371, 371, 371, 371, 239, 266, 234, 71,
436, 407, 407, 414, 415, 418, 415, 329, 414, 414, 415, 330,
435, 436, 210, 436, 214, 436, 436, 210, 434, 438, 438, 402,
402, 289, 264, 264, 434, 435, 434, 434, 434, 434, 434, 427,
427, 427, 328, 422, 291, 412, 221, 417, 416, 416, 79, 322,
213, 440, 434, 462, 397, 457, 419, 406, 316, 392, 392, 392,
392, 392, 452, 386, 399, 305, 240, 404, 226, 226, 381, 385,
392, 388, 388, 391, 396, 392, 385, 385, 385, 237, 378, 378,
378, 381, 126, 315, 379, 314, 185, 313, 312, 301, 312, 312,
310, 310, 307, 306, 304, 455, 472, 472, 466, 467, 334, 565,
429, 429, 425, 422, 421, 419, 417, 417, 410, 405, 195, 422,
419, 419, 426, 426, 442), zone = c("MO1", "MO1", "MO1", "MO1",
"MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1",
"MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1",
"MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1",
"MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1",
"MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1",
"MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1",
"MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1", "MO1",
"MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2",
"MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2",
"MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2",
"MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2",
"MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2",
"MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2",
"MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2", "MO2",
"MO2", "MO2", "MO2", "MO2", "MO2")), row.names = c(NA, 135L
), class = "data.frame")
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
