HI Rui,
thank you so much for this. I tried with the sqldf but it didn't help.
Next I tried your 2nd method and I was following your steps until:
> res2 <- asign[i2, setdiff(names(asign), names(l4))]
> m=merge(res, res2, by.x = c("chr", "pos"), by.y = c("chr", "pos"))
Error in merge.data.table(re
Hello,
Sometimes sqldf::sqldf tends to save memory. Maybe if you try
library(sqldf)
sqldf('select l4.*, asign.gene, asign.chr_pos, asign.`p.val.Retina`
from l4
inner join asign
on X1 = asign.chr and X2 = asign.pos')
Or you can filter the rows that match first, then merge the
Dear Ana
Since this appears to be genetics data have you thought of looking at
Bioconductor for help? I do not use genetic data-sets but people there
must use big files every day three times before breakfast.
Michael
On 24/10/2019 00:33, Ana Marija wrote:
thanks but I would need solution in
Ana... contributed packages like data.table and dplyr are developed completely
independently from R, have their own versions, and in fact both of them have
recommendations as to how to report bugs in their package descriptions.
As for getting help here, you really need to supply ALL of the infor
On 23/10/2019 7:04 p.m., Ana Marija wrote:
I also tried left_join but I got: Error: std::bad_alloc
df3 <- left_join(l4, asign, by = c("chr","pos")
Error: std::bad_alloc
Looks like bugs in whatever package you're finding "left_join" in (and
previously "merge"). Are those from dplyr and base
I am using R-3.6.1
and these libraries:
library(data.table)
library(dplyr)
On Wed, Oct 23, 2019 at 6:54 PM Duncan Murdoch wrote:
>
> On 23/10/2019 7:04 p.m., Ana Marija wrote:
> > I also tried left_join but I got: Error: std::bad_alloc
> >
> >> df3 <- left_join(l4, asign, by = c("chr","pos")
> >
I don't have it installed - that was merely a suggestion. I notice
that both data.table and dplyr packages are mentioned as possibilities
for "merge big datasets in r". Apparently the best way to do it if you
have a database manager is to read the two datasets into tables and do
the join via SQL or
no can you please send me an example how the command would look like in my case?
On Wed, Oct 23, 2019 at 6:16 PM Jim Lemon wrote:
>
> Yes. Have you tried the bigmemory package?
>
> Jim
>
> On Thu, Oct 24, 2019 at 10:08 AM Ana Marija
> wrote:
> >
> > Hi Jim,
> >
> > I think one of the issue is t
Yes. Have you tried the bigmemory package?
Jim
On Thu, Oct 24, 2019 at 10:08 AM Ana Marija wrote:
>
> Hi Jim,
>
> I think one of the issue is that data frames are so big,
> > dim(l4)
> [1] 166941635 8
> > dim(asign)
> [1] 107371528 5
>
> so my example would not reproduce the erro
thanks but I would need solution in R
On Wed, Oct 23, 2019 at 6:31 PM Jim Lemon wrote:
>
> I don't have it installed - that was merely a suggestion. I notice
> that both data.table and dplyr packages are mentioned as possibilities
> for "merge big datasets in r". Apparently the best way to do it
Hi Jim,
I think one of the issue is that data frames are so big,
> dim(l4)
[1] 166941635 8
> dim(asign)
[1] 107371528 5
so my example would not reproduce the error
On Wed, Oct 23, 2019 at 6:05 PM Jim Lemon wrote:
>
> Hi Ana,
> When I run this example taken from your email:
>
> l
Ah, it looks like a memory allocation problem.
Jim
On Thu, Oct 24, 2019 at 10:05 AM Ana Marija wrote:
>
> I also tried left_join but I got: Error: std::bad_alloc
>
> > df3 <- left_join(l4, asign, by = c("chr","pos"))
> Error: std::bad_alloc
> > dim(l4)
> [1] 166941635 8
> > dim(asign)
>
Hi Ana,
When I run this example taken from your email:
l4<-read.table(text="X1 X2 X3 X4 X5 variant_id pval_nominal gene_id.LCL
chr1 13550 G A b38 1:13550:G:A 0.375614 ENSG0227232
chr1 14671 G C b38 1:14671:G:C 0.474708 ENSG0227232
chr1 14677 G A b38 1:14677:G:A 0.699887
I also tried left_join but I got: Error: std::bad_alloc
> df3 <- left_join(l4, asign, by = c("chr","pos"))
Error: std::bad_alloc
> dim(l4)
[1] 166941635 8
> dim(asign)
[1] 107371528 5
On Wed, Oct 23, 2019 at 5:32 PM Ana Marija wrote:
>
> Hello,
>
> I have two data frames like thi
Hello,
I have two data frames like this:
> head(l4)
X1X2 X3 X4 X5 variant_id pval_nominal gene_id.LCL
1 chr1 13550 G A b38 1:13550:G:A 0.375614 ENSG0227232
2 chr1 14671 G C b38 1:14671:G:C 0.474708 ENSG0227232
3 chr1 14677 G A b38 1:14677:G:A 0.699887 ENSG0
15 matches
Mail list logo
