Thanks, everyone, for your suggestions. I'll start by trying out the bam->wig->bigWig route. I read on the FAQs that wigToBigWig takes more memory than a bedGraphToBigWig. I presume the wiggle file will be smaller to begin with, so it shouldn't pose an issue.
Shankar On Wed, Mar 16, 2011 at 12:13 AM, Angie Hinrichs <[email protected]> wrote: > Hi Shankar, > > Another suggestion: instead of bedGraph, try the more concise fixedStep > wiggle format, and wigToBigWig, as described here: > > https://lists.soe.ucsc.edu/pipermail/genome/2010-December/024318.html > > Hope that helps, > Angie > > ----- "Galt Barber" <[email protected]> wrote: > > > From: "Galt Barber" <[email protected]> > > To: "Hiram Clawson" <[email protected]> > > Cc: [email protected], "Shankar Ajay Subramanian" < > [email protected]> > > Sent: Tuesday, March 15, 2011 7:54:42 PM GMT -08:00 US/Canada Pacific > > Subject: Re: [Genome] bedGraphToBigWig memory issue > > > > Yes. I was thinking that perhaps > > a call or two to needlargemem > > should have called needhugemem > > instead. > > > > However since we could not reproduce it thar makes it hard to be > > sure. > > > > -galt > > > > On Mar 15, 2011, at 6:16 PM, Hiram Clawson <[email protected]> > > wrote: > > > > > > > > Go into the source tree and edit the file: src/lib/memalloc.c > > > change the definition of the maxAlloc variable: > > > > > > static size_t maxAlloc = ... > > > > > > Currently it allows 16 Gb in a single allocation. Add a factor > > > of 4 to make it 64 Gb. > > > > > > --Hiram > > > > > > ----- Original Message ----- > > > From: "Shankar Ajay Subramanian" <[email protected]> > > > To: "Galt Barber" <[email protected]> > > > Cc: [email protected] > > > Sent: Tuesday, March 15, 2011 5:32:41 PM > > > Subject: Re: [Genome] bedGraphToBigWig memory issue > > > > > > Hi Galt, > > > > > > Thanks for your response. I updated the executable yesterday with > > what > > > appeared to be a new version (updated Mar 14 on the UCSC website) > > > but I > > > still end up getting a similar error: > > > > > > needLargeMem: trying to allocate 34345569624 bytes (limit: > > > 17179869184) > > > > > > I forgot to mention that the program writes a partial bigWig file > > > (roughly > > > 6GB) before aborting. > > > > > > Let me know if you need additional information for debugging/ > > > diagnosis. > > > > > > Shankar > > > > > > On Tue, Mar 15, 2011 at 8:23 PM, Galt Barber <[email protected]> > > > wrote: > > > > > >> > > >> Hi, Shankar > > >> > > >> I tried to reproduce this error, but was unable to. > > >> I used a 34GB bedgraph file and had no trouble. > > >> > > >> Since this problem may have been fixed somewhat recently, > > >> I suggest you try downloading a fresh version > > >> of bedGraphToBigWig and run it again. > > >> > > >> -Galt > > >> > > >> > > >> 3/14/2011 8:29 PM, Shankar Ajay Subramanian: > > >> > > >> Hello, > > >>> > > >>> I'm trying to create a per-base coverage track from whole-genome > > >>> sequencing > > >>> data and I opted to use the bigWig format to display this on the > > > > >>> UCSC > > >>> browser. I downloaded the 64-bit linux 'bedGraphToBigWig' binary > > > > >>> utility > > >>> to > > >>> do so (Feb-18 version). > > >>> > > >>> The original read data are in BAM format (200GB), which I then > > >>> converted > > >>> to > > >>> bedGraph (49GB). When I tried to convert intermediate bedGraph > > >>> file to > > >>> bigWig I end up getting the following error even though the > > >>> process runs > > >>> on > > >>> a machine with>100G RAM: > > >>> > > >>> needLargeMem: trying to allocate 34345569624 bytes (limit: > > 4294967296 > > >>> ) > > >>> > > >>> From talking to our sys admin it appears that the utility calls a > > > > >>> library > > >>>> > > >>> where a hard-coded memory limit is specified for a 64-bit machine > > >>> (memalloc.c). Is there any way to get around this? I considered > > >>> down-sampling but didn't know if that would help since the > > >>> bedGraph file > > >>> is > > >>> likely to contain nearly as many data-points with Column4 values > > > > >>> being > > >>> lower. > > >>> > > >>> Appreciate your help. > > >>> > > >>> Best, > > >>> Shankar > > >>> > > >>> Postdoctoral fellow > > >>> NIH/NHGRI > > >>> > > >> > > >> > > > _______________________________________________ > > > Genome maillist - [email protected] > > > https://lists.soe.ucsc.edu/mailman/listinfo/genome > > _______________________________________________ > > Genome maillist - [email protected] > > https://lists.soe.ucsc.edu/mailman/listinfo/genome > _______________________________________________ Genome maillist - [email protected] https://lists.soe.ucsc.edu/mailman/listinfo/genome
