Good Morning Shankar: I'm beginning to believe you have a particular input file that is triggering a problem of some sort in the bedGraphToBigWig command. That should be the most efficient memory usage converter command.
Could you provide a URL to your original source file so we can take a look at the conversion situation. See also: http://genomewiki.ucsc.edu/index.php/Big_file_converters --Hiram Shankar Ajay Subramanian wrote: > Thanks, everyone, for your suggestions. I'll start by trying out the > bam->wig->bigWig route. I read on the FAQs that wigToBigWig takes more > memory than a bedGraphToBigWig. I presume the wiggle file will be smaller to > begin with, so it shouldn't pose an issue. > > Shankar > > On Wed, Mar 16, 2011 at 12:13 AM, Angie Hinrichs <[email protected]> wrote: > >> Hi Shankar, >> >> Another suggestion: instead of bedGraph, try the more concise fixedStep >> wiggle format, and wigToBigWig, as described here: >> >> https://lists.soe.ucsc.edu/pipermail/genome/2010-December/024318.html >> >> Hope that helps, >> Angie >> >> ----- "Galt Barber" <[email protected]> wrote: >> >>> From: "Galt Barber" <[email protected]> >>> To: "Hiram Clawson" <[email protected]> >>> Cc: [email protected], "Shankar Ajay Subramanian" < >> [email protected]> >>> Sent: Tuesday, March 15, 2011 7:54:42 PM GMT -08:00 US/Canada Pacific >>> Subject: Re: [Genome] bedGraphToBigWig memory issue >>> >>> Yes. I was thinking that perhaps >>> a call or two to needlargemem >>> should have called needhugemem >>> instead. >>> >>> However since we could not reproduce it thar makes it hard to be >>> sure. >>> >>> -galt >>> >>> On Mar 15, 2011, at 6:16 PM, Hiram Clawson <[email protected]> >>> wrote: >>> >>>> Go into the source tree and edit the file: src/lib/memalloc.c >>>> change the definition of the maxAlloc variable: >>>> >>>> static size_t maxAlloc = ... >>>> >>>> Currently it allows 16 Gb in a single allocation. Add a factor >>>> of 4 to make it 64 Gb. >>>> >>>> --Hiram >>>> >>>> ----- Original Message ----- >>>> From: "Shankar Ajay Subramanian" <[email protected]> >>>> To: "Galt Barber" <[email protected]> >>>> Cc: [email protected] >>>> Sent: Tuesday, March 15, 2011 5:32:41 PM >>>> Subject: Re: [Genome] bedGraphToBigWig memory issue >>>> >>>> Hi Galt, >>>> >>>> Thanks for your response. I updated the executable yesterday with >>> what >>>> appeared to be a new version (updated Mar 14 on the UCSC website) >>>> but I >>>> still end up getting a similar error: >>>> >>>> needLargeMem: trying to allocate 34345569624 bytes (limit: >>>> 17179869184) >>>> >>>> I forgot to mention that the program writes a partial bigWig file >>>> (roughly >>>> 6GB) before aborting. >>>> >>>> Let me know if you need additional information for debugging/ >>>> diagnosis. >>>> >>>> Shankar >>>> >>>> On Tue, Mar 15, 2011 at 8:23 PM, Galt Barber <[email protected]> >>>> wrote: >>>> >>>>> Hi, Shankar >>>>> >>>>> I tried to reproduce this error, but was unable to. >>>>> I used a 34GB bedgraph file and had no trouble. >>>>> >>>>> Since this problem may have been fixed somewhat recently, >>>>> I suggest you try downloading a fresh version >>>>> of bedGraphToBigWig and run it again. >>>>> >>>>> -Galt >>>>> >>>>> >>>>> 3/14/2011 8:29 PM, Shankar Ajay Subramanian: >>>>> >>>>> Hello, >>>>>> I'm trying to create a per-base coverage track from whole-genome >>>>>> sequencing >>>>>> data and I opted to use the bigWig format to display this on the >>>>>> UCSC >>>>>> browser. I downloaded the 64-bit linux 'bedGraphToBigWig' binary >>>>>> utility >>>>>> to >>>>>> do so (Feb-18 version). >>>>>> >>>>>> The original read data are in BAM format (200GB), which I then >>>>>> converted >>>>>> to >>>>>> bedGraph (49GB). When I tried to convert intermediate bedGraph >>>>>> file to >>>>>> bigWig I end up getting the following error even though the >>>>>> process runs >>>>>> on >>>>>> a machine with>100G RAM: >>>>>> >>>>>> needLargeMem: trying to allocate 34345569624 bytes (limit: >>> 4294967296 >>>>>> ) >>>>>> >>>>>> From talking to our sys admin it appears that the utility calls a >>>>>> library >>>>>> where a hard-coded memory limit is specified for a 64-bit machine >>>>>> (memalloc.c). Is there any way to get around this? I considered >>>>>> down-sampling but didn't know if that would help since the >>>>>> bedGraph file >>>>>> is >>>>>> likely to contain nearly as many data-points with Column4 values >>>>>> being >>>>>> lower. >>>>>> >>>>>> Appreciate your help. >>>>>> >>>>>> Best, >>>>>> Shankar >>>>>> >>>>>> Postdoctoral fellow >>>>>> NIH/NHGRI _______________________________________________ Genome maillist - [email protected] https://lists.soe.ucsc.edu/mailman/listinfo/genome
