Hi Henrik: Thank you for patching this so quickly. I didn't realize the defaults of file.copy() had changed in R 2.13.0.
I'll test this once my current job (n = ~400 CEL files) is complete, and I'll let you know what I find. Taylor On Friday, September 5, 2014 4:15:56 PM UTC-4, Henrik Bengtsson wrote: > > I've updated aroma.affymetrix v2.11.8 and friends. Update by: > > source("http://callr.org/install#aroma.affymetrix") > > I'd be keen to hear whether this solves your problem or not. > > /Henrik > > On Thu, Sep 4, 2014 at 1:47 PM, Henrik Bengtsson <h...@biostat.ucsf.edu > <javascript:>> wrote: > > Hi, > > > > good that it works for you with this workaround. The fix in aroma is > > fairly simple(*). I'll let you know when an updated version is > > available. > > > > (*) DETAILS: This has to do with base::file.copy() and its argument > > 'copy.mode' which was introduced in R 2.13.0 (April 2011). The thing > > is that they set the default to TRUE (whereas the previous behavior > > was equivalent to FALSE), meaning any files copied will inherit the > > file permissions from the source. Various steps in the aroma pipeline > > *copies* the source CEL file and uses that as a template to store > > updates signals. This is what hits you. I'll updating to make use of > > base::file.copy(..., copy.mode=FALSE) such that one can work with also > > read-only CEL files. > > > > Thanks for reporting on this (not too unlikely) use case > > > > Henrik > > > > On Thu, Sep 4, 2014 at 1:35 PM, Taylor Raborn <taylor...@gmail.com > <javascript:>> wrote: > >> Hi Henrik: > >> > >> Thank you for the prompt reply. You are right- the .CEL files are all > read > >> only (see their permissions, below) > >> > >> rtraborn@Mason: > >> > /N/dc2/scratch/rtraborn/ML_Project/NIK_2014/rawData/NIK_2014/HuEx-1_0-st-v2> > > >> ls -lkhtra > >> > >> > >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709338_HSB194-VFC-R.CEL > >> > >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709337_HSB194-VFC-L.CEL > >> > >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709336_HSB194-V1C-R.CEL > >> > >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709335_HSB194-V1C-L.CEL > >> > >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709334_HSB194-STR-L.CEL > >> > >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709333_HSB194-STC-R.CEL > >> > >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709332_HSB194-STC-L.CEL > >> > >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709331_HSB194-S1C-R.CEL > >> > >> -r--r--r-- 1 ... 63M Jan 25 2011 GSM709330_HSB194-S1C-L.CEL > >> > >> > >> After changing the permissions, the job performed without issue (I'm > showing > >> you only the results of 1/9 for the sake of brevity): > >> > >>> csBC <- process(bc,verbose=verbose) > >> > >> Background correcting data set... > >> > >> Number of arrays: 9 > >> > >> Array #1 ('GSM709330_HSB194-S1C-L') of 9... > >> > >> Adjusting PM signals only > >> > >> Obtaining signals... > >> > >> Obtaining signals...done > >> > >> Applying normal+exponential signal model... > >> > >> Applying normal+exponential signal model...done > >> > >> Writing adjusted probe signals... > >> > >> Adding temporary suffix from file... > >> > >> Pathname: > >> probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL > >> > >> Suffix: .tmp > >> > >> Rename existing file?: FALSE > >> > >> Temporary pathname: > >> > probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp > >> > >> Adding temporary suffix from file...done > >> > >> Creating CEL file for results, if missing... > >> > >> Creating CEL file for results, if missing...done > >> > >> Writing adjusted intensities... > >> > >> Writing adjusted intensities...done > >> > >> Dropping temporary suffix from file... > >> > >> Temporary pathname: > >> > probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp > >> > >> Suffix: .tmp > >> > >> Regular expression for suffix: \.tmp$ > >> > >> Pathname: > >> probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL > >> > >> Renaming existing file... > >> > >> Result: TRUE > >> > >> Renaming existing file...done > >> > >> Dropping temporary suffix from file...done > >> > >> Writing adjusted probe signals...done > >> > >> used (Mb) gc trigger (Mb) max used (Mb) > >> > >> Ncells 709078 37.9 1166886 62.4 1166886 62.4 > >> > >> Vcells 31586985 241.0 47054688 359.0 46991036 358.6 > >> > >> AffymetrixCelFile: > >> > >> Name: GSM709330_HSB194-S1C-L > >> > >> Tags: > >> > >> Full name: GSM709330_HSB194-S1C-L > >> > >> Pathname: > >> probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL > >> > >> File size: 62.73 MB (65775403 bytes) > >> > >> RAM: 0.00 MB > >> > >> File format: v4 (binary; XDA) > >> > >> Platform: Affymetrix > >> > >> Chip type: HuEx-1_0-st-v2,coreR3,A20071112,EP > >> > >> Timestamp: 2011-01-25 12:15:03 > >> > >> Array #1 ('GSM709330_HSB194-S1C-L') of 9...done > >> > >> > >> ----- > >> > >> > >> There must be a file permissions transfer behavior that takes place on > a > >> multi-user cluster environment that doesn't take place on other types > of > >> machines, because I was unable to reproduce this error elsewhere. I > didn't > >> bother changing the raw .CEL files' permissions because I never > experienced > >> a problem of this nature with .CEL files. > >> > >> > >> Let me know if you patch this up so I can pull the latest code from the > >> aroma repo. > >> > >> > >> Thanks for your help with this and for taking a look at the issue so > >> quickly. > >> > >> > >> Best regards, > >> > >> > >> Taylor > >> > >> > >> > >> On Thursday, September 4, 2014 1:37:36 PM UTC-4, Henrik Bengtsson > wrote: > >>> > >>> Hi, > >>> > >>> interesting. A quick guess is that the input file, i.e. > >>> > >>> rawData/NIK_2014/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL > >>> > >>> is write protected (e.g. file owned by someone else). This could be > >>> the reason, because internally that file is copied and used as a > >>> template and the copy may inherit the file permissions. What does ls > >>> -l say about the above file? > >>> > >>> If this is the case, I need to update the copying such that the new > >>> file has proper permissions. > >>> > >>> Thanks for the report > >>> > >>> Henrik > >>> > >>> > >>> On Thu, Sep 4, 2014 at 10:15 AM, Taylor Raborn <taylor...@gmail.com> > >>> wrote: > >>> > Hi Henrik: > >>> > > >>> > I just come across an unusual error when using aroma.affymetrix on > our > >>> > high-memory HPC machine. It may or may not be platform-specific, and > >>> > I'll > >>> > give you all the information I can in the hopes of isolating and > >>> > identifying > >>> > the error. > >>> > > >>> > Traceback: > >>> > > >>> >> traceback() > >>> > > >>> > 18: stop(cond) > >>> > > >>> > 17: throw.Exception(Exception(...)) > >>> > > >>> > 16: throw(Exception(...)) > >>> > > >>> > 15: throw.default("No permission to modify existing file: ", > pathname) > >>> > > >>> > 14: throw("No permission to modify existing file: ", pathname) > >>> > > >>> > 13: getWritablePathname.Arguments(static, ...) > >>> > > >>> > 12: getWritablePathname(static, ...) at <text>#1 > >>> > > >>> > 11: Arguments$getWritablePathname(pathname, mustExist = TRUE) > >>> > > >>> > 10: renameFile.default(srcPathname, pathname, ...) > >>> > > >>> > 9: renameFile(srcPathname, pathname, ...) > >>> > > >>> > 8: renameTo.GenericDataFile(res, filename = pathname, verbose = > >>> > less(verbose)) > >>> > > >>> > 7: renameTo(res, filename = pathname, verbose = less(verbose)) > >>> > > >>> > 6: createFrom.AffymetrixCelFile(this, filename = pathnameT, path = > NULL, > >>> > > >>> > verbose = less(verbose)) > >>> > > >>> > 5: createFrom(this, filename = pathnameT, path = NULL, verbose = > >>> > less(verbose)) > >>> > > >>> > 4: bgAdjustRma.AffymetrixCelFile(df, path = outputPath, pmonly = > pmonly, > >>> > > >>> > addJitter = addJitter, jitterSd = jitterSd, overwrite = > force, > >>> > > >>> > verbose = verbose, .deprecated = FALSE) > >>> > > >>> > 3: bgAdjustRma(df, path = outputPath, pmonly = pmonly, addJitter = > >>> > addJitter, > >>> > > >>> > jitterSd = jitterSd, overwrite = force, verbose = verbose, > >>> > > >>> > .deprecated = FALSE) > >>> > > >>> > 2: process.RmaBackgroundCorrection(bc, verbose = verbose) > >>> > > >>> > 1: process(bc, verbose = verbose) > >>> > > >>> > > >>> > ---- > >>> > > >>> > sessionInfo: > >>> > > >>> >> sessionInfo() > >>> > > >>> > R version 3.0.1 (2013-05-16) > >>> > > >>> > Platform: x86_64-unknown-linux-gnu (64-bit) > >>> > > >>> > > >>> > locale: > >>> > > >>> > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C > >>> > > >>> > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 > >>> > > >>> > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 > >>> > > >>> > [7] LC_PAPER=C LC_NAME=C > >>> > > >>> > [9] LC_ADDRESS=C LC_TELEPHONE=C > >>> > > >>> > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C > >>> > > >>> > > >>> > attached base packages: > >>> > > >>> > [1] tools parallel stats graphics grDevices utils > datasets > >>> > > >>> > [8] methods base > >>> > > >>> > > >>> > other attached packages: > >>> > > >>> > [1] R.cache_0.10.0 base64enc_0.1-2 aroma.apd_0.5.0 > >>> > > >>> > [4] preprocessCore_1.24.0 affyio_1.30.0 Biobase_2.20.1 > >>> > > >>> > [7] BiocGenerics_0.6.0 aroma.light_1.32.0 > matrixStats_0.10.0 > >>> > > >>> > [10] aroma.affymetrix_2.12.0 aroma.core_2.12.1 > R.devices_2.11.0 > >>> > > >>> > [13] R.filesets_2.6.0 R.utils_1.33.0 R.oo_1.18.0 > >>> > > >>> > [16] affxparser_1.34.2 affy_1.40.0 > R.methodsS3_1.6.1 > >>> > > >>> > [19] PSCBS_0.43.0 DNAcopy_1.36.0 > >>> > BiocInstaller_1.12.1 > >>> > > >>> > > >>> > loaded via a namespace (and not attached): > >>> > > >>> > [1] digest_0.6.4 R.huge_0.8.0 R.rsp_0.19.0 zlibbioc_1.11.1 > >>> > > >>> > > >>> > > >>> > Here's how I produced the error. Note that my directory structure > >>> > matches > >>> > that on the aroma-project.org site, shown here: > >>> > > >>> >> chipType <- "HuEx-1_0-st-v2" > >>> > > >>> > > >>> >> cdf <- AffymetrixCdfFile$byChipType(chipType, > >>> >> tags="coreR3,A20071112,EP") > >>> > > >>> > > >>> >> print(cdf) > >>> > > >>> > AffymetrixCdfFile: > >>> > > >>> > Path: annotationData/chipTypes/HuEx-1_0-st-v2 > >>> > > >>> > Filename: HuEx-1_0-st-v2,coreR3,A20071112,EP.cdf > >>> > > >>> > File size: 38.25 MB (40108891 bytes) > >>> > > >>> > Chip type: HuEx-1_0-st-v2,coreR3,A20071112,EP > >>> > > >>> > RAM: 0.00MB > >>> > > >>> > File format: v4 (binary; XDA) > >>> > > >>> > Dimension: 2560x2560 > >>> > > >>> > Number of cells: 6553600 > >>> > > >>> > Number of units: 18708 > >>> > > >>> > Cells per unit: 350.31 > >>> > > >>> > Number of QC units: 1 > >>> > > >>> > > >>> >> cs <- AffymetrixCelSet$byName("NIK_2014", cdf=cdf) > >>> > > >>> > > >>> >> print(cs) > >>> > > >>> > AffymetrixCelSet: > >>> > > >>> > Name: NIK_2014 > >>> > > >>> > Tags: > >>> > > >>> > Path: rawData/NIK_2014/HuEx-1_0-st-v2 > >>> > > >>> > Platform: Affymetrix > >>> > > >>> > Chip type: HuEx-1_0-st-v2,coreR3,A20071112,EP > >>> > > >>> > Number of arrays: 9 > >>> > > >>> > Names: GSM709330_HSB194-S1C-L, GSM709331_HSB194-S1C-R, > >>> > GSM709332_HSB194-STC-L, ..., GSM709338_HSB194-VFC-R [9] > >>> > > >>> > Time period: 2011-01-25 12:15:03 -- 2011-01-25 13:48:02 > >>> > > >>> > Total file size: 564.87MB > >>> > > >>> > RAM: 0.02MB > >>> > > >>> > > >>> >> csBC <- process(bc,verbose=verbose) > >>> > > >>> > Background correcting data set... > >>> > > >>> > Number of arrays: 9 > >>> > > >>> > Array #1 ('GSM709330_HSB194-S1C-L') of 9... > >>> > > >>> > Adjusting PM signals only > >>> > > >>> > Obtaining signals... > >>> > > >>> > Obtaining signals...done > >>> > > >>> > Applying normal+exponential signal model... > >>> > > >>> > Applying normal+exponential signal model...done > >>> > > >>> > Writing adjusted probe signals... > >>> > > >>> > Adding temporary suffix from file... > >>> > > >>> > Pathname: > >>> > > probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL > >>> > > >>> > Suffix: .tmp > >>> > > >>> > Rename existing file?: FALSE > >>> > > >>> > Temporary pathname: > >>> > > >>> > > probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp > >>> > > >>> > Adding temporary suffix from file...done > >>> > > >>> > Creating CEL file for results, if missing... > >>> > > >>> > [2014-09-04 12:38:54] Exception: No permission to modify existing > file: > >>> > > >>> > > probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp.tmp > > > >>> > > >>> > > >>> > at #13. getWritablePathname.Arguments(static, ...) > >>> > > >>> > - getWritablePathname.Arguments() is in environment > 'R.utils' > >>> > > >>> > > >>> > at #12. getWritablePathname(static, ...) > >>> > > >>> > - getWritablePathname() is in environment 'R.utils' > >>> > > >>> > - originating from '<text>' > >>> > > >>> > > >>> > at #11. Arguments$getWritablePathname(pathname, mustExist = TRUE) > >>> > > >>> > - Arguments$getWritablePathname() is local of the calling > >>> > function > >>> > > >>> > > >>> > at #10. renameFile.default(srcPathname, pathname, ...) > >>> > > >>> > - renameFile.default() is in environment 'R.utils' > >>> > > >>> > > >>> > at #09. renameFile(srcPathname, pathname, ...) > >>> > > >>> > - renameFile() is in environment 'R.utils' > >>> > > >>> > > >>> > at #08. renameTo.GenericDataFile(res, filename = pathname, verbose > = > >>> > less(verbose)) > >>> > > >>> > - renameTo.GenericDataFile() is in environment > 'R.filesets' > >>> > > >>> > > >>> > at #07. renameTo(res, filename = pathname, verbose = > less(verbose)) > >>> > > >>> > - renameTo() is in environment 'R.filesets' > >>> > > >>> > > >>> > at #06. createFrom.AffymetrixCelFile(this, filename = pathnameT, > path > >>> > = > >>> > NULL, > >>> > > >>> > verbose = less(verbose)) > >>> > > >>> > - createFrom.AffymetrixCelFile() is in environment > >>> > 'aroma.affymetrix' > >>> > > >>> > > >>> > at #05. createFrom(this, filename = pathnameT, path = NULL, > verbose = > >>> > less(verbose)) > >>> > > >>> > - createFrom() is in environment 'aroma.affymetrix' > >>> > > >>> > > >>> > at #04. bgAdjustRma.AffymetrixCelFile(df, path = outputPath, > pmonly = > >>> > pmonly, > >>> > > >>> > addJitter = addJitter, jitterSd = jitterSd, overwrite > = > >>> > force, > >>> > > >>> > verbose = verbose, .deprecated = FALSE) > >>> > > >>> > - bgAdjustRma.AffymetrixCelFile() is in environment > >>> > 'aroma.affymetrix' > >>> > > >>> > > >>> > at #03. bgAdjustRma(df, path = outputPath, pmonly = pmonly, > addJitter > >>> > = > >>> > addJitter, > >>> > > >>> > jitterSd = jitterSd, overwrite = force, verbose = > verbose, > >>> > > >>> > .deprecated = FALSE) > >>> > > >>> > - bgAdjustRma() is in environment 'aroma.affymetrix' > >>> > > >>> > > >>> > at #02. process.RmaBackgroundCorrection(bc, verbose = verbose) > >>> > > >>> > - process.RmaBackgroundCorrection() is in environment > >>> > 'aroma.affymetrix' > >>> > > >>> > > >>> > at #01. process(bc, verbose = verbose) > >>> > > >>> > - process() is in environment 'aroma.core' > >>> > > >>> > > >>> > Error: No permission to modify existing file: > >>> > > >>> > > probeData/NIK_2014,RBC,coreR3/HuEx-1_0-st-v2/GSM709330_HSB194-S1C-L.CEL.tmp.tmp > > > >>> > > >>> > Creating CEL file for results, if missing...done > >>> > > >>> > Writing adjusted probe signals...done > >>> > > >>> > Array #1 ('GSM709330_HSB194-S1C-L') of 9...done > >>> > > >>> > Background correcting data set...done > >>> > > >>> > -------------------- > >>> > > >>> > > >>> > I tried to find a solution from the group archives and other places > to > >>> > no > >>> > avail. It's worth mentioning that I get this problem even if I open > up > >>> > permissions to the entire directory. The *CEL.tmp.tmp file appears > to be > >>> > written with write-protections, which is baffling to me. The machine > is > >>> > a > >>> > large, high-memory cluster from my institution, and the error may be > >>> > associated with this. > >>> > > >>> > > >>> > Thanks in advance for any insight you can provide with this. > >>> > > >>> > > >>> > Best regards, > >>> > > >>> > > >>> > Taylor > >>> > > >>> > > >>> > -- > >>> > -- > >>> > When reporting problems on aroma.affymetrix, make sure 1) to run the > >>> > latest > >>> > version of the package, 2) to report the output of sessionInfo() and > >>> > traceback(), and 3) to post a complete code example. > >>> > > >>> > > >>> > You received this message because you are subscribed to the Google > >>> > Groups > >>> > "aroma.affymetrix" group with website http://www.aroma-project.org/. > > >>> > To post to this group, send email to aroma-af...@googlegroups.com > >>> > To unsubscribe and other options, go to > >>> > http://www.aroma-project.org/forum/ > >>> > > >>> > --- > >>> > You received this message because you are subscribed to the Google > >>> > Groups > >>> > "aroma.affymetrix" group. > >>> > To unsubscribe from this group and stop receiving emails from it, > send > >>> > an > >>> > email to aroma-affymetr...@googlegroups.com. > >>> > For more options, visit https://groups.google.com/d/optout. > >> > >> -- > >> -- > >> When reporting problems on aroma.affymetrix, make sure 1) to run the > latest > >> version of the package, 2) to report the output of sessionInfo() and > >> traceback(), and 3) to post a complete code example. > >> > >> > >> You received this message because you are subscribed to the Google > Groups > >> "aroma.affymetrix" group with website http://www.aroma-project.org/. > >> To post to this group, send email to aroma-af...@googlegroups.com > <javascript:> > >> To unsubscribe and other options, go to > http://www.aroma-project.org/forum/ > >> > >> --- > >> You received this message because you are subscribed to the Google > Groups > >> "aroma.affymetrix" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to aroma-affymetr...@googlegroups.com <javascript:>. > >> For more options, visit https://groups.google.com/d/optout. > -- -- When reporting problems on aroma.affymetrix, make sure 1) to run the latest version of the package, 2) to report the output of sessionInfo() and traceback(), and 3) to post a complete code example. You received this message because you are subscribed to the Google Groups "aroma.affymetrix" group with website http://www.aroma-project.org/. To post to this group, send email to aroma-affymetrix@googlegroups.com To unsubscribe and other options, go to http://www.aroma-project.org/forum/ --- You received this message because you are subscribed to the Google Groups "aroma.affymetrix" group. To unsubscribe from this group and stop receiving emails from it, send an email to aroma-affymetrix+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.