[Bioc-devel] Documentation on how to update a BioC experimental package?

2014-10-23 Thread Henrik Bengtsson
It's been a while since I worked with experimental packages.  Where
can I find documentation on how to (Subversion) update our
AffymetrixDataTestFiles package with additional data files?  All I
know is that the SVN repository only contains a stub of the package
and http://www.bioconductor.org/developers/package-guidelines/#package-types
provides little information and basically only point to the devel
mailing list.

Thanks,

Henrik

___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel



Re: [Bioc-devel] Documentation on how to update a BioC experimental package?

2014-10-23 Thread Vincent Carey
There is a README.txt in the pkgs folder.

I will attach it.  I think this is accurate, but there may be something
else on the site.



On Thu, Oct 23, 2014 at 10:23 PM, Henrik Bengtsson h...@biostat.ucsf.edu
wrote:

 It's been a while since I worked with experimental packages.  Where
 can I find documentation on how to (Subversion) update our
 AffymetrixDataTestFiles package with additional data files?  All I
 know is that the SVN repository only contains a stub of the package
 and
 http://www.bioconductor.org/developers/package-guidelines/#package-types
 provides little information and basically only point to the devel
 mailing list.

 Thanks,

 Henrik

 ___
 Bioc-devel@r-project.org mailing list
 https://stat.ethz.ch/mailman/listinfo/bioc-devel


 BioC Experiment Data Package SVN Repos


:Date: 2006-08-10
:Author: S. Falcon
:svn URL: https://hedgehog.fhcrc.org/bioc-data/trunk/experiment/pkgs


Background
==

This svn directory contains BioC experiment data packages.  Data
packages contain potentially large binary files that do not change
often.  Most updates to these packages involve the package
infrastructure files.

Obtaining a working copy of a data package over a slow connection can
be frustrating, especially when all that is needed is the
infrastructure files and not the actual data.  We have implemented a
scheme that allows separate checkout of infrastructure files and data
files.  This document describes the scheme and provides instructions
for checkout and update of existing data packages as well as for
adding new packages.


How to Create an Infrastructure-Only Workingcopy


You can obtain a checkout of all experiment data package
infrastructure files as follows::

svn checkout \
  https://hedgehog.fhcrc.org/bioc-data/trunk/experiment/pkgs

To obtain the files for a particular package, say ``ALL``::

svn checkout \
  https://hedgehog.fhcrc.org/bioc-data/trunk/experiment/pkgs/ALL

If you want to preview what is available, you might try the
following::

# get the top-level scripts, but don't recurse into subdirs
svn checkout -N \
  https://hedgehog.fhcrc.org/bioc-data/trunk/experiment/pkgs

# see what is there
svn ls https://hedgehog.fhcrc.org/bioc-data/trunk/experiment/pkgs

# get a particular package's infrastructure files
cd pkgs
svn up ALL
# see next section for getting complete working copy w/ data


How to Create a Complete Workingcopy


First create a workingcopy of the infrastructure files as described
above.

Next use the helper script ``add_data.py`` (you will need Python).  It
is located here:

https://hedgehog.fhcrc.org/bioc-data/trunk/experiment/pkgs/add_data.py

Here's a complete example for ``ALL``::

svn checkout \
  https://hedgehog.fhcrc.org/bioc-data/trunk/experiment/pkgs/ALL

python add_data.py ALL

This will add the big data directories (usually data/, but sometimes
also dirs under inst/) to your working copy.  Usually, the svn:ignore
property has been set so that you won't accidentally add these dirs
when working with the package, but please take care anyway.


A note about committing changes to the data
---

If you want to modify the actual data, cd into the appropriate dir
after having run add_data.py and do your commit from there.  The
script adds a full working copy inside the infrastructure working
copy.


How to Add a New Data Package
=

1. Add the infrastructure files under ``pkgs``.

2. Add any large data directories to ../data_store/PKGNAME/.  For
   example, if there is large data in PKGNAME/data and
   PKGNAME/inst/extdata, you would add PKGNAME/data and
   PKGNAME/inst/extdata to ../data_store.

3. Create a file 'external_data_store.txt' listing each dir that is
   stored externally (each on a separate line).  Contining the example
   above, the file would contain::

data
inst/extdata

   This should go in the top-level of the package dir.

4. Add svn ignore properties.  Continuing the example::

   cd PKGNAME
   svn propset svn:ignore '*' ./data/ ## property 'svn:ignore' set on '.'
  ## in the data folder 
   svn propset svn:ignore '*' ./inst/

   or (this might not work anymore)
   svn propedit svn:ignore .   ## add 'data' here
   svn propedit svn:ignore inst  ## add 'extdata' here

5. Commit.


Details of Storage Scheme
=

Experiment data package infrastructure files live in
``experiment/pkgs``.  Package subdirectories that contain large files
are stored under ``experiment/data_store``.  There is no mechanism to
support separate storage of individual files.

Here is an example of how data for the 

Re: [Bioc-devel] Documentation on how to update a BioC experimental package?

2014-10-23 Thread Dan Tenenbaum


- Original Message -
 From: Vincent Carey st...@channing.harvard.edu
 To: Henrik Bengtsson h...@biostat.ucsf.edu
 Cc: bioC-devel bioc-de...@stat.math.ethz.ch
 Sent: Thursday, October 23, 2014 7:45:13 PM
 Subject: Re: [Bioc-devel] Documentation on how to update a BioC experimental  
 package?
 
 There is a README.txt in the pkgs folder.
 
 I will attach it.  I think this is accurate, but there may be
 something
 else on the site.
 

See the Experiment Data Packages section of

http://www.bioconductor.org/developers/how-to/source-control/

Dan


 
 
 On Thu, Oct 23, 2014 at 10:23 PM, Henrik Bengtsson
 h...@biostat.ucsf.edu
 wrote:
 
  It's been a while since I worked with experimental packages.  Where
  can I find documentation on how to (Subversion) update our
  AffymetrixDataTestFiles package with additional data files?  All I
  know is that the SVN repository only contains a stub of the package
  and
  http://www.bioconductor.org/developers/package-guidelines/#package-types
  provides little information and basically only point to the devel
  mailing list.
 
  Thanks,
 
  Henrik
 
  ___
  Bioc-devel@r-project.org mailing list
  https://stat.ethz.ch/mailman/listinfo/bioc-devel
 
 
 ___
 Bioc-devel@r-project.org mailing list
 https://stat.ethz.ch/mailman/listinfo/bioc-devel


___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel


Re: [Bioc-devel] Documentation on how to update a BioC experimental package?

2014-10-23 Thread Hervé Pagès

Hi Henrik,

FWIW there is some additional information here:

  https://stat.ethz.ch/pipermail/bioc-devel/2014-January/005156.html

In particular the section starting at One important difference with
the svn software repository provides some useful hints on how to
commit changes to the data (see step 4.).

H.

On 10/23/2014 07:46 PM, Dan Tenenbaum wrote:



- Original Message -

From: Vincent Carey st...@channing.harvard.edu
To: Henrik Bengtsson h...@biostat.ucsf.edu
Cc: bioC-devel bioc-de...@stat.math.ethz.ch
Sent: Thursday, October 23, 2014 7:45:13 PM
Subject: Re: [Bioc-devel] Documentation on how to update a BioC experimental
package?

There is a README.txt in the pkgs folder.

I will attach it.  I think this is accurate, but there may be
something
else on the site.



See the Experiment Data Packages section of

http://www.bioconductor.org/developers/how-to/source-control/

Dan





On Thu, Oct 23, 2014 at 10:23 PM, Henrik Bengtsson
h...@biostat.ucsf.edu
wrote:


It's been a while since I worked with experimental packages.  Where
can I find documentation on how to (Subversion) update our
AffymetrixDataTestFiles package with additional data files?  All I
know is that the SVN repository only contains a stub of the package
and
http://www.bioconductor.org/developers/package-guidelines/#package-types
provides little information and basically only point to the devel
mailing list.

Thanks,

Henrik

___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel



___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel



___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel



--
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpa...@fhcrc.org
Phone:  (206) 667-5791
Fax:(206) 667-1319

___
Bioc-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/bioc-devel