[
https://issues.apache.org/jira/browse/CLIMATE-825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392853#comment-15392853
]
ASF GitHub Bot commented on CLIMATE-825:
----------------------------------------
GitHub user agoodm opened a pull request:
https://github.com/apache/climate/pull/374
CLIMATE-825 - Coalesce data sources into one module
-Added a new class called `DatasetLoader`
-Added corresponding documentation and unit tests
Read the docstrings for more info on the API, but the gist of it is that it
was designed to easily load the inputs to each data source loader from our
current config files.
This is intended for the 1.2 release.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/agoodm/climate CLIMATE-825
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/climate/pull/374.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #374
----
commit ecea6210b717e9464ce527c883286a38e41a7a08
Author: Alex Goodman <[email protected]>
Date: 2016-07-18T17:11:13Z
Added DatasetLoader class
commit a3764c046adb506740fc7e3abdbabedfea917ad5
Author: Alex Goodman <[email protected]>
Date: 2016-07-18T22:11:41Z
Added additional functionality to DatasetLoader
commit bf5b6eba6251c54317693cc1df2a05e3e8401dfb
Author: Alex Goodman <[email protected]>
Date: 2016-07-19T20:38:14Z
Minor fixes to dataset_loader
commit d937675d0ad1da865d5dab57d895cf77e471f646
Author: Alex Goodman <[email protected]>
Date: 2016-07-19T20:42:25Z
Minor docstring fix
commit 4add58b2e5dd4e6b28509402e35adffbc03692e7
Author: Alex Goodman <[email protected]>
Date: 2016-07-25T22:14:42Z
Preserve data_source info after each load
commit 198de48332c84586dafcaf033fa27a8eb8818ebe
Author: Alex Goodman <[email protected]>
Date: 2016-07-25T23:13:48Z
Fixing some bugs found from testing
commit 98a67d130ad9b2c946f18dba90da6b68c51594de
Author: Alex Goodman <[email protected]>
Date: 2016-07-25T23:14:14Z
Added unit tests
commit b56b690cb21b260d146a6080fff33b9c0c77319e
Author: Alex Goodman <[email protected]>
Date: 2016-07-25T23:21:38Z
Add ASF Licence header to test_dataset_loader.py
commit 541dbe53fcb4d014161d9a64f7b8f890f3360069
Author: Alex Goodman <[email protected]>
Date: 2016-07-25T23:29:28Z
Added dataset_loader.py to Sphinx build
----
> Coalesce data sources into one module
> -------------------------------------
>
> Key: CLIMATE-825
> URL: https://issues.apache.org/jira/browse/CLIMATE-825
> Project: Apache Open Climate Workbench
> Issue Type: Improvement
> Components: data sources
> Affects Versions: 1.0.0
> Reporter: Alex Goodman
> Assignee: Alex Goodman
> Fix For: 1.2.0
>
>
> Kyo and I will be working on overhauling the way data loading is handled in
> the current RCMES workflow. Right now, the user manually specifies the
> sources for each dataset which are currently separated into three categories:
> local files on disk, the RCMES database (RCMED), and the Earth System Grid
> (ESGF). These cases are currently handled in separate modules / function
> calls, but it would be most ideal in the future to create one universal
> function call for all the data loading. An example schematic would be
> something like:
> datasets = load(sources, ...)
> Here datasets would be a list of OCW Dataset objects, sources would be a list
> of source specifications for each requested dataset (eg, 'esgf', 'local', or
> 'rcmed'). Ideally we would also like better support for handling datasets
> spanned by multiple files as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
