[
https://issues.apache.org/jira/browse/SDAP-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas G Loubrieu updated SDAP-281:
-----------------------------------
Description:
While doing the ingestion on bigdata wth configuration file
https://github.com/tloubrieu-jpl/sdap-ingester-config/blob/bigdata/collections.yml
None of the dataset have been ingested so far except mur25 (for 3-4 days). No
tiles are availble in solr for the other datasets.
Some granules have been ingested thought as shown in the solr nexusgranules
collection:
"avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",13752,
"smap-l2b-sss-daily-smap-sss",7032,
"mur25-jpl-l4-glob-v41-analysed-sst",6062,
"modis-l3-aqua-11um-v2014-0-4km-daily-sst",6036,
"measures-sla-jpl-1603-regrid1x1-ssh",1805,
"avhrr-l4-glob-v2-clim-seasonal-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",365]
Here are some information from the logs and netcdf files regarding the possible
issues:
Granule ingester log:
Traceback (most recent call last):
File
"/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/consumer/Consumer.py",
line 79, in _received_message
await pipeline.run()
File
"/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/pipeline/Pipeline.py",
line 187, in run
self._slicer.generate_tiles(dataset, granule_name)]
File
"/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/TileSlicer.py",
line 50, in generate_tiles
self._tile_spec_list = self._generate_slices(dimensions)
File
"/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/SliceFileByStepSize.py",
line 36, in _generate_slices
raise KeyError('Provided dimension "{}" not found in
dataset'.format(dim_name))
KeyError: 'Provided dimension "time" not found in dataset'
Dataset measures-sla-jpl-1603-regrid1x1-ssh has dimension Time with Uppercase T:
float Time(Time) ;
Time:units = "Days since 1985-01-01 00:00:00" ;
Time:long_name = "Time" ;
Time:standard_name = "time" ;
Time:calendar = "gregorian" ;
Time:axis = "T" ;
dataset smap-l2b-sss-daily-smap-sss does not have a time dimension at all.
dataset modis-l3-aqua-11um-v2014-0-4km-daily-sst no time dimension
dataset avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020
has a time dimension.
dataset
avhrr-l4-glob-v2-clim-seasonal-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020
does not have a time dimension.
was:
While doing the ingestion on bigdata wth configuration file
https://github.com/tloubrieu-jpl/sdap-ingester-config/blob/bigdata/collections.yml
None of the dataset have been ingested so farexcept mur25 (for 3-4 days).
Here are some information from the logs and netcdf files regarding the possible
issues:
Granule ingester log:
Traceback (most recent call last):
File
"/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/consumer/Consumer.py",
line 79, in _received_message
await pipeline.run()
File
"/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/pipeline/Pipeline.py",
line 187, in run
self._slicer.generate_tiles(dataset, granule_name)]
File
"/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/TileSlicer.py",
line 50, in generate_tiles
self._tile_spec_list = self._generate_slices(dimensions)
File
"/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/SliceFileByStepSize.py",
line 36, in _generate_slices
raise KeyError('Provided dimension "{}" not found in
dataset'.format(dim_name))
KeyError: 'Provided dimension "time" not found in dataset'
Dataset measures-sla-jpl-1603-regrid1x1-ssh has dimension Time with Uppercase T:
float Time(Time) ;
Time:units = "Days since 1985-01-01 00:00:00" ;
Time:long_name = "Time" ;
Time:standard_name = "time" ;
Time:calendar = "gregorian" ;
Time:axis = "T" ;
dataset smap-l2b-sss-daily-smap-sss does not have a time dimension at all.
dataset modis-l3-aqua-11um-v2014-0-4km-daily-sst no time dimension
dataset avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020
has a time dimension.
dataset
avhrr-l4-glob-v2-clim-seasonal-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020
does not have a time dimension.
> bigdata ingestion feedback
> --------------------------
>
> Key: SDAP-281
> URL: https://issues.apache.org/jira/browse/SDAP-281
> Project: Apache Science Data Analytics Platform
> Issue Type: Bug
> Reporter: Thomas G Loubrieu
> Priority: Major
>
> While doing the ingestion on bigdata wth configuration file
> https://github.com/tloubrieu-jpl/sdap-ingester-config/blob/bigdata/collections.yml
> None of the dataset have been ingested so far except mur25 (for 3-4 days).
> No tiles are availble in solr for the other datasets.
> Some granules have been ingested thought as shown in the solr nexusgranules
> collection:
> "avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",13752,
> "smap-l2b-sss-daily-smap-sss",7032,
> "mur25-jpl-l4-glob-v41-analysed-sst",6062,
> "modis-l3-aqua-11um-v2014-0-4km-daily-sst",6036,
> "measures-sla-jpl-1603-regrid1x1-ssh",1805,
>
> "avhrr-l4-glob-v2-clim-seasonal-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020",365]
> Here are some information from the logs and netcdf files regarding the
> possible issues:
> Granule ingester log:
> Traceback (most recent call last):
> File
> "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/consumer/Consumer.py",
> line 79, in _received_message
> await pipeline.run()
> File
> "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/pipeline/Pipeline.py",
> line 187, in run
> self._slicer.generate_tiles(dataset, granule_name)]
> File
> "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/TileSlicer.py",
> line 50, in generate_tiles
> self._tile_spec_list = self._generate_slices(dimensions)
> File
> "/opt/conda/lib/python3.7/site-packages/sdap_granule_ingester-1.0.0_SNAPSHOT-py3.7.egg/granule_ingester/slicers/SliceFileByStepSize.py",
> line 36, in _generate_slices
> raise KeyError('Provided dimension "{}" not found in
> dataset'.format(dim_name))
> KeyError: 'Provided dimension "time" not found in dataset'
> Dataset measures-sla-jpl-1603-regrid1x1-ssh has dimension Time with Uppercase
> T:
> float Time(Time) ;
> Time:units = "Days since 1985-01-01 00:00:00" ;
> Time:long_name = "Time" ;
> Time:standard_name = "time" ;
> Time:calendar = "gregorian" ;
> Time:axis = "T" ;
> dataset smap-l2b-sss-daily-smap-sss does not have a time dimension at all.
> dataset modis-l3-aqua-11um-v2014-0-4km-daily-sst no time dimension
> dataset avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020
> has a time dimension.
> dataset
> avhrr-l4-glob-v2-clim-seasonal-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020
> does not have a time dimension.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
