[GitHub] [incubator-sdap-nexus] skorper opened a new pull request, #208: Add second NCAR provider

[GitBox]

skorper opened a new pull request, #208:
URL: https://github.com/apache/incubator-sdap-nexus/pull/208

   Added a new provider for NCAR so either the NCAR insitu api or JPL insitu 
api can be specified. This wasn't really possible due to the way this was 
written so I had to add an extra field in the insitu config `short_name`. This 
is optional, and can be specified when executing matching and specifying the 
secondary dataset. 
   
   With these changes, the following secondary ICOADS values can be provided:
   
   - `ICOADS%20Release%203.0` Uses the NCAR In-Situ API (this is the old method 
which still works)
   - `ICOADS_NCAR` Uses the NCAR In-Situ API
   - `ICOADS_JPL` Uses the JPL In-Situ API
   
   Once validated, I'd recommend we revert these changes as they won't be 
useful in a practical scenario.
   
   ---
   
   Testing:
   
   ```bash
   
{{big_data_url}}/match_spark?primary=MUR25-JPL-L4-GLOB-v04.2&secondary=ICOADS%20Release%203.0&startTime=2017-07-01T00%3A00%3A00Z&endTime=2017-07-07T23%3A59%3A59Z&b=-100%2C20%2C-79%2C30&platforms=30&depthMin=-10&depthMax=10&tt=43200&rt=5000&matchOnce=false&resultSizeLimit=0
   ```
   
   ```json
       "details": {
           "timeToComplete": 72,
           "numSecondaryMatched": 258,
           "numPrimaryMatched": 101
       }
   ```
   
   ```bash
   
{{big_data_url}}/match_spark?primary=MUR25-JPL-L4-GLOB-v04.2&secondary=ICOADS_JPL&startTime=2017-07-01T00%3A00%3A00Z&endTime=2017-07-07T23%3A59%3A59Z&b=-100%2C20%2C-79%2C30&platforms=30&depthMin=-10&depthMax=10&tt=43200&rt=5000&matchOnce=false&resultSizeLimit=0
   ```
   
   ```json
       "details": {
           "timeToComplete": 83,
           "numSecondaryMatched": 218,
           "numPrimaryMatched": 82
       }
   ```
   
   ```bash
   
{{big_data_url}}/match_spark?primary=MUR25-JPL-L4-GLOB-v04.2&secondary=ICOADS_NCAR&startTime=2017-07-01T00%3A00%3A00Z&endTime=2017-07-07T23%3A59%3A59Z&b=-100%2C20%2C-79%2C30&platforms=30&depthMin=-10&depthMax=10&tt=43200&rt=5000&matchOnce=false&resultSizeLimit=0
   ```
   
   ```json
       "details": {
           "timeToComplete": 77,
           "numSecondaryMatched": 258,
           "numPrimaryMatched": 101
       }
   ```
   
   Also confirmed through logs that the expected API is hit. Note: the NCAR 
matchup returns different results so we should definitely look into that.
   
   Also tested with the `match_spark_doms` to make sure nothing was broken:
   
   ```json
   
{{big_data_url}}/domssubset?dataset=ASCATB-L2-Coastal&insitu=samos&startTime=2017-05-01T00%3A00%3A00Z&endTime=2017-05-14T23%3A59%3A59Z&b=-100%2C10%2C-79%2C-30&parameter=wind&depthMin=-10&depthMax=10&platforms=1%2C2%2C3%2C4%2C5%2C6%2C7%2C8%2C9
   ```
   
   No errors occured. 
   
   Also tested with unrelated `ASCATA-L2` -> shark matchup request, also worked 
as expected. 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org