skorper opened a new pull request #151: URL: https://github.com/apache/incubator-sdap-nexus/pull/151
[SDAP-362](https://issues.apache.org/jira/browse/SDAP-362) ### Changes made Updated tiles to new format. - Updated gridded tiles so they are reformatted to be in a swath format - Updated swath tiles so they are no longer transformed into gridded format - Updated matchup to work with new formats The following has NOT been done: - Update time series tile - Update unit tests - Once the benchmarks are reviewed, if we decide to go with this method I will update the tests. ### Testing This has been tested on the following 3 requests: L2 --> Insitu ``` {{big_data_url}}/match_spark?primary=ASCATB-L2-Coastal&startTime=2017-02-28T12:03:16Z&endTime=2017-02-28T12:03:16Z&tt=86400&rt=100000&b=100,0,150,20&platforms=1,2,3,4,5,6,7,8,9&depthMin=0&depthMax=5&matchOnce=true&secondary=icoads ``` L4 --> Insitu ``` {{big_data_url}}/match_spark?primary=MUR25-JPL-L4-GLOB-v04.2&startTime=2018-09-24T00:00:00Z&endTime=2018-09-30T00:00:00Z&tt=2592000&rt=1000&b=160,-30,180,-25&platforms=1,2,3,4,5,6,7,8,9¶meter=sst&depthMin=0&depthMax=5&matchOnce=true&secondary=icoads&resultSizeLimit=6000 ``` L4 --> L4 ``` {{big_data_url}}/match_spark?primary=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=2018-10-30T00:00:00Z&endTime=2018-11-05T00:00:00Z&tt=2592000&rt=1000&b=-180,-90,-160,-70&platforms=1,2,3,4,5,6,7,8,9¶meter=sst&depthMin=0&depthMax=5&matchOnce=true&secondary=MUR25-JPL-L4-GLOB-v04.2&resultSizeLimit=6000 ``` Below are the before/after timing metrics for the overall matchup requests: | Request | Seconds Before | Seconds After | | ------------- | -------------- | ------------- | | L2 --> Insitu | 4.63 | 5.43 | | L4 --> Insitu | 7.65 | 5.86 | | L2 --> L4 | 24.64 | 24.77 | <details> <summary>Expand this section for more detailed timing metrics</summary> #### L2 --> Insitu ``` {{big_data_url}}/match_spark?primary=ASCATB-L2-Coastal&startTime=2017-02-28T12:03:16Z&endTime=2017-02-28T12:03:16Z&tt=86400&rt=100000&b=100,0,150,20&platforms=1,2,3,4,5,6,7,8,9&depthMin=0&depthMax=5&matchOnce=true&secondary=icoads ``` ##### Before 4.63s | Step Name | Average(s) | Total(s) | Number of executions | | | ------------------------------------------ | ---------- | -------- | -------------------- | --- | | Time to determine spatial-temporal extents | 0.417002 | 2.502009 | 6 | | | Time to call edge for partition | 0.287948 | 1.727686 | 6 | | | Time to convert match points | 0.006623 | 0.026492 | 4 | | | Time to build matchup tree | 0.000349 | 0.001397 | 4 | | | Time to load tile | 0.309977 | 1.239907 | 4 | | | Time to build primary tree | 0.000162 | 0.000648 | 4 | | | Time to convert primary points for tile | 0.073386 | 0.293543 | 4 | | | Time to query primary tree for tile | 0.000245 | 0.00098 | 4 | | ##### After 5.43 | Step Name | Average(s) | Total(s) | Number of executions | | ------------------------------------------ | ---------- | -------- | -------------------- | | Time to determine spatial-temporal extents | 0.331416 | 1.988495 | 6 | | Time to call edge for partition | 0.268049 | 1.608293 | 6 | | Time to convert match points | 0.006183 | 0.024732 | 4 | | Time to build matchup tree | 0.000321 | 0.001285 | 4 | | Time to load tile | 0.022641 | 0.090565 | 4 | | Time to build primary tree | 0.000142 | 0.000567 | 4 | | Time to convert primary points for tile | 0.00418 | 0.016718 | 4 | | Time to query primary tree for tile | 0.000293 | 0.001173 | 4 | #### L4 --> Insitu ``` {{big_data_url}}/match_spark?primary=MUR25-JPL-L4-GLOB-v04.2&startTime=2018-09-24T00:00:00Z&endTime=2018-09-30T00:00:00Z&tt=2592000&rt=1000&b=160,-30,180,-25&platforms=1,2,3,4,5,6,7,8,9¶meter=sst&depthMin=0&depthMax=5&matchOnce=true&secondary=icoads&resultSizeLimit=6000 ``` ##### Before 7.65s | Step Name | Average(s) | Total(s) | Number of executions | | | ------------------------------------------ | ---------- | --------- | -------------------- | --- | | Time to determine spatial-temporal extents | 0.39519 | 3.161517 | 8 | | | Time to call edge for partition | 1.482238 | 11.857904 | 8 | | | Time to convert match points | 0.089865 | 0.718919 | 8 | | | Time to build matchup tree | 0.000805 | 0.006441 | 8 | | | Time to load tile | 0.0191 | 0.343806 | 18 | | | Time to build primary tree | 0.000316 | 0.00568 | 18 | | | Time to convert primary points for tile | 0.009 | 0.161991 | 18 | | | Time to query primary tree for tile | 0.00057 | 0.010254 | 18 | | ##### After 5.86s | Step Name | Average(s) | Total(s) | Number of executions | | ------------------------------------------ | ---------- | --------- | -------------------- | | Time to determine spatial-temporal extents | 0.355954 | 2.847633 | 8 | | Time to call edge for partition | 1.400643 | 11.205143 | 8 | | Time to convert match points | 0.08807 | 0.704559 | 8 | | Time to build matchup tree | 0.000791 | 0.006331 | 8 | | Time to load tile | 0.017569 | 0.316245 | 18 | | Time to build primary tree | 0.000262 | 0.004724 | 18 | | Time to convert primary points for tile | 0.009964 | 0.179348 | 18 | | Time to query primary tree for tile | 0.000493 | 0.008868 | 18 | #### L4 --> L4 ``` {{big_data_url}}/match_spark?primary=avhrr-l4-glob-v2-daily-ncei-ghrsst-sstblend-avhrr-oi-glob-v020-fv020&startTime=2018-10-30T00:00:00Z&endTime=2018-11-05T00:00:00Z&tt=2592000&rt=1000&b=-180,-90,-160,-70&platforms=1,2,3,4,5,6,7,8,9¶meter=sst&depthMin=0&depthMax=5&matchOnce=true&secondary=MUR25-JPL-L4-GLOB-v04.2&resultSizeLimit=6000 ``` ##### Before 24.64s | Step Name | Average(s) | Total(s) | Number of executions | | | ------------------------------------------ | ---------- | --------- | -------------------- | --- | | Time to determine spatial-temporal extents | 0.391577 | 3.132613 | 8 | | | Time to convert match points | 4.44661 | 35.572884 | 8 | | | Time to build matchup tree | 0.019318 | 0.154546 | 8 | | | Time to load tile | 0.018537 | 0.222443 | 12 | | | Time to build primary tree | 0.000235 | 0.002817 | 12 | | | Time to convert primary points for tile | 0.006429 | 0.077144 | 12 | | | Time to query primary tree for tile | 0.010163 | 0.121958 | 12 | | ##### After 24.77s | Step Name | Average(s) | Total(s) | Number of executions | | | ------------------------------------------ | ---------- | --------- | -------------------- | --- | | Time to determine spatial-temporal extents | 0.338613 | 2.708901 | 8 | | | Time to convert match points | 4.963046 | 39.704364 | 8 | | | Time to build matchup tree | 0.020304 | 0.162434 | 8 | | | Time to load tile | 0.017708 | 0.212499 | 12 | | | Time to build primary tree | 0.000253 | 0.003041 | 12 | | | Time to convert primary points for tile | 0.006224 | 0.07469 | 12 | | | Time to query primary tree for tile | 0.009762 | 0.117138 | 12 | | </details> One last data point is that the L2 30x30 request not works, whereas before with the old Swath/Gridded tile formats the request could not complete due to memory errors. The following request was tested and works as expected (`ascat-l2-coastal-multi-t6` was ingested with size 30x30 on bigdata) ``` {{big_data_url}}/match_spark?primary=ascat-l2-coastal-multi-t6&startTime=2017-01-02T19:55:52Z&endTime=2017-01-02T23:55:52Z&tt=86400&rt=100000&b=0,-45,45,0&platforms=1,2,3,4,5,6,7,8,9&depthMin=0&depthMax=5&matchOnce=true&secondary=icoads ``` ### Conclusions The changes seem to have increased the overall matchup request time for gridded requests, which was expected. Unexpectedly, the time has decreased for L4 -> Insitu. Also unexpected is a slight performance decrease for L2 -> Insitu, though testing on a larger request (that matches with more tiles) may be more telling. The new format does allow larger tiles to be used for L2 collections, though there are no 'before' timing metrics to compare this with because 30x30 tiles do not work with the old format. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
