These changes have been extensively tested and improved through the Cloud-Based Data Matchup Service project's SDAP deployment and have been working like a charm. Queries that were previously infeasible due to their size & computation time are now able to be handled with ease. I will say that storing all matchup results, especially given this PR is focused on particularly large results, we should eventually look for a tool to remove old and failed results to free up space.
+1 (binding) from me On 2023/08/16 00:33:19 Stepheny Perez wrote: > Hi everyone, > > I opened a PR for a major change to SDAP here: > https://github.com/apache/incubator-sdap-nexus/pull/249 > > Because this is a major change, I'll open a 72-hour VOTE thread and see if > there are any concerns about this change. However you vote, please provide > justification. > > This change introduces "async jobs" to SDAP. Currently, SDAP only supports > synchronous jobs, meaning the API call will hang until the analysis is > completed and results are returned to the user. This new async feature will > immediately return a job detail response to the user (via a 300 redirect) > which the user can then poll until the results are ready. This is important > because it adds support for larger jobs; the jobs can take days or weeks if > needed. Please be aware this change is only enabled for the /match_spark > endpoint -- no other algorithms are impacted. In order to enable this feature > for other algorithms, the results would need to be persisted to Cassandra and > the "NexusCalcSparkTornadoHandler" handler would need to be inherited. > > The new endpoints utilize the OGC Coverages specification > (https://ogcapi.ogc.org/coverages/) > > Thanks, > Stepheny >
