Thank you, Stepheny! The Jupyter notebook [1] that we've used to illustrate data match up was updated to be compatible with async job and pagination and works as expected.
+1 (binding) from me. Best, Nga [1] https://github.com/access-cdms/cdms-notebooks/blob/master/CDMS-Match-Up-Demo.ipynb On Wed, Aug 16, 2023 at 7:13 AM Riley Kuttruff <r...@apache.org> wrote: > > These changes have been extensively tested and improved through the > Cloud-Based Data Matchup Service project's SDAP deployment and have been > working like a charm. Queries that were previously infeasible due to their > size & computation time are now able to be handled with ease. > I will say that storing all matchup results, especially given this PR is > focused on particularly large results, we should eventually look for a tool > to remove old and failed results to free up space. > > +1 (binding) from me > > On 2023/08/16 00:33:19 Stepheny Perez wrote: > > Hi everyone, > > > > I opened a PR for a major change to SDAP here: > > https://github.com/apache/incubator-sdap-nexus/pull/249 > > > > Because this is a major change, I'll open a 72-hour VOTE thread and see if > > there are any concerns about this change. However you vote, please provide > > justification. > > > > This change introduces "async jobs" to SDAP. Currently, SDAP only supports > > synchronous jobs, meaning the API call will hang until the analysis is > > completed and results are returned to the user. This new async feature will > > immediately return a job detail response to the user (via a 300 redirect) > > which the user can then poll until the results are ready. This is important > > because it adds support for larger jobs; the jobs can take days or weeks if > > needed. Please be aware this change is only enabled for the /match_spark > > endpoint -- no other algorithms are impacted. In order to enable this > > feature for other algorithms, the results would need to be persisted to > > Cassandra and the "NexusCalcSparkTornadoHandler" handler would need to be > > inherited. > > > > The new endpoints utilize the OGC Coverages specification > > (https://ogcapi.ogc.org/coverages/) > > > > Thanks, > > Stepheny > >
