skorper opened a new pull request, #158:
URL: https://github.com/apache/incubator-sdap-nexus/pull/158
In this PR, I've integrated the AWS Parquet insitu API to the matchup
algorithm.
Please be aware that the insitu API is very slow as of now, so we may want
to hold off on merging/deploying this until those issues have been resolved.
For example, the following request:
```
{{big_data_url}}/match_spark?primary=MUR25-JPL-L4-GLOB-v04.2&startTime=2018-09-24T00:00:00Z&endTime=2018-09-30T00:00:00Z&tt=2592000&rt=1000&b=160,-30,180,-25&depthMin=0&depthMax=5&matchOnce=true&secondary=ICOADS%20Release%203.0&resultSizeLimit=6000&platforms=42
```
takes over 4 minutes to complete. Granted, this is on my local machine which
is only using a single Cassandra node and a single Spark node. However this is
still significantly slower than previous requests using the legacy insitu API.
In addition to the insitu API integration, this PR also includes a new
`MatchupException`. If raised, the message will be passed directly to users.
I'm hoping this will be the start of better and more meaningful error messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
