[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

2022-09-29 Thread GitBox 


RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984137556


##
docker/jupyter/requirements.txt:
##
@@ -1,4 +1,3 @@
-shapely
-requests
-numpy
-cassandra-driver==3.9.0

Review Comment:
   Not sure. The notebook runs just fine without it and it was causing build 
errors.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

2022-09-29 Thread GitBox 


RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984137949


##
docker/jupyter/requirements.txt:
##
@@ -1,4 +1,3 @@
-shapely
-requests
-numpy
-cassandra-driver==3.9.0

Review Comment:
   Possibly the notebook intended to do more. Or the the file was copy pasted 
from one that did and never checked.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

2022-09-29 Thread GitBox 


RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984137320


##
docs/quickstart.rst:
##
@@ -18,43 +18,59 @@ This quickstart guide will walk you through how to install 
and run NEXUS on your
 Prerequisites
 ==
 
-* Docker (tested on v18.03.1-ce)
+* Docker (tested on v20.10.17)
 * Internet Connection
-* bash
+* bash or zsh
 * cURL
-* 500 MB of disk space
+* 10.5 GB of disk space
 
 Prepare
 
 
-Start downloading the Docker images and data files.
+Start downloading the Docker images and set up the Docker bridge network.
 
 .. _quickstart-step1:
 
 Pull Docker Images
 ---
 
-Pull the necessary Docker images from the `SDAP repository 
`_ on Docker Hub. Please check the repository 
for the latest version tag.
+Pull the necessary Docker images from the `NEXUS JPL repository 
`_ on Docker Hub. Please check the 
repository for the latest version tag.
 
 .. code-block:: bash
 
-  export VERSION=1.0.0-rc1
+  export CASSANDRA_VERSION=3.11.6-debian-10-r138
+  export RMQ_VERSION=3.8.9-debian-10-r37
+  export COLLECTION_MANAGER_VERSION=0.1.6a14
+  export GRANULE_INGESTER_VERSION=0.1.6a30
+  export WEBAPP_VERSION=distributed.0.4.5a49
+  export SOLR_VERSION=8.11.1
+  export SOLR_CLOUD_INIT_VERSION=1.0.2
+  export ZK_VERSION=3.5.5
+
+  export JUPYTER_VERSION=1.2
 
 .. code-block:: bash
 
-  docker pull sdap/ningester:${VERSION}
-  docker pull sdap/solr-singlenode:${VERSION}
-  docker pull sdap/cassandra:${VERSION}
-  docker pull sdap/nexus-webapp:standalone.${VERSION}
+  docker pull bitnami/cassandra:${CASSANDRA_VERSION}
+  docker pull bitnami/rabbitmq:${RMQ_VERSION}
+  docker pull nexusjpl/collection-manager:${COLLECTION_MANAGER_VERSION}
+  docker pull nexusjpl/granule-ingester:${GRANULE_INGESTER_VERSION}
+  docker pull nexusjpl/nexus-webapp:${WEBAPP_VERSION}
+  docker pull nexusjpl/solr:${SOLR_VERSION}
+  docker pull nexusjpl/solr-cloud-init:${SOLR_CLOUD_INIT_VERSION}
+  docker pull zookeeper:${ZK_VERSION}
+
+  # docker pull nexusjpl/jupyter:${JUPYTER_VERSION}

Review Comment:
   Ok. I'll let you know after I test the newer version of the image since I 
can't push to nexusjpl



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

2022-09-29 Thread GitBox 


RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984137144


##
docs/quickstart.rst:
##
@@ -64,181 +80,240 @@ The network we will be using for this quickstart will be 
called ``sdap-net``. Cr
 
 .. _quickstart-step3:
 
-Download Sample Data
--
+Start Ingester Components and Ingest Some Science Data

Review Comment:
   Will do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

2022-09-29 Thread GitBox 


RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984136931


##
docs/quickstart.rst:
##
@@ -64,181 +80,240 @@ The network we will be using for this quickstart will be 
called ``sdap-net``. Cr
 
 .. _quickstart-step3:
 
-Download Sample Data
--
+Start Ingester Components and Ingest Some Science Data
+
 
-The data we will be downloading is part of the `AVHRR OI dataset 
`_ which 
measures sea surface temperature. We will download 1 month of data and ingest 
it into a local Solr and Cassandra instance.
+Create Data Directory
+
+
+Let's start by creating the directory to hold the science data to ingest.
 
 Choose a location that is mountable by Docker (typically needs to be under the 
User's home directory) to download the data files to.
 
 .. code-block:: bash
 
-  export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
-  mkdir -p ${DATA_DIRECTORY}
+export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
+mkdir -p ${DATA_DIRECTORY}
 
-Then go ahead and download 1 month worth of AVHRR netCDF files.
+Now we can start up the data storage components. We will be using Solr and 
Cassandra to store the tile metadata and data respectively.
 
-.. code-block:: bash
+.. _quickstart-step4:
 
-  cd $DATA_DIRECTORY
+Start Zookeeper
+---
 
-  export 
URL_LIST="https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/305/2015110112-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/306/2015110212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/307/2015110312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/308/2015110412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/309/2015110512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/
 2015/310/2015110612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/311/2015110712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/312/2015110812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/313/2015110912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/314/2015111012-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/315/201512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 https://podaac-opendap.jpl.nasa.gov:443
 
/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/316/2015111212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/317/2015111312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/318/2015111412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/319/2015111512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/320/2015111612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/321/2015111712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-G
 LOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/322/2015111812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/323/2015111912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

2022-09-29 Thread GitBox 


RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984136730


##
docs/quickstart.rst:
##
@@ -18,43 +18,59 @@ This quickstart guide will walk you through how to install 
and run NEXUS on your
 Prerequisites
 ==
 
-* Docker (tested on v18.03.1-ce)
+* Docker (tested on v20.10.17)
 * Internet Connection
-* bash
+* bash or zsh
 * cURL
-* 500 MB of disk space
+* 10.5 GB of disk space

Review Comment:
   Lot of docker images that are quite sizable.
   
   + a little margin for granule data  so I just rounded up to a half GB or so



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

2022-09-29 Thread GitBox 


RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984136329


##
CHANGELOG.md:
##
@@ -41,9 +41,9 @@ and this project adheres to [Semantic 
Versioning](https://semver.org/spec/v2.0.0
 - Fixed issue where satellite to satellite matchups with the same dataset 
don't return the expected result
 - Fixed CSV and NetCDF matchup output bug
 - Fixed NetCDF output switching latitude and longitude
+- SDAP-399: Updated quickstart guide for standalone docker deployment of SDAP.
+- SDAP-399: Updated quickstart Jupyter notebook
 - Fixed import error causing `/timeSeriesSpark` queries to fail.
 - Fixed bug where domsresults no longer worked after successful matchup
 - Fixed certificate error in Dockerfile
-### Security

Review Comment:
   I think I removed that line by accident. Oops



##
CHANGELOG.md:
##
@@ -41,9 +41,9 @@ and this project adheres to [Semantic 
Versioning](https://semver.org/spec/v2.0.0
 - Fixed issue where satellite to satellite matchups with the same dataset 
don't return the expected result
 - Fixed CSV and NetCDF matchup output bug
 - Fixed NetCDF output switching latitude and longitude
+- SDAP-399: Updated quickstart guide for standalone docker deployment of SDAP.
+- SDAP-399: Updated quickstart Jupyter notebook
 - Fixed import error causing `/timeSeriesSpark` queries to fail.
 - Fixed bug where domsresults no longer worked after successful matchup
 - Fixed certificate error in Dockerfile
-### Security

Review Comment:
   I think I removed that line by accident. Oops.
   
   I'll put it back



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

2022-09-29 Thread GitBox 


RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984136094


##
docker/jupyter/Dockerfile:
##
@@ -29,7 +29,8 @@ ENV CHOWN_HOME_OPTS='-R'
 ENV REBUILD_CODE=true
 
 ARG APACHE_NEXUS=https://github.com/apache/incubator-sdap-nexus.git
-ARG APACHE_NEXUS_BRANCH=master
+ARG APACHE_NEXUS_COMMIT=be19c1d567301b09269e851cc5b5af55fea02c5d

Review Comment:
   I checked out this commit because I was unaware that the code for nexuscli 
had moved (I thought it was removed) so I grabbed a commit that would've been 
what was used to create the old docker image. I'll retry with the current 
version.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on pull request #202: SDAP-402: Update matchOnce logic

2022-09-29 Thread GitBox 


RKuttruff commented on PR #202:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/202#issuecomment-1262998052

   I actually don't know exactly how many matches L2 -> Saildrone _should_ be 
returning. I based the current test case on how many were returning from the 
AWS deployment. However, the current deployment is returning the 'expected' 
number of matches while the new changes have it returning 2 extra. One thing to 
note about these extra matches is that the primary points are actually 
duplicated in the matchup output. 
   
   An example for one of the extra matches:
   
   
Common point 
   
   ```json
   {
"platform": "orbiting satellite",
"device": "scatterometers",
"lon": "-129.94937",
"lat": "26.56576",
"point": "Point(-129.94937 26.56576)",
"time": 1522695015,
"depth": null,
"fileurl": 
"ascat_20180402_180900_metopb_28742_eps_o_coa_2401_ovw.l2.nc",
"id": "44d09fab-f85e-3de5-9495-835b8607d8e2[[9 9 9]]",
"source": "ASCATB-L2-Coastal",
"primary": [
{
"variable_name": "wind_speed",
"cf_variable_name": "wind_speed",
"variable_value": 2.480190734863,
"variable_unit": null
},
{
"variable_name": "wind_dir",
"cf_variable_name": "wind_to_direction",
"variable_value": 283.338964844,
"variable_unit": null
}
],
"matches": [
{
"platform": "3B",
"device": "130",
"lon": "-130.0027776",
"lat": "26.35588",
"point": "Point(-130.0027776 26.35588)",
"time": 1522691460,
"depth": 0.5,
"fileurl": null,
"id": null,
"source": "shark-2018",
"secondary": [
{
"variable_name": 
"sea_water_practical_salinity",
"cf_variable_name": 
"sea_water_practical_salinity",
"variable_value": 34.681,
"variable_unit": "1"
},
{
"variable_name": 
"sea_water_practical_salinity_quality",
"cf_variable_name": 
"sea_water_practical_salinity_quality",
"variable_value": 2,
"variable_unit": null
},
{
"variable_name": 
"sea_water_temperature",
"cf_variable_name": 
"sea_water_temperature",
"variable_value": 19.314,
"variable_unit": "C"
},
{
"variable_name": 
"sea_water_temperature_quality",
"cf_variable_name": 
"sea_water_temperature_quality",
"variable_value": 2,
"variable_unit": null
}
]
}
]
   }
   
   ```
   
   
   
Extra point (from modified SDAP) 
   
   ```json
   {
"platform": "orbiting satellite",
"device": "scatterometers",
"lon": "-129.94937",
"lat": "26.56576",
"point": "Point(-129.94937 26.56576)",
"time": 1522695015,
"depth": null,
"fileurl": 
"ascat_20180402_180900_metopb_28742_eps_o_coa_2401_ovw.l2.nc",
"id": "44d09fab-f85e-3de5-9495-835b8607d8e2[[9 9 9]]",
"source": "ASCATB-L2-Coastal",
"primary": [
{
"variable_name": "wind_speed",
"cf_variable_name": "wind_speed",
"variable_value": 2.480190734863,
"variable_unit": null
},
{
"variable_name": "wind_dir",
"cf_variable_name": "wind_to_direction",
"variable_value": 283.338964844,
"variable_unit": null
}
],
"matches": [
{
"platform": "3B",
"device": "113",
"lon": "-130.0174208",
"lat": "26.3396",
"point":

[GitHub] [incubator-sdap-nexus] skorper commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

2022-09-29 Thread GitBox 


skorper commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984133710


##
docker/jupyter/requirements.txt:
##
@@ -1,4 +1,3 @@
-shapely
-requests
-numpy
-cassandra-driver==3.9.0

Review Comment:
   I'm curious why we were able to remove cassandra-driver? Or why it was 
needed before but not now?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

2022-09-29 Thread GitBox 


skorper commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984133683


##
docs/quickstart.rst:
##
@@ -18,43 +18,59 @@ This quickstart guide will walk you through how to install 
and run NEXUS on your
 Prerequisites
 ==
 
-* Docker (tested on v18.03.1-ce)
+* Docker (tested on v20.10.17)
 * Internet Connection
-* bash
+* bash or zsh
 * cURL
-* 500 MB of disk space
+* 10.5 GB of disk space
 
 Prepare
 
 
-Start downloading the Docker images and data files.
+Start downloading the Docker images and set up the Docker bridge network.
 
 .. _quickstart-step1:
 
 Pull Docker Images
 ---
 
-Pull the necessary Docker images from the `SDAP repository 
`_ on Docker Hub. Please check the repository 
for the latest version tag.
+Pull the necessary Docker images from the `NEXUS JPL repository 
`_ on Docker Hub. Please check the 
repository for the latest version tag.
 
 .. code-block:: bash
 
-  export VERSION=1.0.0-rc1
+  export CASSANDRA_VERSION=3.11.6-debian-10-r138
+  export RMQ_VERSION=3.8.9-debian-10-r37
+  export COLLECTION_MANAGER_VERSION=0.1.6a14
+  export GRANULE_INGESTER_VERSION=0.1.6a30
+  export WEBAPP_VERSION=distributed.0.4.5a49
+  export SOLR_VERSION=8.11.1
+  export SOLR_CLOUD_INIT_VERSION=1.0.2
+  export ZK_VERSION=3.5.5
+
+  export JUPYTER_VERSION=1.2
 
 .. code-block:: bash
 
-  docker pull sdap/ningester:${VERSION}
-  docker pull sdap/solr-singlenode:${VERSION}
-  docker pull sdap/cassandra:${VERSION}
-  docker pull sdap/nexus-webapp:standalone.${VERSION}
+  docker pull bitnami/cassandra:${CASSANDRA_VERSION}
+  docker pull bitnami/rabbitmq:${RMQ_VERSION}
+  docker pull nexusjpl/collection-manager:${COLLECTION_MANAGER_VERSION}
+  docker pull nexusjpl/granule-ingester:${GRANULE_INGESTER_VERSION}
+  docker pull nexusjpl/nexus-webapp:${WEBAPP_VERSION}
+  docker pull nexusjpl/solr:${SOLR_VERSION}
+  docker pull nexusjpl/solr-cloud-init:${SOLR_CLOUD_INIT_VERSION}
+  docker pull zookeeper:${ZK_VERSION}
+
+  # docker pull nexusjpl/jupyter:${JUPYTER_VERSION}

Review Comment:
   let's update this back to nexusjpl before we merge this



##
CHANGELOG.md:
##
@@ -41,9 +41,9 @@ and this project adheres to [Semantic 
Versioning](https://semver.org/spec/v2.0.0
 - Fixed issue where satellite to satellite matchups with the same dataset 
don't return the expected result
 - Fixed CSV and NetCDF matchup output bug
 - Fixed NetCDF output switching latitude and longitude
+- SDAP-399: Updated quickstart guide for standalone docker deployment of SDAP.
+- SDAP-399: Updated quickstart Jupyter notebook
 - Fixed import error causing `/timeSeriesSpark` queries to fail.
 - Fixed bug where domsresults no longer worked after successful matchup
 - Fixed certificate error in Dockerfile
-### Security

Review Comment:
   you can leave this section in empty in case we make a security change



##
docker/jupyter/Dockerfile:
##
@@ -29,7 +29,8 @@ ENV CHOWN_HOME_OPTS='-R'
 ENV REBUILD_CODE=true
 
 ARG APACHE_NEXUS=https://github.com/apache/incubator-sdap-nexus.git
-ARG APACHE_NEXUS_BRANCH=master
+ARG APACHE_NEXUS_COMMIT=be19c1d567301b09269e851cc5b5af55fea02c5d

Review Comment:
   This commit is from 2019... ideally, I think we'd like this to run on the 
latest code? What issues did you run when running this on master?



##
docker/jupyter/requirements.txt:
##
@@ -1,4 +1,3 @@
-shapely
-requests
-numpy
-cassandra-driver==3.9.0

Review Comment:
   I'm curious why we were able to remove this? Or why it was needed before but 
not now?



##
docs/quickstart.rst:
##
@@ -64,181 +80,240 @@ The network we will be using for this quickstart will be 
called ``sdap-net``. Cr
 
 .. _quickstart-step3:
 
-Download Sample Data
--
+Start Ingester Components and Ingest Some Science Data
+
 
-The data we will be downloading is part of the `AVHRR OI dataset 
`_ which 
measures sea surface temperature. We will download 1 month of data and ingest 
it into a local Solr and Cassandra instance.
+Create Data Directory
+
+
+Let's start by creating the directory to hold the science data to ingest.
 
 Choose a location that is mountable by Docker (typically needs to be under the 
User's home directory) to download the data files to.
 
 .. code-block:: bash
 
-  export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
-  mkdir -p ${DATA_DIRECTORY}
+export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
+mkdir -p ${DATA_DIRECTORY}
 
-Then go ahead and download 1 month worth of AVHRR netCDF files.
+Now we can start up the data storage components. We will be using Solr and 
Cassandra to store the tile metadata and data respectively.
 
-.. code-block:: bash
+..

[GitHub] [incubator-sdap-nexus] skorper commented on pull request #202: SDAP-402: Update matchOnce logic

2022-09-29 Thread GitBox 


skorper commented on PR #202:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/202#issuecomment-1262983746

   Note, L2 -> Saildrone is returning more matches than expected for both my 
changes and the version deployed to AWS (Riley correct me if I'm wrong) so this 
discrepancy is maybe unrelated to this change?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (SDAP-404) Experiment with using MUR data in Zarr for match up algorithm

2022-09-29 Thread Nga Thien Chung (Jira) 
Nga Thien Chung created SDAP-404:


 Summary: Experiment with using MUR data in Zarr for match up 
algorithm
 Key: SDAP-404
 URL: https://issues.apache.org/jira/browse/SDAP-404
 Project: Apache Science Data Analytics Platform
  Issue Type: New Feature
Reporter: Nga Thien Chung






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (SDAP-390) Update NetCDF reader tool for data match-up

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung updated SDAP-390:
-
Resolution: Fixed
Status: Done  (was: To Do)

> Update NetCDF reader tool for data match-up
> ---
>
> Key: SDAP-390
> URL: https://issues.apache.org/jira/browse/SDAP-390
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>Reporter: Nga Thien Chung
>Priority: Major
>
> NetCDF reader tool developed for DOMS needs to be updated to be compatible 
> with CDMS enhancements



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (SDAP-371) Fix domssubset API

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung updated SDAP-371:
-
Resolution: Fixed
Status: Done  (was: To Do)

> Fix domssubset API
> --
>
> Key: SDAP-371
> URL: https://issues.apache.org/jira/browse/SDAP-371
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>  Components: analysis
>Reporter: Stepheny Perez
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (SDAP-398) Create CDMS API tests

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung reassigned SDAP-398:


Assignee: (was: Nga Thien Chung)

> Create CDMS API tests
> -
>
> Key: SDAP-398
> URL: https://issues.apache.org/jira/browse/SDAP-398
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>Reporter: Nga Thien Chung
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (SDAP-393) Add optional insitu API endpoint to SDAP ingress

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung updated SDAP-393:
-
Resolution: Fixed
Status: Done  (was: To Do)

https://github.com/apache/incubator-sdap-nexus/pull/167

> Add optional insitu API endpoint to SDAP ingress
> 
>
> Key: SDAP-393
> URL: https://issues.apache.org/jira/browse/SDAP-393
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>Reporter: Nga Thien Chung
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (SDAP-398) Create CDMS API tests

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung updated SDAP-398:
-
Assignee: Nga Thien Chung
  Status: In Progress  (was: To Do)

> Create CDMS API tests
> -
>
> Key: SDAP-398
> URL: https://issues.apache.org/jira/browse/SDAP-398
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>Reporter: Nga Thien Chung
>Assignee: Nga Thien Chung
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (SDAP-403) Catch ReadTimeout exception when adding remote collections

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung updated SDAP-403:
-
Assignee: Nga Thien Chung
  Status: In Progress  (was: To Do)

> Catch ReadTimeout exception when adding remote collections
> --
>
> Key: SDAP-403
> URL: https://issues.apache.org/jira/browse/SDAP-403
> Project: Apache Science Data Analytics Platform
>  Issue Type: Bug
>Reporter: Kevin Marlis
>Assignee: Nga Thien Chung
>Priority: Major
>
> The redirect functionality can break the local SDAP /nexus/list if the remote 
> /nexus/list causes a `requests.exceptions.ReadTimeout` exception. We need to 
> catch that exception.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (SDAP-403) Catch ReadTimeout exception when adding remote collections

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung reassigned SDAP-403:


Assignee: (was: Nga Thien Chung)

> Catch ReadTimeout exception when adding remote collections
> --
>
> Key: SDAP-403
> URL: https://issues.apache.org/jira/browse/SDAP-403
> Project: Apache Science Data Analytics Platform
>  Issue Type: Bug
>Reporter: Kevin Marlis
>Priority: Major
>
> The redirect functionality can break the local SDAP /nexus/list if the remote 
> /nexus/list causes a `requests.exceptions.ReadTimeout` exception. We need to 
> catch that exception.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (SDAP-399) Update SDAP quickstart guide to include instructions for both docker and k8s installation

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung updated SDAP-399:
-
Assignee: Nga Thien Chung
  Status: In Progress  (was: To Do)

> Update SDAP quickstart guide to include instructions for both docker and k8s 
> installation
> -
>
> Key: SDAP-399
> URL: https://issues.apache.org/jira/browse/SDAP-399
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>Reporter: Nga Thien Chung
>Assignee: Nga Thien Chung
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (SDAP-399) Update SDAP quickstart guide to include instructions for both docker and k8s installation

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung reassigned SDAP-399:


Assignee: (was: Nga Thien Chung)

> Update SDAP quickstart guide to include instructions for both docker and k8s 
> installation
> -
>
> Key: SDAP-399
> URL: https://issues.apache.org/jira/browse/SDAP-399
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>Reporter: Nga Thien Chung
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (SDAP-394) In situ subset API needs to return meta column as one of the default columns

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung updated SDAP-394:
-
Resolution: Fixed
Status: Done  (was: To Do)

> In situ subset API needs to return meta column as one of the default columns
> 
>
> Key: SDAP-394
> URL: https://issues.apache.org/jira/browse/SDAP-394
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>  Components: in-situ
>Reporter: Nga Thien Chung
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (SDAP-400) Update matchup endpoint to use NCAR API

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung updated SDAP-400:
-
Resolution: Fixed
Status: Done  (was: To Do)

> Update matchup endpoint to use NCAR API
> ---
>
> Key: SDAP-400
> URL: https://issues.apache.org/jira/browse/SDAP-400
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>Reporter: Nga Thien Chung
>Assignee: Nga Thien Chung
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (SDAP-395) Add in situ JSON schema endpoint to in situ data services

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung updated SDAP-395:
-
Resolution: Fixed
Status: Done  (was: To Do)

> Add in situ JSON schema endpoint to in situ data services
> -
>
> Key: SDAP-395
> URL: https://issues.apache.org/jira/browse/SDAP-395
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>  Components: in-situ
>Reporter: Nga Thien Chung
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (SDAP-387) Update NetCDF matchup output format

2022-09-29 Thread Nga Thien Chung (Jira) 


 [ 
https://issues.apache.org/jira/browse/SDAP-387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nga Thien Chung updated SDAP-387:
-
Resolution: Fixed
Status: Done  (was: To Do)

> Update NetCDF matchup output format
> ---
>
> Key: SDAP-387
> URL: https://issues.apache.org/jira/browse/SDAP-387
> Project: Apache Science Data Analytics Platform
>  Issue Type: Improvement
>  Components: analysis
>Reporter: Stepheny Perez
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [incubator-sdap-nexus] kevinmarlis opened a new pull request, #203: SDAP-403

2022-09-29 Thread GitBox 


kevinmarlis opened a new pull request, #203:
URL: https://github.com/apache/incubator-sdap-nexus/pull/203

   
   On the AQ deployment we found a redirect bug that caused the AQ 
`/nexus/list` to crash when attempting to connect to a remote SDAP. The 
redirect code was catching `requests.exceptions.ConnectTimeout` exceptions but 
`requests.exceptions.ReadTimeout` exceptions could also be raised when the 
remote `/nexus/list` endpoint was unreachable.  This PR addresses that by 
including `requests.exceptions.ReadTimeout` in the except block. 
   
   This PR also includes a fix to the HofMoeller algorithms. There was a change 
made to the name of the nexus point data_val variable (from point.data_val to 
point.data_vals) in nexus model.py's nexus_point_generator. The change in 
variable name was not updated in the HofMoeller algorithms, so this change is 
included in the PR.
   
   The fix to the HofMoeller algorithms has been tested in SLCP's sit 
environment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (SDAP-403) Catch ReadTimeout exception when adding remote collections

2022-09-29 Thread Kevin Marlis (Jira) 
Kevin Marlis created SDAP-403:
-

 Summary: Catch ReadTimeout exception when adding remote collections
 Key: SDAP-403
 URL: https://issues.apache.org/jira/browse/SDAP-403
 Project: Apache Science Data Analytics Platform
  Issue Type: Bug
Reporter: Kevin Marlis


The redirect functionality can break the local SDAP /nexus/list if the remote 
/nexus/list causes a `requests.exceptions.ReadTimeout` exception. We need to 
catch that exception.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)