Forwarding to dev@sdap ---------- Forwarded message --------- From: Truc N. Nguyen <trucngu...@cpp.edu> Date: Mon, Nov 18, 2024 at 12:43 AM Subject: Re: Ex: Re: CS 3700 Re: SDAP Airflow Questions and Meeting with SDAP Team To: Nga Chung <nch...@apache.org>, Marc A. Cruz <marcc...@cpp.edu>
Good morning SDAP team, Thank you for your quick response! For the issues, here is the trail we blazed thus far while working through the SDAP QuickStart process, as well as a few questions regarding the documentation. Progress and Challenges For myself: RabbitMQ issues: I encountered an error (ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN) when trying to run the two ingester containers on macOS. I resolved this by modifying RABBITMQ_HOST to rmq in the granule-ingester.env file, though I’m unsure if this is the recommended solution. Jupyter Notebook Data Issue: While I successfully got the Jupyter Notebook running, I encountered a problem where the data wasn’t found when running the Time Series Examples, despite verifying that the data had been downloaded correctly. For Marc, my partner: Initially, we attempted to use version 1.3.0 of the Docker images, but documentation references version 1.2.0. Due to issues with older images not being available, we proceeded with version 1.4.0. Encountered Docker Desktop issues on Windows, prompting a switch to macOS and Debian 12. For Debian 12: The granule ingester failed to connect to Cassandra, requiring us to troubleshoot by replacing host.docker.internal to localhost ,or other container-specific names (e.g., solr for History URL in connection manager). Updated the granule-ingester.env file as follows: CASSANDRA_CONTACT_POINTS=172.18.0.3 CASSANDRA_USERNAME=cassandra CASSANDRA_PASSWORD=cassandra Here, the CASSANDRA_CONTACT_POINTS value was manually set to the container’s IP address using docker inspect cassandra. Created a new RabbitMQ user with admin access and set the virtual host to /. After getting granule ingester to not abort due to connection with cassandra and rabbitmq. collection-manager was able to run and then Debian 12 docker desktop crashed. Tried with default settings for resources and also near maximum resources. Both cases of docker desktop configuration crashed. We apologize that we can not provide too many details of the crash as Docker Desktop premium is required to determine exact causes of the crashes. To conclude, with all the mentioned troubleshooting, I was able to complete all the steps then a crash occurs. Questions Regarding Documentation For documentation, there seems to be two sdap-nexus.readthedocs.io & incubator-sdap-nexus readthedocs.io. Can you confirm whether these are the same or serve different purposes? For example, https://sdap-nexus.readthedocs.io/en/1.4.0/quickstart.html export COLLECTION_MANAGER_VERSION=1.4.0 export GRANULE_INGESTER_VERSION=1.4.0 https://incubator-sdap-nexus.readthedocs.io/en/latest/quickstart.html export COLLECTION_MANAGER_VERSION=1.2.0 export GRANULE_INGESTER_VERSION=1.2.0 With the code block of all version tags, would the given work? As we are not familiar with how sensitive each container is to new changes found in other containers. Was Kubernetes a pre-requisite to run docker QuickStart? Thank you for the help and support, we also filled out the doodle for meeting time so we're looking forward to seeing everyone! Thank you for your time. Sincerely, Truc Nguyen California State Polytechnic University, Pomona trucngu...@cpp.edu ________________________________ From: Nga Chung <nch...@apache.org> Sent: Sunday, November 17, 2024 9:11 PM To: Truc N. Nguyen <trucngu...@cpp.edu>; Marc A. Cruz <marcc...@cpp.edu> Subject: Ex: Re: CS 3700 Re: SDAP Airflow Questions and Meeting with SDAP Team [You don't often get email from nch...@apache.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Third time's the charm... Sorry by default when we reply to dev@ mailing list it only includes dev@. Looking forward to speaking to you both. Best, Nga On Sun, Nov 17, 2024 at 9:09 PM Nga Chung <nch...@apache.org> wrote: > > Sorry, forget to reply all! Truc, please see my message below. > > Thanks, > Nga > > > > On Sun, Nov 17, 2024 at 9:07 PM Nga Chung <nch...@apache.org> wrote: > > > > Hi Truc, > > > > We'd be happy to meet with you. As Riley mentioned, it would be great > > if you can please provide a brief description of the challenges, so we > > can come to the meeting prepared. > > > > I've also created a doodle poll, so for those who would like to join > > us, please fill out the poll and we will find a time that works for > > everyone. > > > > Link to poll: > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoodle.com%2Fmeeting%2Fparticipate%2Fid%2FaQvQZAlb&data=05%7C02%7Ctrucnguyen%40cpp.edu%7Cf93cdbddbf854256e15608dd078f846b%7C164ba61e39ec4f5d89ffaa1f00a521b4%7C0%7C0%7C638675035179090142%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JD0kfSP6L3poS3Vl1FbfJC4d0SisdY9lKjp60sWBL8U%3D&reserved=0 > > > > Thanks, > > Nga > > > > On Fri, Nov 15, 2024 at 1:06 PM Riley Kuttruff <r...@apache.org> wrote: > > > > > > Hi! > > > > > > I'd be happy to help in any way I can. Could you perhaps provide a brief > > > description of the technical challenges you've been facing? Just so I can > > > have a little time to prepare to make the most of our time. > > > > > > I'm happy to meet next week though my calendar is a bit full so we may > > > need > > > to iterate a little to find a suitable time. What times work for you? > > > > > > Also, I think @nchung may also wish to attend. > > > > > > -Riley > > > > > > On Fri, Nov 15, 2024, 12:52 PM Truc N. Nguyen <trucngu...@cpp.edu> wrote: > > > > > > > Good afternoon, SDAP Team, > > > > > > > > My partner and I are students in Dr. Thomas Huang’s Parallel Processing > > > > class, and we are working on a project about Apache SDAP ETL pipeline > > > > using > > > > Apache Airflow as part of our class presentation. > > > > > > > > Dr. Huang recommended reaching out to your team for guidance. We are > > > > currently facing some technical challenges and would greatly appreciate > > > > any > > > > support or advice you could provide. > > > > > > > > Would it be possible to schedule a meeting with your team sometime next > > > > week? We are also aware of your monthly meeting scheduled for next > > > > Tuesday, > > > > and while we’re unsure if we can attend, we will make an effort to join > > > > if > > > > it's possible. > > > > > > > > Thank you for your time and look forward to hearing from you soon. > > > > > > > > Sincerely, > > > > Truc Nguyen > > > > California State Polytechnic University, Pomona > > > > trucngu...@cpp.edu > > > > CAUTION: This email was not sent from a Cal Poly Pomona service. Exercise caution when clicking links or opening attachments. Please forward suspicious email to suspectem...@cpp.edu<mailto:suspectem...@cpp.edu>.
