Here's I've stripped it down to a very basic curl command and run it on a variety of hosts. There's no header or footer in the file that would contribute to the discrepancy.
Running the following 3 times: curl " https://rcmes.jpl.nasa.gov/query-api/query.php?datasetId=3¶meterId=36&latMin=-45.76&latMax=42.24&lonMin=-24.64&lonMax=60.28&timeStart=19900101T0000Z&timeEnd=20071231T0000Z"; > curl.1 On my Mac: ls -l curl.* rw-rr- 1 michaelanderson staff 419156308 Jan 27 12:48 curl.1 rw-rr- 1 michaelanderson staff 432335472 Jan 27 12:54 curl.2 rw-rr- 1 michaelanderson staff 422500568 Jan 27 12:59 curl.3 On a pretty large AWS with RHEL 7: rw-rw-r-. 1 ec2-user ec2-user 546429902 Jan 27 18:12 curl.1 rw-rw-r-. 1 ec2-user ec2-user 578373399 Jan 27 18:15 curl.2 rw-rw-r-. 1 ec2-user ec2-user 484616136 Jan 27 18:17 curl.3 On an AWS with 16 cpu and 122 GB RAM on RHEL 7: -rw-rw-r--. 1 ec2-user ec2-user 659658707 Jan 27 18:30 curl.1 -rw-rw-r--. 1 ec2-user ec2-user 671670784 Jan 27 18:32 curl.2 -rw-rw-r--. 1 ec2-user ec2-user 686702730 Jan 27 18:34 curl.3 On an AWS with 64 CPU, 488 GB RAM and 25GB network on RHEL 7: -rw-rw-r--. 1 ec2-user ec2-user 675561581 Jan 27 18:41 curl.1 -rw-rw-r--. 1 ec2-user ec2-user 688015459 Jan 27 18:43 curl.2 -rw-rw-r--. 1 ec2-user ec2-user 633800036 Jan 27 18:45 curl.3 On Thu, Jan 25, 2018 at 12:04 AM, lewis john mcgibbney <[email protected]> wrote: > Hi Michael, > To answer your final question, I have absolutely no idea. I've forwarded > this to the RCMES team however and will hopefully have a response from you > sometime soon. > Lewis > > On Mon, Jan 22, 2018 at 8:32 PM, <[email protected]> > wrote: > > > From: Michael Anderson <[email protected]> > > To: [email protected] > > Cc: > > Bcc: > > Date: Sun, 21 Jan 2018 19:30:25 -0500 > > Subject: CLIMATE-744 Cannot load TRMM data from RCMED > > I was working on this JIRA: CLIMATE-744 > > <https://issues.apache.org/jira/browse/CLIMATE-744>. A call to RCMED > for > > TRMM throws an error about not being able to reshape the data. > > > > I wrote the following based on the info given in the JIRA: > > > > from datetime import datetime as dt > > start_time = dt.strptime('1990-01-01', '%Y-%m-%d') > > end_time = dt.strptime('2007-12-31', '%Y-%m-%d') > > TRMM = rcmed.parameter_dataset(3, 36, -45.76, 42.24, -24.64, 60.28, > > start_time, end_time) > > > > In rcmed.py _get_data, the call to urlopen returned a different amount > of > > data on each run of the program. > > > > I repeatedly ran the following URL in my browser and each run also > returned > > a different amount of data. > > > > https://rcmes.jpl.nasa.gov/query-api/query.php?datasetId= > > 3¶meterId=36&latMin=-45.76&latMax=42.24&lonMin=-24.64& > > lonMax=60.28&timeStart=19900101T0000Z&timeEnd=20071231T0000Z > > > > When I constrain the dates to a smaller range, one year, the same number > of > > results are returned consistently with each run of the program. > > > > Does RCMED, by any chance, have any constraints on the amount of data > that > > can be returned? > > > > >
