It seems that our collaborator is not able to run globus-job-submit (or globus-job-run) successfully, the command just hangs.
When I run "globus-job-run transpgrid.pppl.gov /bin/date" from my desktop, I get the output after some delay. However, on the server the job doesn't seem to complete (see below). 40491 7362 1 0 14:55 ? 00:00:00 globus-job-manager -conf /etc/globus/globus-gram-job-manager.conf -type fork -seg-module fork So my question is why globus-job-submit or globus-job-run hang? Are there any ports that need to be opened so that someone from outside our network would be able to submit "globus-job-run" or "globus-job-submit" successfully? /var/log/globus/gram_*.log ts=2016-03-14T17:47:16.183966Z id=30243 event=gram.job.end level=ERROR gramid=/16505988311912234441/18179527716261904531/ job_status=4 status=-73 reason="the job manager failed to open stdout" ts=2016-03-14T17:48:14.051020Z id=30243 event=gram.script_idle.info level=DEBUG msg="closing idle script handle after 58.0 seconds ts=2016-03-14T17:48:16.184372Z id=30243 event=gram.end level=DEBUG ts=2016-03-14T18:55:45.516392Z id=6403 event=gram.script_read.start level=DEBUG gramid=/16505999305245686836/18179527716261915071/ result=0 nbytes=56 ts=2016-03-14T18:55:45.516545Z id=6403 event=gram.script.log level=DEBUG msg="filestageout staging failed with " ts=2016-03-14T18:55:45.516662Z id=6403 event=gram.script_read.start level=DEBUG gramid=/16505999305245686836/18179527716261915071/ result=0 nbytes=295 ts=2016-03-14T18:55:45.516969Z id=6403 event=gram.script_read.start level=DEBUG gramid=/16505999305245686836/18179527716261915071/ result=0 nbytes=22 ts=2016-03-14T18:55:45.517039Z id=6403 event=gram.job.info level=DEBUG gramid=/16505999305245686836/18179527716261915071/ job_status=4 ts=2016-03-14T18:55:45.517120Z id=6403 event=gram.script_read.start level=DEBUG gramid=/16505999305245686836/18179527716261915071/ result=0 nbytes=1 ts=2016-03-14T18:55:45.517206Z id=6403 event=gram.job.info level=DEBUG gramid=/16505999305245686836/18179527716261915071/ job_status=4 ts=2016-03-14T18:55:45.517261Z id=6403 event=gram.callback.start level=DEBUG gramid=/16505999305245686836/18179527716261915071/ state=4 restart_when_done=false ts=2016-03-14T18:55:45.518783Z id=6403 event=gram.callback.end level=DEBUG gramid=/16505999305245686836/18179527716261915071/ state=4 status=0 msg="Done queuing callback messages" ts=2016-03-14T18:55:45.699160Z id=6403 event=gram.directory_destroy.end level=DEBUG gramid=/16505999305245686836/18179527716261915071/ path="/u/tr_nhoward/.globus/tr_nhoward/.globus/job/transpgrid2/16505999305245686836.18179527716261915071" failures=0 status=0 ts=2016-03-14T18:55:45.743314Z id=6403 event=gram.job.end level=ERROR gramid=/16505999305245686836/18179527716261915071/ job_status=4 status=-73 reason="the job manager failed to open stdout" ts=2016-03-14T18:55:52.105262Z id=6403 event=gram.add_reference_by_jobid.end level=DEBUG jobid="5ed6bf0a-ea16-11e5-91d3-288023a84afc:7395" status=-156 msg="Unknown job ID" reason="the job contact string does not match any which the job manager is handling" ts=2016-03-14T18:55:52.105382Z id=6403 event=gram.add_reference_by_jobid.end level=DEBUG jobid="5ed6bf0a-ea16-11e5-91d3-288023a84afc:7395" status=-156 msg="Unknown job ID" reason="the job contact string does not match any which the job manager is handling" Thank you! Irena On Mon, Mar 14, 2016 at 2:58 PM, Irena Johnson <ijohn...@pppl.gov> wrote: > Raj, > > This time globus-url-copy worked (see output below). However, they are now > not able to run "globus-job-run transpgrid.pppl.gov /bin/date". Command > just hangs. > > > 227 Entering Passive Mode (192,55,106,199,195,88) > > debug: sending command to > gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: > ALLO 348 > > debug: response from > gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: > 200 ALLO command successful. > > debug: sending command to > gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: > STOR ~/incoming/281948_CMOD.REQUEST > > debug: response from > gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: > 150 Beginning transfer. > > debug: writing buffer 0x7ffaba8c0010, length 348, offset=0, eof=true > debug: data callback, no error, buffer 0x7ffaba8c0010, length 348, > offset=0, eof=true > debug: response from > gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: > 226 Transfer Complete. > > debug: operation complete > > > On Mon, Mar 14, 2016 at 2:27 PM, Raj Kettimuthu <ketti...@anl.gov> wrote: > >> From the 227 response to PASV command below, the server is listening on >> port 38198 (149*256 + 54) for the data channel connection. It means that >> GLOBUS_TCP_PORT_RANGE is not set as intended. Try adding ‘port_range >> 50000,50099’ in gridftp.conf >> >> On Mar 14, 2016, at 1:21 PM, Irena Johnson <ijohn...@pppl.gov> wrote: >> >> Here is the output. >> >> >> 227 Entering Passive Mode (192,55,106,199,149,54) >> >> debug: sending command to >> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >> ALLO 348 >> >> debug: response from >> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >> 200 ALLO command successful. >> >> debug: sending command to >> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >> STOR ~/incoming/281948_CMOD.REQUEST >> >> >> I am thinking something is not configured correctly on my server. I have >> xinetd running (with /etc/xinetd.d/gridftp, which should run gsiftp) as >> well as globus-gridftp-server. >> If I stop the service globus-gridftp-server, I am getting this error: >> "error: globus_xio: Unable to connect to transpgrid.pppl.gov:2811", and >> nothing is listening on port 2811. >> >> >> If I start the service globus-gridftp-server, at least I can see that the >> server is listening on the port: >> >> # netstat -tupln |grep 2811 >> tcp 0 0 :::2811 :::* >> LISTEN 1975/globus-gridftp >> >> >> On Mon, Mar 14, 2016 at 2:04 PM, Raj Kettimuthu <ketti...@anl.gov> wrote: >> >>> Can you send the output of globus-url-copy -dbg now? Are you sure the >>> ports 50000-50099 are open? >>> >>> On Mar 14, 2016, at 1:00 PM, Irena Johnson <ijohn...@pppl.gov> wrote: >>> >>> Hello Raj, >>> >>> Our collaborator is still not able to use globus-url-copy (command still >>> hangs). >>> >>> Do I need to set GLOBUS_TCP_PORT_RANGE in /etc/gridftp.conf (the range >>> is set in /etc/xinetd.d/gridftp) ? >>> >>> >>> # more /etc/gridftp.conf >>> # globus-gridftp-server configuration file >>> >>> # this is a comment >>> >>> # option names beginning with '$' will be set as environment variables, >>> e.g. >>> # $GLOBUS_ERROR_VERBOSE 1 >>> # $GLOBUS_TCP_PORT_RANGE 50000,51000 >>> >>> # port >>> port 2811 >>> log_level ERROR,WARN,INFO >>> log_single /var/log/gridftp-auth.log >>> log_transfer /var/log/gridftp.log >>> >>> >>> Thanks, >>> Irena >>> >>> >>> On Mon, Mar 14, 2016 at 12:44 PM, Irena Johnson <ijohn...@pppl.gov> >>> wrote: >>> >>>> I was missing the line "env = GLOBUS_TCP_PORT_RANGE=50000,50099" in >>>> /etc/xinetd.d/gridftp >>>> >>>> # cat /etc/xinetd.d/gridftp >>>> service gsiftp >>>> { >>>> instances = 100 >>>> socket_type = stream >>>> wait = no >>>> user = root >>>> *env = GLOBUS_TCP_PORT_RANGE=50000,50099* >>>> server = /usr/sbin/globus-gridftp-server >>>> server_args = -i >>>> log_on_success += DURATION >>>> nice = 10 >>>> disable = yes >>>> } >>>> >>>> I restarted globus-gridftp-server -- and will try again. >>>> >>>> >>>> My hosts file is below: >>>> # cat /etc/hosts >>>> 127.0.0.1 localhost >>>> >>>> >>>> Thank you! >>>> >>>> On Mon, Mar 14, 2016 at 12:41 PM, José Luis Gordillo Ruiz < >>>> j...@super.unam.mx> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I recently got into a similar problem. It’s a hostname resolution >>>>> issue. >>>>> see the thread for >>>>> https://lists.globus.org/mailman/htdig/gt-user/2007-May/003429.html >>>>> >>>>> saludos, >>>>> >>>>> José Luis Gordillo Ruiz >>>>> Coordinación de Supercómputo >>>>> DGTIC - UNAM >>>>> >>>>> >>>>> >>>>> El 14/03/2016, a las 9:57 a.m., Raj Kettimuthu <ketti...@anl.gov> >>>>> escribió: >>>>> >>>>> Looks like a firewall issue to me. Do you have a set of ports open on >>>>> the server for data channel connections? If you have done that already, >>>>> you >>>>> have to set that port range either in the server command line using >>>>> ‘-port-range startport,endport’ or set set the environment variable >>>>> GLOBUS_TCP_PORT_RANGE (GLOBUS_TCP_PORT_RANGE=startport,endport). >>>>> >>>>> On Mar 14, 2016, at 10:46 AM, Irena Johnson <ijohn...@pppl.gov> wrote: >>>>> >>>>> Dear Globus User Support, >>>>> >>>>> Our collaborator is having problem copying a file via command >>>>> "globus-url-copy". >>>>> Please see below. >>>>> >>>>> >>>>> >>>>> >>>>> It hangs up after this: >>>>> >>>>> [nthoward@cmodws87 ~]$ globus-url-copy -v -dbg -nodcau >>>>> file:///home/nthoward/281948_CMOD.REQUEST >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST >>>>> Source: file:///home/nthoward/ >>>>> Dest: gsiftp://transpgrid.pppl.gov/~/incoming/ >>>>> 281948_CMOD.REQUEST >>>>> debug: starting to put >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST >>>>> debug: connecting to >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST >>>>> debug: response from >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> 220 transpgrid2.pppl.gov GridFTP Server 7.20 (gcc64, 1420641370-85) >>>>> [Globus Toolkit 6.0] ready. >>>>> >>>>> debug: authenticating with >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST >>>>> debug: response from >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> 230 User tr_nhoward logged in. >>>>> >>>>> debug: sending command to >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> SITE HELP >>>>> >>>>> debug: response from >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> 214-The following commands are recognized: >>>>> ALLO APPE REST CWD CDUP DCAU EPSV FEAT >>>>> ERET MDTM STAT ESTO HELP LIST MODE NLST >>>>> MLSC MLSD PASV RNFR MLSR MLST NOOP OPTS >>>>> STOR PASS PBSZ PORT PROT SITE EPRT RETR >>>>> SPOR MFMT SCKS TREV PWD QUIT SBUF SIZE >>>>> SPAS STRU SYST RNTO TYPE USER LANG MKD >>>>> RMD DELE CKSM DCSC >>>>> 214 End >>>>> >>>>> debug: sending command to >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> FEAT >>>>> >>>>> debug: response from >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> 211-Extensions supported >>>>> HTTP >>>>> DCSC P,D >>>>> MFMT >>>>> AUTHZ_ASSERT >>>>> MLSR >>>>> MLSC >>>>> UTF8 >>>>> LANG EN >>>>> DCAU >>>>> PARALLEL >>>>> SIZE >>>>> MLST >>>>> >>>>> Type*;Size*;Modify*;Perm*;Charset;UNIX.mode*;UNIX.owner*;UNIX.uid*;UNIX.group*;UNIX.gid*;Unique*;UNIX.slink*;X.count; >>>>> ERET >>>>> ESTO >>>>> SPAS >>>>> SPOR >>>>> REST STREAM >>>>> MDTM >>>>> PASV AllowDelayed; >>>>> 211 End. >>>>> >>>>> debug: sending command to >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> SITE CLIENTINFO scheme=gsiftp;appname="globus-url-copy";appver="9.18 >>>>> (gcc64, 1448068506-85) [unknown]"; >>>>> debug: response from >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> 250 OK. >>>>> >>>>> debug: sending command to >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> TYPE I >>>>> debug: response from >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> 200 Type set to I. >>>>> >>>>> debug: sending command to >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> DCAU N >>>>> >>>>> debug: response from >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> 200 DCAU N. >>>>> >>>>> debug: sending command to >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> PASV >>>>> >>>>> debug: response from >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> 227 Entering Passive Mode (192,55,106,199,164,70) >>>>> >>>>> debug: sending command to >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> ALLO 348 >>>>> >>>>> debug: response from >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> 200 ALLO command successful. >>>>> >>>>> debug: sending command to >>>>> gsiftp://transpgrid.pppl.gov/~/incoming/281948_CMOD.REQUEST: >>>>> STOR ~/incoming/281948_CMOD.REQUEST >>>>> >>>>> >>>>> >>>>> Thank you, >>>>> Irena >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Irena >>>> >>> >>> >>> >>> >> >> >> -- >> Irena >> >> >> > > > -- > Irena >
