Re: [gt-user] GridFTP Troubleshooting

[Ian Foster]

Dear Amit:

Unfortunately I am not up to speed on the certificate issues. Things would be 
simpler if ATLAS used Globus -)

Regards — Ian

> On Aug 28, 2018, at 11:25 AM, Amit Kumar  wrote:
> 
> Hi Ian,
> 
> Thank you for quick response. Yes we have Globus connect server setup along 
> with an endpoint for our institution, and that works fine for us. This is a 
> new setup that I am working on so that I can get out LHC collaborators to use 
> GridFTP along for their ATLAS computing needs. 
> This (Globus connect) installation of GridFTP is a slightly different and 
> older works nicely for our DTN, but I need this new setup to use different 
> authentication (LCMAPS VOMS) hence working on another instance. 
> 
> I think I pretty close to getting this setup if I can figure out certificate 
> issues, seems basic to me but I lack a bit of understanding on this. Any 
> insight here will be a great help.
> 
> Thank you,
> Amit
> 
> On Tue, Aug 28, 2018 at 11:18 AM Foster, Ian T.  <mailto:fos...@anl.gov>> wrote:
> Dear Amit:
> 
> Have you tried installing Globus Connect and using the Globus transfer 
> service? Easier, faster, more reliable, …
> 
> See https://www.globus.org/data-transfer 
> <https://www.globus.org/data-transfer>
> 
> Regards — Ian
> 
>> On Aug 28, 2018, at 11:07 AM, Amit Kumar > <mailto:ihavebeenfor...@gmail.com>> wrote:
>> 
>> Dear GT,
>> 
>> I have setup GridFTP and trying to use globus-url-copy to test it and 
>> running it following errors, and wondering if you can help me here. 
>> 
>> I have a InCommon cert for the gridftp host, I also have InCommonRSAServerCA 
>> cert in the path but I am not able to find what issue certificate is it 
>> complaining about?
>> 
>> Any help here is greatly appreciated.
>> 
>> Thank you,
>> Amit
>> 
>> 
>> $ globus-url-copy -vb -dbg gsiftp://gftp.host.xxx.edu/tmp/zero.source 
>> <http://gftp.host.xxx.edu/tmp/zero.source> file:///tmp/zero.dest <>
>> Source: gsiftp://gftp.host.xxx.edu/tmp/ <http://gftp.host.xxx.edu/tmp/>
>> Dest:   file:///tmp/ <>
>>   zero.source  ->  zero.dest
>> debug: starting to get gsiftp://gftp.host.xxx.edu/tmp/zero.source 
>> <http://gftp.host.xxx.edu/tmp/zero.source>
>> debug: connecting to gsiftp://gftp.host.xxx.edu/tmp/zero.source 
>> <http://gftp.host.xxx.edu/tmp/zero.source>
>> 
>> debug: response from gsiftp://gftp.host.xxx.edu/tmp/zero.source 
>> <http://gftp.host.xxx.edu/tmp/zero.source>:
>> 220 10.10.29.20 GridFTP Server 12.8 (gcc64, 1531931206-85) [Globus Toolkit 
>> 6.0] ready.
>> 
>> debug: authenticating with gsiftp://gftp.host.xxx.edu/tmp/zero.source 
>> <http://gftp.host.xxx.edu/tmp/zero.source>
>> debug: fault on connection to gsiftp://gftp.host.xxx.edu/tmp/zero.source 
>> <http://gftp.host.xxx.edu/tmp/zero.source>: globus_ftp_control: 
>> gss_init_sec_context failed
>> debug: data callback, error globus_ftp_control: gss_init_sec_context failed, 
>> buffer 0x2aaab0ff3010, length 0, offset=0, eof=true
>> debug: operation complete
>> 
>> error: globus_ftp_control: gss_init_sec_context failed
>> OpenSSL Error: s3_clnt.c:1264: in library: SSL routines, function 
>> ssl3_get_server_certificate: certificate verify failed
>> globus_gsi_callback_module: Could not verify credential
>> globus_gsi_callback_module: Could not verify credential: unable to get 
>> issuer certificate
> 
> _
> Ian Foster
> Director, Data Science and Learning Division; Senior Scientist; Distinguished 
> Fellow, Argonne National Laboratory
> Arthur Holly Compton Distinguished Service Professor of Computer Science, 
> University of Chicago
> Fellow, Institute for Molecular Engineering
> Chief Troublemaker, Globus, www.globus.org <http://www.globus.org/> 
> Check out our new book: Cloud Computing for Science and Engineering
> Online at https://cloud4scieng.org <https://cloud4scieng.org/>
> Tel: +1 630 252 4619
> 
> 
> 
> 

[gt-user] Announcement regarding future support for open source Globus Toolkit

[Ian Foster]

[Below is also at 
https://github.com/globus/globus-toolkit/blob/globus_6_branch/support-changes.md
 
]

Support for open source Globus Toolkit will end as of January 2018; The Globus 
cloud service and Globus Connect are unaffected

The Globus team at the University of Chicago has developed and supported the 
open source Globus Toolkit  for close to 20 
years. Globus Toolkit GridFTP and GSI software, in particular, have been widely 
used within the scientific community for data transfer and security. Since 
2010, we have leveraged that experience to develop the Globus cloud service 
, which provides enhanced capabilities for data 
transfer plus new identity and group management, data sharing, data 
publication, and other functions. Most Globus Toolkit users have by now moved 
to the Globus cloud service and the associated Globus Connect endpoint 
software, used by tens of thousands of people to manage billions of files on 
more than 10,000 active endpoints. Importantly, a subscription-based 
sustainability model allows the Globus team to assure Globus cloud service 
users of our long-term viability.

We are announcing that, starting in January 2018, the Globus team at the 
University of Chicago will no longer support the open source Globus Toolkit, 
except for its use with the Globus cloud service by Globus subscribers. By the 
end of 2018, all endpoints connected to the Globus cloud service using the open 
source Globus Toolkit GridFTP server must migrate to Globus Connect. At the end 
of 2018, we will discontinue all maintenance (including security patches) and 
distribution of the open source Globus Toolkit. Endpoints using Globus Connect 
Server or Globus Connect Personal will be unaffected, as long as they continue 
to perform routine software updates.

We realize that this change in long-established practice may create challenges 
for those users who have not migrated to the Globus cloud service, and so we 
explain here first the reason for this change, and second, why we recommend 
that remaining Globus Toolkit users migrate to the Globus cloud service. Note 
that this change in support policy only affects direct users of the open source 
Globus Toolkit. Users of the Globus cloud service, and sites running Globus 
Connect to make their storage systems accessible via the Globus cloud service, 
are unaffected.

Why we are ceasing support of the open source Globus Toolkit. In a word, 
funding. The open source Globus Toolkit, like any software, requires constant 
effort to answer support requests, apply security patches, and perform other 
maintenance. This work has long been financed by grants, primarily from the 
U.S. National Science Foundation (NSF). However, our last such grant ends this 
fall and, after extensive discussions with funding agencies and Globus Toolkit 
users, we see little opportunity for further funding for such support 
activities.

In addition, Globus Connect is quickly diverging from the Globus Toolkit, as 
detailed in the FAQ. We must focus on supporting the Globus Connect code base, 
as this is what our subscribers depend on. Supporting and maintaining the open 
source Globus Toolkit would involve substantial, separate effort, for which 
no-one has shown a willingness to provide funding.

Why we recommend that current Globus Toolkit users migrate to, and subscribe 
to, the Globus cloud service. The Globus cloud service provides more 
functionality than the open source Globus Toolkit, is adding new capabilities 
rapidly, and is underpinned by a proven, sustainable business model that will 
ensure that it persists into the future. Subscribers get high-quality support 
from the Globus team, an assurance of software longevity, and the satisfaction 
of supporting a professional team that develops and operates valuable software 
for the research community. They also get access to capabilities that are 
superior to those provided by the open source Globus Toolkit: for example, 
automatic performance optimization, data sharing, specialized storage 
connectors (e.g., Google Drive, Amazon S3, Ceph, Spectra Logic BlackPearl), 
data publication, improved security, REST APIs, and powerful management 
services. The functionality gap between Globus Toolkit and the Globus cloud 
service will grow even larger as we develop more modern capabilities to support 
the research data management lifecycle, such as more storage connectors, 
web-compatible HTTPS+OAuth2 access to storage, metadata handling, data search 
with access control, and HIPAA compliance. Subscribers are collectively 
investing in, and helping us develop, these capabilities for the benefit of the 
broader research community.

Please contact us with any comments, questions, or concerns, at 
supp...@globus.org .

 

Re: [gt-user] Globus Toolkit with Pidora 18 - ARM6

[Ian Foster]

Dear Fabio:

Thanks for letting us know.

Regards -- Ian.

On Oct 8, 2013, at 8:02 AM, Fabio Moreira souza.fab...@gmail.com wrote:

 Dear Ian,
 
 I've decided to install Fedora Remix instead of Pidora. With Remix I could 
 install all necessary packages to build a GridFTP server with Raspberry.
 
 Best Regards
 
 
 On Mon, Oct 7, 2013 at 7:44 PM, Ian Foster fos...@anl.gov wrote:
 Fábio:
 
 Did you get this working?
 
 Regards -- Ian
 
 On Aug 8, 2013, at 3:18 PM, Fábio Moreira souza.fab...@gmail.com wrote:
 
  Hi,
 
  I would like to use Globus Toolkit with a raspberry-arm6 and I've installed 
  pidora 18. Although many Globus package has already been compiled from 
  Fedora to Pidora, some packages, globus-gridftp-server and myproxy for 
  instance, are missing.
 
  Can anyone help me?
 
  Thanks in advance,
 
  Fabio MS
 
 
 Fábio MS

Re: [gt-user] license about Globus Toolkit

[Ian Foster]

Dear Antonio:

Globus Toolkit components are released under Apache v2. Strangely that 
information seems to have disappeared in a recent web site reorganization. 
We'll get that fixed.

Regards -- Ian.


On Aug 19, 2013, at 9:33 PM, Antonio Miguel Bonacin 
antonio.mig...@tenbu.com.br wrote:

 Hello,
 
 I want to know about the part of the Globus Toolkit license for installation 
 on RedHat.
 
 I'll installer a grid computing using Globus Toolkit and I don't no about the 
 license of the product.
 
 Thanks,
 
  
  
 Att,
  
 image001.jpg
 Antonio Miguel Bonacin
 Consultoria
 +55 (11) 3853-1291
 +55 (11) 98564-8727
 www.tenbu.com.br
  
 Antes de imprimir, pense na sua responsabilidade com o MEIO AMBIENTE.
 Esta mensagem, incluindo seus anexos, pode conter informações privilegiadas 
 e/ou de caráter confidencial. Se você não é o destinatário ou pessoa 
 autorizada a recebê-la, você não está autorizado a utilizar esse material 
 para qualquer fim. Se você recebeu esta mensagem por engano, por favor, nos 
 informe respondendo a este e-mail e apague-o.
  
 Before printing, think about your responsibility toward the ENVIRONMENT
 This message (including its attachments) can be confidential and private. If 
 you are not authorized to receive it you are not allowed to use it for any 
 reason. If you received this by mistake, please contact-us by replying this 
 e-mail and erase it.
  
  
  

Re: [gt-user] Data Replication Service

[Ian Foster]

Dear Walter:

The right tool to use for Data Replication now is Globus Online, in my view. 
See www.globusonline.org. 

Ian.

On Jun 14, 2013, at 4:51 PM, Walter Landry wlan...@caltech.edu wrote:

 Hello,
 
 I was wondering what the status of the Data Replication Service is.
 Is it still included with GT?  I can see documentation for it in GT
 4.0, but I can not find it in the documentation for GT 5.2.4.
 
 Thanks,
 Walter Landry

Re: [gt-user] Using globus-url-copy between ftp - gsiftp

[Ian Foster]

Good. So did I :)

On Jun 7, 2013, at 1:37 PM, Brock Palen bro...@umich.edu wrote:

 Ian,
 
 I put in a plug with the HMB project for the benefits of setting up anon 
 endpoints for their data.
 
 Brock Palen
 www.umich.edu/~brockp
 CAEN Advanced Computing
 bro...@umich.edu
 (734)936-1985
 
 
 
 On Jun 7, 2013, at 11:45 AM, Ian Foster fos...@anl.gov wrote:
 
 we should get as many people as possible to request this.
 
 On Jun 7, 2013, at 10:37 AM, Brock Palen bro...@umich.edu wrote:
 
 Steve, 
 
 We are consumers of data from a resource we don't have control over.  
 Getting that resource on Globus would be wonderful though,
 
 It's the Human Microbiome Project.
 http://www.hmpdacc.org/resources/data_browser.php/
 
 They provide public FTP and HTTP download, getting them in as a public data 
 source on GO would do the community a lot of good. 
 
 
 
 Brock Palen
 www.umich.edu/~brockp
 CAEN Advanced Computing
 bro...@umich.edu
 (734)936-1985
 
 
 
 On Jun 7, 2013, at 9:40 AM, Steve Tuecke tue...@ci.uchicago.edu wrote:
 
 Is it possible to upgrade the FTP server to (anonymous) GridFTP? That 
 should allow it to work with both globus-url-copy and Globus Online.
 
 -Steve
 
 On Jun 7, 2013, at 7:57 AM, Brock Palen bro...@umich.edu wrote:
 
 MIke thanks good to know that it only works with file:///  which kinda 
 defeats the point with what we are trying to do.
 
 Thanks.
 
 Brock Palen
 www.umich.edu/~brockp
 CAEN Advanced Computing
 bro...@umich.edu
 (734)936-1985
 
 
 
 On Jun 7, 2013, at 2:17 AM, Michael Link ml...@mcs.anl.gov wrote:
 
 Hi Brock,
 
 It is likely that those servers don't support third party transfers, or 
 fxp as it is commonly called with standard ftp servers.  While some 
 standard ftp server software supports this, it is often not enabled on 
 public servers.
 
 With the second server, it recognizes that the port you're telling it to 
 connect to isn't the same one that the client is connecting from, and 
 fails.
 
 With the first I suspect it is simply ignoring the address portion of 
 the PORT argument and attempting to connect to the client address.
 
 Either should work if you're able to run globus-url-copy on the same 
 host as the gridftp server, though that may of defeat the purpose.
 
 Mike
 
 On 6/4/2013 4:36 PM, Brock Palen wrote:
 I need to have a user push a lot of data though a host from an FTP only 
 site to our gridftp server.  The user has to do it this way because we 
 don't allow user login on this host  (sftp, and globusonline only)
 
 Because we setup this host with GCMU we have self signed certs etc. but 
 I was able to manualy set subjects.  Problem is I can't get the 
 transfer to work:
 
 globus-url-copy -dbg -v -ds 'subject' 
 ftp://public-ftp.hmpdacc.org/Illumina/stool/SRS045004.tar.bz2 
 gsiftp://gridftp-flux.engin.umich.edu/scratch/support_flux/brockp/tmp/
 
 Just hangs, and the file says size zero.
 
 Last line of debug information is:
 debug: sending command to 
 ftp://public-ftp.hmpdacc.org/Illumina/stool/SRS045004.tar.bz2:
 PORT 141,212,30,14,196,207
 
 I tried using another ftp server and I get a different error:
 
 globus-url-copy  -dbg -v -ds 'subject' 
 ftp://mirror.optus.net/fedora/linux/extras/README 
 gsiftp://gridftp-flux.engin.umich.edu/scratch/support_flux/brockp/tmp/
 
 error: globus_ftp_client: the server responded with an error
 500 Illegal PORT command
 
 Any thoughts on why I can't do this?
 
 Brock Palen
 www.umich.edu/~brockp
 CAEN Advanced Computing
 bro...@umich.edu
 (734)936-1985
 
 
 
 
 
 

Re: [gt-user] How to distribute problems to multiple resources (computers)?

[Ian Foster]

I think that BOINC is what you are looking for. Or Condor.

On Nov 18, 2012, at 5:52 PM, john...@hushmail.com wrote:

 Greetings GT community,
 
 Suppose that a pool of computers are able to donate their idle CPU time, how 
 can a problem (i.e. an piece of code) get executed in them in a distributed 
 manner? 
 
 For example, when I use the command globus-job-submit, or globus-job-run, how 
 will my local machine know where should these jobs to be submitted?
 
 I'm expecting that every resource should register itself to a discovery data 
 base (service) that is hosted on a server(s). And that grid users (e.g. 
 programmers/researchers) submit problems, they submit it somewhere that will 
 dispatch them to multiple resources (CPU donators) according to a scheduler 
 and an execution management plan that decies what to do in case of a failure.
 
 However, I fail to see how the above thoughts map to GT5 after following my 
 reading of the quick start guide in 
 http://www.globus.org/toolkit/docs/5.2/5.2.2/admin/quickstart/ -- what is in 
 the guide is pretty controlled by the user/programmer (e.g. he specifies 
 which computer to execute which commands on).
 
 Rgrds,
 J

Re: [gt-user] (no subject)

[Ian Foster]

Dear Melvin:

Note that Globus Online does NOT move your data to systems you do not control. 

Ian.

On Oct 9, 2012, at 9:47 AM, gridftp user gridftpu...@hotmail.com wrote:

 
 Hi Melvin,
 I recommend that you install Globus Connect 
 Multi-User - https://www.globusonline.org/gcmu/ and use Globus Online 
 (www.globusonline.org) to do the transfer. 
 
 In case of 
 the transfer you mention below, if you add say '-p 4' or '-p 8' to the 
 globus-url-copy command line, you should get much better transfer rate. 
 That said, you should really be using Globus Online as it does 
 autotuning of the parameters to get good performance. 
 
 Raj
 
 
 Hi Raj,
 
 
 
 Thank you for taking the time to respond. Adding the -p switch made no real 
 difference:
 
time globus-url-copy -v -p 8 file:/opt/120_GB_file.random 
 sshftp://172.22.10.206/home/test/bigfile.random
 
 
 
real 17m34.074s
 
user 0m3.116s
 
sys 1m55.821s
 
 
 I thought I must have managed to miss that in my reading, but that 
 information is unfortunately missing
 from all of the on-line documentation that I've read over the past week. In 
 fact, even knowing what to look
 for I am unable to find it mentioned in any site documentation at all. I did 
 find it by typing globus-url-copy
 -help, which gives me some reading that I've not gone through yet.
 
 Any chance the lack of improvement is due to the fact that the traffic is 
 going out one server, through a
 single switch, and back into the receiving server? I ask because I am not a 
 network engineer by any means
 and I am trying to get an understanding of what is going on. If the 
 infrastructure is an issue then an outside
 test is in order.
 
 I appreciate the suggestion of using Globus Online. We looked into that, but 
 we do not want our
 data residing on systems we do not control, even if it is there for a short 
 time. That is why I am
 trying so hard to set up a server/client system, so we can have almost 
 complete control over both
 end-points and transfer parameters.
 
 I'll keep plugging away.
 
 Melvin
 
 

[gt-user] Please help: Seeking information from GT users

[Ian Foster]

Dear friends:

We are preparing a proposal to the National Science Foundation to provide 
continued support for the Globus Toolkit (GT): specifically, for the GRAM, 
GridFTP, Integrated Information Service, Grid Security Infrastructure, and 
jGlobus components.

Our success will depend in part on our ability to get NSF researchers and 
educators to communicate to their program managers the importance of this 
technology for their work. In the absence of such communications, NSF will 
conclude that Globus software isn't that important.

We know that there are many users out there. For example, just in the last 24 
hours, GridFTP servers with usage reporting enabled reported 17M transfers, 
totaling 0.5 PB (that's an average of 200 files/sec and 6 GB/sec). But we need 
to know *who*, *what*, and *why* for this usage--and for the usage of other 
components.

Thus, we would like to request your help as follows:

1) If you an NSF-supported researcher or educator who uses GT in your work, 
please:

a) tell us how and why it is important to you

b) tell us what improvements and enhancements will be most important to you

c) let us know if you are prepared to contact your NSF program manager to tell 
them these things, and who that would be

Note: While we welcome detail and completeness, just a few words on these 
topics will be immensely useful

2) If you are a non-NSF-supported researcher or educator (in the US or 
elsewhere) who uses GT in the work, please:

a) tell us how and why it is important to you

b) tell us what improvements and enhancements will be most important to you

c) tell us who supports your work

Regards -- Ian.

Re: [gt-user] GridFTP + UDT success for bio-mirror.net and code patches

[Ian Foster]

Dear Don:

Thank you very much for the detailed report.

We'll follow up separately on the code patches.

On the performance front: I know nothing about your configuration, but I can 
imagine that you might be able to do even better. Would you be interested in 
having us work with you on identifying the bottleneck(s)?

Regards -- Ian.


On Aug 13, 2010, at 1:04 PM, Don Gilbert wrote:

 
 Dear Globus folks,
 (maybe this should go to -dev rather than -user)
 
 We have been testing GridFTP + UDT (gt5.0.2) for the Bio-mirror.net  project,
 which daily mirrors 100's of GB of biology data around the world. The results
 so far are very good. There have been some installation problems, and
 I have code patches if you want them.  A few of these I think would be
 usefully added to a future GridFTP release.
 
 - Don Gilbert
 
 Summary of GridFTP (gt5.0.2) patches
  -- UDT change udt4/src/channel.cpp for Solaris10:
   add UNIX to ifdef BSD, OSX for setsockopt max buffer
   -- UDT globus configure{.ac}: update for *solaris* and *darwin* flags
   (always compiles with -DLINUX otherwise, also fails to configure
   on Linux-Powerpc, were it tries assembly for wrong processor)
 
   -- gridftp server, anonymous ftp directories:  limit to anonymous  user 
 home dir
   in globus_i_gfs_control.c:globus_l_gfs_get_full_path() : add  anon_user 
 path restrictions
   (I tried chroot() first but that was overly complex to decipher).
 
   -- gridftp server: log_transfers, UDT client IP was 0.0.0.0 always
   globus_i_gfs_data.c:globus_l_gfs_data_end_transfer_kickout() :  small 
 patch
 
   -- globus-url-copy:  set dest file timestamp to match source file  time
 (essential for -sync, usual FTP practice)
   globus_url_copy.c: add globus_l_set_dest_timestamp() and add  int 
 src_mtime
 in globus_l_guc_src_dst_pair_t and globus_l_guc_transfer_t  structures
 
 Here are our results to date, with testing still in progress.
 GridFTP TCP and UDT times for 113 GB transfer
 from Bio-mirror.net (Indiana USA)
 
Ping  Time(min)  TCP/  Distance
 SiteRTT  TCP   UDT  UDTKm  Network Route
 --
 NCSA10   139   138   1200  Indiana - U of Illinois - NCSA
  13.9  13.9Megabytes/sec
 Purdue  17   125   125   1500  In. - Chicago - Purdue, Indiana
  15.3  15.3Mb/sec
 ORNL25   361   120   3   1200  In. - Chi. - Nashv., Tennesee - ORNL
   5.3  15.9Mb/sec
 TACC37   616   120   5   2000  In. - Chi. - Huston, Texas - TACC
   3.1  15.9Mb/sec
 SDSC65----   -   3300  In. - Chi. - LA, California - SDSC
 ...
 CSTNET 274----  14  12000  In. - Michigan - Kreonet, Korea -  
 Beijing, China
  0.48  6.67Mb/sec (from preliminary 3GB test)
 --
   Transfer times (minutes), and below speed in Megabytes/second,
   for TCP and UDT, and the TCP/UDT ratio.
   NCSA, Purdue, ORNL, TACC, SDSC are Teragrid.org sites in USA.
   Land/sea line distance is given in Km.
   RTT is network distance as average round trip ping time in ms.
   TCP and UDT transfers were run simultaneously from each site.
 
 -- d.gilbert--bioinformatics--indiana-u--bloomington-in-47405
 -- gilbe...@indiana.edu -- http://marmot.bio.indiana.edu/
 

[gt-user] Important information on Globus events and plans

[Ian Foster]

 are excited about this re-invogoration of the Globus software and  
community, and the increasing value the Globus community can bring to  
the many multi-institutional scientific and biomedical communities  
whose need for robust Grid computing middleware continues to grow  
unabated.


Viva Globus!

The Globus Team

[1] “The Anatomy of the Grid: Enabling Scalable Virtual  
Organizations,” Ian Foster, Carl Kesselman, Steven Tuecke.  
International Journal of Supercomputer Applications, 15(3), 2001.