Hi Suresh, Please see my inline comments. On Tue, May 15, 2012 at 8:55 AM, Marlon Pierce <[email protected]> wrote:
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I'm thinking of a google spreadsheet.... > > On 5/15/12 8:53 AM, Marlon Pierce wrote: > > We have a large collection of use cases, so it would be a good exercise > to apply the email below to specific applications. > > > > > > Marlon > > > > > > On 5/13/12 9:49 AM, Suresh Marru wrote: > >> Hi All, > > > >> I am trying to revisit the Airavata support for all command line > options we pass to applications. Airavata's goal is to make end users > oblivious to any application execution details, but application service > providers need flexibility to configure all possible application options. > > > >> Some terminology like arguments vs parameters vs attributes get > ambiguous. They differ by definition but in practice they are often used > interchangeably. For Airavata, we should avoid a confusion between whats > exposed in wsdl's vs whats passed to application. This matches the > semantics as well, for instance, an argument is an instance of parameter. > This discussion is about what Airavata passes to the command line > applications. I am not suggesting any changes to wsdl's and schemas which > use xml definitions. For applications I am suggesting to use the > terminology per POSIX standard definitions [1]. I also propose that we > should try and follow the utility syntax guidelines [2]. If an application > does not follow these guidelines, we suggest it be wrapped by a shell > script so we can pass arguments and flags confirming to standard practices. > > > >> Application refers to the commands airavata executes on computational > resources. > > > >> Working directory. Airavata should insist on executing each invocation > in a unique working directory. Some applications try and change to a static > directory, but if proper uniqueness is not followed for output and log > files, we risk overwriting executions producing unintended outputs. Also, > avoid writing to home directories and source directories. This might have > side effects and a overrun log file might fill the disk space and freeze > further usage of that account. > > > >> Arguments: > >> * should support application arguments and provide a way to specify > both required and optional. > >> In the case of optional parameters, the resulting wsdl's attributes > should have minOccurs=0 and airavata should skip passing that value to > application (if not specified). > > > >> * Airavata *should not* support arguments with operands followed by > commands. These additional commands get forked without having control over > the process id and monitoring and exit status of these series of commands > gets tricky. More over, the underlying grid job managers do not like > treating a chain of commands as one executable. Rather encourage explicitly > specifying the execution chain and associated I/O. > > > >> * Airavata should also support flags only ( they serve different > purpose than option flags). Flags normally prefix with '--'. These flags > control the execution of the application like --verbose, --fast, --use-fft, > e.t.c > > > >> * Arguments can be passed to the application as standardinput (with > redirector operator) or as name-value pairs or with option flags. The > option flags should always prefix with the POSIX standard of '-'. > > > >> * If the arguments are preceded by an option flag they do not need to > be ordered. But if the arguments are passed just as values, applications > are sensitive to the order the arguments are passed. In this case, optional > arguments have to carefully handled, as missing an argument in between will > mislead. > +1 for these features. > > > >> * If an argument is a file type, and if the file has a remote supported > protocols of (http, ftp, gsiftp, s3) then the file has to be staged first > and only local path passed to the application. Application should be able > to consume the full local path and if only basename is required, it should > be able to handle it internally. > I think we already support this other than we do not support S3 file transfer in Airavata. > > > >> * If an application requires a remove ftp url as an argument, then it > should be specified as a string, in which case Airavata will skip staging > that url and will pass the url as is to the application. > +1 > > > >> * Implicit Parameters: As much as possible, Airavata should insist on > one-on-one match between inputs specified in service description to whats > passed to application. But there will be exceptions like fortran > applications which uses NAMELIST standard to specify all inputs in a config > file and pass only this file to the application. In these cases, the > application still needs to stage some data files to the remote compute > server but these file names or implicitly specified in the application. The > application typically looks for these files relative to working directory > or to input namelist file. > > > >> Outputs: > >> * Airavata should support standard outputs and errors and optionally > provide a way to specify the names of stdout and stderr. > >> * All outputs required to be staged out of the compute machine or > scratch working directory be explicitly specified. > >> * If the output file name(s) are predetermined or specified at in a > config file, then the name should be specified in application description. > In the cases, where output file names are not deterministic, a regular > expression or a containing directory should be specified. > >> * If the application requires the output file name be passed at command > line like -out output.txt, then airavata should provide support for these > outputs flags. > I don't think this is a valid requirement ... Have you seen any applications which provide outputs like this ? > >> * Airavata should support outputs which can be optionally produced. If > an optional output is not generated but application exits with exit code 0, > then the application should be marked as success. (A different discussion > on application execution success criteria is needed). > If we are going to support this how are we going to find an erroneous situation with optional outputs. right now we throw errors if any of the output is empty. > >> * A default output data directory should be created on the remote > compute resource. The application description should be able to specific an > overriding name for this directory. > We already support this. > >> * Airavata should support applications/shell script wrappers which > print name-value pairs of output content or file paths to standard out. > I am not again clear about this usecase and how are we going to suppoort this, I htink this is unnecessary feature if nobody is going to give output with name value pairs. Lahiru > > > >> Once we discuss this topic, we should raise JIRAs for any missing > features and also add these on website/wiki. > > > >> Cheers, > >> Suresh > > > >> [1] - > http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html > >> [2] - > http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html#tag_12_02 > > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.16 (Darwin) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQEcBAEBAgAGBQJPslI0AAoJEEfVXEODPFIDOrAIAKI6yUXoWTVx6vrX2xCZlTta > vRxQS/Kpc7OVtO6IFJKtpODfrQ10GCgynweewt8rF7c8JztFbLWqNmSCFiYnRdrc > B+ZAg5EZRDwW+bs9OO0FhFhp/DkcJKE97o0Kx0YRDPsAQj+SS9OCpzneFR/6mbQ8 > 3AI2x/byBIE4jwaBUZjH31hmXzS1M7ibYR5J10gBqO2ONgeTShipWgbR/QyjebFs > /g3dtfaVwiaB99qRa6bVf3dyAB2wIWMtwRvtoAzqQTdYHMnkiE+azF2/02tfRXiu > LIizzd/ErW3XVHVpUbALdu4Grue3YeaOUmG69yjq8Ipzjk9i+BVA22dvaWebKb0= > =4Bss > -----END PGP SIGNATURE----- > -- System Analyst Programmer PTI Lab Indiana University
