Hi Dimuthu, Thanks for working on this and testing this functionality. I think this will be a great new feature.
On the PR I also added an API method to fetch the status of an intermediate
output fetching process:
ProcessStatus getIntermediateOutputProcessStatus(AuthzToken authzToken,
String airavataExperimentId, List<String> outputNames)
This will make it a little easier for clients to figure out the status of the
process.
As for UI, you're right. Currently the output file UI only displays when an
experiment is COMPLETED. I'm thinking of having a panel for each output file
that will display once the experiment is EXECUTING. Each panel will have a
button to fetch the current file. When the fetch is executing the button is
disabled and a spinner displays with a message that the file is being fetched.
Once the file is fetched, if it is a text file (and maybe within some max file
size limit) then it will display in the panel and the user can scroll through
the file. This is the main use case, retrieving the current value of a log file
or STDOUT or something similar. If the file is too large or binary then instead
of displaying the file directly there will be a download button. That's the
general idea.
Some other things, maybe lower priority:
- I need to think about what to do for a URI_COLLECTION (maybe a tabbed
interface).
- Output view providers, as long as they can work with partial files, should be
able to be invoked on these intermediate files as well and so I also need to
think about how to integrate those.
Thanks,
Marcus
> On Jan 2, 2022, at 11:03 PM, DImuthu Upeksha <dimuthu.upeks...@gmail.com>
> wrote:
>
> Hi All
>
> Marcus and I were working on the feature of grabbing output files from an
> already running experiment so that users can determine whether the experiment
> is running as expected without waiting for it to complete. This will be
> tremendously helpful for long-running and resource-intensive jobs in order to
> determine possible early terminations in case of an input or convergence
> issue. We added a new API [1] to invoke this feature and this will create a
> separate process that is only running a subset of output data transfer
> workflows of the parent process. A sample client code might look like below.
> List<String> outputNames = new ArrayList<>();
> outputNames.add("Gaussian-Application-Output");
> apiClient.fetchIntermediateOutputs(token,"exp-id",
> outputNames,
> "PROCESS_98a8c45c-998c-4998-91e4-18ce915ed13e");
> Any failure to this new process will not affect the main experiment but users
> can see those errors in the experiment statistics page under the new process
> section. The attached screenshot shows how the new process might look like on
> the statistics page. Output data will be placed in a separate directory named
> 'intermediate' on the storage side.
>
> I merged this code to dev environment and it is ready for testing purposes.
> This PR [2] will reflect all the file changes from API to Helix level.
>
> Marcus,
> We might need to add new buttons / UI components to make this available at
> the portal level and we can discuss it here if that makes sense.
>
> Thanks and happy new year
> Dimuthu
>
> [1]
> https://github.com/apache/airavata/blob/AIRAVATA-3549/thrift-interface-descriptions/airavata-apis/airavata_api.thrift#L989
>
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairavata%2Fblob%2FAIRAVATA-3549%2Fthrift-interface-descriptions%2Fairavata-apis%2Fairavata_api.thrift%23L989&data=04%7C01%7Cmachrist%40iu.edu%7Cbc8967c09c0a4cddd96d08d9ce6e1308%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C637767794427967806%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RvmFv1e0ICkzprnvSioC4QXfihaytF2HMp9ukisFcH8%3D&reserved=0>
> [2] https://github.com/apache/airavata/pull/276
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairavata%2Fpull%2F276&data=04%7C01%7Cmachrist%40iu.edu%7Cbc8967c09c0a4cddd96d08d9ce6e1308%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C637767794427967806%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=JF6vrvwfWR31rk0avI9hlYOEOO66QyiZN4j40d28zog%3D&reserved=0>
>
> <Screen Shot 2022-01-02 at 10.20.47 PM.png>
smime.p7s
Description: S/MIME cryptographic signature
