Hi Vivek, Yes, since the Django Portal is the user's data store, it should enforce the policy. If I think about the Airavata API in other contexts, I'm not sure it makes sense for the API client to inform the API server of how much space is being used. Also, this mechanism is insufficient because it doesn't take into account output files that get added to the data store when the experiment is executed.
I think it would be fine if the API provided information about the quota limits. Then the Django portal can query those limits and apply them. > On Aug 3, 2020, at 6:32 PM, Bandaru, Vivek Shresta <vivb...@iu.edu> wrote: > > Hi Marcus, > > Thanks for the reply. > > I think it should be up to the data store to calculate the amount of storage > space use and apply the quota. Currently, that's the Django portal. However, > in the future it will likely be MFT. -> Do you mean to say that the > validation should be done in the gateway rather than Airavata? In my current > implementation, Django-Portal calculates the amount of Data used by a User > and sends this to Airavata when an experiment is being created(through a new > Airavata API) where the validation happens. Once the user reaches the storage > limit (validated against the new entry, UserStorageQuota in > StoragePreferences), Airavata throws an exception resulting in the > cancellation of the new experiment. Is this approach fine? > > Regards, > Vivek. > > From: "Christie, Marcus Aaron" <machr...@iu.edu> > Reply-To: <dev@airavata.apache.org> > Date: Monday, August 3, 2020 at 1:43 PM > To: Airavata Dev <dev@airavata.apache.org> > Cc: "Wannipurage, Dimuthu Upeksha" <dwann...@iu.edu> > Subject: Re: Validating user storage quota > > Hi Vivek, > > > On Jul 30, 2020, at 9:31 PM, Bandaru, Vivek Shresta <vivb...@iu.edu > <mailto:vivb...@iu.edu>> wrote: > > Hi All, > > I’ve been working on various approaches to validate the storage quota for a > gateway user. Though the storage for experiments is taken care by the > gateway, the storage quota validation needs to be done on Airavata. This way, > the gateways need not develop their own mechanism to track the quotas for > every user. > > The only way as far as I know for a user to add his files to the storage are > through Storage page and the Create a New experiment page where they can add > input files. Utilizing the existing > > Also, Airavata deposits output files into the experimentDataDir and these > will count against the user's quota. > > > datastore.size(‘directory_path’) api in the Django-portal, the amount of > space used by the user can be tracked, and when the Gateway sends this data > to Airavata through a new API, Airavata validates the space utilized with the > Storage Quota specified in the Storage Preference. > Currently, whenever a user tries to create a new Experiment, even before the > input files are added to the Experiment’s directory, I’ve added the > validation and when the storage limit exceeds, this is how the new experiment > page is rendered: > > <image001.png> > > Questions: > > Since UserStoragePreference feature isn’t being used, I’m not really sure if > I need to use the StorageResource mentioned in UserStoragePreference. Once > the gateways start using this (not sure if it’s only Django-Portal which > isn’t using UserStoragePreference), then the necessary code can be added. > I would like to know the teams thoughts on this. > > I wouldn't worry about UserStoragePreference. That is somewhat deprecated, to > be replaced with a personal GroupResourceProfile. > > > > I feel that maybe a better way to approach this problem would be if Airavata > tracks the size of each and every file going into the user’s directory > instead of the gateway telling the size of the user’s directory. The amount > of space used by an individual user can be tracked in the USERS table through > a new column. This is where I’m currenty stuck. > Airavata currently uses SCPFileTransferWrapper.java for uploading the input > files onto the compute resources. There is no means of knowing the file size > being transferred. > Using the current implementation, one possible approach that I could think of > is to download the file from the gateway to a temporary directory where > Airavata is deployed, and through that URI, the file size can be retrieved. > But this would involve downloading every input file on all the gateways of > all the users onto a temporary location on Airavata and deleting them. > I’m currently looking for a better alternative to track the file size given a > URI. > > I think it should be up to the data store to calculate the amount of storage > space use and apply the quota. Currently, that's the Django portal. However, > in the future it will likely be MFT. > > > > Once MFT is integrated into Airavata, Airavata will not use the existing file > transfer protocols. So, does it make sense to develop this tracking mechanism > on MFT and for now, use the above mentioned validation mechanism(Gateway gets > the size of the user directory)? > > I think definitely understanding how MFT factors in to the future plans will > be good. Maybe Dimuthu can chime in here. > > > > > Any pointers are appreciated. > Thanks for reading. > > Regards, > Vivek.
smime.p7s
Description: S/MIME cryptographic signature
