Are we using any parts of the storage impl we did for ES Regards, /Nuwan
On Mon, Mar 3, 2014 at 12:35 PM, Chan <[email protected]> wrote: > Hi folks, > I have finished the basic functionality of the Storage app[1]. Below are > the next set of functions I'll be working on. > > - Support h2 db (support h2 database) > - Support content-type storage (Store the content type of the file for > better rendering) > - Make engines and storage-providers different (engine stores UUIDs, > tokens etc where as storage-provider is the actual storage) > - Support consumer tokens and validation for consumers > - Route with given name - requirement that came from ES team > (storage/resource/:uuid/:filename) > > Another question I have is whether we have a component that deals with > token generation. I want to generate tokens for file requests. > > [1] - https://github.com/dulichan/storage > > Cheers~ > > > On Fri, Feb 28, 2014 at 9:03 PM, Shani Ranasinghe <[email protected]> wrote: > >> Hi Chan, >> >> In SS we have the HDFS browser facility. This is a feature where you can >> upload/download files to the HDFS. Please refer >> https://docs.wso2.org/display/SS110/HDFS+Browser. Please check if this >> fits your requirement. >> >> Also, as far as UES is concerned, implementing the HDFS UI as a jaggery >> app is a roadmap item https://redmine.wso2.com/issues/1744. >> >> Mutitenancy is yet to be completed and in the current release is not >> production recommended. With the introduction of Hadoop 2.x in SS (in a >> future release) we are planning to look into possibilities of federating >> the file system and thereby address multitenancy. It is yet to be decided. >> >> >> >> On Fri, Feb 28, 2014 at 6:33 AM, Chan <[email protected]> wrote: >> >>> *Simple spec for the Unified Storage proposal * >>> The goal of the Unified Storage proposal is to provide a REST API and >>> jaggery module wrapper which will provide below features - >>> >>> - Persist files >>> - Get files >>> >>> >>> *REST API* >>> * Persist file* >>> >>> URL - storage/resource/ >>> VERB - PUT >>> >>> Headers >>> >>> consumer token - Get the token of the provider >>> >>> consumer user - User that's authenticated to the service provider >>> >>> Body >>> >>> File data >>> >>> Response >>> >>> Returns id >>> >>> >>> *Get file* >>> >>> URL - storage/ >>> resource >>> /:id >>> >>> VERB - GET >>> Attributes >>> >>> id - The ID obtained from the PUT call >>> >>> Headers >>> >>> consumer token - Get the token of the provider >>> >>> consumer user - User that's authenticated to the service provider >>> >>> Response >>> >>> Returns the file >>> >>> *Jaggery Module* >>> A jaggery module will be implemented which wrap the REST API and provide >>> a developer friendly javascript api. >>> >>> var storage = require("storage").storage; >>> /* >>> To be configured with >>> { >>> provider-name: sdf >>> provider-secret: sdf >>> } >>> */ >>> storage.configure({}); >>> var file = new File(); >>> >>> /* >>> persist a file >>> meta will contain >>> { >>> id = id to access the file >>> } >>> */ >>> var meta = storage.put(file); >>> >>> /* >>> request a file by sending the file id >>> */ >>> var file = storage.get(id); >>> >>> >>> We can improve on the spec by providing security, multi-tenancy >>> *Buckets* >>> A bucket is a namespace for files or objects. This will be useful for a >>> use-case where namespaces for files are required. >>> >>> *Multi-tenancy* >>> There are some limitations in multi-tenancy in HDFS (@Deep or @Shani can >>> better explain this). We can implement the tenant isolation in the storage >>> app layer. >>> >>> *Security* >>> A token can be requested from storage for a resource before issuing the >>> url to a client. A use-case is where store takes a token and append it to >>> the url before sending it to the android device. There will be a governance >>> model on the time period and invalidation to secure the executable >>> resource. Also another layer of security will be implemented so that >>> Storage app can trust the Service-Provider (which is Publisher or Store). >>> >>> Any ideas are welcome. >>> >>> Cheers~ >>> >>> >>> On Mon, Feb 24, 2014 at 1:23 PM, Chan <[email protected]> wrote: >>> >>>> @Dilshan as I have mentioned - SS only handles provisioning of storage. >>>> The proposed solution is for File Storage (not for provisioning). >>>> >>>> >>>> On Mon, Feb 24, 2014 at 12:51 PM, Dilshan Edirisuriya <[email protected] >>>> > wrote: >>>> >>>>> Hi Chan, >>>>> >>>>> I think we discussed this few months ago with SS team (Prabath, Deep). >>>>> What was the final conclusion? >>>>> >>>>> Regards, >>>>> >>>>> Dilshan >>>>> >>>>> >>>>> On Thu, Feb 20, 2014 at 7:49 PM, Chan <[email protected]> wrote: >>>>> >>>>>> Hi folks, >>>>>> In our platform - we don't yet have a solution for storing static >>>>>> files in the cloud. A solution that provides a simple service of >>>>>> persisting >>>>>> a file and getting back a URI. The Enterprise Store team built a storage >>>>>> module in the Publisher app that provides persisting of files to a >>>>>> database >>>>>> and getting back URIs. My proposal is to build a jaggery-app that >>>>>> specifically handles file storage with a REST interface extending what >>>>>> the >>>>>> ES team has done. >>>>>> >>>>>> *Use case* >>>>>> I will be using the Enterprise Store as an example for the Storage >>>>>> App. >>>>>> [image: Inline image 1] >>>>>> 1:- Developer uploads a file to the publisher app using an entry form >>>>>> 2:- Publisher app calls the storage module to contact the storage >>>>>> provider >>>>>> 3:- Storage module contacts the Storage app with a HTTP PUT call and >>>>>> returns the URI of the file persisted. Publisher app will persist this >>>>>> URI >>>>>> to the registry along with other information. >>>>>> 4:- A client contacts the server requesting an asset page. >>>>>> 5:- The server will contact the storage module to request a token. It >>>>>> will in turn call the Storage app requesting a token. After retrieving >>>>>> the >>>>>> token It will append the token to the URI when rendering the pages >>>>>> server >>>>>> side. >>>>>> 6:- Browser will contact the Storage app directly when requesting the >>>>>> file using HTTP GET. Since the browser is passing a valid token - Storage >>>>>> app will return the file in the response. >>>>>> >>>>>> *Architecture overview* >>>>>> Storage app will have service providers. These service providers will >>>>>> be apps that require the Storage service from the Storage app. First >>>>>> let's >>>>>> start off with the API. Storage app will provide an HTTP interface as >>>>>> below >>>>>> GET >>>>>> >>>>>> GET interface is used to request a file. The request can be >>>>>> for permanent links or token based links. >>>>>> >>>>>> PUT >>>>>> >>>>>> PUT can only called by a registered service provider. A token >>>>>> (Provider token) has to be passed to validate the service provider. It >>>>>> will >>>>>> also allow options of - security, permanent links etc. >>>>>> >>>>>> DELETE >>>>>> >>>>>> URI is passed to the DELETE API as well as the provider token. Used >>>>>> to delete files. >>>>>> >>>>>> >>>>>> Next is what the real storage options are. For the first cut we can >>>>>> have the storage app connected to a database. Afterwards we can move to >>>>>> HDFS type storage or Cassandra perhaps. >>>>>> >>>>>> As with Multi-tenancy - we can handle multi-tenancy the same way we >>>>>> handle it for jaggery apps (SaaS multi-tenancy). >>>>>> >>>>>> Ultimate objective of this proposal is to build a cloud storage >>>>>> service like Amazon S3, Google Cloud storage [1]. WDYT guys? >>>>>> >>>>>> [1] - >>>>>> http://www.quora.com/Amazon-S3/What-are-the-best-alternatives-to-S3-and-what-are-the-pros-and-cons >>>>>> >>>>>> Cheers~ >>>>>> -- >>>>>> Chan (Dulitha Wijewantha) >>>>>> Software Engineer - Mobile Development >>>>>> WSO2Mobile >>>>>> Lean.Enterprise.Mobileware >>>>>> * ~Email [email protected] <[email protected]>* >>>>>> * ~Mobile +94712112165 <%2B94712112165>* >>>>>> >>>>>> * ~Website dulithawijewantha.com <http://dulithawijewantha.com/> * >>>>>> >>>>>> * ~Blog blog.dulithawijewantha.com >>>>>> <http://dulichan.github.io/chan/>* >>>>>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Dilshan Edirisuriya >>>>> Senior Software Engineer - WSO2Mobile >>>>> Mob: + 94 772245502 >>>>> http://wso2mobile.com/ >>>>> >>>> >>>> >>>> >>>> -- >>>> Chan (Dulitha Wijewantha) >>>> Software Engineer - Mobile Development >>>> WSO2Mobile >>>> Lean.Enterprise.Mobileware >>>> * ~Email [email protected] <[email protected]>* >>>> * ~Mobile +94712112165 <%2B94712112165>* >>>> >>>> * ~Website dulithawijewantha.com <http://dulithawijewantha.com/> * >>>> >>>> * ~Blog blog.dulithawijewantha.com >>>> <http://dulichan.github.io/chan/>* >>>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>>> >>> >>> >>> >>> -- >>> Chan (Dulitha Wijewantha) >>> Software Engineer - Mobile Development >>> WSO2Mobile >>> Lean.Enterprise.Mobileware >>> * ~Email [email protected] <[email protected]>* >>> * ~Mobile +94712112165 <%2B94712112165>* >>> >>> * ~Website dulithawijewantha.com <http://dulithawijewantha.com/> * >>> >>> * ~Blog blog.dulithawijewantha.com >>> <http://dulichan.github.io/chan/>* >>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>> >> >> >> >> -- >> Thanks and Regards >> *, Shani Ranasinghe* >> Software Engineer >> WSO2 Inc.; http://wso2.com >> lean.enterprise.middleware >> >> mobile: +94 77 2273555 >> linked in: lk.linkedin.com/pub/shani-ranasinghe/34/111/ab >> > > > > -- > Chan (Dulitha Wijewantha) > Software Engineer - Mobile Development > WSO2Mobile > Lean.Enterprise.Mobileware > * ~Email [email protected] <[email protected]>* > * ~Mobile +94712112165 <%2B94712112165>* > * ~Website dulitha.me <http://dulitha.me>* > * ~Twitter @dulitharw <https://twitter.com/dulitharw>* > *~SO @chan <http://stackoverflow.com/users/813471/chan>* > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Thanks & Regards,Nuwan BandaraTechnical Lead; **WSO2 Inc. * *lean . enterprise . middleware | http://wso2.com <http://wso2.com> * *blog : http://nuwanbando.com <http://nuwanbando.com>; email: [email protected] <[email protected]>; phone: +1 812 606 7390 * <http://www.nuwanbando.com/>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
