In enterprise gadget development, there are requirements to allow files upload/download up to 512MB. Shindig server is processing the fetched response from target url every time, convert the stream fetched into byte array and store the byte array in HttpResponse, later on the HttpResponse can be copied to a ServletResponse and return to client. This solution has severe problem when the stream is huge: it could easily cause the JVM to exhaust its heap under moderate to heavy load.
Dan's patch about proxied-form-post would solve the upload. We'd like to propose a download solution: streamed request. The idea is to add a new field in gadgets.io.RequestParameters.STREAMED, and update gadgets.io.getProxyUrl() to consider the case that this is a streamed request, and return a proxied URL like this: http://localhost:8080/gadgets/proxy?container=default&refresh=0&url=http%3A%2F%2Flocalhost%3A8080%2Ftest%2Ftest.rar&; streamed=true Then Shindig server side ProxyHandler would be able to tell whether the response stream need to be processed in Shindig server or not. If it is a streamed request, Shindig server won't process it, just set the stream into HttpResponse and return it to ServletReponse. All the other processes such as response headers, security, caching etc will be kept the same as they were. The changes need to be made in Shindig server are Proxy URL handling, HttpResponse, HttpResponseBuilder, ProxyHandler, ProxyServlet, DefaultRequestPipeline and BasicHttpFetcher. Could you guys provide comment on this solution and see if we miss anything? Thanks Best Regards Marshall Shi(Shi Wei) Mark Weitzel <weitzelm.work@gm ail.com> To dev@shindig.apache.org 2012-02-26 21:14 cc Subject Please respond to Re: Authenticated, binary downloads dev@shindig.apach in gadgets e.org Dan, We've been talking about aligning with cmis so the use of files may be misleading even if we add it as osapi On Feb 22, 2012 9:33 AM, "Dan Dumont" <ddum...@us.ibm.com> wrote: > Similar to the proxied-form-post feature that I recently added to shindig, > there is a need to make an authenticated file download in a gadget. > Authentication for a file download is likely needed and since the shindig > server handles that for the gadget, requests directly to the remote > service would be rejected (without some monkeying around with one-time-use > tokens). > > It would seem that the content proxy (getProxyUrl) is inappropriate here, > as it doesn't look like it can or should accept headers and authentication > params. > makeRequest and osapi.http.* apis also seem inappropriate for this use > case, as they contain a bunch of extra information in the response that > isn't wanted (we just need the binary data to let the browser save the > file). > > So it looks like the only option is a new feature that would allow for > streamed file downloads through an OS proxy. Hopefully, like the > proxied-form-post feature, we can reuse most of the existing makeRequest > server logic to do this for us. I'm wondering if we create a > gadgets.files.* namespace to house these new functions, or if we should > keep them in gadgets.io? > > Any thoughts on all of this?
