Re: Binary, non-base 64 upload
Good point Kambiz - with the likes of S3 object storage this is becoming a
common requirement - especially for very large files where loading the whole
file into memory is not appropriate. And much better than storing in a rdbms
blob column.
I suggest a hidden mandatory attribute on a domain object (actually to hold the
stored object ID once it has uploaded) to prevent the form from being submitted
and javascript (delivered via a custom wicket component) using Fine Uploader
Javascript Upload Library since it is Ajax your form is not submitted until
your javascript allows it by storing the uploaded object ID in the hidden
mandatory field. Also the user sees a fine progress bar whilst uploading and
this library supports chunking and direct to S3 and Azure or to traditional
server side (servlet).
What do you think?
Regards,David.
| |
| | | | | | | |
| Fine Uploader Javascript Upload LibraryFine Uploader. A dependency-free,
open-source, native browser upload tool. |
| |
| View on fineuploader.com | Preview by Yahoo |
| |
| |
On Saturday, 6 August 2016 7:31 AM, Martin Grigorov
wrote:
Hi Kambiz,
I guess this code works only because you don't need another request
parameter in the same action.
Latest versions of Isis recommend to have an action per request parameter,
so maybe this is not an issue.
But imagine submitting a form with one or more text fields and one file
upload field. To be able to pass the values of the input fields Isis will
need to call request.getParameter(someName) and this will consume the
request body and later your code won't see anything in the inputstream.
Martin Grigorov
Wicket Training and Consulting
https://twitter.com/mtgrigorov
On Fri, Aug 5, 2016 at 4:27 PM, Kambiz Darabi
wrote:
> Hello Martin,
>
> On 2016-08-05 13:42 CEST, Martin Grigorov wrote:
>
> > [...]
> > The problem here is that you have to make sure that your code is the very
> > first one that reads from the ServletInputStream. Otherwise the body will
> > be already consumed.
> > You may need to override some Wicket/RESTeasy code to be able to do this
> > for the default viewers.
> > And this would be much harder than custom Servlet Filter in front of the
> > ones by Wicket/RESTeasy
>
> If you look at the change in my original post:
>
> https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73
> f16857a628
>
> you can see that I added this functionality to the
> restfulobjects-viewer, so that I can be sure that the full stream is
> handed to the action without any change.
>
> That code is tested against the Isis archetype with this addition to the
> SimpleObjectMenu service:
>
> @Action(semantics = SemanticsOf.SAFE)
> public void upload(HttpServletRequest request, HttpServletResponse
> response) {
> FileOutputStream out = null;
> try {
> File outFile = new File("/tmp/request");
> out = new FileOutputStream(outFile, false);
> IOUtils.copy(request.getInputStream(), out);
> out.close();
> } catch (IOException e) {
> if(out != null) {
> try {
> out.close();
> } catch (IOException e1) {
> // LOG something
> }
> }
> // throw something
> }
> }
>
>
> and the file in /tmp/request is identical to the posted binary file which
> is sent by
> curl:
>
> curl -X POST --data-binary @big-binary-file
> --header "Authorization: Basic c3ZlbjpwYXNz" \
> --header "Accept: application/json;profile=urn:org.apache.isis/v1" \
> --header "Content-Type: application/octet-stream" \
> http://localhost:8080/restful/services/SimpleObjectMenu/
> actions/upload/invoke
>
>
> Cheers
>
>
> Kambiz
>
> >
> >> action with the metadata of the files/request, but it would help a lot
> >> to have such a facility inside Isis instead of writing a separate
> >> servlet and manually integrating its deployment and the calling of
> >> domain actions etc.
> >>
> >> Cheers
> >>
> >>
> >> Kambiz
> >>
> >>
> >> On 2016-08-03 18:53 CEST, Willie Loyd Tandingan <
> tandingan@gmail.com>
> >> wrote:
> >>
> >> > Hi Kambiz,
> >> >
> >> > We had a requirement similar to yours. The problem with injecting
> >> > HttpServletRequest to domain services is that it kind of violates the
> >> DDD's
> >> > hexagonal architecture by letting viewer implementation leak into the
> >> > domain layer.
> >> >
> >> > The specific requirement that we had was to be able to pre-upload
> files
> >> > before submitting an action. We developed a custom blob that stores
> an id
> >> > instead of the actual data, a service layer that manages a blob
> storage
> >> > abstracting and running on top of jclouds (we plan to be able to
> support
> >> > other types of storage aside from file system), and a file service
> that
> >> > manages the state of "transient files" (files that were pre-uploaded
> or
> >> > that are returned from actions as result of export functions, etc.) as
> >> well
> >> > as means to persist t
Return Fields Related as Null
Hi Guys
Problem Description: When I creating the method "listarSolicituesTramite"
in Apache Isis and the method is invoked only reach the fields of
application are flat on the table, but returns the fields come
from relationship as null.
tbl_Solicitud
tbl_Motivo Tramite
tbl_Tramite
Nombre (Flat)
ID_Motivo_Tramite (Relation Field)
ID_Tramite (Relation Field)
In this example, the field "Nombre" return a value, but "ID_Motivo_Tramite"
and "ID Tramite" arrive in null, which does not allow the filter.
Here the Method
@MemberOrder(sequence="10.3")
@Action(semantics = SemanticsOf.*SAFE*)
@ActionLayout(bookmarking = BookmarkPolicy.*AS_ROOT*)
*public* List ListarSolicitudesTramite(*final*
@ParameterLayout(named="Tramite") String tramite) {
List listaSolicitudes= repositoryService
.allInstances(Solicitud.*class*);
String [] arrayTramites= tramite.split("//|");
ArrayList listaTramites= *new* ArrayList();
*for* (*int* i = 0; i < arrayTramites.length; i++) {
listaTramites.add(arrayTramites[i]);
}
*for* (Iterator iterator = listaSolicitudes.iterator();
iterator.hasNext();) {
Solicitud solicitud = iterator.next();
*if*
(!listaTramites.contains(solicitud.getMotivoTramite().getTramite()))
{
iterator.remove();
}
}
*return* listaSolicitudes;
}
--
[image: Logo]
José Alejandro Manaure
*Project Manager*
Tel: (57) 1 703 17 77
Cel: (57) 312 5476188
E-mail: jose.mana...@ticxar.com
Calle 93 # 19b - 66 Ofc 202
Bogotá D.C., Colombia
www.ticxar.com
[image: facebook]
[image:
twitter] [image: linkedIn]
Re: Binary, non-base 64 upload
Hi Kambiz,
I guess this code works only because you don't need another request
parameter in the same action.
Latest versions of Isis recommend to have an action per request parameter,
so maybe this is not an issue.
But imagine submitting a form with one or more text fields and one file
upload field. To be able to pass the values of the input fields Isis will
need to call request.getParameter(someName) and this will consume the
request body and later your code won't see anything in the inputstream.
Martin Grigorov
Wicket Training and Consulting
https://twitter.com/mtgrigorov
On Fri, Aug 5, 2016 at 4:27 PM, Kambiz Darabi
wrote:
> Hello Martin,
>
> On 2016-08-05 13:42 CEST, Martin Grigorov wrote:
>
> > [...]
> > The problem here is that you have to make sure that your code is the very
> > first one that reads from the ServletInputStream. Otherwise the body will
> > be already consumed.
> > You may need to override some Wicket/RESTeasy code to be able to do this
> > for the default viewers.
> > And this would be much harder than custom Servlet Filter in front of the
> > ones by Wicket/RESTeasy
>
> If you look at the change in my original post:
>
> https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73
> f16857a628
>
> you can see that I added this functionality to the
> restfulobjects-viewer, so that I can be sure that the full stream is
> handed to the action without any change.
>
> That code is tested against the Isis archetype with this addition to the
> SimpleObjectMenu service:
>
> @Action(semantics = SemanticsOf.SAFE)
> public void upload(HttpServletRequest request, HttpServletResponse
> response) {
> FileOutputStream out = null;
> try {
> File outFile = new File("/tmp/request");
> out = new FileOutputStream(outFile, false);
> IOUtils.copy(request.getInputStream(), out);
> out.close();
> } catch (IOException e) {
> if(out != null) {
> try {
> out.close();
> } catch (IOException e1) {
> // LOG something
> }
> }
> // throw something
> }
> }
>
>
> and the file in /tmp/request is identical to the posted binary file which
> is sent by
> curl:
>
> curl -X POST --data-binary @big-binary-file
> --header "Authorization: Basic c3ZlbjpwYXNz" \
> --header "Accept: application/json;profile=urn:org.apache.isis/v1" \
> --header "Content-Type: application/octet-stream" \
> http://localhost:8080/restful/services/SimpleObjectMenu/
> actions/upload/invoke
>
>
> Cheers
>
>
> Kambiz
>
> >
> >> action with the metadata of the files/request, but it would help a lot
> >> to have such a facility inside Isis instead of writing a separate
> >> servlet and manually integrating its deployment and the calling of
> >> domain actions etc.
> >>
> >> Cheers
> >>
> >>
> >> Kambiz
> >>
> >>
> >> On 2016-08-03 18:53 CEST, Willie Loyd Tandingan <
> tandingan@gmail.com>
> >> wrote:
> >>
> >> > Hi Kambiz,
> >> >
> >> > We had a requirement similar to yours. The problem with injecting
> >> > HttpServletRequest to domain services is that it kind of violates the
> >> DDD's
> >> > hexagonal architecture by letting viewer implementation leak into the
> >> > domain layer.
> >> >
> >> > The specific requirement that we had was to be able to pre-upload
> files
> >> > before submitting an action. We developed a custom blob that stores
> an id
> >> > instead of the actual data, a service layer that manages a blob
> storage
> >> > abstracting and running on top of jclouds (we plan to be able to
> support
> >> > other types of storage aside from file system), and a file service
> that
> >> > manages the state of "transient files" (files that were pre-uploaded
> or
> >> > that are returned from actions as result of export functions, etc.) as
> >> well
> >> > as means to persist them (i.e. move them to another
> >> > container/bucket/folder).
> >> >
> >> > We then developed a custom servlet that handles download and upload
> and
> >> > integrate them with isis. This enabled us to implement features like
> >> forced
> >> > attachment content disposition, or on-demand image resizing.
> >> >
> >> > Should you have other ideas or questions, feel free to ask.
> >> >
> >> >
> >> > Best regards,
> >> > Willie
> >> > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi
> >> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I have to implement a legacy file upload service for a customer.
> Files
> >> >> of up to some hundred megabytes are uploaded to a service over HTTP
> POST
> >> >> and a unique ID is returned which can be used to refer to that file,
> add
> >> >> metadata to it etc.
> >> >>
> >> >> Using Blobs is not viable as parsing of the base64 encoded binary
> data
> >> >> and the allocation of multiple strings during request processing
> >> >> increases both the time and space requirements.
> >> >>
> >> >> Ideally, I would be able to create an action on one of my domain
> >> >> services which
Re: Binary, non-base 64 upload
Hello Martin,
On 2016-08-05 13:42 CEST, Martin Grigorov wrote:
> [...]
> The problem here is that you have to make sure that your code is the very
> first one that reads from the ServletInputStream. Otherwise the body will
> be already consumed.
> You may need to override some Wicket/RESTeasy code to be able to do this
> for the default viewers.
> And this would be much harder than custom Servlet Filter in front of the
> ones by Wicket/RESTeasy
If you look at the change in my original post:
https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73f16857a628
you can see that I added this functionality to the
restfulobjects-viewer, so that I can be sure that the full stream is
handed to the action without any change.
That code is tested against the Isis archetype with this addition to the
SimpleObjectMenu service:
@Action(semantics = SemanticsOf.SAFE)
public void upload(HttpServletRequest request, HttpServletResponse
response) {
FileOutputStream out = null;
try {
File outFile = new File("/tmp/request");
out = new FileOutputStream(outFile, false);
IOUtils.copy(request.getInputStream(), out);
out.close();
} catch (IOException e) {
if(out != null) {
try {
out.close();
} catch (IOException e1) {
// LOG something
}
}
// throw something
}
}
and the file in /tmp/request is identical to the posted binary file which is
sent by
curl:
curl -X POST --data-binary @big-binary-file
--header "Authorization: Basic c3ZlbjpwYXNz" \
--header "Accept: application/json;profile=urn:org.apache.isis/v1" \
--header "Content-Type: application/octet-stream" \
http://localhost:8080/restful/services/SimpleObjectMenu/actions/upload/invoke
Cheers
Kambiz
>
>> action with the metadata of the files/request, but it would help a lot
>> to have such a facility inside Isis instead of writing a separate
>> servlet and manually integrating its deployment and the calling of
>> domain actions etc.
>>
>> Cheers
>>
>>
>> Kambiz
>>
>>
>> On 2016-08-03 18:53 CEST, Willie Loyd Tandingan
>> wrote:
>>
>> > Hi Kambiz,
>> >
>> > We had a requirement similar to yours. The problem with injecting
>> > HttpServletRequest to domain services is that it kind of violates the
>> DDD's
>> > hexagonal architecture by letting viewer implementation leak into the
>> > domain layer.
>> >
>> > The specific requirement that we had was to be able to pre-upload files
>> > before submitting an action. We developed a custom blob that stores an id
>> > instead of the actual data, a service layer that manages a blob storage
>> > abstracting and running on top of jclouds (we plan to be able to support
>> > other types of storage aside from file system), and a file service that
>> > manages the state of "transient files" (files that were pre-uploaded or
>> > that are returned from actions as result of export functions, etc.) as
>> well
>> > as means to persist them (i.e. move them to another
>> > container/bucket/folder).
>> >
>> > We then developed a custom servlet that handles download and upload and
>> > integrate them with isis. This enabled us to implement features like
>> forced
>> > attachment content disposition, or on-demand image resizing.
>> >
>> > Should you have other ideas or questions, feel free to ask.
>> >
>> >
>> > Best regards,
>> > Willie
>> > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> I have to implement a legacy file upload service for a customer. Files
>> >> of up to some hundred megabytes are uploaded to a service over HTTP POST
>> >> and a unique ID is returned which can be used to refer to that file, add
>> >> metadata to it etc.
>> >>
>> >> Using Blobs is not viable as parsing of the base64 encoded binary data
>> >> and the allocation of multiple strings during request processing
>> >> increases both the time and space requirements.
>> >>
>> >> Ideally, I would be able to create an action on one of my domain
>> >> services which has a HttpServletRequest argument and take care of
>> >> reading the binary data and writing them to the file system.
>> >>
>> >> I have played around with the request processing part of the
>> >> restfulobjects viewer/server and managed to get the request/response
>> >> objects through.
>> >>
>> >> I know that this is a real edge case, but I think that it is important
>> >> to have solutions for edge cases, too, if Isis is to be used as a
>> >> general purpose platform.
>> >>
>> >> Could some committers please review the changes and maybe propose a
>> >> cleaner way of implementing this?
>> >>
>> >>
>> >> https://github.com/m-creations/isis/commit/
>> aa3b16a5cf463466f5abadbcc8cc73f16857a628
>> >>
>> >> Thanks
>> >>
>> >>
>> >> Kambiz
>> >>
>> >>
>>
Re: Binary, non-base 64 upload
Hi Kambiz, On Fri, Aug 5, 2016 at 12:55 PM, Kambiz Darabi wrote: > Hi, > > yes, I fully agree that it violates the DDD idea, but on the other hand, > I don't see a point in implementing part of the system as a separate > servlet which doesn't have access to the underlying Isis > facilities. > > We need access to other services to implement business logic > e.g. depending on the size of the upload or on the file signature (magic > numbers). > > In most cases, we just have to read the InputStream of the servlet and > write it out to a file, which is much more memory-efficient than reading > the whole file as a base64 string into memory, decoding it, and then > calling an action with the Blob. > > I'm not opposed to programming some facility (an Isis module?) which > streams the content of the request to a file - or multiple files in case > of a multipart request - and _then_ calls an appropriately annotated > The problem here is that you have to make sure that your code is the very first one that reads from the ServletInputStream. Otherwise the body will be already consumed. You may need to override some Wicket/RESTeasy code to be able to do this for the default viewers. And this would be much harder than custom Servlet Filter in front of the ones by Wicket/RESTeasy > action with the metadata of the files/request, but it would help a lot > to have such a facility inside Isis instead of writing a separate > servlet and manually integrating its deployment and the calling of > domain actions etc. > > Cheers > > > Kambiz > > > On 2016-08-03 18:53 CEST, Willie Loyd Tandingan > wrote: > > > Hi Kambiz, > > > > We had a requirement similar to yours. The problem with injecting > > HttpServletRequest to domain services is that it kind of violates the > DDD's > > hexagonal architecture by letting viewer implementation leak into the > > domain layer. > > > > The specific requirement that we had was to be able to pre-upload files > > before submitting an action. We developed a custom blob that stores an id > > instead of the actual data, a service layer that manages a blob storage > > abstracting and running on top of jclouds (we plan to be able to support > > other types of storage aside from file system), and a file service that > > manages the state of "transient files" (files that were pre-uploaded or > > that are returned from actions as result of export functions, etc.) as > well > > as means to persist them (i.e. move them to another > > container/bucket/folder). > > > > We then developed a custom servlet that handles download and upload and > > integrate them with isis. This enabled us to implement features like > forced > > attachment content disposition, or on-demand image resizing. > > > > Should you have other ideas or questions, feel free to ask. > > > > > > Best regards, > > Willie > > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi > wrote: > > > >> Hi, > >> > >> I have to implement a legacy file upload service for a customer. Files > >> of up to some hundred megabytes are uploaded to a service over HTTP POST > >> and a unique ID is returned which can be used to refer to that file, add > >> metadata to it etc. > >> > >> Using Blobs is not viable as parsing of the base64 encoded binary data > >> and the allocation of multiple strings during request processing > >> increases both the time and space requirements. > >> > >> Ideally, I would be able to create an action on one of my domain > >> services which has a HttpServletRequest argument and take care of > >> reading the binary data and writing them to the file system. > >> > >> I have played around with the request processing part of the > >> restfulobjects viewer/server and managed to get the request/response > >> objects through. > >> > >> I know that this is a real edge case, but I think that it is important > >> to have solutions for edge cases, too, if Isis is to be used as a > >> general purpose platform. > >> > >> Could some committers please review the changes and maybe propose a > >> cleaner way of implementing this? > >> > >> > >> https://github.com/m-creations/isis/commit/ > aa3b16a5cf463466f5abadbcc8cc73f16857a628 > >> > >> Thanks > >> > >> > >> Kambiz > >> > >> >
Re: Binary, non-base 64 upload
Hi, yes, I fully agree that it violates the DDD idea, but on the other hand, I don't see a point in implementing part of the system as a separate servlet which doesn't have access to the underlying Isis facilities. We need access to other services to implement business logic e.g. depending on the size of the upload or on the file signature (magic numbers). In most cases, we just have to read the InputStream of the servlet and write it out to a file, which is much more memory-efficient than reading the whole file as a base64 string into memory, decoding it, and then calling an action with the Blob. I'm not opposed to programming some facility (an Isis module?) which streams the content of the request to a file - or multiple files in case of a multipart request - and _then_ calls an appropriately annotated action with the metadata of the files/request, but it would help a lot to have such a facility inside Isis instead of writing a separate servlet and manually integrating its deployment and the calling of domain actions etc. Cheers Kambiz On 2016-08-03 18:53 CEST, Willie Loyd Tandingan wrote: > Hi Kambiz, > > We had a requirement similar to yours. The problem with injecting > HttpServletRequest to domain services is that it kind of violates the DDD's > hexagonal architecture by letting viewer implementation leak into the > domain layer. > > The specific requirement that we had was to be able to pre-upload files > before submitting an action. We developed a custom blob that stores an id > instead of the actual data, a service layer that manages a blob storage > abstracting and running on top of jclouds (we plan to be able to support > other types of storage aside from file system), and a file service that > manages the state of "transient files" (files that were pre-uploaded or > that are returned from actions as result of export functions, etc.) as well > as means to persist them (i.e. move them to another > container/bucket/folder). > > We then developed a custom servlet that handles download and upload and > integrate them with isis. This enabled us to implement features like forced > attachment content disposition, or on-demand image resizing. > > Should you have other ideas or questions, feel free to ask. > > > Best regards, > Willie > On Thu, 4 Aug 2016 at 12:37 AM Kambiz Darabi wrote: > >> Hi, >> >> I have to implement a legacy file upload service for a customer. Files >> of up to some hundred megabytes are uploaded to a service over HTTP POST >> and a unique ID is returned which can be used to refer to that file, add >> metadata to it etc. >> >> Using Blobs is not viable as parsing of the base64 encoded binary data >> and the allocation of multiple strings during request processing >> increases both the time and space requirements. >> >> Ideally, I would be able to create an action on one of my domain >> services which has a HttpServletRequest argument and take care of >> reading the binary data and writing them to the file system. >> >> I have played around with the request processing part of the >> restfulobjects viewer/server and managed to get the request/response >> objects through. >> >> I know that this is a real edge case, but I think that it is important >> to have solutions for edge cases, too, if Isis is to be used as a >> general purpose platform. >> >> Could some committers please review the changes and maybe propose a >> cleaner way of implementing this? >> >> >> https://github.com/m-creations/isis/commit/aa3b16a5cf463466f5abadbcc8cc73f16857a628 >> >> Thanks >> >> >> Kambiz >> >>
