Ah, good to know. Thanks. On Mar 5, 2014, at 2:40 PM, Ben Craig wrote:
> Don't use list<byte>. Large list performance is pretty bad, because > elements tend to be serialized one at a time. Use 'binary' or 'string' > instead. You also get significantly fewer copies this way. > > Rush Manbert <r...@manbert.com> wrote on 03/05/2014 04:31:47 PM: > >> From: Rush Manbert <r...@manbert.com> >> To: dev@thrift.apache.org, >> Date: 03/05/2014 04:32 PM >> Subject: Re: File Upload through thrift >> >> Hi Sachith, >> >> I would define a thrift struct that has metadata about the file, >> followed by a member of type list<byte> that stores the file data. I >> would also define the metadata so it can support transmitting the >> file in chunks (data that says this is chunk m of n, for instance), >> where each chunk is sent in a separate message to the server. I >> would also include some unique cookie value so you can differentiate >> between two client processes sending the same file. >> >> On the client side, figure out how many chunks you need to send the >> file, then split the file data up between that many instances of >> your Thrift class. Iterate over the Thrift classes and send each to >> the server. >> >> On the server side, receive the sendFile messages. Use the unique >> cookie from within the structure to sort the structures you receive >> into a separate bin for each file you are receiving >> (std::map<std::string, std::vector<YourThriftStructType *> > would >> probably work). When you have received all n chunks for a file >> (which you can tell because of the metadata), then use the collected >> Thrift structures to reconstitute the file on the server side. The >> server method can return a value that tells whether or not it thinks >> the transmission has completed. This approach lets you field send >> messages from many clients simultaneously with a single server instance. >> >> That's my first cut. It probably requires some refinement. ;-) >> >> - Rush >> >> >> On Mar 5, 2014, at 1:45 PM, Sachith Withana wrote: >> >>> Thanks Henrique and Jens, >>> >>> Ideally we'd like to support multiple Gigabytes of data. >>> As you said, we will potentially end up having a server to do file >>> uploading and downloading through URLs for Big files. >>> >>> But we'd like to support small file (~10MB) uploads using the Thrift >>> interface at least for now. >>> >>> What would be the best approach in achieving that? >>> >>> >>> On Wed, Mar 5, 2014 at 4:32 PM, Jens Geyer <jensge...@hotmail.com> > wrote: >>> >>>> [...] faster to send a URL and download it separately, [...] >>>>> >>>> >>>> Ack, I was about to write the same. >>>> >>>> >>>> >>>> -----Ursprüngliche Nachricht----- From: Henrique Mendonça >>>> Sent: Wednesday, March 5, 2014 10:24 PM >>>> To: dev@thrift.apache.org >>>> Subject: Re: File Upload through thrift >>>> >>>> >>>> Hi, >>>> You can do this, how big are your files? You might find that some >>>> implementations do have a size limit, not sure about Java, though. >>>> Anyway, is generally faster to send a URL and download it separately, > > if it >>>> fits you... >>>> >>>> >>>> On 5 March 2014 20:56, Sachith Withana <swsach...@gmail.com> wrote: >>>> >>>> I'm planning on converting the file into binary type and then do the >>>>> transfer to the server. >>>>> >>>>> What are the possible drawbacks of using this? >>>>> >>>>> This feature is for Apache Airavata. We are dealing with multiple > language >>>>> clients and a main Java Server. >>>>> We have to upload ( and download) large files too. >>>>> >>>>> Thanks. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Mar 5, 2014 at 1:47 PM, Sachith Withana > <swsach...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I need to transfer files between the Thrift clients and Servers. >>>>>> There can be large files to be transferred. >>>>>> >>>>>> Can someone please suggest a way of obtaining this? >>>>>> >>>>>> -- >>>>>> Thanks, >>>>>> Sachith Withana >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Sachith Withana >>>>> >>>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Sachith Withana >>
