In Camel, we have different mode: with local file caching or using streaming
when possible (it depends of the body in the Exchange).
So, I think we can do the same in Beam.
Regards
JB
On 07/24/2017 09:38 PM, Eugene Kirpichov wrote:
I think Camille may have referred to python standard library
I think Camille may have referred to python standard library class StringIO
which means collecting the output into a string - and then I suppose
uploading the string to FTP. That could work (similar stuff exists in Java
library) but would limit us to files whose content fits in memory.
On Mon,
I guess TextIO ? ;)
Regards
JB
On Jul 24, 2017, 21:27, at 21:27, Eugene Kirpichov
wrote:
>What is StringIO?
>
>On Mon, Jul 24, 2017 at 1:47 AM Tolsa, Camille
>
>wrote:
>
>> Not necessary with StringIO
>>
>> On 24 July 2017 at 09:47,
What is StringIO?
On Mon, Jul 24, 2017 at 1:47 AM Tolsa, Camille
wrote:
> Not necessary with StringIO
>
> On 24 July 2017 at 09:47, Reuven Lax wrote:
>
> > This would require writing data to local files in order to upload it to
> the
> >
Not necessary with StringIO
On 24 July 2017 at 09:47, Reuven Lax wrote:
> This would require writing data to local files in order to upload it to the
> remote FTP, right?
>
> On Mon, Jul 24, 2017 at 12:31 AM, Jean-Baptiste Onofré
> wrote:
>
> > Hi
This would require writing data to local files in order to upload it to the
remote FTP, right?
On Mon, Jul 24, 2017 at 12:31 AM, Jean-Baptiste Onofré
wrote:
> Hi Lucas,
>
> IMHO, it's not a IO, it's a filesystem that TextIO and others can support
> (like GFS or HDFS).
>
>
Hello,
I would definitively appreciate this feature.
If i can help somehow tell me
Camille.
On 24 July 2017 at 09:31, Jean-Baptiste Onofré wrote:
> Hi Lucas,
>
> IMHO, it's not a IO, it's a filesystem that TextIO and others can support
> (like GFS or HDFS).
>
> It's what we
Hi Lucas,
IMHO, it's not a IO, it's a filesystem that TextIO and others can support (like
GFS or HDFS).
It's what we did in Camel: the ftp component is just an extend of file
component.
It means that we would be able to do:
pipeline.apply(TextIO.from("ftp://...;)).
Thoughts ?
If agree, I
Hi Beam folks,
I would like to suggest the creation of a Pipeline I/O to support FTP/SFTP
as both source and sink locations for data processing. I've done some
research and it looks like there isn't any kind of development ongoing for
this (at least not on Jira).
I'd like to know your thoughts