Re: Scio 0.4.7 released

2018-01-04 Thread Jean-Baptiste Onofré 

Hi

I remember when working with Scio, there was some specific IOs in Scio not using 
the ones provided by Beam.


I didn't check recently, maybe it has changed.

Regards
JB

On 01/04/2018 09:51 PM, Neville Li wrote:
Not sure what you mean. We use Beam file IOs whenever possible which should 
already use the FileSystems API.


On Thu, Jan 4, 2018 at 3:48 PM Jean-Baptiste Onofré > wrote:


Hi Neville,

Great  Thanks for the update.

By the way, any plan to leverage the Beam filesystems in Scio now ?

Regards
JB

On 01/04/2018 09:41 PM, Neville Li wrote:
 > Hi all,
 >
 > We just release Scio 0.4.7. This release fixed a join performance
regression and
 > introduced several improvements.
 >
 > https://github.com/spotify/scio/releases/tag/v0.4.7
 >
 > /"Hydrochoerus hydrochaeris"/
 >
 >
 >       Features
 >
 >   * Add support for TFRecordSpec #990

 >   * Add convenience methods to randomSpit #987
 >     
 >   * Add BigtableDoFn #922  
#931
 >     
 >   * Add optional arguments validation #979
 >     
 >   * Performance improvement in Avro and BigQuery macro converters #989
 >     
 >   * Update new bigquery datetime and timestamp specification #982
 >     
 >
 >
 >       Bug fixes
 >
 >   * Fix join performance regression introduced in 0.4.4 #976
 >      #983
 >     
 >   * Fine tune dynamic sink API and and integration tests #993
 >      #995
 >     
 >

--
Jean-Baptiste Onofré
jbono...@apache.org 
http://blog.nanthrax.net
Talend - http://www.talend.com



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Scio 0.4.7 released

2018-01-04 Thread Neville Li 
Not sure what you mean. We use Beam file IOs whenever possible which should
already use the FileSystems API.

On Thu, Jan 4, 2018 at 3:48 PM Jean-Baptiste Onofré  wrote:

> Hi Neville,
>
> Great  Thanks for the update.
>
> By the way, any plan to leverage the Beam filesystems in Scio now ?
>
> Regards
> JB
>
> On 01/04/2018 09:41 PM, Neville Li wrote:
> > Hi all,
> >
> > We just release Scio 0.4.7. This release fixed a join performance
> regression and
> > introduced several improvements.
> >
> > https://github.com/spotify/scio/releases/tag/v0.4.7
> >
> > /"Hydrochoerus hydrochaeris"/
> >
> >
> >   Features
> >
> >   * Add support for TFRecordSpec #990 <
> https://github.com/spotify/scio/pull/990>
> >   * Add convenience methods to randomSpit #987
> > 
> >   * Add BigtableDoFn #922 
> #931
> > 
> >   * Add optional arguments validation #979
> > 
> >   * Performance improvement in Avro and BigQuery macro converters #989
> > 
> >   * Update new bigquery datetime and timestamp specification #982
> > 
> >
> >
> >   Bug fixes
> >
> >   * Fix join performance regression introduced in 0.4.4 #976
> >  #983
> > 
> >   * Fine tune dynamic sink API and and integration tests #993
> >  #995
> > 
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Scio 0.4.7 released

2018-01-04 Thread Jean-Baptiste Onofré 

Hi Neville,

Great  Thanks for the update.

By the way, any plan to leverage the Beam filesystems in Scio now ?

Regards
JB

On 01/04/2018 09:41 PM, Neville Li wrote:

Hi all,

We just release Scio 0.4.7. This release fixed a join performance regression and 
introduced several improvements.


https://github.com/spotify/scio/releases/tag/v0.4.7

/"Hydrochoerus hydrochaeris"/


  Features

  * Add support for TFRecordSpec #990 
  * Add convenience methods to randomSpit #987

  * Add BigtableDoFn #922  #931

  * Add optional arguments validation #979

  * Performance improvement in Avro and BigQuery macro converters #989

  * Update new bigquery datetime and timestamp specification #982



  Bug fixes

  * Fix join performance regression introduced in 0.4.4 #976
 #983

  * Fine tune dynamic sink API and and integration tests #993
 #995




--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com