Scio 0.10.0 released

Hi all, We just released Scio 0.10.0. Here's a short summary of the notable changes since 0.9.x: - Better decoupled Google Cloud Platform dependencies - Simplify coder implicits for faster compilation - Sort Merge Bucket performance improvements and bug fixes - Type-safe Parquet support -

Scio 0.9.5 released

Hi all, We just released Scio 0.9.5. This release upgrades Beam to the latest 2.24.0 and includes several improvements and bug fixes, including Parquet Avro dynamic destinations, Scalable Bloom Filter and many others. This will also likely be the last 0.9.x release before we start working on the

Re: Scio community hangout

So this is happening next Thu Oct 1 12PM US EDT/9AM PDT/16:00 UTC. Drop in if you want to chat or see what we're up to. See you there! meet.google.com/vze-suyd-kwd On Mon, Sep 14, 2020 at 1:51 PM Neville Li wrote: > Cross posting here in case you're interested or already using Scio >

Fwd: Scio community hangout

Cross posting here in case you're interested or already using Scio <https://github.com/spotify/scio>, the Scala API for Apache Beam. -- Forwarded message - From: Neville Li Date: Mon, Sep 14, 2020 at 1:49 PM Subject: Scio community hangout To: Scio Users

Scio 0.9.4 released

Hi all, We just released Scio 0.9.4. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.9.4 *"Deletrius"* There are no breaking changes in this release, but some were introduced with v0.9.0: See v0.9.0 Migration Guide

Scio 0.9.3 released

Hi all, We just released Scio 0.9.3. This bumps Beam SDK to 2.23.0 and includes a lot of improvements & bug fixes. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.9.3 *"Petrificus Totalus"* There are no breaking changes in this release, but some were introduced with v0.9.0:

Scio 0.9.0 released

Hi all, We just released Scio 0.9.0. The biggest change was dropping Scala 2.11 and adding 2.13 support. Also included are Guava/magnolia powered Bloom Filter, improved test messages and other improvements. For those using Featran for feature engineering, the

Re: Scio 0.8.3 released

ble slightly lazy (#2749 <https://github.com/spotify/scio/pull/2749>)" #2791 <https://github.com/spotify/scio/pull/2791> On Fri, Mar 20, 2020 at 2:10 PM Neville Li wrote: > Hi all, > > We just released Scio 0.8.3. This is mainly a bug fix release with some > min

Scio 0.8.3 released

Hi all, We just released Scio 0.8.3. This is mainly a bug fix release with some minor improvements. *Notes on JDK 11* JDK 11 is now supported starting with 0.8.2. To upgrade, you'll need to build your code and submit jobs to Dataflow with JDK 11. *Notes on Scala 2.11* Scio 0.8.x will be the

Scio 0.8. 2 released

Hi all, We just released Scio 0.8.2. This is mainly a bug fix release with some small improvements. Thanks to all the contributors! Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.8.2 *"Capacious Extremis"* Features - Update Beam to 2.19.0 (#2665

Scio 0.8.0 released

Hi all, We just released Scio 0.8.0. This is based on the most recent Beam 2.17.0 release and includes a lot of new features & bug fixes over the past 10 months. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.8.0 *"Amato Animo Animato Animagus"* Breaking changes & deprecations

Scio 0.8.0-beta2 released

Hi all, We've just released Scio 0.8.0-beta2 (Scala API for Apache Beam for those on the Beam mailing list). This will likely be the last beta before 0.8.0 stable release. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.8.0-beta2 Features - Bump Beam to 2.16.0 (#2292

SDK support status clarification

Hi all, more specifically Googlers here, I want to clarify the Beam SDK support status w.r.t. Dataflow runner here: https://cloud.google.com/dataflow/docs/support/sdk-version-support-status When a Beam SDK is deprecated, what does it mean for users running it on Dataflow? The page mentions that

Re: AvroIO read SpecificRecord with custom reader schema?

wrote: > Does AvroIO.readGenericRecords() work ? > > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java#L333 > > Thanks, > Cham > > On Thu, Jun 13, 2019 at 1:46 PM Neville Li wrote: > >> Hi, >> >>

AvroIO read SpecificRecord with custom reader schema?

Hi, Is it just me or is there no way for AvroIO to read SpecificRecords with a custom reader schema? AvroIO.read(Class recordClass) will use the schema of T and there's no way to override it. Cheers, Neville

Scio 0.7.4 released

Hi all, We just released Scio 0.7.4. The biggest change was upgrading to Apache Beam 2.11.0. https://github.com/spotify/scio/releases/tag/v0.7.4 *"Watsonula wautieri"* Features - Add sequence example support to scio-tensorflow (#1757 ) - Add

Scio 0.6.1 released

Hi all, We just released Scio 0.6.1. This is mainly a bug fix release. Also just a heads up, we'll be releasing alpha/beta of 0.7.0 soon which will include some major breaking changes. Keep an eye on this section of

Scio 0.5.4 released

Hi all, We just released Scio 0.5.4. This release includes 2 important serialization fixes and a few new features. We recommend upgrading if you're on 0.5.x. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.5.4 *"Marmota monax"* Features - Add satisfySingleValue SCollection

Scio 0.5.2 released

Hi all, We just released Scio 0.5.2 with a few enhancements and bug fixes. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.5.2 *"Kobus kob"* Features - Add Java Converters #1013 #1076

Re: Scio 0.5.0 released

On Fri, Mar 9, 2018 at 4:31 PM 'Eugene Kirpichov' via Scio Users < scio-us...@googlegroups.com> wrote: > Hi! > > On Fri, Mar 9, 2018 at 1:22 PM Rafal Wojdyla wrote: > >> Hi all, >> >> We have just released Scio 0.5.0. This is a major/breaking release - make >> sure to read the

Fwd: Scio 0.5.0-alpha2 released

+user@beam.apache.org <user@beam.apache.org> -- Forwarded message - From: Neville Li <neville@gmail.com> Date: Mon, Jan 29, 2018 at 6:54 PM Subject: Scio 0.5.0-alpha2 released To: d...@beam.apache.org <d...@beam.apache.org> Hi all, We just released

Re: Trying to understand Unable to encode element exceptions

Here's a fix to #1020 https://github.com/spotify/scio/pull/1032 On Sun, Jan 21, 2018 at 4:36 PM Neville Li <neville@gmail.com> wrote: > Awesome! > We have't wrapped any stateful processing API in scala but if you have > working snippet or ideas it'd be great to share

Re: Trying to understand Unable to encode element exceptions

Thanks for your help!! > > As a side note, I found this issue: > https://github.com/spotify/scio/issues/448 > > I can share/help there with our experience, as our job, with scio + > stateful + timely processing is working fine as of today > > Regards!! > > On Fri

Re: Trying to understand Unable to encode element exceptions

ple? > > Thanks for your help! > > On Fri, Jan 19, 2018 at 5:04 PM Neville Li <neville@gmail.com> wrote: > >> That happens when you mix beam transforms into scio and defeats the >> safety we have in place. Map the values into something beam-serializable >>

Re: Trying to understand Unable to encode element exceptions

too newbie, but this is my first experience > with Beam... > > Thanks! > > On Fri, Jan 19, 2018 at 5:19 PM Neville Li <neville@gmail.com> wrote: > >> In this case it's probably easiest to map the scala `Map[K, V]` into a >> `java.util.Map<K, V>` and e

Re: Trying to understand Unable to encode element exceptions

In this case it's probably easiest to map the scala `Map[K, V]` into a `java.util.Map<K, V>` and explicitly set a `MapCoder<K, V>` so you don't have to deal with internal coder inference. On Fri, Jan 19, 2018 at 11:03 AM Neville Li <neville@gmail.com> wrote: > That happ

Re: Trying to understand Unable to encode element exceptions

ually, this exception arises: > https://pastebin.com/xxdDMXSf > > Thanks! > > On Fri, Jan 19, 2018 at 4:35 PM Neville Li <neville@gmail.com> wrote: > >> You shouldn't manually set coder in most cases. It defaults to >> KryoAtomicCoder for most Scala types.

Re: Trying to understand Unable to encode element exceptions

hing I've always wondered. How does one specify a coder for a > case class? > > Regards > > On Fri, 19 Jan 2018 at 15:51, Neville Li <neville@gmail.com> wrote: > >> Not sure why it falls back to SerializableCoder. Can you file an GH issue >> with ideally a sni

Re: Trying to understand Unable to encode element exceptions

Not sure why it falls back to SerializableCoder. Can you file an GH issue with ideally a snippet that can reproduce the problem? On Fri, Jan 19, 2018, 7:43 AM Carlos Alonso wrote: > Hi everyone!! > > I'm building a pipeline to store items from a Google PubSub subscription

Scio 0.5.0-alpha1 is out

Hi all, We just released Scio 0.5.0-alpha1. This release includes a typed BigQuery performance improvement by bypassing intermediate TableRow JSONs. It has shown a 2x speed up in some of our benchmarks. Cheers, Neville https://github.com/spotify/scio/releases/tag/v0.5.0-alpha1 *"Ia Io"*

Re: Scio 0.4.7 released

to leverage the Beam filesystems in Scio now ? > > Regards > JB > > On 01/04/2018 09:41 PM, Neville Li wrote: > > Hi all, > > > > We just release Scio 0.4.7. This release fixed a join performance > regression and > > introduced several improvemen