Re: Slack request

Sent. Welcome! On Fri, Jan 19, 2018 at 11:35 AM, Thomas Pelletier wrote: > Would love an invite too please. > > On Fri, Jan 19, 2018 at 11:24 AM, Lukasz Cwik wrote: > >> Invite sent, welcome. >> >> On Fri, Jan 19, 2018 at 11:23 AM, Allan Wilson

Re: Slack request

Would love an invite too please. On Fri, Jan 19, 2018 at 11:24 AM, Lukasz Cwik wrote: > Invite sent, welcome. > > On Fri, Jan 19, 2018 at 11:23 AM, Allan Wilson > wrote: > >> Can you add me as well? >> >> Allan Wilson >> Principal Software Engineer >>

Re: Slack request

Can you add me as well? Allan Wilson Principal Software Engineer Pandora Media Inc Thanks From: Lukasz Cwik > Reply-To: "user@beam.apache.org" > Date: Friday, January 19,

Re: Slack request

Invite sent, welcome. On Fri, Jan 19, 2018 at 10:52 AM, Logan Hennessy < logan.henne...@zonarsystems.com> wrote: > Request for invitation to join Slack channel. > > > > *Logan Hennessy *| Software Dev Engineer > > Zonar Systems >

Re: Strata Conference this March 6-8

I'll be talking about streaming SQL at Strata SJC. It's related to the SQL work happening in Beam that Mingmin, James, and Anton are working on, but the talk itself is relatively conceptual, hence no Beam in the title. Is the BoF thing happening? If so, is Wed 3:20pm the confirmed time? I'll be

Re: Request for invitation to Slack team

Just sent, welcome! On Fri, Jan 19, 2018 at 10:29 AM, Andrew Nguonly wrote: > Request for invitation to Slack team > -- Mingmin

Request for invitation to Slack team

Request for invitation to Slack team

Re: Requesting Slack Access

Invite sent, welcome. On Fri, Jan 19, 2018 at 10:17 AM, David Squier wrote: > Hello, > > I am an engineer at Honey Science (joinhoney.com) and we're currently > using Apache Beam. May I have an invite to the Apache Beam Slack channel? > > Thanks much! > > David

Requesting Slack Access

Hello, I am an engineer at Honey Science (joinhoney.com) and we're currently using Apache Beam. May I have an invite to the Apache Beam Slack channel? Thanks much! David Squier

Re: Trying to understand Unable to encode element exceptions

Welcome. Added an issue so we may improve this in the future: https://github.com/spotify/scio/issues/1020 On Fri, Jan 19, 2018 at 11:14 AM Carlos Alonso wrote: > To build the beam transform I was following this example: >

Re: Trying to understand Unable to encode element exceptions

Ok, I’ll try that. Thanks a lot for your help!! On Fri, 19 Jan 2018 at 17:37, Neville Li wrote: > Didn't realize the map is in a case class which is serializable, but > `java.util.Map` is not. So this won't work transitively. > You best bet is to write a custom Coder (you

Re: Trying to understand Unable to encode element exceptions

Didn't realize the map is in a case class which is serializable, but `java.util.Map` is not. So this won't work transitively. You best bet is to write a custom Coder (you can compose a map coder for the map field) for the entire case class and set it as part of the KvCoder. On Fri, Jan 19, 2018

Re: Trying to understand Unable to encode element exceptions

You mean replacing the Map[String, String] from the case class into a java.util.Map? And then, how could I set that MapCoder for that bit? Sorry if those questions are too newbie, but this is my first experience with Beam... Thanks! On Fri, Jan 19, 2018 at 5:19

Re: Trying to understand Unable to encode element exceptions

In this case it's probably easiest to map the scala `Map[K, V]` into a `java.util.Map` and explicitly set a `MapCoder` so you don't have to deal with internal coder inference. On Fri, Jan 19, 2018 at 11:03 AM Neville Li wrote: > That happens when you mix beam

Re: Trying to understand Unable to encode element exceptions

To build the beam transform I was following this example: https://github.com/spotify/scio/blob/master/scio-examples/src/main/scala/com/spotify/scio/examples/extra/DoFnExample.scala To be honest I don't know how to apply timely and stateful processing without using a beam transform or how to

Re: Trying to understand Unable to encode element exceptions

That happens when you mix beam transforms into scio and defeats the safety we have in place. Map the values into something beam-serializable first or rewrite the transform with a scio built-in which takes care of KvCoder. On Fri, Jan 19, 2018, 10:56 AM Carlos Alonso wrote:

Re: Trying to understand Unable to encode element exceptions

I'm following this example: https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L60 because I'm building something very similar to a group into batches functionality. If I don't set the coder manually, this exception arises:

Re: Trying to understand Unable to encode element exceptions

You shouldn't manually set coder in most cases. It defaults to KryoAtomicCoder for most Scala types. More details: https://github.com/spotify/scio/wiki/Scio%2C-Beam-and-Dataflow#coders On Fri, Jan 19, 2018, 10:27 AM Carlos Alonso wrote: > May it be because I’m using >

Re: Trying to understand Unable to encode element exceptions

May it be because I’m using .setCoder(KvCoder.of(StringUtf8Coder.of(), CoderRegistry.createDefault().getCoder(classOf[MessageWithAttributes]))) at some point in the pipeline (CoderRegistry.createDefault().getCoder(classOf[MessageWithAttributes]) outputs a SerializableCoder)? This is something

Re: Trying to understand Unable to encode element exceptions

Not sure why it falls back to SerializableCoder. Can you file an GH issue with ideally a snippet that can reproduce the problem? On Fri, Jan 19, 2018, 7:43 AM Carlos Alonso wrote: > Hi everyone!! > > I'm building a pipeline to store items from a Google PubSub subscription