Thanks Brian and Luke!
I'm curious whether Schema supports optional fields like protobuf. In my
use case, most of the fields will be optional and my application only
accesses these field when the value is presented. Also it seems like if I
want to use Schema to transfer data across sdk, I need to
Hi Luke,
This seems like a _much_ better solution than attempting to manage the state /
buffer internally within the DoFn. I’ll give this a shot.
Much appreciated!
Rion
> On Jun 12, 2020, at 5:59 PM, Luke Cwik wrote:
>
>
> Check out GroupIntoBatches.
>
>> On Fri, Jun 12, 2020 at 3:53
> are unknown fields propagated through if the user only reads/modifies a
row?
I'm not sure I understand this question. Are you asking about handling
schema changes?
The wire format includes the number of fields in the schema, specifically
so that we can detect when the schema changes. This is
Check out GroupIntoBatches.
On Fri, Jun 12, 2020 at 3:53 PM Rion Williams wrote:
> Hi Luke,
>
> I think that’s likely my mistake. I had forgotten that was tied to a given
> key-window. In this example use case, all of the data is keyed differently
> (and thus not associated to a window or a
Hi Luke,
I think that’s likely my mistake. I had forgotten that was tied to a given
key-window. In this example use case, all of the data is keyed differently (and
thus not associated to a window or a key), so knowing that I’m quite sure it’s
the issue.
In this scenario I’m experimenting with
What is the update / compat story around schemas?
* are unknown fields propagated through if the user only reads/modifies a
row?
* how does it work in a pipeline update scenario (downgrade / upgrade)?
Boyuan has been working on a Kafka via SDF source and have been trying to
figure out which
+Boyuan Zhang
On Fri, Jun 12, 2020 at 3:32 PM Luke Cwik wrote:
> What is the update / compat story around schemas?
> * are unknown fields propagated through if the user only reads/modifies a
> row?
> * how does it work in a pipeline update scenario (downgrade / upgrade)?
>
> Boyuan has been
Simple question, you are expecting to see prior results under the same
window and key which you are not seeing (since state is per key and window)?
On Fri, Jun 12, 2020 at 3:09 PM Rion Williams wrote:
> Hi all,
>
> I've been toying around with stateful DoFns recently and was attempting
> some
Hi all,
I've been toying around with stateful DoFns recently and was attempting some
approaches involving buffering when I realized that it seemed that my existing
state was being ignored in the following DoFn:
```
class ExampleStatefulDoFn(): DoFn, KV>() {
@StateId("count")
private
Thank you for clarifying.
I attempted to use FlinkRunner with 2.22 and I am getting the following
error, which I am not sure how to debug:
ERROR:root:java.lang.UnsupportedOperationException: The ActiveBundle does
not have a registered bundle checkpoint handler.
Thanks! I see there are jiras for SpannerIO and JdbcIO as part of that. Are
you planning on using row coder for them?
If so I want to make sure you're aware of
https://s.apache.org/beam-schema-io (sent to the dev list last week
[1]). +Scott
Lukas will be working on building out the ideas there
I'm not very familiar with this effort.
Were there ITs / POCs created for these changes? (to surface any obvious
bugs)
Are these changes usable in DirectRunner?
On Fri, Jun 12, 2020 at 8:50 AM Luke Cwik wrote:
> A few months back there was a discussion[1] about performing work to
> stabilize
Hello Jie!
Welcome. I've added you as a contributor!
Best
-P.
On Fri, Jun 12, 2020 at 12:10 PM Jie Fan wrote:
> Hi,
>
> This is Jie from Google, I started to look into making contributions to
> the Go SDK recently, please add me as a contributor. My JIRA username is
> lastomato.
>
> Thanks,
>
Hi,
This is Jie from Google, I started to look into making contributions to the
Go SDK recently, please add me as a contributor. My JIRA username is
lastomato.
Thanks,
Jie
This is great! Looking forward to it.
Would any metrics need to be migrated over to the new Jenkins?
http://metrics.beam.apache.org/
On Fri, Jun 12, 2020 at 9:57 AM Tyson Hamilton wrote:
> Very exciting! Thanks for the advanced notice Damian.
>
> On Fri, Jun 12, 2020 at 7:58 AM Damian
Very exciting! Thanks for the advanced notice Damian.
On Fri, Jun 12, 2020 at 7:58 AM Damian Gadomski
wrote:
> Hello,
>
> During the last few days, I was preparing for the Beam Jenkins migration
> from builds.apache.org to ci-beam.apache.org. The new Jenkins Master will
> be dedicated only for
Hi Cynthia,
Thank you for the introduction and your interest to work on Apache Beam
documentation with Season of Docs. To participate in the program you need
to follow the guides here [1] [2]. If you are new to the program, we
suggest:
Start by studying our proposed project ideas and expected
Hi Ayush,
Thank you for the introduction and your interest to work on Apache Beam
documentation with Season of Docs. To participate in the program you need
to follow the guides here [1] [2]. If you are new to the program, we
suggest:
Start by studying our proposed project ideas and expected
A few months back there was a discussion[1] about performing work to
stabilize the protos used for pipeline execution looking forward to cross
language pipelines and runners who want to use them across SDK versions
(Dataflow).
All the proposed incompatible clean-up tasks were done and made it
Hello,
During the last few days, I was preparing for the Beam Jenkins migration
from builds.apache.org to ci-beam.apache.org. The new Jenkins Master will
be dedicated only for Beam related jobs, all Beam Committers will have
build configure access, and Beam PMC will have Admin (GUI) Access.
We
20 matches
Mail list logo
