Thanks guys for the details and update.
I'm updating my WriteFn and Coder accordingly.
Regards
JB
On 06/20/2017 05:52 PM, Lukasz Cwik wrote:
Filed https://issues.apache.org/jira/browse/BEAM-2482 for updating
CodedValueMutationDetector
On Tue, Jun 20, 2017 at 8:29 AM, Kenneth Knowles
wrote:
Hi all,
Splittable DoFn is now available in Dataflow streaming runner, as of
https://github.com/apache/beam/pull/1898 !
Meanwhile, Flink support got disabled due to some churn as part of First
Stable Release, but it should be not hard to fix - tracked in
https://issues.apache.org/jira/browse/BEAM
Filed https://issues.apache.org/jira/browse/BEAM-2482 for updating
CodedValueMutationDetector
On Tue, Jun 20, 2017 at 8:29 AM, Kenneth Knowles
wrote:
> Replacing full encoding with structural value is a good way to provide an
> opportunity for a fast past. File a starter JIRA?
>
> The equals che
Replacing full encoding with structural value is a good way to provide an
opportunity for a fast past. File a starter JIRA?
The equals check should be retained since it will sometimes be even faster,
and structural value falls back to full encoding.
On Tue, Jun 20, 2017 at 8:19 AM, Lukasz Cwik
w
I think the mutation detector could be updated to use the coder's
structural value and the coder could then provide a structural value which
wraps the message and does the equality comparison however it chooses.
https://github.com/apache/beam/blob/01b3f87f977d44eac23eb5488074bbc638858a9d/sdks/java/
Either Java object equality or its coder needs to be deterministic for that
check to hold.
On Tue, Jun 20, 2017 at 7:49 AM, Reuven Lax
wrote:
> Him
>
> That is only a fast path. If equals returns false, it then encodes the
> values to a byte array and checks the byte array for equality. So as lo
Him
That is only a fast path. If equals returns false, it then encodes the
values to a byte array and checks the byte array for equality. So as long
as you havev a correct coder, this should work.
On Tue, Jun 20, 2017 at 2:06 AM, Jean-Baptiste Onofré
wrote:
> Hi Kenn,
>
> I checked in MutationD
Hi team,
I would like to run beam pipelines from an AppEngine service. I tried to
run them in a standard environment and I'm facing issues beam related.
This line crashes
import apache_beam as beam
Trace (file: dill/dill.py:68):
ImportError: Cannot re-init internal module __main__
I wonder if i
Hi Kenn,
I checked in MutationDetectors, and we use the CodedValueMutationDetector(T
value, Coder coder).
To verify mutation, we use the verifyUnmodified() method calling
verifyUnmodifiedThrowingCheckedExceptions().
In the verifyUnmodifiedThrowingCheckedExceptions() method, basically, we do
