[jira] [Commented] (BEAM-5638) Add exception handling to single message transforms in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638817#comment-16638817 ] Jeff Klukas commented on BEAM-5638: --- Looks like I'm going to run into https://issues.apache.org/jira/browse/BEAM-1891 where AutoValue classes don't have a default coder. I'm turning Failure into an AutoValue class and I'm going to have to give a default coder of SerializableCoder, which is not ideal. > Add exception handling to single message transforms in Java SDK > --- > > Key: BEAM-5638 > URL: https://issues.apache.org/jira/browse/BEAM-5638 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Jeff Klukas >Assignee: Jeff Klukas >Priority: Minor > Original Estimate: 168h > Remaining Estimate: 168h > > Add methods to MapElements, FlatMapElements, and Filter that allow users to > specify expected exceptions and tuple tags to associate with the with > collections of the successfully and unsuccessfully processed elements. > See discussion on dev list: > https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5638) Add exception handling to single message transforms in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637636#comment-16637636 ] Jeff Klukas commented on BEAM-5638: --- Work in progress PR: https://github.com/apache/beam/pull/6518 > Add exception handling to single message transforms in Java SDK > --- > > Key: BEAM-5638 > URL: https://issues.apache.org/jira/browse/BEAM-5638 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Jeff Klukas >Assignee: Kenneth Knowles >Priority: Minor > Original Estimate: 168h > Remaining Estimate: 168h > > Add methods to MapElements, FlatMapElements, and Filter that allow users to > specify expected exceptions and tuple tags to associate with the with > collections of the successfully and unsuccessfully processed elements. > See discussion on dev list: > https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5639) Add exception handling to single message transforms in Python SDK
[ https://issues.apache.org/jira/browse/BEAM-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Klukas updated BEAM-5639: -- Related issue for Java SDK: https://issues.apache.org/jira/browse/BEAM-5638 > Add exception handling to single message transforms in Python SDK > - > > Key: BEAM-5639 > URL: https://issues.apache.org/jira/browse/BEAM-5639 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Jeff Klukas >Assignee: Ahmet Altay >Priority: Minor > > Add methods to python SDK Map and FlatMap that allow users to specify > expected exceptions and tuple tags to associate with the with collections of > the successfully and unsuccessfully processed elements. > See discussion on dev list: > [https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5639) Add exception handling to single message transforms in Python SDK
Jeff Klukas created BEAM-5639: - Summary: Add exception handling to single message transforms in Python SDK Key: BEAM-5639 URL: https://issues.apache.org/jira/browse/BEAM-5639 Project: Beam Issue Type: Improvement Components: sdk-py-core Reporter: Jeff Klukas Assignee: Ahmet Altay Add methods to python SDK Map and FlatMap that allow users to specify expected exceptions and tuple tags to associate with the with collections of the successfully and unsuccessfully processed elements. See discussion on dev list: [https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5638) Add exception handling to single message transforms in Java SDK
Jeff Klukas created BEAM-5638: - Summary: Add exception handling to single message transforms in Java SDK Key: BEAM-5638 URL: https://issues.apache.org/jira/browse/BEAM-5638 Project: Beam Issue Type: Improvement Components: sdk-java-core Reporter: Jeff Klukas Assignee: Kenneth Knowles Add methods to MapElements, FlatMapElements, and Filter that allow users to specify expected exceptions and tuple tags to associate with the with collections of the successfully and unsuccessfully processed elements. See discussion on dev list: https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5413) Add method for defining composite transforms as lambda expressions
[
https://issues.apache.org/jira/browse/BEAM-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Klukas resolved BEAM-5413.
---
Resolution: Fixed
This has been merged.
> Add method for defining composite transforms as lambda expressions
> --
>
> Key: BEAM-5413
> URL: https://issues.apache.org/jira/browse/BEAM-5413
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.8.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Defining a composite transform today requires writing a full named subclass
> of PTransform (as [the programming guide
> documents|https://beam.apache.org/documentation/programming-guide/#composite-transforms]
> but there are cases where users may want to define a fairly trivial
> composite transform using a less verbose Java 8 lambda expression.
> Consider an example where the user has defined MyDeserializationTransform
> that attempts to deserialize byte arrays into some object, returning a
> PCollectionTuple with tags for successfully deserialized records (mainTag)
> and for errors (errorTag).
> If we introduce a PTransform::compose method that takes in a
> SerializableFunction, the user can handle errors in a small lambda expression:
>
> {code:java}
> byteArrays
> .apply("attempt to deserialize messages",
> new MyDeserializationTransform())
> .apply("write deserialization errors",
> PTransform.compose((PCollectionTuple input) -> {
> input
> .get(errorTag)
> .apply(new MyErrorOutputTransform());
> return input.get(mainTag);
> })
> .apply("more processing on the deserialized messages",
> new MyOtherTransform())
> {code}
> This style allows a more concise and fluent pipeline definition than is
> currently possible.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5413) Add method for defining composite transforms as lambda expressions
[
https://issues.apache.org/jira/browse/BEAM-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630377#comment-16630377
]
Jeff Klukas commented on BEAM-5413:
---
This was merged in with @Experimental annotation. It's my understanding it will
be included with 2.8.0.
> Add method for defining composite transforms as lambda expressions
> --
>
> Key: BEAM-5413
> URL: https://issues.apache.org/jira/browse/BEAM-5413
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.8.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Defining a composite transform today requires writing a full named subclass
> of PTransform (as [the programming guide
> documents|https://beam.apache.org/documentation/programming-guide/#composite-transforms]
> but there are cases where users may want to define a fairly trivial
> composite transform using a less verbose Java 8 lambda expression.
> Consider an example where the user has defined MyDeserializationTransform
> that attempts to deserialize byte arrays into some object, returning a
> PCollectionTuple with tags for successfully deserialized records (mainTag)
> and for errors (errorTag).
> If we introduce a PTransform::compose method that takes in a
> SerializableFunction, the user can handle errors in a small lambda expression:
>
> {code:java}
> byteArrays
> .apply("attempt to deserialize messages",
> new MyDeserializationTransform())
> .apply("write deserialization errors",
> PTransform.compose((PCollectionTuple input) -> {
> input
> .get(errorTag)
> .apply(new MyErrorOutputTransform());
> return input.get(mainTag);
> })
> .apply("more processing on the deserialized messages",
> new MyOtherTransform())
> {code}
> This style allows a more concise and fluent pipeline definition than is
> currently possible.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5413) Add method for defining composite transforms as lambda expressions
[
https://issues.apache.org/jira/browse/BEAM-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Klukas updated BEAM-5413:
--
Description:
Defining a composite transform today requires writing a full named subclass of
PTransform (as [the programming guide
documents|[https://beam.apache.org/documentation/programming-guide/#composite-transforms]]
but there are cases where users may want to define a fairly trivial composite
transform using a less verbose Java 8 lambda expression.
Consider an example where the user has defined MyDeserializationTransform that
attempts to deserialize byte arrays into some object, returning a
PCollectionTuple with tags for successfully deserialized records (mainTag) and
for errors (errorTag).
If we introduce a PTransform::compose method that takes in a
SerializableFunction, the user can handle errors in a small lambda expression:
{code:java}
byteArrays
.apply("attempt to deserialize messages",
new MyDeserializationTransform())
.apply("write deserialization errors",
PTransform.compose((PCollectionTuple input) -> {
input
.get(errorTag)
.apply(new MyErrorOutputTransform());
return input.get(mainTag);
})
.apply("more processing on the deserialized messages",
new MyOtherTransform())
{code}
This style allows a more concise and fluent pipeline definition than is
currently possible.
was:
Defining a composite transform today requires writing a full named subclass of
PTransform (as [the programming guide
documents|[https://beam.apache.org/documentation/programming-guide/#composite-transforms]),]
but there are cases where users may want to define a fairly trivial composite
transform using a less verbose Java 8 lambda expression.
Consider an example where the user has defined MyDeserializationTransform that
attempts to deserialize byte arrays into some object, returning a
PCollectionTuple with tags for successfully deserialized records (mainTag) and
for errors (errorTag).
If we introduce a PTransform::compose method that takes in a
SerializableFunction, the user can handle errors in a small lambda expression:
{code:java}
byteArrays
.apply("attempt to deserialize messages",
new MyDeserializationTransform())
.apply("write deserialization errors",
PTransform.compose((PCollectionTuple input) -> {
input
.get(errorTag)
.apply(new MyErrorOutputTransform());
return input.get(mainTag);
})
.apply("more processing on the deserialized messages",
new MyOtherTransform())
{code}
This style allows a more concise and fluent pipeline definition than is
currently possible.
> Add method for defining composite transforms as lambda expressions
> --
>
> Key: BEAM-5413
> URL: https://issues.apache.org/jira/browse/BEAM-5413
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.8.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Defining a composite transform today requires writing a full named subclass
> of PTransform (as [the programming guide
> documents|[https://beam.apache.org/documentation/programming-guide/#composite-transforms]]
> but there are cases where users may want to define a fairly trivial
> composite transform using a less verbose Java 8 lambda expression.
> Consider an example where the user has defined MyDeserializationTransform
> that attempts to deserialize byte arrays into some object, returning a
> PCollectionTuple with tags for successfully deserialized records (mainTag)
> and for errors (errorTag).
> If we introduce a PTransform::compose method that takes in a
> SerializableFunction, the user can handle errors in a small lambda expression:
>
> {code:java}
> byteArrays
> .apply("attempt to deserialize messages",
> new MyDeserializationTransform())
> .apply("write deserialization errors",
> PTransform.compose((PCollectionTuple input) -> {
> input
> .get(errorTag)
> .apply(new MyErrorOutputTransform());
> return input.get(mainTag);
> })
> .apply("more processing on the deserialized messages",
> new MyOtherTransform())
> {code}
> This style allows a more concise and fluent pipeline definition than is
> currently possible.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-5413) Add method for defining composite transforms as lambda expressions
[
https://issues.apache.org/jira/browse/BEAM-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Klukas updated BEAM-5413:
--
Description:
Defining a composite transform today requires writing a full named subclass of
PTransform (as [the programming guide
documents|https://beam.apache.org/documentation/programming-guide/#composite-transforms]
but there are cases where users may want to define a fairly trivial composite
transform using a less verbose Java 8 lambda expression.
Consider an example where the user has defined MyDeserializationTransform that
attempts to deserialize byte arrays into some object, returning a
PCollectionTuple with tags for successfully deserialized records (mainTag) and
for errors (errorTag).
If we introduce a PTransform::compose method that takes in a
SerializableFunction, the user can handle errors in a small lambda expression:
{code:java}
byteArrays
.apply("attempt to deserialize messages",
new MyDeserializationTransform())
.apply("write deserialization errors",
PTransform.compose((PCollectionTuple input) -> {
input
.get(errorTag)
.apply(new MyErrorOutputTransform());
return input.get(mainTag);
})
.apply("more processing on the deserialized messages",
new MyOtherTransform())
{code}
This style allows a more concise and fluent pipeline definition than is
currently possible.
was:
Defining a composite transform today requires writing a full named subclass of
PTransform (as [the programming guide
documents|[https://beam.apache.org/documentation/programming-guide/#composite-transforms]]
but there are cases where users may want to define a fairly trivial composite
transform using a less verbose Java 8 lambda expression.
Consider an example where the user has defined MyDeserializationTransform that
attempts to deserialize byte arrays into some object, returning a
PCollectionTuple with tags for successfully deserialized records (mainTag) and
for errors (errorTag).
If we introduce a PTransform::compose method that takes in a
SerializableFunction, the user can handle errors in a small lambda expression:
{code:java}
byteArrays
.apply("attempt to deserialize messages",
new MyDeserializationTransform())
.apply("write deserialization errors",
PTransform.compose((PCollectionTuple input) -> {
input
.get(errorTag)
.apply(new MyErrorOutputTransform());
return input.get(mainTag);
})
.apply("more processing on the deserialized messages",
new MyOtherTransform())
{code}
This style allows a more concise and fluent pipeline definition than is
currently possible.
> Add method for defining composite transforms as lambda expressions
> --
>
> Key: BEAM-5413
> URL: https://issues.apache.org/jira/browse/BEAM-5413
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.8.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Defining a composite transform today requires writing a full named subclass
> of PTransform (as [the programming guide
> documents|https://beam.apache.org/documentation/programming-guide/#composite-transforms]
> but there are cases where users may want to define a fairly trivial
> composite transform using a less verbose Java 8 lambda expression.
> Consider an example where the user has defined MyDeserializationTransform
> that attempts to deserialize byte arrays into some object, returning a
> PCollectionTuple with tags for successfully deserialized records (mainTag)
> and for errors (errorTag).
> If we introduce a PTransform::compose method that takes in a
> SerializableFunction, the user can handle errors in a small lambda expression:
>
> {code:java}
> byteArrays
> .apply("attempt to deserialize messages",
> new MyDeserializationTransform())
> .apply("write deserialization errors",
> PTransform.compose((PCollectionTuple input) -> {
> input
> .get(errorTag)
> .apply(new MyErrorOutputTransform());
> return input.get(mainTag);
> })
> .apply("more processing on the deserialized messages",
> new MyOtherTransform())
> {code}
> This style allows a more concise and fluent pipeline definition than is
> currently possible.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5413) Add method for defining composite transforms as lambda expressions
Jeff Klukas created BEAM-5413:
-
Summary: Add method for defining composite transforms as lambda
expressions
Key: BEAM-5413
URL: https://issues.apache.org/jira/browse/BEAM-5413
Project: Beam
Issue Type: Improvement
Components: sdk-java-core
Reporter: Jeff Klukas
Assignee: Kenneth Knowles
Fix For: 2.8.0
Defining a composite transform today requires writing a full named subclass of
PTransform (as [the programming guide
documents|[https://beam.apache.org/documentation/programming-guide/#composite-transforms]),]
but there are cases where users may want to define a fairly trivial composite
transform using a less verbose Java 8 lambda expression.
Consider an example where the user has defined MyDeserializationTransform that
attempts to deserialize byte arrays into some object, returning a
PCollectionTuple with tags for successfully deserialized records (mainTag) and
for errors (errorTag).
If we introduce a PTransform::compose method that takes in a
SerializableFunction, the user can handle errors in a small lambda expression:
{code:java}
byteArrays
.apply("attempt to deserialize messages",
new MyDeserializationTransform())
.apply("write deserialization errors",
PTransform.compose((PCollectionTuple input) -> {
input
.get(errorTag)
.apply(new MyErrorOutputTransform());
return input.get(mainTag);
})
.apply("more processing on the deserialized messages",
new MyOtherTransform())
{code}
This style allows a more concise and fluent pipeline definition than is
currently possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
