[jira] [Commented] (BEAM-469) NullableCoder optimized encoding via passthrough context
[ https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768525#comment-15768525 ] ASF GitHub Bot commented on BEAM-469: - GitHub user dhalperi opened a pull request: https://github.com/apache/incubator-beam/pull/1680 [BEAM-XXX] Make KVCoder more efficient by removing unnecessary nesting See [BEAM-469] for more information about why this is correct. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dhalperi/incubator-beam efficient-nested-coders Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1680.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1680 commit 621e8250c9535d773c4f4440a34ea0833912b51f Author: Dan Halperin Date: 2016-12-21T23:37:49Z [BEAM-XXX] Make KVCoder more efficient by removing unnecessary nesting See [BEAM-469] for more information about why this is correct. > NullableCoder optimized encoding via passthrough context > > > Key: BEAM-469 > URL: https://issues.apache.org/jira/browse/BEAM-469 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Thomas Groh >Priority: Trivial > Labels: backward-incompatible > Fix For: 0.3.0-incubating > > > NullableCoder should encode using the context given and not always use the > nested context. For coders which can efficiently encode in the outer context > such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix > themselves with their length. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-469) NullableCoder optimized encoding via passthrough context
[ https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768476#comment-15768476 ] Daniel Halperin commented on BEAM-469: -- Sorry I missed this JIRA comment, [~mariusz89016]! A bit late, but... Say a coder C does not have the nested context. Then we actually have the guarantee that no one will put later elements. So if {{NullableCoder}} does not have the nested context, then no one will put more elements after whatever the {{NullableCoder}} puts. If the NC puts {{0}} then that's it -- the element is null. But if the NC puts {{1}}, then we know that all remaining bytes in the encoded string belong to the inner coder. That is effectively saying that the inner coder also does not need to have the nested context, so it does not need to write its own length. In your example, the {{NullableCoder}} is used in an inner context. So the inner coder needs to also use the inner context, because there may be more encoded elements later. In either case: the context of the nullable coder can be the same as the context of the inner coder. This is why in the patch here, we simply pass the NC's context down into the inner coder. All we have removed is the _additional_ nesting that was used. https://patch-diff.githubusercontent.com/raw/apache/incubator-beam/pull/992.patch > NullableCoder optimized encoding via passthrough context > > > Key: BEAM-469 > URL: https://issues.apache.org/jira/browse/BEAM-469 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Thomas Groh >Priority: Trivial > Labels: backward-incompatible > Fix For: 0.3.0-incubating > > > NullableCoder should encode using the context given and not always use the > nested context. For coders which can efficiently encode in the outer context > such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix > themselves with their length. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-469) NullableCoder optimized encoding via passthrough context
[ https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583205#comment-15583205 ] Daniel Halperin commented on BEAM-469: -- Backwards compatibility: this is a change to the encoding of data, which will only affect Cloud Dataflow streaming pipelines across update. This is not yet part of the Beam model, and no one in Beam depends on it. > NullableCoder optimized encoding via passthrough context > > > Key: BEAM-469 > URL: https://issues.apache.org/jira/browse/BEAM-469 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Assignee: Thomas Groh >Priority: Trivial > Labels: backward-incompatible > Fix For: 0.3.0-incubating > > > NullableCoder should encode using the context given and not always use the > nested context. For coders which can efficiently encode in the outer context > such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix > themselves with their length. -- This message was sent by Atlassian JIRA (v6.3.4#6332)