[jira] [Commented] (FLINK-2882) Improve performance of string conversions
[
https://issues.apache.org/jira/browse/FLINK-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965649#comment-14965649
]
Ufuk Celebi commented on FLINK-2882:
Regarding {{toShortString}}: This is exclusively used in the network stack and
there it's only used for debugging as far as I know, as part of the
{{toString}} methods of result partitions, because they require two IDs to
identify their source/target (which gets very long even as hex strings). Can
you tell what fraction of calls come from {{toShortString}}? We can also think
about removing that variant as it was mostly useful in the early days during
debugging the network stack.
In general, I'm wondering why the ID toString methods are called so often. Can
you give the top 10 stack traces leading to it or so? And what LOG level you
are using?
In any case, both your suggestions sound reasonable to me.
> Improve performance of string conversions
> -
>
> Key: FLINK-2882
> URL: https://issues.apache.org/jira/browse/FLINK-2882
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> {{AbstractID.toString()}} and {{AbstractID.toShortString()}} call
> {{StringUtils.byteToHexString(...)}} which uses a StringBuilder to convert
> from binary to hex. This is a hotspot when scaling the number of workers.
> While testing on my single node with parallelism=512 jvisualvm reports
> 600,000 calls taking 13.4 seconds. Improving
> {{StringUtils.byteToHexString(...)}} reduces the time to 1.3 seconds.
> Additionally memoizing the string values in {{AbstractID}} reduce the time to
> 350 ms and the number of calls to {{StringUtils.byteToHexString(...)}} to
> ~1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2882) Improve performance of string conversions
[
https://issues.apache.org/jira/browse/FLINK-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14966670#comment-14966670
]
Stephan Ewen commented on FLINK-2882:
-
Interesting find!
I am curious, how often to we need to call this method, actually? When we
transfer IDs, we do not convert them to strings. So is this all done for the
sake of logging?
If so, is the logging by itself also becoming a bottleneck when increasing the
DOP?
> Improve performance of string conversions
> -
>
> Key: FLINK-2882
> URL: https://issues.apache.org/jira/browse/FLINK-2882
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.10
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> {{AbstractID.toString()}} and {{AbstractID.toShortString()}} call
> {{StringUtils.byteToHexString(...)}} which uses a StringBuilder to convert
> from binary to hex. This is a hotspot when scaling the number of workers.
> While testing on my single node with parallelism=512 jvisualvm reports
> 600,000 calls taking 13.4 seconds. Improving
> {{StringUtils.byteToHexString(...)}} reduces the time to 1.3 seconds.
> Additionally memoizing the string values in {{AbstractID}} reduce the time to
> 350 ms and the number of calls to {{StringUtils.byteToHexString(...)}} to
> ~1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2882) Improve performance of string conversions
[
https://issues.apache.org/jira/browse/FLINK-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057957#comment-15057957
]
ASF GitHub Bot commented on FLINK-2882:
---
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/1455
[FLINK-2882] [core] Improve performance of string conversions
Memoize string representations of AbstractID.
Use lookup table for byte-to-hex conversion in StringUtils.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink
2882_improve_performance_of_string_conversions
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1455.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1455
commit e23322855f94245783d7f0a5accb1558d2816152
Author: Greg Hogan
Date: 2015-12-14T19:47:33Z
[FLINK-2882] [core] Improve performance of string conversions
Memoize string representations of AbstractID.
Use lookup table for byte-to-hex conversion in StringUtils.
> Improve performance of string conversions
> -
>
> Key: FLINK-2882
> URL: https://issues.apache.org/jira/browse/FLINK-2882
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.10.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> {{AbstractID.toString()}} and {{AbstractID.toShortString()}} call
> {{StringUtils.byteToHexString(...)}} which uses a StringBuilder to convert
> from binary to hex. This is a hotspot when scaling the number of workers.
> While testing on my single node with parallelism=512 jvisualvm reports
> 600,000 calls taking 13.4 seconds. Improving
> {{StringUtils.byteToHexString(...)}} reduces the time to 1.3 seconds.
> Additionally memoizing the string values in {{AbstractID}} reduce the time to
> 350 ms and the number of calls to {{StringUtils.byteToHexString(...)}} to
> ~1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2882) Improve performance of string conversions
[
https://issues.apache.org/jira/browse/FLINK-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058169#comment-15058169
]
ASF GitHub Bot commented on FLINK-2882:
---
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/1455#issuecomment-164790091
Thanks for your contribution @greghogan. Efficiency improvements look
really good. The test failures seem to be unrelated.
Would be great if you could add a small test for the StringUtils method.
Apart from that, +1 for merging.
> Improve performance of string conversions
> -
>
> Key: FLINK-2882
> URL: https://issues.apache.org/jira/browse/FLINK-2882
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.10.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> {{AbstractID.toString()}} and {{AbstractID.toShortString()}} call
> {{StringUtils.byteToHexString(...)}} which uses a StringBuilder to convert
> from binary to hex. This is a hotspot when scaling the number of workers.
> While testing on my single node with parallelism=512 jvisualvm reports
> 600,000 calls taking 13.4 seconds. Improving
> {{StringUtils.byteToHexString(...)}} reduces the time to 1.3 seconds.
> Additionally memoizing the string values in {{AbstractID}} reduce the time to
> 350 ms and the number of calls to {{StringUtils.byteToHexString(...)}} to
> ~1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2882) Improve performance of string conversions
[
https://issues.apache.org/jira/browse/FLINK-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058174#comment-15058174
]
ASF GitHub Bot commented on FLINK-2882:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1455#issuecomment-164790880
Looks very good.
I am wondering if we can get rid of the toShortString method. What is that
one actually used for?
> Improve performance of string conversions
> -
>
> Key: FLINK-2882
> URL: https://issues.apache.org/jira/browse/FLINK-2882
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.10.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> {{AbstractID.toString()}} and {{AbstractID.toShortString()}} call
> {{StringUtils.byteToHexString(...)}} which uses a StringBuilder to convert
> from binary to hex. This is a hotspot when scaling the number of workers.
> While testing on my single node with parallelism=512 jvisualvm reports
> 600,000 calls taking 13.4 seconds. Improving
> {{StringUtils.byteToHexString(...)}} reduces the time to 1.3 seconds.
> Additionally memoizing the string values in {{AbstractID}} reduce the time to
> 350 ms and the number of calls to {{StringUtils.byteToHexString(...)}} to
> ~1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2882) Improve performance of string conversions
[
https://issues.apache.org/jira/browse/FLINK-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058223#comment-15058223
]
ASF GitHub Bot commented on FLINK-2882:
---
Github user uce commented on the pull request:
https://github.com/apache/flink/pull/1455#issuecomment-164802103
Yes, that's good to remove. I pointed that out as well when Greg opened the
initial issue.
We can either do it as part of this PR or I can do it as a follow up.
> Improve performance of string conversions
> -
>
> Key: FLINK-2882
> URL: https://issues.apache.org/jira/browse/FLINK-2882
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.10.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> {{AbstractID.toString()}} and {{AbstractID.toShortString()}} call
> {{StringUtils.byteToHexString(...)}} which uses a StringBuilder to convert
> from binary to hex. This is a hotspot when scaling the number of workers.
> While testing on my single node with parallelism=512 jvisualvm reports
> 600,000 calls taking 13.4 seconds. Improving
> {{StringUtils.byteToHexString(...)}} reduces the time to 1.3 seconds.
> Additionally memoizing the string values in {{AbstractID}} reduce the time to
> 350 ms and the number of calls to {{StringUtils.byteToHexString(...)}} to
> ~1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2882) Improve performance of string conversions
[
https://issues.apache.org/jira/browse/FLINK-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058304#comment-15058304
]
ASF GitHub Bot commented on FLINK-2882:
---
Github user greghogan commented on the pull request:
https://github.com/apache/flink/pull/1455#issuecomment-164824368
@tillrohrmann There exists StringUtilsTest with small test.
One of the TravisCI errors was a timeout and the other killed by the
watchdog. Are timeouts being reworked to use RetryOnException or similar?
@uce It looks like toShortString is used for more than debugging and tests
so I'd go for a separate PR.
> Improve performance of string conversions
> -
>
> Key: FLINK-2882
> URL: https://issues.apache.org/jira/browse/FLINK-2882
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.10.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> {{AbstractID.toString()}} and {{AbstractID.toShortString()}} call
> {{StringUtils.byteToHexString(...)}} which uses a StringBuilder to convert
> from binary to hex. This is a hotspot when scaling the number of workers.
> While testing on my single node with parallelism=512 jvisualvm reports
> 600,000 calls taking 13.4 seconds. Improving
> {{StringUtils.byteToHexString(...)}} reduces the time to 1.3 seconds.
> Additionally memoizing the string values in {{AbstractID}} reduce the time to
> 350 ms and the number of calls to {{StringUtils.byteToHexString(...)}} to
> ~1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2882) Improve performance of string conversions
[
https://issues.apache.org/jira/browse/FLINK-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058690#comment-15058690
]
ASF GitHub Bot commented on FLINK-2882:
---
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/1455#issuecomment-164880926
Looks good. 1+ for removing `toShortString` in a separate issue.
I'll merge this PR.
> Improve performance of string conversions
> -
>
> Key: FLINK-2882
> URL: https://issues.apache.org/jira/browse/FLINK-2882
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.10.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> {{AbstractID.toString()}} and {{AbstractID.toShortString()}} call
> {{StringUtils.byteToHexString(...)}} which uses a StringBuilder to convert
> from binary to hex. This is a hotspot when scaling the number of workers.
> While testing on my single node with parallelism=512 jvisualvm reports
> 600,000 calls taking 13.4 seconds. Improving
> {{StringUtils.byteToHexString(...)}} reduces the time to 1.3 seconds.
> Additionally memoizing the string values in {{AbstractID}} reduce the time to
> 350 ms and the number of calls to {{StringUtils.byteToHexString(...)}} to
> ~1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-2882) Improve performance of string conversions
[
https://issues.apache.org/jira/browse/FLINK-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058749#comment-15058749
]
ASF GitHub Bot commented on FLINK-2882:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1455
> Improve performance of string conversions
> -
>
> Key: FLINK-2882
> URL: https://issues.apache.org/jira/browse/FLINK-2882
> Project: Flink
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.10.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>
> {{AbstractID.toString()}} and {{AbstractID.toShortString()}} call
> {{StringUtils.byteToHexString(...)}} which uses a StringBuilder to convert
> from binary to hex. This is a hotspot when scaling the number of workers.
> While testing on my single node with parallelism=512 jvisualvm reports
> 600,000 calls taking 13.4 seconds. Improving
> {{StringUtils.byteToHexString(...)}} reduces the time to 1.3 seconds.
> Additionally memoizing the string values in {{AbstractID}} reduce the time to
> 350 ms and the number of calls to {{StringUtils.byteToHexString(...)}} to
> ~1000.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
