[
https://issues.apache.org/jira/browse/GIRAPH-329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460343#comment-13460343
]
Avery Ching commented on GIRAPH-329:
------------------------------------
Actually, I've decided that we might not need to split it. 95% of the changes
are related to changing configuration to be Giraph specific (inspired by
GIRAPH-214). The remaining changes are quite simple and easy to understand.
Major changes:
- Configuration have been deprecated in favor of GiraphConfiguration for easy
get/set methods. Then I have ImmutableClassesGiraphConfiguration that will
generate the classes as final variables to be thread-safe. This is a big
improvement over Configuration#getClass(), which is very slow and makes
deserialization very slow. This represents the bulk of the changes as we have
to change code everywhere. All configuration variables have moved from
GiraphJob to GiraphConfiguration. Note that I didn't fix the vertex input /
output APIs to use ImmutableClassesGiraphConfiguration and just generate it
there for now. That can be done in a later change.
- There is a new ImmutableClassesGiraphConfigurable that replaces Configurable
and our ReflectionUtils understands this to make sure to newInstance an object
and set its ImmutableClassesGiraphConfiguration if it is
ImmutableClassesGiraphConfigurable.
- Upgraded/downgraded log messages to trace/debug to make debug more usable.
I'm able to run most jobs in debug with about a 10% loss in performance, not
too bad. Debug is now quite useful to figuring out performance issues, general
problems.
- Added an optimization to SimpleMessageStore#addPartitionMessages to avoid
getting the partition map for every vertex. This was one of the bottlenecks.
- Added client and server ExecutionHandlers to Netty to avoid doing the
business logic with I/O threads (This used to clog up the pipe)
- Added WrappedAdaptiveReceiveBufferSizePredictorFactory to debug/predict the
size of the incoming messages and provide better performance tuning on the
server instead of receiving a bunch of small messages.
Minor changes:
- Added BspUtilsTest to try various ways of testing class creation, performance
and correctness.
- Upgrade junit from 4.0 -> 4.8 to allow getting the test name (See
BspUtilsTest.java).
> Improve network performance
> ---------------------------
>
> Key: GIRAPH-329
> URL: https://issues.apache.org/jira/browse/GIRAPH-329
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-329.patch, nettyImprovement.png,
> NettyImprovements.xlsx
>
>
> I have been working on some network improvements that appear to give 1.5x to
> 2x. Unfortunately, this is a very large change and very hard to split out.
> Will give an update soon.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
