GitHub user rednaxelafx opened a pull request:
https://github.com/apache/spark/pull/22103
[SPARK-25113][SQL] Add logging to CodeGenerator when any generated method's
bytecode size goes above HugeMethodLimit
## What changes were proposed in this pull request?
Add logging for all generated methods from the `CodeGenerator` whose
bytecode size goes above 8000 bytes.
This is to help with gathering stats on how often Spark is generating
methods too big to be JIT'd. It covers all codegen scenarios, include
whole-stage codegen and also individual expression codegen, e.g. unsafe
projection, mutable projection, etc.
## How was this patch tested?
Manually tested that logging did happen when generated method was above
8000 bytes.
Also added a new unit test case to `CodeGenerationSuite` to verify that the
logging did happen.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rednaxelafx/apache-spark codegen-8k-logging
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22103.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22103
----
commit 640c9cd3b99d51f38c9b1a0c3f94bae676d11e51
Author: Kris Mok <kris.mok@...>
Date: 2018-08-14T10:50:01Z
SPARK-25113: Add logging to CodeGenerator when any generated method's
bytecode size goes above HugeMethodLimit
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
