[
https://issues.apache.org/jira/browse/SPARK-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Wendell updated SPARK-2549:
-----------------------------------
Description:
If we have a function declaration inside of another function, it still triggers
mima failures. We should look at how that is implemented in byte code and just
always exclude functions like that.
{code}
def a() = {
/* Changing b() should not trigger failures, but it does. */
def b() = {}
}
{code}
I dug into the byte code for inner functions a bit more. I noticed that they
tend to use `$$` before the function name.
There is more information on that string sequence here:
https://github.com/scala/scala/blob/2.10.x/src/reflect/scala/reflect/internal/StdNames.scala#L286
I did a cursory look and it appears that symbol is mostly (exclusively?) used
for anonymous or inner functions:
{code}
# in RDD package classes
$ ls *.class | xargs -I {} javap {} |grep "\\$\\$"
public final java.lang.Object
org$apache$spark$rdd$PairRDDFunctions$$createZero$1(scala.reflect.ClassTag,
byte[], scala.runtime.ObjectRef, scala.runtime.VolatileByteRef);
public final java.lang.Object
org$apache$spark$rdd$PairRDDFunctions$$createZero$2(byte[],
scala.runtime.ObjectRef, scala.runtime.VolatileByteRef);
public final scala.collection.Iterator
org$apache$spark$rdd$PairRDDFunctions$$reducePartition$1(scala.collection.Iterator,
scala.Function2);
public final java.util.HashMap
org$apache$spark$rdd$PairRDDFunctions$$mergeMaps$1(java.util.HashMap,
java.util.HashMap, scala.Function2);
...
public final class org.apache.spark.rdd.AsyncRDDActions$$anonfun$countAsync$1
extends scala.runtime.AbstractFunction0$mcJ$sp implements scala.Serializable {
public
org.apache.spark.rdd.AsyncRDDActions$$anonfun$countAsync$1(org.apache.spark.rdd.AsyncRDDActions<T>);
public final class org.apache.spark.rdd.AsyncRDDActions$$anonfun$countAsync$2
extends scala.runtime.AbstractFunction2$mcVIJ$sp implements scala.Serializable {
public
org.apache.spark.rdd.AsyncRDDActions$$anonfun$countAsync$2(org.apache.spark.rdd.AsyncRDDActions<T>);
{code}
was:
If we have a function declaration inside of another function, it still triggers
mima failures. We should look at how that is implemented in byte code and just
always exclude functions like that.
{code}
def a() = {
/* Changing b() should not trigger failures, but it does. */
def b() = {}
}
{code}
> Functions defined inside of other functions trigger failures
> ------------------------------------------------------------
>
> Key: SPARK-2549
> URL: https://issues.apache.org/jira/browse/SPARK-2549
> Project: Spark
> Issue Type: Sub-task
> Components: Build
> Reporter: Patrick Wendell
> Assignee: Prashant Sharma
> Fix For: 1.1.0
>
>
> If we have a function declaration inside of another function, it still
> triggers mima failures. We should look at how that is implemented in byte
> code and just always exclude functions like that.
> {code}
> def a() = {
> /* Changing b() should not trigger failures, but it does. */
> def b() = {}
> }
> {code}
> I dug into the byte code for inner functions a bit more. I noticed that they
> tend to use `$$` before the function name.
> There is more information on that string sequence here:
> https://github.com/scala/scala/blob/2.10.x/src/reflect/scala/reflect/internal/StdNames.scala#L286
> I did a cursory look and it appears that symbol is mostly (exclusively?) used
> for anonymous or inner functions:
> {code}
> # in RDD package classes
> $ ls *.class | xargs -I {} javap {} |grep "\\$\\$"
> public final java.lang.Object
> org$apache$spark$rdd$PairRDDFunctions$$createZero$1(scala.reflect.ClassTag,
> byte[], scala.runtime.ObjectRef, scala.runtime.VolatileByteRef);
> public final java.lang.Object
> org$apache$spark$rdd$PairRDDFunctions$$createZero$2(byte[],
> scala.runtime.ObjectRef, scala.runtime.VolatileByteRef);
> public final scala.collection.Iterator
> org$apache$spark$rdd$PairRDDFunctions$$reducePartition$1(scala.collection.Iterator,
> scala.Function2);
> public final java.util.HashMap
> org$apache$spark$rdd$PairRDDFunctions$$mergeMaps$1(java.util.HashMap,
> java.util.HashMap, scala.Function2);
> ...
> public final class org.apache.spark.rdd.AsyncRDDActions$$anonfun$countAsync$1
> extends scala.runtime.AbstractFunction0$mcJ$sp implements scala.Serializable {
> public
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$countAsync$1(org.apache.spark.rdd.AsyncRDDActions<T>);
> public final class org.apache.spark.rdd.AsyncRDDActions$$anonfun$countAsync$2
> extends scala.runtime.AbstractFunction2$mcVIJ$sp implements
> scala.Serializable {
> public
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$countAsync$2(org.apache.spark.rdd.AsyncRDDActions<T>);
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
