ShuDaoNan created SPARK-35217:
---------------------------------
Summary: com.google.protobuf.Parser.parseFrom() method Can't use
in spark
Key: SPARK-35217
URL: https://issues.apache.org/jira/browse/SPARK-35217
Project: Spark
Issue Type: Bug
Components: Deploy, ML
Affects Versions: 2.4.7
Environment: *platform：*
Linux ip-172-17-1-1 4.14.219-164.354.amzn2.x86_64
[#1|https://github.com/apache/spark/pull/1] SMP Mon Feb 22 21:18:39 UTC 2021
x86_64 x86_64 x86_64 GNU/Linux
Hive 2.3.7, ZooKeeper 3.4.14, Spark 2.4.7, TensorFlow 2.4.1
(Windows10 do not have this problem.)
Reporter: ShuDaoNan
*platform：*
Linux ip-172-17-1-1 4.14.219-164.354.amzn2.x86_64
[#1|https://github.com/apache/spark/pull/1] SMP Mon Feb 22 21:18:39 UTC 2021
x86_64 x86_64 x86_64 GNU/Linux
Hive 2.3.7, ZooKeeper 3.4.14, Spark 2.4.7, TensorFlow 2.4.1
(Windows10 do not have this problem.)
*problem：*
Can not load TensorFlow model(Java or scala):
{code:java}
[hadoop@ip-172-17-1-1 ~]$ spark-shell --master local[2] --jars
s3://jars/jasypt-1.9.2.jar,s3://jars/commons-pool2-2.0.jar,s3://jars/tensorflow-core-api-0.3.1.jar,s3://jars/tensorflow-core-api-0.3.1-linux-x86_64-mkl.jar,s3://jars/ndarray-0.3.1.jar,s3://jars/javacpp-1.5.4.jar,s3://jars/tensorflow-core-platform-0.3.1.jar,s3://jars/protobuf-java-3.8.0.jar
Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.7-amzn-1 /_/ Using
Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_282) Type in
expressions to have them evaluated. Type :help for more information. scala>
import org.{tensorflow => tf} import org.{tensorflow=>tf} scala> val bundle =
tf.SavedModelBundle.load("/home/hadoop/xDeepFM","serve") 2021-04-23
07:32:56.223881: I
external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading
SavedModel from: /home/hadoop/xDeepFM 2021-04-23 07:32:56.266424: I
external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:55] Reading meta
graph with tags { serve } 2021-04-23 07:32:56.266468: I
external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:93] Reading
SavedModel debug info (if present) from: /home/hadoop/xDeepFM 2021-04-23
07:32:56.346757: I
external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:206] Restoring
SavedModel bundle. 2021-04-23 07:32:56.873838: I
external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running
initialization op on SavedModel bundle at path: /home/hadoop/xDeepFM 2021-04-23
07:32:56.928656: I
external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel
load for tags { serve }; Status: success: OK. Took 704788 microseconds.
java.lang.NoSuchMethodError:
com.google.protobuf.Parser.parseFrom(Ljava/nio/ByteBuffer;)Ljava/lang/Object;
at
org.tensorflow.proto.framework.MetaGraphDef.parseFrom(MetaGraphDef.java:3067)
at org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:422) at
org.tensorflow.SavedModelBundle.access$000(SavedModelBundle.java:59) at
org.tensorflow.SavedModelBundle$Loader.load(SavedModelBundle.java:68) at
org.tensorflow.SavedModelBundle.load(SavedModelBundle.java:242) ... 49 elided
scala>
{code}
*But I can load the model in a single 'Testtf.java' like that:*
*{{}}*
{code:java}
[hadoop@ip-172-17-1-1 ~]$ vi Testtf.java
import org.tensorflow.SavedModelBundle;
public class Testtf {
public static void main(String[] args) {
System.out.println("test load...");
SavedModelBundle bundle =
org.tensorflow.SavedModelBundle.load("/home/hadoop/xDeepFM","serve");
System.out.println("loaded bundle...");
System.out.println(bundle);
}
}
[hadoop@ip-172-17-1-1 ~]$ javac -cp
javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.3.1-linux-x86_64.jar:tensorflow-core-api-0.3.1.jar:ndarray-0.3.1.jar:javacpp-1.5.4.jar:tensorflow-core-platform-0.3.1.jar:tensorflow-core-platform-mkl-0.3.1.jar:protobuf-java-3.8.0.jar
Testtf.java
[hadoop@ip-172-17-1-1 ~]$ java -cp
javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.3.1-linux-x86_64.jar:tensorflow-core-api-0.3.1.jar:ndarray-0.3.1.jar:javacpp-1.5.4.jar:tensorflow-core-platform-0.3.1.jar:tensorflow-core-platform-mkl-0.3.1.jar:protobuf-java-3.8.0.jar:.
Testtf
test load...
2021-04-25 02:33:47.247120: I
external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:32] Reading
SavedModel from: /home/hadoop/xDeepFM
2021-04-25 02:33:47.275349: I
external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:55] Reading meta
graph with tags { serve }
2021-04-25 02:33:47.275405: I
external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:93] Reading
SavedModel debug info (if present) from: /home/hadoop/xDeepFM
2021-04-25 02:33:47.275480: I
external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] This
TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)
to use the following CPU instructions in performance-critical operations: AVX2
FMA
To enable them in other operations, rebuild TensorFlow with the appropriate
compiler flags.
2021-04-25 02:33:47.370208: I
external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:206] Restoring
SavedModel bundle.
2021-04-25 02:33:47.381656: I
external/org_tensorflow/tensorflow/core/platform/profile_utils/cpu_utils.cc:112]
CPU Frequency: 2300065000 Hz
2021-04-25 02:33:47.667941: I
external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running
initialization op on SavedModel bundle at path: /home/hadoop/xDeepFM
2021-04-25 02:33:47.725458: I
external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel
load for tags { serve }; Status: success: OK. Took 478354 microseconds.
loaded bundle...
org.tensorflow.SavedModelBundle@80503
[hadoop@ip-172-17-1-1 ~]$ {code}
*{{}}*
*Thanks!*
refer: [tensorflow/java#298|https://github.com/tensorflow/java/issues/298]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
