libenchao commented on code in PR #23162:
URL: https://github.com/apache/flink/pull/23162#discussion_r1390600656
##########
flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/serialize/RowToProtoConverter.java:
##########
@@ -84,11 +85,18 @@ public RowToProtoConverter(RowType rowType, PbFormatConfig
formatConfig)
PbCodegenSerializer codegenSer =
PbCodegenSerializeFactory.getPbCodegenTopRowSer(
descriptor, rowType, formatContext);
+ LOG.info("Fast-pb generate split serialize code");
Review Comment:
remove unnecessary log.
##########
flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/serialize/RowToProtoConverter.java:
##########
@@ -109,4 +117,8 @@ public byte[] convertRowToProtoBinary(RowData rowData)
throws Exception {
AbstractMessage message = (AbstractMessage) encodeMethod.invoke(null,
rowData);
return message.toByteArray();
}
+
+ public boolean isCodeSplit() {
Review Comment:
for testing.
##########
flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/deserialize/ProtoToRowConverter.java:
##########
@@ -104,10 +106,19 @@ public ProtoToRowConverter(RowType rowType,
PbFormatConfig formatConfig)
PbCodegenDeserializer codegenDes =
PbCodegenDeserializeFactory.getPbCodegenTopRowDes(
descriptor, rowType, pbFormatContext);
- String genCode = codegenDes.codegen("rowData", "message", 0);
+ // if codgen generate code size over threshod then split the code
+ PbCodeSplitter pbCodeSplitter = new PbCodeSplitter();
+ LOG.info("Fast-pb generate split deserialize code");
Review Comment:
ping, it seems you are missing this one.
##########
flink-formats/flink-protobuf/src/test/proto/test_big_pb.proto:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+
+package org.apache.flink.formats.protobuf.testproto;
+
+option java_package = "org.apache.flink.formats.protobuf.testproto";
+option java_outer_classname = "BigPbClass";
+
+message BigPbMessage {
+ int32 int_field_1 = 1;
+ bool bool_field_2 = 2;
+ string string_field_3 = 3;
+ bytes bytes_field_4 = 4;
+ double double_field_5 = 5;
+ float float_field_6 = 6;
+ uint32 uint32_field_7 = 7;
+ int64 int64_field_8 = 8;
+ uint64 uint64_field_9 = 9;
+ bytes bytes_field_10 = 10;
+ double double_field_11 = 11;
+ bytes bytes_field_12 = 12;
+ bool bool_field_13 = 13;
+ string string_field_14 = 14;
+ float float_field_15 = 15;
+ int32 int32_field_16 = 16;
+ bytes bytes_field_17 = 17;
+ bool bool_field_18 = 18;
+ string string_field_19 = 19;
+ float float_field_20 = 20;
+ fixed32 fixed32_field_21 = 21;
+ fixed64 fixed64_field_22 = 22;
+ sfixed32 sfixed32_field_23 = 23;
+ sfixed64 sfixed64_field_24 = 24;
+ double double_field_25 = 25;
+ uint32 uint32_field_26 = 26;
+ uint64 uint64_field_27 = 27;
+ bool bool_field_28 = 28;
+ repeated string field_29 = 29;
+ float float_field_30 = 30;
+ string string_field_31 = 31;
+
+ map<string, bytes> map_field_32 = 32;
+ map<string, string> map_field_33 = 33;
+}
Review Comment:
Always put a `\n` in the final line, then it will not complain that "Now new
line at the end of file".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
