Ali Alsuliman has submitted this change and it was merged. ( https://asterix-gerrit.ics.uci.edu/3398 )
Change subject: [ASTERIXDB-2567][RT] Fix serializing nullable record fields ...................................................................... [ASTERIXDB-2567][RT] Fix serializing nullable record fields - user model changes: no - storage format changes: no - interface changes: no Details: When serializing a record, a field should be checked if it's nullable. The value to be serialized should be checked if it's null or value to serialize it correctly. Exceptions could happen when the record is in an array if record fields are not checked. Change-Id: I42105965e69c60bc4b123337296476b4d75f0ad8 Reviewed-on: https://asterix-gerrit.ics.uci.edu/3398 Contrib: Jenkins <jenk...@fulliautomatix.ics.uci.edu> Tested-by: Jenkins <jenk...@fulliautomatix.ics.uci.edu> Reviewed-by: Murtadha Hubail <mhub...@apache.org> Integration-Tests: Jenkins <jenk...@fulliautomatix.ics.uci.edu> --- A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.1.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.2.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.3.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.5.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.6.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.7.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.8.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.1.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.2.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.3.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.4.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.5.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.6.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.7.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.8.adm M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml M asterixdb/asterix-om/src/main/java/org/apache/asterix/dataflow/data/nontagged/serde/ARecordSerializerDeserializer.java 18 files changed, 207 insertions(+), 9 deletions(-) Approvals: Jenkins: Verified; ; Verified Murtadha Hubail: Looks good to me, approved Anon. E. Moose (1000171): Objections: Jenkins: Violations found diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.1.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.1.query.sqlpp new file mode 100644 index 0000000..8d50408 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.1.query.sqlpp @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// testing fix for record serialization with nullable fields where the record is in an array and will be constant folded +[{"f1": "fieldStr", "f2": 1 < "text"}]; \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.2.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.2.query.sqlpp new file mode 100644 index 0000000..d2b017b --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.2.query.sqlpp @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// testing fix for record serialization with nullable fields where the record is in an array and will be constant folded +[{"f1": "fieldStr", "f2": "test" like null}]; \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.3.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.3.query.sqlpp new file mode 100644 index 0000000..4284b49b --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.3.query.sqlpp @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// testing fix for record serialization with nullable fields where the record is in an array and will be constant folded +FROM [{"f1": "fieldStr", "f2": 1 < "text"}] as v select value v; \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.4.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.4.query.sqlpp new file mode 100644 index 0000000..88b3077 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.4.query.sqlpp @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// testing fix for record serialization with nullable fields where the record is in an array and will be constant folded +FROM [{"f1": "fieldStr", "f2": 1 < "text"}, {"f1": "fieldStr2", "f2": 1 < 6}] as v select value v; \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.5.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.5.query.sqlpp new file mode 100644 index 0000000..b52bbb5 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.5.query.sqlpp @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// testing fix for record serialization with nullable fields where the record is in an array and will be constant folded +FROM [{"f1": "fieldStr", "f2": 1 < "text"}, {"f3": "fieldStr2", "f4": 1 < 6}] as v select value v; \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.6.query.sqlpp new file mode 100644 index 0000000..6cc5a16 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.6.query.sqlpp @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// testing fix for record serialization with nullable fields where the record is in an array and will be constant folded +FROM [{"f1": "fieldStr", "f2": 1 < "text"}, {"f2": 1 < 6, "f1": "fieldStr2"}] as v select value v; \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.7.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.7.query.sqlpp new file mode 100644 index 0000000..c278a21 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.7.query.sqlpp @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// testing fix for record serialization with nullable fields where the record is in an array and will be constant folded +FROM [{"f1": "fieldStr", "f2": 1 < "text"}, {"f1": "fieldStr2", "f2": "string field"}] as v select value v; \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.8.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.8.query.sqlpp new file mode 100644 index 0000000..e473447 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.8.query.sqlpp @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +// testing fix for record serialization with nullable fields where the record is in an array and will be constant folded +FROM [{"f1": "fieldStr", "f2": 1 < "text"}, {"f1": "fieldStr2", "f2": 1 < 6}, {"f1": "fieldStr3", "f2": missing < 6}] as v +select value v; \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.1.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.1.adm new file mode 100644 index 0000000..031c4f3 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.1.adm @@ -0,0 +1 @@ +[ { "f1": "fieldStr", "f2": null } ] \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.2.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.2.adm new file mode 100644 index 0000000..031c4f3 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.2.adm @@ -0,0 +1 @@ +[ { "f1": "fieldStr", "f2": null } ] \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.3.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.3.adm new file mode 100644 index 0000000..6deaa91 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.3.adm @@ -0,0 +1 @@ +{ "f1": "fieldStr", "f2": null } \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.4.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.4.adm new file mode 100644 index 0000000..16b5e5e --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.4.adm @@ -0,0 +1,2 @@ +{ "f1": "fieldStr", "f2": null } +{ "f1": "fieldStr2", "f2": true } \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.5.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.5.adm new file mode 100644 index 0000000..21ec9e4 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.5.adm @@ -0,0 +1,2 @@ +{ "f1": "fieldStr", "f2": null } +{ "f3": "fieldStr2", "f4": true } \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.6.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.6.adm new file mode 100644 index 0000000..ab5ba1b --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.6.adm @@ -0,0 +1,2 @@ +{ "f1": "fieldStr", "f2": null } +{ "f2": true, "f1": "fieldStr2" } \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.7.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.7.adm new file mode 100644 index 0000000..39d4120 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.7.adm @@ -0,0 +1,2 @@ +{ "f1": "fieldStr", "f2": null } +{ "f1": "fieldStr2", "f2": "string field" } \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.8.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.8.adm new file mode 100644 index 0000000..83bb027 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/record-serialization-ASTERIXDB-2567/record-serialization-ASTERIXDB-2567.8.adm @@ -0,0 +1,3 @@ +{ "f1": "fieldStr", "f2": null } +{ "f1": "fieldStr2", "f2": true } +{ "f1": "fieldStr3" } \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml index c9a5e37..45f86ec 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml +++ b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml @@ -5620,6 +5620,11 @@ </test-group> <test-group name="misc"> <test-case FilePath="misc"> + <compilation-unit name="record-serialization-ASTERIXDB-2567"> + <output-dir compare="Text">record-serialization-ASTERIXDB-2567</output-dir> + </compilation-unit> + </test-case> + <test-case FilePath="misc"> <compilation-unit name="comp-ASTERIXDB-2415"> <output-dir compare="Text">query-ASTERIXDB-1671</output-dir> </compilation-unit> diff --git a/asterixdb/asterix-om/src/main/java/org/apache/asterix/dataflow/data/nontagged/serde/ARecordSerializerDeserializer.java b/asterixdb/asterix-om/src/main/java/org/apache/asterix/dataflow/data/nontagged/serde/ARecordSerializerDeserializer.java index 0446e22..1d891e1 100644 --- a/asterixdb/asterix-om/src/main/java/org/apache/asterix/dataflow/data/nontagged/serde/ARecordSerializerDeserializer.java +++ b/asterixdb/asterix-om/src/main/java/org/apache/asterix/dataflow/data/nontagged/serde/ARecordSerializerDeserializer.java @@ -47,15 +47,15 @@ import org.apache.hyracks.util.string.UTF8StringUtil; public class ARecordSerializerDeserializer implements ISerializerDeserializer<ARecord> { + private static final long serialVersionUID = 1L; - + // TODO(ali): move PointableHelper to a lower package where this can see and reuse code from PointableHelper + private static final byte[] NULL_BYTES = new byte[] { ATypeTag.SERIALIZED_NULL_TYPE_TAG }; + private static final byte[] MISSING_BYTES = new byte[] { ATypeTag.SERIALIZED_MISSING_TYPE_TAG }; public static final ARecordSerializerDeserializer SCHEMALESS_INSTANCE = new ARecordSerializerDeserializer(); - private static final IAObject[] NO_FIELDS = new IAObject[0]; - private final ARecordType recordType; private final int numberOfSchemaFields; - @SuppressWarnings("rawtypes") private final ISerializerDeserializer[] serializers; @SuppressWarnings("rawtypes") @@ -156,6 +156,7 @@ } IAObject[] schemaFields = new IAObject[numberOfSchemaFields]; for (int fieldId = 0; fieldId < numberOfSchemaFields; fieldId++) { + // TODO: null/missing formula is duplicated across the codebase. should be in a central place. if (hasOptionalFields && ((nullBitMap[fieldId / 4] & (1 << (7 - 2 * (fieldId % 4)))) == 0)) { schemaFields[fieldId] = ANull.NULL; } else if (hasOptionalFields && ((nullBitMap[fieldId / 4] & (1 << (7 - 2 * (fieldId % 4) - 1))) == 0)) { @@ -180,10 +181,19 @@ recordBuilder.reset(recordType); recordBuilder.init(); if (recordType != null) { - int fieldIndex = 0; - for (; fieldIndex < recordType.getFieldNames().length; ++fieldIndex) { + IAType[] fieldTypes = recordType.getFieldTypes(); + for (int fieldIndex = 0; fieldIndex < recordType.getFieldNames().length; ++fieldIndex) { fieldValue.reset(); - serializers[fieldIndex].serialize(instance.getValueByPos(fieldIndex), fieldValue.getDataOutput()); + IAObject value = instance.getValueByPos(fieldIndex); + ATypeTag valueTag = value.getType().getTypeTag(); + boolean fieldIsOptional = NonTaggedFormatUtil.isOptional(fieldTypes[fieldIndex]); + if (fieldIsOptional && valueTag == ATypeTag.NULL) { + fieldValue.set(NULL_BYTES, 0, NULL_BYTES.length); + } else if (fieldIsOptional && valueTag == ATypeTag.MISSING) { + fieldValue.set(MISSING_BYTES, 0, MISSING_BYTES.length); + } else { + serializers[fieldIndex].serialize(value, fieldValue.getDataOutput()); + } recordBuilder.addField(fieldIndex, fieldValue); } recordBuilder.write(out, writeTypeTag); @@ -193,7 +203,7 @@ } @SuppressWarnings({ "unchecked", "rawtypes" }) - public static void serializeSchemalessRecord(ARecord record, DataOutput dataOutput, boolean writeTypeTag) + private static void serializeSchemalessRecord(ARecord record, DataOutput dataOutput, boolean writeTypeTag) throws HyracksDataException { ISerializerDeserializer<AString> stringSerde = SerializerDeserializerProvider.INSTANCE.getSerializerDeserializer(BuiltinType.ASTRING); @@ -262,7 +272,7 @@ return new ARecordType(null, fieldNames, fieldTypes, true); } - public static final int getRecordLength(byte[] serRecord, int offset) { + public static int getRecordLength(byte[] serRecord, int offset) { return AInt32SerializerDeserializer.getInt(serRecord, offset); } -- To view, visit https://asterix-gerrit.ics.uci.edu/3398 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I42105965e69c60bc4b123337296476b4d75f0ad8 Gerrit-Change-Number: 3398 Gerrit-PatchSet: 5 Gerrit-Owner: Ali Alsuliman <ali.al.solai...@gmail.com> Gerrit-Reviewer: Ali Alsuliman <ali.al.solai...@gmail.com> Gerrit-Reviewer: Anon. E. Moose (1000171) Gerrit-Reviewer: Dmitry Lychagin <dmitry.lycha...@couchbase.com> Gerrit-Reviewer: Jenkins <jenk...@fulliautomatix.ics.uci.edu> Gerrit-Reviewer: Michael Blow <mb...@apache.org> Gerrit-Reviewer: Murtadha Hubail <mhub...@apache.org> Gerrit-Reviewer: Till Westmann <ti...@apache.org>
