This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fory.git
The following commit(s) were added to refs/heads/main by this push:
new 80784a79d feat(go): Implement compatible mode with metashare mode
(#2607)
80784a79d is described below
commit 80784a79da01af3257f1e63f666b24e1470cc830
Author: Zhong Junjie <[email protected]>
AuthorDate: Tue Sep 16 22:05:44 2025 +0800
feat(go): Implement compatible mode with metashare mode (#2607)
## Why?
Type-forward and backward-compatible serialization are crucial for
online services where different services update their data schemas and
deploy at different times. This ensures that a service running an older
schema can still communicate with a service using a newer one, and vice
versa.This PR is part of the work of it. The previous PR is #2554 , and
following pr are still work in process.
## What does this PR do?
This PR implements a metashare mode for the fory-go.
Specifically, it:
Provides a user option to enable compatible mode.
If compatible mode is enabled, the serialization process not only
serializes the data but also its detailed type information into a binary
format.
The reading peer, if compatible mode is enabled, will first attempt to
read this type information before deserializing the data. This allows it
to handle data serialized with a different schema.
This implementation is heavily inspired by the existing Java and Python
implementations, ensuring consistency across framework.
All progress and todos:
- [x] metadata encoding and decoding
- [x] Integrate into the main flow to enable metadata sharing
- [ ] Add support for collection types (map, slice, set, etc.)
- [ ] Add support for compressed type definitions
- [ ] Test with python
## Related issues
#2192
## Does this PR introduce any user-facing change?
<!--
If any user-facing interface changes, please [open an
issue](https://github.com/apache/fory/issues/new/choose) describing the
need to do so and update the document if necessary.
Delete section if not applicable.
-->
- [x] Does this PR introduce any public API change? Yes.
- [x] Does this PR introduce any binary protocol compatibility change?
No.
## Benchmark
<!--
When the PR has an impact on performance (if you don't know whether the
PR will have an impact on performance, you can submit the PR first, and
if it will have impact on performance, the code reviewer will explain
it), be sure to attach a benchmark data here.
Delete section if not applicable.
-->
---
.gitignore | 1 +
go/README.md | 32 +++++++
go/fory/context.go | 72 ++++++++++++++++
go/fory/fory.go | 131 ++++++++++++++++++++++++----
go/fory/fory_metashare_test.go | 181 +++++++++++++++++++++++++++++++++++++++
go/fory/set.go | 2 +-
go/fory/slice.go | 4 +-
go/fory/struct.go | 81 ++++++++++++++++--
go/fory/type.go | 110 ++++++++++++++++++++++--
go/fory/type_def.go | 68 ++++++++++++++-
go/fory/type_def_decoder.go | 13 ++-
go/fory/type_def_encoder_test.go | 2 +-
go/fory/type_test.go | 8 +-
13 files changed, 662 insertions(+), 43 deletions(-)
diff --git a/.gitignore b/.gitignore
index c1af3a735..ca557d35e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -40,3 +40,4 @@ MODULE.bazel
MODULE.bazel.lock
.DS_Store
**/.DS_Store
+.vscode/
diff --git a/go/README.md b/go/README.md
index d4e9ca894..7dd44f912 100644
--- a/go/README.md
+++ b/go/README.md
@@ -146,6 +146,38 @@ fory --force -file <your file>
- Does generated code work across Go versions? Yes, it’s plain Go code; keep
your toolchain consistent in CI.
- Can I mix generated and non-generated structs? Yes, adoption is incremental
and per file.
+## Configuration Options
+
+Fory Go supports several configuration options through the functional options
pattern:
+
+### Compatible Mode (Metashare)
+
+Compatible mode enables meta information sharing, which allows for schema
evolution:
+
+```go
+// Enable compatible mode with metashare
+fory := NewForyWithOptions(WithCompatible(true))
+```
+
+### Reference Tracking
+
+Enable reference tracking:
+
+```go
+fory := NewForyWithOptions(WithRefTracking(true))
+```
+
+### Combined Options
+
+You can combine multiple options:
+
+```go
+fory := NewForyWithOptions(
+ WithCompatible(true),
+ WithRefTracking(true),
+)
+```
+
## How to test
```bash
diff --git a/go/fory/context.go b/go/fory/context.go
new file mode 100644
index 000000000..69ac98822
--- /dev/null
+++ b/go/fory/context.go
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package fory
+
+import "reflect"
+
+// MetaContext used to share data across multiple serialization calls
+type MetaContext struct {
+ // typeMap make sure each type just fully serialize once, the following
serialization will use the index
+ typeMap map[reflect.Type]uint32
+ // record typeDefs need to be serialized during one serialization
+ writingTypeDefs []*TypeDef
+ // read from peer
+ readTypeInfos []TypeInfo
+ // scopedMetaShareEnabled controls whether meta sharing is scoped to
single serialization
+ scopedMetaShareEnabled bool
+}
+
+// NewMetaContext creates a new MetaContext
+func NewMetaContext(scopedMetaShareEnabled bool) *MetaContext {
+ return &MetaContext{
+ typeMap: make(map[reflect.Type]uint32),
+ scopedMetaShareEnabled: scopedMetaShareEnabled,
+ }
+}
+
+// resetRead resets the read-related state of the MetaContext
+func (mc *MetaContext) resetRead() {
+ if mc.scopedMetaShareEnabled {
+ mc.readTypeInfos = mc.readTypeInfos[:0] // Reset slice but keep
capacity
+ } else {
+ mc.readTypeInfos = nil
+ }
+}
+
+// resetWrite resets the write-related state of the MetaContext
+func (mc *MetaContext) resetWrite() {
+ if mc.scopedMetaShareEnabled {
+ for k := range mc.typeMap {
+ delete(mc.typeMap, k)
+ }
+ mc.writingTypeDefs = mc.writingTypeDefs[:0] // Reset slice but
keep capacity
+ } else {
+ mc.typeMap = nil
+ mc.writingTypeDefs = nil
+ }
+}
+
+// SetScopedMetaShareEnabled sets the scoped meta share mode
+func (mc *MetaContext) SetScopedMetaShareEnabled(enabled bool) {
+ mc.scopedMetaShareEnabled = enabled
+}
+
+// IsScopedMetaShareEnabled returns whether scoped meta sharing is enabled
+func (mc *MetaContext) IsScopedMetaShareEnabled() bool {
+ return mc.scopedMetaShareEnabled
+}
diff --git a/go/fory/fory.go b/go/fory/fory.go
index becca516d..fc3b9eab3 100644
--- a/go/fory/fory.go
+++ b/go/fory/fory.go
@@ -24,15 +24,72 @@ import (
"sync"
)
-func NewFory(referenceTracking bool) *Fory {
+// Option represents a configuration option for Fory instances.
+// This follows the functional options pattern, allowing flexible configuration
+// by passing variadic option functions to constructors like
NewForyWithOptions.
+type Option func(*Fory)
+
+// WithCompatible sets the compatible mode for the Fory instance.
+// When compatible=true, scoped meta sharing is automatically enabled.
+func WithCompatible(compatible bool) Option {
+ return func(f *Fory) {
+ f.compatible = compatible
+ if compatible {
+ f.metaContext = NewMetaContext(true) // Enable scoped
meta sharing
+ } else {
+ f.metaContext = nil
+ }
+ }
+}
+
+// WithRefTracking sets the reference tracking mode for the Fory instance
+func WithRefTracking(refTracking bool) Option {
+ return func(f *Fory) {
+ f.refTracking = refTracking
+ }
+}
+
+// WithScopedMetaShare sets the scoped meta share mode for the Fory instance.
+// Note: Compatible mode automatically enables scoped meta sharing.
+// This option is mainly used for fine-grained control when compatible mode is
already enabled.
+func WithScopedMetaShare(enabled bool) Option {
+ return func(f *Fory) {
+ if f.metaContext == nil {
+ f.metaContext = NewMetaContext(enabled)
+ } else {
+ f.metaContext.SetScopedMetaShareEnabled(enabled)
+ }
+ }
+}
+
+func NewFory(refTracking bool) *Fory {
+ return NewForyWithOptions(WithRefTracking(refTracking))
+}
+
+// NewForyWithOptions creates a Fory instance with configurable options
+func NewForyWithOptions(options ...Option) *Fory {
fory := &Fory{
- refResolver: newRefResolver(referenceTracking),
- referenceTracking: referenceTracking,
- language: XLANG,
- buffer: NewByteBuffer(nil),
+ refResolver: nil,
+ refTracking: false,
+ language: XLANG,
+ buffer: NewByteBuffer(nil),
+ compatible: false,
}
- // Create a new type resolver for this instance
+
+ // Apply options
+ for _, option := range options {
+ option(fory)
+ }
+
+ // Create a new type resolver for this instance but copy generated
serializers from global resolver
fory.typeResolver = newTypeResolver(fory)
+
+ // Initialize meta context if compatible mode is enabled
+ if fory.compatible {
+ fory.metaContext = NewMetaContext(true)
+ }
+
+ fory.refResolver = newRefResolver(fory.refTracking)
return fory
}
@@ -105,14 +162,16 @@ const (
const MAGIC_NUMBER int16 = 0x62D4
type Fory struct {
- typeResolver *typeResolver
- refResolver *RefResolver
- referenceTracking bool
- language Language
- bufferCallback BufferCallback
- peerLanguage Language
- buffer *ByteBuffer
- buffers []*ByteBuffer
+ typeResolver *typeResolver
+ refResolver *RefResolver
+ refTracking bool
+ language Language
+ bufferCallback BufferCallback
+ peerLanguage Language
+ buffer *ByteBuffer
+ buffers []*ByteBuffer
+ compatible bool
+ metaContext *MetaContext
}
func (f *Fory) RegisterTagType(tag string, v interface{}) error {
@@ -246,7 +305,18 @@ func (f *Fory) readLength(buffer *ByteBuffer) int {
}
func (f *Fory) WriteReferencable(buffer *ByteBuffer, value reflect.Value)
error {
- return f.writeReferencableBySerializer(buffer, value, nil)
+ metaOffset := buffer.writerIndex
+ if f.compatible {
+ buffer.WriteInt32(-1)
+ }
+ if err := f.writeReferencableBySerializer(buffer, value, nil); err !=
nil {
+ return err
+ }
+ if f.compatible && f.metaContext != nil &&
len(f.metaContext.writingTypeDefs) > 0 {
+ buffer.PutInt32(metaOffset,
int32(buffer.writerIndex-metaOffset-4))
+ f.typeResolver.writeTypeDefs(buffer)
+ }
+ return nil
}
func (f *Fory) writeReferencableBySerializer(buffer *ByteBuffer, value
reflect.Value, serializer Serializer) error {
@@ -367,6 +437,19 @@ func (f *Fory) Deserialize(buf *ByteBuffer, v interface{},
buffers []*ByteBuffer
"produced with buffer_callback null")
}
}
+
+ if f.compatible {
+ typeDefOffset := buf.ReadInt32()
+ if typeDefOffset >= 0 {
+ save := buf.readerIndex
+ buf.SetReaderIndex(save + int(typeDefOffset))
+ if err := f.typeResolver.readTypeDefs(buf); err != nil {
+ return fmt.Errorf("failed to read typeDefs:
%w", err)
+ }
+ buf.SetReaderIndex(save)
+ }
+ }
+
if isXLangFlag {
return f.ReadReferencable(buf, reflect.ValueOf(v).Elem())
} else {
@@ -475,11 +558,25 @@ func (f *Fory) Reset() {
func (f *Fory) resetWrite() {
f.typeResolver.resetWrite()
f.refResolver.resetWrite()
+ if f.metaContext != nil {
+ if f.metaContext.IsScopedMetaShareEnabled() {
+ f.metaContext.resetWrite()
+ } else {
+ f.metaContext = nil
+ }
+ }
}
func (f *Fory) resetRead() {
f.typeResolver.resetRead()
f.refResolver.resetRead()
+ if f.metaContext != nil {
+ if f.metaContext.IsScopedMetaShareEnabled() {
+ f.metaContext.resetRead()
+ } else {
+ f.metaContext = nil
+ }
+ }
}
// methods for configure fory.
@@ -488,6 +585,6 @@ func (f *Fory) SetLanguage(language Language) {
f.language = language
}
-func (f *Fory) SetReferenceTracking(referenceTracking bool) {
- f.referenceTracking = referenceTracking
+func (f *Fory) SetRefTracking(refTracking bool) {
+ f.refTracking = refTracking
}
diff --git a/go/fory/fory_metashare_test.go b/go/fory/fory_metashare_test.go
new file mode 100644
index 000000000..cc71c2639
--- /dev/null
+++ b/go/fory/fory_metashare_test.go
@@ -0,0 +1,181 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package fory
+
+import (
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+)
+
+// Test structs for metashare testing
+
+type SimpleDataClass struct {
+ Name string
+ Age int32
+ Active bool
+}
+type ExtendedDataClass struct {
+ Name string
+ Age int32
+ Active bool
+ Email string // Additional field
+}
+
+type ReducedDataClass struct {
+ Name string
+ Age int32
+ // Missing 'active' field
+}
+
+func TestMetaShareEnabled(t *testing.T) {
+ fory := NewForyWithOptions(WithCompatible(true))
+
+ assert.True(t, fory.compatible, "Expected compatible mode to be
enabled")
+ assert.NotNil(t, fory.metaContext, "Expected metaContext to be
initialized when compatible=true")
+ assert.True(t, fory.metaContext.IsScopedMetaShareEnabled(), "Expected
scoped meta share to be enabled by default when compatible=true")
+}
+
+func TestMetaShareDisabled(t *testing.T) {
+ fory := NewForyWithOptions(WithCompatible(false))
+
+ assert.False(t, fory.compatible, "Expected compatible mode to be
disabled")
+ assert.Nil(t, fory.metaContext, "Expected metaContext to be nil when
compatible=false")
+}
+
+func TestSimpleDataClassSerialization(t *testing.T) {
+ fory := NewForyWithOptions(WithCompatible(true))
+
+ // Register the struct
+ err := fory.RegisterTagType("SimpleDataClass", SimpleDataClass{})
+ assert.NoError(t, err, "Failed to register type")
+
+ obj := SimpleDataClass{Name: "test", Age: 25, Active: true}
+
+ // Serialize
+ data, err := fory.Marshal(obj)
+ assert.NoError(t, err, "Failed to marshal")
+
+ // Deserialize
+ var deserialized SimpleDataClass
+ err = fory.Unmarshal(data, &deserialized)
+ assert.NoError(t, err, "Failed to unmarshal")
+
+ // Verify
+ assert.Equal(t, obj.Name, deserialized.Name)
+ assert.Equal(t, obj.Age, deserialized.Age)
+ assert.Equal(t, obj.Active, deserialized.Active)
+}
+
+func TestFieldSortingOrder(t *testing.T) {
+ fory := NewForyWithOptions(WithCompatible(true))
+
+ // Create a struct with fields in non-optimal order (only implemented
types)
+ // the final order should be: FloatField, IntField, BoolField,
ByteField, StringField
+ type UnsortedStruct struct {
+ StringField string
+ FloatField float64
+ BoolField bool
+ IntField int32
+ ByteField byte
+ }
+
+ err := fory.RegisterTagType("UnsortedStruct", UnsortedStruct{})
+ assert.NoError(t, err, "Failed to register type")
+
+ obj := UnsortedStruct{
+ StringField: "test",
+ FloatField: 3.14,
+ BoolField: true,
+ IntField: 42,
+ ByteField: 255,
+ }
+
+ // Serialize
+ data, err := fory.Marshal(obj)
+ assert.NoError(t, err, "Failed to marshal")
+
+ // Deserialize
+ var deserialized UnsortedStruct
+ err = fory.Unmarshal(data, &deserialized)
+ assert.NoError(t, err, "Failed to unmarshal")
+
+ // Verify all fields are correctly serialized/deserialized regardless
of order
+ assert.Equal(t, obj.FloatField, deserialized.FloatField)
+ assert.Equal(t, obj.IntField, deserialized.IntField)
+ assert.Equal(t, obj.BoolField, deserialized.BoolField)
+ assert.Equal(t, obj.ByteField, deserialized.ByteField)
+ assert.Equal(t, obj.StringField, deserialized.StringField)
+
+ t.Logf("Field sorting test passed - optimal order is applied during
field definition creation")
+}
+
+func TestSchemaEvolutionAddField(t *testing.T) {
+ // Test adding fields to existing struct using predefined types
+
+ // Serialize with SimpleDataClass (3 fields)
+ fory1 := NewForyWithOptions(WithCompatible(true))
+ err := fory1.RegisterTagType("TestStruct", SimpleDataClass{})
+ assert.NoError(t, err, "Failed to register SimpleDataClass")
+
+ originalObj := SimpleDataClass{Name: "test", Age: 25, Active: true}
+ data, err := fory1.Marshal(originalObj)
+ assert.NoError(t, err, "Failed to marshal SimpleDataClass")
+
+ // Deserialize with ExtendedDataClass (4 fields - adds Email field)
+ fory2 := NewForyWithOptions(WithCompatible(true))
+ err = fory2.RegisterTagType("TestStruct", ExtendedDataClass{})
+ assert.NoError(t, err, "Failed to register ExtendedDataClass")
+
+ var deserialized ExtendedDataClass
+ err = fory2.Unmarshal(data, &deserialized)
+ assert.NoError(t, err, "Failed to unmarshal to ExtendedDataClass")
+
+ // Verify common fields and default value for new field
+ assert.Equal(t, originalObj.Name, deserialized.Name)
+ assert.Equal(t, originalObj.Age, deserialized.Age)
+ assert.Equal(t, originalObj.Active, deserialized.Active)
+ assert.Equal(t, "", deserialized.Email, "Expected Email to be its
default value (empty string)")
+}
+
+func TestSchemaEvolutionRemoveField(t *testing.T) {
+ // Test removing fields from existing struct using predefined types
+
+ // Serialize with SimpleDataClass (3 fields)
+ fory1 := NewForyWithOptions(WithCompatible(true))
+ err := fory1.RegisterTagType("TestStruct", SimpleDataClass{})
+ assert.NoError(t, err, "Failed to register SimpleDataClass")
+
+ originalObj := SimpleDataClass{Name: "test", Age: 25, Active: true}
+ data, err := fory1.Marshal(originalObj)
+ assert.NoError(t, err, "Failed to marshal SimpleDataClass")
+
+ // Deserialize with ReducedDataClass (2 fields - removes Active)
+ fory2 := NewForyWithOptions(WithCompatible(true))
+ err = fory2.RegisterTagType("TestStruct", ReducedDataClass{})
+ assert.NoError(t, err, "Failed to register ReducedDataClass")
+
+ var deserialized ReducedDataClass
+ err = fory2.Unmarshal(data, &deserialized)
+ assert.NoError(t, err, "Failed to unmarshal to ReducedDataClass")
+
+ // Verify common fields
+ assert.Equal(t, originalObj.Name, deserialized.Name)
+ assert.Equal(t, originalObj.Age, deserialized.Age)
+ // Active field is removed, so it should not be present in deserialized
ReducedDataClass
+}
diff --git a/go/fory/set.go b/go/fory/set.go
index 0b3838c66..d64982b6a 100644
--- a/go/fory/set.go
+++ b/go/fory/set.go
@@ -110,7 +110,7 @@ func (s setSerializer) writeHeader(f *Fory, buf
*ByteBuffer, keys []reflect.Valu
}
// Enable reference tracking if configured
- if f.referenceTracking {
+ if f.refTracking {
collectFlag |= CollectionTrackingRef
}
diff --git a/go/fory/slice.go b/go/fory/slice.go
index 6a171cdc4..ef3810abd 100644
--- a/go/fory/slice.go
+++ b/go/fory/slice.go
@@ -102,7 +102,7 @@ func (s sliceSerializer) writeHeader(f *Fory, buf
*ByteBuffer, value reflect.Val
}
// Enable reference tracking if configured
- if f.referenceTracking {
+ if f.refTracking {
collectFlag |= CollectionTrackingRef
}
@@ -1093,7 +1093,7 @@ func (s stringSliceSerializer) Read(f *Fory, buf
*ByteBuffer, type_ reflect.Type
}
}
elem := readString(buf)
- if f.referenceTracking && refFlag == RefValueFlag {
+ if f.refTracking && refFlag == RefValueFlag {
// If value is not nil(reflect), then value is
a pointer to some variable, we can update the `value`,
// then record `value` in the reference
resolver.
f.refResolver.SetReadObject(nextReadRefId,
reflect.ValueOf(elem))
diff --git a/go/fory/struct.go b/go/fory/struct.go
index f65216005..2fd679a35 100644
--- a/go/fory/struct.go
+++ b/go/fory/struct.go
@@ -30,6 +30,7 @@ type structSerializer struct {
type_ reflect.Type
fieldsInfo structFieldsInfo
structHash int32
+ fieldDefs []FieldDef // defs obtained during reading
}
var UNKNOWN_TYPE_ID = int16(-1)
@@ -86,10 +87,20 @@ func (s *structSerializer) Read(f *Fory, buf *ByteBuffer,
type_ reflect.Type, va
value = value.Elem()
}
if s.fieldsInfo == nil {
- if infos, err := createStructFieldInfos(f, s.type_); err != nil
{
- return err
+ if len(s.fieldDefs) == 0 {
+ // Normal case: create from reflection
+ if infos, err := createStructFieldInfos(f, s.type_);
err != nil {
+ return err
+ } else {
+ s.fieldsInfo = infos
+ }
} else {
- s.fieldsInfo = infos
+ // Create from fieldDefs for forward/backward
compatibility
+ if infos, err := createStructFieldInfosFromFieldDefs(f,
s.fieldDefs, type_); err != nil {
+ return err
+ } else {
+ s.fieldsInfo = infos
+ }
}
}
if s.structHash == 0 {
@@ -100,12 +111,21 @@ func (s *structSerializer) Read(f *Fory, buf *ByteBuffer,
type_ reflect.Type, va
}
}
structHash := buf.ReadInt32()
- if structHash != s.structHash {
+ if !f.compatible && structHash != s.structHash {
return fmt.Errorf("hash %d is not consistent with %d for type
%s",
structHash, s.structHash, s.type_)
}
for _, fieldInfo_ := range s.fieldsInfo {
- fieldValue := value.Field(fieldInfo_.fieldIndex)
+ var fieldValue reflect.Value
+
+ if fieldInfo_.fieldIndex >= 0 {
+ // Field exists in current struct version
+ fieldValue = value.Field(fieldInfo_.fieldIndex)
+ } else {
+ // Field doesn't exist in current struct version,
create temporary value to discard
+ fieldValue = reflect.New(fieldInfo_.type_).Elem()
+ }
+
fieldSerializer := fieldInfo_.serializer
if fieldSerializer != nil {
if err := readBySerializer(f, buf, fieldValue,
fieldSerializer, fieldInfo_.referencable); err != nil {
@@ -204,6 +224,57 @@ func createStructFieldInfos(f *Fory, type_ reflect.Type)
(structFieldsInfo, erro
return fields, nil
}
+// createStructFieldInfosFromFieldDefs creates structFieldsInfo from fieldDefs
and builds field name mapping
+func createStructFieldInfosFromFieldDefs(f *Fory, fieldDefs []FieldDef, type_
reflect.Type) (structFieldsInfo, error) {
+ fieldNameToIndex := make(map[string]int)
+
+ // Build map from field names to struct field indices for quick lookup
+ for i := 0; i < type_.NumField(); i++ {
+ field := type_.Field(i)
+ fieldName := SnakeCase(field.Name)
+ fieldNameToIndex[fieldName] = i
+ }
+
+ var fields structFieldsInfo
+ current_field_names := make(map[string]int)
+
+ for i, def := range fieldDefs {
+ current_field_names[def.name] = i
+
+ fieldIndex := -1 // Default to -1 if field doesn't exist in
current struct
+ var fieldType reflect.Type
+
+ if structFieldIndex, exists := fieldNameToIndex[def.name];
exists {
+ fieldIndex = structFieldIndex
+ fieldType = type_.Field(structFieldIndex).Type
+ } else {
+ // Field doesn't exist in current struct version, we
need the type from FieldDef
+ if info, exists :=
f.typeResolver.typeIDToTypeInfo[int32(def.fieldType.TypeId())]; exists {
+ fieldType = info.Type
+ } else {
+ return nil, fmt.Errorf("unknown type for field
%s with typeId %d", def.name, def.fieldType.TypeId())
+ }
+ }
+
+ fieldSerializer, err := def.fieldType.getSerializer(f)
+ if err != nil {
+ return nil, fmt.Errorf("failed to get serializer for
field %s: %w", def.name, err)
+ }
+
+ fieldInfo := &fieldInfo{
+ name: def.name,
+ fieldIndex: fieldIndex,
+ type_: fieldType,
+ referencable: def.nullable,
+ serializer: fieldSerializer,
+ }
+
+ fields = append(fields, fieldInfo)
+ }
+
+ return fields, nil
+}
+
type triple struct {
typeID int16
serializer Serializer
diff --git a/go/fory/type.go b/go/fory/type.go
index 8d1bc68f8..de0a65446 100644
--- a/go/fory/type.go
+++ b/go/fory/type.go
@@ -306,6 +306,10 @@ type typeResolver struct {
namespaceDecoder *meta.Decoder
typeNameEncoder *meta.Encoder
typeNameDecoder *meta.Decoder
+
+ // meta share related
+ typeToTypeDef map[reflect.Type]*TypeDef
+ defIdToTypeDef map[int64]*TypeDef
}
func newTypeResolver(fory *Fory) *typeResolver {
@@ -341,6 +345,9 @@ func newTypeResolver(fory *Fory) *typeResolver {
namespaceDecoder: meta.NewDecoder('.', '_'),
typeNameEncoder: meta.NewEncoder('$', '_'),
typeNameDecoder: meta.NewDecoder('$', '_'),
+
+ typeToTypeDef: make(map[reflect.Type]*TypeDef),
+ defIdToTypeDef: make(map[int64]*TypeDef),
}
// base type info for encode/decode types.
// composite types info will be constructed dynamically.
@@ -532,15 +539,10 @@ func (r *typeResolver) getTypeInfo(value reflect.Value,
create bool) (TypeInfo,
}
clean := strings.TrimPrefix(rawInfo, "*@")
clean = strings.TrimPrefix(clean, "@")
+ typeName = clean
if idx := strings.LastIndex(clean, "."); idx != -1 {
pkgPath = clean[:idx]
- } else {
- pkgPath = clean
- }
- if type_.Kind() == reflect.Ptr {
- typeName = type_.Elem().Name()
- } else {
- typeName = type_.Name()
+ typeName = clean[idx+1:]
}
// Handle special types that require explicit registration
@@ -716,6 +718,10 @@ func calcTypeHash(type_ reflect.Type) uint64 {
return h.Sum64()
}
+func (r *typeResolver) metaShareEnabled() bool {
+ return r.fory != nil && r.fory.metaContext != nil && r.fory.compatible
+}
+
func (r *typeResolver) writeTypeInfo(buffer *ByteBuffer, typeInfo TypeInfo)
error {
// Extract the internal type ID (lower 8 bits)
typeID := typeInfo.TypeID
@@ -729,6 +735,12 @@ func (r *typeResolver) writeTypeInfo(buffer *ByteBuffer,
typeInfo TypeInfo) erro
// For namespaced types, write additional metadata:
if IsNamespacedType(TypeId(internalTypeID)) {
+ if r.metaShareEnabled() {
+ if err := r.writeSharedTypeMeta(buffer, typeInfo); err
!= nil {
+ return err
+ }
+ return nil
+ }
// Write package path (namespace) metadata
if err := r.metaStringResolver.WriteMetaStringBytes(buffer,
typeInfo.PkgPathBytes); err != nil {
return err
@@ -742,6 +754,87 @@ func (r *typeResolver) writeTypeInfo(buffer *ByteBuffer,
typeInfo TypeInfo) erro
return nil
}
+func (r *typeResolver) writeSharedTypeMeta(buffer *ByteBuffer, typeInfo
TypeInfo) error {
+ context := r.fory.metaContext
+ typ := typeInfo.Type
+
+ if index, exists := context.typeMap[typ]; exists {
+ buffer.WriteVarUint32(index)
+ return nil
+ }
+
+ newIndex := uint32(len(context.typeMap))
+ buffer.WriteVarUint32(newIndex)
+ context.typeMap[typ] = newIndex
+
+ typeDef, err := r.getOrCreateTypeDef(typeInfo.Type)
+ if err != nil {
+ return err
+ }
+ context.writingTypeDefs = append(context.writingTypeDefs, typeDef)
+ return nil
+}
+
+func (r *typeResolver) getOrCreateTypeDef(typ reflect.Type) (*TypeDef, error) {
+ if existingTypeDef, exists := r.typeToTypeDef[typ]; exists {
+ return existingTypeDef, nil
+ }
+
+ zero := reflect.Zero(typ)
+ typeDef, err := buildTypeDef(r.fory, zero)
+ if err != nil {
+ return nil, err
+ }
+ r.typeToTypeDef[typ] = typeDef
+ return typeDef, nil
+}
+
+func (r *typeResolver) readSharedTypeMeta(buffer *ByteBuffer) (TypeInfo,
error) {
+ context := r.fory.metaContext
+ index := buffer.ReadVarInt32() // shared meta index id
+ if index < 0 || index >= int32(len(context.readTypeInfos)) {
+ return TypeInfo{}, fmt.Errorf("TypeInfo not found for index
%d", index)
+ }
+ info := context.readTypeInfos[index]
+ return info, nil
+}
+
+func (r *typeResolver) writeTypeDefs(buffer *ByteBuffer) {
+ context := r.fory.metaContext
+ sz := len(context.writingTypeDefs)
+ buffer.WriteVarUint32Small7(uint32(sz))
+ for _, typeDef := range context.writingTypeDefs {
+ typeDef.writeTypeDef(buffer)
+ }
+ context.writingTypeDefs = nil
+}
+
+func (r *typeResolver) readTypeDefs(buffer *ByteBuffer) error {
+ numTypeDefs := buffer.ReadVarUint32Small7()
+ context := r.fory.metaContext
+ for i := 0; i < numTypeDefs; i++ {
+ id := buffer.ReadInt64()
+ var td *TypeDef
+ if existingTd, exists := r.defIdToTypeDef[id]; exists {
+ skipTypeDef(buffer, id)
+ td = existingTd
+ } else {
+ newTd, err := readTypeDef(r.fory, buffer, id)
+ if err != nil {
+ return err
+ }
+ r.defIdToTypeDef[id] = newTd
+ td = newTd
+ }
+ typeInfo, err := td.buildTypeInfo()
+ if err != nil {
+ return err
+ }
+ context.readTypeInfos = append(context.readTypeInfos, typeInfo)
+ }
+ return nil
+}
+
func (r *typeResolver) createSerializer(type_ reflect.Type, mapInStruct bool)
(s Serializer, err error) {
kind := type_.Kind()
switch kind {
@@ -1015,6 +1108,9 @@ func (r *typeResolver) readTypeInfo(buffer *ByteBuffer)
(TypeInfo, error) {
internalTypeID = -internalTypeID
}
if IsNamespacedType(TypeId(internalTypeID)) {
+ if r.metaShareEnabled() {
+ return r.readSharedTypeMeta(buffer)
+ }
// Read namespace and type name metadata bytes
nsBytes, err := r.metaStringResolver.ReadMetaStringBytes(buffer)
if err != nil {
diff --git a/go/fory/type_def.go b/go/fory/type_def.go
index 193dde6f3..6d59bc05f 100644
--- a/go/fory/type_def.go
+++ b/go/fory/type_def.go
@@ -66,8 +66,21 @@ func (td *TypeDef) writeTypeDef(buffer *ByteBuffer) {
buffer.WriteBinary(td.encoded)
}
-func readTypeDef(fory *Fory, buffer *ByteBuffer) (*TypeDef, error) {
- return decodeTypeDef(fory, buffer)
+// buildTypeInfo constructs a TypeInfo from the TypeDef
+func (td *TypeDef) buildTypeInfo() (TypeInfo, error) {
+ info := TypeInfo{
+ Type: td.type_,
+ TypeID: int32(td.typeId),
+ Serializer: &structSerializer{type_: td.type_, fieldDefs:
td.fieldDefs},
+ PkgPathBytes: td.nsName,
+ NameBytes: td.typeName,
+ IsDynamic: td.typeId < 0,
+ }
+ return info, nil
+}
+
+func readTypeDef(fory *Fory, buffer *ByteBuffer, header int64) (*TypeDef,
error) {
+ return decodeTypeDef(fory, buffer, header)
}
func skipTypeDef(buffer *ByteBuffer, header int64) {
@@ -128,7 +141,7 @@ func buildFieldDefs(fory *Fory, value reflect.Value)
([]FieldDef, error) {
fieldValue := value.Field(i)
var fieldInfo FieldDef
- fieldName := field.Name
+ fieldName := SnakeCase(field.Name)
nameEncoding :=
fory.typeResolver.typeNameEncoder.ComputeEncodingWith(fieldName,
fieldNameEncodings)
@@ -140,11 +153,45 @@ func buildFieldDefs(fory *Fory, value reflect.Value)
([]FieldDef, error) {
name: fieldName,
nameEncoding: nameEncoding,
nullable: nullable(field.Type),
- trackingRef: fory.referenceTracking,
+ trackingRef: fory.refTracking,
fieldType: ft,
}
fieldInfos = append(fieldInfos, fieldInfo)
}
+
+ // Sort field definitions
+ if len(fieldInfos) > 1 {
+ // Extract serializers and names for sorting
+ serializers := make([]Serializer, len(fieldInfos))
+ fieldNames := make([]string, len(fieldInfos))
+ for i, fieldInfo := range fieldInfos {
+ serializer, err :=
fieldInfo.fieldType.getSerializer(fory)
+ if err != nil {
+ // If we can't get serializer, use nil (will be
handled by sortFields)
+ serializers[i] = nil
+ } else {
+ serializers[i] = serializer
+ }
+ fieldNames[i] = fieldInfo.name
+ }
+
+ // Use existing sortFields function to get optimal order
+ _, sortedNames := sortFields(fory.typeResolver, fieldNames,
serializers)
+
+ // Rebuild fieldInfos in the sorted order
+ nameToFieldInfo := make(map[string]FieldDef)
+ for _, fieldInfo := range fieldInfos {
+ nameToFieldInfo[fieldInfo.name] = fieldInfo
+ }
+
+ sortedFieldInfos := make([]FieldDef, len(fieldInfos))
+ for i, name := range sortedNames {
+ sortedFieldInfos[i] = nameToFieldInfo[name]
+ }
+
+ fieldInfos = sortedFieldInfos
+ }
+
return fieldInfos, nil
}
@@ -152,6 +199,7 @@ func buildFieldDefs(fory *Fory, value reflect.Value)
([]FieldDef, error) {
type FieldType interface {
TypeId() TypeId
write(*ByteBuffer)
+ getSerializer(*Fory) (Serializer, error)
}
// BaseFieldType provides common functionality for field types
@@ -164,6 +212,18 @@ func (b *BaseFieldType) write(buffer *ByteBuffer) {
buffer.WriteVarUint32Small7(uint32(b.typeId))
}
+func (o *BaseFieldType) getSerializer(fory *Fory) (Serializer, error) {
+ if o.typeId == EXTENSION || o.typeId == STRUCT || o.typeId ==
NAMED_STRUCT ||
+ o.typeId == COMPATIBLE_STRUCT || o.typeId ==
NAMED_COMPATIBLE_STRUCT || o.typeId == UNKNOWN_TYPE_ID {
+ return nil, nil
+ }
+ info, err := fory.typeResolver.getTypeInfoById(o.typeId)
+ if err != nil {
+ return nil, err
+ }
+ return info.Serializer, nil
+}
+
// readFieldInfo reads field type info from the buffer according to the TypeId
func readFieldType(buffer *ByteBuffer) (FieldType, error) {
typeId := buffer.ReadVarUint32Small7()
diff --git a/go/fory/type_def_decoder.go b/go/fory/type_def_decoder.go
index f4b2d8ec2..d98cff6a8 100644
--- a/go/fory/type_def_decoder.go
+++ b/go/fory/type_def_decoder.go
@@ -19,6 +19,7 @@ package fory
import (
"fmt"
+ "reflect"
)
/*
@@ -29,9 +30,9 @@ typeDef are layout as following:
- next variable bytes: type id (varint) or ns name + type name
- next variable bytes: field definitions (see below)
*/
-func decodeTypeDef(fory *Fory, buffer *ByteBuffer) (*TypeDef, error) {
+func decodeTypeDef(fory *Fory, buffer *ByteBuffer, header int64) (*TypeDef,
error) {
// Read 8-byte global header
- globalHeader := uint64(buffer.ReadInt64())
+ globalHeader := header
hasFieldsMeta := (globalHeader & HAS_FIELDS_META_FLAG) != 0
isCompressed := (globalHeader & COMPRESS_META_FLAG) != 0
metaSize := int(globalHeader & META_SIZE_MASK)
@@ -61,6 +62,7 @@ func decodeTypeDef(fory *Fory, buffer *ByteBuffer) (*TypeDef,
error) {
// Read name or type ID according to the registerByName flag
var typeId TypeId
var nsBytes, nameBytes *MetaStringBytes
+ var type_ reflect.Type
if registeredByName {
// Read namespace and type name for namespaced types
readingNsBytes, err :=
fory.typeResolver.metaStringResolver.ReadMetaStringBytes(metaBuffer)
@@ -78,8 +80,14 @@ func decodeTypeDef(fory *Fory, buffer *ByteBuffer)
(*TypeDef, error) {
return nil, fmt.Errorf("type not registered")
}
typeId = TypeId(info.TypeID)
+ type_ = info.Type
} else {
typeId = TypeId(metaBuffer.ReadVarInt32())
+ if info, err := fory.typeResolver.getTypeInfoById(typeId); err
!= nil {
+ return nil, fmt.Errorf("failed to get type info by id
%d: %w", typeId, err)
+ } else {
+ type_ = info.Type
+ }
}
// Read fields information
@@ -97,6 +105,7 @@ func decodeTypeDef(fory *Fory, buffer *ByteBuffer)
(*TypeDef, error) {
// Create TypeDef
typeDef := NewTypeDef(typeId, nsBytes, nameBytes, registeredByName,
isCompressed, fieldInfos)
typeDef.encoded = encoded
+ typeDef.type_ = type_
return typeDef, nil
}
diff --git a/go/fory/type_def_encoder_test.go b/go/fory/type_def_encoder_test.go
index 0c74954fb..e65474d32 100644
--- a/go/fory/type_def_encoder_test.go
+++ b/go/fory/type_def_encoder_test.go
@@ -57,7 +57,7 @@ func TestTypeDefEncodingDecoding(t *testing.T) {
originalTypeDef.writeTypeDef(buffer)
// Decode the TypeDef
- decodedTypeDef, err := readTypeDef(fory, buffer)
+ decodedTypeDef, err := readTypeDef(fory, buffer,
int64(buffer.ReadInt64()))
if err != nil {
t.Fatalf("Failed to decode TypeDef: %v", err)
}
diff --git a/go/fory/type_test.go b/go/fory/type_test.go
index 5e0add767..b26c5674a 100644
--- a/go/fory/type_test.go
+++ b/go/fory/type_test.go
@@ -25,10 +25,10 @@ import (
func TestTypeResolver(t *testing.T) {
fory := &Fory{
- refResolver: newRefResolver(false),
- referenceTracking: false,
- language: XLANG,
- buffer: NewByteBuffer(nil),
+ refResolver: newRefResolver(false),
+ refTracking: false,
+ language: XLANG,
+ buffer: NewByteBuffer(nil),
}
typeResolver := newTypeResolver(fory)
type A struct {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
