Ok thanks, I was trying to avoid this, to compile and use the generated classes. Now we are currently processing the raw protobuf message as a map of UnknownFieldSet, and I was wondering if there was a choice to make this "rewrite" without the need to compile and use the class, just traversing the protobuf.
I will check and try. Thanks, Joan. On Wednesday, September 20, 2023 at 12:30:26 PM UTC+2 Florian Enner wrote: > As the very first step you should compile your schema and work with the > generated classes: > https://protobuf.dev/getting-started/javatutorial/#compiling-protocol-buffers > > Don't mess with the reflection API if you can avoid it. > > > On Wednesday, September 20, 2023 at 12:19:38 PM UTC+2 Joan Balagueró wrote: > >> Hi Florian, >> >> Not sure what I can do with this code. I only have a byte array >> representing the above protobuf, I don't have any object or message. At >> most I have a map with the list of the "UnknownFieldSet.Field" fields after >> parsing the byte array (this is java code). >> >> I'm relatively new to protobuf, I read a lot on the Internet trying to >> find a solution but I did not find anything. >> >> That's why I tried to traverse the map of fields and write them to the >> new byte array. It's the only solution I could think of, but I'm doing >> something wrong. >> >> Not sure if you can help me a bit more to solve this. >> >> Anyways thanks. >> >> Joan. >> >> >> >> >> >> On Wednesday, September 20, 2023 at 11:56:19 AM UTC+2 Florian Enner wrote: >> >>> Messages are serialized with a length delimiter, so changing the content >>> produces a mismatch and invalid message. >>> >>> Your schema has no affected repeated fields, so appending a delta should >>> work. I've never used the C# API, but here is some hopefully understandable >>> pseudo code: >>> >>> var delta = Request.newInstance(); >>> delta.getMutableMeta().getMutableCutOffTime() >>> .setValue(value) >>> .setScale(TimeSpanScale.MINMAX) >>> byte[] output = append(unmodifiedInputBytes, delta.toByteArray()); >>> >>> If the server expects a length delimiter you'd need to update it to the >>> new combined length. >>> >>> - Florian >>> >>> On Wednesday, September 20, 2023 at 11:03:46 AM UTC+2 Joan Balagueró >>> wrote: >>> >>>> Hi Florian, >>>> >>>> Thanks for your quick response. I'm stuck on this. >>>> >>>> 1) It's not working. When I send the protobuf to the backend server >>>> (it's not our api nor server) using the first method, I get a right >>>> response. But using the second method I receive this error: >>>> ProtoBuf.ProtoException: Invalid wire-type; this usually means you have >>>> over-written a file without truncating or setting the length; see >>>> https://stackoverflow.com/q/2152978/23354 >>>> at ProtoBuf.ProtoReader.StartSubItem(ProtoReader reader) in >>>> C:\code\protobuf-net\src\protobuf-net\ProtoReader.cs:line 637 >>>> at ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key, >>>> ProtoReader reader, Type type) in >>>> C:\code\protobuf-net\src\protobuf-net\ProtoReader.cs:line 584 >>>> at proto_40(Object , ProtoReader ) >>>> at ProtoBuf.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type >>>> type, Object value, Boolean noAutoCreate) in >>>> C:\code\protobuf-net\src\protobuf-net\Meta\TypeModel.cs:line 722 >>>> at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, >>>> Type type, SerializationContext context) in >>>> C:\code\protobuf-net\src\protobuf-net\Meta\TypeModel.cs:line 599 >>>> at >>>> WebBeds.Connect.AspNetCore.Formatters.ProtobufInputFormatter.ReadRequestBodyAsync(InputFormatterContext >>>> >>>> context) >>>> at >>>> Microsoft.AspNetCore.Mvc.Formatters.InputFormatter.ReadAsync(InputFormatterContext >>>> >>>> context) >>>> at >>>> Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext >>>> >>>> bindingContext) >>>> at >>>> Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext >>>> >>>> actionContext, IModelBinder modelBinder, IValueProvider valueProvider, >>>> ParameterDescriptor parameter, ModelMetadata metadata, Object value, >>>> Object >>>> container) >>>> at >>>> Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext() >>>> --- End of stack trace from previous location --- >>>> at >>>> Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker >>>> >>>> invoker, Task lastTask, State next, Scope scope, Object state, Boolean >>>> isCompleted) >>>> at >>>> Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker >>>> >>>> invoker, Task lastTask, State next, Scope scope, Object state, Boolean >>>> isCompleted >>>> >>>> >>>> 2) This is the proto: >>>> >>>> The request with the 'Meta' element: >>>> >>>> message Request { >>>> Meta Meta = 1; >>>> repeated int32 Hotels = 2 [packed = false]; >>>> Country Market = 3; >>>> repeated Room Rooms = 4; >>>> .bcl.DateTime CheckIn = 5; >>>> .bcl.DateTime CheckOut = 6; >>>> OptionalCriteria OptionalCriteria = 7; >>>> } >>>> >>>> The 'Meta' element that contains the 'CutOffTime' that we want to >>>> modify: >>>> >>>> message Meta { >>>> int32 Client = 1; >>>> int32 Brand = 2; >>>> bool UseCache = 3; >>>> .bcl.TimeSpan CutOffTime = 4; >>>> bool B2C = 5; >>>> Language Language = 6; >>>> Currency Currency = 7; >>>> bool IncludeProviderAuditData = 8; >>>> SalesChannel SalesChannel = 9; >>>> string AgentId = 10; >>>> } >>>> >>>> The 'CutOffTime': >>>> >>>> message TimeSpan { >>>> sint64 value = 1; // the size of the timespan (in units of the >>>> selected scale) >>>> TimeSpanScale scale = 2; // the scale of the timespan [default = DAYS] >>>> enum TimeSpanScale { >>>> DAYS = 0; >>>> HOURS = 1; >>>> MINUTES = 2; >>>> SECONDS = 3; >>>> MILLISECONDS = 4; >>>> TICKS = 5; >>>> >>>> MINMAX = 15; // dubious >>>> } >>>> } >>>> >>>> Thanks, >>>> >>>> Joan. >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Wednesday, September 20, 2023 at 10:40:08 AM UTC+2 Florian Enner >>>> wrote: >>>> >>>>> 1) A "varint" is a "variable length integer". When you replace a large >>>>> number with a small one, it's entirely possible to lose some bytes and >>>>> still be valid. You need to check the actual output. >>>>> >>>>> 2) Can you provide the proto definition of the field you want to >>>>> modify? Scalar fields get set to the last encountered value, so the >>>>> easiest >>>>> option may be to copy the original bytes and append a delta containing >>>>> the >>>>> differences. >>>>> >>>>> >>>>> >>>>> >>>>> On Wednesday, September 20, 2023 at 10:19:07 AM UTC+2 Joan Balagueró >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I have a protobuf message like this into a byte array: >>>>>> >>>>>> 1: { // META element >>>>>> 1: 2 >>>>>> 2: 1 >>>>>> 3: 1 >>>>>> 4: { // CutOffTime element within META >>>>>> 1: 10 >>>>>> 2: 3 >>>>>> } >>>>>> 5: 1 >>>>>> 6: 40 >>>>>> } >>>>>> 2: 9836 // HOTEL element >>>>>> 3: 724 // MARKET element >>>>>> >>>>>> >>>>>> We need to traverse this message and write it to a >>>>>> 'CodeOutputStream', but changing the values of the 'cutoff' element to, >>>>>> for >>>>>> example to '4: { 1: 7, 2: 4 }'. I'm not able to do it, I need some help. >>>>>> >>>>>> A basic algorithm that writes the same protobuf to another byte array >>>>>> but without changing anything works. Here I try the special case of >>>>>> 'META' >>>>>> (key = 1) that contains the 'cutoff' element. >>>>>> >>>>>> Map<Integer, UnknownFieldSet.Field> rootFields = >>>>>> UnknownFieldSet.parseFrom(document).asMap(); >>>>>> >>>>>> for (Map.Entry<Integer, UnknownFieldSet.Field> entry : >>>>>> rootFields.entrySet()) { >>>>>> if (entry.getKey() == 1) { >>>>>> ByteString bs = >>>>>> entry.getValue().getLengthDelimitedList().get(0); >>>>>> output.writeByteArray(1, bs.toByteArray()); >>>>>> } >>>>>> else { >>>>>> entry.getValue().writeTo(entry.getKey(), output); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> Now I try to go a bit further, trying to read the cuotff element, >>>>>> change the values and rewrite them to the 'output'. And here is when I'm >>>>>> not able to solve the problem. Below my try that does not work, it >>>>>> generates a byte array >>>>>> of 69 bytes instead of 73 (I'm losing 4 bytes): >>>>>> >>>>>> Map<Integer, UnknownFieldSet.Field> rootFields = >>>>>> UnknownFieldSet.parseFrom(document).asMap(); >>>>>> >>>>>> for (Map.Entry<Integer, UnknownFieldSet.Field> entry : >>>>>> rootFields.entrySet()) { >>>>>> if (entry.getKey() == 1) { >>>>>> ByteString bs = >>>>>> entry.getValue().getLengthDelimitedList().get(0); >>>>>> Map<Integer, UnknownFieldSet.Field> ufs = >>>>>> UnknownFieldSet.parseFrom(bs).asMap(); >>>>>> >>>>>> for (Map.Entry<Integer, UnknownFieldSet.Field> >>>>>> item : ufs.entrySet()) { >>>>>> if (item.getKey() == 4) { >>>>>> Map<Integer, UnknownFieldSet.Field> >>>>>> cutoff = >>>>>> UnknownFieldSet.parseFrom(item.getValue().getLengthDelimitedList().get(0)).asMap(); >>>>>> cutoff.put(1, >>>>>> UnknownFieldSet.Field.newBuilder().addVarint(7).build()).writeTo(1, >>>>>> output); >>>>>> cutoff.put(2, >>>>>> UnknownFieldSet.Field.newBuilder().addVarint(4).build()).writeTo(1, >>>>>> output); >>>>>> } >>>>>> else { >>>>>> item.getValue().writeTo(item.getKey(), >>>>>> output); >>>>>> } >>>>>> } >>>>>> } >>>>>> else { >>>>>> entry.getValue().writeTo(entry.getKey(), output); >>>>>> } >>>>>> } >>>>>> >>>>>> Any help would be really appreciated. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Joan. >>>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/301116cd-25eb-4360-84b8-e659e3c41c49n%40googlegroups.com.
