Re: [protobuf] A first chance exception of type 'System.IO.EndOfStreamException' occurred in protobuf-net.DLL

[Marc Gravell]

This looks to be protobuf-net specific. Note that protobuf-net includes
SerializeWithLengthPrefix and DeserializeWithLengthPrefix which can be used
to simplify working with multiple separate logical messages on a single
stream (such as against a socket), without ever over-reading.

Also - it should never error with an EOF unless the socket becomes broken
mid-message. More likely, it would simply block waiting on incoming data -
or throw a timeout if timeouts are enabled on the underlying stream.

Marc
On 22 Apr 2014 16:58, Empi gmpe...@gmail.com wrote:

 Hi and thanks for this lovely serializer.

 I'm working on  aproject on which local desktop application is talking
 tcp/ip with local window 8 application (metro) through loopback.
 I did of course loopback exemption, and all worked great!!! until...
 yesterday I've added an additional String field to the serialized class...
 and I get the above error.
 If I remove the String it works great again.

 Can you help me solve this mystery?

 The classes I serialize has the following structure (last field is a class
 on which the string is added, I removed field's name and left only the
 types).

 [ProtoContract]
 public class cMessage
 {
 [ProtoMember(1)]
 public uint nPacketNumber { get; set; }
 [ProtoMember(2)]
 public int nBatteryStatus { get; set; }
 [ProtoMember(3)]
 public Boolean isCharging { get; set; }
 [ProtoMember(4)]
 public COMM_STATUS commStatus { get; set; }
 [ProtoMember(5)]
 public Boolean isDownloading { get; set; }
 [ProtoMember(6)]
 public COMM_ERRORS commErrorCode { get; set; } //
 [ProtoMember(10)]
 public cGeneralInfo cInfo;
 }

 [ProtoContract]
 public class cGeneralInfo
 {
 [ProtoMember(100)]
 public DateTime dtLastPacket;
 [ProtoMember(101)]
 public uint unBatteryStatus;
 [ProtoMember(102)]
 public uint unChargingStatus;
 [ProtoMember(103)]
 public double dSoftwareVersion;
 [ProtoMember(104)]
 public String sInfo1
 [ProtoMember(105)]
 public uint
 [ProtoMember(106)]
 public uint
 [ProtoMember(107)]
 public uint
 [ProtoMember(108)]
 public uint
 [ProtoMember(109)]
 public uint
 [ProtoMember(110)]
 public uint
 [ProtoMember(111)]
 public double
 [ProtoMember(112)]
 public int
 [ProtoMember(113)]
 public int
 [ProtoMember(114)]
 public int
 [ProtoMember(115)]
 public int
 [ProtoMember(116)]
 public int
 [ProtoMember(117)]
 public int
 [ProtoMember(118)]
 public int
 [ProtoMember(119)]
 public int
 [ProtoMember(120)]
 public int
 [ProtoMember(121)]
 public int
 [ProtoMember(122)]
 public int
 [ProtoMember(123)]
 public int
 [ProtoMember(124)]
 public int
 [ProtoMember(125)]
 public int
 [ProtoMember(126)]
 public string
 [ProtoMember(127)]
 public int
 [ProtoMember(128)]
 public String    This is the filed started causing
 the error.
 }


 Thanks!!!.


 Empi



  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] I don't understand why we use protocol buffer

[Marc Gravell]

Protocol buffers in the public project consists primarily of the
serialization framework; serialization is always necessary when
communicating, and frameworks/formats are aplenty, including xml, json,
etc. The primary features of protocol buffers (protobuf) are:

- efficient binary on the wire (small bandwidth)
- efficient to encode (small CPU)
- efficient to decode (small CPU)
- cross platform and public specification (contrast with the inbuilt Java
or .NET binary serialization, which is single platform and closed
specification)

As I understand it from this forum, inside google, the protobuf framework
includes a full RPC stack; however, this piece has never been released to
the wider community, so while the .proto definitions include stubs for RPC
hooks, they are not very useful. That is a valid criticism.

However! Efficient and portable serialization **by itself** is very
valuable. Once you can serialize your data down to a blob (typically a
`byte[]`, or whatever is the equivalent in your framework) you can send it
anywhere and rehydrate at the other end. That could be local disk storage,
storage inside a database or key/value store (redis etc), transmission as
an http payload, or raw socket data. If your chosen comm's stack only
supports strings, then convert it to base-64 and you're sorted.

I suspect the key point here is in your last comment; I also don't
understand what is serialize mean?!. Serialization is converting
structured data in some form down to a chunk of binary (or a string, noting
that a string is of course just a special-case of binary) suitable for
storage or transmission.

To emphasize: the public protobuf drop is not an RPC stack. If you look to
it for RPC, you will indeed be somewhat disappointed. But look at it as a
serialization stack, and you'll be pretty happy.

Marc


On 26 April 2014 13:17, Flowra white flowrathe...@gmail.com wrote:












 *Hi,,I was reading a book about Google and that book was talking about the
 communication paradigms and as i understand google use RPCbut the book
 mentioned something named protocol buffer I don't get why we use it and
 what is the relation between it and RPC I searched for meaning but all i
 get that it is a way to serialize structured data I also don't understand
 what is serialize mean?! sorry for bothering , and I wish anyone could
 help me .thanks,,*

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.




-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Use different encoding while serializing strings

[Marc Gravell]

protobuf is a binary-safe protocol, and is not impacted by contents such as
\r, \n or \t. In particular, text content is utf-8 encoded and
length-prefixed - it simply *does not care* what is inside the text. I
suspect any problem you are having relates to how you are transporting and
processing the payload, not to protobuf itself. Because protobuf does not
check for \r, \n or \t at any point.

Marc


On 15 May 2014 00:56, Ganesh Sangle saygane...@gmail.com wrote:

 Hi Guys,
 I am trying to use protobufs between two entities - one in python and
 another in c++.
 The advantage of using protobuf is i dont have to write
 serializing/deserializing code.

 However, there is a complication.
 The message that I create in python world, when i serialize it to be sent
 over to the other side, has '\r\n'.
 The code on the other side is already using \n as a delimiter and cutting
 out the strings.
 So what happens is that the message string gets split and when I try to
 re-assemble it it doent work.

 Since my code is a smaller part of a larger existing system, I wanted to
 know if there is a way to customize/configure protobuf so that it uses some
 other characters instead of '\n\t', or just use a different representation
 (all hex numbers or whatever) when converting it to string.

 Thanks,
 Ganesh

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.




-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What causes TargetInvocationException during serialization

[Marc Gravell]

Oops, meant to reply-all!
On 16 May 2014 19:34, Marc Gravell marc.grav...@gmail.com wrote:

 This is specifically protobuf-net; I have received a repro case from Marco
 separately via email, so I will investigate and post back here and
 directly, but short version don't know off the top of my head; at first
 glance, it looks like it must be a bug in the CF-specific code of
 protobuf-net.

 Marc


 On 16 May 2014 14:58, Marco P. marco...@gmail.com wrote:

 I am trying to serialize something like this (i'll summarize as the
 object is more complex):

 class Anag {

 }

 System.Reflection.TargetInvocationException was unhandled
   Message=TargetInvocationException
   StackTrace:
 at System.Reflection.RuntimePropertyInfo.GetValue()
 at ProtoBuf.Serializers.TupleSerializer.GetValue()
 at ProtoBuf.Serializers.TupleSerializer.Write()
 at ProtoBuf.Meta.RuntimeTypeModel.Serialize()
 at ProtoBuf.ProtoWriter.WriteObject()
 at
 ProtoBuf.Serializers.SubItemSerializer.ProtoBuf.Serializers.IProtoSerializer.Write()
 at ProtoBuf.Serializers.TagDecorator.Write()
 at ProtoBuf.Serializers.ListDecorator.Write()
 at ProtoBuf.Serializers.PropertyDecorator.Write()
 at ProtoBuf.Serializers.TypeSerializer.Write()
 at ProtoBuf.Meta.RuntimeTypeModel.Serialize()
 at ProtoBuf.Meta.TypeModel.SerializeCore()
 at ProtoBuf.Meta.TypeModel.Serialize()
 at ProtoBuf.Meta.TypeModel.Serialize()
 at ProtoBuf.Serializer.Serialize()
  [...]
   InnerException:

  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.




 --
 Regards,

 Marc


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] ProtoContractAttribute.Name Usage?

[Marc Gravell]

Oops; meant to press reply-all, not reply, but:

This relates to protobuf-net. The only time the Name property is used is if
you reverse-generate from code to .proto via Serializer.GetProto (or the
similar method on RuntimeTypeModel).


On 23 May 2014 13:23, Sam Eaton nuluv...@gmail.com wrote:

 Hello,

 I was wondering if anyone could tell me what the Name property on the
 ProtoContractAttribute class is actually used for.

 Thanks for your time and help.

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.




-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf-net - list of objects with parent reference

[Marc Gravell]

The protobuf specification doesn't have the notion of object identity.
protobuf-net *does* (as an optionally enabled feature), but it doesn't
currently work for list items directly, although I suspect it probably
should. Since it would break the format, though, it would need explicit
enabling if I fixed this.

But the following code works today - note that what I have done here is to
wrap the top-level list items in a wrapper that just encapsulates the
ProtoItem, but in doing so enables reference-tracking. Not ideal, but: it
works.

using ProtoBuf;
using System.Collections.Generic;
using System.IO;

[ProtoContract(AsReferenceDefault=true)]
public class ProtoItem
{
[ProtoMember(1)]
public int Value { get; set; }

[ProtoMember(2)]
public ProtoItem BaseItem { get; set; }
}
[ProtoContract]
public class Wrapper
{
[ProtoMember(1, DataFormat = DataFormat.Group)]
public ProtoItem Item { get;set; }

public static implicit operator ProtoItem(Wrapper value)
{
return value == null ? null : value.Item;
}
public static implicit operator Wrapper(ProtoItem value)
{
return value == null ? null : new Wrapper { Item = value };
}
}



static class Program
{

static void Main()
{
var itemParent = new ProtoItem { Value = 1 };
var item2 = new ProtoItem { Value = 2, BaseItem = itemParent };
var item3 = new ProtoItem { Value = 3, BaseItem = itemParent };

var parentListToWrite = new ListWrapper { itemParent, item2,
item3 };

const string file = protofile.txt;
try
{ File.Delete(file); }
catch
{ };

using (var fs = File.OpenWrite(file))
{
Serializer.Serialize(fs,
parentListToWrite);
}

ListWrapper readList;
using (var fs = File.OpenRead(file))
{
readList = Serializer.DeserializeListWrapper(fs);
}

if (readList[0].Item == readList[2].Item.BaseItem)
{
//how to make it equal?
System.Console.WriteLine(eq);
}
if (readList[0].Item == readList[1].Item.BaseItem)
{
//how to make it equal?
System.Console.WriteLine(eq);
}
}
}


On 9 June 2014 18:12, Jan Kowalski tramwaj...@gmail.com wrote:

 I have a simple class with reference to parent object. All objects are in
 one list (even parent objects). Is it possible to keep references
 references after deserialization?

 In my code I have something like this:

 [ProtoContract]
 public class ProtoItem
 {
 [ProtoMember(1)]
 public int Value { get; set; }

 [ProtoMember(2, AsReference = true)]
 public ProtoItem BaseItem { get; set; }
 }

 And I use it like that:


  static void Main()
 {
 var itemParent = new ProtoItem { Value = 1 };
 var item2 = new ProtoItem { Value = 2, BaseItem = itemParent };
 var item3 = new ProtoItem { Value = 3, BaseItem = itemParent };

 var parentListToWrite = new ListProtoItem {itemParent, item2, 
 item3};

 const string file = protofile.txt;
 try { File.Delete(file); }
 catch { };

 using (var fs = File.OpenWrite(file)) { Serializer.Serialize(fs,
  parentListToWrite); }

 ListProtoItem readList;
 using (var fs = File.OpenRead(file)) { readList =
 Serializer.DeserializeListProtoItem(fs); }

 if (readList[0] == readList[2].BaseItem)
 {
 //how to make it equal?
 }
 if (readList[0] == readList[1].BaseItem)
 {
 //how to make it equal?
 }
 }


 Is that possible to deserialize it that last two equations work?
 jan.

  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.




-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Issue 269 in protobuf: Would like to have byte or int8 type for the message definition

[Marc Gravell]

The simple answer would be use the existing oversized types, and cast at
the caller. Varint data in particular will either be 1 byte or 2 (50%
each) for byte values. For longer sequences (rgba etc) there are existing
fixed-32 and bytes.

If the intent is to add a wire type to precisely represent a byte: that is
hugely problematic. Adding a wire type is a significant thing, as:

- the numbers of wire types is strictly limited, and there isn't much spare
(unless the next wire type somehow means check the next byte for the
longer wire type, or something)
- all existing clients, libraries and deployments would need updating to
even touch data with that wire type - it isn't even possible to ignore the
data if you don't understand the wire type.

Marc
On 30 Jun 2014 18:20, proto...@googlecode.com wrote:


 Comment #4 on issue 269 by thecodem...@gmail.com: Would like to have byte
 or int8 type for the message definition
 http://code.google.com/p/protobuf/issues/detail?id=269

 I don't need a fixed length string; I want to encode RGB or RGBA data as 3
 or 4 bytes. I could pack them into an int32, but it seems like a byte type
 is a perfectly reasonable request.

 --
 You received this message because this project is configured to send all
 issue notifications to this address.
 You may adjust your notification preferences at:
 https://code.google.com/hosting/settings

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] [Protobuf] Can protobuf serialize datatable (C#)

[Marc Gravell]

Ooh, the idea of allowing SQL over a service boundary sends all kinds of
shivers up my back, and not the good kind of shivers. On your head be it,
but: I wouldn't do that myself.

I once did some experimental datatable serialisation code - it isn't in the
release binary, but it is still inside the solution; but: I really can't
say how robust this would be, and you'd need to serialize it manually and
store it in a byte[]. Of course, by the time you've done all that you might
as well have simply asked the datatable to serialize itself using its own
internal storage format (make sure you switch from XML to binary) - and
again: shove it in a byte[] member. IIRC you can set the RemotingFormat and
use BinaryFormatter to get the internal binary storage format.

Marc
On 10 Jul 2014 19:56, Desmond Davids deme...@gmail.com wrote:

 Hey Marc,

 I know this is an old article. but i need to ask you something regarding
 this.

 We have a WCF service which we use to execute SqlCommands to our DB. We
 send Commands with DataContractSerialization. I came across protobuf-net
 and I have converted my WCF service to use protobuf. We now sit with a
 problem. we use table valued parameters in our sqlcommands. which means we
 need to serialize DataTables as part of an object. I there a way we can
 achieve this with protobuf.

 Do you have any workaround/alternative or suggestion?

 Regards
 Desmond Davids

 On Friday, 16 July 2010 08:00:25 UTC+2, Marc Gravell wrote:

 From the message, that *sounds* like protobuf-net...

 There is no built-in handling of this, but it is possibly something that
 could be added, especially in v2 which has a much more flexible model.
 For example, you *could* argue that there is an implicit schema that uses a
 repeated element per row, and the column ordinal for a field.

 There are a couple of points here thought:

 - protobuf /normally/ doesn't include schema information; so when
 deserializing *either* you would have to set up the schema (columns etc)
 *first* and then use Merge, /or/ I could include schema information as an
 exception
 - it would need some thought re the difference between DataSet and
 DataTable - a data-set has relations etc, which don't necessarily fit very
 well here

 I'm interested in your thoughts with this; are you just after something
 to load/save the data? Or interop with other platforms?

 And the other important question: /an option/ is to move the data to/from
 a DTO during serialization; the DTO would serialize fine with protobuf-net.

 Marc

 On 15 July 2010 09:43, Ferryandi Chai fjch...@gmail.com wrote:

 Hi All,

 Im just wondering can protobuf serialize datatable.
 Im a c# developer, and when i tried to serialize datatable it prompt
 this error:

 Only data-contract classes (and lists/arrays of such) can be
 processed (error processing DataTable)

 Tried to google this problem but cannot find solutions.


 Thx
 F!

 --
 You received this message because you are subscribed to the Google
 Groups Protocol Buffers group.
 To post to this group, send email to prot...@googlegroups.com.
 To unsubscribe from this group, send email to protobuf+u...@
 googlegroups.com.
 For more options, visit this group at http://groups.google.com/
 group/protobuf?hl=en.




 --
 Regards,

 Marc

  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] [Protobuf] Can protobuf serialize datatable (C#)

[Marc Gravell]

Yes, a `byte[]` can be declared as a `ProtoMember` - it maps to the `bytes`
type in the protobuf spec. I would strongly advise *against* DynamicType
here; that should be used as a last resort only. If you know the data you
want to serialize is actually a finite set of types, I would use that if
possible. For example, let's say you need to support `string`, `int` and
`DataTable`; you could do something like (this is completely untested -
writing in email client here, but the concept is sound):

[ProtoContact]
[ProtoInclude(10, typeof(ParameterValueint)]
[ProtoInclude(11, typeof(ParameterValuestring)]
[ProtoInclude(12, typeof(ParameterValueDataTableShim)]
public abstract class ParameterValue {
 public abstract Type Type {get;}

 public abstract object Value {get;set}
}

[ProtoContract]
public sealed class ParameterValueT : ParameterValue {
public override Type Type { get { return typeof(T); } }

[ProtoMember(1)]
public T TypedValue {get;set;}

public override object Value {
get { return TypedValue; }
set { TypedValue = (T)value; }
}
}

[ProtoContract]
public sealed class DataTableShim {
public DataTable Table {get;set;}

[ProtoMember(1)]
public byte[] TableSerialized {
 get { /* serialize Table to a MemoryStream, and return
ms.ToArray(); */ }
set { /* create a new MemoryStream(value), and deserialize to
Table */ }
}
}



On 11 July 2014 07:54, Desmond Davids deme...@gmail.com wrote:

 Hey Marc,

 Thanks for the reply. We not sending the actual SQL commands over the
 line. we sending the info in such a way that way know which commands to
 execute on which parameters. Problem is our parameters are almost always
 Table Valued Parameters (TVP). and needs to sit in the request as DataTable
 properties. This is our problem protobuf-net cannot serialize DataTable
 type. It also has a problem with object type but i think setting
 DynamicType = true does the trick here as the types will always be
 primitive types. Am i correct? So we need some way to get these TVPs over
 the line using protobuf-net as serializer in WCF.

 Can you tell if byte[] can be declared as a ProtoMember. I can therefore
 use protobuf-net-data to serialize the DataTable object and assign the this
 byte[] ProtoMember. When it comes to assigning my TVP I can just
 Deserialize with protobuf-net-data again.

 I am looking for the fastest way I can do this as the sole reason we using
 this is that we love the performance of protobuf-net. If meaning we have to
 sacrifice the performance to serialize every single TVP we might as well
 revert back to DataContractSerializer which is definitely not what we want.

 I think also you suggestion regarding converting DataTable to a DTO will
 be difficult as we dont really know what the schema of the DataTable will
 be. I will have to think about this more.

 Looking forward to hear from you.

 Regards
 Desmond


 On Thu, Jul 10, 2014 at 9:18 PM, Marc Gravell marc.grav...@gmail.com
 wrote:

 Ooh, the idea of allowing SQL over a service boundary sends all kinds of
 shivers up my back, and not the good kind of shivers. On your head be it,
 but: I wouldn't do that myself.

 I once did some experimental datatable serialisation code - it isn't in
 the release binary, but it is still inside the solution; but: I really
 can't say how robust this would be, and you'd need to serialize it manually
 and store it in a byte[]. Of course, by the time you've done all that you
 might as well have simply asked the datatable to serialize itself using its
 own internal storage format (make sure you switch from XML to binary) - and
 again: shove it in a byte[] member. IIRC you can set the RemotingFormat and
 use BinaryFormatter to get the internal binary storage format.

 Marc
 On 10 Jul 2014 19:56, Desmond Davids deme...@gmail.com wrote:

  Hey Marc,

 I know this is an old article. but i need to ask you something regarding
 this.

 We have a WCF service which we use to execute SqlCommands to our DB. We
 send Commands with DataContractSerialization. I came across protobuf-net
 and I have converted my WCF service to use protobuf. We now sit with a
 problem. we use table valued parameters in our sqlcommands. which means we
 need to serialize DataTables as part of an object. I there a way we can
 achieve this with protobuf.

 Do you have any workaround/alternative or suggestion?

 Regards
 Desmond Davids

 On Friday, 16 July 2010 08:00:25 UTC+2, Marc Gravell wrote:

 From the message, that *sounds* like protobuf-net...

 There is no built-in handling of this, but it is possibly something
 that could be added, especially in v2 which has a much more flexible
 model. For example, you *could* argue that there is an implicit schema that
 uses a repeated element per row, and the column ordinal for a field.

 There are a couple of points here thought

Re: [protobuf] Protogen and wcf json serialization

[Marc Gravell]

This is specifically protobuf-net; firstly, you should be able to add
-p:lightFramework to the command - that omits a few things that don't work
on all frameworks, including [Serializable] iirc.

However, you can also just edit CSharp.xslt to make any necessary changes.
Stick it alongside protogen.exe and it should be used automatically.

Marc
On 15 Jul 2014 21:01, Neta Nir n...@biocatch.com wrote:

 Hello,
 We are using Protogen in .Net environment. The  auto generated class from
 protgen is used as a  response in a service contract between a wcf service
 (http binding)  and a browser client. WCF uses JSON  serializer, the built
 in. The problem we are having is that the c# Response class (which is auto
 generated from protogen) is created with Properties, for example:


 *[global::System.Serializable,
 global::ProtoBuf.ProtoContract(Name = @Response)]*
 *public partial class Response : global::ProtoBuf.IExtensible*
 *{*
 *public Response() { } *
  *private bool _success;*
 *[global::ProtoBuf.ProtoMember(1, IsRequired = true, Name =
 @success, DataFormat = global::ProtoBuf.DataFormat.Default)]*
 *public bool success*
 *{*
 *get { return _success; }*
 *set { _success = value; }*
 *}*

 This class is serialized by the WCF default json serializer and returned
 to the client. The problem is that the json serialize output is composed
 from the *private *member of the class and not from the public
 properties, meaning that the browser response is returned in the following
 way:
 {_success:value} instead of {success:true}.
 This is because of the [*System.Serializable*] tag  on the class which is
 makes  all of the class  *fields* serialized  (no properties).
 Is there a way to tell protogen to create the class with only public
 fields and not properties? Any other suggestion will be welcome as well
 Thanks

  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Newbie here.. pass serializer functions to protobuf-net at runtime?

[Marc Gravell]

This is specific to protobuf-net. The library does not currently expose any
custom serialization extension point; to do that in a way that is genuinely
useful, while not allowing the caller to break the wire format, is quite
tricky. To date: it hasn't been necessary. If you can clarify *why* you
feel you need that, I might be able to help more. At the moment, the idea
is: you let the serializer know the model (whether that is explicit
configuration, attribute-based configuration, or inference from
constructors), and the the serializer worries about the rest.

Marc (protobuf-net)
On 11 Aug 2014 18:07, jens.bart...@googlemail.com wrote:

 Hi everyone,
 My problem is, that i can only determine at runtime how to serialize a
 certain class, called A for clarity.
 Accordingly, I was hoping for a way to tell the RuntimeTypeModel how to
 serialize A through a function/method call, for example by passing a
 delegate to it, or sth. similar.
 Is this possible? And if so, how?

 Thank you for your time
 Jens

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Proto-net Deserialization exception

[Marc Gravell]

My first thought is: what makes you think that you have read an entire
frame of data, and a *single* frame of data? socket read very rarely
conveniently forms itself into complete frames. Please see http://tiny.cc/io.
My second thought is: you told it to use the *entire* buffer, not just the
valo bytes that you know are valid - in which case the message is
*basically* right - you have garbage at the end of your buffer. You should
use the MemoryStream constructor that lets you control this:

new MemoryStream(buffer, 0, valo);

(note this should be after checking valo is positive, and you also need
to think about things like back-buffers if your messages turn out to be
larger than buffer.Length)

Thirdly: how was the data serialized? Did it use the WithLengthPrefix and
the same PrefixStyle?

Marc (protobuf-net)

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Deserialization problem with protobuf-net

[Marc Gravell]

Also; don't swallow exceptions. In fact, if you can't do anything useful,
don't even try/catch - just let it explode.

Marc

On 12 September 2014 10:44, Antonio Ramos ninira...@gmail.com wrote:

 Hi all, this is my first time with protobuf protocol and protobuf-net.

 I have the following question. I read in a buffer of bytes one response
 wich contains two proto messages. I´m trying deserializate those messages.

 I can deserializate the first one perfectly. But when i try to
 deserializate de second one i have a deserialization wired exception: 
 *Invalid
 wire-type; this usually means you have over-written a file without
 truncating or setting the length; see
 http://stackoverflow.com/q/2152978/23354
 http://stackoverflow.com/q/2152978/23354*

 My code is de following:

 int valo = networkStream.Read(buffer, 0, buffer.Length);

 using (MemoryStream streamR = new MemoryStream(buffer))
 {
 try
 {
 Command comando =
 Serializer.DeserializeWithLengthPrefixCommand(streamR,
 PrefixStyle.Base128);
 ChannelInfo chInfo =
 Serializer.DeserializeWithLengthPrefixChannelInfo(streamR,
 PrefixStyle.Base128);
 }
 catch (Exception ex)
 {
 }
 }

 What can i do???

 thanks!!

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.




-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Proto-net Deserialization exception

[Marc Gravell]

You need to know the message size to know that you have an entire message
to actually deserialize. That requirement cannot be glossed over, although
it can be handed to the library to worry about. What you have at the moment
is simply: broken socket code - nothing to do with protobuf.

If you don't want to have to worry about that: just ask the library to read
from the NetworkStream for you. When using the DeserializeWithLengthPrefix
methods it deliberately reads the prefix bytes very carefully to prevent
over-reading.

Marc
On 12 Sep 2014 22:45, Antonio Ramos ninira...@gmail.com wrote:

 But for use this constructor i have to know the message size, is´nt it?
 how can i do it? with a sizeof of the proto class??

 2014-09-12 21:22 GMT+02:00 Marc Gravell marc.grav...@gmail.com:

 My first thought is: what makes you think that you have read an entire
 frame of data, and a *single* frame of data? socket read very rarely
 conveniently forms itself into complete frames. Please see
 http://tiny.cc/io. My second thought is: you told it to use the *entire*
 buffer, not just the valo bytes that you know are valid - in which case
 the message is *basically* right - you have garbage at the end of your
 buffer. You should use the MemoryStream constructor that lets you control
 this:

 new MemoryStream(buffer, 0, valo);

 (note this should be after checking valo is positive, and you also need
 to think about things like back-buffers if your messages turn out to be
 larger than buffer.Length)

 Thirdly: how was the data serialized? Did it use the WithLengthPrefix and
 the same PrefixStyle?

 Marc (protobuf-net)




 --
 Antonio Ramos Recoder. http://antonioramosfotografia.es


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to define Structured Binary file

[Marc Gravell]

You cannot. What you describe is not what protobuf offers.
On 7 Oct 2014 00:55, RPMASA rudram...@gmail.com wrote:

 Hi,

 I have structured binary file specifying size of each element in bytes.
 How can I mentioned them in ptoto file?



No. Description Data Value Variable Type  Number of bytes  1 A1 0=unuse,
 1=use byte 1  2 A2 0=unuse, 1=use byte 1  3 A3 false=unuse, true=use byte
 1 2-2. Object[200] - this object can contains up to 200 times
No. Description Data Value Variable Type  Number of bytes  1 B1 1
 to 200 int 4  2 B2   string 40  3 B3   string 52  4 B4 0=unuse, 1=use byte
 1  5 B5 MMDD string 16  6 B6   string 30  7 B7 0=unuse, 1=use byte 1


 In above example how can I define that fields B2 is string and can have up
 to 40 bytes of data?

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Null Nested Messages--whats expected?

[Marc Gravell]

I can't reproduce this outputting a 0x0; my example is below, and
outputs 22-0A-12-08-73-6F-6D-65-43-69-74-79 - the null nested message is
simply completely omitted; the contents are:

- 2 bytes field-header and length-prefix for the location member
- 2 bytes field-header and length-prefix for the city member
- 8 bytes payload for the text someCity

Happy to investigate, but a repro would really help. To clarify: it is
incorrect to emit an 0x0 at the point of a field-header; that is never
valid.

As a random shot-in-the-dark guess, is it possible that you're actually
using a `MemoryStream` with `GetBuffer` but without the `Length`? if so,
there will be trailing garbage (which will default to 0x0) in the buffer,
precisely because `GetBuffer` returns the *oversized* backing buffer. To
get a right-sized array from `MemoryStream`, use `ToArray` instead (but
note that this involves an extra allocation / copy).

Marc


using ProtoBuf;
using System;
using System.IO;

static class Program
{
static void Main()
{
var a = new A {
location = new Location {
city = someCity,
coordinate = null
}
};
var ms = new MemoryStream();
Serializer.Serialize(ms, a);
var hex = BitConverter.ToString(
ms.GetBuffer(), 0, (int)ms.Length);
Console.WriteLine(hex);
// outputs: 22-0A-12-08-73-6F-6D-65-43-69-74-79
}
}
[ProtoContract]
class A
{
[ProtoMember(2)]
public string someField { get; set; }

[ProtoMember(4)]
public Location location { get; set; }
}
[ProtoContract]
class Location
{
[ProtoMember(1)]
public Coordinate coordinate { get; set; }
[ProtoMember(2)]
public string city { get; set; }
}
[ProtoContract]
class Coordinate
{
//...
}

On 14 October 2014 18:55, Johannes Elgh johannes.e...@tink.se wrote:

 Hello everyone,

 Firstly I need to apologize for any misuse or misunderstand of terms and
 concepts. I'm mostly familiar with the Protocol Buffer implementations
 protostuff and protobuf-net.


 I have a case that two different implementations (the C# one: protobuf-net
 and the Java one: protostuff) do differently and I'd like to know which way
 is the correct according to the Protocol Buffers specification.

 I have these messages:

 message A {
 optional string someField = 2;
 optional Location location = 4;
 }

 message Location {
 optional Coordinate coordinate = 1;
 optional string city = 2;
 }

 message Coordinate {
 ...
 }


 In the following case we have an object A, with a populated Location with 
 coordinate
 = null and city = someCity.

 When protobuf-net (a client) sends this object A (populated as described
 above) to protostuff (the backend) it sends 0x0 in the place of the 
 coordinate.
 This results into that protostuff reads a 0 whilst populating Location,
 and therefore thinks that Location is done. When protostuff then reads
 the next field number it finds 2, city, which has the same number as
 someField in A and populates that field instead (city overwrites someField
 ).

 So the problem is that protobuf-net sends 0x0 for a null nested messages
 whereas protostuff doesn't even expect it to be there (since it's null).

 Best regards,
 Johannes

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.




-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Null Nested Messages--whats expected?

[Marc Gravell]

Just for visibility; we've discussed this more on github, and it *looks* to 
be a protostuff decoding issue, not a protobuf-net encoding issue; Johannes 
will take our findings to protostuff for further investigation.

Arbitrary link is 
arbitrary: https://github.com/mgravell/protobuf-net/issues/35

Marc

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] generate .proto file from C# code

[Marc Gravell]

That output is from protobuf-net, not protoc. I already showed the code to
ask protobuf-net to have a go at writing a .proto schema from a type. Note
that it is not perfect and you may want to tweak names etc to match
convention. But names don't matter much to protobuf.

On 12 Sep 2016 1:59 pm, "Gayatri Shah" <gayatr...@gmail.com> wrote:

> Thank you Marc.  Do you have sample of .proto file?  What did you have to
> do to make it compile? I just wrote few lines of .proto file and I m using 
> *protoc-3.0.0-win32.zip
> <https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-win32.zip>
>  for
> compiler?   Can you also give me ur input (.proto file) for that you gout
> output *https://gist.github.com/mgravell/4967b490d40f13300919b018af23b282
>  .
>
> Thanks for your time.
>
>
> On Fri, Sep 9, 2016 at 7:08 PM, Marc Gravell <marc.grav...@gmail.com>
> wrote:
>
>> You'd be best off creating it by hand, but if you want a starting point,
>> you can use protobuf-net to get some hints; add
>> [ProtoContract(ImplicitFields = ImplicitFields.AllPublic)] to your root
>> type (UAVState?), and use something like 
>> Console.WriteLine(Serializer.GetProto());
>> to see the content. I had to invent a lot of stuff to get it to compile,
>> but that gives output like https://gist.github.com/m
>> gravell/4967b490d40f13300919b018af23b282 - how useful this is for actual
>> serialization depends a lot on your system.
>>
>> On 7 September 2016 at 13:41, Gayatri Shah <gayatr...@gmail.com> wrote:
>>
>>> I am new to Protocol buffer.  I have huge file that is in C# and I want
>>> to convert that to protocol buffer definition.
>>>
>>> How do I do that?  Do i need to write Protocol definition line by line?
>>>
>>> See attached file (Anything marked as non-serialized could be omitted;
>>> pretty much anything that’s a list can be skipped.)
>>>
>>> thanks in advance.
>>>
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Regards,
>>
>> Marc
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] generate .proto file from C# code

[Marc Gravell]

You'd be best off creating it by hand, but if you want a starting point,
you can use protobuf-net to get some hints; add
[ProtoContract(ImplicitFields = ImplicitFields.AllPublic)] to your root
type (UAVState?), and use something
like Console.WriteLine(Serializer.GetProto()); to see the
content. I had to invent a lot of stuff to get it to compile, but that
gives output like
https://gist.github.com/mgravell/4967b490d40f13300919b018af23b282 - how
useful this is for actual serialization depends a lot on your system.

On 7 September 2016 at 13:41, Gayatri Shah  wrote:

> I am new to Protocol buffer.  I have huge file that is in C# and I want to
> convert that to protocol buffer definition.
>
> How do I do that?  Do i need to write Protocol definition line by line?
>
> See attached file (Anything marked as non-serialized could be omitted;
> pretty much anything that’s a list can be skipped.)
>
> thanks in advance.
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] C# oneoof help

[Marc Gravell]

the language guide is very clear and explicit that this is intentional; it
is nothing to do with the C# part, and will behave the same for any
language:

> You can add fields of any type, but cannot use the required, optional, or
repeated keywords.

https://developers.google.com/protocol-buffers/docs/proto#oneof

And:

> A oneof cannot be repeated.

https://developers.google.com/protocol-buffers/docs/proto3#oneof

So: no, you can't do what you want here. Maybe use encapsulation. So: have
a message that *contains* a repeated field, and use that message type in
the one-of.


On 15 Jul 2017 2:53 p.m., "David Vescovi"  wrote:

First I tried this simple protobuf with compiler 3.3:

syntax = "proto3";


message ProxyMessage {

int32 Signal = 1;

oneof payload

 {

  repeated int32 a= 6;

  repeated float b = 7;

  }

}


while this seem quite simple it resulted in the following error:
protoc --csharp_out=. proxy.proto
proxy.proto:7:7: Fields in oneofs must not have labels (required / optional
/ repeated).
proxy.proto:8:7: Fields in oneofs must not have labels (required / optional
/ repeated).


Is there a workaround? Is this a known bug?


I know you can, in effect,  do this same thing in proto3 by not using the
oneof and rely on "optional by default" feature.
The reason I need the oneof is the fact on the other end I am using C++ and
I need the oneof union feature to keep the buffer allocation down.
That being said I really need something like:


syntax = "proto3";

message A

{

   repeated int32 Int32Parameters = 1;

}


message B

{

   repeated float FloatParameters = 1;

}

message ProxyMessage {

int32 Signal = 1;

oneof payload

  {

  A a= 6;

  B b = 7;

  }

}

This actually compiles but when I try to construct a message by doing
something like this:

public MainPage()

{

  this.InitializeComponent();

  RepeatedField myInts = new RepeatedField();


  myInts.Add(2);

  myInts.Add(3);


  ProxyMessage proxy = new ProxyMessage();

  proxy.Signal = 5;

  proxy.A.Int32Parameters.AddRange(myInts);

}


I get a NullReferenceException when it hits the AddRange line.


I also tried adding the line:

proxy.PayloadCase = ProxyMessage.PayloadOneofCase.A;


but this will not compile with "..can not assigned to --it is read only"



any ideas?


Dave




-- 
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Are protobufs serialised canonically?

[Marc Gravell]

Formally: no.

Practically: almost always

You shouldn't **demand** it.

Basically, it goes like this:

- the spec asks that writers *should* write fields in order
- the spec asks that readers *must* allow fields in any order
- data can be concatenated as a merge, meaning fields can appear out of
order
- how extension / unexpected fields are handled is often... inconsistent
- but in most cases: it'll probably work in the way you expect - just: it
isn't **required** to be identical

I'm aware of cases even in "protoc" (so: the C++ version) where it writes
fields out of order (relating to extension fields).

Additionally, the format itself allows for the same value to be expressed
in different ways - in particular as varint. Example: the value 1 is
usually expressed as the byte { 1 }, but actually { 129, 0 } and { 129,
128, 0 } should (if I have my math right) evaluate as 1. I am not aware of
anything that makes this alternative byte sequence formally invalid,
although I'd love to see an explicit / official answer on that.

Does that help any?

On 25 Jul 2017 9:26 p.m.,  wrote:

(Attempt 2 - I'm not sure if messages to this list and moderated or whether
my first one just got lost in the ether)

Are protobufs serialised canonically?  By that I mean is the same message
with the same fields populated with the same data guaranteed to serialise
to the same sequence of bytes, regardless of language/implementation
version?

The reason I ask is we're using serialised single-field protos as keys in a
Kafka topic, and I'm now wondering if this is safe.  The idea was to avoid
ambuiguity about how to interpret the contents of the key.  However, a
critical requirement of a key is that the same key must serialise to the
same bytes (Kafka is only aware of bytes - nothing else).

Is what I'm doing safe?  Is it safe for some subset of proto field types
(for example, I'm already pretty sure it's not safe for map types because
there is no canonical ordering, right?)?

Thanks,
Alex

-- 
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] why not 150

[Marc Gravell]

can you perhaps clarify the question?

On 2 Jul 2017 2:18 p.m., "Jing Lupeng"  wrote:

>
> https://developers.google.com/protocol-buffers/docs/encoding#structure
>
>
>
> 
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: How can i get bytes from a serialized message without the refix-length before each field?

[Marc Gravell]

(duplicate with existing answers 
here: 
https://stackoverflow.com/questions/45501605/is-it-possible-to-use-google-protobuffer-to-serialize-data-without-prefixing-the)

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Practical example for protobuf

[Marc Gravell]

wow, auto-complete went mad with that, sorry. "nay" => may; "red is" =>
redis, etc.

On 21 Jun 2017 8:22 a.m., "Marc Gravell" <marc.grav...@gmail.com> wrote:

> "You wish to express data in a mechanism that can be expressed as a byte
> stream, and/or communicate that between systems that nay or may not be
> using different architectures, in a way that is efficient both
> computationally and in terms of bandwidth; and a platform independent tool
> to express the data format of such data"
>
> This could include:
>
> - persisting state - for example  to a file in a single application, or as
> a row blob in a nosql data store
> - an API between two systems - which could be two systems in the same org,
> or could be between two independent orgs
>
> Basically, the need to store and/or communicate data comes up **all the
> time**. It is one of the most common things you'll ever be doing in
> programming. If you're in the http+ajax/ws world, you might default to JSON
> - but even in that scenario there nay be benefits to more efficient
> transfer mechanisms (and note: protobuf does include a JSON layer too). But
> outside of http+ajax/ws, efficiency may be a much much higher incentive.
>
> That's what protobuf offers.
>
> The public google API *alone* consists of over 200 .proto schemas for
> various APIs. I use protobuf for storing data in red is, and for various
> file formats.
>
>
>
> On 21 Jun 2017 4:47 a.m., <exoticaru...@gmail.com> wrote:
>
>> Being a student, I need to know where we really use protobufs in. A
>> clearer picture is required. Also tell me how it can be applied in the
>> client - server communication cases.
>> Thank you.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Practical example for protobuf

[Marc Gravell]

"You wish to express data in a mechanism that can be expressed as a byte
stream, and/or communicate that between systems that nay or may not be
using different architectures, in a way that is efficient both
computationally and in terms of bandwidth; and a platform independent tool
to express the data format of such data"

This could include:

- persisting state - for example  to a file in a single application, or as
a row blob in a nosql data store
- an API between two systems - which could be two systems in the same org,
or could be between two independent orgs

Basically, the need to store and/or communicate data comes up **all the
time**. It is one of the most common things you'll ever be doing in
programming. If you're in the http+ajax/ws world, you might default to JSON
- but even in that scenario there nay be benefits to more efficient
transfer mechanisms (and note: protobuf does include a JSON layer too). But
outside of http+ajax/ws, efficiency may be a much much higher incentive.

That's what protobuf offers.

The public google API *alone* consists of over 200 .proto schemas for
various APIs. I use protobuf for storing data in red is, and for various
file formats.



On 21 Jun 2017 4:47 a.m.,  wrote:

> Being a student, I need to know where we really use protobufs in. A
> clearer picture is required. Also tell me how it can be applied in the
> client - server communication cases.
> Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Maps and append-as-merge; expectation clarification please

[Marc Gravell]

Basically, if I have

message Foo {
int32 x = 1;
int32 y = 2;
}
message Bar {
map items = 1;
}

And I serialize 2 fragments appended:

items = { 1: { x: 123 } }

and (as the second appended chunk)

items = { 1: { y: 456 } }

And now I deserialize the entire chunk; is the map item with key 1:

- { x: 123, y: 456 }, or
- { y: 456 }, or
- something else

Or to put it another way: are message-based map items merged or strictly
replaced?

My understanding based on the docs is "replaced", so { y: 456 } - but: can
I get a confirmation on this?

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Practical example for protobuf

[Marc Gravell]

javascript (including browsers and node.js) can handle binary data just
fine, and my understanding is that this functionality is exposed by the
javascript implementation. I can't opine on that hugely, because my use of
protobuf is primarily backend - server-to-server, or server<--->storage.

Advantages:
- performance
- size
- schema definition

On 23 Jun 2017 7:53 a.m.,  wrote:

Thank you for your reply. i'm able to get a better picture now.

I also read that the byte stream which is produced can't be read by the
modern browsers which is why we have to convert it into json using
gwt(google web toolkit) or something like that. So automatically json is
used here. Then what makes protobufs advantageous than json. Is there any
advantage other than protobufs are structured? what makes google use
protobufs almost everywhere in their data transmission?


On Wednesday, June 21, 2017 at 9:17:15 AM UTC+5:30, exotic...@gmail.com
wrote:
>
> Being a student, I need to know where we really use protobufs in. A
> clearer picture is required. Also tell me how it can be applied in the
> client - server communication cases.
> Thank you.
>
-- 
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] online .proto parser available

[Marc Gravell]

Hey all; I've been doing some work with protobuf-net lately to bring it up 
to date with proto3, "timestamp", and all those things. As part of that, 
I've been reworking my entire code-gen pipeline, with the result that I 
accidentally wrote an interactive online editor while I was avoiding 
writing "msbuild" tooling support.

So; if it is any use to anyone, this site works (at least, it does 
today): https://protogen.marcgravell.com/

Current features / restrictions:

- supports protobuf-net's custom parser / generator, or any of the inbuilt 
"protoc" 3.3.0 languages
- single file in, single file out only (I need to switch to a tabbed 
editor, I think)
- import is disabled (I was nervous about abusing "import" to access system 
files)
- syntax highlighting for warnings and errors (don't get the full token via 
"protoc" currently, but... meh)

It will continue to evolve, but any feedback (good or bad) is appreciated.

I'm also OK with adding any well-known 3rd party plugins as long as they 
use the protoc plugin API and run on Windows; theoretically I *could* move 
the server to linux (it is asp.net core), but I haven't tested that!

Cheers;

Marc (protobuf-net)

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Generating an (C#) Interface and extendable data model from .proto, namespacing and inheritence

[Marc Gravell]

I can't speak for the design choices - but *as I understand it*, the key
point of protobuf is to enable things to work well cross-platform. That
means that at the DTO level, things need to be *possible* to implement in a
wide range of languages, and the reality is that not all frameworks would
allow what you describe. So building them into the protocol would limit the
target scope, or force several to be second-class citizens.

In the case of C#, we are blessed by things like "partial classes" and
"extension methods" (and hopefully "extension everything" in a future
language version); that combined with encapsulation means you already have
a huge range of options. But - something I've said time and time again, and
coming from someone who *deliberately set out* to create a protobuf
implementation that works well against your pre-existing POCOs:

Serializing your domain model is fine, as long as it works perfectly. As
soon as it gets *even remotely* complicated - or you start having to fight
either your POCO model or the serializer, then the best thing to do is to
very deliberately and neatly disconnect the two worlds. Have one object
model that is your domain model - the POCO world. And have a *completely
separate* model for serialization - the DTO world. How you map between them
is an uninteresting implementation detail, IMO. There are tools that help,
or static conversion operators, or whatever else you like.

That said: if altering the code generator works for you: great! do that!

As to your questions:

1: if I understand you correctly, then that won't impact the serialization
or reflection/descriptor APIs at all
2: as it happens, protobuf-net already models inheritance via optional
members, and I will be migrating that to "oneof" in proto3; as part of
this, it is fully my plan to implement a custom markers (via the extension
API) that work in my own codegen, to help it recognize when to emit C#
inheritance from a .proto; so ... you wouldn't be alone in that idea (note:
my implementation is not the official Google one)
3:  what generator are you using, specifically (including version number)?
Is it possible to give a minimal example of a .proto and the C# that is
offending you, and what you want it to be instead? It would be much easier
to discuss with a concrete example.

Marc

On 30 May 2017 at 12:41, KarrokDC  wrote:

> Hi all,
>
> I was wondering if any thought has gone into generating an interface to go
> along with the generated data classes from a proto.
> And if it had been considered were there any reasons to not do it?
>
> Quick sketch of my current situation:
>
> I have a project that has a complex and messy internal data structure
> (assume for now its too expensive to refactor).
> On top of this I created an interface that already bears semblance to a
> .proto model we use. And an implementation of this interface which has a
> reference to an internal object and 'transmutes' the data from the internal
> model to the external format. So aside from the reference to the internal
> object, no data is held by the interface implementation. It basically acts
> as an adapter.
>
> The implemented getters/setters allow the transmutation of internal data
> structures to 'external' structures
>
> 
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> I am now however looking into simply generating this interface and part of
> the implementation to allow this model to be exported easily, while still
> having a clean API, which would (in time) also allow for code written
> against the generated API (also C++/python) to be portable.
>
> For this I've been hacking away a bit at the C# proto generation code in
> an effort to generate something that would fit into the design below:
>
>
> 
>
>
>
>
> What I've done so far
>
>
>- Added a mirror of csharp_reflection_class as
>csharp_reflection_interface (ReflectionInterfaceGenerator)  to generate
>what I would like my interface to look like
>- Added interface-ended generation options to protoc cli
>ie. --csharp_opt=generate_interface=true
>- Changed/added namespaces to use in generation (because in my case,
>the interface is in a different namespace than the implementation)
>- Added a way to extract (ie not generate) property get;set; (similar
>to how https://silentorbit.com/protobuf/ generates from proto), the
>developer will need to implement these themselves, but having the interface
>enforces it.
>
> Now there are a number of open questions/issues that I thought proto
> devs/experienced C# developers might be able to weigh in.
>
>1. In various areas, the package name defined in the .proto is used as
>a way to create the namespace, 

Re: [protobuf] map support in protobuf version 2

[Marc Gravell]

It is supported in the proto2 *language*, but you'll need to use a *proto
compiler version* that understands it. Depending on the implementation
(i.e. which language and library you are generating for), you *may* also
need to use a *library* version that has additional features to support
maps.

Note that the binary format does not change between version 2 and version
3, but the proto compiler has changes (to understand the back-ported syntax
changes), and the library may or may not have changes.

Basically, that guidance should be read as:

"You can use map in your proto2 schemas without needing to change to proto3
DSL syntax, but you will need to use an up to data version of protoc and
related tools"


If you can't update your tooling and library versions, then you can still
*read and write* map data. You can just use:

message MyMapEntry {
optional int32 key = 1 [default = 0]; // change per your TKey
optional string value = 2 [default = ""]; // change per your TValue
}
repeated MyMapEntry data = 1;

with the slight caveat that it will materialize as a list/array rather than
a native dictionary.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] map support in protobuf version 2

[Marc Gravell]

Oops; I meant to reply to group; email is hard...

(repeats response)

What protoc compiler version are you using? what does protoc --version say?
It should be 3.3.0 or similar.

The following works fine using protoc (note it defaults to protogen -
you'll need to change the drop-down to protoc 3.3.0, for example "(protoc)
Java"): https://protogen.marcgravell.com/#ga1fce91670098b10f326bf20f99ae7d6

(which is just https://protogen.marcgravell.com/ which hosts "protoc" and
"protogen", pre-loading with gist https://gist.github.com/m
gravell/a1fce91670098b10f326bf20f99ae7d6 via client-side javascript)

On 13 June 2017 at 11:04, Nagarjuna Reddy  wrote:

> Hi,
>
> As per the below link https://developers.google.com/protocol-buffers/
> docs/proto#maps, I see that maps are supported in protobuf2.
> But when I use them I get the following error
>
> Expected "required", "optional", or "repeated".
>
> Is the feature really supported in protobuf2? As per some of the queries
> on this forum others are also facing problem in protobuf2.
>
> Regards,
> Nagarjuna
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Is the extension registry address still monitored?

[Marc Gravell]

In descriptor.proto, it asks public authors to request extension numbers (for 
the DSL) via protobuf-global-extension-regis...@google.com

I've done this, but without response. Is this address being monitored?

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] [libprotobuf ERROR google/protobuf/message_lite.cc:118] Can't parse message of type "RegisterWatcherResponse" because it is missing required fields: rc

[Marc Gravell]

(I'm assumig the "optional" isn't there in the real example)

You can't simply change between proto2 and proto3. In proto3 everything is
optional with an implicit zero default. This is fundamentally incompatible
with consumers who have a "required" expectation **unless** you happen to
never send the value zero. Because whenever your value is zero, it will be
omitted and the consumer will complain about a missing required field.

Options:

- update the consumers **first** so that they don't have a "required"
expectation
- never send zero in that field
- don't change to proto3

Marc

On 16 Jun 2017 10:04 p.m., "yihao yang"  wrote:

> I updated a proto file from proto2 to proto3 as follows.
>
> Previous proto2:
> message RegisterWatcherResponse {
>  required int32 rc = 1;
>  optional uint64 sendTime = 2;
> }
>
>
> Now proto3:
> message RegisterWatcherResponse {
>   int32 rc = 1;
>   optional uint64 sendTime = 2;
> }
>
> I tested compatibility by using old version client with using proto2
> RegisterWatcherResponse and receiving responses from a server using
> proto3 RegisterWatcherResponse.
> It turns out clients got failure message like follows:
> [libprotobuf ERROR google/protobuf/message_lite.cc:118] Can't parse
> message of type "RegisterWatcherResponse" because it is missing required
> fields: rc
> I'm sure the server side responses have rc filed.
> Can anyone help me why I got this error? Is proto3 compatible with proto2
> in this scenario?
> What can I do to make it work?
>
> Thanks,
> Yihao
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Additional tool for ripping apart schema-less protobufs

[Marc Gravell]

This will mostly be useful to library maintainers, etc. I know that protoc 
has some tools in this area, but I found them ... "unhelpful" for my 
purposes.

I very often end up digging through protobuf files of unknown origin 
without a schema etc, to either help support a stuck user, or to understand 
differences between different library implementations. I've got tired of 
doing this by hand or having to get my usual tools in place, so I've added 
an online decoder / visualizer here: https://protogen.marcgravell.com/decode

Input: protobuf data (without schema) - as a file, base-64, or hex

Output: hierarchical walk through the data with typical and alternative 
decoding values.

If the collective "you" can find a use for it: great!

(if your data is "mega top secret" you can also pull it from git and 
self-host with the network cable unplugged, and then immediately shred your 
PC afterwards and deny that it ever existed - or whatever infosec 
compliance means to you)

Marc (protobuf-net)

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Transfer actual code with Protobuffers

[Marc Gravell]

That's not something that protobuf targets. If you have a syntax that
allows you to convey an expression as a string or similar, then by all
means do that - but protobuf will just see it as an opaque string. Note
that there are usually a range of concerns when transferring executable
code (of any kind) - less of a concern for server-to-client, but not
removed. Think "XSS", for a trivial example.

On 30 May 2017 7:18 p.m., "Alex TS"  wrote:

Hey everyone,
I'm quite new to Protocol Buffers and don't really have a background on the
theme, so pardon me if this sounds absurd or if this is not the right forum.

With that out of the way, if I'd like to transfer say, a method, from the
server to execute on the client. Is that possible at all using the `bytes`
type or something similar?

The issue being that in an ideal world I'd like to run a method on the
server with the arguments provided by the client and get it back resolved
for injection.

As a silly example, something like this, in JS:

*ServiceX*
const privateMethod = (x) => x * x
const publicMethod = (elementId, y) => document.getElementById(elementId).
innerHTML(privateMethod(y))


*ClientY*
server.publicMethod("root", 3)


Is it possible to transfer the `publicMethod` itself from the server for a
later execution triggered by the client and having the side effects also on
the client? My idea here being basically to be able to create an
application in one service in an isolated way and inject it on another
service that doesn't have to know any details of its implementation details.

Thanks for your help!

-- 
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Null String in C# (CheckForNull)

[Marc Gravell]

Fields are optional but the implicit default for a string is a zero length
string, not a null length string. To be honest, either approach seems
perfectly reasonable as long as it is documented and any exception is clear
and obvious. For my separate implementation I chose to interpret nulls as
"meh, not set, ignore it", but... either is fine IMO.

On 3 October 2017 at 19:35, Everton Araujo  wrote:

> Hi all,
>
> I'm facing the situation where a string value is not allowed to be null in
> Protobuf 3 using C# (CheckNotNull).
> I do not understand it considering that fields are optional by default in
> proto3.
> Does anyone knows the original reason for this constraint? As far as I
> could see, it happens for string and bytearrays.
>
> Regards,
>
> Everton
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Not able to serialize Object type

[Marc Gravell]

 I'm assuming this is protobuf-net; the message is right : the library
can't work with "object itemField". Perhaps the best thing would be to
treat this like a "oneof" (in .proto terms) and have an accessor per
possible type. For example:

[ProtoMember(n)]
private int ValueInt32 {
get => (int)itemField;
set => itemField = value;
}
private bool ShouldSerializeValueInt32() => itemField is int;

(with a different "n" per possible type)

The "ShouldSerialize..." pattern is a .net idiom supported by protobuf-net
that provides memberwise conditional serialization.



On 23 Oct 2017 8:07 a.m., "Nihar Mishra"  wrote:

>
> Hi,
>
> I have below class which is generated from .net client.
> When i try to save in redis cache, i get the "Not able to serialize Object
> type" error.
> Please help me how to resolve it.
>
>   public partial class AttributeValueType
> {
> [DataMember]
> private object itemField;
>
>
> [System.Xml.Serialization.XmlElementAttribute("IntValue",
> typeof(int))]
> [System.Xml.Serialization.XmlElementAttribute("StringValue",
> typeof(string))]
> [System.Xml.Serialization.XmlElementAttribute("dateValue",
> typeof(System.DateTime), DataType = "date")]
> [System.Xml.Serialization.XmlElementAttribute("doubleValue",
> typeof(double))]
> public object Item
> {
> get { return this.itemField; }
> set { this.itemField = value; }
> }
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Not able to serialize Object type

[Marc Gravell]

Yes, something like that; example: https://pastebin.com/CUvWz00L

On 23 October 2017 at 09:38, Nihar Mishra <itnmis...@gmail.com> wrote:

> Hi Marc,
> Thanks a lot. Yes this is protobuf-net.
> Still i don't get how to modify AttributeValueType class.
>
> Do you mean that i should write 4 protomembers for the 4 types?
>
> best Regards,
> Nihar
>
>
> On Mon, Oct 23, 2017 at 9:19 AM, Marc Gravell <marc.grav...@gmail.com>
> wrote:
>
>>  I'm assuming this is protobuf-net; the message is right : the library
>> can't work with "object itemField". Perhaps the best thing would be to
>> treat this like a "oneof" (in .proto terms) and have an accessor per
>> possible type. For example:
>>
>> [ProtoMember(n)]
>> private int ValueInt32 {
>> get => (int)itemField;
>> set => itemField = value;
>> }
>> private bool ShouldSerializeValueInt32() => itemField is int;
>>
>> (with a different "n" per possible type)
>>
>> The "ShouldSerialize..." pattern is a .net idiom supported by
>> protobuf-net that provides memberwise conditional serialization.
>>
>>
>>
>> On 23 Oct 2017 8:07 a.m., "Nihar Mishra" <itnmis...@gmail.com> wrote:
>>
>>>
>>> Hi,
>>>
>>> I have below class which is generated from .net client.
>>> When i try to save in redis cache, i get the "Not able to serialize
>>> Object type" error.
>>> Please help me how to resolve it.
>>>
>>>   public partial class AttributeValueType
>>> {
>>> [DataMember]
>>> private object itemField;
>>>
>>>
>>> [System.Xml.Serialization.XmlElementAttribute("IntValue",
>>> typeof(int))]
>>> [System.Xml.Serialization.XmlElementAttribute("StringValue",
>>> typeof(string))]
>>> [System.Xml.Serialization.XmlElementAttribute("dateValue",
>>> typeof(System.DateTime), DataType = "date")]
>>> [System.Xml.Serialization.XmlElementAttribute("doubleValue",
>>> typeof(double))]
>>> public object Item
>>> {
>>> get { return this.itemField; }
>>> set { this.itemField = value; }
>>> }
>>> }
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Not able to serialize Object type

[Marc Gravell]

For completeness (mostly in case someone finds this via search in the
future), if you were using .proto schema-based generation, "oneof" is your
friend: https://protogen.marcgravell.com/#g01989f8da1f4ce8ab358756115478c37
(click "Generate" once the gist has loaded)

On 23 October 2017 at 09:48, Marc Gravell <marc.grav...@gmail.com> wrote:

> Yes, something like that; example: https://pastebin.com/CUvWz00L
>
> On 23 October 2017 at 09:38, Nihar Mishra <itnmis...@gmail.com> wrote:
>
>> Hi Marc,
>> Thanks a lot. Yes this is protobuf-net.
>> Still i don't get how to modify AttributeValueType class.
>>
>> Do you mean that i should write 4 protomembers for the 4 types?
>>
>> best Regards,
>> Nihar
>>
>>
>> On Mon, Oct 23, 2017 at 9:19 AM, Marc Gravell <marc.grav...@gmail.com>
>> wrote:
>>
>>>  I'm assuming this is protobuf-net; the message is right : the library
>>> can't work with "object itemField". Perhaps the best thing would be to
>>> treat this like a "oneof" (in .proto terms) and have an accessor per
>>> possible type. For example:
>>>
>>> [ProtoMember(n)]
>>> private int ValueInt32 {
>>> get => (int)itemField;
>>> set => itemField = value;
>>> }
>>> private bool ShouldSerializeValueInt32() => itemField is int;
>>>
>>> (with a different "n" per possible type)
>>>
>>> The "ShouldSerialize..." pattern is a .net idiom supported by
>>> protobuf-net that provides memberwise conditional serialization.
>>>
>>>
>>>
>>> On 23 Oct 2017 8:07 a.m., "Nihar Mishra" <itnmis...@gmail.com> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I have below class which is generated from .net client.
>>>> When i try to save in redis cache, i get the "Not able to serialize
>>>> Object type" error.
>>>> Please help me how to resolve it.
>>>>
>>>>   public partial class AttributeValueType
>>>> {
>>>> [DataMember]
>>>> private object itemField;
>>>>
>>>>
>>>> [System.Xml.Serialization.XmlElementAttribute("IntValue",
>>>> typeof(int))]
>>>> [System.Xml.Serialization.XmlElementAttribute("StringValue",
>>>> typeof(string))]
>>>> [System.Xml.Serialization.XmlElementAttribute("dateValue",
>>>> typeof(System.DateTime), DataType = "date")]
>>>> [System.Xml.Serialization.XmlElementAttribute("doubleValue",
>>>> typeof(double))]
>>>> public object Item
>>>> {
>>>> get { return this.itemField; }
>>>> set { this.itemField = value; }
>>>> }
>>>> }
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Protocol Buffers" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to protobuf+unsubscr...@googlegroups.com.
>>>> To post to this group, send email to protobuf@googlegroups.com.
>>>> Visit this group at https://groups.google.com/group/protobuf.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>
>
>
> --
> Regards,
>
> Marc
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to combine FieldOptions with Default Values in Messages

[Marc Gravell]

requested claritication and *possibly* answered here:
https://stackoverflow.com/questions/47246474/protobuf-how-to-combine-fieldoptions-with-default-values-in-messages

On 12 Nov 2017 7:12 a.m.,  wrote:

> I have an message using FieldOptions but I also want to use default values
> as well, how do I combine two of those things ?
>
> extend google.protobuf.MessageOptions {
> optional CommandInfo cmd = 5678;
> }
>
>
> message TestCmd {
>
> optional bool BoolFlag = 1 [(info) = { name: "bool-flag" value: "false" 
> shorthand: "c" usage: "test"} ];
> optional string StringFlag = 2 [ (info) = { name: "string-flag" value: "" 
> shorthand: "o" usage: "test"}];
>
> }
>
>
>
> I want that TestCmd.BoolFlag default value to be bool value  "false", 
> currently I'm using string value then convert it to bool, How do I
>
> specify the default value in a type-safe way ?
>
>
>
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Upgrading float fields to double

[Marc Gravell]

Not really, no. They take different amounts of space on the wire, and have
a different declared wire type (header). Some libraries may choose to be
gracious and apply the conversion silently, but other libraries could just
say "unexpected wire type" and stop processing.

You could perhaps do it as a `oneof`, and have the client check both?

On 15 Nov 2017 11:51 p.m., "James Philbin"  wrote:

> Hi,
>
> Is it safe to upgrade float fields to doubles for serialized protos? If
> not, why not?
>
> Thanks,
> James
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Guid conventions

[Marc Gravell]

One thought on the bytes vs string aspect: as shown in the SO question, it
isn't beyond imagining to want to use a guid/uuid with "map" - which
wouldn't be possible with (raw) bytes.

As an additional consideration for bytes : there are at least two binary
representations of guid/uuid - i.e. which order the bytes come vs the
visual string representation (either left to right byte equivalent, or a
crazy-endian version where individual portions at different 1/2/4-byte
lengths are individually reversed).

It is these ambiguities that makes me think - after hacking around it for a
long time - that a specific type might be useful.

Proposal

For efficiency, I would propose a new primitive keyword in .proto terms,
that is essentially the same as "bytes", but is mapped (when possible) to a
target platforms natural unique identifier primitive. If no such primitive
is available (JavaScript, for example), it would be decoded *creating a
string*, but expanding to the hex representations - so the byte 0xB7 would
contribute the two characters "B7" to the string. To be clarified: with of
without hyphen group separators. The reason for this string is to
facilitate use in maps, and the correct unit semantics, even on platforms
without a native uuid type. In terms of the binary payload: it would be
left-to-right byte-for-byte with the visual representation, so the uuid
starting "010203040506..." would contribute bytes 0x010203040506...

The size would always be 16 bytes but would be written with the usual
length prefix, so: 17 bytes overall. Any unexpected length prefix
encountered would result in a deserialization failure. Where a platform
doesn't have a native uuid type this 16 byte restriction would be enforced
at the point of assignment/addition.

Additional thought: "repeated uuid" (or whatever the keyword) *would be
allowed to use "packed" encoding. A receiver would divide the packed length
prefix by 16 to determine the number of elements.

In JSON it would be formatted in the text version described above for
platforms without a native uuid type.

---end

So: that's my idea of how uuids *should* be handled. But there's quite a
bit of work, and it sounds like not much dramatic call for them to be
added. So it probably won't happen, being realistic. But I just wanted to
brain-dump that.

Marc



On 8 Dec 2017 12:39 a.m., "Adam Cozzette" <acozze...@google.com> wrote:

> I haven't had to store a GUID/UUID in a proto before but it seems like
> string or bytes would be the best choice. You would definitely want to use
> bytes (not string) if you're using the binary representation, since string
> fields are for UTF-8 only. We could consider eventually creating a
> well-known type but I'm not sure how much demand there is for one.
>
> On Wed, Dec 6, 2017 at 11:46 AM, Marc Gravell <marc.grav...@gmail.com>
> wrote:
>
>> A question on Stack Overflow earlier (https://stackoverflow.com/que
>> stions/47674930/google-protobuf-proto-file-query/4767629) reminded me
>> that I'm not fully "up" on the conventions for using guids in protobuf.
>>
>> There's no primitive / keyword for them, and AFAIK no "well known type".
>> So : how do folks tend to handle guids? Strings? Bytes? *Should* there be a
>> stronger guid story? Or is this just a non-issue?
>>
>> Thoughts?
>>
>> Marc
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Guid conventions

[Marc Gravell]

A question on Stack Overflow earlier (
https://stackoverflow.com/questions/47674930/google-protobuf-proto-file-query/4767629)
reminded me that I'm not fully "up" on the conventions for using guids in
protobuf.

There's no primitive / keyword for them, and AFAIK no "well known type". So
: how do folks tend to handle guids? Strings? Bytes? *Should* there be a
stronger guid story? Or is this just a non-issue?

Thoughts?

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Validate .proto file in Java

[Marc Gravell]

Apparently https://github.com/square/wire/ includes a runtime .proto parser
for Java. That might help? It isn't the official one, note.

On 6 Dec 2017 8:28 p.m., "Omar Al-Safi"  wrote:

> Hello folks,
>
> I stubbled upon a requirements that I receive a textual representation of
> a proto file from an external service and then I need to save it. I want
> first to check if the text is a valid protobuf schema representation, is
> there way a to validate the syntax of a proto file during the runtime? I
> know that I can use the protoc compiler to validate a file and produce the
> compiled schema but my intention here just only to* validate the syntax*
> of the textual representation of a text if is a valid proto schema syntax.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any way to "extract all" using protoc or any other tool?

[Marc Gravell]

You can and it does. The problem is that the wire format by itself doesn't
tell it **what message type** the root object is. So you need to tell it in
the additional parameter to --decode

On 10 Dec 2017 3:14 p.m., "Jim Baldwin"  wrote:

It's not really just a sequence; it's a hierarchy, isn't it?  Why can't I
use --decode  or something like that?


On Saturday, December 9, 2017 at 4:20:15 PM UTC-8, Ilia Mirkin wrote:

> On Sat, Dec 9, 2017 at 5:52 PM, Jim Baldwin  wrote:
> > I have a protobuf file, and a .proto file that describes the schema.
> >
> > The .proto describes dozens of different messages that may be in the
> > protobuf file.
> >
> > I would like to know what messages can be found in the file.  I do a
> protoc
> > --decode_raw and get something out, but I don't see how to use that to
> > figure out how to extract messages from the file.
> >
> > I assume there's something I don't get about protobufs, but it seems to
> me I
> > should be able to take a protobuf data file and corresponding .proto and
> > turn it into a file that lets me see what the message hierarchy is in
> the
> > file.  JSON would be a great way to do that.
> >
> > What am I missing?
>
> An encoded protobuf is just a sequence of (tag, value) pairs. If you
> don't know which proto it is, decode_raw is the best you can do. If
> you do know which proto it is, you can use --decode instead and pass
> it a proto name to use for the decoding.
>
> Cheers,
>
>   -ilia
>
-- 
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any way to "extract all" using protoc or any other tool?

[Marc Gravell]

oh, there is a root. if i had to guess it has something like:


message TheRootMessageType {
required string name = 1;
repeated SomeNoun items = 100;
}



On 10 Dec 2017 4:42 p.m., "Jim Baldwin" <j...@baldw.in> wrote:

> In this case, it looks like there isn’t a root.  Rather, this format is a
> series of top-level parameters.  So, I have to give it the Parameter I’m
> looking for.  The problem I have with this is the order of parameters
> _might_ matter, and I lose that by only looking for one.
>
> On Dec 10, 2017, at 8:35 AM, Marc Gravell <marc.grav...@gmail.com> wrote:
>
> as I ubderstand it, --decode *will do that*. it doesn't decode *just* the
> root : but, it needs to know the root message type in order to correctly
> interpret the data
>
> On 10 Dec 2017 4:32 p.m., "Jim Baldwin" <jmbl...@gmail.com> wrote:
>
>> OK, this helps.  I need to figure out what the "root" message is.  It
>> seems like an omission in the whole PB thing that you can't specify the
>> .proto and do a --decode_everything.
>>
>> On Sunday, December 10, 2017 at 8:23:12 AM UTC-8, Marc Gravell wrote:
>>>
>>> You can and it does. The problem is that the wire format by itself
>>> doesn't tell it **what message type** the root object is. So you need to
>>> tell it in the additional parameter to --decode
>>>
>>> On 10 Dec 2017 3:14 p.m., "Jim Baldwin" <jmb...@gmail.com> wrote:
>>>
>>> It's not really just a sequence; it's a hierarchy, isn't it?  Why can't
>>> I use --decode  or something like that?
>>>
>>>
>>> On Saturday, December 9, 2017 at 4:20:15 PM UTC-8, Ilia Mirkin wrote:
>>>
>>>> On Sat, Dec 9, 2017 at 5:52 PM, Jim Baldwin <jmb...@gmail.com> wrote:
>>>> > I have a protobuf file, and a .proto file that describes the schema.
>>>> >
>>>> > The .proto describes dozens of different messages that may be in the
>>>> > protobuf file.
>>>> >
>>>> > I would like to know what messages can be found in the file.  I do a
>>>> protoc
>>>> > --decode_raw and get something out, but I don't see how to use that to
>>>>
>>>> > figure out how to extract messages from the file.
>>>> >
>>>> > I assume there's something I don't get about protobufs, but it seems
>>>> to me I
>>>> > should be able to take a protobuf data file and corresponding .proto
>>>> and
>>>> > turn it into a file that lets me see what the message hierarchy is in
>>>> the
>>>> > file.  JSON would be a great way to do that.
>>>> >
>>>> > What am I missing?
>>>>
>>>> An encoded protobuf is just a sequence of (tag, value) pairs. If you
>>>> don't know which proto it is, decode_raw is the best you can do. If
>>>> you do know which proto it is, you can use --decode instead and pass
>>>> it a proto name to use for the decoding.
>>>>
>>>> Cheers,
>>>>
>>>>   -ilia
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Any way to "extract all" using protoc or any other tool?

[Marc Gravell]

as I ubderstand it, --decode *will do that*. it doesn't decode *just* the
root : but, it needs to know the root message type in order to correctly
interpret the data

On 10 Dec 2017 4:32 p.m., "Jim Baldwin" <jmbl...@gmail.com> wrote:

> OK, this helps.  I need to figure out what the "root" message is.  It
> seems like an omission in the whole PB thing that you can't specify the
> .proto and do a --decode_everything.
>
> On Sunday, December 10, 2017 at 8:23:12 AM UTC-8, Marc Gravell wrote:
>>
>> You can and it does. The problem is that the wire format by itself
>> doesn't tell it **what message type** the root object is. So you need to
>> tell it in the additional parameter to --decode
>>
>> On 10 Dec 2017 3:14 p.m., "Jim Baldwin" <jmb...@gmail.com> wrote:
>>
>> It's not really just a sequence; it's a hierarchy, isn't it?  Why can't I
>> use --decode  or something like that?
>>
>>
>> On Saturday, December 9, 2017 at 4:20:15 PM UTC-8, Ilia Mirkin wrote:
>>
>>> On Sat, Dec 9, 2017 at 5:52 PM, Jim Baldwin <jmb...@gmail.com> wrote:
>>> > I have a protobuf file, and a .proto file that describes the schema.
>>> >
>>> > The .proto describes dozens of different messages that may be in the
>>> > protobuf file.
>>> >
>>> > I would like to know what messages can be found in the file.  I do a
>>> protoc
>>> > --decode_raw and get something out, but I don't see how to use that to
>>> > figure out how to extract messages from the file.
>>> >
>>> > I assume there's something I don't get about protobufs, but it seems
>>> to me I
>>> > should be able to take a protobuf data file and corresponding .proto
>>> and
>>> > turn it into a file that lets me see what the message hierarchy is in
>>> the
>>> > file.  JSON would be a great way to do that.
>>> >
>>> > What am I missing?
>>>
>>> An encoded protobuf is just a sequence of (tag, value) pairs. If you
>>> don't know which proto it is, decode_raw is the best you can do. If
>>> you do know which proto it is, you can use --decode instead and pass
>>> it a proto name to use for the decoding.
>>>
>>> Cheers,
>>>
>>>   -ilia
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+u...@googlegroups.com.
>> To post to this group, send email to prot...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Any way to "extract all" using protoc or any other tool?

[Marc Gravell]

They are field numbers. They don't mean anything by themselves other than
to identify each field. If you want to know the logical *name* of each
field, you need the .proto schema.

On 10 Dec 2017 4:23 p.m., "Jim Baldwin"  wrote:

> Perhaps it might help if I understood the output of protoc --decode_raw.
>
> Here's an example of a .caffemodel file I'm trying to inspect.  Is there a
> description of what the numbers mean in this file?
>
> 1: "VGG_ILSVRC_16_layers"
> 100 {
>   1: "input-data"
>   2: "Python"
>   4: "data"
>   4: "im_info"
>   4: "gt_boxes"
>   10: 0
>   130 {
> 1: "roi_data_layer.layer"
> 2: "RoIDataLayer"
> 3: "\'num_classes\': 2"
>   }
> }
> 100 {
>   1: "data_input-data_0_split"
>   2: "Split"
>   3: "data"
>   4: "data_input-data_0_split_0"
>   4: "data_input-data_0_split_1"
>   10: 0
> }
>
> On Saturday, December 9, 2017 at 2:52:10 PM UTC-8, Jim Baldwin wrote:
>>
>> I have a protobuf file, and a .proto file that describes the schema.
>>
>> The .proto describes dozens of different messages that may be in the
>> protobuf file.
>>
>> I would like to know what messages can be found in the file.  I do a
>> protoc --decode_raw and get something out, but I don't see how to use that
>> to figure out how to extract messages from the file.
>>
>> I assume there's something I don't get about protobufs, but it seems to
>> me I should be able to take a protobuf data file and corresponding .proto
>> and turn it into a file that lets me see what the message hierarchy is in
>> the file.  JSON would be a great way to do that.
>>
>> What am I missing?
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: validating protobuf messages

[Marc Gravell]

a zero tag is never valid in any protobuf data, although it wouldn't be
unheard of for folks to use a zero tag as a sentinel value to demark
multiple root messages. Protoc has some facilities to check the insides of
a message that might help you figure out how likely it is to be a match,
but it won't make a zero tag work.

On 10 Dec 2017 6:35 p.m., "Annu Bansal"  wrote:

> Hey Jeroen!
> Dis you get any method to check if a .pb file is generated from givem
> .proto file? Actually, I'm getting error while deserializing a .pb file,
> "The protocol message contained invalid tag(zero)." I was wondering if the
> two files are compatible.
> Thanks in advance.
>
> On Saturday, November 17, 2012 at 5:38:16 AM UTC+5:30, Jeroen Ooms wrote:
>>
>> I am using the RProtoBuf package, which interfaces to the c++ protobuf
>> library. I was wondering if there is any way of 'validating' a pb
>> message, in the sense that you can check if it actually is a valid
>> message for a given pb description. Currently, RProtoBuf is very
>> permissive and will basically accept anything. E.g the code below will
>> not throw an error, even though the file obviously is not a valid
>> tutorial.Person message:
>>
>> library(RProtoBuf)
>> read(tutorial.Person, "/etc/passwd")
>>
>> In practice, it is quite easy to introduce errors by accidentally reading
>> a message using the wrong description. It would be very helpful if there
>> was a way of preventing this kind of problems by some sort of validation.
>>
>> Is there support for this in the c++ library, or does protobuf basically
>> assume that the user/application has some other way of validating a message?
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf3 InvalidProtocolBufferException with some strings

[Marc Gravell]

(mainly for the list) see also the stackoverflow question:
https://stackoverflow.com/q/50387660/23354

On Thu, 17 May 2018 at 10:42, Alexey Vishnyakov 
wrote:

> Hello
>
> We using protobuf v.3 to transfer messages from C# client to Java server
> over HTTP.
>
> The message proto looks like this:
>
> message CLIENT_MESSAGE {
> string message = 1;
> }
>
> Both client and server uses UTF-8 character encoding for strings.
>
> Everything is fine whe we are using short string values like "abc", but
> when we trying to transfer string with 198 chars in it, we catchig an
> Exception:
>
>
>com.google.protobuf.InvalidProtocolBufferException:
> While parsing a protocol message, the input ended unexpectedly in the
> middle of a field. This could mean either that the input has been
> truncated or that an embedded message misreported its own length.
>
>
> We tried to compare even byte array containing protobuf data, and didn't
> found a solution.
> For "aaa" string byte array starts with this bytes:
>
> 10 3 97 97 97
>
>
> Where 10 is protobuf field number, and 3 is string length, 69 65 67 is
> "aaa".
>
> For string
>
>
>> "aa"
>
>
> which contains 198 characters in it, byte array starts with this:
>
> 10 198 1 97 97 97
>
>
> Where 10 is protobuf field number, and 198 is string length, and 1 seems
> to be like string identifier, or what?
>
>
> And why protobuf cannot parse this message?
>
> Already spent almost a day on looking for solution for this problem, any
> help appreciated.
>
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is the relation between length of data block and file length in varint?

[Marc Gravell]

I wonder if you're decoding it incorrectly. Maybe if you could post the
bytes that you're decoding to reach this contradictory result, we can
advise you.

On Thu, 24 May 2018 at 04:59, Ruman Ahmed Rizvi <
ruman.ahmed.rizvi@gmail.com> wrote:

> When working with protocol buffers and encoding and decoding of these
> there is one issue occurred. In proto stream when decoding the file size
> and data block length in varint, the data block length is bigger than the
> file size? I am getting data block length greater than file length.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is the relation between length of data block and file length in varint?

[Marc Gravell]

I added a much longer version of this here:
https://stackoverflow.com/questions/50500626/what-is-the-relation-between-length-of-data-block-and-file-length-in-varint/50503626#50503626

On Thu, 24 May 2018 at 08:33, Marc Gravell <marc.grav...@gmail.com> wrote:

> I wonder if you're decoding it incorrectly. Maybe if you could post the
> bytes that you're decoding to reach this contradictory result, we can
> advise you.
>
> On Thu, 24 May 2018 at 04:59, Ruman Ahmed Rizvi <
> ruman.ahmed.rizvi@gmail.com> wrote:
>
>> When working with protocol buffers and encoding and decoding of these
>> there is one issue occurred. In proto stream when decoding the file size
>> and data block length in varint, the data block length is bigger than the
>> file size? I am getting data block length greater than file length.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Regards,
>
> Marc
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] first time using protobuf INTERESTING SCENARIO

[Marc Gravell]

It would probably help if you could be specific about what is happening
now. Have you got as far as running "protoc", or is the problem getting
"protoc" ready? If you have "protoc", what command are you using? Are there
error messages? If so: what do they say? Do you have an example .proto file?

Note: if the barrier is getting "protoc" to work at all, then as a
workaround you might be able to use https://protogen.marcgravell.com -
change the drop-down to "(protoc) C++", paste in your .proto, and click
"generate". Then copy the generated code from the bottom panel.


On Mon, 21 May 2018, 07:20 matt egler,  wrote:

> Greetings! I am new to google proto buf and I have over a million
> phonenumbers and I am designing a phone system for my company that requires
> the use of protobuf in order to complete. I am having trouble compiling a
> .proto file in a qubes linux evnvironment but I am using Eclipse IDE in a
> fedora 26 qube virtual machine... any help on how to compile a .proto file
> in terminal on essentialy fedora -26 linux would be greatly appreciated!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Reg: ProtocolBuf-net

[Marc Gravell]

k; you have a csv file... if you have a csv *file*, it is ... already
serialized ... as ... csv. So I'm unclear how this relates to protobuf. It
kinda sounds like you've asked a csv / aspx question on a thread that is
*completely unrelated* to either of those concepts. I'm going to give very
little answer here, mostly because I don't think this is the best place to
*get* the answer you want.

1: paging
2: don't display thousands of lines to a user - that is rarely useful
3: inbuilt controls like gridviews
4: perhaps ask on Stack Overflow with a question that gives *much* more
context and makes it clear exactly what you've tried and what didn't work
when you did so

Marc

On 26 October 2017 at 10:55, Mamy Andriamasinoro <
adrianemo.proj...@gmail.com> wrote:

> Hi Marc;
>
> I'm trying to serialize a csv file which contains thousands lines of query
> results. It works fine, but I'm stuck on how to display the results on an
> aspx page. Any idea please?
>
>
> Le jeudi 27 mai 2010 09:30:03 UTC+3, Marc Gravell a écrit :
>
>> With "full" .NET to .NET WCF, then switching the serialization layer to
>> use protobuf-net can (depending on a few factors) be as simple as a
>> configuration file change:
>> http://marcgravell.blogspot.com/2009/11/controlling-wcf-prot
>> obuf-net-at.html
>>
>>
>> <http://marcgravell.blogspot.com/2009/11/controlling-wcf-protobuf-net-at.html>*Unfortunately*,
>> this doesn't apply to Silverlight since Silverlight lacks that extension
>> point. There is no *silent* way of switching the serializer for
>> Silverlight. Instead the main mechanism for this would instead resolve
>> around exposing (on your WCF API) binary data - either byte[] or
>> Stream. You would, of course, then have to serialize/deserialize
>> manually. Nowhere near as elegant as you can achieve in full .NET, but it
>> still works. WCF essentially then becomes a plumbing layer, rather than
>> representing the actual data API.
>>
>> On the topic of data-contracts, you have a few options - if you have
>> existing data-contracts of the type:
>>
>> [DataContract]
>> public class MyData {
>> [DataMember]
>> public int Foo {get;set;}
>>
>> [DataMember]
>> public string Bar {get;set;}
>> }
>>
>> then all that is required is to associate a unique number with each
>> member (unique within the type, if you see what I mean). This can be done
>> using the Order property:
>>
>> [DataContract]
>> public class MyData {
>> [DataMember(Order=1)]
>> public int Foo {get;set;}
>>
>> [DataMember(Order=2)]
>> public string Bar {get;set;}
>> }
>>
>> or using protobuf-net's own attributes:
>>
>> [ProtoContract]
>> public class MyData {
>> [ProtoMember(1)]
>> public int Foo {get;set;}
>>
>> [ProtoMember(2)]
>> public string Bar {get;set;}
>> }
>>
>> It is important that the client and server agree about the numbers.
>>
>> Alternatively, if you *don't* have existing types you can also start from
>> a .proto file; the generator included with protobuf-net can emit suitable
>> types for you.
>>
>> Re transport; if you are using WCF over HTTP, it is usually worthwhile
>> enable MTOM; but I can't recall whether this is supported under Silverlight.
>>
>> Re serialization; if we assume you are throwing byte[] up and down the
>> wire, you should just need something like:
>>
>> MyData obj = new MyData { Foo = 123, Bar = "abc" };
>> byte[] blob;
>> using(var ms = new MemoryStream()) {
>> Serializer.Serialize(ms,obj);
>> blob = ms.ToArray();
>> }
>>
>> and then to deserialize:
>>
>> MyData obj;
>> using(var ms = new MemoryStream(blob)) {
>> obj = Serializer.Deserialize(ms);
>> }
>>
>> That should cover the main points. If there are more questions, please
>> let me know.
>>
>> Marc Gravell
>> (protobuf-net)
>>
>>
>>
>> On 27 May 2010 05:52, vikram <vikram...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am new to Protocol Buffer.
>>> I wanted to implement protocolbuffer for binary serialization for my
>>> WCF service.
>>>
>>> My requirement goes as given below:
>>>
>>> FrontEnd: SilverLight 3.0
>>> Middlelayer:  WCF Service (.NET 3.5)
>>>
>>> I would like to know what things need to be incopora

Re: [protobuf] Integrate proto2 messages from a device to C#

[Marc Gravell]

protobuf-net should work with proto 2 - try here:
https://protogen.marcgravell.com

On 27 Oct 2017 6:16 p.m., "cleal"  wrote:

> I have to integrate a third part protocol from a gps device, they send me
> the .proto files, they are using the schema proto2.
>
> I should receive the binary messages and I need desarialize it in order to
> work with the location data.  But I see in forums that C# only works for
> proto3.  There is no way to migrate the messages because they are from a
> hardware device from a third part.
>
> I have never working with the Protocol Buffers, how I should start?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Precompiled Version of probuf for VS2017

[Marc Gravell]

Protoc is available for multiple OSes here:
https://github.com/google/protobuf/releases/tag/v3.5.1

Note sure about pre-compiled libs; for Java, they're on mvnrepository; for
C# they're on NuGet, etc.

On 26 January 2018 at 15:22, 'Frank Willen' via Protocol Buffers <
protobuf@googlegroups.com> wrote:

> Hi,
>
> I want to use the protocol buffers for a mixed solution (C+, C#, both in
> VS2017). Where can I download precompiled libs, headers and the protoc.exe?
> Or is it neccessary to build the probuffers for everyone?
>
> Thanks for your support?
>
> Grettings
> Frank
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] what is TypeModel.cs and why is it missing ?

[Marc Gravell]

This is protobuf-net, so I suppose I should chime in :)

Can you please give a precise copy of the error message? And how did you
reference the library? Did you just add the NuGet package as normal? Or did
you do something more exotic? .NET libraries are almost usually distributed
as compiled assemblies - not as source code, so there is no reason for it
to be looking for TypeModel.cs - a source file. To understand the problem,
I'd need to see the actual error message.

No, this isn't expected.



On 31 Jan 2018 1:51 am, "Erick Treetops"  wrote:

Writing my first GTFS Realtime app in .net..  I had some issues getting out
of our proxy but now I'm stuck because the protobuf is looking for a file
called TypeModel.cs  and can't find it.  I've searched and haven't found
anyone else with this issue.  The error occurs when I try and serialize the
request (last line).
Can anyone point me in the right direction


WebRequest req = HttpWebRequest.Create("https://gtfsrt.api.translink.com.au/
Feed/SEQ");

IWebProxy proxy = req.Proxy;

proxy.Credentials = System.Net.CredentialCache.DefaultCredentials;

FeedMessage feed = Serializer.Deserialize(req.GetResponse().
GetResponseStream());



Erick Treetops

-- 
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Migrating legacy code to use Protobuff

[Marc Gravell]

Protobuf doesn't touch security, so we can ignore that one.

Modelling datasets/DataTable is awkward. It isn't really a natural fit, but
it can be manually forced. However, the first thing I'd say is: have you
set the "RemotingFormat" on the dataset to **binary** before using your
existing serialization? Assuming you are using BinaryFormatter, setting the
"RemotingFormat" can have a *huge* efficiency benefit, and might allow you
to get most of what you wanted without doing much work.

On 13 Feb 2018 8:08 am, "Som Shankar Bhattacharyya" <
bhattacharyya...@gmail.com> wrote:

> So i work in a legacy Visual Basic project. It is a client server
> application. The client application sends up some table data to the server
> that persists to a file system. A separate windows service then reads this
> data and saves to the server database.
> Now the data that is sent up from the client is a datatable in a raw .NET
> serialized format. The service that reads this information has the model on
> its path and is hence able to reconstruct the objects.
>
> Now i want to get rid of this flow and use a more secure as well as better
> performant way of doing the same.
> Can someone give me  an idea of how to approach this ?
>
> How do i model a dataset in the protobuff language ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Migrating legacy code to use Protobuff

[Marc Gravell]

Great! Give that a go, and see if it gets you what you need. As a footnote:
I advocate *not* using data tables as a data transfer tool, except in very
specific circumstances (such as ad-hoc query systems where the structure is
very flexible). But that is a huge thing to change.

On 14 Feb 2018 7:07 am, "Som Shankar Bhattacharyya" <
bhattacharyya...@gmail.com> wrote:

> I see no remting format set. Looks like it used the default xml format.
>
> On Wed, Feb 14, 2018 at 12:11 AM, Marc Gravell <marc.grav...@gmail.com>
> wrote:
>
>> Protobuf doesn't touch security, so we can ignore that one.
>>
>> Modelling datasets/DataTable is awkward. It isn't really a natural fit,
>> but it can be manually forced. However, the first thing I'd say is: have
>> you set the "RemotingFormat" on the dataset to **binary** before using your
>> existing serialization? Assuming you are using BinaryFormatter, setting the
>> "RemotingFormat" can have a *huge* efficiency benefit, and might allow you
>> to get most of what you wanted without doing much work.
>>
>> On 13 Feb 2018 8:08 am, "Som Shankar Bhattacharyya" <
>> bhattacharyya...@gmail.com> wrote:
>>
>>> So i work in a legacy Visual Basic project. It is a client server
>>> application. The client application sends up some table data to the server
>>> that persists to a file system. A separate windows service then reads this
>>> data and saves to the server database.
>>> Now the data that is sent up from the client is a datatable in a raw
>>> .NET serialized format. The service that reads this information has the
>>> model on its path and is hence able to reconstruct the objects.
>>>
>>> Now i want to get rid of this flow and use a more secure as well as
>>> better performant way of doing the same.
>>> Can someone give me  an idea of how to approach this ?
>>>
>>> How do i model a dataset in the protobuff language ?
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: How to speedup serialization and deserialization process?

[Marc Gravell]

I did a test locally using protobuf-net (since that is what I'm most
familiar with); to get an output of about 3,784,000 I used a count of
172000 items in the inner array - does that sound about right? Then I
tested it in a loop as per this gist: https://gist.github.com/mgravell/
10a21970531485008731d700b89ec732

The timings I get there are about 25ms to serialize, 40ms to deserialize -
although my machine is quite fast, so this may mean you're pretty much
getting "about right" numbers. What sort of numbers are you looking for
here? Note that in .NET the first run will always be slightly slower due to
JIT.

Additionally, I know nothing about the zmq overheads - are you including
the zmq cost in your numbers?

If you want to squeeze the last few drops of performance, you usually can -
for example, by looking at whether zmq allows you to pass a stream or span
or an *oversized* array - again, I'm not as familiar with the Google C#
version as I am with protobuf-net, but *if* (and it is a huge "if") the
"ToByteArray()" is essentially writing to a MemoryStream then calling
ToArray, you can probably avoid an extra blit and some allocs by providing
your own re-used memory-stream and using GetBuffer to access the oversized
array, remembering to limit to just the first .Length bytes. But again a
lot of this depends on specifics of zmq and the Google C# version. It is
almost certainly diminishing returns.

So: what numbers are you *looking to get*? what would be "acceptable"? And
how complex is your data? is what you've shown the *only* data you need to
transfer? if so, it might not be a bad candidate for fully manual explicit
serialization not involving a library - just a payload of:

Height [int32 fixed 4 bytes]
Width [int32 fixed 4 bytes]
Time [int64 fixed 8 bytes]
ElementCount [int32 fixed 4 bytes]
then for each element: 16 bytes consisting of
X [float fixed 4 bytes]
Y [float fixed 4 bytes]
Z [float fixed 4 bytes]
RGB [int32 fixed 4 bytes]

this would require manual coding, but would typically outperform other
options - but would be more brittle and would require you to be reasonably
good at IO code.

Personally, I'd probably leave it alone...

Marc



On 22 December 2017 at 13:03, Ravi  wrote:

> Any suggestions, please?
>
>
> On Wednesday, December 20, 2017 at 10:00:31 PM UTC+9, Ravi wrote:
>>
>> I have defined the Protocol Buffers message file as follows:
>>
>> syntax = "proto3";
>> package Tutorial;
>> import "google/protobuf/timestamp.proto";
>> option csharp_namespace = "Tutorial";
>>
>> message PointCloud {
>>   int32 width  = 1;
>>   int32 height = 2;
>>
>>   message Point {
>> float x = 1;
>> float y = 2;
>> float z = 3;
>> fixed32 rgb = 4;
>>   }
>>   repeated Point points = 3;
>>   google.protobuf.Timestamp timestamp = 4;
>> }
>>
>> This is how I am preparing the data and serializing it in C#:
>> using Google.Protobuf;
>> using Tutorial;
>> using ZeroMQ;
>>
>> PointCloud pointCloud = new PointCloud();
>> pointCloud.Height = Height
>> pointCloud.Width = Width;
>> pointCloud.Timestamp = Timestamp.FromDateTime(DateTime.UtcNow);
>>
>> for (var index = 0; index < POINTS3D_COUNT; index++) {
>>   PointCloud.Types.Point point = new PointCloud.Types.Point {
>> X = points3D[index].X,
>> Y = points3D[index].Y,
>> Z = points3D[index].Z,
>> Rgb = (uint)((red << 16) | (green << 8) | blue)
>>   };
>>
>>   pointCloud.Points.Add(point);
>> }
>>
>> zmqPublisher.Send(new ZFrame(pointCloud.ToByteArray()));
>>
>> This is how I deserialize the data in C++:
>> while (receive) {
>>   zmq::message_t msg;
>>   int rc = zmq_socket.recv();
>>   if (rc) {
>> Tutorial::PointCloud point_cloud;
>> point_cloud.ParseFromArray(msg.data(), msg.size());
>>   }point_cloud.ParseFromArray(msg.data(), msg.size())
>> }
>>
>> I am able to get the data back properly. However, the serialization and
>> deserialization process seems slow.
>>
>>- I used *System.Diagnostics.Stopwatch *in C# and noticed that
>>*pointCloud.ToByteArray()* is taking 100ms approximately.
>>- Similarly I used *std::chrono::steady_clock::now()* in C++ and
>>noticed that *point_cloud.ParseFromArray(msg.data(), msg.size())* is
>>taking 96ms approximately.
>>- Just for information, the length of the byte array is roughly
>>3,784,000.
>>
>>
>> *I want to know how to speed up serialization and deserialization
>> process?*
>>
>> -
>> Thanks
>> Ravi
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the 

Re: [protobuf] Is protobuffers right for me?

[Marc Gravell]

>  I didn't see anything about lists though

See: "repeated"; this is pretty much a synonym for "list".

>  Some items may reference other incoming packets.

No, there is no concept of an object id, whether for in-message or
out-of-message referencing.

On Sun, 29 Jul 2018 at 22:38, M P  wrote:

> I will be sending* hundreds of results of data in the form of a list as
> part of a protocol* I'm designing. Protobuffers looks ideal to help speed
> development across multiple services. I didn't see anything about lists
> though. *Is it possible to do? *
>
> Some items may reference other incoming packets. *Is this also possible?*
> I know I can just assign IDs and do lookups the traditional way but I want
> to know if protobuffers has a built-in solution for this as well.
>
> Thank you
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Change Timestamp Format

[Marc Gravell]

It is a little unclear what you mean, since the javascript Date constructor
itself takes the year first, but that is nothing to do with protobuf or
JSON, and nothing to do with the format.


If you mean the JSON, then:

In the JSON format, timestamp is always RFC 3339. In the binary format,
timestamp is just epoch integers (or two).

If you need different control over the JSON format, you might want a
general purpose JSON library, not s strongly opinionated tool like
protobuf. Alternatively: store a string.


On Sun, 19 Aug 2018, 03:01 Eduardo Demar,  wrote:

> Hi, I just need change default format of timestamp to dd/mm/
>
> I am using this code, on https://script.google.com:
>
>   input = {
> "timestamp": new Date(DD/MM/)
>   };
>
> I really appreciate this help!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is the most efficient protobuf type (in C++) for storing ipv4 or ipv6 address? My address is a boost::asio::ip::address_v4 (or v6)

[Marc Gravell]

You should be able to encode ipv4 in 4 bytes, making fixed32 ideal, since
you can avoid the length prefix. For ipv6, you're going to need 16 bytes,
so "bytes" is probably your best bet, since it will only require a single
header. You can then create a union of those:

oneof ip_addr {
fixed32 v4 = 1;
bytes v6 = 2;
}

That seems pretty optimal to me.

Marc

On Wed, 18 Jul 2018, 08:16 sanjana gupta, 
wrote:

> I read that protobuf has a type called "*bytes*" which can store
> arbitrary number of bytes and is the equivalent of "C++ string".
>
> The reason why I don't prefer to use "bytes" is that it expects input as a
> C++ string i.e., boost IP will need to be converted to a string.
>
>
> Now my concern lies here : I want to perform serialize and send the
> encoded protobuf message over TCP socket. I want to ensure that the *encoded
> message size is as small as possible*.
>
>
> Currently, I am using the below .proto file :
>
> syntax = "proto2";
>
> message profile
>
> {
>
> repeated *uint32* localEndpoint = 1;
>
> repeated *uint32* remoteEndpoint = 2;
>
> }
>
>
> In order to save boost IP in the protobuf message, I am first converting
> boost IP into byte-format array by using
> "boost::asio::ip::address_v4::to_bytes()". So for a v4 IP, resultant array
> size is 8. Then I am converting 1st 4 bytes from the resultant byte-array
> into one uint32_t number and then storing in "localEndpoint" field of the
> protobuf message. Likewise, I repeat for the next 4 bytes. I am taking 4
> bytes at a time so as to utilize full 32 bits of the uint32.
>
>
> Hence for a v4 address, 2 occurrence of "localEndpoint" field is used.
> Similarly, for a v6 address, 4 occurrence of "localEndpoint" field is used.
>
>
> Please allow me to highlight that if I had used "bytes" here, my input
> string itself would have been of size 15 bytes for a v4 ip like
> 111.111.111.111
>
>
> Using uint32 instead of "bytes" does save me some encoded-data-size but I
> am looking for a more efficient protobuf type requiring lesser number of
> bytes.
>
>
> Sorry for a long description but I wanted to explain my query in details.
> Please help me.. Thanks a lot in advance :)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is the most efficient protobuf type (in C++) for storing ipv4 or ipv6 address? My address is a boost::asio::ip::address_v4 (or v6)

[Marc Gravell]

At that point I'd probably use "repeated bytes", then. It'll cost you an
extra byte on v4 addresses, but it is simple.

On Wed, 18 Jul 2018, 09:36 sanjana gupta, 
wrote:

> Thanks Marc for your reply! By mistake I wrote 8-bytes for ipv4. I have
> corrected it as 4-bytes in my question. Thanks for that :)
>
> One thing I want to ask you is that can I use "oneof" if my field
> "localEndpoint" is repeated?
> I mean to say that my one "profile" message can have any among :
> -> 2 ipv4 addresses
> -> OR 2 ipv6 addresses
> -> OR 1 ipv4 + 1 ipv6 addresses
> in the field "localEndpoint".
>
> So in that case, is "oneof" usage correct? I think that a "oneof" cannot
> be repeated. Please correct me if I am wrong.
>
> On Wednesday, July 18, 2018 at 12:54:39 PM UTC+5:30, Marc Gravell wrote:
>>
>> You should be able to encode ipv4 in 4 bytes, making fixed32 ideal, since
>> you can avoid the length prefix. For ipv6, you're going to need 16 bytes,
>> so "bytes" is probably your best bet, since it will only require a single
>> header. You can then create a union of those:
>>
>> oneof ip_addr {
>> fixed32 v4 = 1;
>> bytes v6 = 2;
>> }
>>
>> That seems pretty optimal to me.
>>
>> Marc
>>
>> On Wed, 18 Jul 2018, 08:16 sanjana gupta,  wrote:
>>
>>> I read that protobuf has a type called "*bytes*" which can store
>>> arbitrary number of bytes and is the equivalent of "C++ string".
>>>
>>> The reason why I don't prefer to use "bytes" is that it expects input as
>>> a C++ string i.e., boost IP will need to be converted to a string.
>>>
>>>
>>> Now my concern lies here : I want to perform serialize and send the
>>> encoded protobuf message over TCP socket. I want to ensure that the *encoded
>>> message size is as small as possible*.
>>>
>>>
>>> Currently, I am using the below .proto file :
>>>
>>> syntax = "proto2";
>>>
>>> message profile
>>>
>>> {
>>>
>>> repeated *uint32* localEndpoint = 1;
>>>
>>> repeated *uint32* remoteEndpoint = 2;
>>>
>>> }
>>>
>>>
>>> In order to save boost IP in the protobuf message, I am first converting
>>> boost IP into byte-format array by using
>>> "boost::asio::ip::address_v4::to_bytes()". So for a v4 IP, resultant array
>>> size is 8. Then I am converting 1st 4 bytes from the resultant byte-array
>>> into one uint32_t number and then storing in "localEndpoint" field of the
>>> protobuf message. Likewise, I repeat for the next 4 bytes. I am taking 4
>>> bytes at a time so as to utilize full 32 bits of the uint32.
>>>
>>>
>>> Hence for a v4 address, 2 occurrence of "localEndpoint" field is used.
>>> Similarly, for a v6 address, 4 occurrence of "localEndpoint" field is used.
>>>
>>>
>>> Please allow me to highlight that if I had used "bytes" here, my input
>>> string itself would have been of size 15 bytes for a v4 ip like
>>> 111.111.111.111
>>>
>>>
>>> Using uint32 instead of "bytes" does save me some encoded-data-size but
>>> I am looking for a more efficient protobuf type requiring lesser number of
>>> bytes.
>>>
>>>
>>> Sorry for a long description but I wanted to explain my query in
>>> details. Please help me.. Thanks a lot in advance :)
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is the most efficient protobuf type (in C++) for storing ipv4 or ipv6 address? My address is a boost::asio::ip::address_v4 (or v6)

[Marc Gravell]

I would be amazed if there isn't a way of getting the raw underlying bytes
from the IP address, rather than the text (ASCII). The address
111.111.111.111 is literally the 4 bytes with decimal values 111, 111, 111
and 111. No ASCII required.

On Wed, 18 Jul 2018, 10:28 sanjana gupta, 
wrote:

> Hey Marc,
>
> I am sorry if I am repeating my words. Please enlighten me on this thing :
>
> "bytes" requires me to give a std::string (c++) value as input. The
> problem with string for me is that a usual 4-byte ipv4 address like
> 111.111.111.111 becomes 15-bytes string "111.111.111.111"
>
> Having said that, I feel discouraged to use bytes over uint32 (or int type
> for that matter) as it already increases the *size of the data to be
> encoded*. As a result, we can say that :
> size of (encoded message using "bytes" type)  >  size of (encoded message
> using "int" type)
>
> Is my understanding correct? Thanks!
>
>
> On Wednesday, July 18, 2018 at 2:40:01 PM UTC+5:30, Marc Gravell wrote:
>>
>> At that point I'd probably use "repeated bytes", then. It'll cost you an
>> extra byte on v4 addresses, but it is simple.
>>
>> On Wed, 18 Jul 2018, 09:36 sanjana gupta,  wrote:
>>
>>> Thanks Marc for your reply! By mistake I wrote 8-bytes for ipv4. I have
>>> corrected it as 4-bytes in my question. Thanks for that :)
>>>
>>> One thing I want to ask you is that can I use "oneof" if my field
>>> "localEndpoint" is repeated?
>>> I mean to say that my one "profile" message can have any among :
>>> -> 2 ipv4 addresses
>>> -> OR 2 ipv6 addresses
>>> -> OR 1 ipv4 + 1 ipv6 addresses
>>> in the field "localEndpoint".
>>>
>>> So in that case, is "oneof" usage correct? I think that a "oneof" cannot
>>> be repeated. Please correct me if I am wrong.
>>>
>>> On Wednesday, July 18, 2018 at 12:54:39 PM UTC+5:30, Marc Gravell wrote:
>>>>
>>>> You should be able to encode ipv4 in 4 bytes, making fixed32 ideal,
>>>> since you can avoid the length prefix. For ipv6, you're going to need 16
>>>> bytes, so "bytes" is probably your best bet, since it will only require a
>>>> single header. You can then create a union of those:
>>>>
>>>> oneof ip_addr {
>>>> fixed32 v4 = 1;
>>>> bytes v6 = 2;
>>>> }
>>>>
>>>> That seems pretty optimal to me.
>>>>
>>>> Marc
>>>>
>>>> On Wed, 18 Jul 2018, 08:16 sanjana gupta, 
>>>> wrote:
>>>>
>>>>> I read that protobuf has a type called "*bytes*" which can store
>>>>> arbitrary number of bytes and is the equivalent of "C++ string".
>>>>>
>>>>> The reason why I don't prefer to use "bytes" is that it expects input
>>>>> as a C++ string i.e., boost IP will need to be converted to a string.
>>>>>
>>>>>
>>>>> Now my concern lies here : I want to perform serialize and send the
>>>>> encoded protobuf message over TCP socket. I want to ensure that the 
>>>>> *encoded
>>>>> message size is as small as possible*.
>>>>>
>>>>>
>>>>> Currently, I am using the below .proto file :
>>>>>
>>>>> syntax = "proto2";
>>>>>
>>>>> message profile
>>>>>
>>>>> {
>>>>>
>>>>> repeated *uint32* localEndpoint = 1;
>>>>>
>>>>> repeated *uint32* remoteEndpoint = 2;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> In order to save boost IP in the protobuf message, I am first
>>>>> converting boost IP into byte-format array by using
>>>>> "boost::asio::ip::address_v4::to_bytes()". So for a v4 IP, resultant array
>>>>> size is 8. Then I am converting 1st 4 bytes from the resultant byte-array
>>>>> into one uint32_t number and then storing in "localEndpoint" field of the
>>>>> protobuf message. Likewise, I repeat for the next 4 bytes. I am taking 4
>>>>> bytes at a time so as to utilize full 32 bits of the uint32.
>>>>>
>>>>>
>>>>> Hence for a v4 address, 2 occurrence of "localEndpoint" field is used.
>>>>> Similarly, for a v6 ad

Re: [protobuf] Decoding message protobuff

[Marc Gravell]

Btw you might find https://protogen.marcgravell.com/decode useful - it
explains how each byte is interpreted. It doesn't give the details (so: it
doesn't explain how varint works), but it at least makes it explicit which
bytes have contributed to which outcomes.

On Tue, 17 Jul 2018, 08:22 Marc Gravell,  wrote:

> AA 06 is indeed 101, that's "varint" encoding. The MSB of each byte is
> continuation, 7 bits payload, and adjust for endianness. The idea of
> decode-raw is that it is enough to start guessing at how to reverse
> engineer a message, for example you can see values that are clearly string
> IP-addresses. However, the other fields are much harder to interpret
> without context. If you do have the original .proto schema, it would help
> tremendously.
>
> On Tue, 17 Jul 2018, 05:17 ,  wrote:
>
>> How to decode protobuff embedded message. Please give any pseudo code. I
>> see aa 06 gets decoded as 101. Please let me know how?
>>
>> Protobuff data:
>>
>> echo -n "0a 0b 31 39 32 2e 31 30 2e 31 31 2e 31 10 01 aa 06 88 01 e2 a4
>> 01 83 01 1a 80 01 0a 1e 0a 06 70 6f 72 74 5f 32 10 80 b7 e3 88 08 18 02 32
>> 0c 08 00 10 00 28 00 30 00 38 00 40 00 0a 1e 0a 06 70 6f 72 74 5f 33 10 80
>> b7 e3 88 08 18 03 32 0c 08 00 10 00 28 00 30 00 38 00 40 00 0a 1e 0a 06 70
>> 6f 72 74 5f 34 10 80 b7 e3 88 08 18 04 32 0c 08 00 10 00 28 00 30 00 38 00
>> 40 00 0a 1e 0a 06 70 6f 72 74 5f 35 10 80 b7 e3 88 08 18 05 32 0c 08 00 10
>> 00 28 00 30 00 38 00 40 00" | xxd -r -p | protoc --decode_raw
>> 1: "192.10.11.1"
>> 2: 1
>> 101 {
>>   2636 {
>> 3 {
>>   1 {
>> 1: "port_2"
>> 2: 2165889920
>> 3: 2
>> 6 {
>>   1: 0
>>   2: 0
>>   5: 0
>>   6: 0
>>   7: 0
>>   8: 0
>> }
>>   }
>>   1 {
>> 1: "port_3"
>> 2: 2165889920
>> 3: 3
>> 6 {
>>   1: 0
>>   2: 0
>>   5: 0
>>   6: 0
>>   7: 0
>>   8: 0
>> }
>>   }
>>   1 {
>> 1: "port_4"
>> 2: 2165889920
>> 3: 4
>> 6 {
>>   1: 0
>>   2: 0
>>   5: 0
>>   6: 0
>>   7: 0
>>   8: 0
>> }
>>   }
>>   1 {
>> 1: "port_5"
>> 2: 2165889920
>> 3: 5
>> 6 {
>>   1: 0
>>   2: 0
>>   5: 0
>>   6: 0
>>   7: 0
>>   8: 0
>> }
>>   }
>> }
>>   }
>> }
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Decoding message protobuff

[Marc Gravell]

AA 06 is indeed 101, that's "varint" encoding. The MSB of each byte is
continuation, 7 bits payload, and adjust for endianness. The idea of
decode-raw is that it is enough to start guessing at how to reverse
engineer a message, for example you can see values that are clearly string
IP-addresses. However, the other fields are much harder to interpret
without context. If you do have the original .proto schema, it would help
tremendously.

On Tue, 17 Jul 2018, 05:17 ,  wrote:

> How to decode protobuff embedded message. Please give any pseudo code. I
> see aa 06 gets decoded as 101. Please let me know how?
>
> Protobuff data:
>
> echo -n "0a 0b 31 39 32 2e 31 30 2e 31 31 2e 31 10 01 aa 06 88 01 e2 a4
> 01 83 01 1a 80 01 0a 1e 0a 06 70 6f 72 74 5f 32 10 80 b7 e3 88 08 18 02 32
> 0c 08 00 10 00 28 00 30 00 38 00 40 00 0a 1e 0a 06 70 6f 72 74 5f 33 10 80
> b7 e3 88 08 18 03 32 0c 08 00 10 00 28 00 30 00 38 00 40 00 0a 1e 0a 06 70
> 6f 72 74 5f 34 10 80 b7 e3 88 08 18 04 32 0c 08 00 10 00 28 00 30 00 38 00
> 40 00 0a 1e 0a 06 70 6f 72 74 5f 35 10 80 b7 e3 88 08 18 05 32 0c 08 00 10
> 00 28 00 30 00 38 00 40 00" | xxd -r -p | protoc --decode_raw
> 1: "192.10.11.1"
> 2: 1
> 101 {
>   2636 {
> 3 {
>   1 {
> 1: "port_2"
> 2: 2165889920
> 3: 2
> 6 {
>   1: 0
>   2: 0
>   5: 0
>   6: 0
>   7: 0
>   8: 0
> }
>   }
>   1 {
> 1: "port_3"
> 2: 2165889920
> 3: 3
> 6 {
>   1: 0
>   2: 0
>   5: 0
>   6: 0
>   7: 0
>   8: 0
> }
>   }
>   1 {
> 1: "port_4"
> 2: 2165889920
> 3: 4
> 6 {
>   1: 0
>   2: 0
>   5: 0
>   6: 0
>   7: 0
>   8: 0
> }
>   }
>   1 {
> 1: "port_5"
> 2: 2165889920
> 3: 5
> 6 {
>   1: 0
>   2: 0
>   5: 0
>   6: 0
>   7: 0
>   8: 0
> }
>   }
> }
>   }
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is the most efficient protobuf type (in C++) for storing ipv4 or ipv6 address? My address is a boost::asio::ip::address_v4 (or v6)

[Marc Gravell]

The first step, then, is to find how to get your ipv4 in a 4 byte
std::string rather than a 15 byte std::string - I can't advise on that, but
it should absolutely be possible. You don't need it as ASCII - you just
want the bytes.

As for will it be longer? No, especially when compared to uint32; a 4-byte
"bytes" will take 5 bytes to encode (1 byte for the length prefix, 4 bytes
payload); the uint32 with value composed of 4 bytes with value 111 will be
varint-encoded, so will be 5 bytes (since the penultimate high bit is set);
in fact, anything with the most-significant byte having value 16 or above
will require 5 bytes to encode as uint32. Now, you *could* encode an ipv4
using "fixed32", which will always take 4 bytes, but... if you do that, you
can't conveniently store ipv6 in the same field. So: since you mention
needing to store both ipv4 and ipv6, "repeated bytes" is your simplest
option. And as above: it isn't any more expensive than "uint32" (for most
IPs).

On Wed, 18 Jul 2018 at 10:58, sanjana gupta 
wrote:

> I would like to understand from the protobuf generated API point of view:
>
> If I have,
> repeated bytes localEndpoint = 1;
>
> The generated file profile.pb.h shows me the following APIs to add+set
> values in it :
>
>   // repeated bytes localEndpoint = 1;
> *  inline void add_localendpoint(const ::std::string& value);*
> *  inline void add_localendpoint(const char* value);*
> *  inline void add_localendpoint(const void* value, size_t size);*
>
> If I use any of them, perhaps my boost::asio::ip::address_v4
> (111.111.111.111) needs to be converted to a std::string and then only I
> can use add_localendpoint() to set a value.
> Am I correct?
>
> If the conversion to string is done, isn't the size of protobuf message
> going to be more compared to the case of using
> repeated uint32 localEndpoint = 1;
> ?
>
> Please let me know if I have confused you :) Thanks again!
>
> On Wednesday, July 18, 2018 at 3:08:52 PM UTC+5:30, Marc Gravell wrote:
>>
>> I would be amazed if there isn't a way of getting the raw underlying
>> bytes from the IP address, rather than the text (ASCII). The address
>> 111.111.111.111 is literally the 4 bytes with decimal values 111, 111, 111
>> and 111. No ASCII required.
>>
>> On Wed, 18 Jul 2018, 10:28 sanjana gupta,  wrote:
>>
>>> Hey Marc,
>>>
>>> I am sorry if I am repeating my words. Please enlighten me on this thing
>>> :
>>>
>>> "bytes" requires me to give a std::string (c++) value as input. The
>>> problem with string for me is that a usual 4-byte ipv4 address like
>>> 111.111.111.111 becomes 15-bytes string "111.111.111.111"
>>>
>>> Having said that, I feel discouraged to use bytes over uint32 (or int
>>> type for that matter) as it already increases the *size of the data to
>>> be encoded*. As a result, we can say that :
>>> size of (encoded message using "bytes" type)  >  size of (encoded
>>> message using "int" type)
>>>
>>> Is my understanding correct? Thanks!
>>>
>>>
>>> On Wednesday, July 18, 2018 at 2:40:01 PM UTC+5:30, Marc Gravell wrote:
>>>>
>>>> At that point I'd probably use "repeated bytes", then. It'll cost you
>>>> an extra byte on v4 addresses, but it is simple.
>>>>
>>>> On Wed, 18 Jul 2018, 09:36 sanjana gupta, 
>>>> wrote:
>>>>
>>>>> Thanks Marc for your reply! By mistake I wrote 8-bytes for ipv4. I
>>>>> have corrected it as 4-bytes in my question. Thanks for that :)
>>>>>
>>>>> One thing I want to ask you is that can I use "oneof" if my field
>>>>> "localEndpoint" is repeated?
>>>>> I mean to say that my one "profile" message can have any among :
>>>>> -> 2 ipv4 addresses
>>>>> -> OR 2 ipv6 addresses
>>>>> -> OR 1 ipv4 + 1 ipv6 addresses
>>>>> in the field "localEndpoint".
>>>>>
>>>>> So in that case, is "oneof" usage correct? I think that a "oneof"
>>>>> cannot be repeated. Please correct me if I am wrong.
>>>>>
>>>>> On Wednesday, July 18, 2018 at 12:54:39 PM UTC+5:30, Marc Gravell
>>>>> wrote:
>>>>>>
>>>>>> You should be able to encode ipv4 in 4 bytes, making fixed32 ideal,
>>>>>> since you can avoid the length prefix. For ipv6, you're going to need 16
>>>>>> bytes, so "bytes" is

Re: [protobuf] protobuf version 3.6 a compile libprotobuf?

[Marc Gravell]

You mention both 3.5 and 3.6 - which did you mean?

As for both compiling and links: it kinda depends what platform and/or
language you are targeting. So: what are you targeting? You'll usually have
to run the protoc output through your own build tools.

On Thu, 6 Sep 2018, 07:47 Natalia Duality,  wrote:

> Hello,
>
> is it possible to create static link  with protobuf version 3.5???
>
> And second question,
>
>
> I found protoc.exe to generate files, but hot to compile dll?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is wrong with my proto file here? Is it not possible to have 2 repeated fields of different types?

[Marc Gravell]

 datavalues_srarray  and  datavalues_prarray   need to have different field
numbers

On Thu, 12 Jul 2018 at 22:50, Himabindu Dittakavi 
wrote:

> Hi All,
>
> New to protobuf.
> Here is my proto file. If i use the code highlighted in yellow it works
> but when I add the one in yellow - there is no .cs file generated...what is
> wrong with the code below? TIA. Also, is there a proto file validator by
> any chance?
>
> syntax = "proto3";
> package tutorial;
> option csharp_namespace = "Google.Protobuf.Examples.SS";
>
> message SS {
>   int32 id = 1;  // Unique ID number for this message or ss.
>   enum GeneratedDataType {
> SR = 0;
> PR = 1;
> SE = 2;
>   }
>message DataValueSr {
> GeneratedDataType type = 0;
> string value = 100;
>   }
>message DataValuePr {
> GeneratedDataType type = 1;
> string value = 100;
>   }
>
> repeated DataValueSr datavalues_srarray = 1000;
> repeated DataValuePr datavalues_prarray = 1000;
>  }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] What is wrong with my proto file here? Is it not possible to have 2 repeated fields of different types?

[Marc Gravell]

also, field zero (type) is invalid; you are welcome to use this as an
online check tool - but protoc also outputs similar messages:

https://protogen.marcgravell.com#gc4759103e204eec4ae1a11ce6089a4bf

On Thu, 12 Jul 2018 at 23:01, Marc Gravell  wrote:

> datavalues_srarray  and  datavalues_prarray   need to have different
> field numbers
>
> On Thu, 12 Jul 2018 at 22:50, Himabindu Dittakavi 
> wrote:
>
>> Hi All,
>>
>> New to protobuf.
>> Here is my proto file. If i use the code highlighted in yellow it works
>> but when I add the one in yellow - there is no .cs file generated...what is
>> wrong with the code below? TIA. Also, is there a proto file validator by
>> any chance?
>>
>> syntax = "proto3";
>> package tutorial;
>> option csharp_namespace = "Google.Protobuf.Examples.SS";
>>
>> message SS {
>>   int32 id = 1;  // Unique ID number for this message or ss.
>>   enum GeneratedDataType {
>> SR = 0;
>> PR = 1;
>> SE = 2;
>>   }
>>message DataValueSr {
>> GeneratedDataType type = 0;
>> string value = 100;
>>   }
>>message DataValuePr {
>> GeneratedDataType type = 1;
>> string value = 100;
>>   }
>>
>> repeated DataValueSr datavalues_srarray = 1000;
>> repeated DataValuePr datavalues_prarray = 1000;
>>  }
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Regards,
>
> Marc
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Tags and values

[Marc Gravell]

Yes, tags (field numbers) can be non-contiguous.

Yes, hex is accepted by protoc

No, repeated fields cannot be required

On 22 January 2018 at 23:08, Ashwin Kini  wrote:

> Hi all,
>
> While defining messages can the *tags *be non continous? The
> documentation never mentions the same
>
> message abc {
> optional uint32 xyz = *100*;
> optional uint32 def = *200*;
> optional unit32 abc = *0xDEAD01*;
>
> }
>
> The documentation says it needs to be unique. Agreed. Can they be non
> continous as explained in example?  . Also does it take HEX values as far
> as it falls in the range mentioned in the doc?
>
> One more question is, can repeated fields be made required? I did a lot of
> digging, seems its not possible. Can anybody confirm? I understand required
> should be completely avoided.
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Defining wrapper types in proto file

[Marc Gravell]

That's pretty vague. Could you clarify what you're trying to do?

On Wed, 12 Sep 2018, 09:00 qplc,  wrote:

> Hi,
>
> How do I define wrapper Long value in .proto file. Can anyone help in this?
>
>
> Thanks,
> qplc
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Marc Gravell]

Well, if it was me: the first thing I'd do is isolate whether it is
serialize or deserialize that is failing - by taking your currently
serialized data as a flat file or byte dump, and checking whether the
values are right or wrong.

Depending on the volume and layout of the data, you might be able to use
protoc's inbuilt decode mechanisms, or (as long as it isn't huge) you're
welcome to try using https://protogen.marcgravell.com/decode

That would allow you to focus your checks.

Marc

On 10 April 2018 at 14:20, Tony Tony <xxteknolus...@gmail.com> wrote:

> c++/windows
>
> I'll upgrade protobuf and see if issue goes away. Just wanted to make sure
> I'm not missing anything. Is there any additional troubleshooting I can
> look into to troubleshoot further if the recent release reproduce the issue?
>
> On Tuesday, April 10, 2018 at 1:46:50 AM UTC-4, Marc Gravell wrote:
>>
>> First thought: what language / platform is this? the C++ generated code
>> is very different to the Java generated code, for example
>>
>> Second thought: 2.6 is pretty old; it is very possible that a bug existed
>> and has been fixed since then (Aug 2014) - does it still happen with more
>> recent releases?
>>
>> The scenario you present seems fine and reasonable.
>>
>> Marc
>>
>> On 10 April 2018 at 03:46, Tony Tony <xxtekn...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a strange case I'm dealing with. Using protobuf compiler 2.6.
>>>
>>> Every time I send a certain set of values over my socket, it ends up
>>> with missing data. Although I've compared the string received after
>>> serialization on the server and on receiving the string on the client and
>>> they are exactly the same, yet the values are all -1.
>>>
>>> I've attached a snippet of the proto file below.
>>>
>>> At this point these are below combos that always fail
>>>
>>> type=99, count= 0, index = 1 (all values are -1 after deserialization)
>>> type=99, count=99, index= 1 (works)
>>> type=9, count=99, index= 1 (works)
>>>
>>> - Basically anytime count is 0, all the values seem to be -1. However if
>>> I make type 0, the same behavior does not occur. Am I missing something or
>>> there is some underlying bug in my code? Again I've checked multiple times
>>> the correct raw string data is being sent (with correct size). I use the
>>> parseFromString method to parse the receiving string data.
>>>
>>> - I've sent plenty of other types of data just fine.
>>>
>>> - There are other messages defined in the same protobuf packet
>>> definition file, can that make a difference?
>>>
>>> message packet {
>>> required int32 type = 1 [default = -1];
>>> required int32 count = 2 [default = -1];
>>> required int32 index = 3 [default = -1];
>>> }
>>>
>>> Thanks
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Regards,
>>
>> Marc
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Marc Gravell]

First thought: what language / platform is this? the C++ generated code is
very different to the Java generated code, for example

Second thought: 2.6 is pretty old; it is very possible that a bug existed
and has been fixed since then (Aug 2014) - does it still happen with more
recent releases?

The scenario you present seems fine and reasonable.

Marc

On 10 April 2018 at 03:46, Tony Tony  wrote:

> Hi,
>
> I have a strange case I'm dealing with. Using protobuf compiler 2.6.
>
> Every time I send a certain set of values over my socket, it ends up with
> missing data. Although I've compared the string received after
> serialization on the server and on receiving the string on the client and
> they are exactly the same, yet the values are all -1.
>
> I've attached a snippet of the proto file below.
>
> At this point these are below combos that always fail
>
> type=99, count= 0, index = 1 (all values are -1 after deserialization)
> type=99, count=99, index= 1 (works)
> type=9, count=99, index= 1 (works)
>
> - Basically anytime count is 0, all the values seem to be -1. However if I
> make type 0, the same behavior does not occur. Am I missing something or
> there is some underlying bug in my code? Again I've checked multiple times
> the correct raw string data is being sent (with correct size). I use the
> parseFromString method to parse the receiving string data.
>
> - I've sent plenty of other types of data just fine.
>
> - There are other messages defined in the same protobuf packet definition
> file, can that make a difference?
>
> message packet {
> required int32 type = 1 [default = -1];
> required int32 count = 2 [default = -1];
> required int32 index = 3 [default = -1];
> }
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protobuf 2.6 serialization weird behavior

[Marc Gravell]

Oddly neither of those is correct! The actual hex should be:

08-03-1A-06-08-01-10-63-18-01

which is the first version minus some trailing data. The second version is
not valid at all - it isn't just missing data: it is an incomplete and
invalid payload. Unfortunately I'm not familiar enough with the C++ API to
advise on whether you're using the wrong method vs this being a bug.


On Wed, 11 Apr 2018, 01:12 Tony Tony, <xxteknolus...@gmail.com> wrote:

> So I upgraded to 3.5.0, same issue.
>
> I guess I should give some more information: I'm basically using protobuf
> to serialize/deserialize string data I'm sending through a games socket
> implementation via hooks/code caves in the games network engine. I
> currently can validate that the bytes after serialization are sent over
> completely to the client. After looking at the decoded output that you guys
> recommended, I do see some differences.
>
> - When I send type=1, count=99, index=1, the server shows the below bytes
> after serialization and the client shows these bytes after receiving the
> data. The decoding tool shows the correct content.
>
> 08 03 1A 06 08 01 10 63 18 01 FD FD FD FD
>
> - When I send type=1, count=0, index=1, the server shows the below bytes
> after serialization and the client shows these bytes after receiving the
> data. The decoding tool basically shows no content, which is obvious from
> the missing data.
>
> 08 03 1A 06 08
>
> The biggest difference is that it seems the during serialization on the
> server, this combo of data seems to break everything. I guess I'm not sure
> why this combo of data would break during serialization. I posted some
> pseudo code of where the problem occurs for that combo of data below. It's
> just strange that only that combo of data above results in this behavior. I
> am able to send large strings just fine, plenty of other data types with no
> issues (well none that I've found yet..)
>
> Packet pak;
> pak.set_type(Packet_Type_set_info);
>
> set_info *setter = pak.mutable_set_info();
> setter->set_count(99);
> setter->set_type(0);
> setter->set_index(1);
>
> char* SendBuf = new char[pak.ByteSize()];
> pak.SerializeToArray(SendBuf, pak.ByteSize());
> //TODO: the bytes seem to be wrong right after this point for the second
> combo of data
> //we send before we delete the sendBuf
>
> delete[] SendBuf;
>
>
> On Tuesday, April 10, 2018 at 12:41:30 PM UTC-4, Adam Cozzette wrote:
>>
>> I think Marc is right that it would be best to isolate the problem by
>> manually checking that the bytes are valid.
>>
>> Tony, how are you framing the messages? Protocol buffers do not describe
>> their own size and so you have to know what size to expect before you parse
>> a proto. The typical approach is to frame the serialized messages by
>> prefixing them with their size. It's easy to get this wrong, though, so
>> that could potentially be the reason that the messages aren't showing up as
>> you expect.
>>
>> On Tue, Apr 10, 2018 at 7:28 AM Marc Gravell <marc.g...@gmail.com> wrote:
>>
>>> Well, if it was me: the first thing I'd do is isolate whether it is
>>> serialize or deserialize that is failing - by taking your currently
>>> serialized data as a flat file or byte dump, and checking whether the
>>> values are right or wrong.
>>>
>>> Depending on the volume and layout of the data, you might be able to use
>>> protoc's inbuilt decode mechanisms, or (as long as it isn't huge) you're
>>> welcome to try using https://protogen.marcgravell.com/decode
>>>
>>> That would allow you to focus your checks.
>>>
>>> Marc
>>>
>>> On 10 April 2018 at 14:20, Tony Tony <xxtekn...@gmail.com> wrote:
>>>
>>>> c++/windows
>>>>
>>>> I'll upgrade protobuf and see if issue goes away. Just wanted to make
>>>> sure I'm not missing anything. Is there any additional troubleshooting I
>>>> can look into to troubleshoot further if the recent release reproduce the
>>>> issue?
>>>>
>>>> On Tuesday, April 10, 2018 at 1:46:50 AM UTC-4, Marc Gravell wrote:
>>>>>
>>>>> First thought: what language / platform is this? the C++ generated
>>>>> code is very different to the Java generated code, for example
>>>>>
>>>>> Second thought: 2.6 is pretty old; it is very possible that a bug
>>>>> existed and has been fixed since then (Aug 2014) - does it still happen
>>>>> with more recent releases?
>>>>>
>>>>> The scenario you present seems fine an

Re: [protobuf] Inheriting protobuf messages

[Marc Gravell]

I'm going to share some thoughts here simply for discussion purposes - I
don't expect them to be directly applicable.

FWIW, protobuf-net has spoofed inheritance for many many years. I'm able to
do this because protobuf-net only needs to target .NET, which has good
inheritance support.
I don't really expect the main "protobuf" project will add inheritance in
any timescale, because it might need to target platforms that don't have
inheritance support.
It would also require ".proto" language considerations.

That said... if anyone at Google does ever want to reignite the discussion
around inheritance, I'm all ears :)

---

The way it protobuf-net does this can be *essentially* represented by
"oneof" - so going back to the original example from (wow, a long time ago):

message MSG1
 {
   required string account = 1;
   required string symbol = 2;
 }
message MSG2 extends MSG1
 {
   required int32  id  = 2;
 }

the way protobuf-net does this is essentially:

 message MSG1
 {
   required string account = 1;
   required string symbol = 2;
   oneof _subtype { // keep in mind that this is actually "zero-or-one-of"
  MSG2 msg2 = 3; // this field-number needs to be unique in MSG1
}
 }
 message MSG2
 {
   required int32  id  = 2; // no restrictions on this field-number
 }

but protobuf-net does some tricks so that when deserializing a MSG1, it
might end up *actually* creating an instance of MSG2 instead (where MSG2 :
MSG1).
For LSP purposes, everything is *always* serialized from the outside in, so
if you had a list/array of MSG2, they would still write the MSG1 fields
first - essentially
it would be like a list/array of MSG1, but where everything *happens to be*
an MSG2. You're right to say that the base type needs to know about the
derived types,
or at least have lib support to help it there, with some kind of deferred
registration.

But: I wonder whether the same approach - just without the actual
inheritance in the generated types - might still be useful to you.

Concretely, from a code-first angle:
https://gist.github.com/mgravell/a33755800e823ee77ba01183be7084df

Or maybe it is completely unhelpful and you're worse off for reading
this... who knows!

Marc




On Mon, 3 Sep 2018 at 11:34, SangemV  wrote:

> I see two problems with this approach. It is neither cut & paste nor
> personal preference issue as I see. The real issues I see are:
>
> 1) The base message (MSG in the example)  and the extended message (MSG2
> in the example ) can belong to different package owned by different
> group/org. The package defining base message (MSG) (call it Pkg1) does not
> event know the existing of the package defining extended message (MSG2)
> (call it Pkg2). How can the base message foresee all the extended message
> from it? Even if Pkg1 and Pkg2 are owned by the same group/org, making Pkg1
> aware of Pkg2 is not a good idea as it can potentially create cyclic
> dependencies.
>
> 2) Polymorphic Lists: I have a list of MSG types which can potentially
> have both MSG and MSG2 types. How is this modeled in proto3 using the
> proposed solution?
>
> These are real issues which I am facing right now. Any suggestions to
> handle the above problems would be of great help.
>
>> ​
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Decode google-protocol-buffer binary data using hex?

[Marc Gravell]

Presumably: convert the hex to bytes, and run it though the normal API for
your library/platform?

On Fri, 4 Jan 2019, 22:35  I have the java schema and the .proto schema for the data.
>
> The data is binary represented as hex. Is it possible to decode the data
> with the schemas I have? if so how?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] which protobuf type should be used for byte arrays

[Marc Gravell]

"bytes" is correct. The problem here is usually "treating binary data as
nul-terminated strings", which is almost certainly something *outside* of
the protobuf code, but in your code. Basically, you can't ever treat
protobuf data as nul-terminated strings.

On Wed, 23 Jan 2019, 00:52  What is the correct type to use when encoding byte arrays that contain 0s?
> If I use "bytes" then arrays that contain 0s are truncated.
>
> This email and its contents are confidential. If you are not the intended
> recipient, please do not disclose or use the information within this email
> or its attachments. If you have received this email in error, please report
> the error to the sender by return email and delete this communication from
> your records.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Regarding serializing multiple .proto files

[Marc Gravell]

without knowledge of the specific library/framework/language that you're
using, I doubt anyone can answer this.

On Tue, 18 Dec 2018 at 05:11, Ishaan Aggarwal 
wrote:

> I have multiple .proto files in my project. I came to know that there can
> only be a single serializer and de-serializer file in the project (please
> confirm this understanding). Is there a way to serdes them?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Getting IOException while deserializing a proto which contains a repeated JKL and JKL has a string type set ( works for repeated JKL and JKL has the same with int type set )

[Marc Gravell]

there's nothing wrong with those message definitions; you haven't told us
which platform and/or library you're using, but: my hunch would be that
you're transporting the data incorrectly in some way, meaning: corrupting
it in *your* code - or the code you're using to compress/decompress is
slightly wrong. Unfortunately we can't see any of that. But: speaking
generally: there's no reason it shouldn't work, so: we'd need more details
to provide useful help.

On Thu, 13 Dec 2018 at 18:52, Shibha Malik  wrote:

>
>
> On Thursday, December 13, 2018 at 10:32:52 AM UTC-8, Shibha Malik wrote:
>>
>> Hi Team,
>>
>> i am getting IOException while deserializing a proto which contains a
>> repeated Data and Data has a string type set ( works for repeated Data and
>> Data has the same with int type set )
>>
>> I have a proto which has the following structure :
>>
>> message XYZData
>> {
>>   ABCId abc_id = 1;
>>   DEF def= 2;
>>   google.protobuf.Timestamp ghi= 3;
>>   repeated JKL jkl = 4;
>>   MNO mno = 5;
>> }
>>
>> where JKL is
>> message JKL
>> {
>>   Culprit culprit = 1;
>>   PQR pqr = 2;
>> }
>> where Culprit is
>> message Culprit
>> {
>>   oneof culprit
>>   {
>> int32 int_culprit = 1;
>> string string_culprit = 2;
>>   }
>> }
>>
>> When i use the int_culprit and use GZIP compression. it serializes and
>> deserializes fine.
>> BUT when i use the string_culprit and use GZIP compression. it serializes
>> fine and deserialization gives me IOException.
>>
>>
>> PLEASE help us on this as this is blocking a major deliverable for us.
>>
>> Thanks,
>> Shibha
>>
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] define Set> message ,what should I do?

[Marc Gravell]

"set" isn't a protobuf concept - the closest that exists is "repeated";
likewise "object" - ideally using some defined message type. Putting those
together, that gives you:

syntax = "proto3";

message SomeRoot {
  repeated SomeLeaf items = 1;
}
message SomeLeaf {
  map values = 1;
}
message SomeVal {
  // ...
}

On Fri, 23 Nov 2018 at 08:31,  wrote:

> I want define a message. Example: Set> please help me.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Re: Defining wrapper types in proto file

[Marc Gravell]

Are you talking about Java's Long here? If so, frankly I'd say: in your DTO
layer, use whatever type protoc wants to use for your data, and just use:

syntax = "proto3";
message TestMessage{
string attribute1 = 1;
map attribute2 = 2;
int64 attribute3 = 3;
int32 attribute4 = 4;
int64 attribute5 = 5;
int32 attribute6 = 6;
repeated int64 attribute7 = 7;
}

However; there *is* also wrappers.proto (
https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto)
which allows something similar, but that
uses com.google.protobuf.Int64Value, not Java.Lang.Long; but if you want to
try it:

syntax = "proto3";
import "google/protobuf/wrappers.proto";
message TestMessage{
string attribute1 = 1;
map attribute2 = 2;
int64 attribute3 = 3;
int32 attribute4 = 4;
.google.protobuf.Int64Value attribute5 = 5;
.google.protobuf.Int32Value attribute6 = 6;
repeated .google.protobuf.Int64Value attribute7 = 7;
}



On Wed, 12 Sep 2018 at 09:44, qplc  wrote:

> Hi Marc,
>
> Thanks for you reply. Here, I'm trying to create a user defined message
> given follow.
>
> Class TestMessage{
>   String attribute1;
>   Map attribute2;
>   long attribute3;
>   int attribute4;
>   Long attribute5;
>   Integer attribute6;
>   Set attribute7;
> }
>
> I want to define above message format in .proto file. Hence, I've created
> following message.
> message TestMessage{
>   string attribute1 = 1;
>   map attribute2 = 2;
>   int64 attribute3 = 3;
> }
>
> But, I'm not sure how do I define attribute5, attribute6 and attribute7 in
> proto file. Please help me on this.
>
>
> On Wednesday, September 12, 2018 at 1:30:14 PM UTC+5:30, qplc wrote:
>>
>> Hi,
>>
>> How do I define wrapper Long value in .proto file. Can anyone help in
>> this?
>>
>>
>> Thanks,
>> qplc
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf-net desrialization taking too long

[Marc Gravell]

Hi; I'd love to help you on this - I'm the protobuf-net author, and I also
know more than a little about redis; it *might* be a little off-piste for
this group though. Running some tests locally with 5000 instances, I get
times like 1ms, but it might be that I'm misunderstanding your object
model. It would be great to perhaps see a realistic payload that I can play
with to see what's going on here.

Question: how fast is "fast enough" here, and how big are your payloads?

I have some incomplete changes in the pipe which at a stroke double the
throughput, but without knowing the answers to these questions, it is hard
to know whether that is sufficient

Depending on your target scenario, if the problem here turns out to be
allocations (of either the SubObj or the array), I have recently been doing
a lot of work in "arena allocators" which might be directly relevant - I
have a future plan to allow protobuf-net to consume / use the arena
allocator I've been working on, which would eliminate both of those
completely.

Example output from my test (the first serialize/deserialize is always more
expensive) - which is around 68k in payload size:

Serialize: 47ms
Serialize: 1ms
Serialize: 1ms
Serialize: 1ms
Serialize: 1ms
Deserialize: 6ms
Deserialize: 1ms
Deserialize: 1ms
Deserialize: 1ms
Deserialize: 1ms

My test code:
https://gist.github.com/mgravell/7afc08f432661f60138f6798efe2b15b

If you want to email me directly (especially if your data contains "real"
things that you don't want to send to the entire group), please feel free
to do so.

On Tue, 26 Mar 2019 at 18:32, Shweta Sharma  wrote:

> I am prototyping an application to store data in redis using
> StackExhange.Redis client. I am using Protobuf for serializing my objects
> and storing them as key/value pair in redis. When reading 20K keys using
> pipeline (splitting keys in batches of 5000), I observe deserialization
> times in 60-300 ms for a batch of say 5000 objects. Serialization code is
> using protobuf-net library version. Deserialization code is using
> protobuf-net 2.4.0.0.
>
> Is there anyway I can speed up desrialization time since this will impact
> my api performance.
>
> The object I am trying to desrialize is not too complex.
>
> [ProtoContract]
> public struct MyObject
> {
> [ProtoMember(1)]
> public int Key { get; set; }
> [ProtoMember(2)]
> public byte Attr1 { get; set; }
> [ProtoMember(3)]
> public bool IsUsed { get; set; }
> [ProtoMember(4)]
> public byte Attr2 { get; set; }
> [ProtoMember(5)]
> public SubObj[] SubObjects { get; set; }
>
> public static MyObject Null
> {
> get
> {
> return default(MyObject);
> }
> }
>
> public bool IsNull
> {
> get { return Attr1 == default(byte) && SubObjects == null; }
> }
> }
> [ProtoContract]
> public class SubObj
> {
> [ProtoMember(1)]
> public int StartOffset { get; set; }
> [ProtoMember(2)]
> public int EndOffset { get; set; }
> [ProtoMember(3)]
> public byte Attr1 { get; set; }
> [ProtoMember(4)]
> public byte Attr2 { get; set; }
> [ProtoMember(5)]
> public bool IsUsed { get; set; }
>
> public static SubObj Null
> {
> get
> {
> return default(SubObj);
> }
> }
>
> public bool IsNull
> {
> get { return Attr1 == default(byte); }
> }
> }
>
> Here're some sample timings using StopWatch:
>
> Time to desrialize 5000 keys from redis:134
> Time to desrialize 5000 keys from redis:147
> Time to desrialize 5000 keys from redis:160
> Time to desrialize 5000 keys from redis:65
> Time to desrialize 5000 keys from redis:67
> Time to desrialize 5000 keys from redis:242
> Time to desrialize 5000 keys from redis:371
> Time to desrialize 5000 keys from redis:190
>
> Thanks,
> Shweta
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf-net desrialization taking too long

[Marc Gravell]

>  Where can I send you mail? My e-mail is ...

I'll ping you

On Wed, 27 Mar 2019 at 15:12, Shweta Sharma  wrote:

> Thanks again Marc. Yes Deserialization code is as you listed below. I was
> using protobuf-net 2.4.0.0 but then moved to older version currently being
> used across our code base, 2.0.0.668, to see any impact. Where can I send
> you mail? My e-mail is shwetad...@gmail.com. I can get these objects
> serialized to a binary file and send over to you.
>
> public T Deserialize(byte[] serializedObject)
> {
> using (var ms = new MemoryStream(serializedObject))
> {
> return (T)Serializer.Deserialize(ms, null, typeof(T));
> }
> }
>
> Thanks much!
> Shweta
>
> On Wednesday, March 27, 2019 at 7:34:58 AM UTC-7, Marc Gravell wrote:
>>
>> > using StackExchange.Redis MGET
>>
>> Yeah, there's really no way for me to dodge this, is there? ;p
>>
>> Minor note: your parallel code currently doesn't actually allow any
>> meaningful parallelism - you *might* want to move the "lock" so that you
>> only "lock" around the "Add". You're also currently doing a "sync over
>> async" here - I'd probably use:
>>
>> var results = db.StringGet(rKeys);
>>
>> but that is unrelated to the real question here. The
>> _serializer.Deserialize(RedisValue) API you're using isn't something that
>> is directly exposed on protobuf-net, so I'd be grateful if I could see that
>> code. I'm *guessing* it is something like:
>>
>> using (var ms = new MemoryStream((byte[])value))
>> {
>> return Serializer.Deserialize(ms);
>> }
>>
>> ?
>>
>> If that is right, then ... it gets tricky; to give a really meaningful
>> analysis, I'd kinda have to see something akin to realistic data. Is there
>> anything I can do to setup an indicative minimal repro here? Or is there
>> anything you can share with me that I can work with to try and repro what
>> you're seeing? My preference here is a randomized data creation script like
>> my example, but if the data doesn't contain anything privileged, an RDB
>> file (from redis) might work, too.
>>
>> As I say: I can a range of in-progress changes that might help *anyway*,
>> but it would be great if I could actually test things to see how it
>> performs. If this allows me to confirm that the proposed changes make a
>> good improvement, I can always just ship those! But there may be other
>> things I can do here too. The key thing is again, to emphasize: me being
>> able to reproduce what you're seeing - because right now, I'm getting ~1ms
>> times.
>>
>> Marc
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf-net desrialization taking too long

[Marc Gravell]

> using StackExchange.Redis MGET

Yeah, there's really no way for me to dodge this, is there? ;p

Minor note: your parallel code currently doesn't actually allow any
meaningful parallelism - you *might* want to move the "lock" so that you
only "lock" around the "Add". You're also currently doing a "sync over
async" here - I'd probably use:

var results = db.StringGet(rKeys);

but that is unrelated to the real question here. The
_serializer.Deserialize(RedisValue) API you're using isn't something that
is directly exposed on protobuf-net, so I'd be grateful if I could see that
code. I'm *guessing* it is something like:

using (var ms = new MemoryStream((byte[])value))
{
return Serializer.Deserialize(ms);
}

?

If that is right, then ... it gets tricky; to give a really meaningful
analysis, I'd kinda have to see something akin to realistic data. Is there
anything I can do to setup an indicative minimal repro here? Or is there
anything you can share with me that I can work with to try and repro what
you're seeing? My preference here is a randomized data creation script like
my example, but if the data doesn't contain anything privileged, an RDB
file (from redis) might work, too.

As I say: I can a range of in-progress changes that might help *anyway*,
but it would be great if I could actually test things to see how it
performs. If this allows me to confirm that the proposed changes make a
good improvement, I can always just ship those! But there may be other
things I can do here too. The key thing is again, to emphasize: me being
able to reproduce what you're seeing - because right now, I'm getting ~1ms
times.

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] windows cmb protoc error

[Marc Gravell]

that just looks like `protoc` isn't in your path, and to be honest I
wouldn't really expect it to be; you'll need to either add it to your
system path list, or just use it from where-ever it installed

On Wed, 3 Apr 2019 at 16:20, abdelrahman hamdy <
hamdyabdelrahman...@gmail.com> wrote:

> I installed protobuf successfully but when I tried to install protocol I
> didn't get it always pops up an error that its invalid syntax, any help?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Can protocol buffer support partial reading?

[Marc Gravell]

What exactly happens? Protocol buffers are designed to be forwards
compatible, so adding fields etc is usually fine and expected. Depending on
the implementation unexpected fields may be ignored or stored as extension
fields. So: is that what you did? And: what happened?

Note: changing the data type (or meaning) of an existing field is *not*
forwards compatible.

On Mon, 27 May 2019, 12:26 ill W,  wrote:

> I am using protocol buffer in C++ and I notice that if I use an old
> protocol buffer generated code and parse a new protocol buffer file, it
> will crash.
>
> Is there any way to partial parse the file like just ignore the
> unrecognized structure in my new file and get the supported total file?
>
> For example, If I support new layer in caffe model, can I parse the caffe
> model without compiling my code and get all the supported layer in my old
> code?
>
> I have read the protobuf api and find ParsePartialFromIstream function,
> but it still don't work.
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/f65a1540-da98-4813-b012-afad4f665841%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxZjR8g0TLg0n4bqzohYTkPeXEAQnbBKGqY_YjmxipGrA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] C# - Can I compile Proto 2 with compiler Proto 3

[Marc Gravell]

Unless it has been updated recently, then no - the "protoc"/Google C#
implementation is (was?) proto3-only.

On Sun, 2 Jun 2019, 17:23 Derek Perez,  wrote:

> Yes it should be fine, just set syntax = "proto2" in your file, the
> compiler understands both I believe.
>
> On Sun, Jun 2, 2019 at 8:59 AM Marc Gravell 
> wrote:
>
>> If it helps, protobuf-net's schema parser can do this. It is a slightly
>> different API to the Google implementation, though.
>>
>> You can test it online at https://protogen.marcgravell.com/ - or the
>> command-line tool is available as a standalone utility via various
>> mechanisms (the standalone command-line tool is more versatile than the
>> online version, especially when processing multiple files).
>>
>> On Sun, 2 Jun 2019, 14:44 Marina Karmely, 
>> wrote:
>>
>>> HI
>>>
>>> Is there some way to compile ptoto 2 with a compiler of proto 3 in VS
>>> 2019 for C#?
>>> For example it can be done easily in Java(Eclipse and C++)
>>>
>>> But with the latest version of protoc 3.8.0 when I try to compile proto
>>> 2 with compiler of proto 3 I have got the following exception* "C# code
>>> generation only supports proto3 syntax"..*
>>>
>>> *Do you have any advise?*
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To post to this group, send email to protobuf@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/protobuf/0bc3b2c5-f604-432e-9ad9-c679e40f1455%40googlegroups.com
>>> <https://groups.google.com/d/msgid/protobuf/0bc3b2c5-f604-432e-9ad9-c679e40f1455%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to protobuf+unsubscr...@googlegroups.com.
>> To post to this group, send email to protobuf@googlegroups.com.
>> Visit this group at https://groups.google.com/group/protobuf.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/protobuf/CAF95VAyE8hVG8p5qh98ETM91vaaigDbVihXwNX8R1NNb%3DZ3E_A%40mail.gmail.com
>> <https://groups.google.com/d/msgid/protobuf/CAF95VAyE8hVG8p5qh98ETM91vaaigDbVihXwNX8R1NNb%3DZ3E_A%40mail.gmail.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAzUK9LHm6bikE%2BNwax-R%3DzbznP7cGC70_6mEG8bqWdkuQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] C# - Can I compile Proto 2 with compiler Proto 3

[Marc Gravell]

If it helps, protobuf-net's schema parser can do this. It is a slightly
different API to the Google implementation, though.

You can test it online at https://protogen.marcgravell.com/ - or the
command-line tool is available as a standalone utility via various
mechanisms (the standalone command-line tool is more versatile than the
online version, especially when processing multiple files).

On Sun, 2 Jun 2019, 14:44 Marina Karmely,  wrote:

> HI
>
> Is there some way to compile ptoto 2 with a compiler of proto 3 in VS 2019
> for C#?
> For example it can be done easily in Java(Eclipse and C++)
>
> But with the latest version of protoc 3.8.0 when I try to compile proto 2
> with compiler of proto 3 I have got the following exception* "C# code
> generation only supports proto3 syntax"..*
>
> *Do you have any advise?*
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/0bc3b2c5-f604-432e-9ad9-c679e40f1455%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAyE8hVG8p5qh98ETM91vaaigDbVihXwNX8R1NNb%3DZ3E_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protobuf encoding with FIX prtocol

[Marc Gravell]

That definition is simply adding metadata to the field definitions; they
won't directly impact protobuf if you're using the default tools, but if
you're using custom tools, or a layer on top of protobuf that knows how to
inspect metadata, it can do additional things with the information. The
metadata isn't part of *payload* data; it exists independent of any
individual messages.

If you want more info, perhaps try clarifying the question so we understand
which bit it unclear.

On Thu, 11 Jul 2019 at 12:49, abdelrahman hamdy <
hamdyabdelrahman...@gmail.com> wrote:

> can anyone help me understanding this block of code and haw file have this
> value in the same line, example in FIX protocol tage = value so deliver
> companyID  tage=value is 128 = IBM for example.
> i need help please .
>  string deliver_to_comp_id = 3   [(fix.tag)=128, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>
> message StandardHeader {
>string begin_string = 1 [(fix.tag)=8, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>sfixed64 body_length = 2[(fix.tag)=9, (fix.type)=
> DATATYPE_LENGTH, (fix.field_added)=VERSION_FIX_4_0];
>string deliver_to_comp_id = 3   [(fix.tag)=128, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>string deliver_to_sub_id = 4[(fix.tag)=129, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>sfixed64 msg_seq_num = 5[(fix.tag)=34, (fix.type)=
> DATATYPE_SEQ_NUM, (fix.field_added)=VERSION_FIX_4_0];
>MsgTypeEnum msg_type = 6[(fix.tag)=35, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>string on_behalf_of_comp_id = 7 [(fix.tag)=115, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>string on_behalf_of_sub_id = 8  [(fix.tag)=116, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>fix.Timestamp orig_sending_time = 9 [(fix.tag)=122, (fix.type)=
> DATATYPE_UTC_TIMESTAMP, (fix.field_added)=VERSION_FIX_4_0];
>bool poss_dup_flag = 10 [(fix.tag)=43, (fix.type)=
> DATATYPE_BOOLEAN, (fix.field_added)=VERSION_FIX_4_0];
>bool poss_resend = 11   [(fix.tag)=97, (fix.type)=
> DATATYPE_BOOLEAN, (fix.field_added)=VERSION_FIX_4_0];
>string secure_data = 12 [(fix.tag)=91, (fix.type)=
> DATATYPE_DATA, (fix.field_added)=VERSION_FIX_4_0];
>sfixed64 secure_data_len = 13   [(fix.tag)=90, (fix.type)=
> DATATYPE_LENGTH, (fix.field_added)=VERSION_FIX_4_0];
>string sender_comp_id = 14  [(fix.tag)=49, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>string sender_sub_id = 15   [(fix.tag)=50, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>fix.Timestamp sending_time = 16 [(fix.tag)=52, (fix.type)=
> DATATYPE_UTC_TIMESTAMP, (fix.field_added)=VERSION_FIX_4_0];
>string target_comp_id = 17  [(fix.tag)=56, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>string target_sub_id = 18   [(fix.tag)=57, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_0];
>string deliver_to_location_id = 19  [(fix.tag)=145, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_1];
>string on_behalf_of_location_id = 20[(fix.tag)=144,
> (fix.type)=DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_1];
>string sender_location_id = 21  [(fix.tag)=142, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_1];
>string target_location_id = 22  [(fix.tag)=143, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_1];
>sfixed64 last_msg_seq_num_processed = 23[(fix.tag)=369,
> (fix.type)=DATATYPE_SEQ_NUM, (fix.field_added)=VERSION_FIX_4_2];
>string message_encoding = 24[(fix.tag)=347, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_2];
>string xml_data = 25[(fix.tag)=213, (fix.type)=
> DATATYPE_DATA, (fix.field_added)=VERSION_FIX_4_2];
>sfixed64 xml_data_len = 26  [(fix.tag)=212, (fix.type)=
> DATATYPE_LENGTH, (fix.field_added)=VERSION_FIX_4_2];
>ApplVerIDEnum appl_ver_id = 27  [(fix.tag)=1128, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_4];
>string cstm_appl_ver_id = 28[(fix.tag)=1129, (fix.type)=
> DATATYPE_STRING, (fix.field_added)=VERSION_FIX_4_4];
>repeated HopGrp hop_grp = 29
>  [(fix.field_added)=VERSION_FIX_4_4];
>sfixed64 appl_ext_id = 30   [(fix.tag)=1156, (fix.type)=
> DATATYPE_INT, (fix.field_added)=VERSION_FIX_5_0];
> }
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at 

Re: [protobuf] scalar types in service RPCs and partial messages

[Marc Gravell]

The "why" is because the marshaller assumes the root is a message; however,
you should look at "wrappers.proto" - there are well-known wrappers for
single values of most common types, including string. Also, prefer
"empty.proto" for empty, not your own.

On Sat, 29 Jun 2019 at 13:25, Ananya Bhat  wrote:

> Hi,
> I am new to protobuf.
> I have question related to specifying partial messages as RPC parameters
> and using scalar types in RPCs for returning simple strings
> Why message wrappers are required?
> Please clarify my doubts to decide better message specification
> I have definition something like below
>
> Option A
>
> message Student
> {
> message Empty
> {
> }
>
> //Name of the Student
> optional string name = 1;
> // Year of Passing
> optional string YOP  = 2;
> message Certificate
> {
> // Certificate name
> optional string name= 1;
> // Absolute location of the file on the disk
> // e.g. C:\university\2019\
> optional string path= 2;
> //strudent Identification code
> optional string ID  = 3;
> // section e.g. "A", "B"
> optional string section = 4;
> // Certificate type
> optional string type = 5;
> }
> optional Certificate cert = 3;
> }
> service StudentService
> {
> rpc getStudentName(Strudent.Empty) returns (Student);
> rpc getCertificate(Student.Certificate) returns (Student.Certificate);
> }
>
> alternatively
>
> Option B
>
> message Student
> {
>
> //Name of the Student
> optional string name = 1;
>
> message FilePath
> {
> optional string path = 1;
> }
> message Certificate
> {
> optional string   name = 1;
> optional Filepath path = 2;
> //strudent Identification code
> optional string ID  = 3;
> // section e.g. "A", "B"
> optional string section = 4;
> // Certificate type
> optional string type = 5;
>
> }
> optional Certificate cert = 2;
> }
> service StudentService
> {
> rpc getStudentName(Strudent.Empty) returns (Student);
> rpc getCertificate(Student.Certificate.FilePath) returns
> (Student.Certificate);
> }
>
> In Option A, file path can not be specified to get the certificate. Full
> message needs to be passed.
> In Option B, there is a extra type introduced to simplify
> However it adds nested definition for path
> as it has to be cert.path.path for getters and setters
>
> Please clarify the following
> Why scalar types can not be used as IN parameter or OUT parameters of
> Service RPCs?
> How partial messages can be explicitly specified in service RPCs without
> using nesting?
> What are the advantages and disadvantages of using message wrappers?
> Regards
> Ananya
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/544ad006-75ea-44fd-8e60-6d4db311f1ac%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxUPpnoYJXWQ%3DBH-6DWWeDmJ-%2BHEJUTS9bQb0J6_YNBmA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: Nested Types in C# generated code causes serilization issue

[Marc Gravell]

And can we see some code that actually demonstrates this problem? That 
would really help here.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/576e5687-9319-4d3d-bd01-97fbdc79e763%40googlegroups.com.

Re: [protobuf] Re: Nested Types in C# generated code causes serilization issue

[Marc Gravell]

Can you show any of your XmlSerializer code here, so can understand it? I
wouldn't expect the presence of a static type to break XmlSerializer, I
admit! But also: protobuf code doesn't usually guarantee anything other
than that it should work with the corresponding protobuf library.

On Tue, 20 Aug 2019, 06:32 arun kumar,  wrote:

> @Marc Gravell,
>
> I auto-generated ".cs" files from each ".proto" file. . Whereever a
> message declared inside another message in proto, auto-generated cs file is
> generated as " Nested Types " and Types class is generated as static. (I
> guess , its how the generator defined)
>
> And, I am trying to convert my xml data into CS object using XMLSerializer
> , it complains
>
> There was an error reflecting type 'SCL.PC.ClientIdentifier'. --->
>> System.InvalidOperationException: SCL.PC.ClientIdentifier+Types cannot be
>> serialized. Static types cannot be used as parameters or return types.
>
>
> All I am trying to do is ,
>
> Converting XML data --> CS Object (IMessage) --> proto buf Serialize into
>> stream
>
>
>
> Is there any way, to convert the xml data to IMessageObject.. I can see
> MessageParser has ParseJson object.  Likewise, anything available for xml ?
>
>
>var serializer = new XmlSerializer(typeof(SCL.PC.ClientIdentifier));
>
> Auto-Generated CS File :
>
> public sealed partial class ClientIdentifier :
> pb::IMessage {
> private static readonly pb::MessageParser _parser =
> new pb::MessageParser(() => new ClientIdentifier());
> private pb::UnknownFieldSet _unknownFields;
> [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
> public static pb::MessageParser Parser { get {
> return _parser; } }
>
> ---
>
> #region Nested types
> /// Container for nested types declared in the
> ClientIdentifier message type.
> [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
> public static partial class Types {
>   public enum Context {
> /// 
> /// The client's identifier consists of a functional identifier
> (FID) in big-endian format.  Valid values range from 0x to
> 0x.
> /// 
> [pbr::OriginalName("CFID")] Cfid = 0,
> /// 
> /// The client's identifier consists of an ASCII-encoded serial
> number.
> /// 
> [pbr::OriginalName("CSN")] Csn = 1,
>   }
>
> }
> #endregion
>
>   }
>
>   #endregion
>
>
> On Monday, 19 August 2019 19:56:45 UTC+5:30, Marc Gravell wrote:
>>
>> And can we see some code that actually demonstrates this problem? That
>> would really help here.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/a6a61e73-3253-4191-b2a0-27cb63d26bde%40googlegroups.com
> <https://groups.google.com/d/msgid/protobuf/a6a61e73-3253-4191-b2a0-27cb63d26bde%40googlegroups.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAycOVwTeWhrVWJ1e2KPBjVsHeNyrfqfsp6CUadxjhmqWQ%40mail.gmail.com.

Re: [protobuf] Re: Nested Types in C# generated code causes serilization issue

[Marc Gravell]

OK; that helps a bit; now, I've tried that with:

using Google.Protobuf;
using SCL.PC;
using System;
using System.Xml.Serialization;

static class P
{
static void Main()
{
var obj = new ClientIdentifier
{
Context = ClientIdentifier.Types.Context.Cfid,
Value = ByteString.CopyFrom(00, 01, 02, 03)
};
var ser = new XmlSerializer(obj.GetType());
ser.Serialize(Console.Out, obj);
}
}

---

I **cannot** reproduce the error you're seeing, but I *do* get another
error, as shown below.

This tells me, ultimately, that your model *isn't going to work with
XmlSerializer*. And that's fine - the code is only *tested* against the
specific API it is designed for - Google's protobuf library in this case.

If XmlSerializer on some down-level version of .NET has *even more*
problems, that's... unexpected, but not amazingly surprising.

If you want to use multiple serializers, usually you should have a
serialization DTO model *per serializer* - one that is designed for use
with that serializer.

However, there may be another option *if you want*. Google's generated code
that is contributing to the pain here. Looking at your generated code, it
looks like your schema is something like:

syntax = "proto3";
message ClientIdentifier {
  Context context = 1;
  bytes value = 2;
}
enum Context {
  Cfid = 0;
  Csn = 1;
}

Now, I can run that through protobuf-net's tooling (
https://protogen.marcgravell.com/#g50e914251fc5d16b94cdc695c8ebac20) which
gives me the simpler code (attached); if I then run that with the tweaked
code:

static class P
{
static void Main()
{
var obj = new ClientIdentifier
{
Context = Context.Cfid,
Value = new byte[] { 00, 01, 02, 03 }
};
var ser = new XmlSerializer(obj.GetType());
ser.Serialize(Console.Out, obj);
}
}

it now all works fine in my version of .NET, and I'm *guessing* that it'll
work in yours too. This is using a **different library**, specifically
protobuf-net rather than Google.ProtoBuf, but: the key point here is that
they both implement the same specification so should be interchangeable.

Up to you!

---

XmlSerializer exception.

Unhandled exception. System.InvalidOperationException: There was an error
reflecting type 'SCL.PC.ClientIdentifier'.
 ---> System.InvalidOperationException: To be XML serializable, types which
inherit from IEnumerable must have an implementation of Add(System.Object)
at all levels of their inheritance hierarchy. Google.Protobuf.ByteString
does not implement Add(System.Object).
   at System.Xml.Serialization.TypeScope.GetEnumeratorElementType(Type
type, TypeFlags& flags)
   at System.Xml.Serialization.TypeScope.ImportTypeDesc(Type type,
MemberInfo memberInfo, Boolean directReference)
   at System.Xml.Serialization.TypeScope.GetTypeDesc(Type type, MemberInfo
source, Boolean directReference, Boolean throwOnError)
   at System.Xml.Serialization.StructModel.GetPropertyModel(PropertyInfo
propertyInfo)
   at System.Xml.Serialization.StructModel.GetFieldModel(MemberInfo
memberInfo)
   at
System.Xml.Serialization.XmlReflectionImporter.InitializeStructMembers(StructMapping
mapping, StructModel model, Boolean openModel, String typeName,
RecursionLimiter limiter)
   at
System.Xml.Serialization.XmlReflectionImporter.ImportStructLikeMapping(StructModel
model, String ns, Boolean openModel, XmlAttributes a, RecursionLimiter
limiter)
   at
System.Xml.Serialization.XmlReflectionImporter.ImportTypeMapping(TypeModel
model, String ns, ImportContext context, String dataType, XmlAttributes a,
Boolean repeats, Boolean openModel, RecursionLimiter limiter)
   --- End of inner exception stack trace ---
   at
System.Xml.Serialization.XmlReflectionImporter.ImportTypeMapping(TypeModel
model, String ns, ImportContext context, String dataType, XmlAttributes a,
Boolean repeats, Boolean openModel, RecursionLimiter limiter)
   at
System.Xml.Serialization.XmlReflectionImporter.ImportElement(TypeModel
model, XmlRootAttribute root, String defaultNamespace, RecursionLimiter
limiter)
   at System.Xml.Serialization.XmlReflectionImporter.ImportTypeMapping(Type
type, XmlRootAttribute root, String defaultNamespace)
   at System.Xml.Serialization.XmlSerializer..ctor(Type type, String
defaultNamespace)
   at System.Xml.Serialization.XmlSerializer..ctor(Type type)
   at P.Main() in C:\Users\marc\source\repos\ConsoleApp14\Program.cs:line 16


On Wed, 21 Aug 2019 at 10:28, Arun  wrote:

> I did the same but it throwing the error.. I dont know what I am making
> mistake here . All my application are 3.5 .net version,
>
> Please find  actual CS file which I am trying to serialize.
>
> On Wed, 21 Aug 2019 at 14:06, Marc Gravell  wrote:
>
>> K, I took the tiny bit of code you posted, and added something that uses
>> it with XmlSerializer, which basically meant implementing the interface
>> (with NIEs) and addin

Re: [protobuf] Re: Nested Types in C# generated code causes serilization issue

[Marc Gravell]

K, I took the tiny bit of code you posted, and added something that uses it
with XmlSerializer, which basically meant implementing the interface (with
NIEs) and adding:

using System;
using System.Xml.Serialization;
using pb = global::Google.Protobuf;
using pbr = global::Google.Protobuf.Reflection;

static class Program
{
static void Main()
{
var ser = new XmlSerializer(typeof(ClientIdentifier));
var obj = new ClientIdentifier();
ser.Serialize(Console.Out, obj);
}
}

---

And here's the thing: it works fine! So: if anyone is going to be able to
help you here, you're going to need to meet us half way, by *showing us the
code that doesn't work*.

On Wed, 21 Aug 2019 at 07:05, Arun  wrote:

>
> I am not able to initialize XmlSerializer class itself. It is failing
> saying "static Types cannot be serialized "
>
> On Wed, 21 Aug 2019 at 00:20, Marc Gravell  wrote:
>
>> Can you show any of your XmlSerializer code here, so can understand it? I
>> wouldn't expect the presence of a static type to break XmlSerializer, I
>> admit! But also: protobuf code doesn't usually guarantee anything other
>> than that it should work with the corresponding protobuf library.
>>
>> On Tue, 20 Aug 2019, 06:32 arun kumar,  wrote:
>>
>>> @Marc Gravell,
>>>
>>> I auto-generated ".cs" files from each ".proto" file. . Whereever a
>>> message declared inside another message in proto, auto-generated cs file is
>>> generated as " Nested Types " and Types class is generated as static. (I
>>> guess , its how the generator defined)
>>>
>>> And, I am trying to convert my xml data into CS object using
>>> XMLSerializer , it complains
>>>
>>> There was an error reflecting type 'SCL.PC.ClientIdentifier'. --->
>>>> System.InvalidOperationException: SCL.PC.ClientIdentifier+Types cannot be
>>>> serialized. Static types cannot be used as parameters or return types.
>>>
>>>
>>> All I am trying to do is ,
>>>
>>> Converting XML data --> CS Object (IMessage) --> proto buf Serialize
>>>> into stream
>>>
>>>
>>>
>>> Is there any way, to convert the xml data to IMessageObject.. I can see
>>> MessageParser has ParseJson object.  Likewise, anything available for xml ?
>>>
>>>
>>>var serializer = new XmlSerializer(typeof(SCL.PC.ClientIdentifier));
>>>
>>> Auto-Generated CS File :
>>>
>>> public sealed partial class ClientIdentifier :
>>> pb::IMessage {
>>> private static readonly pb::MessageParser _parser
>>> = new pb::MessageParser(() => new ClientIdentifier());
>>> private pb::UnknownFieldSet _unknownFields;
>>> [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
>>> public static pb::MessageParser Parser { get {
>>> return _parser; } }
>>>
>>> ---
>>>
>>> #region Nested types
>>> /// Container for nested types declared in the
>>> ClientIdentifier message type.
>>> [global::System.Diagnostics.DebuggerNonUserCodeAttribute]
>>> public static partial class Types {
>>>   public enum Context {
>>>     /// 
>>> /// The client's identifier consists of a functional identifier
>>> (FID) in big-endian format.  Valid values range from 0x to
>>> 0x.
>>> /// 
>>> [pbr::OriginalName("CFID")] Cfid = 0,
>>> /// 
>>> /// The client's identifier consists of an ASCII-encoded serial
>>> number.
>>> /// 
>>> [pbr::OriginalName("CSN")] Csn = 1,
>>>   }
>>>
>>> }
>>> #endregion
>>>
>>>   }
>>>
>>>   #endregion
>>>
>>>
>>> On Monday, 19 August 2019 19:56:45 UTC+5:30, Marc Gravell wrote:
>>>>
>>>> And can we see some code that actually demonstrates this problem? That
>>>> would really help here.
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to protobuf+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/protobuf/a6a61e73-3253-4191-b2a0-27cb63d26bde%40googlegroups.com
>>> <https://groups.google.com/d/msgid/protobuf/a6a61e73-3253-4191-b2a0-27cb63d26bde%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>>
>
> --
> -- Arun
>
>

-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAwkzCH0uD8JT6McpSND1SEZtw98VObP9iNCta8-jo_P9g%40mail.gmail.com.

Re: [protobuf] What are the encoding formats supported by protobuf?

[Marc Gravell]

Protobuf binary or protobuf-flavored JSON.

On Thu, 18 Jul 2019, 08:35 Pratibha Pruno Xavier, 
wrote:

> Hi all,
>
>  What are the encoding formats supported by protobuf 3?
>
> THanks,
> Pratibha
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/817d2ba0-872c-4f59-a284-1538cd054a5a%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAxWGwPWA1FbP%2B5nzc4mtvtz714T47Fva_m0oAMTDheZxw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Protocol Buffer Wrappers returning single character

[Marc Gravell]

Can you show what you mean by wrappers here? If you dump the payload and
inspect it (protoc, or there's a tool at
https://protogen.marcgravell.com/decode), is the data in the payload? A
minimal example, in some way, would really help

On Tue, 5 Nov 2019, 20:55 SW89,  wrote:

> Good Afternoon,
>
> My team is working to implement Protobuf into our angular/C# web
> application however we have run into a snag regarding the implementation of
> wrapper types.
>
> We needed to support the notion of null values and came across the
> implementation of wrappers to facilitate this with protobuf. We have
> implemented all our non required fields using wrappers and what we have
> seen is that values encapsulated within the Wrappers are only returning a
> single character when inspected server side.
>
> For Example we have a form input in which we enter Name = "Test Name" and
> then when inspecting the server side values all Name is holding is "T".
>
> Has anyone encountered this issue and could shed some light on how to fix
> it.
>
> Thanks,
>
> Sam
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/45b6bfbe-e20b-427d-8bab-26197cfe4293%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAzaA7e%3D5YXEvjUsWEXNRBYTD6sb5j4OTuEtdCY5%2Btg7PQ%40mail.gmail.com.

Re: [protobuf] How to correctly understand the term "wire format ordering" and "on the wire" in document "Language Guide (proto3)"?

[Marc Gravell]

It just means "when serialized" i.e. when written into binary or JSON for
storage / transmission; the "wire format ordering" one is emphasizing that
maps have no defined order whether in-memory or serialized, and you should
not rely on the order in which elements appear when serialized - on disk /
socket / whatever they can appear in *any order they want*, which has
nothing whatsoever to do with the order they were added. The "on the wire"
is used multiple times, but again, always means "when written to binary or
JSON".

On Tue, 5 Nov 2019 at 03:02, ken cao  wrote:

> I really really can't understand clearly these terms.
> thankyou very much.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/2ba6c259-94d7-491f-9cb7-36a840e7bda0%40googlegroups.com
> 
> .
>


-- 
Regards,

Marc

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAyCvqJMjzaf6oyAjaJd_N-kqgr4dphhLL%3D89uhR4_MU%3DA%40mail.gmail.com.

Re: [protobuf] Can the client parse the interface and message structures of the server?

[Marc Gravell]

That would prpbably be a great question for James NK over on the relevant
GitHub repo. IIRC the sample server includes reflection, though:
https://github.com/grpc/grpc-dotnet/

On Sat, 14 Dec 2019, 12:22 erik stroeken,  wrote:

> Thanks Nadav Samet
> I've been trying to implement the concepts described in
> https://github.com/grpc/grpc/blob/master/doc/server-reflection.md and for
> C#:
> https://github.com/grpc/grpc/blob/master/doc/csharp/server_reflection.md in
> the implementation of ASP.NET Core. You have to somehow configure Kestrel
> to enable Reflection first. I just don't succeed and have to give up. Does
> somebody know how to enable serverside reflection in ASP.NET Core with
> Kestrel? Or even better has a complete working example of serverside
> reflection in C# ASP.NET Core?
> Thanks, Erik
>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/protobuf/18d51f1a-8cb4-481e-b9ed-3f3f454d91d8%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/protobuf/CAF95VAwecDj7QsGhr1T-3YtpL7U_oz1V6jUCjPxhV5LHQGmwXQ%40mail.gmail.com.