Assuming you are programming in C or C++, make sure to insert a line of #pragma pack(1)
Which will insure the compiler does not add any padding. Check to insure endianness of both the sender and receiver are the same. Then you will be ready to send a blob of arbitrary bytes. On Tue, Nov 15, 2016 at 1:28 AM Auer, Jens <jens.a...@cgi.com> wrote: > Hi Emmanuel, > > The new mechanism cannot provide a way to ensure any alignment of the > payload buffer for received messages. At least I don't see a way how this > would work because it works directly on the data buffer used to receive the > messages. Since you don't know which and how many messages will be received > you cannot force alignment of the payload part of the messages. > > I don't think that ZeroMQ guaranteed an alignment of the message buffers. > It returns a void* which is guaranteed to be byte-aligned but not anything > else. Personally, I think that you should serialize/deserialize objects > into messages, and this implies copying something into or out of the > buffer. I can see that you want to reduce copy overhead, but since the > message is a zero-copy message now, the copy operation in the > user-application is the only copy you have to do. > > Cheers, > Jens > > -- > Dr. Jens Auer | CGI | Software Engineer > CGI Deutschland Ltd. & Co. KG > Rheinstraße 95 | 64295 Darmstadt | Germany > T: +49 6151 36860 154 > jens.a...@cgi.com > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie > unter de.cgi.com/pflichtangaben. > > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to > CGI Group Inc. and its affiliates may be contained in this message. If you > are not a recipient indicated or intended in this message (or responsible > for delivery of this message to such person), or you think for any reason > that this message may have been addressed to you in error, you may not use > or copy or deliver this message to anyone else. In such case, you should > destroy this message and are asked to notify the sender by reply e-mail. > > > > -----Original Message----- > > From: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] On Behalf > Of > > Emmanuel Taurel > > Sent: 15 November 2016 09:11 > > To: 'ZeroMQ development list' > > Subject: Re: [zeromq-dev] Zmq 4.2.0 aligned memory > > > > Hello gurus, > > > > Thank's very much for your answers. > > Here are some more info related to the case I am debugging. > > As I said, we are using a multipart message with 4 parts. > > On the subscriber side, I have: > > > > Message part 1: Size = 41 bytes - Data ptr = xxxa986 -- zmq_msg_t > instance adr > > = xxx9c90 Message part 2: Size = 1 bytes - Data ptr = xxx9cd8 -- > zmq_msg_t > > instance adr = xxx9cd0 Message part 3: Size = 15 bytes - Data ptr = > xxx9d18 -- > > zmq_msg_t instance adr = xxx9d10 Message part 4: Size = 44 bytes - Data > ptr = > > xxxa9e3 --zmq_msg_t instance adr == xxx9d50 > > > > From my understanding of what happens, it seems that part 2 and 3 are > "vsm" and > > we can see that their buffer is part of the zmq_msg_t instances. > > But parts 1 and 4 (size 41 or 44 bytes) are not "vsm" and their buffers > are in other > > memory chunk that their zmq_msg_t instances. > > But those two buffers are not aligned. This generates troubles in our > buffer decoding > > system! > > > > From Jens answer, I understand that this is related to changes done for > release 4.2 > > Is there a way to retrieve aligned buffers? > > > > Sincerely > > > > Emmanuel > > > > On 11/14/2016 08:34 PM, Jens Auer wrote: > > > > > Hi, > > > > > > I think I have an idea why you are seeing unaligned messages, but this > only > > applies to messages where the payload is not stored in the msg_t object > itself. I > > think the threshold for this is 64 bytes. In ZeroMQ 4.1, receiving > messages was > > done by first receiving from the socket into a static 8kb buffer, and > then a new > > message object was created that allocated memory externally by calling > malloc. > > The payload was then copied from the receive buffer to the message > buffer. The > > malloced message buffer was aligned probably. > > > > > > In ZeroMQ 4.2, this is changed to reduce the number of malloc calls > and copy > > operations. The receive buffer is now dynamically allocated as a 8kb > block, and > > messages are constructed as zero-copy messages using the part of the > receive > > buffer containing the payload. This saves malloc calls and copy > operations and > > increases performance. However, the payload may now start at basically > arbitrary > > addresses. As an example, let's assume that we receive a small message > of 10 > > bytes and a large message of 1kb, both received in a single call to recv > on the > > socket. The engine allocates a new buffer of 8kb, calls recv(socket, > buffer) and the > > data is written to the buffer. A small message is then created which > contains the > > data from byte 2-11 in the msg_t, byte 1 contains the header. At byte 12 > starts the > > header of the next message, and at byte 22(?) starts the payload. The > large > > message is created as a zero-copy message using the pointer to byte 22 > as storage. > > This is not aligned to a 4-byte address. > > > > > > Could you provide some more information about the sizes of the > messages that > > you receive? How do you decode the buffer content? > > > > > > Best wishes, > > > Jens > > > > > > -----Ursprüngliche Nachricht----- > > > Von: zeromq-dev [mailto:zeromq-dev-boun...@lists.zeromq.org] Im > > > Auftrag von Emmanuel Taurel > > > Gesendet: Montag, 14. November 2016 16:49 > > > An: zeromq-dev@lists.zeromq.org > > > Betreff: [zeromq-dev] Zmq 4.2.0 aligned memory > > > > > > Hello all, > > > > > > We are using zeromq since years now without troubles. We have recently > tried our > > software using Zmq 4.2.0 (on linux hosts). > > > For our application, we are using multipart messages with 4 parts in > > publish/subscribe mode. > > > With Zmq 4.0.5, on the subscriber side, when we get the last message > part, the > > received buffer was memory aligned (at least on 0x4 border). > Unfortunately, with > > Zmq 4.2.0, the buffer is not aligned any more. > > > For instance with Zmq 4.0.5, the buffer was at address xxx08 while > with Zmq > > 4.2.0, it is at address xxx23. > > > > > > I don't know if it is relevant but our messages are relatively small > messages (few > > tens of bytes) This is a problem for us in decoding the buffer content. > > > > > > Is there something to be done to have memory aligned buffers? > > > > > > Thank's in advance for your answers > > > > > > Emmanuel > > > _______________________________________________ > > > zeromq-dev mailing list > > > zeromq-dev@lists.zeromq.org > > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > > > > > > --- > > > Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. > > > https://www.avast.com/antivirus > > > > > > > > > _______________________________________________ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
