Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
BTW just to make sure, what is the 2nd and 3rd line going to return? IXpbBuilder* pb = utl->getXpbBuilder(&status, IXpbBuilder::BATCH, NULL, 0); pb->getBufferLength(&status); pb->getBuffer(&status); -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
>> Is that the msgLen in BlrFromMessage::buildBlr? Looking at the code then the >> length for SQL_LONG should be 6 (4 + 2), isn't it? >> > > Yes. > When you will send >1 message - pay attention that there is also > alignedMessageLength, which is 8 in this case. For now I'm just trying to send one message. Baby steps. But still no success. Can you maybe build Firebird with truck-load (and I mean borderline masochistic amount) of logging around the batching, so I can see at what point was something not as expected? This is, high level (low level in parentheses), what I'm sending now. This is not resulting in error, but server is not responding to this operation, so something is missing. op_batch_create (0,0,0,99) p_batch_statement (0,0,0,2) p_batch_blr buffer length (0,0,0,12) p_batch_blr - 1x SQL_LONG (5,2,4,0,2,0,8,0,7,0,255,76) - let's assume the BLR correct, because the same code is used in other parts, like in op_fetch and it works p_batch_msglen (0,0,0,6) p_batch_pb buffer length (0,0,0,1) p_batch_pb - empty PB VERSION1 (1) padding (0,0,0) In fact, I'm not creating a new code, just using other parts and putting in the parameters. So something is lost in my understanding from reading the server code. -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
On 12/16/21 17:11, Jiří Činčura wrote: I'm bit confused here. I thought it's the size of the p_batch_blr (which looked weird). But you're saying it's "message size". What's that exactly? Is that the msgLen in BlrFromMessage::buildBlr? Looking at the code then the length for SQL_LONG should be 6 (4 + 2), isn't it? Yes. When you will send >1 message - pay attention that there is also alignedMessageLength, which is 8 in this case. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
On 12/16/21 16:49, Jiří Činčura wrote: p_batch_msglen is needed for server to check that in matches message size, calculated based on BLR. And it seems that in your case it does not match - may be because you want bigint parameter but use blr_long. I'm bit confused here. I thought it's the size of the p_batch_blr (which looked weird). But you're saying it's "message size". What's that exactly? Size of a message that should be transfered in a format defined by BLR. I.e. inMessage->getMessageLength() Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
> Is that the msgLen in BlrFromMessage::buildBlr? Looking at the code > then the length for SQL_LONG should be 6 (4 + 2), isn't it? *One* SQL_LONG, to be precise. -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
> I'm bit confused here. I thought it's the size of the p_batch_blr > (which looked weird). But you're saying it's "message size". What's > that exactly? Is that the msgLen in BlrFromMessage::buildBlr? Looking at the code then the length for SQL_LONG should be 6 (4 + 2), isn't it? -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
> the core developers don't document the actual protocol, because reverse > engineering from the code and by using things like WireShark is > extremely painful, confusing and error prone. With the batch API it's > even worse IMHO, because even the usage examples seem to be incomplete, > and too basic/simple and glosses over things. I'll add one pain point. Companies/people/developers are using Firebird and expect to have new features available (in .NET in my case) in timely manner. But when I need to keep fishing and trying options here and there the progress is slow and everybody is unhappy (and even looking for other options). -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
> p_batch_msglen is needed for server to check that in matches message > size, calculated based on BLR. And it seems that in your case it does > not match - may be because you want bigint parameter but use blr_long. I'm bit confused here. I thought it's the size of the p_batch_blr (which looked weird). But you're saying it's "message size". What's that exactly? -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
On 2021-12-16 14:23, Alex Peshkoff via Firebird-devel wrote: On 12/16/21 15:12, Mark Rotteveel wrote: On 2021-12-16 10:28, Alex Peshkoff via Firebird-devel wrote: be segmented and swap bytes in segment length when needed. Afraid there is - as always :( - no protocol description except source code. Key ops are op_batch_msg, op_batch_blob_stream & also op_batch_cs (retuning batch execution state to client). That is not entirely true, there is [1] which was basically reverse-engineerd by Carlos Guzman Alvarez, and I updated some of it for protocol 11. Personally, I really don't like the current situation that the core developers don't document the actual protocol, because reverse engineering from the code and by using things like WireShark is extremely painful, confusing and error prone. With the batch API it's even worse IMHO, because even the usage examples seem to be incomplete, and too basic/simple and glosses over things. Mark [1]: https://firebirdsql.org/file/documentation/html/en/firebirddocs/wireprotocol/firebird-wire-protocol.html And where is the source of this docs? It's in the firebird-documentation project[1], specifically https://github.com/FirebirdSQL/firebird-documentation/blob/master/src/docs/asciidoc/en/firebirddocs/wireprotocol/firebird-wire-protocol.adoc Mark [1]: https://github.com/FirebirdSQL/firebird-documentation Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
On 12/16/21 15:12, Mark Rotteveel wrote: On 2021-12-16 10:28, Alex Peshkoff via Firebird-devel wrote: be segmented and swap bytes in segment length when needed. Afraid there is - as always :( - no protocol description except source code. Key ops are op_batch_msg, op_batch_blob_stream & also op_batch_cs (retuning batch execution state to client). That is not entirely true, there is [1] which was basically reverse-engineerd by Carlos Guzman Alvarez, and I updated some of it for protocol 11. Personally, I really don't like the current situation that the core developers don't document the actual protocol, because reverse engineering from the code and by using things like WireShark is extremely painful, confusing and error prone. With the batch API it's even worse IMHO, because even the usage examples seem to be incomplete, and too basic/simple and glosses over things. Mark [1]: https://firebirdsql.org/file/documentation/html/en/firebirddocs/wireprotocol/firebird-wire-protocol.html And where is the source of this docs? Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
On 2021-12-16 10:28, Alex Peshkoff via Firebird-devel wrote: be segmented and swap bytes in segment length when needed. Afraid there is - as always :( - no protocol description except source code. Key ops are op_batch_msg, op_batch_blob_stream & also op_batch_cs (retuning batch execution state to client). That is not entirely true, there is [1] which was basically reverse-engineerd by Carlos Guzman Alvarez, and I updated some of it for protocol 11. Personally, I really don't like the current situation that the core developers don't document the actual protocol, because reverse engineering from the code and by using things like WireShark is extremely painful, confusing and error prone. With the batch API it's even worse IMHO, because even the usage examples seem to be incomplete, and too basic/simple and glosses over things. Mark [1]: https://firebirdsql.org/file/documentation/html/en/firebirddocs/wireprotocol/firebird-wire-protocol.html Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
On 12/16/21 13:48, Jiří Činčura wrote: Afraid there is - as always :( - no protocol description except source code. Key ops are op_batch_msg, op_batch_blob_stream & also op_batch_cs (retuning batch execution state to client). Can you create some simple paragraph with the description? Because fishing it out from source code takes time and also misses often important details of handling some edge cases. I'm trying to make it work now mostly with hardcoded values and later somehow fit it into DbBatch (https://docs.microsoft.com/en-us/dotnet/api/system.data.common.dbbatch?view=net-6.0) which is new in recent .NET 6. But that has broader features than what's possible in Firebird now. I do op_allocate_statement and op_prepare_statement and then I'm trying op_batch_create. That seems to be reasonable order. But I'm getting (in)famous SQL error code = -804, SQLDA error (isc_sqlerr, isc_dsql_sqlda_err). I'm sending: 0,0,0,99 (op_batch_create) 0,0,0,2 (p_batch_statement - (from op_allocate_statement)) 0,0,0,12 (buffer length) 5,2,4,0,2,0,8,0,7,0,255,76 (p_batch_blr buffer) 0,0,0,0,0,0,0,12 (p_batch_msglen) 0,0,0,10 (buffer length) 1,2,4,0,0,0,1,0,0,0 (p_batch_pb buffer - (TAG_RECORD_COUNTS,1)) 0,0 (pad) The BLR is if I'm not mistaken (1 bigint parameter): 5 => blr_version5 2 => blr_begin 4 => blr_message 0 => constant 2,0 => 1 param * 2 8 => blr_long 0 => scale 7 => blr_short 0 => constant 255 => blr_end 76 => blr_eoc That all looks (obviously :)) correct to me. But it's not, why else would server complain? :D What looks weird to me is the length in p_batch_msglen, because it's already sent as part of the buffer sending. I'm playing with that now. What makes me very hard to answer your question is that I've never used to look at wire protocol on that low level like exact bytes sequence when writing and debugging batch code. (I've used to do that once, when fixed sending decfloat values between LE/BE machines, but that was only once - and only that problematic 16 bytes.) There are standard ways to send and receive standard things over the wire, and I need not care what exactly was sent. p_batch_msglen is needed for server to check that in matches message size, calculated based on BLR. And it seems that in your case it does not match - may be because you want bigint parameter but use blr_long. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
> Afraid there is - as always :( - no protocol > description except source code. Key ops are op_batch_msg, > op_batch_blob_stream & also op_batch_cs (retuning batch execution state > to client). Can you create some simple paragraph with the description? Because fishing it out from source code takes time and also misses often important details of handling some edge cases. I'm trying to make it work now mostly with hardcoded values and later somehow fit it into DbBatch (https://docs.microsoft.com/en-us/dotnet/api/system.data.common.dbbatch?view=net-6.0) which is new in recent .NET 6. But that has broader features than what's possible in Firebird now. I do op_allocate_statement and op_prepare_statement and then I'm trying op_batch_create. That seems to be reasonable order. But I'm getting (in)famous SQL error code = -804, SQLDA error (isc_sqlerr, isc_dsql_sqlda_err). I'm sending: 0,0,0,99 (op_batch_create) 0,0,0,2 (p_batch_statement - (from op_allocate_statement)) 0,0,0,12 (buffer length) 5,2,4,0,2,0,8,0,7,0,255,76 (p_batch_blr buffer) 0,0,0,0,0,0,0,12 (p_batch_msglen) 0,0,0,10 (buffer length) 1,2,4,0,0,0,1,0,0,0 (p_batch_pb buffer - (TAG_RECORD_COUNTS,1)) 0,0 (pad) The BLR is if I'm not mistaken (1 bigint parameter): 5 => blr_version5 2 => blr_begin 4 => blr_message 0 => constant 2,0 => 1 param * 2 8 => blr_long 0 => scale 7 => blr_short 0 => constant 255 => blr_end 76 => blr_eoc That all looks (obviously :)) correct to me. But it's not, why else would server complain? :D What looks weird to me is the length in p_batch_msglen, because it's already sent as part of the buffer sending. I'm playing with that now. -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
On 12/16/21 11:51, Jiří Činčura wrote: What should be in p_batch_create.p_batch_pb? Is this some new format or something like DPB? Exactly like DPB v.2 (i.e. 32-bit clumplet lengths). And what should - logically - be inside? Or maybe more broadly, is there some high level description of the batching? I'm failing to find something, except batch interface description in Using_OO_API.html, but that's about API, not about the "protocol". Sooner of all there is no other description except interface description in Using_OO_API. What about particular parameters block - here protocol is very simple, it's as always sent 'as is' (remember, we store data in all parameters block in network form). Sending data over the wire does not differ from other places - it's a stream of messages, no difference with other places where messages are sent. A more complicated thing is blob transfer because blobs are also always sent in a stream but we must take into an account that actual data may be segmented and swap bytes in segment length when needed. Afraid there is - as always :( - no protocol description except source code. Key ops are op_batch_msg, op_batch_blob_stream & also op_batch_cs (retuning batch execution state to client). Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Content for p_batch_create.p_batch_pb
>> What should be in p_batch_create.p_batch_pb? Is this some new format or >> something like DPB? >> > > Exactly like DPB v.2 (i.e. 32-bit clumplet lengths). And what should - logically - be inside? Or maybe more broadly, is there some high level description of the batching? I'm failing to find something, except batch interface description in Using_OO_API.html, but that's about API, not about the "protocol". -- Mgr. Jiří Činčura https://www.tabsoverspaces.com/ Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
