Re: [Firebird-devel] Batch API wire protocol
On 11/10/18 12:12 PM, Mark Rotteveel wrote: I do get the feeling (although I maybe wrong, as I don't think I fully understand the protocol), that this will be hard to implement, Bot too hard. The most complex place is when we need to process segments inside segmented blobs (i.e. change endianness of segment size placed in the beginning of segment). and it feels like the message structures need a lot of upfront knowledge of total message sizes Upfront? May be we can say so... Message size and alignment is received from IMessageMetadata using standard API function, no dark magic here :) But certainy that should be done before processing batch. which could be pretty memory intensive if considering blobs, and just a bit annoying for rows without blobs). Except need to parse segments in segmented blob I see no problems here. What about memory consumption - on client it does not depend upon presence of blobs, all data is gathered into single stream and placed into same buffer. It's size 128 or 256Kb, i.e. not too big for 21 century :) Couldn't this be more like a fetch (that is: streaming rows as individual messages)? Rows should be treated as individual message at one step (when we talk about the wire) - when that message is converted into machine-neutral stream of bytes (i.e. XDR protocol). After it messages are once again gathered into the stream and buffered, first of all to minimize calls to network layer. How are they represented before XDR is implementation-specific thing, and certainly may be done in java client in some other way, not exactly like fbclient does. Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Batch API wire protocol
This doesn't really provide me sufficient information to implement it,
but given I don't have the time and energy to work on this anyway, I'll
park it until I have.
I do get the feeling (although I maybe wrong, as I don't think I fully
understand the protocol), that this will be hard to implement, and it
feels like the message structures need a lot of upfront knowledge of
total message sizes which could be pretty memory intensive if
considering blobs, and just a bit annoying for rows without blobs).
Couldn't this be more like a fetch (that is: streaming rows as
individual messages)?
Mark
On 27-10-2018 19:35, Alex Peshkoff via Firebird-devel wrote:
On 10/27/18 15:02, Mark Rotteveel wrote:
What is the wire protocol of the batch API (that is: what operations,
what message format, etc)?
op_batch_create = 99,
op_batch_msg = 100,
op_batch_exec = 101,
op_batch_rls = 102,
op_batch_cs = 103,
op_batch_regblob = 104,
op_batch_blob_stream = 105,
op_batch_set_bpb = 106,
typedef struct p_batch_create
{
OBJCT p_batch_statement; // statement object
CSTRING_CONST p_batch_blr; // blr describing input messages
ULONG p_batch_msglen; // explicit message length
CSTRING_CONST p_batch_pb; // parameters block
} P_BATCH_CREATE;
typedef struct p_batch_msg
{
OBJCT p_batch_statement; // statement object
ULONG p_batch_messages; // number of messages
CSTRING p_batch_data;
} P_BATCH_MSG;
typedef struct p_batch_exec
{
OBJCT p_batch_statement; // statement object
OBJCT p_batch_transaction; // transaction object
} P_BATCH_EXEC;
typedef struct p_batch_cs // completion state
{
OBJCT p_batch_statement; // statement object
ULONG p_batch_reccount; // total records
ULONG p_batch_updates; // update counters
ULONG p_batch_vectors; // recnum + status vector pairs
ULONG p_batch_errors; // error's recnums
} P_BATCH_CS;
typedef struct p_batch_free
{
OBJCT p_batch_statement; // statement object
} P_BATCH_FREE;
typedef struct p_batch_blob
{
OBJCT p_batch_statement; // statement object
CSTRING p_batch_blob_data; // data
} P_BATCH_BLOB;
typedef struct p_batch_regblob
{
OBJCT p_batch_statement; // statement object
SQUAD p_batch_exist_id; // id of blob to register
SQUAD p_batch_blob_id; // blob id
} P_BATCH_REGBLOB;
typedef struct p_batch_setbpb
{
OBJCT p_batch_statement; // statement object
CSTRING_CONST p_batch_blob_bpb; // BPB
} P_BATCH_SETBPB;
There are 3 complex operations - batch messages (sends N messages, they
should be XDR-encoded during it), batch completion state (you should be
able to receive it - and mention struct is followed on the wire by sets
of data) and batch blob stream (you should learn to XDR-encode it). The
rest are trivial, but this 3 (specially blob stream) have rather complex
format in addition to mentioned structures.
Also, is it possible to use this protocol in a lower protocol version
when talking to Firebird 4 (eg Jaybird currently only implements
protocols v10 - 13), or does it really need to be connected with
protocol v16 (if I have my protocol versions right)?
Mark, as far as I remember I did not add any explicit checks for
protocol version. But I'm sure that batches (speically blob stream) is
at least 90% of changes needed to implement appropriate protocol.
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
--
Mark Rotteveel
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Batch API wire protocol
On 10/27/18 15:02, Mark Rotteveel wrote:
What is the wire protocol of the batch API (that is: what operations,
what message format, etc)?
op_batch_create = 99,
op_batch_msg = 100,
op_batch_exec = 101,
op_batch_rls = 102,
op_batch_cs = 103,
op_batch_regblob = 104,
op_batch_blob_stream = 105,
op_batch_set_bpb = 106,
typedef struct p_batch_create
{
OBJCT p_batch_statement; // statement object
CSTRING_CONST p_batch_blr; // blr describing input messages
ULONG p_batch_msglen; // explicit message length
CSTRING_CONST p_batch_pb; // parameters block
} P_BATCH_CREATE;
typedef struct p_batch_msg
{
OBJCT p_batch_statement; // statement object
ULONG p_batch_messages; // number of messages
CSTRING p_batch_data;
} P_BATCH_MSG;
typedef struct p_batch_exec
{
OBJCT p_batch_statement; // statement object
OBJCT p_batch_transaction; // transaction object
} P_BATCH_EXEC;
typedef struct p_batch_cs // completion state
{
OBJCT p_batch_statement; // statement object
ULONG p_batch_reccount; // total records
ULONG p_batch_updates; // update counters
ULONG p_batch_vectors; // recnum + status vector pairs
ULONG p_batch_errors; // error's recnums
} P_BATCH_CS;
typedef struct p_batch_free
{
OBJCT p_batch_statement; // statement object
} P_BATCH_FREE;
typedef struct p_batch_blob
{
OBJCT p_batch_statement; // statement object
CSTRING p_batch_blob_data; // data
} P_BATCH_BLOB;
typedef struct p_batch_regblob
{
OBJCT p_batch_statement; // statement object
SQUAD p_batch_exist_id; // id of blob to register
SQUAD p_batch_blob_id; // blob id
} P_BATCH_REGBLOB;
typedef struct p_batch_setbpb
{
OBJCT p_batch_statement; // statement object
CSTRING_CONST p_batch_blob_bpb; // BPB
} P_BATCH_SETBPB;
There are 3 complex operations - batch messages (sends N messages, they
should be XDR-encoded during it), batch completion state (you should be
able to receive it - and mention struct is followed on the wire by sets
of data) and batch blob stream (you should learn to XDR-encode it). The
rest are trivial, but this 3 (specially blob stream) have rather complex
format in addition to mentioned structures.
Also, is it possible to use this protocol in a lower protocol version
when talking to Firebird 4 (eg Jaybird currently only implements
protocols v10 - 13), or does it really need to be connected with
protocol v16 (if I have my protocol versions right)?
Mark, as far as I remember I did not add any explicit checks for
protocol version. But I'm sure that batches (speically blob stream) is
at least 90% of changes needed to implement appropriate protocol.
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
[Firebird-devel] Batch API wire protocol
What is the wire protocol of the batch API (that is: what operations, what message format, etc)? Also, is it possible to use this protocol in a lower protocol version when talking to Firebird 4 (eg Jaybird currently only implements protocols v10 - 13), or does it really need to be connected with protocol v16 (if I have my protocol versions right)? Mark -- Mark Rotteveel Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
