[sqlite] Pascal (almost) style BLOBs
On 4/18/2016 6:46 AM, Paul Sanderson wrote: > You could use a binary PList or some other form of structured data - > the BPList blob could then contain, ints, floats, strings byte arrays > or even embedded binary plists. As it is a structured format then to > an extent it is self documenting (in terms of the stored data types) > Paul Of course! I don't have to be that complicated though, do I? I've pretty well convinced myself that adding a type indicator byte to the blob will provide all the information I need to unblob it properly. I haven't heard any major objections from the list here, so I'm going try it out. Thanks for the suggestion, -Bill
[sqlite] Pascal (almost) style BLOBs
You could use a binary PList or some other form of structured data - the BPList blob could then contain, ints, floats, strings byte arrays or even embedded binary plists. As it is a structured format then to an extent it is self documenting (in terms of the stored data types) Paul www.sandersonforensics.com skype: r3scue193 twitter: @sandersonforens Tel +44 (0)1326 572786 http://sandersonforensics.com/forum/content.php?195-SQLite-Forensic-Toolkit -Forensic Toolkit for SQLite email from a work address for a fully functional demo licence On 17 April 2016 at 16:27, William Drago wrote: > On 4/17/2016 9:45 AM, Simon Slavin wrote: >> >> On 17 Apr 2016, at 1:02pm, William Drago wrote: >> >>> I am forced to consider this approach because my function will be >>> receiving blobs but will have no idea what those blobs are. >> >> Why do you care ? A BLOB is a BLOB. It doesn't matter whether it's an >> integer, a string, a car, a peace treaty or the smell of newly cut grass. >> You store it when told to and retrieve it when told to. > > > My job is to find all the blob columns and unblob them into arrays of > whatever type they originally were before they were blob'd. > >>> if this "feature" is not clearly documented someone in the future will >>> have a very hard time figuring out why the blobs don't make sense. >> >> BLOBs don't make sense. They're opaque. You are not meant to understand >> anything about what's in a BLOB. That's for the layer of your software >> which packs the information into a blob or unpacks the blob back into >> information. > > > Understood. I'm writing the code that does the packing and unpacking. The > users are designing their own databases. When they store for example, an > array of 16 bit integers, or an array of 64 bit floats, I want to give them > the same thing back when they query the database not just an array of bytes > that they have to decode themselves. > > Since I have control over how the blobs are generated, I can easily add a > type identifier to each blob so that I know how to unblob it. > > -Bill > ___ > sqlite-users mailing list > sqlite-users at mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Pascal (almost) style BLOBs
I'm not sure I understand why you can't specify the type in a separate column, rather than prepending it to the blob? That seems like a more flexible way to have access to the information. Regards, Ketil On 17 Apr 2016 2:02 p.m., "William Drago" wrote: > All, > > Any thoughts on using the first byte of a BLOB to indicate what kind of > data that BLOB contains? > > For example: > 0 = plain old bytes > 1 = 16 bit integers > 2 = 32 bit integers > 3 = singles > 4 = doubles, etc. > > I am forced to consider this approach because my function will be > receiving blobs but will have no idea what those blobs are. However, I can > ask the sender to prepend the blob with an indicator byte to provide the > necessary information. > > In the past I have used comments in the table structure and even used > custom types (e.g. i16BLOB, for a blob that contains 16 bit ints), but in > this case I will not have access to that information. The data is coming to > me in the form of an ADO.NET DataTable which does not contain such > information. > > Of course tagging the blobs like this will increase the size of the > database, but not by much, and if this "feature" is not clearly documented > someone in the future will have a very hard time figuring out why the blobs > don't make sense. > > Is there anything else I should be aware of? > > Thanks, > Bill > > > > > > > ___ > sqlite-users mailing list > sqlite-users at mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users >
[sqlite] Pascal (almost) style BLOBs
On 17 Apr 2016, at 1:02pm, William Drago wrote: > I am forced to consider this approach because my function will be receiving > blobs but will have no idea what those blobs are. Why do you care ? A BLOB is a BLOB. It doesn't matter whether it's an integer, a string, a car, a peace treaty or the smell of newly cut grass. You store it when told to and retrieve it when told to. > if this "feature" is not clearly documented someone in the future will have a > very hard time figuring out why the blobs don't make sense. BLOBs don't make sense. They're opaque. You are not meant to understand anything about what's in a BLOB. That's for the layer of your software which packs the information into a blob or unpacks the blob back into information. If you actually want to store, for example, integers, you shouldn't be storing a BLOB in the first place. Simon.
[sqlite] Pascal (almost) style BLOBs
On 4/17/2016 10:13 AM, Brad Stiles wrote: > An ADO.NET DataTable *can* have type information for table columns, if you > add a parameter to the WriteXml method to have it write the schema along with > the data. Obviously, *their* data must be typed from their source, or be > inferred, for this to work. If the DataTable is the result of a SELECT that involves more than one table is the schema information included for each of the tables? > See if your provider can do this for you. If they can, you should have little > problem reading it, assuming you're not using .Net yourself. I'm confused by your statement. I am using .NET, that's why I'm dealing with ADO.NET DataTables. Thanks, -Bill
[sqlite] Pascal (almost) style BLOBs
On 4/17/2016 9:45 AM, Simon Slavin wrote: > On 17 Apr 2016, at 1:02pm, William Drago wrote: > >> I am forced to consider this approach because my function will be receiving >> blobs but will have no idea what those blobs are. > Why do you care ? A BLOB is a BLOB. It doesn't matter whether it's an > integer, a string, a car, a peace treaty or the smell of newly cut grass. > You store it when told to and retrieve it when told to. My job is to find all the blob columns and unblob them into arrays of whatever type they originally were before they were blob'd. >> if this "feature" is not clearly documented someone in the future will have >> a very hard time figuring out why the blobs don't make sense. > BLOBs don't make sense. They're opaque. You are not meant to understand > anything about what's in a BLOB. That's for the layer of your software which > packs the information into a blob or unpacks the blob back into information. Understood. I'm writing the code that does the packing and unpacking. The users are designing their own databases. When they store for example, an array of 16 bit integers, or an array of 64 bit floats, I want to give them the same thing back when they query the database not just an array of bytes that they have to decode themselves. Since I have control over how the blobs are generated, I can easily add a type identifier to each blob so that I know how to unblob it. -Bill
[sqlite] Pascal (almost) style BLOBs
On 4/17/2016 9:20 AM, Ketil Froyn wrote: > I'm not sure I understand why you can't specify the type in a separate > column, rather than prepending it to the blob? That seems like a more > flexible way to have access to the information. I have no control over the database schema, table structure, etc.. I also have no control of the select statement that is generating that data. -Bill
[sqlite] Pascal (almost) style BLOBs
On 4/17/2016 9:45 AM, Simon Slavin wrote: > On 17 Apr 2016, at 1:02pm, William Drago wrote: > >> I am forced to consider this approach because my function will be receiving >> blobs but will have no idea what those blobs are. > Why do you care ? A BLOB is a BLOB. It doesn't matter whether it's an > integer, a string, a car, a peace treaty or the smell of newly cut grass. > You store it when told to and retrieve it when told to. My job is to find all the blob columns and unblob them into arrays of whatever type they originally were before they were blob'd. >> if this "feature" is not clearly documented someone in the future will have >> a very hard time figuring out why the blobs don't make sense. > BLOBs don't make sense. They're opaque. You are not meant to understand > anything about what's in a BLOB. That's for the layer of your software which > packs the information into a blob or unpacks the blob back into information. Understood. I'm writing the code that does the packing and unpacking. The users are designing their own databases. When they store for example, an array of 16 bit integers, or an array of 64 bit floats, I want to give them the same thing back when they query the database not just an array of bytes that they have to decode themselves. Since I have control over how the blobs are generated, I can easily add a type identifier to each blob so that I know how to unblob it. -Bill
[sqlite] Pascal (almost) style BLOBs
On 4/17/2016 9:20 AM, Ketil Froyn wrote: > I'm not sure I understand why you can't specify the type in a separate > column, rather than prepending it to the blob? That seems like a more > flexible way to have access to the information. I have no control over the database schema, table structure, etc.. I also have no control of the select statement that is generating that data. -Bill
[sqlite] Pascal (almost) style BLOBs
An ADO.NET DataTable *can* have type information for table columns, if you add a parameter to the WriteXml method to have it write the schema along with the data. Obviously, *their* data must be typed from their source, or be inferred, for this to work. See if your provider can do this for you. If they can, you should have little problem reading it, assuming you're not using .Net yourself. > On Apr 17, 2016, at 07:02, William Drago wrote: > > All, > > Any thoughts on using the first byte of a BLOB to indicate what kind of data > that BLOB contains? > > For example: > 0 = plain old bytes > 1 = 16 bit integers > 2 = 32 bit integers > 3 = singles > 4 = doubles, etc. > > I am forced to consider this approach because my function will be receiving > blobs but will have no idea what those blobs are. However, I can ask the > sender to prepend the blob with an indicator byte to provide the necessary > information. > > In the past I have used comments in the table structure and even used custom > types (e.g. i16BLOB, for a blob that contains 16 bit ints), but in this case > I will not have access to that information. The data is coming to me in the > form of an ADO.NET DataTable which does not contain such information. > > Of course tagging the blobs like this will increase the size of the database, > but not by much, and if this "feature" is not clearly documented someone in > the future will have a very hard time figuring out why the blobs don't make > sense. > > Is there anything else I should be aware of? > > Thanks, > Bill > > > > > > > ___ > sqlite-users mailing list > sqlite-users at mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Pascal (almost) style BLOBs
All, Any thoughts on using the first byte of a BLOB to indicate what kind of data that BLOB contains? For example: 0 = plain old bytes 1 = 16 bit integers 2 = 32 bit integers 3 = singles 4 = doubles, etc. I am forced to consider this approach because my function will be receiving blobs but will have no idea what those blobs are. However, I can ask the sender to prepend the blob with an indicator byte to provide the necessary information. In the past I have used comments in the table structure and even used custom types (e.g. i16BLOB, for a blob that contains 16 bit ints), but in this case I will not have access to that information. The data is coming to me in the form of an ADO.NET DataTable which does not contain such information. Of course tagging the blobs like this will increase the size of the database, but not by much, and if this "feature" is not clearly documented someone in the future will have a very hard time figuring out why the blobs don't make sense. Is there anything else I should be aware of? Thanks, Bill
