[sqlite] Pascal (almost) style BLOBs

2016-04-19 Thread William Drago
On 4/18/2016 6:46 AM, Paul Sanderson wrote:
> You could use a binary PList or some other form of structured data -
> the BPList blob could then contain, ints, floats, strings byte arrays
> or even embedded binary plists. As it is a structured format then to
> an extent it is self documenting (in terms of the stored data types)
> Paul

Of course! I don't have to be that complicated though, do I? 
I've pretty well convinced myself that adding a type 
indicator byte to the blob will provide all the information 
I need to unblob it properly. I haven't heard any major 
objections from the list here, so I'm going try it out.

Thanks for the suggestion,
-Bill



[sqlite] Pascal (almost) style BLOBs

2016-04-18 Thread Paul Sanderson
You could use a binary PList or some other form of structured data -
the BPList blob could then contain, ints, floats, strings byte arrays
or even embedded binary plists. As it is a structured format then to
an extent it is self documenting (in terms of the stored data types)
Paul
www.sandersonforensics.com
skype: r3scue193
twitter: @sandersonforens
Tel +44 (0)1326 572786
http://sandersonforensics.com/forum/content.php?195-SQLite-Forensic-Toolkit
-Forensic Toolkit for SQLite
email from a work address for a fully functional demo licence


On 17 April 2016 at 16:27, William Drago  wrote:
> On 4/17/2016 9:45 AM, Simon Slavin wrote:
>>
>> On 17 Apr 2016, at 1:02pm, William Drago  wrote:
>>
>>> I am forced to consider this approach because my function will be
>>> receiving blobs but will have no idea what those blobs are.
>>
>> Why do you care ?  A BLOB is a BLOB.  It doesn't matter whether it's an
>> integer, a string, a car, a peace treaty or the smell of newly cut grass.
>> You store it when told to and retrieve it when told to.
>
>
> My job is to find all the blob columns and unblob them into arrays of
> whatever type they originally were before they were blob'd.
>
>>> if this "feature" is not clearly documented someone in the future will
>>> have a very hard time figuring out why the blobs don't make sense.
>>
>> BLOBs don't make sense.  They're opaque.  You are not meant to understand
>> anything about what's in a BLOB.  That's for the layer of your software
>> which packs the information into a blob or unpacks the blob back into
>> information.
>
>
> Understood. I'm writing the code that does the packing and unpacking. The
> users are designing their own databases. When they store for example, an
> array of 16 bit integers, or an array of 64 bit floats, I want to give them
> the same thing back when they query the database not just an array of bytes
> that they have to decode themselves.
>
> Since I have control over how the blobs are generated, I can easily add a
> type identifier to each blob so that I know how to unblob it.
>
> -Bill
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread Ketil Froyn
I'm not sure I understand why you can't specify the type in a separate
column, rather than prepending it to the blob? That seems like a more
flexible way to have access to the information.

Regards, Ketil
On 17 Apr 2016 2:02 p.m., "William Drago"  wrote:

> All,
>
> Any thoughts on using the first byte of a BLOB to indicate what kind of
> data that BLOB contains?
>
> For example:
> 0 = plain old bytes
> 1 = 16 bit integers
> 2 = 32 bit integers
> 3 = singles
> 4 = doubles, etc.
>
> I am forced to consider this approach because my function will be
> receiving blobs but will have no idea what those blobs are. However, I can
> ask the sender to prepend the blob with an indicator byte to provide the
> necessary information.
>
> In the past I have used comments in the table structure and even used
> custom types (e.g. i16BLOB, for a blob that contains 16 bit ints), but in
> this case I will not have access to that information. The data is coming to
> me in the form of an ADO.NET DataTable which does not contain such
> information.
>
> Of course tagging the blobs like this will increase the size of the
> database, but not by much, and if this "feature" is not clearly documented
> someone in the future will have a very hard time figuring out why the blobs
> don't make sense.
>
> Is there anything else I should be aware of?
>
> Thanks,
> Bill
>
>
>
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>


[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread Simon Slavin

On 17 Apr 2016, at 1:02pm, William Drago  wrote:

> I am forced to consider this approach because my function will be receiving 
> blobs but will have no idea what those blobs are.

Why do you care ?  A BLOB is a BLOB.  It doesn't matter whether it's an 
integer, a string, a car, a peace treaty or the smell of newly cut grass.  You 
store it when told to and retrieve it when told to.

> if this "feature" is not clearly documented someone in the future will have a 
> very hard time figuring out why the blobs don't make sense.

BLOBs don't make sense.  They're opaque.  You are not meant to understand 
anything about what's in a BLOB.  That's for the layer of your software which 
packs the information into a blob or unpacks the blob back into information.

If you actually want to store, for example, integers, you shouldn't be storing 
a BLOB in the first place.

Simon.


[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread William Drago
On 4/17/2016 10:13 AM, Brad Stiles wrote:
> An ADO.NET DataTable *can* have type information for table columns, if you 
> add a parameter to the WriteXml method to have it write the schema along with 
> the data. Obviously, *their* data must be typed from their source, or be 
> inferred, for this to work.

If the DataTable is the result of a SELECT that involves 
more than one table is the schema information included for 
each of the tables?

> See if your provider can do this for you. If they can, you should have little 
> problem reading it, assuming you're not using .Net yourself.

I'm confused by your statement. I am using .NET, that's why 
I'm dealing with ADO.NET DataTables.

Thanks,
-Bill


[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread William Drago
On 4/17/2016 9:45 AM, Simon Slavin wrote:
> On 17 Apr 2016, at 1:02pm, William Drago  wrote:
>
>> I am forced to consider this approach because my function will be receiving 
>> blobs but will have no idea what those blobs are.
> Why do you care ?  A BLOB is a BLOB.  It doesn't matter whether it's an 
> integer, a string, a car, a peace treaty or the smell of newly cut grass.  
> You store it when told to and retrieve it when told to.
My job is to find all the blob columns and unblob them into 
arrays of whatever type they originally were before they 
were blob'd.
>> if this "feature" is not clearly documented someone in the future will have 
>> a very hard time figuring out why the blobs don't make sense.
> BLOBs don't make sense.  They're opaque.  You are not meant to understand 
> anything about what's in a BLOB.  That's for the layer of your software which 
> packs the information into a blob or unpacks the blob back into information.
Understood. I'm writing the code that does the packing and 
unpacking. The users are designing their own databases. When 
they store for example, an array of 16 bit integers, or an 
array of 64 bit floats, I want to give them the same thing 
back when they query the database not just an array of bytes 
that they have to decode themselves.

Since I have control over how the blobs are generated, I can 
easily add a type identifier to each blob so that I know how 
to unblob it.

-Bill



[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread William Drago


On 4/17/2016 9:20 AM, Ketil Froyn wrote:
> I'm not sure I understand why you can't specify the type in a separate
> column, rather than prepending it to the blob? That seems like a more
> flexible way to have access to the information.
I have no control over the database schema, table structure, 
etc.. I also have no control of the select statement that is 
generating that data.

-Bill


[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread William Drago
On 4/17/2016 9:45 AM, Simon Slavin wrote:
> On 17 Apr 2016, at 1:02pm, William Drago  wrote:
>
>> I am forced to consider this approach because my function will be receiving 
>> blobs but will have no idea what those blobs are.
> Why do you care ?  A BLOB is a BLOB.  It doesn't matter whether it's an 
> integer, a string, a car, a peace treaty or the smell of newly cut grass.  
> You store it when told to and retrieve it when told to.

My job is to find all the blob columns and unblob them into 
arrays of whatever type they originally were before they 
were blob'd.

>> if this "feature" is not clearly documented someone in the future will have 
>> a very hard time figuring out why the blobs don't make sense.
> BLOBs don't make sense.  They're opaque.  You are not meant to understand 
> anything about what's in a BLOB.  That's for the layer of your software which 
> packs the information into a blob or unpacks the blob back into information.

Understood. I'm writing the code that does the packing and 
unpacking. The users are designing their own databases. When 
they store for example, an array of 16 bit integers, or an 
array of 64 bit floats, I want to give them the same thing 
back when they query the database not just an array of bytes 
that they have to decode themselves.

Since I have control over how the blobs are generated, I can 
easily add a type identifier to each blob so that I know how 
to unblob it.

-Bill


[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread William Drago
On 4/17/2016 9:20 AM, Ketil Froyn wrote:
> I'm not sure I understand why you can't specify the type in a separate
> column, rather than prepending it to the blob? That seems like a more
> flexible way to have access to the information.

I have no control over the database schema, table structure, 
etc.. I also have no control of the select statement that is 
generating that data.

-Bill



[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread Brad Stiles
An ADO.NET DataTable *can* have type information for table columns, if you add 
a parameter to the WriteXml method to have it write the schema along with the 
data. Obviously, *their* data must be typed from their source, or be inferred, 
for this to work. 

See if your provider can do this for you. If they can, you should have little 
problem reading it, assuming you're not using .Net yourself. 

> On Apr 17, 2016, at 07:02, William Drago  wrote:
> 
> All,
> 
> Any thoughts on using the first byte of a BLOB to indicate what kind of data 
> that BLOB contains?
> 
> For example:
> 0 = plain old bytes
> 1 = 16 bit integers
> 2 = 32 bit integers
> 3 = singles
> 4 = doubles, etc.
> 
> I am forced to consider this approach because my function will be receiving 
> blobs but will have no idea what those blobs are. However, I can ask the 
> sender to prepend the blob with an indicator byte to provide the necessary 
> information.
> 
> In the past I have used comments in the table structure and even used custom 
> types (e.g. i16BLOB, for a blob that contains 16 bit ints), but in this case 
> I will not have access to that information. The data is coming to me in the 
> form of an ADO.NET DataTable which does not contain such information.
> 
> Of course tagging the blobs like this will increase the size of the database, 
> but not by much, and if this "feature" is not clearly documented someone in 
> the future will have a very hard time figuring out why the blobs don't make 
> sense.
> 
> Is there anything else I should be aware of?
> 
> Thanks,
> Bill
> 
> 
> 
> 
> 
> 
> ___
> sqlite-users mailing list
> sqlite-users at mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


[sqlite] Pascal (almost) style BLOBs

2016-04-17 Thread William Drago
All,

Any thoughts on using the first byte of a BLOB to indicate 
what kind of data that BLOB contains?

For example:
0 = plain old bytes
1 = 16 bit integers
2 = 32 bit integers
3 = singles
4 = doubles, etc.

I am forced to consider this approach because my function 
will be receiving blobs but will have no idea what those 
blobs are. However, I can ask the sender to prepend the blob 
with an indicator byte to provide the necessary information.

In the past I have used comments in the table structure and 
even used custom types (e.g. i16BLOB, for a blob that 
contains 16 bit ints), but in this case I will not have 
access to that information. The data is coming to me in the 
form of an ADO.NET DataTable which does not contain such 
information.

Of course tagging the blobs like this will increase the size 
of the database, but not by much, and if this "feature" is 
not clearly documented someone in the future will have a 
very hard time figuring out why the blobs don't make sense.

Is there anything else I should be aware of?

Thanks,
Bill