Re: External data storage question

[John Baughman via 4D_Tech]

He David,

> Ouch. That definitely sounds like a bug to me, so I'd say file it wherever
> you're meant to. I think that's on the beta release forum in France.

Yep. I am nearly convinced that his is beta bug. I have already found a 
couple of other bugs and reported them. I will wait a bit to see if I hear back 
from any others on the NUG before I post this as a bug on the beta forum.

> I can't offer any real help because the only time I've used the external
> feature for lots of pictures, we went with letting 4D manage where to put
> the file. Then we backed it up into an explicit and sensible file hierarchy
> by hand. Worked fine. Made me feel safer...and made the files reusable. (I
> know I've mentioned this in the past, but there it is again.) Ask Tony or
> me if you've got questions.

Interestingly, I started this many many years ago, before 4D introduced 
external storage, storing externally with my own code. When 4D implemented the 
external data option, I switched to letting 4D handle it all automatically. 

The problem is that over time the external data folder has grown to 
over 200gb in size. The way that 4D implemented the automatic external storage 
is to include the external storage folder in a 4D backup. This has resulted in 
backups that take over 3.5 hours to complete. Unacceptable, in my opinion.

By switching to custom path to file, the backup now only takes about 2 
minutes. As an added benefit now all the files can be easily found manually in 
the finder on the drive. I have them stored by patient name/# and image series. 
4D’s automatic implementation of the storage results in an organization of the 
files in such a way that it is impossible to locate a file in the finder.

These images are also served to an iPad app. With automatic storage I 
have to load the record and save the picture out to a file that I can send to 
the iPad. I am looking at being able to serve the pictures to the iPad directly 
from the storage folder. In theory that should be noticeably faster.

> I'm on the mainland right now but not coming back through Hawaii. Are you
> at home or on the mainland?

Just got back last week.

John

John Baughman
Kailua, Hawaii
(808) 262-0328
john...@hawaii.rr.com





**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: External data storage question

[John Baughman via 4D_Tech]

> On Aug 3, 2017, at 1:45 PM, Jody Bevan via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> With what you have done now, how does 4D know the path of the document 
> (picture) that you saved.

Jody,

With custom pathname you use the command SET EXTERNAL DATA PATH and the 
path is saved with each record. In the table trigger for new and existing 
records…

CREATE FOLDER($imagePath;*)  //$image path is the full path to the picture file 
inside the external storage folder
WRITE PICTURE FILE($imagePath;[Patient Record]Image)  //saves the current 
picture in the field to the storage folder

$currentImagePath:=Get external data path([Patient Record]Image)

If ($currentImagePath#$imagePath)  //If this is a new record currentImagePath 
will be an empty string. This also handles the upgrade to the new custom 
storage when I deploy.
SET EXTERNAL DATA PATH([Patient Record]Image;$imagePath)

End if 

John


John Baughman
Kailua, Hawaii
(808) 262-0328
john...@hawaii.rr.com





**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: External data storage question

[David Adams via 4D_Tech]

John,

Ouch. That definitely sounds like a bug to me, so I'd say file it wherever
you're meant to. I think that's on the beta release forum in France.

I can't offer any real help because the only time I've used the external
feature for lots of pictures, we went with letting 4D manage where to put
the file. Then we backed it up into an explicit and sensible file hierarchy
by hand. Worked fine. Made me feel safer...and made the files reusable. (I
know I've mentioned this in the past, but there it is again.) Ask Tony or
me if you've got questions.

I'm on the mainland right now but not coming back through Hawaii. Are you
at home or on the mainland?
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: External data storage question

[Jody Bevan via 4D_Tech]

John:

We did this big time long before 4D permitted it autosaving the ‘picture’ 
outside of the data file that they looked after. Therefore we had to look after 
the saving, and when loading up the form we had to look after retrieving the 
‘picture’ as well. It sounds like you have done almost all the work an it 
should not take much code for you to be loading the pictures as well. With what 
you have done now, how does 4D know the path of the document (picture) that you 
saved. Is 4D caching the picture in memory so that if you reload it right away 
it uses what is cached instead of going and getting the file again? You can 
test this by shutting down your app in between.

Jody


Jody Bevan
ARGUS Productions Inc.
Developer

Argus Productions Inc. 




> On Aug 3, 2017, at 5:21 PM, John Baughman via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> I recently changed storing a picture field outside the database from 
> automatic location to custom path name to file. Still working on getting it 
> ready for deployment.
> 
> I just noticed that if I open a form with the picture field displayed, then 
> change the picture to a different image, save it, then reopen the record, the 
> picture field continues to display the old picture. The picture file in the 
> storage folder reflects the new picture. I can actually watch it change as 
> the record is saved. The bad thing is if I save the record again the old 
> picture gets saved to the storage folder and the new picture is lost. If I 
> restart the database, without having reentered and saved again, the new 
> picture displays properly.
> 
> I know that I have to manage the creation and modification of the picture on 
> disk which I am doing, but I do not see anything that says I have to load the 
> picture from the storage folder. Am I missing something?
> 
> I also recently started working this db in v16 R4 beta so this may be a bug 
> in the beta? I do not really have an easy way to test this is an earlier 
> version of 4D.
> 
> John
> 
> 
> John Baughman
> Kailua, Hawaii
> (808) 262-0328
> john...@hawaii.rr.com

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

v13 - wierdness with drag n drop

[Chip Scheide via 4D_Tech]

I have a database I am using to test a component. Both component and 
database are interpreted.
it has 2 tables, 1table with 1 field, and the other table with 3 fields.

It has 2 project methods for testing the component methods.


In this (testing) database, I open the component methods exposed in the 
explorer.
I select all, and drag them to one of the test methods. The selection 
of component methods drop into the test method - but they are all in 
inverse alphabetic order?!?!?

any Idea why?
THanks

---
Gas is for washing parts
Alcohol is for drinkin'
Nitromethane is for racing 
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Web Site Security - Observatory by Moziila

[Jim Hays via 4D_Tech]

We have been working on making our web product more secure, and customer's
security testing was giving us a hard time.

I recently found this:  https://observatory.mozilla.org/

It will test your site, and offer step by step advice on making it more
secure.
It's easily the most understandable information I've seen on web security -
especially for those of us that don't spend all our time working on web
apps (yet!).
It took a couple of days to get through it all, but now we pass all the
tests, and I know enough to be dangerous.

I'm curious if anyone has used Mozilla Observatory, if there are any
opinions on it's quality, and if there are other recommendations.

Our setup is:
4D v12.6
Active4D
Apache, with the SSL cert, as reverse proxy in a DMZ

 - Jim Hays
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Blob to Record

[John DeSoi via 4D_Tech]

It would be very handy for a lot of different reasons. A few months ago, I 
added a feature request for it:

http://forums.4d.com/Post/EN/17757394/1/17757395#17757395

Please vote for it if you think it would be useful.

John DeSoi, Ph.D.




> On Aug 3, 2017, at 8:45 AM, Chip Scheide via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> I was thinking about this, and thought a 'Record to Blob' and 'blob to 
> record' would be useful/handy.

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Encrypted Data at Rest

[Jody Bevan via 4D_Tech]

Richard:

Hardware acceleration in the CPU is still not nearly as fast as this task being 
done by the drive controller. All software runs through the CPU. Yes, the CPU 
can support specific types of code which helps. Just like GPUs,  the advanced 
support in the drive controllers makes a big difference. Tim’s explanation 
really highlights why it is faster - good explanation Tim.  

As with all things in 4D and IT, there are vast differences in use and needs. 
Why buy a big server with 24 GB RAM and 2 Terabytes of SSD in RAID 
configuration, and 18 CPUs when there are 20 people connecting - not needed - 
huge waste of money. On the other hand trying to serve a large site with large 
data, and heavy use with a MacMini running FileVault would be a no go. When 
working with large sites with hundreds of people connected and they do ALL 
their work in the system, good hardware, good network, and OS makes a huge 
difference (as well as proven 4D application). This is when pinches in 
throughput really shows. It makes one dig into the details. Reminds me of 
gigabyte switches. Clients would complain that the switches we insisted on were 
way to expensive when they could buy gigabyte switches for 20% of the price. 
Yes each port may be able to support 1 GB throughput, but if the whole device 
only support 2 GB throughput it really is not good for busy sites. The details 
matter at large sites.

The details of performance really matter if your site’s needs require high 
performance. If not - then FileVault2 can be a great solution that makes a lot 
of sense.

Apple backed out of the server world as far as hardware is concerned, but that 
is a whole other topic for those of us that actively supported Apple Servers 
with several hundred Xserves. With a few more important features we could have 
sold several hundred more.

Jody


> On Aug 2, 2017, at 1:59 PM, Richard Wright via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> Not sure why you say for personal use only. FileVault supports 256 AES and 
> benefits from hardware acceleration in the CPU. I’ve never done timing 
> comparisons but there are no noticeable performance effects at all. Some 
> people have tested, years ago, and found at most only 2-3% degradation.
> 

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

RE: Encrypted Data at Rest

[Bruno LEGAY via 4D_Tech]

Hi,

This is an interesting subject...

I would say it depends on what you want to protect from and how deep are your 
pockets...

As people said, it is difficult to encrypt the whole 4D database without 
encryption being supported at the database engine level...
At the moment 4D does not support it.

It is possible to encrypt few fields, but it will then be difficult to search 
on those crypted fields... Decrypting on the fly is ok for targeted purpose but 
not for massive data processing (search, sort, stats, etc...).

For instance, database with people and medical results. One idea would be to 
encrypt the medical result but not encrypt the name, surname, dob information 
(so you can at least search, if authorized, the result of one person).
If you store a pdf of results with name and medical results they need to be 
encrypted as well.

If you wanted to make stats ont the medical data, then you would have to have 
information in "clear" but encrypt the relation between medical data and 
person...

This is just an idea, I don't know if it will hold against certification...

Filesystem/storage level encryption is good for one part of the risks. if 
machine gets stolen, drive is disposed of without being rerased/destroyed...
If you can get away with this, this will be the easiest/cheapest option.

But if a hacker gets into your server (with the same privileges as the owner of 
the datafile), he will be able to copy you data file (and if he is very good he 
will read/extract your data)...
If a hacker is into your server, it means that your network security was not 
that great, that the server configuration/protection was not that great, etc...

Now what about a dishonest/bad/disgruntled employee/admin (or a whistler 
blower)... These things can happen, event to the NSA...

Should the information be accessible to the admin ? the developer ? or just the 
users with correct privileges ?

If you do backups (they will leave the encrypted disk, the machine and the 
building hopefully), you need to encrypt the backup data (7z has an option to 
do AES-256 encryption).

Then there is the question of the keys... Where and how do you store and 
protect the keys ??? 
Hard coded in code ?, in data ? in preference file ? in a Hardware Security 
Module (HSM) ?
The HSM is a kind of very specialized (and very expensive) hardware designed to 
store and protect keys and self-destroy if tempered with... 

S3 is an option. Communication to S3 is secure (https). 
Storage can be done encrypted with AES-256 (transparently), it is just an 
option at object level. Data is encrypted before being written to disk and 
decrypted on the fly when you try to read it (all transparent). Amazon cannot 
read your data (it is encrypted with your secret key).
I read that NSA was suspected to have planted a "system" (with the help of 
Amazon ?) between the  ssl tunnel exit and before the encryption... But that's 
it.
If you don't trust amazon/nsa, you can encrypt before sending.

Finally, Oracle has a concept of "Transparent Data Encryption" since few years 
(11g).
Maybe if you really need this, store your data on Oracle.

HTH





**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Blob to Record

[Chip Scheide via 4D_Tech]

THANKS

Actually, I'm not trying to do anything with this - at the moment.
I am working on a component for managing text, and a couple of routines 
do records to text and record to blob/text, with the idea (no specific 
implementation intent) of exporting to some other program/database.

I was thinking about this, and thought a 'Record to Blob' and 'blob to 
record' would be useful/handy.

Thanks again
Chip
On Thu, 3 Aug 2017 14:37:46 +0200, Koen Van Hooreweghe via 4D_Tech 
wrote:
> Hi Chip,
> 
> It depends on what you want to try to achieve with that command.
> I’m using the code below to copy a complete 4D record binary into a 
> blob. But the only thing you normally can do with the resulting blob 
> is to put it back into a record.
> AFAIK this is also the case with the API Pack (and OTools?) commands.
> 
> But in your pseudo code I noticed you add a 'convert to text' 
> command. So the resulting blob would contain a textual representation 
> of the record. But that would fail with binary fields like pictures, 
> objects and blobs. 
> 
> This is my RecordToBlob method, which uses an intermediate file in 
> the temp directory:
> //usage: $blob:=RecordToBlob(->[myTable]myField)
> C_POINTER($1;$tablePtr)
> C_BLOB($0)
> C_TEXT($path)
> 
> $tablePtr:=$1
> 
> If (Records in selection($tablePtr->)#0)
> $path:=Temporary folder +"temp_rec_to_blob"+String(Current process)
> SET CHANNEL(10;$path)
> SEND RECORD($tablePtr->)
> SET CHANNEL(11)
> 
> DOCUMENT TO BLOB($path;$0)
> Else 
> TRACE
> End if 
> 
> 
> And its BlobToRecord counterpart:
> //usage: RecordToBlob($blob;->[myTable]myField)
> C_BLOB($1;$blob)
> C_POINTER($2;$tablePtr)
> 
> $blob:=$1
> $tablePtr:=$2
> 
> $path:=4D_temp_path +"temp_rec_to_blob"+String(Current process)
> 
> BLOB TO DOCUMENT($path;$blob)
> 
> SET CHANNEL(10;$path)
> RECEIVE RECORD($tablePtr->)
> SET CHANNEL(11)
> SAVE RECORD($tablePtr->)
> 
> 
> Kind regards,
> Koen
> 
>> Op 2 aug. 2017, om 20:20 heeft Chip Scheide via 4D_Tech 
>> <4d_tech@lists.4d.com> het volgende geschreven:
>> 
>> is there simple way to move an entire (arbitrary) record to a blob?
>> 
>> I was hoping for a simple command like:
>> Record to blob([table];Blob)
> 
> 
> 
> 
> Compass bvba
> Koen Van Hooreweghe
> Kloosterstraat 65
> 9910 Knesselare
> Belgium
> tel +32 495 511.653
> 
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
---
Gas is for washing parts
Alcohol is for drinkin'
Nitromethane is for racing 
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Blob to Record

[Koen Van Hooreweghe via 4D_Tech]

Hi Chip,

It depends on what you want to try to achieve with that command.
I’m using the code below to copy a complete 4D record binary into a blob. But 
the only thing you normally can do with the resulting blob is to put it back 
into a record.
AFAIK this is also the case with the API Pack (and OTools?) commands.

But in your pseudo code I noticed you add a 'convert to text' command. So the 
resulting blob would contain a textual representation of the record. But that 
would fail with binary fields like pictures, objects and blobs. 

This is my RecordToBlob method, which uses an intermediate file in the temp 
directory:
//usage: $blob:=RecordToBlob(->[myTable]myField)
C_POINTER($1;$tablePtr)
C_BLOB($0)
C_TEXT($path)

$tablePtr:=$1

If (Records in selection($tablePtr->)#0)
$path:=Temporary folder +"temp_rec_to_blob"+String(Current process)
SET CHANNEL(10;$path)
SEND RECORD($tablePtr->)
SET CHANNEL(11)

DOCUMENT TO BLOB($path;$0)
Else 
TRACE
End if 


And its BlobToRecord counterpart:
//usage: RecordToBlob($blob;->[myTable]myField)
C_BLOB($1;$blob)
C_POINTER($2;$tablePtr)

$blob:=$1
$tablePtr:=$2

$path:=4D_temp_path +"temp_rec_to_blob"+String(Current process)

BLOB TO DOCUMENT($path;$blob)

SET CHANNEL(10;$path)
RECEIVE RECORD($tablePtr->)
SET CHANNEL(11)
SAVE RECORD($tablePtr->)


Kind regards,
Koen

> Op 2 aug. 2017, om 20:20 heeft Chip Scheide via 4D_Tech 
> <4d_tech@lists.4d.com> het volgende geschreven:
> 
> is there simple way to move an entire (arbitrary) record to a blob?
> 
> I was hoping for a simple command like:
> Record to blob([table];Blob)




Compass bvba
Koen Van Hooreweghe
Kloosterstraat 65
9910 Knesselare
Belgium
tel +32 495 511.653

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

JSON/objects and two-dimensional arrays

[Epperlein, Lutz (agendo) via 4D_Tech]

> JSON Validate, Collections and a NULL command!
> 
> http://blog.4d.com/en-whats-new-in-4d-v16-r4/

Nice improvements.
Unfortunately they didn't remove the limitation of the OB SET ARRAY command 
regarding the support of two-dimensional arrays. At least the docs say:

> Note: It is not possible to use two-dimensional arrays.

Because of that I posted a new feature request on the forum. If you like, 
please vote for it:
http://forums.4d.com/Post//19953344/1/

Thanks
Lutz Epperlein
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Blob to Record

[Jörg Knebel via 4D_Tech]

Hi Chip,

> On 3 Aug 2017, at 04:20 AEST, Chip Scheide via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> is there simple way to move an entire (arbitrary) record to a blob?

Maybe you should take a look at ObjectTools commands  OT PutRecord / OT 
PutRecord / OT GetRecordTable.

Cheers
Jörg
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**