Re: [Dev] [IOT]Concerns regarding saving the image and binary files in the File System

[Chathura Ekanayake]

As Charitha and Waruna mentioned, we will have to implement the storage
method as an extension, where it is possible to switch among local file
system (with rsync), FTP, S3, etc based on deployment requirements.

@ChathuraD: We will not be exposing the actual storage to outside and all
accesses will be through IoT server APIs. In that case, do we need to
enforce security by having random file names in the storage? Regarding
possible optimization in image access, we can have
1) caching for frequently accessed images (which is independent of the
underlying storage)
2) separate fast access storage for images
Only if the latter is practical in real expected deployments, I think
option 2 mentioned by Megala is useful. Otherwise I prefer option 1.

Regards,
Chathura

On Thu, Aug 17, 2017 at 7:35 AM, Chathura Dilan  wrote:

> I'm +1 with option 2 since we can have a big storage for binary files and
> small storage for image files. The method of serving images and binary
> files are for two different things. We can optimize image storage for
> serving images because images are accessed frequently. Also in terms of
> security storing binary files in a separate location provides additional
> security for them.
>
> On Wed, Aug 16, 2017 at 11:21 PM, Megala Uthayakumar 
> wrote:
>
>> Hi All,
>>
>> Please fine my responses below,
>>
>> *@Nuwan *
>> Is there a problem is storing this file on the DB instead of the FS?
>>
>> As explained by Chathura and Charitha main reasoning behind not selecting
>> the database is size of the binary file and yes we will consider other
>> possible ways for supporting this as well, given that NFS may not be
>> preferred by the customer.
>>
>> *@Chathura*
>> If you go with the first approach, could you please mention what are the
>> parameters you use to get the image? It is always better to get the image
>> with a random parameters so it prevents hackers from guessing image names.
>>
>> As we know already we will be getting the files using an REST API and in
>> that case, the URL will be something like follow,
>>*https://localhost:8243/applications/getIcon/
>> *
>> So with this URL, nothing can be guessed as in the API level, what we
>> expect to do is something like follow,
>>
>>1. From the UUID, get the auto-generated ID from the database
>>2. We will derive the path for the particular application as follow,
>>   - /ID
>>3. In that path we will search for the file with the name "icon" and
>>we will pass File with the response.
>>
>> In this case, I don't think there is possible security issue where we
>> expose the path name or so. Please do correct me, if it is not the case.By
>> having random names, we need to have additional database operations for
>> saving icon/other image files name and retrieving them.
>>
>> *@Charitha*
>> I'm not clear with the reason behind this statement in under option 2
>>
>> "but this may not be possible in real production scenarios depending on
>> the size of the images.
>>
>> I meant that caching images will not work as in the real-production
>> scenarios, we may have images with higher sizes, so in that case, we cannot
>> efficiently cache and gain performance gain out of them by using option 2.
>>
>> *@Waruna *
>> Storage Location - This should be written as extension where you can
>> store in local directory, FTP, NFS or someone can customize based on
>> requirement.
>>
>> For the storage location, if we are using NFS or local file system, I
>> think om the implementation level it will be the same and we do not need to
>> specially handle them. We have not thought about supporting FTP for the
>> first cut implementation and wondering whether we need that level
>> complexity (i.e. Having a server-server communication) for only saving
>> image files and binary files.
>>
>> Do we need real requirement to search by image name,apk name or other
>> content name. If we have that kind of requiremnt as well then option 1 will
>> have some cons.
>>
>> AFAIK we will not get a requirement to search the apis with the apk name
>> or image name as people are normally interested in searching by using
>> application name, platform name etc.
>>
>> Thanks.
>>
>> Regards,
>> Megala
>>
>> On Wed, Aug 16, 2017 at 6:31 PM, Waruna Jayaweera 
>> wrote:
>>
>>> Hi Megala,
>>>
>>> AFAIU There are two problems as where we store the file
>>> contents(supporting clustering cases) and file structure of application
>>> content.
>>>
>>> Storage Location - This should be written as extension where you can
>>> store in local directory, FTP, NFS or someone can customize based on
>>> requirement.
>>> Application store structure - I prefer option 1 as it will be more
>>> manageable per application which makes separate storage space for
>>> application contents(apk,icon etc).
>>>
>>> As an example we can store like //>> plication_content_name>/content_data
>>> 

Re: [Dev] [IOT]Concerns regarding saving the image and binary files in the File System

[Chathura Dilan]

I'm +1 with option 2 since we can have a big storage for binary files and
small storage for image files. The method of serving images and binary
files are for two different things. We can optimize image storage for
serving images because images are accessed frequently. Also in terms of
security storing binary files in a separate location provides additional
security for them.

On Wed, Aug 16, 2017 at 11:21 PM, Megala Uthayakumar 
wrote:

> Hi All,
>
> Please fine my responses below,
>
> *@Nuwan *
> Is there a problem is storing this file on the DB instead of the FS?
>
> As explained by Chathura and Charitha main reasoning behind not selecting
> the database is size of the binary file and yes we will consider other
> possible ways for supporting this as well, given that NFS may not be
> preferred by the customer.
>
> *@Chathura*
> If you go with the first approach, could you please mention what are the
> parameters you use to get the image? It is always better to get the image
> with a random parameters so it prevents hackers from guessing image names.
>
> As we know already we will be getting the files using an REST API and in
> that case, the URL will be something like follow,
>*https://localhost:8243/applications/getIcon/
> *
> So with this URL, nothing can be guessed as in the API level, what we
> expect to do is something like follow,
>
>1. From the UUID, get the auto-generated ID from the database
>2. We will derive the path for the particular application as follow,
>   - /ID
>3. In that path we will search for the file with the name "icon" and
>we will pass File with the response.
>
> In this case, I don't think there is possible security issue where we
> expose the path name or so. Please do correct me, if it is not the case.By
> having random names, we need to have additional database operations for
> saving icon/other image files name and retrieving them.
>
> *@Charitha*
> I'm not clear with the reason behind this statement in under option 2
>
> "but this may not be possible in real production scenarios depending on
> the size of the images.
>
> I meant that caching images will not work as in the real-production
> scenarios, we may have images with higher sizes, so in that case, we cannot
> efficiently cache and gain performance gain out of them by using option 2.
>
> *@Waruna *
> Storage Location - This should be written as extension where you can store
> in local directory, FTP, NFS or someone can customize based on requirement.
>
> For the storage location, if we are using NFS or local file system, I
> think om the implementation level it will be the same and we do not need to
> specially handle them. We have not thought about supporting FTP for the
> first cut implementation and wondering whether we need that level
> complexity (i.e. Having a server-server communication) for only saving
> image files and binary files.
>
> Do we need real requirement to search by image name,apk name or other
> content name. If we have that kind of requiremnt as well then option 1 will
> have some cons.
>
> AFAIK we will not get a requirement to search the apis with the apk name
> or image name as people are normally interested in searching by using
> application name, platform name etc.
>
> Thanks.
>
> Regards,
> Megala
>
> On Wed, Aug 16, 2017 at 6:31 PM, Waruna Jayaweera 
> wrote:
>
>> Hi Megala,
>>
>> AFAIU There are two problems as where we store the file
>> contents(supporting clustering cases) and file structure of application
>> content.
>>
>> Storage Location - This should be written as extension where you can
>> store in local directory, FTP, NFS or someone can customize based on
>> requirement.
>> Application store structure - I prefer option 1 as it will be more
>> manageable per application which makes separate storage space for
>> application contents(apk,icon etc).
>>
>> As an example we can store like //> plication_content_name>/content_data
>> I option 2 you will have less folders but you may need to concat file
>> names for support for multiple application versions and their contents. In
>> option 1 delete application operation also easy.
>>
>> Do we need real requirement to search by image name,apk name or other
>> content name. If we have that kind of requiremnt as well then option 1 will
>> have some cons.
>>
>> Thanks,
>> Waruna
>>
>> On Wed, Aug 16, 2017 at 6:02 PM, Charitha Goonetilleke <
>> charit...@wso2.com> wrote:
>>
>>> Hi Nuwan,
>>>
>>> As our previous experience with similar approach we have used to store
>>> binaries in DB, it cause for scallability issues in later when growing the
>>> size of the DB in GBs. So storing files as binaries in DB is highly
>>> depending on the DB engine used and the growth rate of the data. IMO, these
>>> concerns and issues might be critical and complex than the complexity we
>>> are going to introduce with FS based solution. So I think 

Re: [Dev] [IOT]Concerns regarding saving the image and binary files in the File System

[Megala Uthayakumar]

Hi All,

Please fine my responses below,

*@Nuwan *
Is there a problem is storing this file on the DB instead of the FS?

As explained by Chathura and Charitha main reasoning behind not selecting
the database is size of the binary file and yes we will consider other
possible ways for supporting this as well, given that NFS may not be
preferred by the customer.

*@Chathura*
If you go with the first approach, could you please mention what are the
parameters you use to get the image? It is always better to get the image
with a random parameters so it prevents hackers from guessing image names.

As we know already we will be getting the files using an REST API and in
that case, the URL will be something like follow,
   *https://localhost:8243/applications/getIcon/
*
So with this URL, nothing can be guessed as in the API level, what we
expect to do is something like follow,

   1. From the UUID, get the auto-generated ID from the database
   2. We will derive the path for the particular application as follow,
  - /ID
   3. In that path we will search for the file with the name "icon" and we
   will pass File with the response.

In this case, I don't think there is possible security issue where we
expose the path name or so. Please do correct me, if it is not the case.By
having random names, we need to have additional database operations for
saving icon/other image files name and retrieving them.

*@Charitha*
I'm not clear with the reason behind this statement in under option 2

"but this may not be possible in real production scenarios depending on the
size of the images.

I meant that caching images will not work as in the real-production
scenarios, we may have images with higher sizes, so in that case, we cannot
efficiently cache and gain performance gain out of them by using option 2.

*@Waruna *
Storage Location - This should be written as extension where you can store
in local directory, FTP, NFS or someone can customize based on requirement.

For the storage location, if we are using NFS or local file system, I think
om the implementation level it will be the same and we do not need to
specially handle them. We have not thought about supporting FTP for the
first cut implementation and wondering whether we need that level
complexity (i.e. Having a server-server communication) for only saving
image files and binary files.

Do we need real requirement to search by image name,apk name or other
content name. If we have that kind of requiremnt as well then option 1 will
have some cons.

AFAIK we will not get a requirement to search the apis with the apk name or
image name as people are normally interested in searching by using
application name, platform name etc.

Thanks.

Regards,
Megala

On Wed, Aug 16, 2017 at 6:31 PM, Waruna Jayaweera  wrote:

> Hi Megala,
>
> AFAIU There are two problems as where we store the file
> contents(supporting clustering cases) and file structure of application
> content.
>
> Storage Location - This should be written as extension where you can store
> in local directory, FTP, NFS or someone can customize based on requirement.
> Application store structure - I prefer option 1 as it will be more
> manageable per application which makes separate storage space for
> application contents(apk,icon etc).
>
> As an example we can store like //<
> application_content_name>/content_data
> I option 2 you will have less folders but you may need to concat file
> names for support for multiple application versions and their contents. In
> option 1 delete application operation also easy.
>
> Do we need real requirement to search by image name,apk name or other
> content name. If we have that kind of requiremnt as well then option 1 will
> have some cons.
>
> Thanks,
> Waruna
>
> On Wed, Aug 16, 2017 at 6:02 PM, Charitha Goonetilleke  > wrote:
>
>> Hi Nuwan,
>>
>> As our previous experience with similar approach we have used to store
>> binaries in DB, it cause for scallability issues in later when growing the
>> size of the DB in GBs. So storing files as binaries in DB is highly
>> depending on the DB engine used and the growth rate of the data. IMO, these
>> concerns and issues might be critical and complex than the complexity we
>> are going to introduce with FS based solution. So I think it is best to
>> store files in FS rather than adding it to DB.
>>
>> Anyway as you have raised, there might be situations where, NFS
>> integration is not possible. So I'm +1 for having few alternative ways to
>> handle this as well. In addition to the RSync, we can also use shared
>> volume or even S3 bucket if some one is going to deploy this in AWS.
>>
>> @Megala, I'm not clear with the reason behind this statement in under
>> option 2
>>
>> "but this may not be possible in real production scenarios depending on
>> the size of the images."
>>
>>
>> On Wed, Aug 16, 2017 at 5:46 PM, Nuwan Dias  

Re: [Dev] [IOT]Concerns regarding saving the image and binary files in the File System

[Waruna Jayaweera]

Hi Megala,

AFAIU There are two problems as where we store the file contents(supporting
clustering cases) and file structure of application content.

Storage Location - This should be written as extension where you can store
in local directory, FTP, NFS or someone can customize based on requirement.
Application store structure - I prefer option 1 as it will be more
manageable per application which makes separate storage space for
application contents(apk,icon etc).

As an example we can store like
///content_data
I option 2 you will have less folders but you may need to concat file names
for support for multiple application versions and their contents. In option
1 delete application operation also easy.

Do we need real requirement to search by image name,apk name or other
content name. If we have that kind of requiremnt as well then option 1 will
have some cons.

Thanks,
Waruna

On Wed, Aug 16, 2017 at 6:02 PM, Charitha Goonetilleke 
wrote:

> Hi Nuwan,
>
> As our previous experience with similar approach we have used to store
> binaries in DB, it cause for scallability issues in later when growing the
> size of the DB in GBs. So storing files as binaries in DB is highly
> depending on the DB engine used and the growth rate of the data. IMO, these
> concerns and issues might be critical and complex than the complexity we
> are going to introduce with FS based solution. So I think it is best to
> store files in FS rather than adding it to DB.
>
> Anyway as you have raised, there might be situations where, NFS
> integration is not possible. So I'm +1 for having few alternative ways to
> handle this as well. In addition to the RSync, we can also use shared
> volume or even S3 bucket if some one is going to deploy this in AWS.
>
> @Megala, I'm not clear with the reason behind this statement in under
> option 2
>
> "but this may not be possible in real production scenarios depending on
> the size of the images."
>
>
> On Wed, Aug 16, 2017 at 5:46 PM, Nuwan Dias  wrote:
>
>> If you require NFS anyway (for other usecases), then I think its fine.
>> But if you're introducing the complexity of NFS just for this use case,
>> then I think it would be an overkill.
>>
>> BTW, not every one is ok with setting up a NFS. So if you need to share
>> files between nodes you may want to have some other recommendations handy
>> as well (RSync).
>>
>> Is there a problem is storing this file on the DB instead of the FS?
>>
>> On Wed, Aug 16, 2017 at 5:41 PM, Megala Uthayakumar 
>> wrote:
>>
>>> Hi Nuwan,
>>>
>>> As mentioned by Harshan user needs to configure NFS when configuring
>>> this in HA setup. We do accept local file path or NFS url in the
>>> configuration file for the path.
>>>
>>> Thanks.
>>>
>>> On Wed, Aug 16, 2017 at 5:01 PM, Nuwan Dias  wrote:
>>>
 When you have two nodes of the IoT server running for HA, and if a
 given image/file is stored on the file system of one server, how do you
 make that image/file available for a request that's served via the other
 server?

 On Wed, Aug 16, 2017 at 4:54 PM, Megala Uthayakumar 
 wrote:

> Hi All,
>
> Currently we are working on writing a mobile application store for IOT
> server. For that we have decided to save the images and binary files
> related with applications in the file system(This decision was taken
> considering the file size). File location will be derived from a
> configuration. While implementing this, we have come-up with the following
> suggestions for saving files as per the meeting we had internally.
>
> *Option 1*
> For each application we will have a folder with the name as the
> generated ID for the application from the database. Each will hold all the
> relevant artifacts such as icon, screen-shots and binary files.
> In this case, there were 2 suggestions
>
>1. For each app, the icon and screen-shots will be saved with the
>same name. Ex - For icons the image file name will be 'icon' and 
> like-wise
>for others as well. So in this case, we do not need to save image 
> names in
>the database.
>2. Need to dynamically generate some random names.
>
> If we consider first approach, if we use the same name for all the
> application, there is possible chance an attacker may get all the image
> files if the name is known to them.  AFAIU this can happen even if we do
> not use the 1st approach and use the second approach in which we use 
> random
> names, as this can happen only if the root path for saving the artifacts
> are compromised and without the name also attacker can do a "listFiles"
> request  and get all the data.
>
> *Option 2*
> For binary files and image files we will have separate location and
> each respective files will be saved in these location separately by

Re: [Dev] [IOT]Concerns regarding saving the image and binary files in the File System

[Charitha Goonetilleke]

Hi Nuwan,

As our previous experience with similar approach we have used to store
binaries in DB, it cause for scallability issues in later when growing the
size of the DB in GBs. So storing files as binaries in DB is highly
depending on the DB engine used and the growth rate of the data. IMO, these
concerns and issues might be critical and complex than the complexity we
are going to introduce with FS based solution. So I think it is best to
store files in FS rather than adding it to DB.

Anyway as you have raised, there might be situations where, NFS integration
is not possible. So I'm +1 for having few alternative ways to handle this
as well. In addition to the RSync, we can also use shared volume or even S3
bucket if some one is going to deploy this in AWS.

@Megala, I'm not clear with the reason behind this statement in under
option 2

"but this may not be possible in real production scenarios depending on the
size of the images."


On Wed, Aug 16, 2017 at 5:46 PM, Nuwan Dias  wrote:

> If you require NFS anyway (for other usecases), then I think its fine. But
> if you're introducing the complexity of NFS just for this use case, then I
> think it would be an overkill.
>
> BTW, not every one is ok with setting up a NFS. So if you need to share
> files between nodes you may want to have some other recommendations handy
> as well (RSync).
>
> Is there a problem is storing this file on the DB instead of the FS?
>
> On Wed, Aug 16, 2017 at 5:41 PM, Megala Uthayakumar 
> wrote:
>
>> Hi Nuwan,
>>
>> As mentioned by Harshan user needs to configure NFS when configuring this
>> in HA setup. We do accept local file path or NFS url in the configuration
>> file for the path.
>>
>> Thanks.
>>
>> On Wed, Aug 16, 2017 at 5:01 PM, Nuwan Dias  wrote:
>>
>>> When you have two nodes of the IoT server running for HA, and if a given
>>> image/file is stored on the file system of one server, how do you make that
>>> image/file available for a request that's served via the other server?
>>>
>>> On Wed, Aug 16, 2017 at 4:54 PM, Megala Uthayakumar 
>>> wrote:
>>>
 Hi All,

 Currently we are working on writing a mobile application store for IOT
 server. For that we have decided to save the images and binary files
 related with applications in the file system(This decision was taken
 considering the file size). File location will be derived from a
 configuration. While implementing this, we have come-up with the following
 suggestions for saving files as per the meeting we had internally.

 *Option 1*
 For each application we will have a folder with the name as the
 generated ID for the application from the database. Each will hold all the
 relevant artifacts such as icon, screen-shots and binary files.
 In this case, there were 2 suggestions

1. For each app, the icon and screen-shots will be saved with the
same name. Ex - For icons the image file name will be 'icon' and 
 like-wise
for others as well. So in this case, we do not need to save image names 
 in
the database.
2. Need to dynamically generate some random names.

 If we consider first approach, if we use the same name for all the
 application, there is possible chance an attacker may get all the image
 files if the name is known to them.  AFAIU this can happen even if we do
 not use the 1st approach and use the second approach in which we use random
 names, as this can happen only if the root path for saving the artifacts
 are compromised and without the name also attacker can do a "listFiles"
 request  and get all the data.

 *Option 2*
 For binary files and image files we will have separate location and
 each respective files will be saved in these location separately by
 generating a unique name. This may be helpful if we can cache the images
 and improve the performance, but this may not be possible in real
 production scenarios depending on the size of the images.

 What would be the most preferable option? Comments and suggestions on
 this regard is highly appreciated.

 Thanks.

 Regards,
 Megala

 --
 Megala Uthayakumar

 Software Engineer
 Mobile : 0779967122

 --
 You received this message because you are subscribed to the Google
 Groups "WSO2 IoT Team Group" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to iot-group+unsubscr...@wso2.com.
 For more options, visit https://groups.google.com/a/wso2.com/d/optout.

>>>
>>>
>>>
>>> --
>>> Nuwan Dias
>>>
>>> Software Architect - WSO2, Inc. http://wso2.com
>>> email : nuw...@wso2.com
>>> Phone : +94 777 775 729 <+94%2077%20777%205729>
>>>
>>
>>
>>
>> --
>> Megala Uthayakumar
>>
>> Software Engineer
>> Mobile : 0779967122
>>
>
>
>
> --
> Nuwan Dias
>
> 

Re: [Dev] [IOT]Concerns regarding saving the image and binary files in the File System

[Chathura Dilan]

Hi Megala,

If you go with the first approach, could you please mention what are the
parameters you use to get the image? It is always better to get the image
with a random parameters so it prevents hackers from guessing image names.

@Nuwan. Some binary files could be large as 500MB in size. So we cannot
store them in the  database. Also rsync consumes lot of bandwidth, but you
can go with that approach based on the scenario. But IMO NFS with high
availability would be best approach.

On Wed, Aug 16, 2017 at 5:41 PM, Megala Uthayakumar  wrote:

> Hi Nuwan,
>
> As mentioned by Harshan user needs to configure NFS when configuring this
> in HA setup. We do accept local file path or NFS url in the configuration
> file for the path.
>
> Thanks.
>
> On Wed, Aug 16, 2017 at 5:01 PM, Nuwan Dias  wrote:
>
>> When you have two nodes of the IoT server running for HA, and if a given
>> image/file is stored on the file system of one server, how do you make that
>> image/file available for a request that's served via the other server?
>>
>> On Wed, Aug 16, 2017 at 4:54 PM, Megala Uthayakumar 
>> wrote:
>>
>>> Hi All,
>>>
>>> Currently we are working on writing a mobile application store for IOT
>>> server. For that we have decided to save the images and binary files
>>> related with applications in the file system(This decision was taken
>>> considering the file size). File location will be derived from a
>>> configuration. While implementing this, we have come-up with the following
>>> suggestions for saving files as per the meeting we had internally.
>>>
>>> *Option 1*
>>> For each application we will have a folder with the name as the
>>> generated ID for the application from the database. Each will hold all the
>>> relevant artifacts such as icon, screen-shots and binary files.
>>> In this case, there were 2 suggestions
>>>
>>>1. For each app, the icon and screen-shots will be saved with the
>>>same name. Ex - For icons the image file name will be 'icon' and 
>>> like-wise
>>>for others as well. So in this case, we do not need to save image names 
>>> in
>>>the database.
>>>2. Need to dynamically generate some random names.
>>>
>>> If we consider first approach, if we use the same name for all the
>>> application, there is possible chance an attacker may get all the image
>>> files if the name is known to them.  AFAIU this can happen even if we do
>>> not use the 1st approach and use the second approach in which we use random
>>> names, as this can happen only if the root path for saving the artifacts
>>> are compromised and without the name also attacker can do a "listFiles"
>>> request  and get all the data.
>>>
>>> *Option 2*
>>> For binary files and image files we will have separate location and each
>>> respective files will be saved in these location separately by generating a
>>> unique name. This may be helpful if we can cache the images and improve the
>>> performance, but this may not be possible in real production scenarios
>>> depending on the size of the images.
>>>
>>> What would be the most preferable option? Comments and suggestions on
>>> this regard is highly appreciated.
>>>
>>> Thanks.
>>>
>>> Regards,
>>> Megala
>>>
>>> --
>>> Megala Uthayakumar
>>>
>>> Software Engineer
>>> Mobile : 0779967122
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "WSO2 IoT Team Group" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to iot-group+unsubscr...@wso2.com.
>>> For more options, visit https://groups.google.com/a/wso2.com/d/optout.
>>>
>>
>>
>>
>> --
>> Nuwan Dias
>>
>> Software Architect - WSO2, Inc. http://wso2.com
>> email : nuw...@wso2.com
>> Phone : +94 777 775 729 <+94%2077%20777%205729>
>>
>
>
>
> --
> Megala Uthayakumar
>
> Software Engineer
> Mobile : 0779967122
>



-- 
Thanks and Best Regards,

Chatura Dilan Perera
*Associate Tech Lead** - WSO2 Inc.*
+94777266673
www.dilan.me
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [IOT]Concerns regarding saving the image and binary files in the File System

[Nuwan Dias]

If you require NFS anyway (for other usecases), then I think its fine. But
if you're introducing the complexity of NFS just for this use case, then I
think it would be an overkill.

BTW, not every one is ok with setting up a NFS. So if you need to share
files between nodes you may want to have some other recommendations handy
as well (RSync).

Is there a problem is storing this file on the DB instead of the FS?

On Wed, Aug 16, 2017 at 5:41 PM, Megala Uthayakumar  wrote:

> Hi Nuwan,
>
> As mentioned by Harshan user needs to configure NFS when configuring this
> in HA setup. We do accept local file path or NFS url in the configuration
> file for the path.
>
> Thanks.
>
> On Wed, Aug 16, 2017 at 5:01 PM, Nuwan Dias  wrote:
>
>> When you have two nodes of the IoT server running for HA, and if a given
>> image/file is stored on the file system of one server, how do you make that
>> image/file available for a request that's served via the other server?
>>
>> On Wed, Aug 16, 2017 at 4:54 PM, Megala Uthayakumar 
>> wrote:
>>
>>> Hi All,
>>>
>>> Currently we are working on writing a mobile application store for IOT
>>> server. For that we have decided to save the images and binary files
>>> related with applications in the file system(This decision was taken
>>> considering the file size). File location will be derived from a
>>> configuration. While implementing this, we have come-up with the following
>>> suggestions for saving files as per the meeting we had internally.
>>>
>>> *Option 1*
>>> For each application we will have a folder with the name as the
>>> generated ID for the application from the database. Each will hold all the
>>> relevant artifacts such as icon, screen-shots and binary files.
>>> In this case, there were 2 suggestions
>>>
>>>1. For each app, the icon and screen-shots will be saved with the
>>>same name. Ex - For icons the image file name will be 'icon' and 
>>> like-wise
>>>for others as well. So in this case, we do not need to save image names 
>>> in
>>>the database.
>>>2. Need to dynamically generate some random names.
>>>
>>> If we consider first approach, if we use the same name for all the
>>> application, there is possible chance an attacker may get all the image
>>> files if the name is known to them.  AFAIU this can happen even if we do
>>> not use the 1st approach and use the second approach in which we use random
>>> names, as this can happen only if the root path for saving the artifacts
>>> are compromised and without the name also attacker can do a "listFiles"
>>> request  and get all the data.
>>>
>>> *Option 2*
>>> For binary files and image files we will have separate location and each
>>> respective files will be saved in these location separately by generating a
>>> unique name. This may be helpful if we can cache the images and improve the
>>> performance, but this may not be possible in real production scenarios
>>> depending on the size of the images.
>>>
>>> What would be the most preferable option? Comments and suggestions on
>>> this regard is highly appreciated.
>>>
>>> Thanks.
>>>
>>> Regards,
>>> Megala
>>>
>>> --
>>> Megala Uthayakumar
>>>
>>> Software Engineer
>>> Mobile : 0779967122
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "WSO2 IoT Team Group" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to iot-group+unsubscr...@wso2.com.
>>> For more options, visit https://groups.google.com/a/wso2.com/d/optout.
>>>
>>
>>
>>
>> --
>> Nuwan Dias
>>
>> Software Architect - WSO2, Inc. http://wso2.com
>> email : nuw...@wso2.com
>> Phone : +94 777 775 729 <+94%2077%20777%205729>
>>
>
>
>
> --
> Megala Uthayakumar
>
> Software Engineer
> Mobile : 0779967122
>



-- 
Nuwan Dias

Software Architect - WSO2, Inc. http://wso2.com
email : nuw...@wso2.com
Phone : +94 777 775 729
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [IOT]Concerns regarding saving the image and binary files in the File System

[Megala Uthayakumar]

Hi Nuwan,

As mentioned by Harshan user needs to configure NFS when configuring this
in HA setup. We do accept local file path or NFS url in the configuration
file for the path.

Thanks.

On Wed, Aug 16, 2017 at 5:01 PM, Nuwan Dias  wrote:

> When you have two nodes of the IoT server running for HA, and if a given
> image/file is stored on the file system of one server, how do you make that
> image/file available for a request that's served via the other server?
>
> On Wed, Aug 16, 2017 at 4:54 PM, Megala Uthayakumar 
> wrote:
>
>> Hi All,
>>
>> Currently we are working on writing a mobile application store for IOT
>> server. For that we have decided to save the images and binary files
>> related with applications in the file system(This decision was taken
>> considering the file size). File location will be derived from a
>> configuration. While implementing this, we have come-up with the following
>> suggestions for saving files as per the meeting we had internally.
>>
>> *Option 1*
>> For each application we will have a folder with the name as the generated
>> ID for the application from the database. Each will hold all the relevant
>> artifacts such as icon, screen-shots and binary files.
>> In this case, there were 2 suggestions
>>
>>1. For each app, the icon and screen-shots will be saved with the
>>same name. Ex - For icons the image file name will be 'icon' and like-wise
>>for others as well. So in this case, we do not need to save image names in
>>the database.
>>2. Need to dynamically generate some random names.
>>
>> If we consider first approach, if we use the same name for all the
>> application, there is possible chance an attacker may get all the image
>> files if the name is known to them.  AFAIU this can happen even if we do
>> not use the 1st approach and use the second approach in which we use random
>> names, as this can happen only if the root path for saving the artifacts
>> are compromised and without the name also attacker can do a "listFiles"
>> request  and get all the data.
>>
>> *Option 2*
>> For binary files and image files we will have separate location and each
>> respective files will be saved in these location separately by generating a
>> unique name. This may be helpful if we can cache the images and improve the
>> performance, but this may not be possible in real production scenarios
>> depending on the size of the images.
>>
>> What would be the most preferable option? Comments and suggestions on
>> this regard is highly appreciated.
>>
>> Thanks.
>>
>> Regards,
>> Megala
>>
>> --
>> Megala Uthayakumar
>>
>> Software Engineer
>> Mobile : 0779967122
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "WSO2 IoT Team Group" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to iot-group+unsubscr...@wso2.com.
>> For more options, visit https://groups.google.com/a/wso2.com/d/optout.
>>
>
>
>
> --
> Nuwan Dias
>
> Software Architect - WSO2, Inc. http://wso2.com
> email : nuw...@wso2.com
> Phone : +94 777 775 729 <+94%2077%20777%205729>
>



-- 
Megala Uthayakumar

Software Engineer
Mobile : 0779967122
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [IOT]Concerns regarding saving the image and binary files in the File System

[Harshan Liyanage]

Hi,

@Megala : I think having too many folders (lets say if there are 100's of
apps) won't be a good idea. So I'm +1 with the option 2.

As Nuwan has mentioned we need to think of serving this on HA mode. So if
we are to go with filesystem based approach we might need to have NFS
in-between IoT nodes when the IoT is running on HA mode.

Thanks,

Harshan Liyanage
EMM/IoT TG
Mobile: *+94765672894*
Email: hars...@wso2.com
Blog : http://harshanliyanage.blogspot.com/
Medium : https://medium.com/@harshan.dll
*WSO2, Inc. :** wso2.com *
lean.enterprise.middleware.

On Wed, Aug 16, 2017 at 5:01 PM, Nuwan Dias  wrote:

> When you have two nodes of the IoT server running for HA, and if a given
> image/file is stored on the file system of one server, how do you make that
> image/file available for a request that's served via the other server?
>
> On Wed, Aug 16, 2017 at 4:54 PM, Megala Uthayakumar 
> wrote:
>
>> Hi All,
>>
>> Currently we are working on writing a mobile application store for IOT
>> server. For that we have decided to save the images and binary files
>> related with applications in the file system(This decision was taken
>> considering the file size). File location will be derived from a
>> configuration. While implementing this, we have come-up with the following
>> suggestions for saving files as per the meeting we had internally.
>>
>> *Option 1*
>> For each application we will have a folder with the name as the generated
>> ID for the application from the database. Each will hold all the relevant
>> artifacts such as icon, screen-shots and binary files.
>> In this case, there were 2 suggestions
>>
>>1. For each app, the icon and screen-shots will be saved with the
>>same name. Ex - For icons the image file name will be 'icon' and like-wise
>>for others as well. So in this case, we do not need to save image names in
>>the database.
>>2. Need to dynamically generate some random names.
>>
>> If we consider first approach, if we use the same name for all the
>> application, there is possible chance an attacker may get all the image
>> files if the name is known to them.  AFAIU this can happen even if we do
>> not use the 1st approach and use the second approach in which we use random
>> names, as this can happen only if the root path for saving the artifacts
>> are compromised and without the name also attacker can do a "listFiles"
>> request  and get all the data.
>>
>> *Option 2*
>> For binary files and image files we will have separate location and each
>> respective files will be saved in these location separately by generating a
>> unique name. This may be helpful if we can cache the images and improve the
>> performance, but this may not be possible in real production scenarios
>> depending on the size of the images.
>>
>> What would be the most preferable option? Comments and suggestions on
>> this regard is highly appreciated.
>>
>> Thanks.
>>
>> Regards,
>> Megala
>>
>> --
>> Megala Uthayakumar
>>
>> Software Engineer
>> Mobile : 0779967122
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "WSO2 IoT Team Group" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to iot-group+unsubscr...@wso2.com.
>> For more options, visit https://groups.google.com/a/wso2.com/d/optout.
>>
>
>
>
> --
> Nuwan Dias
>
> Software Architect - WSO2, Inc. http://wso2.com
> email : nuw...@wso2.com
> Phone : +94 777 775 729 <+94%2077%20777%205729>
>
> --
> You received this message because you are subscribed to the Google Groups
> "WSO2 IoT Team Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to iot-group+unsubscr...@wso2.com.
> For more options, visit https://groups.google.com/a/wso2.com/d/optout.
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [IOT]Concerns regarding saving the image and binary files in the File System

[Nuwan Dias]

When you have two nodes of the IoT server running for HA, and if a given
image/file is stored on the file system of one server, how do you make that
image/file available for a request that's served via the other server?

On Wed, Aug 16, 2017 at 4:54 PM, Megala Uthayakumar  wrote:

> Hi All,
>
> Currently we are working on writing a mobile application store for IOT
> server. For that we have decided to save the images and binary files
> related with applications in the file system(This decision was taken
> considering the file size). File location will be derived from a
> configuration. While implementing this, we have come-up with the following
> suggestions for saving files as per the meeting we had internally.
>
> *Option 1*
> For each application we will have a folder with the name as the generated
> ID for the application from the database. Each will hold all the relevant
> artifacts such as icon, screen-shots and binary files.
> In this case, there were 2 suggestions
>
>1. For each app, the icon and screen-shots will be saved with the same
>name. Ex - For icons the image file name will be 'icon' and like-wise for
>others as well. So in this case, we do not need to save image names in the
>database.
>2. Need to dynamically generate some random names.
>
> If we consider first approach, if we use the same name for all the
> application, there is possible chance an attacker may get all the image
> files if the name is known to them.  AFAIU this can happen even if we do
> not use the 1st approach and use the second approach in which we use random
> names, as this can happen only if the root path for saving the artifacts
> are compromised and without the name also attacker can do a "listFiles"
> request  and get all the data.
>
> *Option 2*
> For binary files and image files we will have separate location and each
> respective files will be saved in these location separately by generating a
> unique name. This may be helpful if we can cache the images and improve the
> performance, but this may not be possible in real production scenarios
> depending on the size of the images.
>
> What would be the most preferable option? Comments and suggestions on this
> regard is highly appreciated.
>
> Thanks.
>
> Regards,
> Megala
>
> --
> Megala Uthayakumar
>
> Software Engineer
> Mobile : 0779967122
>
> --
> You received this message because you are subscribed to the Google Groups
> "WSO2 IoT Team Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to iot-group+unsubscr...@wso2.com.
> For more options, visit https://groups.google.com/a/wso2.com/d/optout.
>



-- 
Nuwan Dias

Software Architect - WSO2, Inc. http://wso2.com
email : nuw...@wso2.com
Phone : +94 777 775 729
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] [IOT]Concerns regarding saving the image and binary files in the File System

[Megala Uthayakumar]

Hi All,

Currently we are working on writing a mobile application store for IOT
server. For that we have decided to save the images and binary files
related with applications in the file system(This decision was taken
considering the file size). File location will be derived from a
configuration. While implementing this, we have come-up with the following
suggestions for saving files as per the meeting we had internally.

*Option 1*
For each application we will have a folder with the name as the generated
ID for the application from the database. Each will hold all the relevant
artifacts such as icon, screen-shots and binary files.
In this case, there were 2 suggestions

   1. For each app, the icon and screen-shots will be saved with the same
   name. Ex - For icons the image file name will be 'icon' and like-wise for
   others as well. So in this case, we do not need to save image names in the
   database.
   2. Need to dynamically generate some random names.

If we consider first approach, if we use the same name for all the
application, there is possible chance an attacker may get all the image
files if the name is known to them.  AFAIU this can happen even if we do
not use the 1st approach and use the second approach in which we use random
names, as this can happen only if the root path for saving the artifacts
are compromised and without the name also attacker can do a "listFiles"
request  and get all the data.

*Option 2*
For binary files and image files we will have separate location and each
respective files will be saved in these location separately by generating a
unique name. This may be helpful if we can cache the images and improve the
performance, but this may not be possible in real production scenarios
depending on the size of the images.

What would be the most preferable option? Comments and suggestions on this
regard is highly appreciated.

Thanks.

Regards,
Megala

-- 
Megala Uthayakumar

Software Engineer
Mobile : 0779967122
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev