Hi All,
Please fine my responses below,
*@Nuwan *
Is there a problem is storing this file on the DB instead of the FS?
As explained by Chathura and Charitha main reasoning behind not selecting
the database is size of the binary file and yes we will consider other
possible ways for supporting this as well, given that NFS may not be
preferred by the customer.
*@Chathura*
If you go with the first approach, could you please mention what are the
parameters you use to get the image? It is always better to get the image
with a random parameters so it prevents hackers from guessing image names.
As we know already we will be getting the files using an REST API and in
that case, the URL will be something like follow,
*https://localhost:8243/applications/getIcon/
<https://localhost:8243/applications/getIcon/><UUID>*
So with this URL, nothing can be guessed as in the API level, what we
expect to do is something like follow,
1. From the UUID, get the auto-generated ID from the database
2. We will derive the path for the particular application as follow,
- <Path in Configuration>/ID
3. In that path we will search for the file with the name "icon" and we
will pass File with the response.
In this case, I don't think there is possible security issue where we
expose the path name or so. Please do correct me, if it is not the case.By
having random names, we need to have additional database operations for
saving icon/other image files name and retrieving them.
*@Charitha*
I'm not clear with the reason behind this statement in under option 2
"but this may not be possible in real production scenarios depending on the
size of the images.
I meant that caching images will not work as in the real-production
scenarios, we may have images with higher sizes, so in that case, we cannot
efficiently cache and gain performance gain out of them by using option 2.
*@Waruna *
Storage Location - This should be written as extension where you can store
in local directory, FTP, NFS or someone can customize based on requirement.
For the storage location, if we are using NFS or local file system, I think
om the implementation level it will be the same and we do not need to
specially handle them. We have not thought about supporting FTP for the
first cut implementation and wondering whether we need that level
complexity (i.e. Having a server-server communication) for only saving
image files and binary files.
Do we need real requirement to search by image name,apk name or other
content name. If we have that kind of requiremnt as well then option 1 will
have some cons.
AFAIK we will not get a requirement to search the apis with the apk name or
image name as people are normally interested in searching by using
application name, platform name etc.
Thanks.
Regards,
Megala
On Wed, Aug 16, 2017 at 6:31 PM, Waruna Jayaweera <waru...@wso2.com> wrote:
> Hi Megala,
>
> AFAIU There are two problems as where we store the file
> contents(supporting clustering cases) and file structure of application
> content.
>
> Storage Location - This should be written as extension where you can store
> in local directory, FTP, NFS or someone can customize based on requirement.
> Application store structure - I prefer option 1 as it will be more
> manageable per application which makes separate storage space for
> application contents(apk,icon etc).
>
> As an example we can store like <application_id>/<version>/<
> application_content_name>/content_data
> I option 2 you will have less folders but you may need to concat file
> names for support for multiple application versions and their contents. In
> option 1 delete application operation also easy.
>
> Do we need real requirement to search by image name,apk name or other
> content name. If we have that kind of requiremnt as well then option 1 will
> have some cons.
>
> Thanks,
> Waruna
>
> On Wed, Aug 16, 2017 at 6:02 PM, Charitha Goonetilleke <charit...@wso2.com
> > wrote:
>
>> Hi Nuwan,
>>
>> As our previous experience with similar approach we have used to store
>> binaries in DB, it cause for scallability issues in later when growing the
>> size of the DB in GBs. So storing files as binaries in DB is highly
>> depending on the DB engine used and the growth rate of the data. IMO, these
>> concerns and issues might be critical and complex than the complexity we
>> are going to introduce with FS based solution. So I think it is best to
>> store files in FS rather than adding it to DB.
>>
>> Anyway as you have raised, there might be situations where, NFS
>> integration is not possible. So I'm +1 for having few alternative ways to
>> handle this as well. In addition to the RSync, we can also use shared
>> volume or even S3 bucket if some one is going to deploy this in AWS.
>>
>> @Megala, I'm not clear with the reason behind this statement in under
>> option 2
>>
>> "but this may not be possible in real production scenarios depending on
>> the size of the images."
>>
>>
>> On Wed, Aug 16, 2017 at 5:46 PM, Nuwan Dias <nuw...@wso2.com> wrote:
>>
>>> If you require NFS anyway (for other usecases), then I think its fine.
>>> But if you're introducing the complexity of NFS just for this use case,
>>> then I think it would be an overkill.
>>>
>>> BTW, not every one is ok with setting up a NFS. So if you need to share
>>> files between nodes you may want to have some other recommendations handy
>>> as well (RSync).
>>>
>>> Is there a problem is storing this file on the DB instead of the FS?
>>>
>>> On Wed, Aug 16, 2017 at 5:41 PM, Megala Uthayakumar <meg...@wso2.com>
>>> wrote:
>>>
>>>> Hi Nuwan,
>>>>
>>>> As mentioned by Harshan user needs to configure NFS when configuring
>>>> this in HA setup. We do accept local file path or NFS url in the
>>>> configuration file for the path.
>>>>
>>>> Thanks.
>>>>
>>>> On Wed, Aug 16, 2017 at 5:01 PM, Nuwan Dias <nuw...@wso2.com> wrote:
>>>>
>>>>> When you have two nodes of the IoT server running for HA, and if a
>>>>> given image/file is stored on the file system of one server, how do you
>>>>> make that image/file available for a request that's served via the other
>>>>> server?
>>>>>
>>>>> On Wed, Aug 16, 2017 at 4:54 PM, Megala Uthayakumar <meg...@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> Currently we are working on writing a mobile application store for
>>>>>> IOT server. For that we have decided to save the images and binary files
>>>>>> related with applications in the file system(This decision was taken
>>>>>> considering the file size). File location will be derived from a
>>>>>> configuration. While implementing this, we have come-up with the
>>>>>> following
>>>>>> suggestions for saving files as per the meeting we had internally.
>>>>>>
>>>>>> *Option 1*
>>>>>> For each application we will have a folder with the name as the
>>>>>> generated ID for the application from the database. Each will hold all
>>>>>> the
>>>>>> relevant artifacts such as icon, screen-shots and binary files.
>>>>>> In this case, there were 2 suggestions
>>>>>>
>>>>>> 1. For each app, the icon and screen-shots will be saved with the
>>>>>> same name. Ex - For icons the image file name will be 'icon' and
>>>>>> like-wise
>>>>>> for others as well. So in this case, we do not need to save image
>>>>>> names in
>>>>>> the database.
>>>>>> 2. Need to dynamically generate some random names.
>>>>>>
>>>>>> If we consider first approach, if we use the same name for all the
>>>>>> application, there is possible chance an attacker may get all the image
>>>>>> files if the name is known to them. AFAIU this can happen even if we do
>>>>>> not use the 1st approach and use the second approach in which we use
>>>>>> random
>>>>>> names, as this can happen only if the root path for saving the artifacts
>>>>>> are compromised and without the name also attacker can do a "listFiles"
>>>>>> request and get all the data.
>>>>>>
>>>>>> *Option 2*
>>>>>> For binary files and image files we will have separate location and
>>>>>> each respective files will be saved in these location separately by
>>>>>> generating a unique name. This may be helpful if we can cache the images
>>>>>> and improve the performance, but this may not be possible in real
>>>>>> production scenarios depending on the size of the images.
>>>>>>
>>>>>> What would be the most preferable option? Comments and suggestions on
>>>>>> this regard is highly appreciated.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> Regards,
>>>>>> Megala
>>>>>>
>>>>>> --
>>>>>> Megala Uthayakumar
>>>>>>
>>>>>> Software Engineer
>>>>>> Mobile : 0779967122
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "WSO2 IoT Team Group" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to iot-group+unsubscr...@wso2.com.
>>>>>> For more options, visit https://groups.google.com/a/wso2.com/d/optout
>>>>>> .
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Nuwan Dias
>>>>>
>>>>> Software Architect - WSO2, Inc. http://wso2.com
>>>>> email : nuw...@wso2.com
>>>>> Phone : +94 777 775 729 <+94%2077%20777%205729>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Megala Uthayakumar
>>>>
>>>> Software Engineer
>>>> Mobile : 0779967122
>>>>
>>>
>>>
>>>
>>> --
>>> Nuwan Dias
>>>
>>> Software Architect - WSO2, Inc. http://wso2.com
>>> email : nuw...@wso2.com
>>> Phone : +94 777 775 729 <+94%2077%20777%205729>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "WSO2 IoT Team Group" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to iot-group+unsubscr...@wso2.com.
>>> For more options, visit https://groups.google.com/a/wso2.com/d/optout.
>>>
>>
>>
>>
>> --
>> *Charitha Goonetilleke*
>> Senior Software Engineer
>> WSO2 Inc.; http://wso2.com
>> lean.enterprise.middleware
>>
>> mobile: +94 77 751 3669 <%2B94777513669>
>> Twitter:@CharithaWs <https://twitter.com/CharithaWs>, fb: charithag
>> <https://www.facebook.com/charithag>, linkedin: charithag
>> <http://www.linkedin.com/in/charithag>
>>
>> <http://wso2.com/signature>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "WSO2 IoT Team Group" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to iot-group+unsubscr...@wso2.com.
>> For more options, visit https://groups.google.com/a/wso2.com/d/optout.
>>
>
>
>
> --
> Regards,
>
> Waruna Lakshitha Jayaweera
> Senior Software Engineer
> WSO2 Inc; http://wso2.com
> phone: +94713255198 <+94%2071%20325%205198>
> http://waruapz.blogspot.com/
>
>
--
Megala Uthayakumar
Software Engineer
Mobile : 0779967122
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
