Re: [Dev] WSO2 Docker images layered composition

[Imesh Gunaratne]

On Wed, Jul 6, 2016 at 9:55 AM, Isuru Haththotuwa  wrote:


> The rationale behind my previous statement is that once the docker images
> are pushed to a registry, the users do not have to continuously build them,
> but rather can extend from the product Docker images and do development and
> testing.
>
> Thanks for the response Isuru! Yes that's true!

Thanks
​

> However, I do agree that this change is indeed useful for scenarios where
> you need to build images in network isolated environments. In such cases
> having a base image is essential, otherwise it would not be possible to
> build the Docker images. Therefore,  +1 for adding a base image.
>
>>
>> Thanks
>>
>>
>>> Anyway, lets give this a go and try to improve ii as we go along. Anyway
>>> we should be able to push proper product images to the Dockerhub once we
>>> have OpenJDK support.
>>>

 Thanks

>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* *
>
>
>


 --
 *Imesh Gunaratne*
 Software Architect
 WSO2 Inc: http://wso2.com
 T: +94 11 214 5345 M: +94 77 374 2057
 W: https://medium.com/@imesh TW: @imesh
 Lean . Enterprise . Middleware


>>>
>>>
>>> --
>>> Thanks and Regards,
>>>
>>> Isuru H.
>>> +94 716 358 048* *
>>>
>>>
>>>
>>
>>
>> --
>> *Imesh Gunaratne*
>> Software Architect
>> WSO2 Inc: http://wso2.com
>> T: +94 11 214 5345 M: +94 77 374 2057
>> W: https://medium.com/@imesh TW: @imesh
>> Lean . Enterprise . Middleware
>>
>>
>
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* *
>
>
>


-- 
*Imesh Gunaratne*
Software Architect
WSO2 Inc: http://wso2.com
T: +94 11 214 5345 M: +94 77 374 2057
W: https://medium.com/@imesh TW: @imesh
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Isuru Haththotuwa]

Hi Imesh,

Sorry for the late response.

On Tue, Jun 28, 2016 at 9:06 AM, Imesh Gunaratne  wrote:

>
>
> On Mon, Jun 27, 2016 at 11:44 PM, Isuru Haththotuwa 
> wrote:
>
>> Hi Imesh,
>>
>> Agreed on the size impact. My concern is that this change does not bring
>> in a significant improvement for the model that we have now.
>>
>
> ​I think it has a significant improvement. If we don't use a base image​,
> each product image will build on top of an OS image and each build will
> download all the prerequisites. As a result the internet bandwidth will get
> utilized unnecessarily. AFAIK we all faced this issue and created our own
> base images to solve the problem. Therefore I think it's better to have a
> base image in the remote repository.
>
> At the time a production environment is being created, users can change
> the base image to an OS image and get the image size optimizations done.
> WDYT?
>
The rationale behind my previous statement is that once the docker images
are pushed to a registry, the users do not have to continuously build them,
but rather can extend from the product Docker images and do development and
testing.

However, I do agree that this change is indeed useful for scenarios where
you need to build images in network isolated environments. In such cases
having a base image is essential, otherwise it would not be possible to
build the Docker images. Therefore,  +1 for adding a base image.

>
> Thanks
>
>
>> Anyway, lets give this a go and try to improve ii as we go along. Anyway
>> we should be able to push proper product images to the Dockerhub once we
>> have OpenJDK support.
>>
>>>
>>> Thanks
>>>

 --
 Thanks and Regards,

 Isuru H.
 +94 716 358 048* *



>>>
>>>
>>> --
>>> *Imesh Gunaratne*
>>> Software Architect
>>> WSO2 Inc: http://wso2.com
>>> T: +94 11 214 5345 M: +94 77 374 2057
>>> W: https://medium.com/@imesh TW: @imesh
>>> Lean . Enterprise . Middleware
>>>
>>>
>>
>>
>> --
>> Thanks and Regards,
>>
>> Isuru H.
>> +94 716 358 048* *
>>
>>
>>
>
>
> --
> *Imesh Gunaratne*
> Software Architect
> WSO2 Inc: http://wso2.com
> T: +94 11 214 5345 M: +94 77 374 2057
> W: https://medium.com/@imesh TW: @imesh
> Lean . Enterprise . Middleware
>
>


-- 
Thanks and Regards,

Isuru H.
+94 716 358 048* *
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Akila Ravihansa Perera]

Hi,


>
>> Regarding tag names, it was done on purpose do multiple Puppet module
>> releases for a specific WSO2 product version. I know it's ugly, but would
>> like to hear others' thoughts/suggestions.
>>
>
> ​I'm sorry I do not get this.​
>
> ​Can you please explain this in more detail?
>

I take back what I said :) Let me put it this way; we used Puppet to
provision the images which got pushed to DockerHub. We had named the Puppet
modules as wso2-: so that
multiple Puppet module releases were possible for a one product version.
That's why (I think) these images were named the same way Puppet modules
were which was a mistake on our part. Thanks Ramon for pointing this out.

Also regarding platform specific Docker images; eg: "mesos", "kubernetes"
etc. there is a separate thread to discuss it. Refer to mail thread "Naming
convention for WSO2 docker images based on the platform". In summary,
current approach is to append the platform name to the Docker image name
itself.
@Ramon: Do you have any objections against this? You are most welcome to
share your thoughts on that thread.

Thanks.

-- 
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Imesh Gunaratne]

On Mon, Jun 27, 2016 at 11:44 PM, Isuru Haththotuwa  wrote:

> Hi Imesh,
>
> Agreed on the size impact. My concern is that this change does not bring
> in a significant improvement for the model that we have now.
>

​I think it has a significant improvement. If we don't use a base image​,
each product image will build on top of an OS image and each build will
download all the prerequisites. As a result the internet bandwidth will get
utilized unnecessarily. AFAIK we all faced this issue and created our own
base images to solve the problem. Therefore I think it's better to have a
base image in the remote repository.

At the time a production environment is being created, users can change the
base image to an OS image and get the image size optimizations done. WDYT?

Thanks


> Anyway, lets give this a go and try to improve ii as we go along. Anyway
> we should be able to push proper product images to the Dockerhub once we
> have OpenJDK support.
>
>>
>> Thanks
>>
>>>
>>> --
>>> Thanks and Regards,
>>>
>>> Isuru H.
>>> +94 716 358 048* *
>>>
>>>
>>>
>>
>>
>> --
>> *Imesh Gunaratne*
>> Software Architect
>> WSO2 Inc: http://wso2.com
>> T: +94 11 214 5345 M: +94 77 374 2057
>> W: https://medium.com/@imesh TW: @imesh
>> Lean . Enterprise . Middleware
>>
>>
>
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* *
>
>
>


-- 
*Imesh Gunaratne*
Software Architect
WSO2 Inc: http://wso2.com
T: +94 11 214 5345 M: +94 77 374 2057
W: https://medium.com/@imesh TW: @imesh
Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Imesh Gunaratne]

Hi Akila,

On Mon, Jun 27, 2016 at 7:50 PM, Akila Ravihansa Perera 
wrote:

>
> Regarding tag names, it was done on purpose do multiple Puppet module
> releases for a specific WSO2 product version. I know it's ugly, but would
> like to hear others' thoughts/suggestions.
>

​I'm sorry I do not get this.​

​Can you please explain this in more detail?

Thanks​

-- 
*Imesh Gunaratne*
Software Architect
WSO2 Inc: http://wso2.com
T: +94 11 214 5345 M: +94 77 374 2057
W: https://medium.com/@imesh TW: @imesh
Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Isuru Haththotuwa]

Hi Imesh,

On Thu, Jun 23, 2016 at 3:06 PM, Imesh Gunaratne  wrote:

> Hi Isuru/Akila,
>
> On Tue, Jun 21, 2016 at 3:19 PM, Isuru Haththotuwa 
> wrote:
>
> To summarize:
>
>- Publish a wso2 product specific docker image to a local docker
>repository (we can even push to dockerhub, without the JDK distribution and
>ask the users to provide the JDK when spinning up a container from the
>image).
>
> A good suggestion, since we currently track product downloads for
> analytics we might need to find a way to do the same with Docker if we were
> to publicly share WSO2 Docker images. Otherwise using a local registry with
> a base/product image would do.
>
>- Guide the users/developers to create their own Dockerfile extending
>from the original wso2 product Docker image.
>
> We might need to solve 1, to do this.
>
> On Thu, Jun 23, 2016 at 9:42 AM, Isuru Haththotuwa 
> wrote:
>
>>
>> On Tue, Jun 21, 2016 at 6:54 PM, Akila Ravihansa Perera <
>> raviha...@wso2.com> wrote:
>>
>>>
>>> I'm not against having a base image for the WSO2 Docker images. But,
>> IMHO the advantages should be significant for doing so. AFAIU there is no
>> significant advantage of doing so currently. But, this is my take only.
>> Therefore lets get the input from the community on this matter.
>>
>> @Imesh: WDYT?
>>
>> ​Currently almost all of us use a base image locally to reduce the
> internet bandwidth usage due to executing docker builds from scratch.
> Therefore I think it would be better to push that to the remote repository.
>
> As Akila has mentioned the only downside of this is the image size being
> increased by around 100 MB. ​When creating Docker images for production
> deployment those optimizations can be taken into account.
>
Agreed on the size impact. My concern is that this change does not bring in
a significant improvement for the model that we have now.

Anyway, lets give this a go and try to improve ii as we go along. Anyway we
should be able to push proper product images to the Dockerhub once we have
OpenJDK support.

>
> Thanks
>
>>
>> --
>> Thanks and Regards,
>>
>> Isuru H.
>> +94 716 358 048* *
>>
>>
>>
>
>
> --
> *Imesh Gunaratne*
> Software Architect
> WSO2 Inc: http://wso2.com
> T: +94 11 214 5345 M: +94 77 374 2057
> W: https://medium.com/@imesh TW: @imesh
> Lean . Enterprise . Middleware
>
>


-- 
Thanks and Regards,

Isuru H.
+94 716 358 048* *
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Akila Ravihansa Perera]

Hi Ramon,

Thanks for sharing this but we will not be able make these official due to
OracleJDK license issues. OpenJDK will be fully supported in Carbon 5 based
products but that will take some time.

Regarding tag names, it was done on purpose do multiple Puppet module
releases for a specific WSO2 product version. I know it's ugly, but would
like to hear others' thoughts/suggestions.

Thanks.
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Imesh Gunaratne]

On Sat, Jun 25, 2016 at 10:21 AM, Akila Ravihansa Perera  wrote:

> Hi,
>
> I've merged the PR [1] which introduces wso2base image. This base image
> will include packages that were previously installed from Puppet (zip and
> unzip), and those are removed from Puppet/Hiera side. With that we should
> be able to build WSO2 Docker images in offline mode.
>
> wso2base image is pushed to DockerHub [2] as well. You don't need to build
> it locally.
>

​Great work! Will try this out!

Thanks​


>
> [1] https://github.com/wso2/dockerfiles/pull/46
> [2] https://hub.docker.com/r/wso2/wso2base/
>
> Thanks.
>
> On Thu, Jun 23, 2016 at 3:06 PM, Imesh Gunaratne  wrote:
>
>> Hi Isuru/Akila,
>>
>> On Tue, Jun 21, 2016 at 3:19 PM, Isuru Haththotuwa 
>> wrote:
>>
>> To summarize:
>>
>>- Publish a wso2 product specific docker image to a local docker
>>repository (we can even push to dockerhub, without the JDK distribution 
>> and
>>ask the users to provide the JDK when spinning up a container from the
>>image).
>>
>> A good suggestion, since we currently track product downloads for
>> analytics we might need to find a way to do the same with Docker if we were
>> to publicly share WSO2 Docker images. Otherwise using a local registry with
>> a base/product image would do.
>>
>>- Guide the users/developers to create their own Dockerfile extending
>>from the original wso2 product Docker image.
>>
>> We might need to solve 1, to do this.
>>
>> On Thu, Jun 23, 2016 at 9:42 AM, Isuru Haththotuwa 
>> wrote:
>>
>>>
>>> On Tue, Jun 21, 2016 at 6:54 PM, Akila Ravihansa Perera <
>>> raviha...@wso2.com> wrote:
>>>

 I'm not against having a base image for the WSO2 Docker images. But,
>>> IMHO the advantages should be significant for doing so. AFAIU there is no
>>> significant advantage of doing so currently. But, this is my take only.
>>> Therefore lets get the input from the community on this matter.
>>>
>>> @Imesh: WDYT?
>>>
>>> ​Currently almost all of us use a base image locally to reduce the
>> internet bandwidth usage due to executing docker builds from scratch.
>> Therefore I think it would be better to push that to the remote repository.
>>
>> As Akila has mentioned the only downside of this is the image size being
>> increased by around 100 MB. ​When creating Docker images for production
>> deployment those optimizations can be taken into account.
>>
>> Thanks
>>
>>>
>>> --
>>> Thanks and Regards,
>>>
>>> Isuru H.
>>> +94 716 358 048* *
>>>
>>>
>>>
>>
>>
>> --
>> *Imesh Gunaratne*
>> Software Architect
>> WSO2 Inc: http://wso2.com
>> T: +94 11 214 5345 M: +94 77 374 2057
>> W: https://medium.com/@imesh TW: @imesh
>> Lean . Enterprise . Middleware
>>
>>
>
>
> --
> Akila Ravihansa Perera
> WSO2 Inc.;  http://wso2.com/
>
> Blog: http://ravihansa3000.blogspot.com
>



-- 
*Imesh Gunaratne*
Software Architect
WSO2 Inc: http://wso2.com
T: +94 11 214 5345 M: +94 77 374 2057
W: https://medium.com/@imesh TW: @imesh
Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Akila Ravihansa Perera]

Hi,

I've merged the PR [1] which introduces wso2base image. This base image
will include packages that were previously installed from Puppet (zip and
unzip), and those are removed from Puppet/Hiera side. With that we should
be able to build WSO2 Docker images in offline mode.

wso2base image is pushed to DockerHub [2] as well. You don't need to build
it locally.

[1] https://github.com/wso2/dockerfiles/pull/46
[2] https://hub.docker.com/r/wso2/wso2base/

Thanks.

On Thu, Jun 23, 2016 at 3:06 PM, Imesh Gunaratne  wrote:

> Hi Isuru/Akila,
>
> On Tue, Jun 21, 2016 at 3:19 PM, Isuru Haththotuwa 
> wrote:
>
> To summarize:
>
>- Publish a wso2 product specific docker image to a local docker
>repository (we can even push to dockerhub, without the JDK distribution and
>ask the users to provide the JDK when spinning up a container from the
>image).
>
> A good suggestion, since we currently track product downloads for
> analytics we might need to find a way to do the same with Docker if we were
> to publicly share WSO2 Docker images. Otherwise using a local registry with
> a base/product image would do.
>
>- Guide the users/developers to create their own Dockerfile extending
>from the original wso2 product Docker image.
>
> We might need to solve 1, to do this.
>
> On Thu, Jun 23, 2016 at 9:42 AM, Isuru Haththotuwa 
> wrote:
>
>>
>> On Tue, Jun 21, 2016 at 6:54 PM, Akila Ravihansa Perera <
>> raviha...@wso2.com> wrote:
>>
>>>
>>> I'm not against having a base image for the WSO2 Docker images. But,
>> IMHO the advantages should be significant for doing so. AFAIU there is no
>> significant advantage of doing so currently. But, this is my take only.
>> Therefore lets get the input from the community on this matter.
>>
>> @Imesh: WDYT?
>>
>> ​Currently almost all of us use a base image locally to reduce the
> internet bandwidth usage due to executing docker builds from scratch.
> Therefore I think it would be better to push that to the remote repository.
>
> As Akila has mentioned the only downside of this is the image size being
> increased by around 100 MB. ​When creating Docker images for production
> deployment those optimizations can be taken into account.
>
> Thanks
>
>>
>> --
>> Thanks and Regards,
>>
>> Isuru H.
>> +94 716 358 048* *
>>
>>
>>
>
>
> --
> *Imesh Gunaratne*
> Software Architect
> WSO2 Inc: http://wso2.com
> T: +94 11 214 5345 M: +94 77 374 2057
> W: https://medium.com/@imesh TW: @imesh
> Lean . Enterprise . Middleware
>
>


-- 
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Imesh Gunaratne]

Hi Isuru/Akila,

On Tue, Jun 21, 2016 at 3:19 PM, Isuru Haththotuwa  wrote:

To summarize:

   - Publish a wso2 product specific docker image to a local docker
   repository (we can even push to dockerhub, without the JDK distribution and
   ask the users to provide the JDK when spinning up a container from the
   image).

A good suggestion, since we currently track product downloads for analytics
we might need to find a way to do the same with Docker if we were to
publicly share WSO2 Docker images. Otherwise using a local registry with a
base/product image would do.

   - Guide the users/developers to create their own Dockerfile extending
   from the original wso2 product Docker image.

We might need to solve 1, to do this.

On Thu, Jun 23, 2016 at 9:42 AM, Isuru Haththotuwa  wrote:

>
> On Tue, Jun 21, 2016 at 6:54 PM, Akila Ravihansa Perera <
> raviha...@wso2.com> wrote:
>
>>
>> I'm not against having a base image for the WSO2 Docker images. But, IMHO
> the advantages should be significant for doing so. AFAIU there is no
> significant advantage of doing so currently. But, this is my take only.
> Therefore lets get the input from the community on this matter.
>
> @Imesh: WDYT?
>
> ​Currently almost all of us use a base image locally to reduce the
internet bandwidth usage due to executing docker builds from scratch.
Therefore I think it would be better to push that to the remote repository.

As Akila has mentioned the only downside of this is the image size being
increased by around 100 MB. ​When creating Docker images for production
deployment those optimizations can be taken into account.

Thanks

>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* *
>
>
>


-- 
*Imesh Gunaratne*
Software Architect
WSO2 Inc: http://wso2.com
T: +94 11 214 5345 M: +94 77 374 2057
W: https://medium.com/@imesh TW: @imesh
Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Isuru Haththotuwa]

Hi Akila,

On Tue, Jun 21, 2016 at 6:54 PM, Akila Ravihansa Perera 
wrote:

> Hi Isuru,
>
> One drawback of not having a wso2base is that in a restricted environment
> where Internet connectivity is not available, a private build server will
> not be able to build the images using Puppet provisioning if there are any
> additional packages to be installed. But if there was a wso2base image,
> those packages could have been installed to that and manually loaded into
> the build server as a one time task.
>
Yes this is true. However, still there is a possibility of this occurring
even at a base image building time, isn't it?

>
> Thanks.
>
I'm not against having a base image for the WSO2 Docker images. But, IMHO
the advantages should be significant for doing so. AFAIU there is no
significant advantage of doing so currently. But, this is my take only.
Therefore lets get the input from the community on this matter.

@Imesh: WDYT?


-- 
Thanks and Regards,

Isuru H.
+94 716 358 048* *
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Akila Ravihansa Perera]

Hi Isuru,

One drawback of not having a wso2base is that in a restricted environment
where Internet connectivity is not available, a private build server will
not be able to build the images using Puppet provisioning if there are any
additional packages to be installed. But if there was a wso2base image,
those packages could have been installed to that and manually loaded into
the build server as a one time task.

Thanks.
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Akila Ravihansa Perera]

Hi Isuru,

+1 for the suggested approach. I think wso2base image will only complement
the model that you have described. We can definitely provide guidelines to
users so that they can extend from our product specific images (excluding
Oracle JDK).

If we have the right layered model, i.e.; wso2base -> wso2product ->
wso2product-customized
then we can cut down the network traffic when push/pull'ing images to/from
Docker registries or even locally save/load'ing images. This is because
Docker will pull only the changed top level layer since bottom layers are
do not change for a specific Dockerfiles release. Since wso2base is common
across all products, this will cut down push/pull time in a multi-product
deployment.

wdyt?

Thanks.

On Tue, Jun 21, 2016 at 3:19 PM, Isuru Haththotuwa  wrote:

> Hi Akila,
>
> If we use a base image, true that we can avoid the time taken to run
> apt-get update and software installation. But still the users will have to
> run the provisioning again and again when doing development/testing. The
> model I suggest is a way to eliminate apt-get update and also the
> provisioning over and over again.
>
> To summarize:
>
>- Publish a wso2 product specific docker image to a local docker
>repository (we can even push to dockerhub, without the JDK distribution and
>ask the users to provide the JDK when spinning up a container from the
>image).
>- Guide the users/developers to create their own Dockerfile extending
>from the original wso2 product Docker image.
>- Use the extension provided to copy artifacts/configurations to the
>container [1]. At development time, ideally the developer would only need
>to modify the configurations/copy artifacts to the carbon server.
>- After the development phase, collect all the changes and apply them
>to the provisioning method (puppet, etc.) that is used, and build the final
>production ready Docker image.
>
> I'm +1 to push a Docker image with a configured product to Dockerhub,
> excluding the JDK and ask the user to provide a JDK at runtime/creating a
> new Dockerfile extending from the original product Dockerfile.
>
> [1].
> https://docs.wso2.com/display/DF110/Running+WSO2+Docker+Images#RunningWSO2DockerImages-Copyingartifactsfromhostmachinetothecontainer
>
> On Mon, Jun 20, 2016 at 12:37 PM, Akila Ravihansa Perera <
> raviha...@wso2.com> wrote:
>
>> Hi Isuru,
>>
>> Do you have any specific concerns about using a wso2base image? On
>> average there is a spike of 100MB when using a wso2base image, but that's a
>> small price to pay when considering benefits users would gain out of this.
>> Note that users can use "default" provisioning method to package any
>> jars/configs to the image. The main problem here is the time taken to run
>> apt-get update and package installation. Especially having to download
>> files over I'net every time a Docker image is built which I think is
>> unnecessary. wdyt?
>>
>> Thanks.
>>
>> On Wed, Jun 1, 2016 at 10:31 PM, Isuru Haththotuwa 
>> wrote:
>>
>>> Hi Akila and Imesh,
>>>
>>> Without creating a base image, if we package everything in a product
>>> specific image, host it in a local registry, and provide instructions to
>>> create a local Dockerfile extending from that image, the time taken to
>>> build the local image would be lesser. This will speedup the
>>> development/testing procedure even more. The developers can package any
>>> apps/jars/configs via the local Dockerfile (using volume mounts, etc.), for
>>> testing. I have seen this pattern used in some Docker images, for an
>>> example in the haproxy Docker image [1].
>>>
>>> One downside is, as you mentioned, we cannot push the image to Dockerhub
>>> with Oracle JDK distribution included. Maybe we can create the product
>>> Docker image with the relevant utilities (wget, zip, unzip, etc.) + the
>>> product distribution, and push it to Docker hub. Then provide an option to
>>> package the JDK via a local Dockerfile extending from the product
>>> Dockerfile.
>>>
>>> WDYT?
>>>
>>> [1]. https://hub.docker.com/_/haproxy/
>>>
>>> On Mon, May 30, 2016 at 12:48 PM, Akila Ravihansa Perera <
>>> raviha...@wso2.com> wrote:
>>>
 Hi,

 Currently we don't have a base image for WSO2 Docker images due to
 image size issues we had with it. But later we found out that this was
 mainly due to multiple RUN commands, which caused Docker to add unnecessary
 layers.

 I think we should bring back the base image for couple of reasons;

 1. wso2base image with minimum packages installed (wget, puppet etc)
 will eliminate the need for having to run apt-get commands (which are very
 expensive, time-wise) each and every time

 2. wso2base image can be hosted publicly (without Oracle JDK of course)
 so that users don't have to build wso2base but rather pull it from
 DockerHub.

 3. wso2base image provides a Docker layer which 

Re: [Dev] WSO2 Docker images layered composition

[Isuru Haththotuwa]

Hi Akila,

If we use a base image, true that we can avoid the time taken to run
apt-get update and software installation. But still the users will have to
run the provisioning again and again when doing development/testing. The
model I suggest is a way to eliminate apt-get update and also the
provisioning over and over again.

To summarize:

   - Publish a wso2 product specific docker image to a local docker
   repository (we can even push to dockerhub, without the JDK distribution and
   ask the users to provide the JDK when spinning up a container from the
   image).
   - Guide the users/developers to create their own Dockerfile extending
   from the original wso2 product Docker image.
   - Use the extension provided to copy artifacts/configurations to the
   container [1]. At development time, ideally the developer would only need
   to modify the configurations/copy artifacts to the carbon server.
   - After the development phase, collect all the changes and apply them to
   the provisioning method (puppet, etc.) that is used, and build the final
   production ready Docker image.

I'm +1 to push a Docker image with a configured product to Dockerhub,
excluding the JDK and ask the user to provide a JDK at runtime/creating a
new Dockerfile extending from the original product Dockerfile.

[1].
https://docs.wso2.com/display/DF110/Running+WSO2+Docker+Images#RunningWSO2DockerImages-Copyingartifactsfromhostmachinetothecontainer

On Mon, Jun 20, 2016 at 12:37 PM, Akila Ravihansa Perera  wrote:

> Hi Isuru,
>
> Do you have any specific concerns about using a wso2base image? On average
> there is a spike of 100MB when using a wso2base image, but that's a small
> price to pay when considering benefits users would gain out of this. Note
> that users can use "default" provisioning method to package any
> jars/configs to the image. The main problem here is the time taken to run
> apt-get update and package installation. Especially having to download
> files over I'net every time a Docker image is built which I think is
> unnecessary. wdyt?
>
> Thanks.
>
> On Wed, Jun 1, 2016 at 10:31 PM, Isuru Haththotuwa 
> wrote:
>
>> Hi Akila and Imesh,
>>
>> Without creating a base image, if we package everything in a product
>> specific image, host it in a local registry, and provide instructions to
>> create a local Dockerfile extending from that image, the time taken to
>> build the local image would be lesser. This will speedup the
>> development/testing procedure even more. The developers can package any
>> apps/jars/configs via the local Dockerfile (using volume mounts, etc.), for
>> testing. I have seen this pattern used in some Docker images, for an
>> example in the haproxy Docker image [1].
>>
>> One downside is, as you mentioned, we cannot push the image to Dockerhub
>> with Oracle JDK distribution included. Maybe we can create the product
>> Docker image with the relevant utilities (wget, zip, unzip, etc.) + the
>> product distribution, and push it to Docker hub. Then provide an option to
>> package the JDK via a local Dockerfile extending from the product
>> Dockerfile.
>>
>> WDYT?
>>
>> [1]. https://hub.docker.com/_/haproxy/
>>
>> On Mon, May 30, 2016 at 12:48 PM, Akila Ravihansa Perera <
>> raviha...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Currently we don't have a base image for WSO2 Docker images due to image
>>> size issues we had with it. But later we found out that this was mainly due
>>> to multiple RUN commands, which caused Docker to add unnecessary layers.
>>>
>>> I think we should bring back the base image for couple of reasons;
>>>
>>> 1. wso2base image with minimum packages installed (wget, puppet etc)
>>> will eliminate the need for having to run apt-get commands (which are very
>>> expensive, time-wise) each and every time
>>>
>>> 2. wso2base image can be hosted publicly (without Oracle JDK of course)
>>> so that users don't have to build wso2base but rather pull it from
>>> DockerHub.
>>>
>>> 3. wso2base image provides a Docker layer which reduces the amount of
>>> traffic carried when pulling images from a Docker registry over a long
>>> period of time. This is because base image is very unlikely to be changed,
>>> but product images can change very rapidly (due to artifacts or
>>> configuration change, product version etc.). It is unnecessary to carry the
>>> information in the lower level layers each time a product image is changed.
>>>
>>> If you must use a single image, I found this cool Docker tool written in
>>> Python which can be used to squash Docker layers called docker-squash [1].
>>> Following is a comparison of Docker images built with/without wso2base and
>>> squashed images;
>>>
>>> [image: Inline image 3]
>>>
>>> In summary;
>>> wso2am-1.10.0 with base image: 1.079GB
>>> wso2am-1.10.0 with base image and *squashed*: 948MB
>>> wso2am-1.10.0 *without* base image: 969MB
>>>
>>> wso2esb-4.9.0 with base image: 794MB
>>> wso2esb-4.9.0 with base image and 

Re: [Dev] WSO2 Docker images layered composition

[Akila Ravihansa Perera]

Hi Isuru,

Do you have any specific concerns about using a wso2base image? On average
there is a spike of 100MB when using a wso2base image, but that's a small
price to pay when considering benefits users would gain out of this. Note
that users can use "default" provisioning method to package any
jars/configs to the image. The main problem here is the time taken to run
apt-get update and package installation. Especially having to download
files over I'net every time a Docker image is built which I think is
unnecessary. wdyt?

Thanks.

On Wed, Jun 1, 2016 at 10:31 PM, Isuru Haththotuwa  wrote:

> Hi Akila and Imesh,
>
> Without creating a base image, if we package everything in a product
> specific image, host it in a local registry, and provide instructions to
> create a local Dockerfile extending from that image, the time taken to
> build the local image would be lesser. This will speedup the
> development/testing procedure even more. The developers can package any
> apps/jars/configs via the local Dockerfile (using volume mounts, etc.), for
> testing. I have seen this pattern used in some Docker images, for an
> example in the haproxy Docker image [1].
>
> One downside is, as you mentioned, we cannot push the image to Dockerhub
> with Oracle JDK distribution included. Maybe we can create the product
> Docker image with the relevant utilities (wget, zip, unzip, etc.) + the
> product distribution, and push it to Docker hub. Then provide an option to
> package the JDK via a local Dockerfile extending from the product
> Dockerfile.
>
> WDYT?
>
> [1]. https://hub.docker.com/_/haproxy/
>
> On Mon, May 30, 2016 at 12:48 PM, Akila Ravihansa Perera <
> raviha...@wso2.com> wrote:
>
>> Hi,
>>
>> Currently we don't have a base image for WSO2 Docker images due to image
>> size issues we had with it. But later we found out that this was mainly due
>> to multiple RUN commands, which caused Docker to add unnecessary layers.
>>
>> I think we should bring back the base image for couple of reasons;
>>
>> 1. wso2base image with minimum packages installed (wget, puppet etc) will
>> eliminate the need for having to run apt-get commands (which are very
>> expensive, time-wise) each and every time
>>
>> 2. wso2base image can be hosted publicly (without Oracle JDK of course)
>> so that users don't have to build wso2base but rather pull it from
>> DockerHub.
>>
>> 3. wso2base image provides a Docker layer which reduces the amount of
>> traffic carried when pulling images from a Docker registry over a long
>> period of time. This is because base image is very unlikely to be changed,
>> but product images can change very rapidly (due to artifacts or
>> configuration change, product version etc.). It is unnecessary to carry the
>> information in the lower level layers each time a product image is changed.
>>
>> If you must use a single image, I found this cool Docker tool written in
>> Python which can be used to squash Docker layers called docker-squash [1].
>> Following is a comparison of Docker images built with/without wso2base and
>> squashed images;
>>
>> [image: Inline image 3]
>>
>> In summary;
>> wso2am-1.10.0 with base image: 1.079GB
>> wso2am-1.10.0 with base image and *squashed*: 948MB
>> wso2am-1.10.0 *without* base image: 969MB
>>
>> wso2esb-4.9.0 with base image: 794MB
>> wso2esb-4.9.0 with base image and *squashed*: 664MB
>> wso2esb-4.9.0 *without* base image: 685MB
>>
>> In conclusion, introducing a base image with minimum packages has a image
>> size overhead of around 100+MB, but having individual images without a
>> hierarchy can cause a significant amount of unnecessary information being
>> transferred over the network when pull/push'ing to/from a registry.
>>
>> I've created a PR for this feature at [2]. Also used
>> "--no-install-recommends" option to apt-get when installing packages to
>> prevent unnecessary packages being installed. Please share your thoughts.
>>
>> [1] https://github.com/goldmann/docker-squash
>> [2] https://github.com/wso2/dockerfiles/pull/46/files
>>
>> Thanks.
>> --
>> Akila Ravihansa Perera
>> WSO2 Inc.;  http://wso2.com/
>>
>> Blog: http://ravihansa3000.blogspot.com
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* *
>
>
>


-- 
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Isuru Haththotuwa]

Hi Akila and Imesh,

Without creating a base image, if we package everything in a product
specific image, host it in a local registry, and provide instructions to
create a local Dockerfile extending from that image, the time taken to
build the local image would be lesser. This will speedup the
development/testing procedure even more. The developers can package any
apps/jars/configs via the local Dockerfile (using volume mounts, etc.), for
testing. I have seen this pattern used in some Docker images, for an
example in the haproxy Docker image [1].

One downside is, as you mentioned, we cannot push the image to Dockerhub
with Oracle JDK distribution included. Maybe we can create the product
Docker image with the relevant utilities (wget, zip, unzip, etc.) + the
product distribution, and push it to Docker hub. Then provide an option to
package the JDK via a local Dockerfile extending from the product
Dockerfile.

WDYT?

[1]. https://hub.docker.com/_/haproxy/

On Mon, May 30, 2016 at 12:48 PM, Akila Ravihansa Perera  wrote:

> Hi,
>
> Currently we don't have a base image for WSO2 Docker images due to image
> size issues we had with it. But later we found out that this was mainly due
> to multiple RUN commands, which caused Docker to add unnecessary layers.
>
> I think we should bring back the base image for couple of reasons;
>
> 1. wso2base image with minimum packages installed (wget, puppet etc) will
> eliminate the need for having to run apt-get commands (which are very
> expensive, time-wise) each and every time
>
> 2. wso2base image can be hosted publicly (without Oracle JDK of course) so
> that users don't have to build wso2base but rather pull it from DockerHub.
>
> 3. wso2base image provides a Docker layer which reduces the amount of
> traffic carried when pulling images from a Docker registry over a long
> period of time. This is because base image is very unlikely to be changed,
> but product images can change very rapidly (due to artifacts or
> configuration change, product version etc.). It is unnecessary to carry the
> information in the lower level layers each time a product image is changed.
>
> If you must use a single image, I found this cool Docker tool written in
> Python which can be used to squash Docker layers called docker-squash [1].
> Following is a comparison of Docker images built with/without wso2base and
> squashed images;
>
> [image: Inline image 3]
>
> In summary;
> wso2am-1.10.0 with base image: 1.079GB
> wso2am-1.10.0 with base image and *squashed*: 948MB
> wso2am-1.10.0 *without* base image: 969MB
>
> wso2esb-4.9.0 with base image: 794MB
> wso2esb-4.9.0 with base image and *squashed*: 664MB
> wso2esb-4.9.0 *without* base image: 685MB
>
> In conclusion, introducing a base image with minimum packages has a image
> size overhead of around 100+MB, but having individual images without a
> hierarchy can cause a significant amount of unnecessary information being
> transferred over the network when pull/push'ing to/from a registry.
>
> I've created a PR for this feature at [2]. Also used
> "--no-install-recommends" option to apt-get when installing packages to
> prevent unnecessary packages being installed. Please share your thoughts.
>
> [1] https://github.com/goldmann/docker-squash
> [2] https://github.com/wso2/dockerfiles/pull/46/files
>
> Thanks.
> --
> Akila Ravihansa Perera
> WSO2 Inc.;  http://wso2.com/
>
> Blog: http://ravihansa3000.blogspot.com
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
Thanks and Regards,

Isuru H.
+94 716 358 048* *
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] WSO2 Docker images layered composition

[Imesh Gunaratne]

+1 for the idea Akila!

I agree with all the points listed above, we experienced those during past
couple of weeks after removing the base image. Will include this change in
the next Dockerfiles release.

Thanks

On Mon, May 30, 2016 at 12:48 PM, Akila Ravihansa Perera  wrote:

> Hi,
>
> Currently we don't have a base image for WSO2 Docker images due to image
> size issues we had with it. But later we found out that this was mainly due
> to multiple RUN commands, which caused Docker to add unnecessary layers.
>
> I think we should bring back the base image for couple of reasons;
>
> 1. wso2base image with minimum packages installed (wget, puppet etc) will
> eliminate the need for having to run apt-get commands (which are very
> expensive, time-wise) each and every time
>
> 2. wso2base image can be hosted publicly (without Oracle JDK of course) so
> that users don't have to build wso2base but rather pull it from DockerHub.
>
> 3. wso2base image provides a Docker layer which reduces the amount of
> traffic carried when pulling images from a Docker registry over a long
> period of time. This is because base image is very unlikely to be changed,
> but product images can change very rapidly (due to artifacts or
> configuration change, product version etc.). It is unnecessary to carry the
> information in the lower level layers each time a product image is changed.
>
> If you must use a single image, I found this cool Docker tool written in
> Python which can be used to squash Docker layers called docker-squash [1].
> Following is a comparison of Docker images built with/without wso2base and
> squashed images;
>
> [image: Inline image 3]
>
> In summary;
> wso2am-1.10.0 with base image: 1.079GB
> wso2am-1.10.0 with base image and *squashed*: 948MB
> wso2am-1.10.0 *without* base image: 969MB
>
> wso2esb-4.9.0 with base image: 794MB
> wso2esb-4.9.0 with base image and *squashed*: 664MB
> wso2esb-4.9.0 *without* base image: 685MB
>
> In conclusion, introducing a base image with minimum packages has a image
> size overhead of around 100+MB, but having individual images without a
> hierarchy can cause a significant amount of unnecessary information being
> transferred over the network when pull/push'ing to/from a registry.
>
> I've created a PR for this feature at [2]. Also used
> "--no-install-recommends" option to apt-get when installing packages to
> prevent unnecessary packages being installed. Please share your thoughts.
>
> [1] https://github.com/goldmann/docker-squash
> [2] https://github.com/wso2/dockerfiles/pull/46/files
>
> Thanks.
> --
> Akila Ravihansa Perera
> WSO2 Inc.;  http://wso2.com/
>
> Blog: http://ravihansa3000.blogspot.com
>



-- 
*Imesh Gunaratne*
Software Architect
WSO2 Inc: http://wso2.com
T: +94 11 214 5345 M: +94 77 374 2057
W: https://medium.com/@imesh TW: @imesh
Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] WSO2 Docker images layered composition

[Akila Ravihansa Perera]

Hi,

Currently we don't have a base image for WSO2 Docker images due to image
size issues we had with it. But later we found out that this was mainly due
to multiple RUN commands, which caused Docker to add unnecessary layers.

I think we should bring back the base image for couple of reasons;

1. wso2base image with minimum packages installed (wget, puppet etc) will
eliminate the need for having to run apt-get commands (which are very
expensive, time-wise) each and every time

2. wso2base image can be hosted publicly (without Oracle JDK of course) so
that users don't have to build wso2base but rather pull it from DockerHub.

3. wso2base image provides a Docker layer which reduces the amount of
traffic carried when pulling images from a Docker registry over a long
period of time. This is because base image is very unlikely to be changed,
but product images can change very rapidly (due to artifacts or
configuration change, product version etc.). It is unnecessary to carry the
information in the lower level layers each time a product image is changed.

If you must use a single image, I found this cool Docker tool written in
Python which can be used to squash Docker layers called docker-squash [1].
Following is a comparison of Docker images built with/without wso2base and
squashed images;

[image: Inline image 3]

In summary;
wso2am-1.10.0 with base image: 1.079GB
wso2am-1.10.0 with base image and *squashed*: 948MB
wso2am-1.10.0 *without* base image: 969MB

wso2esb-4.9.0 with base image: 794MB
wso2esb-4.9.0 with base image and *squashed*: 664MB
wso2esb-4.9.0 *without* base image: 685MB

In conclusion, introducing a base image with minimum packages has a image
size overhead of around 100+MB, but having individual images without a
hierarchy can cause a significant amount of unnecessary information being
transferred over the network when pull/push'ing to/from a registry.

I've created a PR for this feature at [2]. Also used
"--no-install-recommends" option to apt-get when installing packages to
prevent unnecessary packages being installed. Please share your thoughts.

[1] https://github.com/goldmann/docker-squash
[2] https://github.com/wso2/dockerfiles/pull/46/files

Thanks.
-- 
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev