Re: OODT docker builds
I was thinking about the outputs. Currently everything is 1 docker file which is fine for some deployments and not for others. In the RADIX build we should also build individual containers for each component. For the deployment we could also have a docker-compose file and a K8S Helm setup so that you could deploy a distributed OODT setup from your RADIX output, this could also have the ZK stuff in it so we can properly utilise the distributed nature we started constructing with the FM ZK changes. On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha...@cse.mrt.ac.lk) wrote: Hi Tom, I think this will be great since people are adopting docker more and more and even for a user once they have built a customized docker image, they can share it among the peers reducing the time spent for configuration by each individual. Also we have another option ;-) With distributed configuration management which I implemented, users can ask OODT components to download configuration published in zookeeper. But this will require zookeeper to be running (either as a container or standalone). As per my understanding, configuration is the problem we need to solve when using a pre-built docker image? In future, if we are able to implement https://issues.apache.org/jira/browse/OODT-977 we will be able to run multiple file managers in multiple containers which will allow the load to be distributed and query all at once. So, docker will be the way to go as I see it. What do you think? Thanks, Imesha On Sun, 14 Oct 2018 at 15:40, Tom Barber wrote: > I’m interested in the Dockerization of OODT but also conscious that most > people use RADIX to build their stuff, which make’s overriding bits of a > prebuilt image tricky. > > I’m wondering if its worth adding an optional Docker profile to RADIX to > add a Docker build step to the backend of people’s RADIX builds if they so > wanted. > > Thoughts? > > -- > > > Spicule Limited is registered in England & Wales. Company Number: > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > > > All engagements > are subject to Spicule Terms and Conditions of Business. This email and > its > contents are intended solely for the individual to whom it is addressed > and > may contain information that is confidential, privileged or otherwise > protected from disclosure, distributing or copying. Any views or opinions > presented in this email are solely those of the author and do not > necessarily represent those of Spicule Limited. The company accepts no > liability for any damage caused by any virus transmitted by this email. If > you have received this message in error, please notify us immediately by > reply email before deleting it from your system. Service of legal notice > cannot be effected on Spicule Limited by email. > -- Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891. All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidential, privileged or otherwise protected from disclosure, distributing or copying. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Spicule Limited. The company accepts no liability for any damage caused by any virus transmitted by this email. If you have received this message in error, please notify us immediately by reply email before deleting it from your system. Service of legal notice cannot be effected on Spicule Limited by email.
Re: OODT docker builds
Yes. Agree with you. That is something I have been planning to ask you for long; why aren't we doing so. I like the idea of having a docker image per component and as you suggested we can create docker-compose or kubernetes setup for deployments. I like that direction ;-) As an starting point, we can add an all-in-one docker image to be built in the RADIX build, right? If you start off, I will be able to join you along the way. Thanks, Imesha On Mon, 15 Oct 2018 at 16:33, Tom Barber wrote: > I was thinking about the outputs. Currently everything is 1 docker file > which is fine for some deployments and not for others. > > In the RADIX build we should also build individual containers for each > component. For the deployment we could also have a docker-compose file and > a K8S Helm setup so that you could deploy a distributed OODT setup from > your RADIX output, this could also have the ZK stuff in it so we can > properly utilise the distributed nature we started constructing with the FM > ZK changes. > > > On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha...@cse.mrt.ac.lk) > wrote: > > Hi Tom, > > I think this will be great since people are adopting docker more and more > and even for a user > once they have built a customized docker image, they can share it among > the > peers > reducing the time spent for configuration by each individual. > > Also we have another option ;-) With distributed configuration management > which I implemented, > users can ask OODT components to download configuration published in > zookeeper. But this will > require zookeeper to be running (either as a container or standalone). As > per my understanding, > configuration is the problem we need to solve when using a pre-built > docker > image? > > In future, if we are able to implement > https://issues.apache.org/jira/browse/OODT-977 > we will be able to run multiple file managers in multiple containers which > will allow > the load to be distributed and query all at once. So, docker will be the > way to go as I see it. > What do you think? > > Thanks, > Imesha > > > On Sun, 14 Oct 2018 at 15:40, Tom Barber wrote: > > > I’m interested in the Dockerization of OODT but also conscious that most > > people use RADIX to build their stuff, which make’s overriding bits of a > > prebuilt image tricky. > > > > I’m wondering if its worth adding an optional Docker profile to RADIX to > > add a Docker build step to the backend of people’s RADIX builds if they > so > > wanted. > > > > Thoughts? > > > > -- > > > > > > Spicule Limited is registered in England & Wales. Company Number: > > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > > > > > > > > All engagements > > are subject to Spicule Terms and Conditions of Business. This email and > > its > > contents are intended solely for the individual to whom it is addressed > > and > > may contain information that is confidential, privileged or otherwise > > protected from disclosure, distributing or copying. Any views or > opinions > > presented in this email are solely those of the author and do not > > necessarily represent those of Spicule Limited. The company accepts no > > liability for any damage caused by any virus transmitted by this email. > If > > you have received this message in error, please notify us immediately by > > reply email before deleting it from your system. Service of legal notice > > cannot be effected on Spicule Limited by email. > > > > > Spicule Limited is registered in England & Wales. Company Number: > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > All engagements are subject to Spicule Terms and Conditions of Business. > This email and its contents are intended solely for the individual to whom > it is addressed and may contain information that is confidential, > privileged or otherwise protected from disclosure, distributing or copying. > Any views or opinions presented in this email are solely those of the > author and do not necessarily represent those of Spicule Limited. The > company accepts no liability for any damage caused by any virus transmitted > by this email. If you have received this message in error, please notify us > immediately by reply email before deleting it from your system. Service of > legal notice cannot be effected on Spicule Limited by email. >
Re: OODT docker builds
Why aren’t we doing so?! :) Lack of cycles and young kids ;) I’ll take a stab at it and see where we get to outside of RADIX to get the stack in distinct containers and then we’ll look at integrating it into the main build then. Tom On 15 October 2018 at 13:07:56, Imesha Sudasingha (imesha...@cse.mrt.ac.lk) wrote: Yes. Agree with you. That is something I have been planning to ask you for long; why aren't we doing so. I like the idea of having a docker image per component and as you suggested we can create docker-compose or kubernetes setup for deployments. I like that direction ;-) As an starting point, we can add an all-in-one docker image to be built in the RADIX build, right? If you start off, I will be able to join you along the way. Thanks, Imesha On Mon, 15 Oct 2018 at 16:33, Tom Barber wrote: > I was thinking about the outputs. Currently everything is 1 docker file > which is fine for some deployments and not for others. > > In the RADIX build we should also build individual containers for each > component. For the deployment we could also have a docker-compose file and > a K8S Helm setup so that you could deploy a distributed OODT setup from > your RADIX output, this could also have the ZK stuff in it so we can > properly utilise the distributed nature we started constructing with the FM > ZK changes. > > > On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha...@cse.mrt.ac.lk) > wrote: > > Hi Tom, > > I think this will be great since people are adopting docker more and more > and even for a user > once they have built a customized docker image, they can share it among the > peers > reducing the time spent for configuration by each individual. > > Also we have another option ;-) With distributed configuration management > which I implemented, > users can ask OODT components to download configuration published in > zookeeper. But this will > require zookeeper to be running (either as a container or standalone). As > per my understanding, > configuration is the problem we need to solve when using a pre-built docker > image? > > In future, if we are able to implement > https://issues.apache.org/jira/browse/OODT-977 > we will be able to run multiple file managers in multiple containers which > will allow > the load to be distributed and query all at once. So, docker will be the > way to go as I see it. > What do you think? > > Thanks, > Imesha > > > On Sun, 14 Oct 2018 at 15:40, Tom Barber wrote: > > > I’m interested in the Dockerization of OODT but also conscious that most > > people use RADIX to build their stuff, which make’s overriding bits of a > > prebuilt image tricky. > > > > I’m wondering if its worth adding an optional Docker profile to RADIX to > > add a Docker build step to the backend of people’s RADIX builds if they > so > > wanted. > > > > Thoughts? > > > > -- > > > > > > Spicule Limited is registered in England & Wales. Company Number: > > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > > > > > > > > All engagements > > are subject to Spicule Terms and Conditions of Business. This email and > > its > > contents are intended solely for the individual to whom it is addressed > > and > > may contain information that is confidential, privileged or otherwise > > protected from disclosure, distributing or copying. Any views or opinions > > presented in this email are solely those of the author and do not > > necessarily represent those of Spicule Limited. The company accepts no > > liability for any damage caused by any virus transmitted by this email. > If > > you have received this message in error, please notify us immediately by > > reply email before deleting it from your system. Service of legal notice > > cannot be effected on Spicule Limited by email. > > > > > Spicule Limited is registered in England & Wales. Company Number: > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > All engagements are subject to Spicule Terms and Conditions of Business. > This email and its contents are intended solely for the individual to whom > it is addressed and may contain information that is confidential, > privileged or otherwise protected from disclosure, distributing or copying. > Any views or opinions presented in this email are solely those of the > author and do not necessarily represent those of Spicule Limited. The > company accepts no liability for any damage caused by any virus transmitted > by this email. If you have received this message in error, please notify us > immediately by reply email before deleting it from your system. Service of > legal notice cannot be effected on Spicule Limited by email. > -- Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 25147889
Re: OODT docker builds
Isn’t an image per component really heavyweight? From: Tom Barber Date: Monday, October 15, 2018 at 8:26 AM To: Imesha Sudasingha Cc: dev , Chris Mattmann Subject: Re: OODT docker builds Why aren’t we doing so?! :) Lack of cycles and young kids ;) I’ll take a stab at it and see where we get to outside of RADIX to get the stack in distinct containers and then we’ll look at integrating it into the main build then. Tom On 15 October 2018 at 13:07:56, Imesha Sudasingha (imesha...@cse.mrt.ac.lk) wrote: Yes. Agree with you. That is something I have been planning to ask you for long; why aren't we doing so. I like the idea of having a docker image per component and as you suggested we can create docker-compose or kubernetes setup for deployments. I like that direction ;-) As an starting point, we can add an all-in-one docker image to be built in the RADIX build, right? If you start off, I will be able to join you along the way. Thanks, Imesha On Mon, 15 Oct 2018 at 16:33, Tom Barber wrote: I was thinking about the outputs. Currently everything is 1 docker file which is fine for some deployments and not for others. In the RADIX build we should also build individual containers for each component. For the deployment we could also have a docker-compose file and a K8S Helm setup so that you could deploy a distributed OODT setup from your RADIX output, this could also have the ZK stuff in it so we can properly utilise the distributed nature we started constructing with the FM ZK changes. On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha...@cse.mrt.ac.lk) wrote: Hi Tom, I think this will be great since people are adopting docker more and more and even for a user once they have built a customized docker image, they can share it among the peers reducing the time spent for configuration by each individual. Also we have another option ;-) With distributed configuration management which I implemented, users can ask OODT components to download configuration published in zookeeper. But this will require zookeeper to be running (either as a container or standalone). As per my understanding, configuration is the problem we need to solve when using a pre-built docker image? In future, if we are able to implement https://issues.apache.org/jira/browse/OODT-977 we will be able to run multiple file managers in multiple containers which will allow the load to be distributed and query all at once. So, docker will be the way to go as I see it. What do you think? Thanks, Imesha On Sun, 14 Oct 2018 at 15:40, Tom Barber wrote: > I’m interested in the Dockerization of OODT but also conscious that most > people use RADIX to build their stuff, which make’s overriding bits of a > prebuilt image tricky. > > I’m wondering if its worth adding an optional Docker profile to RADIX to > add a Docker build step to the backend of people’s RADIX builds if they so > wanted. > > Thoughts? > > -- > > > Spicule Limited is registered in England & Wales. Company Number: > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > > > All engagements > are subject to Spicule Terms and Conditions of Business. This email and > its > contents are intended solely for the individual to whom it is addressed > and > may contain information that is confidential, privileged or otherwise > protected from disclosure, distributing or copying. Any views or opinions > presented in this email are solely those of the author and do not > necessarily represent those of Spicule Limited. The company accepts no > liability for any damage caused by any virus transmitted by this email. If > you have received this message in error, please notify us immediately by > reply email before deleting it from your system. Service of legal notice > cannot be effected on Spicule Limited by email. > Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891. All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidential, privileged or otherwise protected from disclosure, distributing or copying. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Spicule Limited. The company accepts no liability for any damage caused by any virus transmitted by this email. If you have received this message in error, please notify us immediately by reply email before deleting it from your system. Service of legal notice cannot be effected on Spicule Limited by email. Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor
Re: OODT docker builds
Not at all and if you want to run OODT on Kubernetes for example, that would be how you’d do it, that way you can upgrade, scale, restart and fail components without the entire stack falling over. In terms of disk space, don’t forget each image is built on layers, so for example, openjdk-8 on alpine is 56mb, that layer would then be used across all base images so its only installed once on each host, then you’ve got your file manager for example which would check in currently at 62MB, so the entire image size would be 118MB which you could then deploy on 1 node, or 100 nodes. Then say you’ve got opsui as another dependency, that would be tomcat:8 (463mb)+ opsui(73mb)==536mb But say you have no interest in workflow etc, thats all you’d deploy. In reality it would be much more flexible and much more inline with how docker containers should be deployed, which is as a single process container not as a bunch of processes all stuck into 1 unit. On 15 October 2018 at 17:32:32, Chris Mattmann (mattm...@apache.org) wrote: Isn’t an image per component really heavyweight? *From: *Tom Barber *Date: *Monday, October 15, 2018 at 8:26 AM *To: *Imesha Sudasingha *Cc: *dev , Chris Mattmann *Subject: *Re: OODT docker builds Why aren’t we doing so?! :) Lack of cycles and young kids ;) I’ll take a stab at it and see where we get to outside of RADIX to get the stack in distinct containers and then we’ll look at integrating it into the main build then. Tom On 15 October 2018 at 13:07:56, Imesha Sudasingha (imesha...@cse.mrt.ac.lk) wrote: Yes. Agree with you. That is something I have been planning to ask you for long; why aren't we doing so. I like the idea of having a docker image per component and as you suggested we can create docker-compose or kubernetes setup for deployments. I like that direction ;-) As an starting point, we can add an all-in-one docker image to be built in the RADIX build, right? If you start off, I will be able to join you along the way. Thanks, Imesha On Mon, 15 Oct 2018 at 16:33, Tom Barber wrote: I was thinking about the outputs. Currently everything is 1 docker file which is fine for some deployments and not for others. In the RADIX build we should also build individual containers for each component. For the deployment we could also have a docker-compose file and a K8S Helm setup so that you could deploy a distributed OODT setup from your RADIX output, this could also have the ZK stuff in it so we can properly utilise the distributed nature we started constructing with the FM ZK changes. On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha...@cse.mrt.ac.lk) wrote: Hi Tom, I think this will be great since people are adopting docker more and more and even for a user once they have built a customized docker image, they can share it among the peers reducing the time spent for configuration by each individual. Also we have another option ;-) With distributed configuration management which I implemented, users can ask OODT components to download configuration published in zookeeper. But this will require zookeeper to be running (either as a container or standalone). As per my understanding, configuration is the problem we need to solve when using a pre-built docker image? In future, if we are able to implement https://issues.apache.org/jira/browse/OODT-977 we will be able to run multiple file managers in multiple containers which will allow the load to be distributed and query all at once. So, docker will be the way to go as I see it. What do you think? Thanks, Imesha On Sun, 14 Oct 2018 at 15:40, Tom Barber wrote: > I’m interested in the Dockerization of OODT but also conscious that most > people use RADIX to build their stuff, which make’s overriding bits of a > prebuilt image tricky. > > I’m wondering if its worth adding an optional Docker profile to RADIX to > add a Docker build step to the backend of people’s RADIX builds if they so > wanted. > > Thoughts? > > -- > > > Spicule Limited is registered in England & Wales. Company Number: > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > > > All engagements > are subject to Spicule Terms and Conditions of Business. This email and > its > contents are intended solely for the individual to whom it is addressed > and > may contain information that is confidential, privileged or otherwise > protected from disclosure, distributing or copying. Any views or opinions > presented in this email are solely those of the author and do not > necessarily represent those of Spicule Limited. The company accepts no > liability for any damage caused by any virus transmitted by this email. If > you have received this message in error, please notify us immediately by > reply email before deleting it from your system. Service of legal notice > cannot be effected on Spicule Limited by email. > Spicule Limited is
Re: OODT docker builds
Also don’t forget, Docker containers, don’t ship Kernel’s etc and all the system stuff, which is what keeps them lightweight. On 15 October 2018 at 17:43:37, Tom Barber (t...@spicule.co.uk) wrote: Not at all and if you want to run OODT on Kubernetes for example, that would be how you’d do it, that way you can upgrade, scale, restart and fail components without the entire stack falling over. In terms of disk space, don’t forget each image is built on layers, so for example, openjdk-8 on alpine is 56mb, that layer would then be used across all base images so its only installed once on each host, then you’ve got your file manager for example which would check in currently at 62MB, so the entire image size would be 118MB which you could then deploy on 1 node, or 100 nodes. Then say you’ve got opsui as another dependency, that would be tomcat:8 (463mb)+ opsui(73mb)==536mb But say you have no interest in workflow etc, thats all you’d deploy. In reality it would be much more flexible and much more inline with how docker containers should be deployed, which is as a single process container not as a bunch of processes all stuck into 1 unit. On 15 October 2018 at 17:32:32, Chris Mattmann (mattm...@apache.org) wrote: Isn’t an image per component really heavyweight? *From:* Tom Barber *Date:* Monday, October 15, 2018 at 8:26 AM *To:* Imesha Sudasingha *Cc:* dev , Chris Mattmann *Subject:* Re: OODT docker builds Why aren’t we doing so?! :) Lack of cycles and young kids ;) I’ll take a stab at it and see where we get to outside of RADIX to get the stack in distinct containers and then we’ll look at integrating it into the main build then. Tom On 15 October 2018 at 13:07:56, Imesha Sudasingha (imesha...@cse.mrt.ac.lk) wrote: Yes. Agree with you. That is something I have been planning to ask you for long; why aren't we doing so. I like the idea of having a docker image per component and as you suggested we can create docker-compose or kubernetes setup for deployments. I like that direction ;-) As an starting point, we can add an all-in-one docker image to be built in the RADIX build, right? If you start off, I will be able to join you along the way. Thanks, Imesha On Mon, 15 Oct 2018 at 16:33, Tom Barber wrote: I was thinking about the outputs. Currently everything is 1 docker file which is fine for some deployments and not for others. In the RADIX build we should also build individual containers for each component. For the deployment we could also have a docker-compose file and a K8S Helm setup so that you could deploy a distributed OODT setup from your RADIX output, this could also have the ZK stuff in it so we can properly utilise the distributed nature we started constructing with the FM ZK changes. On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha...@cse.mrt.ac.lk) wrote: Hi Tom, I think this will be great since people are adopting docker more and more and even for a user once they have built a customized docker image, they can share it among the peers reducing the time spent for configuration by each individual. Also we have another option ;-) With distributed configuration management which I implemented, users can ask OODT components to download configuration published in zookeeper. But this will require zookeeper to be running (either as a container or standalone). As per my understanding, configuration is the problem we need to solve when using a pre-built docker image? In future, if we are able to implement https://issues.apache.org/jira/browse/OODT-977 we will be able to run multiple file managers in multiple containers which will allow the load to be distributed and query all at once. So, docker will be the way to go as I see it. What do you think? Thanks, Imesha On Sun, 14 Oct 2018 at 15:40, Tom Barber wrote: > I’m interested in the Dockerization of OODT but also conscious that most > people use RADIX to build their stuff, which make’s overriding bits of a > prebuilt image tricky. > > I’m wondering if its worth adding an optional Docker profile to RADIX to > add a Docker build step to the backend of people’s RADIX builds if they so > wanted. > > Thoughts? > > -- > > > Spicule Limited is registered in England & Wales. Company Number: > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston > Road, Brighton, England, BN1 6AF. VAT No. 251478891. > > > > > All engagements > are subject to Spicule Terms and Conditions of Business. This email and > its > contents are intended solely for the individual to whom it is addressed > and > may contain information that is confidential, privileged or otherwise > protected from disclosure, distributing or copying. Any views or opinions > presented in this email are solely those of the author and do not > necessarily represent those of Spicule Limited. The company accepts no > liability for any damage caused by any virus transmitted by this email. If > you have received thi
Re: OODT docker builds
Interesting OK. B/c the more containers for me on my local laptop typically eat up way more memory and CPU…but maybe that’s just me on a Mac lol From: Tom Barber Reply-To: "dev@oodt.apache.org" Date: Monday, October 15, 2018 at 9:44 AM To: Imesha Sudasingha , Chris Mattmann Cc: dev Subject: Re: OODT docker builds Not at all and if you want to run OODT on Kubernetes for example, that would be how you’d do it, that way you can upgrade, scale, restart and fail components without the entire stack falling over. In terms of disk space, don’t forget each image is built on layers, so for example, openjdk-8 on alpine is 56mb, that layer would then be used across all base images so its only installed once on each host, then you’ve got your file manager for example which would check in currently at 62MB, so the entire image size would be 118MB which you could then deploy on 1 node, or 100 nodes. Then say you’ve got opsui as another dependency, that would be tomcat:8 (463mb)+ opsui(73mb)==536mb But say you have no interest in workflow etc, thats all you’d deploy. In reality it would be much more flexible and much more inline with how docker containers should be deployed, which is as a single process container not as a bunch of processes all stuck into 1 unit. On 15 October 2018 at 17:32:32, Chris Mattmann (mattm...@apache.org) wrote: Isn’t an image per component really heavyweight? *From: *Tom Barber *Date: *Monday, October 15, 2018 at 8:26 AM *To: *Imesha Sudasingha *Cc: *dev , Chris Mattmann *Subject: *Re: OODT docker builds Why aren’t we doing so?! :) Lack of cycles and young kids ;) I’ll take a stab at it and see where we get to outside of RADIX to get the stack in distinct containers and then we’ll look at integrating it into the main build then. Tom On 15 October 2018 at 13:07:56, Imesha Sudasingha (imesha...@cse.mrt.ac.lk) wrote: Yes. Agree with you. That is something I have been planning to ask you for long; why aren't we doing so. I like the idea of having a docker image per component and as you suggested we can create docker-compose or kubernetes setup for deployments. I like that direction ;-) As an starting point, we can add an all-in-one docker image to be built in the RADIX build, right? If you start off, I will be able to join you along the way. Thanks, Imesha On Mon, 15 Oct 2018 at 16:33, Tom Barber wrote: I was thinking about the outputs. Currently everything is 1 docker file which is fine for some deployments and not for others. In the RADIX build we should also build individual containers for each component. For the deployment we could also have a docker-compose file and a K8S Helm setup so that you could deploy a distributed OODT setup from your RADIX output, this could also have the ZK stuff in it so we can properly utilise the distributed nature we started constructing with the FM ZK changes. On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha...@cse.mrt.ac.lk) wrote: Hi Tom, I think this will be great since people are adopting docker more and more and even for a user once they have built a customized docker image, they can share it among the peers reducing the time spent for configuration by each individual. Also we have another option ;-) With distributed configuration management which I implemented, users can ask OODT components to download configuration published in zookeeper. But this will require zookeeper to be running (either as a container or standalone). As per my understanding, configuration is the problem we need to solve when using a pre-built docker image? In future, if we are able to implement https://issues.apache.org/jira/browse/OODT-977 we will be able to run multiple file managers in multiple containers which will allow the load to be distributed and query all at once. So, docker will be the way to go as I see it. What do you think? Thanks, Imesha On Sun, 14 Oct 2018 at 15:40, Tom Barber wrote: I’m interested in the Dockerization of OODT but also conscious that most people use RADIX to build their stuff, which make’s overriding bits of a prebuilt image tricky. I’m wondering if its worth adding an optional Docker profile to RADIX to add a Docker build step to the backend of people’s RADIX builds if they so wanted. Thoughts? -- Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891. All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidentia
Re: OODT docker builds
internally as they only 1 process they should run with very little overhead. I can vouch also though that IO certainly sucks really really badly on my Mac compared to linux box, and if SK reads this I'm sure he can also relay some docker horror stories on Mac. On Mac it's better since they got rid of the virtualbox bridge, but it's still pretty bad. Tom On Mon, 15 Oct 2018, 19:20 Chris Mattmann, wrote: > Interesting OK. B/c the more containers for me on my local laptop > typically eat up way more memory and CPU…but maybe that’s just > me on a Mac lol > > > > > > > > > > From: Tom Barber > Reply-To: "dev@oodt.apache.org" > Date: Monday, October 15, 2018 at 9:44 AM > To: Imesha Sudasingha , Chris Mattmann < > mattm...@apache.org> > Cc: dev > Subject: Re: OODT docker builds > > > > Not at all and if you want to run OODT on Kubernetes for example, that > > would be how you’d do it, that way you can upgrade, scale, restart and fail > > components without the entire stack falling over. > > > > In terms of disk space, don’t forget each image is built on layers, so for > > example, openjdk-8 on alpine is 56mb, that layer would then be used across > > all base images so its only installed once on each host, then you’ve got > > your file manager for example which would check in currently at 62MB, so > > the entire image size would be 118MB which you could then deploy on 1 node, > > or 100 nodes. > > > > Then say you’ve got opsui as another dependency, that would be > > > > tomcat:8 (463mb)+ opsui(73mb)==536mb > > > > But say you have no interest in workflow etc, thats all you’d deploy. > > > > In reality it would be much more flexible and much more inline with how > > docker containers should be deployed, which is as a single process > > container not as a bunch of processes all stuck into 1 unit. > > > > > > > > > > > > On 15 October 2018 at 17:32:32, Chris Mattmann (mattm...@apache.org) > wrote: > > > > Isn’t an image per component really heavyweight? > > > > > > > > > > > > > > > > *From: *Tom Barber > > *Date: *Monday, October 15, 2018 at 8:26 AM > > *To: *Imesha Sudasingha > > *Cc: *dev , Chris Mattmann > > *Subject: *Re: OODT docker builds > > > > > > > > Why aren’t we doing so?! :) > > > > > > > > Lack of cycles and young kids ;) > > > > > > > > I’ll take a stab at it and see where we get to outside of RADIX to get the > > stack in distinct containers and then we’ll look at integrating it into the > > main build then. > > > > > > > > > > > > Tom > > > > > > > > On 15 October 2018 at 13:07:56, Imesha Sudasingha (imesha...@cse.mrt.ac.lk > ) > > wrote: > > > > Yes. Agree with you. That is something I have been planning to ask you for > > long; why aren't we doing so. > > > > I like the idea of having a docker image per component and as you suggested > > we can create docker-compose > > > > or kubernetes setup for deployments. I like that direction ;-) > > > > > > > > As an starting point, we can add an all-in-one docker image to be built in > > the RADIX build, right? > > > > If you start off, I will be able to join you along the way. > > > > > > > > Thanks, > > > > Imesha > > > > > > > > > > > > > > > > On Mon, 15 Oct 2018 at 16:33, Tom Barber wrote: > > > > I was thinking about the outputs. Currently everything is 1 docker file > > which is fine for some deployments and not for others. > > > > > > > > In the RADIX build we should also build individual containers for each > > component. For the deployment we could also have a docker-compose file and > > a K8S Helm setup so that you could deploy a distributed OODT setup from > > your RADIX output, this could also have the ZK stuff in it so we can > > properly utilise the distributed nature we started constructing with the FM > > ZK changes. > > > > > > > > > > > > On 15 October 2018 at 05:08:41, Imesha Sudasingha (imesha...@cse.mrt.ac.lk > ) > > wrote: > > > > Hi Tom, > > > > I think this will be great since people are adopting docker more and more > > and even for a user > > once they have built a customized docker image, they can share it among the > > peers > > reducing the time spent for configuration by each individual. > > > > Also we have another option ;-) With distributed configuration management > > which I implemented, > > users can ask OODT components to download configuration published in > > zookeeper. But this will > > require zookeeper to be running (either as a container or standalone). As > > per my understanding, > > configuration is the problem we need to solve when using a pre-built docker > > image? > > > > In future, if we are able to implement > > https://issues.apache.org/jira/browse/OODT-977 > > we will be able to run multiple file managers in multiple containers which > > will allow > > the load to be distributed and query all at once. So, docker will be the > > way to go as I see it. > > What do you think? > > > > Thanks, > > Imesha > > > > > > On Sun, 14 Oct 2018 at 15:40, Tom Barber wrote: > > > > I’m interested in
[jira] [Created] (OODT-997) JAXRS endpoints broken
Tom Barber created OODT-997: --- Summary: JAXRS endpoints broken Key: OODT-997 URL: https://issues.apache.org/jira/browse/OODT-997 Project: OODT Issue Type: Bug Reporter: Tom Barber Assignee: Tom Barber Calling something like: :8080/fmprod/jaxrs/product?productID=ad4b2df4-d0c5-11e8-90f6-ad8c0b53f3f7 Throws: javax.servlet.ServletException: No Resource class org.apache.cxf.jaxrs.provider.json.JSONProvider, can be found org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.loadClass(CXFNonSpringJaxrsServlet.java:392) org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.loadClass(CXFNonSpringJaxrsServlet.java:385) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [jira] [Created] (OODT-997) JAXRS endpoints broken
You’re missing an env var, see here: https://github.com/apache/drat/wiki/Installation#bash FMPROD* Cheers, Chris From: "Tom Barber (JIRA)" Reply-To: "dev@oodt.apache.org" Date: Monday, October 15, 2018 at 3:12 PM To: "dev@oodt.apache.org" Subject: [jira] [Created] (OODT-997) JAXRS endpoints broken Tom Barber created OODT-997: --- Summary: JAXRS endpoints broken Key: OODT-997 URL: https://issues.apache.org/jira/browse/OODT-997 Project: OODT Issue Type: Bug Reporter: Tom Barber Assignee: Tom Barber Calling something like: :8080/fmprod/jaxrs/product?productID=ad4b2df4-d0c5-11e8-90f6-ad8c0b53f3f7 Throws: javax.servlet.ServletException: No Resource class org.apache.cxf.jaxrs.provider.json.JSONProvider, can be found org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.loadClass(CXFNonSpringJaxrsServlet.java:392) org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.loadClass(CXFNonSpringJaxrsServlet.java:385) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [jira] [Created] (OODT-997) JAXRS endpoints broken
Hmm its a ClassNotFound exception. If I update CXF to the latest 3.2.6 it works without any changes to my deployment. On 15 October 2018 at 23:13:34, Chris Mattmann (mattm...@apache.org) wrote: You’re missing an env var, see here: https://github.com/apache/drat/wiki/Installation#bash FMPROD* Cheers, Chris From: "Tom Barber (JIRA)" Reply-To: "dev@oodt.apache.org" Date: Monday, October 15, 2018 at 3:12 PM To: "dev@oodt.apache.org" Subject: [jira] [Created] (OODT-997) JAXRS endpoints broken Tom Barber created OODT-997: --- Summary: JAXRS endpoints broken Key: OODT-997 URL: https://issues.apache.org/jira/browse/OODT-997 Project: OODT Issue Type: Bug Reporter: Tom Barber Assignee: Tom Barber Calling something like: :8080/fmprod/jaxrs/product?productID=ad4b2df4-d0c5-11e8-90f6-ad8c0b53f3f7 Throws: javax.servlet.ServletException: No Resource class org.apache.cxf.jaxrs.provider.json.JSONProvider, can be found org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.loadClass(CXFNonSpringJaxrsServlet.java:392) org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.loadClass(CXFNonSpringJaxrsServlet.java:385) -- This message was sent by Atlassian JIRA (v7.6.3#76005) -- Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891. All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidential, privileged or otherwise protected from disclosure, distributing or copying. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Spicule Limited. The company accepts no liability for any damage caused by any virus transmitted by this email. If you have received this message in error, please notify us immediately by reply email before deleting it from your system. Service of legal notice cannot be effected on Spicule Limited by email.
React based OPSUI
Hi folks, I’m mocking and stubbing out ideas for a React based OPSUI. Its not designed to reinvent the wheel, but modernise, allow easier theming/branding, be modular for people wanting to embed specific components into apps and so on. I have plans in my head for a couple of extra screens for file management, workflow management and I’ve still got work to do on the mocks, but here’s a Balsamiq link for people to start to mull over: https://balsamiq.cloud/sgp43nx/pdsiadc If you were designing OPSUI again, whats missing? Anything? Nothing, just needs to throw less stack traces? ;) Cheers Tom -- Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891. All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidential, privileged or otherwise protected from disclosure, distributing or copying. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Spicule Limited. The company accepts no liability for any damage caused by any virus transmitted by this email. If you have received this message in error, please notify us immediately by reply email before deleting it from your system. Service of legal notice cannot be effected on Spicule Limited by email.
Re: [jira] [Created] (OODT-997) JAXRS endpoints broken
Ahh, good to hear. Please fix…why would this break though since it worked before? From: Tom Barber Reply-To: "dev@oodt.apache.org" Date: Monday, October 15, 2018 at 4:40 PM To: "dev@oodt.apache.org" , Chris Mattmann Subject: Re: [jira] [Created] (OODT-997) JAXRS endpoints broken Hmm its a ClassNotFound exception. If I update CXF to the latest 3.2.6 it works without any changes to my deployment. On 15 October 2018 at 23:13:34, Chris Mattmann (mattm...@apache.org) wrote: You’re missing an env var, see here: https://github.com/apache/drat/wiki/Installation#bash FMPROD* Cheers, Chris From: "Tom Barber (JIRA)" Reply-To: "dev@oodt.apache.org" Date: Monday, October 15, 2018 at 3:12 PM To: "dev@oodt.apache.org" Subject: [jira] [Created] (OODT-997) JAXRS endpoints broken Tom Barber created OODT-997: --- Summary: JAXRS endpoints broken Key: OODT-997 URL: https://issues.apache.org/jira/browse/OODT-997 Project: OODT Issue Type: Bug Reporter: Tom Barber Assignee: Tom Barber Calling something like: :8080/fmprod/jaxrs/product?productID=ad4b2df4-d0c5-11e8-90f6-ad8c0b53f3f7 Throws: javax.servlet.ServletException: No Resource class org.apache.cxf.jaxrs.provider.json.JSONProvider, can be found org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.loadClass(CXFNonSpringJaxrsServlet.java:392) org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.loadClass(CXFNonSpringJaxrsServlet.java:385) -- This message was sent by Atlassian JIRA (v7.6.3#76005) -- Spicule Limited is registered in England & Wales. Company Number: 09954122. Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, England, BN1 6AF. VAT No. 251478891. All engagements are subject to Spicule Terms and Conditions of Business. This email and its contents are intended solely for the individual to whom it is addressed and may contain information that is confidential, privileged or otherwise protected from disclosure, distributing or copying. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Spicule Limited. The company accepts no liability for any damage caused by any virus transmitted by this email. If you have received this message in error, please notify us immediately by reply email before deleting it from your system. Service of legal notice cannot be effected on Spicule Limited by email.
Re: OODT docker builds
Nice explanation Tom. In addition to that, we will be able to run these containers in multiple pods within a kubernetes cluster when being used for an actual use case (production). This way, it will be easy to deploy, scale and manage. Under those conditions, we may be able to run one component per node as well with Kube-DNS is there to avoid hardcoding IPs and allow service discovery without a problem. On Mon, 15 Oct 2018 at 23:55, Tom Barber wrote: > internally as they only 1 process they should run with very little > overhead. > > > I can vouch also though that IO certainly sucks really really badly on my > Mac compared to linux box, and if SK reads this I'm sure he can also relay > some docker horror stories on Mac. > > On Mac it's better since they got rid of the virtualbox bridge, but it's > still pretty bad. > > Tom > > On Mon, 15 Oct 2018, 19:20 Chris Mattmann, wrote: > > > Interesting OK. B/c the more containers for me on my local laptop > > typically eat up way more memory and CPU…but maybe that’s just > > me on a Mac lol > > > > > > > > > > > > > > > > > > > > From: Tom Barber > > Reply-To: "dev@oodt.apache.org" > > Date: Monday, October 15, 2018 at 9:44 AM > > To: Imesha Sudasingha , Chris Mattmann < > > mattm...@apache.org> > > Cc: dev > > Subject: Re: OODT docker builds > > > > > > > > Not at all and if you want to run OODT on Kubernetes for example, that > > > > would be how you’d do it, that way you can upgrade, scale, restart and > fail > > > > components without the entire stack falling over. > > > > > > > > In terms of disk space, don’t forget each image is built on layers, so > for > > > > example, openjdk-8 on alpine is 56mb, that layer would then be used > across > > > > all base images so its only installed once on each host, then you’ve got > > > > your file manager for example which would check in currently at 62MB, so > > > > the entire image size would be 118MB which you could then deploy on 1 > node, > > > > or 100 nodes. > > > > > > > > Then say you’ve got opsui as another dependency, that would be > > > > > > > > tomcat:8 (463mb)+ opsui(73mb)==536mb > > > > > > > > But say you have no interest in workflow etc, thats all you’d deploy. > > > > > > > > In reality it would be much more flexible and much more inline with how > > > > docker containers should be deployed, which is as a single process > > > > container not as a bunch of processes all stuck into 1 unit. > > > > > > > > > > > > > > > > > > > > > > > > On 15 October 2018 at 17:32:32, Chris Mattmann (mattm...@apache.org) > > wrote: > > > > > > > > Isn’t an image per component really heavyweight? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *From: *Tom Barber > > > > *Date: *Monday, October 15, 2018 at 8:26 AM > > > > *To: *Imesha Sudasingha > > > > *Cc: *dev , Chris Mattmann > > > > *Subject: *Re: OODT docker builds > > > > > > > > > > > > > > > > Why aren’t we doing so?! :) > > > > > > > > > > > > > > > > Lack of cycles and young kids ;) > > > > > > > > > > > > > > > > I’ll take a stab at it and see where we get to outside of RADIX to get > the > > > > stack in distinct containers and then we’ll look at integrating it into > the > > > > main build then. > > > > > > > > > > > > > > > > > > > > > > > > Tom > > > > > > > > > > > > > > > > On 15 October 2018 at 13:07:56, Imesha Sudasingha ( > imesha...@cse.mrt.ac.lk > > ) > > > > wrote: > > > > > > > > Yes. Agree with you. That is something I have been planning to ask you > for > > > > long; why aren't we doing so. > > > > > > > > I like the idea of having a docker image per component and as you > suggested > > > > we can create docker-compose > > > > > > > > or kubernetes setup for deployments. I like that direction ;-) > > > > > > > > > > > > > > > > As an starting point, we can add an all-in-one docker image to be built > in > > > > the RADIX build, right? > > > > > > > > If you start off, I will be able to join you along the way. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Imesha > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 15 Oct 2018 at 16:33, Tom Barber wrote: > > > > > > > > I was thinking about the outputs. Currently everything is 1 docker file > > > > which is fine for some deployments and not for others. > > > > > > > > > > > > > > > > In the RADIX build we should also build individual containers for each > > > > component. For the deployment we could also have a docker-compose file > and > > > > a K8S Helm setup so that you could deploy a distributed OODT setup from > > > > your RADIX output, this could also have the ZK stuff in it so we can > > > > properly utilise the distributed nature we started constructing with the > FM > > > > ZK changes. > > > > > > > > > > > > > > > > > > > > > > > > On 15 October 2018 at 05:08:41, Imesha Sudasingha ( > imesha...@cse.mrt.ac.lk > > ) > > > > wrote: > > > > > > > > Hi Tom, > > > > > > > > I think this will be great since people are adopting docker mor
