This sounds very very familiar: https://github.com/kubernetes/kubernetes/issues/30637
Particularly comment: https://github.com/kubernetes/kubernetes/issues/30637#issuecomment-243276076 That is a nasty bug. How can I upgrade Kubernetes in my cluster? My current versions are -bash-4.2$ oc version oc v1.3.0 kubernetes v1.3.0+52492b4 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://poc-docker01.aipo.gov.au:8443 openshift v1.3.0 kubernetes v1.3.0+52492b4 On 18 November 2016 at 18:18, Lionel Orellana <lione...@gmail.com> wrote: > Files in other dirs in the same NFS server don't get deleted (e.g. <server > name>/poc_runtime/test/) > > There is something in my Openshift node deleting files in <server > name>/poc_runtime/evs as soon as I put them there! > > On 18 November 2016 at 18:04, Lionel Orellana <lione...@gmail.com> wrote: > >> >> In fact, whatever is deleting my files is still doing it: >> >> [root@poc-docker03 evs]# touch x >> [root@poc-docker03 evs]# ls >> [root@poc-docker03 evs]# >> >> evs is a path on an NFS volume that I have added directly to some >> deployment configs >> >> - >> name: evs >> nfs: >> server: <server name> >> path: /poc_runtime/evs >> >> If I stop the origin-service on one particular node the file doesn't >> disappear. >> >> [root@poc-docker03 evs]# touch x >> [root@poc-docker03 evs]# ls >> x >> [root@poc-docker03 evs]# >> >> When I restart the origin-node service I see a lot of errors like this >> >> Failed cleaning pods: [remove /var/lib/origin/openshift.loca >> l.volumes/pods/1b7e3a16-ab08-11e6-8618-005056915814/volumes/kubernetes.io~nfs >> device or resource bus >> Failed to remove orphaned pod xxxxx dir; err: remove >> /var/lib/origin/openshift.local.volumes/pods/xxxx/volumes/kubernetes.io >> ~nfs/*evs*: device or resource bus >> >> Despite the fact that the error says that it couldn't remove it, what >> exactly is it trying to do here? Is it possible that this process >> previously deleted the data in the evs folder? >> >> >> >> >> On 18 November 2016 at 16:45, Lionel Orellana <lione...@gmail.com> wrote: >> >>> What about NFS volumes added directly in build configs. >>> >>> volumes: >>> - >>> name: jenkins-volume-1 >>> nfs: >>> server: <server name> >>> path: /poc_runtime/jenkins/home >>> >>> >>> We just restarted all the servers hosting my openshift cluster and the >>> all data in the path above disappeared. Simply by restarting the host VM! >>> >>> >>> >>> On 18 November 2016 at 16:19, Lionel Orellana <lione...@gmail.com> >>> wrote: >>> >>>> Thanks Mark >>>> >>>> On 18 November 2016 at 15:09, Mark Turansky <mtura...@redhat.com> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Thu, Nov 17, 2016 at 10:41 PM, Lionel Orellana <lione...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Couple of questions regarding Persistent Volumes, in particular NFS >>>>>> ones. >>>>>> >>>>>> 1) If I have a PV configured with the Retain policy it is not clear >>>>>> to me how this PV can be reused after the bound PVC is deleted. Deleting >>>>>> the PVC makes the PV status "Released". How do I make it "Available" >>>>>> again >>>>>> without losing the data? >>>>>> >>>>> >>>>> You can keep the PVC around longer if you intend to reuse it between >>>>> pods. There is no way for a PV to go from Released to Available again in >>>>> your scenario. You would have to delete and recreate the PV. It's a >>>>> pointer >>>>> to real storage (the NFS share), so you're just recreating the pointer. >>>>> The >>>>> data in the NFS volume itself is untouched. >>>>> >>>>> >>>>> >>>>>> >>>>>> 2) Is there anything (e.g. all nodes crashing due to some underlying >>>>>> infrastructure failure) that would cause the data in a "Retain" volume to >>>>>> be wiped out? We had a problem with all our vmware servers (where I host >>>>>> my openshift POC) and all my NFS mounted volumes were wiped out. The >>>>>> storage guys assure me that nothing at their end caused that and it must >>>>>> have been a running process that did it. >>>>>> >>>>> >>>>> "Retain" is just a flag to the recycling process to leave that PV >>>>> alone when it's Released. The PV's retention policy wouldn't cause >>>>> everything to be deleted. NFS volumes on the node are no different than if >>>>> you called "mount" yourself. There is nothing inherent in OpenShift itself >>>>> that is running in that share that would wipe out data. >>>>> >>>>> >>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> Lionel. >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> users@lists.openshift.redhat.com >>>>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
