Re: [pve-devel] Task List Migration
Am 08.10.2014 um 09:34 schrieb Eneko Lacunza: I think shared storage is the correct solution. If you want the feature, you'll have to provide the shared storage which is not difficult to do, otherwise current local storage works ok. This would keep things as simple as possible, which is quite good for reliable logs. Yes but especially for this a replicated storage over all nodes which is may be slow but reliable is much better. Stefan On 06/10/14 10:43, Dietmar Maurer wrote: So i would like to implement something pretty simple f.e. rsync? Note: as mentioned earlier, the rsync approach does not work for HA, because you cannot sync the logs for a dead node. ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
Am 08.10.2014 um 09:34 schrieb Eneko Lacunza: I think shared storage is the correct solution. If you want the feature, you'll have to provide the shared storage which is not difficult to do, otherwise current local storage works ok. This would keep things as simple as possible, which is quite good for reliable logs. Yes but especially for this a replicated storage over all nodes which is may be slow but reliable is much better. Not sure what you suggest here? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
I think shared storage is the correct solution. If you want the feature, you'll have to provide the shared storage which is not difficult to do, otherwise current local storage works ok. This would keep things as simple as possible, which is quite good for reliable logs. On 06/10/14 10:43, Dietmar Maurer wrote: So i would like to implement something pretty simple f.e. rsync? Note: as mentioned earlier, the rsync approach does not work for HA, because you cannot sync the logs for a dead node. ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943575997 943493611 Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa) www.binovo.es ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
Am 04.10.2014 um 10:37 schrieb Dietmar Maurer: I also like the rsync logs on migration. The only missing part is HA migration here. Can i ignore this one? So you would also need to sync the log on VM start. But from which node ?? Thought a bit more about that, and I think the only real solution is to but the VM task logs on shared storage. I can imagine to have a new datacenter.cfg option ---/etc/pve/datacenter.cfg vm_log_store: storageID So the user can store the logs on NFS, GlusterFS, or CephFS ... (or 'local' storage by default) But this is not an easy task - also you need such a storage. Generally i think something like a clustered leightweight database would be much better instead of an FS for stuff like that. But this is out of my knowledge and something more longterm may be PVE 4? So i would like to implement something pretty simple f.e. rsync? Stefan But we currently have a single big task list per node. We need to split that into several independent lists, i.e. we need to use one extra task list for each VM. This is important to avoid locking problems on the shared storage, and to make it easy to delete all logs for a specific VM. ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
So the user can store the logs on NFS, GlusterFS, or CephFS ... (or 'local' storage by default) But this is not an easy task - also you need such a storage. Generally i think something like a clustered leightweight database would be much better instead of an FS for stuff like that. IMHO a clustered database is extremely complex (compared to shared storage). But this is out of my knowledge and something more longterm may be PVE 4? So i would like to implement something pretty simple f.e. rsync? I am against such hacks. Also, It is perfectly valid to argue that VM task logs belongs to the host, so I am not really sure if we need that feature. ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
So i would like to implement something pretty simple f.e. rsync? Note: as mentioned earlier, the rsync approach does not work for HA, because you cannot sync the logs for a dead node. ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
I also like the rsync logs on migration. The only missing part is HA migration here. Can i ignore this one? So you would also need to sync the log on VM start. But from which node ?? Thought a bit more about that, and I think the only real solution is to but the VM task logs on shared storage. I can imagine to have a new datacenter.cfg option ---/etc/pve/datacenter.cfg vm_log_store: storageID So the user can store the logs on NFS, GlusterFS, or CephFS ... (or 'local' storage by default) But we currently have a single big task list per node. We need to split that into several independent lists, i.e. we need to use one extra task list for each VM. This is important to avoid locking problems on the shared storage, and to make it easy to delete all logs for a specific VM. ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
Migrate the VM then migrate the logs for that VM. Or use some sort of central logging system/distributed database. A distributed NoSQL database seems like a good place to store the task logs, each task can simply be a document. Three ways to deal with task logs when a VM is deleted: a) never reuse a VMID or b) Delete all task logs for that VM when it is deleted or c)generate a GUID on VM creation, store GUID in the VM conf file. Include GUID as part of the log index eg VMID+GUID If VM is deleted and VMID reused, the tasks with the old GUID would not be displayed for the same VMID with new GUID On Thu, Oct 2, 2014 at 3:18 PM, Dietmar Maurer diet...@proxmox.com wrote: I also like the rsync logs on migration. The only missing part is HA migration here. Can i ignore this one? So you would also need to sync the log on VM start. But from which node ?? What do you think regarding deletion of VM? no opinion ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
+1 On Oct 2, 2014 6:46 AM, Stefan Priebe - Profihost AG s.pri...@profihost.ag wrote: Hi, i have asked this question already in the past but didn't had the time to proceed. I'm still missing that in case of a migration the task history migrates too. I really like to know the history of a VM and would like to be able to see it after migration too. Another problem to me is, that the task history is kept even after deleting the VM. So after recreating the VM i see the history of an old vm. Any ideas how to archieve this? Is this a feature somebody else likes? Greets, Stefan ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
Any ideas how to archieve this. Task logs and syslog are node local. But I guess it would be possible to: a.) rsync logs on migration. b.) store logs on shared storage other ideas? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
We *could* put them in /etc/pve someplace, so they automatically sync throughout the cluster, but that would probably cause more issues than it solves, especially as regards disk space and change frequency. The fact that VMs themselves are sometimes node local, and sometimes on shared storage, means there's some inconsistency involved with not storing the logs on the same storage as the VMs (they're currently under /var/log, which generally makes sense). However, rsync on migrate would cause the fewest compatibility issues, and the fewest code changes. As to deleted VMs, we could easily just delete the task logs when we delete the VM. However, there might be some benefit from keeping the old logs for later review. So, perhaps rename them (they're stored one directory to a VM, one file to a task, so rename the directory) to include the VM deletion date/time (or some other unique identifier, if we have one I've forgotten), so they don't persist through to any new VM(s) using the same ID, but they also stick around on disk, even if multiple VMs with the same ID get deleted over time. We'd probably want a mechanism in the API/web UI for reviewing/removing such logs. On Oct 2, 2014 11:31 AM, Dietmar Maurer diet...@proxmox.com wrote: Any ideas how to archieve this. Task logs and syslog are node local. But I guess it would be possible to: a.) rsync logs on migration. b.) store logs on shared storage other ideas? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
Am 02.10.2014 19:31, schrieb Dietmar Maurer: Any ideas how to archieve this. Task logs and syslog are node local. But I guess it would be possible to: a.) rsync logs on migration. b.) store logs on shared storage other ideas? I also like the rsync logs on migration. The only missing part is HA migration here. Can i ignore this one? What do you think regarding deletion of VM? Greets, Stefan ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
We *could* put them in /etc/pve someplace, so they automatically sync throughout the cluster, No, you cannot put large files there! (Everything is stored in RAM!) ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Task List Migration
I also like the rsync logs on migration. The only missing part is HA migration here. Can i ignore this one? So you would also need to sync the log on VM start. But from which node ?? What do you think regarding deletion of VM? no opinion ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
