[GitHub] cloudstack pull request:
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/commit/59a9db39b1d17056753385c0e091bfe26d2df522#commitcomment-14184110 Issue CLOUDSTACK-9024 refers to this commit. https://issues.apache.org/jira/browse/CLOUDSTACK-9024 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9058
GitHub user dsclose opened a pull request: https://github.com/apache/cloudstack/pull/1062 CLOUDSTACK-9058 Respond with "saved_password" if no password is to be issued. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-9058 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1062.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1062 commit 8a7deefe64cab0b3c49ebc510c6524b1fad1f884 Author: dsclose Date: 2015-11-12T08:05:57Z CLOUDSTACK-9058 Respond with "saved_password" if no password is to be issued. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9050
GitHub user dsclose opened a pull request: https://github.com/apache/cloudstack/pull/1063 CLOUDSTACK-9050 Changed grep to match IP address only. Solves issue where virtual router was NATing IP addresses to the wrong interface if the IP address was a substring of the broadcast address on another interface. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-9050 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1063.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1063 commit e15195bf889ff372589b5a903d91ec5fd6cecb48 Author: dsclose Date: 2015-11-10T08:50:44Z CLOUDSTACK-9050 - Match IP address only. Virtual router was NATing IP addresses to the wrong interface if the IP address was a substring of the broadcast address on another interface. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9050
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1063#issuecomment-156103018 I've only used situational tests for this fix. I'll looking into the docs regarding getting the integration tests up and running and will get back to you in the next few days. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9058 Password server should re...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1062#issuecomment-156103338 As with my other pull request, I only used situational tests for this. I'll look into the docs to get the marvin tests up and running and will update this issue in a few days. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9058 Password server should re...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1062#issuecomment-157409534 This was tested situationally. Not perfect, but we don't have the Cloudstack test-suite setup yet. Not sure I will have time to do that now. The following steps were taken on a Cloudstack 4.5 router: 1. Demonstrate default response when no password is to be issued. 1. Open a session on a linux VM 2. Use the arp table to get the IP address of the master virtual router (10.2.13.58 in my case) 3. Query the router for a password and save the result in pass.txt: wget -t 3 -T 20 -O pass.txt --header 'DomU_Request: send_my_password' 10.2.13.58:8080 4. Verify that pass.txt is empty, delete pass.txt 2. Demonstrate correct response when there is a password to be issued. 1. Obtain the IP address of a VM you want to issue a password for (10.2.13.22 in my case) 2. Set up the router to serve a password. 1. Open a session on the router. 2. Save a password for the VM with the value of âtest_passwordâ: 1. /opt/cloud/bin/savepassword.sh âv 10.2.13.22 âp test_password 1. Verify that the password has been saved: cat /var/cache/cloud/passwords-10.2.13.58 10.2.13.22=test 3. Query the router: 1. Open a session on the linux VM 2. Run the same query command as above: wget -t 3 -T 20 -O pass.txt --header 'DomU_Request: send_my_password' 10.2.13.58:8080 3. pass.txt should now contain test_password Those same steps were followed for a Cloudstack 4.3 router (we upgraded from 4.3 to 4.5). It was observed that the Cloudstack 4.3 router replied with "saved_password" when there was no password to be served. Once we established the difference in behaviour we implemented the given fix, rebuilt the systemvm.iso and pushed it out to our hypervisors. The solution was initially tested: 1. By rebuilding a test network, repeating the steps above to verify that we got the expected behaviour. 2. By reproducing the password reset issue on a Windows guest VM running on a network that used the Cloudstack 4.5 system VM. We then rebuilt that network and observed that the Windows VM maintained its password. 1. We also observed that the windows VM repeatedly queried the password server for 30 minutes. 2. This has been seen each time a network is rebuilt so far. 3. We restarted the cloud service on Windows Guest VMs to see if that would cause the password to be lost. The password was maintained. 4. We manually set the administrative password on a windows guest VM and then rebooted. The password was maintained. 5. We triggered a password reset on a windows guest VM, a new password was assigned and adopted by the VM. 6. We repeated steps 4 and 5 on a linux VM. The Linux VM behaved as expected. Subsequent to this, we rebuilt all of our networks with the fixed router. This rebuilt several-hundred virtual routers. There are several-hundred Windows guests. Whereas we were getting alerted by our clients regularly about password issues connected with reboots before the fix, we have not seen a repeat of the behaviour since this fixed was rolled out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9058 - Respond with "saved_pas...
GitHub user dsclose opened a pull request: https://github.com/apache/cloudstack/pull/1079 CLOUDSTACK-9058 - Respond with "saved_password" if no password is to be issued. The password server on the virtual router should respond with "saved_password" if no password is to be issued. This allows for backwards compatibility with Windows Guest VMs which require the "saved_password" response. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-9058 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1079.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1079 commit 8a7deefe64cab0b3c49ebc510c6524b1fad1f884 Author: dsclose Date: 2015-11-12T08:05:57Z CLOUDSTACK-9058 Respond with "saved_password" if no password is to be issued. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9058 Password server should re...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1062#issuecomment-157411446 I've created the PR against 4.6 now. https://github.com/apache/cloudstack/pull/1079 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9058 - Respond with "saved_pas...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1079#issuecomment-157411634 This is a duplicate of PR https://github.com/apache/cloudstack/pull/1062 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9058 - Respond with "saved_pas...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1079#issuecomment-157416663 @wilderrodrigues no, i was asked to open this pull request by @remibergsma --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9058 Password server should re...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1062#issuecomment-157418578 Closing this pull request now that it has been created against the 4.6 branch in PR #1079 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9058 Password server should re...
Github user dsclose closed the pull request at: https://github.com/apache/cloudstack/pull/1062 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9058 - Respond with "saved_pas...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1079#issuecomment-157672288 @wilderrodrigues I'd be pleased to write a Marvin test though I've not worked with them before. I think the first step would be to get the tests running on my local system. Do you have any docs for that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9050 Virtual router should onl...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1063#issuecomment-157678223 CLosing this pull request as the patched script is already deprecated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9050 Virtual router should onl...
Github user dsclose closed the pull request at: https://github.com/apache/cloudstack/pull/1063 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1514#issuecomment-220918123 @swill Pardon me - I was on leave for a week. I'll get this force pushed momentarily. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
GitHub user dsclose reopened a pull request: https://github.com/apache/cloudstack/pull/1514 CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR Rebase of PR #1509 against the 4.7 branch as requested by @swill One LGTM from @ustcweizhou carried from previous PR. Previous PR will be closed. Description from PR #1509: CLOUDSTACK-6975 refers to service monitoring bringing up dnsmasq but this is no-longer accurate, as service monitoring is not active on the post-4.6 routers. These routers still suffer an essentially identical issue, however, because "dnsmasq needs to be restarted each time configure.py is called in order to avoid lease problems." As such, dnsmasq is still running on backup RvRs, causing the issues described in CLOUDSTACK-6975. This PR is based on a patch submitted by @ustcweizhou. The code now checks the redundant state of the router before restarting dnsmasq. RvR networks without this patch have dnsmasq running on both master and backup routers. RvR networks with this patch have dnsmasq running on only the master router. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-6975 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1514.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1514 commit 38b3bdd488745b3ded6d54ae2d3093842bc4d763 Author: dean.close Date: 2016-05-09T10:34:47Z CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose closed the pull request at: https://github.com/apache/cloudstack/pull/1514 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1514#issuecomment-220964156 @luhaijiao I've seen the same error in our XenServer setup but wasn't sure what the cause is. I'd be surprised if this was the cause but it's worth exploring. I will try to boot some system VMs now without this patch to see if the password server starts properly and then apply the patch to see if it subsequently fails. I'll report back here shortly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1514#issuecomment-220982534 @luhaijiao I reverted to a vanilla Cloudstack 4.8.0 ISO. The cloud-passwd-srvr service failed to start. The following appears in /var/log/messages on the router: `Password server failed with error code 1. Restarting it...` The same was true with this patch as well. In both cases, manually restarting the service (as the root user) was successful. I think we should treat the password server issue as separate and raise that elsewhere. I'll kick off the tests again here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose closed the pull request at: https://github.com/apache/cloudstack/pull/1514 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
GitHub user dsclose reopened a pull request: https://github.com/apache/cloudstack/pull/1514 CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR Rebase of PR #1509 against the 4.7 branch as requested by @swill One LGTM from @ustcweizhou carried from previous PR. Previous PR will be closed. Description from PR #1509: CLOUDSTACK-6975 refers to service monitoring bringing up dnsmasq but this is no-longer accurate, as service monitoring is not active on the post-4.6 routers. These routers still suffer an essentially identical issue, however, because "dnsmasq needs to be restarted each time configure.py is called in order to avoid lease problems." As such, dnsmasq is still running on backup RvRs, causing the issues described in CLOUDSTACK-6975. This PR is based on a patch submitted by @ustcweizhou. The code now checks the redundant state of the router before restarting dnsmasq. RvR networks without this patch have dnsmasq running on both master and backup routers. RvR networks with this patch have dnsmasq running on only the master router. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-6975 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1514.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1514 commit 38b3bdd488745b3ded6d54ae2d3093842bc4d763 Author: dean.close Date: 2016-05-09T10:34:47Z CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1514#issuecomment-221304669 @swill if you are happy to merge this without including the tests requested by @jburwell then this PR is ready to merge. @luhaijiao I've created CLOUDSTACK-9385 to deal with the password server not running on redundant virtual routers. Please can you post your findings there if they do not match what I've described? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-221501553 @swill taking a look now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose commented on a diff in the pull request: https://github.com/apache/cloudstack/pull/1519#discussion_r64537215 --- Diff: systemvm/patches/debian/config/opt/cloud/bin/cs/CsAddress.py --- @@ -288,14 +290,16 @@ def post_configure(self, address): """ The steps that must be done after a device is configured """ route = CsRoute() if not self.get_type() in ["control"]: -route.add_table(self.dev) -CsRule(self.dev).addMark() +if self.dev != 'eth0': +route.add_table(self.dev) +CsRule(self.dev).addMark() +self.set_mark() interfaces = [CsInterface(address, self.config)] CsHelper.reconfigure_interfaces(self.cl, interfaces) --- End diff -- Lines 299 and 300 were added as part of c41edc1. I've not investigated what those lines do, I've just naively overcome the merge conflicts. I'll look at the results of CI and follow from there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
GitHub user dsclose reopened a pull request: https://github.com/apache/cloudstack/pull/1519 Cloudstack 9339: Virtual Routers do not handle Multiple Public Interfaces This PR addresses CLOUDSTACK-9339 and may need a code review from someone familiar with the System VM scripts. In particular, this PR has not been tested in a VPC RvR context. Only standalone routers and RvR routers have been demonstrated. - **d582358: Leave public interfaces down in backup redundant routers.** Previously backup routers were bringing all interfaces up and thus arping public IPs away from the master router. - **9ee1eb6: Add the default gateway to the main routing table when interfaces are configured.** The gateway for the first public IP was always being added to the main routing table. Sometimes a router would consequently add the gateway for an IP other than the default source-NAT IP. This would prevent outbound connectivity for guest VMs. - **ad9d72f: Add default gateway to device-specific routing tables.** Link-level routes were being put into the device-specific routing tables (accessed via firewall marks) but these are unnecessary. Instead, the default gateway is needed to allow the kernel to make an appropriate routing decision. - **8db879e: Only mark guest connections when they are part of a static-NAT.** Guest connections were being marked with a zero. This added no functionality and prevented static-NAT rules from routing outbound traffic properly as device-specific routing tables would not be used. Instead, all traffic would be routed out via the default public interface. - **788b1be: Allow forwarding and collect network stats on any public interface.** Forwarding rules and network stats were limited to eth2 on RvR networks. This needed to be decoupled from eth2 and reapplied to whichever interface was under consideration. - **b19e8aa: Ensure that CONNMARK --restore-mark only appears once.** This is a bit of a hack and can do with being improved. The CONNMARK rule was not being picked up by the de-duplication logic in CsNetfilter and was being added twice. This caused checksum errors on packets traversing NAT. - **bf285e1: Transition to master state should add all necessary routes.** Now that backup routers keep their interfaces down, the route logic executed at configuration-time cannot be applied. Instead, once the interface is brought up during a transition to master, routers must re-evaluate what routes are needed and add them. Unfortunately I couldn't see a way to re-use the existing route logic with the variables that I had in scope so there is some duplication. In some cases, routers did not successfully arp IPs away from the old master so some arp logic was added. During a failover most connections with guest VMs will be maintained with only minor packet loss. SSH sessions implemented via port-forwarding rules on an interface other than the source-NAT interface consistently get dropped, however, so the failover isn't quite seamless. It's possible that there's an easy fix for that. I expect that a number of tests may need to be modified/written as part of this PR. Any feedback or pointers would be useful as initially I'll be relying on the CI failures to tell me where to look. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-9339 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1519.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1519 commit e7a63be161bdd14c985a8b483bffe4bfdaa3f5d4 Author: dean.close Date: 2016-05-09T10:31:26Z CLOUDSTACK-9339: Handle multiple public subnets on virtual routers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose closed the pull request at: https://github.com/apache/cloudstack/pull/1519 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose closed the pull request at: https://github.com/apache/cloudstack/pull/1519 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
GitHub user dsclose reopened a pull request: https://github.com/apache/cloudstack/pull/1519 Cloudstack 9339: Virtual Routers do not handle Multiple Public Interfaces This PR addresses CLOUDSTACK-9339 and may need a code review from someone familiar with the System VM scripts. In particular, this PR has not been tested in a VPC RvR context. Only standalone routers and RvR routers have been demonstrated. - **d582358: Leave public interfaces down in backup redundant routers.** Previously backup routers were bringing all interfaces up and thus arping public IPs away from the master router. - **9ee1eb6: Add the default gateway to the main routing table when interfaces are configured.** The gateway for the first public IP was always being added to the main routing table. Sometimes a router would consequently add the gateway for an IP other than the default source-NAT IP. This would prevent outbound connectivity for guest VMs. - **ad9d72f: Add default gateway to device-specific routing tables.** Link-level routes were being put into the device-specific routing tables (accessed via firewall marks) but these are unnecessary. Instead, the default gateway is needed to allow the kernel to make an appropriate routing decision. - **8db879e: Only mark guest connections when they are part of a static-NAT.** Guest connections were being marked with a zero. This added no functionality and prevented static-NAT rules from routing outbound traffic properly as device-specific routing tables would not be used. Instead, all traffic would be routed out via the default public interface. - **788b1be: Allow forwarding and collect network stats on any public interface.** Forwarding rules and network stats were limited to eth2 on RvR networks. This needed to be decoupled from eth2 and reapplied to whichever interface was under consideration. - **b19e8aa: Ensure that CONNMARK --restore-mark only appears once.** This is a bit of a hack and can do with being improved. The CONNMARK rule was not being picked up by the de-duplication logic in CsNetfilter and was being added twice. This caused checksum errors on packets traversing NAT. - **bf285e1: Transition to master state should add all necessary routes.** Now that backup routers keep their interfaces down, the route logic executed at configuration-time cannot be applied. Instead, once the interface is brought up during a transition to master, routers must re-evaluate what routes are needed and add them. Unfortunately I couldn't see a way to re-use the existing route logic with the variables that I had in scope so there is some duplication. In some cases, routers did not successfully arp IPs away from the old master so some arp logic was added. During a failover most connections with guest VMs will be maintained with only minor packet loss. SSH sessions implemented via port-forwarding rules on an interface other than the source-NAT interface consistently get dropped, however, so the failover isn't quite seamless. It's possible that there's an easy fix for that. I expect that a number of tests may need to be modified/written as part of this PR. Any feedback or pointers would be useful as initially I'll be relying on the CI failures to tell me where to look. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-9339 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1519.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1519 commit e7a63be161bdd14c985a8b483bffe4bfdaa3f5d4 Author: dean.close Date: 2016-05-09T10:31:26Z CLOUDSTACK-9339: Handle multiple public subnets on virtual routers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-221600590 > No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself. > >The build has been terminated Kicking it off again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-221611699 Same issue. I don't think this is related to the PR. I'll leave this be for now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose closed the pull request at: https://github.com/apache/cloudstack/pull/1519 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
GitHub user dsclose reopened a pull request: https://github.com/apache/cloudstack/pull/1519 Cloudstack 9339: Virtual Routers do not handle Multiple Public Interfaces This PR addresses CLOUDSTACK-9339 and may need a code review from someone familiar with the System VM scripts. In particular, this PR has not been tested in a VPC RvR context. Only standalone routers and RvR routers have been demonstrated. - **d582358: Leave public interfaces down in backup redundant routers.** Previously backup routers were bringing all interfaces up and thus arping public IPs away from the master router. - **9ee1eb6: Add the default gateway to the main routing table when interfaces are configured.** The gateway for the first public IP was always being added to the main routing table. Sometimes a router would consequently add the gateway for an IP other than the default source-NAT IP. This would prevent outbound connectivity for guest VMs. - **ad9d72f: Add default gateway to device-specific routing tables.** Link-level routes were being put into the device-specific routing tables (accessed via firewall marks) but these are unnecessary. Instead, the default gateway is needed to allow the kernel to make an appropriate routing decision. - **8db879e: Only mark guest connections when they are part of a static-NAT.** Guest connections were being marked with a zero. This added no functionality and prevented static-NAT rules from routing outbound traffic properly as device-specific routing tables would not be used. Instead, all traffic would be routed out via the default public interface. - **788b1be: Allow forwarding and collect network stats on any public interface.** Forwarding rules and network stats were limited to eth2 on RvR networks. This needed to be decoupled from eth2 and reapplied to whichever interface was under consideration. - **b19e8aa: Ensure that CONNMARK --restore-mark only appears once.** This is a bit of a hack and can do with being improved. The CONNMARK rule was not being picked up by the de-duplication logic in CsNetfilter and was being added twice. This caused checksum errors on packets traversing NAT. - **bf285e1: Transition to master state should add all necessary routes.** Now that backup routers keep their interfaces down, the route logic executed at configuration-time cannot be applied. Instead, once the interface is brought up during a transition to master, routers must re-evaluate what routes are needed and add them. Unfortunately I couldn't see a way to re-use the existing route logic with the variables that I had in scope so there is some duplication. In some cases, routers did not successfully arp IPs away from the old master so some arp logic was added. During a failover most connections with guest VMs will be maintained with only minor packet loss. SSH sessions implemented via port-forwarding rules on an interface other than the source-NAT interface consistently get dropped, however, so the failover isn't quite seamless. It's possible that there's an easy fix for that. I expect that a number of tests may need to be modified/written as part of this PR. Any feedback or pointers would be useful as initially I'll be relying on the CI failures to tell me where to look. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-9339 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1519.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1519 commit e7a63be161bdd14c985a8b483bffe4bfdaa3f5d4 Author: dean.close Date: 2016-05-09T10:31:26Z CLOUDSTACK-9339: Handle multiple public subnets on virtual routers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose closed the pull request at: https://github.com/apache/cloudstack/pull/1519 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
GitHub user dsclose reopened a pull request: https://github.com/apache/cloudstack/pull/1519 Cloudstack 9339: Virtual Routers do not handle Multiple Public Interfaces This PR addresses CLOUDSTACK-9339 and may need a code review from someone familiar with the System VM scripts. In particular, this PR has not been tested in a VPC RvR context. Only standalone routers and RvR routers have been demonstrated. - **d582358: Leave public interfaces down in backup redundant routers.** Previously backup routers were bringing all interfaces up and thus arping public IPs away from the master router. - **9ee1eb6: Add the default gateway to the main routing table when interfaces are configured.** The gateway for the first public IP was always being added to the main routing table. Sometimes a router would consequently add the gateway for an IP other than the default source-NAT IP. This would prevent outbound connectivity for guest VMs. - **ad9d72f: Add default gateway to device-specific routing tables.** Link-level routes were being put into the device-specific routing tables (accessed via firewall marks) but these are unnecessary. Instead, the default gateway is needed to allow the kernel to make an appropriate routing decision. - **8db879e: Only mark guest connections when they are part of a static-NAT.** Guest connections were being marked with a zero. This added no functionality and prevented static-NAT rules from routing outbound traffic properly as device-specific routing tables would not be used. Instead, all traffic would be routed out via the default public interface. - **788b1be: Allow forwarding and collect network stats on any public interface.** Forwarding rules and network stats were limited to eth2 on RvR networks. This needed to be decoupled from eth2 and reapplied to whichever interface was under consideration. - **b19e8aa: Ensure that CONNMARK --restore-mark only appears once.** This is a bit of a hack and can do with being improved. The CONNMARK rule was not being picked up by the de-duplication logic in CsNetfilter and was being added twice. This caused checksum errors on packets traversing NAT. - **bf285e1: Transition to master state should add all necessary routes.** Now that backup routers keep their interfaces down, the route logic executed at configuration-time cannot be applied. Instead, once the interface is brought up during a transition to master, routers must re-evaluate what routes are needed and add them. Unfortunately I couldn't see a way to re-use the existing route logic with the variables that I had in scope so there is some duplication. In some cases, routers did not successfully arp IPs away from the old master so some arp logic was added. During a failover most connections with guest VMs will be maintained with only minor packet loss. SSH sessions implemented via port-forwarding rules on an interface other than the source-NAT interface consistently get dropped, however, so the failover isn't quite seamless. It's possible that there's an easy fix for that. I expect that a number of tests may need to be modified/written as part of this PR. Any feedback or pointers would be useful as initially I'll be relying on the CI failures to tell me where to look. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-9339 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1519.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1519 commit e7a63be161bdd14c985a8b483bffe4bfdaa3f5d4 Author: dean.close Date: 2016-05-09T10:31:26Z CLOUDSTACK-9339: Handle multiple public subnets on virtual routers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-221867807 @swill no worries. Happy to do that - the CI output was vast and I wasn't looking forward to combing through it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-222085574 @swill agreed; in particular, my note concerning the merge of lines 299 and 300 - i've not even tested that locally - not had a chance to even figure out what it's doing! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-222195257 @luhaijiao I'd recommend trying e7a63be Port forwarding works for me. I've not tried the VPN functionality. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9385 - Ensure passwd srvr is r...
GitHub user dsclose opened a pull request: https://github.com/apache/cloudstack/pull/1568 CLOUDSTACK-9385 - Ensure passwd srvr is restarts after keepalived transitions Based on @ustcweizhou code suggestion. Tried on redundant and standalone virtual routers and works fine. Needs to be checked on VPC routers. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-9385 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1568.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1568 commit c535e22e8cf310ca288a1b6566318b85bd98c347 Author: dean.close Date: 2016-05-27T17:09:31Z CLOUDSTACK-9385 - Ensure that the password server is restarted after keepalived transitions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9385 - Ensure passwd srvr is r...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1568#issuecomment-222308460 **NB**: This PR should be considered controversial because it restarts the password server after each keepalived transition, rather than stopping the service on transitions to backup or fault states. This decision was taken in anticipation of CLOUDSTACK-9035, which suggests that the password server should be running on backup redundant routers. However, this may be ill-advised. I'm happy to incorporate any feedback into this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9385 - Ensure passwd srvr is restarts af...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1568 @remibergsma @ustcweizhou Well that clears that up! I'll see about adjusting the PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #1519: Cloudstack 9339: Virtual Routers do not handle Multi...
Github user dsclose commented on the issue: https://github.com/apache/cloudstack/pull/1519 I'd recommend merging this if the tests are passing. There is an issue with hairpin NATs which can be solved by adding a route to the guest subnet (scope link) in each routing table. This is easy to do on Redundant routers (as routes are handled in one spot in the transition to master) but needs a bit more work on standalone routers - it looks like the guest subnet and device need to be passed down from CsIP.process all the way to CsAddress.post_change_config. Unfortunately I won't be available to do further work on this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request #1568: CLOUDSTACK-9385 - Ensure passwd srvr is resta...
Github user dsclose closed the pull request at: https://github.com/apache/cloudstack/pull/1568 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #1568: CLOUDSTACK-9385 - Ensure passwd srvr is restarts aft...
Github user dsclose commented on the issue: https://github.com/apache/cloudstack/pull/1568 Unfortunately I won't be available to do further work on this PR. I'll leave the commits available incase they are useful. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #1659: CLOUDSTACK-9339 Virtual Routers don't handle Multipl...
Github user dsclose commented on the issue: https://github.com/apache/cloudstack/pull/1659 @murali-reddy - thank you for picking this up. Let me know when you think it's appropriate to close PR #1519 and I shall do so. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9287 - Fix unique mac address ...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1413#issuecomment-205716220 @wilderrodrigues @ustcweizhou Sorry to arrive late to the party but this appears to break/ignore some RvR functionality. 4 things in particular stand out: 1. All public interfaces should be down in both redundanant virtual routers. Non-public interfaces should be up. The master.py script will bring up public interfaces on master routers. I had been discussing this with @ustcweizhou on the Cloudstack mailing list, Wei had provided a solution based upon the check_is_up method of the CsIP class (in CsAddress.py) but the commit at 8bbea5eeb6598b213b78a324c184841dbba69280 in #1413 appears to contradict what we discussed. 2. Redundant virtual routers can have multiple public interfaces. Presumably, so can VPC routers - although I've not experimented with that. The source-NAT IP will be assigned to eth2 - as will any public IP on the same subnet. Public IPs on different subnets will be assigned to eth3, eth4, eth5 etc. with a new device being created for each separate subnet. The commit 11e61f7054234e17343e9a11948f1804d732ac6c in #1413 does not account for this. 3. Because of point 2, we know that a new interface will be created on an RvR when an IP on a new subnet is assigned to it. Because of point 1, we know that that interface will be down because the master.py script will not have run since it was created. Currently a failover must be induced before this interface will be brought up - I'm yet to address this issue because of point 4, next. 4. Currently static-NATs do not work on RvR setup when they are on a different subnet to the source-NAT IP. There are two reasons for this; one is missing iptables config allowing traffic across interfaces other than eth2. The second reason is that connmarks are being assigned to outbound traffic from eth0 (guest network) which are being used to match traffic to an interface in the IP rules. Traffic intended for a public network therefore gets routed out of the default gateway interface (eth2), breaking any static-NAT traffic on public IPs not assigned to eth2. I'd like to contribute to points 3 and 4 but issue #1413 appears to have been marked as closed. To remedy these points I might need to overwrite code committed as part of this issue, so it seems best to start a conversation about it before rolling the sleeves up and getting stuck in. Are these issues being considered else where? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9287 - Fix unique mac address ...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1413#issuecomment-205831656 Thank you. I've raised CLOUDSTACK-9339 on Jira about the non-VPC RvR issues. As my company needs those issues solved in the short term I'll begin working on a fix from the 4.7 branch. If that fix is later useful upstream then I shall create a PR. If I need to re-work things from a different branch then I'll be happy to do so subsequently --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-9287 - Fix unique mac address ...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1483#issuecomment-208755644 @kiwiflyer I'm currently working on CLOUDSTACK-9339 to resolve the issues that I mentioned in #1413 - I don't think this PR breaks anything that currently works so CLOUDSTACK-9339 probably shouldn't hold this PR up. Dean --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
GitHub user dsclose opened a pull request: https://github.com/apache/cloudstack/pull/1509 CLOUDSTACK-6975: Prevent dnsmasq from starting on backup RvR CLOUDSTACK-6975 refers to service monitoring bringing up dnsmasq but this is no-longer accurate, as service monitoring is not active on the post-4.6 routers. These routers still suffer an essentially identical issue, however, because "dnsmasq needs to be restarted each time configure.py is called in order to avoid lease problems." As such, dnsmasq is still running on backup RvRs, causing the issues described in CLOUDSTACK-6975. This PR is based on a patch submitted by @ustcweizhou. The code now checks the redundant state of the router before restarting dnsmasq. RvR networks without this patch have dnsmasq running on both master and backup routers. RvR networks with this patch have dnsmasq running on only the master router. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-6975 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1509.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1509 commit e17a085b208ca04a436600cb1b3420400547f01e Author: dean.close Date: 2016-04-22T08:38:42Z CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant routers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1509#issuecomment-213437832 @swill I raised the PR against `apache:4.7.1-RC20160120T2318` because it seemed sensible to include it in the latest 4.7 branch. I'd assumed it could be forward merged but as this is only my second PR for Cloudstack I don't think I should be making any recommendations. It's not clear to me how to determine which branch I should raise PRs against - but that's probably a question for the mailing list rather than here. Let me know if you want it based off something else and I'll sort that out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
GitHub user dsclose opened a pull request: https://github.com/apache/cloudstack/pull/1514 CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR Rebase of PR #1509 against the 4.7 branch as requested by @swill One LGTM from @ustcweizhou carried from previous PR. Previous PR will be closed. Description from PR #1509: CLOUDSTACK-6975 refers to service monitoring bringing up dnsmasq but this is no-longer accurate, as service monitoring is not active on the post-4.6 routers. These routers still suffer an essentially identical issue, however, because "dnsmasq needs to be restarted each time configure.py is called in order to avoid lease problems." As such, dnsmasq is still running on backup RvRs, causing the issues described in CLOUDSTACK-6975. This PR is based on a patch submitted by @ustcweizhou. The code now checks the redundant state of the router before restarting dnsmasq. RvR networks without this patch have dnsmasq running on both master and backup routers. RvR networks with this patch have dnsmasq running on only the master router. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-6975 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1514.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1514 commit cb16ce7e6c8197ae289d105bbf829c68db6d5086 Author: dean.close Date: 2016-04-22T14:58:16Z CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant routers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose closed the pull request at: https://github.com/apache/cloudstack/pull/1509 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1509#issuecomment-213465470 Rebased off of 4.7 branch and opened PR #1514 - closing this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1509#issuecomment-213467207 @swill and thank you for you're support - much appreciated :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1514#issuecomment-214246821 @jburwell - sure, I should be able to look at that this week. I'll investigate why the CI build failed as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers don't ha...
GitHub user dsclose opened a pull request: https://github.com/apache/cloudstack/pull/1519 Cloudstack 9339: Virtual Routers don't handle Multiple Public Interfaces This PR addresses CLOUDSTACK-9339 and may need a code review from someone familiar with the System VM scripts. In particular, this PR has not been tested in a VPC RvR context. Only standalone routers and RvR routers have been demonstrated. - **d582358: Leave public interfaces down in backup redundant routers.** Previously backup routers were bringing all interfaces up and thus arping public IPs away from the master router. - **9ee1eb6: Add the default gateway to the main routing table when interfaces are configured.** The gateway for the first public IP was always being added to the main routing table. Sometimes a router would consequently add the gateway for an IP other than the default source-NAT IP. This would prevent outbound connectivity for guest VMs. - **ad9d72f: Add default gateway to device-specific routing tables.** Link-level routes were being put into the device-specific routing tables (accessed via firewall marks) but these are unnecessary. Instead, the default gateway is needed to allow the kernel to make an appropriate routing decision. - **8db879e: Only mark guest connections when they are part of a static-NAT.** Guest connections were being marked with a zero. This added no functionality and prevented static-NAT rules from routing outbound traffic properly as device-specific routing tables would not be used. Instead, all traffic would be routed out via the default public interface. - **788b1be: Allow forwarding and collect network stats on any public interface.** Forwarding rules and network stats were limited to eth2 on RvR networks. This needed to be decoupled from eth2 and reapplied to whichever interface was under consideration. - **b19e8aa: Ensure that CONNMARK --restore-mark only appears once.** This is a bit of a hack and can do with being improved. The CONNMARK rule was not being picked up by the de-duplication logic in CsNetfilter and was being added twice. This caused checksum errors on packets traversing NAT. - **bf285e1: Transition to master state should add all necessary routes.** Now that backup routers keep their interfaces down, the route logic executed at configuration-time cannot be applied. Instead, once the interface is brought up during a transition to master, routers must re-evaluate what routes are needed and add them. Unfortunately I couldn't see a way to re-use the existing route logic with the variables that I had in scope so there is some duplication. In some cases, routers did not successfully arp IPs away from the old master so some arp logic was added. During a failover most connections with guest VMs will be maintained with only minor packet loss. SSH sessions implemented via port-forwarding rules on an interface other than the source-NAT interface consistently get dropped, however, so the failover isn't quite seamless. It's possible that there's an easy fix for that. I expect that a number of tests may need to be modified/written as part of this PR. Any feedback or pointers would be useful as initially I'll be relying on the CI failures to tell me where to look. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dsclose/cloudstack CLOUDSTACK-9339 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/cloudstack/pull/1519.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1519 commit d5823589c8f14f943d921ef0072119b3643893ef Author: dean.close Date: 2016-04-27T08:30:30Z CLOUDSTACK-9339: Leave public interfaces down in backup redundant routers. commit 9ee1eb6be9ccc6b297a2d0bd74c6ded629a04461 Author: dean.close Date: 2016-04-27T08:35:49Z CLOUDSTACK-9339: Add the default gateway to the main routing table when the interfaces are configured. commit ad9d72f288070096c5817d9dac36b1cf8c37b7b2 Author: dean.close Date: 2016-04-27T08:49:25Z CLOUDSTACK-9339: Add default gateway to device-specific routing tables. commit 8db879e3f0a880836ba11129ce66649fc1260d95 Author: dean.close Date: 2016-04-27T09:07:06Z CLOUDSTACK-9339: Only mark guest connections when they are part of a static-NAT. commit 788b1be3366435bb8c1dc3c0a7f14e6a24ac45c0 Author: dean.close Date: 2016-04-27T09:08:23Z CLOUDSTACK-9339: Allow forwarding and collect network stats on any public interface. commit b19e8aab2725cdad048162505153dfcd10fd8810 Author: dean.close Date: 2016-04-27T09:10:04Z CLOUDSTACK-9339: Ensure that CONNMARK --restore-mark only appears once. This avoids checksum errors. commit bf285e185d674238f228a546b600aed694699166 Author: dean.close Date: 2016-0
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers don't ha...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-215480948 @ustcweizhou How would you recommend I separate this? I can imagine separating the issues broadly into two parts: 1. **Routing tables and iptables rules should be properly configured for multiple public NICs.** This PR should allow all features to work on standalone virtual routers. To achieve this, we'd need commits 9ee1eb6, ad9d72f, 8db879e, 788b1be and b19e8aa. 2. **Public Interfaces should be down on backup RvRs.** This PR would also need to ensure that the transition to master restores the proper routing and tables and iptables rules. Therefore it would need commits d582358 and bf285e1. I'm reticent to do this because the first PR would not allow multiple subnets to work on RvR setups. Do you agree with this separation and if so how should it be handled? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers don't ha...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-217110131 Currently I'm investigating @ustcweizhou suggestions above. He's quite correct about not adding the mark for eth0 and I think I've reproduced the problem he reported on NICs higher than eth3. Once we trigger the creation of eth4, the routing table Table_eth4 gets added multiple times - but only once with the fwmark limitation. This prevents any traffic being routed correctly on the VR - even to the point of not being able to connect to it from the HV. @swill @rhtyd - I'll squash and force push this soon. I want to incorporate the snippets given by @ustcweizhou and verify that they work first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers don't ha...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-217279039 That worked a treat. The suggestions made by @ustcweizhou resulted in a very clean set of IP rules and I was able to add IPs on eth4 and eth5 without breaking the router. I'll do a bit more testing tomorrow before squashing and force pushing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers don't ha...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-217290925 @kiwiflyer I have not. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers don't ha...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-217383124 Squashed and force pushed. Tasks remaining: - [ ] Pass CI - [ ] Situational testing on VPC RvR - [ ] Some automated tests Regarding the automated tests for the routers, where should I look for these? If I see some examples I should be able to adjust/add to them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose commented on the pull request:
https://github.com/apache/cloudstack/pull/1514#issuecomment-217392996
@DaanHoogland yes, the line `CsHelper.service("dnsmasq", "restart")` in
`set_master` is in
systemvm/patches/debian/config/opt/cloud/bin/cs/CsRedundant.py. When a backup
router transitions to master, it successfully launches the dnsmasq service.
I should point out that I have only verified that the service is running. I
have not confirmed that it is successfully configured.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-217397801 Looks like the build environment isn't sufficient on jenkins-test-a20: ``` [ERROR] Java HotSpot(TM) 64-Bit Server VM warning: Insufficient space for shared memory file: 30583 Try using the -Djava.io.tmpdir= option to select an alternate temp location. ``` Doing another force push. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: CLOUDSTACK-6975: Prevent dnsmasq from sta...
Github user dsclose commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1514#discussion_r62344011
--- Diff: systemvm/patches/debian/config/opt/cloud/bin/cs/CsDhcp.py ---
@@ -54,7 +54,8 @@ def process(self):
self.cloud.commit()
# We restart DNSMASQ every time the configure.py is called in
order to avoid lease problems.
-CsHelper.service("dnsmasq", "restart")
+if not self.cl.is_redundant() or self.cl.is_master():
--- End diff --
@jburwell Whilst that check might be convenient in some cases, I think that
convenience would be a curse. For example "bring up dnsmasq on non-redundant or
master routers" is a nice explicit declaration of behaviour. Simplifying that
to "bring up dnsmasq on master routers" might do the same thing but it
increases the learning curve for someone reading the code for the first time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-217489945 I rebased against the latest 4.7 before force pushing. Has an error been introduced along the way? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-217490305 Same error over at #1519 - something introduced in an earlier commit? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request: Cloudstack 9339: Virtual Routers do not h...
Github user dsclose commented on the pull request: https://github.com/apache/cloudstack/pull/1519#issuecomment-217495575 @swill ok, thanks, good luck with the master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #1519: Cloudstack 9339: Virtual Routers do not handle Multi...
Github user dsclose commented on the issue: https://github.com/apache/cloudstack/pull/1519 PR #1659 appears to have superseded this PR. As such I'm concerned about recent activity on this PR. Is anyone available to clarify what should be done? I'd anticipated closing this PR when PR #1659 matured. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack issue #1519: Cloudstack 9339: Virtual Routers do not handle Multi...
Github user dsclose commented on the issue: https://github.com/apache/cloudstack/pull/1519 @murali-reddy I haven't worked on Cloudstack for many months but one thing I do recall; without d582358 and bf285e1, networks with redundant virtual routers will simply not work. Whether the commits are appropriate for the current codebase, however, is unclear. I have no way of testing them so I shall not be confusing matters by raising a speculative PR. I shall close this PR now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] cloudstack pull request #1519: Cloudstack 9339: Virtual Routers do not handl...
Github user dsclose closed the pull request at: https://github.com/apache/cloudstack/pull/1519 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
