Re: [Openstack] heads up regarding keystone dev venv on an Ubuntu VM (VirtualBox)
I'm not sure what pftp is, nor do I have a Mac handy to test with, but I am testing from my Linux firewall box at home, and can't reproduce this problem. The box is running CentOS 5, connected to Comcast cable, with no VPN tunneling to these boxes (the reason I chose it for testing). I did an ftp tummy.com, entered anonymous and j...@tummy.com and then did an ls and it worked. I then ran passive and it said passive mode off. Then I tried an ls, and it failed... I checked the client and I didn't have the ip_nat_ftp module loaded, so I loaded that and did another ls and it worked. So the issue here was a firewall on the client side. While I had the problem, the wire dump showed that the FTP server was trying to connect to the client, and the client was sending back a Host administratively prohibited message. Are you *SURE* that tummy.com is sending the host administratively prohibited message and not the Mac? This would explain why the virtualbox doesn't see it as well, the mac is generating it back towards tummy.com. If this is the case, using passive mode or fixing the firewall should resolve the problem. If you have the network dump that is being referred to here, I'd love to see it. I'd be happy to work with you to resolve this problem. I'll be away from the computer for the next couple of hours, 2pm to 4pm Mountain (GMT-6), but feel free to phone or text me at 970-430-5432 and we can get to the bottom of this. Or if I'm online look for me on Google talk at jaf...@gmail.com, AIM as hrayesci, or irc.community.tummy.com on #hackingsociety. Sean On 05/04/2012 10:39 PM, Duncan McGreggor wrote: (Adding Sean at tummy, in case this is useful info for him.) Nice detective work, Jeremy! d On Fri, May 4, 2012 at 8:27 PM, Jeremy Hanmer jeremy.han...@dreamhost.com wrote: After quite a bit of experimentation, I've noticed that the crash is triggered by pip when accessing ftp://tummy.com specifically. After doing some network dumps, it's apparent that after the connection is initiated we receive an ICMP Host Administratively Prohibited message. My assumption is that this message is seen by VirtualBox, who then tears down the NAT session but fails to pass the ICMP message along to the guest (hard to verify because of how quickly the guest gets killed off). When the guest tries to transfer the first bit of data over the NAT link, I'm guessing it causes something along the lines of a null pointer deref. I'll throw in some of the debug info I've got but the problem is trivial to reproduce: simply run pftp tummy.com, log in anonymously, and try an ls. Process: VirtualBoxVM [16548] Path:/Applications/VirtualBox.app/Contents/MacOS/VirtualBoxVM Identifier: VirtualBoxVM Version: ??? (???) Code Type: X86-64 (Native) Parent Process: VBoxSVC [16535] Date/Time: 2012-05-04 17:14:19.640 -0700 OS Version: Mac OS X 10.7.3 (11D50b) Report Version: 9 Crashed Thread: 22 Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x0010 On May 4, 2012, at 2:31 PM, Duncan McGreggor wrote: Update: Jeremy Hanmer said that it looks like it's a port lookup that's failing and taking cfprefsd with it. He's gone over my head, but I'm sure that's meaningful to someone out there ;-) d On Fri, May 4, 2012 at 12:59 PM, Duncan McGreggor dun...@dreamhost.com wrote: Updates: * Doug Hellmann narrowed this down to the network access that was happening with pip * Mark McClain further narrowed it down to VirtualBox's networking: with a NATed interface, big probs -- with a bridged interface, things go well. I haven't taken the time to check this on my own system, since I've got a working solution right now, but when I need to rebuild, I will check. Mark also mentioned that VBox networking sometimes does some weird stuff (rewriting headers or something) and that might be contributing to the problem. Hope this helps, d On Fri, May 4, 2012 at 12:40 PM, Duncan McGreggor dun...@dreamhost.com wrote: Hey folks, We're really pressed for time right now, so there are certain rabbit holes we can't dive down, but I wanted to bring this up in case it hasn't been seen yet. On Mac OS X 10.6 and 10.7, when running a 12.04 Ubuntu VM and setting up the dev env for Keystone, we get some madness. 10.6: VirtualBox instance aborts, leaving no traces of issue in system logs (that I could see) 10.7: VB dies, OS X kernel panics The second time, I watched carefully, and it happened as python-memcached was getting installed via pip in the .venv. So I built a third. That burned down, fell over, then sank into the swamp. But the fourth one stayed up after I removed .venv and changed tools/install_venv.py to enable system site-package use. d ___ Mailing list: https://launchpad.net/~openstack Post to :
Re: [Openstack] heads up regarding keystone dev venv on an Ubuntu VM (VirtualBox)
In the tummy.dump you provided, packet 16 is the client asking the server to go into passive mode. Packet 17 is the server telling the client to use port 42318 for the passive connection. Packet 19 is the client trying to connect on port 20133 and packet 20 is the client trying to connect on port 42318. Packet 21 is the server reporting that port 20133 is not allowed. Seems like the server is acting correctly in this case, the client is connecting to a port that the server didn't tell it to connect on, and is rejecting that connection. Thoughts? Sean On 05/06/2012 04:32 PM, Jeremy Hanmer wrote: pftp is just an alias to ftp -p. i.e. it simply initiates a passive FTP connection. This bug has also been replicated from a minimum of 3 source locations so I don't believe it's a client firewall issue at all. Both network dumps I have (one to tummy.com and one to kernel.org for comparison) are both up at the following URL now: http://karategerbil.com/vboxdumps/ On May 5, 2012, at 12:54 PM, Sean Reifschneider j...@tummy.com wrote: I'm not sure what pftp is, nor do I have a Mac handy to test with, but I am testing from my Linux firewall box at home, and can't reproduce this problem. The box is running CentOS 5, connected to Comcast cable, with no VPN tunneling to these boxes (the reason I chose it for testing). I did an ftp tummy.com, entered anonymous and j...@tummy.com and then did an ls and it worked. I then ran passive and it said passive mode off. Then I tried an ls, and it failed... I checked the client and I didn't have the ip_nat_ftp module loaded, so I loaded that and did another ls and it worked. So the issue here was a firewall on the client side. While I had the problem, the wire dump showed that the FTP server was trying to connect to the client, and the client was sending back a Host administratively prohibited message. Are you *SURE* that tummy.com is sending the host administratively prohibited message and not the Mac? This would explain why the virtualbox doesn't see it as well, the mac is generating it back towards tummy.com. If this is the case, using passive mode or fixing the firewall should resolve the problem. If you have the network dump that is being referred to here, I'd love to see it. I'd be happy to work with you to resolve this problem. I'll be away from the computer for the next couple of hours, 2pm to 4pm Mountain (GMT-6), but feel free to phone or text me at 970-430-5432 and we can get to the bottom of this. Or if I'm online look for me on Google talk at jaf...@gmail.com, AIM as hrayesci, or irc.community.tummy.com on #hackingsociety. Sean On 05/04/2012 10:39 PM, Duncan McGreggor wrote: (Adding Sean at tummy, in case this is useful info for him.) Nice detective work, Jeremy! d On Fri, May 4, 2012 at 8:27 PM, Jeremy Hanmer jeremy.han...@dreamhost.com wrote: After quite a bit of experimentation, I've noticed that the crash is triggered by pip when accessing ftp://tummy.com specifically. After doing some network dumps, it's apparent that after the connection is initiated we receive an ICMP Host Administratively Prohibited message. My assumption is that this message is seen by VirtualBox, who then tears down the NAT session but fails to pass the ICMP message along to the guest (hard to verify because of how quickly the guest gets killed off). When the guest tries to transfer the first bit of data over the NAT link, I'm guessing it causes something along the lines of a null pointer deref. I'll throw in some of the debug info I've got but the problem is trivial to reproduce: simply run pftp tummy.com, log in anonymously, and try an ls. Process: VirtualBoxVM [16548] Path:/Applications/VirtualBox.app/Contents/MacOS/VirtualBoxVM Identifier: VirtualBoxVM Version: ??? (???) Code Type: X86-64 (Native) Parent Process: VBoxSVC [16535] Date/Time: 2012-05-04 17:14:19.640 -0700 OS Version: Mac OS X 10.7.3 (11D50b) Report Version: 9 Crashed Thread: 22 Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x0010 On May 4, 2012, at 2:31 PM, Duncan McGreggor wrote: Update: Jeremy Hanmer said that it looks like it's a port lookup that's failing and taking cfprefsd with it. He's gone over my head, but I'm sure that's meaningful to someone out there ;-) d On Fri, May 4, 2012 at 12:59 PM, Duncan McGreggor dun...@dreamhost.com wrote: Updates: * Doug Hellmann narrowed this down to the network access that was happening with pip * Mark McClain further narrowed it down to VirtualBox's networking: with a NATed interface, big probs -- with a bridged interface, things go well. I haven't taken the time to check this on my own system, since I've got a working solution right now, but when I need to rebuild, I will check. Mark also mentioned that VBox networking sometimes does
