Your description and the symptoms are very similar to bandwidth saturation. When your link is saturated with traffic, latency will be very high with increased packet loss. This behavior can be caused by saturation in either direction, up or down. With only 5Mbps up, that would be fairly easy.
I would plug in each device individually, wait some time and watch. This will narrow the cause down to a specific device. With that knowledge you can then look at the device itself and find the cause. /* PLUG: http://plug.org, #utah on irc.freenode.net Unsubscribe: http://plug.org/mailman/options/plug Don't fear the penguin. */