If I had to upload 270mb from my home I'd be waiting 3 weeks..... what's the timeout set to? ;) On 20 Jun 2016 19:30, "Merlijn Sebrechts" <merlijn.sebrec...@gmail.com> wrote:
> Thanks for the explanation, Jay. > > I did some further testing. Charm upload fails for a 270MB Charm both from > my home, my work and our datacenter. > > - The datacenter is connected directly to Belnet (upload bandwith > ~300Mbit/s). > - My upload bandwidth at home is ~ 3 Mbps (speedtest.net) although > during upload, system monitor shows ~400KiB/s. > > This causes me to think there is more at play here then large file + slow > internet... Let me know if I can help to further debug this problem. > > > As an aside; I don't consider 270MB to be that large. Some examples: > > > - Kubernetes is ~1G > - Ubuntu docker base image is ~200MB > > > I think this is stuff we should be able to handle... > > 2016-06-20 18:41 GMT+02:00 Jay Wren <jay.w...@canonical.com>: > >> Yes, files are broken up into many TCP packets, and they are all >> transmitted over a single TCP connection. TCP is a complex protocol which >> is well documented, so I'll not repeat that here. If you want lots of >> details, wikipedia is not bad: >> https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Protocol_operation >> >> In the abstract, when you connect to a server using TCP it is identified >> by a 4-tuple of source address, source port, target ip address, target >> port. These connections consume server resources and indeed a connection >> exhaustion is a popular denial of service attack. >> >> You are getting a tcp timeout due to of file size because the time it >> takes to send the entire content is longer than the TCP connection timeout. >> >> Yes, the resource upload command to charmstore will also be affected by >> this. Luckily, resources also have the ability to be uploaded specifically >> to a model, which might have greater network data rates from the resource >> uploader. >> >> -- >> Jay >> >> >> On Mon, Jun 20, 2016 at 7:04 PM, Merlijn Sebrechts < >> merlijn.sebrec...@gmail.com> wrote: >> >>> Thanks for looking into this! I'll try the compression and see if that >>> works. >>> >>> Just curious; why does filesize affect tcp connection timeout? Aren't >>> the files broken up into a bunch of smaller tcp packets? Filesize should >>> only affect the number of tcp packets, not the size of the tcp packets? So >>> getting a tcp timeout because of filesize seems strange to me... Any idea >>> what exactly goes wrong here? >>> >>> Also, now that I think of it, the resource upload command might also be >>> affected by this, if it uses the same library and similar backend >>> infrastructure? I'll test this out. >>> >>> >>> Op maandag 20 juni 2016 heeft Jay Wren <jay.w...@canonical.com> het >>> volgende geschreven: >>> > Hello Merlijn, >>> > I can replicate the problem and I can work around it by using a faster >>> internet connection. >>> > At some point, tcp connections have to time out. I can only replicate >>> the issue when that timeout is reached. If you have the means to relocate >>> to a faster internet connection temporarily for pushing to charmstore, >>> please do. You might also try recompressing any items in the charm using a >>> higher compression level. xz -9 instead of gzip -3 or whatever things may >>> be using now. >>> > We are aware this is a poor longterm solution. We are investigating >>> better solutions for uploads. As you've mentioned, resources will also help >>> the situation. >>> > I am sorry that I do not have a better solution. >>> > -- >>> > Jay >>> > On Fri, Jun 17, 2016 at 4:29 PM, Rick Harding < >>> rick.hard...@canonical.com> wrote: >>> >> >>> >> Merlijn, thanks. I'm going to bet there's an issue with http request >>> sizes for the charmstore that the charm command talks do as we've got some >>> layers (Apache, Squid) in front of the actual application. The team is >>> looking into it. Thanks for giving us the heads up. >>> >> On Wed, Jun 15, 2016 at 10:28 AM Merlijn Sebrechts < >>> merlijn.sebrec...@gmail.com> wrote: >>> >>> >>> >>> Hi all >>> >>> >>> >>> I've hit a roadblock in setting up my CI pipeline. I have a charm >>> with a Java sdk blob of ~200MB. Pushing this charm to the store causes the >>> tool to crash. I put a bug report here: >>> https://bugs.launchpad.net/juju/+bug/1592822 >>> >>> Juju resources would fix my problem, but then I'd need to move to >>> Juju 2.0 and I'm not ready to do that yet. >>> >>> >>> >>> >>> >>> Kind regards >>> >>> Merlijn Sebrechts >>> >>> -- >>> >>> Juju mailing list >>> >>> Juju@lists.ubuntu.com >>> >>> Modify settings or unsubscribe at: >>> https://lists.ubuntu.com/mailman/listinfo/juju >>> >> >>> >> -- >>> >> Juju mailing list >>> >> Juju@lists.ubuntu.com >>> >> Modify settings or unsubscribe at: >>> https://lists.ubuntu.com/mailman/listinfo/juju >>> >> >>> > >>> > >>> >> >> > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju > >
-- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
