[Cloud] NFS bandwidth to VPS nodes?

[Roy Smith]

I'm exploring various ways of working with the XML data dumps on 
/publib/dumps/public/enwiki.  I've got a process which runs through all of the 
enwiki-20210301-pages-articles[123456789]*.xml* files in about 6 hours.  If 
I've done the math right, that's just about 18 GB of data, or 3 GB/h, or 8 MB/s 
that I'm slurping off NFS.

If I were to spin up 8 VPS nodes and run 8 jobs in parallel, in theory I could 
process 64 MB/s (512 Mb/s).  Is that realistic?  Or am I just going to beat the 
hell out of the poor NFS server, or peg some backbone network link, or hit some 
other rate limiting bottleneck long before I run out of CPU?  Hitting a 
bottleneck doesn't bother me so much as not wanting to trash a shared resource 
by doing something stupid to it.

Putting it another way, would trying this be a bad idea?


___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Publishing out of a VPS node?

[Roy Smith]

I appreciate the offer, but it's not so much a matter of getting it to work, 
but keeping it simple.  For the few files I need to export, I can just copy 
them manually to someplace else.

> On Apr 18, 2021, at 9:06 AM, Alex Monk  wrote:
> 
> You might try:
> sudo a2enmod mpm_prefork
> Failing that, what's the name of your instance and project? I can probably 
> sort it out for you
> 
> On Sun, 18 Apr 2021 at 13:30, Roy Smith  > wrote:
> Hmmm, it turns out, this doesn't work.  Apache fails to start with:
> 
> AH00534: apache2: Configuration error: No MPM loaded.
> 
> I found a bunch of explanations of why this is happening (sounds like a bug 
> in the default apache2 config?), and how to fix it, but by the time I'm into 
> hacking apache2 config files, we out of "quick and dirty" territory, so I'm 
> going to pass.  Thanks for the suggestion anyway.
> 
> 
> 
>> On Apr 17, 2021, at 6:44 PM, Daniel Zahn > > wrote:
>> 
>> Untested, but this should work I think:
>> 
>> sudo apt-get install apache2
>> sudo a2enmod userdir
>> mkdir ~/public_html/
>> 
>> copy files to public_html as you already mentioned
>> 
>> then click in Horizon to create a proxy, select a name and the
>> instance it should point to
>> access data under that name + /~username/
>> 
>> On Sat, Apr 17, 2021 at 3:27 PM Roy Smith > > wrote:
>>> 
>>> I've computed some data on a VPS node that I want to show to people.  Is 
>>> there some quick and dirty way to publish a file so it's visible to the 
>>> outside?  I'm thinking something along the lines of a public_html directory.
>>> ___
>>> Wikimedia Cloud Services mailing list
>>> Cloud@lists.wikimedia.org  (formerly 
>>> lab...@lists.wikimedia.org )
>>> https://lists.wikimedia.org/mailman/listinfo/cloud 
>>> 
>> 
>> 
>> 
>> -- 
>> Daniel Zahn mailto:dz...@wikimedia.org>>
>> Operations Engineer
>> 
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org  (formerly 
>> lab...@lists.wikimedia.org )
>> https://lists.wikimedia.org/mailman/listinfo/cloud 
>> 
> 
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org  (formerly 
> lab...@lists.wikimedia.org )
> https://lists.wikimedia.org/mailman/listinfo/cloud 
> 
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud

___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Publishing out of a VPS node?

[Alex Monk]

You might try:

sudo a2enmod mpm_prefork

Failing that, what's the name of your instance and project? I can probably
sort it out for you

On Sun, 18 Apr 2021 at 13:30, Roy Smith  wrote:

> Hmmm, it turns out, this doesn't work.  Apache fails to start with:
>
> AH00534: apache2: Configuration error: No MPM loaded.
>
> I found a bunch of explanations of why this is happening (sounds like a
> bug in the default apache2 config?), and how to fix it, but by the time I'm
> into hacking apache2 config files, we out of "quick and dirty" territory,
> so I'm going to pass.  Thanks for the suggestion anyway.
>
>
>
> On Apr 17, 2021, at 6:44 PM, Daniel Zahn  wrote:
>
> Untested, but this should work I think:
>
> sudo apt-get install apache2
> sudo a2enmod userdir
> mkdir ~/public_html/
>
> copy files to public_html as you already mentioned
>
> then click in Horizon to create a proxy, select a name and the
> instance it should point to
> access data under that name + /~username/
>
> On Sat, Apr 17, 2021 at 3:27 PM Roy Smith  wrote:
>
>
> I've computed some data on a VPS node that I want to show to people.  Is
> there some quick and dirty way to publish a file so it's visible to the
> outside?  I'm thinking something along the lines of a public_html directory.
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
>
>
>
>
> --
> Daniel Zahn 
> Operations Engineer
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
>
>
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
>
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] Publishing out of a VPS node?

[Roy Smith]

Hmmm, it turns out, this doesn't work.  Apache fails to start with:

AH00534: apache2: Configuration error: No MPM loaded.

I found a bunch of explanations of why this is happening (sounds like a bug in 
the default apache2 config?), and how to fix it, but by the time I'm into 
hacking apache2 config files, we out of "quick and dirty" territory, so I'm 
going to pass.  Thanks for the suggestion anyway.



> On Apr 17, 2021, at 6:44 PM, Daniel Zahn  wrote:
> 
> Untested, but this should work I think:
> 
> sudo apt-get install apache2
> sudo a2enmod userdir
> mkdir ~/public_html/
> 
> copy files to public_html as you already mentioned
> 
> then click in Horizon to create a proxy, select a name and the
> instance it should point to
> access data under that name + /~username/
> 
> On Sat, Apr 17, 2021 at 3:27 PM Roy Smith  wrote:
>> 
>> I've computed some data on a VPS node that I want to show to people.  Is 
>> there some quick and dirty way to publish a file so it's visible to the 
>> outside?  I'm thinking something along the lines of a public_html directory.
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
> 
> 
> 
> -- 
> Daniel Zahn 
> Operations Engineer
> 
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud

___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud