Re: [Toolserver-l] SGE can now be used for *all* tools

[River Tarnell]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Sorry for the flood of announcements about SGE today.  I think this should be 
the last one.

> This policy has now changed, and SGE can be used for all jobs, including 
> long-running jobs.  This effectively obsoletes Phoenix and similar tools, 
> since 
> SGE with cronsub provides the same functionality.

To make this work better, there is now a 'longrun' queue that should be used 
instead of the default 'all.q' for long-running jobs:

 % qsub -q longrun -N myjob myjob.sh
or:
 % cronsub -l myjob myjob.sh

- river.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.16 (FreeBSD)

iEYEARECAAYFAkziKb4ACgkQIXd7fCuc5vIoUACfcd5cJ0HLdCJlsmPpYKWbreHp
Vr8AnA/hSUPKEzWEiDw7IN8ulSlUW2iY
=XACa
-END PGP SIGNATURE-

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] SGE can now be used for *all* tools

[River Tarnell]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Alex Brollo:
> Thanks. A question. I'm a new toolserver user, presently working at a long
> list of personal "Hello world" tests. Next scheduled "Hello world"  was cron
> test. Is your suggestion to step over cron, and to try to learn SGE from the
> beginning?

If you're new to Unix, I would suggesting learning cron first, then adding SGE 
later.  You'll need to use cron to run your SGE jobs anyway.

Converting an existing cron job to SGE is quite simple:
  

- river.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.16 (FreeBSD)

iEYEARECAAYFAkziHvAACgkQIXd7fCuc5vLs8gCgnWvjXTp7dZvVTdUhAbPGNHw/
07EAn1AymtluBv+TJs9wuqetTkreWST8
=WHM/
-END PGP SIGNATURE-

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] SGE can now be used for *all* tools

[Alex Brollo]

2010/11/16 River Tarnell 

>
> This policy has now changed, and SGE can be used for all jobs, including
> long-running jobs.  This effectively obsoletes Phoenix and similar tools,
> since
> SGE with cronsub provides the same functionality.
>

Thanks. A question. I'm a new toolserver user, presently working at a long
list of personal "Hello world" tests. Next scheduled "Hello world"  was cron
test. Is your suggestion to step over cron, and to try to learn SGE from the
beginning?

Alex brollo
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] SGE can now be used for *all* tools

[River Tarnell]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

In the past, our policy on SGE (batch job scheduling) was that it should only 
be used for jobs which run once then exit, and that it should not be used for 
jobs which run continuously.

This policy has now changed, and SGE can be used for all jobs, including 
long-running jobs.  This effectively obsoletes Phoenix and similar tools, since 
SGE with cronsub provides the same functionality.

Because SGE makes it easier for us to scale the Toolserver cluster, as well as 
making your tools run more efficiently (and therefore quicker), we recommend 
that all tools be converted to SGE where possible.  The wiki page now has a 
quick-start section explaining how to do this:

  

(While SGE has a lot of features and can be complicated, we've tried to make it 
easy to use for the typical cases; if something is not clear, please let us 
know.)

- river.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.16 (FreeBSD)

iEYEARECAAYFAkziD9YACgkQIXd7fCuc5vLmewCeLTy/5zyomUajAttXfPoG7j+e
vToAnRD5yVeL5iccv/SfzIgGicHSoAS0
=fV93
-END PGP SIGNATURE-

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] [Toolserver-announce] SGE on web servers

[River Tarnell]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

> The webservers (wolfsbane and ortelius) have been added as SGE submit hosts, 

Since the webservers have a lot of idle CPU (about 90%), I've now added them as 
SGE execution hosts as well, so SGE jobs will run on these systems.  We'll 
monitor to make sure this doesn't create too much load.

Note that running ordinary tools on these systems still isn't allowed.  If you 
want to take advantage of the extra capacity, you would need to use SGE.

- river.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.16 (FreeBSD)

iEYEARECAAYFAkzh9j4ACgkQIXd7fCuc5vIC9wCdGPDSFh3BXfgk0hkII6uBVk52
ECwAnifrbqQaYp9Z+KICs9hgASX9bx+9
=Utd1
-END PGP SIGNATURE-

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Monthly pageviews

[MZMcBride]

Magnus Manske wrote:
> I know there are lots'o'files for daily (hourly?) pageview stats on
> the toolserver.
> 
> Are there aggregated counts for the whole month? So I only have to
> check 1 file instead of hundreds (the aggregated file would, of
> course, be smaller than the concatenated hourly ones).
> Or maybe even as a database? (Onecan dream...)
> 
> If not, does anyone volunteer to generate them? They'd really help
> with my GLAM tools, increase Wikimedia outreach etc.

Pageview stats are still a mess and there's no centralized or clean
database, as far as I'm aware. Henrik's tool (stats.grok.se) has an API you
can hit for monthly stats: http://stats.grok.se/json/en/201006/Barack_Obama

That's probably your best bet right now.

>From what I understand, Wikimedia is devoting resources to setting up Open
Web Analytics. The first test run is supposed to be this week, I think.

MZMcBride



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] acctrenew / setlicense

[wikipedia]

For info;

If you're finding the 'setlicense' script is ending unexpectedly, please be 
aware that it likes you to type 'yes' rather than just 'y' in response to 
questions.

Ho hum.  Another half hour well spent ;)


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] Monthly pageviews

[Magnus Manske]

Hi,

I know there are lots'o'files for daily (hourly?) pageview stats on
the toolserver.

Are there aggregated counts for the whole month? So I only have to
check 1 file instead of hundreds (the aggregated file would, of
course, be smaller than the concatenated hourly ones).
Or maybe even as a database? (Onecan dream...)

If not, does anyone volunteer to generate them? They'd really help
with my GLAM tools, increase Wikimedia outreach etc.

Cheers,
Magnus

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette