Re: [Toolserver-l] Cronjob starting cronsub

[MZMcBride]

Magnus Manske wrote:
> I understand that, and I didn't say lfs was better (it has plenty of
> its own quirks), or that SGE was a bad choice - it seems to run well
> enough, all things considered. It's just that things like "SGE might
> interpret comments in your scripts as commands and die" or "you can't
> run qsub from within qsub" are types of behaviour that are unusual to
> say the least, and the toolserver users are left to the mercy of these
> issues (I'd call them bugs) without warning or documentation. (Yes,
> there is a wiki pages that says it can interpret special comments in
> your script. That's somewhat different from "it will reinterpret your
> Perl script and fall over").

Having learned some lessons today, you should write improved documentation
on the wiki. :-)

MZMcBride



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Disambiguation

[Mashiah Davidson]

> select pl_title from pagelinks,page where
> page_title="Disambiguationspage" and page_namespace=8 and
> pl_from=page_id and pl_namespace=10;
>

unfortunately not all of the languages have this page configured
properly, the first table here:
http://toolserver.org/~lvova/cgi-bin/go.sh?&interface=en corresponds
to configured languages and the second to missconfigured ones.

mashiah

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Solving problems with qsub with -b y option

[Marcin Cieslak]

>> Magnus Manske  wrote:

 Wow, what a mess. Why does cronsub/qsub try to divine hidden vodoo
 meaning from my scripts (which predate qsub availability) anyway? If
 that is desired for some cases, why is that activated by default? Is
 there at least a qsub option to turn it off? Will that option work
 with cronsub?
>>>
>>> To quote man qsub(1):
>>
>> There is another option that's quite useful:
>>
>> qsub -b y path-to-program
>
> Thanks, very helpful. Do you know if cronsub supports this?

Have a look at /opt/local/bin/cronsub, it's not a very complicated script.
Cronsub does not currently allow for adding custom qsub parameters.
But what you can of course use own modified copy or whatever. 

>> This causes path-to-program (use full path like /usr/bin/something or
>> path relative to your $HOME or it won't work) to be executed
>> as a "binary" program. Your script will not be copied to a temporary
>> directory on the SGE cluster and will be ran in place.
>
> I believe that is default behaviour anyway now? I vaguely remember a
> mail concerning that WRT Python.

cronsub currently copies the contents of the script to the temporary file
that will be executed and copied by SGE, leaving original script intact
in-place. 

-b y has its disadvantages, though. When I execute the program
(I run a binary C program for test) and it abruptly stops by dumping
core without generating any output. I combine "-b y" with "-m ae"
to receive a mail when the job is completed, in case of crash it's reported
in the e-mail:

To: sa...@clematis.toolserver.org
Subject: Job 211410 (howto2) Aborted
Date: Mon, 28 Feb 2011 20:57:51 + (UTC)
From: r...@wolfsbane.toolserver.org (Super-User)

Job 211410 (howto2) Aborted
 Exit Status  = 139
 Signal   = SEGV
 User = saper
 Queue= al...@wolfsbane.toolserver.org
 Host = wolfsbane.toolserver.org
 Start Time   = 02/28/2011 20:57:50
 End Time = 02/28/2011 20:57:51
 CPU  = 00:00:00
 Max vmem = NA
failed assumedly after job because:
job 211410.1 died through signal SEGV (11)

With shell scripts it is much more obvious, since shells complain
to stderr (as in your case at least you could see the message).

//Marcin



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Disambiguation

[Magnus Manske]

On Mon, Feb 28, 2011 at 7:57 PM, Stefan Kühn  wrote:
> Hmm,
>
> but I will get the templates with a script. And when I show:
>
> http://en.wikipedia.org/wiki/MediaWiki:Disambiguationspage
> http://de.wikipedia.org/wiki/MediaWiki:Disambiguationspage
> http://fr.wikipedia.org/wiki/MediaWiki:Disambiguationspage
>
> then I can't see an standard format in this pages.
>
> Is there an other possibility?

For each language:
select pl_title from pagelinks,page where
page_title="Disambiguationspage" and page_namespace=8 and
pl_from=page_id and pl_namespace=10;

seems to work...

Cheers,
Magnus

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Disambiguation

[Steef389]

Hi,

You can use the API:


Regards,
Steef389

Am 28.02.2011 20:57, schrieb Stefan Kühn:
> Hmm,
> 
> but I will get the templates with a script. And when I show:
> 
> http://en.wikipedia.org/wiki/MediaWiki:Disambiguationspage
> http://de.wikipedia.org/wiki/MediaWiki:Disambiguationspage
> http://fr.wikipedia.org/wiki/MediaWiki:Disambiguationspage
> 
> then I can't see an standard format in this pages.
> 
> Is there an other possibility?
> 
> Stefan
> 
> 
> Am 28.02.2011 19:56, schrieb Steef389:
>> Hi,
>>
>> You can find the name(s) of the template(s) on
>> MediaWiki:Disambiguationspage on every wiki.
>>
>> Steef389
>>
>> Am 28.02.2011 19:47, schrieb Stefan Kühn:
>>> Hi,
>>>
>>> is there an offical list with all templates for disambiguations of all
>>> projects at the toolserver?
>>>
>>> I found only this page:
>>> https://fisheye.toolserver.org/browse/~raw,r=8465/pywikipedia/branches/rewrite/pywikibot/families/wikipedia_family.py
>>>
>>> Maybe there exist a page in the Mediawiki:API? But with this request I
>>> don't found this special templates.
>>> http://de.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=general|namespaces|namespacealiases|statistics|magicwords
>>>
>>> Thanks for your help.
>>> Stefan (sk)
>>>
>>> ___
>>> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
>>> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
>>> Posting guidelines for this list: 
>>> https://wiki.toolserver.org/view/Mailing_list_etiquette
>>
>>
>>
>> ___
>> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
>> Posting guidelines for this list: 
>> https://wiki.toolserver.org/view/Mailing_list_etiquette
> 
> 
> 
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Disambiguation

[Stefan Kühn]

Hmm,

but I will get the templates with a script. And when I show:

http://en.wikipedia.org/wiki/MediaWiki:Disambiguationspage
http://de.wikipedia.org/wiki/MediaWiki:Disambiguationspage
http://fr.wikipedia.org/wiki/MediaWiki:Disambiguationspage

then I can't see an standard format in this pages.

Is there an other possibility?

Stefan


Am 28.02.2011 19:56, schrieb Steef389:
> Hi,
>
> You can find the name(s) of the template(s) on
> MediaWiki:Disambiguationspage on every wiki.
>
> Steef389
>
> Am 28.02.2011 19:47, schrieb Stefan Kühn:
>> Hi,
>>
>> is there an offical list with all templates for disambiguations of all
>> projects at the toolserver?
>>
>> I found only this page:
>> https://fisheye.toolserver.org/browse/~raw,r=8465/pywikipedia/branches/rewrite/pywikibot/families/wikipedia_family.py
>>
>> Maybe there exist a page in the Mediawiki:API? But with this request I
>> don't found this special templates.
>> http://de.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=general|namespaces|namespacealiases|statistics|magicwords
>>
>> Thanks for your help.
>> Stefan (sk)
>>
>> ___
>> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
>> Posting guidelines for this list: 
>> https://wiki.toolserver.org/view/Mailing_list_etiquette
>
>
>
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Disambiguation

[Steef389]

Hi,

You can find the name(s) of the template(s) on
MediaWiki:Disambiguationspage on every wiki.

Steef389

Am 28.02.2011 19:47, schrieb Stefan Kühn:
> Hi,
> 
> is there an offical list with all templates for disambiguations of all 
> projects at the toolserver?
> 
> I found only this page:
> https://fisheye.toolserver.org/browse/~raw,r=8465/pywikipedia/branches/rewrite/pywikibot/families/wikipedia_family.py
> 
> Maybe there exist a page in the Mediawiki:API? But with this request I 
> don't found this special templates.
> http://de.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=general|namespaces|namespacealiases|statistics|magicwords
> 
> Thanks for your help.
> Stefan (sk)
> 
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette



signature.asc
Description: OpenPGP digital signature
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] Disambiguation

[Stefan Kühn]

Hi,

is there an offical list with all templates for disambiguations of all 
projects at the toolserver?

I found only this page:
https://fisheye.toolserver.org/browse/~raw,r=8465/pywikipedia/branches/rewrite/pywikibot/families/wikipedia_family.py

Maybe there exist a page in the Mediawiki:API? But with this request I 
don't found this special templates.
http://de.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=general|namespaces|namespacealiases|statistics|magicwords

Thanks for your help.
Stefan (sk)

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[River Tarnell]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

In article ,
Magnus Manske   wrote:
> Thankfully, Marcin was actually helpful in answering this. Your
> attempt to distract from the actual issues at hand by "literal
> interpretation" has been duly noted.

I'm sorry you didn't find my post helpful, but I don't see any need for 
this sort of attitude.  I'm trying to help you.

> It's just that things like "SGE might interpret comments in your 
> scripts as commands and die" or "you can't run qsub from within qsub"  
> are types of behaviour that are unusual to say the least, and the 
> toolserver users are left to the mercy of these issues (I'd call them 
> bugs) without warning or documentation.

Recursive qsub not working was not a known issue until you posted it on 
the list just now, so it's unreasonable to complain about lack of 
documentation -- there's no way we can document an issue we don't even 
know about.

You are also the first person to have a problem with an unrelated #$ in 
a script.  Now that we can see that might be an issue, the documentation
can be updated.

The documentation is a wiki, so perhaps *you* could update it if you 
don't think it's very good.

- river.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (NetBSD)

iEYEARECAAYFAk1r0EwACgkQIXd7fCuc5vIXxwCgiDpQKCFf14bnWJrDyAujDIkr
P7gAn2TNI3NrIEOZUfm3/xvZ0pVMK48t
=1TEW
-END PGP SIGNATURE-

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Solving problems with qsub with -b y option

[Magnus Manske]

On Mon, Feb 28, 2011 at 3:24 PM, Marcin Cieslak  wrote:
>>> Marcin Cieslak  wrote:
 Magnus Manske  wrote:
>>> On Mon, Feb 28, 2011 at 2:28 PM, Marcin Cieslak  wrote:
>> Magnus Manske  wrote:
>>
>> cronsub update_dupes /home/magnus/update_dupes.pl
> qsub: Unknown option
>
> qsub: Unknown option
>
> qsub: Unknown option

 This is because you have in your script

 #$cmd = "./cron_sql.sh" unless -e $cmd ;

 and this is recognized by qsub(1) as the embedded qsub option.

 If you change it to:

 # $cmd =

 it works again.
>>>
>>> Thanks!
>>>
>>> Wow, what a mess. Why does cronsub/qsub try to divine hidden vodoo
>>> meaning from my scripts (which predate qsub availability) anyway? If
>>> that is desired for some cases, why is that activated by default? Is
>>> there at least a qsub option to turn it off? Will that option work
>>> with cronsub?
>>
>> To quote man qsub(1):
>
> There is another option that's quite useful:
>
> qsub -b y path-to-program

Thanks, very helpful. Do you know if cronsub supports this?

> This causes path-to-program (use full path like /usr/bin/something or
> path relative to your $HOME or it won't work) to be executed
> as a "binary" program. Your script will not be copied to a temporary
> directory on the SGE cluster and will be ran in place.

I believe that is default behaviour anyway now? I vaguely remember a
mail concerning that WRT Python.

Cheers,
Magnus

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[Magnus Manske]

On Mon, Feb 28, 2011 at 3:24 PM, River Tarnell  wrote:
> In article ,
> Magnus Manske   wrote:
>> Wow, what a mess. Why does cronsub/qsub try to divine hidden vodoo
>> meaning from my scripts (which predate qsub availability) anyway?
>
> Because it means you can include directives in your script indicating
> how it should run:
>
> #! /bin/ksh
> #$ -o $HOME/mytool.out
> #$ -l h_rt=3:0:0
> #$ -t 1-10
> #$ -tc 3
> # rest of script...
>
> This is easier than having a separate script just to start that one.

And more complicated than passing these parameters to qsub directly.


>> Is there at least a qsub option to turn it off?
>
> Yes.

Thankfully, Marcin was actually helpful in answering this. Your
attempt to distract from the actual issues at hand by "literal
interpretation" has been duly noted.

>> In my "day job", I work with a large compute farm (lfs, ~10K cores),
>> so I'm more than happy to work with, and migrate existing cronjobs to,
>> this system, but it would help if it weren't a deliberate minefield...
>
> I didn't write SGE.  I considered several existing batch scheduling
> systems and SGE seemed to be the best/most useful.  Of course that
> doesn't mean every user will like every feature it has.

I understand that, and I didn't say lfs was better (it has plenty of
its own quirks), or that SGE was a bad choice - it seems to run well
enough, all things considered. It's just that things like "SGE might
interpret comments in your scripts as commands and die" or "you can't
run qsub from within qsub" are types of behaviour that are unusual to
say the least, and the toolserver users are left to the mercy of these
issues (I'd call them bugs) without warning or documentation. (Yes,
there is a wiki pages that says it can interpret special comments in
your script. That's somewhat different from "it will reinterpret your
Perl script and fall over").

Magnus

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Solving problems with qsub with -b y option

[Marcin Cieslak]

>> Marcin Cieslak  wrote:
>>> Magnus Manske  wrote:
>> On Mon, Feb 28, 2011 at 2:28 PM, Marcin Cieslak  wrote:
> Magnus Manske  wrote:
>
> cronsub update_dupes /home/magnus/update_dupes.pl
 qsub: Unknown option

 qsub: Unknown option

 qsub: Unknown option
>>>
>>> This is because you have in your script
>>>
>>> #$cmd = "./cron_sql.sh" unless -e $cmd ;
>>>
>>> and this is recognized by qsub(1) as the embedded qsub option.
>>>
>>> If you change it to:
>>>
>>> # $cmd =
>>>
>>> it works again.
>>
>> Thanks!
>>
>> Wow, what a mess. Why does cronsub/qsub try to divine hidden vodoo
>> meaning from my scripts (which predate qsub availability) anyway? If
>> that is desired for some cases, why is that activated by default? Is
>> there at least a qsub option to turn it off? Will that option work
>> with cronsub?
>
> To quote man qsub(1):

There is another option that's quite useful:

qsub -b y path-to-program

This causes path-to-program (use full path like /usr/bin/something or 
path relative to your $HOME or it won't work) to be executed
as a "binary" program. Your script will not be copied to a temporary
directory on the SGE cluster and will be ran in place.

Using "-b y" for Python scripts fixes the python import issue
mentioned earlier on this list 
http://thread.gmane.org/gmane.org.wikimedia.toolserver/3069
(item two)

Of course then job submission does not attempt to interpret
your script in any way (including hidden voodoo to read
qsub parameters from it), so "#$" won't work.

Shell scripts work fine using "-b y".

//Marcin



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[River Tarnell]

In article ,
Magnus Manske   wrote:
> Wow, what a mess. Why does cronsub/qsub try to divine hidden vodoo
> meaning from my scripts (which predate qsub availability) anyway?

Because it means you can include directives in your script indicating 
how it should run:

#! /bin/ksh
#$ -o $HOME/mytool.out
#$ -l h_rt=3:0:0
#$ -t 1-10
#$ -tc 3
# rest of script...

This is easier than having a separate script just to start that one.

> If that is desired for some cases, why is that activated by default?

With SGE, it's usually desirable for *all* cases (or at least the vast 
majority).  The cronsub script makes some of the most common uses 
unnecessary (e.g. options controlling output file), so it's used less 
there.  However, turning it off for cronsub while it was enabled for 
qsub seemed like it would be more confusing for users.

> Is there at least a qsub option to turn it off?

Yes.

> Will that option work with cronsub?

No.  But you can invoke qsub yourself if you prefer that.  (This is a 
known issue but it's somewhat difficult to fix because of how cronsub 
works.)
 
> In my "day job", I work with a large compute farm (lfs, ~10K cores),
> so I'm more than happy to work with, and migrate existing cronjobs to,
> this system, but it would help if it weren't a deliberate minefield...

I didn't write SGE.  I considered several existing batch scheduling 
systems and SGE seemed to be the best/most useful.  Of course that 
doesn't mean every user will like every feature it has.

- river.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[Magnus Manske]

On Mon, Feb 28, 2011 at 3:01 PM, Marcin Cieslak  wrote:
>>> Magnus Manske  wrote:
>> On Mon, Feb 28, 2011 at 11:43 AM, Ilmari Karonen  wrote:
>>> On 02/28/2011 01:00 PM, Magnus Manske wrote:

 OK, so I gave up on starting my perl script using cronsub, which seems
 to work only if I give the perl script a parameter (WTF?)
>>>
>>> Any parameter?  I smell a line break encoding issue.
>>>
>>> Try making sure there are no carriage returns in your crontab or in any
>>> of the scripts involved.
>>
>> Well, since that happens also when I start cronsub fron the shell, I doubt 
>> it...
>
> It seems like qsub dumps core when executed from inside of the SGE job.
>
> /sge62/default/spool/wolfsbane/job_scripts/210756[3]: 1253 Segmentation 
> Fault(coredump)
>
> I have filed https://jira.toolserver.org/browse/TS-963 about this. Really 
> strange.

Maybe some safety mechanism to prevent infinite recursion gone bad?

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[Marcin Cieslak]

>> Magnus Manske  wrote:
> On Mon, Feb 28, 2011 at 2:28 PM, Marcin Cieslak  wrote:
 Magnus Manske  wrote:

 cronsub update_dupes /home/magnus/update_dupes.pl
>>> qsub: Unknown option
>>>
>>> qsub: Unknown option
>>>
>>> qsub: Unknown option
>>
>> This is because you have in your script
>>
>> #$cmd = "./cron_sql.sh" unless -e $cmd ;
>>
>> and this is recognized by qsub(1) as the embedded qsub option.
>>
>> If you change it to:
>>
>> # $cmd =
>>
>> it works again.
>
> Thanks!
>
> Wow, what a mess. Why does cronsub/qsub try to divine hidden vodoo
> meaning from my scripts (which predate qsub availability) anyway? If
> that is desired for some cases, why is that activated by default? Is
> there at least a qsub option to turn it off? Will that option work
> with cronsub?

To quote man qsub(1):

 -C prefix_string
  Available for qsub and qrsh with script submission  (-b
  n).

  Prefix_string defines the prefix that declares a direc-
  tive  in  the  job's  command.  The prefix is not a job
  attribute, but affects the behavior of qsub  and  qrsh.
  If  prefix  is  a  null string, the command will not be
  scanned for embedded directives.
  The directive prefix consists of two  ASCII  characters
  which,  when  appearing  in  the  first  two bytes of a
  script line, indicate that what follows is an Sun  Grid
  Engine command.  The default is "#$".
  The user should be aware that changing the first delim-
  iting character can produce unforeseen side effects. If
  the script file contains  anything  other  than  a  "#"
  character  in  the first byte position of the line, the
  shell processor for the job will reject  the  line  and
  may exit the job prematurely.
  If the -C option is present in the script file,  it  is
  ignored.

However, please be warned that cronsub currently messes up custom
qsub options.

//Marcin


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[Marcin Cieslak]

>> Magnus Manske  wrote:
> On Mon, Feb 28, 2011 at 11:43 AM, Ilmari Karonen  wrote:
>> On 02/28/2011 01:00 PM, Magnus Manske wrote:
>>>
>>> OK, so I gave up on starting my perl script using cronsub, which seems
>>> to work only if I give the perl script a parameter (WTF?)
>>
>> Any parameter?  I smell a line break encoding issue.
>>
>> Try making sure there are no carriage returns in your crontab or in any
>> of the scripts involved.
>
> Well, since that happens also when I start cronsub fron the shell, I doubt 
> it...

It seems like qsub dumps core when executed from inside of the SGE job.

/sge62/default/spool/wolfsbane/job_scripts/210756[3]: 1253 Segmentation 
Fault(coredump)

I have filed https://jira.toolserver.org/browse/TS-963 about this. Really 
strange.

//Marcin


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[Magnus Manske]

On Mon, Feb 28, 2011 at 2:28 PM, Marcin Cieslak  wrote:
>>> Magnus Manske  wrote:
>> On Mon, Feb 28, 2011 at 11:37 AM, River Tarnell  wrote:
>>> In article ,
>>> Magnus Manske   wrote:
 OK, so I gave up on starting my perl script using cronsub, which seems
 to work only if I give the perl script a parameter (WTF?), and instead
 altered it to be run manually, generating a shell script that could
 then start via cronsub.
>>>
>>> This should not be the case.  Please provide the exact script and
>>> command line you used and the error message.
>>
>>> cronsub update_dupes /home/magnus/update_dupes.pl
>> qsub: Unknown option
>>
>> qsub: Unknown option
>>
>> qsub: Unknown option
>
> This is because you have in your script
>
> #$cmd = "./cron_sql.sh" unless -e $cmd ;
>
> and this is recognized by qsub(1) as the embedded qsub option.
>
> If you change it to:
>
> # $cmd =
>
> it works again.

Thanks!

Wow, what a mess. Why does cronsub/qsub try to divine hidden vodoo
meaning from my scripts (which predate qsub availability) anyway? If
that is desired for some cases, why is that activated by default? Is
there at least a qsub option to turn it off? Will that option work
with cronsub?

In my "day job", I work with a large compute farm (lfs, ~10K cores),
so I'm more than happy to work with, and migrate existing cronjobs to,
this system, but it would help if it weren't a deliberate minefield...

Cheers,
Magnus

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[Marcin Cieslak]

>> Magnus Manske  wrote:
> On Mon, Feb 28, 2011 at 11:37 AM, River Tarnell  wrote:
>> In article ,
>> Magnus Manske   wrote:
>>> OK, so I gave up on starting my perl script using cronsub, which seems
>>> to work only if I give the perl script a parameter (WTF?), and instead
>>> altered it to be run manually, generating a shell script that could
>>> then start via cronsub.
>>
>> This should not be the case.  Please provide the exact script and
>> command line you used and the error message.
>
>> cronsub update_dupes /home/magnus/update_dupes.pl
> qsub: Unknown option
>
> qsub: Unknown option
>
> qsub: Unknown option

This is because you have in your script

#$cmd = "./cron_sql.sh" unless -e $cmd ;

and this is recognized by qsub(1) as the embedded qsub option.

If you change it to:

# $cmd = 

it works again. 

You could have also used -C '##' to change the command prefix to
(for example) two hash signs, but giving options to qsub
does not work with the current cronsub 
(https://jira.toolserver.org/browse/TS-958).

//Marcin



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[Magnus Manske]

On Mon, Feb 28, 2011 at 1:59 PM, River Tarnell  wrote:
> In article ,
> Magnus Manske   wrote:
>> > cronsub update_dupes /home/magnus/update_dupes.pl
>> qsub: Unknown option
>>
>> qsub: Unknown option
>>
>> qsub: Unknown option
>
> What is the output from this?
>
> $ ksh -x /opt/local/bin/cronsub update_dupes /home/magnus/update_dupes.pl


+ . /sge62/default/common/settings.sh
+ SGE_ROOT=/sge62
+ export SGE_ROOT
+ + /sge62/util/arch
ARCH=sol-amd64
+ + /sge62/util/arch -m
DEFAULTMANPATH=/usr/share/man
+ + /sge62/util/arch -mt
MANTYPE=man
+ SGE_CELL=default
+ export SGE_CELL
+ SGE_CLUSTER_NAME=toolserver
+ export SGE_CLUSTER_NAME
+ unset SGE_QMASTER_PORT
+ unset SGE_EXECD_PORT
+ [ 
/sge62/man:/opt/ts/gnu/share/man:/opt/ts/share/man:/opt/SUNWspro/man:/usr/postgres/8.3/man:/opt/ts/mysql/5.1/share/man:/opt/ts/python/2.6/share/man:/opt/ts/perl/5.10/share/man:/opt/ts/ruby/1.8/share/man:/usr/share/man:/usr/sfw/share/man:/usr/cluster/man:/opt/jobserver/man
=  ]
+ 
MANPATH=/sge62/man:/sge62/man:/opt/ts/gnu/share/man:/opt/ts/share/man:/opt/SUNWspro/man:/usr/postgres/8.3/man:/opt/ts/mysql/5.1/share/man:/opt/ts/python/2.6/share/man:/opt/ts/perl/5.10/share/man:/opt/ts/ruby/1.8/share/man:/usr/share/man:/usr/sfw/share/man:/usr/cluster/man:/opt/jobserver/man
+ export MANPATH
+ 
PATH=/sge62/bin/sol-amd64:/sge62/bin/sol-amd64:/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/mysql/5.1/bin:/opt/ts/perl/5.12/bin:/opt/ts/perl/5/bin:/opt/ts/python/2.7/bin:/opt/ts/php/5.3/bin:/opt/ts/ruby/1.8/bin:/opt/ts/mono/2.0/bin:/opt/ts/tcl/8.5/bin:/usr/ccs/bin:/sge62/bin/sol-amd64:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/postgres/8.3/bin:/usr/cluster/bin:/opt/jobserver/bin
+ export PATH
+ unset ARCH DEFAULTMANPATH MANTYPE
+ 
PATH=/usr/bin:/sge62/bin/sol-amd64:/sge62/bin/sol-amd64:/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/mysql/5.1/bin:/opt/ts/perl/5.12/bin:/opt/ts/perl/5/bin:/opt/ts/python/2.7/bin:/opt/ts/php/5.3/bin:/opt/ts/ruby/1.8/bin:/opt/ts/mono/2.0/bin:/opt/ts/tcl/8.5/bin:/usr/ccs/bin:/sge62/bin/sol-amd64:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/postgres/8.3/bin:/usr/cluster/bin:/opt/jobserver/bin
+ sflag=
+ QUEUE=all.q
+ OPTS=
+ hrt=
+ getopts slh f
+ shift 0
+ [[ 2 -ge 1 ]]
+ JOBNAME=update_dupes
+ shift
+ [ -z  ]
+ [[ 1 -ge 1 ]]
+ SCRIPT=/home/magnus/update_dupes.pl
+ + mktemp
TMPF=/tmp/tmp.5jaiw2
+ chmod 700 /tmp/tmp.5jaiw2
+ trap rm -f /tmp/tmp.5jaiw2 0
+ cat
+ 1> /tmp/tmp.5jaiw2 0<<
#! /bin/sh
cd $HOME
exec "$@"
exit 1

+ grep ^#\$
+ 0< /home/magnus/update_dupes.pl 1>> /tmp/tmp.5jaiw2
+ grep Full jobname:[ \t]*update_dupes$
+ 1> /dev/null
+ qstat -r
+ qsub -q all.q -N update_dupes /tmp/tmp.5jaiw2 /home/magnus/update_dupes.pl
+ 1> /dev/null
qsub: Unknown option

qsub: Unknown option

qsub: Unknown option

+ rm -f /tmp/tmp.5jaiw2

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[River Tarnell]

In article ,
Magnus Manske   wrote:
> > cronsub update_dupes /home/magnus/update_dupes.pl
> qsub: Unknown option
> 
> qsub: Unknown option
> 
> qsub: Unknown option

What is the output from this?

$ ksh -x /opt/local/bin/cronsub update_dupes /home/magnus/update_dupes.pl

- river.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] Maintenance notice: Tuesday, March 1

[River Tarnell]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

On the morning (UTC) of March 1 we will install some new hardware at the 
Toolserver.  Services will be affected as follows:

 Service | Expected impact
 +
 Databases: s3, s4, s6, s7   | User databases unavailable for
 |  < 10 minutes.
 user-store filesystem   | Unavailable for < 10 minutes.

Start time: Tuesday, 1st March, 1PM UTC (estimated)
http://time.tcx.org.uk/utc/2011/3/1/13/0

End time: Tuesday, 1st March, 4PM UTC (estimated)
http://time.tcx.org.uk/utc/2011/3/1/16/0

Details:

We will install new storage for databases and the user-store filesystem.  
This requires installing HBAs in three servers: hemlock, cassia, and 
hyacinth, which requires downtime.  We will try to stagger the 
installation of the two database servers so that those clusters 
remaining accessible during the maintenance, but user databases will be 
unavailable while hyacinth is down.

Before the maintenance on hemlock, we will forcibly unmount the 
user-store filesystem.  Any program accessing it at the time will 
receive an I/O error.

- river.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (NetBSD)

iEYEARECAAYFAk1ro3AACgkQIXd7fCuc5vLhsQCfbKKphnxKxXCM3VNCJC+YOsz+
E5sAn3EBKJjF1PeU4WZCT5W9KFNuqGqM
=TGse
-END PGP SIGNATURE-

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[Magnus Manske]

On Mon, Feb 28, 2011 at 11:43 AM, Ilmari Karonen  wrote:
> On 02/28/2011 01:00 PM, Magnus Manske wrote:
>>
>> OK, so I gave up on starting my perl script using cronsub, which seems
>> to work only if I give the perl script a parameter (WTF?)
>
> Any parameter?  I smell a line break encoding issue.
>
> Try making sure there are no carriage returns in your crontab or in any
> of the scripts involved.

Well, since that happens also when I start cronsub fron the shell, I doubt it...

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[Ilmari Karonen]

On 02/28/2011 01:00 PM, Magnus Manske wrote:
>
> OK, so I gave up on starting my perl script using cronsub, which seems
> to work only if I give the perl script a parameter (WTF?)

Any parameter?  I smell a line break encoding issue.

Try making sure there are no carriage returns in your crontab or in any 
of the scripts involved.

-- 
Ilmari Karonen

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[Magnus Manske]

On Mon, Feb 28, 2011 at 11:37 AM, River Tarnell  wrote:
> In article ,
> Magnus Manske   wrote:
>> OK, so I gave up on starting my perl script using cronsub, which seems
>> to work only if I give the perl script a parameter (WTF?), and instead
>> altered it to be run manually, generating a shell script that could
>> then start via cronsub.
>
> This should not be the case.  Please provide the exact script and
> command line you used and the error message.

> cronsub update_dupes /home/magnus/update_dupes.pl
qsub: Unknown option

qsub: Unknown option

qsub: Unknown option

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[River Tarnell]

In article ,
Magnus Manske   wrote:
> OK, so I gave up on starting my perl script using cronsub, which seems
> to work only if I give the perl script a parameter (WTF?), and instead
> altered it to be run manually, generating a shell script that could
> then start via cronsub.

This should not be the case.  Please provide the exact script and 
command line you used and the error message.

- river.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronjob starting cronsub

[Magnus Manske]

On Sun, Feb 27, 2011 at 1:10 PM, River Tarnell  wrote:
> In article ,
> Marcin Cieslak   wrote:
>> >> Magnus Manske  wrote:
>> > I have a wrapper script that starts a handful of jobs on the
>> > toolserver (via qsub). But, when I put that wrapper script as a
>> > "normal" cronjob, I get some error messages about qsub not found.
>> > Starting the wrapper via cronsub doesn't work either.
>
> You should first run cronsub from the command-line with the same
> arguments.  If that still doesn't work, post the entire command line and
> any output from it.
>
> Once you have it working from the command-line then it should be fine
> fromc ron.
>
>> Actually cronsub does a litle bit more than sourcing
>> /sge62/default/common/settings.sh file, so I don't know
>> why it's not working for you. Maybe you could share some output?
>
> cronsub makes it easier to submit longrun jobs, automatically sets the
> default output file, runs the script from $HOME and prevents duplicate
> jobs being submitted.  Otherwise it's identical to qsub.


OK, so I gave up on starting my perl script using cronsub, which seems
to work only if I give the perl script a parameter (WTF?), and instead
altered it to be run manually, generating a shell script that could
then start via cronsub.

Script ( /home/magnus/cronjobs/most_wanted_wrapper.sh ) :
#!/bin/ksh
qsub -N mw_dewiki_p -j Y -o
/home/magnus/cronjobs/update_most_wanted.dewiki_p.out -l sqlprocs-s5=1
/home/magnus/cronjobs/update_most_wanted.pl dewiki_p
qsub -N mw_enwiki_p -j Y -o
/home/magnus/cronjobs/update_most_wanted.enwiki_p.out -l sqlprocs-s1=1
/home/magnus/cronjobs/update_most_wanted.pl enwiki_p
qsub -N mw_frwiki_p -j Y -o
/home/magnus/cronjobs/update_most_wanted.frwiki_p.out -l sqlprocs-s6=1
/home/magnus/cronjobs/update_most_wanted.pl frwiki_p
qsub -N mw_plwiki_p -j Y -o
/home/magnus/cronjobs/update_most_wanted.plwiki_p.out -l sqlprocs-s2=1
/home/magnus/cronjobs/update_most_wanted.pl plwiki_p

Running this script, or any of its individual lines, directly from
shell, works absolutely fine. However, on

> cronsub mostwanted /home/magnus/cronjobs/most_wanted_wrapper.sh

I get (in the error file):
/home/magnus/cronjobs/most_wanted_wrapper.sh[2]: 7274 Segmentation
Fault(coredump)
/home/magnus/cronjobs/most_wanted_wrapper.sh[3]: 7289 Segmentation
Fault(coredump)
/home/magnus/cronjobs/most_wanted_wrapper.sh[4]: 7290 Segmentation
Fault(coredump)
/home/magnus/cronjobs/most_wanted_wrapper.sh[5]: 7298 Segmentation
Fault(coredump)

I tried replacing qsub with cronsub, only to be met with a flood of
errors relating to all the options it doesn't know, including the sql
slot reservation (I guess that's my punishment for trying to be a good
citizen...).

Removing basically all options, using only lines like this:
cronsub -s mw_dewiki_p /home/magnus/cronjobs/update_most_wanted.pl dewiki_p
in the wrapper script, again runs peachy from shell, but barfs
coredumps again when run via cronsub.


So, as far as I am concerned, it's broken.

Cheers,
Magnus

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette