Re: [galaxy-dev] AD Intergration

[Chen, John (IT)]

Jelle,

After messing around a few time with the config file.. I have it working 
already.   I am able to authenticate against AD now.. 

Matthias,
I saw that posting, but that wasn’t the issue, but thanks anyway.



John Chen
Tel: 646-524-0080

Cell: 347-587-9655
https://nyumc.webex.com/join/chenj29

Email: john.ch...@nyumc.org

On 6/16/17, 12:30 PM, "galaxy-dev on behalf of Matthias Bernt" 
 wrote:

Hi Jelle,

I just (in this very moment) solved the "option error" issue for our 
galaxy installation.

see my comment on the first issue mentioned by john:


https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_galaxyproject_galaxy_issues_3178-23issuecomment-2D306538866&d=DQIGaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=bOqvdGabzr80lh6GA_AnYh1-lz5wZ9iCLk4PxBK4Z3M&m=N2VIuDVupElUxyy8Q_CJmDB_VsT9Ck4MTnE5Fpqep3o&s=s0_hB56pHAp1xLeW2_kGub14aZ7Ci_JANFHwVRT93sg&e=
 

Maybe you do not need to compile everything from source (as I needed to).

Best,
Matthias

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.galaxyproject.org_&d=DQIGaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=bOqvdGabzr80lh6GA_AnYh1-lz5wZ9iCLk4PxBK4Z3M&m=N2VIuDVupElUxyy8Q_CJmDB_VsT9Ck4MTnE5Fpqep3o&s=DhvWm0WIbuaJhb5Oafp1-aFc-5JiwLcOUpmaw8OKJEs&e=
 

To search Galaxy mailing lists use the unified search at:
  
https://urldefense.proofpoint.com/v2/url?u=http-3A__galaxyproject.org_search_&d=DQIGaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=bOqvdGabzr80lh6GA_AnYh1-lz5wZ9iCLk4PxBK4Z3M&m=N2VIuDVupElUxyy8Q_CJmDB_VsT9Ck4MTnE5Fpqep3o&s=oQLWs0ho9aQMZlAaN3v9VFZ09Oa1o6xdnEHylLVQgx4&e=
 



This email message, including any attachments, is for the sole use of the 
intended recipient(s) and may contain information that is proprietary, 
confidential, and exempt from disclosure under applicable law. Any unauthorized 
review, use, disclosure, or distribution is prohibited. If you have received 
this email in error please notify the sender by return email and delete the 
original message. Please note, the recipient should check this email and any 
attachments for the presence of viruses. The organization accepts no liability 
for any damage caused by any virus transmitted by this email.
=
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] AD Intergration

[Matthias Bernt]

Hi Jelle,

I just (in this very moment) solved the "option error" issue for our 
galaxy installation.


see my comment on the first issue mentioned by john:

https://github.com/galaxyproject/galaxy/issues/3178#issuecomment-306538866

Maybe you do not need to compile everything from source (as I needed to).

Best,
Matthias

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/

Re: [galaxy-dev] Singularity Container forks ?

[Björn Grüning]

Hi Nikos,

we are discussion various enhancement with the Singularity developers,
one is to suppress warnings.

Can you try this for the time being? This is working fine for me.

$tool_directory:rw,$working_directory:rw,$default_file_path:rw

Looking more closely to your error message it seems to be that
Singularity is interpreting the default_ro, which should not happen.
This should be replace by either ro or rw. Where do you specify this string?

Cheers,
Bjoern


> hi,
> 
> i have question when submitting jobs as real user
> 
> defaults = "$galaxy_root:default_ro,$tool_directory:default_ro"
> 
> singularity gives a WARNING since the home dir is already mounted
> "WARNING: Not mounting requested bind point (already mounted in
> container): default_ro"
> 
> can this be avoided ?
> 
> 
> Thanks,
> 
> 
> 
> 
> On 14/06/2017 07:46 μμ, John Chilton wrote:
>> Sorry for the late response - but earlier this week Björn Grüning and
>> I added various support to Galaxy's development branch for Singularity.
>>
>> The following pull request added Singularity support in job running
>> (https://github.com/galaxyproject/galaxy/pull/4175) - here job
>> destinations may describe the paths to Singularity containers. It may
>> also work with Singularity 2.3 to just specify Docker containers and
>> let it auto-convert - I haven't tested that.
>>
>> For "best practice" tools - these are tools that have only
>> requirements that can resolve via Conda in the best practice Conda
>> channels we can do even more - we can build Singularity containers and
>> use them automatically - either ahead of time or build on the demand
>> during job execution.
>>
>> Here is the PR that added the on-demand building support for such
>> Singularity containers to Galaxy
>> https://github.com/galaxyproject/galaxy/pull/4185 and here is the
>> tooling PR (https://github.com/galaxyproject/galaxy-lib/pull/64) and
>> cached container PR
>> (https://github.com/galaxyproject/galaxy/pull/4179) support. If you or
>> anyone has a chance to try it out and beta test - let the list know
>> how it goes. I plan to talk a little about this work and related
>> Docker and Conda work at the GCC so I hope to see you there!
>>
>> -John
>>
>>
>>
>> On Tue, May 2, 2017 at 8:51 AM, Nikos Nikoloutsakos
>> mailto:nikolou...@admin.grnet.gr>> wrote:
>>
>> Hello,
>>
>> Are there any galaxy forks using Singularity images instead of
>> Docker ?
>> Is it trivial enough to replace "docker run" with "singularity exec" ?
>>
>> Thank you,
>> Nikos
>>
>>
>> -- 
>> Nikos Nikoloutsakos
>> Greek Research and Technology Network (GRNET)
>> ☎ +30 210 7471127 
>> ✉ nikolou...@grnet.gr 
>>  hpc.grnet.gr 
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/ 
>>
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/ 
>>
>>
> 
> -- 
> Nikos Nikoloutsakos
> Greek Research and Technology Network (GRNET)
> ☎ +30 210 7471127
> ✉ nikolou...@grnet.gr
>  hpc.grnet.gr
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Singularity Container forks ?

[Nikos Nikoloutsakos]

hi,

i have question when submitting jobs as real user

defaults = "$galaxy_root:default_ro,$tool_directory:default_ro"

singularity gives a WARNING since the home dir is already mounted 
"WARNING: Not mounting requested bind point (already mounted in 
container): default_ro"


can this be avoided ?


Thanks,




On 14/06/2017 07:46 μμ, John Chilton wrote:
Sorry for the late response - but earlier this week Björn Grüning and 
I added various support to Galaxy's development branch for Singularity.


The following pull request added Singularity support in job running 
(https://github.com/galaxyproject/galaxy/pull/4175) - here job 
destinations may describe the paths to Singularity containers. It may 
also work with Singularity 2.3 to just specify Docker containers and 
let it auto-convert - I haven't tested that.


For "best practice" tools - these are tools that have only 
requirements that can resolve via Conda in the best practice Conda 
channels we can do even more - we can build Singularity containers and 
use them automatically - either ahead of time or build on the demand 
during job execution.


Here is the PR that added the on-demand building support for such 
Singularity containers to Galaxy 
https://github.com/galaxyproject/galaxy/pull/4185 and here is the 
tooling PR (https://github.com/galaxyproject/galaxy-lib/pull/64) and 
cached container PR 
(https://github.com/galaxyproject/galaxy/pull/4179) support. If you or 
anyone has a chance to try it out and beta test - let the list know 
how it goes. I plan to talk a little about this work and related 
Docker and Conda work at the GCC so I hope to see you there!


-John



On Tue, May 2, 2017 at 8:51 AM, Nikos Nikoloutsakos 
mailto:nikolou...@admin.grnet.gr>> wrote:


Hello,

Are there any galaxy forks using Singularity images instead of
Docker ?
Is it trivial enough to replace "docker run" with "singularity exec" ?

Thank you,
Nikos


-- 
Nikos Nikoloutsakos

Greek Research and Technology Network (GRNET)
☎ +30 210 7471127 
✉ nikolou...@grnet.gr 
hpc.grnet.gr 

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/ 

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/ 




--
Nikos Nikoloutsakos
Greek Research and Technology Network (GRNET)
☎ +30 210 7471127
✉ nikolou...@grnet.gr
 hpc.grnet.gr

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] drmaa resubmission

[Matthias Bernt]

Dear John,

thanks a lot for all the information. I guess I will need some time to 
dig into this.


For drmaa the wait() function of the python library seems to return 
quite bit of useful information:  hasExited, hasCoreDump, hasSignal, and 
terminateSignal. I guess this would be of help.


The problem seems to be that when the external run script is used jobs 
can not be queried properly (see my other post). But I did not 
understand this completely.


Cheers,
Matthias

On 15.06.2017 19:05, John Chilton wrote:

We've done a lot of work in Galaxy dev on this problem over the last
few years - I'm not sure how much concrete progress we have made.

Nate started it and I did some work at the end of last year. Just to
summarize my most recent work on this - in
https://github.com/galaxyproject/galaxy/pull/3291/commits/b78287f1508db2c06f0c309ed8d3747adb4d17fa
I added some test cases for the existing job runner resubmission stuff
- it was just my sense to understand what was there - hopefully the
examples in the form of test cases help you as well.. This includes a
little test job_conf.xml file that describes how you can catch job
walltime and memory limit hits registered by the job runner and send
jobs to different destinations. This requires the job runner knows how
to record these problems - which the SLURM job runner does - other job
runners like the generic drmaa runner may need to be subclassed to
check for these things in general.

In 
https://github.com/galaxyproject/galaxy/pull/3291/commits/7d52b28ab2ab0314cd4fa31108a6750cb9750ef3
I created a little DSL for resubmissions to make what can be expressed
in job_conf more powerful. Then I added variables to expressions
language such as seconds_since_queued,
seconds_running(https://github.com/galaxyproject/galaxy/pull/3291/commits/18eb1c8d0e4c3f7616d44fd177c90943695b7053),
and attempt number
(https://github.com/galaxyproject/galaxy/pull/3291/commits/7e338d790964f594ae67b33e6a72e1777e774b8c).
I also added the ability to resubmit on unknown job runner problems
here 
(https://github.com/galaxyproject/galaxy/pull/3291/commits/0559cff6e94b250ddd98275b119ab51b36491e34).

None of this is really documented outside the test cases - it is
waiting for someone to come along and find it useful.

I think the next thing I'd like to see for job resubmission besides
documentation and more job runner support for common runners is
described in this issue
(https://github.com/galaxyproject/galaxy/issues/3320) - all the
existing resubmission logic is based on errors detected from job
runners - if the underlying error exhibits itself as a tool failure -
we need a way to reason about that and we cannot currently.

Hope this helps.

-John

On Thu, Jun 15, 2017 at 10:37 AM, Matthias Bernt  wrote:

Dear list,

I was thinking about implementing the job resubmission feature for drmaa.

I hope that I can simplify the job configuration for our installation (and
probably others as well) by escalating through different queues (or
ressource limits). Thereby I hope to reduce the number of special cases that
I need to take care.

I was wondering if there are others

- who are also interested in this feature and want to join? I would try to
give this project a head start in the next week.

- that may have started to work on this feature or just started to think
about it and want to share code/experience

Best,
Matthias
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/

Re: [galaxy-dev] drmaa job status

[Matthias Bernt]

Dear Nate and Curtis,

I read a bit in the documentation of the python and underlying C 
library.. and played a bit around.


I can't speak for GridEngine's specific behavior because I haven't used 
it in a long time, but it's not surprising that jobs "disappear" as soon 
as they've exited. Unfortunately, Galaxy uses periodic polling rather 
than waiting on completion. We'd need to create a thread-per-submitted 
job unless you can still get job exit details by looping over jobs with 
a timeout wait.

> > You can gain some control over how Galaxy handles InvalidJobException

exceptions with drmaa job runner plugin params, see here:

https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced#L9

However, if normally finished jobs also result in InvalidJobException, 
that probably won't help. Alternatively, you could create a 
DRMAAJobRunner subclass for GridEngine like we've done for Slurm that 
does some digging to learn more about terminal jobs.


I found a way to get the information. The problem in my script was that 
wait() "reaps" (that the term used by python-drmaa) all information on 
the job from the session(?). Hence calls to jobStatus after wait() will 
fail. The solution here would be to use synchronize() with parameter 
dispose=False, see attached file -- alternatively one can also wait 
until job status is DONE or FAILED.


But this seems not to be the source of the problem within Galaxy, since 
it never calls wait(..). The problem seems to be that an external python 
script submits the job in another session (when jobs are submitted as 
real user). The problem is then that jobs created in another session can 
not be queried (The documentation is a bit vague here: in the C library 
documentation of SGE I read that it is definitely not possible to query 
finished jobs).


I tried if it is possible to pickle the session -- without success.

Does anyone has an idea how one could get the active drmaa session from 
galaxy to the external script?


> We have had this problem on our SGE based installation for years. We
> referred to it as the "green screen of death" - as it would allow a
> biologist to continue analysis using output that was partial, at best,
> often resulting in seemingly successful completion of the entire
> analysis, but completely bogus results (say, cuffdiff killed half way
> through the genome, but it's green in galaxy, so no transcripts on the
> smaller chromosomes, but no error, either).

Did you use submission as real user? Or does the problem also appear if 
jobs are submitted as the single user running galaxy.


> We ended up implementing an external reaper that detected these killed
> jobs from SGE, and notified the user and galaxy post-hoc. It was not a
> very satisfactory solution. We are currently moving to SLURM for other
> reasons and hope the problem will not be present there.

I was also thinking about not using the python library at all, but using 
system calls to qsub, qkill, qacct, etc? Any opinions on this idea?


I guess your reaper could be of interest also for others. Is this 
available somewhere?


Best,
Matthias


--nate

On Thu, Jun 15, 2017 at 10:27 AM, Matthias Bernt > wrote:


Dear list,

I have two question for all DRMAA users. Here is the first one.

I was checking how our queuing system (univa GridEngine) and Galaxy
react if jobs are submitted that exceed run time or memory limits.

I found out that the python drmaa library cannot query the job
status after the job is finished (for both successful and
unsuccessful jobs).

In lib/galaxy/jobs/runners/drmaa.py the call gives an exception
 self.ds.job_status( external_job_id )

Is this always the case? Or might this be a problem with our GridEngine?

I have attached some code for testing. Here the first call to
s.jobStatus(jobid) works, but the second after s.wait(...) doesn't.
But I get "drmaa.errors.InvalidJobException: code 18: The job
specified by the 'jobid' does not exist."

The same error pops up in the Galaxy logs. The consequence is that
jobs that reached the limits are shown as completed successfully in
Galaxy.

Interestingly, quite a bit of information can be obtained from the
return value of s.wait. I was wondering if this can be used to
differentiate successful from failed jobs. In particular hasExited,
hasSignal, and terminateSignal are different in the two cases.

Cheers,
Matthias


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/ 

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/ 


#!/usr/bin/env pyth

Re: [galaxy-dev] AD Intergration

[Jelle Scholtalbers]

Hi John,

this error looks familiar:
https://github.com/galaxyproject/galaxy/issues/3178
https://github.com/galaxyproject/galaxy/issues/4153
https://github.com/galaxyproject/starforge/issues/130

To remedy the 'option' error:

source /home/galaxy/galaxy/.venv/bin/activate
pip install --upgrade python-ldap


Hope this brings you a step further.

- Jelle

On Wed, Jun 14, 2017 at 6:09 PM, John Chen  wrote:

> Jelle
>
> I did all that and it looks correct.. it is retrieving the correct field.
> This is the error i am still getting..  I am using pretty much the same
> option in other apps..
>
>
>
> galaxy.webapps.galaxy.controllers.user DEBUG 2017-06-14 12:04:40,648
> trans.app.config.auth_config_file: ./config/auth_conf.xml
> galaxy.auth.providers.ldap_ad DEBUG 2017-06-14 12:04:40,648 LDAP
> authenticate: email is johnu...@example.org
> galaxy.auth.providers.ldap_ad DEBUG 2017-06-14 12:04:40,648 LDAP
> authenticate: username is None
> galaxy.auth.providers.ldap_ad DEBUG 2017-06-14 12:04:40,648 LDAP
> authenticate: options are {'bind-user': '{dn}', 'search-fields':
> 'uid,mail', 'login-use-username': 'False', 'allow-register': 'True',
> 'ldap-options': 'OPT_X_TLS_REQUIRE_CERT=OPT_X_TLS_ALLOW',
> 'auto-register-email': '{email}', 'server': 'ldap://ldap.nyumc.org',
> 'auto-register': 'True', 'search-base': 'DC=example,DC=org',
> 'search-filter': '(mail={email})', 'continue-on-failure': 'True',
> 'auto-register-username': '{sAMAccountName', 'bind-password': '{password}',
> 'allow-password-change': 'False'}
> galaxy.auth.providers.ldap_ad DEBUG 2017-06-14 12:04:40,648 LDAP
> authenticate: Valid LDAP option pair OPT_X_TLS_REQUIRE_CERT=OPT_X_TLS_ALLOW
> -> 24582=3
> galaxy.auth.providers.ldap_ad ERROR 2017-06-14 12:04:40,648 LDAP
> authenticate: search exception
> Traceback (most recent call last):
>   File "/home/galaxy/galaxy/lib/galaxy/auth/providers/ldap_ad.py", line
> 118, in authenticate
> ldap.set_option(*opt)
>   File 
> "/home/galaxy/galaxy/.venv/lib/python2.7/site-packages/ldap/functions.py",
> line 135, in set_option
> return _ldap_function_call(None,_ldap.set_option,option,invalue)
>   File 
> "/home/galaxy/galaxy/.venv/lib/python2.7/site-packages/ldap/functions.py",
> line 66, in _ldap_function_call
> result = func(*args,**kwargs)
> ValueError: option error
>
>
> Are you running MS AD ?   if so, could i take a look at your config file?
>
> Thanks
> John
>
>
> --
> *From:* Jelle Scholtalbers 
> *To:* Hans-Rudolf Hotz 
> *Cc:* John Chen ; Galaxy Dev List  galaxyproject.org>
> *Sent:* Monday, June 12, 2017 3:16 AM
> *Subject:* Re: [galaxy-dev] AD Intergration
>
> Hi John,
>
> as a tip, you can use the tool "ldapsearch", from e.g. the package
> "openldap-client", to figure out with which attributes you search and which
> attributes you can retrieve.
>
> Examples:
> $ ldapsearch -vv -x -H ldap://dc1.example.com -b
> cn=Users,dc=exampke,dc=org" # retrieve all AD/ldap entries
> $ ldapsearch -vv -x -H ldap://dc1.example.com -b
> cn=Users,dc=exampke,dc=org "uid=a_username"  # retrieve all attributes for
> user with uid "a_username"
> $ ldapsearch -vv -x -H ldap://dc1.example.com -b
> cn=Users,dc=exampke,dc=org "sAMAccountName=a_username" mail # only
> retrieve the mail attribute by searching for the sAMAccountName
>
>
> In addition, if you get it working, you might want to switch to the more
> secure ldap*s* if that is supported by your IT.
>
> Cheers,
> Jelle
>
>
>
> On Mon, Jun 12, 2017 at 8:32 AM, Hans-Rudolf Hotz  wrote:
>
>
>
> On 06/09/2017 03:29 PM, John Chen wrote:
>
> Hans-Rudolf,
>
> That got me past the error, but I i am now having issue authenticating
> with against AD, as if its not able to search for the users.  Do I need
> a binding service account to search AD object?  Does the bottow 5 lines
> look correct?
>
>
> They look right, but I can't say whether they are correct. You need to
> discuss this with the person who has set up your Active Directory
>
>
> Hans-Rudolf
>
>
>
>
> cn=galaxy,ou=Secu rity,ou=somegroup,dc=example,
> dc=org
>
> (&(objectCl ass=user)(sAMAccountName={
> username}))
>  ADsearchAccount< /search-user>
>  AD_Search_Pa sswrd
>  {sAMAccountName}
>
> The logs show that it found the userID and email, but gets an invalid
> password on the webportal
>
> galaxy.webapps.galaxy.controll ers.user DEBUG 2017-06-09 09:26:34,592
> trans.app.config.auth_config_f ile: ./config/auth_conf.xml
> galaxy.auth.providers.ldap_ad DEBUG 2017-06-09 09:26:34,592 LDAP
> authenticate: email is testuser.n...@example.org
> galaxy.auth.providers.ldap_ad DEBUG 2017-06-09 09:26:34,592 LDAP
> authenticate: username is testUser
> galaxy.auth.providers.ldap_ad DEBUG 2017-06-09 09:26:34,592 LDAP
> authenticate: options are {'bind-user': '{sAMAccountName}',
> 'search-fields': 'sAMAccountName,mail', 'login-use-username': 'True',
> 'allow-register': 'False', 'auto-register-email': '{mail}', 'server':
> 'ldap://xxx.xxx.xx', 'auto-register': '