[cp-discuss] Advices on instructing Git lesson without password

[Maxime Boissonneault]

Hi,

I don't know if everyone is aware, but starting on August 13, 2021, 
Github is no longer going to accept password authentications:


https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations

This can prove a significant challenge in a teaching environment. I 
don't see myself explaining SSH keys or password managers or MFA in the 
context of teaching git.



Has anyone given some thoughts on this ?


Maxime Boissonneault


--
This list is for the purpose of general discussion about The Carpentries 
including community activities, upcoming events, and announcements.  Some other 
lists you may also be interested in include discuss-hpc, discuss-r, and  our 
local groups. Visit https://carpentries.topicbox.com/groups/ to learn more. All 
activity on this and other Carpentries spaces should abide by The Carpentries 
Code of Conduct found here: 
https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html

The Carpentries: discuss
Permalink: 
https://carpentries.topicbox.com/groups/discuss/T101ca604a0f953b0-M81ef7fe5ab9058a2618c0393
Delivery options: https://carpentries.topicbox.com/groups/discuss/subscription

Re: [discuss] Software Carpentry style lesson for Conda

[Maxime Boissonneault]

On 2019-06-12 10:56 AM, Michael Sarahan wrote:
That's a good point, but rather than say "don't use conda at all" - 
that's more reason to have custom channels where conda is set up to 
comply with those needs.  Conda need not be mutually exclusive with 
these things, but it does take some setup to get them working together.


That's not what I'm saying. What I'm saying is consult with your local 
experts.


On our clusters, *we* tell users don't use conda.

We provide a comprehensive list of precompiled python wheels. There is 
absolutely no need for conda in 99% of the cases.


I don't see why we would support custom conda channels when we can just 
as well support python wheels that don't require conda.



Maxime




Saying "don't use conda at all" is ignoring the work that has to 
happen either way.  Either you have to reproduce what conda is 
providing somehow, or you have to make conda use the part on the 
system side.  That's definitely a case-by-case scenario for everyone, 
and we need to document both paths.


For your example of MPI, conda packages are setup to explicitly 
require some MPI implementation where necessary. That package can come 
from an actual conda MPICH package, or it can come from a known binary 
compatible system installation that has a conda package setup to 
reference it.  Conda is not dogmatic about being hermetic (unlike, 
say, bazel).  Binary compatibility with external libraries can be 
pretty tricky, though.


On Wed, Jun 12, 2019 at 9:48 AM Maxime Boissonneault 
<mailto:maxime.boissonnea...@calculquebec.ca>> wrote:


Hi,
How about including a part about when *not* to use Conda ?

In particular, if they are going to be computing on a
supercomputer, they should consult with your cluster specialists
first.
Conda works well on somebody's desktop, but it creates a lot of
problems on supercomputers, because it does crazy stuff like
installing MPI by itself instead of relying on staff-installed
modules and software packages.

Cheers,

Maxime


On 2019-06-12 9:49 AM, David Pugh wrote:

All,

I have developed a Software Carpentry style lesson for Conda and
would be keen to get feedback from the community!

Website:

https://kaust-vislab.github.io/introduction-to-conda-for-data-scientists/

Repo:

https://github.com/kaust-vislab/introduction-to-conda-for-data-scientists

Thanks and look forward to hearing from you!

David


*The Carpentries <https://carpentries.topicbox.com/latest>* / discuss 
/ see discussions <https://carpentries.topicbox.com/groups/discuss> + 
participants <https://carpentries.topicbox.com/groups/discuss/members> 
+ delivery options 
<https://carpentries.topicbox.com/groups/discuss/subscription> 
Permalink 
<https://carpentries.topicbox.com/groups/discuss/Tb12fc97e5ee621f2-M1342d710fc4e431e3998ff41> 




--
-
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Président - Comité de coordination du soutien à la recherche de Calcul Québec
Team lead - Research Support National Team, Compute Canada
Instructeur Software Carpentry
Ph. D. en physique


--
The Carpentries: discuss
Permalink: 
https://carpentries.topicbox.com/groups/discuss/Tb12fc97e5ee621f2-M594b7e821f7b475e303c2ceb
Delivery options: https://carpentries.topicbox.com/groups/discuss/subscription

Re: [discuss] Software Carpentry style lesson for Conda

[Maxime Boissonneault]

Hi,
How about including a part about when *not* to use Conda ?

In particular, if they are going to be computing on a supercomputer, 
they should consult with your cluster specialists first.
Conda works well on somebody's desktop, but it creates a lot of problems 
on supercomputers, because it does crazy stuff like installing MPI by 
itself instead of relying on staff-installed modules and software packages.


Cheers,

Maxime


On 2019-06-12 9:49 AM, David Pugh wrote:

All,

I have developed a Software Carpentry style lesson for Conda and would 
be keen to get feedback from the community!


Website:

https://kaust-vislab.github.io/introduction-to-conda-for-data-scientists/

Repo:

https://github.com/kaust-vislab/introduction-to-conda-for-data-scientists

Thanks and look forward to hearing from you!

David
*The Carpentries * / discuss 
/ see discussions  + 
participants  
+ delivery options 
 
Permalink 
 



--
The Carpentries: discuss
Permalink: 
https://carpentries.topicbox.com/groups/discuss/Tb12fc97e5ee621f2-M3bf5ece417a2cb1b32686e63
Delivery options: https://carpentries.topicbox.com/groups/discuss/subscription

Re: [discuss] Re: SWC workshop spread over two weekends?

[Maxime Boissonneault]

Hi all,
Some feedback from me. We have attempted to have a split workshop over 
two weeks, but our experience is a large amount of "no-show" on the 
second week. I would not recommend it.



--
-----
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Président - Comité de coordination du soutien à la recherche de Calcul Québec
Team lead - Research Support National Team, Compute Canada
Instructeur Software Carpentry
Ph. D. en physique




On 2018-12-05 5:22 PM, nicholdav via discuss wrote:

Hi Raniere, Meredith, and Maneesha,

Okay, good to know (a) I'm not the only one who felt weird about this 
and (b) other people have experimented with a more spread out version 
and reported back. Looks like they address some of the exact questions 
we had. Thanks!


We definitely plan to include all the core SWC lessons.

If we end up spreading it over two weekends, we will report back and 
let you know about our experience.


Appreciate it!
David
*The Carpentries <https://carpentries.topicbox.com/latest>* / discuss 
/ see discussions <https://carpentries.topicbox.com/groups/discuss> + 
participants <https://carpentries.topicbox.com/groups/discuss/members> 
+ delivery options 
<https://carpentries.topicbox.com/groups/discuss/subscription> 
Permalink 
<https://carpentries.topicbox.com/groups/discuss/T1e9023d7a45042e7-Mf84383f1e4caee365c01b592> 



--
The Carpentries: discuss
Permalink: 
https://carpentries.topicbox.com/groups/discuss/T1e9023d7a45042e7-M29f9d4b5cc26bceb27990faa
Delivery options: https://carpentries.topicbox.com/groups/discuss/subscription

Re: [discuss] Carpentry lessons material in multiple (spoken) languages

[Maxime Boissonneault]

My two cents from a bilingual organization in Canada :

From our experience at Compute Canada (we produce most of our written 
content in both French and English), having *separate* content is a 
recipe for one of them to become outdated. What we found works well is 
MediaWiki with the Language Extension bundle. I wrote a 
software-carpentry-style lesson about OpenACC programming using that here :

https://docs.computecanada.ca/wiki/OpenACC_Tutorial

What makes things easier with this way is that
1) each page is disentangled into multiple translation units 
automatically, by the extensions
2) the extensions provide us with a quick way to spot outdated or new 
translation units, which we can then translate :

https://docs.computecanada.ca/mediawiki/index.php?title=Special%3ALanguageStats=D=fr=1


I don't know that it is possible to do something nearly as convenient 
with just gh-pages...


Regards,

Maxime

On 2018-08-28 6:30 PM, Rémi Rampin wrote:
2018-08-28 05:11 EDT, "DVD PS" <mailto:discuss@lists.carpentries.org>>:


So, over the last months, I've been working to get something that
eases the work of the translators and provides all the languages
the same visibility - i.e. one page for all!


Hi David,

I wonder if this is the right approach. There is no real benefit in 
having everything be in the same repository, so long as you have links 
pointing to the other languages. In fact, I am not sure if 
git-novice-_*es*_/01-basics/ is worse than 
git-novice/*_es/_*01-basics/ (or the current 
git-novice/*_es/_episodes/_*01-basics/).


I think having separate repos (or branches) for different languages 
would make it much easier to keep the translation up-to-date, since 
you can use Git merges in the usual way to see what have changed in 
the upstream (e.g. the English lesson) and update your translation.


I see that you are not far from this since you are in fact using a 
submodule, however:


  * submodules are error prone (especially in sub-directories), it is
likely that contributors or even maintainers will get this wrong
  * the build process gets more complicated (do contributors to the
translation have to clone the outer project instead of the
translation?)
  * the submodules for each translation will need to be updated in the
outer repo for changes in the translation to appear on the site
  * you cannot have the translation be a branch that you merge from
the English upstream (infinite recursion)

What do you think?

--
Rémi
*The Carpentries <https://carpentries.topicbox.com/latest>* / discuss 
/ see discussions <https://carpentries.topicbox.com/groups/discuss> + 
participants <https://carpentries.topicbox.com/groups/discuss/members> 
+ delivery options 
<https://carpentries.topicbox.com/groups/discuss/subscription> 
Permalink 
<https://carpentries.topicbox.com/groups/discuss/Tdb042c4bc0ecf365-M16279ab9bf8409e8faa1e004> 




--
-
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Président - Comité de coordination du soutien à la recherche de Calcul Québec
Team lead - Research Support National Team, Compute Canada
Instructeur Software Carpentry
Ph. D. en physique


--
The Carpentries: discuss
Permalink: 
https://carpentries.topicbox.com/groups/discuss/Tdb042c4bc0ecf365-M0ae9399ed69dc884844c20a8
Delivery options: https://carpentries.topicbox.com/groups/discuss/subscription

Re: [discuss] Slide of Joel Grus' JupyterCon Talk "I Don't Like Notebooks"

[Maxime Boissonneault]

Hi Carol,
I don't think this is where the subthread about Conda is heading. 
Jupyter notbooks is orthogonal to Anaconda. You can definitely have 
Jupyter without Conda. From a teaching perspective, both Conda and 
Jupyter notebooks do a fine job. But just as it would be beneficial to 
warn users about notebook caveats (hidden states and such), it would 
also be good to do the same for conda caveats (performance).


Cheers,

Maxime




On 2018-08-28 6:29 PM, Carol Willing wrote:

Hi all,

There's positive discussion that has been started by Joel's talk. While I liked 
his talk and there are some good points re: improving support for software 
engineering best practices in Jupyter and JupyterLab notebooks, I'm a bit 
concerned about the direction that this conversation is going.

While all are entitled to their personal opinions and the Carpentries will use 
notebooks when and if needed, I believe that the Carpentries would be doing its 
students a disservice by warning people not to use the notebooks or conda.

The notebooks are a popular and effective tool for scientists and data scientists to have 
in their toolbox. Project Jupyter won the ACM Software System Award recently, and the ACM 
stated "These tools, which include IPython, the Jupyter Notebook and JupyterHub, 
have become a de facto standard for data analysis in research, education, journalism and 
industry." https://awards.acm.org/software-system

While it's great for folks to have different personal perspectives, I want to 
make sure that the Carpentries and its lessons do not recommend that the 
Jupyter Notebooks, IPython, and JupyterHub should be avoided by scientists and 
data scientists.

Thanks,

Carol Willing



On 28 Aug 2018, at 11:38, Maxime Boissonneault 
 wrote:

These kinds of things are rather hard to track in time, because everything is a 
moving target (conda and other package managers constantly get updated, but 
also version of packages changes), but here is a bit more details :

- The 10x performance difference was with a user code, which I unfortunately 
can't share (nor do I still have a copy of it). It was about numpy, which may 
or may not have changed since MKL can now be shipped with Anaconda.

- FFTW, 2x performance gain : These slides compare between Conda-provided (and 
those provided by other package managers) FFTW, and one which was built on an 
avx2 cluster, the performance gain is 2x (see slides 28 and 29 :
https://archive.fosdem.org/2018/schedule/event/installing_software_for_scientists/attachments/slides/2437/export/events/attachments/installing_software_for_scientists/slides/2437/20180204_installing_software_for_scientists.pdf


- Tensorflow, 7x gain for CPU version, slide 28 of this talk : 
https://archive.fosdem.org/2018/schedule/event/how_to_make_package_managers_cry/attachments/slides/2297/export/events/attachments/how_to_make_package_managers_cry/slides/2297/how_to_make_package_managers_cry.pdf

   This one was not comparing Conda itself, but manylinux python wheels 
provided by the Tensorflow team, but no doubt Conda has the same issue if they 
build for generic architectures.



Basically, any package that is compiled in a portable manner, such as what 
Conda and manylinux wheels do, will have some degree of speedup if compiled for 
the target architecture instead. This is typically achieved by the team of 
analysts who manage a cluster.

Cheers,

Maxime


On 2018-08-28 2:20 PM, Ashwin Srinath wrote:

I'm very interested to see these examples? We use and advocate the use
of conda environments and I'm happy to be convinced otherwise.

Thanks,
Ashwin

On Tue, Aug 28, 2018 at 2:17 PM, Maxime Boissonneault
 wrote:

Regarding performance, we have example of code using Anaconda-provided
packages that run 10 times slower than the same code using locally built
packages, optimized for the cluster architectures. That's not *a bit*
slower, that's a lot slower.

Regarding "cheating on your partner", that analogy is not by me, but the
point he is trying to carry is that Anaconda basically replaces any cluster
provided versions, which HPC center people are working hard to optimize.
Recent versions of Anaconda are even worse, by packaging things like
compilers and linkers, creating conflicts with cluster-provided system
libraries and tools, and creating a lot of debugging problems for users and
support people alike.

Regards,

Maxime


On 2018-08-28 12:48 PM, Rémi Rampin wrote:

2018-08-28 12:27 EDT, Maxime Boissonneault
:

As a side-discussion, I think we should also be wary of using Anaconda,
and tell users not to use it in a cluster environment. For reasons, see
here :
https://twitter.com/mboisso/status/1034476890353020928

Hi Maxime,

All I see in this thread is that "it's like cheating on your partner" (!!!)
and it's "generically optimized software" that might be a bit slower than
locally-built libs (interesting concern when using Python, an interpreted
scripting language

Re: [discuss] Slide of Joel Grus' JupyterCon Talk "I Don't Like Notebooks"

[Maxime Boissonneault]

These kinds of things are rather hard to track in time, because 
everything is a moving target (conda and other package managers 
constantly get updated, but also version of packages changes), but here 
is a bit more details :


- The 10x performance difference was with a user code, which I 
unfortunately can't share (nor do I still have a copy of it). It was 
about numpy, which may or may not have changed since MKL can now be 
shipped with Anaconda.


- FFTW, 2x performance gain : These slides compare between 
Conda-provided (and those provided by other package managers) FFTW, and 
one which was built on an avx2 cluster, the performance gain is 2x (see 
slides 28 and 29 :

https://archive.fosdem.org/2018/schedule/event/installing_software_for_scientists/attachments/slides/2437/export/events/attachments/installing_software_for_scientists/slides/2437/20180204_installing_software_for_scientists.pdf


- Tensorflow, 7x gain for CPU version, slide 28 of this talk : 
https://archive.fosdem.org/2018/schedule/event/how_to_make_package_managers_cry/attachments/slides/2297/export/events/attachments/how_to_make_package_managers_cry/slides/2297/how_to_make_package_managers_cry.pdf


  This one was not comparing Conda itself, but manylinux python wheels 
provided by the Tensorflow team, but no doubt Conda has the same issue 
if they build for generic architectures.




Basically, any package that is compiled in a portable manner, such as 
what Conda and manylinux wheels do, will have some degree of speedup if 
compiled for the target architecture instead. This is typically achieved 
by the team of analysts who manage a cluster.


Cheers,

Maxime


On 2018-08-28 2:20 PM, Ashwin Srinath wrote:

I'm very interested to see these examples? We use and advocate the use
of conda environments and I'm happy to be convinced otherwise.

Thanks,
Ashwin

On Tue, Aug 28, 2018 at 2:17 PM, Maxime Boissonneault
 wrote:

Regarding performance, we have example of code using Anaconda-provided
packages that run 10 times slower than the same code using locally built
packages, optimized for the cluster architectures. That's not *a bit*
slower, that's a lot slower.

Regarding "cheating on your partner", that analogy is not by me, but the
point he is trying to carry is that Anaconda basically replaces any cluster
provided versions, which HPC center people are working hard to optimize.
Recent versions of Anaconda are even worse, by packaging things like
compilers and linkers, creating conflicts with cluster-provided system
libraries and tools, and creating a lot of debugging problems for users and
support people alike.

Regards,

Maxime


On 2018-08-28 12:48 PM, Rémi Rampin wrote:

2018-08-28 12:27 EDT, Maxime Boissonneault
:

As a side-discussion, I think we should also be wary of using Anaconda,
and tell users not to use it in a cluster environment. For reasons, see
here :
https://twitter.com/mboisso/status/1034476890353020928


Hi Maxime,

All I see in this thread is that "it's like cheating on your partner" (!!!)
and it's "generically optimized software" that might be a bit slower than
locally-built libs (interesting concern when using Python, an interpreted
scripting language (and on the slow side too)).

Could you elaborate on those reasons?

Best
--
Rémi


The Carpentries / discuss / see discussions + participants + delivery
options Permalink

--
The Carpentries: discuss
Permalink: 
https://carpentries.topicbox.com/groups/discuss/T1505f74d7f6e32f8-Mad4fadc6a6da6de2b5f2aeb9
Delivery options: https://carpentries.topicbox.com/groups/discuss/subscription



--
-----
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Président - Comité de coordination du soutien à la recherche de Calcul Québec
Team lead - Research Support National Team, Compute Canada
Instructeur Software Carpentry
Ph. D. en physique


--
The Carpentries: discuss
Permalink: 
https://carpentries.topicbox.com/groups/discuss/T1505f74d7f6e32f8-M37b48fabbde91daec4f331d7
Delivery options: https://carpentries.topicbox.com/groups/discuss/subscription

Re: [discuss] Slide of Joel Grus' JupyterCon Talk "I Don't Like Notebooks"

[Maxime Boissonneault]

Regarding performance, we have example of code using Anaconda-provided 
packages that run 10 times slower than the same code using locally built 
packages, optimized for the cluster architectures. That's not *a bit* 
slower, that's a lot slower.


Regarding "cheating on your partner", that analogy is not by me, but the 
point he is trying to carry is that Anaconda basically replaces any 
cluster provided versions, which HPC center people are working hard to 
optimize. Recent versions of Anaconda are even worse, by packaging 
things like compilers and linkers, creating conflicts with 
cluster-provided system libraries and tools, and creating a lot of 
debugging problems for users and support people alike.


Regards,

Maxime


On 2018-08-28 12:48 PM, Rémi Rampin wrote:
2018-08-28 12:27 EDT, Maxime Boissonneault 
<mailto:maxime.boissonnea...@calculquebec.ca>>:


As a side-discussion, I think we should also be wary of using
Anaconda,
and tell users not to use it in a cluster environment. For
reasons, see
here :
https://twitter.com/mboisso/status/1034476890353020928


Hi Maxime,

All I see in this thread is that "it's like cheating on your partner" 
(!!!) and it's "generically optimized software" that might be a bit 
slower than locally-built libs (interesting concern when using Python, 
an interpreted scripting language (and on the slow side too)).


Could you elaborate on those reasons?

Best
--
Rémi


--
The Carpentries: discuss
Permalink: 
https://carpentries.topicbox.com/groups/discuss/T1505f74d7f6e32f8-Mb3de150d1dd911032d34b1e8
Delivery options: https://carpentries.topicbox.com/groups/discuss/subscription

Re: [discuss] Slide of Joel Grus' JupyterCon Talk "I Don't Like Notebooks"

[Maxime Boissonneault]

As a side-discussion, I think we should also be wary of using Anaconda, 
and tell users not to use it in a cluster environment. For reasons, see 
here :

https://twitter.com/mboisso/status/1034476890353020928

Maxime


On 2018-08-24 4:47 PM, Konrad Förstner wrote:

Dear all,

these are the slide of Joel Grus' Jupyter Con talk "I Don't Like
Notebooks":

https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit#slide=id.g362da58057_0_1 



Beside the fact that this talk is it really funny, it raises a lot of
issues that I can confirm from my experience:

- hidden states
- encouraging bad habits and discouraging good habits
- less powerful help tooltip than in a proper IDEs
- copy and paste between different media is hard

I personally really like Jupyter Notebooks for teaching (while it
never made into my data analysis tool box) but this talked motivated
me to rethink its usage and maybe I will try to jump earlier than
before into a proper IDE when I am teaching Python. Anaconda comes
with Spyder so there is a good alternative already at hand.

I would be interested in hearing your thoughts regarding this.

Best wishes

Konrad


--
The Carpentries: discuss
Permalink: 
https://carpentries.topicbox.com/groups/discuss/T1505f74d7f6e32f8-M84f62eac740739ea0a0f32af
Delivery options: 
https://carpentries.topicbox.com/groups/discuss/subscription



--
-----
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Président - Comité de coordination du soutien à la recherche de Calcul Québec
Team lead - Research Support National Team, Compute Canada
Instructeur Software Carpentry
Ph. D. en physique


--
The Carpentries: discuss
Permalink: 
https://carpentries.topicbox.com/groups/discuss/T1505f74d7f6e32f8-Mc3ce4dda4ce68f5225691f61
Delivery options: https://carpentries.topicbox.com/groups/discuss/subscription