Re: [Ifeffit] How to use Larch as backend for Athena and Artemis on Windows

[Matt Newville]

Hi Hiroyuki, Bruce, Carmelo, Luke, all,

Sorry for the delay in responding.  I was at the Q2XAFS workshop spending
some of my time discussing Larch.

As a relevant aside, at this meeting Yohei Uemura  showed me a very nice
Python GUI that he's been working on for fitting multiple EXAFS spectra.
He also had a very interesting poster of results of a rather amazing study
of photo-excited WO3.  If I understand correctly, Yohei used Larch as the
EXAFS calculation engine for ~30 million configurations using around 30
core-hours (that is, it took around 2 real hours on a 14 core machine).
I think we can conclude it will be good to handle 100 spectra ;).

For installing Larch, Hiroyuki's approach can work but is pretty involved.
We're recommending Anaconda Python (as Luke mentioned), as it includes many
of the needed packages by default, and has a good package manager to
install Larch and everything it needs.  It also makes pushing updates easy
for us (and easy for the end user to pull and install the updates too).

Also, Larch on Python 3.6 works for me, and I believe the XAFS code should
be clean.  But I also know we're not testing every corner of the code as
well as we should, and I would not be surprised to see some Python3 bugs.
These may be more likely if you're using HDF5 files and/or the XRF Mapping
GUI tools.

Anyway, I'm very happy to hear others getting the Larch+Demeter combination
working on Windows! For sure, please let us know what could make that
easier and better.

Cheers,


On Thu, Aug 17, 2017 at 4:21 AM, ASAKURA, Hiroyuki <
asak...@moleng.kyoto-u.ac.jp> wrote:

> Dear Bruce,
>
> I do appreciate your prompt response!
>
> Now, I managed to run Athena with Larch backend on Windows 10 and handle
> about 100 XAS data at once.
> Athena with Ifeffit backend stalls when handling about 40 XAS spectra at
> least in my case.
> Unfortunately, there seems to be a bit severe performance issue with a
> large number of spectra, as noted.
>
> Here is what I did.
>
> 0. Install an official Git package
> 1-1. Download "64-bit Git for Windows Setup" via
> https://git-scm.com/download/win
> 1-2. Install Git with default options except for "Use Git from the Windows
> Command Prompt" and "Use Windows' default console window"
> Note: You can use any options you want.
>
> 1. Install an official Demeter package
> 1-1. Download "Demeter 0.9.25 (32 bit)" via https://bruceravel.github.io/d
> emeter/
> 1-2. Install Demeter with default options
> Note: I cannot run 64 bit version with Larch.
>
> 2. Install an official Python package
> 2-1. Download "Windows x86-64 web-based installer" via
> https://www.python.org/downloads/release/python-362/
> 2-2. Custom Install Python with default options except for "Add Python 3.6
> to PATH" (checked)
>
> 3. Install unofficial Numpy and Scipy, Python libraries
> 3-1. Download numpy‑1.13.1+mkl‑cp36‑cp36m‑win_amd64.whl and
> scipy‑0.19.1‑cp36‑cp36m‑win_amd64.whl via http://www.lfd.uci.edu/~gohlke
> /pythonlibs/
> 3-2. pip install numpy-1.13.1+mkl-cp36-cp36m-win_amd64.whl
> 3-3. pip install scipy‑0.19.1‑cp36‑cp36m‑win_amd64.whl
>
> 4. Install Larch and its dependencies
> 4-1. Start a command prompt
> 4-2. git clone https://github.com/xraypy/xraylarch.git
> 4-3. cd xraylarch
> 4-4. pip install lmfit
> 4-5. pip install h5py
> 4-6. pip install pillow
> 4-7. pip install requests
> 4-8. pip install sqlalchemy
> 4-9. pip install matplotlib
> 4-10. pip install pypiwin32
> 4-11. pip install wxpython
> 4-12. pip install wxmplot
> 4-13. pip install wxutils
> 4-14. python setup.py install
> 4-15. larch --version
> 4-16. You will see "larch command-line version 0.2".
> Note: I installed Larch dependencies except for pyFAI, CifFile, fabio,
> tomopy, scikit-image.
>
> 5. Run Larch and Athena with Larch
> 5-1. Start a new command prompt
> 5-2. larch -rc
> 5-3. Start another new command prompt
> 5-4.cd %APPDATA%\DemeterPerl\perl\site\bin
> 5-5. lathena.bat
>
> It works.
> Athena is the most user-friendly XAS processing tool I have used.
>
> Thanks so much!
>
> Best regards,
>
> ASAKURA, Hiroyuki
>
>
>
> On 2017/08/17 2:17, Bruce Ravel wrote:
>
>>
>> Hi Hiroyuki,
>>
>> Sorry I didn't respond yesterday.  Yesterday was a bit of crazy day...
>>
>> I suppose that this is still a work in progress.  I believe that Demeter
>> works with Larch on Windows, but it is certainly not as easy or convenient
>> yet as it should be.
>>
>> In the place where Demeter got installed on your computer, there is a
>> folder called \perl\site\bin -- you can look at the properties of the
>> Athena desktop icon to see where that is.  In that folder is the
>> "dathena.bat" file which the Athena desktop icon points at.
>>
>> You should also find a file called "lathena.bat".  Try running that or
>> pointing your desktop icon at it instead.
>>
>> The difference between "lathena.bat" and "dathena.bat" is line 25 that
>> sets an environment variable that Demeter uses to decide which of ifeffit
>> or larch to use.
>>
>> Obviously, there 

[Ifeffit] How to use Larch as backend for Athena and Artemis on Mac (Re: Ifeffit Digest, Vol 174, Issue 8)

[Luke Higgins]

Dear All,

Following the excellent guide that Hiroyuki & Bruce gave for running larch on 
Demeter on Windows (Ifeffit Digest, Vol 174, Issue 8). The following steps 
worked for me to run larch/Demeter on MacOS (Sierra  10.12.6).

1) Install Demeter using the advice from: 
https://bruceravel.github.io/demeter/documents/SinglePage/macinstallation.html
2) Installed Anaconda and larch using the following guide: 
http://cars.uchicago.edu/xraylarch/installation/index.html (Python & larch from 
source is fine: but then use Hiroyuki’s step 3&4 advice [attached below]) 

3) In a new terminal window check larch by: 
   larch --version 
   You should see "larch command-line version 0.2".
   
4) Running larch and Demeter, run in a new terminal window: 
   larch --port=4966 –rc
   
5) Open Demeter.pm in a text editor (which for me was buried in: 
/opt/local/lib/perl5/site_perl/5.24/darwin-thread-multi-2level/Demeter.pm) and 
change line 44 to: 

   $ENV{DEMETER_BACKEND} ||= 'larch';
   
6) This has changed both Athena and Artemis to using larch – haven’t fully 
tested yet, but looks good.

Best Wishes,

Luke

(from Hhiroyuki, installing python and larch from source files:)
3. Install unofficial Numpy and Scipy, Python libraries
3-1. Download numpy‑1.13.1+mkl‑cp36‑cp36m‑win_amd64.whl and 
scipy‑0.19.1‑cp36‑cp36m‑win_amd64.whl via 
http://www.lfd.uci.edu/~gohlke/pythonlibs/
3-2. pip install numpy-1.13.1+mkl-cp36-cp36m-win_amd64.whl
3-3. pip install scipy‑0.19.1‑cp36‑cp36m‑win_amd64.whl

4. Install Larch and its dependencies
4-1. Start a command prompt
4-2. git clone https://github.com/xraypy/xraylarch.git
4-3. cd xraylarch
4-4. pip install lmfit
4-5. pip install h5py
4-6. pip install pillow
4-7. pip install requests
4-8. pip install sqlalchemy
4-9. pip install matplotlib
4-10. pip install pypiwin32
4-11. pip install wxpython
4-12. pip install wxmplot
4-13. pip install wxutils
4-14. python setup.py install
4-15. larch --version
4-16. You will see "larch command-line version 0.2".
Note: I installed Larch dependencies except for pyFAI, CifFile, fabio, tomopy, 
scikit-image. 


___
Ifeffit mailing list
Ifeffit@millenia.cars.aps.anl.gov
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

Re: [Ifeffit] Ifeffit Digest, Vol 174, Issue 8

[Bruce Ravel]

On 08/17/2017 05:16 PM, Bruce Ravel wrote:
That's a thumbs-up from me, Hiroyuki.  Carmelo set out to use Larch 
specifically to address the problem of processing many MANY files.  I 
think Athena and PrestoPronto could both have a good place in anyone's 
toolbox.



Although I was thrilled to hear that you got Larch+Demeter+Windows 
working, Hiroyuki.  It is very gratifying that a third party -- not me, 
not Matt -- was able to do so.  Whew!


B

--
 Bruce Ravel   bra...@bnl.gov

 National Institute of Standards and Technology
 Synchrotron Science Group at NSLS-II
 Building 743, Room 114
 Upton NY, 11973

 Homepage:http://bruceravel.github.io/home/
 Software:https://github.com/bruceravel
 Demeter: http://bruceravel.github.io/demeter/
___
Ifeffit mailing list
Ifeffit@millenia.cars.aps.anl.gov
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
Unsubscribe: http://millenia.cars.aps.anl.gov/mailman/options/ifeffit

Re: [Ifeffit] Ifeffit Digest, Vol 174, Issue 8

[Bruce Ravel]

That's a thumbs-up from me, Hiroyuki.  Carmelo set out to use Larch 
specifically to address the problem of processing many MANY files.  I 
think Athena and PrestoPronto could both have a good place in anyone's 
toolbox.


B


On 08/17/2017 04:29 PM, Carmelo Prestipino wrote:

Hi Hiroyuki,
I done a small interface to Larch to work with a lot of files 
http://soonready.github.io/PrestoPronto/
It is not nice as Athena and it has not all its possibility but if you 
are used to  Athena  you should not get lost (the name of parameter is 
the same).

It is not fast how it should be but give a try, maybe could be useful
Carmelo


On Thu, Aug 17, 2017 at 7:00 PM, 
> wrote:


Send Ifeffit mailing list submissions to
ifeffit@millenia.cars.aps.anl.gov


To subscribe or unsubscribe via the World Wide Web, visit
http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit

or, via email, send a message with subject or body 'help' to
ifeffit-requ...@millenia.cars.aps.anl.gov


You can reach the person managing the list at
ifeffit-ow...@millenia.cars.aps.anl.gov


When replying, please edit your Subject line so it is more specific
than "Re: Contents of Ifeffit digest..."


Today's Topics:

1. Re: How to use Larch as backend for Athena and Artemis on
   Windows (Bruce Ravel)
2. Re: How to use Larch as backend for Athena and Artemis on
   Windows (ASAKURA, Hiroyuki)


--

Message: 1
Date: Wed, 16 Aug 2017 13:17:13 -0400
From: Bruce Ravel >
To: XAFS Analysis using Ifeffit >
Subject: Re: [Ifeffit] How to use Larch as backend for Athena and
 Artemis on Windows
Message-ID: <1956f39f-85f4-7a92-a989-24a5436df...@bnl.gov
>
Content-Type: text/plain; charset=utf-8; format=flowed


Hi Hiroyuki,

Sorry I didn't respond yesterday.  Yesterday was a bit of crazy day...

I suppose that this is still a work in progress.  I believe that Demeter
works with Larch on Windows, but it is certainly not as easy or
convenient yet as it should be.

In the place where Demeter got installed on your computer, there is a
folder called \perl\site\bin -- you can look at the properties of the
Athena desktop icon to see where that is.  In that folder is the
"dathena.bat" file which the Athena desktop icon points at.

You should also find a file called "lathena.bat".  Try running that or
pointing your desktop icon at it instead.

The difference between "lathena.bat" and "dathena.bat" is line 25 that
sets an environment variable that Demeter uses to decide which of
ifeffit or larch to use.

Obviously, there are number of other ways to get this environment
variable set.  The bottom line is, yes, there is a switch.  The switch
is the DEMETER_BACKEND environment variable.

Unfortunately, that switch cannot be in the configuration files,
although what you tired in item 2 of your email was a sensible guess.
It turns out that the choice of larch/ifeffit has to made long before
the configuration files are read.

Do let me know how you get on.  While Demeter + larch should work, there
are still almost certainly problems.  I need to get a new version of the
Demeter installer out soon, so it would be nice to get some feedback in
this area.

As for using Demeter + larch + 100s of file -- it should work after a
fashion.  But I worry that Athena's performance might not be what you
are hoping for.  Athena does a lot of things that are more user-friendly
than performance-friendly


Cheers,
B


On 08/15/2017 06:10 AM, ASAKURA, Hiroyuki wrote:
 > Dear all,
 >
 > I would like to use the latest Larch as backend for Athena and
Artemis
 > on Windows to handle hundreds of XAS data (, hopefully).
 > But, I'm stuck. I would appreciate any comments and suggestions.
 >
 > Matt announced it is now possible on this list, but I still
cannot do it
 > on Windows.
 >
 > [Ifeffit] Larch 0.9.33
 >
http://millenia.cars.aps.anl.gov/pipermail/ifeffit/2017-April/009091.html

 >
 >> The "server mode" for Athena and Artemis now works on all
platforms and I
 >> strongly encourage everyone to use this over the ifeffit back end.
 >
 > Here is what I did.

Re: [Ifeffit] Ifeffit Digest, Vol 174, Issue 8

[Carmelo Prestipino]

Hi Hiroyuki,
I done a small interface to Larch to work with a lot of files
http://soonready.github.io/PrestoPronto/
It is not nice as Athena and it has not all its possibility but if you are
used to  Athena  you should not get lost (the name of parameter is the same
).
It is not fast how it should be but give a try, maybe could be useful
Carmelo


On Thu, Aug 17, 2017 at 7:00 PM, 
wrote:

> Send Ifeffit mailing list submissions to
> ifeffit@millenia.cars.aps.anl.gov
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://millenia.cars.aps.anl.gov/mailman/listinfo/ifeffit
> or, via email, send a message with subject or body 'help' to
> ifeffit-requ...@millenia.cars.aps.anl.gov
>
> You can reach the person managing the list at
> ifeffit-ow...@millenia.cars.aps.anl.gov
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Ifeffit digest..."
>
>
> Today's Topics:
>
>1. Re: How to use Larch as backend for Athena and Artemis on
>   Windows (Bruce Ravel)
>2. Re: How to use Larch as backend for Athena and Artemis on
>   Windows (ASAKURA, Hiroyuki)
>
>
> --
>
> Message: 1
> Date: Wed, 16 Aug 2017 13:17:13 -0400
> From: Bruce Ravel 
> To: XAFS Analysis using Ifeffit 
> Subject: Re: [Ifeffit] How to use Larch as backend for Athena and
> Artemis on Windows
> Message-ID: <1956f39f-85f4-7a92-a989-24a5436df...@bnl.gov>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
>
> Hi Hiroyuki,
>
> Sorry I didn't respond yesterday.  Yesterday was a bit of crazy day...
>
> I suppose that this is still a work in progress.  I believe that Demeter
> works with Larch on Windows, but it is certainly not as easy or
> convenient yet as it should be.
>
> In the place where Demeter got installed on your computer, there is a
> folder called \perl\site\bin -- you can look at the properties of the
> Athena desktop icon to see where that is.  In that folder is the
> "dathena.bat" file which the Athena desktop icon points at.
>
> You should also find a file called "lathena.bat".  Try running that or
> pointing your desktop icon at it instead.
>
> The difference between "lathena.bat" and "dathena.bat" is line 25 that
> sets an environment variable that Demeter uses to decide which of
> ifeffit or larch to use.
>
> Obviously, there are number of other ways to get this environment
> variable set.  The bottom line is, yes, there is a switch.  The switch
> is the DEMETER_BACKEND environment variable.
>
> Unfortunately, that switch cannot be in the configuration files,
> although what you tired in item 2 of your email was a sensible guess.
> It turns out that the choice of larch/ifeffit has to made long before
> the configuration files are read.
>
> Do let me know how you get on.  While Demeter + larch should work, there
> are still almost certainly problems.  I need to get a new version of the
> Demeter installer out soon, so it would be nice to get some feedback in
> this area.
>
> As for using Demeter + larch + 100s of file -- it should work after a
> fashion.  But I worry that Athena's performance might not be what you
> are hoping for.  Athena does a lot of things that are more user-friendly
> than performance-friendly
>
>
> Cheers,
> B
>
>
> On 08/15/2017 06:10 AM, ASAKURA, Hiroyuki wrote:
> > Dear all,
> >
> > I would like to use the latest Larch as backend for Athena and Artemis
> > on Windows to handle hundreds of XAS data (, hopefully).
> > But, I'm stuck. I would appreciate any comments and suggestions.
> >
> > Matt announced it is now possible on this list, but I still cannot do it
> > on Windows.
> >
> > [Ifeffit] Larch 0.9.33
> > http://millenia.cars.aps.anl.gov/pipermail/ifeffit/2017-
> April/009091.html
> >
> >> The "server mode" for Athena and Artemis now works on all platforms and
> I
> >> strongly encourage everyone to use this over the ifeffit back end.
> >
> > Here is what I did.
> >
> > Environment:
> > Windows 10 Pro (Japanese)
> > Demeter 0.9.25 (via official website)
> > Python 3.6 (official distribution)
> >
> > 1. Installation of the latest Larch
> >
> >> git clone https://github.com/xraypy/xraylarch.git (sha1:
> >> 5a372cfe1368a8ac10c2ebfc5bef4e9a9a325f6d)
> >> cd xraylarch
> >> python setup.py install
> > # Some dependency related to XRD is not fulfilled.
> >
> > 2. Start Larch server
> >
> >> cd %LOCALAPPDATA%\Programs\Python\Python36\Scripts
> >> python larch_server start
> > larch_server port=4966: started
> >
> > I found a parameter of peakfit backend in demeter.ini and changed to
> > larch, but did not work (as expected).
> >
> > In the dathena.log, Athena still uses ifeffit as backend.
> >
> >> perl version: v5.18.2
> >> backend: ifeffit
> >
> > Should I compile Demeter package by myself to use the Larch backend?
> > Or, is there any 

Re: [Ifeffit] How to use Larch as backend for Athena and Artemis on Windows

[ASAKURA, Hiroyuki]

Dear Bruce,

I do appreciate your prompt response!

Now, I managed to run Athena with Larch backend on Windows 10 and handle about 
100 XAS data at once.
Athena with Ifeffit backend stalls when handling about 40 XAS spectra at least 
in my case.
Unfortunately, there seems to be a bit severe performance issue with a large 
number of spectra, as noted.

Here is what I did.

0. Install an official Git package
1-1. Download "64-bit Git for Windows Setup" via 
https://git-scm.com/download/win
1-2. Install Git with default options except for "Use Git from the Windows Command 
Prompt" and "Use Windows' default console window"
Note: You can use any options you want.

1. Install an official Demeter package
1-1. Download "Demeter 0.9.25 (32 bit)" via 
https://bruceravel.github.io/demeter/
1-2. Install Demeter with default options
Note: I cannot run 64 bit version with Larch.

2. Install an official Python package
2-1. Download "Windows x86-64 web-based installer" via 
https://www.python.org/downloads/release/python-362/
2-2. Custom Install Python with default options except for "Add Python 3.6 to 
PATH" (checked)

3. Install unofficial Numpy and Scipy, Python libraries
3-1. Download numpy‑1.13.1+mkl‑cp36‑cp36m‑win_amd64.whl and 
scipy‑0.19.1‑cp36‑cp36m‑win_amd64.whl via 
http://www.lfd.uci.edu/~gohlke/pythonlibs/
3-2. pip install numpy-1.13.1+mkl-cp36-cp36m-win_amd64.whl
3-3. pip install scipy‑0.19.1‑cp36‑cp36m‑win_amd64.whl

4. Install Larch and its dependencies
4-1. Start a command prompt
4-2. git clone https://github.com/xraypy/xraylarch.git
4-3. cd xraylarch
4-4. pip install lmfit
4-5. pip install h5py
4-6. pip install pillow
4-7. pip install requests
4-8. pip install sqlalchemy
4-9. pip install matplotlib
4-10. pip install pypiwin32
4-11. pip install wxpython
4-12. pip install wxmplot
4-13. pip install wxutils
4-14. python setup.py install
4-15. larch --version
4-16. You will see "larch command-line version 0.2".
Note: I installed Larch dependencies except for pyFAI, CifFile, fabio, tomopy, 
scikit-image.

5. Run Larch and Athena with Larch
5-1. Start a new command prompt
5-2. larch -rc
5-3. Start another new command prompt
5-4.cd %APPDATA%\DemeterPerl\perl\site\bin
5-5. lathena.bat

It works.
Athena is the most user-friendly XAS processing tool I have used.

Thanks so much!

Best regards,

ASAKURA, Hiroyuki


On 2017/08/17 2:17, Bruce Ravel wrote:


Hi Hiroyuki,

Sorry I didn't respond yesterday.  Yesterday was a bit of crazy day...

I suppose that this is still a work in progress.  I believe that Demeter works 
with Larch on Windows, but it is certainly not as easy or convenient yet as it 
should be.

In the place where Demeter got installed on your computer, there is a folder called 
\perl\site\bin -- you can look at the properties of the Athena desktop icon to see where 
that is.  In that folder is the "dathena.bat" file which the Athena desktop 
icon points at.

You should also find a file called "lathena.bat".  Try running that or pointing 
your desktop icon at it instead.

The difference between "lathena.bat" and "dathena.bat" is line 25 that sets an 
environment variable that Demeter uses to decide which of ifeffit or larch to use.

Obviously, there are number of other ways to get this environment variable set. 
 The bottom line is, yes, there is a switch.  The switch is the DEMETER_BACKEND 
environment variable.

Unfortunately, that switch cannot be in the configuration files, although what 
you tired in item 2 of your email was a sensible guess. It turns out that the 
choice of larch/ifeffit has to made long before the configuration files are 
read.

Do let me know how you get on.  While Demeter + larch should work, there are 
still almost certainly problems.  I need to get a new version of the Demeter 
installer out soon, so it would be nice to get some feedback in this area.

As for using Demeter + larch + 100s of file -- it should work after a fashion.  
But I worry that Athena's performance might not be what you are hoping for.  
Athena does a lot of things that are more user-friendly than 
performance-friendly


Cheers,
B


On 08/15/2017 06:10 AM, ASAKURA, Hiroyuki wrote:

Dear all,

I would like to use the latest Larch as backend for Athena and Artemis on 
Windows to handle hundreds of XAS data (, hopefully).
But, I'm stuck. I would appreciate any comments and suggestions.

Matt announced it is now possible on this list, but I still cannot do it on 
Windows.

[Ifeffit] Larch 0.9.33
http://millenia.cars.aps.anl.gov/pipermail/ifeffit/2017-April/009091.html


The "server mode" for Athena and Artemis now works on all platforms and I
strongly encourage everyone to use this over the ifeffit back end.


Here is what I did.

Environment:
Windows 10 Pro (Japanese)
Demeter 0.9.25 (via official website)
Python 3.6 (official distribution)

1. Installation of the latest Larch


git clone https://github.com/xraypy/xraylarch.git (sha1: 
5a372cfe1368a8ac10c2ebfc5bef4e9a9a325f6d)
cd xraylarch
python