[Distutils] Re: setuptools reading simple index takes 30sec

2019-08-02 Thread Cralle, Chris 
Well I think I found the culprit.  I knew it must be something simple.  And it 
was.
Our pydisutils.cfg had both an index-url and find_links, both with the same 
simple url.
When I removed the find_links, all the time sinks went away.  What did take 3hr 
is now back to 24min.  /facepalm

fwiw, this is just a python setup.py test, and uses tox virtualenv setuptools.  
I also thought that setuptools just used pip, but now I see it's actually 
easy-install, which I didn't realize was a different thing from pip.
Good luck on https://github.com/pypa/setuptools/issues/917
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/LQDOSF37TR7HNGRAUCOWW4CCVPAQ33IZ/

[Distutils] Re: setuptools reading simple index takes 30sec

2019-08-02 Thread Chris C 
Well I think I found the culprit.  I knew it must be something simple.  And it 
was.
Our pydisutils.cfg had both a index-url and find_links, both with the same 
simple url.
When I removed the find_links, all the time sinks went away.  What did take 3hr 
is now back to 24min.  /facepalm

fwiw, this is just a python setup.py test, and uses tox virtualenv setuptools.  
I also thought that setuptools just used pip, but now I see it's actually 
easy-install, which I didn't realize was a different thing from pip.
Good luck on https://github.com/pypa/setuptools/issues/917
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/URXRD674YKY4G5C5YNYOT3GOB5JQQMAR/

[Distutils] Re: setuptools reading simple index takes 30sec

2019-08-01 Thread Tzu-ping Chung 
What command are you using to result in the output? The root page lists all 
available packages on that index, and with 11MB of data there must be literally 
millions of entries to parse, and would definitely take a long time. But it 
definitely don’t need to be done under normal circumstances, and pip/setuptools 
definitely does not do it under normal circumstances.


Sent from Mail for Windows 10

From: chris.cra...@rsa.com
Sent: 02 August 2019 05:06
To: distutils-sig@python.org
Subject: [Distutils] setuptools reading simple index takes 30sec

Hello,
I have a python dev team that I support in our local jenkins/artifactory 
environment.  Recently this team switched over from an old local pypi legacy 
repository to a current pypi remote/virtual of pypi.org.  So we could get them 
updated libraries and such. Anyway, I am banging my head to explain the time 
taken for reading the simple index.  This index is about 11MB, takes 1 second 
to download itself, but then it takes ~30 seconds for setuptools to scan this 
and find the package it's looking for.  Multiple this by the ~170 libraries 
they have in requirements, and the build has gone from 30minutes to almost 3 
hours.  And 90% comes from reading the simple index for each dependency.

Here's a sample output with timestamps
00:57:00.036 Searching for tenacity==4.8.0
00:57:00.037 Reading 
https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/
00:57:27.650 Reading 
https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/tenacity/
00:57:27.881 Downloading 
https://repo1.local/artifactory/api/pypi/py-pypi-virt/packages/packages/fc/e9/5499018e0d420f8d03a215c310ee7bc6e1a7e84adaa63f6ea208e864bdb6/tenacity-4.8.0-py2.py3-none-any.whl#sha256=efcf0672547f52fd49f96c2c1912e0f0e77d78a6630823aad54f99227a3c332d
00:57:28.037 Best match: tenacity 4.8.0
00:57:28.037 Processing tenacity-4.8.0-py2.py3-none-any.whl
00:57:28.038 Installing tenacity-4.8.0-py2.py3-none-any.whl to 
/home/build/workspace/workflows-pr/workflows-extension/.eggs
00:57:28.069 writing requirements to 
/home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg/EGG-INFO/requires.txt
00:57:28.114 Installed 
/home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg

Note the time on the first Reading.  I'm hoping there is some obvious simple 
solution to make this faster.
Any ideas or help would be greatly appreciated?

Thanks
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/O7M274ZYQMMU3WT52PY2YIVJMCNOQJIR/

--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/M3R6JQ5X6MN4TRYAM5JYVL2OLO52BJW7/