Re: [Distutils] GSoC 2017 - RFC on Proposal

[Nick Coghlan]

On 26 March 2017 at 01:46, Pradyun Gedam  wrote:
> Hello Everyone!
>
> I had previously sent a mail on this list, stating that I would like to work
> on pip's dependency resolution for my GSoC 2017. I now have drafted a
> proposal for the same; with help from my mentors - Donald Stufft and Justin
> Cappos. I'd also take this opportunity to thank them for agreeing to be my
> mentors for this GSoC.

The problem description and proposed resolution plan both look
excellent to me - thank you to you all, and I hope the project goes
well!

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] GSoC 2017 - RFC on Proposal

[Pradyun Gedam]

I apologize for the duplicate mail. Please respond on this thread.

On Sat, 25 Mar 2017 at 21:16 Pradyun Gedam  wrote:

> Hello Everyone!
>
> I had previously sent a mail on this list, stating that I would like to
> work on pip's dependency resolution for my GSoC 2017. I now have drafted a
> proposal for the same; with help from my mentors - Donald Stufft and Justin
> Cappos. I'd also take this opportunity to thank them for agreeing to be my
> mentors for this GSoC.
>
> I would like to request for comments on my proposal for GSoC - it is
> hosted at
> https://gist.github.com/pradyunsg/5cf4a35b81f08b6432f280aba6f511eb.
>
> Please find trailing a MarkDown version of the proposal.
>
> Thanks,
> Pradyun Gedam
>
> -
>
> # Adding Proper Dependency Resolution to pip
>
> - **Name:** Pradyun S. Gedam
> - **Email:** [pradyu...@gmail.com][mailto-email]
> - **Github:** [pradyunsg][github-profile]
> - **University:** [VIT University, Vellore, India][vit-homepage]
> - **Course:** Bachelor of Technology in Computer Science and Engineering
> - **Course Term:** 2016/17 - 2019/20 (4 Year)
> - **Timezone:** IST (GMT +5:30)
> - **GSoC Blog RSS Feed URL:** <
> https://pradyunsg.github.io/gsoc-2017/feed.xml>
>
> [github-profile]: http://github.com/pradyunsg/
> [vit-homepage]: http://vit.ac.in/
> [mailto-email]: mailto:pradyu...@gmail.com
>
> ## About Me
>
> I was introduced to Python about five years ago, through "Core Python
> Programming" by Weasley J Chun. After the initial struggle with Python
> 2/3, the
> ball was set rolling and hasn't stopped since. I have fiddled around with
> Game Programming (PyGame), Computer Vision (OpenCV), Data Analytics
> (Pandas,
> SciPy, NumPy), transcompilers (Py2C) and more.
>
> As a high school student, I did internship at Enthought in 2013 and 2014.
> The two summers that I spent at Enthought were a great learning experience
> and
> I thoroughly enjoyed the environment there.
>
> Other than Python, I have also used C, C++, Web Technologies (JavaScript,
> HTML,
> CSS) and preprocessors (Pug, TypeScript, LESS/SASS/SCSS), Java and
> Bash/Zsh for
> some other projects.
>
> Curious to understand how pip works, I began looking into pip's source
> code.
> I started contributing to pip in May 2016. I am now fairly familiar with
> the
> design of pip and have a fair understanding of how it works, due to the
> various
> contributions I have made to pip in the past year.
>
> ## Mentors
>
> - Donald Stufft (@dstufft on GitHub)
> - Justin Cappos (@JustinCappos on GitHub)
>
> Communication with the mentors will be done mostly on issues and pull
> requests
> on pip's GitHub repository. If at any point in time, a real time
> discussion is
> to be done with the mentors, it would be done over IRC or Skype. Email can
> also
> be used if needed.
>
> ## Proposal
>
> This project is regarding improving dependency resolution performed within
> pip
> by implementing a dependency resolver within it.
>
> ### Abstract
>
> Currently, pip does not resolve dependencies correctly when there are
> conflicting requirements. The lack of dependency resolution has caused
> hard-to-debug bugs/failures due to the installation of incompatible
> packages.
> The lack of a dependency resolver is also a blocker for various other
> features -
> adding an upgrade-all functionality to pip and properly determining
> build-time
> dependencies for packages are two such features.
>
> ### Deliverables
>
> At the end of this project, pip will have the ability to:
>
> - detect requirement conflicts
> - resolve conflicting dependency requirements where possible
>
> ### Implementation
>
> The implementation language will be Python. The code will maintain the
> compatibility requirements of pip - the same source code will support the
> multiple Python implementations and version, including but not limited to,
> CPython 2.7, CPython 3.3+, PyPy 2, PyPy3.
>
> New Tests for the functionality introduced will be added to pip's current
> test
> suite.
>
> User documentation would not be a major part of this project. The only
> changes
> would be to mention pip can now resolve dependencies properly. There are
> certain
> sections that might need updating.
>
>  Overview
>
> The project will be composed of the following stages:
>
> 1. Refactor the dependency resolution logic of pip into a separate module.
> 1. Implement dependency information caching in pip.
> 1. Implement a backtracking dependency resolver to resolve the
> dependencies.
>
> Every stage depends on the previous ones being completed. This step-wise
> approach would make incremental improvements so that there is a constant
> feedback on the work being done as well as making it easy to change course
> without losing existing work; if needed for some unforeseen reason.
>
>  Discussion
>
> There is a class in pip - `RequirementSet`, which is currently a God
> class. It
> is responsible for the following:
>
>   1. Holding a list of things to be installed
>   1. 

[Distutils] GSoC 2017 - RFC on Proposal

[Pradyun Gedam]

Hello Everyone!

I had previously sent a mail on this list, stating that I would like to
work on pip's dependency resolution for my GSoC 2017. I now have drafted a
proposal for the same; with help from my mentors - Donald Stufft and Justin
Cappos. I'd also take this opportunity to thank them for agreeing to be my
mentors for this GSoC.

I would like to request for comments on my proposal for GSoC - it is hosted
at https://gist.github.com/pradyunsg/5cf4a35b81f08b6432f280aba6f511eb.

Please find trailing a MarkDown version of the proposal.

Thanks,
Pradyun Gedam

-

# Adding Proper Dependency Resolution to pip

- **Name:** Pradyun S. Gedam
- **Email:** [pradyu...@gmail.com][mailto-email]
- **Github:** [pradyunsg][github-profile]
- **University:** [VIT University, Vellore, India][vit-homepage]
- **Course:** Bachelor of Technology in Computer Science and Engineering
- **Course Term:** 2016/17 - 2019/20 (4 Year)
- **Timezone:** IST (GMT +5:30)
- **GSoC Blog RSS Feed URL:** <
https://pradyunsg.github.io/gsoc-2017/feed.xml>

[github-profile]: http://github.com/pradyunsg/
[vit-homepage]: http://vit.ac.in/
[mailto-email]: mailto:pradyu...@gmail.com

## About Me

I was introduced to Python about five years ago, through "Core Python
Programming" by Weasley J Chun. After the initial struggle with Python 2/3,
the
ball was set rolling and hasn't stopped since. I have fiddled around with
Game Programming (PyGame), Computer Vision (OpenCV), Data Analytics (Pandas,
SciPy, NumPy), transcompilers (Py2C) and more.

As a high school student, I did internship at Enthought in 2013 and 2014.
The two summers that I spent at Enthought were a great learning experience
and
I thoroughly enjoyed the environment there.

Other than Python, I have also used C, C++, Web Technologies (JavaScript,
HTML,
CSS) and preprocessors (Pug, TypeScript, LESS/SASS/SCSS), Java and Bash/Zsh
for
some other projects.

Curious to understand how pip works, I began looking into pip's source code.
I started contributing to pip in May 2016. I am now fairly familiar with the
design of pip and have a fair understanding of how it works, due to the
various
contributions I have made to pip in the past year.

## Mentors

- Donald Stufft (@dstufft on GitHub)
- Justin Cappos (@JustinCappos on GitHub)

Communication with the mentors will be done mostly on issues and pull
requests
on pip's GitHub repository. If at any point in time, a real time discussion
is
to be done with the mentors, it would be done over IRC or Skype. Email can
also
be used if needed.

## Proposal

This project is regarding improving dependency resolution performed within
pip
by implementing a dependency resolver within it.

### Abstract

Currently, pip does not resolve dependencies correctly when there are
conflicting requirements. The lack of dependency resolution has caused
hard-to-debug bugs/failures due to the installation of incompatible
packages.
The lack of a dependency resolver is also a blocker for various other
features -
adding an upgrade-all functionality to pip and properly determining
build-time
dependencies for packages are two such features.

### Deliverables

At the end of this project, pip will have the ability to:

- detect requirement conflicts
- resolve conflicting dependency requirements where possible

### Implementation

The implementation language will be Python. The code will maintain the
compatibility requirements of pip - the same source code will support the
multiple Python implementations and version, including but not limited to,
CPython 2.7, CPython 3.3+, PyPy 2, PyPy3.

New Tests for the functionality introduced will be added to pip's current
test
suite.

User documentation would not be a major part of this project. The only
changes
would be to mention pip can now resolve dependencies properly. There are
certain
sections that might need updating.

 Overview

The project will be composed of the following stages:

1. Refactor the dependency resolution logic of pip into a separate module.
1. Implement dependency information caching in pip.
1. Implement a backtracking dependency resolver to resolve the dependencies.

Every stage depends on the previous ones being completed. This step-wise
approach would make incremental improvements so that there is a constant
feedback on the work being done as well as making it easy to change course
without losing existing work; if needed for some unforeseen reason.

 Discussion

There is a class in pip - `RequirementSet`, which is currently a God class.
It
is responsible for the following:

  1. Holding a list of things to be installed
  1. Downloading Files
  1. Dependency Resolution
  1. Unpacking downloaded files (preparation for installation)
  1. Uninstalling packages
  1. Installing packages

This is clearly a bad situation since this is most of the heavy lifting
involved in pip. These responsibilities need to be separated and moved out
into
their independent modules/classes, to allow for simplification of