[Impala-ASF-CR] Speed up Python dependencies.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 5: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Tue, 01 May 2018 22:12:38 +
Gerrit-HasComments: No

[Impala-ASF-CR] Speed up Python dependencies.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..

Speed up Python dependencies.

This parallelizes downloading some Python libraries, giving a speedup of
$IMPALA_HOME/infra/python/deps/download_requirements.  I've seen this
take from 7-15 seconds before and from 2-5 seconds after.

I also checked that we always have at least Python 2.6 when
building Impala, so I was able to remove the try/except
handling in bootstrap_toolchain.

Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Reviewed-on: http://gerrit.cloudera.org:8080/10234
Reviewed-by: Philip Zeyliger 
Tested-by: Impala Public Jenkins 
---
M bin/bootstrap_toolchain.py
M infra/python/deps/pip_download.py
2 files changed, 27 insertions(+), 33 deletions(-)

Approvals:
  Philip Zeyliger: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 6
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 

[Impala-ASF-CR] Speed up Python dependencies.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2391/


--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Tue, 01 May 2018 18:23:10 +
Gerrit-HasComments: No

[Impala-ASF-CR] Speed up Python dependencies.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 5: Code-Review+2

(3 comments)

I impelemented your suggestions. I ran "buildall" long enough to get to the C++ 
compilation; verification to be done by GVO.

http://gerrit.cloudera.org:8080/#/c/10234/4/infra/python/deps/pip_download.py
File infra/python/deps/pip_download.py:

http://gerrit.cloudera.org:8080/#/c/10234/4/infra/python/deps/pip_download.py@141
PS4, Line 141: for line in open(requirements_file):
> Nit: prefer open (https://docs.python.org/2/library/functions.html#file)
Done


http://gerrit.cloudera.org:8080/#/c/10234/4/infra/python/deps/pip_download.py@150
PS4, Line 150:   continue
 :   pkg_name, pkg_version = l.split('==')
 :   results.append(pool.apply_async(
 : download_packa
> These four lines can be simplified as:
Done


http://gerrit.cloudera.org:8080/#/c/10234/4/infra/python/deps/pip_download.py@154
PS4, Line 154:
> Invert check and use continue to reduce nesting:
Done



--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Tue, 01 May 2018 18:22:43 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] Speed up Python dependencies.

[Philip Zeyliger (Code Review)]

Hello Taras Bobrovytsky, Lars Volker, Michael Brown, Joe McDonnell,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/10234

to look at the new patch set (#5).

Change subject: Speed up Python dependencies.
..

Speed up Python dependencies.

This parallelizes downloading some Python libraries, giving a speedup of
$IMPALA_HOME/infra/python/deps/download_requirements.  I've seen this
take from 7-15 seconds before and from 2-5 seconds after.

I also checked that we always have at least Python 2.6 when
building Impala, so I was able to remove the try/except
handling in bootstrap_toolchain.

Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
---
M bin/bootstrap_toolchain.py
M infra/python/deps/pip_download.py
2 files changed, 27 insertions(+), 33 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/10234/5
--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 

[Impala-ASF-CR] Speed up Python dependencies.

[Lars Volker (Code Review)]

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 4: Code-Review+2

(3 comments)

I left a few nits that could reduce the code a bit, but I'm also good to limit 
this change to adding parallelism. Apologies for the incremental review.

http://gerrit.cloudera.org:8080/#/c/10234/4/infra/python/deps/pip_download.py
File infra/python/deps/pip_download.py:

http://gerrit.cloudera.org:8080/#/c/10234/4/infra/python/deps/pip_download.py@141
PS4, Line 141: for line in file(requirements_file):
Nit: prefer open (https://docs.python.org/2/library/functions.html#file)


http://gerrit.cloudera.org:8080/#/c/10234/4/infra/python/deps/pip_download.py@150
PS4, Line 150: semi_colon_index = line.find(';')
 :   if semi_colon_index != -1:
 : line = line[:semi_colon_index]
 :   l = line.strip()
These four lines can be simplified as:

line.split(';')[0].strip()

The same works for hash_index above.


http://gerrit.cloudera.org:8080/#/c/10234/4/infra/python/deps/pip_download.py@154
PS4, Line 154:   if len(l) > 0:
Invert check and use continue to reduce nesting:

  if not line:
continue



--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Tue, 01 May 2018 04:21:29 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] Speed up Python dependencies.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/10234/3/infra/python/deps/pip_download.py
File infra/python/deps/pip_download.py:

http://gerrit.cloudera.org:8080/#/c/10234/3/infra/python/deps/pip_download.py@140
PS3, Line 140: f = open(requirements_file, 'r')
> While you are making changes here, you could actually change this to "for l
Done


http://gerrit.cloudera.org:8080/#/c/10234/3/infra/python/deps/pip_download.py@158
PS3, Line 158:   if pool:
> This check seems unnecessary now
Done



--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Tue, 01 May 2018 04:03:09 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] Speed up Python dependencies.

[Philip Zeyliger (Code Review)]

Hello Taras Bobrovytsky, Lars Volker, Michael Brown, Joe McDonnell,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/10234

to look at the new patch set (#4).

Change subject: Speed up Python dependencies.
..

Speed up Python dependencies.

This parallelizes downloading some Python libraries, giving a speedup of
$IMPALA_HOME/infra/python/deps/download_requirements.  I've seen this
take from 7-15 seconds before and from 2-5 seconds after.

I also checked that we always have at least Python 2.6 when
building Impala, so I was able to remove the try/except
handling in bootstrap_toolchain.

Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
---
M bin/bootstrap_toolchain.py
M infra/python/deps/pip_download.py
2 files changed, 31 insertions(+), 33 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/10234/4
--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 

[Impala-ASF-CR] Speed up Python dependencies.

[Lars Volker (Code Review)]

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/10234/3/infra/python/deps/pip_download.py
File infra/python/deps/pip_download.py:

http://gerrit.cloudera.org:8080/#/c/10234/3/infra/python/deps/pip_download.py@140
PS3, Line 140: f = open(requirements_file, 'r')
While you are making changes here, you could actually change this to "for line 
in open(...):", which takes care of closing the file, even if it hits an 
exception.


http://gerrit.cloudera.org:8080/#/c/10234/3/infra/python/deps/pip_download.py@158
PS3, Line 158:   if pool:
This check seems unnecessary now



--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Tue, 01 May 2018 03:53:09 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] Speed up Python dependencies.

[Philip Zeyliger (Code Review)]

Hello Taras Bobrovytsky, Lars Volker, Michael Brown, Joe McDonnell,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/10234

to look at the new patch set (#3).

Change subject: Speed up Python dependencies.
..

Speed up Python dependencies.

This parallelizes downloading some Python libraries, giving a speedup of
$IMPALA_HOME/infra/python/deps/download_requirements.  I've seen this
take from 7-15 seconds before and from 2-5 seconds after.

I also checked that we always have at least Python 2.6 when
building Impala, so I was able to remove the try/except
handling in bootstrap_toolchain.

Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
---
M bin/bootstrap_toolchain.py
M infra/python/deps/pip_download.py
2 files changed, 18 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/10234/3
--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 

[Impala-ASF-CR] Speed up Python dependencies.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 2:

(3 comments)

I'm re-running tests on the updated patch.

http://gerrit.cloudera.org:8080/#/c/10234/2/infra/python/deps/pip_download.py
File infra/python/deps/pip_download.py:

http://gerrit.cloudera.org:8080/#/c/10234/2/infra/python/deps/pip_download.py@137
PS2, Line 137: results = []
> Hoist this out of the loop (see my comment below).
Done


http://gerrit.cloudera.org:8080/#/c/10234/2/infra/python/deps/pip_download.py@139
PS2, Line 139: # Multiprocessing was introduced in Python2.6; fail 
gracefully if
> If it was introduced in 2.6, can we not rely on it (I thought that was the
I think we can. I observed that even when we build for CentOS 5, we have a copy 
of Python2.6 for our build. (I observed it doing threading in 
bootstrap_toolchain.py in the log.)


http://gerrit.cloudera.org:8080/#/c/10234/2/infra/python/deps/pip_download.py@172
PS2, Line 172: if results:
> Nice catch. We can get rid of the "if results:" line
Done



--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Tue, 01 May 2018 03:15:44 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] Speed up Python dependencies.

[Taras Bobrovytsky (Code Review)]

Taras Bobrovytsky has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10234/2/infra/python/deps/pip_download.py
File infra/python/deps/pip_download.py:

http://gerrit.cloudera.org:8080/#/c/10234/2/infra/python/deps/pip_download.py@172
PS2, Line 172: if results:
> If the import above fails, then results = [] will never execute and this st
Nice catch. We can get rid of the "if results:" line



--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Fri, 27 Apr 2018 22:03:35 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] Speed up Python dependencies.

[Lars Volker (Code Review)]

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 2: Code-Review-1

(3 comments)

I think this has a bug when multiprocessing fails to import.

http://gerrit.cloudera.org:8080/#/c/10234/2/infra/python/deps/pip_download.py
File infra/python/deps/pip_download.py:

http://gerrit.cloudera.org:8080/#/c/10234/2/infra/python/deps/pip_download.py@137
PS2, Line 137: results = []
Hoist this out of the loop (see my comment below).


http://gerrit.cloudera.org:8080/#/c/10234/2/infra/python/deps/pip_download.py@139
PS2, Line 139: # Multiprocessing was introduced in Python2.6; fail 
gracefully if
If it was introduced in 2.6, can we not rely on it (I thought that was the 
earliest Python version we support).


http://gerrit.cloudera.org:8080/#/c/10234/2/infra/python/deps/pip_download.py@172
PS2, Line 172: if results:
If the import above fails, then results = [] will never execute and this 
statement will fail (unless I'm missing something). Moving it out of the loop 
above also allows to get rid of the "if ".



--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Fri, 27 Apr 2018 21:44:55 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] Speed up Python dependencies.

[Taras Bobrovytsky (Code Review)]

Taras Bobrovytsky has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 2: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Fri, 27 Apr 2018 21:34:17 +
Gerrit-HasComments: No

[Impala-ASF-CR] Speed up Python dependencies.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/10234/1/infra/python/deps/pip_download.py
File infra/python/deps/pip_download.py:

http://gerrit.cloudera.org:8080/#/c/10234/1/infra/python/deps/pip_download.py@165
PS1, Line 165: results.append(pool.apply_async(download_package, 
args=[pkg_name.strip(), pkg_version.strip()]))
> You can use "git diff HEAD^ | impala-flake8 --diff" and similar patterns to
Done


http://gerrit.cloudera.org:8080/#/c/10234/1/infra/python/deps/pip_download.py@173
PS1, Line 173: x.get()
> What happens if the download fails for some reason? Will this be detected?
Yes, it works correctly, I believe.

>>> from multiprocessing.pool import ThreadPool
>>> p = ThreadPool()
>>> def f(a):
...   raise Exception("hey")
...
>>> x = p.apply_async(f, [1])
>>> x.get()
Traceback (most recent call last):
  File "", line 1, in 
  File "/Users/philip/anaconda/lib/python2.7/multiprocessing/pool.py", line 
567, in get
raise self._value
Exception: hey



--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Fri, 27 Apr 2018 20:17:03 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] Speed up Python dependencies.

[Philip Zeyliger (Code Review)]

Hello Taras Bobrovytsky, Michael Brown, Joe McDonnell,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/10234

to look at the new patch set (#2).

Change subject: Speed up Python dependencies.
..

Speed up Python dependencies.

This parallelizes downloading some Python libraries, giving a speedup of
$IMPALA_HOME/infra/python/deps/download_requirements.  I've seen this
take from 7-15 seconds before and from 2-5 seconds after.

Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
---
M infra/python/deps/pip_download.py
1 file changed, 19 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/10234/2
--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Taras Bobrovytsky 

[Impala-ASF-CR] Speed up Python dependencies.

[Michael Brown (Code Review)]

Michael Brown has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10234/1/infra/python/deps/pip_download.py
File infra/python/deps/pip_download.py:

http://gerrit.cloudera.org:8080/#/c/10234/1/infra/python/deps/pip_download.py@165
PS1, Line 165: results.append(pool.apply_async(download_package, 
args=[pkg_name.strip(), pkg_version.strip()]))
> long line
You can use "git diff HEAD^ | impala-flake8 --diff" and similar patterns to 
find problems like this.



--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Fri, 27 Apr 2018 20:12:03 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] Speed up Python dependencies.

[Taras Bobrovytsky (Code Review)]

Taras Bobrovytsky has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10234 )

Change subject: Speed up Python dependencies.
..


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/10234/1/infra/python/deps/pip_download.py
File infra/python/deps/pip_download.py:

http://gerrit.cloudera.org:8080/#/c/10234/1/infra/python/deps/pip_download.py@165
PS1, Line 165: results.append(pool.apply_async(download_package, 
args=[pkg_name.strip(), pkg_version.strip()]))
long line


http://gerrit.cloudera.org:8080/#/c/10234/1/infra/python/deps/pip_download.py@173
PS1, Line 173: x.get()
What happens if the download fails for some reason? Will this be detected?



--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Comment-Date: Fri, 27 Apr 2018 20:03:56 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] Speed up Python dependencies.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10234


Change subject: Speed up Python dependencies.
..

Speed up Python dependencies.

This parallelizes downloading some Python libraries, giving a speedup of
$IMPALA_HOME/infra/python/deps/download_requirements.  I've seen this
take from 7-15 seconds before and from 2-5 seconds after.

Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
---
M infra/python/deps/pip_download.py
1 file changed, 18 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/10234/1
--
To view, visit http://gerrit.cloudera.org:8080/10234
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7cbf622adb7d037f1a53c519402dcd8ae3c0fe30
Gerrit-Change-Number: 10234
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger