[jira] [Updated] (THRIFT-4026) TSSLSocket doesn't work with Python < 2.7.9
[
https://issues.apache.org/jira/browse/THRIFT-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aki Sukegawa updated THRIFT-4026:
-
Summary: TSSLSocket doesn't work with Python < 2.7.9 (was: TSSLSocket
doesn't work with Python 2.7)
> TSSLSocket doesn't work with Python < 2.7.9
> ---
>
> Key: THRIFT-4026
> URL: https://issues.apache.org/jira/browse/THRIFT-4026
> Project: Thrift
> Issue Type: Bug
> Components: Python - Library
>Affects Versions: 0.10.0
> Environment: Python 2.7.5
>Reporter: Benoit Sigoure
>Priority: Critical
>
> As a result of a change made in THRIFT-3917, {{TSSLSocket}} now fails with a
> {{TypeError}} in {{open()}}:
> {code}
> File
> "/bld/thrift/Artools-rpmbuild/thrift-0.10.0/lib/py/build/lib.linux-i686-2.7/thrift/transport/TSSLSocket.py",
> line 293, in open
> self._validate_callback(self.peercert, self._server_hostname)
> TypeError: legacy_validate_callback() takes exactly 3 arguments (2 given)
> {code}
> Indeed, {{legacy_validate_callback()}} takes an extra {{self}} argument,
> which cannot possibly be compatible with the signature of
> [{{ssl.match_hostname()}}|https://docs.python.org/3/library/ssl.html#ssl.match_hostname]
> (which was added in Python 3.2).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (THRIFT-4026) TSSLSocket doesn't work with Python 2.7
[
https://issues.apache.org/jira/browse/THRIFT-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843901#comment-15843901
]
Aki Sukegawa commented on THRIFT-4026:
--
[~tsuna] thanks, it seems indeed a bug.
For anyone who wants to workaround this without managing local patch/fork or
upgrading to python >= 2.7.9,
a recommended workaround is to install backports.ssl_match_hostname package.
> TSSLSocket doesn't work with Python 2.7
> ---
>
> Key: THRIFT-4026
> URL: https://issues.apache.org/jira/browse/THRIFT-4026
> Project: Thrift
> Issue Type: Bug
> Components: Python - Library
>Affects Versions: 0.10.0
> Environment: Python 2.7.5
>Reporter: Benoit Sigoure
>Priority: Critical
>
> As a result of a change made in THRIFT-3917, {{TSSLSocket}} now fails with a
> {{TypeError}} in {{open()}}:
> {code}
> File
> "/bld/thrift/Artools-rpmbuild/thrift-0.10.0/lib/py/build/lib.linux-i686-2.7/thrift/transport/TSSLSocket.py",
> line 293, in open
> self._validate_callback(self.peercert, self._server_hostname)
> TypeError: legacy_validate_callback() takes exactly 3 arguments (2 given)
> {code}
> Indeed, {{legacy_validate_callback()}} takes an extra {{self}} argument,
> which cannot possibly be compatible with the signature of
> [{{ssl.match_hostname()}}|https://docs.python.org/3/library/ssl.html#ssl.match_hostname]
> (which was added in Python 3.2).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (THRIFT-4026) TSSLSocket doesn't work with Python < 2.7.9
[
https://issues.apache.org/jira/browse/THRIFT-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aki Sukegawa reassigned THRIFT-4026:
Assignee: Aki Sukegawa
> TSSLSocket doesn't work with Python < 2.7.9
> ---
>
> Key: THRIFT-4026
> URL: https://issues.apache.org/jira/browse/THRIFT-4026
> Project: Thrift
> Issue Type: Bug
> Components: Python - Library
>Affects Versions: 0.10.0
> Environment: Python 2.7.5
>Reporter: Benoit Sigoure
>Assignee: Aki Sukegawa
>Priority: Critical
>
> As a result of a change made in THRIFT-3917, {{TSSLSocket}} now fails with a
> {{TypeError}} in {{open()}}:
> {code}
> File
> "/bld/thrift/Artools-rpmbuild/thrift-0.10.0/lib/py/build/lib.linux-i686-2.7/thrift/transport/TSSLSocket.py",
> line 293, in open
> self._validate_callback(self.peercert, self._server_hostname)
> TypeError: legacy_validate_callback() takes exactly 3 arguments (2 given)
> {code}
> Indeed, {{legacy_validate_callback()}} takes an extra {{self}} argument,
> which cannot possibly be compatible with the signature of
> [{{ssl.match_hostname()}}|https://docs.python.org/3/library/ssl.html#ssl.match_hostname]
> (which was added in Python 3.2).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (THRIFT-3855) In the go simple server, if Stop() is called multiple times it hangs
[
https://issues.apache.org/jira/browse/THRIFT-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843896#comment-15843896
]
ASF GitHub Bot commented on THRIFT-3855:
Github user pbennett commented on the issue:
https://github.com/apache/thrift/pull/1113
I was asked to close this pull request in
https://issues.apache.org/jira/browse/THRIFT-3855 but I'm not the submitter of
this pull request nor can I accept it.
> In the go simple server, if Stop() is called multiple times it hangs
>
>
> Key: THRIFT-3855
> URL: https://issues.apache.org/jira/browse/THRIFT-3855
> Project: Thrift
> Issue Type: Bug
> Components: Go - Library
>Affects Versions: 0.9.3
>Reporter: James E. King, III
>Assignee: Paul Finkelshteyn
>Priority: Minor
> Fix For: 0.10.0
>
>
> From the submitter huaiwan:
> {quote}
> huaiyun commented 18 hours ago
> When Stop() is called twice or more, and no new connection accepted from
> AcceptLoop(), the Stop() will be blocked because the quit channel is full.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] thrift issue #1113: Updated Stop by comment from Patrick Benett
Github user pbennett commented on the issue: https://github.com/apache/thrift/pull/1113 I was asked to close this pull request in https://issues.apache.org/jira/browse/THRIFT-3855 but I'm not the submitter of this pull request nor can I accept it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] thrift issue #1083: THRIFT-3921 Add C++ ostream operator<< functions for enu...
Github user jeking3 commented on the issue: https://github.com/apache/thrift/pull/1083 Hello - when I upgraded a project to thrift-0.10.0 I found I needed to provide the operator << for enumerations. I am a bit concerned we're not providing an operator >> as well; I think it would be a good idea to make sure the implementation can be overridden. That said, if you could rebase on master and force push it will kick a new CI build; I would like to see things passing (save maybe build job 17 which has had some trouble lately). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (THRIFT-3921) C++ code should print enums as strings
[
https://issues.apache.org/jira/browse/THRIFT-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843856#comment-15843856
]
ASF GitHub Bot commented on THRIFT-3921:
Github user jeking3 commented on the issue:
https://github.com/apache/thrift/pull/1083
Hello - when I upgraded a project to thrift-0.10.0 I found I needed to
provide the operator << for enumerations. I am a bit concerned we're not
providing an operator >> as well; I think it would be a good idea to make sure
the implementation can be overridden. That said, if you could rebase on master
and force push it will kick a new CI build; I would like to see things passing
(save maybe build job 17 which has had some trouble lately).
> C++ code should print enums as strings
> --
>
> Key: THRIFT-3921
> URL: https://issues.apache.org/jira/browse/THRIFT-3921
> Project: Thrift
> Issue Type: Improvement
> Components: C++ - Compiler
>Reporter: Vivek Jain
>
> THRIFT-2067 added {{operator<<}} to the C++ generated code, which is great,
> but enums are printed as numbers rather than their string representation. It
> would be great if the generated code printed them as a string instead. There
> might be some backwards-compatibility concerns (if users have already defined
> their own versions, then thrift's version would cause compile errors), not
> sure how important you guys think those are.
> Other thoughts/concerns? I am willing to work on a patch if no one else is
> able to work on it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (THRIFT-3855) In the go simple server, if Stop() is called multiple times it hangs
[
https://issues.apache.org/jira/browse/THRIFT-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843854#comment-15843854
]
James E. King, III commented on THRIFT-3855:
[~patrickb] If you believe that the issue has already been resolved, could you
close your pull request (https://github.com/apache/thrift/pull/1113)? Thanks.
> In the go simple server, if Stop() is called multiple times it hangs
>
>
> Key: THRIFT-3855
> URL: https://issues.apache.org/jira/browse/THRIFT-3855
> Project: Thrift
> Issue Type: Bug
> Components: Go - Library
>Affects Versions: 0.9.3
>Reporter: James E. King, III
>Assignee: Paul Finkelshteyn
>Priority: Minor
> Fix For: 0.10.0
>
>
> From the submitter huaiwan:
> {quote}
> huaiyun commented 18 hours ago
> When Stop() is called twice or more, and no new connection accepted from
> AcceptLoop(), the Stop() will be blocked because the quit channel is full.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (THRIFT-4007) Micro-optimization of TTransport.py
[
https://issues.apache.org/jira/browse/THRIFT-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843852#comment-15843852
]
ASF GitHub Bot commented on THRIFT-4007:
Github user jeking3 commented on the issue:
https://github.com/apache/thrift/pull/1121
Hello - please rebase on master and force push. I would like to see a CI
build pass before merging; save perhaps job #17 which has been failing.
> Micro-optimization of TTransport.py
> ---
>
> Key: THRIFT-4007
> URL: https://issues.apache.org/jira/browse/THRIFT-4007
> Project: Thrift
> Issue Type: Improvement
> Components: Python - Library
>Reporter: Nathan Jensen
>Priority: Trivial
> Labels: easyfix, performance
>
> Method readAll(self, sz) in class TTransportBase in TTransport.py can be
> optimized to only call len(chunk) once. It currently calls len(chunk) twice,
> i.e.
> {code}
> def readAll(self, sz):
> buff = b''
> have = 0
> while (have < sz):
> chunk = self.read(sz - have)
> have += len(chunk)
> buff += chunk
> if len(chunk) == 0:
> raise EOFError()
> return buff
> {code}
> That results in two method invocations to len(chunk) but it can be reduced to
> one method invocation to len(chunk). Depending on the amount of data being
> read, the extra len(chunk) method invocations can add up to slightly slower
> performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] thrift issue #1121: THRIFT-4007: TTransport.py micro-optimization: don't cal...
Github user jeking3 commented on the issue: https://github.com/apache/thrift/pull/1121 Hello - please rebase on master and force push. I would like to see a CI build pass before merging; save perhaps job #17 which has been failing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] thrift pull request #520: Remove moot `version` property from bower.json
Github user asfgit closed the pull request at: https://github.com/apache/thrift/pull/520 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (THRIFT-4042) ExtractionError when using accelerated thrift in a multiprocess test
[
https://issues.apache.org/jira/browse/THRIFT-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843844#comment-15843844
]
ASF GitHub Bot commented on THRIFT-4042:
Github user cjmay commented on the issue:
https://github.com/apache/thrift/pull/1165
Thank you!
> ExtractionError when using accelerated thrift in a multiprocess test
>
>
> Key: THRIFT-4042
> URL: https://issues.apache.org/jira/browse/THRIFT-4042
> Project: Thrift
> Issue Type: Bug
> Components: Python - Library
>Affects Versions: 0.10.0
>Reporter: Chandler May
>Assignee: James E. King, III
> Fix For: 0.11.0
>
>
> We recently switched to thrift 0.10.0 with accelerated protocols and started
> seeing ExtractionError being sporadically raised in tests that use the
> accelerated protocols (fastbinary module) and multiprocessing module.
> Example traceback:
> {code}
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File "/tmp/thrift-4042/thrift4042.py", line 15, in
> load_fastbinary()
> File "/tmp/thrift-4042/thrift4042.py", line 9, in load_fastbinary
> return TCompactProtocolAccelerated(None, fallback=False)
> File "build/bdist.linux-x86_64/egg/thrift/protocol/TCompactProtocol.py",
> line 449, in __init__
> File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 7,
> in
> File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 4,
> in __bootstrap__
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1203, in resource_filename
> self, resource_name
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1715, in get_resource_filename
> self._extract_resource(manager, self._eager_to_zip(name))
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1745, in _extract_resource
> self.egg_name, self._parts(zip_path)
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1270, in get_cache_path
> self.extraction_error()
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1250, in extraction_error
> raise err
> pkg_resources.ExtractionError: Can't extract file(s) to egg cache
> The following error occurred while trying to extract file(s) to the Python egg
> cache:
> [Errno 17] File exists: '/root/.cache/Python-Eggs'
> The Python egg cache directory is currently set to:
> /root/.cache/Python-Eggs
> Perhaps your account does not have write access to this directory? You can
> change the cache directory by setting the PYTHON_EGG_CACHE environment
> variable to point to an accessible directory.
> {code}
> There is a MWE using Docker at https://github.com/cjmay/thrift-4042 . I
> reproduced the error on the first run, but you may need to try several runs
> to reproduce it as it is a race condition.
> This bug only happens when using the new accelerated protocol in thrift
> 0.10.0 and only happens when thrift has been installed as an egg. When the
> accelerated protocol is used the fastbinary module is extracted from the
> installed thrift egg to the Python egg cache. I believe the extraction of
> the module file itself is atomic (using the rename syscall) but the creation
> of its parent directory is non-atomic and sometimes results in an error if
> the fastbinary module is loaded simultaneously by several processes. Note
> this only occurs the first time the module is loaded, as after that point the
> parent directory already exists. (This is the justification for using Docker
> for the MWE.)
> I believe this is the same error as:
> http://dev.list.galaxyproject.org/python-egg-cache-exists-error-td4656276.html
> http://www.georgevreilly.com/blog/2015/01/28/PythonEggCache.html
> The documentation indicates we can prevent packaging the module into an egg
> by setting {{zip_safe}} to {{False}} in {{setup.py}}:
> http://setuptools.readthedocs.io/en/latest/setuptools.html
> I tried this in the repository linked above and it worked. There are public
> builds demonstrating it: https://travis-ci.org/cjmay/thrift-4042
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] thrift issue #1165: THRIFT-4042 set zip_safe=False in python build to preven...
Github user cjmay commented on the issue: https://github.com/apache/thrift/pull/1165 Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (THRIFT-3854) allow users to clear read buffers
[ https://issues.apache.org/jira/browse/THRIFT-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843841#comment-15843841 ] ASF GitHub Bot commented on THRIFT-3854: Github user asfgit closed the pull request at: https://github.com/apache/thrift/pull/1081 > allow users to clear read buffers > - > > Key: THRIFT-3854 > URL: https://issues.apache.org/jira/browse/THRIFT-3854 > Project: Thrift > Issue Type: Improvement > Components: Java - Library >Reporter: Chris Lockfort >Priority: Minor > Fix For: 0.11.0 > > Attachments: thrift-3854-clearable-read-buffers.patch > > > My use case (which I believe many others would share): > - I have a large number of Thrift connections in a connection pool > - Upon return of the used connection to the pool I would like to eagerly > clean up the read buffer because I know it will not be used again until the > connection is checked out and a new read puts a new buffer in its place. > - Eagerly clearing the read buffers of idle connections saves a considerable > amount of heap memory in my application, vs. having all of the idle > connections keep all of their read buffers allocated. > - Currently, it looks like someone thought about this and there is a > TMemoryInputTransport#clear() , but it isn't callable from people with a > TFramedTransport which uses a TMemoryInputTransport for its read buffer > (unless you count doing gross and brittle reflection magic). > I've included a patch, but I'm very flexible / open to opinions of how > someone more familiar to this project would want to implement a feature like > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (THRIFT-3854) allow users to clear read buffers
[ https://issues.apache.org/jira/browse/THRIFT-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843840#comment-15843840 ] ASF GitHub Bot commented on THRIFT-3854: Github user asfgit closed the pull request at: https://github.com/apache/thrift/pull/1164 > allow users to clear read buffers > - > > Key: THRIFT-3854 > URL: https://issues.apache.org/jira/browse/THRIFT-3854 > Project: Thrift > Issue Type: Improvement > Components: Java - Library >Reporter: Chris Lockfort >Priority: Minor > Fix For: 0.11.0 > > Attachments: thrift-3854-clearable-read-buffers.patch > > > My use case (which I believe many others would share): > - I have a large number of Thrift connections in a connection pool > - Upon return of the used connection to the pool I would like to eagerly > clean up the read buffer because I know it will not be used again until the > connection is checked out and a new read puts a new buffer in its place. > - Eagerly clearing the read buffers of idle connections saves a considerable > amount of heap memory in my application, vs. having all of the idle > connections keep all of their read buffers allocated. > - Currently, it looks like someone thought about this and there is a > TMemoryInputTransport#clear() , but it isn't callable from people with a > TFramedTransport which uses a TMemoryInputTransport for its read buffer > (unless you count doing gross and brittle reflection magic). > I've included a patch, but I'm very flexible / open to opinions of how > someone more familiar to this project would want to implement a feature like > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (THRIFT-3854) allow users to clear read buffers
[ https://issues.apache.org/jira/browse/THRIFT-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James E. King, III resolved THRIFT-3854. Resolution: Fixed Assignee: James E. King, III Fix Version/s: 0.11.0 > allow users to clear read buffers > - > > Key: THRIFT-3854 > URL: https://issues.apache.org/jira/browse/THRIFT-3854 > Project: Thrift > Issue Type: Improvement > Components: Java - Library >Reporter: Chris Lockfort >Assignee: James E. King, III >Priority: Minor > Fix For: 0.11.0 > > Attachments: thrift-3854-clearable-read-buffers.patch > > > My use case (which I believe many others would share): > - I have a large number of Thrift connections in a connection pool > - Upon return of the used connection to the pool I would like to eagerly > clean up the read buffer because I know it will not be used again until the > connection is checked out and a new read puts a new buffer in its place. > - Eagerly clearing the read buffers of idle connections saves a considerable > amount of heap memory in my application, vs. having all of the idle > connections keep all of their read buffers allocated. > - Currently, it looks like someone thought about this and there is a > TMemoryInputTransport#clear() , but it isn't callable from people with a > TFramedTransport which uses a TMemoryInputTransport for its read buffer > (unless you count doing gross and brittle reflection magic). > I've included a patch, but I'm very flexible / open to opinions of how > someone more familiar to this project would want to implement a feature like > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] thrift pull request #1081: THRIFT-3854 add a way in java to clear TFramedTra...
Github user asfgit closed the pull request at: https://github.com/apache/thrift/pull/1081 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] thrift pull request #1164: THRIFT-3854 add a way in java to clear TFramedTra...
Github user asfgit closed the pull request at: https://github.com/apache/thrift/pull/1164 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (THRIFT-4042) ExtractionError when using accelerated thrift in a multiprocess test
[
https://issues.apache.org/jira/browse/THRIFT-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843836#comment-15843836
]
ASF GitHub Bot commented on THRIFT-4042:
Github user asfgit closed the pull request at:
https://github.com/apache/thrift/pull/1165
> ExtractionError when using accelerated thrift in a multiprocess test
>
>
> Key: THRIFT-4042
> URL: https://issues.apache.org/jira/browse/THRIFT-4042
> Project: Thrift
> Issue Type: Bug
> Components: Python - Library
>Affects Versions: 0.10.0
>Reporter: Chandler May
>Assignee: James E. King, III
> Fix For: 0.11.0
>
>
> We recently switched to thrift 0.10.0 with accelerated protocols and started
> seeing ExtractionError being sporadically raised in tests that use the
> accelerated protocols (fastbinary module) and multiprocessing module.
> Example traceback:
> {code}
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File "/tmp/thrift-4042/thrift4042.py", line 15, in
> load_fastbinary()
> File "/tmp/thrift-4042/thrift4042.py", line 9, in load_fastbinary
> return TCompactProtocolAccelerated(None, fallback=False)
> File "build/bdist.linux-x86_64/egg/thrift/protocol/TCompactProtocol.py",
> line 449, in __init__
> File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 7,
> in
> File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 4,
> in __bootstrap__
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1203, in resource_filename
> self, resource_name
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1715, in get_resource_filename
> self._extract_resource(manager, self._eager_to_zip(name))
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1745, in _extract_resource
> self.egg_name, self._parts(zip_path)
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1270, in get_cache_path
> self.extraction_error()
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1250, in extraction_error
> raise err
> pkg_resources.ExtractionError: Can't extract file(s) to egg cache
> The following error occurred while trying to extract file(s) to the Python egg
> cache:
> [Errno 17] File exists: '/root/.cache/Python-Eggs'
> The Python egg cache directory is currently set to:
> /root/.cache/Python-Eggs
> Perhaps your account does not have write access to this directory? You can
> change the cache directory by setting the PYTHON_EGG_CACHE environment
> variable to point to an accessible directory.
> {code}
> There is a MWE using Docker at https://github.com/cjmay/thrift-4042 . I
> reproduced the error on the first run, but you may need to try several runs
> to reproduce it as it is a race condition.
> This bug only happens when using the new accelerated protocol in thrift
> 0.10.0 and only happens when thrift has been installed as an egg. When the
> accelerated protocol is used the fastbinary module is extracted from the
> installed thrift egg to the Python egg cache. I believe the extraction of
> the module file itself is atomic (using the rename syscall) but the creation
> of its parent directory is non-atomic and sometimes results in an error if
> the fastbinary module is loaded simultaneously by several processes. Note
> this only occurs the first time the module is loaded, as after that point the
> parent directory already exists. (This is the justification for using Docker
> for the MWE.)
> I believe this is the same error as:
> http://dev.list.galaxyproject.org/python-egg-cache-exists-error-td4656276.html
> http://www.georgevreilly.com/blog/2015/01/28/PythonEggCache.html
> The documentation indicates we can prevent packaging the module into an egg
> by setting {{zip_safe}} to {{False}} in {{setup.py}}:
> http://setuptools.readthedocs.io/en/latest/setuptools.html
> I tried this in the repository linked above and it worked. There are public
> builds demonstrating it: https://travis-ci.org/cjmay/thrift-4042
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (THRIFT-4042) ExtractionError when using accelerated thrift in a multiprocess test
[
https://issues.apache.org/jira/browse/THRIFT-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James E. King, III resolved THRIFT-4042.
Resolution: Fixed
Assignee: James E. King, III
Fix Version/s: 0.11.0
> ExtractionError when using accelerated thrift in a multiprocess test
>
>
> Key: THRIFT-4042
> URL: https://issues.apache.org/jira/browse/THRIFT-4042
> Project: Thrift
> Issue Type: Bug
> Components: Python - Library
>Affects Versions: 0.10.0
>Reporter: Chandler May
>Assignee: James E. King, III
> Fix For: 0.11.0
>
>
> We recently switched to thrift 0.10.0 with accelerated protocols and started
> seeing ExtractionError being sporadically raised in tests that use the
> accelerated protocols (fastbinary module) and multiprocessing module.
> Example traceback:
> {code}
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File "/tmp/thrift-4042/thrift4042.py", line 15, in
> load_fastbinary()
> File "/tmp/thrift-4042/thrift4042.py", line 9, in load_fastbinary
> return TCompactProtocolAccelerated(None, fallback=False)
> File "build/bdist.linux-x86_64/egg/thrift/protocol/TCompactProtocol.py",
> line 449, in __init__
> File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 7,
> in
> File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 4,
> in __bootstrap__
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1203, in resource_filename
> self, resource_name
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1715, in get_resource_filename
> self._extract_resource(manager, self._eager_to_zip(name))
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1745, in _extract_resource
> self.egg_name, self._parts(zip_path)
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1270, in get_cache_path
> self.extraction_error()
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1250, in extraction_error
> raise err
> pkg_resources.ExtractionError: Can't extract file(s) to egg cache
> The following error occurred while trying to extract file(s) to the Python egg
> cache:
> [Errno 17] File exists: '/root/.cache/Python-Eggs'
> The Python egg cache directory is currently set to:
> /root/.cache/Python-Eggs
> Perhaps your account does not have write access to this directory? You can
> change the cache directory by setting the PYTHON_EGG_CACHE environment
> variable to point to an accessible directory.
> {code}
> There is a MWE using Docker at https://github.com/cjmay/thrift-4042 . I
> reproduced the error on the first run, but you may need to try several runs
> to reproduce it as it is a race condition.
> This bug only happens when using the new accelerated protocol in thrift
> 0.10.0 and only happens when thrift has been installed as an egg. When the
> accelerated protocol is used the fastbinary module is extracted from the
> installed thrift egg to the Python egg cache. I believe the extraction of
> the module file itself is atomic (using the rename syscall) but the creation
> of its parent directory is non-atomic and sometimes results in an error if
> the fastbinary module is loaded simultaneously by several processes. Note
> this only occurs the first time the module is loaded, as after that point the
> parent directory already exists. (This is the justification for using Docker
> for the MWE.)
> I believe this is the same error as:
> http://dev.list.galaxyproject.org/python-egg-cache-exists-error-td4656276.html
> http://www.georgevreilly.com/blog/2015/01/28/PythonEggCache.html
> The documentation indicates we can prevent packaging the module into an egg
> by setting {{zip_safe}} to {{False}} in {{setup.py}}:
> http://setuptools.readthedocs.io/en/latest/setuptools.html
> I tried this in the repository linked above and it worked. There are public
> builds demonstrating it: https://travis-ci.org/cjmay/thrift-4042
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] thrift pull request #1165: THRIFT-4042 set zip_safe=False in python build to...
Github user asfgit closed the pull request at: https://github.com/apache/thrift/pull/1165 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (THRIFT-3854) allow users to clear read buffers
[ https://issues.apache.org/jira/browse/THRIFT-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843640#comment-15843640 ] ASF GitHub Bot commented on THRIFT-3854: Github user jeking3 commented on the issue: https://github.com/apache/thrift/pull/1164 Job 17 continues to fail on builds. Something to do with haskell. I am going to ignore it here. > allow users to clear read buffers > - > > Key: THRIFT-3854 > URL: https://issues.apache.org/jira/browse/THRIFT-3854 > Project: Thrift > Issue Type: Improvement > Components: Java - Library >Reporter: Chris Lockfort >Priority: Minor > Attachments: thrift-3854-clearable-read-buffers.patch > > > My use case (which I believe many others would share): > - I have a large number of Thrift connections in a connection pool > - Upon return of the used connection to the pool I would like to eagerly > clean up the read buffer because I know it will not be used again until the > connection is checked out and a new read puts a new buffer in its place. > - Eagerly clearing the read buffers of idle connections saves a considerable > amount of heap memory in my application, vs. having all of the idle > connections keep all of their read buffers allocated. > - Currently, it looks like someone thought about this and there is a > TMemoryInputTransport#clear() , but it isn't callable from people with a > TFramedTransport which uses a TMemoryInputTransport for its read buffer > (unless you count doing gross and brittle reflection magic). > I've included a patch, but I'm very flexible / open to opinions of how > someone more familiar to this project would want to implement a feature like > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] thrift issue #1164: THRIFT-3854 add a way in java to clear TFramedTransport ...
Github user jeking3 commented on the issue: https://github.com/apache/thrift/pull/1164 Job 17 continues to fail on builds. Something to do with haskell. I am going to ignore it here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (THRIFT-4042) ExtractionError when using accelerated thrift in a multiprocess test
[
https://issues.apache.org/jira/browse/THRIFT-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843567#comment-15843567
]
Chandler May commented on THRIFT-4042:
--
I updated the description: setting {{zip_safe=False}} in {{setup()}} does fix
the issue.
> ExtractionError when using accelerated thrift in a multiprocess test
>
>
> Key: THRIFT-4042
> URL: https://issues.apache.org/jira/browse/THRIFT-4042
> Project: Thrift
> Issue Type: Bug
> Components: Python - Library
>Affects Versions: 0.10.0
>Reporter: Chandler May
>
> We recently switched to thrift 0.10.0 with accelerated protocols and started
> seeing ExtractionError being sporadically raised in tests that use the
> accelerated protocols (fastbinary module) and multiprocessing module.
> Example traceback:
> {code}
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File "/tmp/thrift-4042/thrift4042.py", line 15, in
> load_fastbinary()
> File "/tmp/thrift-4042/thrift4042.py", line 9, in load_fastbinary
> return TCompactProtocolAccelerated(None, fallback=False)
> File "build/bdist.linux-x86_64/egg/thrift/protocol/TCompactProtocol.py",
> line 449, in __init__
> File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 7,
> in
> File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 4,
> in __bootstrap__
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1203, in resource_filename
> self, resource_name
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1715, in get_resource_filename
> self._extract_resource(manager, self._eager_to_zip(name))
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1745, in _extract_resource
> self.egg_name, self._parts(zip_path)
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1270, in get_cache_path
> self.extraction_error()
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1250, in extraction_error
> raise err
> pkg_resources.ExtractionError: Can't extract file(s) to egg cache
> The following error occurred while trying to extract file(s) to the Python egg
> cache:
> [Errno 17] File exists: '/root/.cache/Python-Eggs'
> The Python egg cache directory is currently set to:
> /root/.cache/Python-Eggs
> Perhaps your account does not have write access to this directory? You can
> change the cache directory by setting the PYTHON_EGG_CACHE environment
> variable to point to an accessible directory.
> {code}
> There is a MWE using Docker at https://github.com/cjmay/thrift-4042 . I
> reproduced the error on the first run, but you may need to try several runs
> to reproduce it as it is a race condition.
> This bug only happens when using the new accelerated protocol in thrift
> 0.10.0 and only happens when thrift has been installed as an egg. When the
> accelerated protocol is used the fastbinary module is extracted from the
> installed thrift egg to the Python egg cache. I believe the extraction of
> the module file itself is atomic (using the rename syscall) but the creation
> of its parent directory is non-atomic and sometimes results in an error if
> the fastbinary module is loaded simultaneously by several processes. Note
> this only occurs the first time the module is loaded, as after that point the
> parent directory already exists. (This is the justification for using Docker
> for the MWE.)
> I believe this is the same error as:
> http://dev.list.galaxyproject.org/python-egg-cache-exists-error-td4656276.html
> http://www.georgevreilly.com/blog/2015/01/28/PythonEggCache.html
> The documentation indicates we can prevent packaging the module into an egg
> by setting {{zip_safe}} to {{False}} in {{setup.py}}:
> http://setuptools.readthedocs.io/en/latest/setuptools.html
> I tried this in the repository linked above and it worked. There are public
> builds demonstrating it: https://travis-ci.org/cjmay/thrift-4042
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (THRIFT-4042) ExtractionError when using accelerated thrift in a multiprocess test
[
https://issues.apache.org/jira/browse/THRIFT-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandler May updated THRIFT-4042:
-
Description:
We recently switched to thrift 0.10.0 with accelerated protocols and started
seeing ExtractionError being sporadically raised in tests that use the
accelerated protocols (fastbinary module) and multiprocessing module. Example
traceback:
{code}
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/tmp/thrift-4042/thrift4042.py", line 15, in
load_fastbinary()
File "/tmp/thrift-4042/thrift4042.py", line 9, in load_fastbinary
return TCompactProtocolAccelerated(None, fallback=False)
File "build/bdist.linux-x86_64/egg/thrift/protocol/TCompactProtocol.py", line
449, in __init__
File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 7, in
File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 4, in
__bootstrap__
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1203,
in resource_filename
self, resource_name
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1715,
in get_resource_filename
self._extract_resource(manager, self._eager_to_zip(name))
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1745,
in _extract_resource
self.egg_name, self._parts(zip_path)
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1270,
in get_cache_path
self.extraction_error()
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1250,
in extraction_error
raise err
pkg_resources.ExtractionError: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg
cache:
[Errno 17] File exists: '/root/.cache/Python-Eggs'
The Python egg cache directory is currently set to:
/root/.cache/Python-Eggs
Perhaps your account does not have write access to this directory? You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
{code}
There is a MWE using Docker at https://github.com/cjmay/thrift-4042 . I
reproduced the error on the first run, but you may need to try several runs to
reproduce it as it is a race condition.
This bug only happens when using the new accelerated protocol in thrift 0.10.0
and only happens when thrift has been installed as an egg. When the
accelerated protocol is used the fastbinary module is extracted from the
installed thrift egg to the Python egg cache. I believe the extraction of the
module file itself is atomic (using the rename syscall) but the creation of its
parent directory is non-atomic and sometimes results in an error if the
fastbinary module is loaded simultaneously by several processes. Note this
only occurs the first time the module is loaded, as after that point the parent
directory already exists. (This is the justification for using Docker for the
MWE.)
I believe this is the same error as:
http://dev.list.galaxyproject.org/python-egg-cache-exists-error-td4656276.html
http://www.georgevreilly.com/blog/2015/01/28/PythonEggCache.html
The documentation indicates we can prevent packaging the module into an egg by
setting {{zip_safe}} to {{False}} in {{setup.py}}:
http://setuptools.readthedocs.io/en/latest/setuptools.html
I tried this in the repository linked above and it worked. There are public
builds demonstrating it: https://travis-ci.org/cjmay/thrift-4042
was:
We recently switched to thrift 0.10.0 with accelerated protocols and started
seeing ExtractionError being sporadically raised in tests that use the
accelerated protocols (fastbinary module) and multiprocessing module. Example
traceback:
{code}
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/tmp/thrift-4042/thrift4042.py", line 15, in
load_fastbinary()
File "/tmp/thrift-4042/thrift4042.py", line 9, in load_fastbinary
return TCompactProtocolAccelerated(None, fallback=False)
File "build/bdist.linux-x86_64/egg/thrift/protocol/TCompactProtocol.py", line
449, in __init__
File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 7, in
File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 4, in
__bootstrap__
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1203,
in resource_filename
self, resource_name
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1715,
in get_resource_filename
self._extract_
[jira] [Commented] (THRIFT-4042) ExtractionError when using accelerated thrift in a multiprocess test
[
https://issues.apache.org/jira/browse/THRIFT-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843532#comment-15843532
]
ASF GitHub Bot commented on THRIFT-4042:
GitHub user cjmay opened a pull request:
https://github.com/apache/thrift/pull/1165
THRIFT-4042 set zip_safe=False in python build to prevent egg
ExtractionError
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cjmay/thrift thrift-4042
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/thrift/pull/1165.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1165
commit 11ee1da2bb9e85990953a854181d106e73a0cadb
Author: Chandler May
Date: 2017-01-27T21:21:40Z
THRIFT-4042 set zip_safe=False in python build to prevent egg
ExtractionError
> ExtractionError when using accelerated thrift in a multiprocess test
>
>
> Key: THRIFT-4042
> URL: https://issues.apache.org/jira/browse/THRIFT-4042
> Project: Thrift
> Issue Type: Bug
> Components: Python - Library
>Affects Versions: 0.10.0
>Reporter: Chandler May
>
> We recently switched to thrift 0.10.0 with accelerated protocols and started
> seeing ExtractionError being sporadically raised in tests that use the
> accelerated protocols (fastbinary module) and multiprocessing module.
> Example traceback:
> {code}
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File "/tmp/thrift-4042/thrift4042.py", line 15, in
> load_fastbinary()
> File "/tmp/thrift-4042/thrift4042.py", line 9, in load_fastbinary
> return TCompactProtocolAccelerated(None, fallback=False)
> File "build/bdist.linux-x86_64/egg/thrift/protocol/TCompactProtocol.py",
> line 449, in __init__
> File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 7,
> in
> File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 4,
> in __bootstrap__
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1203, in resource_filename
> self, resource_name
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1715, in get_resource_filename
> self._extract_resource(manager, self._eager_to_zip(name))
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1745, in _extract_resource
> self.egg_name, self._parts(zip_path)
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1270, in get_cache_path
> self.extraction_error()
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1250, in extraction_error
> raise err
> pkg_resources.ExtractionError: Can't extract file(s) to egg cache
> The following error occurred while trying to extract file(s) to the Python egg
> cache:
> [Errno 17] File exists: '/root/.cache/Python-Eggs'
> The Python egg cache directory is currently set to:
> /root/.cache/Python-Eggs
> Perhaps your account does not have write access to this directory? You can
> change the cache directory by setting the PYTHON_EGG_CACHE environment
> variable to point to an accessible directory.
> {code}
> There is a MWE using Docker at https://github.com/cjmay/thrift-4042 . I
> reproduced the error on the first run, but you may need to try several runs
> to reproduce it as it is a race condition.
> This bug only happens when using the new accelerated protocol in thrift
> 0.10.0 and only happens when thrift has been installed as an egg. When the
> accelerated protocol is used the fastbinary module is extracted from the
> installed thrift egg to the Python egg cache. I believe the extraction of
> the module file itself is atomic (using the rename syscall) but the creation
> of its parent directory is non-atomic and sometimes results in an error if
> the fastbinary module is loaded simultaneously by several processes. Note
> this only occurs the first time the module is loaded, as after that point the
> parent directory already exists. (This is the justification for using Docker
> for the MWE.)
> I believe this is the same error as:
> http://dev.list.galaxyproject.org/python-egg-cache-exists-error-td4656276.html
> http://www.georgevreilly.com/blog/2015/01/28/PythonEggCache.html
> I believe the documentation indicates this problem can be worked around by
> setting {{zip_safe}} to {{False}} in {{setup.py}}:
> http://setuptools.readthedocs.io/en/latest/setuptools.html
--
This message was
[GitHub] thrift pull request #1165: THRIFT-4042 set zip_safe=False in python build to...
GitHub user cjmay opened a pull request: https://github.com/apache/thrift/pull/1165 THRIFT-4042 set zip_safe=False in python build to prevent egg ExtractionError You can merge this pull request into a Git repository by running: $ git pull https://github.com/cjmay/thrift thrift-4042 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/thrift/pull/1165.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1165 commit 11ee1da2bb9e85990953a854181d106e73a0cadb Author: Chandler May Date: 2017-01-27T21:21:40Z THRIFT-4042 set zip_safe=False in python build to prevent egg ExtractionError --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] thrift pull request #1164: THRIFT-3854 add a way in java to clear TFramedTra...
GitHub user jeking3 opened a pull request: https://github.com/apache/thrift/pull/1164 THRIFT-3854 add a way in java to clear TFramedTransport read buffers I took https://github.com/apache/thrift/pull/1081 and applied changes to address the code comments. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jeking3/thrift defect/THRIFT-3854 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/thrift/pull/1164.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1164 commit dd8aa452d1651c827333056d5bd2b8fb422cdebe Author: James E. King, III Date: 2017-01-27T21:17:08Z THRIFT-3854 add a way in java to clear TFramedTransport read buffers --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (THRIFT-3854) allow users to clear read buffers
[ https://issues.apache.org/jira/browse/THRIFT-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843503#comment-15843503 ] ASF GitHub Bot commented on THRIFT-3854: GitHub user jeking3 opened a pull request: https://github.com/apache/thrift/pull/1164 THRIFT-3854 add a way in java to clear TFramedTransport read buffers I took https://github.com/apache/thrift/pull/1081 and applied changes to address the code comments. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jeking3/thrift defect/THRIFT-3854 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/thrift/pull/1164.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1164 commit dd8aa452d1651c827333056d5bd2b8fb422cdebe Author: James E. King, III Date: 2017-01-27T21:17:08Z THRIFT-3854 add a way in java to clear TFramedTransport read buffers > allow users to clear read buffers > - > > Key: THRIFT-3854 > URL: https://issues.apache.org/jira/browse/THRIFT-3854 > Project: Thrift > Issue Type: Improvement > Components: Java - Library >Reporter: Chris Lockfort >Priority: Minor > Attachments: thrift-3854-clearable-read-buffers.patch > > > My use case (which I believe many others would share): > - I have a large number of Thrift connections in a connection pool > - Upon return of the used connection to the pool I would like to eagerly > clean up the read buffer because I know it will not be used again until the > connection is checked out and a new read puts a new buffer in its place. > - Eagerly clearing the read buffers of idle connections saves a considerable > amount of heap memory in my application, vs. having all of the idle > connections keep all of their read buffers allocated. > - Currently, it looks like someone thought about this and there is a > TMemoryInputTransport#clear() , but it isn't callable from people with a > TFramedTransport which uses a TMemoryInputTransport for its read buffer > (unless you count doing gross and brittle reflection magic). > I've included a patch, but I'm very flexible / open to opinions of how > someone more familiar to this project would want to implement a feature like > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (THRIFT-3854) allow users to clear read buffers
[ https://issues.apache.org/jira/browse/THRIFT-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843481#comment-15843481 ] ASF GitHub Bot commented on THRIFT-3854: Github user jeking3 commented on the issue: https://github.com/apache/thrift/pull/1081 I made the requested changes locally but I don't know if I can push on top of this, so I will likely open a new pull request to obsolete this one. > allow users to clear read buffers > - > > Key: THRIFT-3854 > URL: https://issues.apache.org/jira/browse/THRIFT-3854 > Project: Thrift > Issue Type: Improvement > Components: Java - Library >Reporter: Chris Lockfort >Priority: Minor > Attachments: thrift-3854-clearable-read-buffers.patch > > > My use case (which I believe many others would share): > - I have a large number of Thrift connections in a connection pool > - Upon return of the used connection to the pool I would like to eagerly > clean up the read buffer because I know it will not be used again until the > connection is checked out and a new read puts a new buffer in its place. > - Eagerly clearing the read buffers of idle connections saves a considerable > amount of heap memory in my application, vs. having all of the idle > connections keep all of their read buffers allocated. > - Currently, it looks like someone thought about this and there is a > TMemoryInputTransport#clear() , but it isn't callable from people with a > TFramedTransport which uses a TMemoryInputTransport for its read buffer > (unless you count doing gross and brittle reflection magic). > I've included a patch, but I'm very flexible / open to opinions of how > someone more familiar to this project would want to implement a feature like > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] thrift issue #1081: THRIFT-3854 add a way in java to clear TFramedTransport ...
Github user jeking3 commented on the issue: https://github.com/apache/thrift/pull/1081 I made the requested changes locally but I don't know if I can push on top of this, so I will likely open a new pull request to obsolete this one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (THRIFT-4042) ExtractionError when using accelerated thrift in a multiprocess test
[
https://issues.apache.org/jira/browse/THRIFT-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843432#comment-15843432
]
Chandler May commented on THRIFT-4042:
--
Updated with minimal working example.
> ExtractionError when using accelerated thrift in a multiprocess test
>
>
> Key: THRIFT-4042
> URL: https://issues.apache.org/jira/browse/THRIFT-4042
> Project: Thrift
> Issue Type: Bug
> Components: Python - Library
>Affects Versions: 0.10.0
>Reporter: Chandler May
>
> We recently switched to thrift 0.10.0 with accelerated protocols and started
> seeing ExtractionError being sporadically raised in tests that use the
> accelerated protocols (fastbinary module) and multiprocessing module.
> Example traceback:
> {code}
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File "/tmp/thrift-4042/thrift4042.py", line 15, in
> load_fastbinary()
> File "/tmp/thrift-4042/thrift4042.py", line 9, in load_fastbinary
> return TCompactProtocolAccelerated(None, fallback=False)
> File "build/bdist.linux-x86_64/egg/thrift/protocol/TCompactProtocol.py",
> line 449, in __init__
> File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 7,
> in
> File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 4,
> in __bootstrap__
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1203, in resource_filename
> self, resource_name
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1715, in get_resource_filename
> self._extract_resource(manager, self._eager_to_zip(name))
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1745, in _extract_resource
> self.egg_name, self._parts(zip_path)
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1270, in get_cache_path
> self.extraction_error()
> File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line
> 1250, in extraction_error
> raise err
> pkg_resources.ExtractionError: Can't extract file(s) to egg cache
> The following error occurred while trying to extract file(s) to the Python egg
> cache:
> [Errno 17] File exists: '/root/.cache/Python-Eggs'
> The Python egg cache directory is currently set to:
> /root/.cache/Python-Eggs
> Perhaps your account does not have write access to this directory? You can
> change the cache directory by setting the PYTHON_EGG_CACHE environment
> variable to point to an accessible directory.
> {code}
> There is a MWE using Docker at https://github.com/cjmay/thrift-4042 . I
> reproduced the error on the first run, but you may need to try several runs
> to reproduce it as it is a race condition.
> This bug only happens when using the new accelerated protocol in thrift
> 0.10.0 and only happens when thrift has been installed as an egg. When the
> accelerated protocol is used the fastbinary module is extracted from the
> installed thrift egg to the Python egg cache. I believe the extraction of
> the module file itself is atomic (using the rename syscall) but the creation
> of its parent directory is non-atomic and sometimes results in an error if
> the fastbinary module is loaded simultaneously by several processes. Note
> this only occurs the first time the module is loaded, as after that point the
> parent directory already exists. (This is the justification for using Docker
> for the MWE.)
> I believe this is the same error as:
> http://dev.list.galaxyproject.org/python-egg-cache-exists-error-td4656276.html
> http://www.georgevreilly.com/blog/2015/01/28/PythonEggCache.html
> I believe the documentation indicates this problem can be worked around by
> setting {{zip_safe}} to {{False}} in {{setup.py}}:
> http://setuptools.readthedocs.io/en/latest/setuptools.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (THRIFT-4042) ExtractionError when using accelerated thrift in a multiprocess test
[
https://issues.apache.org/jira/browse/THRIFT-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandler May updated THRIFT-4042:
-
Description:
We recently switched to thrift 0.10.0 with accelerated protocols and started
seeing ExtractionError being sporadically raised in tests that use the
accelerated protocols (fastbinary module) and multiprocessing module. Example
traceback:
{code}
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/tmp/thrift-4042/thrift4042.py", line 15, in
load_fastbinary()
File "/tmp/thrift-4042/thrift4042.py", line 9, in load_fastbinary
return TCompactProtocolAccelerated(None, fallback=False)
File "build/bdist.linux-x86_64/egg/thrift/protocol/TCompactProtocol.py", line
449, in __init__
File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 7, in
File "build/bdist.linux-x86_64/egg/thrift/protocol/fastbinary.py", line 4, in
__bootstrap__
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1203,
in resource_filename
self, resource_name
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1715,
in get_resource_filename
self._extract_resource(manager, self._eager_to_zip(name))
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1745,
in _extract_resource
self.egg_name, self._parts(zip_path)
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1270,
in get_cache_path
self.extraction_error()
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1250,
in extraction_error
raise err
pkg_resources.ExtractionError: Can't extract file(s) to egg cache
The following error occurred while trying to extract file(s) to the Python egg
cache:
[Errno 17] File exists: '/root/.cache/Python-Eggs'
The Python egg cache directory is currently set to:
/root/.cache/Python-Eggs
Perhaps your account does not have write access to this directory? You can
change the cache directory by setting the PYTHON_EGG_CACHE environment
variable to point to an accessible directory.
{code}
There is a MWE using Docker at https://github.com/cjmay/thrift-4042 . I
reproduced the error on the first run, but you may need to try several runs to
reproduce it as it is a race condition.
This bug only happens when using the new accelerated protocol in thrift 0.10.0
and only happens when thrift has been installed as an egg. When the
accelerated protocol is used the fastbinary module is extracted from the
installed thrift egg to the Python egg cache. I believe the extraction of the
module file itself is atomic (using the rename syscall) but the creation of its
parent directory is non-atomic and sometimes results in an error if the
fastbinary module is loaded simultaneously by several processes. Note this
only occurs the first time the module is loaded, as after that point the parent
directory already exists. (This is the justification for using Docker for the
MWE.)
I believe this is the same error as:
http://dev.list.galaxyproject.org/python-egg-cache-exists-error-td4656276.html
http://www.georgevreilly.com/blog/2015/01/28/PythonEggCache.html
I believe the documentation indicates this problem can be worked around by
setting {{zip_safe}} to {{False}} in {{setup.py}}:
http://setuptools.readthedocs.io/en/latest/setuptools.html
was:
We recently switched to thrift 0.10.0 with accelerated protocols and started
getting sporadic errors in tests that use the multiprocessing module of the
form:
{code}
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python2.7/multiprocessing/pool.py:250: in map
return self.map_async(func, iterable, chunksize).get()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = , timeout = None
def get(self, timeout=None):
self.wait(timeout)
if not self._ready:
raise TimeoutError
if self._success:
return self._value
else:
> raise self._value
E ExtractionError: Can't extract file(s) to egg cache
E
E The following error occurred while trying to extract file(s) to the
Python egg
E cache:
E
E [Errno 17] File exists: '/home/concrete/.cache/Python-Eggs'
E
E The Python egg cache directory is currently set to:
E
E /home/concrete/.cache/Python-Eggs
E
E Perhaps your account does not have write access to this directory?
You can
E change the cache directory by setting the PYTHON_EGG_CACHE
environment
E variable to point to an accessible directory.
/usr/lib64/pytho
[jira] [Commented] (THRIFT-3550) Latest indexed version of the perl lib on CPAN is 0.8.0, although 0.9.0 is present
[ https://issues.apache.org/jira/browse/THRIFT-3550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843137#comment-15843137 ] James E. King, III commented on THRIFT-3550: I'm waiting for co-ownership permission currently owned by GSLIN on CPAN for the namespace "Thrift" as GSLIN published thrift-0.8.0 and thrift-0.9.0 in the past. I have already contacted GSLIN, will keep folks updated. > Latest indexed version of the perl lib on CPAN is 0.8.0, although 0.9.0 is > present > -- > > Key: THRIFT-3550 > URL: https://issues.apache.org/jira/browse/THRIFT-3550 > Project: Thrift > Issue Type: Wish > Components: Perl - Library >Affects Versions: 0.9.1, 0.9.2, 0.9.3, 0.10.0 >Reporter: David Morel >Assignee: James E. King, III > > Running cpanm or cpan installs version 0.8.0 of the perl lib, although 0.9.0 > has been uploaded on PAUSE. > Also, would be nice to have v 0.9.3 since it's available? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (THRIFT-3854) allow users to clear read buffers
[
https://issues.apache.org/jira/browse/THRIFT-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843064#comment-15843064
]
ASF GitHub Bot commented on THRIFT-3854:
Github user clockfort commented on a diff in the pull request:
https://github.com/apache/thrift/pull/1081#discussion_r98236369
--- Diff: lib/java/src/org/apache/thrift/transport/TFramedTransport.java ---
@@ -123,6 +123,12 @@ public void consumeBuffer(int len) {
readBuffer_.consumeBuffer(len);
}
+ public void clear() {
+if (readBuffer_ != null) {
--- End diff --
(I'm the person who originally wrote this patch / feature request)
I agree, and can't find any reflected use cases in at least just the thrift
codebase.
> allow users to clear read buffers
> -
>
> Key: THRIFT-3854
> URL: https://issues.apache.org/jira/browse/THRIFT-3854
> Project: Thrift
> Issue Type: Improvement
> Components: Java - Library
>Reporter: Chris Lockfort
>Priority: Minor
> Attachments: thrift-3854-clearable-read-buffers.patch
>
>
> My use case (which I believe many others would share):
> - I have a large number of Thrift connections in a connection pool
> - Upon return of the used connection to the pool I would like to eagerly
> clean up the read buffer because I know it will not be used again until the
> connection is checked out and a new read puts a new buffer in its place.
> - Eagerly clearing the read buffers of idle connections saves a considerable
> amount of heap memory in my application, vs. having all of the idle
> connections keep all of their read buffers allocated.
> - Currently, it looks like someone thought about this and there is a
> TMemoryInputTransport#clear() , but it isn't callable from people with a
> TFramedTransport which uses a TMemoryInputTransport for its read buffer
> (unless you count doing gross and brittle reflection magic).
> I've included a patch, but I'm very flexible / open to opinions of how
> someone more familiar to this project would want to implement a feature like
> this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] thrift pull request #1081: THRIFT-3854 add a way in java to clear TFramedTra...
Github user clockfort commented on a diff in the pull request:
https://github.com/apache/thrift/pull/1081#discussion_r98236369
--- Diff: lib/java/src/org/apache/thrift/transport/TFramedTransport.java ---
@@ -123,6 +123,12 @@ public void consumeBuffer(int len) {
readBuffer_.consumeBuffer(len);
}
+ public void clear() {
+if (readBuffer_ != null) {
--- End diff --
(I'm the person who originally wrote this patch / feature request)
I agree, and can't find any reflected use cases in at least just the thrift
codebase.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (THRIFT-3854) allow users to clear read buffers
[
https://issues.apache.org/jira/browse/THRIFT-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843062#comment-15843062
]
ASF GitHub Bot commented on THRIFT-3854:
Github user clockfort commented on a diff in the pull request:
https://github.com/apache/thrift/pull/1081#discussion_r98235950
--- Diff:
lib/java/test/org/apache/thrift/transport/TestTFastFramedTransport.java ---
@@ -19,6 +19,8 @@
package org.apache.thrift.transport;
public class TestTFastFramedTransport extends TestTFramedTransport {
+ protected final static int INITIAL_CAPACITY = 50;
+
@Override
protected TTransport getTransport(TTransport underlying) {
return new TFastFramedTransport(underlying, 50, 10 * 1024 * 1024);
--- End diff --
Yes.
> allow users to clear read buffers
> -
>
> Key: THRIFT-3854
> URL: https://issues.apache.org/jira/browse/THRIFT-3854
> Project: Thrift
> Issue Type: Improvement
> Components: Java - Library
>Reporter: Chris Lockfort
>Priority: Minor
> Attachments: thrift-3854-clearable-read-buffers.patch
>
>
> My use case (which I believe many others would share):
> - I have a large number of Thrift connections in a connection pool
> - Upon return of the used connection to the pool I would like to eagerly
> clean up the read buffer because I know it will not be used again until the
> connection is checked out and a new read puts a new buffer in its place.
> - Eagerly clearing the read buffers of idle connections saves a considerable
> amount of heap memory in my application, vs. having all of the idle
> connections keep all of their read buffers allocated.
> - Currently, it looks like someone thought about this and there is a
> TMemoryInputTransport#clear() , but it isn't callable from people with a
> TFramedTransport which uses a TMemoryInputTransport for its read buffer
> (unless you count doing gross and brittle reflection magic).
> I've included a patch, but I'm very flexible / open to opinions of how
> someone more familiar to this project would want to implement a feature like
> this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] thrift pull request #1081: THRIFT-3854 add a way in java to clear TFramedTra...
Github user clockfort commented on a diff in the pull request:
https://github.com/apache/thrift/pull/1081#discussion_r98235950
--- Diff:
lib/java/test/org/apache/thrift/transport/TestTFastFramedTransport.java ---
@@ -19,6 +19,8 @@
package org.apache.thrift.transport;
public class TestTFastFramedTransport extends TestTFramedTransport {
+ protected final static int INITIAL_CAPACITY = 50;
+
@Override
protected TTransport getTransport(TTransport underlying) {
return new TFastFramedTransport(underlying, 50, 10 * 1024 * 1024);
--- End diff --
Yes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (THRIFT-3550) Latest indexed version of the perl lib on CPAN is 0.8.0, although 0.9.0 is present
[ https://issues.apache.org/jira/browse/THRIFT-3550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15842930#comment-15842930 ] James E. King, III commented on THRIFT-3550: I'm waiting for my PAUSE account to be approved so I can upload to CPAN. > Latest indexed version of the perl lib on CPAN is 0.8.0, although 0.9.0 is > present > -- > > Key: THRIFT-3550 > URL: https://issues.apache.org/jira/browse/THRIFT-3550 > Project: Thrift > Issue Type: Wish > Components: Perl - Library >Affects Versions: 0.9.1, 0.9.2, 0.9.3, 0.10.0 >Reporter: David Morel >Assignee: James E. King, III > > Running cpanm or cpan installs version 0.8.0 of the perl lib, although 0.9.0 > has been uploaded on PAUSE. > Also, would be nice to have v 0.9.3 since it's available? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (THRIFT-4043) thrift perl debian package is placing files in the wrong place
James E. King, III created THRIFT-4043: -- Summary: thrift perl debian package is placing files in the wrong place Key: THRIFT-4043 URL: https://issues.apache.org/jira/browse/THRIFT-4043 Project: Thrift Issue Type: Bug Components: Build Process, Perl - Library Affects Versions: 0.10.0 Environment: Ubuntu docker build environment, installing on Ubuntu 14.04 LTS Reporter: James E. King, III The libthrift-perl-0.10.0 package is installing things into /usr/lib, when it should be going into /usr/share/perl5. This means "Thrift.pm" is being dropped directly into /usr/lib/Thrift.pm right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
