Here is what I see in our Ansible scripts:
- name: Install python-requests module
pip:
name: requests
version: 2.6.1
I would try installing that exact version with pip. Also, have you tried
installing that dependency?
On Tue, Feb 27, 2018 at 1:14 PM, David McGinnis <mcginn...@avalonconsult.com
> wrote:
> I've tried that and it didn't fix the issue. I upgraded to the newest
> version of requests (2.18.4), and now instead of in that location, I am
> getting the same error in a dependency of requests (urllib3). See call
> stack below.
>
> Traceback (most recent call last):
> File
> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
> line 59, in run_file
> imp.load_source('__main__', script)
> File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/
> package/scripts/indexing_master.py", line 18, in <module>
> import requests
> File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 43,
> in <module>
> import urllib3
> File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 8, in
> <module>
> from .connectionpool import (
> File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line
> 11, in <module>
> from .exceptions import (
> File "/usr/lib/python2.7/site-packages/urllib3/exceptions.py", line 2,
> in <module>
> from .packages.six.moves.http_client import (
> File "/usr/lib/python2.7/site-packages/urllib3/packages/six.py", line
> 198, in load_module
> return sys.modules[fullname]
> AttributeError: 'NoneType' object has no attribute 'modules'
>
> On Tue, Feb 27, 2018 at 1:12 PM, Ryan Merriman <merrim...@gmail.com>
> wrote:
>
>> I'm not a python expect but I would either uninstall then install with
>> pip or install dependencies as needed.
>>
>> On Tue, Feb 27, 2018 at 1:09 PM, David McGinnis <
>> mcginn...@avalonconsult.com> wrote:
>>
>>> Is there a fix available already, or how would you go about making this
>>> fix? I'm open to figuring it out if you have any ideas on how we should fix
>>> the issue. Requests was installed for me already without pip even being
>>> installed, likely through the MPack process or as part of a previously
>>> installed component. Pip reported requests had been installed through
>>> distutils.
>>>
>>> On Tue, Feb 27, 2018 at 1:06 PM, Ryan Merriman <merrim...@gmail.com>
>>> wrote:
>>>
>>>> Yep you found it. You were unlucky enough to install a version of
>>>> master that had this regression. How did you install requests? Using pip
>>>> should also install the necessary dependencies. Either way everything
>>>> should still work, you'll just have to manually start/stop indexing since
>>>> you won't ever be able to stop it.
>>>>
>>>> You are correct about HDP 2.6.
>>>>
>>>> On Tue, Feb 27, 2018 at 12:57 PM, David McGinnis <
>>>> mcginn...@avalonconsult.com> wrote:
>>>>
>>>>> Looks like this issue was introduced in this commit:
>>>>> https://github.com/apache/metron/commit/1c9437c41c4eb626bd44
>>>>> 193ec6545f9356f3f7b2 . Looking past this, there is only one commit
>>>>> that seems to deal with this, but it is for the Ansible deployment, so I
>>>>> don't believe it would affect an MPack installation:
>>>>> https://github.com/apache/metron/commit/fa86663ecdb166f889c9
>>>>> c0c4c34373aec68dbd30 .
>>>>>
>>>>> I installed a newer version of requests just to see if that would fix
>>>>> the issue, and I'm getting a similar issue in a different package relied
>>>>> upon by requests now (urllib3). It seems as if something is causing sys to
>>>>> not exist or not point to what it should. I wonder if it has to do with
>>>>> the
>>>>> python2.6 vs python2.7 issue.
>>>>>
>>>>> Also, as I was looking through the commits, it looks like using the
>>>>> last official release wouldn't work for me, since we are on HDP 2.6, and
>>>>> changes went in since the last release to support that version of HDP.
>>>>>
>>>>> On Tue, Feb 27, 2018 at 12:12 PM, David McGinnis <
>>>>> mcginn...@avalonconsult.com> wrote:
>>>>>
>>>>>> *Simon:*
>>>>>>
>>>>>> There are some errors from processing showing up in the indexing
>>>>>> topic, but no data is making it through without restarting Metron
>>>>>> completely (restarting just the indexing service doesn't seem to do it).
>>>>>>
>>>>>> I don't have easy access to the Storm UI at the moment. I have a work
>>>>>> around to get the logs that I'm going to look at shortly.
>>>>>>
>>>>>> I understand, but I didn't realize I had not pulled from a release
>>>>>> until I had already installed, and this is just a prototype cluster
>>>>>> anyways. I'll likely pull from an established release for the production
>>>>>> deployment.
>>>>>>
>>>>>> *Ryan:*
>>>>>>
>>>>>> I actually found something interesting in the ambari-agent logs on
>>>>>> the Metron node. See the callstack below.
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>> File
>>>>>> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
>>>>>> line 59, in run_file
>>>>>> imp.load_source('__main__', script)
>>>>>> File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/pa
>>>>>> ckage/scripts/indexing_master.py", line 18, in <module>
>>>>>> import requests
>>>>>> File "/usr/lib/python2.7/site-packages/requests/__init__.py", line
>>>>>> 53, in <module>
>>>>>> from .packages.urllib3.contrib import pyopenssl
>>>>>> File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py",
>>>>>> line 61, in load_module
>>>>>> if name in sys.modules:
>>>>>> AttributeError: 'NoneType' object has no attribute 'modules'
>>>>>>
>>>>>> Perhaps there is a missing component that needs to be installed?
>>>>>>
>>>>>> On Tue, Feb 27, 2018 at 12:01 PM, Ryan Merriman <merrim...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> If there was an error happening during the service check, it will be
>>>>>>> in the Ambari agent logs. I would look there first.
>>>>>>>
>>>>>>> On Feb 27, 2018, at 11:45 AM, Simon Elliston Ball <
>>>>>>> si...@simonellistonball.com> wrote:
>>>>>>>
>>>>>>> Anything appearing on the indexing topic in kafka?
>>>>>>>
>>>>>>> Anything in the logs for the indexing topology in Storm UI?
>>>>>>>
>>>>>>> Master isn’t always the best place to start, might be worth sticking
>>>>>>> to a released build to kick the tyres.
>>>>>>>
>>>>>>> Simon
>>>>>>>
>>>>>>> On 27 Feb 2018, at 17:38, David McGinnis <
>>>>>>> mcginn...@avalonconsult.com> wrote:
>>>>>>>
>>>>>>> All,
>>>>>>>
>>>>>>> I have a kerberized HDP 2.6.3 cluster which I have installed Metron
>>>>>>> on through an MPack. I am using code straight from the master branch of
>>>>>>> the
>>>>>>> github as of last week.
>>>>>>>
>>>>>>> When I start Metron, all of the components correct start up. Shortly
>>>>>>> after, however, the Indexing Service stops working. I have not been
>>>>>>> able to
>>>>>>> find any logs that seem to capture any issues, and both indexing storm
>>>>>>> topologies are still active and running after the failure. Those
>>>>>>> topologies
>>>>>>> only report a warning about one of my parsers using the default HDFS
>>>>>>> writer
>>>>>>> config, which appears to be normal.
>>>>>>>
>>>>>>> Not only is this making my dashboard red when it shouldn't be, but
>>>>>>> also it seems as if data doesn't go through the pipeline until the
>>>>>>> entire
>>>>>>> Metron system is restarted. I am not confident this is related, but I
>>>>>>> suspect it is.
>>>>>>>
>>>>>>> Where should I go to find logs about this service, or does anyone
>>>>>>> have any ideas why this would be happening?
>>>>>>>
>>>>>>> --
>>>>>>> David McGinnis
>>>>>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>>>>>> <http://www.avalonconsult.com/>M: (513) 439-0082
>>>>>>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> |
>>>>>>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter
>>>>>>> <https://twitter.com/avalonconsult>
>>>>>>> ------------------------------------------------------------
>>>>>>> -------------------------------------------------
>>>>>>> This message (including any attachments) contains confidential
>>>>>>> information
>>>>>>> intended for a specific individual and purpose, and is protected by
>>>>>>> law. If
>>>>>>> you are not the intended recipient, you should delete this message.
>>>>>>> Any
>>>>>>> disclosure, copying, or distribution of this message, or the taking
>>>>>>> of any
>>>>>>> action based on it, is strictly prohibited.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> David McGinnis
>>>>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>>>>> <http://www.avalonconsult.com/>M: (513) 439-0082
>>>>>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> |
>>>>>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter
>>>>>> <https://twitter.com/avalonconsult>
>>>>>> ------------------------------------------------------------
>>>>>> -------------------------------------------------
>>>>>> This message (including any attachments) contains confidential
>>>>>> information
>>>>>> intended for a specific individual and purpose, and is protected by
>>>>>> law. If
>>>>>> you are not the intended recipient, you should delete this message.
>>>>>> Any
>>>>>> disclosure, copying, or distribution of this message, or the taking
>>>>>> of any
>>>>>> action based on it, is strictly prohibited.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> David McGinnis
>>>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>>>> <http://www.avalonconsult.com/>M: (513) 439-0082
>>>>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> |
>>>>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter
>>>>> <https://twitter.com/avalonconsult>
>>>>> ------------------------------------------------------------
>>>>> -------------------------------------------------
>>>>> This message (including any attachments) contains confidential
>>>>> information
>>>>> intended for a specific individual and purpose, and is protected by
>>>>> law. If
>>>>> you are not the intended recipient, you should delete this message.
>>>>> Any
>>>>> disclosure, copying, or distribution of this message, or the taking of
>>>>> any
>>>>> action based on it, is strictly prohibited.
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> David McGinnis
>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>> <http://www.avalonconsult.com/>M: (513) 439-0082
>>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> |
>>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter
>>> <https://twitter.com/avalonconsult>
>>> ------------------------------------------------------------
>>> -------------------------------------------------
>>> This message (including any attachments) contains confidential
>>> information
>>> intended for a specific individual and purpose, and is protected by law.
>>> If
>>> you are not the intended recipient, you should delete this message. Any
>>> disclosure, copying, or distribution of this message, or the taking of
>>> any
>>> action based on it, is strictly prohibited.
>>>
>>
>>
>
>
> --
> David McGinnis
> Staff Hadoop Consultant | Avalon Consulting, LLC
> <http://www.avalonconsult.com/>M: (513) 439-0082
> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> | Google+
> <http://www.google.com/+AvalonConsultingLLC> | Twitter
> <https://twitter.com/avalonconsult>
> ------------------------------------------------------------
> -------------------------------------------------
> This message (including any attachments) contains confidential information
> intended for a specific individual and purpose, and is protected by law.
> If
> you are not the intended recipient, you should delete this message. Any
> disclosure, copying, or distribution of this message, or the taking of any
> action based on it, is strictly prohibited.
>
