David - I also ran into this issue. I had to manually install the correct
version (as Ryan mentioned), but I also had to restart the Ambari Agent
running on the node experiencing the problem. Without restarting the
agent, the same issue would recur. I suspect that is your problem.
yum install python2-pip -y
pip install requests
ambari-agent restart
On Tue, Feb 27, 2018 at 3:45 PM, David McGinnis <mcginn...@avalonconsult.com
> wrote:
> Uninstalled requests and urllib3, and installed requests 2.6.1, and am
> still seeing a similar issue on launch.
>
> On Tue, Feb 27, 2018 at 2:19 PM, Ryan Merriman <merrim...@gmail.com>
> wrote:
>
>> Here is what I see in our Ansible scripts:
>>
>> - name: Install python-requests module
>> pip:
>> name: requests
>> version: 2.6.1
>>
>> I would try installing that exact version with pip. Also, have you tried
>> installing that dependency?
>>
>> On Tue, Feb 27, 2018 at 1:14 PM, David McGinnis <
>> mcginn...@avalonconsult.com> wrote:
>>
>>> I've tried that and it didn't fix the issue. I upgraded to the newest
>>> version of requests (2.18.4), and now instead of in that location, I am
>>> getting the same error in a dependency of requests (urllib3). See call
>>> stack below.
>>>
>>> Traceback (most recent call last):
>>> File
>>> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
>>> line 59, in run_file
>>> imp.load_source('__main__', script)
>>> File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/pa
>>> ckage/scripts/indexing_master.py", line 18, in <module>
>>> import requests
>>> File "/usr/lib/python2.7/site-packages/requests/__init__.py", line
>>> 43, in <module>
>>> import urllib3
>>> File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 8,
>>> in <module>
>>> from .connectionpool import (
>>> File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py",
>>> line 11, in <module>
>>> from .exceptions import (
>>> File "/usr/lib/python2.7/site-packages/urllib3/exceptions.py", line
>>> 2, in <module>
>>> from .packages.six.moves.http_client import (
>>> File "/usr/lib/python2.7/site-packages/urllib3/packages/six.py", line
>>> 198, in load_module
>>> return sys.modules[fullname]
>>> AttributeError: 'NoneType' object has no attribute 'modules'
>>>
>>> On Tue, Feb 27, 2018 at 1:12 PM, Ryan Merriman <merrim...@gmail.com>
>>> wrote:
>>>
>>>> I'm not a python expect but I would either uninstall then install with
>>>> pip or install dependencies as needed.
>>>>
>>>> On Tue, Feb 27, 2018 at 1:09 PM, David McGinnis <
>>>> mcginn...@avalonconsult.com> wrote:
>>>>
>>>>> Is there a fix available already, or how would you go about making
>>>>> this fix? I'm open to figuring it out if you have any ideas on how we
>>>>> should fix the issue. Requests was installed for me already without pip
>>>>> even being installed, likely through the MPack process or as part of a
>>>>> previously installed component. Pip reported requests had been installed
>>>>> through distutils.
>>>>>
>>>>> On Tue, Feb 27, 2018 at 1:06 PM, Ryan Merriman <merrim...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yep you found it. You were unlucky enough to install a version of
>>>>>> master that had this regression. How did you install requests? Using
>>>>>> pip
>>>>>> should also install the necessary dependencies. Either way everything
>>>>>> should still work, you'll just have to manually start/stop indexing since
>>>>>> you won't ever be able to stop it.
>>>>>>
>>>>>> You are correct about HDP 2.6.
>>>>>>
>>>>>> On Tue, Feb 27, 2018 at 12:57 PM, David McGinnis <
>>>>>> mcginn...@avalonconsult.com> wrote:
>>>>>>
>>>>>>> Looks like this issue was introduced in this commit:
>>>>>>> https://github.com/apache/metron/commit/1c9437c41c4eb626bd44
>>>>>>> 193ec6545f9356f3f7b2 . Looking past this, there is only one commit
>>>>>>> that seems to deal with this, but it is for the Ansible deployment, so I
>>>>>>> don't believe it would affect an MPack installation:
>>>>>>> https://github.com/apache/metron/commit/fa86663ecdb166f889c9
>>>>>>> c0c4c34373aec68dbd30 .
>>>>>>>
>>>>>>> I installed a newer version of requests just to see if that would
>>>>>>> fix the issue, and I'm getting a similar issue in a different package
>>>>>>> relied upon by requests now (urllib3). It seems as if something is
>>>>>>> causing
>>>>>>> sys to not exist or not point to what it should. I wonder if it has to
>>>>>>> do
>>>>>>> with the python2.6 vs python2.7 issue.
>>>>>>>
>>>>>>> Also, as I was looking through the commits, it looks like using the
>>>>>>> last official release wouldn't work for me, since we are on HDP 2.6, and
>>>>>>> changes went in since the last release to support that version of HDP.
>>>>>>>
>>>>>>> On Tue, Feb 27, 2018 at 12:12 PM, David McGinnis <
>>>>>>> mcginn...@avalonconsult.com> wrote:
>>>>>>>
>>>>>>>> *Simon:*
>>>>>>>>
>>>>>>>> There are some errors from processing showing up in the indexing
>>>>>>>> topic, but no data is making it through without restarting Metron
>>>>>>>> completely (restarting just the indexing service doesn't seem to do
>>>>>>>> it).
>>>>>>>>
>>>>>>>> I don't have easy access to the Storm UI at the moment. I have a
>>>>>>>> work around to get the logs that I'm going to look at shortly.
>>>>>>>>
>>>>>>>> I understand, but I didn't realize I had not pulled from a release
>>>>>>>> until I had already installed, and this is just a prototype cluster
>>>>>>>> anyways. I'll likely pull from an established release for the
>>>>>>>> production
>>>>>>>> deployment.
>>>>>>>>
>>>>>>>> *Ryan:*
>>>>>>>>
>>>>>>>> I actually found something interesting in the ambari-agent logs on
>>>>>>>> the Metron node. See the callstack below.
>>>>>>>>
>>>>>>>> Traceback (most recent call last):
>>>>>>>> File
>>>>>>>> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
>>>>>>>> line 59, in run_file
>>>>>>>> imp.load_source('__main__', script)
>>>>>>>> File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/pa
>>>>>>>> ckage/scripts/indexing_master.py", line 18, in <module>
>>>>>>>> import requests
>>>>>>>> File "/usr/lib/python2.7/site-packages/requests/__init__.py",
>>>>>>>> line 53, in <module>
>>>>>>>> from .packages.urllib3.contrib import pyopenssl
>>>>>>>> File
>>>>>>>> "/usr/lib/python2.7/site-packages/requests/packages/__init__.py",
>>>>>>>> line 61, in load_module
>>>>>>>> if name in sys.modules:
>>>>>>>> AttributeError: 'NoneType' object has no attribute 'modules'
>>>>>>>>
>>>>>>>> Perhaps there is a missing component that needs to be installed?
>>>>>>>>
>>>>>>>> On Tue, Feb 27, 2018 at 12:01 PM, Ryan Merriman <
>>>>>>>> merrim...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> If there was an error happening during the service check, it will
>>>>>>>>> be in the Ambari agent logs. I would look there first.
>>>>>>>>>
>>>>>>>>> On Feb 27, 2018, at 11:45 AM, Simon Elliston Ball <
>>>>>>>>> si...@simonellistonball.com> wrote:
>>>>>>>>>
>>>>>>>>> Anything appearing on the indexing topic in kafka?
>>>>>>>>>
>>>>>>>>> Anything in the logs for the indexing topology in Storm UI?
>>>>>>>>>
>>>>>>>>> Master isn’t always the best place to start, might be worth
>>>>>>>>> sticking to a released build to kick the tyres.
>>>>>>>>>
>>>>>>>>> Simon
>>>>>>>>>
>>>>>>>>> On 27 Feb 2018, at 17:38, David McGinnis <
>>>>>>>>> mcginn...@avalonconsult.com> wrote:
>>>>>>>>>
>>>>>>>>> All,
>>>>>>>>>
>>>>>>>>> I have a kerberized HDP 2.6.3 cluster which I have installed
>>>>>>>>> Metron on through an MPack. I am using code straight from the master
>>>>>>>>> branch
>>>>>>>>> of the github as of last week.
>>>>>>>>>
>>>>>>>>> When I start Metron, all of the components correct start up.
>>>>>>>>> Shortly after, however, the Indexing Service stops working. I have
>>>>>>>>> not been
>>>>>>>>> able to find any logs that seem to capture any issues, and both
>>>>>>>>> indexing
>>>>>>>>> storm topologies are still active and running after the failure. Those
>>>>>>>>> topologies only report a warning about one of my parsers using the
>>>>>>>>> default
>>>>>>>>> HDFS writer config, which appears to be normal.
>>>>>>>>>
>>>>>>>>> Not only is this making my dashboard red when it shouldn't be, but
>>>>>>>>> also it seems as if data doesn't go through the pipeline until the
>>>>>>>>> entire
>>>>>>>>> Metron system is restarted. I am not confident this is related, but I
>>>>>>>>> suspect it is.
>>>>>>>>>
>>>>>>>>> Where should I go to find logs about this service, or does anyone
>>>>>>>>> have any ideas why this would be happening?
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> David McGinnis
>>>>>>>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>>>>>>>> <http://www.avalonconsult.com/>M: (513) 439-0082
>>>>>>>>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc>
>>>>>>>>> | Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter
>>>>>>>>> <https://twitter.com/avalonconsult>
>>>>>>>>> ------------------------------------------------------------
>>>>>>>>> -------------------------------------------------
>>>>>>>>> This message (including any attachments) contains confidential
>>>>>>>>> information
>>>>>>>>> intended for a specific individual and purpose, and is protected
>>>>>>>>> by law. If
>>>>>>>>> you are not the intended recipient, you should delete this
>>>>>>>>> message. Any
>>>>>>>>> disclosure, copying, or distribution of this message, or the
>>>>>>>>> taking of any
>>>>>>>>> action based on it, is strictly prohibited.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> David McGinnis
>>>>>>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>>>>>>> <http://www.avalonconsult.com/>M: (513) 439-0082
>>>>>>>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> |
>>>>>>>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter
>>>>>>>> <https://twitter.com/avalonconsult>
>>>>>>>> ------------------------------------------------------------
>>>>>>>> -------------------------------------------------
>>>>>>>> This message (including any attachments) contains confidential
>>>>>>>> information
>>>>>>>> intended for a specific individual and purpose, and is protected by
>>>>>>>> law. If
>>>>>>>> you are not the intended recipient, you should delete this message.
>>>>>>>> Any
>>>>>>>> disclosure, copying, or distribution of this message, or the taking
>>>>>>>> of any
>>>>>>>> action based on it, is strictly prohibited.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> David McGinnis
>>>>>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>>>>>> <http://www.avalonconsult.com/>M: (513) 439-0082
>>>>>>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> |
>>>>>>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter
>>>>>>> <https://twitter.com/avalonconsult>
>>>>>>> ------------------------------------------------------------
>>>>>>> -------------------------------------------------
>>>>>>> This message (including any attachments) contains confidential
>>>>>>> information
>>>>>>> intended for a specific individual and purpose, and is protected by
>>>>>>> law. If
>>>>>>> you are not the intended recipient, you should delete this message.
>>>>>>> Any
>>>>>>> disclosure, copying, or distribution of this message, or the taking
>>>>>>> of any
>>>>>>> action based on it, is strictly prohibited.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> David McGinnis
>>>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>>>> <http://www.avalonconsult.com/>M: (513) 439-0082
>>>>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> |
>>>>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter
>>>>> <https://twitter.com/avalonconsult>
>>>>> ------------------------------------------------------------
>>>>> -------------------------------------------------
>>>>> This message (including any attachments) contains confidential
>>>>> information
>>>>> intended for a specific individual and purpose, and is protected by
>>>>> law. If
>>>>> you are not the intended recipient, you should delete this message.
>>>>> Any
>>>>> disclosure, copying, or distribution of this message, or the taking of
>>>>> any
>>>>> action based on it, is strictly prohibited.
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> David McGinnis
>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>> <http://www.avalonconsult.com/>M: (513) 439-0082
>>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> |
>>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter
>>> <https://twitter.com/avalonconsult>
>>> ------------------------------------------------------------
>>> -------------------------------------------------
>>> This message (including any attachments) contains confidential
>>> information
>>> intended for a specific individual and purpose, and is protected by law.
>>> If
>>> you are not the intended recipient, you should delete this message. Any
>>> disclosure, copying, or distribution of this message, or the taking of
>>> any
>>> action based on it, is strictly prohibited.
>>>
>>
>>
>
>
> --
> David McGinnis
> Staff Hadoop Consultant | Avalon Consulting, LLC
> <http://www.avalonconsult.com/>M: (513) 439-0082
> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> | Google+
> <http://www.google.com/+AvalonConsultingLLC> | Twitter
> <https://twitter.com/avalonconsult>
> ------------------------------------------------------------
> -------------------------------------------------
> This message (including any attachments) contains confidential information
> intended for a specific individual and purpose, and is protected by law.
> If
> you are not the intended recipient, you should delete this message. Any
> disclosure, copying, or distribution of this message, or the taking of any
> action based on it, is strictly prohibited.
>
