Re: Metron Indexing Service Failing Shortly After Start

[Nick Allen]

David - I also ran into this issue.  I had to manually install the correct
version (as Ryan mentioned), but I also had to restart the Ambari Agent
running on the node experiencing the problem.  Without restarting the
agent, the same issue would recur.  I suspect that is your problem.

yum install python2-pip -y
pip install requests
​ambari-agent restart​



On Tue, Feb 27, 2018 at 3:45 PM, David McGinnis  wrote:

> Uninstalled requests and urllib3, and installed requests 2.6.1, and am
> still seeing a similar issue on launch.
>
> On Tue, Feb 27, 2018 at 2:19 PM, Ryan Merriman 
> wrote:
>
>> Here is what I see in our Ansible scripts:
>>
>> - name: Install python-requests module
>>   pip:
>> name: requests
>> version: 2.6.1
>>
>> I would try installing that exact version with pip.  Also, have you tried
>> installing that dependency?
>>
>> On Tue, Feb 27, 2018 at 1:14 PM, David McGinnis <
>> mcginn...@avalonconsult.com> wrote:
>>
>>> I've tried that and it didn't fix the issue. I upgraded to the newest
>>> version of requests (2.18.4), and now instead of in that location, I am
>>> getting the same error in a dependency of requests (urllib3). See call
>>> stack below.
>>>
>>> Traceback (most recent call last):
>>>   File 
>>> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
>>> line 59, in run_file
>>> imp.load_source('__main__', script)
>>>   File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/pa
>>> ckage/scripts/indexing_master.py", line 18, in 
>>> import requests
>>>   File "/usr/lib/python2.7/site-packages/requests/__init__.py", line
>>> 43, in 
>>> import urllib3
>>>   File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 8,
>>> in 
>>> from .connectionpool import (
>>>   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py",
>>> line 11, in 
>>> from .exceptions import (
>>>   File "/usr/lib/python2.7/site-packages/urllib3/exceptions.py", line
>>> 2, in 
>>> from .packages.six.moves.http_client import (
>>>   File "/usr/lib/python2.7/site-packages/urllib3/packages/six.py", line
>>> 198, in load_module
>>> return sys.modules[fullname]
>>> AttributeError: 'NoneType' object has no attribute 'modules'
>>>
>>> On Tue, Feb 27, 2018 at 1:12 PM, Ryan Merriman 
>>> wrote:
>>>
 I'm not a python expect but I would either uninstall then install with
 pip or install dependencies as needed.

 On Tue, Feb 27, 2018 at 1:09 PM, David McGinnis <
 mcginn...@avalonconsult.com> wrote:

> Is there a fix available already, or how would you go about making
> this fix? I'm open to figuring it out if you have any ideas on how we
> should fix the issue. Requests was installed for me already without pip
> even being installed, likely through the MPack process or as part of a
> previously installed component. Pip reported requests had been installed
> through distutils.
>
> On Tue, Feb 27, 2018 at 1:06 PM, Ryan Merriman 
> wrote:
>
>> Yep you found it.  You were unlucky enough to install a version of
>> master that had this regression.  How did you install requests?  Using 
>> pip
>> should also install the necessary dependencies.  Either way everything
>> should still work, you'll just have to manually start/stop indexing since
>> you won't ever be able to stop it.
>>
>> You are correct about HDP 2.6.
>>
>> On Tue, Feb 27, 2018 at 12:57 PM, David McGinnis <
>> mcginn...@avalonconsult.com> wrote:
>>
>>> Looks like this issue was introduced in this commit:
>>> https://github.com/apache/metron/commit/1c9437c41c4eb626bd44
>>> 193ec6545f9356f3f7b2 . Looking past this, there is only one commit
>>> that seems to deal with this, but it is for the Ansible deployment, so I
>>> don't believe it would affect an MPack installation:
>>> https://github.com/apache/metron/commit/fa86663ecdb166f889c9
>>> c0c4c34373aec68dbd30 .
>>>
>>> I installed a newer version of requests just to see if that would
>>> fix the issue, and I'm getting a similar issue in a different package
>>> relied upon by requests now (urllib3). It seems as if something is 
>>> causing
>>> sys to not exist or not point to what it should. I wonder if it has to 
>>> do
>>> with the python2.6 vs python2.7 issue.
>>>
>>> Also, as I was looking through the commits, it looks like using the
>>> last official release wouldn't work for me, since we are on HDP 2.6, and
>>> changes went in since the last release to support that version of HDP.
>>>
>>> On Tue, Feb 27, 2018 at 12:12 PM, David McGinnis <
>>> mcginn...@avalonconsult.com> wrote:
>>>
 *Simon:*

 There are some errors from processing showing up in the indexing
 topic, but no data is making it through without 

Re: Metron Indexing Service Failing Shortly After Start

[David McGinnis]

Uninstalled requests and urllib3, and installed requests 2.6.1, and am
still seeing a similar issue on launch.

On Tue, Feb 27, 2018 at 2:19 PM, Ryan Merriman  wrote:

> Here is what I see in our Ansible scripts:
>
> - name: Install python-requests module
>   pip:
> name: requests
> version: 2.6.1
>
> I would try installing that exact version with pip.  Also, have you tried
> installing that dependency?
>
> On Tue, Feb 27, 2018 at 1:14 PM, David McGinnis <
> mcginn...@avalonconsult.com> wrote:
>
>> I've tried that and it didn't fix the issue. I upgraded to the newest
>> version of requests (2.18.4), and now instead of in that location, I am
>> getting the same error in a dependency of requests (urllib3). See call
>> stack below.
>>
>> Traceback (most recent call last):
>>   File 
>> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
>> line 59, in run_file
>> imp.load_source('__main__', script)
>>   File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/pa
>> ckage/scripts/indexing_master.py", line 18, in 
>> import requests
>>   File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 43,
>> in 
>> import urllib3
>>   File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 8,
>> in 
>> from .connectionpool import (
>>   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py",
>> line 11, in 
>> from .exceptions import (
>>   File "/usr/lib/python2.7/site-packages/urllib3/exceptions.py", line 2,
>> in 
>> from .packages.six.moves.http_client import (
>>   File "/usr/lib/python2.7/site-packages/urllib3/packages/six.py", line
>> 198, in load_module
>> return sys.modules[fullname]
>> AttributeError: 'NoneType' object has no attribute 'modules'
>>
>> On Tue, Feb 27, 2018 at 1:12 PM, Ryan Merriman 
>> wrote:
>>
>>> I'm not a python expect but I would either uninstall then install with
>>> pip or install dependencies as needed.
>>>
>>> On Tue, Feb 27, 2018 at 1:09 PM, David McGinnis <
>>> mcginn...@avalonconsult.com> wrote:
>>>
 Is there a fix available already, or how would you go about making this
 fix? I'm open to figuring it out if you have any ideas on how we should fix
 the issue. Requests was installed for me already without pip even being
 installed, likely through the MPack process or as part of a previously
 installed component. Pip reported requests had been installed through
 distutils.

 On Tue, Feb 27, 2018 at 1:06 PM, Ryan Merriman 
 wrote:

> Yep you found it.  You were unlucky enough to install a version of
> master that had this regression.  How did you install requests?  Using pip
> should also install the necessary dependencies.  Either way everything
> should still work, you'll just have to manually start/stop indexing since
> you won't ever be able to stop it.
>
> You are correct about HDP 2.6.
>
> On Tue, Feb 27, 2018 at 12:57 PM, David McGinnis <
> mcginn...@avalonconsult.com> wrote:
>
>> Looks like this issue was introduced in this commit:
>> https://github.com/apache/metron/commit/1c9437c41c4eb626bd44
>> 193ec6545f9356f3f7b2 . Looking past this, there is only one commit
>> that seems to deal with this, but it is for the Ansible deployment, so I
>> don't believe it would affect an MPack installation:
>> https://github.com/apache/metron/commit/fa86663ecdb166f889c9
>> c0c4c34373aec68dbd30 .
>>
>> I installed a newer version of requests just to see if that would fix
>> the issue, and I'm getting a similar issue in a different package relied
>> upon by requests now (urllib3). It seems as if something is causing sys 
>> to
>> not exist or not point to what it should. I wonder if it has to do with 
>> the
>> python2.6 vs python2.7 issue.
>>
>> Also, as I was looking through the commits, it looks like using the
>> last official release wouldn't work for me, since we are on HDP 2.6, and
>> changes went in since the last release to support that version of HDP.
>>
>> On Tue, Feb 27, 2018 at 12:12 PM, David McGinnis <
>> mcginn...@avalonconsult.com> wrote:
>>
>>> *Simon:*
>>>
>>> There are some errors from processing showing up in the indexing
>>> topic, but no data is making it through without restarting Metron
>>> completely (restarting just the indexing service doesn't seem to do it).
>>>
>>> I don't have easy access to the Storm UI at the moment. I have a
>>> work around to get the logs that I'm going to look at shortly.
>>>
>>> I understand, but I didn't realize I had not pulled from a release
>>> until I had already installed, and this is just a prototype cluster
>>> anyways. I'll likely pull from an established release for the production
>>> deployment.
>>>
>>> *Ryan:*
>>>
>>> I actually 

Re: Metron Indexing Service Failing Shortly After Start

[Ryan Merriman]

Here is what I see in our Ansible scripts:

- name: Install python-requests module
  pip:
name: requests
version: 2.6.1

I would try installing that exact version with pip.  Also, have you tried
installing that dependency?

On Tue, Feb 27, 2018 at 1:14 PM, David McGinnis  wrote:

> I've tried that and it didn't fix the issue. I upgraded to the newest
> version of requests (2.18.4), and now instead of in that location, I am
> getting the same error in a dependency of requests (urllib3). See call
> stack below.
>
> Traceback (most recent call last):
>   File 
> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
> line 59, in run_file
> imp.load_source('__main__', script)
>   File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/
> package/scripts/indexing_master.py", line 18, in 
> import requests
>   File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 43,
> in 
> import urllib3
>   File "/usr/lib/python2.7/site-packages/urllib3/__init__.py", line 8, in
> 
> from .connectionpool import (
>   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line
> 11, in 
> from .exceptions import (
>   File "/usr/lib/python2.7/site-packages/urllib3/exceptions.py", line 2,
> in 
> from .packages.six.moves.http_client import (
>   File "/usr/lib/python2.7/site-packages/urllib3/packages/six.py", line
> 198, in load_module
> return sys.modules[fullname]
> AttributeError: 'NoneType' object has no attribute 'modules'
>
> On Tue, Feb 27, 2018 at 1:12 PM, Ryan Merriman 
> wrote:
>
>> I'm not a python expect but I would either uninstall then install with
>> pip or install dependencies as needed.
>>
>> On Tue, Feb 27, 2018 at 1:09 PM, David McGinnis <
>> mcginn...@avalonconsult.com> wrote:
>>
>>> Is there a fix available already, or how would you go about making this
>>> fix? I'm open to figuring it out if you have any ideas on how we should fix
>>> the issue. Requests was installed for me already without pip even being
>>> installed, likely through the MPack process or as part of a previously
>>> installed component. Pip reported requests had been installed through
>>> distutils.
>>>
>>> On Tue, Feb 27, 2018 at 1:06 PM, Ryan Merriman 
>>> wrote:
>>>
 Yep you found it.  You were unlucky enough to install a version of
 master that had this regression.  How did you install requests?  Using pip
 should also install the necessary dependencies.  Either way everything
 should still work, you'll just have to manually start/stop indexing since
 you won't ever be able to stop it.

 You are correct about HDP 2.6.

 On Tue, Feb 27, 2018 at 12:57 PM, David McGinnis <
 mcginn...@avalonconsult.com> wrote:

> Looks like this issue was introduced in this commit:
> https://github.com/apache/metron/commit/1c9437c41c4eb626bd44
> 193ec6545f9356f3f7b2 . Looking past this, there is only one commit
> that seems to deal with this, but it is for the Ansible deployment, so I
> don't believe it would affect an MPack installation:
> https://github.com/apache/metron/commit/fa86663ecdb166f889c9
> c0c4c34373aec68dbd30 .
>
> I installed a newer version of requests just to see if that would fix
> the issue, and I'm getting a similar issue in a different package relied
> upon by requests now (urllib3). It seems as if something is causing sys to
> not exist or not point to what it should. I wonder if it has to do with 
> the
> python2.6 vs python2.7 issue.
>
> Also, as I was looking through the commits, it looks like using the
> last official release wouldn't work for me, since we are on HDP 2.6, and
> changes went in since the last release to support that version of HDP.
>
> On Tue, Feb 27, 2018 at 12:12 PM, David McGinnis <
> mcginn...@avalonconsult.com> wrote:
>
>> *Simon:*
>>
>> There are some errors from processing showing up in the indexing
>> topic, but no data is making it through without restarting Metron
>> completely (restarting just the indexing service doesn't seem to do it).
>>
>> I don't have easy access to the Storm UI at the moment. I have a work
>> around to get the logs that I'm going to look at shortly.
>>
>> I understand, but I didn't realize I had not pulled from a release
>> until I had already installed, and this is just a prototype cluster
>> anyways. I'll likely pull from an established release for the production
>> deployment.
>>
>> *Ryan:*
>>
>> I actually found something interesting in the ambari-agent logs on
>> the Metron node. See the callstack below.
>>
>> Traceback (most recent call last):
>>   File 
>> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
>> line 59, in run_file
>> imp.load_source('__main__', 

Re: Metron Indexing Service Failing Shortly After Start

[David McGinnis]

Is there a fix available already, or how would you go about making this
fix? I'm open to figuring it out if you have any ideas on how we should fix
the issue. Requests was installed for me already without pip even being
installed, likely through the MPack process or as part of a previously
installed component. Pip reported requests had been installed through
distutils.

On Tue, Feb 27, 2018 at 1:06 PM, Ryan Merriman  wrote:

> Yep you found it.  You were unlucky enough to install a version of master
> that had this regression.  How did you install requests?  Using pip should
> also install the necessary dependencies.  Either way everything should
> still work, you'll just have to manually start/stop indexing since you
> won't ever be able to stop it.
>
> You are correct about HDP 2.6.
>
> On Tue, Feb 27, 2018 at 12:57 PM, David McGinnis <
> mcginn...@avalonconsult.com> wrote:
>
>> Looks like this issue was introduced in this commit:
>> https://github.com/apache/metron/commit/1c9437c41c4eb626bd44
>> 193ec6545f9356f3f7b2 . Looking past this, there is only one commit that
>> seems to deal with this, but it is for the Ansible deployment, so I don't
>> believe it would affect an MPack installation:
>> https://github.com/apache/metron/commit/fa86663ecdb166f889c9
>> c0c4c34373aec68dbd30 .
>>
>> I installed a newer version of requests just to see if that would fix the
>> issue, and I'm getting a similar issue in a different package relied upon
>> by requests now (urllib3). It seems as if something is causing sys to not
>> exist or not point to what it should. I wonder if it has to do with the
>> python2.6 vs python2.7 issue.
>>
>> Also, as I was looking through the commits, it looks like using the last
>> official release wouldn't work for me, since we are on HDP 2.6, and changes
>> went in since the last release to support that version of HDP.
>>
>> On Tue, Feb 27, 2018 at 12:12 PM, David McGinnis <
>> mcginn...@avalonconsult.com> wrote:
>>
>>> *Simon:*
>>>
>>> There are some errors from processing showing up in the indexing topic,
>>> but no data is making it through without restarting Metron completely
>>> (restarting just the indexing service doesn't seem to do it).
>>>
>>> I don't have easy access to the Storm UI at the moment. I have a work
>>> around to get the logs that I'm going to look at shortly.
>>>
>>> I understand, but I didn't realize I had not pulled from a release until
>>> I had already installed, and this is just a prototype cluster anyways. I'll
>>> likely pull from an established release for the production deployment.
>>>
>>> *Ryan:*
>>>
>>> I actually found something interesting in the ambari-agent logs on the
>>> Metron node. See the callstack below.
>>>
>>> Traceback (most recent call last):
>>>   File 
>>> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
>>> line 59, in run_file
>>> imp.load_source('__main__', script)
>>>   File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/pa
>>> ckage/scripts/indexing_master.py", line 18, in 
>>> import requests
>>>   File "/usr/lib/python2.7/site-packages/requests/__init__.py", line
>>> 53, in 
>>> from .packages.urllib3.contrib import pyopenssl
>>>   File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py",
>>> line 61, in load_module
>>> if name in sys.modules:
>>> AttributeError: 'NoneType' object has no attribute 'modules'
>>>
>>> Perhaps there is a missing component that needs to be installed?
>>>
>>> On Tue, Feb 27, 2018 at 12:01 PM, Ryan Merriman 
>>> wrote:
>>>
 If there was an error happening during the service check, it will be in
 the Ambari agent logs.  I would look there first.

 On Feb 27, 2018, at 11:45 AM, Simon Elliston Ball <
 si...@simonellistonball.com> wrote:

 Anything appearing on the indexing topic in kafka?

 Anything in the logs for the indexing topology in Storm UI?

 Master isn’t always the best place to start, might be worth sticking to
 a released build to kick the tyres.

 Simon

 On 27 Feb 2018, at 17:38, David McGinnis 
 wrote:

 All,

 I have a kerberized HDP 2.6.3 cluster which I have installed Metron on
 through an MPack. I am using code straight from the master branch of the
 github as of last week.

 When I start Metron, all of the components correct start up. Shortly
 after, however, the Indexing Service stops working. I have not been able to
 find any logs that seem to capture any issues, and both indexing storm
 topologies are still active and running after the failure. Those topologies
 only report a warning about one of my parsers using the default HDFS writer
 config, which appears to be normal.

 Not only is this making my dashboard red when it shouldn't be, but also
 it seems as if data doesn't go through the pipeline until the entire 

Re: Metron Indexing Service Failing Shortly After Start

[Ryan Merriman]

Yep you found it.  You were unlucky enough to install a version of master
that had this regression.  How did you install requests?  Using pip should
also install the necessary dependencies.  Either way everything should
still work, you'll just have to manually start/stop indexing since you
won't ever be able to stop it.

You are correct about HDP 2.6.

On Tue, Feb 27, 2018 at 12:57 PM, David McGinnis <
mcginn...@avalonconsult.com> wrote:

> Looks like this issue was introduced in this commit:
> https://github.com/apache/metron/commit/1c9437c41c4eb626bd44
> 193ec6545f9356f3f7b2 . Looking past this, there is only one commit that
> seems to deal with this, but it is for the Ansible deployment, so I don't
> believe it would affect an MPack installation:
> https://github.com/apache/metron/commit/fa86663ecdb166f889c9
> c0c4c34373aec68dbd30 .
>
> I installed a newer version of requests just to see if that would fix the
> issue, and I'm getting a similar issue in a different package relied upon
> by requests now (urllib3). It seems as if something is causing sys to not
> exist or not point to what it should. I wonder if it has to do with the
> python2.6 vs python2.7 issue.
>
> Also, as I was looking through the commits, it looks like using the last
> official release wouldn't work for me, since we are on HDP 2.6, and changes
> went in since the last release to support that version of HDP.
>
> On Tue, Feb 27, 2018 at 12:12 PM, David McGinnis <
> mcginn...@avalonconsult.com> wrote:
>
>> *Simon:*
>>
>> There are some errors from processing showing up in the indexing topic,
>> but no data is making it through without restarting Metron completely
>> (restarting just the indexing service doesn't seem to do it).
>>
>> I don't have easy access to the Storm UI at the moment. I have a work
>> around to get the logs that I'm going to look at shortly.
>>
>> I understand, but I didn't realize I had not pulled from a release until
>> I had already installed, and this is just a prototype cluster anyways. I'll
>> likely pull from an established release for the production deployment.
>>
>> *Ryan:*
>>
>> I actually found something interesting in the ambari-agent logs on the
>> Metron node. See the callstack below.
>>
>> Traceback (most recent call last):
>>   File 
>> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
>> line 59, in run_file
>> imp.load_source('__main__', script)
>>   File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/pa
>> ckage/scripts/indexing_master.py", line 18, in 
>> import requests
>>   File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 53,
>> in 
>> from .packages.urllib3.contrib import pyopenssl
>>   File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py",
>> line 61, in load_module
>> if name in sys.modules:
>> AttributeError: 'NoneType' object has no attribute 'modules'
>>
>> Perhaps there is a missing component that needs to be installed?
>>
>> On Tue, Feb 27, 2018 at 12:01 PM, Ryan Merriman 
>> wrote:
>>
>>> If there was an error happening during the service check, it will be in
>>> the Ambari agent logs.  I would look there first.
>>>
>>> On Feb 27, 2018, at 11:45 AM, Simon Elliston Ball <
>>> si...@simonellistonball.com> wrote:
>>>
>>> Anything appearing on the indexing topic in kafka?
>>>
>>> Anything in the logs for the indexing topology in Storm UI?
>>>
>>> Master isn’t always the best place to start, might be worth sticking to
>>> a released build to kick the tyres.
>>>
>>> Simon
>>>
>>> On 27 Feb 2018, at 17:38, David McGinnis 
>>> wrote:
>>>
>>> All,
>>>
>>> I have a kerberized HDP 2.6.3 cluster which I have installed Metron on
>>> through an MPack. I am using code straight from the master branch of the
>>> github as of last week.
>>>
>>> When I start Metron, all of the components correct start up. Shortly
>>> after, however, the Indexing Service stops working. I have not been able to
>>> find any logs that seem to capture any issues, and both indexing storm
>>> topologies are still active and running after the failure. Those topologies
>>> only report a warning about one of my parsers using the default HDFS writer
>>> config, which appears to be normal.
>>>
>>> Not only is this making my dashboard red when it shouldn't be, but also
>>> it seems as if data doesn't go through the pipeline until the entire Metron
>>> system is restarted. I am not confident this is related, but I suspect it
>>> is.
>>>
>>> Where should I go to find logs about this service, or does anyone have
>>> any ideas why this would be happening?
>>>
>>> --
>>> David McGinnis
>>> Staff Hadoop Consultant | Avalon Consulting, LLC
>>> M: (513) 439-0082
>>> LinkedIn  |
>>> Google+  | Twitter
>>> 
>>> 

Re: Metron Indexing Service Failing Shortly After Start

[David McGinnis]

Looks like this issue was introduced in this commit:
https://github.com/apache/metron/commit/1c9437c41c4eb626bd44
193ec6545f9356f3f7b2 . Looking past this, there is only one commit that
seems to deal with this, but it is for the Ansible deployment, so I don't
believe it would affect an MPack installation: https://github.com/apache/
metron/commit/fa86663ecdb166f889c9c0c4c34373aec68dbd30 .

I installed a newer version of requests just to see if that would fix the
issue, and I'm getting a similar issue in a different package relied upon
by requests now (urllib3). It seems as if something is causing sys to not
exist or not point to what it should. I wonder if it has to do with the
python2.6 vs python2.7 issue.

Also, as I was looking through the commits, it looks like using the last
official release wouldn't work for me, since we are on HDP 2.6, and changes
went in since the last release to support that version of HDP.

On Tue, Feb 27, 2018 at 12:12 PM, David McGinnis <
mcginn...@avalonconsult.com> wrote:

> *Simon:*
>
> There are some errors from processing showing up in the indexing topic,
> but no data is making it through without restarting Metron completely
> (restarting just the indexing service doesn't seem to do it).
>
> I don't have easy access to the Storm UI at the moment. I have a work
> around to get the logs that I'm going to look at shortly.
>
> I understand, but I didn't realize I had not pulled from a release until I
> had already installed, and this is just a prototype cluster anyways. I'll
> likely pull from an established release for the production deployment.
>
> *Ryan:*
>
> I actually found something interesting in the ambari-agent logs on the
> Metron node. See the callstack below.
>
> Traceback (most recent call last):
>   File 
> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
> line 59, in run_file
> imp.load_source('__main__', script)
>   File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/
> package/scripts/indexing_master.py", line 18, in 
> import requests
>   File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 53,
> in 
> from .packages.urllib3.contrib import pyopenssl
>   File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py",
> line 61, in load_module
> if name in sys.modules:
> AttributeError: 'NoneType' object has no attribute 'modules'
>
> Perhaps there is a missing component that needs to be installed?
>
> On Tue, Feb 27, 2018 at 12:01 PM, Ryan Merriman 
> wrote:
>
>> If there was an error happening during the service check, it will be in
>> the Ambari agent logs.  I would look there first.
>>
>> On Feb 27, 2018, at 11:45 AM, Simon Elliston Ball <
>> si...@simonellistonball.com> wrote:
>>
>> Anything appearing on the indexing topic in kafka?
>>
>> Anything in the logs for the indexing topology in Storm UI?
>>
>> Master isn’t always the best place to start, might be worth sticking to a
>> released build to kick the tyres.
>>
>> Simon
>>
>> On 27 Feb 2018, at 17:38, David McGinnis 
>> wrote:
>>
>> All,
>>
>> I have a kerberized HDP 2.6.3 cluster which I have installed Metron on
>> through an MPack. I am using code straight from the master branch of the
>> github as of last week.
>>
>> When I start Metron, all of the components correct start up. Shortly
>> after, however, the Indexing Service stops working. I have not been able to
>> find any logs that seem to capture any issues, and both indexing storm
>> topologies are still active and running after the failure. Those topologies
>> only report a warning about one of my parsers using the default HDFS writer
>> config, which appears to be normal.
>>
>> Not only is this making my dashboard red when it shouldn't be, but also
>> it seems as if data doesn't go through the pipeline until the entire Metron
>> system is restarted. I am not confident this is related, but I suspect it
>> is.
>>
>> Where should I go to find logs about this service, or does anyone have
>> any ideas why this would be happening?
>>
>> --
>> David McGinnis
>> Staff Hadoop Consultant | Avalon Consulting, LLC
>> M: (513) 439-0082
>> LinkedIn  |
>> Google+  | Twitter
>> 
>> 
>> -
>> This message (including any attachments) contains confidential
>> information
>> intended for a specific individual and purpose, and is protected by law.
>> If
>> you are not the intended recipient, you should delete this message. Any
>> disclosure, copying, or distribution of this message, or the taking of
>> any
>> action based on it, is strictly prohibited.
>>
>>
>>
>
>
> --
> David McGinnis
> Staff Hadoop Consultant | Avalon Consulting, LLC
> M: 

Re: Metron Indexing Service Failing Shortly After Start

[David McGinnis]

*Simon:*

There are some errors from processing showing up in the indexing topic, but
no data is making it through without restarting Metron completely
(restarting just the indexing service doesn't seem to do it).

I don't have easy access to the Storm UI at the moment. I have a work
around to get the logs that I'm going to look at shortly.

I understand, but I didn't realize I had not pulled from a release until I
had already installed, and this is just a prototype cluster anyways. I'll
likely pull from an established release for the production deployment.

*Ryan:*

I actually found something interesting in the ambari-agent logs on the
Metron node. See the callstack below.

Traceback (most recent call last):
  File
"/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
line 59, in run_file
imp.load_source('__main__', script)
  File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.3/package/scripts/indexing_master.py",
line 18, in 
import requests
  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 53, in

from .packages.urllib3.contrib import pyopenssl
  File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py",
line 61, in load_module
if name in sys.modules:
AttributeError: 'NoneType' object has no attribute 'modules'

Perhaps there is a missing component that needs to be installed?

On Tue, Feb 27, 2018 at 12:01 PM, Ryan Merriman  wrote:

> If there was an error happening during the service check, it will be in
> the Ambari agent logs.  I would look there first.
>
> On Feb 27, 2018, at 11:45 AM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> Anything appearing on the indexing topic in kafka?
>
> Anything in the logs for the indexing topology in Storm UI?
>
> Master isn’t always the best place to start, might be worth sticking to a
> released build to kick the tyres.
>
> Simon
>
> On 27 Feb 2018, at 17:38, David McGinnis 
> wrote:
>
> All,
>
> I have a kerberized HDP 2.6.3 cluster which I have installed Metron on
> through an MPack. I am using code straight from the master branch of the
> github as of last week.
>
> When I start Metron, all of the components correct start up. Shortly
> after, however, the Indexing Service stops working. I have not been able to
> find any logs that seem to capture any issues, and both indexing storm
> topologies are still active and running after the failure. Those topologies
> only report a warning about one of my parsers using the default HDFS writer
> config, which appears to be normal.
>
> Not only is this making my dashboard red when it shouldn't be, but also it
> seems as if data doesn't go through the pipeline until the entire Metron
> system is restarted. I am not confident this is related, but I suspect it
> is.
>
> Where should I go to find logs about this service, or does anyone have any
> ideas why this would be happening?
>
> --
> David McGinnis
> Staff Hadoop Consultant | Avalon Consulting, LLC
> M: (513) 439-0082
> LinkedIn  | Google+
>  | Twitter
> 
> 
> -
> This message (including any attachments) contains confidential information
> intended for a specific individual and purpose, and is protected by law.
> If
> you are not the intended recipient, you should delete this message. Any
> disclosure, copying, or distribution of this message, or the taking of any
> action based on it, is strictly prohibited.
>
>
>


-- 
David McGinnis
Staff Hadoop Consultant | Avalon Consulting, LLC
M: (513) 439-0082
LinkedIn  | Google+
 | Twitter

-
This message (including any attachments) contains confidential information
intended for a specific individual and purpose, and is protected by law. If
you are not the intended recipient, you should delete this message. Any
disclosure, copying, or distribution of this message, or the taking of any
action based on it, is strictly prohibited.