Re: Error building in Travis after taking master

[Otto Fowler]

I am not sure I know how to do that in Travis.  I will look when I get back

On May 3, 2017 at 11:35:34, Ryan Merriman (merrim...@gmail.com) wrote:

> You might want to try clearing the mvn cache. I've had travis get into a
> bad state before because of corrupt maven artifacts.
>
> On Wed, May 3, 2017 at 10:26 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> https://travis-ci.org/ottobackwards/incubator-metron/builds/228364894?utm_
> source=email_medium=notification
>
> On May 3, 2017 at 11:12:15, Justin Leet (justinjl...@gmail.com) wrote:
>
> Jacoco is introduced from
> https://github.com/apache/incubator-metron/pull/459.
>
> I'm not sure why you'd be getting a plugin error for it though, because
> it's available in the standard repos and I was able to build off a
> completely clean maven cache (and Travis has been able to build it as
> well).
>
> What exactly you were running?
>
> On Wed, May 3, 2017 at 11:03 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> Anyone know what this is?
>
> [ERROR] No plugin found for prefix 'jacoco' in the current project and in
> the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available
> from the repositories [local (/home/travis/.m2/repository), central (
> https://repo.maven.apache.org/maven2)] -> [Help 1]
>
>
>
>

[DISCUSS] platform_info.sh and PR template / Dev Guidelines

[Otto Fowler]

I think we should have documented in the template and guidelines something
to the extent ( appropriate for the document ) of :

“Metron provides a platform_info.sh script in metron-deployment/scripts
that outputs information that may be important to troubleshooting build and
deployment issues.  It is important if any development introduces new
technology to the stack that effects the build and deployment by it’s
presence or version, that the platform_info.sh be updated to reflect that
information.”


pr:
[ ] platform_info.sh updated for new dependencies

Re: Request double-check on Ambari config logic (ES network_host)

[Otto Fowler]

of actual bind addresses.  If you
> give it the wildcard value (“0.0.0.0” for ipv4), it will bind to all
> available addresses.
> -network.publish_host is the address Elasticsearch “publishes” for
> clients and other servers to connect to. It will publish only one address.
> If you give it a set of addresses, it picks the most “desirable” of the set
> – it assures it actually is accessible, and it prefers ipv4 (or 6,
> depending on another config), then  global, then site-local, then
> link-local, then loopback. Within each category it orders by numeric
> magnitude of the IP address, which is hardly meaningful.  This means the
> published address can be wrong on a multi-homed server or VM, if you don’t
> appropriately constrain it.
> -The parameter values can be network addresses, network interface
> names, host names (to be dereferenced via DNS), “special” names denoting
> predefined sets of addresses, and combinations of the above.
> -Wildcard and loopback addresses are allowed.
> -If the wildcard is provided it must be the ONLY value provided (list
> of length == 1), or ES will throw an error.
>
> Discussion item 1:  If you use network.host, the same list of addresses
> get sent to both network.bind_host and network.publish_host.  The algorithm
> for picking the single publish_host address is not good enough, at least in
> ES 2.3, to give certainty that the right address will be published, on
> multi-homed servers or VMs (although on non-multi-homed, it should
> generally work fine).
>
> It seems to me that specifying exactly one of _local_, _site_, or _global_
> will usually give the right result, but that too can fail if the server has
> multiple addresses within the same category.
>
> I think network.bind_host and network.publish_host should be separately
> configured, as they are with Hadoop.
> There’s an article here:
> https://community.hortonworks.com/content/kbentry/24277/parameters-for-multi-homing.html
> that discusses these issues at some length, and clarifies why they must be
> separately configured.
>
> What do you-all think?
>
> Discussion item 2:  While it’s fine to use 0.0.0.0 for the bind address,
> it gives no guidance at all to the needed publish_host value. Using _local_
> for QuickDev and single-node deployments, and _site_  for FullDev
> deployments and all cluster deployments, is probably a reasonable choice
> for publish_host.
>
> What do you-all think?
>
> Discussion item 3: Should we attempt to further the “hadoop style” of
> config parameter, and silently add the square brackets and perhaps
> substring quotes in python processing?  Or should we say users need to
> understand ES configuration, and tell them to put the list in square
> brackets themselves, if they need a list entry in this parameter, per
> https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-network.html
> ?
>
> Please share your thoughts,
> Thanks,
> --Matt
>
>
> On 5/2/17, 9:57 PM, "Matt Foley" <mfo...@hortonworks.com> wrote:
>
> Hi Otto,
> This event derives from this line of code:
> https://github.com/elastic/elasticsearch/blob/2.3/core/src/main/java/org/elasticsearch/action/support/master/TransportMasterNodeAction.java#L148
> which suggests that a cluster action has been requested on a local
> (loopback) address.  This is not
>     surprising given what I’ve learned about the semantics of network.host
> with wildcard address.
> See next message, item C.  Basically, while the wildcard causes ES to
> “listen” on all IP addresses, it
> only *publishes* one, and on a multi-homed server it can be the wrong
> one.  I can’t be certain
> this causes what you’re seeing, but it seems feasible.
>
> From: Otto Fowler <ottobackwa...@gmail.com>
> Date: Tuesday, May 2, 2017 at 8:30 PM
> To: "d...@metron.incubator.apache.org" <d...@metron.incubator.apache.org>,
> Matt Foley <mfo...@hortonworks.com>, "dev@metron.apache.org" <
> dev@metron.apache.org>, "zeo...@gmail.com" <zeo...@gmail.com>
> Subject: Re: Request double-check on Ambari config logic (ES
> network_host)
>
> OK.
> I tried it using this method, and master ( adding [] ).  In both
> cases, I can hit 9200 from other machines, but in both cases I’m getting ES
> master errors:
>
> ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not
> recovered / initialized];]
> at
> org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:174)
> at
> org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:66)
>

Re: Request double-check on Ambari config logic (ES network_host)

[Otto Fowler]

OK.
I tried it using this method, and master ( adding [] ).  In both cases, I
can hit 9200 from other machines, but in both cases I’m getting ES master
errors:

ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not
recovered / initialized];]
at
org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:174)
at
org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:66)
at
org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:41)
at
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:148)
at
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:140)
at
org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:107)
at
org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:51)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)
at
org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:98)
at
org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:66)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
at
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52)
at
org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.doExecute(BaseRestHandler.java:83)
at
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
at
org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:371)
at
org.elasticsearch.rest.action.index.RestIndexAction.handleRequest(RestIndexAction.java:102)
at
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54)
at
org.elasticsearch.rest.RestController.executeHandler(RestController.java:205)
at
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166)
at
org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:128)
at
org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:86)
at
org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServ

and kibana is not good.

not sure what that error means.
I have 5 nodes, and put es master on #5, with #3,4 as datanodes.

Sorry, but I don’t think my setup is going to be much help at this point.



On May 2, 2017 at 17:19:43, Matt Foley (mfo...@hortonworks.com) wrote:

The default will now be “0.0.0.0”, and not eth0. And this will work if
suggestions from various community members and a suggestion in the old 1.x
documentation for ES are correct. The 2.x documentation (we specify ES 2.3)
doesn’t mention “0.0.0.0”, but I think it’s likely to still work, but it
needs testing.

Thanks,
--Matt

From: Otto Fowler <ottobackwa...@gmail.com>
Date: Tuesday, May 2, 2017 at 11:27 AM
To: "d...@metron.incubator.apache.org" <d...@metron.incubator.apache.org>,
Matt Foley <mfo...@hortonworks.com>, "dev@metron.apache.org" <
dev@metron.apache.org>, "zeo...@gmail.com" <zeo...@gmail.com>
Subject: Re: Request double-check on Ambari config logic (ES network_host)

Are you saying that the defaults should work now?
Or they should work, but I still need to change the interface from eth0?




On May 2, 2017 at 13:36:11, Matt Foley (mfo...@hortonworks.com) wrote:
Hi Otto,
The basic change to use “0.0.0.0” as the default binding, and put the
square brackets in the template text instead of the parameter value, is now
available in
https://github.com/mattf-horton/incubator-metron branch METRON-905 commit
e879719a0c3fb

I’m having some trouble with my test env, so if you wanted to give it a
try, that would be great.
If the “0.0.0.0” doesn’t work, then we should use
"_local_", "_site_"
that being the ES special values that mean aprx the same.

I’m going to have to do trial-and-error to determine the exact behavior of
multi-item lists, and then write the python code to strip redundant square
brackets if included in the parameter value.
Thanks,
--Matt


On 5/2/17, 6:44 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote:

I am working on a centos 7 cluster deploy for testing the steps.
I have this issue ( along with the wrong interface name ) and can test when
you have it.

An eta would help?


On May 2, 2017 at 09:14:10, zeo...@gmail.com (zeo...@gmail.com) wrote:

Are you working on this one? The JIRA doesn't look like it's currently
assigned. Thanks,

Jon

On Mon, May 

Re: Request double-check on Ambari config logic (ES network_host)

[Otto Fowler]

Thanks!


On May 2, 2017 at 14:58:37, Nick Allen (n...@nickallen.org) wrote:

I don't know how to uninstall, but you can reinstall by passing the --force
flag


On Tue, May 2, 2017 at 2:33 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Do you know how to uninstall an mpack from the cli?
>
>
> On May 2, 2017 at 14:27:02, Otto Fowler (ottobackwa...@gmail.com) wrote:
>
> Are you saying that the defaults should work now?
> Or they should work, but I still need to change the interface from eth0?
>
>
>
> On May 2, 2017 at 13:36:11, Matt Foley (mfo...@hortonworks.com) wrote:
>
> Hi Otto,
> The basic change to use “0.0.0.0” as the default binding, and put the
> square brackets in the template text instead of the parameter value, is
now
> available in
> https://github.com/mattf-horton/incubator-metron branch METRON-905 commit
> e879719a0c3fb
>
> I’m having some trouble with my test env, so if you wanted to give it a
> try, that would be great.
> If the “0.0.0.0” doesn’t work, then we should use
> "_local_", "_site_"
> that being the ES special values that mean aprx the same.
>
> I’m going to have to do trial-and-error to determine the exact behavior
of
> multi-item lists, and then write the python code to strip redundant
square
> brackets if included in the parameter value.
> Thanks,
> --Matt
>
>
> On 5/2/17, 6:44 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote:
>
> I am working on a centos 7 cluster deploy for testing the steps.
> I have this issue ( along with the wrong interface name ) and can test
when
> you have it.
>
> An eta would help?
>
>
> On May 2, 2017 at 09:14:10, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> Are you working on this one? The JIRA doesn't look like it's currently
> assigned. Thanks,
>
> Jon
>
> On Mon, May 1, 2017 at 6:40 PM Matt Foley <mfo...@hortonworks.com> wrote:
>
> > Ah, I see I mis-read METRON-897, and Nick specifically says
> > "lo:ipv4","eth0:ipv4" did not work for him, but
> ["_lo:ipv4_","_eth0:ipv4_"]
> > did work.
> >
> > So I went back and dug a little deeper, and realized that in the
> > environment where "lo:ipv4","eth0:ipv4" worked for me, I had modified
the
> > yaml.j2 template to include the square brackets.
> >
> > So the below theory is wrong. Back to the drawing board.
> > Thanks,
> > --Matt
> >
> > On 5/1/17, 3:08 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > Hi, there have been widely varying statements about what needs to be
> > in the Elasticsearch config parameter “network_host”. I think I may
have
> a
> > rationale for what works and what doesn’t, but I’d like your input or
> > correction.
> >
> > I am focusing on what worked in terms of punctuation (quotes and
> > square brackets) with the old _lo:ip4_,_eth0:ip4_. I would like to
ignore
> > for the moment, please, whether eth0 was the correct name for a given
> env,
> > and whether we can use 0.0.0.0. Instead, for systems where eth0 WAS the
> > correct name, I’d like to understand what worked and why.
> >
> > It’s complicated because the value starts out in xml, is read into
> > python, printed by jinja, then consumed by yaml.
> >
> > I think there were two constructs that actually worked for this
> > param. Please say whether this is consistent or inconsistent with your
> > experience:
> >
> > "_lo:ip4_","_eth0:ip4_"
> > This worked for me. I think this was read from XML into python as a
> > list of strings, then output in jinja ‘print statement‘
> > {{ network_host }} as a python literal list with form:
> > [ "_lo:ip4_", "_eth0:ip4_" ]
> > In other words, the print statement for a python list object injected
> > the needed square brackets.
> >
> > and
> > "[ _lo:ip4_, _eth0:ip4_ ]"
> > Nick and Anand, please confirm if this is the form that worked for
> > you. I think this was read from XML into python as a single string, and
> > output in the same jinja print statement as:
> > [ _lo:ip4_, _eth0:ip4_ ]
> > because the print statement for a python string object does not
> > produce quote marks.
> >
> > In either case, yaml (the consumer of the jinja output) saw what it
> > interprets as a list of strings (since quotes are optional for yaml
> > strings).
> >
> > What didn’t work was:
> >
> > * "_lo:ip4_, _eth0:ip4_"
> > This would be read in and output as a single string, and no square
> > brackets would ever be introduced.
> >
> > * _lo:ip4_, _eth0:ip4_ or [ _lo:ip4_, _eth0:ip4_ ]
> > (without quotes) I think the unquoted colons messed up the python
> > parsing
> >
> > Finally, I don’t know whether
> > * [ "_lo:ip4_", "_eth0:ip4_" ]
> > worked or not, I’m not sure anyone ever tried it. By the above logic
> > it probably should work.
> >
> > Please give me your input if you have touched on these issues.
> > Thanks,
> > --Matt
> >
> >
> >
> >
> >
> >
> > --
>
> Jon
>

Re: introduction

[Otto Fowler]

Welcome!


On May 2, 2017 at 07:55:58, Christian Tramnitz (tramn...@trasec.de) wrote:

Hello Metron developers,

I thought I’d first introduce myself before I ask any questions:
I’m new to Metron, but we are running a fairly complex OpenSOC-based
installation for a customer. Since Metron evolved from OpenSOC our
code-base drifted apart somewhat and I don’t know if we can contribute back
some of the stuff we added with that customer, but in parallel I would like
to make myself familiar with Metron for internal use, other projects and
reusing a few components (i.e. the indexing topology) in the existing
environment.



Best regards,
Christian Tramnitz

Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)

[Otto Fowler]

Do we have any async methods in the current api I could look at?


On April 27, 2017 at 10:45:09, Otto Fowler (ottobackwa...@gmail.com) wrote:

I thought as much.  I have not worked with the framework before however.


On April 27, 2017 at 10:20:20, Ryan Merriman (merrim...@gmail.com) wrote:

I think leveraging the REST application would work for this use case.
Services already exist for most of the functions listed in your pseudocode
(HDFS read/write, Zookeeper read/write). Asynchronous functions are also
supported so no issues there.

On Thu, Apr 27, 2017 at 9:09 AM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out,
including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers. This will be the
> effort to actually install 3rd party parser and other extensions ( stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this. We would use the current services, or orchestrate them to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser - unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way? Should we do an ambari view instead?
> * Is this too much to do in a rest call? Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.
>

Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)

[Otto Fowler]

I thought as much.  I have not worked with the framework before however.


On April 27, 2017 at 10:20:20, Ryan Merriman (merrim...@gmail.com) wrote:

I think leveraging the REST application would work for this use case.
Services already exist for most of the functions listed in your pseudocode
(HDFS read/write, Zookeeper read/write). Asynchronous functions are also
supported so no issues there.

On Thu, Apr 27, 2017 at 9:09 AM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out,
including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers. This will be the
> effort to actually install 3rd party parser and other extensions (
stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this. We would use the current services, or orchestrate them
to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser -
unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way? Should we do an ambari view instead?
> * Is this too much to do in a rest call? Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.
>

Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)

[Otto Fowler]

I think this approach should be extended to the parsers too down the road
as I’ve mentioned before.
Installing a parser installed the configurations as templates, and from
there you create new instances ( named ).
Those instances are equivalent to what we have now.

Install parser -> configurations to ‘available area’ ( ES, ParserConfig,
 )
UI -> Create Parser Instance -> take available, clone, edit save/deploy for
XXX


On April 27, 2017 at 10:28:18, Otto Fowler (ottobackwa...@gmail.com) wrote:

Maybe the templates can be installed somewhere, not ES but some store.  The
UI can list the available templates, and you can edit and deploy from the
UI?
They can still be packaged with the parser, just not go to ES.  Does that
address your concerns?


On April 27, 2017 at 10:25:37, Otto Fowler (ottobackwa...@gmail.com) wrote:

“As a parser developer, I want to build, test, and maintain my parser
extension as a single deliverable ‘package’”.
I think from the 3rd party developer perspective, everything required to
make the parser work, should be packaged and installed with the parser.
The point is they may only be working with the parser code, and never touch
ambari or other ‘metron’ source.  Having essential components
managed otherwise makes it harder for the developers.

I know this pov is true for my teams.


On April 27, 2017 at 10:15:55, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Otto,

Happy to help around this.

Couple of questions and things to maybe think about…

REST seems fine for sending the payload, but the unpacking job should
probably be async (triggered by the job, but not responsible for replying,
i.e. the initial call gets an ACCEPTED response immediately before the
heavy lifting is done.

I’m still not 100% sold on the idea of ES templates being included in the
bundle, and would prefer that to be a broader “register the schema for the
parser” somewhere from which the Metron framework can generate templates,
this allows us to account for things like enrichments in templates, but the
principle you describe holds if we make it non-ES specific (some people use
solr for instance).

If we do this as a REST call, we can easily incorporate it to the UI, or
maybe add an ambari view later, so I’d say the REST approach is definitely
the right first step.

One other thing to think about is whether this belongs in the metron-rest
package (personally i think it probably does) or whether it is a separate
micro-service.

Anyway, happy to help in any way I can on this.

Simon


> On 27 Apr 2017, at 15:09, Otto Fowler <ottobackwa...@gmail.com> wrote:
>
> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwa...@gmail.com)
wrote:
>
> So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out,
including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers. This will be the
> effort to actually install 3rd party parser and other extensions ( stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this. We would use the current services, or orchestrate them to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser - unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way? Should we do an ambari view instead?
> * Is this too much to do in a rest call? Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.

Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)

[Otto Fowler]

Maybe the templates can be installed somewhere, not ES but some store.  The
UI can list the available templates, and you can edit and deploy from the
UI?
They can still be packaged with the parser, just not go to ES.  Does that
address your concerns?


On April 27, 2017 at 10:25:37, Otto Fowler (ottobackwa...@gmail.com) wrote:

“As a parser developer, I want to build, test, and maintain my parser
extension as a single deliverable ‘package’”.
I think from the 3rd party developer perspective, everything required to
make the parser work, should be packaged and installed with the parser.
The point is they may only be working with the parser code, and never touch
ambari or other ‘metron’ source.  Having essential components
managed otherwise makes it harder for the developers.

I know this pov is true for my teams.


On April 27, 2017 at 10:15:55, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Otto,

Happy to help around this.

Couple of questions and things to maybe think about…

REST seems fine for sending the payload, but the unpacking job should
probably be async (triggered by the job, but not responsible for replying,
i.e. the initial call gets an ACCEPTED response immediately before the
heavy lifting is done.

I’m still not 100% sold on the idea of ES templates being included in the
bundle, and would prefer that to be a broader “register the schema for the
parser” somewhere from which the Metron framework can generate templates,
this allows us to account for things like enrichments in templates, but the
principle you describe holds if we make it non-ES specific (some people use
solr for instance).

If we do this as a REST call, we can easily incorporate it to the UI, or
maybe add an ambari view later, so I’d say the REST approach is definitely
the right first step.

One other thing to think about is whether this belongs in the metron-rest
package (personally i think it probably does) or whether it is a separate
micro-service.

Anyway, happy to help in any way I can on this.

Simon


> On 27 Apr 2017, at 15:09, Otto Fowler <ottobackwa...@gmail.com> wrote:
>
> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwa...@gmail.com)
wrote:
>
> So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out,
including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers. This will be the
> effort to actually install 3rd party parser and other extensions ( stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this. We would use the current services, or orchestrate them to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser - unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way? Should we do an ambari view instead?
> * Is this too much to do in a rest call? Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.

Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)

[Otto Fowler]

“As a parser developer, I want to build, test, and maintain my parser
extension as a single deliverable ‘package’”.
I think from the 3rd party developer perspective, everything required to
make the parser work, should be packaged and installed with the parser.
The point is they may only be working with the parser code, and never touch
ambari or other ‘metron’ source.  Having essential components
managed otherwise makes it harder for the developers.

I know this pov is true for my teams.


On April 27, 2017 at 10:15:55, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Otto,

Happy to help around this.

Couple of questions and things to maybe think about…

REST seems fine for sending the payload, but the unpacking job should
probably be async (triggered by the job, but not responsible for replying,
i.e. the initial call gets an ACCEPTED response immediately before the
heavy lifting is done.

I’m still not 100% sold on the idea of ES templates being included in the
bundle, and would prefer that to be a broader “register the schema for the
parser” somewhere from which the Metron framework can generate templates,
this allows us to account for things like enrichments in templates, but the
principle you describe holds if we make it non-ES specific (some people use
solr for instance).

If we do this as a REST call, we can easily incorporate it to the UI, or
maybe add an ambari view later, so I’d say the REST approach is definitely
the right first step.

One other thing to think about is whether this belongs in the metron-rest
package (personally i think it probably does) or whether it is a separate
micro-service.

Anyway, happy to help in any way I can on this.

Simon


> On 27 Apr 2017, at 15:09, Otto Fowler <ottobackwa...@gmail.com> wrote:
>
> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwa...@gmail.com)
wrote:
>
> So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out,
including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers. This will be the
> effort to actually install 3rd party parser and other extensions (
stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this. We would use the current services, or orchestrate them
to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser -
unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way? Should we do an ambari view instead?
> * Is this too much to do in a rest call? Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.

Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)

[Otto Fowler]

Also, if you want to help that would be great ;)


On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwa...@gmail.com) wrote:

So,  assuming ( I know I know ) that METRON-777 eventually lands, it will
have lain the framework for the extension system for parsers out, including
the capability to create parsers outside the metron tree.

The next step is METRON-258, side loading of parsers.  This will be the
effort to actually install 3rd party parser and other extensions ( stellar
libs for example ) into metron.

My first inclination is to add new capabilities to the REST services to
accomplish this.  We would use the current services, or orchestrate them to
accomplish this.

The ‘process’ required would be the following:

I’ll pseudo code it out:

[rest endpoint]
installExtension(FILE extensionTgz, string extensionType)

workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)

deployExtensionBundleToHdfs(workingDir)

if(parser == extenstionType)
pushConfigurationToZK(workingDir)
pushESTemplate(workingDir) *possibly depending on 777 review
setupLogRotate(workingDIr) * possibly depending on 777 review
saveExtensionTgzSomewhere(extensionTgz)


Then, the configuration ui would be extended to front the new api.

* there is still a question of how we get a parser to the ambari
configuration, such that when starting parsers it starts a parser - unless
that happens and I don’t see it


I would like some feedback on this approach.

* Is rest the right way?  Should we do an ambari view instead?
* Is this too much to do in a rest call?  Will it timeout etc?
???


Any ideas would be appreciated.

[DISCUSS] Metron Rest to Install Parser (METRON-258)

[Otto Fowler]

So,  assuming ( I know I know ) that METRON-777 eventually lands, it will
have lain the framework for the extension system for parsers out, including
the capability to create parsers outside the metron tree.

The next step is METRON-258, side loading of parsers.  This will be the
effort to actually install 3rd party parser and other extensions ( stellar
libs for example ) into metron.

My first inclination is to add new capabilities to the REST services to
accomplish this.  We would use the current services, or orchestrate them to
accomplish this.

The ‘process’ required would be the following:

I’ll pseudo code it out:

[rest endpoint]
installExtension(FILE extensionTgz, string extensionType)

workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)

deployExtensionBundleToHdfs(workingDir)

if(parser == extenstionType)
pushConfigurationToZK(workingDir)
pushESTemplate(workingDir) *possibly depending on 777 review
setupLogRotate(workingDIr) * possibly depending on 777 review
saveExtensionTgzSomewhere(extensionTgz)


Then, the configuration ui would be extended to front the new api.

* there is still a question of how we get a parser to the ambari
configuration, such that when starting parsers it starts a parser - unless
that happens and I don’t see it


I would like some feedback on this approach.

* Is rest the right way?  Should we do an ambari view instead?
* Is this too much to do in a rest call?  Will it timeout etc?
???


Any ideas would be appreciated.

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

Thanks Nick.  I don’t think we catch exceptions at all from the execute
calls.  I wonder what the best practice is?
I can’t seem to find the ambari mpack programming documentation ;)


On April 26, 2017 at 16:50:18, Nick Allen (n...@nickallen.org) wrote:

I can create the JIRA.  I capture the logs.

On Wed, Apr 26, 2017 at 4:48 PM, Nick Allen <n...@nickallen.org> wrote:

> Yes, Otto.  I just experienced that myself.  It is a bug that we should
> create a JIRA for.
>
> I was able to work around it by just using "start" instead of "restart".
>
> On Wed, Apr 26, 2017 at 3:41 PM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
>> Ok, I did ifconfig, i mean ip a and got the right values
>> that and the brackets and it looks like es is up.
>>
>> Now my next problem is prob. a bug.
>>
>> Ambari thinks it needs to restart ‘metron’.
>> the enrichment restart crashes because we don’t catch the storm exception
>> that you are stopping a topology that was not running.
>>
>> The question is - is the state wrong _and_ the exception catch or not.
>>
>> I am guessing I should start a Jira for that one?
>>
>>
>>
>> On April 26, 2017 at 15:31:05, Otto Fowler (ottobackwa...@gmail.com)
>> wrote:
>>
>> So now I’m getting errors for ETH0
>> I think that is one of the things that has to change on centos7?
>> Can you check what you have?
>>
>>
>>
>> On April 26, 2017 at 15:20:18, Nick Allen (n...@nickallen.org) wrote:
>>
>> Yes, I am also running on CentOS 7.
>>
>> On Wed, Apr 26, 2017 at 3:19 PM, Otto Fowler <ottobackwa...@gmail.com>
>> wrote:
>>
>>> Would i have to change that on Centos 7?
>>>
>>>
>>> On April 26, 2017 at 15:12:38, Nick Allen (n...@nickallen.org) wrote:
>>>
>>> Check on the `network_hosts` setting. I just ran into the same issue. I
>>> had to add brackets around it otherwise the file was not valid YAML and
>>> it
>>> would crash trying to load the config file.
>>>
>>> network.host: ["_lo:ipv4_","_eth0:ipv4_"]
>>>
>>> On Wed, Apr 26, 2017 at 3:09 PM, Otto Fowler <ottobackwa...@gmail.com>
>>> wrote:
>>>
>>> > I think it is literally crashing trying to load the config file
>>> >
>>> >
>>> >
>>> > On April 26, 2017 at 15:00:25, Michael Miklavcic (
>>> > michael.miklav...@gmail.com) wrote:
>>> >
>>> > A couple more things you should check, given this setup:
>>> >
>>> > - 1 master node (not as data node)
>>> > - 2 data nodes
>>> >
>>> > I'd use something like this:
>>> >
>>> > - gateway_recover_after_data_nodes=1 or 2
>>> > - index_number_of_replicas=0 or 1 (2 data nodes, with 2 total copies of
>>> > the index - if you set this value to 2, it would need 1 more data node
>>> > to
>>> > attain 1 orig index + 2 replicas)
>>> >
>>> >
>>> >
>>> > On Wed, Apr 26, 2017 at 12:02 PM, Otto Fowler <ottobackwa...@gmail.com
>>> >
>>> > wrote:
>>> >
>>> > > Actually never mind. I am not sure how to read the exception. Is it
>>> > > failing to read the configuration?
>>> > >
>>> > >
>>> > > On April 26, 2017 at 14:01:26, Otto Fowler (ottobackwa...@gmail.com)
>>> > > wrote:
>>> > >
>>> > > So the elasticsearch service is disabled. That is the issue. When ES
>>> > > calls get status ( service elasticsearch status ) it fails.
>>> > > I don’t know systemd too well, but does that mean we didn’t do the
>>> equiv
>>> > > of chkconfig on?
>>> > >
>>> > >
>>> > >
>>> > > On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com)
>>> > > wrote:
>>> > >
>>> > > Getting the same error in ES as before from what I can see
>>> > >
>>> > >
>>> > > On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com)
>>> > > wrote:
>>> > >
>>> > > I still can’t keep ES running, it dies.
>>> > > Also, I’m getting exceptions from enrichment trying to restart when
>>> the
>>> > > topology wasn’t running.
>>> > >
>>> > > Is there some coordination in this work? Should I just log jiras?
>>> &

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

I can’t reproduce now, and the log rolled, so I’m not going to log it until
I see it again.



On April 26, 2017 at 15:47:48, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

I thought we had addressed that at some point, so if it's blowing up still
on restart when parsers are down, then I would file a Jira.

On Wed, Apr 26, 2017 at 1:41 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Ok, I did ifconfig, i mean ip a and got the right values
> that and the brackets and it looks like es is up.
>
> Now my next problem is prob. a bug.
>
> Ambari thinks it needs to restart ‘metron’.
> the enrichment restart crashes because we don’t catch the storm exception
> that you are stopping a topology that was not running.
>
> The question is - is the state wrong _and_ the exception catch or not.
>
> I am guessing I should start a Jira for that one?
>
>
>
> On April 26, 2017 at 15:31:05, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> So now I’m getting errors for ETH0
> I think that is one of the things that has to change on centos7?
> Can you check what you have?
>
>
>
> On April 26, 2017 at 15:20:18, Nick Allen (n...@nickallen.org) wrote:
>
> Yes, I am also running on CentOS 7.
>
> On Wed, Apr 26, 2017 at 3:19 PM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > Would i have to change that on Centos 7?
> >
> >
> > On April 26, 2017 at 15:12:38, Nick Allen (n...@nickallen.org) wrote:
> >
> > Check on the `network_hosts` setting. I just ran into the same issue. I
> > had to add brackets around it otherwise the file was not valid YAML and
> it
> > would crash trying to load the config file.
> >
> > network.host: ["_lo:ipv4_","_eth0:ipv4_"]
> >
> > On Wed, Apr 26, 2017 at 3:09 PM, Otto Fowler <ottobackwa...@gmail.com>
> > wrote:
> >
> > > I think it is literally crashing trying to load the config file
> > >
> > >
> > >
> > > On April 26, 2017 at 15:00:25, Michael Miklavcic (
> > > michael.miklav...@gmail.com) wrote:
> > >
> > > A couple more things you should check, given this setup:
> > >
> > > - 1 master node (not as data node)
> > > - 2 data nodes
> > >
> > > I'd use something like this:
> > >
> > > - gateway_recover_after_data_nodes=1 or 2
> > > - index_number_of_replicas=0 or 1 (2 data nodes, with 2 total copies
of
> > > the index - if you set this value to 2, it would need 1 more data
node
> > > to
> > > attain 1 orig index + 2 replicas)
> > >
> > >
> > >
> > > On Wed, Apr 26, 2017 at 12:02 PM, Otto Fowler <ottobackwa...@gmail.com
> >
> > > wrote:
> > >
> > > > Actually never mind. I am not sure how to read the exception. Is it
> > > > failing to read the configuration?
> > > >
> > > >
> > > > On April 26, 2017 at 14:01:26, Otto Fowler (ottobackwa...@gmail.com)

> > > > wrote:
> > > >
> > > > So the elasticsearch service is disabled. That is the issue. When
ES
> > > > calls get status ( service elasticsearch status ) it fails.
> > > > I don’t know systemd too well, but does that mean we didn’t do the
> > equiv
> > > > of chkconfig on?
> > > >
> > > >
> > > >
> > > > On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com)

> > > > wrote:
> > > >
> > > > Getting the same error in ES as before from what I can see
> > > >
> > > >
> > > > On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com)

> > > > wrote:
> > > >
> > > > I still can’t keep ES running, it dies.
> > > > Also, I’m getting exceptions from enrichment trying to restart when
> the
> > > > topology wasn’t running.
> > > >
> > > > Is there some coordination in this work? Should I just log jiras?
> > > > If someone is already sussing out ES, we can sync up
> > > >
> > > >
> > > >
> > > > On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com)
> > > wrote:
> > > >
> > > > I'm also interested to know why that's important at such a small
> scale.
> > > >
> > > > Jon
> > > >
> > > > On Wed, Apr 26, 2017, 10:51 AM Otto Fowler <ottobackwa...@gmail.com>

> > > > wrote:
> > > >
> > > > > I am following
> > > > >
> > >

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

Would i have to change that on Centos 7?


On April 26, 2017 at 15:12:38, Nick Allen (n...@nickallen.org) wrote:

Check on the `network_hosts` setting. I just ran into the same issue. I
had to add brackets around it otherwise the file was not valid YAML and it
would crash trying to load the config file.

network.host: ["_lo:ipv4_","_eth0:ipv4_"]

On Wed, Apr 26, 2017 at 3:09 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> I think it is literally crashing trying to load the config file
>
>
>
> On April 26, 2017 at 15:00:25, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> A couple more things you should check, given this setup:
>
> - 1 master node (not as data node)
> - 2 data nodes
>
> I'd use something like this:
>
> - gateway_recover_after_data_nodes=1 or 2
> - index_number_of_replicas=0 or 1 (2 data nodes, with 2 total copies of
> the index - if you set this value to 2, it would need 1 more data node
> to
> attain 1 orig index + 2 replicas)
>
>
>
> On Wed, Apr 26, 2017 at 12:02 PM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > Actually never mind. I am not sure how to read the exception. Is it
> > failing to read the configuration?
> >
> >
> > On April 26, 2017 at 14:01:26, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > So the elasticsearch service is disabled. That is the issue. When ES
> > calls get status ( service elasticsearch status ) it fails.
> > I don’t know systemd too well, but does that mean we didn’t do the
equiv
> > of chkconfig on?
> >
> >
> >
> > On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > Getting the same error in ES as before from what I can see
> >
> >
> > On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > I still can’t keep ES running, it dies.
> > Also, I’m getting exceptions from enrichment trying to restart when the
> > topology wasn’t running.
> >
> > Is there some coordination in this work? Should I just log jiras?
> > If someone is already sussing out ES, we can sync up
> >
> >
> >
> > On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
> >
> > I'm also interested to know why that's important at such a small scale.
> >
> > Jon
> >
> > On Wed, Apr 26, 2017, 10:51 AM Otto Fowler <ottobackwa...@gmail.com>
> > wrote:
> >
> > > I am following
> > >
> > > https://community.hortonworks.com/articles/60805/deploying-
> > a-fresh-metron-cluster-using-ambari-serv.html
> > > I DO have master and data node together.
> > > Why is that a problem?
> > >
> > > I will try again with master and data node separate.
> > >
> > >
> > > On April 26, 2017 at 10:41:59, Michael Miklavcic (
> > > michael.miklav...@gmail.com) wrote:
> > >
> > > Hey Otto,
> > >
> > > How do you have the ES nodes configured? For a base install I would
> > setup 1
> > > master (NOT as data node) and 2 data nodes (NOT on the same node as
the
> > > master). This is the install configuration I got working. You can
also
> > > modify some configuration properties around master node as data node,
> > index
> > > replicas, and gateway recovery to get it working differently, but
this
> is
> > > what will work OOTB with the default config settings from the mpack.
If
> > > you've already setup a master node and a data node on the same host,
> > we'll
> > > need to re-install.
> > >
> > > Mike
> > >
> > >
> > > On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler <ottobackwa...@gmail.com>

> > > wrote:
> > >
> > > > File
> > > > "/usr/lib/python2.6/site-packages/resource_management/
> core/shell.py",
> > > line
> > > > 293, in _call
> > > >
> > > > raise ExecutionFailed(err_msg, code, out, err)
> > > >
> > > > ExecutionFailed: Execution of 'service elasticsearch status'
returned
> > 3.
> > > ●
> > > > elasticsearch.service - Elasticsearch
> > > >
> > > > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > > > disabled;
> > > > vendor preset: disabled)
> > > >
> > > > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58
EDT
> > > > <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>; 10h ago
> > > >
>

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

Actually never mind.  I am not sure how to read the exception. Is it
failing to read the configuration?


On April 26, 2017 at 14:01:26, Otto Fowler (ottobackwa...@gmail.com) wrote:

So the elasticsearch service is disabled.  That is the issue.  When ES
calls get status ( service elasticsearch status ) it fails.
I don’t know systemd too well, but does that mean we didn’t do the equiv of
chkconfig on?



On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com) wrote:

Getting the same error in ES as before from what I can see


On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com) wrote:

I still can’t keep ES running, it dies.
Also, I’m getting exceptions from enrichment trying to restart when the
topology wasn’t running.

Is there some coordination in this work?  Should I just log jiras?
If someone is already sussing out ES, we can sync up



On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com) wrote:

I'm also interested to know why that's important at such a small scale.

Jon

On Wed, Apr 26, 2017, 10:51 AM Otto Fowler <ottobackwa...@gmail.com> wrote:

> I am following
>
>
https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
> I DO have master and data node together.
> Why is that a problem?
>
> I will try again with master and data node separate.
>
>
> On April 26, 2017 at 10:41:59, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> How do you have the ES nodes configured? For a base install I would setup
1
> master (NOT as data node) and 2 data nodes (NOT on the same node as the
> master). This is the install configuration I got working. You can also
> modify some configuration properties around master node as data node,
index
> replicas, and gateway recovery to get it working differently, but this is
> what will work OOTB with the default config settings from the mpack. If
> you've already setup a master node and a data node on the same host, we'll
> need to re-install.
>
> Mike
>
>
> On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > File
> > "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line
> > 293, in _call
> >
> > raise ExecutionFailed(err_msg, code, out, err)
> >
> > ExecutionFailed: Execution of 'service elasticsearch status' returned 3.
> ●
> > elasticsearch.service - Elasticsearch
> >
> > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > disabled;
> > vendor preset: disabled)
> >
> > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> > <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>; 10h ago
> >
> > Docs: http://www.elastic.co
> >
> > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> > -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> > HOME}
> > -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> > -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
> >
> > Process: 16819
> > ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> > (code=exited, status=0/SUCCESS)
> >
> > Main PID: 16821 (code=exited, status=1/FAILURE)
> >
> >
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromStream(Settings.java:1080)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromPath(Settings.java:1067)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.node.internal.InternalSettingsPreparer.
> > prepareEnvironment(InternalSettingsPreparer.java:88)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> >
org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20E

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

So the elasticsearch service is disabled.  That is the issue.  When ES
calls get status ( service elasticsearch status ) it fails.
I don’t know systemd too well, but does that mean we didn’t do the equiv of
chkconfig on?



On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com) wrote:

Getting the same error in ES as before from what I can see


On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com) wrote:

I still can’t keep ES running, it dies.
Also, I’m getting exceptions from enrichment trying to restart when the
topology wasn’t running.

Is there some coordination in this work?  Should I just log jiras?
If someone is already sussing out ES, we can sync up



On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com) wrote:

I'm also interested to know why that's important at such a small scale.

Jon

On Wed, Apr 26, 2017, 10:51 AM Otto Fowler <ottobackwa...@gmail.com> wrote:

> I am following
>
>
https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
> I DO have master and data node together.
> Why is that a problem?
>
> I will try again with master and data node separate.
>
>
> On April 26, 2017 at 10:41:59, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> How do you have the ES nodes configured? For a base install I would setup
1
> master (NOT as data node) and 2 data nodes (NOT on the same node as the
> master). This is the install configuration I got working. You can also
> modify some configuration properties around master node as data node,
index
> replicas, and gateway recovery to get it working differently, but this is
> what will work OOTB with the default config settings from the mpack. If
> you've already setup a master node and a data node on the same host, we'll
> need to re-install.
>
> Mike
>
>
> On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > File
> > "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line
> > 293, in _call
> >
> > raise ExecutionFailed(err_msg, code, out, err)
> >
> > ExecutionFailed: Execution of 'service elasticsearch status' returned 3.
> ●
> > elasticsearch.service - Elasticsearch
> >
> > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > disabled;
> > vendor preset: disabled)
> >
> > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> > <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>; 10h ago
> >
> > Docs: http://www.elastic.co
> >
> > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> > -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> > HOME}
> > -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> > -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
> >
> > Process: 16819
> > ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> > (code=exited, status=0/SUCCESS)
> >
> > Main PID: 16821 (code=exited, status=1/FAILURE)
> >
> >
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromStream(Settings.java:1080)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromPath(Settings.java:1067)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.node.internal.InternalSettingsPreparer.
> > prepareEnvironment(InternalSettingsPreparer.java:88)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> >
org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> >
>

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

Getting the same error in ES as before from what I can see


On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com) wrote:

I still can’t keep ES running, it dies.
Also, I’m getting exceptions from enrichment trying to restart when the
topology wasn’t running.

Is there some coordination in this work?  Should I just log jiras?
If someone is already sussing out ES, we can sync up



On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com) wrote:

I'm also interested to know why that's important at such a small scale.

Jon

On Wed, Apr 26, 2017, 10:51 AM Otto Fowler <ottobackwa...@gmail.com> wrote:

> I am following
>
>
https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
> I DO have master and data node together.
> Why is that a problem?
>
> I will try again with master and data node separate.
>
>
> On April 26, 2017 at 10:41:59, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> How do you have the ES nodes configured? For a base install I would setup
1
> master (NOT as data node) and 2 data nodes (NOT on the same node as the
> master). This is the install configuration I got working. You can also
> modify some configuration properties around master node as data node,
index
> replicas, and gateway recovery to get it working differently, but this is
> what will work OOTB with the default config settings from the mpack. If
> you've already setup a master node and a data node on the same host, we'll
> need to re-install.
>
> Mike
>
>
> On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > File
> > "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line
> > 293, in _call
> >
> > raise ExecutionFailed(err_msg, code, out, err)
> >
> > ExecutionFailed: Execution of 'service elasticsearch status' returned 3.
> ●
> > elasticsearch.service - Elasticsearch
> >
> > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > disabled;
> > vendor preset: disabled)
> >
> > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> > <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>; 10h ago
> >
> > Docs: http://www.elastic.co
> >
> > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> > -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> > HOME}
> > -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> > -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
> >
> > Process: 16819
> > ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> > (code=exited, status=0/SUCCESS)
> >
> > Main PID: 16821 (code=exited, status=1/FAILURE)
> >
> >
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromStream(Settings.java:1080)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromPath(Settings.java:1067)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.node.internal.InternalSettingsPreparer.
> > prepareEnvironment(InternalSettingsPreparer.java:88)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> >
org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: Refer
> > to
> > the log for complete error details.
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%2

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

I still can’t keep ES running, it dies.
Also, I’m getting exceptions from enrichment trying to restart when the
topology wasn’t running.

Is there some coordination in this work?  Should I just log jiras?
If someone is already sussing out ES, we can sync up



On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com) wrote:

I'm also interested to know why that's important at such a small scale.

Jon

On Wed, Apr 26, 2017, 10:51 AM Otto Fowler <ottobackwa...@gmail.com> wrote:

> I am following
>
>
https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
> I DO have master and data node together.
> Why is that a problem?
>
> I will try again with master and data node separate.
>
>
> On April 26, 2017 at 10:41:59, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> How do you have the ES nodes configured? For a base install I would setup
1
> master (NOT as data node) and 2 data nodes (NOT on the same node as the
> master). This is the install configuration I got working. You can also
> modify some configuration properties around master node as data node,
index
> replicas, and gateway recovery to get it working differently, but this is
> what will work OOTB with the default config settings from the mpack. If
> you've already setup a master node and a data node on the same host,
we'll
> need to re-install.
>
> Mike
>
>
> On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > File
> > "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line
> > 293, in _call
> >
> > raise ExecutionFailed(err_msg, code, out, err)
> >
> > ExecutionFailed: Execution of 'service elasticsearch status' returned
3.
> ●
> > elasticsearch.service - Elasticsearch
> >
> > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > disabled;
> > vendor preset: disabled)
> >
> > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> > <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>; 10h ago
> >
> > Docs: http://www.elastic.co
> >
> > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> > -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> > HOME}
> > -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> > -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
> >
> > Process: 16819
> >
ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> > (code=exited, status=0/SUCCESS)
> >
> > Main PID: 16821 (code=exited, status=1/FAILURE)
> >
> >
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromStream(Settings.java:1080)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromPath(Settings.java:1067)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.node.internal.InternalSettingsPreparer.
> > prepareEnvironment(InternalSettingsPreparer.java:88)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> >
org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]:
Refer
> > to
> > the log for complete error details.
> >
> > Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> > elasticsearch.service: main process exited, code

Re: Reliable place to get latest release info for script

[Otto Fowler]

No,  Take a look at METRON-777 PR.
What I want is an extension mechanism, where you can:

1. create a project from a metron maven archetype
2. produce an extension ( parser, enrichment, stellar lib, ?? )
3. install that extension through the management UI and configure

All without touching the metron code.



On April 26, 2017 at 12:38:28, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Regarding parsers and enrichment, new java based topologies for bluecoat,
msexchange, msserver, asa, etc.

Batching, we use NiFi + Kylo for both stream and batch, but batch data send
to java based tool that passes data through parsers/enrichments classes and
stores in Hive.

So what you suggesting is to have internal repo with all our changes alone
and to merge it with Apache Metron whenever we need to get full build?

- Dima

On 04/26/2017 09:09 PM, Otto Fowler wrote:

Thanks!

Parsers, and Enrichments, we have a plan or idea about. When you say
extends batch processing… can you say what tech/component/part of stack you
extended?

New topologies?  New WriterBolts? etc?

I would like as complete a picture as possible of the things for which
anyone would say:
“If I could write this outside the metron tree, i could not have to
maintain a private company fork”




On April 26, 2017 at 11:22:21, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Otto,

Yes, we developed custom parsers and enrichments (in future we seek to open
source them, but have no time to up them to the decent level currently). We
also merged our custom tools that extends Metron batching processing and
store data in HDFS and Hive to be used by tableau, zeppelin, etc.

Is that answers your question? Let me know if you want to know more.

- Dima

On 04/26/2017 08:29 PM, Otto Fowler wrote:

Can I ask Dima, as comfortable you are in describing, what areas do you
make changes or additions to that require you to maintain a fork?

Custom parsers?  Enrichments? Other?

I have done some work, and have more planned and lined up to try to
eliminate the requirement to develop in the metron tree to extend the
product, and would be interested in your cases.


On April 26, 2017 at 10:46:03, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Hello,

We want to merge latest metron release branch with out develpoment every
time it is get released. What is the path we should take here in order
to identify what release happened successfully?

Latest release is 0.3.1 currently, but since you're preparing for 0.4.0
(which will take a while) following links already have 0.4.0:
https://github.com/apache/incubator-metron/blob/master/pom.xml
https://github.com/apache/incubator-metron/releases

The only place I have found with 0.3.1 (current latest release) is:
https://metron.apache.org/documentation/#releases
But there is no way for a script to figure what commit or branch to pick
up from github for merging.

Can you please suggest?
Thank you.

- Dima

Re: Quick Dev - Atlas Images

[Otto Fowler]

I know there is a lot of anti-ansible sentiment.  But having gone
spelunking through the MPack scripts and the metron.spec to more -777 let
me just say I missed ansible’s flexibility, and documentation.


On April 26, 2017 at 12:12:08, Nick Allen (n...@nickallen.org) wrote:

> I think you can have vagrant use docker as a back end too right?

I don't know about using Docker as a backend for Vagrant.  I don't think
Vagrant is our major sticking point.  I think Ansible is the problem.  We
have a lot of deployment functionality still in Ansible.  Much of that has
been moved to Ambari which helps a ton, but we have a fair amount left
AFAIK.

> If it is docker, then it is just the Dockerfiles.

Agreed, the storage size would be lighter weight with Docker.  But there
are a lot of other comparisons to make when thinking about Docker too. Many
of which I don't know the answers to right now.

   - Would Docker offer a more "production-like" environment? In that each
   component can run on isolated components?
   - Can we avoid some of the resource constraints that we currently have
   in running single-node Metron?
   - Can we avoid re-writing the entire deployment process?
   - How does the "time to start" compare for a "canned" release image as
   Dave suggested?







On Tue, Apr 25, 2017 at 4:09 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> If it is docker, then it is just the Dockerfiles.
> I think you can have vagrant use docker as a back end too right?
>
>
>
> On April 25, 2017 at 14:34:14, Nick Allen (n...@nickallen.org) wrote:
>
> >> I hadn't really reasoned about the notion of a "released" Quick Dev
> image,
> but I can see a lot of value in having a versioned sandbox type image- but
> maybe not quick dev, maybe not even Vagrant? We could actually pre-package
> everything needed and save some provisioning time on released versions.
>
> I really like the idea. I think it would be very beneficial to have a
> single pre-packaged image for each release that users can download and take
> new features for a spin.
>
> If we do stick with Vagrant for this I think Atlas works just fine. Who
> else is going to host a 5.1 GB image for us? :) Although I am very open to
> alternative implementations of this idea.
>
>
> >> I always thought of Quick Dev as a developer tool, so our obligation is
> to make it work with the current master and any branches currently used by
> devs.
>
> Would be interested to get other opinions on this. I am good with making
> that assumption. Whatever the community agrees on for Quick Dev though,
> we should document it as such. Right now, I think it would be reasonable
> to assume that a user would download a release and expect to be able to
> launch Quick Dev.
>
>
>
>
>
>
>
>
>
> On Mon, Apr 24, 2017 at 11:51 AM, David Lyle <dlyle65...@gmail.com> wrote:
>
> > I think it's a really good idea. There is some complexity:
> >
> > a) Image releases do not map 1:1 with Metron releases and Atlas doesn't
> > allow -SNAPSHOT in their release number scheme. That is, we'll have
> > different versions of the image that work with a 0.4.1-SNAPSHOT Metron
> and
> > some released versions of Metron that won't require a new image (A guy's
> > gotta believe). I think that can be easily worked around.
> >
> > b) If the Quick Dev image becomes one of our release artifacts, Atlas is
> > likely the wrong place to host it.
> >
> > I always thought of Quick Dev as a developer tool, so our obligation is
> to
> > make it work with the current master and any branches currently used by
> > devs. I hadn't really reasoned about the notion of a "released" Quick Dev
> > image, but I can see a lot of value in having a versioned sandbox type
> > image- but maybe not quick dev, maybe not even Vagrant? We could actually
> > pre-package everything needed and save some provisioning time on released
> > versions. It'd just come up ready to go. I think, should we want one, we
> > should release it as a convenience binary signed and hosted alongside the
> > other release artifacts. Meantime, we could keep the incremental versions
> > of Quick Dev in Atlas.
> >
> > Anyway, I think it's a really interesting notion.
> >
> > -D...
> >
> >
> > On Mon, Apr 24, 2017 at 11:26 AM, Nick Allen <n...@nickallen.org> wrote:
> >
> > > Right now, we have the images that get pushed to Atlas for Quick Dev
> > > <https://atlas.hashicorp.com/metron/boxes/quick_dev> versioned
> > > independently from the rest of Metron. We currently have versions 0.1.0
> > > and 0.2.0.
> > >
> > > What happen

Re: Reliable place to get latest release info for script

[Otto Fowler]

Thanks!

Parsers, and Enrichments, we have a plan or idea about. When you say
extends batch processing… can you say what tech/component/part of stack you
extended?

New topologies?  New WriterBolts? etc?

I would like as complete a picture as possible of the things for which
anyone would say:
“If I could write this outside the metron tree, i could not have to
maintain a private company fork”




On April 26, 2017 at 11:22:21, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Otto,

Yes, we developed custom parsers and enrichments (in future we seek to open
source them, but have no time to up them to the decent level currently). We
also merged our custom tools that extends Metron batching processing and
store data in HDFS and Hive to be used by tableau, zeppelin, etc.

Is that answers your question? Let me know if you want to know more.

- Dima

On 04/26/2017 08:29 PM, Otto Fowler wrote:

Can I ask Dima, as comfortable you are in describing, what areas do you
make changes or additions to that require you to maintain a fork?

Custom parsers?  Enrichments? Other?

I have done some work, and have more planned and lined up to try to
eliminate the requirement to develop in the metron tree to extend the
product, and would be interested in your cases.


On April 26, 2017 at 10:46:03, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Hello,

We want to merge latest metron release branch with out develpoment every
time it is get released. What is the path we should take here in order
to identify what release happened successfully?

Latest release is 0.3.1 currently, but since you're preparing for 0.4.0
(which will take a while) following links already have 0.4.0:
https://github.com/apache/incubator-metron/blob/master/pom.xml
https://github.com/apache/incubator-metron/releases

The only place I have found with 0.3.1 (current latest release) is:
https://metron.apache.org/documentation/#releases
But there is no way for a script to figure what commit or branch to pick
up from github for merging.

Can you please suggest?
Thank you.

- Dima

Re: Reliable place to get latest release info for script

[Otto Fowler]

Can I ask Dima, as comfortable you are in describing, what areas do you
make changes or additions to that require you to maintain a fork?

Custom parsers?  Enrichments? Other?

I have done some work, and have more planned and lined up to try to
eliminate the requirement to develop in the metron tree to extend the
product, and would be interested in your cases.


On April 26, 2017 at 10:46:03, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Hello,

We want to merge latest metron release branch with out develpoment every
time it is get released. What is the path we should take here in order
to identify what release happened successfully?

Latest release is 0.3.1 currently, but since you're preparing for 0.4.0
(which will take a while) following links already have 0.4.0:
https://github.com/apache/incubator-metron/blob/master/pom.xml
https://github.com/apache/incubator-metron/releases

The only place I have found with 0.3.1 (current latest release) is:
https://metron.apache.org/documentation/#releases
But there is no way for a script to figure what commit or branch to pick
up from github for merging.

Can you please suggest?
Thank you.

- Dima

Re: auto-install on bare metal

[Otto Fowler]

Well, let’s be sure what Dima’s looking for.

That site is most def. what I’m looking for however.


On April 26, 2017 at 10:29:36, Nick Allen (n...@nickallen.org) wrote:

> But that still requires to pre-install Ambari first, right?

No. Just like what happens when deploying "Full Dev", Ansible will install
Ambari.

On Wed, Apr 26, 2017 at 10:25 AM, Dima Kovalyov <dima.koval...@sstech.us>
wrote:

> But that still requires to pre-install Ambari first, right?
>
> - Dima
>
> On 04/26/2017 07:54 PM, Nick Allen wrote:
> > Ok, then I must have totally misunderstood what you're looking for.
> Sorry.
> >
> > On Wed, Apr 26, 2017 at 9:59 AM, Otto Fowler <ottobackwa...@gmail.com>
> > wrote:
> >
> >> Right, I think this : https://cwiki.apache.org/
> confluence/pages/viewpage.
> >> action?pageId=65144361 is the flow,
> >> but I need to verify it post recent changes to allow building in
docker
> >> again.
> >>
> >>
> >> On April 26, 2017 at 09:54:26, Nick Allen (n...@nickallen.org) wrote:
> >>
> >> Here is an example of how you might do that. I created this quite a
> while
> >> ago, but it shows you the structure and how you could manage multiple
> >> environments with this method.
> >>
> >> https://github.com/nickwallen/metron-environments
> >>
> >> On Tue, Apr 25, 2017 at 9:46 PM, Otto Fowler <ottobackwa...@gmail.com>
> >> wrote:
> >>
> >>> I failed at this today, but maybe it was the way I tried.
> >>> An example would be great.
> >>>
> >>>
> >>>
> >>> On April 25, 2017 at 20:11:26, David Lyle (dlyle65...@gmail.com)
> wrote:
> >>>
> >>> Hi Dima,
> >>>
> >>> The same Ansible playbooks that work for EC2 and Vagrant will work
for
> >> bare
> >>> metal installations. The only difference is that you would need to
> >>> pre-provision your machines and hand-build your inventory file. The
AWS
> >>> playbooks only provision the machines. All deployment of Metron is
> >> handled
> >>> (for all installation types) by the metron_full_install playbook [1].
> >>>
> >>> -D...
> >>>
> >>> [1]
> >>> https://github.com/apache/incubator-metron/blob/master/
> >>> metron-deployment/playbooks/metron_full_install.yml
> >>>
> >>> On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov <
> dima.koval...@sstech.us>
> >>> wrote:
> >>>
> >>>> Hello Metron Team,
> >>>>
> >>>> We have developed a script that performs auto-install of the Metron
on
> >>>> bare metal machines, but still working on few issues here and there.
> >>>>
> >>>> I am curios as to what automate solutions we do have for Metron
> >>>> installation right now?
> >>>> The ones I am aware of are in
> >>>> https://github.com/apache/incubator-metron/tree/master/
> >>> metron-deployment/:
> >>>
> >>>> a) AWS Ansible install (1 or 10 nodes)
> >>>> b) Vagrant local VM setup
> >>>>
> >>>> Is there any other solution available? Has anyone managed to use AWS
> >>>> Ansible playbooks for bare metal installation?
> >>>>
> >>>> - Dima
> >>>>
> >>>>
> >>>>
> >>
>
>

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

I am following
https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
I DO have master and data node together.
Why is that a problem?

I will try again with master and data node separate.


On April 26, 2017 at 10:41:59, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Hey Otto,

How do you have the ES nodes configured? For a base install I would setup 1
master (NOT as data node) and 2 data nodes (NOT on the same node as the
master). This is the install configuration I got working. You can also
modify some configuration properties around master node as data node, index
replicas, and gateway recovery to get it working differently, but this is
what will work OOTB with the default config settings from the mpack. If
you've already setup a master node and a data node on the same host, we'll
need to re-install.

Mike


On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> File
> "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line
> 293, in _call
>
> raise ExecutionFailed(err_msg, code, out, err)
>
> ExecutionFailed: Execution of 'service elasticsearch status' returned 3.
●
> elasticsearch.service - Elasticsearch
>
> Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> disabled;
> vendor preset: disabled)
>
> Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>; 10h ago
>
> Docs: http://www.elastic.co
>
> Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> HOME}
> -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
>
> Process: 16819
> ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> (code=exited, status=0/SUCCESS)
>
> Main PID: 16821 (code=exited, status=1/FAILURE)
>
>
>
> Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.common.settings.Settings$Builder.
> loadFromStream(Settings.java:1080)
>
> Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.common.settings.Settings$Builder.
> loadFromPath(Settings.java:1067)
>
> Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.node.internal.InternalSettingsPreparer.
> prepareEnvironment(InternalSettingsPreparer.java:88)
>
> Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
>
> Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
>
> Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
>
> Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: Refer
> to
> the log for complete error details.
>
> Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> elasticsearch.service: main process exited, code=exited, status=1/FAILURE
>
> Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]: Unit
> elasticsearch.service entered failed state.
>
> Apr 25 22:14:58 <http://airmail.calendar/2017-04-25%2022:14:58%20EDT>
> ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> elasticsearch.service failed.
>
> INFO 2017-04-26 08:55:19,962
> <http://airmail.calendar/2017-04-26%2008:55:19%20EDT> Contr
>
>
> On April 26, 2017 at 08:27:27, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Can you describe what ES issues you are working on?
> Fixing the repos got everything installed, but my ES components don’t
stay
> running.
> I need to harvest the errors.
>
>
>
> On April 25, 2017 at 16:46:00, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Nm. sorry. I fixed it.
>
>
>
> On April 25, 2017 a

Re: auto-install on bare metal

[Otto Fowler]

Right, I think this :
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361
is the flow,
but I need to verify it post recent changes to allow building in docker
again.


On April 26, 2017 at 09:54:26, Nick Allen (n...@nickallen.org) wrote:

Here is an example of how you might do that. I created this quite a while
ago, but it shows you the structure and how you could manage multiple
environments with this method.

https://github.com/nickwallen/metron-environments

On Tue, Apr 25, 2017 at 9:46 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> I failed at this today, but maybe it was the way I tried.
> An example would be great.
>
>
>
> On April 25, 2017 at 20:11:26, David Lyle (dlyle65...@gmail.com) wrote:
>
> Hi Dima,
>
> The same Ansible playbooks that work for EC2 and Vagrant will work for
bare
> metal installations. The only difference is that you would need to
> pre-provision your machines and hand-build your inventory file. The AWS
> playbooks only provision the machines. All deployment of Metron is
handled
> (for all installation types) by the metron_full_install playbook [1].
>
> -D...
>
> [1]
> https://github.com/apache/incubator-metron/blob/master/
> metron-deployment/playbooks/metron_full_install.yml
>
> On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov <dima.koval...@sstech.us>
> wrote:
>
> > Hello Metron Team,
> >
> > We have developed a script that performs auto-install of the Metron on
> > bare metal machines, but still working on few issues here and there.
> >
> > I am curios as to what automate solutions we do have for Metron
> > installation right now?
> > The ones I am aware of are in
> > https://github.com/apache/incubator-metron/tree/master/
> metron-deployment/:
>
> > a) AWS Ansible install (1 or 10 nodes)
> > b) Vagrant local VM setup
> >
> > Is there any other solution available? Has anyone managed to use AWS
> > Ansible playbooks for bare metal installation?
> >
> > - Dima
> >
> >
> >
>

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

Can you describe what ES issues you are working on?
Fixing the repos got everything installed, but my ES components don’t stay
running.
I need to harvest the errors.



On April 25, 2017 at 16:46:00, Otto Fowler (ottobackwa...@gmail.com) wrote:

Nm.  sorry.  I fixed it.



On April 25, 2017 at 16:42:05, Otto Fowler (ottobackwa...@gmail.com) wrote:

Ok, now I see the repos in the ‘pick version’ screen, but it is erring on
the f://localrepo
even though the folder exists, there is no repodata/repomd.xml.

What is the command to create a local repo?



On April 25, 2017 at 16:05:17, Otto Fowler (ottobackwa...@gmail.com) wrote:

I was going by the HW community page.

Ok, Let me try it



On April 25, 2017 at 16:04:07, David Lyle (dlyle65...@gmail.com) wrote:

That would do it. It requires 2.4.2+. I would have sworn I put that in the
README, but I must have only annotated the PR. :(

I'll get that in asap.

-D...


On Tue, Apr 25, 2017 at 3:44 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> 2.4.1 as a matter of fact
>
>
> On April 25, 2017 at 15:29:43, David Lyle (dlyle65...@gmail.com) wrote:
>
> Any chance they're running Ambari < 2.4.2?
>
> -D...
>
>
> On Tue, Apr 25, 2017 at 3:23 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Hey Otto, I don't have wizard screenshots in front of me right now to
> say
> > for sure, but I do see a repoinfo.xml in the mpack. I haven't run into
> > anything like that, but next time I run through the install I can take
> > another look.
> >
> > https://github.com/apache/incubator-metron/blob/master/
> > metron-deployment/packaging/ambari/metron-mpack/src/main/
> > resources/addon-services/ELASTICSEARCH/2.3.3/repos/repoinfo.xml
> >
> >
> > On Tue, Apr 25, 2017 at 12:16 PM, Otto Fowler <ottobackwa...@gmail.com>
> > wrote:
> >
> > > stderr:
> > >
> > > Traceback (most recent call last):
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 71, in 
> > >
> > > Elasticsearch().execute()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 280, in execute
> > >
> > > method(env)
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 32, in install
> > >
> > > self.install_packages(env)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 567, in install_packages
> > >
> > > retry_count=agent_stack_retry_count)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/base.py",
> > > line 155, in __init__
> > >
> > > self.env.run()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 160, in run
> > >
> > > self.run_action(resource, action)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 124, in run_action
> > >
> > > provider_action()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 54, in action_install
> > >
> > > self.install_package(package_name, self.resource.use_repos,
> > > self.resource.skip_repos)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/yumrpm.py", line 49, in install_package
> > >
> > > self.checked_call_with_retries(cmd, sudo=True,
> > > logoutput=self.get_logoutput())
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 83, in
> > checked_call_with_retries
> > >
> > > return self._call_with_retries(cmd, is_checked=True, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 91, in _call_with_retries
> > >
> > > code, out = func(cmd, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 71, in inner
> > >
> > > result = function(command, **kwar

Re: [DISCUSS] Regression introduced in Full Dev

[Otto Fowler]

So I have a fix, vagrant full installs for me, with the rpm’s in
/localrepo.  I do have issues
post deploy cluster, but I think they are resource related.

How should I go about this?  What Jira would I post the PR against?  The
same as the original?



On April 26, 2017 at 07:39:07, Justin Leet (justinjl...@gmail.com) wrote:

I have it (and had it yesterday). Subject is: "[DISCUSS] The various
methods and incantations to spin up Metron".

On Wed, Apr 26, 2017 at 7:33 AM, zeo...@gmail.com <zeo...@gmail.com> wrote:

> Yeah, I don't see the other thread either. Stuck in the outbox Casey?
>
> Jon
>
> On Wed, Apr 26, 2017, 6:53 AM Otto Fowler <ottobackwa...@gmail.com>
wrote:
>
> > What other thread?
> >
> >
> > On April 25, 2017 at 19:56:56, Casey Stella (ceste...@gmail.com) wrote:
> >
> > Ok, I spun up that discussion in another thread. Hopefully we can get
> some
> > better sense about the various ways to spin up metron and a centralized
> > place to direct people to along with with guidance on when some
approach
> > would be better than another.
> >
> > I'll be honest, I've totally lost track and never really consider
> anything
> > outside of full-dev anymore since it's the one that is generally stable
> > (quick-dev gets out of date quickly because mpack changes cause it to
get
> > stale) and is sufficient for validating PRs. Most of the other ones
tend
> > to either not have all of the system spun up (i.e. the hadoop
components)
> > and therefore end up with me having to test in full-dev anyway or just
> > weren't apparent to me and have unknown pros and cons. ;)
> >
> > On Tue, Apr 25, 2017 at 7:21 PM, Casey Stella <ceste...@gmail.com>
> wrote:
> >
> > > Yeah, I tend to agree that a rundown of the various methods and when
> you
> > > would use them is in order. I will say that full-dev is especially
> > > important to have working since it is required for validating PRs.
> > >
> > > On Tue, Apr 25, 2017 at 18:56 zeo...@gmail.com <zeo...@gmail.com>
> wrote:
> > >
> > >> Can somebody map out all of the current methods and procedures to
spin
> > up
> > >> Metron components? I swear I find out about new ones every month.
> > >> Metron-docker, the 4 vagrants, rpm-docker, ansible-docker, any
others?
> > >> Perhaps an agreed upon write up of when to use what would be
helpful.
> > >>
> > >> Jon
> > >>
> > >> On Tue, Apr 25, 2017, 6:17 PM Ryan Merriman <merrim...@gmail.com>
> > wrote:
> > >>
> > >> > A regression was introduced recently that breaks full dev. I've
> > >> narrowed
> > >> > down the commit that introduced it and have submitted a PR to
revert
> > >> that
> > >> > commit: https://github.com/apache/incubator-metron/pull/549.
> > >> >
> > >> > Given there has been confusion recently over our deployment build
> > >> process,
> > >> > I think it's appropriate that we discuss and come to a consensus
and
> > on
> > >> > how this should work.
> > >> >
> > >> > Ryan
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> >
> --
>
> Jon
>

Re: [DISCUSS] Regression introduced in Full Dev

[Otto Fowler]

http://stackoverflow.com/questions/22150209/maven-changes-the-order-of-plugins-of-different-profiles
https://issues.apache.org/jira/browse/MNG-2258


On April 25, 2017 at 22:33:58, Otto Fowler (ottobackwa...@gmail.com) wrote:

So, the copy resources configuration is still there, and not governed by
the profile.
Is it possible that since they both run in package, that they are out of
order?


On April 25, 2017 at 22:31:27, Otto Fowler (ottobackwa...@gmail.com) wrote:

Having said that, i see the problem.
I am very sorry


How do I go about resubmitting if I can fix it?



On April 25, 2017 at 22:29:56, Otto Fowler (ottobackwa...@gmail.com) wrote:

OK, I’ll take a look.  Sorry - I thought they read out of the RPMS dir and
verified that they were produced both ways.
I am not sure how my change would change the target copy.


On April 25, 2017 at 22:25:04, Justin Leet (justinjl...@gmail.com) wrote:

The problem seems to be that full/quick dev rely on having the RPMs in the
/target folder

>From metron-deployment/roles/metron-rpms/defaults:
metron_rpm_glob: "{{ playbook_dir
}}/../packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm"

However, that PR appears to no longer copy to the /target folder run as
expected, so quick and full dev are unable to find the RPMs.

On Tue, Apr 25, 2017 at 9:26 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Ryan, did you see why it was not working?
>
>
> On April 25, 2017 at 18:17:29, Ryan Merriman (merrim...@gmail.com) wrote:
>
> A regression was introduced recently that breaks full dev. I've narrowed
> down the commit that introduced it and have submitted a PR to revert that
> commit: https://github.com/apache/incubator-metron/pull/549.
>
> Given there has been confusion recently over our deployment build process,
> I think it's appropriate that we discuss and come to a consensus and on
> how this should work.
>
> Ryan
>

Re: [DISCUSS] Regression introduced in Full Dev

[Otto Fowler]

So, the copy resources configuration is still there, and not governed by
the profile.
Is it possible that since they both run in package, that they are out of
order?


On April 25, 2017 at 22:31:27, Otto Fowler (ottobackwa...@gmail.com) wrote:

Having said that, i see the problem.
I am very sorry


How do I go about resubmitting if I can fix it?



On April 25, 2017 at 22:29:56, Otto Fowler (ottobackwa...@gmail.com) wrote:

OK, I’ll take a look.  Sorry - I thought they read out of the RPMS dir and
verified that they were produced both ways.
I am not sure how my change would change the target copy.


On April 25, 2017 at 22:25:04, Justin Leet (justinjl...@gmail.com) wrote:

The problem seems to be that full/quick dev rely on having the RPMs in the
/target folder

>From metron-deployment/roles/metron-rpms/defaults:
metron_rpm_glob: "{{ playbook_dir
}}/../packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm"

However, that PR appears to no longer copy to the /target folder run as
expected, so quick and full dev are unable to find the RPMs.

On Tue, Apr 25, 2017 at 9:26 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Ryan, did you see why it was not working?
>
>
> On April 25, 2017 at 18:17:29, Ryan Merriman (merrim...@gmail.com) wrote:
>
> A regression was introduced recently that breaks full dev. I've narrowed
> down the commit that introduced it and have submitted a PR to revert that
> commit: https://github.com/apache/incubator-metron/pull/549.
>
> Given there has been confusion recently over our deployment build process,
> I think it's appropriate that we discuss and come to a consensus and on
> how this should work.
>
> Ryan
>

Re: [DISCUSS] Regression introduced in Full Dev

[Otto Fowler]

Having said that, i see the problem.
I am very sorry


How do I go about resubmitting if I can fix it?



On April 25, 2017 at 22:29:56, Otto Fowler (ottobackwa...@gmail.com) wrote:

OK, I’ll take a look.  Sorry - I thought they read out of the RPMS dir and
verified that they were produced both ways.
I am not sure how my change would change the target copy.


On April 25, 2017 at 22:25:04, Justin Leet (justinjl...@gmail.com) wrote:

The problem seems to be that full/quick dev rely on having the RPMs in the
/target folder

>From metron-deployment/roles/metron-rpms/defaults:
metron_rpm_glob: "{{ playbook_dir
}}/../packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm"

However, that PR appears to no longer copy to the /target folder run as
expected, so quick and full dev are unable to find the RPMs.

On Tue, Apr 25, 2017 at 9:26 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Ryan, did you see why it was not working?
>
>
> On April 25, 2017 at 18:17:29, Ryan Merriman (merrim...@gmail.com) wrote:
>
> A regression was introduced recently that breaks full dev. I've narrowed
> down the commit that introduced it and have submitted a PR to revert that
> commit: https://github.com/apache/incubator-metron/pull/549.
>
> Given there has been confusion recently over our deployment build process,
> I think it's appropriate that we discuss and come to a consensus and on
> how this should work.
>
> Ryan
>

Re: auto-install on bare metal

[Otto Fowler]

I failed at this today, but maybe it was the way I tried.
An example would be great.



On April 25, 2017 at 20:11:26, David Lyle (dlyle65...@gmail.com) wrote:

Hi Dima,

The same Ansible playbooks that work for EC2 and Vagrant will work for bare
metal installations. The only difference is that you would need to
pre-provision your machines and hand-build your inventory file. The AWS
playbooks only provision the machines. All deployment of Metron is handled
(for all installation types) by the metron_full_install playbook [1].

-D...

[1]
https://github.com/apache/incubator-metron/blob/master/metron-deployment/playbooks/metron_full_install.yml

On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov 
wrote:

> Hello Metron Team,
>
> We have developed a script that performs auto-install of the Metron on
> bare metal machines, but still working on few issues here and there.
>
> I am curios as to what automate solutions we do have for Metron
> installation right now?
> The ones I am aware of are in
> https://github.com/apache/incubator-metron/tree/master/metron-deployment/:

> a) AWS Ansible install (1 or 10 nodes)
> b) Vagrant local VM setup
>
> Is there any other solution available? Has anyone managed to use AWS
> Ansible playbooks for bare metal installation?
>
> - Dima
>
>
>

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

Nm.  sorry.  I fixed it.



On April 25, 2017 at 16:42:05, Otto Fowler (ottobackwa...@gmail.com) wrote:

Ok, now I see the repos in the ‘pick version’ screen, but it is erring on
the f://localrepo
even though the folder exists, there is no repodata/repomd.xml.

What is the command to create a local repo?



On April 25, 2017 at 16:05:17, Otto Fowler (ottobackwa...@gmail.com) wrote:

I was going by the HW community page.

Ok, Let me try it



On April 25, 2017 at 16:04:07, David Lyle (dlyle65...@gmail.com) wrote:

That would do it. It requires 2.4.2+. I would have sworn I put that in the
README, but I must have only annotated the PR. :(

I'll get that in asap.

-D...


On Tue, Apr 25, 2017 at 3:44 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> 2.4.1 as a matter of fact
>
>
> On April 25, 2017 at 15:29:43, David Lyle (dlyle65...@gmail.com) wrote:
>
> Any chance they're running Ambari < 2.4.2?
>
> -D...
>
>
> On Tue, Apr 25, 2017 at 3:23 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Hey Otto, I don't have wizard screenshots in front of me right now to
> say
> > for sure, but I do see a repoinfo.xml in the mpack. I haven't run into
> > anything like that, but next time I run through the install I can take
> > another look.
> >
> > https://github.com/apache/incubator-metron/blob/master/
> > metron-deployment/packaging/ambari/metron-mpack/src/main/
> > resources/addon-services/ELASTICSEARCH/2.3.3/repos/repoinfo.xml
> >
> >
> > On Tue, Apr 25, 2017 at 12:16 PM, Otto Fowler <ottobackwa...@gmail.com>
> > wrote:
> >
> > > stderr:
> > >
> > > Traceback (most recent call last):
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 71, in 
> > >
> > > Elasticsearch().execute()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 280, in execute
> > >
> > > method(env)
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 32, in install
> > >
> > > self.install_packages(env)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 567, in install_packages
> > >
> > > retry_count=agent_stack_retry_count)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/base.py",
> > > line 155, in __init__
> > >
> > > self.env.run()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 160, in run
> > >
> > > self.run_action(resource, action)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 124, in run_action
> > >
> > > provider_action()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 54, in action_install
> > >
> > > self.install_package(package_name, self.resource.use_repos,
> > > self.resource.skip_repos)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/yumrpm.py", line 49, in install_package
> > >
> > > self.checked_call_with_retries(cmd, sudo=True,
> > > logoutput=self.get_logoutput())
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 83, in
> > checked_call_with_retries
> > >
> > > return self._call_with_retries(cmd, is_checked=True, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 91, in _call_with_retries
> > >
> > > code, out = func(cmd, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 71, in inner
> > >
> > > result = function(command, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 93, in checked_call
> > >
> > > tries=tries, try_sleep=try_sleep)
> > &g

Re: Quick Dev - Atlas Images

[Otto Fowler]

If it is docker, then it is just the Dockerfiles.
I think you can have vagrant use docker as a back end too right?



On April 25, 2017 at 14:34:14, Nick Allen (n...@nickallen.org) wrote:

>> I hadn't really reasoned about the notion of a "released" Quick Dev
image,
but I can see a lot of value in having a versioned sandbox type image- but
maybe not quick dev, maybe not even Vagrant? We could actually pre-package
everything needed and save some provisioning time on released versions.

I really like the idea. I think it would be very beneficial to have a
single pre-packaged image for each release that users can download and take
new features for a spin.

If we do stick with Vagrant for this I think Atlas works just fine. Who
else is going to host a 5.1 GB image for us? :) Although I am very open to
alternative implementations of this idea.


>> I always thought of Quick Dev as a developer tool, so our obligation is
to make it work with the current master and any branches currently used by
devs.

Would be interested to get other opinions on this. I am good with making
that assumption. Whatever the community agrees on for Quick Dev though,
we should document it as such. Right now, I think it would be reasonable
to assume that a user would download a release and expect to be able to
launch Quick Dev.









On Mon, Apr 24, 2017 at 11:51 AM, David Lyle  wrote:

> I think it's a really good idea. There is some complexity:
>
> a) Image releases do not map 1:1 with Metron releases and Atlas doesn't
> allow -SNAPSHOT in their release number scheme. That is, we'll have
> different versions of the image that work with a 0.4.1-SNAPSHOT Metron
and
> some released versions of Metron that won't require a new image (A guy's
> gotta believe). I think that can be easily worked around.
>
> b) If the Quick Dev image becomes one of our release artifacts, Atlas is
> likely the wrong place to host it.
>
> I always thought of Quick Dev as a developer tool, so our obligation is
to
> make it work with the current master and any branches currently used by
> devs. I hadn't really reasoned about the notion of a "released" Quick Dev
> image, but I can see a lot of value in having a versioned sandbox type
> image- but maybe not quick dev, maybe not even Vagrant? We could actually
> pre-package everything needed and save some provisioning time on released
> versions. It'd just come up ready to go. I think, should we want one, we
> should release it as a convenience binary signed and hosted alongside the
> other release artifacts. Meantime, we could keep the incremental versions
> of Quick Dev in Atlas.
>
> Anyway, I think it's a really interesting notion.
>
> -D...
>
>
> On Mon, Apr 24, 2017 at 11:26 AM, Nick Allen  wrote:
>
> > Right now, we have the images that get pushed to Atlas for Quick Dev
> >  versioned
> > independently from the rest of Metron. We currently have versions 0.1.0
> > and 0.2.0.
> >
> > What happens when a user downloads an official release of Metron, like
> > 0.3.1, and attempts to run Quick Dev? I would assume that the code
would
> > download the latest image version, which we may have been updated since
> the
> > release. This would cause it to fail for the release version. Am I
> wrong?
> >
> > If we had the Atlas images follow Metron's versioning scheme, would
this
> > solve the problem? Are there other cons of doing this?
> >
> > Thanks
> >
>

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

I was going by the HW community page.

Ok, Let me try it



On April 25, 2017 at 16:04:07, David Lyle (dlyle65...@gmail.com) wrote:

That would do it. It requires 2.4.2+. I would have sworn I put that in the
README, but I must have only annotated the PR. :(

I'll get that in asap.

-D...


On Tue, Apr 25, 2017 at 3:44 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> 2.4.1 as a matter of fact
>
>
> On April 25, 2017 at 15:29:43, David Lyle (dlyle65...@gmail.com) wrote:
>
> Any chance they're running Ambari < 2.4.2?
>
> -D...
>
>
> On Tue, Apr 25, 2017 at 3:23 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Hey Otto, I don't have wizard screenshots in front of me right now to
> say
> > for sure, but I do see a repoinfo.xml in the mpack. I haven't run into
> > anything like that, but next time I run through the install I can take
> > another look.
> >
> > https://github.com/apache/incubator-metron/blob/master/
> > metron-deployment/packaging/ambari/metron-mpack/src/main/
> > resources/addon-services/ELASTICSEARCH/2.3.3/repos/repoinfo.xml
> >
> >
> > On Tue, Apr 25, 2017 at 12:16 PM, Otto Fowler <ottobackwa...@gmail.com>
> > wrote:
> >
> > > stderr:
> > >
> > > Traceback (most recent call last):
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 71, in 
> > >
> > > Elasticsearch().execute()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 280, in execute
> > >
> > > method(env)
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 32, in install
> > >
> > > self.install_packages(env)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 567, in install_packages
> > >
> > > retry_count=agent_stack_retry_count)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/base.py",
> > > line 155, in __init__
> > >
> > > self.env.run()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 160, in run
> > >
> > > self.run_action(resource, action)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 124, in run_action
> > >
> > > provider_action()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 54, in action_install
> > >
> > > self.install_package(package_name, self.resource.use_repos,
> > > self.resource.skip_repos)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/yumrpm.py", line 49, in install_package
> > >
> > > self.checked_call_with_retries(cmd, sudo=True,
> > > logoutput=self.get_logoutput())
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 83, in
> > checked_call_with_retries
> > >
> > > return self._call_with_retries(cmd, is_checked=True, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 91, in _call_with_retries
> > >
> > > code, out = func(cmd, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 71, in inner
> > >
> > > result = function(command, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 93, in checked_call
> > >
> > > tries=tries, try_sleep=try_sleep)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 141, in _call_wrapper
> > >
> > > result = _call(command, **kwargs_copy)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 294, in _call
>

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

2.4.1 as a matter of fact


On April 25, 2017 at 15:29:43, David Lyle (dlyle65...@gmail.com) wrote:

Any chance they're running Ambari < 2.4.2?

-D...


On Tue, Apr 25, 2017 at 3:23 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Hey Otto, I don't have wizard screenshots in front of me right now to say
> for sure, but I do see a repoinfo.xml in the mpack. I haven't run into
> anything like that, but next time I run through the install I can take
> another look.
>
> https://github.com/apache/incubator-metron/blob/master/
> metron-deployment/packaging/ambari/metron-mpack/src/main/
> resources/addon-services/ELASTICSEARCH/2.3.3/repos/repoinfo.xml
>
>
> On Tue, Apr 25, 2017 at 12:16 PM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > stderr:
> >
> > Traceback (most recent call last):
> >
> > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > 2.3.3/package/scripts/elastic_slave.py", line 71, in 
> >
> > Elasticsearch().execute()
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> libraries/script/script.py",
> > line 280, in execute
> >
> > method(env)
> >
> > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > 2.3.3/package/scripts/elastic_slave.py", line 32, in install
> >
> > self.install_packages(env)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> libraries/script/script.py",
> > line 567, in install_packages
> >
> > retry_count=agent_stack_retry_count)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/base.py",
> > line 155, in __init__
> >
> > self.env.run()
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/environment.py",
> > line 160, in run
> >
> > self.run_action(resource, action)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/environment.py",
> > line 124, in run_action
> >
> > provider_action()
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/providers/package/__init__.py", line 54, in action_install
> >
> > self.install_package(package_name, self.resource.use_repos,
> > self.resource.skip_repos)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/providers/package/yumrpm.py", line 49, in install_package
> >
> > self.checked_call_with_retries(cmd, sudo=True,
> > logoutput=self.get_logoutput())
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/providers/package/__init__.py", line 83, in
> checked_call_with_retries
> >
> > return self._call_with_retries(cmd, is_checked=True, **kwargs)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/providers/package/__init__.py", line 91, in _call_with_retries
> >
> > code, out = func(cmd, **kwargs)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/shell.py",
> > line 71, in inner
> >
> > result = function(command, **kwargs)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/shell.py",
> > line 93, in checked_call
> >
> > tries=tries, try_sleep=try_sleep)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/shell.py",
> > line 141, in _call_wrapper
> >
> > result = _call(command, **kwargs_copy)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/shell.py",
> > line 294, in _call
> >
> > raise Fail(err_msg)
> >
> > resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d
> 0
> > -e 0 -y install elasticsearch-2.3.3' returned 1. Error: Nothing to do
> >
> > stdout:
> >
> > 2017-04-25 14:12:48,669
> > <http://airmail.calendar/2017-04-25%2014:12:48%20EDT> - Using hadoop
> conf
> > dir: /usr/hdp/current/hadoop-client/conf
> >
> > 2017-04-25 14:12:48,671
> > <http://airmail.calendar/2017-04-25%2014:12:48%20EDT> - Group['metron']
> {}
> >
> > 2017-04-25 14:12:48,673
> > <http://airmail.calendar/2017-04-25%2014:12:48%20EDT> - Adding group
> > Group['metron']
> >
> > 2017-04-25 14:12:48,740
> > <http://airmail.calendar/2017-04-25%2014:12:48%20EDT> - Group['livy']
{}
> >
> > 2017-04-25 14:12:48,741
> > <http://airmail.calenda

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

wnloadSource('
http://css-fx-met-poc-82:8080/resources//jdk-8u77-linux-x64.tar.gz'),
'not_if': 'test -f /var/lib/ambari-agent/tmp/jdk-8u77-linux-x64.tar.gz'}

2017-04-25 14:12:50,504
<http://airmail.calendar/2017-04-25%2014:12:50%20EDT> - Downloading the
file from http://css-fx-met-poc-82:8080/resources//jdk-8u77-linux-x64.tar.gz

2017-04-25 14:12:53,793
<http://airmail.calendar/2017-04-25%2014:12:53%20EDT> -
Directory['/usr/jdk64'] {}

2017-04-25 14:12:53,793
<http://airmail.calendar/2017-04-25%2014:12:53%20EDT> - Creating directory
Directory['/usr/jdk64'] since it doesn't exist.

2017-04-25 14:12:53,794
<http://airmail.calendar/2017-04-25%2014:12:53%20EDT> - Execute[('chmod',
'a+x', u'/usr/jdk64')] {'sudo': True}

2017-04-25 14:12:53,824
<http://airmail.calendar/2017-04-25%2014:12:53%20EDT> - Execute['cd
/var/lib/ambari-agent/tmp/jdk_tmp_1BRMsS && tar -xf
/var/lib/ambari-agent/tmp/jdk-8u77-linux-x64.tar.gz && ambari-sudo.sh cp
-rp /var/lib/ambari-agent/tmp/jdk_tmp_1BRMsS/* /usr/jdk64'] {}

2017-04-25 14:13:00,439
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> -
Directory['/var/lib/ambari-agent/tmp/jdk_tmp_1BRMsS'] {'action': ['delete']}

2017-04-25 14:13:00,440
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> - Removing directory
Directory['/var/lib/ambari-agent/tmp/jdk_tmp_1BRMsS'] and all its content

2017-04-25 14:13:00,620
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> -
File['/usr/jdk64/jdk1.8.0_77/bin/java'] {'mode': 0755, 'cd_access': 'a'}

2017-04-25 14:13:00,621
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> - Execute[('chmod',
'-R', '755', u'/usr/jdk64/jdk1.8.0_77')] {'sudo': True}

2017-04-25 14:13:00,682
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> - Initializing 2
repositories

2017-04-25 14:13:00,683
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> -
Repository['HDP-2.5'] {'base_url':
'http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.5.0',
'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template':
'[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list
%}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif
%}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP',
'mirror_list': None}

2017-04-25 14:13:00,709
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> -
File['/etc/yum.repos.d/HDP.repo'] {'content': InlineTemplate(...)}

2017-04-25 14:13:00,711
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> - Writing
File['/etc/yum.repos.d/HDP.repo'] because it doesn't exist

2017-04-25 14:13:00,711
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> -
Repository['HDP-UTILS-1.1.0.21'] {'base_url':
'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7',
'action': ['create'], 'components': [u'HDP-UTILS', 'main'],
'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list
%}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif
%}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS',
'mirror_list': None}

2017-04-25 14:13:00,717
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> -
File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': InlineTemplate(...)}

2017-04-25 14:13:00,717
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> - Writing
File['/etc/yum.repos.d/HDP-UTILS.repo'] because it doesn't exist

2017-04-25 14:13:00,718
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> - Package['unzip']
{'retry_on_repo_unavailability': False, 'retry_count': 5}

2017-04-25 14:13:00,846
<http://airmail.calendar/2017-04-25%2014:13:00%20EDT> - Installing package
unzip ('/usr/bin/yum -d 0 -e 0 -y install unzip')

2017-04-25 14:13:03,413
<http://airmail.calendar/2017-04-25%2014:13:03%20EDT> - Package['curl']
{'retry_on_repo_unavailability': False, 'retry_count': 5}

2017-04-25 14:13:03,429
<http://airmail.calendar/2017-04-25%2014:13:03%20EDT> - Skipping
installation of existing package curl

2017-04-25 14:13:03,430
<http://airmail.calendar/2017-04-25%2014:13:03%20EDT> -
Package['hdp-select'] {'retry_on_repo_unavailability': False,
'retry_count': 5}

2017-04-25 14:13:03,451
<http://airmail.calendar/2017-04-25%2014:13:03%20EDT> - Installing package
hdp-select ('/usr/bin/yum -d 0 -e 0 -y install hdp-select')

2017-04-25 14:13:04,948
<http://airmail.calendar/2017-04-25%2014:13:04%20EDT> - Install ES Data Node

2017-04-25 14:13:04,952
<http://airmail.calendar/2017-04-25%2014:13:04%20EDT> -
Package['elasticsearch-2.3.3'] {'retry_on_repo_unavailability': False,
'retry_count': 5}

2017-04-25 14:13:05,086
<http://airmail.calendar/2017-04-25%2014:13:05%20EDT> - Installing package
elasticsearch-2.3.3 ('/usr/bin/yum -d 0 -e 0 -y install
elasticsearch-2.3.3')



Command failed after 1 tries





On April 25, 2017 at 13:47:17, Otto Fowler (ottobackwa...@gmail.com) wrote:

Yes, I’m going to run through again and get the logs.  The basic issue is,
no repo tab -> if I should have one,
and the ES yum command failing with nothing to do, which stops deployment
on all the nodes.



On April 25, 2017 at 13:36:25, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

I've been doing an internal hwx deployment (HCP) recently for testing
purposes. I have some ES tweaks to work out, but other than that it has
been running smoothly. Can you share any log info or screenshots?

Mike

On Apr 25, 2017 11:30 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote:

> Hi,
> Should I see the repo tab in the ambari wizard for metron at this point?
> Asking for a friend who can’t get the wizard to work :)
>

Re: Status of METRON-153

[Otto Fowler]

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361

This was broken when the rpm builds were introduced ( docker in docker no
no ).
This should work now ( i can indeed build the rpms etc and run ansible
inside of docker now with METRON-857 landed ) with
centos 6x ( although I have not tried it since my testing cluster is centos
7 right now ).




On April 25, 2017 at 13:30:02, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

We didn't come up with another approach Otto. I closed the PR at the time
bc I didn't think we wanted to break with Centos 6 yet, and my PR probably
would have done so. Can you elaborate on "prior capability wrt centos 6.x
work again" a bit?

On Apr 25, 2017 10:59 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote:

> Also, I can’t find any list discussion of the issue or other approaches.
>
>
> On April 25, 2017 at 12:48:46, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> What I am looking for is the prior capability wrt centos 6.x work again (
> which I think my
> pr fixing the docker in docker issue would do ), and have it extended to
> work with 7.x.
>
> I don’t think re-writing all the stuff that is in metron roles is very
> attractive.
>
>
> On April 25, 2017 at 12:10:47, David Lyle (dlyle65...@gmail.com) wrote:
>
> So, the current Ansible deployment actually does use blueprints. It
> constructs them from one of small_cluster.yml or single_node_vm.yml using
> the ambari_custer_state module.
>
> Ambari blueprints [1] and the Ambari Install Wizard are actually two
> separate things that are only related because both are available via
> Ambari.
>
> Currently, you should be able to create a blueprint and instantiate a
> cluster on Centos7 without using Ansible at all, though if you wished to,
> you could realize your blueprint in the same way that small_cluster and
> single_node_vm do, or alternatively create some Ansible roles to make the
> REST calls that Ambari requires to stand up a cluster.
>
> -D...
>
>
> [1] https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
>
> On Tue, Apr 25, 2017 at 11:35 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > Let me clarify, that this is support for automated blueprint deployment
> > with ansible, not the ambari wizard.
> >
> >
> >
> > On April 25, 2017 at 11:31:37, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
> >
> > Just tagging on here to indicate my interest in this - in order to have
> > someone other than me manage the OSs in my Metron cluster, I must run
on
> > RHEL 7. I assume that will be common across many enterprises.
> > Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a
> rough
> > go and I dropped it down on my priority list.
> >
> > Jon
> >
> > On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler <ottobackwa...@gmail.com>
> > wrote:
> >
> > > https://issues.apache.org/jira/browse/METRON-153
> > >
> > > What is the other approach that is mentioned here? Was it
implemented?
> > > The ansible build still does not support this and I was thinking of
> > looking
> > > at it.
> > >
> > --
> >
> > Jon
> >
>

Ambari Wizard: Repo Tab

[Otto Fowler]

Hi,
Should I see the repo tab in the ambari wizard for metron at this point?
Asking for a friend who can’t get the wizard to work :)

Re: Status of METRON-153

[Otto Fowler]

Also, I can’t find any list discussion of the issue or other approaches.


On April 25, 2017 at 12:48:46, Otto Fowler (ottobackwa...@gmail.com) wrote:

What I am looking for is the prior capability wrt centos 6.x work again (
which I think my
pr fixing the docker in docker issue would do ), and have it extended to
work with 7.x.

I don’t think re-writing all the stuff that is in metron roles is very
attractive.


On April 25, 2017 at 12:10:47, David Lyle (dlyle65...@gmail.com) wrote:

So, the current Ansible deployment actually does use blueprints. It
constructs them from one of small_cluster.yml or single_node_vm.yml using
the ambari_custer_state module.

Ambari blueprints [1] and the Ambari Install Wizard are actually two
separate things that are only related because both are available via
Ambari.

Currently, you should be able to create a blueprint and instantiate a
cluster on Centos7 without using Ansible at all, though if you wished to,
you could realize your blueprint in the same way that small_cluster and
single_node_vm do, or alternatively create some Ansible roles to make the
REST calls that Ambari requires to stand up a cluster.

-D...


[1] https://cwiki.apache.org/confluence/display/AMBARI/Blueprints

On Tue, Apr 25, 2017 at 11:35 AM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Let me clarify, that this is support for automated blueprint deployment
> with ansible, not the ambari wizard.
>
>
>
> On April 25, 2017 at 11:31:37, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> Just tagging on here to indicate my interest in this - in order to have
> someone other than me manage the OSs in my Metron cluster, I must run on
> RHEL 7. I assume that will be common across many enterprises.
> Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a rough
> go and I dropped it down on my priority list.
>
> Jon
>
> On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > https://issues.apache.org/jira/browse/METRON-153
> >
> > What is the other approach that is mentioned here? Was it implemented?
> > The ansible build still does not support this and I was thinking of
> looking
> > at it.
> >
> --
>
> Jon
>

Re: metron_full_install playbook with small_cluster and multiple hosts

[Otto Fowler]

I have a 5 node cluster ( with deploy machine/docker ).  I want used to be
able
to run the full install with a custom inventory for this cluster to deploy.

I am trying it again ( post docker fix ), but with centos 7.

EC2 doesn’t work for everyone.  unless you are saying I can use that to a
local cluster.



On April 25, 2017 at 12:32:17, David Lyle (dlyle65...@gmail.com) wrote:

Hi Otto,

I don't use a custom inventory that I create, but I do run an EC2 build
very frequently, most recently on Friday. That does use the small_cluster
file.

1. It's used because the Ansible Vagrant provisioner doesn't add localhost
(as it should) to the inventory, so [local] works around that. If you're
running under Vagrant, you'll want to set it to 127.0.0.1 or omit it
otherwise.
2. Are you sure? I used run_once and local_action to prevent that when I
originally did it. Through multiple distributed deployments, I only saw one
build being executed. I don't know what you mean by "set to become", Become
is set to false because I didn't want to sudo to execute the build.

-D...


On Tue, Apr 25, 2017 at 10:51 AM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Now that I have docker sorted with building the rpms, I’m trying to
deploy
> using a custom inventory and metron_full_install.
> So the question is, has anyone else tried to do this recently? I may be
> doing something wrong here with a couple of things.
>
> 1. what is the [local] host type in the inventory hosts file for? what
> should it be set to?
> 2. metron_build is set to become for ALL hosts, so it is trying to build
on
> all the hosts. That isn’t right, it should build locally not on the
hosts.
>

Re: Status of METRON-153

[Otto Fowler]

What I am looking for is the prior capability wrt centos 6.x work again (
which I think my
pr fixing the docker in docker issue would do ), and have it extended to
work with 7.x.

I don’t think re-writing all the stuff that is in metron roles is very
attractive.


On April 25, 2017 at 12:10:47, David Lyle (dlyle65...@gmail.com) wrote:

So, the current Ansible deployment actually does use blueprints. It
constructs them from one of small_cluster.yml or single_node_vm.yml using
the ambari_custer_state module.

Ambari blueprints [1] and the Ambari Install Wizard are actually two
separate things that are only related because both are available via
Ambari.

Currently, you should be able to create a blueprint and instantiate a
cluster on Centos7 without using Ansible at all, though if you wished to,
you could realize your blueprint in the same way that small_cluster and
single_node_vm do, or alternatively create some Ansible roles to make the
REST calls that Ambari requires to stand up a cluster.

-D...


[1] https://cwiki.apache.org/confluence/display/AMBARI/Blueprints

On Tue, Apr 25, 2017 at 11:35 AM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Let me clarify, that this is support for automated blueprint deployment
> with ansible, not the ambari wizard.
>
>
>
> On April 25, 2017 at 11:31:37, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> Just tagging on here to indicate my interest in this - in order to have
> someone other than me manage the OSs in my Metron cluster, I must run on
> RHEL 7. I assume that will be common across many enterprises.
> Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a
rough
> go and I dropped it down on my priority list.
>
> Jon
>
> On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > https://issues.apache.org/jira/browse/METRON-153
> >
> > What is the other approach that is mentioned here? Was it implemented?
> > The ansible build still does not support this and I was thinking of
> looking
> > at it.
> >
> --
>
> Jon
>

Re: Status of METRON-153

[Otto Fowler]

Let me clarify, that this is support for automated blueprint deployment
with ansible, not the ambari wizard.



On April 25, 2017 at 11:31:37, zeo...@gmail.com (zeo...@gmail.com) wrote:

Just tagging on here to indicate my interest in this - in order to have
someone other than me manage the OSs in my Metron cluster, I must run on
RHEL 7. I assume that will be common across many enterprises.
Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a rough
go and I dropped it down on my priority list.

Jon

On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler <ottobackwa...@gmail.com>
wrote:

> https://issues.apache.org/jira/browse/METRON-153
>
> What is the other approach that is mentioned here? Was it implemented?
> The ansible build still does not support this and I was thinking of
looking
> at it.
>
-- 

Jon

Status of METRON-153

[Otto Fowler]

https://issues.apache.org/jira/browse/METRON-153

What is the other approach that is mentioned here?  Was it implemented?
The ansible build still does not support this and I was thinking of looking
at it.

Re: Ansible Docker fails to build latest

[Otto Fowler]

OK,
the issue is the old old maven that is in the ansible image.
Changing it to 3.3.9 resolved.



On April 21, 2017 at 17:21:02, Otto Fowler (ottobackwa...@gmail.com) wrote:

Yes.   I am following the readme in there and starting the image, then
just running mvn package -DskipTests



On April 21, 2017 at 15:28:15, Casey Stella (ceste...@gmail.com) wrote:

Sorry, can you go through how you're getting to this error? I'm not super
familiar with this part of the stack..is this compiling Metron inside of a
docker image?

On Tue, Apr 18, 2017 at 1:45 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Is it something to do with the relocation of google.common that happens in
> the other shaded jars but not in the storm kafka client?
>
>
>
> On April 18, 2017 at 12:56:12, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> I’m trying to build latest in ansible docker, but I’m seeing the following
> error. Note that metron-common is a dependency in this module.
>
> [INFO] 
> 
> [INFO] Building metron-storm-kafka 0.4.0
> [INFO] 
> 
> Downloading: http://clojars.org/repo/org/apache/storm/storm-kafka-
> client/1.0.1.2.5.0.0-1245/storm-kafka-client-1.0.1.2.5.0.0-1245.pom
> Downloading: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm-kafka-client/1.0.1.2.5.0.0-1245/
> storm-kafka-client-1.0.1.2.5.0.0-1245.pom
> Downloaded: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm-kafka-client/1.0.1.2.5.0.0-1245/
> storm-kafka-client-1.0.1.2.5.0.0-1245.pom (4 KB at 31.5 KB/sec)
> Downloading: http://clojars.org/repo/org/apache/storm/storm/1.0.1.2.5.
> 0.0-1245/storm-1.0.1.2.5.0.0-1245.pom
> Downloading: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm/1.0.1.2.5.0.0-1245/storm-1.0.1.2.5.0.0-1245.pom
> Downloaded: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm/1.0.1.2.5.0.0-1245/storm-1.0.1.2.5.0.0-1245.pom
> (49 KB at 433.0 KB/sec)
> Downloading: http://clojars.org/repo/org/apache/kafka/kafka-clients/0.
> 10.0.2.5.0.0-1245/kafka-clients-0.10.0.2.5.0.0-1245.pom
> Downloading: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/kafka/kafka-clients/0.10.0.2.5.0.0-1245/
> kafka-clients-0.10.0.2.5.0.0-1245.pom
> Downloaded: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/kafka/kafka-clients/0.10.0.2.5.0.0-1245/
> kafka-clients-0.10.0.2.5.0.0-1245.pom (2 KB at 21.1 KB/sec)
> Downloading: http://clojars.org/repo/org/apache/storm/storm-kafka-
> client/1.0.1.2.5.0.0-1245/storm-kafka-client-1.0.1.2.5.0.0-1245.jar
> Downloading: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm-kafka-client/1.0.1.2.5.0.0-1245/
> storm-kafka-client-1.0.1.2.5.0.0-1245.jar
> Downloaded: http://repo.hortonworks.com/content/repositories/releases/
> org/apache/storm/storm-kafka-client/1.0.1.2.5.0.0-1245/
> storm-kafka-client-1.0.1.2.5.0.0-1245.jar (64 KB at 417.1 KB/sec)
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
> metron-storm-kafka ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] skip non existing resourceDirectory /root/incubator-metron/metron-
> platform/metron-storm-kafka/src/main/resources
> [INFO]
> [INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @
> metron-storm-kafka ---
> [INFO] Changes detected - recompiling the module!
> [INFO] Compiling 7 source files to /root/incubator-metron/metron-
> platform/metron-storm-kafka/target/classes
> /root/incubator-metron/metron-platform/metron-storm-kafka/
> src/main/java/org/apache/metron/storm/kafka/flux/
> SimpleStormKafkaBuilder.java:21: error: package com.google.common.base
> does not exist
> import com.google.common.base.Joiner;
>
> Has anyone tried the ansible docker image recently? Or seen this error?
>
>
>

Re: Reducing Warnings in Build

[Otto Fowler]

Until there is a use case supporting changing to other then the 80% plus
case, this should not be changed.
or a customer requirement etc.  This would need some design and discussion
time as well to map out all the
implications for the complete pipeline.

I think configurable charset should be separated from the cleanup.


On April 21, 2017 at 12:26:36, JJ Meyer (jjmey...@gmail.com) wrote:

Nick,

You're not off base at all. This is exactly the push back I wanted to hear.
I wasn't really sure if what I was proposing was the proper way of going
about it. My understanding of charsets is limited, and I've usually just
defaulted to UTF-8. That being said, right or wrong, my thought process for
including multiple configs was potential use of multiple charsets that may
be compatible (if that's possible?). But as I'm typing this out, I'm
probably over doing it. Especially since currently we usually take system
default, which a lot of times is UTF-8. So, you are right, it is probably
best to start with the path of least resistance and include one
configuration that is set to UTF-8. Then expand if we ever see a need to do
so.

Thanks,
JJ

On Fri, Apr 21, 2017 at 10:45 AM, Ryan Merriman 
wrote:

> I think Nick brings up some good points. Would there ever be a reason to
> not use UTF8 as the default from parsing a message on? All the tools we
> use for analytics work with UTF8 (am I wrong?).
>
> The only case I can see needing a configurable charset would be if a
> message coming from a sensor were encoded in a charset other than UTF8.
In
> that case there would need to be a configurable charset per parser (in
> parser config?) for decoding but the message could be encoded/decoded
with
> UTF8 after that. Or we could just make UTF encoding a prerequisite for
> sending messages to Metron.
>
> On Fri, Apr 21, 2017 at 10:32 AM, Nick Allen  wrote:
>
> > Per (2), I think it makes sense to make the charset configurable, but
> with
> > the proposal of 3 separate settings, wouldn't things blow up horribly
if
> > the Parsers are producing UTF-8, but Enrichment is expecting UTF-16?
> They
> > are not even speaking the same language, no?
> >
> > This makes me think that we need to understand the scenarios under
which
> a
> > user would want to change the charset, before we know what kinds of
> levers
> > to bake-in. What sort of use case would drive someone to change the
> > charset? Would there be a particular sensor producing telemetry with a
> > different charset from others? If one source of telemetry is different
> > than the others, would the entire system even work with varying
charsets?
> >
> > Without a good understanding of use cases, I think the only mildly safe
> > thing to do right now, is to have a single, global charset setting. The
> > user would have to ensure that their entire environment and all the
JVMs
> > driving it are set to use that charset.
> >
> > Perhaps my questioning comes from a lack of understanding of charsets.
I
> > can't remember ever having to deviate from the default. Please chime in
> > and educate me, if I am off base.
> >
> >
> >
> >
> >
> >
> > On Fri, Apr 21, 2017 at 8:50 AM, JJ Meyer  wrote:
> >
> > > Hello everybody,
> > >
> > > Currently our build has a significant amount of warnings (most from
the
> > > error prone plugin). A lot of this has been documented here:
> > > https://issues.apache.org/jira/browse/METRON-617
> > >
> > > I want to continue to work on this Jira. I really want to reduce the
> > number
> > > of warnings in our build. As the Jira points out, a lot of them are
> > > unchecked casts or the implicit use of default charset.
> > >
> > > When starting this, I want to start with a specific module. I plan on
> > > starting with `metron-interface` as it's fairly self contained and is
> > > pretty new. Below I want to layout what I plan on doing. Please let
me
> > know
> > > what you all think:
> > >
> > > 1. Suppress warnings where generics are not supported or checking
type
> is
> > > not possible.
> > > 2. For all unit tests that require supplying a charset I'll supply
the
> > > UTF-8 charset.
> > > 3. Update the API to have a charset configuration for each resource
> that
> > > needs one.
> > > 4. Remove @SuppressWarnings("ALL") on all unit tests. I found out
error
> > > prone doesn't support this level of suppression. Which is probably a
> good
> > > thing. We will need to explicitly state what we want to suppress
> instead.
> > >
> > > Two big questions came up right away when I was thinking about the
> above:
> > >
> > > 1. When suppressing warnings. I see we sometimes cast a JSONObject to
> > > Map. I know it extends Map, but is it really safe to
do
> > > something like the following? Should we have a utility that truly
> builds
> > a
> > > map that uses generics? I plan on doing some testing around this, but
> if
> > > anyone has any experience with this it would be greatly appreciated.
> 

Re: [GitHub] incubator-metron issue #538: METRON-868 Fix documentation on building RPMs

[Otto Fowler]

Maybe it is the relative parent path?


On April 20, 2017 at 20:05:31, mmiklavc (g...@git.apache.org) wrote:

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/538

Further, the reason that install worked is not because it needs the
artifacts in the cache, per se. It's because of something in the way our
multi-module build is structured that isn't allowing artifacts in the
metron-deployment project hierarchy to see projects in metron-platform. The
reason "package" would normally work on a typical build (without ever
having done an install) is because sibling modules are able to reference
the pom info of each other, and therefore the appropriate target
directories containing the standard build artifacts, e.g. jars.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Failing build

[Otto Fowler]

Maybe it is a container thing?  I don’t think my PR for extensions has the
problem,
or has had it in my travis - and it is a vm in travis.



On April 20, 2017 at 09:06:21, Casey Stella (ceste...@gmail.com) wrote:

oh man, yeah, it's pissing me off too. I'm still looking at ways to get
around Storm to not have its slots jam up at the end of tests. It's
particularly bad because it just doesn't seem to happen to me locally.
Also, while it does happen periodically on my personal travis, it's way
more infrequent . At this point, I'm going to try to spin the in memory
components up once and then reuse them across the parser integration tests,
which is mostly where I see the problem occurring. That would at least get
us closer to being able to run against external infrastructure (i.e. docker
images). It's just been slow going since it's a night and weekend project.

Anyway, if anyone else wants to tackle it, do chime in; I'd be willing to
give it up. ;)

On Wed, Apr 19, 2017 at 9:56 PM, zeo...@gmail.com  wrote:

> I was just looking at the commit history and noticed that out of the last
> commits that had travis runs, at least 7 of them failed (mine is still
> running). Given that we require a clean build on the branch prior to
> merging, that's a pretty unfortunate track record.
>
> Jon
>
> On Thu, Apr 6, 2017 at 1:31 PM Casey Stella  wrote:
>
> > Yeah, this is an intermittent test failure and has to do with the
> migration
> > to storm 1.0.3 and how it handles shutting down in local mode. The slot
> > worker refuses to shut down and freezes and therefore we wait forever.
I
> > thought I had it fixed by manually clobbering the slots, but alas it
> > appears that I only made the problem less frequent. When I found it as
> > part of METRON-793, I ran the fix 20 times locally as well as at least
10
> > times on my personal travis without repetition.
> >
> > Bottom line: we should correct it. I'll have to think a bit more about
> how
> > to fix it and if anyone else wants to take a crack at it, feel free. :)
> >
> > On Thu, Apr 6, 2017 at 1:26 PM, zeo...@gmail.com 
> wrote:
> >
> > > We appear to have a failed build again:
> > >
> > > No output has been received in the last 10m0s, this potentially
> > indicates a
> > > stalled build or something wrong with the build itself.
> > >
> > > https://travis-ci.org/apache/incubator-metron/builds/219261745
> > >
> > > Jon
> > > --
> > >
> > > Jon
> > >
> >
> --
>
> Jon
>