[jira] [Commented] (METRON-1650) Packaging docker containers are too large
[ https://issues.apache.org/jira/browse/METRON-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544296#comment-16544296 ] ASF GitHub Bot commented on METRON-1650: Github user jameslamb commented on the issue: https://github.com/apache/metron/pull/1091 Hey, awesome! Thanks to you and the rest of the metron community for your patience and support > Packaging docker containers are too large > - > > Key: METRON-1650 > URL: https://issues.apache.org/jira/browse/METRON-1650 > Project: Metron > Issue Type: Improvement >Reporter: James Lamb >Priority: Minor > > I have been looking through the project source code, and found that the > packaging docker containers are bigger than they need to be. > The containers' size could be cut substantially by: > * Grouping as many commands as possible into as few RUN stages as possible > * Taking advantage of `yum clean` and `apt-get clean` > * Removing artifacts pulled with `wget` > * Removing unused libraries or libraries only needed at build time (like > `tar` and `wget`) > I am creating this issue but will soon submit a PR to address it. > Thank you for considering my proposal! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user jameslamb commented on the issue: https://github.com/apache/metron/pull/1091 Hey, awesome! Thanks to you and the rest of the metron community for your patience and support ---
[jira] [Commented] (METRON-1650) Packaging docker containers are too large
[ https://issues.apache.org/jira/browse/METRON-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544294#comment-16544294 ] ASF GitHub Bot commented on METRON-1650: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 +1 from me. I was able to do the above, along with building metron from the instructions ansible-docker's readme.md. Thanks for sticking with it. > Packaging docker containers are too large > - > > Key: METRON-1650 > URL: https://issues.apache.org/jira/browse/METRON-1650 > Project: Metron > Issue Type: Improvement >Reporter: James Lamb >Priority: Minor > > I have been looking through the project source code, and found that the > packaging docker containers are bigger than they need to be. > The containers' size could be cut substantially by: > * Grouping as many commands as possible into as few RUN stages as possible > * Taking advantage of `yum clean` and `apt-get clean` > * Removing artifacts pulled with `wget` > * Removing unused libraries or libraries only needed at build time (like > `tar` and `wget`) > I am creating this issue but will soon submit a PR to address it. > Thank you for considering my proposal! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 +1 from me. I was able to do the above, along with building metron from the instructions ansible-docker's readme.md. Thanks for sticking with it. ---
[jira] [Commented] (METRON-1657) Parser aggregation in storm
[ https://issues.apache.org/jira/browse/METRON-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544263#comment-16544263 ] ASF GitHub Bot commented on METRON-1657: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 I have been on vacation, but will be reviewing Monday and Tuesday. Please do not commit > Parser aggregation in storm > --- > > Key: METRON-1657 > URL: https://issues.apache.org/jira/browse/METRON-1657 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Currently our parsing solution requires one storm topology per sensor. It has > been complained that this may be wasteful of resources and that, rather than > one storm topology per sensor, it would be advantageous to have multiple > sensors in the same topology. The benefit to this is that it would require > fewer storm slots. > The issue with this is that whenever we've aggregated functionality like this > before, we've run into issues appropriately being able to scale storm (e.g. > batch vs random access indexing in the same topology). The main point in > addressing this is to recommend that parsers with similar velocities and > complexity are grouped together. > Particularly for a first cut, leave the configuration mostly as-is, while > allowing for comma separated lists of sensors in start_parser_topology.sh > (e.g. bro,yaf creates a aggregated parser consisting of those two). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1099 I have been on vacation, but will be reviewing Monday and Tuesday. Please do not commit ---
[jira] [Commented] (METRON-1650) Packaging docker containers are too large
[ https://issues.apache.org/jira/browse/METRON-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544250#comment-16544250 ] ASF GitHub Bot commented on METRON-1650: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 Great! I will give this a try asap > Packaging docker containers are too large > - > > Key: METRON-1650 > URL: https://issues.apache.org/jira/browse/METRON-1650 > Project: Metron > Issue Type: Improvement >Reporter: James Lamb >Priority: Minor > > I have been looking through the project source code, and found that the > packaging docker containers are bigger than they need to be. > The containers' size could be cut substantially by: > * Grouping as many commands as possible into as few RUN stages as possible > * Taking advantage of `yum clean` and `apt-get clean` > * Removing artifacts pulled with `wget` > * Removing unused libraries or libraries only needed at build time (like > `tar` and `wget`) > I am creating this issue but will soon submit a PR to address it. > Thank you for considering my proposal! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1091 Great! I will give this a try asap ---
[jira] [Commented] (METRON-1650) Packaging docker containers are too large
[ https://issues.apache.org/jira/browse/METRON-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544247#comment-16544247 ] ASF GitHub Bot commented on METRON-1650: Github user jameslamb commented on the issue: https://github.com/apache/metron/pull/1091 @merrimanr @ottobackwards Hey I think I got this working! See my recent commit. I had accidentally smashed together two commands, which is why node stuff was failing. I was able to get successful builds locally by running the following (after installing Java 8, per recommendations above): ``` mvn clean install -DskipTests -Drat.skip=true cd metron-deployment/ mvn clean package -DskipTests -Pbuild-rpms mvn clean package -DskipTests -Pbuild-debs ``` > Packaging docker containers are too large > - > > Key: METRON-1650 > URL: https://issues.apache.org/jira/browse/METRON-1650 > Project: Metron > Issue Type: Improvement >Reporter: James Lamb >Priority: Minor > > I have been looking through the project source code, and found that the > packaging docker containers are bigger than they need to be. > The containers' size could be cut substantially by: > * Grouping as many commands as possible into as few RUN stages as possible > * Taking advantage of `yum clean` and `apt-get clean` > * Removing artifacts pulled with `wget` > * Removing unused libraries or libraries only needed at build time (like > `tar` and `wget`) > I am creating this issue but will soon submit a PR to address it. > Thank you for considering my proposal! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user jameslamb commented on the issue: https://github.com/apache/metron/pull/1091 @merrimanr @ottobackwards Hey I think I got this working! See my recent commit. I had accidentally smashed together two commands, which is why node stuff was failing. I was able to get successful builds locally by running the following (after installing Java 8, per recommendations above): ``` mvn clean install -DskipTests -Drat.skip=true cd metron-deployment/ mvn clean package -DskipTests -Pbuild-rpms mvn clean package -DskipTests -Pbuild-debs ``` ---
[jira] [Commented] (METRON-1650) Packaging docker containers are too large
[ https://issues.apache.org/jira/browse/METRON-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544237#comment-16544237 ] ASF GitHub Bot commented on METRON-1650: Github user jameslamb commented on the issue: https://github.com/apache/metron/pull/1091 @nickwallen Thank you so much!! I was able to get `master` building with your help and the advice of the other commenters. Now that I know I can build, I'll be able to test my changes. Will comment on here once I think I have something that works. > Packaging docker containers are too large > - > > Key: METRON-1650 > URL: https://issues.apache.org/jira/browse/METRON-1650 > Project: Metron > Issue Type: Improvement >Reporter: James Lamb >Priority: Minor > > I have been looking through the project source code, and found that the > packaging docker containers are bigger than they need to be. > The containers' size could be cut substantially by: > * Grouping as many commands as possible into as few RUN stages as possible > * Taking advantage of `yum clean` and `apt-get clean` > * Removing artifacts pulled with `wget` > * Removing unused libraries or libraries only needed at build time (like > `tar` and `wget`) > I am creating this issue but will soon submit a PR to address it. > Thank you for considering my proposal! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1091: METRON-1650: Cut size of packaging docker containers
Github user jameslamb commented on the issue: https://github.com/apache/metron/pull/1091 @nickwallen Thank you so much!! I was able to get `master` building with your help and the advice of the other commenters. Now that I know I can build, I'll be able to test my changes. Will comment on here once I think I have something that works. ---
