- Date range limits on queries
I will add a warning in the Job cleanup PR. That seems like an appropriate
place for it (ie. make sure you don't cause health issues in your cluster).
- UI should manage a queue/history of jobs
I can add some documentation around killing jobs manually with the
- Date range limits on queries
I took the point the wrong way apparently, sorry, I withdraw. I thought
you meant allow specifying a limit on the query, not the system imposing a
limit.
This should be documented with a warning or something
- UI should manage a queue/history of jobs
I was
Thanks for the feedback Otto. I have created a sub task for documenting
the Job cleanup documentation:
https://issues.apache.org/jira/browse/METRON-1737. I completely agree with
you there, this needs to be documented. For the others you marked "Follow
on" I will create follow on tasks in Jira.
Do you have any suggestions for what would make sense as a delimiter?
On 9 August 2018 at 05:57, Ali Nazemian wrote:
> Hi All,
>
> I was wondering if we can change the field separators in Metron to be able
> to make it Hive/ORC friendly. I could find the following PR, but neither
> dot nor
Yep, I'm wondering whether our parser interface should have the ability to
create schema either like that, or well, that, which would be helpful
within Metron as well.
@Otto, the one thing missing from the record reader api, is that if you
don't emit any records at all for a flow file, it errors,
Also, If we are doing the record readers, we can have a reader for a
parser type and explicitly set the schema, as seen here :
If we can do the record readers ourselves ( with the parsers inside them )
we can handle the returns.
I’ll be doing the net flow 5 readers once the net flow 5 processor PR ( not
mine ) is in.
I don’t think having a generic class loading parsers foo and having to
manage all that is preferable to
- Job cleanup/TTL
Documented at least, or a helper script to help yourself if you are in a
situation
- Expose the Query filter (vs Fixed) in the UI
Follow on
- Date range limits on queries
I don’t see how this won’t be immediately required. I would do this for
minimum viable.
- Pcap query
Maybe the edge use case will clarify the config issue a little. The reason
I would want to be able to push Metron parsers into NiFi would be so I can
pre-parse and filter on the edge to save bandwidth from remote locations. I
would expect to be able to parse at the edge and use NiFi to prioritise
