The ideal would be if your app could strip the newlines before writing to the logs. If that's not possible, you need a script that will tail the logs and parse the JSON (or at least parse it enough to find the beginning and ending curly braces) and strip out the new line characters.
Using morphlines to go from JSON to Avro is a good idea. -Joey On Sun, Dec 14, 2014 at 9:13 AM, Jean <[email protected]> wrote: > Ok, my app logs json event in local files and i also can pipe to nc or > logger so no avro. > > I'll check too if i can strip out new lines (with nc) > > Do you think tail files with flume (with morphline to parse json) is a good > way ? > > > > >> Le 12 déc. 2014 à 19:05, Joey Echeverria <[email protected]> a écrit : >> >> You'd either have to write a new source or use one of the RPC sources >> and send the JSON in a container. For example, you could use the Avro >> source to send just a string with the content of the JSON. You'll >> have to deserialize somewhere else in your flow. >> >> Alternatively, you could pre-process the JSON to strip out new lines >> before sending them to the netcat source. >> >> -Joey >> >>> On Fri, Dec 12, 2014 at 12:01 AM, jean garutti <[email protected]> wrote: >>> Hi, >>> I want flume to listen to JSON event send by netcat. >>> But the event is not correct because each new line result in a new flume >>> event. >>> >>> How can i send json event to flume by tcp/udp and have flume handle >>> correctly the json format ? >>> >>> thanks >> >> >> >> -- >> Joey Echeverria -- Joey Echeverria
