gdelt to mongo
gdeltmongof2e603c1-abcf-40c7-bc0c-003d7feb042bb9038bf9-0042-4f77-91e7-f4b0edbda7b20 MB0b9038bf9-0042-4f77-91e7-f4b0edbda7b258e1487b-10cd-48c8-bbad-9850908d198bPROCESSOR0 sec1matchedb9038bf9-0042-4f77-91e7-f4b0edbda7b203f21d9f-74d1-41d3-9067-dfa807af205ePROCESSOR04b19c2a7-abcf-4669-b64f-2de17e9de05eb9038bf9-0042-4f77-91e7-f4b0edbda7b20 MB0b9038bf9-0042-4f77-91e7-f4b0edbda7b2480877a4-f349-4ef7-9538-8e3e3e108e06PROCESSOR0 sec1unmatchedb9038bf9-0042-4f77-91e7-f4b0edbda7b203f21d9f-74d1-41d3-9067-dfa807af205ePROCESSOR03da101c7-107c-47da-a83f-588389894f45b9038bf9-0042-4f77-91e7-f4b0edbda7b20 MB0b9038bf9-0042-4f77-91e7-f4b0edbda7b203f21d9f-74d1-41d3-9067-dfa807af205ePROCESSOR0 sec1successb9038bf9-0042-4f77-91e7-f4b0edbda7b23d629abc-d882-471a-938e-8e45a48752b5PROCESSOR003f21d9f-74d1-41d3-9067-dfa807af205eb9038bf9-0042-4f77-91e7-f4b0edbda7b2551.1774260344026152.396133415593WARN1TIMER_DRIVEN1EVENT_DRIVEN0CRON_DRIVEN1TIMER_DRIVEN0 secCRON_DRIVEN* * * * * ?Character SetUTF-8The Character Set in which the file is encodedCharacter SetfalseCharacter SettruefalsefalseMaximum Buffer Size1 MBSpecifies the maximum amount of data to buffer (per file) in order to apply the regular expressions. Files larger than the specified maximum will not be fully evaluated.Maximum Buffer SizefalseMaximum Buffer SizetruefalsefalseMaximum Capture Group Length1024Specifies the maximum number of characters a given capture group value can have. Any characters beyond the max will be truncated.Maximum Capture Group LengthfalseMaximum Capture Group LengthfalsefalsefalseEnable Canonical EquivalencetruetruefalsefalsefalseIndicates that two characters match only when their full canonical decompositions match.Enable Canonical EquivalencefalseEnable Canonical EquivalencetruefalsefalseEnable Case-insensitive MatchingtruetruefalsefalsefalseIndicates that two characters match even if they are in a different case. Can also be specified via the embeded flag (?i).Enable Case-insensitive MatchingfalseEnable Case-insensitive MatchingtruefalsefalsePermit Whitespace and Comments in PatterntruetruefalsefalsefalseIn this mode, whitespace is ignored, and embedded comments starting with # are ignored until the end of a line. Can also be specified via the embeded flag (?x).Permit Whitespace and Comments in PatternfalsePermit Whitespace and Comments in PatterntruefalsefalseEnable DOTALL ModetruetruefalsefalsefalseIndicates that the expression '.' should match any character, including a line terminator. Can also be specified via the embeded flag (?s).Enable DOTALL ModefalseEnable DOTALL ModetruefalsefalseEnable Literal Parsing of the PatterntruetruefalsefalsefalseIndicates that Metacharacters and escape characters should be given no special meaning.Enable Literal Parsing of the PatternfalseEnable Literal Parsing of the PatterntruefalsefalseEnable Multiline ModetruetruefalsefalsefalseIndicates that '^' and '$' should match just after and just before a line terminator or end of sequence, instead of only the begining or end of the entire input. Can also be specified via the embeded flag (?m).Enable Multiline ModefalseEnable Multiline ModetruefalsefalseEnable Unicode-aware Case FoldingtruetruefalsefalsefalseWhen used with 'Enable Case-insensitive Matching', matches in a manner consistent with the Unicode Standard. Can also be specified via the embeded flag (?u).Enable Unicode-aware Case FoldingfalseEnable Unicode-aware Case FoldingtruefalsefalseEnable Unicode Predefined Character ClassestruetruefalsefalsefalseSpecifies conformance with the Unicode Technical Standard #18: Unicode Regular Expression Annex C: Compatibility Properties. Can also be specified via the embeded flag (?U).Enable Unicode Predefined Character ClassesfalseEnable Unicode Predefined Character ClassestruefalsefalseEnable Unix Lines ModetruetruefalsefalsefalseIndicates that only the '
' line terminator is recognized in the behavior of '.', '^', and '$'. Can also be specified via the embeded flag (?d).Enable Unix Lines ModefalseEnable Unix Lines ModetruefalsefalseInclude Capture Group 0truetruefalsefalsetrueIndicates that Capture Group 0 should be included as an attribute. Capture Group 0 represents the entirety of the regular expression match, is typically not used, and could have considerable length.Include Capture Group 0falseInclude Capture Group 0truefalsefalsefalse30 secCharacter SetMaximum Buffer SizeMaximum Capture Group LengthEnable Canonical EquivalenceEnable Case-insensitive MatchingPermit Whitespace and Comments in PatternEnable DOTALL ModeEnable Literal Parsing of the PatternEnable Multiline ModeEnable Unicode-aware Case FoldingEnable Unicode Predefined Character ClassesEnable Unix Lines ModeInclude Capture Group 000 secTIMER_DRIVEN1 secExtractTextfalseFlowFiles are routed to this relationship when the Regular Expression is successfully evaluated and the FlowFile is modified as a resultmatchedfalseFlowFiles are routed to this relationship when no provided Regular Expression matches the content of the FlowFileunmatchedSTOPPEDtruetrueorg.apache.nifi.processors.standard.ExtractText480877a4-f349-4ef7-9538-8e3e3e108e06b9038bf9-0042-4f77-91e7-f4b0edbda7b2970.2632674873253117.5702899657357WARN1TIMER_DRIVEN1EVENT_DRIVEN0CRON_DRIVEN1TIMER_DRIVEN0 secCRON_DRIVEN* * * * * ?Mongo URIMongoURI, typically of the form: mongodb://host1[:port1][,host2[:port2],...]Mongo URIfalseMongo URItruefalsefalseMongo Database NameThe name of the database to useMongo Database NamefalseMongo Database NametruefalsefalseMongo Collection NameThe name of the collection to useMongo Collection NamefalseMongo Collection NametruefalsefalseModeinsertinsertupdateupdateinsertIndicates whether the processor should insert or update contentModefalseModetruefalsefalseUpserttruetruefalsefalsefalseWhen true, inserts a document if no document matches the update query criteria; this property is valid only when using update mode, otherwise it is ignoredUpsertfalseUpserttruefalsefalseUpdate Query Key_idKey name used to build the update query criteria; this property is valid only when using update mode, otherwise it is ignoredUpdate Query KeyfalseUpdate Query KeytruefalsefalseWrite ConcernACKNOWLEDGEDACKNOWLEDGEDUNACKNOWLEDGEDUNACKNOWLEDGEDFSYNCEDFSYNCEDJOURNALEDJOURNALEDREPLICA_ACKNOWLEDGEDREPLICA_ACKNOWLEDGEDMAJORITYMAJORITYACKNOWLEDGEDThe write concern to useWrite ConcernfalseWrite ConcerntruefalsefalseCharacter SetUTF-8The Character Set in which the data is encodedCharacter SetfalseCharacter Settruefalsefalsefalse30 secMongo URImongodb://localhost:27017Mongo Database NamegdelttestMongo Collection NameunmatchedModeUpsertUpdate Query KeyWrite ConcernCharacter Set250 secTIMER_DRIVEN1 secPutMongotrueAll FlowFiles that cannot be written to MongoDB are routed to this relationshipfailuretrueAll FlowFiles that are written to MongoDB are routed to this relationshipsuccessSTOPPEDtruetrueorg.apache.nifi.processors.mongodb.PutMongo3d629abc-d882-471a-938e-8e45a48752b5b9038bf9-0042-4f77-91e7-f4b0edbda7b2555.1774129728792-10.603865851985063WARN1TIMER_DRIVEN1EVENT_DRIVEN0CRON_DRIVEN1TIMER_DRIVEN0 secCRON_DRIVEN* * * * * ?Input DirectoryThe input directory from which to pull filesInput DirectoryfalseInput DirectorytruefalsetrueFile Filter[^\.].*Only files whose names match the given regular expression will be picked upFile FilterfalseFile FiltertruefalsefalsePath FilterWhen Recurse Subdirectories is true, then only subdirectories whose path matches the given regular expression will be scannedPath FilterfalsePath FilterfalsefalsefalseBatch Size10The maximum number of files to pull in each iterationBatch SizefalseBatch SizetruefalsefalseKeep Source FiletruetruefalsefalsefalseIf true, the file is not deleted after it has been copied to the Content Repository; this causes the file to be picked up continually and is useful for testing purposes. If not keeping original NiFi will need write permissions on the directory it is pulling from otherwise it will ignore the file.Keep Source FilefalseKeep Source FiletruefalsefalseRecurse SubdirectoriestruetruefalsefalsetrueIndicates whether or not to pull files from subdirectoriesRecurse SubdirectoriesfalseRecurse SubdirectoriestruefalsefalsePolling Interval0 secIndicates how long to wait before performing a directory listingPolling IntervalfalsePolling IntervaltruefalsefalseIgnore Hidden FilestruetruefalsefalsetrueIndicates whether or not hidden files should be ignoredIgnore Hidden FilesfalseIgnore Hidden FilestruefalsefalseMinimum File Age0 secThe minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignoredMinimum File AgefalseMinimum File AgetruefalsefalseMaximum File AgeThe maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignoredMaximum File AgefalseMaximum File AgefalsefalsefalseMinimum File Size0 BThe minimum size that a file must be in order to be pulledMinimum File SizefalseMinimum File SizetruefalsefalseMaximum File SizeThe maximum size that a file can be in order to be pulledMaximum File SizefalseMaximum File Sizefalsefalsefalsefalse30 secInput Directory/datadumpFile FilterPath FilterBatch SizeKeep Source FileRecurse SubdirectoriesPolling IntervalIgnore Hidden FilesMinimum File AgeMaximum File AgeMinimum File SizeMaximum File Size250 secTIMER_DRIVEN1 secGetFilefalseAll files are routed to successsuccessSTOPPEDfalsetrueorg.apache.nifi.processors.standard.GetFile58e1487b-10cd-48c8-bbad-9850908d198bb9038bf9-0042-4f77-91e7-f4b0edbda7b2555.9934548429962323.3081455005539WARN1TIMER_DRIVEN1EVENT_DRIVEN0CRON_DRIVEN1TIMER_DRIVEN0 secCRON_DRIVEN* * * * * ?Mongo URIMongoURI, typically of the form: mongodb://host1[:port1][,host2[:port2],...]Mongo URIfalseMongo URItruefalsefalseMongo Database NameThe name of the database to useMongo Database NamefalseMongo Database NametruefalsefalseMongo Collection NameThe name of the collection to useMongo Collection NamefalseMongo Collection NametruefalsefalseModeinsertinsertupdateupdateinsertIndicates whether the processor should insert or update contentModefalseModetruefalsefalseUpserttruetruefalsefalsefalseWhen true, inserts a document if no document matches the update query criteria; this property is valid only when using update mode, otherwise it is ignoredUpsertfalseUpserttruefalsefalseUpdate Query Key_idKey name used to build the update query criteria; this property is valid only when using update mode, otherwise it is ignoredUpdate Query KeyfalseUpdate Query KeytruefalsefalseWrite ConcernACKNOWLEDGEDACKNOWLEDGEDUNACKNOWLEDGEDUNACKNOWLEDGEDFSYNCEDFSYNCEDJOURNALEDJOURNALEDREPLICA_ACKNOWLEDGEDREPLICA_ACKNOWLEDGEDMAJORITYMAJORITYACKNOWLEDGEDThe write concern to useWrite ConcernfalseWrite ConcerntruefalsefalseCharacter SetUTF-8The Character Set in which the data is encodedCharacter SetfalseCharacter Settruefalsefalsefalse30 secMongo URImongodb://localhost:27017Mongo Database NamegdelttestMongo Collection NamegdeltModeUpsertUpdate Query KeyWrite ConcernCharacter Set250 secTIMER_DRIVEN1 secPutMongotrueAll FlowFiles that cannot be written to MongoDB are routed to this relationshipfailuretrueAll FlowFiles that are written to MongoDB are routed to this relationshipsuccessSTOPPEDtruetrueorg.apache.nifi.processors.mongodb.PutMongo09/21/2015 15:14:27 EDT