Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/1831 @bbende I very much like the idea of this PR. I tried to test it out, though, and it didn't behave as I was hoping. I downloaded the GeoLite2 database for ASN and for City. I set the service to lookup all fields. I then copied the City database to "db.mmdb" and configured the lookup to load db.mmdb file. While running, I tried overwriting db.mmdb with the ASN database so that we can see if it reloads properly. Unfortunately, though, as soon as I overwrote the file, I got the following stack trace: ``` 2017-05-19 17:02:19,287 ERROR [Timer-Driven Process Thread-4] o.a.n.processors.standard.LookupRecord LookupRecord[id=17a48af1-015c-1000-29d3-2b7ebd7d4241] Failed to process StandardFlowFileRecord[uuid=c25e503e-5f28-4b59-a05f-7462df2424b8,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1495227698674-1, container=default, section=1], offset=36678, length=91],offset=0,name=1599597234048490,size=91]: org.apache.nifi.processor.exception.ProcessException: Failed to lookup value '216.58.195.228' in Lookup Service org.apache.nifi.processor.exception.ProcessException: Failed to lookup value '216.58.195.228' in Lookup Service at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:218) at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:58) at org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:136) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2136) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2106) at org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:129) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.nifi.lookup.LookupFailureException: Failed to lookup City information for IP Address /216.58.195.228 at org.apache.nifi.lookup.maxmind.IPLookupService.lookup(IPLookupService.java:211) at sun.reflect.GeneratedMethodAccessor349.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89) at com.sun.proxy.$Proxy77.lookup(Unknown Source) at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:216) ... 17 common frames omitted Caused by: com.maxmind.db.InvalidDatabaseException: The MaxMind DB file's search tree is corrupt: contains pointer larger than the database. at com.maxmind.db.Reader.resolveDataPointer(Reader.java:196) at com.maxmind.db.Reader.get(Reader.java:103) at org.apache.nifi.lookup.maxmind.DatabaseReader.get(DatabaseReader.java:150) at org.apache.nifi.lookup.maxmind.DatabaseReader.city(DatabaseReader.java:194) at org.apache.nifi.lookup.maxmind.IPLookupService.lookup(IPLookupService.java:209) ... 23 common frames omitted ``` If we are going to support reloading, then we need to support having the file overwritten. I think we should wrap the entire lookup method in a try/catch and catch InvalidDatabaseException. If this is caught, we should reload the database and try again. If it fails a second time, then we should just throw a LookupFailedException. Thoughts?
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---