Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: rjollos Type: defect | Status: review Priority: major | Milestone: Release 8 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Comment (by rjollos): The issue is due to caching behavior in the mutliproduct module, so I'd prefer if we can fix it inside that module rather than further patching `trac.web.main`. I'm not going to push a change unless I can see the use case as to why this fix is necessary, and that there isn't a better way to fix the issue. -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:34 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is
restarted
+---
Reporter: rjollos| Owner: rjollos
Type: defect | Status: new
Priority: major | Milestone: Release 8
Component: dashboard |Version:
Resolution: | Keywords: tracadmin command
+---
Changes (by olemis):
* status: closed = new
* resolution: fixed =
Comment:
Replying to [comment:27 rjollos]:
I can confirm that [attachment:t539_r1560613_hooks_update_config.diff]
fixes the issue with enabling and disabling components. Calling
`parse_if_needed` in `open_environment`
(`multiproduct.hooks:MultiProductEnvironmentFactory.open_environment`)
addresses my concern in comment:11.
I'm actually reopening this ticket to suggest a minor (but important
improvement) in order to reload env config outside hooks definition . The
point is , if other web bootstrap handler is installed (e.g. for product
sub-domains) then still ensure that product config is reloaded before
entering the dispatching loop .
{{{#!sh
$ hg qapplied
t539/t539_r1562687_hooks_update_config.v2.diff
$ hg log -r qparent
changeset: 535:2ad16ce035d6
tag: qparent
tag: svn-1562687
user:rjollos@13f79535-47bb-0310-9956-ffa450edef68
date:Thu Jan 30 03:46:39 2014 +
summary: 0.8dev: Changes to the product configuration will take effect
immediately. Refs #539.
}}}
--
Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:28
Apache Bloodhound https://issues.apache.org/bloodhound/
The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: rjollos Type: defect | Status: review Priority: major | Milestone: Release 8 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Changes (by olemis): * owner: olemis = rjollos * status: accepted = review Comment: please review ... -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:30 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: olemis Type: defect | Status: accepted Priority: major | Milestone: Release 8 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Changes (by olemis): * owner: rjollos = olemis * status: new = accepted -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:29 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is
restarted
+---
Reporter: rjollos| Owner: olemis
Type: defect | Status: assigned
Priority: major | Milestone: Release 8
Component: dashboard |Version:
Resolution: | Keywords: tracadmin command
+---
Changes (by olemis):
* status: review = assigned
* owner: rjollos = olemis
Comment:
Replying to [comment:20 olemis]:
[...]
Replying to [comment:17 rjollos]:
[...]
To reproduce the issue with component enabling/disabling not taking
effect until `tracd` is restarted,
1. Open trac-admin:
{{{#!sh
$ trac-admin $env
product admin prod1 config set components trac.wiki.* disabled
}}}
1. Refresh `/products/prod1/wiki`. The component will not be disabled
until `tracd` is restarted.
I'll start from here , and try to reproduce ( ... and fix if a problem
is found )
confirmed , not fixed .
/me working on it
--
Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:22
Apache Bloodhound https://issues.apache.org/bloodhound/
The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is
restarted
+---
Reporter: rjollos| Owner: olemis
Type: defect | Status: assigned
Priority: major | Milestone: Release 8
Component: dashboard |Version:
Resolution: | Keywords: tracadmin command
+---
Comment (by olemis):
I have added a [attachment:t539_r1560613_hooks_update_config.diff new
patch] that will fix the issues mentioned in comment:17 and
comment:description . Details :
{{{#!sh
$ hg qapplied
t539/t539_r1516428_test_product_config_sync.diff
t539/t539_r1516428_product_config_sync.diff
t539/t539_r1516428_test_product_config_sync.2.diff
t539/t539_r1545950_product_component_dyn_status.diff
t539/t539_r1560613_hooks_update_config.diff
$ hg log -r qparent --template={svnrev}\n
1560613
}}}
I suggest that all other (unrelated) issues about DB caching behaviors
will be fixed elsewhere in separate tickets .
--
Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:24
Apache Bloodhound https://issues.apache.org/bloodhound/
The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: olemis Type: defect | Status: assigned Priority: major | Milestone: Release 8 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Comment (by olemis): btw , I refreshed the patches to apply them on top of r1560613 -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:25 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: rjollos Type: defect | Status: review Priority: major | Milestone: Release 8 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Changes (by olemis): * owner: olemis = rjollos * status: assigned = review -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:26 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is
restarted
+---
Reporter: rjollos| Owner: rjollos
Type: defect | Status: review
Priority: major | Milestone: Release 8
Component: dashboard |Version:
Resolution: | Keywords: tracadmin command
+---
Comment (by rjollos):
I've tested with the latest patches, but I'm still seeing the issues
described in comment:description and comment:3:
1. Grant a permission to a subject through web admin.
1. Refresh the page. On repeated refresh the action will sometimes be
associated with the subject, and other times not.
This was also reported on the
[http://markmail.org/message/rfurv6tzjbqhgomt mailing list] today.
The same general issue can be seen when renaming wiki pages and refreshing
the TitleIndex. The old or new page will randomly appear with each
refresh.
1. Open trac-admin:
{{{#!sh
$ trac-admin $env
product admin prod1 config set components trac.wiki.* disabled
}}}
1. Refresh `/products/prod1/wiki`. The component will not be disabled
until `tracd` is restarted.
All of these issues were seen with TracStandalone (make sure to //not//
pass the `-r` flag to `tracd`).
The `ProductEnvironment` is retrieved from the cache when the page is
refreshed, and I don't see a mechanism by which the cache will be
invalidated when a request is sent and the `ProductEnvironment` is
retrieved from the cache.
--
Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:17
Apache Bloodhound https://issues.apache.org/bloodhound/
The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: rjollos Type: defect | Status: review Priority: major | Milestone: Release 8 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Comment (by rjollos): Replying to [comment:4 astaric]: What was the reasoning behind the decision that the product configs should be stored in the database instead of the .ini files? File configs would allow us to reuse trac functionality for detecting modification of configuration. This was discussed a few times early on, in #115 and on the [http://markmail.org/thread/p4agqsmokub2c34q mailing list]. I seem to remember some additional in-depth discussion on the mailing list that I'm not able to locate now. -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:18 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: rjollos Type: defect | Status: review Priority: major | Milestone: Release 8 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Comment (by rjollos): The issues in comment:17 were reproduced using SQLite. -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:19 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is
restarted
+---
Reporter: rjollos| Owner: rjollos
Type: defect | Status: review
Priority: major | Milestone: Release 8
Component: dashboard |Version:
Resolution: | Keywords: tracadmin command
+---
Comment (by olemis):
In general I suspect the issues in comment:description are not related to
those mentioned in comment:3 . The former is about product config sync
whereas the later is about a mysterious DB caching , I've not been able to
detect its root cause .
Replying to [comment:17 rjollos]:
[...]
To reproduce the issue with component enabling/disabling not taking
effect until `tracd` is restarted,
1. Open trac-admin:
{{{#!sh
$ trac-admin $env
product admin prod1 config set components trac.wiki.* disabled
}}}
1. Refresh `/products/prod1/wiki`. The component will not be disabled
until `tracd` is restarted.
I'll start from here , and try to reproduce ( ... and fix if a problem is
found )
All of these issues were seen with TracStandalone (make sure to //not//
pass the `-r` flag to `tracd`).
The `ProductEnvironment` is retrieved from the cache when the page is
refreshed, and I don't see a mechanism by which the cache will be
invalidated when a request is sent and the `ProductEnvironment` is
retrieved from the cache.
The `ProductEnvironment` cache does not need to be invalidated at all ,
just product config cache . Anyway , let me see what's wrong with your
scenario .
--
Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:20
Apache Bloodhound https://issues.apache.org/bloodhound/
The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: rjollos Type: defect | Status: review Priority: major | Milestone: Release 8 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Comment (by rjollos): Change from t:comment:35:ticket:9536 was pushed in [trac 12285]. -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:16 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: rjollos Type: defect | Status: review Priority: major | Milestone: Release 8 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Comment (by rjollos): I'll take a look, but will probably have more questions later. Regarding the issue you found in the Trac core, I added a comment about that in t:comment:35:ticket:9536 and will try to push that fix in Trac 1.0.2. -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:15 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is
restarted
+---
Reporter: rjollos| Owner: rjollos
Type: defect | Status: review
Priority: major | Milestone: Release 8
Component: dashboard |Version:
Resolution: | Keywords: tracadmin command
+---
Comment (by olemis):
I just remembered now what were these patches all about . Once upon a time
I just submitted the code to synchronize product config files, which was a
subtle feature request considering comment:4 .
Like I just said in comment:13 components on/off status is influenced by
two levels of cache stored in environment's `enabled` and `_rules`
attributes. The goal of
[attachment:t539_r1545950_product_component_dyn_status.diff attached
patch] is to watch for modification time of product env's config lock file
(introduced in previous patches ;) . If values loaded in memory are
outdated then firstly product env's config is reloaded. Cached data
influencing component state is cleared as well.
It is a fact that components will be enabled in product context only if
they are enabled in the global environment as well. Therefore cache
invalidation takes place for both product and global env. In its current
form this approach may be inefficient (and there's an inline FIXME tag in
for this ;) e.g. in a global env having n products with updated config
values then even if global config does not change at all its cache will
invalidated O(n) times . This means that global [components] section will
be (systematicly and unnecessarily) reloaded every time a configuration
value is added in a product. This might lead to serious performance
penalties and overhead, but works as expected. @rjollos : since this
belongs in our Trac copy and maight be backported to Trac then I thought
I'd rather delegate such improvements for you to move forward with this
ticket if you do not mind , otherwise let me know and I'll focus on a new
patch.
Ideally cache invalidation should work like this
||= Global config =||= Product config =||= Global on/off cache =||=
Product on/off cache =||
|| outdated|| unchanged|| invalidate (clear)||
invalidate (clear) ||
|| outdated|| outdated || invalidate (clear)||
invalidate (clear) ||
|| unchanged || outdated || keep in-memory copy ||
invalidate (clear) ||
|| unchanged || unchanged|| keep in-memory copy || keep
in-memory copy||
... and another potentially useful optimization may consist in computing a
diff upon [components] section (rather than whole config modification
time) to decide whether cached on/off data have to be cleared or not .
This may be overkill though
The [http://pastebin.com/urQsMENh test report] illustrates that there are
no regressions . Details follow
{{{#!sh
$ hg qapplied
t539/t539_r1516428_test_product_config_sync.diff
t539/t539_r1516428_product_config_sync.diff
t539/t539_r1516428_test_product_config_sync.2.diff
t539/t539_r1545950_product_component_dyn_status.diff
$ hg log -r qparent
changeset: 497:e260cdce0b83
tag: qparent
tag: svn-1545950
user:rjollos@13f79535-47bb-0310-9956-ffa450edef68
date:Wed Nov 27 07:21:58 2013 +
summary: 0.8dev: Integrate change from [trac 12268].
}}}
PS: There are a number of adjustments in test code to consider values not
formerly written in environment stub's config at testing time .
--
Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:14
Apache Bloodhound https://issues.apache.org/bloodhound/
The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: rjollos Type: defect | Status: review Priority: major | Milestone: Release 8 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Comment (by olemis): Replying to [comment:11 rjollos]: I've tested the patch and it doesn't seem to resolve the issue reported in this ticket. I'm not sure why this is yet, but it seems that `parse_if_needed` is not being called on the `multiproduct.config.Configuration` instance in `open_environment`. I am not sure I quite understand this. Function open_environment is aimed at loading a global env by path. OTOH Db config objects are only used by instances of ProductEnvironment. Nevertheless I'll try the patches against latest versions and let you know of the results I get asap -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:12 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: rjollos Type: defect | Status: review Priority: major | Milestone: Release 8 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Comment (by rjollos): I've tested the patch and it doesn't seem to resolve the issue reported in this ticket. I'm not sure why this is yet, but it seems that `parse_if_needed` is not being called on the `multiproduct.config.Configuration` instance in `open_environment`. -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:11 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: olemis Type: defect | Status: accepted Priority: major | Milestone: Release 8 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Comment (by olemis): Regressions fixed in [attachment:t539_r1516428_test_product_config_sync.2.diff this patch] . See [http://goo.gl/po7ovF test results] . [comment:7 Patch order] udated accordingly. -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:8 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: olemis Type: defect | Status: accepted Priority: major | Milestone: Release 6 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Comment (by astaric): Config values in Product Configuration are cached to avoid hitting the database every time the config is accessed. Cache for config option is invalidated only when this option is set using the same Configuration instance, which is not the case when configuration is being changed using the trac-admin. Trac uses changes times of its configuration files to detect changes and reload the configuration (it actually creates a new environment, see open_environment in trac.env). To provide similar behaviour for product configurations we would need to somehow detect that the changes were made to the configuration in the database. If could store last modification data as a special value in a bloodhound_productconfig table, but this would require a hit to the database each time a value is read from the config so simply disabling the cache would be easier. What was the reasoning behind the decision that the product configs should be stored in the database instead of the .ini files? File configs would allow us to reuse trac functionality for detecting modification of configuration. Anyway, I do not think that this issue is ready to be fixed before 0.6. We can disable caching of config values which should be enough to close this ticket and open a separate ticket for caching discussion, or move this ticket to the next milestone. -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:4 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: olemis Type: defect | Status: accepted Priority: major | Milestone: Release 7 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Changes (by olemis): * milestone: Release 6 = Release 7 Comment: Deferring this ticket to [milestone:Release 7] -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:5 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: olemis Type: defect | Status: accepted Priority: major | Milestone: Release 6 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Comment (by olemis): I have also noticed the effects of this failure mode when renaming wiki pages . Immediately after doing so the changes will not be reflected in TitleIndex . After sending `HUP` signal to ''apache2'' server everything works as expected . -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:3 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
[Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is
restarted
---+---
Reporter: rjollos| Owner: olemis
Type: defect | Status: new
Priority: major | Milestone: Release 6
Component: dashboard |Version:
Keywords: tracadmin command |
---+---
The following issue was noticed while working on #518. With
TracStandalone,
{{{#!sh
config set components trac.wiki.* disabled
}}}
and a page refresh immediately shows that the wiki is disabled. However,
when the corresponding product command is run, `tracd` must be killed and
restarted for the change to take effect.
{{{#!sh
product admin prod1 config set components trac.wiki.* disabled
}}}
I tested while disabling/enabling other components and had the same
result. Another possibly-related result of this is that the TracAdmin
console must be exited before the allowed commands for a disabled
component (displayed by ` product admin prod1 help`) are no longer shown.
--
Ticket URL: https://issues.apache.org/bloodhound/ticket/539
Apache Bloodhound https://issues.apache.org/bloodhound/
The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is restarted +--- Reporter: rjollos| Owner: olemis Type: defect | Status: accepted Priority: major | Milestone: Release 6 Component: dashboard |Version: Resolution: | Keywords: tracadmin command +--- Changes (by olemis): * status: new = accepted -- Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:1 Apache Bloodhound https://issues.apache.org/bloodhound/ The Apache Bloodhound issue tracker
Re: [Apache Bloodhound] #539: Product-scope TracAdmin commands don't take effect until tracd is restarted
#539: Product-scope TracAdmin commands don't take effect until tracd is
restarted
+---
Reporter: rjollos| Owner: olemis
Type: defect | Status: accepted
Priority: major | Milestone: Release 6
Component: dashboard |Version:
Resolution: | Keywords: tracadmin command
+---
Comment (by olemis):
Replying to [ticket:539 rjollos]:
[...]
However, when the corresponding product command is run, `tracd` must be
killed and restarted for the change to take effect.
{{{#!sh
product admin prod1 config set components trac.wiki.* disabled
}}}
[...]
Another possibly-related result of this is that the TracAdmin console
must be exited before the allowed commands for a disabled component
(displayed by ` product admin prod1 help`) are no longer shown.
The two situations are effects of the same failure mode .
--
Ticket URL: https://issues.apache.org/bloodhound/ticket/539#comment:2
Apache Bloodhound https://issues.apache.org/bloodhound/
The Apache Bloodhound issue tracker
