[xwiki-devs] Security cache doing a large clear every 4 hours

[ktc]

Hi, we are running into an issue with our Wiki security cache where we notice
a large dump of the cache every 4 hours. The reason this happens every 4
hours is because we have configured out infinispan expiry time to 4 hours;
however, we would expect a gradual expiration of the cache. 

After investigating the DefaultSecurityCache logic, we discovered that the
dispose() method is being called whenever infinispan attempts to expire a
security cache entry. When disposing an entry, it will rightfully disconnect
the entry from its parent(s) and remove all children.

What happens is that the XWiki root page ("xwiki:XWiki") is one of the first
entries to be created. As such, it is one of the first entries to be
expired. When it is expired, it removes all children from the cache as well.
This results in all user pages ("xwiki:XWiki.user1", "xwiki:XWiki.user2",
...) as well as our permission groups (stored under
"xwiki:XWiki.POSIX.group1", "xwiki:XWiki.LDAP.group2", ...). This also
removes all children linking to documents ("xwiki:user1@@Document"). 

In order to fix this, we were planning on adding logic around the
DefaultSecurityCache.dispose() method to skip disposal of certain core pages
("xwiki:XWiki.POSIX", and "xwiki:XWiki.LDAP") as there is no real benefit of
removing them from the security cache, but need to further investigate
possible risks or side effects.

Some questions we had for the XWiki dev team were:

1. Where are XWiki groups stored? Are they not under the document
"xwiki:XWiki.group1"?
2. If they are stored under a shared parent, shouldn't they run into the
same issue as we are experiencing where clearing the "xwiki:XWiki" entry
also removes all user entries and groups?



--
Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html

Re: [xwiki-devs] Activating Wiki on a single host in fleet clears security cache for entire fleet

[ktc]

Hi Thomas,

Thanks for the reply! Yes, the link you provided is the right one with the
bug we face. The thing is we may not be able to upgrade to a newer version
of XWiki recently, so we plan to add a title for XWikiServerXwiki page as
the workaround. Do you think it is safe? And do you have any specific idea
why an internal XWiki page exists without a title and how we can prevent
this from happening again? Thank you!



--
Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html

Re: [xwiki-devs] Activating Wiki on a single host in fleet clears security cache for entire fleet

[ktc]

Hi Denis,

Thank you for your reply! We found out the reason is a document called
XWiki.XWikiServerXwiki whose title is empty. During the XWiki initialization
in 8.4.4 version(the version we use),  setDocumentFields

  
will mark the document as needsUpdate if the title is blank(It only has
StringUtils.isBlank(document.getTitle()) in if statement for 8.4.4 version).
So it will then remove the security cache and cause the issue. 

For the fix, we plan to add a title for XWiki.XWikiServerXwiki page, which
solved the issue in beta testing. Right now we didn't see any problem except
it will remove the security caches once more. Do you think it's safe doing
that? Thanks! 



--
Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html

[xwiki-devs] Activating Wiki on a single host in fleet clears security cache for entire fleet

[ktc]

Hi, we noticed an issue during host activation where a single host being
activated would cause all hosts in the fleet to then clear their security
cache. Upon investigation, we discovered that this was due to a JGroups
message being sent for a DocumentUpdatedEvent. Here is a sample message
received by one of the hosts in the fleet:

Fri May 11 16:18:04 0995 GMT Wiki -@ [DEBUG]  (null) Received
JGroups remote event [event:
[org.xwiki.bridge.event.DocumentUpdatedEvent@16e3c8e3], source:
[{docversion=41.1, doclanguage=, origdocversion=41.1, origdoclanguage=,
docname=xwiki:XWiki.XWikiServerXwiki}], data: [{contextwiki=xwiki,
contextuser=XWiki.XWikiGuest}]], id:  timestamp: [Fri May 11
16:18:04 UTC 2018] air time: 22 ms RID=, SESSION=

>From what we can tell, the initialization triggers the
XWikiServerXwikiDocumentInitializer's updateDocument() event. As expected,
this document update produces the DocumentUpdatedEvent shown above. When the
hosts receive this event through JGroups, it triggers [this condition in the
DefaultSecurityCacheRulesInvalidator](https://github.com/xwiki/xwiki-platform/blob/eb0354b7ac7730108751a137a4dbc80187cbaf37/xwiki-platform-core/xwiki-platform-security/xwiki-platform-security-bridge/src/main/java/org/xwiki/security/authorization/internal/DefaultSecurityCacheRulesInvalidatorListener.java#L220-L227),
which then proceeds to remove the 'xwiki' node from the security cache (and
everything else in the cache as well since they are all connected to that
node).

What is the purpose of this update document on initialization and is there
any way for us to avoid it? For example, we could perhaps override the
DefaultSecurityCacheRulesInvalidatorListener and comment out that condition
if there are no major repercussions. Thanks!



--
Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html

[xwiki-devs] Weird issue in security cache

[ktc]

Hi XWiki devs,
 
We ran into a weird issue on XWiki security cache. Our entire security cache
is purged every 4 hours.
 
Some background about our setup:
We retrieve users’ group membership from LDAP
We set a 4 hour expiration time on the cache(as below) because LDAP records
may update at any time.
 

  
  <-- 4 hours -->
  


To our understanding, each cache entry should have its own timer which
expires after 4 hours.
But what we observe from JMX dashboard is that the entire security cache is
being purged every 4 hours.
 
Security cache JMX diagram:
 
 
Our user group has similar issue but purge irregularly:
 

This problem is cause big performance issue in our system. We can’t figure
out the root case. Any pointer will be highly appreciated.



--
Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html

Re: [xwiki-devs] Questions related to underlying logics for XWiki security and group cache

[ktc]

Hi devs,

Here are the graphs we mentioned in the above post. 
 
The first is the security authorization cache, as you can see it will wipe
out most entries every 4 hour.
 
The second is the group cache, which is very weird. 

BTW, if there are some documents related to security/group cache, that will
be very helpful, too. 

Thanks in advance! 



--
Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html

[xwiki-devs] Questions related to underlying logics for XWiki security and group cache

[ktc]

Hi devs,

We run into a complicated issue related to XWiki cache and permissions. In
order to fix the issue, we would like to know some underlying logics for
XWiki security and group cache:

1. Why XWiki cache purge happens every 4 hours?
2. From our dashboards, we notice that every 4 hours, the security
authorization cache purge will wipe out most entries(from 50K to 5K). Why
this happens rather than selectively expiring 4-hour-old entries over time
gradually?
3. From our dashboards, we notice the user group cache dumps on an irregular
cadence. The number of entries will drop to nearly 0 on 12:00 and bounce
back at around 16:00 nearly every weekday. Do you know some reasons behind
that?
4. What's the underlying logic of XWiki purges SecurityCache entries? We
assume it will happen in one or both of the following cases:
1. when a given page's permissions are changed
2. when a given user is removed from a (XWiki-internal) group
Is that correct? 
5. We have a custom code called LDAP service. When our wiki user calls that
service, we wish to keep the cache for only a few minutes and then
invalidate the cache. So if our user calls again after a few minutes, it
will try to get reply from the service instead of the cache. For our case,
what method do you suggest to use when we invalidate SecurityCache entries?
Is there any side effect if we do that?

Thanks a lot for your help!



--
Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html

[xwiki-devs] Will XWiki Activity Stream deprecation affect activitystream_events table?

[ktc]

Hi devs,

Right now we are building a feature based on XWiki Activity Stream and get
data from activitystream_events table from database. We've heard that XWiki
will replace Activity Stream with Notification. Does this mean
activitystream_events table will be deprecated as well? If it is, do you
have any suggestions for replacement like where we can get similar data from
database and which application/extension with similar support features that
we can rely on? Thanks a lot!



--
Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html

[xwiki-devs] XWiki Security Cache global read/write lock scalability

[ktc]

We ran into an issue recently where users were not able to access the Wiki
due to the XWiki security cache holding up access to read requests as we
were experiencing heap exhaustion. A thread dump of a host showed that there
were 211 security cache read locks and 2 security cache write locks during
the event. Read locks are supposedly parallelizable, so we suspect that it
was the write locks that blocked the read requests.

Looking into the write locks, we found
*DefaultSecurityCacheRulesInvalidatorListener* shares a read-write lock
(*SecurityCacheRulesInvalidator*) with the security cache loader
(*DefaultSecurityCacheLoader*). It appears that these write requests to the
security cache were potentially creating a deadlock, blocking any further
read requests.

Our investigation of this event resulted in the following questions that we
are hoping to get answered by reaching out here:

1. What exactly happens in the event of a write failure to the Security
Cache? Are we correct in understanding that the write request will continue
to hold a lock on the cache, preventing all read requests from proceeding?
2. If so, are there any mitigations around the global read-write lock on the
security cache? Or, perhaps, is the best action is to simply move toward a
more scalable caching mechanism (i.e. a centralized cache such as Elastic
Cache)?
3. It looks as if all threads doing a read/write request to the
SecurityCache were all parked waiting for the same thread (3081f946);
however, the thread is not showing up in the thread dump so we are not sure
what it is stuck on. Does the XWiki team have any suspicions as to what
could be hold up the requests to the SecurityCache?

Thank you!

---

For more information on the event:

>From our investigation, we concluded that the outage happened because:

1. Remote events triggered a security cache update and requested a couple of
write locks
2. The writes could not execute (perhaps due to the heap exhaustion,
although root cause of write failure is unknown)
3. ReentrantReadWriteLock lock followed a fair policy, which releases locks
in time order, so read requests were forced to wait for any previous write
requests to finish. Because the write requests couldn't finish, the
remaining 211 threads were blocked.

* Possible remote events that trigger security cache update: changes to an
XWiki group such as leaving the admin group, user first time visiting XWiki,
etc..



*LDAP Connection Lock example:*
 
Thread t@225108
   java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
- waiting on <7249fc39> (a com.novell.ldap.Connection)
at com.novell.ldap.Connection.waitForReader(Unknown Source)
at com.novell.ldap.Connection.startReader(Unknown Source)
at com.novell.ldap.Connection.connect(Unknown Source)
at com.novell.ldap.Connection.connect(Unknown Source)
at com.novell.ldap.LDAPConnection.connect(Unknown Source)
at
com.ownership.groups.ldap.DefaultLdapConnection.connect(DefaultLdapConnection.java:25)
at
com.ownership.groups.ldap.LdapClient.queryGroups(LdapClient.java:99)
at
com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:64)
at
com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:59)
at
com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:113)
- locked <7cc1cfca> (a java.lang.String)
at
com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:75)
at
org.xwiki.security.internal.DefaultUserBridge.getGroupsReferencesFor(DefaultUserBridge.java:122)
at
org.xwiki.security.internal.DefaultUserBridge.getAllGroupsFor(DefaultUserBridge.java:78)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:350)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:342)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserEntry(DefaultSecurityCacheLoader.java:318)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadAccessEntries(DefaultSecurityCacheLoader.java:221)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadRequiredEntries(DefaultSecurityCacheLoader.java:193)
at
org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:148)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159)
at
org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152)
at
org.xwiki.

[xwiki-devs] Bug in XObjects, or am I using them incorrectly?

[ktc]

I'm running into what appears to be a bug in how XObjects are removed and
added, and wanted to ask if I am perhaps using them incorrectly.

Basically I am performing the following sequence of steps:

1. Insert some XObjects to an XWikiDocument using document.newXObject().
Save the document and reload.
2. Delete one of the XObjects in the middle of the list of XObjects, using
document.removeXObject(). Save the document and reload.
3. Add another XObject using document.newXObject(). Save.

Here is that the document's xObjects data looks like after each step
(simplified for demonstration purposes):

AFTER STEP 1:
index=0 object="...0..."
index=1 object="...1..."
index=2 object="...2..."

AFTER STEP 2 (assume I delete the xObject with index=1):
index=0 object="...0..."
index=1 object=null
index=2 object="...2..."

AFTER STEP 3:
index=0 object="...0..."
index=1 object=null
index=2 object="...2..."
index=3 object="...2..."


As you can see there are two related problems here:

1. A null object is being preserved in the list of XObjects, even after
being rehydrated from Hibernate following a server restart.
2. The index=3 object has been given a 'number' of '2' -- but that number is
already in use by index=2 object.

Because of this, saving the document actually fails: I am receiving a
NonUniqueObjectException as follows:

org.hibernate.NonUniqueObjectException: a different object with the same
identifier value was already associated with the session:
[com.xpn.xwiki.objects.BaseObject#-1957000305211198111]

Looking in the XWiki code it looks like this is essentially happening
because the value of 'number' is computed as 'list size - 1'. So that 'null'
that gets preserved in the list seems to be throwing off the logic.

Am I using newXObject()/removeXObject() improperly?

Please advise.

Thanks!




--
View this message in context: 
http://xwiki.475771.n2.nabble.com/Bug-in-XObjects-or-am-I-using-them-incorrectly-tp7603477.html
Sent from the XWiki- Dev mailing list archive at Nabble.com.

Re: [xwiki-devs] (RESEND) Nesting within Annotated XHTML

[ktc]

Thanks for the input Marius.

Could you advise on a starting point for implementing an 'element-based'
equivalent of the Annotated XHTML syntax? Is there a particular class(es)
that could be extended or mimicked to achieve this (ideally building on
existing logic for most of the heavy lifting)?

Thanks!



--
View this message in context: 
http://xwiki.475771.n2.nabble.com/RESEND-Nesting-within-Annotated-XHTML-tp7603271p7603277.html
Sent from the XWiki- Dev mailing list archive at Nabble.com.

[xwiki-devs] (RESEND) Nesting within Annotated XHTML

[ktc]

EDITED: Resending this message to fix some escaping issues when the email
went out the first time.

I have a use case where I need to nest macros to 2+ levels deep using
XWiki's Annotated XHTML syntax. We are using this syntax as an input to a
transformation process that utilizes XWiki's rendering engine to convert the
Annotated XHTML into XWiki-native markup, similarly to how CKEditor does its
work.

The problem is that with Annotated XHTML, macros are encoded as HTML
comments -- but HTML comments cannot be nested within other HTML comments.

Here is a demonstrative example: (NOTE: in all examples below I have added
extra spaces around the ! chars to prevent these from getting stripped out
in this email)

  < ! --startmacro:outermacro|-||-|outer text before < !
--startmacro:innermacro|-||-|inner text-->< ! --stopmacro--> outer text
after-->< ! --stopmacro-->

The intention is for this to be convertible to this XWiki syntax:

  {{outermacro}}outer text before {{innermacro}}inner text{{/innermacro}}
outer text after{{/outermacro}}

However, this doesn't work because the HTML parser sees the first "-->" it
encounters as the "end of comment" marker, thus breaking the intended
nesting. 

Thus, when converted to XWiki syntax, this is what comes out instead:

  {{outermacro}}outer text before <==startmacro:innermacro|-||-|inner
text{{/outermacro}} outer text after ~-~->


We have considered a couple possible solutions:

1. Mimic what CKEditor does in this scenario, which is to use an HTML
comment for the outermost macro-block, but have any nested macros (embedded
inside the HTML comment) be rendered as XWiki-native syntax. Like this:

  < ! --startmacro:outermacro|-||-|outer text before {{innermacro}}inner
text{{//innermacro}} outer text after-->< ! --stopmacro-->

This approach seems to work, but it's not ideal because our Annotated XHTML
generation code will now need to support two varieties of output depending
on the current nesting level -- Annotated XHTML for the outermost depth, and
XWiki-native for anything nested deeper.

2. Implement a variation of the existing Annotated XHTML parsing logic
which, instead of expecting macros encoded in HTML comments, will expect the
use of an XHTML structure that models the same information, but doesn't
suffer from the nesting restrictions of HTML comments. Something like this:

  
outer text before

  inner text

outer text after
  


I am looking for input from XWiki experts regarding what approach would make
the most sense. Option 2 above seems like the most elegant solution overall,
but I'm not sure how difficult this would be to get working in the current
XWiki codebase, or what file(s) would need to be extended, etc. Is there
another approach that might solve this problem more efficiently?


Note: For brevity, I've referred to "nested macros" above, but for our use
case we will actually need to handle all the different comment types
recognized in XWikiCommentHandler.java.


References:
AnnotatedXHTMLRenderer.java:
https://github.com/xwiki/xwiki-rendering/blob/master/xwiki-rendering-syntaxes/xwiki-rendering-syntax-annotatedxhtml/src/main/java/org/xwiki/rendering/internal/renderer/xhtml/AnnotatedXHTMLRenderer.java
XWikiCommentHandler.java:
https://github.com/xwiki/xwiki-rendering/blob/master/xwiki-rendering-syntaxes/xwiki-rendering-syntax-xhtml/src/main/java/org/xwiki/rendering/internal/parser/xhtml/wikimodel/XWikiCommentHandler.java#L90-L99



--
View this message in context: 
http://xwiki.475771.n2.nabble.com/RESEND-Nesting-within-Annotated-XHTML-tp7603271.html
Sent from the XWiki- Dev mailing list archive at Nabble.com.

[xwiki-devs] Nesting within Annotated XHTML

[ktc]

I have a use case where I need to nest macros to 2+ levels deep using XWiki's
Annotated XHTML syntax. We are using this syntax as an input to a
transformation process that utilizes XWiki's rendering engine to convert the
Annotated XHTML into XWiki-native markup, similarly to how CKEditor does its
work.

The problem is that with Annotated XHTML, macros are encoded as HTML
comments -- but HTML comments cannot be nested within other HTML comments.

Here is a demonstrative example:

   outer text after-->

The intention is for this to be convertible to this XWiki syntax:

  {{outermacro}}outer text before {{innermacro}}inner text{{/innermacro}}
outer text after{{/outermacro}}

However, this doesn't work because the HTML parser sees the first "-->" it
encounters as the "end of comment" marker, thus breaking the intended
nesting. 

Thus, when converted to XWiki syntax, this is what comes out instead:

  {{outermacro}}outer text before <==startmacro:innermacro|-||-|inner
text{{/outermacro}} outer text after ~-~->


We have considered a couple possible solutions:

1. Mimic what CKEditor does in this scenario, which is to use an HTML
comment for the outermost macro-block, but have any nested macros (embedded
inside the HTML comment) be rendered as XWiki-native syntax. Like this:

  

This approach seems to work, but it's not ideal because our Annotated XHTML
generation code will now need to support two varieties of output depending
on the current nesting level -- Annotated XHTML for the outermost depth, and
XWiki-native for anything nested deeper.

2. Implement a variation of the existing Annotated XHTML parsing logic
which, instead of expecting macros encoded in HTML comments, will expect the
use of an XHTML structure that models the same information, but doesn't
suffer from the nesting restrictions of HTML comments. Something like this:

  
outer text before

  inner text

outer text after
  


I am looking for input from XWiki experts regarding what approach would make
the most sense. Option 2 above seems like the most elegant solution overall,
but I'm not sure how difficult this would be to get working in the current
XWiki codebase, or what file(s) would need to be extended, etc. Is there
another approach that might solve this problem more efficiently?


Note: For brevity, I've referred to "nested macros" above, but for our use
case we will actually need to handle all the different comment types
recognized in XWikiCommentHandler.java.


References:
AnnotatedXHTMLRenderer.java:
https://github.com/xwiki/xwiki-rendering/blob/master/xwiki-rendering-syntaxes/xwiki-rendering-syntax-annotatedxhtml/src/main/java/org/xwiki/rendering/internal/renderer/xhtml/AnnotatedXHTMLRenderer.java
XWikiCommentHandler.java:
https://github.com/xwiki/xwiki-rendering/blob/master/xwiki-rendering-syntaxes/xwiki-rendering-syntax-xhtml/src/main/java/org/xwiki/rendering/internal/parser/xhtml/wikimodel/XWikiCommentHandler.java#L90-L99



--
View this message in context: 
http://xwiki.475771.n2.nabble.com/Nesting-within-Annotated-XHTML-tp7603270.html
Sent from the XWiki- Dev mailing list archive at Nabble.com.

Re: [xwiki-devs] Page Complexity limiter

[ktc]

That extension is on the right track to what we would need, unfortunately it
only works for groovy.

I wasn't thinking that it would have to be at the JVM level but maybe at the
rendering context level or context level in general?  The context could then
can keep track of how deeply nested the inclusions are as well as keep track
of how long the request has been running.  These limits can then be checked
at critical times while rendering is occurring and then throw an Exception
to abort the process should a limit be broken.  It wouldn't necessarily have
to guarantee 100% accuracy in the first iteration of the feature, but a best
effort could be good for some protection.



--
View this message in context: 
http://xwiki.475771.n2.nabble.com/Page-Complexity-limiter-tp7602880p7602882.html
Sent from the XWiki- Dev mailing list archive at Nabble.com.

[xwiki-devs] Page Complexity limiter

[ktc]

Devs,

In MediaWiki there are  preprocessor limits
   that track how
complex a page is so that rendering can gracefully bail out if any of these
limits are broken to keep the server stable.  MediaWiki will include this
information in a comment in the rendered page like:
  

We have cases where our users will want to include pages as templates into
other pages and these can get pretty deeply nested as well as contain some
complex content that is expensive and/or takes a long time to render.  We
want to make sure that a user cannot bring our servers down accidentally or
intentionally due to using too many expensive/long running inclusions on
their pages.  Testing this on our instance of XWiki is showing that XWiki
will run until a) the page is finally able to render which could be minutes
in some cases or b) the server runs out of memory and falls over.

I have been looking but have been unable to find if XWiki has this sort of
feature to turn on and configure.  I was also unable to find it in the XWiki
Jira or in this forum so I wanted to ask the Devs directly.  Does this sort
of limiting exist on XWiki?  And if so, how can I turn it on?

Thanks!



--
View this message in context: 
http://xwiki.475771.n2.nabble.com/Page-Complexity-limiter-tp7602880.html
Sent from the XWiki- Dev mailing list archive at Nabble.com.