Re: memory visibility of @Reference service references

[Peter Kriens]

Late to the discussion but just want to shed some light on this. This 
discussion has been held before.

Volatile fields are not required for STATIC references because there is an 
implicit lock in the execution path from BundleContext.getService to the 
instance. Since this is the only way you can officially get an instance it is 
perfectly safe to not make the field volatile.

The spec prescribes or implies the following:

The activation is executed exactly once
No caller of getService can see the object before activate has finished

So what must happen under the hood

Tscr: Register Service Factory SF for S
Tscr: notify service event
T1 -> getService S
T2 -> getService S

Since the spec mandates that an activate for instance I must be run exactly 
once AND you cannot access an unactivated instance, SF MUST have a lock. Since 
T1 was first, it gets it, T2 is blocked

T1: get lock L -> pass
T2: get L -> block
T1: create instance I
T1: inject fields in I
T1: activate I
T1: unlock L
T2: get lock L -> pass
T2: get initialized instance I
T2: unlock L
T2: use I
T1: use I

Since I can only be reached through the getService call, which will call the 
ServiceFactory.getService, it is impossible for a thread Tx to not pass the 
lock in the Service Factory SF for S. Thus, T2 always sees I after T1 finished 
activating it. Ergo, there is no need to use volatile for a static variable.

Kind regards,

Peter Kriens

Re: memory visibility of @Reference service references

[Julian Sedding]

Hi Neil

On Tue, May 2, 2017 at 10:10 PM, Neil Bartlett  wrote:
> I think I get what Dirk is saying, and although I disagree with some of his 
> analysis there is arguably a gap in the DS specification that makes this 
> unclear. I hope somebody can point at the vital sentence I have missed, but 
> here is my analysis:
>
> The JLS defines happens-before thus: "Two actions can be ordered by a 
> happens-before relationship. If one action happens-before another, then the 
> first is visible to and ordered before the second.” 
> (https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5 
> ).
>
> Define two actions, A1 and A2. A1 is the setting of an @Reference field in a 
> component. A2 is reading the value of that field. It is Dirk’s contention 
> that no provable happens-before relationship edge exists between A1 and A2, 
> and therefore it’s possible for A2 to access the value of the @Reference 
> field that obtained before A1. Let’s see.
>
> A1 is an action that is taken by SCR on an object that it creates (the 
> component instance). A2 can only occur in the following circumstances:
>
> 1. SCR itself reads the field.
> SCR is in full control of its own code and I am assuming a correct SCR 
> implementation.
>
> 2. The activate method of the component uses the value of the field.
> Section 112.3.8.1 of OSGi Compendium R6 states "there is a happens-before 
> relationship between setting the field and activating the component 
> instance”. Also section 112.3.6.1 states “The bind method is called and/or 
> the field is set before the component instance is activated”, which is less 
> clear about the happens-before but I think the intent is there, and it also 
> clarifies that this applies to bind methods as well as injected fields. 
> Therefore action A1 happens-before A2 in this case.
>
> Note that the spec does not tell the implementation HOW to achieve this 
> happens-before edge. As Dirk points out, if it is done with a synchronized 
> block then it would have to be on the same monitor. For example, the 
> implementation can achieve this by synchronizing on the component instance 
> itself when binding fields and calling the activate method.
>
> 3. The activate method starts a thread which uses the value of the field (I 
> think this maps to Dirk’s HTTP-serving example).
> From point 2 above, we know that A1 happens-before the activate method is 
> invoked. JLS 17.4.5 states that “a call to start() on a thread happens-before 
> any actions in the started thread”. JLS also states that happens-before is 
> transitive, i.e. if X happens-before Y and Y happens-before Z then X 
> happens-before Z.
>
> Therefore A1 happens-before A2 in this case.
>
> 4. Service method(s) on the component use the value of the field.
> The analysis here diverges for immediate vs delayed components.
>
> For immediate components, Compendium section 112.5.3 states: "If an immediate 
> component configuration is satisfied and specifies a service, SCR must 
> register the component configuration as a service in the service registry and 
> then activate the component configuration”. This does NOT state that the 
> registration of the service must happen after the binding of the reference 
> fields. So it seems an implementation is free to hand the service object to a 
> consumer before the static references are bound. This surprised me.

Agreed, I am especially surprised at the order of "[...] register the
component configuration as a service in the service registry and then
activate the component configuration [...]". I would have expected it
is first activated and then registered as a service.

>
> For delayed components, Compendium section 112.5.4 states: "When the service 
> is requested … SCR must create and activate a [unique] component 
> configuration”. It does NOT explicitly state that the static references must 
> be bound before the service object is handed to the caller of getService(). 
> Again this surprised me.
>

I think "112.5.6 Activation" clarifies that "activation" means more
than just calling the activate method. It summarizes the following
steps with the term "activation":

1. Load the component implementation class.
2. Create the component instance and component context.
3. Bind the target services.
4. Call the activate method, if present.

So you could even consider section 112.5.4 a bit verbose, it could be
reduced to "... SCR must activate a [unique] component configuration"
;)

On a more serious note: I think due to the definition of "activation",
it is clear that all references must be set before the caller gets the
object.

Regards
Julian

>
> In summary, it’s clear to me that the *intention* of the specification is for 
> static references to be bound before the service becomes accessible, and that 
> component writers should not have to declare those fields volatile, but this 
> intention is 

Re: memory visibility of @Reference service references

[Julian Sedding]

Hi Dirk

Thanks for bringing up this discussion. I find it educational.

On Tue, May 2, 2017 at 10:27 PM, Dirk Hogan  wrote:
> Hi Neil-
> OK - now we are getting somewhere.
>
> You need to dig a bit deeper to see what establishes a happens-before
> relationship. For within a single thread, it is the Program Order: the
> underlying virtual machine cannot allow the visibility of memory effects
> which contradict the sequence of instructions defined in your .java file,
> even though these commands will be re-ordered by the compiler, and
> pipelined and executed concurrently by machine cores.
>
> Between threads, it is the Synchronization Order, which is NOT Program
> Order. Between threads, the memory effects of your .java file writes/reads
> do NOT have to be honored. So if you say 'But Felix writes the reference
> before publishing the service', and thus concluding that the reference will
> always be present in the service, you are correct, but ONLY from the
> perspective of the thread which wrote the reference and published the
> service. For ALL other threads, all bets are off, unless there is a
> Synchronization Action (volatile reference read/write, monitor lock/unlock
> on the same monitor) which dictates a Synchronization Order of the
> visibility of memory effects between DISTINCT threads.

I read some of your references and according to my understanding you
are disregarding the transitive nature of different orderings here.
Let me explain:
- assume we have a component C with a field f (annotated with @Reference)
- assume C is a delayed component providing a service
- assume the component is satisfied and its corresponding
ServiceFactory sf is registered with the service-registry
- assume no other thread has requested the service before
- thread T1 requests C from the service registry
- thread T2 requests C from the service registry (assume T1 wins the race)
  - T2: waits for a CountDownLatch (that's how it's implemented in Felix)
  - T1: SCR creates an instance of C called c
  - T1: SCR injects @Reference fields, i.e. it sets c.f
  - T1: SCR calls c.activate()
  - T1: c is published by setting it in a volatile field of the
ServiceFactory: sf.componentInstance
  - T1: "releases" the CountDownLatch allowing T2 to proceed
  - T2: reads c from sf.componentInstance
  - T2: reads c.f

All actions that happen in T1 are governed by Program Order. Setting
sf.componentInstance is visible across threads due to Synchronization
Order (it is a volatile field). Because actions in T1 are bound by PO,
the field sf.componentInstance is set AFTER c.f is set. T2 can only
get a reference to c AFTER sf.componentInstance is set. Therefore,
transitively, T2 can only get a reference to c AFTER c.f is set.

This is in line with my understanding of the following quote from
https://shipilev.net/blog/2014/jmm-pragmatics/, at the end of the
section "Happens-Before: Publication":

"Release can be thought of as the operation which releases all the
preceding state updates into the wild, and acquire is the paired
operation which receives those updates. So, after a successful
acquire, we can see all the updates preceding the paired release."

Regards
Julian

>
> The definition of Synchronization Actions, how they combine to create a
> Synchronization Order, and how this, in turn, allows a happens-before
> relationship to be established, is a formal set of rules defined in the
> JMM. There is no ambiguity here. A read of section 17.4 will illuminate as
> much. It is very dense - that is why I turned to the other references I
> provided in the links above. They are not a quick/easy read, but are
> necessary to coherently reason about memory visibility across threads.
>
> Thanks
>
> Dirk
>
> On Tue, May 2, 2017 at 1:10 PM, Neil Bartlett  wrote:
>
>> I think I get what Dirk is saying, and although I disagree with some of
>> his analysis there is arguably a gap in the DS specification that makes
>> this unclear. I hope somebody can point at the vital sentence I have
>> missed, but here is my analysis:
>>
>> The JLS defines happens-before thus: "Two actions can be ordered by a
>> happens-before relationship. If one action happens-before another, then the
>> first is visible to and ordered before the second.” (
>> https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5 <
>> https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5
>> >).
>>
>> Define two actions, A1 and A2. A1 is the setting of an @Reference field in
>> a component. A2 is reading the value of that field. It is Dirk’s contention
>> that no provable happens-before relationship edge exists between A1 and A2,
>> and therefore it’s possible for A2 to access the value of the @Reference
>> field that obtained before A1. Let’s see.
>>
>> A1 is an action that is taken by SCR on an object that it creates (the
>> component instance). A2 can only occur in the following circumstances:
>>
>> 1. SCR itself 

Re: memory visibility of @Reference service references

[Dirk Hogan]

I have no stomach for further discussion/argument.

Thanks for the responses guys.

Dirk

On Tue, May 2, 2017 at 4:50 PM, Richard S. Hall 
wrote:

> On 5/2/17 19:41 , Richard S. Hall wrote:
>
>> Resending to the mailing list...
>>
>> On 5/2/17 18:55 , Dirk Hogan wrote:
>>
>>> Hi Richard-
>>> I have a embedded jetty OSGi component that exposes the conglomeration of
>>> OSGi services to the outside world. Felix would need to set these service
>>> references in a thread-safe way. You appear to be suggesting that Felix
>>> can
>>> be only consumed by the Felix threads themselves, as only that would not
>>> bypass 'the hidden synchronization inside the framework' - whatever that
>>> is.
>>>
>>
>> I'm suggesting that if you pass objects to threads then you need to do so
>> in a thread safe way.
>>
>> Assume some something like:
>>
>> class FooThread {
>> private MyService service;
>> pubic void run() {
>> while (true) {
>> if (service != null) {
>> service.sayHello();
>> }
>> }
>> }
>> public void setService(MyService s) {
>> service = s;
>> }
>> }
>>
>> So lets assume that there is some thread spinning on run(). If in another
>> component you create a FooThread then get a service object from the service
>> registry and call myFooThread.setService() (where the MyService instance
>> contains your injected service), who is responsible for safely publishing
>> your service to your FooThread?
>>
>> Clearly, if you just call setService() you are not safely handing off the
>> service object to your thread (this is the equivalent of your shutdown flag
>> example). You are implying that it is the frameworks responsibility to
>> provide a barrier to FooThreads member service variable.
>>
>> Certainly, one solution to this is making your reference volatile, but
>> that it between you and FooThread, not between your service and the
>> framework.
>>
>
> Just some further elaboration and generalization, since the above may be
> confusing...
>
> The main point is you have some thread that you somehow have given access
> to an object and internally that object invokes some OSGi service. If you
> hand that object reference to your thread in a safe way, then there will be
> no problem.
>
> At the end of the day, the thread invoking this object had to be given the
> object reference by you, so you are responsible for giving it the reference
> using a barrier.
>
> -> richard
>
>
>
>> -> richard
>>
>>
>>> Let me try one more time: a while back, Felix mandated that DYNAMIC
>>> @References be volatile. Presumably that decision was made by someone who
>>> knows the JMM, and thus could provide a response to the question in my
>>> initial post:
>>>
>>> Perhaps an explanation as to why a DYNAMIC @Reference does need to be
>>> volatile would be helpful - e.g. why does component
>>> de-activation/re-activation provide memory visibility guarantees for
>>> STATIC
>>> references?
>>>
>>> I don't mean to be contentious, and there is nothing preventing me from
>>> declaring my @References volatile. I just was hoping that there was
>>> JMM-knowledge active in this forum which could provide an answer to my
>>> original question. I may well have my answer.
>>> Thanks
>>>
>>> Dirk
>>>
>>>
>>> On Tue, May 2, 2017 at 2:29 PM, Richard S. Hall 
>>> wrote:
>>>
>>> Dirk,

 Allow I'm not well versed on the details of JMM. I think this boils down
 to typical OSGi component code will be going through the hidden
 barriers in
 the the framework when they try to retrieve and access the service
 objects.
 You appear to be concocting a situation where you have threads from
 outside
 that you have given access to OSGi service components and bypassing the
 hidden synchronization inside the framework. If you can create such a
 scenario, then it is up to you to safely publish these objects when you
 hand them off to this other thread. And the only way this is really
 problematic is if your handoff is done in such a way that you bypass all
 barriers to begin with.

 -> richard



 On 5/2/17 16:27 , Dirk Hogan wrote:

 Hi Neil-
> OK - now we are getting somewhere.
>
> You need to dig a bit deeper to see what establishes a happens-before
> relationship. For within a single thread, it is the Program Order: the
> underlying virtual machine cannot allow the visibility of memory
> effects
> which contradict the sequence of instructions defined in your .java
> file,
> even though these commands will be re-ordered by the compiler, and
> pipelined and executed concurrently by machine cores.
>
> Between threads, it is the Synchronization Order, which is NOT Program
> Order. Between threads, the memory effects of your .java file
> writes/reads
> do NOT have to be honored. So if you say 'But Felix writes 

Re: memory visibility of @Reference service references

[Richard S. Hall]

On 5/2/17 19:41 , Richard S. Hall wrote:

Resending to the mailing list...

On 5/2/17 18:55 , Dirk Hogan wrote:

Hi Richard-
I have a embedded jetty OSGi component that exposes the 
conglomeration of
OSGi services to the outside world. Felix would need to set these 
service
references in a thread-safe way. You appear to be suggesting that 
Felix can

be only consumed by the Felix threads themselves, as only that would not
bypass 'the hidden synchronization inside the framework' - whatever 
that is.


I'm suggesting that if you pass objects to threads then you need to do 
so in a thread safe way.


Assume some something like:

class FooThread {
private MyService service;
pubic void run() {
while (true) {
if (service != null) {
service.sayHello();
}
}
}
public void setService(MyService s) {
service = s;
}
}

So lets assume that there is some thread spinning on run(). If in 
another component you create a FooThread then get a service object 
from the service registry and call myFooThread.setService() (where the 
MyService instance contains your injected service), who is responsible 
for safely publishing your service to your FooThread?


Clearly, if you just call setService() you are not safely handing off 
the service object to your thread (this is the equivalent of your 
shutdown flag example). You are implying that it is the frameworks 
responsibility to provide a barrier to FooThreads member service 
variable.


Certainly, one solution to this is making your reference volatile, but 
that it between you and FooThread, not between your service and the 
framework.


Just some further elaboration and generalization, since the above may be 
confusing...


The main point is you have some thread that you somehow have given 
access to an object and internally that object invokes some OSGi 
service. If you hand that object reference to your thread in a safe way, 
then there will be no problem.


At the end of the day, the thread invoking this object had to be given 
the object reference by you, so you are responsible for giving it the 
reference using a barrier.


-> richard



-> richard



Let me try one more time: a while back, Felix mandated that DYNAMIC
@References be volatile. Presumably that decision was made by someone 
who

knows the JMM, and thus could provide a response to the question in my
initial post:

Perhaps an explanation as to why a DYNAMIC @Reference does need to be
volatile would be helpful - e.g. why does component
de-activation/re-activation provide memory visibility guarantees for 
STATIC

references?

I don't mean to be contentious, and there is nothing preventing me from
declaring my @References volatile. I just was hoping that there was
JMM-knowledge active in this forum which could provide an answer to my
original question. I may well have my answer.
Thanks

Dirk


On Tue, May 2, 2017 at 2:29 PM, Richard S. Hall 
wrote:


Dirk,

Allow I'm not well versed on the details of JMM. I think this boils 
down
to typical OSGi component code will be going through the hidden 
barriers in
the the framework when they try to retrieve and access the service 
objects.
You appear to be concocting a situation where you have threads from 
outside

that you have given access to OSGi service components and bypassing the
hidden synchronization inside the framework. If you can create such a
scenario, then it is up to you to safely publish these objects when you
hand them off to this other thread. And the only way this is really
problematic is if your handoff is done in such a way that you bypass 
all

barriers to begin with.

-> richard



On 5/2/17 16:27 , Dirk Hogan wrote:


Hi Neil-
OK - now we are getting somewhere.

You need to dig a bit deeper to see what establishes a happens-before
relationship. For within a single thread, it is the Program Order: the
underlying virtual machine cannot allow the visibility of memory 
effects
which contradict the sequence of instructions defined in your .java 
file,

even though these commands will be re-ordered by the compiler, and
pipelined and executed concurrently by machine cores.

Between threads, it is the Synchronization Order, which is NOT Program
Order. Between threads, the memory effects of your .java file 
writes/reads
do NOT have to be honored. So if you say 'But Felix writes the 
reference

before publishing the service', and thus concluding that the reference
will
always be present in the service, you are correct, but ONLY from the
perspective of the thread which wrote the reference and published the
service. For ALL other threads, all bets are off, unless there is a
Synchronization Action (volatile reference read/write, monitor 
lock/unlock

on the same monitor) which dictates a Synchronization Order of the
visibility of memory effects between DISTINCT threads.

The definition of Synchronization Actions, how they combine to 
create a

Re: memory visibility of @Reference service references

[Richard S. Hall]

p.s. If you really want to make this a spec discussion vs an 
implementation issue, you could bring it up on the 
osgi-...@mail.osgi.org mailing list.


On 5/2/17 18:55 , Dirk Hogan wrote:

Hi Richard-
I have a embedded jetty OSGi component that exposes the conglomeration of
OSGi services to the outside world. Felix would need to set these service
references in a thread-safe way. You appear to be suggesting that Felix can
be only consumed by the Felix threads themselves, as only that would not
bypass 'the hidden synchronization inside the framework' - whatever that is.

Let me try one more time: a while back, Felix mandated that DYNAMIC
@References be volatile. Presumably that decision was made by someone who
knows the JMM, and thus could provide a response to the question in my
initial post:

Perhaps an explanation as to why a DYNAMIC @Reference does need to be
volatile would be helpful - e.g. why does component
de-activation/re-activation provide memory visibility guarantees for STATIC
references?

I don't mean to be contentious, and there is nothing preventing me from
declaring my @References volatile. I just was hoping that there was
JMM-knowledge active in this forum which could provide an answer to my
original question. I may well have my answer.

Thanks

Dirk


On Tue, May 2, 2017 at 2:29 PM, Richard S. Hall 
wrote:


Dirk,

Allow I'm not well versed on the details of JMM. I think this boils down
to typical OSGi component code will be going through the hidden barriers in
the the framework when they try to retrieve and access the service objects.
You appear to be concocting a situation where you have threads from outside
that you have given access to OSGi service components and bypassing the
hidden synchronization inside the framework. If you can create such a
scenario, then it is up to you to safely publish these objects when you
hand them off to this other thread. And the only way this is really
problematic is if your handoff is done in such a way that you bypass all
barriers to begin with.

-> richard



On 5/2/17 16:27 , Dirk Hogan wrote:


Hi Neil-
OK - now we are getting somewhere.

You need to dig a bit deeper to see what establishes a happens-before
relationship. For within a single thread, it is the Program Order: the
underlying virtual machine cannot allow the visibility of memory effects
which contradict the sequence of instructions defined in your .java file,
even though these commands will be re-ordered by the compiler, and
pipelined and executed concurrently by machine cores.

Between threads, it is the Synchronization Order, which is NOT Program
Order. Between threads, the memory effects of your .java file writes/reads
do NOT have to be honored. So if you say 'But Felix writes the reference
before publishing the service', and thus concluding that the reference
will
always be present in the service, you are correct, but ONLY from the
perspective of the thread which wrote the reference and published the
service. For ALL other threads, all bets are off, unless there is a
Synchronization Action (volatile reference read/write, monitor lock/unlock
on the same monitor) which dictates a Synchronization Order of the
visibility of memory effects between DISTINCT threads.

The definition of Synchronization Actions, how they combine to create a
Synchronization Order, and how this, in turn, allows a happens-before
relationship to be established, is a formal set of rules defined in the
JMM. There is no ambiguity here. A read of section 17.4 will illuminate as
much. It is very dense - that is why I turned to the other references I
provided in the links above. They are not a quick/easy read, but are
necessary to coherently reason about memory visibility across threads.

Thanks

Dirk

On Tue, May 2, 2017 at 1:10 PM, Neil Bartlett 
wrote:

I think I get what Dirk is saying, and although I disagree with some of

his analysis there is arguably a gap in the DS specification that makes
this unclear. I hope somebody can point at the vital sentence I have
missed, but here is my analysis:

The JLS defines happens-before thus: "Two actions can be ordered by a
happens-before relationship. If one action happens-before another, then
the
first is visible to and ordered before the second.” (
https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5
<
https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5


).


Define two actions, A1 and A2. A1 is the setting of an @Reference field
in
a component. A2 is reading the value of that field. It is Dirk’s
contention
that no provable happens-before relationship edge exists between A1 and
A2,
and therefore it’s possible for A2 to access the value of the @Reference
field that obtained before A1. Let’s see.

A1 is an action that is taken by SCR on an object that it creates (the
component instance). A2 can only occur in the following circumstances:

1. SCR itself reads the field.
SCR is in full 

Re: memory visibility of @Reference service references

[Richard S. Hall]

Resending to the mailing list...

On 5/2/17 18:55 , Dirk Hogan wrote:

Hi Richard-
I have a embedded jetty OSGi component that exposes the conglomeration of
OSGi services to the outside world. Felix would need to set these service
references in a thread-safe way. You appear to be suggesting that Felix can
be only consumed by the Felix threads themselves, as only that would not
bypass 'the hidden synchronization inside the framework' - whatever that is.


I'm suggesting that if you pass objects to threads then you need to do 
so in a thread safe way.


Assume some something like:

class FooThread {
private MyService service;
pubic void run() {
while (true) {
if (service != null) {
service.sayHello();
}
}
}
public void setService(MyService s) {
service = s;
}
}

So lets assume that there is some thread spinning on run(). If in 
another component you create a FooThread then get a service object from 
the service registry and call myFooThread.setService() (where the 
MyService instance contains your injected service), who is responsible 
for safely publishing your service to your FooThread?


Clearly, if you just call setService() you are not safely handing off 
the service object to your thread (this is the equivalent of your 
shutdown flag example). You are implying that it is the frameworks 
responsibility to provide a barrier to FooThreads member service variable.


Certainly, one solution to this is making your reference volatile, but 
that it between you and FooThread, not between your service and the 
framework.


-> richard



Let me try one more time: a while back, Felix mandated that DYNAMIC
@References be volatile. Presumably that decision was made by someone who
knows the JMM, and thus could provide a response to the question in my
initial post:

Perhaps an explanation as to why a DYNAMIC @Reference does need to be
volatile would be helpful - e.g. why does component
de-activation/re-activation provide memory visibility guarantees for STATIC
references?

I don't mean to be contentious, and there is nothing preventing me from
declaring my @References volatile. I just was hoping that there was
JMM-knowledge active in this forum which could provide an answer to my
original question. I may well have my answer.
Thanks

Dirk


On Tue, May 2, 2017 at 2:29 PM, Richard S. Hall 
wrote:


Dirk,

Allow I'm not well versed on the details of JMM. I think this boils down
to typical OSGi component code will be going through the hidden barriers in
the the framework when they try to retrieve and access the service objects.
You appear to be concocting a situation where you have threads from outside
that you have given access to OSGi service components and bypassing the
hidden synchronization inside the framework. If you can create such a
scenario, then it is up to you to safely publish these objects when you
hand them off to this other thread. And the only way this is really
problematic is if your handoff is done in such a way that you bypass all
barriers to begin with.

-> richard



On 5/2/17 16:27 , Dirk Hogan wrote:


Hi Neil-
OK - now we are getting somewhere.

You need to dig a bit deeper to see what establishes a happens-before
relationship. For within a single thread, it is the Program Order: the
underlying virtual machine cannot allow the visibility of memory effects
which contradict the sequence of instructions defined in your .java file,
even though these commands will be re-ordered by the compiler, and
pipelined and executed concurrently by machine cores.

Between threads, it is the Synchronization Order, which is NOT Program
Order. Between threads, the memory effects of your .java file writes/reads
do NOT have to be honored. So if you say 'But Felix writes the reference
before publishing the service', and thus concluding that the reference
will
always be present in the service, you are correct, but ONLY from the
perspective of the thread which wrote the reference and published the
service. For ALL other threads, all bets are off, unless there is a
Synchronization Action (volatile reference read/write, monitor lock/unlock
on the same monitor) which dictates a Synchronization Order of the
visibility of memory effects between DISTINCT threads.

The definition of Synchronization Actions, how they combine to create a
Synchronization Order, and how this, in turn, allows a happens-before
relationship to be established, is a formal set of rules defined in the
JMM. There is no ambiguity here. A read of section 17.4 will illuminate as
much. It is very dense - that is why I turned to the other references I
provided in the links above. They are not a quick/easy read, but are
necessary to coherently reason about memory visibility across threads.

Thanks

Dirk

On Tue, May 2, 2017 at 1:10 PM, Neil Bartlett 
wrote:

I think I get what Dirk is saying, and although I disagree with some of

his 

Re: memory visibility of @Reference service references

[Dirk Hogan]

Hi Richard-
I have a embedded jetty OSGi component that exposes the conglomeration of
OSGi services to the outside world. Felix would need to set these service
references in a thread-safe way. You appear to be suggesting that Felix can
be only consumed by the Felix threads themselves, as only that would not
bypass 'the hidden synchronization inside the framework' - whatever that is.

Let me try one more time: a while back, Felix mandated that DYNAMIC
@References be volatile. Presumably that decision was made by someone who
knows the JMM, and thus could provide a response to the question in my
initial post:

Perhaps an explanation as to why a DYNAMIC @Reference does need to be
volatile would be helpful - e.g. why does component
de-activation/re-activation provide memory visibility guarantees for STATIC
references?

I don't mean to be contentious, and there is nothing preventing me from
declaring my @References volatile. I just was hoping that there was
JMM-knowledge active in this forum which could provide an answer to my
original question. I may well have my answer.

Thanks

Dirk


On Tue, May 2, 2017 at 2:29 PM, Richard S. Hall 
wrote:

> Dirk,
>
> Allow I'm not well versed on the details of JMM. I think this boils down
> to typical OSGi component code will be going through the hidden barriers in
> the the framework when they try to retrieve and access the service objects.
> You appear to be concocting a situation where you have threads from outside
> that you have given access to OSGi service components and bypassing the
> hidden synchronization inside the framework. If you can create such a
> scenario, then it is up to you to safely publish these objects when you
> hand them off to this other thread. And the only way this is really
> problematic is if your handoff is done in such a way that you bypass all
> barriers to begin with.
>
> -> richard
>
>
>
> On 5/2/17 16:27 , Dirk Hogan wrote:
>
>> Hi Neil-
>> OK - now we are getting somewhere.
>>
>> You need to dig a bit deeper to see what establishes a happens-before
>> relationship. For within a single thread, it is the Program Order: the
>> underlying virtual machine cannot allow the visibility of memory effects
>> which contradict the sequence of instructions defined in your .java file,
>> even though these commands will be re-ordered by the compiler, and
>> pipelined and executed concurrently by machine cores.
>>
>> Between threads, it is the Synchronization Order, which is NOT Program
>> Order. Between threads, the memory effects of your .java file writes/reads
>> do NOT have to be honored. So if you say 'But Felix writes the reference
>> before publishing the service', and thus concluding that the reference
>> will
>> always be present in the service, you are correct, but ONLY from the
>> perspective of the thread which wrote the reference and published the
>> service. For ALL other threads, all bets are off, unless there is a
>> Synchronization Action (volatile reference read/write, monitor lock/unlock
>> on the same monitor) which dictates a Synchronization Order of the
>> visibility of memory effects between DISTINCT threads.
>>
>> The definition of Synchronization Actions, how they combine to create a
>> Synchronization Order, and how this, in turn, allows a happens-before
>> relationship to be established, is a formal set of rules defined in the
>> JMM. There is no ambiguity here. A read of section 17.4 will illuminate as
>> much. It is very dense - that is why I turned to the other references I
>> provided in the links above. They are not a quick/easy read, but are
>> necessary to coherently reason about memory visibility across threads.
>>
>> Thanks
>>
>> Dirk
>>
>> On Tue, May 2, 2017 at 1:10 PM, Neil Bartlett 
>> wrote:
>>
>> I think I get what Dirk is saying, and although I disagree with some of
>>> his analysis there is arguably a gap in the DS specification that makes
>>> this unclear. I hope somebody can point at the vital sentence I have
>>> missed, but here is my analysis:
>>>
>>> The JLS defines happens-before thus: "Two actions can be ordered by a
>>> happens-before relationship. If one action happens-before another, then
>>> the
>>> first is visible to and ordered before the second.” (
>>> https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5
>>> <
>>> https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5
>>>
 ).

>>> Define two actions, A1 and A2. A1 is the setting of an @Reference field
>>> in
>>> a component. A2 is reading the value of that field. It is Dirk’s
>>> contention
>>> that no provable happens-before relationship edge exists between A1 and
>>> A2,
>>> and therefore it’s possible for A2 to access the value of the @Reference
>>> field that obtained before A1. Let’s see.
>>>
>>> A1 is an action that is taken by SCR on an object that it creates (the
>>> component instance). A2 can only occur in the following circumstances:
>>>
>>> 1. 

Re: memory visibility of @Reference service references

[Richard S. Hall]

Dirk,

Allow I'm not well versed on the details of JMM. I think this boils down 
to typical OSGi component code will be going through the hidden barriers 
in the the framework when they try to retrieve and access the service 
objects. You appear to be concocting a situation where you have threads 
from outside that you have given access to OSGi service components and 
bypassing the hidden synchronization inside the framework. If you can 
create such a scenario, then it is up to you to safely publish these 
objects when you hand them off to this other thread. And the only way 
this is really problematic is if your handoff is done in such a way that 
you bypass all barriers to begin with.


-> richard


On 5/2/17 16:27 , Dirk Hogan wrote:

Hi Neil-
OK - now we are getting somewhere.

You need to dig a bit deeper to see what establishes a happens-before
relationship. For within a single thread, it is the Program Order: the
underlying virtual machine cannot allow the visibility of memory effects
which contradict the sequence of instructions defined in your .java file,
even though these commands will be re-ordered by the compiler, and
pipelined and executed concurrently by machine cores.

Between threads, it is the Synchronization Order, which is NOT Program
Order. Between threads, the memory effects of your .java file writes/reads
do NOT have to be honored. So if you say 'But Felix writes the reference
before publishing the service', and thus concluding that the reference will
always be present in the service, you are correct, but ONLY from the
perspective of the thread which wrote the reference and published the
service. For ALL other threads, all bets are off, unless there is a
Synchronization Action (volatile reference read/write, monitor lock/unlock
on the same monitor) which dictates a Synchronization Order of the
visibility of memory effects between DISTINCT threads.

The definition of Synchronization Actions, how they combine to create a
Synchronization Order, and how this, in turn, allows a happens-before
relationship to be established, is a formal set of rules defined in the
JMM. There is no ambiguity here. A read of section 17.4 will illuminate as
much. It is very dense - that is why I turned to the other references I
provided in the links above. They are not a quick/easy read, but are
necessary to coherently reason about memory visibility across threads.

Thanks

Dirk

On Tue, May 2, 2017 at 1:10 PM, Neil Bartlett  wrote:


I think I get what Dirk is saying, and although I disagree with some of
his analysis there is arguably a gap in the DS specification that makes
this unclear. I hope somebody can point at the vital sentence I have
missed, but here is my analysis:

The JLS defines happens-before thus: "Two actions can be ordered by a
happens-before relationship. If one action happens-before another, then the
first is visible to and ordered before the second.” (
https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5 <
https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5

).

Define two actions, A1 and A2. A1 is the setting of an @Reference field in
a component. A2 is reading the value of that field. It is Dirk’s contention
that no provable happens-before relationship edge exists between A1 and A2,
and therefore it’s possible for A2 to access the value of the @Reference
field that obtained before A1. Let’s see.

A1 is an action that is taken by SCR on an object that it creates (the
component instance). A2 can only occur in the following circumstances:

1. SCR itself reads the field.
SCR is in full control of its own code and I am assuming a correct SCR
implementation.

2. The activate method of the component uses the value of the field.
Section 112.3.8.1 of OSGi Compendium R6 states "there is a happens-before
relationship between setting the field and activating the component
instance”. Also section 112.3.6.1 states “The bind method is called and/or
the field is set before the component instance is activated”, which is less
clear about the happens-before but I think the intent is there, and it also
clarifies that this applies to bind methods as well as injected fields.
Therefore action A1 happens-before A2 in this case.

Note that the spec does not tell the implementation HOW to achieve this
happens-before edge. As Dirk points out, if it is done with a synchronized
block then it would have to be on the same monitor. For example, the
implementation can achieve this by synchronizing on the component instance
itself when binding fields and calling the activate method.

3. The activate method starts a thread which uses the value of the field
(I think this maps to Dirk’s HTTP-serving example).
 From point 2 above, we know that A1 happens-before the activate method is
invoked. JLS 17.4.5 states that “a call to start() on a thread
happens-before any actions in the started thread”. JLS also states that
happens-before is transitive, i.e. if X 

Re: memory visibility of @Reference service references

[Dirk Hogan]

Hi Neil-
OK - now we are getting somewhere.

You need to dig a bit deeper to see what establishes a happens-before
relationship. For within a single thread, it is the Program Order: the
underlying virtual machine cannot allow the visibility of memory effects
which contradict the sequence of instructions defined in your .java file,
even though these commands will be re-ordered by the compiler, and
pipelined and executed concurrently by machine cores.

Between threads, it is the Synchronization Order, which is NOT Program
Order. Between threads, the memory effects of your .java file writes/reads
do NOT have to be honored. So if you say 'But Felix writes the reference
before publishing the service', and thus concluding that the reference will
always be present in the service, you are correct, but ONLY from the
perspective of the thread which wrote the reference and published the
service. For ALL other threads, all bets are off, unless there is a
Synchronization Action (volatile reference read/write, monitor lock/unlock
on the same monitor) which dictates a Synchronization Order of the
visibility of memory effects between DISTINCT threads.

The definition of Synchronization Actions, how they combine to create a
Synchronization Order, and how this, in turn, allows a happens-before
relationship to be established, is a formal set of rules defined in the
JMM. There is no ambiguity here. A read of section 17.4 will illuminate as
much. It is very dense - that is why I turned to the other references I
provided in the links above. They are not a quick/easy read, but are
necessary to coherently reason about memory visibility across threads.

Thanks

Dirk

On Tue, May 2, 2017 at 1:10 PM, Neil Bartlett  wrote:

> I think I get what Dirk is saying, and although I disagree with some of
> his analysis there is arguably a gap in the DS specification that makes
> this unclear. I hope somebody can point at the vital sentence I have
> missed, but here is my analysis:
>
> The JLS defines happens-before thus: "Two actions can be ordered by a
> happens-before relationship. If one action happens-before another, then the
> first is visible to and ordered before the second.” (
> https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5 <
> https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5
> >).
>
> Define two actions, A1 and A2. A1 is the setting of an @Reference field in
> a component. A2 is reading the value of that field. It is Dirk’s contention
> that no provable happens-before relationship edge exists between A1 and A2,
> and therefore it’s possible for A2 to access the value of the @Reference
> field that obtained before A1. Let’s see.
>
> A1 is an action that is taken by SCR on an object that it creates (the
> component instance). A2 can only occur in the following circumstances:
>
> 1. SCR itself reads the field.
> SCR is in full control of its own code and I am assuming a correct SCR
> implementation.
>
> 2. The activate method of the component uses the value of the field.
> Section 112.3.8.1 of OSGi Compendium R6 states "there is a happens-before
> relationship between setting the field and activating the component
> instance”. Also section 112.3.6.1 states “The bind method is called and/or
> the field is set before the component instance is activated”, which is less
> clear about the happens-before but I think the intent is there, and it also
> clarifies that this applies to bind methods as well as injected fields.
> Therefore action A1 happens-before A2 in this case.
>
> Note that the spec does not tell the implementation HOW to achieve this
> happens-before edge. As Dirk points out, if it is done with a synchronized
> block then it would have to be on the same monitor. For example, the
> implementation can achieve this by synchronizing on the component instance
> itself when binding fields and calling the activate method.
>
> 3. The activate method starts a thread which uses the value of the field
> (I think this maps to Dirk’s HTTP-serving example).
> From point 2 above, we know that A1 happens-before the activate method is
> invoked. JLS 17.4.5 states that “a call to start() on a thread
> happens-before any actions in the started thread”. JLS also states that
> happens-before is transitive, i.e. if X happens-before Y and Y
> happens-before Z then X happens-before Z.
>
> Therefore A1 happens-before A2 in this case.
>
> 4. Service method(s) on the component use the value of the field.
> The analysis here diverges for immediate vs delayed components.
>
> For immediate components, Compendium section 112.5.3 states: "If an
> immediate component configuration is satisfied and specifies a service, SCR
> must register the component configuration as a service in the service
> registry and then activate the component configuration”. This does NOT
> state that the registration of the service must happen after the binding of
> the reference fields. So it seems an 

Re: memory visibility of @Reference service references

[Neil Bartlett]

I think I get what Dirk is saying, and although I disagree with some of his 
analysis there is arguably a gap in the DS specification that makes this 
unclear. I hope somebody can point at the vital sentence I have missed, but 
here is my analysis:

The JLS defines happens-before thus: "Two actions can be ordered by a 
happens-before relationship. If one action happens-before another, then the 
first is visible to and ordered before the second.” 
(https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.4.5 
).

Define two actions, A1 and A2. A1 is the setting of an @Reference field in a 
component. A2 is reading the value of that field. It is Dirk’s contention that 
no provable happens-before relationship edge exists between A1 and A2, and 
therefore it’s possible for A2 to access the value of the @Reference field that 
obtained before A1. Let’s see.

A1 is an action that is taken by SCR on an object that it creates (the 
component instance). A2 can only occur in the following circumstances:

1. SCR itself reads the field.
SCR is in full control of its own code and I am assuming a correct SCR 
implementation.

2. The activate method of the component uses the value of the field.
Section 112.3.8.1 of OSGi Compendium R6 states "there is a happens-before 
relationship between setting the field and activating the component instance”. 
Also section 112.3.6.1 states “The bind method is called and/or the field is 
set before the component instance is activated”, which is less clear about the 
happens-before but I think the intent is there, and it also clarifies that this 
applies to bind methods as well as injected fields. Therefore action A1 
happens-before A2 in this case.

Note that the spec does not tell the implementation HOW to achieve this 
happens-before edge. As Dirk points out, if it is done with a synchronized 
block then it would have to be on the same monitor. For example, the 
implementation can achieve this by synchronizing on the component instance 
itself when binding fields and calling the activate method.

3. The activate method starts a thread which uses the value of the field (I 
think this maps to Dirk’s HTTP-serving example).
From point 2 above, we know that A1 happens-before the activate method is 
invoked. JLS 17.4.5 states that “a call to start() on a thread happens-before 
any actions in the started thread”. JLS also states that happens-before is 
transitive, i.e. if X happens-before Y and Y happens-before Z then X 
happens-before Z.

Therefore A1 happens-before A2 in this case.

4. Service method(s) on the component use the value of the field.
The analysis here diverges for immediate vs delayed components.

For immediate components, Compendium section 112.5.3 states: "If an immediate 
component configuration is satisfied and specifies a service, SCR must register 
the component configuration as a service in the service registry and then 
activate the component configuration”. This does NOT state that the 
registration of the service must happen after the binding of the reference 
fields. So it seems an implementation is free to hand the service object to a 
consumer before the static references are bound. This surprised me.

For delayed components, Compendium section 112.5.4 states: "When the service is 
requested … SCR must create and activate a [unique] component configuration”. 
It does NOT explicitly state that the static references must be bound before 
the service object is handed to the caller of getService(). Again this 
surprised me.


In summary, it’s clear to me that the *intention* of the specification is for 
static references to be bound before the service becomes accessible, and that 
component writers should not have to declare those fields volatile, but this 
intention is not adequately spelled out. The best statement I could find is in 
section 112.3.6.1: "A component instance never sees any of the dynamics of the 
static reference”, which is very vague.

Again, I may well have missed something or made an error in my analysis. I look 
forward to be being corrected.


Neil


> On 2 May 2017, at 19:56, David Jencks  wrote:
> 
> I don’t understand your example or I don’t understand how your example 
> relates to an osgi or ds problem.
> 
> I think you are proposing that the DS component is an http server and starts 
> threads etc.  If so, it’s that components responsibility to make sure its’ 
> internal operations are thread safe, and this has little to do with osgi or 
> ds.
> 
> If you are asking about osgi, for instance if the component exposes 
> HttpServlet and is registered with the whiteboard http service and receives 
> calls from it, then the whiteboard had to call getService to get the DS 
> component instance. In this case I think that there are plenty of 
> synchronization barriers in the service registry and in e.g. 
> 

Re: memory visibility of @Reference service references

[Dirk Hogan]

Hi David-
I am asserting that unless the @Reference is volatile, regardless of
whether it is STATIC or DYNAMIC, a data race is present.

The JMM states synchronization barriers on DISTINCT monitors have no memory
effects for the references read/written within the synchronization blocks.
That is stated in the first bullet point under section 17.4.4 here:
http://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4

Here's an example demonstrating that fallacy:
https://shipilev.net/blog/2016/close-encounters-of-jmm-kind/#wishful-unobserved-sync

The inter-thread memory visibility of writes/reads are guaranteed only if
performed when synchronized on the SAME monitor. Seeing as Felix does not
expose some global monitor to synchronize on in order to guarantee the
visibility of the @References it has written, looking at the code to
determine the presence of 'enough synchronization barriers' does not seem
to make sense.

Thanks

Dirk






On Tue, May 2, 2017 at 11:56 AM, David Jencks 
wrote:

> I don’t understand your example or I don’t understand how your example
> relates to an osgi or ds problem.
>
> I think you are proposing that the DS component is an http server and
> starts threads etc.  If so, it’s that components responsibility to make
> sure its’ internal operations are thread safe, and this has little to do
> with osgi or ds.
>
> If you are asking about osgi, for instance if the component exposes
> HttpServlet and is registered with the whiteboard http service and receives
> calls from it, then the whiteboard had to call getService to get the DS
> component instance. In this case I think that there are plenty of
> synchronization barriers in the service registry and in e.g.
> SingleComponentManager.getService to assure that the view a request
> thread has of the component is the same as the view the activation thread
> had.
>
> I could be wrong about whether there are actually enough synchronization
> barriers.  Have you looked at the code to see?
>
> thanks
> david jencks
>
> > On May 2, 2017, at 10:45 AM, Dirk Hogan 
> wrote:
> >
> > Hi Raymond-
> > Assume Felix does it's thing: sets all @References, activates all
> > components, publishes all services. It has resolved all dependencies, and
> > set references accordingly.
> >
> > Thus, in the example above, myOSGiComponent has its @Reference to
> > SomeOtherOSGiService resolved - Felix has set this reference. At the
> > conclusion of the activate method, myOSGiComponent has exposed some
> > functionality via http, even successfully dereferenced myOSGiComponent,
> and
> > now requests start to come it, serviced by the http-server's thread-pool.
> > Part of servicing this request involves dereferencing (reading) the
> > @Reference myServiceReference, which, I agree has been set (written) by a
> > Felix thread.
> >
> > The point of the Java Memory Model is that memory state, including
> > references, read and written across threads, do not have to have
> > determinate state, unless there is a Synchronization Action which ensures
> > that the state written by thread A is visible to any other thread - a
> > volatile reference provides such a Synchronization Action. So code in
> > myOSGiComponent activate method can successfully dereference the
> > SomeOtherOSGiService reference, because the same thread which set the
> > reference is calling activate. But this does NOT mean that another
> thread,
> > reading the SAME reference, will see a properly initialized reference.
> >
> > Again, this should be self-evident, and the various rules of the JMM
> model
> > are defined in section 17.4 of the Java Language Specification. If it is
> > not clear, then an understanding if the JMM is required. This
> understanding
> > can be gained by reading the JLS, reading the links above, the chapter
> > in *Concurrency
> > in Practice* on the JMM,  or http://gee.cs.oswego.edu/dl/cpj/jmm.html
> > provides a good intro.
> >
> > The bottom line: memory state, including references, shared among
> multiple
> > threads can have indeterminate state when written/read, unless explicit
> > steps, defined in the JMM, are taken to ensure this visibility and thus
> > coherent state. This is not an opinion.
> >
> > Thanks
> >
> > Dirk
> >
> >
> >
> > On Tue, May 2, 2017 at 10:23 AM, Raymond Auge 
> > wrote:
> >
> >> On Tue, May 2, 2017 at 1:18 PM, Raymond Auge 
> >> wrote:
> >>
> >>>
> >>>
> >>> On Tue, May 2, 2017 at 1:14 PM, Raymond Auge  >
> >>> wrote:
> >>>
> 
>  On Tue, May 2, 2017 at 11:46 AM, Dirk Hogan  >
>  wrote:
> 
> > But no similar guarantee applies when another thread hits the
> > component,
> >
> 
>  I believe the discussion boils down to why you believe the above is
>  possible?
> 
>  If SCR has not yet made the component available to other threads
> (which,

Re: memory visibility of @Reference service references

[David Jencks]

I don’t understand your example or I don’t understand how your example relates 
to an osgi or ds problem.

I think you are proposing that the DS component is an http server and starts 
threads etc.  If so, it’s that components responsibility to make sure its’ 
internal operations are thread safe, and this has little to do with osgi or ds.

If you are asking about osgi, for instance if the component exposes HttpServlet 
and is registered with the whiteboard http service and receives calls from it, 
then the whiteboard had to call getService to get the DS component instance. In 
this case I think that there are plenty of synchronization barriers in the 
service registry and in e.g. SingleComponentManager.getService to assure that 
the view a request thread has of the component is the same as the view the 
activation thread had.

I could be wrong about whether there are actually enough synchronization 
barriers.  Have you looked at the code to see?

thanks
david jencks

> On May 2, 2017, at 10:45 AM, Dirk Hogan  wrote:
> 
> Hi Raymond-
> Assume Felix does it's thing: sets all @References, activates all
> components, publishes all services. It has resolved all dependencies, and
> set references accordingly.
> 
> Thus, in the example above, myOSGiComponent has its @Reference to
> SomeOtherOSGiService resolved - Felix has set this reference. At the
> conclusion of the activate method, myOSGiComponent has exposed some
> functionality via http, even successfully dereferenced myOSGiComponent, and
> now requests start to come it, serviced by the http-server's thread-pool.
> Part of servicing this request involves dereferencing (reading) the
> @Reference myServiceReference, which, I agree has been set (written) by a
> Felix thread.
> 
> The point of the Java Memory Model is that memory state, including
> references, read and written across threads, do not have to have
> determinate state, unless there is a Synchronization Action which ensures
> that the state written by thread A is visible to any other thread - a
> volatile reference provides such a Synchronization Action. So code in
> myOSGiComponent activate method can successfully dereference the
> SomeOtherOSGiService reference, because the same thread which set the
> reference is calling activate. But this does NOT mean that another thread,
> reading the SAME reference, will see a properly initialized reference.
> 
> Again, this should be self-evident, and the various rules of the JMM model
> are defined in section 17.4 of the Java Language Specification. If it is
> not clear, then an understanding if the JMM is required. This understanding
> can be gained by reading the JLS, reading the links above, the chapter
> in *Concurrency
> in Practice* on the JMM,  or http://gee.cs.oswego.edu/dl/cpj/jmm.html
> provides a good intro.
> 
> The bottom line: memory state, including references, shared among multiple
> threads can have indeterminate state when written/read, unless explicit
> steps, defined in the JMM, are taken to ensure this visibility and thus
> coherent state. This is not an opinion.
> 
> Thanks
> 
> Dirk
> 
> 
> 
> On Tue, May 2, 2017 at 10:23 AM, Raymond Auge 
> wrote:
> 
>> On Tue, May 2, 2017 at 1:18 PM, Raymond Auge 
>> wrote:
>> 
>>> 
>>> 
>>> On Tue, May 2, 2017 at 1:14 PM, Raymond Auge 
>>> wrote:
>>> 
 
 On Tue, May 2, 2017 at 11:46 AM, Dirk Hogan 
 wrote:
 
> But no similar guarantee applies when another thread hits the
> component,
> 
 
 I believe the discussion boils down to why you believe the above is
 possible?
 
 If SCR has not yet made the component available to other threads (which,
 to my knowledge, it only does by registering it as a service to the
>> service
 registry) how can _another_ thread access the component at all?
 
 I don't even see a way for components within the same DS bundle to
 interact with each other other than via the service registry because
 @Reference only works through the registry.
 
>>> 
>>> Correction, components could interact with each other via the
>>> ComponentContext.. but I guess we'd have to know if that's part of your
>> use
>>> case.
>>> 
>> 
>> Actually, no, it's not even possible via the ComponentContext. It only
>> allows access to Services.
>> 
>>> 
>>> - Ray
>>> 
>>> 
 
 Please, can you explain in more detail what case might allow for the
 above? Because I believe the assumption is that it's not currently
 possible. Certainly there could be a bug and I'm certain everyone would
 want to see that fixed.
 
 --
 *Raymond Augé* 
 (@rotty3000)
 Senior Software Architect *Liferay, Inc.* 
 (@Liferay)
 Board Member & EEG Co-Chair, OSGi Alliance 
 (@OSGiAlliance)

Re: memory visibility of @Reference service references

[Dirk Hogan]

Hi Raymond-
Assume Felix does it's thing: sets all @References, activates all
components, publishes all services. It has resolved all dependencies, and
set references accordingly.

Thus, in the example above, myOSGiComponent has its @Reference to
SomeOtherOSGiService resolved - Felix has set this reference. At the
conclusion of the activate method, myOSGiComponent has exposed some
functionality via http, even successfully dereferenced myOSGiComponent, and
now requests start to come it, serviced by the http-server's thread-pool.
Part of servicing this request involves dereferencing (reading) the
@Reference myServiceReference, which, I agree has been set (written) by a
Felix thread.

The point of the Java Memory Model is that memory state, including
references, read and written across threads, do not have to have
determinate state, unless there is a Synchronization Action which ensures
that the state written by thread A is visible to any other thread - a
volatile reference provides such a Synchronization Action. So code in
myOSGiComponent activate method can successfully dereference the
SomeOtherOSGiService reference, because the same thread which set the
reference is calling activate. But this does NOT mean that another thread,
reading the SAME reference, will see a properly initialized reference.

Again, this should be self-evident, and the various rules of the JMM model
are defined in section 17.4 of the Java Language Specification. If it is
not clear, then an understanding if the JMM is required. This understanding
can be gained by reading the JLS, reading the links above, the chapter
in *Concurrency
in Practice* on the JMM,  or http://gee.cs.oswego.edu/dl/cpj/jmm.html
provides a good intro.

The bottom line: memory state, including references, shared among multiple
threads can have indeterminate state when written/read, unless explicit
steps, defined in the JMM, are taken to ensure this visibility and thus
coherent state. This is not an opinion.

Thanks

Dirk



On Tue, May 2, 2017 at 10:23 AM, Raymond Auge 
wrote:

> On Tue, May 2, 2017 at 1:18 PM, Raymond Auge 
> wrote:
>
> >
> >
> > On Tue, May 2, 2017 at 1:14 PM, Raymond Auge 
> > wrote:
> >
> >>
> >> On Tue, May 2, 2017 at 11:46 AM, Dirk Hogan 
> >> wrote:
> >>
> >>> But no similar guarantee applies when another thread hits the
> >>> component,
> >>>
> >>
> >> I believe the discussion boils down to why you believe the above is
> >> possible?
> >>
> >> If SCR has not yet made the component available to other threads (which,
> >> to my knowledge, it only does by registering it as a service to the
> service
> >> registry) how can _another_ thread access the component at all?
> >>
> >> I don't even see a way for components within the same DS bundle to
> >> interact with each other other than via the service registry because
> >> @Reference only works through the registry.
> >>
> >
> > Correction, components could interact with each other via the
> > ComponentContext.. but I guess we'd have to know if that's part of your
> use
> > case.
> >
>
> Actually, no, it's not even possible via the ComponentContext. It only
> allows access to Services.
>
> >
> > - Ray
> >
> >
> >>
> >> Please, can you explain in more detail what case might allow for the
> >> above? Because I believe the assumption is that it's not currently
> >> possible. Certainly there could be a bug and I'm certain everyone would
> >> want to see that fixed.
> >>
> >> --
> >> *Raymond Augé* 
> >>  (@rotty3000)
> >> Senior Software Architect *Liferay, Inc.* 
> >>  (@Liferay)
> >> Board Member & EEG Co-Chair, OSGi Alliance 
> >> (@OSGiAlliance)
> >>
> >
> >
> >
> > --
> > *Raymond Augé* 
> >  (@rotty3000)
> > Senior Software Architect *Liferay, Inc.* 
> >  (@Liferay)
> > Board Member & EEG Co-Chair, OSGi Alliance 
> > (@OSGiAlliance)
> >
>
>
>
> --
> *Raymond Augé* 
>  (@rotty3000)
> Senior Software Architect *Liferay, Inc.* 
>  (@Liferay)
> Board Member & EEG Co-Chair, OSGi Alliance 
> (@OSGiAlliance)
>

Re: memory visibility of @Reference service references

[Raymond Auge]

On Tue, May 2, 2017 at 1:18 PM, Raymond Auge 
wrote:

>
>
> On Tue, May 2, 2017 at 1:14 PM, Raymond Auge 
> wrote:
>
>>
>> On Tue, May 2, 2017 at 11:46 AM, Dirk Hogan 
>> wrote:
>>
>>> But no similar guarantee applies when another thread hits the
>>> component,
>>>
>>
>> I believe the discussion boils down to why you believe the above is
>> possible?
>>
>> If SCR has not yet made the component available to other threads (which,
>> to my knowledge, it only does by registering it as a service to the service
>> registry) how can _another_ thread access the component at all?
>>
>> I don't even see a way for components within the same DS bundle to
>> interact with each other other than via the service registry because
>> @Reference only works through the registry.
>>
>
> Correction, components could interact with each other via the
> ComponentContext.. but I guess we'd have to know if that's part of your use
> case.
>

Actually, no, it's not even possible via the ComponentContext. It only
allows access to Services.

>
> - Ray
>
>
>>
>> Please, can you explain in more detail what case might allow for the
>> above? Because I believe the assumption is that it's not currently
>> possible. Certainly there could be a bug and I'm certain everyone would
>> want to see that fixed.
>>
>> --
>> *Raymond Augé* 
>>  (@rotty3000)
>> Senior Software Architect *Liferay, Inc.* 
>>  (@Liferay)
>> Board Member & EEG Co-Chair, OSGi Alliance 
>> (@OSGiAlliance)
>>
>
>
>
> --
> *Raymond Augé* 
>  (@rotty3000)
> Senior Software Architect *Liferay, Inc.* 
>  (@Liferay)
> Board Member & EEG Co-Chair, OSGi Alliance 
> (@OSGiAlliance)
>



-- 
*Raymond Augé* 
 (@rotty3000)
Senior Software Architect *Liferay, Inc.* 
 (@Liferay)
Board Member & EEG Co-Chair, OSGi Alliance  (@OSGiAlliance)

Re: memory visibility of @Reference service references

[Raymond Auge]

On Tue, May 2, 2017 at 1:14 PM, Raymond Auge 
wrote:

>
> On Tue, May 2, 2017 at 11:46 AM, Dirk Hogan 
> wrote:
>
>> But no similar guarantee applies when another thread hits the
>> component,
>>
>
> I believe the discussion boils down to why you believe the above is
> possible?
>
> If SCR has not yet made the component available to other threads (which,
> to my knowledge, it only does by registering it as a service to the service
> registry) how can _another_ thread access the component at all?
>
> I don't even see a way for components within the same DS bundle to
> interact with each other other than via the service registry because
> @Reference only works through the registry.
>

Correction, components could interact with each other via the
ComponentContext.. but I guess we'd have to know if that's part of your use
case.

- Ray


>
> Please, can you explain in more detail what case might allow for the
> above? Because I believe the assumption is that it's not currently
> possible. Certainly there could be a bug and I'm certain everyone would
> want to see that fixed.
>
> --
> *Raymond Augé* 
>  (@rotty3000)
> Senior Software Architect *Liferay, Inc.* 
>  (@Liferay)
> Board Member & EEG Co-Chair, OSGi Alliance 
> (@OSGiAlliance)
>



-- 
*Raymond Augé* 
 (@rotty3000)
Senior Software Architect *Liferay, Inc.* 
 (@Liferay)
Board Member & EEG Co-Chair, OSGi Alliance  (@OSGiAlliance)

Re: memory visibility of @Reference service references

[Raymond Auge]

On Tue, May 2, 2017 at 11:46 AM, Dirk Hogan 
wrote:

> But no similar guarantee applies when another thread hits the
> component,
>

I believe the discussion boils down to why you believe the above is
possible?

If SCR has not yet made the component available to other threads (which, to
my knowledge, it only does by registering it as a service to the service
registry) how can _another_ thread access the component at all?

I don't even see a way for components within the same DS bundle to interact
with each other other than via the service registry because @Reference only
works through the registry.

Please, can you explain in more detail what case might allow for the above?
Because I believe the assumption is that it's not currently possible.
Certainly there could be a bug and I'm certain everyone would want to see
that fixed.

-- 
*Raymond Augé* 
 (@rotty3000)
Senior Software Architect *Liferay, Inc.* 
 (@Liferay)
Board Member & EEG Co-Chair, OSGi Alliance  (@OSGiAlliance)

Re: memory visibility of @Reference service references

[Dirk Hogan]

Hi Neil-
I agree that, upon publication, the service has been activated, and thus
all references are visible to the thread which performed the publication.
But the java memory model explicitly distinguishes between:
---Program order: which guarantees that the underlying virtual machine must
ensure that memory state matches program execution sequence despite
omnipresent compiler re-orderings. This only applies to the execution
thread - i.e. intra-thread visibility
---Synchronization order: which dictates memory visibility across threads.
Unless there is an explicit Synchronization Action, there is NO guarantee
that thread A can read what thread B wrote.

So yes, when Felix initializes a component and publishes a service, all of
its references are set and visible to the Felix thread which performed this
action. But no similar guarantee applies when another thread hits the
component, unless there is an explicit Synchronization Action between the
write performed by the Felix thread, and the read performed by another
thread.

Remember the 'thread shutdown idiom', where you had a thread A running in a
while(!shutdown) loop, and a shutdown() method to toggle shutdown to true
so that thread A shuts down? The shutdown boolean had to be declared as
volatile so the write to the shutdown boolean performed by thread X would
be visible to the read performed by thread A in the while(!shutdown) loop.
In the shutdown() method, after having set shutdown to true, thread X is
guaranteed to always see a value of true for the shutdown boolean, but it
is completely valid for thread A to see either true or false.

These two links are a great resource:
https://shipilev.net/blog/2016/close-encounters-of-jmm-kind/
https://shipilev.net/blog/2014/jmm-pragmatics/

Thanks

Dirk

On Mon, May 1, 2017 at 4:03 PM, Neil Bartlett  wrote:

> Dirk,
>
> But this is why the happens-before relationship is important. The
> component is not published as a service until after activation, therefore
> no other thread can obtain visibility in order to “hit” it until after the
> references are bound.
>
> For this reason, static @Reference fields do not need to be volatile.
>
> Regards,
> Neil
>
>
>
> > On 1 May 2017, at 22:05, Dirk Hogan  wrote:
> >
> > Hi Neil-
> >
> > I was not so worried about the happens-before relationship between
> > setting the @Reference and activating the component. This relationship
> > could be provided by the memory visibility guarantees of the program
> > order defined in the JMM, provided @Reference set and component
> > activation occurred in the same thread.
> >
> >
> > I was more worried about some other thread (e.g. client request)
> > hitting this component, and, in the process, dereferencing the
> > @Reference, as this will involve inter-thread memory visibility, and
> > as such, a synchronization action to ensure that a memory read sees
> > the state written by OSGi thread which wrote the @Reference.
> >
> >
> > Thanks
> >
> >
> > Dirk
> >
> >
> >
> > The Declarative Services spec requires that there is a happens-before
> > relationship between SCR setting the field and activating the component
> > instance — see OSGi Compendium 6.0.0 section 112.3.8.1.
> >
> > There are various ways of achieving this under the Java Memory Model. For
> > example the SCR implementation can use internal synchronized blocks.
> >
> > Regards,
> > Neil
> >
> >> On 27 Apr 2017, at 17:33, Dirk Hogan  wrote:
> >>
> >> I have been puzzled about the visibility of references written/read
> across
> >> threads in Felix.
> >>
> >> Scenario:
> >>
> >> public class myOSGiComponent {
> >>
> >> @Reference
> >>
> >> SomeOtherOSGiService myServiceReference;
> >>
> >> public void activate(ComponentContext cc) {
> >>
> >> // myServiceReference will be set - use it to initialize other
> >> dependencies, etc
> >>
> >> }
> >>
> >> public void clientRequestViaEmbeddedJetty() {
> >>
> >> myServiceReference.foo();
> >>
> >> }
> >>
> >> The bottom line is that a STATIC reference does not have to be
> volatile. I
> >> can see that the write to myServiceReference is guaranteed to be
> visible in
> >> the activate method, provided that same Felix thread which set
> >> myServiceReference invokes activate, and thus the visibility is
> guaranteed
> >> by program order (17.4.3 of the Java Language Spec).
> >>
> >> However, in the absence of the synchronization action provided by e.g. a
> >> volatile @Reference declaration, I do not see how Felix can provide a
> Java
> >> Memory Model compliant guarantee that myServiceReference will be
> visible in
> >> clientRequestViaEmbeddedJetty, as the thread which set the reference,
> and
> >> the thread which invokes clientRequestViaEmbeddedJetty, will be
> distinct.
> >> In the absence of a volatile @Reference declaration, there is no
> >> synchronization action (17.4.2 of JLS) which would guarantee that the
> >> reference write is visible to the 

Re: memory visibility of @Reference service references

[Neil Bartlett]

Dirk,

But this is why the happens-before relationship is important. The component is 
not published as a service until after activation, therefore no other thread 
can obtain visibility in order to “hit” it until after the references are bound.

For this reason, static @Reference fields do not need to be volatile.

Regards,
Neil



> On 1 May 2017, at 22:05, Dirk Hogan  wrote:
> 
> Hi Neil-
> 
> I was not so worried about the happens-before relationship between
> setting the @Reference and activating the component. This relationship
> could be provided by the memory visibility guarantees of the program
> order defined in the JMM, provided @Reference set and component
> activation occurred in the same thread.
> 
> 
> I was more worried about some other thread (e.g. client request)
> hitting this component, and, in the process, dereferencing the
> @Reference, as this will involve inter-thread memory visibility, and
> as such, a synchronization action to ensure that a memory read sees
> the state written by OSGi thread which wrote the @Reference.
> 
> 
> Thanks
> 
> 
> Dirk
> 
> 
> 
> The Declarative Services spec requires that there is a happens-before
> relationship between SCR setting the field and activating the component
> instance — see OSGi Compendium 6.0.0 section 112.3.8.1.
> 
> There are various ways of achieving this under the Java Memory Model. For
> example the SCR implementation can use internal synchronized blocks.
> 
> Regards,
> Neil
> 
>> On 27 Apr 2017, at 17:33, Dirk Hogan  wrote:
>> 
>> I have been puzzled about the visibility of references written/read across
>> threads in Felix.
>> 
>> Scenario:
>> 
>> public class myOSGiComponent {
>> 
>> @Reference
>> 
>> SomeOtherOSGiService myServiceReference;
>> 
>> public void activate(ComponentContext cc) {
>> 
>> // myServiceReference will be set - use it to initialize other
>> dependencies, etc
>> 
>> }
>> 
>> public void clientRequestViaEmbeddedJetty() {
>> 
>> myServiceReference.foo();
>> 
>> }
>> 
>> The bottom line is that a STATIC reference does not have to be volatile. I
>> can see that the write to myServiceReference is guaranteed to be visible in
>> the activate method, provided that same Felix thread which set
>> myServiceReference invokes activate, and thus the visibility is guaranteed
>> by program order (17.4.3 of the Java Language Spec).
>> 
>> However, in the absence of the synchronization action provided by e.g. a
>> volatile @Reference declaration, I do not see how Felix can provide a Java
>> Memory Model compliant guarantee that myServiceReference will be visible in
>> clientRequestViaEmbeddedJetty, as the thread which set the reference, and
>> the thread which invokes clientRequestViaEmbeddedJetty, will be distinct.
>> In the absence of a volatile @Reference declaration, there is no
>> synchronization action (17.4.2 of JLS) which would guarantee that the
>> reference write is visible to the reference read, as they are being
>> performed by distinct threads.
>> 
>> Perhaps an explanation as to why a DYNAMIC @Reference does need to be
>> volatile would be helpful - e.g. why does component
>> de-activation/re-activation provide memory visibility guarantees for STATIC
>> references?
>> 
>> 
>> Thanks
>> 
>> 
>> Dirk

Re: memory visibility of @Reference service references

[Dirk Hogan]

Thanks for the discussion guys. I failed to subscribe before posting -
hopefully this ends up in the right place.

Referencing S1 from a volatile field will guarantee that that the OSGi
context which writes/reads the reference will see the written reference,
even if distinct threads write and read the reference.

However, this does nothing if the @Reference field in my Component is not
volatile, as the the Synchronization Action defined by a volatile field
write/read only applies to that volatile field. So even if the correct S1
reference is written to the @Reference by the OSGi runtime, if the
Component @Reference is non-volatile, there is no guarantee that this S1
state will be visible when this @Reference is read by another thread.
What's worse, it could be that this initial reference is correct, but
additional references in S1 will not be visible from the @Reference. In
fact, any non-volatile, non-final references in S1 are not guaranteed to be
visible, even if the Service @Reference is non-null.

And as far as synchronized blocks are concerned, they only provide a
guarantee for the inter-thread memory visibility of references written/read
within the synchronized block of the SAME monitor (first bullet point in
17.4.4 of
http://docs.oracle.com/javase/specs/jls/se8/html/jls-17.html#jls-17.4. Or
the section under Visibility here: http://gee.cs.oswego.edu/dl/cpj/jmm.html).
So initializing S1 in a synchronized block will do nothing for these
references if the @Reference is not read when synchronized on the same
monitor.

Not to be pedantic, but
https://shipilev.net/blog/2016/close-encounters-of-jmm-kind/
has some good examples of the mind-blowing implications of the JMM.

The bottom line: I think STATIC @References need to be volatile.

Thanks

Dirk



Hi David

On Fri, Apr 28, 2017 at 5:19 PM, David Jencks  wrote:
> Step 5 happens before step 1.

You are right. SCR registers a ServiceFactory with the framework
before step 1 (at least for a delayed component). And on top of that,
I got the terminology pretty wrong.

Let me try again (the following assumes a delayed component):

1. the component becomes satisfied
2. DS (or rather SCR) registers a ServiceFactory with the framework's
service registry
3. the service is requested from the service registry by thread T1
3.1. DS creates a new instances of the implementation object S1 in thread T1
3.2. DS injects S1 with the (static) @Reference service in thread T1
3.3. DS calls activate() in thread T1
3.4. at this time no other thread has a reference to service S1
3.5. DS makes S1 accessible and returns it from the
ServiceFactory#getService() method
4. at this time other threads can get a fully initialized reference to
service S1

So regarding Java's memory model, I still assume that full activation
of the implementation object S1 before it leaves the scope of
ServiceFactory#getService() is the key. FWIW, the Felix implementation
keeps the reference to S1 in a volatile field that can only be
accessed once a volatile boolean is set to true.

Does this sound better?

Regards
Julian

>
> I’m not sure that java by itself has sufficient synchronization to assure
> that a requesting thread gets the fully correct memory set up by thread T1.
> However I Think that with the synchronization in Felix DS requesting threads
> will get correct information.
>
> thanks
> david jencks
>
>> On Apr 28, 2017, at 1:13 AM, Julian Sedding  wrote:
>>
>> My theory is as follows:
>>
>> 1. DS creates a new instances of the service S1 in thread T1
>> 2. DS injects the (static) @Reference service in thread T1
>> 3. DS calls activate() in thread T1
>> 4. at this time no other thread has a reference to service S1
>> 5. DS publishes the service in the OSGi service registry
>> 6. at this time other thread can get a fully initialized reference to
>> service S1
>>
>> I believe that because up to 4. the reference to S1 is "private" to
>> thread T1. In 5. S1 is "published" and thus first allowed to cross a
>> thread boundary (I assume Java internally creates some sort of copy of
>> S1's memory for a new thread T2). However, at that point S1 is fully
>> initialized and immutable. Should the static @Reference on S1 change,
>> it is deactivated and a new instance S1' is created, activated and
>> registered.
>>
>> In contrast, with a dynamic @Reference, the field's value may change
>> after it is published. Thus it needs to be volatile.
>>
>> As I said, this is a theory ;) I have no references to back me up.
>>
>> Regards
>> Julian
>>
>>
>> On Thu, Apr 27, 2017 at 7:06 PM, David Jencks 
>> wrote:
>>> My theory about why this is ok is as follows, perhaps you can see something
>>> wrong with it.
>>>
>>> Felix DS has some synchronization after activate is called, therefore the
>>> state of the static reference is definitely written to memory.  This
>>> reference field won’t change after that, since it’s (SCR) static.
>>> Any getService 

Re: memory visibility of @Reference service references

[Dirk Hogan]

Hi Neil-

I was not so worried about the happens-before relationship between
setting the @Reference and activating the component. This relationship
could be provided by the memory visibility guarantees of the program
order defined in the JMM, provided @Reference set and component
activation occurred in the same thread.


I was more worried about some other thread (e.g. client request)
hitting this component, and, in the process, dereferencing the
@Reference, as this will involve inter-thread memory visibility, and
as such, a synchronization action to ensure that a memory read sees
the state written by OSGi thread which wrote the @Reference.


Thanks


Dirk



The Declarative Services spec requires that there is a happens-before
relationship between SCR setting the field and activating the component
instance — see OSGi Compendium 6.0.0 section 112.3.8.1.

There are various ways of achieving this under the Java Memory Model. For
example the SCR implementation can use internal synchronized blocks.

Regards,
Neil

> On 27 Apr 2017, at 17:33, Dirk Hogan  wrote:
>
> I have been puzzled about the visibility of references written/read across
> threads in Felix.
>
> Scenario:
>
> public class myOSGiComponent {
>
> @Reference
>
> SomeOtherOSGiService myServiceReference;
>
> public void activate(ComponentContext cc) {
>
> // myServiceReference will be set - use it to initialize other
> dependencies, etc
>
> }
>
> public void clientRequestViaEmbeddedJetty() {
>
> myServiceReference.foo();
>
> }
>
> The bottom line is that a STATIC reference does not have to be volatile. I
> can see that the write to myServiceReference is guaranteed to be visible in
> the activate method, provided that same Felix thread which set
> myServiceReference invokes activate, and thus the visibility is guaranteed
> by program order (17.4.3 of the Java Language Spec).
>
> However, in the absence of the synchronization action provided by e.g. a
> volatile @Reference declaration, I do not see how Felix can provide a Java
> Memory Model compliant guarantee that myServiceReference will be visible in
> clientRequestViaEmbeddedJetty, as the thread which set the reference, and
> the thread which invokes clientRequestViaEmbeddedJetty, will be distinct.
> In the absence of a volatile @Reference declaration, there is no
> synchronization action (17.4.2 of JLS) which would guarantee that the
> reference write is visible to the reference read, as they are being
> performed by distinct threads.
>
> Perhaps an explanation as to why a DYNAMIC @Reference does need to be
> volatile would be helpful - e.g. why does component
> de-activation/re-activation provide memory visibility guarantees for STATIC
> references?
>
>
> Thanks
>
>
> Dirk

Re: memory visibility of @Reference service references

[Julian Sedding]

Hi David

On Fri, Apr 28, 2017 at 5:19 PM, David Jencks  wrote:
> Step 5 happens before step 1.

You are right. SCR registers a ServiceFactory with the framework
before step 1 (at least for a delayed component). And on top of that,
I got the terminology pretty wrong.

Let me try again (the following assumes a delayed component):

1. the component becomes satisfied
2. DS (or rather SCR) registers a ServiceFactory with the framework's
service registry
3. the service is requested from the service registry by thread T1
3.1. DS creates a new instances of the implementation object S1 in thread T1
3.2. DS injects S1 with the (static) @Reference service in thread T1
3.3. DS calls activate() in thread T1
3.4. at this time no other thread has a reference to service S1
3.5. DS makes S1 accessible and returns it from the
ServiceFactory#getService() method
4. at this time other threads can get a fully initialized reference to
service S1

So regarding Java's memory model, I still assume that full activation
of the implementation object S1 before it leaves the scope of
ServiceFactory#getService() is the key. FWIW, the Felix implementation
keeps the reference to S1 in a volatile field that can only be
accessed once a volatile boolean is set to true.

Does this sound better?

Regards
Julian

>
> I’m not sure that java by itself has sufficient synchronization to assure 
> that a requesting thread gets the fully correct memory set up by thread T1.  
> However I Think that with the synchronization in Felix DS requesting threads 
> will get correct information.
>
> thanks
> david jencks
>
>> On Apr 28, 2017, at 1:13 AM, Julian Sedding  wrote:
>>
>> My theory is as follows:
>>
>> 1. DS creates a new instances of the service S1 in thread T1
>> 2. DS injects the (static) @Reference service in thread T1
>> 3. DS calls activate() in thread T1
>> 4. at this time no other thread has a reference to service S1
>> 5. DS publishes the service in the OSGi service registry
>> 6. at this time other thread can get a fully initialized reference to 
>> service S1
>>
>> I believe that because up to 4. the reference to S1 is "private" to
>> thread T1. In 5. S1 is "published" and thus first allowed to cross a
>> thread boundary (I assume Java internally creates some sort of copy of
>> S1's memory for a new thread T2). However, at that point S1 is fully
>> initialized and immutable. Should the static @Reference on S1 change,
>> it is deactivated and a new instance S1' is created, activated and
>> registered.
>>
>> In contrast, with a dynamic @Reference, the field's value may change
>> after it is published. Thus it needs to be volatile.
>>
>> As I said, this is a theory ;) I have no references to back me up.
>>
>> Regards
>> Julian
>>
>>
>> On Thu, Apr 27, 2017 at 7:06 PM, David Jencks  
>> wrote:
>>> My theory about why this is ok is as follows, perhaps you can see something 
>>> wrong with it.
>>>
>>> Felix DS has some synchronization after activate is called, therefore the 
>>> state of the static reference is definitely written to memory.  This 
>>> reference field won’t change after that, since it’s (SCR) static.
>>> Any getService call on the service reference also has sufficient 
>>> synchronization that it happens after the above write.  Therefore any other 
>>> thread which obtained the referenced service will get an object with the 
>>> correct field value.  Therefore any access to this possibly cached object 
>>> by any thread will have the correct value.
>>>
>>> thanks
>>>
>>> david jencks
>>>
 On Apr 27, 2017, at 9:33 AM, Dirk Hogan  wrote:

 I have been puzzled about the visibility of references written/read across
 threads in Felix.

 Scenario:

 public class myOSGiComponent {

 @Reference

 SomeOtherOSGiService myServiceReference;

 public void activate(ComponentContext cc) {

 // myServiceReference will be set - use it to initialize other
 dependencies, etc

 }

 public void clientRequestViaEmbeddedJetty() {

 myServiceReference.foo();

 }

 The bottom line is that a STATIC reference does not have to be volatile. I
 can see that the write to myServiceReference is guaranteed to be visible in
 the activate method, provided that same Felix thread which set
 myServiceReference invokes activate, and thus the visibility is guaranteed
 by program order (17.4.3 of the Java Language Spec).

 However, in the absence of the synchronization action provided by e.g. a
 volatile @Reference declaration, I do not see how Felix can provide a Java
 Memory Model compliant guarantee that myServiceReference will be visible in
 clientRequestViaEmbeddedJetty, as the thread which set the reference, and
 the thread which invokes clientRequestViaEmbeddedJetty, will be distinct.
 In the absence of a volatile 

Re: memory visibility of @Reference service references

[David Jencks]

Step 5 happens before step 1.

I’m not sure that java by itself has sufficient synchronization to assure that 
a requesting thread gets the fully correct memory set up by thread T1.  However 
I Think that with the synchronization in Felix DS requesting threads will get 
correct information.

thanks
david jencks

> On Apr 28, 2017, at 1:13 AM, Julian Sedding  wrote:
> 
> My theory is as follows:
> 
> 1. DS creates a new instances of the service S1 in thread T1
> 2. DS injects the (static) @Reference service in thread T1
> 3. DS calls activate() in thread T1
> 4. at this time no other thread has a reference to service S1
> 5. DS publishes the service in the OSGi service registry
> 6. at this time other thread can get a fully initialized reference to service 
> S1
> 
> I believe that because up to 4. the reference to S1 is "private" to
> thread T1. In 5. S1 is "published" and thus first allowed to cross a
> thread boundary (I assume Java internally creates some sort of copy of
> S1's memory for a new thread T2). However, at that point S1 is fully
> initialized and immutable. Should the static @Reference on S1 change,
> it is deactivated and a new instance S1' is created, activated and
> registered.
> 
> In contrast, with a dynamic @Reference, the field's value may change
> after it is published. Thus it needs to be volatile.
> 
> As I said, this is a theory ;) I have no references to back me up.
> 
> Regards
> Julian
> 
> 
> On Thu, Apr 27, 2017 at 7:06 PM, David Jencks  
> wrote:
>> My theory about why this is ok is as follows, perhaps you can see something 
>> wrong with it.
>> 
>> Felix DS has some synchronization after activate is called, therefore the 
>> state of the static reference is definitely written to memory.  This 
>> reference field won’t change after that, since it’s (SCR) static.
>> Any getService call on the service reference also has sufficient 
>> synchronization that it happens after the above write.  Therefore any other 
>> thread which obtained the referenced service will get an object with the 
>> correct field value.  Therefore any access to this possibly cached object by 
>> any thread will have the correct value.
>> 
>> thanks
>> 
>> david jencks
>> 
>>> On Apr 27, 2017, at 9:33 AM, Dirk Hogan  wrote:
>>> 
>>> I have been puzzled about the visibility of references written/read across
>>> threads in Felix.
>>> 
>>> Scenario:
>>> 
>>> public class myOSGiComponent {
>>> 
>>> @Reference
>>> 
>>> SomeOtherOSGiService myServiceReference;
>>> 
>>> public void activate(ComponentContext cc) {
>>> 
>>> // myServiceReference will be set - use it to initialize other
>>> dependencies, etc
>>> 
>>> }
>>> 
>>> public void clientRequestViaEmbeddedJetty() {
>>> 
>>> myServiceReference.foo();
>>> 
>>> }
>>> 
>>> The bottom line is that a STATIC reference does not have to be volatile. I
>>> can see that the write to myServiceReference is guaranteed to be visible in
>>> the activate method, provided that same Felix thread which set
>>> myServiceReference invokes activate, and thus the visibility is guaranteed
>>> by program order (17.4.3 of the Java Language Spec).
>>> 
>>> However, in the absence of the synchronization action provided by e.g. a
>>> volatile @Reference declaration, I do not see how Felix can provide a Java
>>> Memory Model compliant guarantee that myServiceReference will be visible in
>>> clientRequestViaEmbeddedJetty, as the thread which set the reference, and
>>> the thread which invokes clientRequestViaEmbeddedJetty, will be distinct.
>>> In the absence of a volatile @Reference declaration, there is no
>>> synchronization action (17.4.2 of JLS) which would guarantee that the
>>> reference write is visible to the reference read, as they are being
>>> performed by distinct threads.
>>> 
>>> Perhaps an explanation as to why a DYNAMIC @Reference does need to be
>>> volatile would be helpful - e.g. why does component
>>> de-activation/re-activation provide memory visibility guarantees for STATIC
>>> references?
>>> 
>>> 
>>> Thanks
>>> 
>>> 
>>> Dirk
>> 

Re: memory visibility of @Reference service references

[Julian Sedding]

My theory is as follows:

1. DS creates a new instances of the service S1 in thread T1
2. DS injects the (static) @Reference service in thread T1
3. DS calls activate() in thread T1
4. at this time no other thread has a reference to service S1
5. DS publishes the service in the OSGi service registry
6. at this time other thread can get a fully initialized reference to service S1

I believe that because up to 4. the reference to S1 is "private" to
thread T1. In 5. S1 is "published" and thus first allowed to cross a
thread boundary (I assume Java internally creates some sort of copy of
S1's memory for a new thread T2). However, at that point S1 is fully
initialized and immutable. Should the static @Reference on S1 change,
it is deactivated and a new instance S1' is created, activated and
registered.

In contrast, with a dynamic @Reference, the field's value may change
after it is published. Thus it needs to be volatile.

As I said, this is a theory ;) I have no references to back me up.

Regards
Julian


On Thu, Apr 27, 2017 at 7:06 PM, David Jencks  wrote:
> My theory about why this is ok is as follows, perhaps you can see something 
> wrong with it.
>
> Felix DS has some synchronization after activate is called, therefore the 
> state of the static reference is definitely written to memory.  This 
> reference field won’t change after that, since it’s (SCR) static.
> Any getService call on the service reference also has sufficient 
> synchronization that it happens after the above write.  Therefore any other 
> thread which obtained the referenced service will get an object with the 
> correct field value.  Therefore any access to this possibly cached object by 
> any thread will have the correct value.
>
> thanks
>
> david jencks
>
>> On Apr 27, 2017, at 9:33 AM, Dirk Hogan  wrote:
>>
>> I have been puzzled about the visibility of references written/read across
>> threads in Felix.
>>
>> Scenario:
>>
>> public class myOSGiComponent {
>>
>> @Reference
>>
>> SomeOtherOSGiService myServiceReference;
>>
>> public void activate(ComponentContext cc) {
>>
>> // myServiceReference will be set - use it to initialize other
>> dependencies, etc
>>
>> }
>>
>> public void clientRequestViaEmbeddedJetty() {
>>
>> myServiceReference.foo();
>>
>> }
>>
>> The bottom line is that a STATIC reference does not have to be volatile. I
>> can see that the write to myServiceReference is guaranteed to be visible in
>> the activate method, provided that same Felix thread which set
>> myServiceReference invokes activate, and thus the visibility is guaranteed
>> by program order (17.4.3 of the Java Language Spec).
>>
>> However, in the absence of the synchronization action provided by e.g. a
>> volatile @Reference declaration, I do not see how Felix can provide a Java
>> Memory Model compliant guarantee that myServiceReference will be visible in
>> clientRequestViaEmbeddedJetty, as the thread which set the reference, and
>> the thread which invokes clientRequestViaEmbeddedJetty, will be distinct.
>> In the absence of a volatile @Reference declaration, there is no
>> synchronization action (17.4.2 of JLS) which would guarantee that the
>> reference write is visible to the reference read, as they are being
>> performed by distinct threads.
>>
>> Perhaps an explanation as to why a DYNAMIC @Reference does need to be
>> volatile would be helpful - e.g. why does component
>> de-activation/re-activation provide memory visibility guarantees for STATIC
>> references?
>>
>>
>> Thanks
>>
>>
>> Dirk
>

Re: memory visibility of @Reference service references

[David Jencks]

My theory about why this is ok is as follows, perhaps you can see something 
wrong with it.

Felix DS has some synchronization after activate is called, therefore the state 
of the static reference is definitely written to memory.  This reference field 
won’t change after that, since it’s (SCR) static.
Any getService call on the service reference also has sufficient 
synchronization that it happens after the above write.  Therefore any other 
thread which obtained the referenced service will get an object with the 
correct field value.  Therefore any access to this possibly cached object by 
any thread will have the correct value.

thanks

david jencks

> On Apr 27, 2017, at 9:33 AM, Dirk Hogan  wrote:
> 
> I have been puzzled about the visibility of references written/read across
> threads in Felix.
> 
> Scenario:
> 
> public class myOSGiComponent {
> 
> @Reference
> 
> SomeOtherOSGiService myServiceReference;
> 
> public void activate(ComponentContext cc) {
> 
> // myServiceReference will be set - use it to initialize other
> dependencies, etc
> 
> }
> 
> public void clientRequestViaEmbeddedJetty() {
> 
> myServiceReference.foo();
> 
> }
> 
> The bottom line is that a STATIC reference does not have to be volatile. I
> can see that the write to myServiceReference is guaranteed to be visible in
> the activate method, provided that same Felix thread which set
> myServiceReference invokes activate, and thus the visibility is guaranteed
> by program order (17.4.3 of the Java Language Spec).
> 
> However, in the absence of the synchronization action provided by e.g. a
> volatile @Reference declaration, I do not see how Felix can provide a Java
> Memory Model compliant guarantee that myServiceReference will be visible in
> clientRequestViaEmbeddedJetty, as the thread which set the reference, and
> the thread which invokes clientRequestViaEmbeddedJetty, will be distinct.
> In the absence of a volatile @Reference declaration, there is no
> synchronization action (17.4.2 of JLS) which would guarantee that the
> reference write is visible to the reference read, as they are being
> performed by distinct threads.
> 
> Perhaps an explanation as to why a DYNAMIC @Reference does need to be
> volatile would be helpful - e.g. why does component
> de-activation/re-activation provide memory visibility guarantees for STATIC
> references?
> 
> 
> Thanks
> 
> 
> Dirk

Re: memory visibility of @Reference service references

[Neil Bartlett]

The Declarative Services spec requires that there is a happens-before 
relationship between SCR setting the field and activating the component 
instance — see OSGi Compendium 6.0.0 section 112.3.8.1.

There are various ways of achieving this under the Java Memory Model. For 
example the SCR implementation can use internal synchronized blocks.

Regards,
Neil

> On 27 Apr 2017, at 17:33, Dirk Hogan  wrote:
> 
> I have been puzzled about the visibility of references written/read across
> threads in Felix.
> 
> Scenario:
> 
> public class myOSGiComponent {
> 
> @Reference
> 
> SomeOtherOSGiService myServiceReference;
> 
> public void activate(ComponentContext cc) {
> 
> // myServiceReference will be set - use it to initialize other
> dependencies, etc
> 
> }
> 
> public void clientRequestViaEmbeddedJetty() {
> 
> myServiceReference.foo();
> 
> }
> 
> The bottom line is that a STATIC reference does not have to be volatile. I
> can see that the write to myServiceReference is guaranteed to be visible in
> the activate method, provided that same Felix thread which set
> myServiceReference invokes activate, and thus the visibility is guaranteed
> by program order (17.4.3 of the Java Language Spec).
> 
> However, in the absence of the synchronization action provided by e.g. a
> volatile @Reference declaration, I do not see how Felix can provide a Java
> Memory Model compliant guarantee that myServiceReference will be visible in
> clientRequestViaEmbeddedJetty, as the thread which set the reference, and
> the thread which invokes clientRequestViaEmbeddedJetty, will be distinct.
> In the absence of a volatile @Reference declaration, there is no
> synchronization action (17.4.2 of JLS) which would guarantee that the
> reference write is visible to the reference read, as they are being
> performed by distinct threads.
> 
> Perhaps an explanation as to why a DYNAMIC @Reference does need to be
> volatile would be helpful - e.g. why does component
> de-activation/re-activation provide memory visibility guarantees for STATIC
> references?
> 
> 
> Thanks
> 
> 
> Dirk