I have made a live document for this thread to make it easier to track the consensus, we can gather the agreements there and focus the threads on the remaining issues
https://collaborate.linaro.org/display/ODP/odp_thread+and+shmem+debate On 3 June 2016 at 05:15, Christophe Milard <christophe.mil...@linaro.org> wrote: > since V3: Update following Bill's comments > since V2: Update following Barry and Bill's comments > since V1: Update following arch call 31 may 2016 > > This is a tentative to sum up the discussions around the thread/process > that have been happening these last weeks. > Sorry for the formalism of this mail, but it seems we need accuracy > here... > > This summary is organized as follows: > > It is a set of statements, each of them expecting a separate answer > from you. When no specific ODP version is specified, the statement > regards the"ultimate" goal (i.e what we want eventually to achieve). > Each statement is prefixed with: > - a statement number for further reference (e.g. S1) > - a status word (one of 'agreed' or 'open', or 'closed'). > Agreed statements expect a yes/no answers: 'yes' meaning that you > acknowledge that this is your understanding of the agreement and will > not nack an implementation based on this statement. You can comment > after a yes, but your comment will not block any implementation based > on the agreed statement. A 'no' implies that the statement does not > reflect your understanding of the agreement, or you refuse the > proposal. > Any 'no' received on an 'agreed' statement will push it back as 'open'. > Open statements are fully open for further discussion. > > S1 -agreed: an ODP thread is an OS/platform concurrent execution > environment object (as opposed to an ODP objects). No more specific > definition is given by the ODP API itself. > > Barry: YES > Bill: Yes > > --------------------------- > > S2 -agreed: Each ODP implementation must tell what is allowed to be > used as ODP thread for that specific implementation: a linux-based > implementation, for instance, will have to state whether odp threads > can be linux pthread, linux processes, or both, or any other type of > concurrent execution environment. ODP implementations can put any > restriction they wish on what an ODP thread is allowed to be. This > should be documented in the ODP implementation documentation. > > Barry: YES > Bill: Yes > > --------------------------- > > S3 -agreed: in the linux generic ODP implementation a odpthread will be > either: > * a linux process descendant (or same as) the odp instantiation > process. > * a pthread 'member' of a linux process descendant (or same > as) the odp instantiation process. > > Barry: YES > Bill: Yes > > --------------------------- > > S4 -agreed: For monarch, the linux generic ODP implementation only > supports odp thread as pthread member of the instantiation process. > > Barry: YES > Bill: Yes > > --------------------------- > > S5 -agreed: whether multiple instances of ODP can be run on the same > machine is left as a implementation decision. The ODP implementation > document should state what is supported and any restriction is allowed. > > Barry: YES > Bill: Yes > > --------------------------- > > S6 -agreed: The l-g odp implementation will support multiple odp > instances whose instantiation processes are different and not > ancestor/descendant of each others. Different instances of ODP will, > of course, be restricted in sharing common OS ressources (The total > amount of memory available for each ODP instances may decrease as the > number of instances increases, the access to network interfaces will > probably be granted to the first instance grabbing the interface and > denied to others... some other rule may apply when sharing other > common ODP ressources.) > > Bill: Yes > > --------------------------- > > S7 -agreed: the l-g odp implementation will not support multiple ODP > instances initiated from the same linux process (calling > odp_init_global() multiple times). > As an illustration, This means that a single process P is not allowed > to execute the following calls (in any order) > instance1 = odp_init_global() > instance2 = odp_init_global() > pthread_create (and, in that thread, run odp_local_init(instance1) ) > pthread_create (and, in that thread, run odp_local_init(instance2) ) > > Bill: Yes > > ------------------- > > S8 -agreed: the l-g odp implementation will not support multiple ODP > instances initiated from related linux processes (descendant/ancestor > of each other), hence enabling ODP 'sub-instance'? As an illustration, > this means that the following is not supported: > instance1 = odp_init_global() > pthread_create (and, in that thread, run odp_local_init(instance1) ) > if (fork()==0) { > instance2 = odp_init_global() > pthread_create (and, in that thread, run odp_local_init(instance2) ) > } > > Bill: Yes > > -------------------- > > S9 -agreed: the odp instance passed as parameter to odp_local_init() > must always be one of the odp_instance returned by odp_global_init() > > Barry: YES > Bill: Yes > > --------------------------- > > S10 -agreed: For l-g, if the answer to S7 and S8 are 'yes', then due to S3, > the odp_instance an odp_thread can attach to is completely defined by > the ancestor of the thread, making the odp_instance parameter of > odp_init_local redundant. The odp l-g implementation guide will > enlighten this > redundancy, but will stress that even in this case the parameter to > odp_local_init() still have to be set correctly, as its usage is > internal to the implementation. > > Barry: I think so > Bill: This practice also ensures that applications behave unchanged if > and when multi-instance support is added, so I don't think we need to > be apologetic about this parameter requirement. > > --------------------------- > > S11 -agreed: at odp_global_init() time, the application will provide 3 > sets of cpu (i.e 3 cpu masks): > -the control cpu mask > -the worker cpu mask > -the odp service cpu mask (i.e the set of cpu odp can take for > its own usage) > Note: The service CPU mask will be introdused post monarch > > Bill: Yes > Barry: YES > --------------------------- > > S12 -agreed: the odp implementation may return an error at > odp_init_global() call if the number of cpu in the odp service mask > (or their 'position') does not match the ODP implementation need. > > Barry: YES > Bill: Yes. However, an implementation may fail an odp_init_global() call > for any resource insufficiency, not just cpus. > > > --------------------------- > > S13 -agreed: the application is fully responsible of pinning its own > odp threads to different cpus, and this is done directly through OS > system calls, or via helper functions (as opposed to ODP API calls). > This pinning should be made among cpus member of the worker cpu mask > or the control cpu mask. > > Barry: YES, but I support the existence of helper functions to do this > – including the > important case of pinning the main thread > > Bill: Yes. And agree an ODP helper is useful here (which is why odp-linux > provides one). > > --------------------------- > > S14 -agreed: whether more than one odp thread can be pinned to the > same cpu is left as an implementation choice (and the answer to that > question can be different for the service, worker and control > threads). This choice should be well documented in the implementation > user manual. > > Barry: YES > Bill: Yes > > --------------------------- > > S15 -agreed: the odp implementation is responsible of pinning its own > service threads among the cpu member of the odp service cpu mask. > > Barry: YES, in principle – BUT be aware that currently the l-g ODP > implementation > (and perhaps many others) cannot call the helper functions (unless > inlined), > so this internal pinning may not be well coordinated with the helpers. > > Bill: Yes. And I agree with Barry on the helper recursion issue. We should > fix that so there is no conflict between implementation internal pinning > and application pinning attempts. > > --------------------------- > > S16 -open: why does the odp implementation need to know the control and > worker mask? If S13 is true, shoudln't these two masks be part of the > helper only? (meaning that S11 is wrong) > > Barry: Currently it probably doesn’t NEED them, but perhaps in the > future, with some > new API’s and capabilities, it might benefit from this information, > and so I would leave them in. > > Bill: The implementation sees these because they are how a provisioning > agent (e.g., OpenDaylight) would pass higher-level configuration > information through the application to the underlying ODP implementation. > The initial masks specified on odp_init_global() are used in the > implementation of the odp_cpumask_default_worker(), > odp_cpumask_default_control(), and odp_cpumask_all_available() APIs. > > --------------------------- > > S17 -open: should masks passed as parameter to odp_init_global() have the > same "namespace" as those used internally within ODP? > > Barry: YES > Bill: Yes. I'm not sure what it would mean for them to be in a different > namespace. How would those be bridged if they weren't? > > > --------------------------- > > S18 -agreed: ODP handles are valid over the whole ODP instance, i.e. > any odp handle remains valid among all the odpthreads of the ODP > instance regardless of the odp thread type (process, thread or > whatever): an ODP thread A can pass an odp handle to onother ODP > thread B (using any kind of IPC), and B can use the handle. > > Bill: Yes > > ----------------- > > S19 -open : any pointer retrieved by an ODP call (such as > odp_*_get_addr()) follows the rules defined by the OS, with the > possible exception defined in S21. For the linux generic ODP > implementation, this means that > pointers are fully shareable when using pthreads and that pointers > pointing to shared mem areas will be shareable as long as the fork() > happens after the shm_reserve(). > > Barry: NO. Disagree. I would prefer to see a consistent ODP answer on > this topic, and in > particular I don’t even believe that most OS’s “have rules defining …”, > since > in fact one can make programs run under Linux which can share pointers > regardless > the ordering of fork() calls. Most OS have lots of (continually > evolving) capabilities > in the category of sharing memory and so “following the rules of the OS” > is not > well defined. > Instead, I prefer a simpler rule. Memory reserved using the special > flag is guaranteed > to use the same addresses across processes, and all other pointers are > not guaranteed > to be the same nor guaranteed to be different, so the ODP programmer > should avoid > any such assumptions for maximum portability. But of course programmers > often > only consider a subset of possible targets (e.g. how many programmers > consider porting > to an 8-bit CPU or a machine with a 36-bit word length), and so they > may happily take advantage > of certain non-guaranteed assumptions. > > > Bill: As I noted earlier we have to distinguish between different types of > memory and where these pointers come from. If the application is using > malloc() or some other OS API to get memory and then using that memory's > address as, for example, a queue context pointer, then it is taking > responsibility for ensuring that these pointers are meaningful to whoever > sees them. ODP isn't going to do anything to help there. So this question > really only refers to addresses returned from ODP APIs. If we look for void > * returns in the ODP API we see that the only addresses ODP returns are: > > 1) Those that enable addressability to buffers and packets > (odp_buffer_addr(), odp_packet_data(), odp_packet_offset(), etc.) > > These addresses are intended to be used within the scope of the calling > thread and should not be assumed to have any validity outside of that > context because the buffer/packet is the durable object and any addresses > are just (potentially transient) mappings of that object for use by that > thread. Packet and buffer handles (not addresses) are passed between > threads via queues and the receiver issues its own such calls on the > received handles to get its own addressability to these objects. Whether or > not these addresses are the same is purely internal to an ODP > implementation and is not visible to the application. > > 2) Packet user areas (odp_packet_user_area()). This API returns the > address of a preallocated user area associated with the packet (size > defined by the pool that the packet was drawn from at odp_pool_create() > time by the max_uarea_size entry in the odp_pool_param_t). Since this is > metadata associated with the packet this API may be called by any thread > that obtains the odp_packet_t for the packet that contains that user area. > However, like the packet itself, the scope of this returned address is the > calling thread. So the address returned by odp_packet_user_area() should > not be cached or passed to any other thread. Each thread that needs > addressability to this area makes its own call and whether these returned > addresses are the same or different is again internal to the implementation > and not visible to the application. Note that just as two threads should > not ownership of an odp_packet_t at the same time, two threads should not > be trying to access the user area associated with a packet either. > > 3) Context pointer getters (odp_queue_context(), odp_packet_user_ptr(), > odp_timeout_user_ptr(), odp_tm_node_context(), odp_tm_queue_context(), and > the user context pointer carried in the odp_crypto_params_t struct) > > These are set by the application using corresponding setter APIs or > provided as values in structs, so the application either obtains these > pointers on its own, in which case it is responsible for ensuring that they > are meaningful to whoever retrieves them, or from an odp_shm_t. So these > are not a special case in themselves. > > 4) ODP shared memory (odp_shm_addr(), odp_shm_info()). These APIs return > addresses to odp_shm_t objects that are specifically created to support > sharing. The rule here is simple: the scope of any returned shm address is > determined by the sharing flag specified at odp_shm_reserve() time. ODP > currently defines two such flags: ODP_SHM_SW_ONLY and ODP_SHM_PROC. We > simply need to define precisely the intended sharing scope of these two (or > any new flags we define) to answer this question. Note that context > pointers drawn from odp_shm_t objects would then have whatever sharing > attributes that the shm object has, thus completely defining case (3). > > --------------------- > > S20 -open: by default, shmem addresses (returned by odp_shm_addr()) > follow the OS rules, as defined by S19. > > Ola: The question is which OS rules apply (an OS can have different rules for > different OS objects, e.g. memory regions allocated using malloc and mmap > will behave differently). I think the answer depends on ODP shmem objects > are implemented. Only the ODP implementation knows how ODP shmem objects > are created (e.g. use some OS system call, manipulate the page tables > directly). So essentially the sharability of pointers is ODP implementation > specific (although ODP implementations on the same OS can be expected to > behave the same). Conclusion: we actually don't specify anything at all > here, it is completely up to the ODP implementation. > What is required/expected by ODP applications? If we don't make > applications happy, ODP is unlikely to succeed. > I think many applications are happy with a single-process thread model > where all memory is shared and pointers can be shared freely. > I hear of some applications that require multi-process thread model, I > expect that those applications also want to be able to share memory and > pointers freely between them, at least memory that was specifically > allocated to be shared (so called shared memory regions, what's otherwise > the purpose of such memory regions?). > > Barry: Disagree with the same comments as in S19. > > Bill: I believe my discourse on S19 completely resolves this question. This > is controlled by the share flag specified at odp_shm_reserve() time. We > just need to specify the sharing scope implied by each of these and then it > is up to each implementation to see that such scope is realized. > > --------------------- > > S21 -open: shm will support and extra flag at shm_reserve() call time: > SHM_XXX. The usage of this flag will allocate shared memory guaranteed > to be located at the same virtual address on all odpthreads of the > odp_instance. Pointers to this shared memory type are therefore fully > sharable, even on odpthreads running on different VA space (e.g. > processes). The amount of memory which can be allocated using this > flag can be > limited to any value by the ODP implementation, down to zero bytes, > meaning that some odp implementation may not support this option at > all. The shm_reserve() will return an error in this case.The usage of > this flag by the application is therefore not recommended. The ODP > implementation may require a hint about the size of this area at > odp_init_global() call time. > > Barry: Mostly agree, except for the comment about the special flag not > being recommended. > > Ola: Agree. Some/many applications will want to share memory between > threads/processes and must be able to do so. Some ODP platforms may have > limitations to the amount of memory (if any) that can be shared and may > thus fail to run certain applications. Such is life. I don't see a problem > with that. Possibly we should remove the phrase "not recommended" and just > state that portability may be limited. > > > Bill: Yes. As noted in S19 and S20 the intent of the share flag is to > specify desired addressability scope for the returned odp_shm_t. It's > perfectly reasonable to define multiple such scopes that may have different > intended uses (and implementation costs). > > ------------------ > > S22 -open: please put here your name suggestions for this SHM_XXX flag > :-). > > Ola: > SHM_I_REALLY_WANT_TO_SHARE_THIS_MEMORY > > Bill: I previously suggested ODP_SHM_INSTANCE that specifies that the > sharing scope of this odp_shm_t is the entire ODP instance. > > ------------------ > > S23 -open: The rules above define relatively well the behaviour of > pointer retrieved by the call to odp_shm_get_addr(). But many points > needs tobe defined regarding other ODP objects pointers: What is the > validity of a pointer to a packet, for instance? If process A creates > a packet pool P, then forks B and C, and B allocate a packet from P > and retrieves a pointer to a packet allocated from this P... Is this > pointer valid in A and C? In the current l-g implementation, it > will... Is this behaviour > something we wish to enforce on any odp implementation? What about > other objects: buffers, atomics ... Some clear rule has to be defined > here... How things behave and if this behaviour is a part of the ODP > API or just specific to different implementations... > > Ola: Perhaps we need the option to specify the > I_REALLY_WANT_TO_SHARE_THIS_MEMORY flag when creating all types of ODP > pools? > An ODP implementation can always fail to create such a pool if the > sharability requirement can not be satisfied. > Allocation of locations used for atomic operations is the responsibility > of the application which can (and must) choose a suitable type of memory. > It is better that sharability is an explicit requirement from the > application. It should be specified as a flag parameter to the different > calls that create/allocate regions of memory (shmem, different types of > pools). > > > Barry: > Again refer to S19 answer. Specifically it is about what is > GUARANTEED regarding > pointer validity, not whether the pointers in certain cases will happen > to be > the same. So for your example, the pointer is not guaranteed to be > valid in A and C, > but the programmer might well believe that for all the ODP platforms > and implementations > they expect to run on, this is very likely to be the case, in which > case we can’t stop them > from constraining their program’s portability – no more than requiring > them to be able to > port to a ternary (3-valued “bit”) architecture. > > > > --------------------- > > Thanks for your feedback! > > > -- Mike Holmes Technical Manager - Linaro Networking Group Linaro.org │ Open source software for ARM SoCs "Work should be fun and collaborative, the rest follows" _______________________________________________ lng-odp mailing list lng-odp@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lng-odp
