On Thu, Jun 09, 2016 at 02:13:49PM +0200, Christophe Milard wrote: > Bill: > S19: When you write:"These addresses are intended to be used within > the scope of the calling thread and should not be assumed to have any > validity outside of that context", do you really mean this, regardless > on how ODP threads are implemented? > > Jerrin: > S18: your answer to S18 is confusing me. see my returned question in > the doc. please answer it :-)
|Christophe to Jerin: if 2 different ODP threads are expected to retrieve |each its own handle |via lookup calls, aren't you saying that handles are NOT valid over the |whole ODP |instance? If you are just saying that ODP may provide additional APIs to |retrieve handles, |then fine, as long as the usage of this APIs is optionnal and passing |the handles via IPC is as good We do have ODP API for the name to handle lookup. For instance a _true_ multi-process ODP implementation(i.e Second process NOT derived from first process(two ODP process started at two terminals on different cores), one as primary and other one secondary). In those cases only way to share the handles is through ODP lookup API) Jerin > > All: I have updated the doc with a summary table and the comments from > Jon Rosen. Please, check that What I wrote matches what you think. > > I have also added a list of possible behaviour (numbered A to F) for > S19. Can you read these and pick your choice? If none of these > alternative match your choice, please add your description. > > Christophe > > On 9 June 2016 at 02:53, Bill Fischofer <bill.fischo...@linaro.org> wrote: > > > > > > On Wed, Jun 8, 2016 at 10:54 AM, Yi He <yi...@linaro.org> wrote: > >> > >> Hi, Christophe and team > >> > >> For the shmem part: > >> > >> S18: yes > >> S19, S20: I agree with Bill's comments are very accurate and formalized. > >> > >> S21: to cover the case shm addresses(pointers) passed within data > >> structure > >> > >> especially think of cases: > >> 3) Context pointer getters (odp_queue_context(), > >> odp_packet_user_ptr(), odp_timeout_user_ptr(), > >> odp_tm_node_context(), odp_tm_queue_context(), and > >> I agree with the solution, only feel it may meet difficulties > >> in runtime: > >> > >> According to man page: > >> Using shmat() with shmaddr equal to NULL is the preferred, > >> portable way of attaching a shared memory segment. Be aware > >> that the shared memory segment attached in this way may be > >> attached at different addresses in different processes. > >> Therefore, any pointers maintained within the shared memory > >> must be made relative (typically to the starting address of > >> the segment), rather than absolute. > >> I found an alternate sweetheart only available in Windows, > >> called based pointers (C++) > >> https://msdn.microsoft.com/en-us/library/57a97k4e.aspx > >> > >> Maybe we can spent some time to look for a counterpart in > >> std C world, that will be perfect. > > > > > > For students of programming history, or those old enough to remember, the > > concept of BASED storage originated in the (now ancient) programming > > language PL/I. Pointers to BASED storage were stored as offsets and the > > compiler automatically handled the relative addressing. They are very > > convenient for this sort of purpose. > > > >> > >> > >> S23: agree with Bill's comments covered the cases. > >> > >> Thanks and best regards, Yi > >> > >> On 8 June 2016 at 17:04, Christophe Milard <christophe.mil...@linaro.org> > >> wrote: > >>> > >>> OK. good that you agree (and please update the shared doc so it > >>> becomes the central point of information). > >>> There is something I like though, in your willingness to decouple the > >>> function and the pinning... > >>> Even if I am not sure this can be enforced by ODP at all time (as > >>> already stated), there is definitively a point in helping the > >>> application that with to do so. So please keep an eye on that! > >>> > >>> Your opinion on S19 S20 and S21 would be very welcome as well... This > >>> is the main hurting point.... > >>> > >>> Christophe > >>> > >>> On 8 June 2016 at 09:41, Yi He <yi...@linaro.org> wrote: > >>> > Hi, thanks Christophe and happy to discuss with and learn from team in > >>> > yesterday's ARCH call :) > >>> > > >>> > The question which triggers this kind of thinking is: how to use ODP as > >>> > a > >>> > framework to produce re-usable building blocks to compose "Network > >>> > Function > >>> > Instance" in runtime, since until runtime it will prepare resources for > >>> > function to settle down, thus comes the thought of seperating function > >>> > implementation and launching. > >>> > > >>> > I agree your point it seems upper layer consideration, I'll have some > >>> > time > >>> > to gain deeper understanding and knowledge on how upstair playings, > >>> > thus I > >>> > agree to the current S11, S12, S13, S14, S15, S16, S17 approach and we > >>> > can > >>> > revisit if really realized upper layer programming practise/model > >>> > affects > >>> > ODP's design in this aspect. > >>> > > >>> > Best Regards, Yi > >>> > > >>> > On 7 June 2016 at 16:47, Christophe Milard > >>> > <christophe.mil...@linaro.org> > >>> > wrote: > >>> >> > >>> >> On 7 June 2016 at 10:22, Yi He <yi...@linaro.org> wrote: > >>> >> > Hi, team > >>> >> > > >>> >> > I send my thoughts on the ODP thread part: > >>> >> > > >>> >> > S1, S2, S3, S4 > >>> >> > Yi: Yes > >>> >> > This set of statements defines ODP thread concept as a higher > >>> >> > level abstraction of underlying concurrent execution context. > >>> >> > > >>> >> > S5, S6, S7, S8, S9, S10: > >>> >> > Yi: Yes > >>> >> > This set of statements add several constraints upon ODP > >>> >> > instance concept. > >>> >> > > >>> >> > S11, S12, S13, S14, S15, S16, S17: > >>> >> > Yi: Not very much > >>> >> > > >>> >> > Currently ODP application still mixes the Function Implementation > >>> >> > and its Deployment, by this I mean in ODP application code, there > >>> >> > is code to implement Function, there is also code to deploy the > >>> >> > Function onto platform resources (CPU cores). > >>> >> > > >>> >> > I'd like to propose a programming practice/model as an attempt to > >>> >> > decouple these two things, ODP application code only focuses on > >>> >> > implementing Function, and leave it to a deployment script or > >>> >> > launcher to take care the deployment with different resource spec > >>> >> > (and sufficiency check also). > >>> >> > >>> >> Well, that goes straight against Barry's will that apps should pin > >>> >> their tasks on cpus... > >>> >> Please join the public call today: if Barry is there you can discuss > >>> >> this: I am happy to hear other voices in this discussion > >>> >> > >>> >> > > >>> >> > /* Use Case: a upper layer orchestrator, say Daylight is deploying > >>> >> > * an ODP application (can be a VNF instance in my opinion) onto > >>> >> > * some platform: > >>> >> > */ > >>> >> > > >>> >> > /* The application can accept command line options */ > >>> >> > --list-launchable > >>> >> > list all algorithm functions that need to be arranged > >>> >> > into an execution unit. > >>> >> > > >>> >> > --preparing-threads > >>> >> > control<1,2,3>@cpu<1>, worker<1,2,3,4>@cpu<2,3,4,5> > >>> >> > I'm not sure if the control/worker distinguish is needed > >>> >> > anymore, DPDK has similar concept lcore. > >>> >> > > >>> >> > --wait-launching-signal > >>> >> > main process can pause after preparing above threads but > >>> >> > before launching algorithm functions, here orchestrator can > >>> >> > further fine-tuning the threads by scripting (cgroups, etc). > >>> >> > CPU, disk/net IO, memory quotas, interrupt bindings, etc. > >>> >> > > >>> >> > --launching > >>> >> > main@control<1>,algorithm_one@control<2>, > >>> >> > algorithm_two@worker<1,2,3,4>... > >>> >> > > >>> >> > In source code, the only thing ODP library and application need to > >>> >> > do > >>> >> > related to deployment is to declare launchable algorithm functions > >>> >> > by > >>> >> > adding them into special text section: > >>> >> > > >>> >> > int main(...) __attribute__((section(".launchable"))); > >>> >> > int algorithm_one(void *) __attribute__((section(".launchable"))); > >>> >> > >>> >> interresting ... Does it need to be part of ODP, though? Cannot the > >>> >> ODP api provide means of pinning, and the apps that wish it can > >>> >> provide the options you mentioned to do it: i.e. the application that > >>> >> wants would pin according to its command line interface and the > >>> >> application that can't/does not want would pin manually... > >>> >> I mean one could also imagine cases where the app needs to pin: if > >>> >> specific HW exists connected to some cpus, or if serialisation exists > >>> >> (e.g. something like: pin tasks 1,2,and 3 to cpu 1,2,3, and then, when > >>> >> this work is done (e.g. after a barrier joining task 1,2,3) pin taks > >>> >> 4,5,6 on cpu 1,2,3 again.) There could be theoriticaly complex such > >>> >> dependency graphs where all cpus cannot be pinned from the beginning, > >>> >> and where your approach seem too restrictive. > >>> >> > >>> >> But I am glad to hear your voice on the arch call :-) > >>> >> > >>> >> Christophe. > >>> >> > > >>> >> > Best Regards, Yi > >>> >> > > >>> >> > > >>> >> > On 4 June 2016 at 05:56, Bill Fischofer <bill.fischo...@linaro.org> > >>> >> > wrote: > >>> >> >> > >>> >> >> I realized I forgot to respond to s23. Corrected here. > >>> >> >> > >>> >> >> On Fri, Jun 3, 2016 at 4:15 AM, Christophe Milard < > >>> >> >> christophe.mil...@linaro.org> wrote: > >>> >> >> > >>> >> >> > since V3: Update following Bill's comments > >>> >> >> > since V2: Update following Barry and Bill's comments > >>> >> >> > since V1: Update following arch call 31 may 2016 > >>> >> >> > > >>> >> >> > This is a tentative to sum up the discussions around the > >>> >> >> > thread/process > >>> >> >> > that have been happening these last weeks. > >>> >> >> > Sorry for the formalism of this mail, but it seems we need > >>> >> >> > accuracy > >>> >> >> > here... > >>> >> >> > > >>> >> >> > This summary is organized as follows: > >>> >> >> > > >>> >> >> > It is a set of statements, each of them expecting a separate > >>> >> >> > answer > >>> >> >> > from you. When no specific ODP version is specified, the > >>> >> >> > statement > >>> >> >> > regards the"ultimate" goal (i.e what we want eventually to > >>> >> >> > achieve). > >>> >> >> > Each statement is prefixed with: > >>> >> >> > - a statement number for further reference (e.g. S1) > >>> >> >> > - a status word (one of 'agreed' or 'open', or 'closed'). > >>> >> >> > Agreed statements expect a yes/no answers: 'yes' meaning that you > >>> >> >> > acknowledge that this is your understanding of the agreement and > >>> >> >> > will > >>> >> >> > not nack an implementation based on this statement. You can > >>> >> >> > comment > >>> >> >> > after a yes, but your comment will not block any implementation > >>> >> >> > based > >>> >> >> > on the agreed statement. A 'no' implies that the statement does > >>> >> >> > not > >>> >> >> > reflect your understanding of the agreement, or you refuse the > >>> >> >> > proposal. > >>> >> >> > Any 'no' received on an 'agreed' statement will push it back as > >>> >> >> > 'open'. > >>> >> >> > Open statements are fully open for further discussion. > >>> >> >> > > >>> >> >> > S1 -agreed: an ODP thread is an OS/platform concurrent execution > >>> >> >> > environment object (as opposed to an ODP objects). No more > >>> >> >> > specific > >>> >> >> > definition is given by the ODP API itself. > >>> >> >> > > >>> >> >> > Barry: YES > >>> >> >> > Bill: Yes > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S2 -agreed: Each ODP implementation must tell what is allowed to > >>> >> >> > be > >>> >> >> > used as ODP thread for that specific implementation: a > >>> >> >> > linux-based > >>> >> >> > implementation, for instance, will have to state whether odp > >>> >> >> > threads > >>> >> >> > can be linux pthread, linux processes, or both, or any other type > >>> >> >> > of > >>> >> >> > concurrent execution environment. ODP implementations can put any > >>> >> >> > restriction they wish on what an ODP thread is allowed to be. > >>> >> >> > This > >>> >> >> > should be documented in the ODP implementation documentation. > >>> >> >> > > >>> >> >> > Barry: YES > >>> >> >> > Bill: Yes > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S3 -agreed: in the linux generic ODP implementation a odpthread > >>> >> >> > will > >>> >> >> > be > >>> >> >> > either: > >>> >> >> > * a linux process descendant (or same as) the odp > >>> >> >> > instantiation > >>> >> >> > process. > >>> >> >> > * a pthread 'member' of a linux process descendant (or > >>> >> >> > same > >>> >> >> > as) the odp instantiation process. > >>> >> >> > > >>> >> >> > Barry: YES > >>> >> >> > Bill: Yes > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S4 -agreed: For monarch, the linux generic ODP implementation > >>> >> >> > only > >>> >> >> > supports odp thread as pthread member of the instantiation > >>> >> >> > process. > >>> >> >> > > >>> >> >> > Barry: YES > >>> >> >> > Bill: Yes > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S5 -agreed: whether multiple instances of ODP can be run on the > >>> >> >> > same > >>> >> >> > machine is left as a implementation decision. The ODP > >>> >> >> > implementation > >>> >> >> > document should state what is supported and any restriction is > >>> >> >> > allowed. > >>> >> >> > > >>> >> >> > Barry: YES > >>> >> >> > Bill: Yes > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S6 -agreed: The l-g odp implementation will support multiple odp > >>> >> >> > instances whose instantiation processes are different and not > >>> >> >> > ancestor/descendant of each others. Different instances of ODP > >>> >> >> > will, > >>> >> >> > of course, be restricted in sharing common OS ressources (The > >>> >> >> > total > >>> >> >> > amount of memory available for each ODP instances may decrease as > >>> >> >> > the > >>> >> >> > number of instances increases, the access to network interfaces > >>> >> >> > will > >>> >> >> > probably be granted to the first instance grabbing the interface > >>> >> >> > and > >>> >> >> > denied to others... some other rule may apply when sharing other > >>> >> >> > common ODP ressources.) > >>> >> >> > > >>> >> >> > Bill: Yes > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S7 -agreed: the l-g odp implementation will not support multiple > >>> >> >> > ODP > >>> >> >> > instances initiated from the same linux process (calling > >>> >> >> > odp_init_global() multiple times). > >>> >> >> > As an illustration, This means that a single process P is not > >>> >> >> > allowed > >>> >> >> > to execute the following calls (in any order) > >>> >> >> > instance1 = odp_init_global() > >>> >> >> > instance2 = odp_init_global() > >>> >> >> > pthread_create (and, in that thread, run > >>> >> >> > odp_local_init(instance1) ) > >>> >> >> > pthread_create (and, in that thread, run > >>> >> >> > odp_local_init(instance2) ) > >>> >> >> > > >>> >> >> > Bill: Yes > >>> >> >> > > >>> >> >> > ------------------- > >>> >> >> > > >>> >> >> > S8 -agreed: the l-g odp implementation will not support multiple > >>> >> >> > ODP > >>> >> >> > instances initiated from related linux processes > >>> >> >> > (descendant/ancestor > >>> >> >> > of each other), hence enabling ODP 'sub-instance'? As an > >>> >> >> > illustration, > >>> >> >> > this means that the following is not supported: > >>> >> >> > instance1 = odp_init_global() > >>> >> >> > pthread_create (and, in that thread, run > >>> >> >> > odp_local_init(instance1) ) > >>> >> >> > if (fork()==0) { > >>> >> >> > instance2 = odp_init_global() > >>> >> >> > pthread_create (and, in that thread, run > >>> >> >> > odp_local_init(instance2) ) > >>> >> >> > } > >>> >> >> > > >>> >> >> > Bill: Yes > >>> >> >> > > >>> >> >> > -------------------- > >>> >> >> > > >>> >> >> > S9 -agreed: the odp instance passed as parameter to > >>> >> >> > odp_local_init() > >>> >> >> > must always be one of the odp_instance returned by > >>> >> >> > odp_global_init() > >>> >> >> > > >>> >> >> > Barry: YES > >>> >> >> > Bill: Yes > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S10 -agreed: For l-g, if the answer to S7 and S8 are 'yes', then > >>> >> >> > due > >>> >> >> > to > >>> >> >> > S3, > >>> >> >> > the odp_instance an odp_thread can attach to is completely > >>> >> >> > defined by > >>> >> >> > the ancestor of the thread, making the odp_instance parameter of > >>> >> >> > odp_init_local redundant. The odp l-g implementation guide will > >>> >> >> > enlighten this > >>> >> >> > redundancy, but will stress that even in this case the parameter > >>> >> >> > to > >>> >> >> > odp_local_init() still have to be set correctly, as its usage is > >>> >> >> > internal to the implementation. > >>> >> >> > > >>> >> >> > Barry: I think so > >>> >> >> > Bill: This practice also ensures that applications behave > >>> >> >> > unchanged > >>> >> >> > if > >>> >> >> > and when multi-instance support is added, so I don't think we > >>> >> >> > need to > >>> >> >> > be apologetic about this parameter requirement. > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S11 -agreed: at odp_global_init() time, the application will > >>> >> >> > provide > >>> >> >> > 3 > >>> >> >> > sets of cpu (i.e 3 cpu masks): > >>> >> >> > -the control cpu mask > >>> >> >> > -the worker cpu mask > >>> >> >> > -the odp service cpu mask (i.e the set of cpu odp can > >>> >> >> > take > >>> >> >> > for > >>> >> >> > its own usage) > >>> >> >> > Note: The service CPU mask will be introdused post monarch > >>> >> >> > > >>> >> >> > Bill: Yes > >>> >> >> > Barry: YES > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S12 -agreed: the odp implementation may return an error at > >>> >> >> > odp_init_global() call if the number of cpu in the odp service > >>> >> >> > mask > >>> >> >> > (or their 'position') does not match the ODP implementation need. > >>> >> >> > > >>> >> >> > Barry: YES > >>> >> >> > Bill: Yes. However, an implementation may fail an > >>> >> >> > odp_init_global() > >>> >> >> > call > >>> >> >> > for any resource insufficiency, not just cpus. > >>> >> >> > > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S13 -agreed: the application is fully responsible of pinning its > >>> >> >> > own > >>> >> >> > odp threads to different cpus, and this is done directly through > >>> >> >> > OS > >>> >> >> > system calls, or via helper functions (as opposed to ODP API > >>> >> >> > calls). > >>> >> >> > This pinning should be made among cpus member of the worker cpu > >>> >> >> > mask > >>> >> >> > or the control cpu mask. > >>> >> >> > > >>> >> >> > Barry: YES, but I support the existence of helper functions to do > >>> >> >> > this > >>> >> >> > – including the > >>> >> >> > important case of pinning the main thread > >>> >> >> > > >>> >> >> > Bill: Yes. And agree an ODP helper is useful here (which is why > >>> >> >> > odp-linux > >>> >> >> > provides one). > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S14 -agreed: whether more than one odp thread can be pinned to > >>> >> >> > the > >>> >> >> > same cpu is left as an implementation choice (and the answer to > >>> >> >> > that > >>> >> >> > question can be different for the service, worker and control > >>> >> >> > threads). This choice should be well documented in the > >>> >> >> > implementation > >>> >> >> > user manual. > >>> >> >> > > >>> >> >> > Barry: YES > >>> >> >> > Bill: Yes > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S15 -agreed: the odp implementation is responsible of pinning its > >>> >> >> > own > >>> >> >> > service threads among the cpu member of the odp service cpu mask. > >>> >> >> > > >>> >> >> > Barry: YES, in principle – BUT be aware that currently the l-g > >>> >> >> > ODP > >>> >> >> > implementation > >>> >> >> > (and perhaps many others) cannot call the helper functions > >>> >> >> > (unless > >>> >> >> > inlined), > >>> >> >> > so this internal pinning may not be well coordinated with the > >>> >> >> > helpers. > >>> >> >> > > >>> >> >> > Bill: Yes. And I agree with Barry on the helper recursion issue. > >>> >> >> > We > >>> >> >> > should > >>> >> >> > fix that so there is no conflict between implementation internal > >>> >> >> > pinning > >>> >> >> > and application pinning attempts. > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S16 -open: why does the odp implementation need to know the > >>> >> >> > control > >>> >> >> > and > >>> >> >> > worker mask? If S13 is true, shoudln't these two masks be part of > >>> >> >> > the > >>> >> >> > helper only? (meaning that S11 is wrong) > >>> >> >> > > >>> >> >> > Barry: Currently it probably doesn’t NEED them, but perhaps in > >>> >> >> > the > >>> >> >> > future, with some > >>> >> >> > new API’s and capabilities, it might benefit from this > >>> >> >> > information, > >>> >> >> > and so I would leave them in. > >>> >> >> > > >>> >> >> > Bill: The implementation sees these because they are how a > >>> >> >> > provisioning > >>> >> >> > agent (e.g., OpenDaylight) would pass higher-level configuration > >>> >> >> > information through the application to the underlying ODP > >>> >> >> > implementation. > >>> >> >> > The initial masks specified on odp_init_global() are used in the > >>> >> >> > implementation of the odp_cpumask_default_worker(), > >>> >> >> > odp_cpumask_default_control(), and odp_cpumask_all_available() > >>> >> >> > APIs. > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S17 -open: should masks passed as parameter to odp_init_global() > >>> >> >> > have > >>> >> >> > the > >>> >> >> > same "namespace" as those used internally within ODP? > >>> >> >> > > >>> >> >> > Barry: YES > >>> >> >> > Bill: Yes. I'm not sure what it would mean for them to be in a > >>> >> >> > different > >>> >> >> > namespace. How would those be bridged if they weren't? > >>> >> >> > > >>> >> >> > > >>> >> >> > --------------------------- > >>> >> >> > > >>> >> >> > S18 -agreed: ODP handles are valid over the whole ODP instance, > >>> >> >> > i.e. > >>> >> >> > any odp handle remains valid among all the odpthreads of the ODP > >>> >> >> > instance regardless of the odp thread type (process, thread or > >>> >> >> > whatever): an ODP thread A can pass an odp handle to onother ODP > >>> >> >> > thread B (using any kind of IPC), and B can use the handle. > >>> >> >> > > >>> >> >> > Bill: Yes > >>> >> >> > > >>> >> >> > ----------------- > >>> >> >> > > >>> >> >> > S19 -open : any pointer retrieved by an ODP call (such as > >>> >> >> > odp_*_get_addr()) follows the rules defined by the OS, with the > >>> >> >> > possible exception defined in S21. For the linux generic ODP > >>> >> >> > implementation, this means that > >>> >> >> > pointers are fully shareable when using pthreads and that > >>> >> >> > pointers > >>> >> >> > pointing to shared mem areas will be shareable as long as the > >>> >> >> > fork() > >>> >> >> > happens after the shm_reserve(). > >>> >> >> > > >>> >> >> > Barry: NO. Disagree. I would prefer to see a consistent ODP > >>> >> >> > answer > >>> >> >> > on > >>> >> >> > this topic, and in > >>> >> >> > particular I don’t even believe that most OS’s “have rules > >>> >> >> > defining > >>> >> >> > …”, > >>> >> >> > since > >>> >> >> > in fact one can make programs run under Linux which can share > >>> >> >> > pointers > >>> >> >> > regardless > >>> >> >> > the ordering of fork() calls. Most OS have lots of (continually > >>> >> >> > evolving) capabilities > >>> >> >> > in the category of sharing memory and so “following the rules of > >>> >> >> > the > >>> >> >> > OS” > >>> >> >> > is not > >>> >> >> > well defined. > >>> >> >> > Instead, I prefer a simpler rule. Memory reserved using the > >>> >> >> > special > >>> >> >> > flag is guaranteed > >>> >> >> > to use the same addresses across processes, and all other > >>> >> >> > pointers > >>> >> >> > are > >>> >> >> > not guaranteed > >>> >> >> > to be the same nor guaranteed to be different, so the ODP > >>> >> >> > programmer > >>> >> >> > should avoid > >>> >> >> > any such assumptions for maximum portability. But of course > >>> >> >> > programmers > >>> >> >> > often > >>> >> >> > only consider a subset of possible targets (e.g. how many > >>> >> >> > programmers > >>> >> >> > consider porting > >>> >> >> > to an 8-bit CPU or a machine with a 36-bit word length), and so > >>> >> >> > they > >>> >> >> > may happily take advantage > >>> >> >> > of certain non-guaranteed assumptions. > >>> >> >> > > >>> >> >> > > >>> >> >> > Bill: As I noted earlier we have to distinguish between different > >>> >> >> > types > >>> >> >> > of > >>> >> >> > memory and where these pointers come from. If the application is > >>> >> >> > using > >>> >> >> > malloc() or some other OS API to get memory and then using that > >>> >> >> > memory's > >>> >> >> > address as, for example, a queue context pointer, then it is > >>> >> >> > taking > >>> >> >> > responsibility for ensuring that these pointers are meaningful to > >>> >> >> > whoever > >>> >> >> > sees them. ODP isn't going to do anything to help there. So this > >>> >> >> > question > >>> >> >> > really only refers to addresses returned from ODP APIs. If we > >>> >> >> > look > >>> >> >> > for > >>> >> >> > void > >>> >> >> > * returns in the ODP API we see that the only addresses ODP > >>> >> >> > returns > >>> >> >> > are: > >>> >> >> > > >>> >> >> > 1) Those that enable addressability to buffers and packets > >>> >> >> > (odp_buffer_addr(), odp_packet_data(), odp_packet_offset(), etc.) > >>> >> >> > > >>> >> >> > These addresses are intended to be used within the scope of the > >>> >> >> > calling > >>> >> >> > thread and should not be assumed to have any validity outside of > >>> >> >> > that > >>> >> >> > context because the buffer/packet is the durable object and any > >>> >> >> > addresses > >>> >> >> > are just (potentially transient) mappings of that object for use > >>> >> >> > by > >>> >> >> > that > >>> >> >> > thread. Packet and buffer handles (not addresses) are passed > >>> >> >> > between > >>> >> >> > threads via queues and the receiver issues its own such calls on > >>> >> >> > the > >>> >> >> > received handles to get its own addressability to these objects. > >>> >> >> > Whether > >>> >> >> > or > >>> >> >> > not these addresses are the same is purely internal to an ODP > >>> >> >> > implementation and is not visible to the application. > >>> >> >> > > >>> >> >> > 2) Packet user areas (odp_packet_user_area()). This API returns > >>> >> >> > the > >>> >> >> > address of a preallocated user area associated with the packet > >>> >> >> > (size > >>> >> >> > defined by the pool that the packet was drawn from at > >>> >> >> > odp_pool_create() > >>> >> >> > time by the max_uarea_size entry in the odp_pool_param_t). Since > >>> >> >> > this > >>> >> >> > is > >>> >> >> > metadata associated with the packet this API may be called by any > >>> >> >> > thread > >>> >> >> > that obtains the odp_packet_t for the packet that contains that > >>> >> >> > user > >>> >> >> > area. > >>> >> >> > However, like the packet itself, the scope of this returned > >>> >> >> > address > >>> >> >> > is > >>> >> >> > the > >>> >> >> > calling thread. So the address returned by odp_packet_user_area() > >>> >> >> > should > >>> >> >> > not be cached or passed to any other thread. Each thread that > >>> >> >> > needs > >>> >> >> > addressability to this area makes its own call and whether these > >>> >> >> > returned > >>> >> >> > addresses are the same or different is again internal to the > >>> >> >> > implementation > >>> >> >> > and not visible to the application. Note that just as two threads > >>> >> >> > should > >>> >> >> > not ownership of an odp_packet_t at the same time, two threads > >>> >> >> > should > >>> >> >> > not > >>> >> >> > be trying to access the user area associated with a packet > >>> >> >> > either. > >>> >> >> > > >>> >> >> > 3) Context pointer getters (odp_queue_context(), > >>> >> >> > odp_packet_user_ptr(), > >>> >> >> > odp_timeout_user_ptr(), odp_tm_node_context(), > >>> >> >> > odp_tm_queue_context(), > >>> >> >> > and > >>> >> >> > the user context pointer carried in the odp_crypto_params_t > >>> >> >> > struct) > >>> >> >> > > >>> >> >> > These are set by the application using corresponding setter APIs > >>> >> >> > or > >>> >> >> > provided as values in structs, so the application either obtains > >>> >> >> > these > >>> >> >> > pointers on its own, in which case it is responsible for ensuring > >>> >> >> > that > >>> >> >> > they > >>> >> >> > are meaningful to whoever retrieves them, or from an odp_shm_t. > >>> >> >> > So > >>> >> >> > these > >>> >> >> > are not a special case in themselves. > >>> >> >> > > >>> >> >> > 4) ODP shared memory (odp_shm_addr(), odp_shm_info()). These > >>> >> >> > APIs > >>> >> >> > return > >>> >> >> > addresses to odp_shm_t objects that are specifically created to > >>> >> >> > support > >>> >> >> > sharing. The rule here is simple: the scope of any returned shm > >>> >> >> > address > >>> >> >> > is > >>> >> >> > determined by the sharing flag specified at odp_shm_reserve() > >>> >> >> > time. > >>> >> >> > ODP > >>> >> >> > currently defines two such flags: ODP_SHM_SW_ONLY and > >>> >> >> > ODP_SHM_PROC. > >>> >> >> > We > >>> >> >> > simply need to define precisely the intended sharing scope of > >>> >> >> > these > >>> >> >> > two > >>> >> >> > (or > >>> >> >> > any new flags we define) to answer this question. Note that > >>> >> >> > context > >>> >> >> > pointers drawn from odp_shm_t objects would then have whatever > >>> >> >> > sharing > >>> >> >> > attributes that the shm object has, thus completely defining case > >>> >> >> > (3). > >>> >> >> > > >>> >> >> > --------------------- > >>> >> >> > > >>> >> >> > S20 -open: by default, shmem addresses (returned by > >>> >> >> > odp_shm_addr()) > >>> >> >> > follow the OS rules, as defined by S19. > >>> >> >> > > >>> >> >> > Ola: The question is which OS rules apply (an OS can have > >>> >> >> > different > >>> >> >> > rules > >>> >> >> > for > >>> >> >> > different OS objects, e.g. memory regions allocated using malloc > >>> >> >> > and > >>> >> >> > mmap > >>> >> >> > will behave differently). I think the answer depends on ODP shmem > >>> >> >> > objects > >>> >> >> > are implemented. Only the ODP implementation knows how ODP shmem > >>> >> >> > objects > >>> >> >> > are created (e.g. use some OS system call, manipulate the page > >>> >> >> > tables > >>> >> >> > directly). So essentially the sharability of pointers is ODP > >>> >> >> > implementation > >>> >> >> > specific (although ODP implementations on the same OS can be > >>> >> >> > expected > >>> >> >> > to > >>> >> >> > behave the same). Conclusion: we actually don't specify anything > >>> >> >> > at > >>> >> >> > all > >>> >> >> > here, it is completely up to the ODP implementation. > >>> >> >> > What is required/expected by ODP applications? If we don't make > >>> >> >> > applications happy, ODP is unlikely to succeed. > >>> >> >> > I think many applications are happy with a single-process thread > >>> >> >> > model > >>> >> >> > where all memory is shared and pointers can be shared freely. > >>> >> >> > I hear of some applications that require multi-process thread > >>> >> >> > model, > >>> >> >> > I > >>> >> >> > expect that those applications also want to be able to share > >>> >> >> > memory > >>> >> >> > and > >>> >> >> > pointers freely between them, at least memory that was > >>> >> >> > specifically > >>> >> >> > allocated to be shared (so called shared memory regions, what's > >>> >> >> > otherwise > >>> >> >> > the purpose of such memory regions?). > >>> >> >> > > >>> >> >> > Barry: Disagree with the same comments as in S19. > >>> >> >> > > >>> >> >> > Bill: I believe my discourse on S19 completely resolves this > >>> >> >> > question. > >>> >> >> > This > >>> >> >> > is controlled by the share flag specified at odp_shm_reserve() > >>> >> >> > time. > >>> >> >> > We > >>> >> >> > just need to specify the sharing scope implied by each of these > >>> >> >> > and > >>> >> >> > then > >>> >> >> > it > >>> >> >> > is up to each implementation to see that such scope is realized. > >>> >> >> > > >>> >> >> > --------------------- > >>> >> >> > > >>> >> >> > S21 -open: shm will support and extra flag at shm_reserve() call > >>> >> >> > time: > >>> >> >> > SHM_XXX. The usage of this flag will allocate shared memory > >>> >> >> > guaranteed > >>> >> >> > to be located at the same virtual address on all odpthreads of > >>> >> >> > the > >>> >> >> > odp_instance. Pointers to this shared memory type are therefore > >>> >> >> > fully > >>> >> >> > sharable, even on odpthreads running on different VA space (e.g. > >>> >> >> > processes). The amount of memory which can be allocated using > >>> >> >> > this > >>> >> >> > flag can be > >>> >> >> > limited to any value by the ODP implementation, down to zero > >>> >> >> > bytes, > >>> >> >> > meaning that some odp implementation may not support this option > >>> >> >> > at > >>> >> >> > all. The shm_reserve() will return an error in this case.The > >>> >> >> > usage of > >>> >> >> > this flag by the application is therefore not recommended. The > >>> >> >> > ODP > >>> >> >> > implementation may require a hint about the size of this area at > >>> >> >> > odp_init_global() call time. > >>> >> >> > > >>> >> >> > Barry: Mostly agree, except for the comment about the special > >>> >> >> > flag > >>> >> >> > not > >>> >> >> > being recommended. > >>> >> >> > > >>> >> >> > Ola: Agree. Some/many applications will want to share memory > >>> >> >> > between > >>> >> >> > threads/processes and must be able to do so. Some ODP platforms > >>> >> >> > may > >>> >> >> > have > >>> >> >> > limitations to the amount of memory (if any) that can be shared > >>> >> >> > and > >>> >> >> > may > >>> >> >> > thus fail to run certain applications. Such is life. I don't see > >>> >> >> > a > >>> >> >> > problem > >>> >> >> > with that. Possibly we should remove the phrase "not recommended" > >>> >> >> > and > >>> >> >> > just > >>> >> >> > state that portability may be limited. > >>> >> >> > > >>> >> >> > > >>> >> >> > Bill: Yes. As noted in S19 and S20 the intent of the share flag > >>> >> >> > is to > >>> >> >> > specify desired addressability scope for the returned odp_shm_t. > >>> >> >> > It's > >>> >> >> > perfectly reasonable to define multiple such scopes that may have > >>> >> >> > different > >>> >> >> > intended uses (and implementation costs). > >>> >> >> > > >>> >> >> > ------------------ > >>> >> >> > > >>> >> >> > S22 -open: please put here your name suggestions for this SHM_XXX > >>> >> >> > flag > >>> >> >> > :-). > >>> >> >> > > >>> >> >> > Ola: > >>> >> >> > SHM_I_REALLY_WANT_TO_SHARE_THIS_MEMORY > >>> >> >> > > >>> >> >> > Bill: I previously suggested ODP_SHM_INSTANCE that specifies that > >>> >> >> > the > >>> >> >> > sharing scope of this odp_shm_t is the entire ODP instance. > >>> >> >> > > >>> >> >> > ------------------ > >>> >> >> > > >>> >> >> > S23 -open: The rules above define relatively well the behaviour > >>> >> >> > of > >>> >> >> > pointer retrieved by the call to odp_shm_get_addr(). But many > >>> >> >> > points > >>> >> >> > needs tobe defined regarding other ODP objects pointers: What is > >>> >> >> > the > >>> >> >> > validity of a pointer to a packet, for instance? If process A > >>> >> >> > creates > >>> >> >> > a packet pool P, then forks B and C, and B allocate a packet from > >>> >> >> > P > >>> >> >> > and retrieves a pointer to a packet allocated from this P... Is > >>> >> >> > this > >>> >> >> > pointer valid in A and C? In the current l-g implementation, it > >>> >> >> > will... Is this behaviour > >>> >> >> > something we wish to enforce on any odp implementation? What > >>> >> >> > about > >>> >> >> > other objects: buffers, atomics ... Some clear rule has to be > >>> >> >> > defined > >>> >> >> > here... How things behave and if this behaviour is a part of the > >>> >> >> > ODP > >>> >> >> > API or just specific to different implementations... > >>> >> >> > > >>> >> >> > Ola: Perhaps we need the option to specify the > >>> >> >> > I_REALLY_WANT_TO_SHARE_THIS_MEMORY flag when creating all types > >>> >> >> > of > >>> >> >> > ODP > >>> >> >> > pools? > >>> >> >> > An ODP implementation can always fail to create such a pool if > >>> >> >> > the > >>> >> >> > sharability requirement can not be satisfied. > >>> >> >> > Allocation of locations used for atomic operations is the > >>> >> >> > responsibility > >>> >> >> > of the application which can (and must) choose a suitable type of > >>> >> >> > memory. > >>> >> >> > It is better that sharability is an explicit requirement from the > >>> >> >> > application. It should be specified as a flag parameter to the > >>> >> >> > different > >>> >> >> > calls that create/allocate regions of memory (shmem, different > >>> >> >> > types > >>> >> >> > of > >>> >> >> > pools). > >>> >> >> > > >>> >> >> > > >>> >> >> > Barry: > >>> >> >> > Again refer to S19 answer. Specifically it is about what is > >>> >> >> > GUARANTEED regarding > >>> >> >> > pointer validity, not whether the pointers in certain cases will > >>> >> >> > happen > >>> >> >> > to be > >>> >> >> > the same. So for your example, the pointer is not guaranteed to > >>> >> >> > be > >>> >> >> > valid in A and C, > >>> >> >> > but the programmer might well believe that for all the ODP > >>> >> >> > platforms > >>> >> >> > and implementations > >>> >> >> > they expect to run on, this is very likely to be the case, in > >>> >> >> > which > >>> >> >> > case we can’t stop them > >>> >> >> > from constraining their program’s portability – no more than > >>> >> >> > requiring > >>> >> >> > them to be able to > >>> >> >> > port to a ternary (3-valued “bit”) architecture. > >>> >> >> > > >>> >> >> > >>> >> >> > >>> >> >> Bill: See s19 response, which answers all these questions. The > >>> >> >> fork > >>> >> >> case > >>> >> >> you mention is moot because addresses returned by ODP packet > >>> >> >> operations > >>> >> >> are > >>> >> >> only valid in the thread that makes those calls. The handle must be > >>> >> >> valid > >>> >> >> throughout the instance so that may affect how the implementation > >>> >> >> chooses > >>> >> >> to implement packet pool creation, but such questions are internal > >>> >> >> to > >>> >> >> each > >>> >> >> implementation and not part of the API. > >>> >> >> > >>> >> >> Atomics are application declared entities and so memory sharing is > >>> >> >> again > >>> >> >> covered by my response to s19. If the atomic is in storage the > >>> >> >> application > >>> >> >> allocated from the OS, then its sharability is the responsibility > >>> >> >> of > >>> >> >> the > >>> >> >> application. If the atomic is part of a struct that resides in an > >>> >> >> odp_shm_t, then its sharability is governed by the share flag of > >>> >> >> the > >>> >> >> odp_shm it is taken from. > >>> >> >> > >>> >> >> No additional parameters are required for pools because what is > >>> >> >> shared > >>> >> >> from > >>> >> >> them is handles, not addresses. As noted, when these handles are > >>> >> >> converted > >>> >> >> to addresses those addresses are only valid for the calling thread. > >>> >> >> All > >>> >> >> other threads that have access to the handle make their own calls > >>> >> >> and > >>> >> >> whether or not those two addresses have any relationship to each > >>> >> >> other > >>> >> >> is > >>> >> >> not visible to the application. > >>> >> >> > >>> >> >> > >>> >> >> > > >>> >> >> > >>> >> >> > --------------------- > >>> >> >> > > >>> >> >> > Thanks for your feedback! > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> > > >>> >> >> _______________________________________________ > >>> >> >> lng-odp mailing list > >>> >> >> lng-odp@lists.linaro.org > >>> >> >> https://lists.linaro.org/mailman/listinfo/lng-odp > >>> >> > > >>> >> > > >>> > > >>> > > >> > >> > > _______________________________________________ lng-odp mailing list lng-odp@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lng-odp
