On Fri, Jul 21, 2017 at 08:01:07PM +0800, Bob Liu wrote: > On Fri, Jul 21, 2017 at 10:10 AM, Bob Liu <liub...@huawei.com> wrote: > > On 2017/7/21 9:41, Jerome Glisse wrote: > >> On Fri, Jul 21, 2017 at 09:15:29AM +0800, Bob Liu wrote: > >>> On 2017/7/20 23:03, Jerome Glisse wrote: > >>>> On Wed, Jul 19, 2017 at 05:09:04PM +0800, Bob Liu wrote: > >>>>> On 2017/7/19 10:25, Jerome Glisse wrote: > >>>>>> On Wed, Jul 19, 2017 at 09:46:10AM +0800, Bob Liu wrote: > >>>>>>> On 2017/7/18 23:38, Jerome Glisse wrote: > >>>>>>>> On Tue, Jul 18, 2017 at 11:26:51AM +0800, Bob Liu wrote: > >>>>>>>>> On 2017/7/14 5:15, Jérôme Glisse wrote: > >> > >> [...] > >> > >>>>> Then it's more like replace the numa node solution(CDM) with ZONE_DEVICE > >>>>> (type MEMORY_DEVICE_PUBLIC). But the problem is the same, e.g how to > >>>>> make > >>>>> sure the device memory say HBM won't be occupied by normal CPU > >>>>> allocation. > >>>>> Things will be more complex if there are multi GPU connected by nvlink > >>>>> (also cache coherent) in a system, each GPU has their own HBM. > >>>>> > >>>>> How to decide allocate physical memory from local HBM/DDR or remote HBM/ > >>>>> DDR? > >>>>> > >>>>> If using numa(CDM) approach there are NUMA mempolicy and autonuma > >>>>> mechanism > >>>>> at least. > >>>> > >>>> NUMA is not as easy as you think. First like i said we want the device > >>>> memory to be isolated from most existing mm mechanism. Because memory > >>>> is unreliable and also because device might need to be able to evict > >>>> memory to make contiguous physical memory allocation for graphics. > >>>> > >>> > >>> Right, but we need isolation any way. > >>> For hmm-cdm, the isolation is not adding device memory to lru list, and > >>> many > >>> if (is_device_public_page(page)) ... > >>> > >>> But how to evict device memory? > >> > >> What you mean by evict ? Device driver can evict whenever they see the need > >> to do so. CPU page fault will evict too. Process exit or munmap() will free > >> the device memory. > >> > >> Are you refering to evict in the sense of memory reclaim under pressure ? > >> > >> So the way it flows for memory pressure is that if device driver want to > >> make room it can evict stuff to system memory and if there is not enough > > > > Yes, I mean this. > > So every driver have to maintain their own LRU-similar list instead of > > reuse what already in linux kernel.
Regarding LRU it is again not as easy. First we do necessarily have access information like CPU page table for device page table. Second the mmu_notifier callback on per page basis is costly. Finaly device are use differently than CPU, usualy you schedule a job and once that job is done you can safely evict memory it was using. Existing device driver already have quite large memory management code of their own because of that different usage model. LRU might make sense at one point but so far i doubt it is the right solution for device memory. > > And how HMM-CDM can handle multiple devices or device with multiple > device memories(may with different properties also)? > This kind of hardware platform would be very common when CCIX is out soon. A) Multiple device is under control of device driver. Multiple devices link to each other through dedicated link can have themself a complex topology and remote access between device is highly tie to the device (how to program the device mmu and device registers) and thus to the device driver. If we identify common design pattern between different hardware then we might start thinking about factoring out some common code to help those cases. B) Multiple different device is an harder problem. Each device provide their own userspace API and that is through that API that you will get memory placement advise. If several device fight for placement of same chunk of memory one can argue that the application is broken or device is broken. But for now we assume that device and application will behave. Rate limiting migration is hard, you need to keep migration statistics and that need memory. So unless we really need to do that i would rather avoid doing that. Again this is a thing for which we will have to wait and see how thing panout. Maybe i should stress that HMM is a set of helpers for device memory and it is not intended to be a policy maker or to manage device memory. Intention is that device driver will keep managing device memory as they already do today. A deeper integration with process memory management is probably bound to happen but for now it is just about having toolbox for device driver. Jérôme
