On Tuesday, April 30, 2013 05:38:38 PM Toshi Kani wrote:
> On Mon, 2013-04-29 at 14:26 +0200, Rafael J. Wysocki wrote:
> > From: Rafael J. Wysocki <rafael.j.wyso...@intel.com>
> >
> > In some cases, graceful hot-removal of devices is not possible,
> > although in principle the devices in question support hotplug.
> > For example, that may happen for the last CPU in the system or
> > for memory modules holding kernel memory.
> >
> > In those cases it is nice to be able to check if the given device
> > can be safely hot-removed before triggering a removal procedure
> > that cannot be aborted or reversed. Unfortunately, however, the
> > kernel currently doesn't provide any support for that.
> >
> > To address that deficiency, introduce support for offline and
> > online operations that can be performed on devices, respectively,
> > before a hot-removal and in case when it is necessary (or convenient)
> > to put a device back online after a successful offline (that has not
> > been followed by removal). The idea is that the offline will fail
> > whenever the given device cannot be gracefully removed from the
> > system and it will not be allowed to use the device after a
> > successful offline (until a subsequent online) in analogy with the
> > existing CPU offline/online mechanism.
> >
> > For now, the offline and online operations are introduced at the
> > bus type level, as that should be sufficient for the most urgent use
> > cases (CPUs and memory modules). In the future, however, the
> > approach may be extended to cover some more complicated device
> > offline/online scenarios involving device drivers etc.
>
> I like this approach much better than the user space approach we
> considered before. :) My comments below.
Great! :-)
> > Signed-off-by: Rafael J. Wysocki <rafael.j.wyso...@intel.com>
> > ---
> > Documentation/ABI/testing/sysfs-devices-online | 19 +++
> > drivers/base/core.c | 134
> > +++++++++++++++++++++++++
> > include/linux/device.h | 21 +++
> > 3 files changed, 174 insertions(+)
> >
> > Index: linux-pm/include/linux/device.h
> > ===================================================================
> > --- linux-pm.orig/include/linux/device.h
> > +++ linux-pm/include/linux/device.h
> > @@ -70,6 +70,10 @@ extern void bus_remove_file(struct bus_t
> > * the specific driver's probe to initial the matched device.
> > * @remove: Called when a device removed from this bus.
> > * @shutdown: Called at shut-down time to quiesce the device.
> > + *
> > + * @online: Called to put the device back online (after offlining
> > it).
> > + * @offline: Called to put the device offline for hot-removal. May
> > fail.
> > + *
> > * @suspend: Called when a device on this bus wants to go to sleep
> > mode.
> > * @resume: Called to bring a device on this bus out of sleep mode.
> > * @pm: Power management operations of this bus, callback the
> > specific
> > @@ -103,6 +107,9 @@ struct bus_type {
> > int (*remove)(struct device *dev);
> > void (*shutdown)(struct device *dev);
> >
> > + int (*online)(struct device *dev);
> > + int (*offline)(struct device *dev);
> > +
> > int (*suspend)(struct device *dev, pm_message_t state);
> > int (*resume)(struct device *dev);
> >
> > @@ -646,6 +653,8 @@ struct acpi_dev_node {
> > * @release: Callback to free the device after all references have
> > * gone away. This should be set by the allocator of the
> > * device (i.e. the bus driver that discovered the device).
> > + * @offline_disabled: If set, the device is permanently online.
> > + * @offline: Set after successful invocation of bus type's
> > .offline().
> > *
> > * At the lowest level, every device in a Linux system is represented by an
> > * instance of struct device. The device structure contains the information
> > @@ -718,6 +727,9 @@ struct device {
> >
> > void (*release)(struct device *dev);
> > struct iommu_group *iommu_group;
> > +
> > + bool offline_disabled:1;
> > + bool offline:1;
> > };
> >
> > static inline struct device *kobj_to_dev(struct kobject *kobj)
> > @@ -853,6 +865,15 @@ extern const char *device_get_devnode(st
> > extern void *dev_get_drvdata(const struct device *dev);
> > extern int dev_set_drvdata(struct device *dev, void *data);
> >
> > +static inline bool device_supports_offline(struct device *dev)
> > +{
> > + return dev->bus && dev->bus->offline && dev->bus->online;
> > +}
> > +
> > +extern void lock_device_offline(void);
> > +extern void unlock_device_offline(void);
> > +extern int device_offline(struct device *dev);
> > +extern int device_online(struct device *dev);
> > /*
> > * Root device objects for grouping under /sys/devices
> > */
> > Index: linux-pm/drivers/base/core.c
> > ===================================================================
> > --- linux-pm.orig/drivers/base/core.c
> > +++ linux-pm/drivers/base/core.c
> > @@ -397,6 +397,40 @@ static ssize_t store_uevent(struct devic
> > static struct device_attribute uevent_attr =
> > __ATTR(uevent, S_IRUGO | S_IWUSR, show_uevent, store_uevent);
> >
> > +static ssize_t show_online(struct device *dev, struct device_attribute
> > *attr,
> > + char *buf)
> > +{
> > + bool ret;
> > +
> > + lock_device_offline();
> > + ret = !dev->offline;
> > + unlock_device_offline();
> > + return sprintf(buf, "%u\n", ret);
> > +}
> > +
> > +static ssize_t store_online(struct device *dev, struct device_attribute
> > *attr,
> > + const char *buf, size_t count)
> > +{
> > + int ret;
> > +
> > + lock_device_offline();
> > + switch (buf[0]) {
> > + case '0':
> > + ret = device_offline(dev);
> > + break;
> > + case '1':
> > + ret = device_online(dev);
> > + break;
>
> memblk has multiple types of online operations specific to memory
> devices, such as "online_kernel" and "online_movable". As memblk needs
> to be integrated into this framework for addressing the crash issue, we
> need to think about how they can be generalized into this operation.
Sure.
> > + default:
> > + ret = -EINVAL;
> > + }
> > + unlock_device_offline();
> > + return ret < 0 ? ret : count;
> > +}
> > +
> > +static struct device_attribute online_attr =
> > + __ATTR(online, S_IRUGO | S_IWUSR, show_online, store_online);
> > +
> > static int device_add_attributes(struct device *dev,
> > struct device_attribute *attrs)
> > {
> > @@ -510,6 +544,12 @@ static int device_add_attrs(struct devic
> > if (error)
> > goto err_remove_type_groups;
> >
> > + if (device_supports_offline(dev) && !dev->offline_disabled) {
> > + error = device_create_file(dev, &online_attr);
> > + if (error)
> > + goto err_remove_type_groups;
> > + }
> > +
> > return 0;
> >
> > err_remove_type_groups:
> > @@ -530,6 +570,7 @@ static void device_remove_attrs(struct d
> > struct class *class = dev->class;
> > const struct device_type *type = dev->type;
> >
> > + device_remove_file(dev, &online_attr);
> > device_remove_groups(dev, dev->groups);
> >
> > if (type)
> > @@ -1415,6 +1456,99 @@ EXPORT_SYMBOL_GPL(put_device);
> > EXPORT_SYMBOL_GPL(device_create_file);
> > EXPORT_SYMBOL_GPL(device_remove_file);
> >
> > +static DEFINE_MUTEX(device_offline_lock);
> > +
> > +void lock_device_offline(void)
> > +{
> > + mutex_lock(&device_offline_lock);
> > +}
> > +
> > +void unlock_device_offline(void)
> > +{
> > + mutex_unlock(&device_offline_lock);
> > +}
> > +
> > +static int device_check_offline(struct device *dev, void *not_used)
> > +{
> > + int ret;
> > +
> > + ret = device_for_each_child(dev, NULL, device_check_offline);
> > + if (ret)
> > + return ret;
> > +
> > + return device_supports_offline(dev) && !dev->offline ? -EBUSY : 0;
> > +}
> > +
> > +/**
> > + * device_offline - Prepare the device for hot-removal.
> > + * @dev: Device to be put offline.
> > + *
> > + * Execute the device bus type's .offline() callback, if present, to
> > prepare
> > + * the device for a subsequent hot-removal. If that succeeds, the device
> > must
> > + * not be used until either it is removed or its bus type's .online()
> > callback
> > + * is executed.
> > + *
> > + * Call under device_offline_lock.
> > + */
> > +int device_offline(struct device *dev)
> > +{
> > + int ret;
> > +
> > + if (dev->offline_disabled)
> > + return -EPERM;
> > +
> > + ret = device_for_each_child(dev, NULL, device_check_offline);
> > + if (ret)
> > + return ret;
> > +
> > + device_lock(dev);
> > + if (device_supports_offline(dev)) {
> > + if (dev->offline) {
> > + ret = 1;
> > + } else {
> > + ret = dev->bus->offline(dev);
> > + if (!ret) {
> > + kobject_uevent(&dev->kobj, KOBJ_OFFLINE);
> > + dev->offline = true;
>
> Shouldn't this offline flag be set before sending KOBJ_OFFLINE?
>
> > + }
> > + }
> > + }
> > + device_unlock(dev);
> > +
> > + return ret;
> > +}
> > +
> > +/**
> > + * device_online - Put the device back online after successful
> > device_offline().
> > + * @dev: Device to be put back online.
> > + *
> > + * If device_offline() has been successfully executed for @dev, but the
> > device
> > + * has not been removed subsequently, execute its bus type's .online()
> > callback
> > + * to indicate that the device can be used again.
>
> There is another use-case for online(). When a device like CPU is
> hot-added, it is added in offline. I am not sure why, but it has been
> this way. So, we need to call online() to make a new device available
> for use after a hot-add.
Actually, in the CPU case that is left to user space as far as I can say.
That is, the device appears initially offline and user space is supposed to
bring it online via sysfs.
> > + *
> > + * Call under device_offline_lock.
> > + */
> > +int device_online(struct device *dev)
> > +{
> > + int ret = 0;
> > +
> > + device_lock(dev);
> > + if (device_supports_offline(dev)) {
> > + if (dev->offline) {
> > + ret = dev->bus->online(dev);
> > + if (!ret) {
> > + kobject_uevent(&dev->kobj, KOBJ_ONLINE);
> > + dev->offline = false;
>
> Same comment as KOBJ_OFFLINE.
I wonder why the ordering may be important?
> > + }
> > + } else {
> > + ret = 1;
>
> This case has a problem in the hot-add use-case I mentioned above. When
> a new device is added, dev->offline is set to 0. So, device_online()
> thinks it is online already.
Then whoever adds the device needs to set dev->offline.
Thanks,
Rafael
--
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
