Re: [PATCH] scsi: fix race condition when removing target

2017-12-01 Thread Jason Yan


On 2017/12/1 7:56, James Bottomley wrote:

On Thu, 2017-11-30 at 16:08 +, Bart Van Assche wrote:

On Thu, 2017-11-30 at 09:18 +0800, Jason Yan wrote:


Hi Bart, I chose the approach in my patch because it has been used
in scsi_device_get() for years and been proved safe. I think using
kobject_get_unless_zero() is safe here and can fix this issue too.
And this approach is beneficial to all users.


Hello Jason,

A possible approach is that we start with your patch and defer any
get_device() changes until after your patch has been applied.


It's possible, but not quite good enough: the same race can be produced
with any of our sdev lists that are deleted in the release callback,
because there could be a released device on any one of them.  The only
way to mediate it properly is to get a reference in the iterator using
kobject_get_unless_zero().

It's a bit like a huge can of worms, there's another problem every time
I look.  However, this is something like the mechanism that could work
(and if get_device() ever gets fixed, we can put it in place of
kobject_get_unless_zero()).

James

---

diff --git a/drivers/scsi/53c700.c b/drivers/scsi/53c700.c
index 6be77b3aa8a5..c3246f26c02c 100644
--- a/drivers/scsi/53c700.c
+++ b/drivers/scsi/53c700.c
@@ -1169,6 +1169,7 @@ process_script_interrupt(__u32 dsps, __u32 dsp, struct 
scsi_cmnd *SCp,


}
+   put_device(&SDp->sdev_gendev);
} else if(dsps == A_RESELECTED_DURING_SELECTION) {

/* This section is full of debugging code because I've
diff --git a/drivers/scsi/esp_scsi.c b/drivers/scsi/esp_scsi.c
index c3fc34b9964d..7736f3fb2501 100644
--- a/drivers/scsi/esp_scsi.c
+++ b/drivers/scsi/esp_scsi.c
@@ -1198,6 +1198,7 @@ static int esp_reconnect(struct esp *esp)
goto do_reset;
}
lp = dev->hostdata;
+   put_device(&dev->sdev_gendev);

ent = lp->non_tagged_cmd;
if (!ent) {
diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
index a7e4fba724b7..c96c11716152 100644
--- a/drivers/scsi/scsi.c
+++ b/drivers/scsi/scsi.c
@@ -677,11 +677,10 @@ struct scsi_device *__scsi_device_lookup_by_target(struct 
scsi_target *starget,
  {
struct scsi_device *sdev;

-   list_for_each_entry(sdev, &starget->devices, same_target_siblings) {
-   if (sdev->sdev_state == SDEV_DEL)
-   continue;
-   if (sdev->lun ==lun)
+   __sdev_for_each_get(sdev, &starget->devices, same_target_siblings) {
+   if (sdev->sdev_state != SDEV_DEL && sdev->lun ==lun)
return sdev;
+   put_device(&sdev->sdev_gendev);
}

return NULL;
@@ -700,15 +699,16 @@ EXPORT_SYMBOL(__scsi_device_lookup_by_target);
  struct scsi_device *scsi_device_lookup_by_target(struct scsi_target *starget,
 u64 lun)
  {
-   struct scsi_device *sdev;
+  struct scsi_device *sdev, *sdev_copy;
struct Scsi_Host *shost = dev_to_shost(starget->dev.parent);
unsigned long flags;

spin_lock_irqsave(shost->host_lock, flags);
-   sdev = __scsi_device_lookup_by_target(starget, lun);
+   sdev_copy = sdev = __scsi_device_lookup_by_target(starget, lun);
+   spin_unlock_irqrestore(shost->host_lock, flags);
if (sdev && scsi_device_get(sdev))
sdev = NULL;
-   spin_unlock_irqrestore(shost->host_lock, flags);
+   put_device(&sdev_copy->sdev_gendev);

return sdev;
  }
@@ -735,12 +735,12 @@ struct scsi_device *__scsi_device_lookup(struct Scsi_Host 
*shost,
  {
struct scsi_device *sdev;

-   list_for_each_entry(sdev, &shost->__devices, siblings) {
-   if (sdev->sdev_state == SDEV_DEL)
-   continue;
-   if (sdev->channel == channel && sdev->id == id &&
-   sdev->lun ==lun)
+   __sdev_for_each_get(sdev, &shost->__devices, siblings) {
+   if (sdev->sdev_state != SDEV_DEL &&
+   sdev->channel == channel && sdev->id == id &&
+   sdev->lun ==lun)
return sdev;
+   put_device(&sdev->sdev_gendev);
}

return NULL;
@@ -761,14 +761,15 @@ EXPORT_SYMBOL(__scsi_device_lookup);
  struct scsi_device *scsi_device_lookup(struct Scsi_Host *shost,
uint channel, uint id, u64 lun)
  {
-   struct scsi_device *sdev;
+  struct scsi_device *sdev, *sdev_copy;
unsigned long flags;

spin_lock_irqsave(shost->host_lock, flags);
-   sdev = __scsi_device_lookup(shost, channel, id, lun);
+   sdev_copy = sdev = __scsi_device_lookup(shost, channel, id, lun);
+   spin_unlock_irqrestore(shost->host_lock, flags);
if (sdev && scsi_device_get(sdev))
sdev = NULL;
-   spin_unlock_irqrestore(shost->host_lock, flags);
+   put_device(&sdev_copy->sdev_gendev);

return sde

Re: [PATCH 1/2] Ensure that the SCSI error handler gets woken up

2017-12-01 Thread Pavel Tikhomirov



On 12/01/2017 01:44 AM, Bart Van Assche wrote:

If scsi_eh_scmd_add() is called concurrently with
scsi_host_queue_ready() while shost->host_blocked > 0 then it can
happen that neither function wakes up the SCSI error handler. Fix
this by making every function that decreases the host_busy counter
wake up the error handler if necessary and by protecting the
host_failed checks with the SCSI host lock.

Reported-by: Pavel Tikhomirov 
Fixes: commit 746650160866 ("scsi: convert host_busy to atomic_t")
Signed-off-by: Bart Van Assche 
Cc: Konstantin Khorenko 
Cc: Stuart Hayes 
Cc: Pavel Tikhomirov 
Cc: Christoph Hellwig 
Cc: Hannes Reinecke 
Cc: Johannes Thumshirn 
Cc: 
---
  drivers/scsi/scsi_error.c |  8 +++-
  drivers/scsi/scsi_lib.c   | 39 ---
  2 files changed, 35 insertions(+), 12 deletions(-)

diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
index 5e89049e9b4e..b22a9a23c74c 100644
--- a/drivers/scsi/scsi_error.c
+++ b/drivers/scsi/scsi_error.c
@@ -233,19 +233,25 @@ static void scsi_eh_reset(struct scsi_cmnd *scmd)
  void scsi_eh_scmd_add(struct scsi_cmnd *scmd)
  {
struct Scsi_Host *shost = scmd->device->host;
+   enum scsi_host_state shost_state;
unsigned long flags;
int ret;
  
  	WARN_ON_ONCE(!shost->ehandler);
  
  	spin_lock_irqsave(shost->host_lock, flags);

+   shost_state = shost->shost_state;
if (scsi_host_set_state(shost, SHOST_RECOVERY)) {
ret = scsi_host_set_state(shost, SHOST_CANCEL_RECOVERY);
WARN_ON_ONCE(ret);
}
if (shost->eh_deadline != -1 && !shost->last_reset)
shost->last_reset = jiffies;
-
+   if (shost_state != shost->shost_state) {
+   spin_unlock_irqrestore(shost->host_lock, flags);
+   synchronize_rcu();


We can come here from interrupt context, so may be we should use 
call_rcu() here instead, possible backtrace:


 => scsi_eh_scmd_add
 => scsi_times_out
 => blk_rq_timed_out
 => blk_abort_request
 => ata_qc_schedule_eh
 => ata_qc_complete
 => ata_do_link_abort
 => ata_port_abort
 => ahci_handle_port_interrupt
 => ahci_single_irq_intr
 => __handle_irq_event_percpu
 => handle_irq_event_percpu
 => handle_irq_event
 => handle_edge_irq
 => handle_irq
 => do_IRQ


+   spin_lock_irqsave(shost->host_lock, flags);
+   }
scsi_eh_reset(scmd);
list_add_tail(&scmd->eh_entry, &shost->eh_cmd_q);
shost->host_failed++;
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index b6d3842b6809..7d18fb245d7d 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -318,22 +318,39 @@ static void scsi_init_cmd_errh(struct scsi_cmnd *cmd)
cmd->cmd_len = scsi_command_size(cmd->cmnd);
  }
  
-void scsi_device_unbusy(struct scsi_device *sdev)

+/*
+ * Decrement the host_busy counter and wake up the error handler if necessary.
+ * Avoid as follows that the error handler is not woken up if shost->host_busy
+ * == shost->host_failed: use synchronize_rcu() in scsi_eh_scmd_add() in
+ * combination with an RCU read lock in this function to ensure that this
+ * function in its entirety either finishes before scsi_eh_scmd_add()
+ * increases the host_failed counter or that it notices the shost state change
+ * made by scsi_eh_scmd_add().
+ */
+static void scsi_dec_host_busy(struct Scsi_Host *shost)
  {
-   struct Scsi_Host *shost = sdev->host;
-   struct scsi_target *starget = scsi_target(sdev);
unsigned long flags;
  
+	rcu_read_lock();

atomic_dec(&shost->host_busy);
-   if (starget->can_queue > 0)
-   atomic_dec(&starget->target_busy);
-
-   if (unlikely(scsi_host_in_recovery(shost) &&
-(shost->host_failed || shost->host_eh_scheduled))) {
+   if (unlikely(scsi_host_in_recovery(shost))) {
spin_lock_irqsave(shost->host_lock, flags);
-   scsi_eh_wakeup(shost);
+   if (shost->host_failed || shost->host_eh_scheduled)
+   scsi_eh_wakeup(shost);
spin_unlock_irqrestore(shost->host_lock, flags);
}
+   rcu_read_unlock();
+}
+
+void scsi_device_unbusy(struct scsi_device *sdev)
+{
+   struct Scsi_Host *shost = sdev->host;
+   struct scsi_target *starget = scsi_target(sdev);
+
+   scsi_dec_host_busy(shost);
+
+   if (starget->can_queue > 0)
+   atomic_dec(&starget->target_busy);
  
  	atomic_dec(&sdev->device_busy);

  }
@@ -1531,7 +1548,7 @@ static inline int scsi_host_queue_ready(struct 
request_queue *q,
list_add_tail(&sdev->starved_entry, &shost->starved_list);
spin_unlock_irq(shost->host_lock);
  out_dec:
-   atomic_dec(&shost->host_busy);
+   scsi_dec_host_busy(shost);
return 0;
  }
  
@@ -2017,7 +2034,7 @@ static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,

return BLK_STS_OK;
  
  out_dec_host_busy:

-   atomic_

Re: [PATCH 1/2] Ensure that the SCSI error handler gets woken up

2017-12-01 Thread Johannes Thumshirn
Hi Bart,

Bart Van Assche  writes:
[...]

> + if (shost_state != shost->shost_state) {
> + spin_unlock_irqrestore(shost->host_lock, flags);
> + synchronize_rcu();
> + spin_lock_irqsave(shost->host_lock, flags);
> + }

Plese correct me if I'm wrong, but once you drop the host lock all
assumptions about states it protects are void, aren't they? 

-- 
Johannes Thumshirn  Storage
jthumsh...@suse.de+49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


Re: [PATCH 2/2] Convert a source code comment into a runtime check

2017-12-01 Thread Johannes Thumshirn
Looks good,
Reviewed-by: Johannes Thumshirn 
-- 
Johannes Thumshirn  Storage
jthumsh...@suse.de+49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850


[PATCH] scsi: bnx2fc: fix spelling mistake: "Couldnt" -> "Couldn't"

2017-12-01 Thread Colin King
From: Colin Ian King 

Trivial fix to spelling mistake in error message text.

Signed-off-by: Colin Ian King 
---
 drivers/scsi/bnx2fc/bnx2fc_fcoe.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/bnx2fc/bnx2fc_fcoe.c 
b/drivers/scsi/bnx2fc/bnx2fc_fcoe.c
index e6b9de7d41ac..65de1d0578a1 100644
--- a/drivers/scsi/bnx2fc/bnx2fc_fcoe.c
+++ b/drivers/scsi/bnx2fc/bnx2fc_fcoe.c
@@ -1552,7 +1552,7 @@ static struct fc_lport *bnx2fc_if_create(struct 
bnx2fc_interface *interface,
 
rc = bnx2fc_shost_config(lport, parent);
if (rc) {
-   printk(KERN_ERR PFX "Couldnt configure shost for %s\n",
+   printk(KERN_ERR PFX "Couldn't configure shost for %s\n",
interface->netdev->name);
goto lp_config_err;
}
@@ -1560,7 +1560,7 @@ static struct fc_lport *bnx2fc_if_create(struct 
bnx2fc_interface *interface,
/* Initialize the libfc library */
rc = bnx2fc_libfc_config(lport);
if (rc) {
-   printk(KERN_ERR PFX "Couldnt configure libfc\n");
+   printk(KERN_ERR PFX "Couldn't configure libfc\n");
goto shost_err;
}
fc_host_port_type(lport->host) = FC_PORTTYPE_UNKNOWN;
-- 
2.14.1



[PATCH] scsi: csiostor: fix spelling mistake: "Couldnt" -> "Couldn't"

2017-12-01 Thread Colin King
From: Colin Ian King 

Trivial fix to spelling mistake in error message text.

Signed-off-by: Colin Ian King 
---
 drivers/scsi/csiostor/csio_mb.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/csiostor/csio_mb.c b/drivers/scsi/csiostor/csio_mb.c
index 931b1d8f9f3e..5f4e0a787bd1 100644
--- a/drivers/scsi/csiostor/csio_mb.c
+++ b/drivers/scsi/csiostor/csio_mb.c
@@ -1216,7 +1216,7 @@ csio_mb_issue(struct csio_hw *hw, struct csio_mb *mbp)
/* Queue mbox cmd, if another mbox cmd is active */
if (mbp->mb_cbfn == NULL) {
rv = -EBUSY;
-   csio_dbg(hw, "Couldnt own Mailbox %x op:0x%x\n",
+   csio_dbg(hw, "Couldn't own Mailbox %x op:0x%x\n",
hw->pfn, *((uint8_t *)mbp->mb));
 
goto error_out;
@@ -1244,14 +1244,14 @@ csio_mb_issue(struct csio_hw *hw, struct csio_mb *mbp)
rv = owner ? -EBUSY : -ETIMEDOUT;
 
csio_dbg(hw,
-"Couldnt own Mailbox %x op:0x%x "
+"Couldn't own Mailbox %x op:0x%x "
 "owner:%x\n",
 hw->pfn, *((uint8_t *)mbp->mb), owner);
goto error_out;
} else {
if (mbm->mcurrent == NULL) {
csio_err(hw,
-"Couldnt own Mailbox %x "
+"Couldn't own Mailbox %x "
 "op:0x%x owner:%x\n",
 hw->pfn, *((uint8_t *)mbp->mb),
 owner);
-- 
2.14.1



[PATCH] scsi: ipr: fix incorrect indentation of assignment statement

2017-12-01 Thread Colin King
From: Colin Ian King 

Remove one extraneous level of indentation on an assignment statement.

Signed-off-by: Colin Ian King 
---
 drivers/scsi/ipr.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/ipr.c b/drivers/scsi/ipr.c
index cc0187965eee..e07dd990e585 100644
--- a/drivers/scsi/ipr.c
+++ b/drivers/scsi/ipr.c
@@ -9653,8 +9653,8 @@ static int ipr_alloc_cmd_blks(struct ipr_ioa_cfg *ioa_cfg)
if (i == 0) {
entries_each_hrrq = IPR_NUM_INTERNAL_CMD_BLKS;
ioa_cfg->hrrq[i].min_cmd_id = 0;
-   ioa_cfg->hrrq[i].max_cmd_id =
-   (entries_each_hrrq - 1);
+   ioa_cfg->hrrq[i].max_cmd_id =
+   (entries_each_hrrq - 1);
} else {
entries_each_hrrq =
IPR_NUM_BASE_CMD_BLKS/
-- 
2.14.1



Re: [PATCH] scsi: fix race condition when removing target

2017-12-01 Thread Ewan D. Milne
We have another test case that demonstrates this issue involving
duplicate invocations of scsi_device_dev_release() on the same
device.  This other test case involves repeated log in / log out
of an iSCSI target.  (The first test case I mentioned in an earlier
mail was an oscillating FC port with a low dev_loss_tmo value.)

The iSCSI test was not fixed by Jason Yan's patch, however adding Bart's
change to use kobject_get_unless_zero() in get_device() as well seems to
have resolved it.  We are going to try with just Bart's change next.

-Ewan




Re: [PATCH] scsi: fix race condition when removing target

2017-12-01 Thread James Bottomley
On Fri, 2017-12-01 at 16:40 +0800, Jason Yan wrote:
> On 2017/12/1 7:56, James Bottomley wrote:
> > b/include/scsi/scsi_device.h
> > index 571ddb49b926..2e4d48d8cd68 100644
> > --- a/include/scsi/scsi_device.h
> > +++ b/include/scsi/scsi_device.h
> > @@ -380,6 +380,23 @@ extern struct scsi_device
> > *__scsi_iterate_devices(struct Scsi_Host *,
> >   #define __shost_for_each_device(sdev, shost) \
> >     list_for_each_entry((sdev), &((shost)->__devices),
> > siblings)
> > 
> 
> Seems that __shost_for_each_device() is still not safe. scsi device
> been deleted stays in the list and put_device() can be called
> anywhere out of the host lock.

Not if it's used with scsi_get_device().  As I said, I only did a
cursory inspectiont, so if I've missed a loop, please specify.

The point was more a demonstration of how we could fix the problem if
we don't change get_device().

James



Re: [PATCH] scsi: ipr: fix incorrect indentation of assignment statement

2017-12-01 Thread Brian King
On 12/01/2017 07:33 AM, Colin King wrote:
> From: Colin Ian King 
> 
> Remove one extraneous level of indentation on an assignment statement.
> 
> Signed-off-by: Colin Ian King 
> ---
>  drivers/scsi/ipr.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/scsi/ipr.c b/drivers/scsi/ipr.c
> index cc0187965eee..e07dd990e585 100644
> --- a/drivers/scsi/ipr.c
> +++ b/drivers/scsi/ipr.c
> @@ -9653,8 +9653,8 @@ static int ipr_alloc_cmd_blks(struct ipr_ioa_cfg 
> *ioa_cfg)
>   if (i == 0) {
>   entries_each_hrrq = IPR_NUM_INTERNAL_CMD_BLKS;
>   ioa_cfg->hrrq[i].min_cmd_id = 0;
> - ioa_cfg->hrrq[i].max_cmd_id =
> - (entries_each_hrrq - 1);
> + ioa_cfg->hrrq[i].max_cmd_id =
> + (entries_each_hrrq - 1);
>   } else {
>   entries_each_hrrq =
>   IPR_NUM_BASE_CMD_BLKS/
> 

Acked-by: Brian King 

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center



crash in iscsi/scsi initiator with linux-4.15.0-rc1

2017-12-01 Thread Steve Wise
Hey,

I'm  seeing this null pointer dereference with linux-4.15.0-rc1.  To reproduce
it, I connect two ram disks via iscsi/TCP, and start an fio:

iscsiadm -m discovery --op update --type sendtargets -p 172.16.1.10:3260
iscsiadm -m node -p 172.16.1.10:3260 -l
ISCSI_DISKS=/dev/sdd:/dev/sde; fio --rw=randrw --name=random --norandommap
--ioengine=libaio --size=400m --group_reporting --exitall --fsync_on_close=1
--invalidate=1 --direct=1 --filename=$ISCSI_DISKS --time_based --runtime=300
--iodepth=128 --numjobs=8 --unit_base=1 --bs=64k --kb_base=1000

Then on the initiator node, while the fio test is running, I detach the devices:

iscsiadm -m node -p 172.16.1.10:3260 -I iser -u

Then I hit this crash.  Has anyone else encountered this issue?  Wondering if
there is a fix handy. :)

Thanks,

Steve.



[  127.175953] scsi 8:0:0:0: alua: Detached
[  127.175955] scsi 8:0:0:0: alua: Detached
[  127.175981] [ cut here ]
[  127.175984] list_del corruption. prev->next should be 8803382f1240, but
was 88039ab0f780
[  127.176010] WARNING: CPU: 5 PID: 373 at lib/list_debug.c:53
__list_del_entry_valid+0x7c/0xa0
[  127.176011] Modules linked in: iscsi_tcp libiscsi_tcp rpcrdma ib_isert
iscsi_target_mod libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp
scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm
iw_cm libcxgb mlx5_ib ext4 ib_core dm_mirror dm_region_hash dm_log dm_mod
mbcache jbd2 coretemp kvm iTCO_wdt ppdev irqbypass iTCO_vendor_support gpio_ich
i2c_i801 pcspkr lpc_ich parport_pc i5400_edac sg parport i5k_amb shpchp nfsd
auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod nouveau
cdrom sd_mod ata_generic pata_acpi video mxm_wmi wmi drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm mlx5_core drm igb cxgb4 ahci firewire_ohci
ata_piix libahci firewire_core dca i2c_algo_bit devlink libata ptp serio_raw
i2c_core crc_itu_t pps_core [last unloaded: ib_iser]
[  127.176055] CPU: 5 PID: 373 Comm: kworker/u16:4 Not tainted 4.15.0-rc1+ #6
[  127.176056] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
[  127.176074] Workqueue: scsi_wq_9 __iscsi_unbind_session
[scsi_transport_iscsi]
[  127.176075] task: 88039a498000 task.stack: c9000288
[  127.176076] RIP: 0010:__list_del_entry_valid+0x7c/0xa0
[  127.176076] RSP: 0018:c90002883d38 EFLAGS: 00010082
[  127.176077] RAX:  RBX: 8803382f1240 RCX: 
[  127.176078] RDX: 0001 RSI: 0002 RDI: 0092
[  127.176079] RBP: 8803982129c0 R08: 0054 R09: 823d60e0
[  127.176079] R10: 0473 R11:  R12: 880398212800
[  127.176080] R13: 880396701800 R14: 880396701800 R15: 8801afc31000
[  127.176081] FS:  () GS:8803bfd4()
knlGS:
[  127.176082] CS:  0010 DS:  ES:  CR0: 80050033
[  127.176083] CR2: 7f6a80028038 CR3: 00039a957000 CR4: 06e0
[  127.176084] Call Trace:
[  127.176091]  alua_bus_detach+0x5c/0xc0
[  127.176095]  scsi_dh_release_device+0x18/0x50
[  127.176098]  scsi_device_dev_release_usercontext+0x25/0x230
[  127.176107]  execute_in_process_context+0x58/0x60
[  127.176110]  device_release+0x2d/0x80
[  127.176113]  kobject_cleanup+0x5e/0x180
[  127.176115]  scsi_remove_target+0x16b/0x1b0
[  127.176119]  __iscsi_unbind_session+0xb3/0x160 [scsi_transport_iscsi]
[  127.176121]  process_one_work+0x141/0x340
[  127.176123]  worker_thread+0x47/0x3e0
[  127.176124]  kthread+0xf5/0x130
[  127.176126]  ? rescuer_thread+0x380/0x380
[  127.176127]  ? kthread_associate_blkcg+0x90/0x90
[  127.176129]  ret_from_fork+0x1f/0x30
[  127.176130] Code: ff 31 c0 c3 48 89 fe 31 c0 48 c7 c7 60 19 a9 81 e8 3a 33 d0
ff 0f ff 31 c0 c3 48 89 fe 31 c0 48 c7 c7 20 19 a9 81 e8 24 33 d0 ff <0f> ff 31
c0 c3 48 89 fe 31 c0 48 c7 c7 e8 18 a9 81 e8 0e 33 d0
[  127.176145] ---[ end trace e7e378e0f32966e0 ]---
[  127.176148] scsi 9:0:0:0: alua: Detached
[  127.466362] BUG: unable to handle kernel NULL pointer dereference at
(null)
[  127.474355] IP: _raw_spin_lock_irqsave+0x1e/0x40
[  127.479136] PGD 399e70067 P4D 399e70067 PUD 3966cd067 PMD 0
[  127.484961] Oops: 0002 [#1] SMP
[  127.488269] Modules linked in: iscsi_tcp libiscsi_tcp rpcrdma ib_isert
iscsi_target_mod libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp
scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm
iw_cm libcxgb mlx5_ib ext4 ib_core dm_mirror dm_region_hash dm_log dm_mod
mbcache jbd2 coretemp kvm iTCO_wdt ppdev irqbypass iTCO_vendor_support gpio_ich
i2c_i801 pcspkr lpc_ich parport_pc i5400_edac sg parport i5k_amb shpchp nfsd
auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod nouveau
cdrom sd_mod ata_generic pata_acpi video mxm_wmi wmi drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm mlx5_core drm igb cxgb4 ahci firewire_ohci
ata_piix libahci firewire_c

Re: [PATCH 1/2] Ensure that the SCSI error handler gets woken up

2017-12-01 Thread Bart Van Assche
On Fri, 2017-12-01 at 09:45 +0100, Johannes Thumshirn wrote:
> Bart Van Assche  writes:
> [...]
> 
> > +   if (shost_state != shost->shost_state) {
> > +   spin_unlock_irqrestore(shost->host_lock, flags);
> > +   synchronize_rcu();
> > +   spin_lock_irqsave(shost->host_lock, flags);
> > +   }
> 
> Plese correct me if I'm wrong, but once you drop the host lock all
> assumptions about states it protects are void, aren't they? 

Hello Johannes,

That's a good question. I think it is safe to drop the host lock at that point
because waking up the error handler thread will only happen after host_failed
has been incremented.

Bart.

Re: [PATCH 1/2] Ensure that the SCSI error handler gets woken up

2017-12-01 Thread Bart Van Assche
On Fri, 2017-12-01 at 11:42 +0300, Pavel Tikhomirov wrote:
> On 12/01/2017 01:44 AM, Bart Van Assche wrote:
> > +   if (shost_state != shost->shost_state) {
> > +   spin_unlock_irqrestore(shost->host_lock, flags);
> > +   synchronize_rcu();
> 
> We can come here from interrupt context, so may be we should use 
> call_rcu() here instead.

Hello Pavel,

I will rework this patch such that it uses call_rcu() instead of
synchronize_rcu().

Bart.

Re: crash in iscsi/scsi initiator with linux-4.15.0-rc1

2017-12-01 Thread Ewan D. Milne
On Fri, 2017-12-01 at 11:00 -0600, Steve Wise wrote:
> Hey,
> 
> I'm  seeing this null pointer dereference with linux-4.15.0-rc1.  To reproduce
> it, I connect two ram disks via iscsi/TCP, and start an fio:
> 
> iscsiadm -m discovery --op update --type sendtargets -p 172.16.1.10:3260
> iscsiadm -m node -p 172.16.1.10:3260 -l
> ISCSI_DISKS=/dev/sdd:/dev/sde; fio --rw=randrw --name=random --norandommap
> --ioengine=libaio --size=400m --group_reporting --exitall --fsync_on_close=1
> --invalidate=1 --direct=1 --filename=$ISCSI_DISKS --time_based --runtime=300
> --iodepth=128 --numjobs=8 --unit_base=1 --bs=64k --kb_base=1000
> 
> Then on the initiator node, while the fio test is running, I detach the 
> devices:
> 
> iscsiadm -m node -p 172.16.1.10:3260 -I iser -u
> 
> Then I hit this crash.  Has anyone else encountered this issue?  Wondering if
> there is a fix handy. :)
> 

This is the same problem that is being discussed under the thread:
"[PATCH] scsi: fix race condition when removing target".

We had good test results with both Jason Yan's patch and Bart's patch
applied, however the ultimate solution is still in progress, see James'
comments.

You could also try reverting fbce4d97fd "scsi: fixup kernel warning
during rmmod()" if you just need to get past this.

-Ewan

> Thanks,
> 
> Steve.
> 
> 
> 
> [  127.175953] scsi 8:0:0:0: alua: Detached
> [  127.175955] scsi 8:0:0:0: alua: Detached
> [  127.175981] [ cut here ]
> [  127.175984] list_del corruption. prev->next should be 8803382f1240, but
> was 88039ab0f780
> [  127.176010] WARNING: CPU: 5 PID: 373 at lib/list_debug.c:53
> __list_del_entry_valid+0x7c/0xa0
> [  127.176011] Modules linked in: iscsi_tcp libiscsi_tcp rpcrdma ib_isert
> iscsi_target_mod libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp
> scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm
> iw_cm libcxgb mlx5_ib ext4 ib_core dm_mirror dm_region_hash dm_log dm_mod
> mbcache jbd2 coretemp kvm iTCO_wdt ppdev irqbypass iTCO_vendor_support 
> gpio_ich
> i2c_i801 pcspkr lpc_ich parport_pc i5400_edac sg parport i5k_amb shpchp nfsd
> auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod nouveau
> cdrom sd_mod ata_generic pata_acpi video mxm_wmi wmi drm_kms_helper 
> syscopyarea
> sysfillrect sysimgblt fb_sys_fops ttm mlx5_core drm igb cxgb4 ahci 
> firewire_ohci
> ata_piix libahci firewire_core dca i2c_algo_bit devlink libata ptp serio_raw
> i2c_core crc_itu_t pps_core [last unloaded: ib_iser]
> [  127.176055] CPU: 5 PID: 373 Comm: kworker/u16:4 Not tainted 4.15.0-rc1+ #6
> [  127.176056] Hardware name: Supermicro X7DWA/X7DWA, BIOS 6.00 12/21/2007
> [  127.176074] Workqueue: scsi_wq_9 __iscsi_unbind_session
> [scsi_transport_iscsi]
> [  127.176075] task: 88039a498000 task.stack: c9000288
> [  127.176076] RIP: 0010:__list_del_entry_valid+0x7c/0xa0
> [  127.176076] RSP: 0018:c90002883d38 EFLAGS: 00010082
> [  127.176077] RAX:  RBX: 8803382f1240 RCX: 
> 
> [  127.176078] RDX: 0001 RSI: 0002 RDI: 
> 0092
> [  127.176079] RBP: 8803982129c0 R08: 0054 R09: 
> 823d60e0
> [  127.176079] R10: 0473 R11:  R12: 
> 880398212800
> [  127.176080] R13: 880396701800 R14: 880396701800 R15: 
> 8801afc31000
> [  127.176081] FS:  () GS:8803bfd4()
> knlGS:
> [  127.176082] CS:  0010 DS:  ES:  CR0: 80050033
> [  127.176083] CR2: 7f6a80028038 CR3: 00039a957000 CR4: 
> 06e0
> [  127.176084] Call Trace:
> [  127.176091]  alua_bus_detach+0x5c/0xc0
> [  127.176095]  scsi_dh_release_device+0x18/0x50
> [  127.176098]  scsi_device_dev_release_usercontext+0x25/0x230
> [  127.176107]  execute_in_process_context+0x58/0x60
> [  127.176110]  device_release+0x2d/0x80
> [  127.176113]  kobject_cleanup+0x5e/0x180
> [  127.176115]  scsi_remove_target+0x16b/0x1b0
> [  127.176119]  __iscsi_unbind_session+0xb3/0x160 [scsi_transport_iscsi]
> [  127.176121]  process_one_work+0x141/0x340
> [  127.176123]  worker_thread+0x47/0x3e0
> [  127.176124]  kthread+0xf5/0x130
> [  127.176126]  ? rescuer_thread+0x380/0x380
> [  127.176127]  ? kthread_associate_blkcg+0x90/0x90
> [  127.176129]  ret_from_fork+0x1f/0x30
> [  127.176130] Code: ff 31 c0 c3 48 89 fe 31 c0 48 c7 c7 60 19 a9 81 e8 3a 33 
> d0
> ff 0f ff 31 c0 c3 48 89 fe 31 c0 48 c7 c7 20 19 a9 81 e8 24 33 d0 ff <0f> ff 
> 31
> c0 c3 48 89 fe 31 c0 48 c7 c7 e8 18 a9 81 e8 0e 33 d0
> [  127.176145] ---[ end trace e7e378e0f32966e0 ]---
> [  127.176148] scsi 9:0:0:0: alua: Detached
> [  127.466362] BUG: unable to handle kernel NULL pointer dereference at
> (null)
> [  127.474355] IP: _raw_spin_lock_irqsave+0x1e/0x40
> [  127.479136] PGD 399e70067 P4D 399e70067 PUD 3966cd067 PMD 0
> [  127.484961] Oops: 0002 [#1] SMP
> [  127.488269] Modules linked in: iscsi_tcp libiscsi_tcp rpc

RE: crash in iscsi/scsi initiator with linux-4.15.0-rc1

2017-12-01 Thread Steve Wise
> > Then I hit this crash.  Has anyone else encountered this issue?  Wondering 
> > if
> > there is a fix handy. :)
> >
> 
> This is the same problem that is being discussed under the thread:
> "[PATCH] scsi: fix race condition when removing target".
> 
> We had good test results with both Jason Yan's patch and Bart's patch
> applied, however the ultimate solution is still in progress, see James'
> comments.
> 
> You could also try reverting fbce4d97fd "scsi: fixup kernel warning
> during rmmod()" if you just need to get past this.
> 
> -Ewan


Thanks Ewan, I'll back up that commit just to verify I'm seeing the same issue. 
 I'm also happy to test any final fix.

Steve.



★ lintondowntown --“2018世界复合材料展览及会议”将于“3月”在“法国巴黎”举行 (地右P1-L-Me)

2017-12-01 Thread linux-scsi-owner
尊敬的 lintondownt...@megagate.com 企业领导/公司负责人/业界专家,您好:
  
  
新材料为21世纪三大共性关键技术之一,已成为全球经济迅速增长的源动力和提升核心竞争力的战略焦点。材料作为制造业的基础,特别是新材料研究和产业发展的水平与规模,已经成为衡量一个国家科技进步和综合实力的重要标志。在新材料发展与应用中,复合材料占有相当重要的地位,特别广泛的应用在汽车、交通、风能、航空、航天、兵器、船舶、国防、机械、电子、化工、建筑、农业、渔业、纺织、运动器材等领域,一直是世界各国优先发展和竞争激烈的重要行业。
  
  “JEC世界复合材料展览及会议”(JEC world Composites Show & 
Conferences)创办于1963年,每年举办一届,至2017年总共举办了52届,主办单位是法国JEC复合材料发展促进会/JEC集团,中国总展团展商组织单位为映德国际会展集团中国代表处,在北京、上海等地设有分支机构,负责该展会在中国的推广和招商工作(JEC中国总展团报名热线:4000-680-860转8144、5220)。JEC复合材料展已成为世界上历史最悠久、规模最大的复合材料行业专业展览会,展示和反映了当前复合材料行业的最新技术和应用成果。
  
  为了增进国内外复合材料行业的交流与合作,同时展示我国复合材料产业的发展与成就,帮助境内企业开拓国内外市场,中国国际复材协会、映德国际会展集团(YOND 
EXPO)中国代表处已近十年组织中国企业参与该展会,为中国复合材料集团、中材科技、中钢集团、中国建材集团、中国商飞、北京玻钢院、上海杰事杰新材料集团、重庆国际复合材料、中南控股集团、秦皇岛耀华玻璃钢、烟台氨纶、天马集团、华东理工大学、哈尔滨工业大学、巨石集团、中冶集团、金光集团、江苏恒神纤维材料、重庆大学、上海玻璃钢研究院、中南大学、哈尔滨玻璃钢研究院等众多行业巨头和知名机构提供了优质高效的境外展贸服务。
  
  “JEC world 2018 
第五十三世界复合材料展览及会议”将于“3月06-08日”在“法国巴黎展览会议中心”再度举行,我们诚邀全国各地复合材料及新材料相关单位与业界人士加入咱们的中国总展团前往参展参观。
  
  
  有关参展参观“JEC世界复合材料展”事宜,请联络【中国总展团】组办方—— 
全国统一客服热线:4000-580-850(转5220、8144、)、010—6923-6944; 邮箱/QQ:12809395#qq.com; 
微信: CanZhanXiaoXi(参展消息)、ZhanShangZhiJia(展商之家); 
微博:http://weibo.com/jecshow(展会)、http://weibo.com/yingdehuizhan(公司)。
  
  参加JEC展会是一个复合材料及新材料企业走向国际化的标志和途径!
  
  
  
  
__
  
(百万群发系统|为您发送|如不希望再收到此行业资讯|请回复“TD+JEC”至邮箱1055800...@qq.com)


Re: [PATCH] bsg: update bsg_device_list to utilize static hash table implementation.

2017-12-01 Thread Tim Hansen
This patch was submitted as an effort to standardize on Sasha's hashtable
implementation.

Just a friendly ping to get some comments on this patch, been about 2
months with no comments on it at all.  Are there any changes to be
requested or issues raised with this change?


答复:1...@2099099.htm——InnoTrans 2018 第十二届德国柏林国际轨道交通技术展  1...@2099099.htm

2017-12-01 Thread 玻纤布
1440
(回信专用接收邮箱“12809...@qq.com”报名参展)
  
尊敬的 企业领导/公司负责人/业界专家:
  您好! 现将“全世界轨道交通行业规模最大,发展最快,专业观众最多的国际展览盛会”「InnoTrans 
2018第十二届德国柏林国际轨道交通技术展」资讯发送给您。
  请积极参与、共享盛会;  提前安排、赢得先机  —— 
  
InnoTrans 2018 第十二届德国柏林国际轨道交通技术展览会
  —— 参展InnoTrans是一个轨道交通企业走向国际化的标志和途径。
  
  
1、 基 本 信 息 :
  
展会名称: 2018第十二届德国柏林国际轨道交通技术展览会(InnoTrans 2018)
展会时间: 2018年09月18—21日
展会地点: 德国柏林国际展览中心 (Messe Berlin)
展会周期: 两年一届 (2018年为第12届)
报名截止: 2017年12月18日
  
支持单位: 欧洲铁路联合会、德国铁路运输协会 欧洲铁路基础设施管理公司共同体、美国铁路保养工程协会、欧洲铁路轨道工程承包商会、欧洲轨道基础设施管理者协会
主办单位: 德国柏林国际展览有限公司
组团单位: 中国国际贸促会、 映德国际会展集团中国代表处、 映德会展(中国)有限公司
在线客服: QQ/邮箱/ 82775...@qq.com; 微信/ yondexpo; 微博/ http://weibo.com/guidaojiaotong 
 
咨询电话: 4000-680-860(转、8114); 139-1031-8144; 010—8699-7155、6923-6944;
  
  
2、 市 场 背 景 :
  
  
随着世界经济的快速发展,人们对安全、快捷、环保的交通运输需求越来越高。作为一种能够使用各种能源种类的交通运输方式,轨道交通具有占地少、能耗低、污染小、运量大、全天候等多种优势,是节能环保型的运输工具。因此,当今世界各国纷纷对轨道交通建设投入巨资。根据世界顶级铁路行业机构——欧洲铁路行业联盟(CER)的调查统计,目前全球铁路市场总额为1033亿欧元,其中718亿欧元为可获取金额。未来十年铁路行业呈现上升趋势,预计年度增长为1.5-2%,十年后的铁路市场增幅将达到20%左右。
  目前,除了西欧、日本和中国专注于发展高速铁路外,很多国家和地区如:印度、巴西、俄罗斯、中东、非洲、东南亚、甚至美国也开始纷纷加快干线铁路和城市轨道交通建设。
  
经过几十年的努力,我国的铁路和轨道交通建设事业取得了举世瞩目的成就,也培养和锻炼了我国自己的铁路工业体系。现在,我国轨道交通企业技术水平和竞争力不断提高,与西方发达国家相比,我国的很多轨道交通技术产品在技术和成本等方面具有自己的优势和特点,有着很强的竞争力。
 
  
  
3、 本 展 简 介 :
  
  
“德国柏林国际轨道交通技术展览会(InnoTrans)”创办于1996年,每两年一届,2018年度为第十二届,十余年来发展迅速,目前是“全世界轨道交通行业规模最大,发展最快,专业观众最多的国际展览盛会”。展会分为“轨道技术展区”、“基础设施展区”、“隧道建设展区”、“公共交通展区”和“车辆内饰展区”等五大展区。由德国柏林国际展览有限公司(Messe
 Berlin GmbH)主办,得到了欧洲铁路联合会、德国铁路协会等机构的大力支持和协助。中国总展团的组办方为映德国际会展集团(YOND 
EXPO)中国代表处,在北京、上海等地设有分支机构,负责该展会在中国的推广和招商工作,以及中国境内企业参展参观的组织管理事宜,InnoTrans德国柏林轨道展中国总展团报名热线:4000-680-860转、8144;139-1031-8144。
  
InnoTrans是世界上轨道交通产品和技术展示与贸易的最佳平台,也是全世界专业人士沟通交流的理想场所,每一届的Innotrans展览会上都会出现很多来自各个国家的采购商、贸易商和相关政府官员。
  “2018第12届德国柏林国际轨道交通技术展览会(InnoTrans 2018)”将于“9月18日至21日”在“柏林国际展览中心” 
再度盛大举行。为了增进国内外轨道交通行业的交流与合作,同时展示我国轨道交通产业的发展与成就,帮助企业开拓国外市场,映德国际会展集团中国代表处将联合国家有关部门继续组团参加InnoTrans轨道交通展。我们诚邀全国各地相关单位与业界人士加入咱们的中国总展团前往参展参观。参加InnoTrans展会是一个轨道交通企业走向国际化的标志和途径。
  
展会组织者除了举行各种与轨道交通有关的高层研讨会以外,还将举办数十场专业会议和论坛等一系列的活动如:对话论坛、国际轨道交通高峰会以及2014年铁路技术设施建设特别会议。除了以上大型系列活动,展会中还有内容丰富的以轨道交通为中心的小型活动,如:“第五届UITP区域性和郊区轨道交通大会”,“第五届VDWG铁道论坛等。规模最大的组委会每届举办的圆桌会议和欧洲-亚洲轨道高峰会议,今年的第五届欧洲-亚洲轨道高峰会议邀请到了来自欧洲和亚洲多个国家的铁路部门部长等高级官员,以及轨道交通行业各大企业的高层人士。德国隧道联合会(Stuva)还专门举办了国际隧道论坛会议。
  
  
4、 区 位 优 势 :
  
  
柏林位于欧洲中部,作为德国交通运输制造业最集中的重要城市,汇集了大约220家世界知名轨道交通技术公司,承办了世界各地大量大规模的相关施工项目。因此,柏林成为举办交通技术展览的理想场所,为众多此类的买家和卖家提供了交流和交易的平台。
 
  欧洲铁路工业每年销售额为600亿欧元左右,占世界总量的70%。
  
  
5、 历 届 回 顾 :
  
  
2014年柏林国际轨道交通技术展展览面积超过了138600平方米,使用了整个柏林国际展览中心的全部室内外场地。吸引了来自46个国家的2351家参展商,国际展商为1065家;展商展出的展品中,铁路设备及技术展52.9%,基础设施建设23.2%,公共交通工具为14.5%,隧道施工1.9%。本届展会接待超过87个国家的专业观众89600人,德国观众占53.5%,海外观众近51000名。德国、法国、美国、日本、英国、巴西等均组织了庞大的参观团到会参观采购。
  轨道技术展区(Rail Technology)是最大的展区,几乎环绕了整个展览馆的中庭花园;
  第二大展区是交通基础建设展区(Railway Infrastructure)其参展商数量从上届的300家增到了353余家;
  隧道建设展区(Tunnel Construction)是2006年开始增加的特色展区,随着随到建设技术的逐渐成熟也越来越引起关注;
  公共交通展区(Public 
Transport)的扩大则充分反映了世界各国城市轨道交通业对通讯需求的重视,面积达到了18000平米,285家展商中有一半来自德国以外的国家,该展区参展企业的展品集中于轨道交通通讯系统、公共交通管理系统、旅客信息系统等领域;
  车辆内饰展区(Interior)的面积也比上届增长了50%,达到了13000平米。
  在展览会举行期间,还同时举办了各种与轨道交通有关的高层研讨会和专业活动,如:铁路对话论坛、欧亚轨道交通高峰会、铁路技术设施建设特别会议等。
  
中国共有96家企业组团参与了本届展会,展示了我国在轨道交通车辆、零部件与配件、车辆内饰、电子、通信通讯等方面的产品和技术,受到了与会者的高度关注,取得了很好的效果。
  
  
6、 展 品 范 围 :
  
◆ 铁道技术展区:
  
轨道交通及铁路机车车辆设备及零部件和机电设备;车辆段设备;供电系统;通信、信号系统;自动售检票系统;升降系统;火灾报警系统;通风、空调与采暖系统;环境与设备监控系统;综合监控系统;票务清分系统;屏蔽门安全系统;轨道交通及铁路建设施工材料、装备、安全、节能、环保技术与维护等;
◆ 基础设施展区:
  铁道设施建设,车站及站场设备,轨道线路铺设、养护、维修装备与技术;铁道土木工程、桥梁道路、给排水、环境工程、风景园林、公用工程等建设;
◆ 隧道建设展区:
  
隧道施工建设机械设备、配件、材料与技术,工程机械,安全装置及设备等;地下交通检测、勘探技术、建设材料与技术设备等;地下线路、信号、桥梁、隧道、供电网、站房等施工机械及配套设备等;
◆ 公共交通展区:
  
智能交通管理系统、道路收费系统设备、智能交通产品及安防监控设备;视频监控设备、交通信息采集设备、交通控制设备、道路收费系统设备;地铁、汽车、巴士等轨道交通运输工具;
◆ 内部装饰展区:
  铁轨车内部设计、更新、装饰服务,门、窗帘,隔断,桌椅,行李架,扶手杆,替换零件,锁柜,安全带,涂料,手推车,洗手间,厨房设备等。
  
  
7、 参 展 形 式 :
  
◆ 参 展: 租赁【展位】,展示企业形象及产品,主要费用有【展位费】【运输费】【搭建费】【人员费】等。
◆ 参 观: 派遣【人员】,莅临展场参观及考察,主要费用有【报名费】【注册费】【交通费】【食宿费】等。
  
  
8、 筹 展 联 络 :
  
有关参展参观“InnoTrans德国柏林轨道展”事宜,请联络【中国总展团】组办方:
参展顾问: 王先生、 杨先生、 孙小姐、 吴小姐
参展热线: 139-1031-8144、 131-2662-5206;  010— 8699- 7155、 6923- 6944.
全国统一客服热线: 4000- 580- 850.(分机: / 分机:5220)
全国统一报名专线: 4000- 680- 860.(兼传真)
在线客服: QQ/ 82775507;  微信/ yondexpo;  邮箱/ 12809395#qq.com;  微博/ 
http://weibo.com/yingdehuizhan 
  
  〔 映德会展 / 诚邀参加2018年度权威实效品牌盛会 ﹞
  
  
【 公众平台 】
  
微信: 参展消息 (ID:CanZhanXiaoXi)—— 品牌扩张的平台 市场开拓的桥梁
微信: 展商之家 (ID:ZhanShangZhiJia)—— 为展商提供最佳营地 为阁下营造参展价值
  
  
---
百万群发系统|为您发送|如不希望再收到此行业资讯|请回复“退订+德国轨道展”至邮箱1055800...@qq.com