subject:"\[Users\] cant remove disks from iscsi domain"

Re: [Users] cant remove disks from iscsi domain

2013-08-22 Thread Doron Fediuck



- Original Message -
| From: Yuriy Demchenko demchenko...@gmail.com
| To: Greg Padgett gpadg...@redhat.com
| Cc: users@ovirt.org
| Sent: Thursday, August 22, 2013 9:55:19 AM
| Subject: Re: [Users] cant remove disks from iscsi domain
| 
| I've done some more tests - and it seems quota error is not related to
| my issue: I tried to remove another disk and this time there were no
| quota errors in engine.log
| New logs in attach.
| 
| What catches my eye in logs is this errors, but maybe that's not the
| root of case:
|  Thread-60725::DEBUG::2013-08-22
|  10:37:45,549::lvm::485::OperationMutex::(_invali  datevgs) Operation
|  'lvm invalidate operation' released the operation mutex
|  Thread-60725::WARNING::2013-08-22
|  10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks)
|  Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da
|  
6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40
|  85-8900-da271affe831. [Errno 2] No such file or directory:
|  '/rhev/data-center/mn
|  
t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da
|  7937fbdfad/dfefc573-de85-4085-8900-da271affe831'
|  Thread-60725::WARNING::2013-08-22
|  10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks)
|  Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da
|  
6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43
|  5d-bdc7-713b445a2326. [Errno 2] No such file or directory:
|  '/rhev/data-center/mn
|  
t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da
|  7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326'
|  Thread-60725::DEBUG::2013-08-22
|  10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks)
|  removed: []
|  Thread-60725::ERROR::2013-08-22
|  10:37:45,549::task::833::TaskManager.Task::(_set  Error)
|  Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error
|  Traceback (most recent call last):
|File /usr/share/vdsm/storage/task.py, line 840, in _run
|  return fn(*args, **kargs)
|File /usr/share/vdsm/logUtils.py, line 42, in wrapper
|  res = f(*args, **kwargs)
|File /usr/share/vdsm/storage/hsm.py, line 1460, in deleteImage
|  dom.deleteImage(sdUUID, imgUUID, volsByImg)
|File /usr/share/vdsm/storage/blockSD.py, line 957, in deleteImage
|  self.rmDCImgDir(imgUUID, volsImgs)
|File /usr/share/vdsm/storage/blockSD.py, line 943, in rmDCImgDir
|  self.log.warning(Can't rmdir %s. %s, imgPath, exc_info=True)
|File /usr/lib64/python2.6/logging/__init__.py, line 1068, in warning
|  self._log(WARNING, msg, args, **kwargs)
|File /usr/lib64/python2.6/logging/__init__.py, line 1173, in _log
|  self.handle(record)
|File /usr/lib64/python2.6/logging/__init__.py, line 1183, in handle
|  self.callHandlers(record)
|File /usr/lib64/python2.6/logging/__init__.py, line 1220, in
|  callHandlers
|  hdlr.handle(record)
|File /usr/lib64/python2.6/logging/__init__.py, line 679, in handle
|  self.emit(record)
|File /usr/lib64/python2.6/logging/handlers.py, line 780, in emit
|  msg = self.format(record)
|File /usr/lib64/python2.6/logging/__init__.py, line 654, in format
|  return fmt.format(record)
|File /usr/lib64/python2.6/logging/__init__.py, line 436, in format
|  record.message = record.getMessage()
|File /usr/lib64/python2.6/logging/__init__.py, line 306, in getMessage
|  msg = msg % self.args
|  TypeError: not enough arguments for format string
| 
| 
| Yuriy Demchenko

Yuri,
just to clarify the quota part, the stack trace you provided
was resolved as the bz Greg described- 
https://bugzilla.redhat.com/show_bug.cgi?id=905891

This is unrelated to the storage issue, as the quota caching
is an independent procedure running in parallel.

So what we now need to focus on is this part:
[Errno 2] No such file or directory:
'/rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-4085-8900-da271affe831'

What we need to understand is how did we get into
this state; Did you have network issues or relevant crashes?
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [Users] cant remove disks from iscsi domain

2013-08-22 Thread Dafna Ron

you said that the disks are deleted although an exception is given but 
the engine is reporting the delete as failed.
I think that I remember a bug reported and fixed on that as well but I 
can't seem to find it.

Adding Ayal and Eduardo



On 08/22/2013 07:55 AM, Yuriy Demchenko wrote:
I've done some more tests - and it seems quota error is not related to 
my issue: I tried to remove another disk and this time there were no 
quota errors in engine.log

New logs in attach.

What catches my eye in logs is this errors, but maybe that's not the 
root of case:
Thread-60725::DEBUG::2013-08-22 
10:37:45,549::lvm::485::OperationMutex::(_invali  datevgs) Operation 
'lvm invalidate operation' released the operation mutex
Thread-60725::WARNING::2013-08-22 
10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) 
Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 
6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 
85-8900-da271affe831. [Errno 2] No such file or directory: 
'/rhev/data-center/mn 
t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 
7937fbdfad/dfefc573-de85-4085-8900-da271affe831'
Thread-60725::WARNING::2013-08-22 
10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks) 
Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da 
6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 
5d-bdc7-713b445a2326. [Errno 2] No such file or directory: 
'/rhev/data-center/mn 
t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 
7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326'
Thread-60725::DEBUG::2013-08-22 
10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks) 
removed: []
Thread-60725::ERROR::2013-08-22 
10:37:45,549::task::833::TaskManager.Task::(_set  Error) 
Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error

Traceback (most recent call last):
  File /usr/share/vdsm/storage/task.py, line 840, in _run
return fn(*args, **kargs)
  File /usr/share/vdsm/logUtils.py, line 42, in wrapper
res = f(*args, **kwargs)
  File /usr/share/vdsm/storage/hsm.py, line 1460, in deleteImage
dom.deleteImage(sdUUID, imgUUID, volsByImg)
  File /usr/share/vdsm/storage/blockSD.py, line 957, in deleteImage
self.rmDCImgDir(imgUUID, volsImgs)
  File /usr/share/vdsm/storage/blockSD.py, line 943, in rmDCImgDir
self.log.warning(Can't rmdir %s. %s, imgPath, exc_info=True)
  File /usr/lib64/python2.6/logging/__init__.py, line 1068, in warning
self._log(WARNING, msg, args, **kwargs)
  File /usr/lib64/python2.6/logging/__init__.py, line 1173, in _log
self.handle(record)
  File /usr/lib64/python2.6/logging/__init__.py, line 1183, in handle
self.callHandlers(record)
  File /usr/lib64/python2.6/logging/__init__.py, line 1220, in 
callHandlers

hdlr.handle(record)
  File /usr/lib64/python2.6/logging/__init__.py, line 679, in handle
self.emit(record)
  File /usr/lib64/python2.6/logging/handlers.py, line 780, in emit
msg = self.format(record)
  File /usr/lib64/python2.6/logging/__init__.py, line 654, in format
return fmt.format(record)
  File /usr/lib64/python2.6/logging/__init__.py, line 436, in format
record.message = record.getMessage()
  File /usr/lib64/python2.6/logging/__init__.py, line 306, in 
getMessage

msg = msg % self.args
TypeError: not enough arguments for format string



Yuriy Demchenko

On 08/22/2013 04:11 AM, Greg Padgett wrote:

On 08/21/2013 04:10 PM, Dafna Ron wrote:

there is a is an exception in the log related to a quota calculation

2013-08-21 17:52:32,694 ERROR
[org.ovirt.engine.core.utils.timer.SchedulerUtilQu   artzImpl]
(DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd
ateQuotaCache: java.lang.reflect.InvocationTargetException
 at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown 
Source)

[:1.7.0   _25]
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43) [rt.jar:1.7.0_25]
 at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_25]
 at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:
60) [scheduler.jar:]
 at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
[quartz.jar:]
 at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j
ava:557) [quartz.jar:]
Caused by: org.springframework.jdbc.BadSqlGrammarException:
PreparedStatementCal   lback; bad SQL grammar [select * from
calculateallstorageusage()]; nested excep   tion is
org.postgresql.util.PSQLException: ERROR: column 
quota_limitation.quota
_id must appear in the GROUP BY clause or be used in an aggregate 
function
   Where: PL/pgSQL function calculateallstorageusage line 3 at 
RETURN QUERY



in any case this is a bug.
I'm adding Doron to this mail, perhaps this was reported in the past 
and

already solved in later versions.
if not it should be reported and fixed.

Dafna




If I'm

Re: [Users] cant remove disks from iscsi domain

2013-08-22 Thread Yuriy Demchenko


Thanks a lot! Patch did the job, a few more disks were deleted successfully.

One last question - how do i remove stale records of disks in illegal 
state?


Yuriy Demchenko

On 08/22/2013 01:25 PM, Eduardo Warszawski wrote:


- Original Message -

you said that the disks are deleted although an exception is given but
the engine is reporting the delete as failed.
I think that I remember a bug reported and fixed on that as well but I
can't seem to find it.
Adding Ayal and Eduardo


The log issue already solved in v4.11.0~380.

commit ad916c79e2b0959dea20dd19a21b99bc702d65ca
Author: Eduardo Warszawski ewars...@redhat.com
Date:   Mon Dec 17 14:32:51 2012 +0200

 Fix typo in negative flow log in blockSD.rmDCImgDir().
 
 Related to BZ#885489.
 
 Change-Id: I951e582acc86e08d709da4249084015660fc4ea0

 Signed-off-by: Eduardo ewars...@redhat.com
 Reviewed-on: http://gerrit.ovirt.org/10153
 Reviewed-by: Yeela Kaplan ykap...@redhat.com
 Reviewed-by: Ayal Baron aba...@redhat.com
 Tested-by: Dan Kenigsberg dan...@redhat.com

diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py
index b5447cd..22a434b 100644
--- a/vdsm/storage/blockSD.py
+++ b/vdsm/storage/blockSD.py
@@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain):
  try:
  os.rmdir(imgPath)
  except OSError:
-self.log.warning(Can't rmdir %s. %s, imgPath, exc_info=True)
+self.log.warning(Can't rmdir %s, imgPath, exc_info=True)
  else:
  self.log.debug(removed image dir: %s, imgPath)
  return imgPath



On 08/22/2013 07:55 AM, Yuriy Demchenko wrote:

I've done some more tests - and it seems quota error is not related to
my issue: I tried to remove another disk and this time there were no
quota errors in engine.log
New logs in attach.

What catches my eye in logs is this errors, but maybe that's not the
root of case:

Thread-60725::DEBUG::2013-08-22
10:37:45,549::lvm::485::OperationMutex::(_invali  datevgs) Operation
'lvm invalidate operation' released the operation mutex
Thread-60725::WARNING::2013-08-22
10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks)
Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da
6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40
85-8900-da271affe831. [Errno 2] No such file or directory:
'/rhev/data-center/mn
t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da
7937fbdfad/dfefc573-de85-4085-8900-da271affe831'
Thread-60725::WARNING::2013-08-22
10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks)
Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da
6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43
5d-bdc7-713b445a2326. [Errno 2] No such file or directory:
'/rhev/data-center/mn
t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da
7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326'
Thread-60725::DEBUG::2013-08-22
10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks)
removed: []
Thread-60725::ERROR::2013-08-22
10:37:45,549::task::833::TaskManager.Task::(_set  Error)
Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error
Traceback (most recent call last):
   File /usr/share/vdsm/storage/task.py, line 840, in _run
 return fn(*args, **kargs)
   File /usr/share/vdsm/logUtils.py, line 42, in wrapper
 res = f(*args, **kwargs)
   File /usr/share/vdsm/storage/hsm.py, line 1460, in deleteImage
 dom.deleteImage(sdUUID, imgUUID, volsByImg)
   File /usr/share/vdsm/storage/blockSD.py, line 957, in deleteImage
 self.rmDCImgDir(imgUUID, volsImgs)
   File /usr/share/vdsm/storage/blockSD.py, line 943, in rmDCImgDir
 self.log.warning(Can't rmdir %s. %s, imgPath, exc_info=True)
   File /usr/lib64/python2.6/logging/__init__.py, line 1068, in warning
 self._log(WARNING, msg, args, **kwargs)
   File /usr/lib64/python2.6/logging/__init__.py, line 1173, in _log
 self.handle(record)
   File /usr/lib64/python2.6/logging/__init__.py, line 1183, in handle
 self.callHandlers(record)
   File /usr/lib64/python2.6/logging/__init__.py, line 1220, in
callHandlers
 hdlr.handle(record)
   File /usr/lib64/python2.6/logging/__init__.py, line 679, in handle
 self.emit(record)
   File /usr/lib64/python2.6/logging/handlers.py, line 780, in emit
 msg = self.format(record)
   File /usr/lib64/python2.6/logging/__init__.py, line 654, in format
 return fmt.format(record)
   File /usr/lib64/python2.6/logging/__init__.py, line 436, in format
 record.message = record.getMessage()
   File /usr/lib64/python2.6/logging/__init__.py, line 306, in
getMessage
 msg = msg % self.args
TypeError: not enough arguments for format string


Yuriy Demchenko

On 08/22/2013 04:11 AM, Greg Padgett wrote:

On 08/21/2013 04:10 PM, Dafna Ron wrote:

there is a is an exception in the log related to a quota calculation

2013-08-21

Re: [Users] cant remove disks from iscsi domain

2013-08-22 Thread Dafna Ron


deleting illegal objects from UI was added to the latest ovirt/vdsm
are you getting any errors when trying to delete them now?



On 08/22/2013 10:46 AM, Yuriy Demchenko wrote:
Thanks a lot! Patch did the job, a few more disks were deleted 
successfully.


One last question - how do i remove stale records of disks in 
illegal state?


Yuriy Demchenko

On 08/22/2013 01:25 PM, Eduardo Warszawski wrote:


- Original Message -

you said that the disks are deleted although an exception is given but
the engine is reporting the delete as failed.
I think that I remember a bug reported and fixed on that as well but I
can't seem to find it.
Adding Ayal and Eduardo


The log issue already solved in v4.11.0~380.

commit ad916c79e2b0959dea20dd19a21b99bc702d65ca
Author: Eduardo Warszawski ewars...@redhat.com
Date:   Mon Dec 17 14:32:51 2012 +0200

 Fix typo in negative flow log in blockSD.rmDCImgDir().
  Related to BZ#885489.
  Change-Id: I951e582acc86e08d709da4249084015660fc4ea0
 Signed-off-by: Eduardo ewars...@redhat.com
 Reviewed-on: http://gerrit.ovirt.org/10153
 Reviewed-by: Yeela Kaplan ykap...@redhat.com
 Reviewed-by: Ayal Baron aba...@redhat.com
 Tested-by: Dan Kenigsberg dan...@redhat.com

diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py
index b5447cd..22a434b 100644
--- a/vdsm/storage/blockSD.py
+++ b/vdsm/storage/blockSD.py
@@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain):
  try:
  os.rmdir(imgPath)
  except OSError:
-self.log.warning(Can't rmdir %s. %s, imgPath, 
exc_info=True)

+self.log.warning(Can't rmdir %s, imgPath, exc_info=True)
  else:
  self.log.debug(removed image dir: %s, imgPath)
  return imgPath



On 08/22/2013 07:55 AM, Yuriy Demchenko wrote:

I've done some more tests - and it seems quota error is not related to
my issue: I tried to remove another disk and this time there were no
quota errors in engine.log
New logs in attach.

What catches my eye in logs is this errors, but maybe that's not the
root of case:

Thread-60725::DEBUG::2013-08-22
10:37:45,549::lvm::485::OperationMutex::(_invali  datevgs) Operation
'lvm invalidate operation' released the operation mutex
Thread-60725::WARNING::2013-08-22
10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks)
Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da
6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 


85-8900-da271affe831. [Errno 2] No such file or directory:
'/rhev/data-center/mn
t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 


7937fbdfad/dfefc573-de85-4085-8900-da271affe831'
Thread-60725::WARNING::2013-08-22
10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks)
Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da
6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 


5d-bdc7-713b445a2326. [Errno 2] No such file or directory:
'/rhev/data-center/mn
t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 


7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326'
Thread-60725::DEBUG::2013-08-22
10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks)
removed: []
Thread-60725::ERROR::2013-08-22
10:37:45,549::task::833::TaskManager.Task::(_set  Error)
Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error
Traceback (most recent call last):
   File /usr/share/vdsm/storage/task.py, line 840, in _run
 return fn(*args, **kargs)
   File /usr/share/vdsm/logUtils.py, line 42, in wrapper
 res = f(*args, **kwargs)
   File /usr/share/vdsm/storage/hsm.py, line 1460, in deleteImage
 dom.deleteImage(sdUUID, imgUUID, volsByImg)
   File /usr/share/vdsm/storage/blockSD.py, line 957, in 
deleteImage

 self.rmDCImgDir(imgUUID, volsImgs)
   File /usr/share/vdsm/storage/blockSD.py, line 943, in rmDCImgDir
 self.log.warning(Can't rmdir %s. %s, imgPath, exc_info=True)
   File /usr/lib64/python2.6/logging/__init__.py, line 1068, in 
warning

 self._log(WARNING, msg, args, **kwargs)
   File /usr/lib64/python2.6/logging/__init__.py, line 1173, in 
_log

 self.handle(record)
   File /usr/lib64/python2.6/logging/__init__.py, line 1183, in 
handle

 self.callHandlers(record)
   File /usr/lib64/python2.6/logging/__init__.py, line 1220, in
callHandlers
 hdlr.handle(record)
   File /usr/lib64/python2.6/logging/__init__.py, line 679, in 
handle

 self.emit(record)
   File /usr/lib64/python2.6/logging/handlers.py, line 780, in emit
 msg = self.format(record)
   File /usr/lib64/python2.6/logging/__init__.py, line 654, in 
format

 return fmt.format(record)
   File /usr/lib64/python2.6/logging/__init__.py, line 436, in 
format

 record.message = record.getMessage()
   File /usr/lib64/python2.6/logging/__init__.py, line 306, in
getMessage
 msg = msg % self.args
TypeError: not enough arguments for

Re: [Users] cant remove disks from iscsi domain

2013-08-22 Thread Dafna Ron

there are actually a lot of tables that have a reference to the image in 
the db.
I am adding Liron who might be able to give you the patch needed for 
deleting these images or tell you in what rpm they were merged to.



On 08/22/2013 11:06 AM, Yuriy Demchenko wrote:
If by latest you mean head/nightly - i'm not using it yet, I'm using 
packets from stable el6 repo, so my test env match production env.


Errors i get:
User admin@internal failed to initiate removing of disk 
virt-dhcp-3_Disk1 from domain iscsi-store


Thread-1196::ERROR::2013-08-22 
14:00:57,736::task::833::TaskManager.Task::(_setError) 
Task=`b691e001-7d8e-46cc-a04e-3d89ffea8fe5`::Unexpected error

Traceback (most recent call last):
  File /usr/share/vdsm/storage/task.py, line 840, in _run
return fn(*args, **kargs)
  File /usr/share/vdsm/logUtils.py, line 42, in wrapper
res = f(*args, **kwargs)
  File /usr/share/vdsm/storage/hsm.py, line 1434, in deleteImage
raise se.ImageDoesNotExistInSD(imgUUID, sdUUID)
ImageDoesNotExistInSD: Image does not exist in domain: 
'image=dde47a09-81d9-4592-87e4-92398ea834ce, 
domain=d786e2d5-05ab-4da6-95fc-1af791a3c113'


Maybe just do some sql-query to remove them from DB?

Yuriy Demchenko

On 08/22/2013 01:55 PM, Dafna Ron wrote:

deleting illegal objects from UI was added to the latest ovirt/vdsm
are you getting any errors when trying to delete them now?



On 08/22/2013 10:46 AM, Yuriy Demchenko wrote:
Thanks a lot! Patch did the job, a few more disks were deleted 
successfully.


One last question - how do i remove stale records of disks in 
illegal state?


Yuriy Demchenko

On 08/22/2013 01:25 PM, Eduardo Warszawski wrote:


- Original Message -
you said that the disks are deleted although an exception is given 
but

the engine is reporting the delete as failed.
I think that I remember a bug reported and fixed on that as well 
but I

can't seem to find it.
Adding Ayal and Eduardo


The log issue already solved in v4.11.0~380.

commit ad916c79e2b0959dea20dd19a21b99bc702d65ca
Author: Eduardo Warszawski ewars...@redhat.com
Date:   Mon Dec 17 14:32:51 2012 +0200

 Fix typo in negative flow log in blockSD.rmDCImgDir().
  Related to BZ#885489.
  Change-Id: I951e582acc86e08d709da4249084015660fc4ea0
 Signed-off-by: Eduardo ewars...@redhat.com
 Reviewed-on: http://gerrit.ovirt.org/10153
 Reviewed-by: Yeela Kaplan ykap...@redhat.com
 Reviewed-by: Ayal Baron aba...@redhat.com
 Tested-by: Dan Kenigsberg dan...@redhat.com

diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py
index b5447cd..22a434b 100644
--- a/vdsm/storage/blockSD.py
+++ b/vdsm/storage/blockSD.py
@@ -978,7 +978,7 @@ class BlockStorageDomain(sd.StorageDomain):
  try:
  os.rmdir(imgPath)
  except OSError:
-self.log.warning(Can't rmdir %s. %s, imgPath, 
exc_info=True)
+self.log.warning(Can't rmdir %s, imgPath, 
exc_info=True)

  else:
  self.log.debug(removed image dir: %s, imgPath)
  return imgPath



On 08/22/2013 07:55 AM, Yuriy Demchenko wrote:
I've done some more tests - and it seems quota error is not 
related to

my issue: I tried to remove another disk and this time there were no
quota errors in engine.log
New logs in attach.

What catches my eye in logs is this errors, but maybe that's not the
root of case:

Thread-60725::DEBUG::2013-08-22
10:37:45,549::lvm::485::OperationMutex::(_invali datevgs) Operation
'lvm invalidate operation' released the operation mutex
Thread-60725::WARNING::2013-08-22
10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks)
Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da
6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/dfefc573-de85-40 


85-8900-da271affe831. [Errno 2] No such file or directory:
'/rhev/data-center/mn
t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 


7937fbdfad/dfefc573-de85-4085-8900-da271affe831'
Thread-60725::WARNING::2013-08-22
10:37:45,549::blockSD::931::Storage.StorageDom ain::(rmDCVolLinks)
Can't unlink /rhev/data-center/mnt/blockSD/d786e2d5-05ab-4da
6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da7937fbdfad/c6cd6d1d-b70f-43 


5d-bdc7-713b445a2326. [Errno 2] No such file or directory:
'/rhev/data-center/mn
t/blockSD/d786e2d5-05ab-4da6-95fc-1af791a3c113/images/5344ca63-302a-43de-9193-da 


7937fbdfad/c6cd6d1d-b70f-435d-bdc7-713b445a2326'
Thread-60725::DEBUG::2013-08-22
10:37:45,549::blockSD::934::Storage.StorageDomai n::(rmDCVolLinks)
removed: []
Thread-60725::ERROR::2013-08-22
10:37:45,549::task::833::TaskManager.Task::(_set Error)
Task=`83867bdc-48cd-4ba0-b453-6f8abbace13e`::Unexpected error
Traceback (most recent call last):
   File /usr/share/vdsm/storage/task.py, line 840, in _run
 return fn(*args, **kargs)
   File /usr/share/vdsm/logUtils.py, line 42, in wrapper
 res = f(*args, **kwargs)
   File /usr/share/vdsm/storage/hsm.py, line 1460, in deleteImage

[Users] cant remove disks from iscsi domain

2013-08-21 Thread Yuriy Demchenko


Hi,

I've recently encountered a problem with removing disks from iscsi 
domain in my test lab - just cant remove any.
Remove operation fails with message User admin@internal failed to 
initiate removing of disk pg-slave1_opt from domain iscsi-store and 
re-elections of SPM. After that - disk is marked as illegal in ovirt 
webinterface, however, it is _in fact_ removed from storage - lvdisplay 
doesn't show it and free space is updated correctly.

And this happens with just about every disk/vm i try to remove.

ovirt 3.2.2-el6
centos 6.4
vdsm-4.10.3-17.el6
lvm2-2.02.98-9.el6

Any tips? logs in attach

--
Yuriy Demchenko

2013-08-21 17:52:21,156 INFO  [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp   --127.0.0.1-8702-8) Lock Acquired to object EngineLock [exclusiveLocks= key: 594   71028-8157-484b-9b2c-98c0eb431ba7 value: DISK
, sharedLocks= key: 2be220e7-6e46-4d12-9ab5-e58416225455 value: VM
]
2013-08-21 17:52:21,273 INFO  [org.ovirt.engine.core.bll.RemoveDiskCommand] (poo   l-3-thread-45) [4b3c8227] Running command: RemoveDiskCommand internal: false. En   tities affected :  ID: 59471028-8157-484b-9b2c-98c0eb431ba7 Type: Disk
2013-08-21 17:52:21,278 INFO  [org.ovirt.engine.core.bll.RemoveImageCommand] (po   ol-3-thread-45) [fe46de8] Running command: RemoveImageCommand internal: true. En   tities affected :  ID: d786e2d5-05ab-4da6-95fc-1af791a3c113 Type: Storage
2013-08-21 17:52:21,299 INFO  [org.ovirt.engine.core.bll.RemoveImageCommand] (po   ol-3-thread-45) [fe46de8] Lock freed to object EngineLock [exclusiveLocks= key:59471028-8157-484b-9b2c-98c0eb431ba7 value: DISK
, sharedLocks= key: 2be220e7-6e46-4d12-9ab5-e58416225455 value: VM
]
2013-08-21 17:52:21,306 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteI   mageGroupVDSCommand] (pool-3-thread-45) [fe46de8] START, DeleteImageGroupVDSComm   and( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit =false, compatabilityVersion = 3.2, storageDomainId = d786e2d5-05ab-4da6-95fc-1a   f791a3c113, imageGroupId = 59471028-8157-484b-9b2c-98c0eb431ba7, postZeros = fal   se, forceDelete = false), log id: 3e928634
2013-08-21 17:52:22,170 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerC   ommandBase] (pool-3-thread-45) [fe46de8] Failed in DeleteImageGroupVDS method
2013-08-21 17:52:22,171 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerC   ommandBase] (pool-3-thread-45) [fe46de8] Error code GeneralException and error m   essage IRSGenericException: IRSErrorException: Failed to DeleteImageGroupVDS, er   ror = not enough arguments for format string
2013-08-21 17:52:22,172 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrok   erCommand] (pool-3-thread-45) [fe46de8] IrsBroker::Failed::DeleteImageGroupVDS d   ue to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to Dele   teImageGroupVDS, error = not enough arguments for format string
2013-08-21 17:52:22,343 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStop   VDSCommand] (pool-3-thread-45) [fe46de8] START, SpmStopVDSCommand(HostName = nod   e-2, HostId = e499def4-61ad-4801-8fe3-36f6f91c70a8, storagePoolId = 5849b030-626   e-47cb-ad90-3ce782d831b3), log id: a2093f6
2013-08-21 17:52:22,378 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStop   VDSCommand] (pool-3-thread-45) [fe46de8] SpmStopVDSCommand::Stopping SPM on vdsnode-2, pool id 5849b030-626e-47cb-ad90-3ce782d831b3
2013-08-21 17:52:23,009 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStop   VDSCommand] (pool-3-thread-45) [fe46de8] FINISH, SpmStopVDSCommand, log id: a209   3f6
2013-08-21 17:52:23,012 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrok   erCommand] (pool-3-thread-45) [fe46de8] Irs placed on server e499def4-61ad-4801-   8fe3-36f6f91c70a8 failed. Proceed Failover
2013-08-21 17:52:23,065 INFO  [org.ovirt.engine.core.bll.storage.SetStoragePoolS   tatusCommand] (pool-3-thread-45) [1377b2ae] Running command: SetStoragePoolStatu   sCommand internal: true. Entities affected :  ID: 5849b030-626e-47cb-ad90-3ce782   d831b3 Type: StoragePool
2013-08-21 17:52:23,222 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrok   erCommand] (pool-3-thread-45) [1377b2ae] hostFromVds::selectedVds - node-1, spmS   tatus Free, storage pool Default
2013-08-21 17:52:23,225 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrok   erCommand] (pool-3-thread-45) [1377b2ae] SPM Init: could not find reported vds o   r not up - pool:Default vds_spm_id: 2
2013-08-21 17:52:23,256 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrok   erCommand] (pool-3-thread-45) [1377b2ae] SPM selection - vds seems as spm node-2
2013-08-21 17:52:23,287 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStop   VDSCommand] (pool-3-thread-45) [1377b2ae] START, SpmStopVDSCommand(HostName = no   de-2, HostId = e499def4-61ad-4801-8fe3-36f6f91c70a8, storagePoolId = 5849b030-62   6e-47cb-ad90-3ce782d831b3), log id: 6bfcf094
2013-08-21 17:52:23,316 INFO

Re: [Users] cant remove disks from iscsi domain

2013-08-21 Thread Dafna Ron


from the logs it appears to be a quota issue.
do you have quota enabled?


On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:

Hi,

I've recently encountered a problem with removing disks from iscsi 
domain in my test lab - just cant remove any.
Remove operation fails with message User admin@internal failed to 
initiate removing of disk pg-slave1_opt from domain iscsi-store and 
re-elections of SPM. After that - disk is marked as illegal in ovirt 
webinterface, however, it is _in fact_ removed from storage - 
lvdisplay doesn't show it and free space is updated correctly.

And this happens with just about every disk/vm i try to remove.

ovirt 3.2.2-el6
centos 6.4
vdsm-4.10.3-17.el6
lvm2-2.02.98-9.el6

Any tips? logs in attach



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users



--
Dafna Ron
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [Users] cant remove disks from iscsi domain

2013-08-21 Thread Yura Demchenko


21.08.2013 19:18, Dafna Ron пишет:

from the logs it appears to be a quota issue.
do you have quota enabled?

Yes, quota in enforced mode. But VMs/disks in question belongs to 
unlimited quota (defined quota policy with no limits on storage/cpu/ram)




On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:

Hi,

I've recently encountered a problem with removing disks from iscsi 
domain in my test lab - just cant remove any.
Remove operation fails with message User admin@internal failed to 
initiate removing of disk pg-slave1_opt from domain iscsi-store and 
re-elections of SPM. After that - disk is marked as illegal in 
ovirt webinterface, however, it is _in fact_ removed from storage - 
lvdisplay doesn't show it and free space is updated correctly.

And this happens with just about every disk/vm i try to remove.

ovirt 3.2.2-el6
centos 6.4
vdsm-4.10.3-17.el6
lvm2-2.02.98-9.el6

Any tips? logs in attach



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users





___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [Users] cant remove disks from iscsi domain

2013-08-21 Thread Dafna Ron


there is a is an exception in the log related to a quota calculation

2013-08-21 17:52:32,694 ERROR 
[org.ovirt.engine.core.utils.timer.SchedulerUtilQu   artzImpl] 
(DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd   
ateQuotaCache: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source) 
[:1.7.0   _25]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces   
sorImpl.java:43) [rt.jar:1.7.0_25]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
at 
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:   60) 
[scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
at 
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j   
ava:557) [quartz.jar:]
Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCal   lback; 
bad SQL grammar [select * from  calculateallstorageusage()]; nested excep   tion is 
org.postgresql.util.PSQLException: ERROR: column quota_limitation.quota   _id 
must appear in the GROUP BY clause or be used in an aggregate function
  Where: PL/pgSQL function calculateallstorageusage line 3 at RETURN QUERY


in any case this is a bug.
I'm adding Doron to this mail, perhaps this was reported in the past and 
already solved in later versions.

if not it should be reported and fixed.

Dafna





On 08/21/2013 05:26 PM, Yura Demchenko wrote:

21.08.2013 19:18, Dafna Ron пишет:

from the logs it appears to be a quota issue.
do you have quota enabled?

Yes, quota in enforced mode. But VMs/disks in question belongs to 
unlimited quota (defined quota policy with no limits on 
storage/cpu/ram)




On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:

Hi,

I've recently encountered a problem with removing disks from iscsi 
domain in my test lab - just cant remove any.
Remove operation fails with message User admin@internal failed to 
initiate removing of disk pg-slave1_opt from domain iscsi-store and 
re-elections of SPM. After that - disk is marked as illegal in 
ovirt webinterface, however, it is _in fact_ removed from storage - 
lvdisplay doesn't show it and free space is updated correctly.

And this happens with just about every disk/vm i try to remove.

ovirt 3.2.2-el6
centos 6.4
vdsm-4.10.3-17.el6
lvm2-2.02.98-9.el6

Any tips? logs in attach



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users








--
Dafna Ron
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [Users] cant remove disks from iscsi domain

2013-08-21 Thread Greg Padgett


On 08/21/2013 04:10 PM, Dafna Ron wrote:

there is a is an exception in the log related to a quota calculation

2013-08-21 17:52:32,694 ERROR
[org.ovirt.engine.core.utils.timer.SchedulerUtilQu   artzImpl]
(DefaultQuartzScheduler_Worker-7) failed to invoke sceduled method upd
ateQuotaCache: java.lang.reflect.InvocationTargetException
 at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source)
[:1.7.0   _25]
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43) [rt.jar:1.7.0_25]
 at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_25]
 at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:
60) [scheduler.jar:]
 at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
[quartz.jar:]
 at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.j
ava:557) [quartz.jar:]
Caused by: org.springframework.jdbc.BadSqlGrammarException:
PreparedStatementCal   lback; bad SQL grammar [select * from
calculateallstorageusage()]; nested excep   tion is
org.postgresql.util.PSQLException: ERROR: column quota_limitation.quota
_id must appear in the GROUP BY clause or be used in an aggregate function
   Where: PL/pgSQL function calculateallstorageusage line 3 at RETURN QUERY


in any case this is a bug.
I'm adding Doron to this mail, perhaps this was reported in the past and
already solved in later versions.
if not it should be reported and fixed.

Dafna




If I'm not mistaken, it looks like this bug:

https://bugzilla.redhat.com/show_bug.cgi?id=905891

Greg





On 08/21/2013 05:26 PM, Yura Demchenko wrote:

21.08.2013 19:18, Dafna Ron пишет:

from the logs it appears to be a quota issue.
do you have quota enabled?


Yes, quota in enforced mode. But VMs/disks in question belongs to
unlimited quota (defined quota policy with no limits on storage/cpu/ram)



On 08/21/2013 03:20 PM, Yuriy Demchenko wrote:

Hi,

I've recently encountered a problem with removing disks from iscsi
domain in my test lab - just cant remove any.
Remove operation fails with message User admin@internal failed to
initiate removing of disk pg-slave1_opt from domain iscsi-store and
re-elections of SPM. After that - disk is marked as illegal in ovirt
webinterface, however, it is _in fact_ removed from storage -
lvdisplay doesn't show it and free space is updated correctly.
And this happens with just about every disk/vm i try to remove.

ovirt 3.2.2-el6
centos 6.4
vdsm-4.10.3-17.el6
lvm2-2.02.98-9.el6

Any tips? logs in attach



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users










___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [Users] cant remove disks from iscsi domain

Re: [Users] cant remove disks from iscsi domain

Re: [Users] cant remove disks from iscsi domain

Re: [Users] cant remove disks from iscsi domain

Re: [Users] cant remove disks from iscsi domain

[Users] cant remove disks from iscsi domain

Re: [Users] cant remove disks from iscsi domain

Re: [Users] cant remove disks from iscsi domain

Re: [Users] cant remove disks from iscsi domain

Re: [Users] cant remove disks from iscsi domain

10 matches

Site Navigation

Mail list logo

Footer information