[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #43, sr#110963 (group freeipmi):

I got a potential candidate for merger in this branch

freeipmi-1-6-0-stable-lenovo_workaround

the workaround I've created is now called "altbridging"


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #42, sr #110963 (project freeipmi):

Not a problem, take your time. I can work with what you've given me so far. I
really appreciate it.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #41, sr #110963 (project freeipmi):

Cool.  If you remove the "printfs" in libfreeipmi/sensor-read/sensor-read.c
you should have something that can work for you temporarily.

I just have to debate how to do it.  I'm a little reluctant to change default
behavior in case it breaks things for current users.  So perhaps another
workaround flag.




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #40, sr #110963 (project freeipmi):

Ha i didn't even check the output, just the debug. Yes it does look like it
works! If you can fold it in somehow, is it possible to 'assume bmc owner'
text string ? But really, whatever allows me to use it is perfect!


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #39, sr #110963 (project freeipmi):

i think it worked

> 292 | Fan 1 Front Tach | Fan  | 14924.00   | RPM   | 'OK'

discounting my debug prints, does it work for all sensors?

if so, i need to think of how to fold this in for real


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #38, sr #110963 (project freeipmi):

Trying again

(file #55392)

___

Additional Item Attachment:

File name: dr_ipmi-sensors-debug-id292_v4.txt Size:63 KB
   




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #37, sr #110963 (project freeipmi):

I notice in your debug output you always run out of a `Hyperscale/SR650v3`
directory.  Are you installing the debug ipmi-sensors there?

Try running out of the cloned build directory instead.  I'm wondering if your
ipmi-sensors could be picking up the system libfreeipmi by accident (vs using
my hacked libfreeipmi in the build directory).


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #36, sr #110963 (project freeipmi):

I get the same thing when I run that command in the cloned free-ipmi
directory. I can send all of the output from when I did the complile and the
make if that would help ?


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #35, sr #110963 (project freeipmi):

something isn't quite right, even my printf debug statements aren't appearing
in your output.  Are my hacks in the branch you are building?  Here's what I
get when I run git log.

```
>git log | head -n5
commit 1db6ac40eaecaf84d7615a5a2fd6f5b63919b2bc
Author: Albert Chu 
Date:   Sat Dec 2 08:38:48 2023 -0800

hacks
```





___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #34, sr #110963 (project freeipmi):

Re-cloned the repo, and did a re-make.

Attached new log

(file #55391)

___

Additional Item Attachment:

File name: dr_ipmi-sensors-debug-id292_v3.txt Size:63 KB
   




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #33, sr #110963 (project freeipmi):

remember to "git fetch" to update the branch (or you can re-clone too).


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #32, sr #110963 (project freeipmi):

crud, i'm sorry, I think I pushed the wrong code to that branch ... can you
try again.  thanks.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #31, sr #110963 (project freeipmi):

Reuploaded

(file #55390)

___

Additional Item Attachment:

File name: dr_ipmi-sensors-debug-id292_v2.txt Size:63 KB
   




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #30, sr #110963 (project freeipmi):

I think you forgot to run with --workaround-flags=assumebmcowner


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #28, sr #110963 (project freeipmi):

Could you show the --debug output for the fan sensor?  Maybe I messed
something up.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #27, sr #110963 (project freeipmi):

I think I followed all the commands correctly, but when I ran the ipmi-sensors
command I got the same information. I'm not sure if you incremented the
version number to ensure I made it properly, but it is listed as 1.6.12


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #26, sr #110963 (project freeipmi):

Here's a hack on a branch ... just trying to see if this logic would work
(hopefully no typos in these build instructions):

https://github.com/chu11/freeipmi-mirror/tree/lenovo_workaround

git clone https://github.com/chu11/freeipmi-mirror.git

cd freeipmi-mirror
git checkout lenovo_workaround
./autogen.sh
./configure --enable-debug
make
cd ipmi-sensors
./ipmi-sensors --workaround-flags=assumebmcowner




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #25, sr #110963 (project freeipmi):

Ok, seems that’s the answer.  Lemme see if I can figure out a workaround.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #24, sr #110963 (project freeipmi):

root@sr630v3-2310:/home/jacob# ipmi-raw 0 0x04 0x2d 0x81
rcvd: 2D 00 00 C0 00 80

root@sr630v3-2310:/home/jacob# ipmi-raw 3 0x04 0x2d 0x81
rcvd: 2D 00 50 C0 C0


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #23, sr #110963 (project freeipmi):

so that sorta worked, although 0 rpm doesn't sound right.

looking through more code, i have a theory what might be going on.

on your board i see

```
172.16.40.114: [  10h] = sensor_owner_id[ 7b]
172.16.40.114: [   3h] = sensor_owner_lun[ 2b]
172.16.40.114: [   0h] = sensor_owner_lun.reserved[ 2b]
172.16.40.114: [   0h] = channel_number[ 4b]
```

the sensor owner is the typical one and the lun is != 0.

but in my code I have


```
if (slave_address == IPMI_SLAVE_ADDRESS_BMC && sensor_owner_lun ==
IPMI_BMC_IPMB_LUN_BMC)
   don't do bridging
else
   do bridging

```

which short answer, I do "bridging" when the address & lun aren't the BMC, but
it appears ipmitool may not.  It may only do bridging when the address is not
the BMC.

My assumption is that the bridging I do just happens to work on most
motherboards, but it doesn't on yours (thus the 0xC3 error we saw earlier). 
But it does work the way ipmitool does it.

I'm not sure how to work around this for the moment, but lets see if we can
verify.

ipmi-raw 0 0x04 0x2d 0x81

should emulate what i'm doing now

ipmi-raw 3 0x04 0x2d 0x81

should emulate what ipmitool is doing

my hope is that we'll see

(first case)
rcvd: 2D 00 00 XX XX

(second case)
rcvd: 2D 00 NN XX XX 


where NN will not be 0 in the second case.





___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #22, sr #110963 (project freeipmi):

Attached

(file #55385)

___

Additional Item Attachment:

File name: ipmi-sensor-assumebmcowner-debug.txt Size:61 KB
   




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #21, sr #110963 (project freeipmi):

Could you try the "assumebmcowner" workaround (without --bridge-sensors). 
AFAICT, ipmitool is not "bridging", even though they should.



___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #20, sr #110963 (project freeipmi):

Just to add an additional detail, no sensor information gets return from this
system, except for the coin battery voltage. Nothing else does. I attached a
spreadsheet that shows the difference between ipmitool and ipmi-sensors

(file #55384)

___

Additional Item Attachment:

File name: ipmi_differences.xlsx  Size:24 KB




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #19, sr #110963 (project freeipmi):

Hey Albert, thanks for all the feedback. I'm working to get those commands
rerun.

So ipmitool is working just fine, and is returning the appropriate RPM values
that I'm seeing from the web-based BMC, so I'm not so concerned about
ipmitool. 

Attached are oob debug commands, one with bridge sensors and one without



(file #55382, file #55383)

___

Additional Item Attachment:

File name: ipmi-sensors-oob-bridge.txtSize:64 KB
   


File name: ipmi-sensors-oob.txt   Size:56 KB




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #18, sr #110963 (project freeipmi):

Actually I just noticed something in your out of band debug output.

```
>> Sending IPMI command payload
>>netfn   : 0x04
>>command : 0x2d
BUILDING A v2 COMMAND
Local RqAddr 0x20 transit 0:0 target 0x20:0 bridgePossible 1
<< IPMI Response Session Header
<<   Authtype: RMCP+
<<   Payload type: IPMI (0)
<<   Session ID  : 0xa0a2a3a4
<<   Sequence: 0x0309
<<   IPMI Msg/Payload Length : 32
<< IPMI Response Message Header
<<   Rq Addr: 81
<<   NetFn  : 05
<<   Rq LUN : 0
<<   Rs Addr: 20
<<   Rq Seq : 0f
<<   Rs Lun : 0
<<   Command: 2d
<<   Compl Code : 0x00

```

This is the "get sensor reading" command, and it apparently doesn't return an
error (completion code == 0).


Could you do the out of band --debug --bridge-sensors output again?  Your
debug output of that before didn't connect to the server.

i.e. ipmi-sensors -r NUMBER --debug --bridge-sensors -h host -u user -p
password



___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #17, sr #110963 (project freeipmi):

That said, why your motherboard is returning errors for that sensor is
something you'll probably want to bring up with your vendor.  Could be a
firmware issue or otherwise.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #16, sr #110963 (project freeipmi):

At this point I'm unconvinced that ipmitool is doing the right thing.  It
appears when the 0xC3 error occurs when reading a sensor, ipmitool will output
data under the assumption the reading was valid.

I've submitted a bug to ipmitool on the mailing list for the time being.  We
can see if anyone responds.




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #15, sr #110963 (project freeipmi):

I looked through the ipmitool code and one chunk looks very concerning in
`ipmi_sdr_read_sensor_value()`

```
if (rsp->ccode) { 
 
if ( !((sr.full&& rsp->ccode == 0xcb) ||  
 
   (sr.compact && rsp->ccode == 0xcd)) ) {
 
lprintf(LOG_DEBUG,
 
"Error reading sensor %s (#%02x): %s",
sr.s_id, 
sensor->keys.sensor_num,  
 
val2str(rsp->ccode, completion_code_vals));   
 
} 
 
return
 
}
```

It seems to suggest that if an error occurs during the sensor reading, output
a debug message (if necessary), and then return the sensor reading as though
it were successful.

So this may be a bug in ipmitool in which it gets an error, but ignores the
error.  

What confidence do you have in the RPM values actually being correct?

The RPM may simply be junk data being output.




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #14, sr #110963 (project freeipmi):

Unfortunately the ipmitool debug did not have much (they seem to exclude
packet dumps around the area I care about).

Lemme peruse their code and see if there's something obvious they are doing. 
Perhaps it's a workaround of sorts for Lenovo.

Can you tell me the OS and version of ipmitool you're using.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #13, sr #110963 (project freeipmi):

The -D made know difference. Attached are the in-band and out of band commands

(file #55379, file #55380)

___

Additional Item Attachment:

File name: ipmitool-ib.txtSize:45 KB


File name: ipmitool-oob.txt   Size:501 KB




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #12, sr #110963 (project freeipmi):

Hmmm I see

> Sensor reading/event_bitmask retrieval error: sensor reading
> cannot be obtained

The error code is 0xC3 indicating a timeout.

question, you are using the linux ipmi driver, right?  Normally reading from
/dev/ipmi0?  If by chance it's not using it, you can force it with
"ipmi-sensors -D openipmi"

could you get the debug output from ipmitool to show how it's getting the
sensor reading?  Perhaps there will be a clue.

I think `ipmitool sdr -vvv get "Fan 1 Front Tach"` should get it.

(If possible both inband and outofband ... unfortunately, it doesn't appear
ipmitool outputs a lot of debug info with inband).




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #11, sr #110963 (project freeipmi):

A colleague of mine ran the debug commands in band on his SR650 v3 system. He
used the id that matched with the fan 1 front tach. Hopefully it helps

(file #55376, file #55377)

___

Additional Item Attachment:

File name: jl_ipmi_sensors-debug-id289.txt Size:17 KB
   


File name: jl_ipmi_sensors-debug-id289-bridge.txt Size:17 KB
   




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #10, sr #110963 (project freeipmi):

Your new debugs failed to connect to the server.  Some random mistake I
assume.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #9, sr #110963 (project freeipmi):

Ah, sorry. I had to replace the board in hopes of getting the video to work. I
see what happened. I changed it to 292 now, which matched the Fan 1 Front Tach
(which was 260 originally).

(file #55373, file #55374)

___

Additional Item Attachment:

File name: ipmi_sensors-debug-id292.txt   Size:16 KB
   


File name: ipmi_sensors-debug-id292-bridge.txt Size:66 KB
   




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #8, sr #110963 (project freeipmi):


> 260 | PVNN PCH AUX PG | Voltage | N/A| N/A   | Unknown

Are you running against a different motherboard?  I used the id of 260 based
on your original screenshot.

Not sure if we need to start over with the original debug output.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #7, sr #110963 (project freeipmi):

I can't do this in-band. For some reason my video output isn't working either
locally or remote.

Attached new debug log

(file #55372)

___

Additional Item Attachment:

File name: ipmi_sensors-bridge-sensors-debug.txt Size:61 KB
   




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #6, sr #110963 (project freeipmi):

Could you provide the --debug for --bridge-sensors.  like

ipmi-sensors --record-ids=260 --debug --bridge-sensors

(if possible could you do inband, so the extra out-of-band packet stuff isn't
there)


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #3, sr #110963 (project freeipmi):

I have attempted using the workaround, as I found it in an earlier bug.
Unfortunately that didn't work, as these sensors are coded for RPM and not
assert/deassert.

I ran the debug command as requested, and have attached it here.

(file #55371)

___

Additional Item Attachment:

File name: ipmi_sensors.txt   Size:57 KB




___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #5, sr #110963 (project freeipmi):

No, that doesn't work either


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #4, sr #110963 (project freeipmi):

Does --bridge-sensors work?


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Albert Chu]

Follow-up Comment #2, sr #110963 (project freeipmi):

Hi, does the "discretereading" workaround work, i.e.
--workaround-flags=discretereading.  I have noticed this issue before only on
HP motherboards.

If not, could you send me some debug info.  For just one sensor would be
enough, like

ipmi-sensors --debug --record-ids=260

and we can see what's going on.  Perhaps it's a new problem that hasn't been
seen before.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[Derek Rubright]

Follow-up Comment #1, sr #110963 (project freeipmi):

I submitted this before I created an account. Hopefully that doesn't lead to
problems reviewing this.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[sr #110963] ipmi-sensors does not return fan rpm on Lenovo SR650 v3 but ipmitool does

[anonymous]

URL:
  

 Summary: ipmi-sensors does not return fan rpm on Lenovo SR650
v3 but ipmitool does
   Group: GNU FreeIPMI
   Submitter: None
   Submitted: Wed 29 Nov 2023 11:53:05 AM UTC
Category: ipmi-sensors
Priority: 5 - Normal
Severity: 3 - Normal
  Status: None
 Privacy: Public
 Assigned to: None
Originator Email: derek.rubri...@gmail.com
 Open/Closed: Open
 Discussion Lock: Any
Operating System: GNU/Linux


___

Follow-up Comments:


---
Date: Wed 29 Nov 2023 11:53:05 AM UTC By: Anonymous
While working on a Lenovo SR650 v3 system, I attempted to pull fan RPM info
via ipmi-sensors command (both in-band and out-of-band), and neither returned
the information. I was able to use ipmitool and was able to get the info. I
have attached screenshots showing the differences.






___
File Attachments:


---
Date: Wed 29 Nov 2023 11:53:05 AM UTC  Name: Screenshot@2023-11...@120831.png 
Size: 122KiB   By: None



___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/