UBUNTU-CVE-2025-68310
In the Linux kernel, the following vulnerability has been resolved: s390/pci: Avoid deadlock between PCI error recovery and mlx5 crdump Do not block PCI config accesses through pci_cfg_access_lock() when executing the s390 variant of PCI error recovery: Acquire just device_lock() instead of pci_dev_lock() as powerpc's EEH and generig PCI AER processing do. During error recovery testing a pair of tasks was reported to be hung: mlx5_core 0000:00:00.1: mlx5_health_try_recover:338:(pid 5553): health recovery flow aborted, PCI reads still not working INFO: task kmcheck:72 blocked for more than 122 seconds. Not tainted 5.14.0-570.12.1.bringup7.el9.s390x #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kmcheck state:D stack:0 pid:72 tgid:72 ppid:2 flags:0x00000000 Call Trace: [<000000065256f030>] __schedule+0x2a0/0x590 [<000000065256f356>] schedule+0x36/0xe0 [<000000065256f572>] schedule_preempt_disabled+0x22/0x30 [<0000000652570a94>] __mutex_lock.constprop.0+0x484/0x8a8 [<000003ff800673a4>] mlx5_unload_one+0x34/0x58 [mlx5_core] [<000003ff8006745c>] mlx5_pci_err_detected+0x94/0x140 [mlx5_core] [<0000000652556c5a>] zpci_event_attempt_error_recovery+0xf2/0x398 [<0000000651b9184a>] __zpci_event_error+0x23a/0x2c0 INFO: task kworker/u1664:6:1514 blocked for more than 122 seconds. Not tainted 5.14.0-570.12.1.bringup7.el9.s390x #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/u1664:6 state:D stack:0 pid:1514 tgid:1514 ppid:2 flags:0x00000000 Workqueue: mlx5_health0000:00:00.0 mlx5_fw_fatal_reporter_err_work [mlx5_core] Call Trace: [<000000065256f030>] __schedule+0x2a0/0x590 [<000000065256f356>] schedule+0x36/0xe0 [<0000000652172e28>] pci_wait_cfg+0x80/0xe8 [<0000000652172f94>] pci_cfg_access_lock+0x74/0x88 [<000003ff800916b6>] mlx5_vsc_gw_lock+0x36/0x178 [mlx5_core] [<000003ff80098824>] mlx5_crdump_collect+0x34/0x1c8 [mlx5_core] [<000003ff80074b62>] mlx5_fw_fatal_reporter_dump+0x6a/0xe8 [mlx5_core] [<0000000652512242>] devlink_health_do_dump.part.0+0x82/0x168 [<0000000652513212>] devlink_health_report+0x19a/0x230 [<000003ff80075a12>] mlx5_fw_fatal_reporter_err_work+0xba/0x1b0 [mlx5_core] No kernel log of the exact same error with an upstream kernel is available - but the very same deadlock situation can be constructed there, too: - task: kmcheck mlx5_unload_one() tries to acquire devlink lock while the PCI error recovery code has set pdev->block_cfg_access by way of pci_cfg_access_lock() - task: kworker mlx5_crdump_collect() tries to set block_cfg_access through pci_cfg_access_lock() while devlink_health_report() had acquired the devlink lock. A similar deadlock situation can be reproduced by requesting a crdump with > devlink health dump show pci/<BDF> reporter fw_fatal while PCI error recovery is executed on the same <BDF> physical function by mlx5_core's pci_error_handlers. On s390 this can be injected with > zpcictl --reset-fw <BDF> Tests with this patch failed to reproduce that second deadlock situation, the devlink command is rejected with "kernel answers: Permission denied" - and we get a kernel log message of: mlx5_core 1ed0:00:00.1: mlx5_crdump_collect:50:(pid 254382): crdump: failed to lock vsc gw err -5 because the config read of VSC_SEMAPHORE is rejected by the underlying hardware. Two prior attempts to address this issue have been discussed and ultimately rejected [see link], with the primary argument that s390's implementation of PCI error recovery is imposing restrictions that neither powerpc's EEH nor PCI AER handling need. Tests show that PCI error recovery on s390 is running to completion even without blocking access to PCI config space.
02 / AFFECTED SOFTWARE
Affected packages
3 explicit affected versions
10 explicit affected versions
11 explicit affected versions
68 explicit affected versions
28 explicit affected versions
51 explicit affected versions
8 explicit affected versions
55 explicit affected versions
5 explicit affected versions
8 explicit affected versions
10 explicit affected versions
7 explicit affected versions
39 explicit affected versions
13 explicit affected versions
13 explicit affected versions
26 explicit affected versions
15 explicit affected versions
6 explicit affected versions
80 explicit affected versions
79 explicit affected versions
13 explicit affected versions
1 explicit affected versions
1 explicit affected versions
1 explicit affected versions
91 explicit affected versions
12 explicit affected versions
7 explicit affected versions
71 explicit affected versions
80 explicit affected versions
11 explicit affected versions
3 explicit affected versions
89 explicit affected versions
50 explicit affected versions
6 explicit affected versions
12 explicit affected versions
19 explicit affected versions
10 explicit affected versions
6 explicit affected versions
100 explicit affected versions
24 explicit affected versions
85 explicit affected versions
84 explicit affected versions
86 explicit affected versions
76 explicit affected versions
10 explicit affected versions
30 explicit affected versions
26 explicit affected versions
90 explicit affected versions
7 explicit affected versions
84 explicit affected versions
34 explicit affected versions
13 explicit affected versions
37 explicit affected versions
35 explicit affected versions
51 explicit affected versions
50 explicit affected versions
18 explicit affected versions
4 explicit affected versions
42 explicit affected versions
64 explicit affected versions
39 explicit affected versions
13 explicit affected versions
10 explicit affected versions
23 explicit affected versions
45 explicit affected versions
1 explicit affected versions
21 explicit affected versions
9 explicit affected versions
16 explicit affected versions
41 explicit affected versions
8 explicit affected versions
7 explicit affected versions
12 explicit affected versions
11 explicit affected versions
7 explicit affected versions
12 explicit affected versions
14 explicit affected versions
10 explicit affected versions
1 explicit affected versions
23 explicit affected versions
7 explicit affected versions
9 explicit affected versions
12 explicit affected versions
1 explicit affected versions
12 explicit affected versions
13 explicit affected versions
34 explicit affected versions
45 explicit affected versions
16 explicit affected versions
9 explicit affected versions
33 explicit affected versions
27 explicit affected versions
10 explicit affected versions
37 explicit affected versions
10 explicit affected versions
1 explicit affected versions
26 explicit affected versions
5 explicit affected versions
13 explicit affected versions
37 explicit affected versions
29 explicit affected versions
13 explicit affected versions
12 explicit affected versions
96 explicit affected versions
34 explicit affected versions
34 explicit affected versions
92 explicit affected versions
55 explicit affected versions
33 explicit affected versions
23 explicit affected versions
60 explicit affected versions
10 explicit affected versions
13 explicit affected versions
12 explicit affected versions
69 explicit affected versions
11 explicit affected versions
5 explicit affected versions
12 explicit affected versions
1 explicit affected versions
7 explicit affected versions
12 explicit affected versions
8 explicit affected versions
5 explicit affected versions
77 explicit affected versions
93 explicit affected versions
10 explicit affected versions
26 explicit affected versions
32 explicit affected versions
1 explicit affected versions
40 explicit affected versions
11 explicit affected versions
1 explicit affected versions
13 explicit affected versions
39 explicit affected versions
16 explicit affected versions
91 explicit affected versions
43 explicit affected versions
3 explicit affected versions
7 explicit affected versions
2 explicit affected versions
8 explicit affected versions
18 explicit affected versions
35 explicit affected versions
12 explicit affected versions
1 explicit affected versions
27 explicit affected versions
66 explicit affected versions
36 explicit affected versions
13 explicit affected versions
7 explicit affected versions
8 explicit affected versions
10 explicit affected versions
12 explicit affected versions
9 explicit affected versions
39 explicit affected versions
44 explicit affected versions
16 explicit affected versions
45 explicit affected versions
37 explicit affected versions
44 explicit affected versions
14 explicit affected versions
38 explicit affected versions
9 explicit affected versions
9 explicit affected versions
11 explicit affected versions
32 explicit affected versions
92 explicit affected versions
8 explicit affected versions
7 explicit affected versions
4 explicit affected versions
81 explicit affected versions
43 explicit affected versions
6 explicit affected versions
10 explicit affected versions
8 explicit affected versions
38 explicit affected versions
14 explicit affected versions
4 explicit affected versions
78 explicit affected versions
6 explicit affected versions
11 explicit affected versions
1 explicit affected versions
12 explicit affected versions
32 explicit affected versions
80 explicit affected versions
03 / CONNECTIONS
Connected vulnerabilities
04 / EVIDENCE
Source records
In the Linux kernel, the following vulnerability has been resolved: s390/pci: Avoid deadlock between PCI error recovery and mlx5 crdump Do not block PCI config accesses through pci_cfg_access_lock() when executing the s390 variant of PCI error recovery: Acquire just device_lock() instead of pci_dev_lock() as powerpc's EEH and generig PCI AER processing do. During error recovery testing a pair of tasks was reported to be hung: mlx5_core 0000:00:00.1: mlx5_health_try_recover:338:(pid 5553): health recovery flow aborted, PCI reads still not working INFO: task kmcheck:72 blocked for more than 122 seconds. Not tainted 5.14.0-570.12.1.bringup7.el9.s390x #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kmcheck state:D stack:0 pid:72 tgid:72 ppid:2 flags:0x00000000 Call Trace: [<000000065256f030>] __schedule+0x2a0/0x590 [<000000065256f356>] schedule+0x36/0xe0 [<000000065256f572>] schedule_preempt_disabled+0x22/0x30 [<0000000652570a94>] __mutex_lock.constprop.0+0x484/0x8a8 [<000003ff800673a4>] mlx5_unload_one+0x34/0x58 [mlx5_core] [<000003ff8006745c>] mlx5_pci_err_detected+0x94/0x140 [mlx5_core] [<0000000652556c5a>] zpci_event_attempt_error_recovery+0xf2/0x398 [<0000000651b9184a>] __zpci_event_error+0x23a/0x2c0 INFO: task kworker/u1664:6:1514 blocked for more than 122 seconds. Not tainted 5.14.0-570.12.1.bringup7.el9.s390x #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/u1664:6 state:D stack:0 pid:1514 tgid:1514 ppid:2 flags:0x00000000 Workqueue: mlx5_health0000:00:00.0 mlx5_fw_fatal_reporter_err_work [mlx5_core] Call Trace: [<000000065256f030>] __schedule+0x2a0/0x590 [<000000065256f356>] schedule+0x36/0xe0 [<0000000652172e28>] pci_wait_cfg+0x80/0xe8 [<0000000652172f94>] pci_cfg_access_lock+0x74/0x88 [<000003ff800916b6>] mlx5_vsc_gw_lock+0x36/0x178 [mlx5_core] [<000003ff80098824>] mlx5_crdump_collect+0x34/0x1c8 [mlx5_core] [<000003ff80074b62>] mlx5_fw_fatal_reporter_dump+0x6a/0xe8 [mlx5_core] [<0000000652512242>] devlink_health_do_dump.part.0+0x82/0x168 [<0000000652513212>] devlink_health_report+0x19a/0x230 [<000003ff80075a12>] mlx5_fw_fatal_reporter_err_work+0xba/0x1b0 [mlx5_core] No kernel log of the exact same error with an upstream kernel is available - but the very same deadlock situation can be constructed there, too: - task: kmcheck mlx5_unload_one() tries to acquire devlink lock while the PCI error recovery code has set pdev->block_cfg_access by way of pci_cfg_access_lock() - task: kworker mlx5_crdump_collect() tries to set block_cfg_access through pci_cfg_access_lock() while devlink_health_report() had acquired the devlink lock. A similar deadlock situation can be reproduced by requesting a crdump with > devlink health dump show pci/<BDF> reporter fw_fatal while PCI error recovery is executed on the same <BDF> physical function by mlx5_core's pci_error_handlers. On s390 this can be injected with > zpcictl --reset-fw <BDF> Tests with this patch failed to reproduce that second deadlock situation, the devlink command is rejected with "kernel answers: Permission denied" - and we get a kernel log message of: mlx5_core 1ed0:00:00.1: mlx5_crdump_collect:50:(pid 254382): crdump: failed to lock vsc gw err -5 because the config read of VSC_SEMAPHORE is rejected by the underlying hardware. Two prior attempts to address this issue have been discussed and ultimately rejected [see link], with the primary argument that s390's implementation of PCI error recovery is imposing restrictions that neither powerpc's EEH nor PCI AER handling need. Tests show that PCI error recovery on s390 is running to completion even without blocking access to PCI config space.
05 / REFERENCES
Further evidence
- https://git.kernel.org/linus/0fd20f65df6aa430454a0deed8f43efa91c54835
- https://git.kernel.org/stable/c/0fd20f65df6aa430454a0deed8f43efa91c54835
- https://git.kernel.org/stable/c/3591d56ea9bfd3e7fbbe70f749bdeed689d415f9
- https://git.kernel.org/stable/c/54f938d9f5693af8ed586a08db4af5d9da1f0f2d
- https://git.kernel.org/stable/c/b63c061be622b17b495cbf78a6d5f2d4c3147f8e
- https://git.kernel.org/stable/c/d0df2503bc3c2be385ca2fd96585daad1870c7c5
- https://ubuntu.com/security/CVE-2025-68310
- https://ubuntu.com/security/notices/USN-8029-1
- https://ubuntu.com/security/notices/USN-8029-2
- https://ubuntu.com/security/notices/USN-8029-3
- https://ubuntu.com/security/notices/USN-8030-1
- https://ubuntu.com/security/notices/USN-8048-1
- https://ubuntu.com/security/notices/USN-8095-1
- https://ubuntu.com/security/notices/USN-8095-2
- https://ubuntu.com/security/notices/USN-8095-3
- https://ubuntu.com/security/notices/USN-8095-4
- https://ubuntu.com/security/notices/USN-8095-5
- https://ubuntu.com/security/notices/USN-8100-1
- https://ubuntu.com/security/notices/USN-8125-1
- https://ubuntu.com/security/notices/USN-8126-1
- https://ubuntu.com/security/notices/USN-8165-1
- https://ubuntu.com/security/notices/USN-8261-1
- https://www.cve.org/CVERecord?id=CVE-2025-68310