UBUNTU-CVE-2023-52587
In the Linux kernel, the following vulnerability has been resolved: IB/ipoib: Fix mcast list locking Releasing the `priv->lock` while iterating the `priv->multicast_list` in `ipoib_mcast_join_task()` opens a window for `ipoib_mcast_dev_flush()` to remove the items while in the middle of iteration. If the mcast is removed while the lock was dropped, the for loop spins forever resulting in a hard lockup (as was reported on RHEL 4.18.0-372.75.1.el8_6 kernel): Task A (kworker/u72:2 below) | Task B (kworker/u72:0 below) -----------------------------------+----------------------------------- ipoib_mcast_join_task(work) | ipoib_ib_dev_flush_light(work) spin_lock_irq(&priv->lock) | __ipoib_ib_dev_flush(priv, ...) list_for_each_entry(mcast, | ipoib_mcast_dev_flush(dev = priv->dev) &priv->multicast_list, list) | ipoib_mcast_join(dev, mcast) | spin_unlock_irq(&priv->lock) | | spin_lock_irqsave(&priv->lock, flags) | list_for_each_entry_safe(mcast, tmcast, | &priv->multicast_list, list) | list_del(&mcast->list); | list_add_tail(&mcast->list, &remove_list) | spin_unlock_irqrestore(&priv->lock, flags) spin_lock_irq(&priv->lock) | | ipoib_mcast_remove_list(&remove_list) (Here, `mcast` is no longer on the | list_for_each_entry_safe(mcast, tmcast, `priv->multicast_list` and we keep | remove_list, list) spinning on the `remove_list` of | >>> wait_for_completion(&mcast->done) the other thread which is blocked | and the list is still valid on | it's stack.) Fix this by keeping the lock held and changing to GFP_ATOMIC to prevent eventual sleeps. Unfortunately we could not reproduce the lockup and confirm this fix but based on the code review I think this fix should address such lockups. crash> bc 31 PID: 747 TASK: ff1c6a1a007e8000 CPU: 31 COMMAND: "kworker/u72:2" -- [exception RIP: ipoib_mcast_join_task+0x1b1] RIP: ffffffffc0944ac1 RSP: ff646f199a8c7e00 RFLAGS: 00000002 RAX: 0000000000000000 RBX: ff1c6a1a04dc82f8 RCX: 0000000000000000 work (&priv->mcast_task{,.work}) RDX: ff1c6a192d60ac68 RSI: 0000000000000286 RDI: ff1c6a1a04dc8000 &mcast->list RBP: ff646f199a8c7e90 R8: ff1c699980019420 R9: ff1c6a1920c9a000 R10: ff646f199a8c7e00 R11: ff1c6a191a7d9800 R12: ff1c6a192d60ac00 mcast R13: ff1c6a1d82200000 R14: ff1c6a1a04dc8000 R15: ff1c6a1a04dc82d8 dev priv (&priv->lock) &priv->multicast_list (aka head) ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- <NMI exception stack> --- #5 [ff646f199a8c7e00] ipoib_mcast_join_task+0x1b1 at ffffffffc0944ac1 [ib_ipoib] #6 [ff646f199a8c7e98] process_one_work+0x1a7 at ffffffff9bf10967 crash> rx ff646f199a8c7e68 ff646f199a8c7e68: ff1c6a1a04dc82f8 <<< work = &priv->mcast_task.work crash> list -hO ipoib_dev_priv.multicast_list ff1c6a1a04dc8000 (empty) crash> ipoib_dev_priv.mcast_task.work.func,mcast_mutex.owner.counter ff1c6a1a04dc8000 mcast_task.work.func = 0xffffffffc0944910 <ipoib_mcast_join_task>, mcast_mutex.owner.counter = 0xff1c69998efec000 crash> b 8 PID: 8 TASK: ff1c69998efec000 CPU: 33 COMMAND: "kworker/u72:0" -- #3 [ff646f1980153d50] wait_for_completion+0x96 at ffffffff9c7d7646 #4 [ff646f1980153d90] ipoib_mcast_remove_list+0x56 at ffffffffc0944dc6 [ib_ipoib] #5 [ff646f1980153de8] ipoib_mcast_dev_flush+0x1a7 at ffffffffc09455a7 [ib_ipoib] #6 [ff646f1980153e58] __ipoib_ib_dev_flush+0x1a4 at ffffffffc09431a4 [ib_ipoib] #7 [ff ---truncated---
02 / AFFECTED SOFTWARE
Affected packages
7 explicit affected versions
29 explicit affected versions
41 explicit affected versions
50 explicit affected versions
6 explicit affected versions
12 explicit affected versions
16 explicit affected versions
6 explicit affected versions
50 explicit affected versions
57 explicit affected versions
36 explicit affected versions
57 explicit affected versions
47 explicit affected versions
36 explicit affected versions
7 explicit affected versions
43 explicit affected versions
191 explicit affected versions
7 explicit affected versions
44 explicit affected versions
84 explicit affected versions
9 explicit affected versions
85 explicit affected versions
35 explicit affected versions
3 explicit affected versions
14 explicit affected versions
18 explicit affected versions
5 explicit affected versions
118 explicit affected versions
30 explicit affected versions
86 explicit affected versions
13 explicit affected versions
4 explicit affected versions
121 explicit affected versions
23 explicit affected versions
1 explicit affected versions
21 explicit affected versions
9 explicit affected versions
16 explicit affected versions
10 explicit affected versions
41 explicit affected versions
7 explicit affected versions
12 explicit affected versions
111 explicit affected versions
34 explicit affected versions
12 explicit affected versions
10 explicit affected versions
7 explicit affected versions
74 explicit affected versions
12 explicit affected versions
89 explicit affected versions
97 explicit affected versions
12 explicit affected versions
85 explicit affected versions
120 explicit affected versions
10 explicit affected versions
10 explicit affected versions
37 explicit affected versions
9 explicit affected versions
10 explicit affected versions
23 explicit affected versions
1 explicit affected versions
27 explicit affected versions
26 explicit affected versions
13 explicit affected versions
144 explicit affected versions
4 explicit affected versions
1 explicit affected versions
29 explicit affected versions
78 explicit affected versions
13 explicit affected versions
47 explicit affected versions
1 explicit affected versions
73 explicit affected versions
45 explicit affected versions
89 explicit affected versions
7 explicit affected versions
33 explicit affected versions
23 explicit affected versions
10 explicit affected versions
188 explicit affected versions
11 explicit affected versions
12 explicit affected versions
7 explicit affected versions
12 explicit affected versions
88 explicit affected versions
138 explicit affected versions
2 explicit affected versions
36 explicit affected versions
28 explicit affected versions
132 explicit affected versions
44 explicit affected versions
165 explicit affected versions
154 explicit affected versions
10 explicit affected versions
26 explicit affected versions
23 explicit affected versions
1 explicit affected versions
13 explicit affected versions
139 explicit affected versions
47 explicit affected versions
16 explicit affected versions
47 explicit affected versions
37 explicit affected versions
116 explicit affected versions
3 explicit affected versions
93 explicit affected versions
1 explicit affected versions
2 explicit affected versions
54 explicit affected versions
51 explicit affected versions
92 explicit affected versions
1 explicit affected versions
12 explicit affected versions
34 explicit affected versions
25 explicit affected versions
20 explicit affected versions
1 explicit affected versions
1 explicit affected versions
98 explicit affected versions
8 explicit affected versions
8 explicit affected versions
5 explicit affected versions
8 explicit affected versions
1 explicit affected versions
10 explicit affected versions
9 explicit affected versions
77 explicit affected versions
13 explicit affected versions
41 explicit affected versions
7 explicit affected versions
140 explicit affected versions
94 explicit affected versions
39 explicit affected versions
10 explicit affected versions
12 explicit affected versions
12 explicit affected versions
76 explicit affected versions
44 explicit affected versions
16 explicit affected versions
3 explicit affected versions
13 explicit affected versions
26 explicit affected versions
33 explicit affected versions
37 explicit affected versions
14 explicit affected versions
38 explicit affected versions
167 explicit affected versions
37 explicit affected versions
9 explicit affected versions
36 explicit affected versions
11 explicit affected versions
44 explicit affected versions
8 explicit affected versions
7 explicit affected versions
18 explicit affected versions
97 explicit affected versions
48 explicit affected versions
121 explicit affected versions
4 explicit affected versions
1 explicit affected versions
1 explicit affected versions
42 explicit affected versions
10 explicit affected versions
43 explicit affected versions
42 explicit affected versions
10 explicit affected versions
8 explicit affected versions
14 explicit affected versions
77 explicit affected versions
4 explicit affected versions
66 explicit affected versions
7 explicit affected versions
1 explicit affected versions
69 explicit affected versions
03 / CONNECTIONS
Connected vulnerabilities
04 / EVIDENCE
Source records
In the Linux kernel, the following vulnerability has been resolved: IB/ipoib: Fix mcast list locking Releasing the `priv->lock` while iterating the `priv->multicast_list` in `ipoib_mcast_join_task()` opens a window for `ipoib_mcast_dev_flush()` to remove the items while in the middle of iteration. If the mcast is removed while the lock was dropped, the for loop spins forever resulting in a hard lockup (as was reported on RHEL 4.18.0-372.75.1.el8_6 kernel): Task A (kworker/u72:2 below) | Task B (kworker/u72:0 below) -----------------------------------+----------------------------------- ipoib_mcast_join_task(work) | ipoib_ib_dev_flush_light(work) spin_lock_irq(&priv->lock) | __ipoib_ib_dev_flush(priv, ...) list_for_each_entry(mcast, | ipoib_mcast_dev_flush(dev = priv->dev) &priv->multicast_list, list) | ipoib_mcast_join(dev, mcast) | spin_unlock_irq(&priv->lock) | | spin_lock_irqsave(&priv->lock, flags) | list_for_each_entry_safe(mcast, tmcast, | &priv->multicast_list, list) | list_del(&mcast->list); | list_add_tail(&mcast->list, &remove_list) | spin_unlock_irqrestore(&priv->lock, flags) spin_lock_irq(&priv->lock) | | ipoib_mcast_remove_list(&remove_list) (Here, `mcast` is no longer on the | list_for_each_entry_safe(mcast, tmcast, `priv->multicast_list` and we keep | remove_list, list) spinning on the `remove_list` of | >>> wait_for_completion(&mcast->done) the other thread which is blocked | and the list is still valid on | it's stack.) Fix this by keeping the lock held and changing to GFP_ATOMIC to prevent eventual sleeps. Unfortunately we could not reproduce the lockup and confirm this fix but based on the code review I think this fix should address such lockups. crash> bc 31 PID: 747 TASK: ff1c6a1a007e8000 CPU: 31 COMMAND: "kworker/u72:2" -- [exception RIP: ipoib_mcast_join_task+0x1b1] RIP: ffffffffc0944ac1 RSP: ff646f199a8c7e00 RFLAGS: 00000002 RAX: 0000000000000000 RBX: ff1c6a1a04dc82f8 RCX: 0000000000000000 work (&priv->mcast_task{,.work}) RDX: ff1c6a192d60ac68 RSI: 0000000000000286 RDI: ff1c6a1a04dc8000 &mcast->list RBP: ff646f199a8c7e90 R8: ff1c699980019420 R9: ff1c6a1920c9a000 R10: ff646f199a8c7e00 R11: ff1c6a191a7d9800 R12: ff1c6a192d60ac00 mcast R13: ff1c6a1d82200000 R14: ff1c6a1a04dc8000 R15: ff1c6a1a04dc82d8 dev priv (&priv->lock) &priv->multicast_list (aka head) ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- <NMI exception stack> --- #5 [ff646f199a8c7e00] ipoib_mcast_join_task+0x1b1 at ffffffffc0944ac1 [ib_ipoib] #6 [ff646f199a8c7e98] process_one_work+0x1a7 at ffffffff9bf10967 crash> rx ff646f199a8c7e68 ff646f199a8c7e68: ff1c6a1a04dc82f8 <<< work = &priv->mcast_task.work crash> list -hO ipoib_dev_priv.multicast_list ff1c6a1a04dc8000 (empty) crash> ipoib_dev_priv.mcast_task.work.func,mcast_mutex.owner.counter ff1c6a1a04dc8000 mcast_task.work.func = 0xffffffffc0944910 <ipoib_mcast_join_task>, mcast_mutex.owner.counter = 0xff1c69998efec000 crash> b 8 PID: 8 TASK: ff1c69998efec000 CPU: 33 COMMAND: "kworker/u72:0" -- #3 [ff646f1980153d50] wait_for_completion+0x96 at ffffffff9c7d7646 #4 [ff646f1980153d90] ipoib_mcast_remove_list+0x56 at ffffffffc0944dc6 [ib_ipoib] #5 [ff646f1980153de8] ipoib_mcast_dev_flush+0x1a7 at ffffffffc09455a7 [ib_ipoib] #6 [ff646f1980153e58] __ipoib_ib_dev_flush+0x1a4 at ffffffffc09431a4 [ib_ipoib] #7 [ff ---truncated---
05 / REFERENCES
Further evidence
- https://git.kernel.org/linus/4f973e211b3b1c6d36f7c6a19239d258856749f9
- https://git.kernel.org/stable/c/342258fb46d66c1b4c7e2c3717ac01e10c03cf18
- https://git.kernel.org/stable/c/4c8922ae8eb8dcc1e4b7d1059d97a8334288d825
- https://git.kernel.org/stable/c/4f973e211b3b1c6d36f7c6a19239d258856749f9
- https://git.kernel.org/stable/c/5108a2dc2db5630fb6cd58b8be80a0c134bc310a
- https://git.kernel.org/stable/c/615e3adc2042b7be4ad122a043fc9135e6342c90
- https://git.kernel.org/stable/c/7c7bd4d561e9dc6f5b7df9e184974915f6701a89
- https://git.kernel.org/stable/c/ac2630fd3c90ffec34a0bfc4d413668538b0e8f2
- https://git.kernel.org/stable/c/ed790bd0903ed3352ebf7f650d910f49b7319b34
- https://ubuntu.com/security/CVE-2023-52587
- https://ubuntu.com/security/notices/USN-6688-1
- https://ubuntu.com/security/notices/USN-6766-1
- https://ubuntu.com/security/notices/USN-6766-2
- https://ubuntu.com/security/notices/USN-6766-3
- https://ubuntu.com/security/notices/USN-6767-1
- https://ubuntu.com/security/notices/USN-6767-2
- https://ubuntu.com/security/notices/USN-6795-1
- https://ubuntu.com/security/notices/USN-6818-1
- https://ubuntu.com/security/notices/USN-6818-2
- https://ubuntu.com/security/notices/USN-6818-3
- https://ubuntu.com/security/notices/USN-6818-4
- https://ubuntu.com/security/notices/USN-6819-1
- https://ubuntu.com/security/notices/USN-6819-2
- https://ubuntu.com/security/notices/USN-6819-3
- https://ubuntu.com/security/notices/USN-6819-4
- https://ubuntu.com/security/notices/USN-6828-1
- https://www.cve.org/CVERecord?id=CVE-2023-52587