UBUNTU-CVE-2024-50095
In the Linux kernel, the following vulnerability has been resolved: RDMA/mad: Improve handling of timed out WRs of mad agent Current timeout handler of mad agent acquires/releases mad_agent_priv lock for every timed out WRs. This causes heavy locking contention when higher no. of WRs are to be handled inside timeout handler. This leads to softlockup with below trace in some use cases where rdma-cm path is used to establish connection between peer nodes Trace: ----- BUG: soft lockup - CPU#4 stuck for 26s! [kworker/u128:3:19767] CPU: 4 PID: 19767 Comm: kworker/u128:3 Kdump: loaded Tainted: G OE ------- --- 5.14.0-427.13.1.el9_4.x86_64 #1 Hardware name: Dell Inc. PowerEdge R740/01YM03, BIOS 2.4.8 11/26/2019 Workqueue: ib_mad1 timeout_sends [ib_core] RIP: 0010:__do_softirq+0x78/0x2ac RSP: 0018:ffffb253449e4f98 EFLAGS: 00000246 RAX: 00000000ffffffff RBX: 0000000000000000 RCX: 000000000000001f RDX: 000000000000001d RSI: 000000003d1879ab RDI: fff363b66fd3a86b RBP: ffffb253604cbcd8 R08: 0000009065635f3b R09: 0000000000000000 R10: 0000000000000040 R11: ffffb253449e4ff8 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000040 FS: 0000000000000000(0000) GS:ffff8caa1fc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fd9ec9db900 CR3: 0000000891934006 CR4: 00000000007706e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: <IRQ> ? show_trace_log_lvl+0x1c4/0x2df ? show_trace_log_lvl+0x1c4/0x2df ? __irq_exit_rcu+0xa1/0xc0 ? watchdog_timer_fn+0x1b2/0x210 ? __pfx_watchdog_timer_fn+0x10/0x10 ? __hrtimer_run_queues+0x127/0x2c0 ? hrtimer_interrupt+0xfc/0x210 ? __sysvec_apic_timer_interrupt+0x5c/0x110 ? sysvec_apic_timer_interrupt+0x37/0x90 ? asm_sysvec_apic_timer_interrupt+0x16/0x20 ? __do_softirq+0x78/0x2ac ? __do_softirq+0x60/0x2ac __irq_exit_rcu+0xa1/0xc0 sysvec_call_function_single+0x72/0x90 </IRQ> <TASK> asm_sysvec_call_function_single+0x16/0x20 RIP: 0010:_raw_spin_unlock_irq+0x14/0x30 RSP: 0018:ffffb253604cbd88 EFLAGS: 00000247 RAX: 000000000001960d RBX: 0000000000000002 RCX: ffff8cad2a064800 RDX: 000000008020001b RSI: 0000000000000001 RDI: ffff8cad5d39f66c RBP: ffff8cad5d39f600 R08: 0000000000000001 R09: 0000000000000000 R10: ffff8caa443e0c00 R11: ffffb253604cbcd8 R12: ffff8cacb8682538 R13: 0000000000000005 R14: ffffb253604cbd90 R15: ffff8cad5d39f66c cm_process_send_error+0x122/0x1d0 [ib_cm] timeout_sends+0x1dd/0x270 [ib_core] process_one_work+0x1e2/0x3b0 ? __pfx_worker_thread+0x10/0x10 worker_thread+0x50/0x3a0 ? __pfx_worker_thread+0x10/0x10 kthread+0xdd/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x29/0x50 </TASK> Simplified timeout handler by creating local list of timed out WRs and invoke send handler post creating the list. The new method acquires/ releases lock once to fetch the list and hence helps to reduce locking contetiong when processing higher no. of WRs
02 / AFFECTED SOFTWARE
Affected packages
10 explicit affected versions
23 explicit affected versions
13 explicit affected versions
19 explicit affected versions
130 explicit affected versions
71 explicit affected versions
16 explicit affected versions
59 explicit affected versions
22 explicit affected versions
47 explicit affected versions
3 explicit affected versions
124 explicit affected versions
20 explicit affected versions
12 explicit affected versions
45 explicit affected versions
84 explicit affected versions
1 explicit affected versions
1 explicit affected versions
2 explicit affected versions
51 explicit affected versions
83 explicit affected versions
11 explicit affected versions
45 explicit affected versions
108 explicit affected versions
39 explicit affected versions
53 explicit affected versions
50 explicit affected versions
6 explicit affected versions
12 explicit affected versions
19 explicit affected versions
6 explicit affected versions
63 explicit affected versions
57 explicit affected versions
19 explicit affected versions
48 explicit affected versions
83 explicit affected versions
58 explicit affected versions
49 explicit affected versions
10 explicit affected versions
11 explicit affected versions
7 explicit affected versions
17 explicit affected versions
55 explicit affected versions
191 explicit affected versions
7 explicit affected versions
54 explicit affected versions
14 explicit affected versions
13 explicit affected versions
77 explicit affected versions
16 explicit affected versions
35 explicit affected versions
15 explicit affected versions
18 explicit affected versions
18 explicit affected versions
16 explicit affected versions
109 explicit affected versions
64 explicit affected versions
19 explicit affected versions
113 explicit affected versions
13 explicit affected versions
4 explicit affected versions
23 explicit affected versions
114 explicit affected versions
24 explicit affected versions
116 explicit affected versions
1 explicit affected versions
1 explicit affected versions
21 explicit affected versions
16 explicit affected versions
10 explicit affected versions
67 explicit affected versions
20 explicit affected versions
7 explicit affected versions
12 explicit affected versions
1 explicit affected versions
46 explicit affected versions
12 explicit affected versions
14 explicit affected versions
7 explicit affected versions
1 explicit affected versions
97 explicit affected versions
12 explicit affected versions
99 explicit affected versions
12 explicit affected versions
14 explicit affected versions
113 explicit affected versions
14 explicit affected versions
11 explicit affected versions
6 explicit affected versions
10 explicit affected versions
181 explicit affected versions
37 explicit affected versions
1 explicit affected versions
13 explicit affected versions
12 explicit affected versions
10 explicit affected versions
33 explicit affected versions
1 explicit affected versions
26 explicit affected versions
13 explicit affected versions
10 explicit affected versions
1 explicit affected versions
29 explicit affected versions
103 explicit affected versions
13 explicit affected versions
60 explicit affected versions
13 explicit affected versions
14 explicit affected versions
1 explicit affected versions
94 explicit affected versions
57 explicit affected versions
19 explicit affected versions
33 explicit affected versions
23 explicit affected versions
10 explicit affected versions
1 explicit affected versions
107 explicit affected versions
11 explicit affected versions
12 explicit affected versions
130 explicit affected versions
7 explicit affected versions
12 explicit affected versions
2 explicit affected versions
46 explicit affected versions
40 explicit affected versions
122 explicit affected versions
56 explicit affected versions
158 explicit affected versions
144 explicit affected versions
10 explicit affected versions
26 explicit affected versions
18 explicit affected versions
1 explicit affected versions
27 explicit affected versions
31 explicit affected versions
113 explicit affected versions
1 explicit affected versions
1 explicit affected versions
88 explicit affected versions
15 explicit affected versions
8 explicit affected versions
20 explicit affected versions
5 explicit affected versions
8 explicit affected versions
1 explicit affected versions
10 explicit affected versions
18 explicit affected versions
13 explicit affected versions
102 explicit affected versions
13 explicit affected versions
64 explicit affected versions
7 explicit affected versions
2 explicit affected versions
85 explicit affected versions
50 explicit affected versions
10 explicit affected versions
14 explicit affected versions
12 explicit affected versions
101 explicit affected versions
74 explicit affected versions
48 explicit affected versions
44 explicit affected versions
16 explicit affected versions
3 explicit affected versions
13 explicit affected versions
26 explicit affected versions
45 explicit affected versions
37 explicit affected versions
23 explicit affected versions
14 explicit affected versions
38 explicit affected versions
49 explicit affected versions
9 explicit affected versions
48 explicit affected versions
11 explicit affected versions
13 explicit affected versions
7 explicit affected versions
56 explicit affected versions
7 explicit affected versions
29 explicit affected versions
90 explicit affected versions
134 explicit affected versions
113 explicit affected versions
4 explicit affected versions
1 explicit affected versions
1 explicit affected versions
1 explicit affected versions
54 explicit affected versions
104 explicit affected versions
13 explicit affected versions
43 explicit affected versions
69 explicit affected versions
137 explicit affected versions
111 explicit affected versions
158 explicit affected versions
10 explicit affected versions
8 explicit affected versions
17 explicit affected versions
81 explicit affected versions
14 explicit affected versions
4 explicit affected versions
78 explicit affected versions
11 explicit affected versions
1 explicit affected versions
60 explicit affected versions
115 explicit affected versions
14 explicit affected versions
03 / CONNECTIONS
Connected vulnerabilities
04 / EVIDENCE
Source records
In the Linux kernel, the following vulnerability has been resolved: RDMA/mad: Improve handling of timed out WRs of mad agent Current timeout handler of mad agent acquires/releases mad_agent_priv lock for every timed out WRs. This causes heavy locking contention when higher no. of WRs are to be handled inside timeout handler. This leads to softlockup with below trace in some use cases where rdma-cm path is used to establish connection between peer nodes Trace: ----- BUG: soft lockup - CPU#4 stuck for 26s! [kworker/u128:3:19767] CPU: 4 PID: 19767 Comm: kworker/u128:3 Kdump: loaded Tainted: G OE ------- --- 5.14.0-427.13.1.el9_4.x86_64 #1 Hardware name: Dell Inc. PowerEdge R740/01YM03, BIOS 2.4.8 11/26/2019 Workqueue: ib_mad1 timeout_sends [ib_core] RIP: 0010:__do_softirq+0x78/0x2ac RSP: 0018:ffffb253449e4f98 EFLAGS: 00000246 RAX: 00000000ffffffff RBX: 0000000000000000 RCX: 000000000000001f RDX: 000000000000001d RSI: 000000003d1879ab RDI: fff363b66fd3a86b RBP: ffffb253604cbcd8 R08: 0000009065635f3b R09: 0000000000000000 R10: 0000000000000040 R11: ffffb253449e4ff8 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000040 FS: 0000000000000000(0000) GS:ffff8caa1fc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fd9ec9db900 CR3: 0000000891934006 CR4: 00000000007706e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: <IRQ> ? show_trace_log_lvl+0x1c4/0x2df ? show_trace_log_lvl+0x1c4/0x2df ? __irq_exit_rcu+0xa1/0xc0 ? watchdog_timer_fn+0x1b2/0x210 ? __pfx_watchdog_timer_fn+0x10/0x10 ? __hrtimer_run_queues+0x127/0x2c0 ? hrtimer_interrupt+0xfc/0x210 ? __sysvec_apic_timer_interrupt+0x5c/0x110 ? sysvec_apic_timer_interrupt+0x37/0x90 ? asm_sysvec_apic_timer_interrupt+0x16/0x20 ? __do_softirq+0x78/0x2ac ? __do_softirq+0x60/0x2ac __irq_exit_rcu+0xa1/0xc0 sysvec_call_function_single+0x72/0x90 </IRQ> <TASK> asm_sysvec_call_function_single+0x16/0x20 RIP: 0010:_raw_spin_unlock_irq+0x14/0x30 RSP: 0018:ffffb253604cbd88 EFLAGS: 00000247 RAX: 000000000001960d RBX: 0000000000000002 RCX: ffff8cad2a064800 RDX: 000000008020001b RSI: 0000000000000001 RDI: ffff8cad5d39f66c RBP: ffff8cad5d39f600 R08: 0000000000000001 R09: 0000000000000000 R10: ffff8caa443e0c00 R11: ffffb253604cbcd8 R12: ffff8cacb8682538 R13: 0000000000000005 R14: ffffb253604cbd90 R15: ffff8cad5d39f66c cm_process_send_error+0x122/0x1d0 [ib_cm] timeout_sends+0x1dd/0x270 [ib_core] process_one_work+0x1e2/0x3b0 ? __pfx_worker_thread+0x10/0x10 worker_thread+0x50/0x3a0 ? __pfx_worker_thread+0x10/0x10 kthread+0xdd/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x29/0x50 </TASK> Simplified timeout handler by creating local list of timed out WRs and invoke send handler post creating the list. The new method acquires/ releases lock once to fetch the list and hence helps to reduce locking contetiong when processing higher no. of WRs
05 / REFERENCES
Further evidence
- https://git.kernel.org/linus/2a777679b8ccd09a9a65ea0716ef10365179caac
- https://git.kernel.org/stable/c/2a777679b8ccd09a9a65ea0716ef10365179caac
- https://git.kernel.org/stable/c/3e799fa463508abe7a738ce5d0f62a8dfd05262a
- https://git.kernel.org/stable/c/7022a517bf1ca37ef5a474365bcc5eafd345a13a
- https://git.kernel.org/stable/c/713adaf0ecfc49405f6e5d9e409d984f628de818
- https://git.kernel.org/stable/c/a195a42dd25ca4f12489687065d00be64939409f
- https://git.kernel.org/stable/c/e80eadb3604a92d2d086e956b8b2692b699d4d0a
- https://ubuntu.com/security/CVE-2024-50095
- https://ubuntu.com/security/notices/USN-7166-1
- https://ubuntu.com/security/notices/USN-7166-2
- https://ubuntu.com/security/notices/USN-7166-3
- https://ubuntu.com/security/notices/USN-7166-4
- https://ubuntu.com/security/notices/USN-7186-1
- https://ubuntu.com/security/notices/USN-7186-2
- https://ubuntu.com/security/notices/USN-7194-1
- https://ubuntu.com/security/notices/USN-7276-1
- https://ubuntu.com/security/notices/USN-7277-1
- https://ubuntu.com/security/notices/USN-7310-1
- https://ubuntu.com/security/notices/USN-7383-1
- https://ubuntu.com/security/notices/USN-7383-2
- https://ubuntu.com/security/notices/USN-7384-1
- https://ubuntu.com/security/notices/USN-7384-2
- https://ubuntu.com/security/notices/USN-7385-1
- https://ubuntu.com/security/notices/USN-7386-1
- https://ubuntu.com/security/notices/USN-7403-1
- https://ubuntu.com/security/notices/USN-7451-1
- https://ubuntu.com/security/notices/USN-7468-1
- https://ubuntu.com/security/notices/USN-7523-1
- https://ubuntu.com/security/notices/USN-7524-1
- https://ubuntu.com/security/notices/USN-7907-1
- https://ubuntu.com/security/notices/USN-7907-2
- https://ubuntu.com/security/notices/USN-7907-3
- https://ubuntu.com/security/notices/USN-7907-4
- https://ubuntu.com/security/notices/USN-7907-5
- https://ubuntu.com/security/notices/USN-7922-1
- https://ubuntu.com/security/notices/USN-7922-2
- https://ubuntu.com/security/notices/USN-7922-3
- https://ubuntu.com/security/notices/USN-7922-4
- https://ubuntu.com/security/notices/USN-7922-5
- https://ubuntu.com/security/notices/USN-7930-1
- https://ubuntu.com/security/notices/USN-7930-2
- https://ubuntu.com/security/notices/USN-7937-1
- https://ubuntu.com/security/notices/USN-7939-1
- https://ubuntu.com/security/notices/USN-7939-2
- https://www.cve.org/CVERecord?id=CVE-2024-50095