Command: /usr/local/sbin/syslog-digest.py Time : 0.7319s Status : exited 0 Output : Syslog digest — 2026-07-17 03:05 EDT Processed 4,863 lines, filtered 2,182 (44%). 2,681 unknown (896 distinct) from 62 host(s). == ai-course (4 distinct, 6 total) == Jul 17 02:28:14 ai-course kernel: INFO: task jbd2/dm-0-8:314 blocked for more than 122 seconds. Jul 17 02:30:04 ai-course kernel: Not tainted 6.8.0-124-generic #124-Ubuntu [x2] Jul 17 02:30:04 ai-course kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x2] Jul 17 02:30:17 ai-course kernel: INFO: task jbd2/dm-0-8:314 blocked for more than 245 seconds. == alecten (7 distinct, 15 total) == Jul 17 02:27:29 alecten kernel: INFO: task kworker/0:0:3047574 blocked for more than 122 seconds. Jul 17 02:27:29 alecten kernel: Not tainted 6.12.0-124.43.1.el10_1.x86_64 #1 [x5] Jul 17 02:27:29 alecten kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x5] Jul 17 02:27:29 alecten kernel: INFO: task kworker/u16:1:3048131 blocked for more than 122 seconds. Jul 17 02:29:32 alecten kernel: INFO: task auditd:861 blocked for more than 122 seconds. Jul 17 02:29:32 alecten kernel: INFO: task kworker/0:0:3047574 blocked for more than 245 seconds. Jul 17 02:29:32 alecten kernel: INFO: task kworker/u16:1:3048131 blocked for more than 245 seconds. == b0b02.cs.umd.edu (4 distinct, 5 total) == Jul 17 02:29:16 b0b02.cs.umd.edu setroubleshoot[3902182]: SELinux is preventing /usr/bin/python3.9 from getattr access on the file /etc/authselect/nsswitch.conf. For complete SELinux messages run: sealert -l 9a1e2952-3fde-4937-a8ab-f7f59791a458 [x2] Jul 17 02:29:18 b0b02.cs.umd.edu setroubleshoot[3902182]: SELinux is preventing /usr/bin/python3.9 from read access on the file /etc/authselect/nsswitch.conf. For complete SELinux messages run: sealert -l 97d74423-e027-4cd6-8810-ad80944900b5 Jul 17 02:29:21 b0b02.cs.umd.edu setroubleshoot[3902182]: SELinux is preventing /usr/bin/python3.9 from open access on the file /etc/authselect/nsswitch.conf. For complete SELinux messages run: sealert -l 4f253f6a-62d6-46dd-8eaf-0b4d29df41c6 Jul 17 03:04:20 b0b02.cs.umd.edu rustdesk[3910588]: cannot open display: == biggulp.cs.umd.edu (8 distinct, 21 total) == Jul 17 02:24:45 biggulp.cs.umd.edu kernel: INFO: task xfsaild/dm-0:603 blocked for more than 122 seconds. Jul 17 02:24:45 biggulp.cs.umd.edu kernel: Not tainted 5.14.0-687.22.1.el9_8.x86_64 #1 [x7] Jul 17 02:24:45 biggulp.cs.umd.edu kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x7] Jul 17 02:24:45 biggulp.cs.umd.edu kernel: INFO: task kworker/3:0:2295150 blocked for more than 122 seconds. Jul 17 02:26:48 biggulp.cs.umd.edu kernel: INFO: task xfsaild/dm-0:603 blocked for more than 245 seconds. Jul 17 02:26:48 biggulp.cs.umd.edu kernel: INFO: task kworker/3:0:2295150 blocked for more than 245 seconds. Jul 17 02:26:48 biggulp.cs.umd.edu kernel: INFO: task kworker/u32:5:2309292 blocked for more than 122 seconds. [x2] Jul 17 02:30:54 biggulp.cs.umd.edu kernel: INFO: task ir_agent:3554 blocked for more than 122 seconds. == borogove.cs.umd.edu (2 distinct, 4 total) == Jul 17 02:20:52 borogove.cs.umd.edu kernel: libceph: mon2 (1)10.73.235.202:6789 socket closed (con state OPEN) [x2] Jul 17 02:21:02 borogove.cs.umd.edu kernel: libceph: mon0 (1)10.73.235.200:6789 socket closed (con state OPEN) [x2] == contest (9 distinct, 42 total) == Jul 17 02:05:29 contest goferd[994]: [ERROR][worker-0] gofer.messaging.adapter.connect:33 - connect: proton+amqps://satellite8.cs.umd.edu:5647, failed: Connection amqps://satellite8.cs.umd.edu:5647 disconnected: Condition('proton.pythonio', 'Connection refused to all addresses') [x34] Jul 17 02:19:34 contest setroubleshoot[591425]: SELinux is preventing /usr/libexec/platform-python3.6 from create access on the directory libdnf.vGuuwr. For complete SELinux messages run: sealert -l cb6b25c6-4765-4cf2-88a0-9b3b508e94e9 Jul 17 02:19:34 contest setroubleshoot[591425]: SELinux is preventing /usr/bin/gpgconf from add_name access on the directory repodata. For complete SELinux messages run: sealert -l 41cf7b1b-0f34-4609-98f2-b66e7825fef7 Jul 17 02:19:34 contest setroubleshoot[591425]: SELinux is preventing /usr/libexec/platform-python3.6 from create access on the file mirrorlist. For complete SELinux messages run: sealert -l fc20ff92-ea83-4f38-a24b-ddfd737a3141 Jul 17 02:19:35 contest setroubleshoot[591425]: SELinux is preventing /usr/libexec/platform-python3.6 from setattr access on the file repomd.xml. For complete SELinux messages run: sealert -l ac675631-fa35-4e93-adc9-b692b18e53e5 Jul 17 02:19:35 contest setroubleshoot[591425]: SELinux is preventing /usr/libexec/platform-python3.6 from read access on the directory libdnf.vGuuwr. For complete SELinux messages run: sealert -l 45d103b6-0810-4890-a705-990118577ce8 Jul 17 02:19:39 contest setroubleshoot[591425]: SELinux is preventing /usr/libexec/platform-python3.6 from remove_name access on the directory repomd.xml. For complete SELinux messages run: sealert -l 7e499113-2fd7-4f67-bec6-3521efbe3637 Jul 17 02:19:42 contest setroubleshoot[591425]: SELinux is preventing rhsmcertd-worke from unlink access on the file repomd.xml. For complete SELinux messages run: sealert -l 22835727-7fef-4cfb-b3b5-881f66888317 Jul 17 02:19:55 contest setroubleshoot[591483]: SELinux is preventing /usr/libexec/platform-python3.6 from rmdir access on the directory repodata. For complete SELinux messages run: sealert -l 3928930b-a1ed-448d-950b-0665ac2adef8 == csfw00a (1 distinct, 250 total) == Jul 17 02:06:55 csfw00a kernel: bnxt_en 0000:09:00.0 eth3: NIC Link is Down [x250] == cskayako (10 distinct, 28 total) == Jul 17 02:27:28 cskayako kernel: Not tainted 5.14.0-611.47.1.el9_7.x86_64 #1 [x10] Jul 17 02:27:28 cskayako kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x10] Jul 17 02:29:30 cskayako kernel: INFO: task xfsaild/dm-0:545 blocked for more than 122 seconds. Jul 17 02:29:30 cskayako kernel: INFO: task ir_agent:1133 blocked for more than 122 seconds. Jul 17 02:29:30 cskayako kernel: INFO: task choria:1072 blocked for more than 122 seconds. Jul 17 02:29:30 cskayako kernel: INFO: task kworker/u24:0:3374229 blocked for more than 122 seconds. Jul 17 02:29:30 cskayako kernel: INFO: task kworker/u24:2:3379421 blocked for more than 122 seconds. Jul 17 02:29:30 cskayako kernel: INFO: task kworker/1:0:3382300 blocked for more than 122 seconds. Jul 17 02:29:30 cskayako kernel: INFO: task kworker/u24:4:3382818 blocked for more than 122 seconds. Jul 17 02:29:30 cskayako kernel: INFO: task kworker/u24:5:3382873 blocked for more than 122 seconds. == florence-ruohan (16 distinct, 118 total) == Jul 17 03:04:00 florence-ruohan nologin[2096282]: Attempted login by UNKNOWN (UID: 65534) on UNKNOWN Jul 17 03:04:00 florence-ruohan pipewire[2096273]: mod.rt: RTKit error: org.freedesktop.DBus.Error.AccessDenied [x2] Jul 17 03:04:00 florence-ruohan pipewire[2096273]: mod.rt: could not set nice-level to -11: Permission denied Jul 17 03:04:00 florence-ruohan kernel: kauditd_printk_skb: 4 callbacks suppressed Jul 17 03:04:00 florence-ruohan pipewire-media-session[2096274]: mod.rt: RTKit error: org.freedesktop.DBus.Error.AccessDenied Jul 17 03:04:00 florence-ruohan pipewire-media-session[2096274]: mod.rt: could not make thread 2096334 realtime using RTKit: Permission denied Jul 17 03:04:00 florence-ruohan pipewire[2096273]: mod.rt: could not make thread 2096336 realtime using RTKit: Permission denied Jul 17 03:04:00 florence-ruohan pulseaudio[2096275]: Failed to create secure directory (/nonexistent/.config/pulse): No such file or directory [x5] Jul 17 03:04:09 florence-ruohan puppet-agent[2096283]: (Puppet::Type::Firewall::ProviderIptables) Skipping unparsable iptables rule: keys (3) and values (8) count mismatch on line: -A PREROUTING -m conntrack --ctstate RELATED,ESTABLISHED -j CONNMARK --restore-mark --nfmask 0xff0000 --ctmask 0xff0000 [x46] Jul 17 03:04:09 florence-ruohan puppet-agent[2096283]: (Puppet::Type::Firewall::ProviderIptables) Skipping unparsable iptables rule: keys (4) and values (9) count mismatch on line: -A OUTPUT -m conntrack --ctstate NEW -m mark ! --mark 0x0/0xff0000 -j CONNMARK --save-mark --nfmask 0xff0000 --ctmask 0xff0000 [x46] Jul 17 03:04:09 florence-ruohan puppet-agent[2096283]: (Puppet::Type::Firewall::ProviderIp6tables) Skipping unparsable iptables rule: keys (3) and values (8) count mismatch on line: -A PREROUTING -m conntrack --ctstate RELATED,ESTABLISHED -j CONNMARK --restore-mark --nfmask 0xff0000 --ctmask 0xff0000 [x4] Jul 17 03:04:09 florence-ruohan puppet-agent[2096283]: (Puppet::Type::Firewall::ProviderIp6tables) Skipping unparsable iptables rule: keys (4) and values (9) count mismatch on line: -A OUTPUT -m conntrack --ctstate NEW -m mark ! --mark 0x0/0xff0000 -j CONNMARK --save-mark --nfmask 0xff0000 --ctmask 0xff0000 [x4] Jul 17 03:04:14 florence-ruohan puppet-agent[2096283]: Could not set 'file' on ensure: No such file or directory - A directory component in /etc/krb5.conf.d/umd_kdc.conf20260717-2096283-1aoex5q.lock does not exist or is a dangling symbolic link (file: /etc/puppetlabs/code/environments/production/site/profile/manifests/base/ditkeytab.pp, line: 155) [x2] Jul 17 03:04:14 florence-ruohan puppet-agent[2096283]: Wrapped exception: Jul 17 03:04:14 florence-ruohan puppet-agent[2096283]: No such file or directory - A directory component in /etc/krb5.conf.d/umd_kdc.conf20260717-2096283-1aoex5q.lock does not exist or is a dangling symbolic link Jul 17 03:04:14 florence-ruohan puppet-agent[2096283]: (/Stage[main]/Profile::Base::Ditkeytab/File[/etc/krb5.conf.d/umd_kdc.conf]/ensure) change from 'absent' to 'file' failed: Could not set 'file' on ensure: No such file or directory - A directory component in /etc/krb5.conf.d/umd_kdc.conf20260717-2096283-1aoex5q.lock does not exist or is a dangling symbolic link (file: /etc/puppetlabs/code/environments/production/site/profile/manifests/base/ditkeytab.pp, line: 155) == forge00 (7 distinct, 18 total) == Jul 17 02:16:24 forge00 kernel: INFO: task xfsaild/dm-0:793 blocked for more than 122 seconds. [x2] Jul 17 02:16:24 forge00 kernel: Not tainted 6.12.0-211.20.1.el10_2.x86_64 #1 [x6] Jul 17 02:16:24 forge00 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x6] Jul 17 02:16:24 forge00 kernel: INFO: task forgejo:3198303 blocked for more than 122 seconds. Jul 17 02:16:24 forge00 kernel: INFO: task kworker/1:2:3370684 blocked for more than 122 seconds. Jul 17 02:24:35 forge00 kernel: INFO: task forgejo:2532 blocked for more than 122 seconds. Jul 17 02:24:35 forge00 kernel: INFO: task kworker/2:0:3367327 blocked for more than 122 seconds. == frad00 (5 distinct, 9 total) == Jul 17 02:28:24 frad00 kernel: INFO: task auditd:792 blocked for more than 122 seconds. Jul 17 02:28:24 frad00 kernel: Not tainted 5.14.0-687.10.1.el9_8.x86_64 #1 [x3] Jul 17 02:28:24 frad00 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x3] Jul 17 02:28:24 frad00 kernel: INFO: task rpm:3099960 blocked for more than 122 seconds. Jul 17 02:30:27 frad00 kernel: INFO: task auditd:792 blocked for more than 245 seconds. == gle-dind01 (4 distinct, 6 total) == Jul 17 02:30:04 gle-dind01 kernel: INFO: task xfsaild/dm-0:634 blocked for more than 122 seconds. Jul 17 02:30:04 gle-dind01 kernel: Not tainted 6.12.0-211.7.3.el10_2.x86_64 #1 [x2] Jul 17 02:30:04 gle-dind01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x2] Jul 17 02:30:04 gle-dind01 kernel: INFO: task kworker/u16:0:3890769 blocked for more than 122 seconds. == gle-omnibus00.cs.umd.edu (12 distinct, 30 total) == Jul 17 02:26:37 gle-omnibus00.cs.umd.edu kernel: INFO: task alertmanager:2734 blocked for more than 122 seconds. Jul 17 02:26:37 gle-omnibus00.cs.umd.edu kernel: Not tainted 5.14.0-687.12.1.el9_8.x86_64 #1 [x10] Jul 17 02:26:37 gle-omnibus00.cs.umd.edu kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x10] Jul 17 02:26:37 gle-omnibus00.cs.umd.edu kernel: INFO: task alertmanager:2823 blocked for more than 122 seconds. Jul 17 02:26:37 gle-omnibus00.cs.umd.edu kernel: INFO: task gitaly:2847 blocked for more than 122 seconds. Jul 17 02:27:09 gle-omnibus00.cs.umd.edu kernel: INFO: task gitaly:459335 blocked for more than 122 seconds. Jul 17 02:27:09 gle-omnibus00.cs.umd.edu kernel: INFO: task kworker/14:1:2960212 blocked for more than 122 seconds. Jul 17 02:27:09 gle-omnibus00.cs.umd.edu kernel: INFO: task kworker/u66:7:2977593 blocked for more than 122 seconds. Jul 17 02:27:09 gle-omnibus00.cs.umd.edu kernel: INFO: task git:2979008 blocked for more than 122 seconds. Jul 17 02:28:40 gle-omnibus00.cs.umd.edu kernel: INFO: task alertmanager:2734 blocked for more than 245 seconds. Jul 17 02:28:53 gle-omnibus00.cs.umd.edu kernel: INFO: task alertmanager:2823 blocked for more than 245 seconds. Jul 17 02:28:53 gle-omnibus00.cs.umd.edu kernel: INFO: task gitaly:2847 blocked for more than 245 seconds. == grafana (8 distinct, 18 total) == Jul 17 02:27:45 grafana kernel: INFO: task xfsaild/dm-0:590 blocked for more than 122 seconds. Jul 17 02:27:45 grafana kernel: Not tainted 5.14.0-611.47.1.el9_7.x86_64 #1 [x6] Jul 17 02:27:45 grafana kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x6] Jul 17 02:27:45 grafana kernel: INFO: task kworker/u18:0:1016387 blocked for more than 122 seconds. Jul 17 02:27:45 grafana kernel: INFO: task kworker/1:2:1017037 blocked for more than 122 seconds. Jul 17 02:29:48 grafana kernel: INFO: task xfsaild/dm-0:590 blocked for more than 245 seconds. Jul 17 02:29:48 grafana kernel: INFO: task kworker/u18:0:1016387 blocked for more than 245 seconds. Jul 17 02:29:48 grafana kernel: INFO: task kworker/1:2:1017037 blocked for more than 245 seconds. == hasaraumd (4 distinct, 9 total) == Jul 17 02:24:25 hasaraumd kernel: INFO: task jbd2/dm-0-8:378 blocked for more than 122 seconds. [x2] Jul 17 02:24:25 hasaraumd kernel: Not tainted 6.8.0-100-generic #100-Ubuntu [x3] Jul 17 02:24:25 hasaraumd kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x3] Jul 17 02:26:28 hasaraumd kernel: INFO: task jbd2/dm-0-8:378 blocked for more than 245 seconds. == hazfs.cs.umd.edu (1 distinct, 1 total) == Jul 17 02:52:31 hazfs.cs.umd.edu zrepl[2444237]: [hourlies_push][pruning][vUfd$LDmZ$LDmZ.WVPf.seOH.HZl7]: target could not destroy snapshots prune_side="receiver" err="destroys failed @zrepl_20260717_063957_000: dataset is busy" == idm00 (8 distinct, 19 total) == Jul 17 02:16:34 idm00 kernel: INFO: task xfsaild/dm-0:624 blocked for more than 122 seconds. [x2] Jul 17 02:16:34 idm00 kernel: Not tainted 5.14.0-570.17.1.el9_6.x86_64 #1 [x6] Jul 17 02:16:34 idm00 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x6] Jul 17 02:16:34 idm00 kernel: INFO: task ns-slapd:2284 blocked for more than 122 seconds. Jul 17 02:16:34 idm00 kernel: INFO: task ns-slapd:2286 blocked for more than 122 seconds. Jul 17 02:16:34 idm00 kernel: INFO: task kworker/u16:5:1201706 blocked for more than 122 seconds. Jul 17 02:24:46 idm00 kernel: INFO: task ns-slapd:2253 blocked for more than 122 seconds. Jul 17 02:50:57 idm00 named[1455]: client @0x7f3310381948 10.73.240.42#57287: update '73.10.in-addr.arpa/IN' denied == idmh1 (200 distinct, 457 total) == Jul 17 02:06:58 idmh1 systemd[1]: check-ldap-alive.service: Watchdog timeout (limit 5s)! [x55] Jul 17 02:06:58 idmh1 systemd-coredump[1014285]: Process 1014259 (ldapsearch) of user 0 dumped core.#012#012Stack trace of thread 1014259:#012#0 0x00007fdac2701ee7 __poll (libc.so.6 + 0x101ee7)#012#1 0x00007fdac2a4bacd ldap_result (libldap.so.2 + 0x1dacd)#012#2 0x00005631e9d3024c tool_bind (ldapsearch + 0xf24c)#012#3 0x00005631e9d2693b main (ldapsearch + 0x593b)#012#4 0x00007fdac26295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#5 0x00007fdac2629680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#6 0x00005631e9d28db5 _start (ldapsearch + 0x7db5)#012ELF object binary architecture: AMD x86-64 Jul 17 02:07:03 idmh1 systemd[1]: check-ldap-alive.service: State 'stop-watchdog' timed out. Killing. [x53] Jul 17 02:07:03 idmh1 systemd[1]: check-ldap-alive.service: Main process exited, code=killed, status=9/KILL [x53] Jul 17 02:07:04 idmh1 systemd[1]: check-http-alive.service: Watchdog timeout (limit 5s)! [x20] Jul 17 02:07:09 idmh1 systemd[1]: check-http-alive.service: State 'stop-watchdog' timed out. Killing. [x18] Jul 17 02:07:09 idmh1 systemd[1]: check-http-alive.service: Main process exited, code=killed, status=9/KILL [x18] Jul 17 02:07:15 idmh1 systemd-coredump[1014355]: Process 1014305 (curl) of user 0 dumped core.#012#012Stack trace of thread 1014305:#012#0 0x00007fa7fae5ae9b rename (libc.so.6 + 0x5ae9b)#012#1 0x00007fa7fb2346c6 Curl_flush_cookies (libcurl.so.4 + 0x1d6c6)#012#2 0x00007fa7fb2968c3 Curl_close.isra.0 (libcurl.so.4 + 0x7f8c3)#012#3 0x00007fa7fb23662c curl_easy_cleanup (libcurl.so.4 + 0x1f62c)#012#4 0x000056478527c6cb post_per_transfer (curl + 0x106cb)#012#5 0x00005647852788ca main (curl + 0xc8ca)#012#6 0x00007fa7fae295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#7 0x00007fa7fae29680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#8 0x0000564785278e35 _start (curl + 0xce35)#012ELF object binary architecture: AMD x86-64 Jul 17 02:07:15 idmh1 systemd-coredump[1014688]: Process 1012314 (check-http-aliv) of user 0 dumped core.#012#012Stack trace of thread 1012314:#012#0 0x00007f04aaa3ee2b kill (libc.so.6 + 0x3ee2b)#012#1 0x000055c8cc0dc930 termsig_handler.part.0 (bash + 0x88930)#012#2 0x000055c8cc15594b waitchld.constprop.0 (bash + 0x10194b)#012#3 0x000055c8cc0b9a7a wait_for (bash + 0x65a7a)#012#4 0x000055c8cc0a1ee8 execute_command_internal (bash + 0x4dee8)#012#5 0x000055c8cc0a2748 execute_command (bash + 0x4e748)#012#6 0x000055c8cc0a12f0 execute_command_internal (bash + 0x4d2f0)#012#7 0x000055c8cc0a13cb execute_command_internal (bash + 0x4d3cb)#012#8 0x000055c8cc0a2748 execute_command (bash + 0x4e748)#012#9 0x000055c8cc0a138e execute_command_internal (bash + 0x4d38e)#012#10 0x000055c8cc0a2748 execute_command (bash + 0x4e748)#012#11 0x000055c8cc0a138e execute_command_internal (bash + 0x4d38e)#012#12 0x000055c8cc0a2748 execute_command (bash + 0x4e748)#012#13 0x000055c8cc0a138e execute_command_ internal (bash + 0x4d38e)#012#14 0x000055c8cc0a2748 execute_command (bash + 0x4e748)#012#15 0x000055c8cc0a28f0 execute_while_or_until (bash + 0x4e8f0)#012#16 0x000055c8cc0a01c0 execute_command_internal (bash + 0x4c1c0)#012#17 0x000055c8cc0a2748 execute_command (bash + 0x4e748)#012#18 0x000055c8cc094109 reader_loop (bash + 0x40109)#012#19 0x000055c8cc08597e main (bash + 0x3197e)#012#20 0x00007f04aaa295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#21 0x00007f04aaa29680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#22 0x000055c8cc085d75 _start (bash + 0x31d75)#012ELF object binary architecture: AMD x86-64 Jul 17 02:07:15 idmh1 systemd-coredump[1015231]: Process 1014332 (systemctl) of user 0 dumped core.#012#012Stack trace of thread 1014332:#012#0 0x00007f9cf7b01fbf ppoll (libc.so.6 + 0x101fbf)#012#1 0x00007f9cf7fc481e ppoll_usec (libsystemd-shared-252.so + 0x1c481e)#012#2 0x00007f9cf802ea79 bus_poll (libsystemd-shared-252.so + 0x22ea79)#012#3 0x00007f9cf80302dd sd_bus_wait (libsystemd-shared-252.so + 0x2302dd)#012#4 0x00007f9cf7eb6881 bus_wait_for_jobs (libsystemd-shared-252.so + 0xb6881)#012#5 0x0000556be3fd3825 verb_start (systemctl + 0x30825)#012#6 0x0000556be3fb3cde main (systemctl + 0x10cde)#012#7 0x00007f9cf7a295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#8 0x00007f9cf7a29680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#9 0x0000556be3fb5565 _start (systemctl + 0x12565)#012ELF object binary architecture: AMD x86-64 Jul 17 02:07:21 idmh1 systemd-coredump[1015332]: Process 1015302 (systemctl) of user 0 dumped core.#012#012Stack trace of thread 1015302:#012#0 0x00007ff555701fbf ppoll (libc.so.6 + 0x101fbf)#012#1 0x00007ff555bc481e ppoll_usec (libsystemd-shared-252.so + 0x1c481e)#012#2 0x00007ff555c2ea79 bus_poll (libsystemd-shared-252.so + 0x22ea79)#012#3 0x00007ff555c302dd sd_bus_wait (libsystemd-shared-252.so + 0x2302dd)#012#4 0x00007ff555ab6881 bus_wait_for_jobs (libsystemd-shared-252.so + 0xb6881)#012#5 0x0000557c081f7825 verb_start (systemctl + 0x30825)#012#6 0x0000557c081d7cde main (systemctl + 0x10cde)#012#7 0x00007ff5556295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#8 0x00007ff555629680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#9 0x0000557c081d9565 _start (systemctl + 0x12565)#012ELF object binary architecture: AMD x86-64 Jul 17 02:07:39 idmh1 systemd-coredump[1015423]: Process 1015386 (systemctl) of user 0 dumped core.#012#012Stack trace of thread 1015386:#012#0 0x00007f3397b01fbf ppoll (libc.so.6 + 0x101fbf)#012#1 0x00007f3397fc481e ppoll_usec (libsystemd-shared-252.so + 0x1c481e)#012#2 0x00007f339802ea79 bus_poll (libsystemd-shared-252.so + 0x22ea79)#012#3 0x00007f33980302dd sd_bus_wait (libsystemd-shared-252.so + 0x2302dd)#012#4 0x00007f3397eb6881 bus_wait_for_jobs (libsystemd-shared-252.so + 0xb6881)#012#5 0x0000561ecb26c825 verb_start (systemctl + 0x30825)#012#6 0x0000561ecb24ccde main (systemctl + 0x10cde)#012#7 0x00007f3397a295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#8 0x00007f3397a29680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#9 0x0000561ecb24e565 _start (systemctl + 0x12565)#012ELF object binary architecture: AMD x86-64 Jul 17 02:07:45 idmh1 systemd-coredump[1015516]: Process 1015466 (systemctl) of user 0 dumped core.#012#012Stack trace of thread 1015466:#012#0 0x00007f0258d01fbf ppoll (libc.so.6 + 0x101fbf)#012#1 0x00007f02591c481e ppoll_usec (libsystemd-shared-252.so + 0x1c481e)#012#2 0x00007f025922ea79 bus_poll (libsystemd-shared-252.so + 0x22ea79)#012#3 0x00007f02592302dd sd_bus_wait (libsystemd-shared-252.so + 0x2302dd)#012#4 0x00007f02590b6881 bus_wait_for_jobs (libsystemd-shared-252.so + 0xb6881)#012#5 0x0000559969278825 verb_start (systemctl + 0x30825)#012#6 0x0000559969258cde main (systemctl + 0x10cde)#012#7 0x00007f0258c295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#8 0x00007f0258c29680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#9 0x000055996925a565 _start (systemctl + 0x12565)#012ELF object binary architecture: AMD x86-64 Jul 17 02:07:46 idmh1 systemd-coredump[1015575]: Process 1015461 (check-ldap-aliv) of user 0 dumped core.#012#012Stack trace of thread 1015461:#012#0 0x00007f5521a3ee2b kill (libc.so.6 + 0x3ee2b)#012#1 0x000055cd965a6930 termsig_handler.part.0 (bash + 0x88930)#012#2 0x000055cd9661f94b waitchld.constprop.0 (bash + 0x10194b)#012#3 0x000055cd96583a7a wait_for (bash + 0x65a7a)#012#4 0x000055cd9656bee8 execute_command_internal (bash + 0x4dee8)#012#5 0x000055cd9656b3cb execute_command_internal (bash + 0x4d3cb)#012#6 0x000055cd9656c748 execute_command (bash + 0x4e748)#012#7 0x000055cd9656acdf execute_command_internal (bash + 0x4ccdf)#012#8 0x000055cd9656b3cb execute_command_internal (bash + 0x4d3cb)#012#9 0x000055cd9656c748 execute_command (bash + 0x4e748)#012#10 0x000055cd9656b38e execute_command_internal (bash + 0x4d38e)#012#11 0x000055cd9656c748 execute_command (bash + 0x4e748)#012#12 0x000055cd9656b38e execute_command_internal (bash + 0x4d38e)#012#13 0x000055cd9656c748 execute _command (bash + 0x4e748)#012#14 0x000055cd9656c8f0 execute_while_or_until (bash + 0x4e8f0)#012#15 0x000055cd9656a1c0 execute_command_internal (bash + 0x4c1c0)#012#16 0x000055cd9656c748 execute_command (bash + 0x4e748)#012#17 0x000055cd9655e109 reader_loop (bash + 0x40109)#012#18 0x000055cd9654f97e main (bash + 0x3197e)#012#19 0x00007f5521a295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#20 0x00007f5521a29680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#21 0x000055cd9654fd75 _start (bash + 0x31d75)#012ELF object binary architecture: AMD x86-64 Jul 17 02:07:46 idmh1 systemd[1]: check-ldap-alive.service: Main process exited, code=dumped, status=6/ABRT [x2] Jul 17 02:08:23 idmh1 systemd-coredump[1015863]: Process 1015828 (ldapsearch) of user 0 dumped core.#012#012Stack trace of thread 1015828:#012#0 0x00007f146f301ee7 __poll (libc.so.6 + 0x101ee7)#012#1 0x00007f146f587acd ldap_result (libldap.so.2 + 0x1dacd)#012#2 0x00005582b18b49ef dosearch.constprop.0 (ldapsearch + 0xf9ef)#012#3 0x00005582b18abf6d main (ldapsearch + 0x6f6d)#012#4 0x00007f146f2295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#5 0x00007f146f229680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#6 0x00005582b18acdb5 _start (ldapsearch + 0x7db5)#012ELF object binary architecture: AMD x86-64 Jul 17 02:08:35 idmh1 systemd-coredump[1015934]: Process 1015909 (systemctl) of user 0 dumped core.#012#012Stack trace of thread 1015909:#012#0 0x00007f834bb01fbf ppoll (libc.so.6 + 0x101fbf)#012#1 0x00007f834bfc481e ppoll_usec (libsystemd-shared-252.so + 0x1c481e)#012#2 0x00007f834c02ea79 bus_poll (libsystemd-shared-252.so + 0x22ea79)#012#3 0x00007f834c0302dd sd_bus_wait (libsystemd-shared-252.so + 0x2302dd)#012#4 0x00007f834beb6881 bus_wait_for_jobs (libsystemd-shared-252.so + 0xb6881)#012#5 0x000055db50276825 verb_start (systemctl + 0x30825)#012#6 0x000055db50256cde main (systemctl + 0x10cde)#012#7 0x00007f834ba295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#8 0x00007f834ba29680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#9 0x000055db50258565 _start (systemctl + 0x12565)#012ELF object binary architecture: AMD x86-64 Jul 17 02:09:17 idmh1 systemd-coredump[1016303]: Process 1016280 (ldapsearch) of user 0 dumped core.#012#012Stack trace of thread 1016280:#012#0 0x00007fb999701ee7 __poll (libc.so.6 + 0x101ee7)#012#1 0x00007fb999a06acd ldap_result (libldap.so.2 + 0x1dacd)#012#2 0x0000563d2fbce9ef dosearch.constprop.0 (ldapsearch + 0xf9ef)#012#3 0x0000563d2fbc5f6d main (ldapsearch + 0x6f6d)#012#4 0x00007fb9996295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#5 0x00007fb999629680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#6 0x0000563d2fbc6db5 _start (ldapsearch + 0x7db5)#012ELF object binary architecture: AMD x86-64 Jul 17 02:09:21 idmh1 systemd-coredump[1016386]: Process 1016347 (systemctl) of user 0 dumped core.#012#012Stack trace of thread 1016347:#012#0 0x00007fbf74901fbf ppoll (libc.so.6 + 0x101fbf)#012#1 0x00007fbf74dc481e ppoll_usec (libsystemd-shared-252.so + 0x1c481e)#012#2 0x00007fbf74e2ea79 bus_poll (libsystemd-shared-252.so + 0x22ea79)#012#3 0x00007fbf74e302dd sd_bus_wait (libsystemd-shared-252.so + 0x2302dd)#012#4 0x00007fbf74cb6881 bus_wait_for_jobs (libsystemd-shared-252.so + 0xb6881)#012#5 0x00005607c15f0825 verb_start (systemctl + 0x30825)#012#6 0x00005607c15d0cde main (systemctl + 0x10cde)#012#7 0x00007fbf748295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#8 0x00007fbf74829680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#9 0x00005607c15d2565 _start (systemctl + 0x12565)#012ELF object binary architecture: AMD x86-64 Jul 17 02:09:25 idmh1 systemd-coredump[1016462]: Process 1016342 (check-ldap-aliv) of user 0 dumped core.#012#012Stack trace of thread 1016342:#012#0 0x00007fc845a3ee2b kill (libc.so.6 + 0x3ee2b)#012#1 0x000055e12bcac930 termsig_handler.part.0 (bash + 0x88930)#012#2 0x000055e12bd2594b waitchld.constprop.0 (bash + 0x10194b)#012#3 0x000055e12bc89a7a wait_for (bash + 0x65a7a)#012#4 0x000055e12bc71ee8 execute_command_internal (bash + 0x4dee8)#012#5 0x000055e12bc713cb execute_command_internal (bash + 0x4d3cb)#012#6 0x000055e12bc72748 execute_command (bash + 0x4e748)#012#7 0x000055e12bc70cdf execute_command_internal (bash + 0x4ccdf)#012#8 0x000055e12bc713cb execute_command_internal (bash + 0x4d3cb)#012#9 0x000055e12bc72748 execute_command (bash + 0x4e748)#012#10 0x000055e12bc7138e execute_command_internal (bash + 0x4d38e)#012#11 0x000055e12bc72748 execute_command (bash + 0x4e748)#012#12 0x000055e12bc7138e execute_command_internal (bash + 0x4d38e)#012#13 0x000055e12bc72748 execute _command (bash + 0x4e748)#012#14 0x000055e12bc728f0 execute_while_or_until (bash + 0x4e8f0)#012#15 0x000055e12bc701c0 execute_command_internal (bash + 0x4c1c0)#012#16 0x000055e12bc72748 execute_command (bash + 0x4e748)#012#17 0x000055e12bc64109 reader_loop (bash + 0x40109)#012#18 0x000055e12bc5597e main (bash + 0x3197e)#012#19 0x00007fc845a295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#20 0x00007fc845a29680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#21 0x000055e12bc55d75 _start (bash + 0x31d75)#012ELF object binary architecture: AMD x86-64 Jul 17 02:11:36 idmh1 systemd-coredump[1017782]: Process 1017753 (ldapsearch) of user 0 dumped core.#012#012Stack trace of thread 1017753:#012#0 0x00007fef58501ee7 __poll (libc.so.6 + 0x101ee7)#012#1 0x00007fef58744acd ldap_result (libldap.so.2 + 0x1dacd)#012#2 0x0000565550ee224c tool_bind (ldapsearch + 0xf24c)#012#3 0x0000565550ed893b main (ldapsearch + 0x593b)#012#4 0x00007fef584295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#5 0x00007fef58429680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#6 0x0000565550edadb5 _start (ldapsearch + 0x7db5)#012ELF object binary architecture: AMD x86-64 Jul 17 02:11:49 idmh1 systemd-coredump[1017852]: Process 1017822 (systemctl) of user 0 dumped core.#012#012Stack trace of thread 1017822:#012#0 0x00007f9303f01fbf ppoll (libc.so.6 + 0x101fbf)#012#1 0x00007f93043c481e ppoll_usec (libsystemd-shared-252.so + 0x1c481e)#012#2 0x00007f930442ea79 bus_poll (libsystemd-shared-252.so + 0x22ea79)#012#3 0x00007f93044302dd sd_bus_wait (libsystemd-shared-252.so + 0x2302dd)#012#4 0x00007f93042b6881 bus_wait_for_jobs (libsystemd-shared-252.so + 0xb6881)#012#5 0x000055bcb7f9d825 verb_start (systemctl + 0x30825)#012#6 0x000055bcb7f7dcde main (systemctl + 0x10cde)#012#7 0x00007f9303e295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#8 0x00007f9303e29680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#9 0x000055bcb7f7f565 _start (systemctl + 0x12565)#012ELF object binary architecture: AMD x86-64 Jul 17 02:11:49 idmh1 sssd_be[1046]: LDAP connection error: unknown error Jul 17 02:12:00 idmh1 systemd-coredump[1017929]: Process 1017897 (systemctl) of user 0 dumped core.#012#012Stack trace of thread 1017897:#012#0 0x00007f9db3701fbf ppoll (libc.so.6 + 0x101fbf)#012#1 0x00007f9db3bc481e ppoll_usec (libsystemd-shared-252.so + 0x1c481e)#012#2 0x00007f9db3c2ea79 bus_poll (libsystemd-shared-252.so + 0x22ea79)#012#3 0x00007f9db3c302dd sd_bus_wait (libsystemd-shared-252.so + 0x2302dd)#012#4 0x00007f9db3ab6881 bus_wait_for_jobs (libsystemd-shared-252.so + 0xb6881)#012#5 0x00005645e02aa825 verb_start (systemctl + 0x30825)#012#6 0x00005645e028acde main (systemctl + 0x10cde)#012#7 0x00007f9db36295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#8 0x00007f9db3629680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#9 0x00005645e028c565 _start (systemctl + 0x12565)#012ELF object binary architecture: AMD x86-64 Jul 17 02:12:31 idmh1 systemd[1]: Cannot find unit for notify message of PID 1018972, ignoring. Jul 17 02:12:50 idmh1 systemd[1]: Cannot find unit for notify message of PID 1019021, ignoring. Jul 17 02:12:52 idmh1 systemd-coredump[1019033]: Process 1018991 (systemctl) of user 0 dumped core.#012#012Stack trace of thread 1018991:#012#0 0x00007fd694701fbf n/a (n/a + 0x0)#012ELF object binary architecture: AMD x86-64 Jul 17 02:12:52 idmh1 systemd-coredump[1018982]: Process 1018930 (systemctl) of user 0 dumped core.#012#012Stack trace of thread 1018930:#012#0 0x00007ffac7701fbf n/a (n/a + 0x0)#012ELF object binary architecture: AMD x86-64 Jul 17 02:12:52 idmh1 systemd-coredump[1017996]: Process 1017969 (curl) of user 0 dumped core.#012#012Stack trace of thread 1017969:#012#0 0x00007fc017a5ae9b rename (libc.so.6 + 0x5ae9b)#012#1 0x00007fc017c536c6 Curl_flush_cookies (libcurl.so.4 + 0x1d6c6)#012#2 0x00007fc017cb58c3 Curl_close.isra.0 (libcurl.so.4 + 0x7f8c3)#012#3 0x00007fc017c5562c curl_easy_cleanup (libcurl.so.4 + 0x1f62c)#012#4 0x0000556b22b946cb post_per_transfer (curl + 0x106cb)#012#5 0x0000556b22b908ca main (curl + 0xc8ca)#012#6 0x00007fc017a295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#7 0x00007fc017a29680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#8 0x0000556b22b90e35 _start (curl + 0xce35)#012ELF object binary architecture: AMD x86-64 Jul 17 02:12:52 idmh1 systemd-coredump[1018252]: Process 1015247 (check-http-aliv) of user 0 dumped core.#012#012Stack trace of thread 1015247:#012#0 0x00007f52f223ee2b kill (libc.so.6 + 0x3ee2b)#012#1 0x0000556cd15c1930 termsig_handler.part.0 (bash + 0x88930)#012#2 0x0000556cd163a94b waitchld.constprop.0 (bash + 0x10194b)#012#3 0x0000556cd159ea7a wait_for (bash + 0x65a7a)#012#4 0x0000556cd1586ee8 execute_command_internal (bash + 0x4dee8)#012#5 0x0000556cd1587748 execute_command (bash + 0x4e748)#012#6 0x0000556cd15862f0 execute_command_internal (bash + 0x4d2f0)#012#7 0x0000556cd15863cb execute_command_internal (bash + 0x4d3cb)#012#8 0x0000556cd1587748 execute_command (bash + 0x4e748)#012#9 0x0000556cd158638e execute_command_internal (bash + 0x4d38e)#012#10 0x0000556cd1587748 execute_command (bash + 0x4e748)#012#11 0x0000556cd158638e execute_command_internal (bash + 0x4d38e)#012#12 0x0000556cd1587748 execute_command (bash + 0x4e748)#012#13 0x0000556cd158638e execute_command_ internal (bash + 0x4d38e)#012#14 0x0000556cd1587748 execute_command (bash + 0x4e748)#012#15 0x0000556cd15878f0 execute_while_or_until (bash + 0x4e8f0)#012#16 0x0000556cd15851c0 execute_command_internal (bash + 0x4c1c0)#012#17 0x0000556cd1587748 execute_command (bash + 0x4e748)#012#18 0x0000556cd1579109 reader_loop (bash + 0x40109)#012#19 0x0000556cd156a97e main (bash + 0x3197e)#012#20 0x00007f52f22295d0 __libc_start_call_main (libc.so.6 + 0x295d0)#012#21 0x00007f52f2229680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680)#012#22 0x0000556cd156ad75 _start (bash + 0x31d75)#012ELF object binary architecture: AMD x86-64 ... 170 more distinct messages not shown == idmp0 (1 distinct, 1 total) == Jul 17 02:35:00 idmp0 named[5198]: client @0x7f9914012c48 10.73.240.43#61207: update '73.10.in-addr.arpa/IN' denied == idmv0 (6 distinct, 6 total) == Jul 17 02:15:42 idmv0 kernel: INFO: task xfsaild/dm-0:669 blocked for more than 122 seconds. Jul 17 02:15:42 idmv0 kernel: Not tainted 5.14.0-570.17.1.el9_6.x86_64 #1 Jul 17 02:15:42 idmv0 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 17 02:18:40 idmv0 systemd[1]: Cannot find unit for notify message of PID 1173897, ignoring. Jul 17 02:21:45 idmv0 systemd[1]: Cannot find unit for notify message of PID 1174575, ignoring. Jul 17 02:21:45 idmv0 systemd[1]: Cannot find unit for notify message of PID 1174584, ignoring. == inbound00.cs.umd.edu (3 distinct, 3 total) == Jul 17 02:24:22 inbound00.cs.umd.edu kernel: INFO: task xfsaild/dm-0:595 blocked for more than 122 seconds. Jul 17 02:24:22 inbound00.cs.umd.edu kernel: Not tainted 5.14.0-687.22.1.el9_8.x86_64 #1 Jul 17 02:24:22 inbound00.cs.umd.edu kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. == inbound01.cs.umd.edu (10 distinct, 22 total) == Jul 17 02:24:28 inbound01.cs.umd.edu kernel: INFO: task xfsaild/dm-0:594 blocked for more than 122 seconds. Jul 17 02:24:28 inbound01.cs.umd.edu kernel: Not tainted 5.14.0-611.47.1.el9_7.x86_64 #1 [x7] Jul 17 02:24:28 inbound01.cs.umd.edu kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x7] Jul 17 02:24:28 inbound01.cs.umd.edu kernel: INFO: task kworker/u16:3:4175356 blocked for more than 122 seconds. Jul 17 02:24:28 inbound01.cs.umd.edu kernel: INFO: task kworker/3:4:4178030 blocked for more than 122 seconds. Jul 17 02:24:28 inbound01.cs.umd.edu kernel: INFO: task cleanup:4178160 blocked for more than 122 seconds. Jul 17 02:26:31 inbound01.cs.umd.edu kernel: INFO: task xfsaild/dm-0:594 blocked for more than 245 seconds. Jul 17 02:26:31 inbound01.cs.umd.edu kernel: INFO: task kworker/u16:3:4175356 blocked for more than 245 seconds. Jul 17 02:26:31 inbound01.cs.umd.edu kernel: INFO: task kworker/3:4:4178030 blocked for more than 245 seconds. Jul 17 02:28:19 inbound01.cs.umd.edu sshd-session[4181026]: error: Protocol major versions differ: 2 vs. 1 == itmon01 (20 distinct, 59 total) == Jul 17 02:16:05 itmon01 kernel: INFO: task xfsaild/dm-1:529 blocked for more than 120 seconds. [x2] Jul 17 02:16:05 itmon01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x5] Jul 17 02:16:05 itmon01 kernel: Call Trace: [x5] Jul 17 02:16:05 itmon01 kernel: [<ffffffff925b7ca9>] schedule+0x29/0x70 [x5] Jul 17 02:16:05 itmon01 kernel: [<ffffffffc05ed687>] xfs_log_force+0x157/0x2e0 [xfs] [x2] Jul 17 02:16:05 itmon01 kernel: [<ffffffff91ee1170>] ? wake_up_state+0x20/0x20 [x5] Jul 17 02:16:05 itmon01 kernel: [<ffffffffc05f9e70>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs] [x4] Jul 17 02:16:05 itmon01 kernel: [<ffffffffc05fa000>] xfsaild+0x190/0x780 [xfs] [x2] Jul 17 02:16:05 itmon01 kernel: [<ffffffff91ecb621>] kthread+0xd1/0xe0 [x2] Jul 17 02:16:05 itmon01 kernel: [<ffffffff91ecb550>] ? insert_kthread_work+0x40/0x40 [x4] Jul 17 02:16:05 itmon01 kernel: [<ffffffff925c51f7>] ret_from_fork_nospec_begin+0x21/0x21 [x2] Jul 17 02:16:05 itmon01 kernel: INFO: task nagios:1856 blocked for more than 120 seconds. Jul 17 02:16:05 itmon01 kernel: [<ffffffffc05edcfc>] xfs_log_force_lsn+0x32c/0x370 [xfs] [x3] Jul 17 02:16:05 itmon01 kernel: [<ffffffffc05cd51d>] xfs_file_fsync+0xfd/0x1c0 [xfs] [x3] Jul 17 02:16:05 itmon01 kernel: [<ffffffff92093195>] do_fsync+0x55/0x80 [x3] Jul 17 02:16:05 itmon01 kernel: [<ffffffff91f46986>] ? __audit_syscall_exit+0x1f6/0x2b0 [x3] Jul 17 02:16:05 itmon01 kernel: [<ffffffff92093460>] SyS_fsync+0x10/0x20 [x3] Jul 17 02:16:05 itmon01 kernel: [<ffffffff925c539a>] system_call_fastpath+0x25/0x2a [x3] Jul 17 02:26:05 itmon01 kernel: INFO: task mysqld:2154 blocked for more than 120 seconds. Jul 17 02:26:05 itmon01 kernel: INFO: task mysqld:26049 blocked for more than 120 seconds. == itmon02 (1 distinct, 45 total) == Jul 17 02:07:48 itmon02 sshd[1904]: error: beginning MaxStartups throttling [x45] == jaws.cs.umd.edu (2 distinct, 2 total) == Jul 17 02:20:52 jaws.cs.umd.edu kernel: libceph: mon2 (1)10.73.235.202:6789 socket closed (con state OPEN) Jul 17 02:21:02 jaws.cs.umd.edu kernel: libceph: mon0 (1)10.73.235.200:6789 socket closed (con state OPEN) == jubjub.cs.umd.edu (2 distinct, 2 total) == Jul 17 02:20:52 jubjub.cs.umd.edu kernel: libceph: mon2 10.73.235.202:6789 socket closed (con state OPEN) Jul 17 02:21:02 jubjub.cs.umd.edu kernel: libceph: mon0 10.73.235.200:6789 socket closed (con state OPEN) == kc00 (8 distinct, 18 total) == Jul 17 02:27:29 kc00 kernel: INFO: task xfsaild/dm-0:550 blocked for more than 122 seconds. Jul 17 02:27:29 kc00 kernel: Not tainted 5.14.0-687.22.1.el9_8.x86_64 #1 [x6] Jul 17 02:27:29 kc00 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x6] Jul 17 02:27:29 kc00 kernel: INFO: task kworker/2:0:221306 blocked for more than 122 seconds. Jul 17 02:27:29 kc00 kernel: INFO: task kworker/u16:3:226820 blocked for more than 122 seconds. Jul 17 02:29:32 kc00 kernel: INFO: task xfsaild/dm-0:550 blocked for more than 245 seconds. Jul 17 02:29:32 kc00 kernel: INFO: task kworker/2:0:221306 blocked for more than 245 seconds. Jul 17 02:29:32 kc00 kernel: INFO: task kworker/u16:3:226820 blocked for more than 245 seconds. == kc01 (8 distinct, 24 total) == Jul 17 02:24:11 kc01 kernel: INFO: task xfsaild/dm-0:548 blocked for more than 122 seconds. [x2] Jul 17 02:24:11 kc01 kernel: Not tainted 5.14.0-687.20.1.el9_8.x86_64 #1 [x8] Jul 17 02:24:11 kc01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x8] Jul 17 02:24:11 kc01 kernel: INFO: task kworker/u16:3:1871582 blocked for more than 122 seconds. [x2] Jul 17 02:28:16 kc01 kernel: INFO: task kworker/2:0:1869320 blocked for more than 122 seconds. Jul 17 02:30:19 kc01 kernel: INFO: task xfsaild/dm-0:548 blocked for more than 245 seconds. Jul 17 02:30:19 kc01 kernel: INFO: task kworker/2:0:1869320 blocked for more than 245 seconds. Jul 17 02:30:19 kc01 kernel: INFO: task kworker/u16:3:1871582 blocked for more than 245 seconds. == mashie (8 distinct, 18 total) == Jul 17 02:26:58 mashie kernel: INFO: task xfsaild/dm-0:575 blocked for more than 122 seconds. Jul 17 02:26:58 mashie kernel: Not tainted 5.14.0-687.10.1.el9_8.x86_64 #1 [x6] Jul 17 02:26:58 mashie kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x6] Jul 17 02:26:58 mashie kernel: INFO: task kworker/u32:4:3535750 blocked for more than 122 seconds. Jul 17 02:26:58 mashie kernel: INFO: task kworker/1:2:3538982 blocked for more than 122 seconds. Jul 17 02:29:09 mashie kernel: INFO: task xfsaild/dm-0:575 blocked for more than 245 seconds. Jul 17 02:29:09 mashie kernel: INFO: task kworker/u32:4:3535750 blocked for more than 245 seconds. Jul 17 02:29:09 mashie kernel: INFO: task kworker/1:2:3538982 blocked for more than 245 seconds. == matrix (15 distinct, 38 total) == Jul 17 02:18:50 matrix systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT Jul 17 02:18:50 matrix systemd-journald[3344825]: File /var/log/journal/037fe7147f464e72b9491eea36fddc58/system.journal corrupted or uncleanly shut down, renaming and replacing. [x2] Jul 17 02:18:50 matrix systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)! [x2] Jul 17 02:31:11 matrix kernel: INFO: task ntfy:8529 blocked for more than 122 seconds. [x2] Jul 17 02:31:11 matrix kernel: Not tainted 6.17.0-35-generic #35~24.04.1-Ubuntu [x10] Jul 17 02:31:11 matrix kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x10] Jul 17 02:31:11 matrix kernel: INFO: task ntfy:3345277 blocked for more than 122 seconds. [x2] Jul 17 02:31:11 matrix kernel: INFO: task postgres:7916 blocked for more than 122 seconds. [x2] Jul 17 02:31:11 matrix systemd[1]: systemd-journald.service: State 'stop-watchdog' timed out. Killing. Jul 17 02:31:11 matrix systemd[1]: systemd-journald.service: Processes still around after SIGKILL. Ignoring. Jul 17 02:31:11 matrix kernel: INFO: task xfsaild/dm-0:278 blocked for more than 122 seconds. Jul 17 02:31:11 matrix kernel: INFO: task choria:735 blocked for more than 122 seconds. Jul 17 02:31:11 matrix kernel: INFO: task postgres:9232 blocked for more than 122 seconds. Jul 17 02:31:11 matrix kernel: INFO: task kworker/u16:3:3313946 blocked for more than 122 seconds. Jul 17 02:31:11 matrix systemd[1]: systemd-journald.service: Main process exited, code=killed, status=9/KILL == mcstic (14 distinct, 24 total) == Jul 17 02:31:11 mcstic kernel: INFO: task jbd2/dm-0-8:266 blocked for more than 122 seconds. Jul 17 02:31:11 mcstic kernel: Not tainted 6.8.0-124-generic #124-Ubuntu [x6] Jul 17 02:31:11 mcstic kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x6] Jul 17 02:31:11 mcstic kernel: INFO: task journal-offline:1225179 blocked for more than 122 seconds. Jul 17 02:31:11 mcstic kernel: INFO: task appstreamcli:1225177 blocked for more than 122 seconds. Jul 17 02:31:11 mcstic systemd[1]: systemd-journald.service: State 'stop-watchdog' timed out. Killing. Jul 17 02:31:11 mcstic systemd[1]: systemd-journald.service: Failed to kill control group /system.slice/systemd-journald.service, ignoring: Invalid argument Jul 17 02:31:11 mcstic kernel: INFO: task jbd2/dm-0-8:266 blocked for more than 245 seconds. Jul 17 02:31:11 mcstic kernel: INFO: task journal-offline:1225179 blocked for more than 245 seconds. Jul 17 02:31:11 mcstic kernel: INFO: task appstreamcli:1225177 blocked for more than 245 seconds. Jul 17 02:31:11 mcstic systemd[1]: systemd-journald.service: Main process exited, code=killed, status=9/KILL Jul 17 02:31:11 mcstic systemd-journald[1225373]: File /var/log/journal/e0b9538e2ed7420ab3fa55c584db854a/system.journal corrupted or uncleanly shut down, renaming and replacing. Jul 17 02:31:11 mcstic systemd[1]: systemd-journald.service: start operation timed out. Terminating. Jul 17 02:31:11 mcstic systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)! == munin (2 distinct, 320 total) == Jul 17 02:12:53 munin sudo[620877]: pam_systemd(sudo:session): Failed to stat() runtime directory '/run/user/0': No such file or directory [x160] Jul 17 02:12:53 munin sudo[620877]: pam_systemd(sudo:session): Not setting $XDG_RUNTIME_DIR, as the directory is not in order. [x160] == naemon-printer-worker (9 distinct, 23 total) == Jul 17 02:28:33 naemon-printer-worker kernel: INFO: task xfsaild/dm-0:599 blocked for more than 122 seconds. Jul 17 02:28:33 naemon-printer-worker kernel: Not tainted 6.12.0-211.20.1.el10_2.x86_64 #1 [x8] Jul 17 02:28:33 naemon-printer-worker kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x8] Jul 17 02:28:33 naemon-printer-worker kernel: INFO: task kworker/u16:1:1432462 blocked for more than 122 seconds. Jul 17 02:28:33 naemon-printer-worker kernel: INFO: task kworker/2:1:1432890 blocked for more than 122 seconds. Jul 17 02:30:36 naemon-printer-worker kernel: INFO: task xfsaild/dm-0:599 blocked for more than 245 seconds. Jul 17 02:30:36 naemon-printer-worker kernel: INFO: task auditd:825 blocked for more than 122 seconds. Jul 17 02:30:36 naemon-printer-worker kernel: INFO: task kworker/u16:1:1432462 blocked for more than 245 seconds. Jul 17 02:30:36 naemon-printer-worker kernel: INFO: task kworker/2:1:1432890 blocked for more than 245 seconds. == naemon-worker-avw1138-admin (12 distinct, 30 total) == Jul 17 02:26:42 naemon-worker-avw1138-admin kernel: INFO: task xfsaild/dm-0:796 blocked for more than 122 seconds. Jul 17 02:26:42 naemon-worker-avw1138-admin kernel: Not tainted 6.12.0-211.26.1.el10_2.x86_64 #1 [x10] Jul 17 02:26:42 naemon-worker-avw1138-admin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x10] Jul 17 02:26:42 naemon-worker-avw1138-admin kernel: INFO: task auditd:1055 blocked for more than 122 seconds. Jul 17 02:26:42 naemon-worker-avw1138-admin kernel: INFO: task kworker/u8:0:4102698 blocked for more than 122 seconds. Jul 17 02:26:42 naemon-worker-avw1138-admin kernel: INFO: task kworker/0:1:4104966 blocked for more than 122 seconds. Jul 17 02:28:45 naemon-worker-avw1138-admin kernel: INFO: task xfsaild/dm-0:796 blocked for more than 245 seconds. Jul 17 02:28:45 naemon-worker-avw1138-admin kernel: INFO: task auditd:1055 blocked for more than 245 seconds. Jul 17 02:28:45 naemon-worker-avw1138-admin kernel: INFO: task kworker/u8:0:4102698 blocked for more than 245 seconds. Jul 17 02:28:45 naemon-worker-avw1138-admin kernel: INFO: task kworker/0:1:4104966 blocked for more than 245 seconds. Jul 17 02:30:48 naemon-worker-avw1138-admin kernel: INFO: task xfsaild/dm-0:796 blocked for more than 368 seconds. Jul 17 02:30:48 naemon-worker-avw1138-admin kernel: INFO: task auditd:1055 blocked for more than 368 seconds. == nagios-worker-avw01.cs.umd.edu (113 distinct, 238 total) == Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: INFO: task xfsaild/dm-0:582 blocked for more than 120 seconds. [x2] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: Not tainted 4.18.0-553.117.1.el8_10.x86_64 #1 [x7] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x7] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: Call Trace: [x7] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: __schedule+0x2d1/0x870 [x7] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: schedule+0x55/0xf0 [x7] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: xlog_wait_on_iclog+0x124/0x180 [xfs] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: ? wake_up_q+0x60/0x60 Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: xfsaild_push+0x373/0x6d0 [xfs] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: ? schedule_timeout+0x1a9/0x320 [x2] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: xfsaild+0x70/0x230 [xfs] [x2] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: ? xfsaild_push+0x6d0/0x6d0 [xfs] [x2] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: kthread+0x134/0x150 [x3] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: ? set_kthread_struct+0x50/0x50 [x3] Jul 17 02:24:47 nagios-worker-avw01.cs.umd.edu kernel: ret_from_fork+0x35/0x40 [x3] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: ? mempool_alloc+0x67/0x180 [x4] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: ? wbt_exit+0x30/0x30 [x12] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: ? __wbt_done+0x40/0x40 [x6] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: io_schedule+0x12/0x40 [x6] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: rq_qos_wait+0xb3/0x130 [x6] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: ? karma_partition+0x1f0/0x1f0 [x6] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: wbt_wait+0x96/0xc0 [x6] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: __rq_qos_throttle+0x23/0x40 [x6] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: blk_mq_make_request+0x131/0x5c0 [x6] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: generic_make_request_no_check+0xe1/0x330 [x6] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: submit_bio+0x3c/0x160 [x6] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: ? bio_add_page+0x46/0x60 Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: _xfs_buf_ioapply+0x2af/0x430 [xfs] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: ? xfs_iextents_copy+0xba/0x170 [xfs] Jul 17 02:30:55 nagios-worker-avw01.cs.umd.edu kernel: ? _xfs_buf_obj_cmp+0x30/0x30 [xfs] ... 83 more distinct messages not shown == nagios-worker-v6-avw00.cs.umd.edu (111 distinct, 264 total) == Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: INFO: task xfsaild/dm-0:586 blocked for more than 120 seconds. [x2] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: Not tainted 4.18.0-553.117.1.el8_10.x86_64 #1 [x9] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x9] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: Call Trace: [x9] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: __schedule+0x2d1/0x870 [x9] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: schedule+0x55/0xf0 [x9] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: xlog_wait_on_iclog+0x124/0x180 [xfs] [x2] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: ? wake_up_q+0x60/0x60 [x2] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: xfsaild_push+0x373/0x6d0 [xfs] [x2] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: ? schedule_timeout+0x1a9/0x320 [x2] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: xfsaild+0x70/0x230 [xfs] [x2] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: ? xfsaild_push+0x6d0/0x6d0 [xfs] [x2] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: kthread+0x134/0x150 [x4] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: ? set_kthread_struct+0x50/0x50 [x4] Jul 17 02:16:21 nagios-worker-v6-avw00.cs.umd.edu kernel: ret_from_fork+0x1f/0x40 [x4] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: INFO: task splunkd:2626 blocked for more than 120 seconds. [x2] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: ? bit_wait_io+0x50/0x50 [x4] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: bit_wait+0xd/0x50 [x2] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: __wait_on_bit+0x2d/0x90 [x2] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: __inode_wait_for_writeback+0xb6/0xf0 [x2] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: ? var_wake_function+0x30/0x30 [x2] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: inode_wait_for_writeback+0x21/0x40 [x2] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: evict+0xbc/0x1a0 [x2] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: __dentry_kill+0xd5/0x170 [x2] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: dentry_kill+0x4d/0x1a0 [x2] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: dput.part.36+0xff/0x150 [x2] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: do_renameat2+0x39b/0x510 [x2] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: __x64_sys_rename+0x1c/0x30 [x2] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: do_syscall_64+0x5b/0x1d0 [x3] Jul 17 02:26:35 nagios-worker-v6-avw00.cs.umd.edu kernel: entry_SYSCALL_64_after_hwframe+0x66/0xcb [x5] ... 81 more distinct messages not shown == neo-gizmonic (12 distinct, 18 total) == Jul 17 02:31:07 neo-gizmonic kernel: INFO: task jbd2/dm-0-8:308 blocked for more than 122 seconds. Jul 17 02:31:07 neo-gizmonic kernel: Not tainted 6.8.0-124-generic #124-Ubuntu [x4] Jul 17 02:31:07 neo-gizmonic kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x4] Jul 17 02:31:07 neo-gizmonic kernel: INFO: task jbd2/dm-0-8:308 blocked for more than 245 seconds. Jul 17 02:31:07 neo-gizmonic kernel: INFO: task journal-offline:2635968 blocked for more than 122 seconds. Jul 17 02:31:07 neo-gizmonic systemd[1]: systemd-journald.service: State 'stop-watchdog' timed out. Killing. Jul 17 02:31:07 neo-gizmonic systemd[1]: systemd-journald.service: Failed to kill control group /system.slice/systemd-journald.service, ignoring: Invalid argument Jul 17 02:31:07 neo-gizmonic kernel: INFO: task systemd-journal:381 blocked for more than 122 seconds. Jul 17 02:31:07 neo-gizmonic systemd[1]: systemd-journald.service: Processes still around after SIGKILL. Ignoring. Jul 17 02:31:07 neo-gizmonic systemd[1]: systemd-journald.service: Main process exited, code=killed, status=9/KILL Jul 17 02:31:07 neo-gizmonic systemd-journald[2636561]: File /var/log/journal/6e19242c522a420989ab18e2268a6536/system.journal corrupted or uncleanly shut down, renaming and replacing. Jul 17 02:31:07 neo-gizmonic systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)! == neograd (9 distinct, 23 total) == Jul 17 02:26:42 neograd kernel: INFO: task xfsaild/dm-0:527 blocked for more than 122 seconds. [x2] Jul 17 02:26:42 neograd kernel: Not tainted 5.14.0-611.47.1.el9_7.x86_64 #1 [x7] Jul 17 02:26:42 neograd kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x7] Jul 17 02:26:42 neograd kernel: INFO: task kworker/u8:1:121158 blocked for more than 122 seconds. [x2] Jul 17 02:27:02 neograd kernel: Process 122693(rpm) has RLIMIT_CORE set to 1 Jul 17 02:27:02 neograd kernel: Aborting core Jul 17 02:30:48 neograd kernel: INFO: task auditd:706 blocked for more than 122 seconds. Jul 17 02:30:48 neograd kernel: INFO: task ir_agent:1126 blocked for more than 122 seconds. Jul 17 02:30:48 neograd kernel: INFO: task kworker/1:0:115366 blocked for more than 122 seconds. == netbox (11 distinct, 29 total) == Jul 17 02:26:56 netbox kernel: INFO: task xfsaild/dm-0:613 blocked for more than 122 seconds. Jul 17 02:26:56 netbox kernel: Not tainted 6.12.0-211.16.1.el10_2.x86_64 #1 [x10] Jul 17 02:26:56 netbox kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x10] Jul 17 02:26:56 netbox kernel: INFO: task kworker/u16:2:3197803 blocked for more than 122 seconds. Jul 17 02:26:56 netbox kernel: INFO: task kworker/u16:0:3202739 blocked for more than 122 seconds. Jul 17 02:26:56 netbox kernel: INFO: task kworker/2:2:3203692 blocked for more than 122 seconds. Jul 17 02:28:59 netbox kernel: INFO: task xfsaild/dm-0:613 blocked for more than 245 seconds. Jul 17 02:28:59 netbox kernel: INFO: task auditd:840 blocked for more than 122 seconds. Jul 17 02:28:59 netbox kernel: INFO: task kworker/u16:2:3197803 blocked for more than 245 seconds. Jul 17 02:28:59 netbox kernel: INFO: task kworker/u16:0:3202739 blocked for more than 245 seconds. Jul 17 02:28:59 netbox kernel: INFO: task kworker/2:2:3203692 blocked for more than 245 seconds. == netman.cs.umd.edu (1 distinct, 1 total) == Jul 17 03:03:49 netman.cs.umd.edu sshd[1084345]: error: PAM: Authentication failure for nobody from 177.174.105.113 == newhonors.cs.umd.edu (49 distinct, 68 total) == Jul 17 02:30:01 newhonors.cs.umd.edu kernel: INFO: task jbd2/dm-0-8:442 blocked for more than 120 seconds. Jul 17 02:30:01 newhonors.cs.umd.edu kernel: Not tainted 4.4.0-128-generic #154-Ubuntu [x2] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x2] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: ffff880035da7ad8 ffff8801abf589e8 ffff8801abffaa80 ffff8801abffbfc0 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: ffff880035da8000 ffff8801b6c172c0 7fffffffffffffff ffffffff8184e3f0 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: ffff880035da7c30 ffff880035da7af0 ffffffff8184dbf5 0000000000000000 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: Call Trace: [x2] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff8184e3f0>] ? bit_wait+0x60/0x60 [x6] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff8184dbf5>] schedule+0x35/0x80 [x2] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff81850d86>] schedule_timeout+0x1b6/0x270 [x2] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff810cec21>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20 [x3] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff8106644e>] ? kvm_clock_get_cycles+0x1e/0x20 [x2] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff810fa29e>] ? ktime_get+0x3e/0xb0 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff8184d364>] io_schedule_timeout+0xa4/0x110 [x2] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff8184e40b>] bit_wait_io+0x1b/0x70 [x2] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff8184df9f>] __wait_on_bit+0x5f/0x90 [x2] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff8184e052>] out_of_line_wait_on_bit+0x82/0xb0 [x2] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff810c7fa0>] ? autoremove_wake_function+0x40/0x40 [x2] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff8124d002>] __wait_on_buffer+0x32/0x40 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff812f6144>] jbd2_journal_commit_transaction+0xf54/0x1890 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff810f072a>] ? try_to_del_timer_sync+0x5a/0x80 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff812fa6aa>] kjournald2+0xca/0x250 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff810c7f60>] ? wake_atomic_t_function+0x60/0x60 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff812fa5e0>] ? commit_timeout+0x10/0x10 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff810a4477>] kthread+0xe7/0x100 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff810a4390>] ? kthread_create_on_node+0x1e0/0x1e0 [x2] Jul 17 02:30:01 newhonors.cs.umd.edu kernel: [<ffffffff81852235>] ret_from_fork+0x55/0x80 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: INFO: task rs:main Q:Reg:854 blocked for more than 120 seconds. Jul 17 02:30:01 newhonors.cs.umd.edu kernel: ffff8801ad1978f8 ffff8801ad197978 ffff8800bbaf6a40 ffff880035d81540 Jul 17 02:30:01 newhonors.cs.umd.edu kernel: ffff8801ad198000 ffff8801b6c172c0 7fffffffffffffff ffffffff8184e3f0 ... 19 more distinct messages not shown == openrepo (3 distinct, 3 total) == Jul 17 02:30:47 openrepo kernel: INFO: task jbd2/sda1-8:265 blocked for more than 120 seconds. Jul 17 02:30:47 openrepo kernel: Not tainted 6.12.74+deb13+1-amd64 #1 Debian 6.12.74-2 Jul 17 02:30:47 openrepo kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. == openvox08 (7 distinct, 24 total) == Jul 17 02:17:40 openvox08 kernel: INFO: task xfsaild/dm-0:615 blocked for more than 122 seconds. [x2] Jul 17 02:17:40 openvox08 kernel: Not tainted 6.12.0-211.16.1.el10_2.x86_64 #1 [x8] Jul 17 02:17:40 openvox08 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x8] Jul 17 02:17:40 openvox08 kernel: INFO: task auditd:840 blocked for more than 122 seconds. [x2] Jul 17 02:17:40 openvox08 kernel: INFO: task kworker/0:0:3062829 blocked for more than 122 seconds. Jul 17 02:17:40 openvox08 kernel: INFO: task kworker/u16:4:3065158 blocked for more than 122 seconds. [x2] Jul 17 02:27:55 openvox08 kernel: INFO: task kworker/2:1:3064640 blocked for more than 122 seconds. == openvox08-cmp1 (4 distinct, 6 total) == Jul 17 02:29:07 openvox08-cmp1 kernel: INFO: task kworker/u16:3:186085 blocked for more than 122 seconds. Jul 17 02:29:07 openvox08-cmp1 kernel: Not tainted 6.12.0-211.34.1.el10_2.x86_64 #1 [x2] Jul 17 02:29:07 openvox08-cmp1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x2] Jul 17 02:29:07 openvox08-cmp1 kernel: INFO: task kworker/2:1:187862 blocked for more than 122 seconds. == openvox08-cmp3 (12 distinct, 30 total) == Jul 17 02:26:12 openvox08-cmp3 kernel: INFO: task xfsaild/dm-0:605 blocked for more than 122 seconds. Jul 17 02:26:12 openvox08-cmp3 kernel: Not tainted 6.12.0-211.31.1.el10_2.x86_64 #1 [x10] Jul 17 02:26:12 openvox08-cmp3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x10] Jul 17 02:26:12 openvox08-cmp3 kernel: INFO: task kworker/u16:3:1101843 blocked for more than 122 seconds. Jul 17 02:26:12 openvox08-cmp3 kernel: INFO: task kworker/u16:0:1102703 blocked for more than 122 seconds. Jul 17 02:28:15 openvox08-cmp3 kernel: INFO: task xfsaild/dm-0:605 blocked for more than 245 seconds. Jul 17 02:28:15 openvox08-cmp3 kernel: INFO: task auditd:838 blocked for more than 122 seconds. Jul 17 02:28:15 openvox08-cmp3 kernel: INFO: task kworker/3:1:1101829 blocked for more than 122 seconds. Jul 17 02:28:15 openvox08-cmp3 kernel: INFO: task kworker/u16:3:1101843 blocked for more than 245 seconds. Jul 17 02:28:15 openvox08-cmp3 kernel: INFO: task kworker/u16:0:1102703 blocked for more than 245 seconds. Jul 17 02:30:18 openvox08-cmp3 kernel: INFO: task xfsaild/dm-0:605 blocked for more than 368 seconds. Jul 17 02:30:18 openvox08-cmp3 kernel: INFO: task auditd:838 blocked for more than 245 seconds. == openvox08-db (11 distinct, 27 total) == Jul 17 02:26:48 openvox08-db kernel: INFO: task xfsaild/dm-0:607 blocked for more than 122 seconds. Jul 17 02:26:48 openvox08-db kernel: Not tainted 6.12.0-211.33.1.el10_2.x86_64 #1 [x9] Jul 17 02:26:48 openvox08-db kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x9] Jul 17 02:26:48 openvox08-db kernel: INFO: task auditd:841 blocked for more than 122 seconds. Jul 17 02:26:48 openvox08-db kernel: INFO: task kworker/u16:3:536118 blocked for more than 122 seconds. Jul 17 02:26:48 openvox08-db kernel: INFO: task kworker/2:3:543544 blocked for more than 122 seconds. Jul 17 02:28:51 openvox08-db kernel: INFO: task xfsaild/dm-0:607 blocked for more than 245 seconds. Jul 17 02:28:51 openvox08-db kernel: INFO: task auditd:841 blocked for more than 245 seconds. Jul 17 02:28:51 openvox08-db kernel: INFO: task qtp549064587-90:492305 blocked for more than 122 seconds. Jul 17 02:28:51 openvox08-db kernel: INFO: task qtp549064587-99:547057 blocked for more than 122 seconds. Jul 17 02:28:51 openvox08-db kernel: INFO: task kworker/u16:3:536118 blocked for more than 245 seconds. == outbound00.cs.umd.edu (10 distinct, 25 total) == Jul 17 02:09:18 outbound00.cs.umd.edu python3[808589]: ldapdb [x4] Jul 17 02:16:16 outbound00.cs.umd.edu kernel: INFO: task xfsaild/dm-0:593 blocked for more than 122 seconds. Jul 17 02:16:16 outbound00.cs.umd.edu kernel: Not tainted 5.14.0-687.22.1.el9_8.x86_64 #1 [x7] Jul 17 02:16:16 outbound00.cs.umd.edu kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x7] Jul 17 02:16:16 outbound00.cs.umd.edu kernel: INFO: task kworker/u16:0:805879 blocked for more than 122 seconds. Jul 17 02:16:16 outbound00.cs.umd.edu kernel: INFO: task bounce:809558 blocked for more than 122 seconds. Jul 17 02:18:19 outbound00.cs.umd.edu kernel: INFO: task xfsaild/dm-0:593 blocked for more than 245 seconds. Jul 17 02:18:19 outbound00.cs.umd.edu kernel: INFO: task kworker/u16:0:805879 blocked for more than 245 seconds. Jul 17 02:18:19 outbound00.cs.umd.edu kernel: INFO: task kworker/1:3:808767 blocked for more than 122 seconds. Jul 17 02:18:19 outbound00.cs.umd.edu kernel: INFO: task bounce:809558 blocked for more than 245 seconds. == outbound01.cs.umd.edu (13 distinct, 31 total) == Jul 17 02:23:45 outbound01.cs.umd.edu python3[4191179]: ldapdb [x5] Jul 17 02:25:39 outbound01.cs.umd.edu kernel: INFO: task xfsaild/dm-0:588 blocked for more than 122 seconds. Jul 17 02:25:39 outbound01.cs.umd.edu kernel: Not tainted 5.14.0-611.47.1.el9_7.x86_64 #1 [x8] Jul 17 02:25:39 outbound01.cs.umd.edu kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x8] Jul 17 02:25:39 outbound01.cs.umd.edu kernel: INFO: task auditd:783 blocked for more than 122 seconds. Jul 17 02:25:39 outbound01.cs.umd.edu kernel: INFO: task kworker/u16:0:4176638 blocked for more than 122 seconds. Jul 17 02:25:39 outbound01.cs.umd.edu kernel: INFO: task kworker/3:1:4187273 blocked for more than 122 seconds. Jul 17 02:25:39 outbound01.cs.umd.edu kernel: INFO: task cleanup:4189253 blocked for more than 122 seconds. Jul 17 02:25:39 outbound01.cs.umd.edu kernel: INFO: task cleanup:4190749 blocked for more than 122 seconds. Jul 17 02:25:39 outbound01.cs.umd.edu kernel: INFO: task cleanup:4190794 blocked for more than 122 seconds. Jul 17 02:27:42 outbound01.cs.umd.edu kernel: INFO: task xfsaild/dm-0:588 blocked for more than 245 seconds. Jul 17 02:36:35 outbound01.cs.umd.edu krb5_child[941]: Client 'admin@UMD.EDU' not found in Kerberos database Jul 17 02:36:50 outbound01.cs.umd.edu sshd-session[974]: error: Protocol major versions differ: 2 vs. 1 == peach.cs.umd.edu (2 distinct, 4 total) == Jul 17 02:22:09 peach.cs.umd.edu setroubleshoot[762940]: SELinux is preventing /usr/bin/qemu-ga from search access on the directory /var/lib/containers/storage/overlay. For complete SELinux messages run: sealert -l 007f2494-1b82-4ff6-bd59-5f449ab9de99 [x2] Jul 17 02:22:09 peach.cs.umd.edu setroubleshoot[762940]: SELinux is preventing /usr/bin/qemu-ga from search access on the directory /var/lib/containers/storage/overlay. For complete SELinux messages run: sealert -l a801d9aa-5cee-4f5c-b9a9-bd22e572596b [x2] == sebatinsky.cs.umd.edu (1 distinct, 1 total) == Jul 17 02:23:35 sebatinsky.cs.umd.edu kernel: hpet1: lost 78 rtc interrupts == smokeping (12 distinct, 22 total) == Jul 17 02:31:13 smokeping kernel: INFO: task jbd2/dm-1-8:564 blocked for more than 120 seconds. Jul 17 02:31:13 smokeping kernel: Not tainted 6.12.73+deb13-amd64 #1 Debian 6.12.73-1 [x6] Jul 17 02:31:13 smokeping kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x6] Jul 17 02:31:13 smokeping kernel: INFO: task choria:915 blocked for more than 120 seconds. Jul 17 02:31:13 smokeping kernel: INFO: task journal-offline:983730 blocked for more than 120 seconds. Jul 17 02:31:13 smokeping kernel: INFO: task kworker/u16:1:982366 blocked for more than 120 seconds. Jul 17 02:31:13 smokeping kernel: INFO: task kworker/u16:3:982865 blocked for more than 120 seconds. Jul 17 02:31:13 smokeping kernel: INFO: task kworker/u16:4:983165 blocked for more than 120 seconds. Jul 17 02:31:13 smokeping systemd[1]: systemd-journald.service: State 'stop-watchdog' timed out. Killing. Jul 17 02:31:13 smokeping systemd[1]: systemd-journald.service: Main process exited, code=killed, status=9/KILL Jul 17 02:31:13 smokeping systemd-journald[983866]: File /var/log/journal/f75f384fcaef4d6e971d5910816b80ee/system.journal corrupted or uncleanly shut down, renaming and replacing. Jul 17 02:31:13 smokeping systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)! == soul (2 distinct, 2 total) == Jul 17 03:00:57 soul krb5_child[257322]: Client 'admin@UMD.EDU' not found in Kerberos database Jul 17 03:02:09 soul sshd-session[257638]: error: Protocol major versions differ: 2 vs. 1 == strawberry (2 distinct, 8 total) == Jul 17 02:09:44 strawberry udisksd[1067]: Error performing housekeeping for drive /org/freedesktop/UDisks2/drives/ST2000DM001_1CH164_W1E8J5KJ: Error updating SMART data: Error sending ATA command CHECK POWER MODE: Unexpected sense data returned:#0120000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................#0120010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................#012 (g-io-error-quark, 0) [x6] Jul 17 02:34:42 strawberry smartd[5314]: Device: /dev/sda [SAT], failed to read SMART Attribute Data [x2] == submit26-b9-2 (9 distinct, 12 total) == Jul 17 02:50:08 submit26-b9-2 setroubleshoot[3576947]: SELinux is preventing /usr/bin/python3.9 from create access on the directory /(null). For complete SELinux messages run: sealert -l d506243c-d61e-48f3-b42a-4592c913deb0 Jul 17 02:50:08 submit26-b9-2 setroubleshoot[3576947]: SELinux is preventing /usr/bin/python3.9 from add_name access on the directory /(null). For complete SELinux messages run: sealert -l 3b4752ac-1b9a-4af9-a05b-32e4c2eb13ab Jul 17 02:50:08 submit26-b9-2 setroubleshoot[3576947]: SELinux is preventing /usr/bin/python3.9 from create access on the file /(null). For complete SELinux messages run: sealert -l 29e5adbd-4381-4467-b40d-a2219101e096 [x2] Jul 17 02:50:08 submit26-b9-2 setroubleshoot[3576947]: SELinux is preventing /usr/bin/python3.9 from open access on the file /tmp/libdnf.0pigeG/repodata/repomd.xml. For complete SELinux messages run: sealert -l 1b38f2ec-01ce-4343-8059-ba7494ecc5c4 [x2] Jul 17 02:50:09 submit26-b9-2 setroubleshoot[3576947]: SELinux is preventing /usr/bin/python3.9 from setattr access on the file repomd.xml. For complete SELinux messages run: sealert -l e5f9af40-1f63-4924-b012-27ab2a7b2266 [x2] Jul 17 02:50:09 submit26-b9-2 setroubleshoot[3576947]: SELinux is preventing /usr/bin/python3.9 from read access on the directory libdnf.0pigeG. For complete SELinux messages run: sealert -l 5c54db87-54a3-4a7c-97b4-b5c48d6c3543 Jul 17 02:50:09 submit26-b9-2 setroubleshoot[3576947]: SELinux is preventing /usr/bin/python3.9 from remove_name access on the directory repomd.xml. For complete SELinux messages run: sealert -l 63cd60e1-1e61-4276-afea-dd8526ffe8b2 Jul 17 02:50:09 submit26-b9-2 setroubleshoot[3576947]: SELinux is preventing /usr/bin/python3.9 from unlink access on the file repomd.xml. For complete SELinux messages run: sealert -l 81c7e9ec-e023-4c6d-b1ad-c02a242e655e Jul 17 02:50:09 submit26-b9-2 setroubleshoot[3576947]: SELinux is preventing /usr/bin/python3.9 from rmdir access on the directory repodata. For complete SELinux messages run: sealert -l 6ae9a597-e123-4d74-8eec-a8ebe0f5e4d0 == talkhosts9 (14 distinct, 28 total) == Jul 17 02:25:25 talkhosts9 kernel: INFO: task kworker/u8:3:1667086 blocked for more than 122 seconds. Jul 17 02:26:34 talkhosts9 kernel: Not tainted 5.14.0-687.22.1.el9_8.x86_64 #1 [x8] Jul 17 02:26:34 talkhosts9 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x8] Jul 17 02:27:19 talkhosts9 kernel: INFO: task kworker/1:0:1668310 blocked for more than 122 seconds. Jul 17 02:27:19 talkhosts9 kernel: INFO: task kworker/u8:4:1668325 blocked for more than 122 seconds. Jul 17 02:31:09 talkhosts9 kernel: INFO: task kworker/u8:5:1668328 blocked for more than 122 seconds. Jul 17 02:31:09 talkhosts9 kernel: INFO: task kworker/u8:7:1668749 blocked for more than 122 seconds. Jul 17 02:31:09 talkhosts9 kernel: INFO: task ir_agent:1054 blocked for more than 122 seconds. Jul 17 02:31:09 talkhosts9 kernel: INFO: task kworker/u8:0:1657063 blocked for more than 122 seconds. Jul 17 02:31:09 talkhosts9 kernel: INFO: task kworker/u8:1:1666170 blocked for more than 122 seconds. Jul 17 02:31:09 talkhosts9 systemd[1]: systemd-journald.service: State 'stop-watchdog' timed out. Killing. Jul 17 02:31:09 talkhosts9 systemd[1]: systemd-journald.service: Main process exited, code=killed, status=9/KILL Jul 17 02:31:09 talkhosts9 systemd-journald[1670397]: File /run/log/journal/82ede0f66ed24ebdaea2923548bfcdea/system.journal corrupted or uncleanly shut down, renaming and replacing. Jul 17 02:31:09 talkhosts9 systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)! == teaching-awards (5 distinct, 5 total) == Jul 17 02:26:43 teaching-awards kernel: INFO: task auditd:795 blocked for more than 122 seconds. Jul 17 02:26:43 teaching-awards kernel: Not tainted 5.14.0-687.22.1.el9_8.x86_64 #1 Jul 17 02:26:43 teaching-awards kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 17 02:30:40 teaching-awards kernel: Process 623728(rpm) has RLIMIT_CORE set to 1 Jul 17 02:30:40 teaching-awards kernel: Aborting core == test-ipv6 (6 distinct, 12 total) == Jul 17 02:27:00 test-ipv6 kernel: INFO: task xfsaild/dm-0:549 blocked for more than 122 seconds. Jul 17 02:27:00 test-ipv6 kernel: Not tainted 5.14.0-611.9.1.el9_7.x86_64 #1 [x4] Jul 17 02:27:00 test-ipv6 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x4] Jul 17 02:27:00 test-ipv6 kernel: INFO: task kworker/u16:4:690588 blocked for more than 122 seconds. Jul 17 02:27:00 test-ipv6 kernel: INFO: task kworker/3:0:696192 blocked for more than 122 seconds. Jul 17 02:29:03 test-ipv6 kernel: INFO: task xfsaild/dm-0:549 blocked for more than 245 seconds. == tulgey.cs.umd.edu (2 distinct, 2 total) == Jul 17 02:20:52 tulgey.cs.umd.edu kernel: libceph: mon2 (1)10.73.235.202:6789 socket closed (con state OPEN) Jul 17 02:21:02 tulgey.cs.umd.edu kernel: libceph: mon0 (1)10.73.235.200:6789 socket closed (con state OPEN) == uffish.cs.umd.edu (1 distinct, 1 total) == Jul 17 02:20:52 uffish.cs.umd.edu kernel: libceph: mon2 10.73.235.202:6789 socket closed (con state OPEN) == unknown (1 distinct, 1 total) == using-explainable-ai == updates01.cs.umd.edu (34 distinct, 85 total) == Jul 17 02:16:16 updates01.cs.umd.edu kernel: INFO: task xfsaild/sda3:462 blocked for more than 120 seconds. [x3] Jul 17 02:16:16 updates01.cs.umd.edu kernel: Not tainted 4.18.0-553.117.1.el8_10.x86_64 #1 [x5] Jul 17 02:16:16 updates01.cs.umd.edu kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x5] Jul 17 02:16:16 updates01.cs.umd.edu kernel: Call Trace: [x5] Jul 17 02:16:16 updates01.cs.umd.edu kernel: __schedule+0x2d1/0x870 [x5] Jul 17 02:16:16 updates01.cs.umd.edu kernel: schedule+0x55/0xf0 [x5] Jul 17 02:16:16 updates01.cs.umd.edu kernel: xlog_wait_on_iclog+0x124/0x180 [xfs] [x3] Jul 17 02:16:16 updates01.cs.umd.edu kernel: ? wake_up_q+0x60/0x60 [x4] Jul 17 02:16:16 updates01.cs.umd.edu kernel: xfsaild_push+0x373/0x6d0 [xfs] [x3] Jul 17 02:16:16 updates01.cs.umd.edu kernel: ? schedule_timeout+0x1a9/0x320 [x3] Jul 17 02:16:16 updates01.cs.umd.edu kernel: xfsaild+0x70/0x230 [xfs] [x3] Jul 17 02:16:16 updates01.cs.umd.edu kernel: ? xfsaild_push+0x6d0/0x6d0 [xfs] [x3] Jul 17 02:16:16 updates01.cs.umd.edu kernel: kthread+0x134/0x150 [x5] Jul 17 02:16:16 updates01.cs.umd.edu kernel: ? set_kthread_struct+0x50/0x50 [x5] Jul 17 02:16:16 updates01.cs.umd.edu kernel: ret_from_fork+0x1f/0x40 [x5] Jul 17 02:30:36 updates01.cs.umd.edu kernel: INFO: task kworker/u2:2:1509734 blocked for more than 120 seconds. Jul 17 02:30:36 updates01.cs.umd.edu kernel: xlog_state_get_iclog_space+0x10b/0x340 [xfs] Jul 17 02:30:36 updates01.cs.umd.edu kernel: xlog_write+0x152/0x740 [xfs] Jul 17 02:30:36 updates01.cs.umd.edu kernel: xlog_cil_push_work+0x2ab/0x550 [xfs] Jul 17 02:30:36 updates01.cs.umd.edu kernel: ? set_next_entity+0xb5/0x1e0 Jul 17 02:30:36 updates01.cs.umd.edu kernel: process_one_work+0x1d3/0x390 [x2] Jul 17 02:30:36 updates01.cs.umd.edu kernel: ? process_one_work+0x390/0x390 [x4] Jul 17 02:30:36 updates01.cs.umd.edu kernel: worker_thread+0x1ce/0x390 Jul 17 02:30:36 updates01.cs.umd.edu kernel: INFO: task kworker/0:0:1620373 blocked for more than 120 seconds. Jul 17 02:30:36 updates01.cs.umd.edu kernel: schedule_timeout+0x281/0x320 Jul 17 02:30:36 updates01.cs.umd.edu kernel: ? enqueue_entity+0x102/0x6b0 Jul 17 02:30:36 updates01.cs.umd.edu kernel: ? check_preempt_wakeup+0x113/0x270 Jul 17 02:30:36 updates01.cs.umd.edu kernel: wait_for_completion+0x96/0x100 Jul 17 02:30:36 updates01.cs.umd.edu kernel: flush_work+0x11d/0x1d0 Jul 17 02:30:36 updates01.cs.umd.edu kernel: ? rcu_free_pwq+0x20/0x20 ... 4 more distinct messages not shown == vault (8 distinct, 20 total) == Jul 17 02:28:45 vault kernel: INFO: task xfsaild/dm-0:558 blocked for more than 122 seconds. Jul 17 02:28:45 vault kernel: Not tainted 5.14.0-611.47.1.el9_7.x86_64 #1 [x7] Jul 17 02:28:45 vault kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [x7] Jul 17 02:28:45 vault kernel: INFO: task kworker/u24:0:3269233 blocked for more than 122 seconds. Jul 17 02:28:45 vault kernel: INFO: task kworker/0:2:3272349 blocked for more than 122 seconds. Jul 17 02:30:48 vault kernel: INFO: task xfsaild/dm-0:558 blocked for more than 245 seconds. Jul 17 02:30:48 vault kernel: INFO: task kworker/u24:0:3269233 blocked for more than 245 seconds. Jul 17 02:30:48 vault kernel: INFO: task kworker/0:2:3272349 blocked for more than 245 seconds.