Portál AbcLinuxu, 6. května 2025 16:31
munin a po chvíli pátrání jsem našel ls /usr/share/munin/plugins/ ls: cannot access /usr/share/munin/plugins/ipmi_: Stale file handle ls: cannot access /usr/share/munin/plugins/ipmi_sensor_: Stale file handle ls: cannot access /usr/share/munin/plugins/ircu: Stale file handle ls: cannot access /usr/share/munin/plugins/irqstats: Stale file handle ls: cannot access /usr/share/munin/plugins/jmx_tomcat_dbpools: Stale file handle ls: cannot access /usr/share/munin/plugins/load: Stale file handle ls: cannot access /usr/share/munin/plugins/loggrep: Stale file handle ls: cannot access /usr/share/munin/plugins/lpar_cpu: Stale file handle ls: cannot access /usr/share/munin/plugins/lpstat: Stale file handle ls: cannot access /usr/share/munin/plugins/mailman: Stale file handle ls: cannot access /usr/share/munin/plugins/mailscanner: Stale file handle ls: cannot access /usr/share/munin/plugins/mbmon_: Stale file handle ls: cannot access /usr/share/munin/plugins/meminfo: Stale file handle ls: cannot access /usr/share/munin/plugins/memory: Stale file handle ls: cannot access /usr/share/munin/plugins/mhttping: Stale file handle ls: cannot access /usr/share/munin/plugins/multiping: Stale file handle ls: cannot access /usr/share/munin/plugins/multips: Stale file handle ls: cannot access /usr/share/munin/plugins/multips_memory: Stale file handle ls: cannot access /usr/share/munin/plugins/munin_stats: Stale file handle ls: cannot access /usr/share/munin/plugins/munin_update: Stale file handle ls: cannot access /usr/share/munin/plugins/mysql_: Stale file handle ls: cannot access /usr/share/munin/plugins/mysql_bytes: Stale file handle ls: cannot access /usr/share/munin/plugins/mysql_innodb: Stale file handle ls: cannot access /usr/share/munin/plugins/mysql_isam_space_: Stale file handle ls: cannot access /usr/share/munin/plugins/mysql_queries: Stale file handle ls: cannot access /usr/share/munin/plugins/mysql_slowqueries: Stale file handle ls: cannot access /usr/share/munin/plugins/mysql_threads: Stale file handle ls: cannot access /usr/share/munin/plugins/named: Stale file handle ls: cannot access /usr/share/munin/plugins/netopia: Stale file handle ls: cannot access /usr/share/munin/plugins/netstat: Stale file handle acpi cps_ fw_packets mailman nginx_request postfix_mailvolume qmailscan-simple snmp__print_supplies squid_requests amavis cpu haproxy_ mailscanner nginx_status postgres_autovacuum quota_usage_ snmp__processes squid_traffic apache_accesses cpuspeed haproxy_ng mbmon_ nomadix_users_ postgres_bgwriter samba snmp__rdp_users surfboard apache_processes cupsys_pages hddtemp meminfo ntp_kernel_err postgres_cache_ selinux_avcstat snmp__sensors_fsc_bx_fan swap apache_volume df hddtemp2 memory ntp_kernel_pll_freq postgres_checkpoints sendmail_mailqueue snmp__sensors_fsc_bx_temp tcp apc_envunit_ df_abs hddtempd mhttping ntp_kernel_pll_off postgres_connections_ sendmail_mailstats snmp__sensors_fsc_fan threads apc_nis df_inode hddtemp_smartctl multiping ntp_offset postgres_connections_db sendmail_mailtraffic snmp__sensors_fsc_temp tomcat_access apt digitemp_ hp2000_ multips ntp_states postgres_locks_ sensors_ snmp__sensors_mbm_fan tomcat_jvm apt_all diskstat_ http_loadtime multips_memory nut_misc postgres_oldest_prepared_xact_ slapd_ snmp__sensors_mbm_temp tomcat_threads asterisk_channels diskstats if_ munin_stats nutups_ postgres_prepared_xacts_ slapd_bdb_cache_ snmp__sensors_mbm_volt tomcat_volume asterisk_channelstypes ejabberd_ if_err_ munin_update nut_volts postgres_querylength_ slony_lag_ snmp__swap uptime asterisk_codecs entropy ifx_concurrent_sessions_ mysql_ nvidia_ postgres_scans_ smart_ snmp__uptime users asterisk_meetme exim_mailqueue interrupts mysql_bytes open_files postgres_size_ snmp__cpuload snmp__users varnish_ asterisk_meetmeusers exim_mailqueue_alt iostat mysql_innodb open_inodes postgres_streaming_ snmp__df snmp__winload vlan_ asterisk_sipchannels exim_mailstats iostat_ios mysql_isam_space_ openvpn postgres_transactions_ snmp__df_ram snmp__winmem vlan_inetuse_ asterisk_sippeers fail2ban ip_ mysql_queries perdition postgres_tuples_ snmp__fc_if_ snort_alerts vlan_linkuse_ asterisk_voicemail foldingathome ipac-ng mysql_slowqueries pgbouncer_connections postgres_users snmp__fc_if_err_ snort_bytes_pkt vmstat bind9 foldingathome_rank ipmi_ mysql_threads pgbouncer_requests postgres_xlog snmp__if_ snort_drop_rate vserver_cpu_ bind9_rndc foldingathome_wu ipmi_sensor_ named ping_ proc snmp__if_err_ snort_pattern_match vserver_loadavg bonding_err_ forks ircu netopia plugin.sh processes snmp__if_multi snort_pkts vserver_resources buddyinfo freeradius_acct irqstats netstat plugins.history proc_pri snmp__load snort_traffic yum cmc_tc_sensor_ freeradius_auth jmx_tomcat_dbpools netstat_multi pm3users_ proxy_plugin snmp__memory spamstats zimbra_ courier_ freeradius_proxy_acct load nfs4_client pop_stats ps_ snmp__netapp_diskusage_ squeezebox_ courier_mta_mailqueue freeradius_proxy_auth loggrep nfs_client port_ psu_ snmp__netapp_inodeusage_ squid_cache courier_mta_mailstats fw_conntrack lpar_cpu nfsd postfix_mailqueue qmailqstat snmp__netstat squid_icp courier_mta_mailvolume fw_forwarded_local lpstat nfsd4 postfix_mailstats qmailscan snmp__print_pages squid_objectsizeJaktože mohu mít
Stale file handle pro soubory na lokálním filesystému. Znám to jen z NFS. A jak je možné to vyřešit?
# su -s /bin/bash munin bash-4.2$ /usr/share/munin/munin-update --debug --nofork 2017/03/05 13:09:17 [DEBUG] Creating new lock file /var/run/munin/munin-update.lock 2017/03/05 13:09:17 [DEBUG] Creating lock : /var/run/munin/munin-update.lock succeeded 2017/03/05 13:09:17 [INFO]: Starting munin-update 2017/03/05 13:09:17 [DEBUG] Lock /var/run/munin/munin-localhost-localhost.lock already exists, checking process 2017/03/05 13:09:17 [DEBUG] Lock contained pid '30317' 2017/03/05 13:09:17 [INFO] Process 30317 is dead, stealing lock, removing file 2017/03/05 13:09:17 [DEBUG] Creating new lock file /var/run/munin/munin-localhost-localhost.lock 2017/03/05 13:09:17 [DEBUG] Creating lock : /var/run/munin/munin-localhost-localhost.lock succeeded 2017/03/05 13:09:17 [DEBUG] Reading state for localhost-localhost in /var/lib/munin/state-localhost-localhost.storable 2017/03/05 13:09:17 [INFO] starting work in 30343 for localhost/127.0.0.1:4949. 2017/03/05 13:09:17 [INFO] node localhost advertised itself as localhost.localdomain instead. 2017/03/05 13:09:17 TLS set to "disabled". 2017/03/05 13:09:17 [DEBUG] Negotiating capabilities 2017/03/05 13:09:17 [DEBUG] Writing to socket: "cap multigraph dirtyconfig ". 2017/03/05 13:09:17 [DEBUG] Node says /cap multigraph dirtyconfig/ 2017/03/05 13:09:17 [DEBUG] Writing to socket: "list localhost.localdomain ". 2017/03/05 13:09:17 [DEBUG] for my open_inodes (open_inodes if_enp4s0 irqstats ntp_states df uptime swap load ntp_offset fw_conntrack cpu df_inode nfsd4 fw_forwarded_local open_files ntp_kernel_err forks diskstats memory vmstat if_enp6s1 fw_packets entropy ntp_kernel_pll_freq if_err_enp4s0 postfix_mailqueue processes users interrupts nfsd hddtemp_smartctl postfix_mailvolume proc_pri threads ntp_kernel_pll_off if_err_enp6s1) 2017/03/05 13:09:17 [DEBUG] Fetching service configuration for 'open_inodes' 2017/03/05 13:09:17 [DEBUG] Writing to socket: "config open_inodes ". 2017/03/05 13:09:17 [DEBUG] Reading from socket: "graph_title Inode table usage\ngraph_args --base 1000 -l 0\ngraph_vlabel number of open inodes\ngraph_category system\ngraph_info This graph monitors the Linux open inode table.\nused.label open inodes\nused.info The number of currently open inodes.\nmax.label inode table size\nmax.info The size of the system inode table. This is dynamically adjusted by the kernel.". 2017/03/05 13:09:17 [DEBUG] config: 0.075336 sec for 'open_inodes' on localhost/127.0.0.1/4949 2017/03/05 13:09:17 [DEBUG] Now parsing config output from plugin open_inodes on localhost 2017/03/05 13:09:17 [DEBUG] update_rate 0 for open_inodes on localhost/127.0.0.1:4949 2017/03/05 13:09:17 [DEBUG] No service data for open_inodes, fetching it 2017/03/05 13:09:17 [DEBUG] Writing to socket: "fetch open_inodes ". 2017/03/05 13:09:17 [DEBUG] data: 0.067993 sec for 'open_inodes' on localhost/127.0.0.1/4949 2017/03/05 13:09:17 [DEBUG] Now parsing fetch output from plugin open_inodes on localhost/127.0.0.1:4949 2017/03/05 13:09:17 [FETCH from open_inodes] used.value 73800 2017/03/05 13:09:17 [FETCH from open_inodes] Storing 73800 in used 2017/03/05 13:09:17 [FETCH from open_inodes] max.value 191015 2017/03/05 13:09:17 [FETCH from open_inodes] Storing 191015 in max 2017/03/05 13:09:17 [DEBUG] asking for a rrd of size : normal 2017/03/05 13:09:17 [DEBUG] rrd filename: /var/lib/munin/localhost/localhost-open_inodes-max-g.rrd 2017/03/05 13:09:17 [DEBUG] Updating /var/lib/munin/localhost/localhost-open_inodes-max-g.rrd with 1488715757:191015 Bus error bash-4.2$Porovnal jsem součty a chyby na FS jsou.
# btrfs scrub start -B /dev/dm-1
scrub done for d27abb02-b5d0-4eb3-86db-1b2d5a293fb2
scrub started at Sun Mar 5 12:52:09 2017 and finished after 00:01:38
total bytes scrubbed: 2.37GiB with 101 errors
error details: verify=7 csum=94
corrected errors: 0, uncorrectable errors: 101, unverified errors: 0
ERROR: there are uncorrectable errors
Takže teď jak to narovnat. Jak zjistit v jakých souborech mám chyby a případně je zrekonstruovat ze záloh.
[65218.369312] BTRFS: unable to fixup (regular) error at logical 1281064960 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 [65275.257209] scrub_handle_errored_block: 13 callbacks suppressed [65275.257395] BTRFS: checksum error at logical 10166935552 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28, sector 21970832, root 257, inode 345151, offset 255217664, length 4096, links 1 (path: crash/127.0.0.1-2017-01-29-12:26:46/vmcore) [65275.257408] btrfs_dev_stat_print_on_error: 13 callbacks suppressed [65275.257413] BTRFS: bdev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 errs: wr 0, rd 0, flush 0, corrupt 307, gen 28 [65275.264563] scrub_handle_errored_block: 13 callbacks suppressed [65275.264568] BTRFS: unable to fixup (regular) error at logical 10166935552 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 [65275.272016] BTRFS: checksum error at logical 10166976512 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28, sector 21970912, root 257, inode 345151, offset 255258624, length 4096, links 1 (path: crash/127.0.0.1-2017-01-29-12:26:46/vmcore) [65275.272028] BTRFS: bdev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 errs: wr 0, rd 0, flush 0, corrupt 308, gen 28 [65275.272149] BTRFS: checksum error at logical 10167107584 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28, sector 21971168, root 257, inode 345151, offset 255389696, length 4096, links 1 (path: crash/127.0.0.1-2017-01-29-12:26:46/vmcore) [65275.272161] BTRFS: bdev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 errs: wr 0, rd 0, flush 0, corrupt 309, gen 28 [65275.272164] BTRFS: unable to fixup (regular) error at logical 10167107584 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 [65275.292885] BTRFS: unable to fixup (regular) error at logical 10166976512 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 [65275.304940] BTRFS: checksum error at logical 10167111680 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28, sector 21971176, root 257, inode 345151, offset 255393792, length 4096, links 1 (path: crash/127.0.0.1-2017-01-29-12:26:46/vmcore) [65275.304971] BTRFS: bdev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 errs: wr 0, rd 0, flush 0, corrupt 310, gen 28 [65275.305901] BTRFS: checksum error at logical 10166980608 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28, sector 21970920, root 257, inode 345151, offset 255262720, length 4096, links 1 (path: crash/127.0.0.1-2017-01-29-12:26:46/vmcore) [65275.305912] BTRFS: bdev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 errs: wr 0, rd 0, flush 0, corrupt 311, gen 28 [65275.305915] BTRFS: unable to fixup (regular) error at logical 10166980608 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 [65275.306360] BTRFS: checksum error at logical 10166984704 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28, sector 21970928, root 257, inode 345151, offset 255266816, length 4096, links 1 (path: crash/127.0.0.1-2017-01-29-12:26:46/vmcore) [65275.306368] BTRFS: bdev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 errs: wr 0, rd 0, flush 0, corrupt 312, gen 28 [65275.306371] BTRFS: unable to fixup (regular) error at logical 10166984704 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 [65275.306775] BTRFS: checksum error at logical 10166988800 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28, sector 21970936, root 257, inode 345151, offset 255270912, length 4096, links 1 (path: crash/127.0.0.1-2017-01-29-12:26:46/vmcore) [65275.306782] BTRFS: bdev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 errs: wr 0, rd 0, flush 0, corrupt 313, gen 28 [65275.306784] BTRFS: unable to fixup (regular) error at logical 10166988800 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 [65275.307142] BTRFS: checksum error at logical 10166992896 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28, sector 21970944, root 257, inode 345151, offset 255275008, length 4096, links 1 (path: crash/127.0.0.1-2017-01-29-12:26:46/vmcore) [65275.307160] BTRFS: bdev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 errs: wr 0, rd 0, flush 0, corrupt 314, gen 28 [65275.307163] BTRFS: unable to fixup (regular) error at logical 10166992896 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 [65275.307507] BTRFS: checksum error at logical 10166996992 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28, sector 21970952, root 257, inode 345151, offset 255279104, length 4096, links 1 (path: crash/127.0.0.1-2017-01-29-12:26:46/vmcore) [65275.307513] BTRFS: bdev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 errs: wr 0, rd 0, flush 0, corrupt 315, gen 28 [65275.307516] BTRFS: unable to fixup (regular) error at logical 10166996992 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 [65275.307861] BTRFS: checksum error at logical 10167001088 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28, sector 21970960, root 257, inode 345151, offset 255283200, length 4096, links 1 (path: crash/127.0.0.1-2017-01-29-12:26:46/vmcore) [65275.307868] BTRFS: bdev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 errs: wr 0, rd 0, flush 0, corrupt 316, gen 28 [65275.307870] BTRFS: unable to fixup (regular) error at logical 10167001088 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28 [65275.308128] BTRFS: unable to fixup (regular) error at logical 10167005184 on dev /dev/mapper/luks-351589ee-7d82-4761-bdc3-652983743c28mám v dmesg
btrfs check --repair
--repair nestačilo --init-csum-tree vyřešilo součty. a ted jeden --init-extent-tree (zatím jede asi 6 hodin. CPU na 100% a iotop má 0 operací s diskem.) uvidíme co dále. Ted už je to spíše v režimu "chci zjistit jestli se to dá opravit" protože zálohu mám a FS který se rozbil je defakto prázdný systém. je tam jen operační systém a konfugurace a konfigurace je zazálohována. Takže asi nejrychlejší by to bylo reinstalovat a přes jedoucí systém obnovit zálohu.
btrfs check --init-extent-tree jede více než 2000 minut (to bylo ráno), což na datech které jsou 2,5 GB je skoto minutu na MB a s tím že CPU jen počítá. Netuším co dělá. Jak dorazím domů tak vše přeinstaluji.
Tiskni
Sdílej:
ISSN 1214-1267, (c) 1999-2007 Stickfish s.r.o.