On Tue, Jul 01, 2008 at 04:46:53PM +0300, Gregory Edigarov wrote:
Добрый день еще раз,
имем такую диагностику:
%% Low on memory; try again later
%% Low on memory; try again later
%% Low on memory; try again later
То есть - у нее тупо не хватает памяти... В результате, процесс ISDN не может аллоцировать себе очередной буфер и заваливает ее по software-forced reload. Рекомендации: - если 12.4 некритичен - попробовать откатиться до 12.3-stable, если критичен - "накатиться" до 12.4-stable, возможно, нехватка памяти - это последствия какого-то memory leak. - промониторить, кто из процессов жрет память, и, по возможности, отказаться от этого процесса. - добить кошку памятью хотя-бы до 256M. Это само по себе от leak'а не избавит (если это leak), но увеличит время между ребутами...
*Jan 1 13:30:58: %OSPF-5-ADJCHG: Process 1, Nbr 193.124.57.65 on FastEthernet0/ 0 from 2WAY to DOWN, Neighbor Down: Interface down or detached *Jan 1 13:30:58: %OSPF-5-ADJCHG: Process 1, Nbr 193.124.57.68 on FastEthernet0/ 0 from FULL to DOWN, Neighbor Down: Interface down or detached *Jan 1 13:30:58: %OSPF-5-ADJCHG: Process 1, Nbr 193.124.57.77 on FastEthernet0/ 0 from FULL to DOWN, Neighbor Down: Interface down or detached *Jan 1 13:30:58: %OSPF-5-ADJCHG: Process 1, Nbr 193.124.57.88 on FastEthernet0/ 0 from EXSTART to DOWN, Neighbor Down: Interface down or detached
%Software-forced reload
13:31:05 KHARKOV Sat Jan 1 2000: Breakpoint exception, CPU signal 23, PC = 0x60 5E51B8
-Traceback= 0x605E51B8 0x605E2EA8 0x600B9040 0x605A9764 0x605A9748 $0 : 00000000, AT : 62F00000, v0 : 63170000, v1 : 62CC0000 a0 : 00000000, a1 : 62F6EC30, a2 : 00000000, a3 : 62CE0000 t0 : 62F00480, t1 : 3400FF01, t2 : 62F00480, t3 : FFFF00FF t4 : 00001F00, t5 : 003F46B4, t6 : FFFFC0FF, t7 : 00000000 s0 : 630A0000, s1 : 00000000, s2 : 62CC0000, s3 : 62CD0000 s4 : 00000000, s5 : 00000000, s6 : 00000000, s7 : 00000000 t8 : 651A1920, t9 : 00000000, k0 : 64D77320, k1 : 605F43B8 gp : 62F07C80, sp : 651A1E98, s8 : 00000000, ra : 605E2EA8 EPC : 605E51B8, ErrorEPC : 605EA3E8, SREG : 3400FF03 MDLO : 00000000, MDHI : 00000000, BadVaddr : 00000020 CacheErr : 00000000, DErrAddr0 : 00000000, DErrAddr1 : 00000000 Cause 01000024 (Code 0x9): Breakpoint exception
Writing crashinfo to flash:crashinfo_20000101-113105
13:31:05 KHARKOV Sat Jan 1 2000: Breakpoint exception, CPU signal 23, PC = 0x60 5E51B8
-Traceback= 0x605E51B8 0x605E2EA8 0x600B9040 0x605A9764 0x605A9748 $0 : 00000000, AT : 62F00000, v0 : 63170000, v1 : 62CC0000 a0 : 00000000, a1 : 62F6EC30, a2 : 00000000, a3 : 62CE0000 t0 : 62F00480, t1 : 3400FF01, t2 : 62F00480, t3 : FFFF00FF t4 : 00001F00, t5 : 003F46B4, t6 : FFFFC0FF, t7 : 00000000 s0 : 630A0000, s1 : 00000000, s2 : 62CC0000, s3 : 62CD0000 s4 : 00000000, s5 : 00000000, s6 : 00000000, s7 : 00000000 t8 : 651A1920, t9 : 00000000, k0 : 64D77320, k1 : 605F43B8 gp : 62F07C80, sp : 651A1E98, s8 : 00000000, ra : 605E2EA8 EPC : 605E51B8, ErrorEPC : 605EA3E8, SREG : 3400FF03 MDLO : 00000000, MDHI : 00000000, BadVaddr : 00000020 CacheErr : 00000000, DErrAddr0 : 00000000, DErrAddr1 : 00000000 Cause 01000024 (Code 0x9): Breakpoint exception
-Traceback= 0x605E51B8 0x605E2EA8 0x600B9040 0x605A9764 0x605A9748
*Jan 1 13:31:04: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 80.92.224.20 sta rted - reconnection *Jan 1 13:31:05: %SYS-2-MALLOCFAIL: Memory allocation of 3348 bytes failed from 0x60136964, alignment 0 Pool: Processor Free: 2872 Cause: Not enough free memory Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "ISDN", ipl= 0, pid= 62 -Traceback= 0x604A6B2C 0x605FBA54 0x606046F8 0x6013696C 0x600D2670 0x600DFD50 0x 600D9780 0x600CFE2C 0x600D7490 0x600EBB7C *Jan 1 13:31:05: -Traceback= 0x60136980 0x600D2670 0x600DFD50 0x600D9780 0x600CFE2C 0x600D7490 0x600EBB7C *Jan 1 13:31:05: -Traceback= 0x60136664 0x601369B0 0x600D2670 0x600DFD50 0x600D9780 0x600CFE2C 0x600D7490 0x600EBB7C *Jan 1 13:31:05: %ISDN-3-ISDN_FATAL: ISDN FATAL ERROR: file lif_common.c, function LIF_GetPkt, message: malloc of Buffer failed *Jan 1 13:31:05: %ISDN-3-ISDN_EXIT: malloc of Buffer failed *Jan 1 13:31:05: -Traceback= 0x60136980 0x600C9D50 0x600CA67C *Jan 1 13:31:05: -Traceback= 0x60136664 0x601369B0 0x600C9D50 0x600CA67C *Jan 1 13:31:05: %ISDN-3-ISDN_FATAL: ISDN FATAL ERROR: file lif_common.c, function LIF_GetPkt, message: malloc of Buffer failed *Jan 1 13:31:05: %SCHED-2-EDISMSCRIT: Critical/high priority process ISDN L2 Process may not dismiss. -Process= "ISDN L2 Process", ipl= 0, pid= 171 -Traceback= 0x604A6B2C 0x605D256C 0x6016A9C8 0x600B9BC8 *Jan 1 13:31:09: -Traceback= 0x60136664 0x6005A908 0x6005B198 0x60068988 0x6006D2A0 0x6005C0E8 0x600B8284 *Jan 1 13:31:09: %ISDN-3-ISDN_FATAL: ISDN FATAL ERROR: file CCPRI, function CCPRI_GetOutInfo, message: Unable to get a package info buffer: 0 *Jan 1 13:31:09: -Traceback= 0x60136980 0x600C9D50 0x600CA67C *Jan 1 13:31:09: -Traceback= 0x60136664 0x601369B0 0x600C9D50 0x600CA67C *Jan 1 13:31:09: %ISDN-3-ISDN_FATAL: ISDN FATAL ERROR: file lif_common.c, function LIF_GetPkt, message: malloc of Buffer failed
=== Flushing messages (13:31:37 KHARKOV Sat Jan 1 2000) ===
Buffered messages: queue 20 -Process= "CC-API_VCM", ipl= 4, pid= 148 -Traceback= 0x604A6B2C 0x605FA4C0 0x6176DE40 0x6176437C 0x617644C8 0x6176D788 0x6176EE50 0x6177EB94 0x6177F198 0x612AB550 0x612ABBF0 0x612AC320 *Jan 1 13:12:25: -Traceback= 0x61764274 0x61764420 0x6176DE48 0x6176437C 0x617644C8 0x6176D788 0x6176EE50 0x6177EB94 0x6177F198 0x612AB550 0x612ABBF0 0x612AC320Attempt to Unlock/Free something that may not be an AFW instance or has already been freed *Jan 1 13:12:25: -Traceback= 0x6176443C 0x6176DE48 0x6176437C 0x617644C8 0x6176D788 0x6176EE50 0x6177EB94 0x6177F198 0x612AB550 0x612ABBF0 0x612AC320 *Jan 1 13:13:17: %ALIGN-3-SPURIOUS: Spurious memory access made at 0x612A9128 reading 0xF8 *Jan 1 13:13:17: %ALIGN-3-TRACE: -Traceback= 0x612A9128 0x612A9488 0x6176DE30 0x6176437C 0x617644C8 0x6176D788 0x6176EE50 0x6177EB94 *Jan 1 13:13:17: %ALIGN-3-TRACE: -Traceback= 0x612A9134 0x612A9488 0x6176DE30 0x6176437C 0x617644C8 0x6176D788 0x6176EE50 0x6177EB94 *Jan 1 13:13:17: %ALIGN-3-TRACE: -Traceback= 0x605BC644 0x612A9148 0x612A9488 0x6176DE30 0x6176437C 0x617644C8 0x6176D788 0x6176EE50 *Jan 1 13:13:17: %ALIGN-3-TRACE: -Traceback= 0x61C9D42C 0x617F6698 0x617F6F40 0x612A9198 0x612A9488 0x6176DE30 0x6176437C 0x617644C8 *Jan 1 13:13:17: %ALIGN-3-TRACE: -Traceback= 0x61C9D460 0x617F6698 0x617F6F40 0x612A9198 0x612A9488 0x6176DE30 0x6176437C 0x617644C8 *Jan 1 13:13:17: %ALIGN-3-TRACE: -Traceback= 0x612A9308 0x612A9488 0x6176DE30 0x6176437C 0x617644C8 0x6176D788 0x6176EE50 0x6177EB94 *Jan 1 13:13:17: %ALIGN-3-TRACE: -Traceback= 0x605FA404 0x6176DE40 0x6176437C 0x617644C8 0x6176D788 0x6176EE50 0x6177EB94 0x6177F198 *Jan 1 13:30:58: %OSPF-5-ADJCHG: Process 1, Nbr 193.124.57.65 on FastEthernet0/0 from 2WAY to DOWN, Neighbor Down: Interface down or detached *Jan 1 13:30:58: %OSPF-5-ADJCHG: Process 1, Nbr 193.124.57.68 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached *Jan 1 13:30:58: %OSPF-5-ADJCHG: Process 1, Nbr 193.124.57.77 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached *Jan 1 13:30:58: %OSPF-5-ADJCHG: Process 1, Nbr 193.124.57.88 on FastEthernet0/0 from EXSTART to DOWN, Neighbor Down: Interface down or detached *Jan 1 13:31:04: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 80.92.224.20 started - reconnection *Jan 1 13:31:05: %SYS-2-MALLOCFAIL: Memory allocation of 3348 bytes failed from 0x60136964, alignment 0 Pool: Processor Free: 2872 Cause: Not enough free memory Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "ISDN", ipl= 0, pid= 62 -Traceback= 0x604A6B2C 0x605FBA54 0x606046F8 0x6013696C 0x600D2670 0x600DFD50 0x600D9780 0x600CFE2C 0x600D7490 0x600EBB7C *Jan 1 13:31:05: -Traceback= 0x60136980 0x600D2670 0x600DFD50 0x600D9780 0x600CFE2C 0x600D7490 0x600EBB7C *Jan 1 13:31:05: -Traceback= 0x60136664 0x601369B0 0x600D2670 0x600DFD50 0x600D9780 0x600CFE2C 0x600D7490 0x600EBB7C *Jan 1 13:31:05: %ISDN-3-ISDN_FATAL: ISDN FATAL ERROR: file lif_common.c, function LIF_GetPkt, message: malloc of Buffer failed *Jan 1 13:31:05: %ISDN-3-ISDN_EXIT: malloc of Buffer failed *Jan 1 13:31:05: -Traceback= 0x60136980 0x600C9D50 0x600CA67C *Jan 1 13:31:05: -Traceback= 0x60136664 0x601369B0 0x600C9D50 0x600CA67C *Jan 1 13:31:05: %ISDN-3-ISDN_FATAL: ISDN FATAL ERROR: file lif_common.c, function LIF_GetPkt, message: malloc of Buffer failed *Jan 1 13:31:05: %SCHED-2-EDISMSCRIT: Critical/high priority process ISDN L2 Process may not dismiss. -Process= "ISDN L2 Process", ipl= 0, pid= 171 -Traceback= 0x604A6B2C 0x605D256C 0x6016A9C8 0x600B9BC8 *Jan 1 13:31:09: -Traceback= 0x60136664 0x6005A908 0x6005B198 0x60068988 0x6006D2A0 0x6005C0E8 0x600B8284 *Jan 1 13:31:09: %ISDN-3-ISDN_FATAL: ISDN FATAL ERROR: file CCPRI, function CCPRI_GetOutInfo, message: Unable to get a package info buffer: 0 *Jan 1 13:31:09: -Traceback= 0x60136980 0x600C9D50 0x600CA67C *Jan 1 13:31:09: -Traceback= 0x60136664 0x601369B0 0x600C9D50 0x600CA67C *Jan 1 13:31:09: %ISDN-3-ISDN_FATAL: ISDN FATAL ERROR: file lif_common.c, function LIF_GetPkt, message: malloc of Buffer failed Queued messages *** System received an unknown failure *** signal= 0x0, code= 0x0, context= 0x63166700 PC = 0x605ecea8, Cause Reg = 0x103c020, Status Reg = 0x34001b02 Interrupt Ctrl Reg = 0x10001082
System Bootstrap, Version 12.2(1r)1, RELEASE SOFTWARE (fc1) TAC Support: http://www.cisco.com/cgi-bin/ibld/view.pl?i=support Copyright (c) 2001 by cisco Systems, Inc. AS5350 platform with 131072 Kbytes of main memory
Self decompressing the image : ########################################################################################################################################################## [OK]
%SYS-4-CONFIG_NEWER: Configuration from version 12.4 may not be correctly understood %AAAA-4-SERVUNDEF: The server-group "radius" is not defined. Please define it. %AAAA-4-SERVUNDEF: The server-group "radius" is not defined. Please define it. %AAAA-4-SERVUNDEF: The server-group "radius" is not defined. Please define it. %AAAA-4-SERVUNDEF: The server-group "radius" is not defined. Please define it. %AAAA-4-SERVUNDEF: The server-group "radius" is not defined. Please define it. %SYS-6-BOOT_MESSAGES: Messages above this line are from the boot loader. Self decompressing the image : ############################################################################################################################################################################################################ [OK]
Restricted Rights Legend
Use, duplication, or disclosure by the Government is subject to restrictions as set forth in subparagraph (c) of the Commercial Computer Software - Restricted Rights clause at FAR sec. 52.227-19 and subparagraph (c) (1) (ii) of the Rights in Technical Data and Computer Software clause at DFARS sec. 252.227-7013.
cisco Systems, Inc. 170 West Tasman Drive San Jose, California 95134-1706
Cisco IOS Software, 5350 Software (C5350-IS-M), Version 12.4(5a), RELEASE SOFTWARE (fc3) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2006 by Cisco Systems, Inc. Compiled Fri 13-Jan-06 20:37 by alnguyen Image text-base: 0x60011084, data-base: 0x61D00000
Cisco AS5350 (R7K) processor (revision T) with 131072K/65536K bytes of memory. Processor board ID JAE07510LQU R7000 CPU at 250MHz, Implementation 39, Rev 1.0, 256KB L2, 2048KB L3 Cache Last reset from warm-reset Manufacture Cookie Info: EEPROM Type 0x0001, EEPROM Version 0x01, Board ID 0x32, Board Hardware Version 3.34, Item Number 800-5171-02, Board Revision C0, Serial Number JAE07510LQU, PLD/ISP Version 2.2, Manufacture Date 31-Dec-2003. Processor 0x14, MAC Address 000d.bd6d.777e Backplane HW Revision 1.0, Flash Type 5V 2 FastEthernet interfaces 6 Serial interfaces 60 terminal lines 2 Channelized E1/PRI ports 512K bytes of NVRAM. 32768K bytes of processor board System flash (Read/Write) 8192K bytes of processor board Boot flash (Read/Write)
Building configuration...
Press RETURN to get started!
*Jan 1 00:00:02.683: %NP_VPD-6-SLOT_INSERTED: Slot 1 (60 ports max) inserted *Jan 1 00:00:02.727: %NP_MD-6-SLOT_INSERTED: Slot 1 (60 ports max) inserted *Jan 1 02:00:17: %SYS-6-CLOCKUPDATE: System clock has been updated from 00:00:17 UTC Sat Jan 1 2000 to 02:00:17 KHARKOV Sat Jan 1 2000, configured from console by console. *Jan 1 02:00:17: %SYS-6-CLOCKUPDATE: System clock has been updated from 02:00:17 KHARKOV Sat Jan 1 2000 to 02:00:17 KHARKOV Sat Jan 1 2000, configured from console by console.
dial1>*Jan 1 02:00:17: %TRUNK_CLOCK-6-TRUNK: Switching to the trunk clock on slot 3 port 0 priority 1 *Jan 1 02:00:24: %SYS-5-CONFIG_I: Configured from memory by console *Jan 1 02:00:24: %NP_BS-6-MODULE_STARTED: NextPort module 1/0/0 Started - 6.27.10.31 *Jan 1 02:00:27: %SYS-5-RESTART: System restarted -- Cisco IOS Software, 5350 Software (C5350-IS-M), Version 12.4(5a), RELEASE SOFTWARE (fc3) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2006 by Cisco Systems, Inc. Compiled Fri 13-Jan-06 20:37 by alnguyen *Jan 1 02:00:27: %SNMP-5-COLDSTART: SNMP agent on host dial1 is undergoing a cold start *Jan 1 02:00:27: %SYS-6-BOOTTIME: Time taken to reboot after reload = 0 seconds *Jan 1 02:00:27: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 80.92.224.20 started - CLI initiated *Jan 1 02:00:28: %NP_BS-6-MODULE_STARTED: NextPort module 1/0/1 Started - 6.27.10.31 *Jan 1 02:00:32: %CONTROLLER-5-UPDOWN: Controller E1 3/0, changed state to up *Jan 1 02:00:32: %CSM-5-PRI: add PRI at slot 3, unit 0, channel 15 with index 0 *Jan 1 02:00:32: %CONTROLLER-5-UPDOWN: Controller E1 3/1, changed state to up *Jan 1 02:00:32: %CSM-5-PRI: add PRI at slot 3, unit 1, channel 15 with index 1 *Jan 1 02:00:33: %NP_VPD-6-MODULE_UP: NextPort module 1/0/0 up *Jan 1 02:00:33: %NP_MD-6-MODULE_UP: NextPort module 1/0/0 up *Jan 1 02:00:37: %NP_VPD-6-MODULE_UP: NextPort module 1/0/1 up *Jan 1 02:00:37: %NP_MD-6-MODULE_UP: NextPort module 1/0/1 up
Грешу на память, но для того, чтобы ее поменять, надо быть уверенным в том, что после того, как мы ее поменяем, ВСЕ будет точно ОК. или возможно есть другие причины, по которым оно может себя так вести?
Заранее очень благодарен.
-- With best regards, Gregory Edigarov