|
半个月来，新的网站服务器运行可靠性急遽下降，主要原因是jvm crash。而引起jvm crash的原因也都与java GC(Garbage Collection）相关。
查了一下sun的文档“Trouble-Shooting and
Diagnostic Guide",总算有了点初步的认识。检查4个jvm 崩溃时的文档(hs_err_pidXXXX)，原因完全一致:
An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x010d2d9e, pid=8089, tid=5417904
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_07-b03 mixed mode)
# Problematic frame:
# V [libjvm.so+0x3efd9e]
#
--------------- T H R E A D ---------------
Current thread (0x080e34a0): VMThread [id=8090]
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000058
Registers:
EAX=0x00000000, EBX=0x01292970, ECX=0x00670076, EDX=0xc82db880
ESP=0x00529d30, EBP=0x00529d48, ESI=0x01283c40, EDI=0x012a2094
EIP=0x010d2d9e, CR2=0x00000058, EFLAGS=0x00210216
Top of Stack: (sp=0x00529d30)
0x00529d30: 00670076 c82db880 0806bc60 0806bc60
0x00529d40: 01292970 0128a3a4 00529d68 010d30ac
0x00529d50: 0128a3a4 0047c800 0047aff4 00000000
0x00529d60: fffffff0 01292970 00529d98 011c2584
0x00529d70: 012a2094 0128a3a4 00529d98 011b3b91
0x00529d80: 005b7cb7 0047cbd0 011f0373 01292970
0x00529d90: 00000002 0806bb40 00529dd8 00ef1bc0
0x00529da0: 012a2094 00000000 00000001 00000001
Instructions: (pc=0x010d2d9e)
0x010d2d8e: 40 08 8b 14 88 8b 42 04 8d 48 08 8b 40 08 52 51
0x010d2d9e: ff 50 58 8b 06 83 c4 10 8b 08 85 c9 75 d9 eb d0
Stack: [0x004aa000,0x0052b000), sp=0x00529d30, free space=511k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x010d2d9e, pid=8089, tid=5417904
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_07-b03 mixed mode)
# Problematic frame:
# V [libjvm.so+0x3efd9e]
#
--------------- T H R E A D ---------------
Current thread (0x080e34a0): VMThread [id=8090]
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000058
Registers:
EAX=0x00000000, EBX=0x01292970, ECX=0x00670076, EDX=0xc82db880
ESP=0x00529d30, EBP=0x00529d48, ESI=0x01283c40, EDI=0x012a2094
EIP=0x010d2d9e, CR2=0x00000058, EFLAGS=0x00210216
Top of Stack: (sp=0x00529d30)
0x00529d30: 00670076 c82db880 0806bc60 0806bc60
0x00529d40: 01292970 0128a3a4 00529d68 010d30ac
0x00529d50: 0128a3a4 0047c800 0047aff4 00000000
0x00529d60: fffffff0 01292970 00529d98 011c2584
0x00529d70: 012a2094 0128a3a4 00529d98 011b3b91
0x00529d80: 005b7cb7 0047cbd0 011f0373 01292970
0x00529d90: 00000002 0806bb40 00529dd8 00ef1bc0
0x00529da0: 012a2094 00000000 00000001 00000001
Instructions: (pc=0x010d2d9e)
0x010d2d8e: 40 08 8b 14 88 8b 42 04 8d 48 08 8b 40 08 52 51
0x010d2d9e: ff 50 58 8b 06 83 c4 10 8b 08 85 c9 75 d9 eb d0
Stack: [0x004aa000,0x0052b000), sp=0x00529d30, free space=511k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
..............
VM_Operation (0x0612cf40): full generation collection, mode: safepoint, requested by thread 0x08a11b30
...............................
Heap
def new generation total 52480K, used 26959K [0xb6800000, 0xba0f0000, 0xbce30000)
eden space 46656K, 55% used [0xb6800000, 0xb811aa28, 0xb9590000)
from space 5824K, 21% used [0xb9590000, 0xb96c92f0, 0xb9b40000)
to space 5824K, 0% used [0xb9b40000, 0xb9b40000, 0xba0f0000)
tenured generation total 466048K, used 242672K [0xbce30000, 0xd9550000, 0xf0000000)
the space 466048K, 52% used [0xbce30000, 0xcbb2c398, 0xcbb2c400, 0xd9550000)
compacting perm gen total 50432K, used 50281K [0xf0000000, 0xf3140000, 0xf4000000)
the space 50432K, 99% used [0xf0000000, 0xf311a608, 0xf311a800, 0xf3140000)
No shared spaces configured.
.......................
再查jboss的文档，初步判断jvm崩溃的原因是permsize不足所致。因为系统大量使用了spring beans，jvm需要较多地使用 Permanent Generation Heap来存储reflective data。
解决方案是：在JAVA_OPTS加上-XX:MaxPermSize=128m，jdk5.0默认的初始值为8Mb(client)/16Mb(server),最大值为64Mb.
再使用jmap观察调整的效果：
cd /opt/jdk/bin
./jmap -heap 9657
Attaching to process ID 9657, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.5.0_07-b03
using thread-local object allocation.
Mark Sweep Compact GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 964689920 (920.0MB)
NewSize = 655360 (0.625MB)
MaxNewSize = 4294901760 (4095.9375MB)
OldSize = 1441792 (1.375MB)
NewRatio = 8
SurvivorRatio = 8
PermSize = 67108864 (64.0MB)
MaxPermSize = 134217728 (128.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 96468992 (92.0MB)
used = 4733136 (4.5138702392578125MB)
free = 91735856 (87.48612976074219MB)
4.906380694845448% used
Eden Space:
capacity = 85786624 (81.8125MB)
used = 4733136 (4.5138702392578125MB)
free = 81053488 (77.29862976074219MB)
5.517335662958365% used
From Space:
capacity = 10682368 (10.1875MB)
used = 0 (0.0MB)
free = 10682368 (10.1875MB)
0.0% used
To Space:
capacity = 10682368 (10.1875MB)
used = 0 (0.0MB)
free = 10682368 (10.1875MB)
0.0% used
tenured generation:
capacity = 857538560 (817.8125MB)
used = 193104168 (184.15848541259766MB)
free = 664434392 (633.6540145874023MB)
22.51842389454767% used
Perm Generation:
capacity = 67108864 (64.0MB)
used = 45904968 (43.77838897705078MB)
free = 21203896 (20.22161102294922MB)
68.40373277664185% used
现在Perm Generation只是使用了68%，而之前4次jvm崩溃是，该值均为99%.
新服务器的可用性问题是否因此得到解决，还有待观察。
|
|
