WAS数据收集及简略CPU占用高分析

WAS数据收集及简单CPU占用高分析
1. 收集完整的数据信息
  参考链接 - MustGather: Performance, hang, or high CPU issues with WebSphere Application Server on AIX

2. 脚本运行结果和日志文件有

aixperf_RESULTS.tar.gz（本地目录生成）three javacores （$WebSphere/profiles/AppSrv01/）server logs (SystemOut.log, native_stderr.log,..../logs/server/)

3. 解压aixperf_RESULTS.tar.gz，打开sleep.prof，其中有占用CPU高的进程和线程列表

Process                                Freq  Total Kernel   User Shared  Other=======                                ====  ===== ======   ==== ======  =====/usr/java5/jre/bin/java                 432  69.30   2.65   0.00   2.01  64.64wait                                     32  29.80  29.80   0.00   0.00   0.00/usr/bin/runmqsc                         96   0.45   0.18   0.00   0.27   0.00/usr/bin/tprof                            1   0.23   0.01   0.19   0.02   0.00vmmd                                      1   0.08   0.08   0.00   0.00   0.00...

Process                   PID      TID  Total Kernel   User Shared  Other=======                   ===      ===  ===== ======   ==== ======  =====r/java5/jre/bin/java  7012560 109772901   4.99   0.01   0.00   0.00   4.98r/java5/jre/bin/java  7012560 85065811   4.92   0.01   0.00   0.00   4.91r/java5/jre/bin/java  7012560 28770661   4.71   0.00   0.00   0.01   4.69r/java5/jre/bin/java  7012560  5439747   4.65   0.00   0.00   0.01   4.63r/java5/jre/bin/java  7012560 78970915   4.49   0.01   0.00   0.03   4.45r/java5/jre/bin/java  7012560 35913987   4.41   0.01   0.00   0.01   4.40r/java5/jre/bin/java  7012560 111083605   4.21   0.01   0.00   0.00   4.20r/java5/jre/bin/java  7012560 87490635   4.04   0.01   0.00   0.01   4.03r/java5/jre/bin/java  7012560 63635485   4.00   0.00   0.00   0.00   4.00r/java5/jre/bin/java  7012560 86376471   3.83   0.00   0.00   0.02   3.81r/java5/jre/bin/java  7012560 74842269   3.83   0.00   0.00   0.00   3.83r/java5/jre/bin/java 11337904 72351955   3.73   0.01   0.00   0.02   3.70r/java5/jre/bin/java  7012560 100335747   3.61   0.00   0.00   0.01   3.60...

本例中7012560和11337904进程号对应的进程是WAS的两个server，可以看到其中一个7012560进程，所开的java线程占用CPU十分高，基本占3.5%以上。记下其中的TID号109772901（Thread ID），此进程号是十进制，转化为十六进制后为0x68B0065。

4. 在对应进程号的javacore文件中，搜索该十六进制进程号0x68B0065，即可查出该线程操作的堆栈数

"batchImportQuertz_Worker-9" J9VMThread:0x0000000032E9AE00, j9thread_t:0x000000012BAB1FA0, java/lang/Thread:0x0000000719D73DA8, state:CW, prio=5       (native thread ID:0x68B0065, native priority:0x5, native policy:UNKNOWN)      Java callstack:          at com/app/xxx/JobMonitorCommonService.triggerComplete(JobMonitorCommonService.java:328(Compiled Code))          at com/app/xxx/XXX.executeInternal(XXX.java:37)          at org/springframework/scheduling/xxx/XXX.execute(QuartzJobBean.java:86)          at org/xxx/core/JobRunShell.run(JobRunShell.java:216)          at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549(Compiled Code))      Native callstack:          _event_wait+0x2b8 (0x0900000000A9E9BC [libpthreads.a+0x169bc])          _cond_wait_local+0x4e4 (0x0900000000AAC768 [libpthreads.a+0x24768])          _cond_wait+0xbc (0x0900000000AACD40 [libpthreads.a+0x24d40])          pthread_cond_wait+0x1a8 (0x0900000000AAD9AC [libpthreads.a+0x259ac])          (0x0900000002A5B17C [libj9thr24.so+0x417c])          (0x0900000002A5AF40 [libj9thr24.so+0x3f40])          (0x0900000002A5AEE0 [libj9thr24.so+0x3ee0])          (0x09000000029DDB6C [libj9vm24.so+0x10b6c])          (0x09000000029E5AE8 [libj9vm24.so+0x18ae8])          (0x090000000305C2C8 [libj9jit24.so+0x54d2c8])          (0x09000000029D8E10 [libj9vm24.so+0xbe10])          (0x0900000002A6CF94 [libj9prt24.so+0x1f94])          (0x09000000029D8D30 [libj9vm24.so+0xbd30])          (0x0900000002A58C70 [libj9thr24.so+0x1c70])          _pthread_body+0xf0 (0x0900000000A8BD54 [libpthreads.a+0x3d54])

查看更多 下一篇