Monday, August 15, 2011

trace core file by STACKX script

I posted "How to debug core dump?". I felt curious what I can do with core dump file from oracle's process.
Thank You Roger Snowden. He suggested me stackx script. STACKX User Guide [ID 362791.1]

stackx is a shell script utility for extracting diagnostic stack trace information from core file on UNIX/LINUX platforms.

I downloaded and tested it(just test on 10g with real core dump file).
$ ls stackx1_2.tar
stackx1_2.tar

$ tar xf stackx1_2.tar

$ ls stackx
stackx

$ ./stackx
Wrong number of parameters.
Usage: stackx [debugger] <corefilename>


$ ./stackx /oracle/product/admin/orcl/cdump/core_29158/core.29158
######################################################################
stackx core / stack trace extraction utility
version 1.2 (c) 2005, 2006, 2007 Oracle Corp. all rights reserved.
Reporting: Mon Aug 15 00:49:24 ICT 2011
Machine : Linux corcl01 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:32:02 EDT 2006 x86_64 x86_64 x86_64 GNU/Linux
CPU Bits : 32
Core file: /oracle/product/admin/orcl/cdump/core_29158/core.29158
Core date: Oct 16 2010
Tool(s) : /usr/bin/gdb
Program : oracle
Core extraction section follows
######################################################################
For: /oracle/product/10.2.0/db_1/bin/oracle executable file found in core,
a core extraction will be attempted...
<debugging session>
(no debugging symbols found)
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
Core was generated by `ora_j002_orcl1'.
Program terminated with signal 6, Aborted.
Reading symbols from /oracle/product/10.2.0/db_1/lib/libskgxp10.so...(no debugging symbols found)...done.
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libskgxp10.so
Reading symbols from /oracle/product/10.2.0/db_1/lib/libhasgen10.so...done.
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libhasgen10.so
Reading symbols from /oracle/product/10.2.0/db_1/lib/libskgxn2.so...done.
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libskgxn2.so
Reading symbols from /oracle/product/10.2.0/db_1/lib/libocr10.so...done.
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libocr10.so
Reading symbols from /oracle/product/10.2.0/db_1/lib/libocrb10.so...done.
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libocrb10.so
Reading symbols from /oracle/product/10.2.0/db_1/lib/libocrutl10.so...done.
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libocrutl10.so
Reading symbols from /oracle/product/10.2.0/db_1/lib/libjox10.so...done.
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libjox10.so
Reading symbols from /oracle/product/10.2.0/db_1/lib/libclsra10.so...done.
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libclsra10.so
Reading symbols from /oracle/product/10.2.0/db_1/lib/libdbcfg10.so...done.
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libdbcfg10.so
Reading symbols from /oracle/product/10.2.0/db_1/lib/libnnz10.so...done.
Loaded symbols for /oracle/product/10.2.0/db_1/lib/libnnz10.so
Reading symbols from /usr/lib64/libaio.so.1...done.
Loaded symbols for /usr/lib64/libaio.so.1
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/tls/libm.so.6...done.
Loaded symbols for /lib64/tls/libm.so.6
Reading symbols from /lib64/tls/libpthread.so.0...done.
Loaded symbols for /lib64/tls/libpthread.so.0
Reading symbols from /lib64/libnsl.so.1...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/tls/libc.so.6...done.
Loaded symbols for /lib64/tls/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /opt/oracle/extapi/64/asm/orcl/1/libasm.so...done.
Loaded symbols for /opt/oracle/extapi/64/asm/orcl/1/libasm.so
#0 0x000000328642e829 in kill () from /lib64/tls/libc.so.6
(gdb) (gdb) <stack>

Thread 1 (process 29158):
#0 0x000000328642e829 in kill () from /lib64/tls/libc.so.6
#1 0x00000000022e21e4 in slcra ()
#2 0x0000000000745aab in ssexhd ()
#3 <signal handler called>
#4 0x0000000002323312 in kpnfch ()
#5 0x0000000002390911 in opifch2 ()
#6 0x000000000238e700 in opifch ()
#7 0x00000000023bc6ce in opipls ()
#8 0x000000000074c338 in opiodr ()
#9 0x0000000000751aea in rpidrus ()
#10 0x0000000003c69316 in skgmstack ()
#11 0x0000000000751e34 in rpidru ()
#12 0x0000000000750f2c in rpiswu2 ()
#13 0x00000000007507bd in rpidrv ()
#14 0x00000000031cbe56 in psddr0 ()
#15 0x00000000031cbbdb in psdnal ()
#16 0x0000000004573411 in pevm_BFTCHC ()
#17 0x0000000004529f05 in pfrinstr_FTCHC ()
#18 0x00000000045248c9 in pfrrun_no_tool ()
#19 0x0000000004523312 in pfrrun ()
#20 0x000000000456086d in plsql_run ()
#21 0x000000000450c7bc in peicnt ()
#22 0x0000000003fc1d50 in kkxexe ()
#23 0x00000000023b3f93 in opiexe ()
#24 0x0000000002325c2c in kpoal8 ()
#25 0x000000000074c338 in opiodr ()
#26 0x0000000002359999 in kpoodr ()
#27 0x00000000038b315e in upirtrc ()
#28 0x000000000382a2a3 in kpurcsc ()
#29 0x00000000037d3266 in kpuexecv8 ()
#30 0x00000000037d0a67 in kpuexec ()
#31 0x00000000038858ef in OCIStmtExecute ()
#32 0x0000000001cd8e65 in jslvec_execcb ()
#33 0x0000000001cd1792 in jslvswu ()
#34 0x0000000001cc84a2 in jslve_execute0 ()
#35 0x0000000001cc7936 in jslve_execute ()
#36 0x0000000000750f2c in rpiswu2 ()
#37 0x0000000001c297cf in kkjex1e ()
#38 0x0000000001c28f17 in kkjsexe ()
#39 0x0000000001c284c4 in kkjrdp ()
#40 0x0000000002380a18 in opirip ()
#41 0x0000000000746086 in opidrv ()
#42 0x000000000074450e in sou2o ()
#43 0x000000000070a565 in opimai_real ()
#44 0x000000000070a41c in main ()
#0 0x000000328642e829 in kill () from /lib64/tls/libc.so.6
#1 0x00000000022e21e4 in slcra ()
#2 0x0000000000745aab in ssexhd ()
#3 <signal handler called>
#4 0x0000000002323312 in kpnfch ()
#5 0x0000000002390911 in opifch2 ()
#6 0x000000000238e700 in opifch ()
#7 0x00000000023bc6ce in opipls ()
#8 0x000000000074c338 in opiodr ()
#9 0x0000000000751aea in rpidrus ()
#10 0x0000000003c69316 in skgmstack ()
#11 0x0000000000751e34 in rpidru ()
#12 0x0000000000750f2c in rpiswu2 ()
#13 0x00000000007507bd in rpidrv ()
#14 0x00000000031cbe56 in psddr0 ()
#15 0x00000000031cbbdb in psdnal ()
#16 0x0000000004573411 in pevm_BFTCHC ()
#17 0x0000000004529f05 in pfrinstr_FTCHC ()
#18 0x00000000045248c9 in pfrrun_no_tool ()
#19 0x0000000004523312 in pfrrun ()
#20 0x000000000456086d in plsql_run ()
#21 0x000000000450c7bc in peicnt ()
#22 0x0000000003fc1d50 in kkxexe ()
#23 0x00000000023b3f93 in opiexe ()
#24 0x0000000002325c2c in kpoal8 ()
#25 0x000000000074c338 in opiodr ()
#26 0x0000000002359999 in kpoodr ()
#27 0x00000000038b315e in upirtrc ()
#28 0x000000000382a2a3 in kpurcsc ()
#29 0x00000000037d3266 in kpuexecv8 ()
#30 0x00000000037d0a67 in kpuexec ()
#31 0x00000000038858ef in OCIStmtExecute ()
#32 0x0000000001cd8e65 in jslvec_execcb ()
#33 0x0000000001cd1792 in jslvswu ()
#34 0x0000000001cc84a2 in jslve_execute0 ()
#35 0x0000000001cc7936 in jslve_execute ()
#36 0x0000000000750f2c in rpiswu2 ()
#37 0x0000000001c297cf in kkjex1e ()
#38 0x0000000001c28f17 in kkjsexe ()
#39 0x0000000001c284c4 in kkjrdp ()
#40 0x0000000002380a18 in opirip ()
#41 0x0000000000746086 in opidrv ()
#42 0x000000000074450e in sou2o ()
#43 0x000000000070a565 in opimai_real ()
#44 0x000000000070a41c in main ()
</stack>
<registers>
rax 0x0 0
rbx 0x1 1
rcx 0xffffffffffffffff -1
rdx 0x71e6 29158
rsi 0x6 6
rdi 0x71e6 29158
rbp 0x2a9715aa40 0x2a9715aa40
rsp 0x2a9715a2d8 0x2a9715a2d8
r8 0x0 0
r9 0x0 0
r10 0x2a9715a201 182923403777
r11 0x206 518
r12 0xbdb71760 3182892896
r13 0x7 7
r14 0x7b 123
r15 0x2a9715ac40 182923406400
rip 0x328642e829 0x328642e829 <kill+9>
eflags 0x206 518
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
st0 0 (raw 0x00000000000000000000)
st1 0 (raw 0x00000000000000000000)
st2 0 (raw 0x00000000000000000000)
st3 0 (raw 0x00000000000000000000)
st4 0 (raw 0x00000000000000000000)
st5 0 (raw 0x00000000000000000000)
st6 0 (raw 0x00000000000000000000)
st7 0 (raw 0x00000000000000000000)
fctrl 0x37f 895
fstat 0x0 0
ftag 0xffff 65535
fiseg 0x0 0
fioff 0x0 0
foseg 0x0 0
fooff 0x0 0
fop 0x0 0
xmm0 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm1 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm2 {f = {0x0, 0x0, 0x0, 0x0}} {f = {4.34402524e-44, 3.36311631e-44, 0, 0}}
xmm3 {f = {0x0, 0x0, 0x0, 0x0}} {f = {3.50324616e-44, 8.40779079e-45, 0, 0}}
xmm4 {f = {0x0, 0x0, 0xc, 0x0}} {f = {6.40969056e-10, 1.14785479e-08, 12.3875132, 2.05823494e-06}}
xmm5 {f = {0x304d, 0x0, 0x31, 0x0}} {f = {12365.0557, 0.00067597907, 49.5529366, 7.21062543e-10}}
xmm6 {f = {0x0, 0x0, 0x0, 0x0}} {f = {1.35631564e-19, 1.3571458e-19, 4.12584313e-08, 6.40982378e-10}}
xmm7 {f = {0x0, 0xa03420, 0x81314c00, 0x0}} {f = {1.35701254e-19, 10499104, 1.07574262e+10, 1.7984663e-19}}
xmm8 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm9 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm10 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm11 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm12 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm13 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm14 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
xmm15 {f = {0x0, 0x0, 0x0, 0x0}} {f = {0, 0, 0, 0}}
mxcsr 0x1f80 8064
</registers>
</debugging session>
######################################################################
stackx done.
Note:
Core stack trace extraction is supported on the following platforms:
- IBM AIX
- Intel Linux (RedHat and SuSE)
- HP-UX (10.* and 11.*)
- HP Tru64
- Sun Solaris (2.5 - 2.10)

Debuggers recognized and discovered by stackx are, in order of preferential priority:
- pstack (Solaris only)
- gdb
- dbx
- adb
- sdb

It's easy with this script.

No comments: