升级11.1.0.7后启动数据库报错error 27103
感谢：http://www.dbsupport.cn/thread-372-1-1.html
?
升级到11.1.0.7.12启动数据库报错：
OS版本：suse linux X86-64
DB版本：11.1.0.7.8 升级到11.1.0.7.12
?
Total System Global Area 1.2560E+10 bytes
Fixed Size ? ? ? ? ? ? ? ? ?2171344 bytes
Variable Size ? ? ? ? ? ?7918849584 bytes
Database Buffers ? ? ? ? 4630511616 bytes
Redo Buffers ? ? ? ? ? ? ? ?8601600 bytes
ORA-03113: end-of-file on communication channel
Process ID: 13056
Session ID: 1105 Serial number: 3
?
alert日志如下：
oracle@PGM_DB1:/oracle/app/diag/rdbms/ora11g/ora11g/trace> tail -100 alert_ora11g.log |more
RECO started with pid=13, OS id=9038?
Tue Dec 10 14:49:52 2013
MMON started with pid=14, OS id=9040?
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
Tue Dec 10 14:49:52 2013
MMNL started with pid=15, OS id=9042?
starting up 1 shared server(s) ...
ORACLE_BASE from environment = /oracle/app
Tue Dec 10 14:49:52 2013
ALTER DATABASE ? MOUNT
Errors in file /oracle/app/diag/rdbms/ora11g/ora11g/trace/ora11g_mman_9025.trc:
ORA-27103: internal error
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: -1
Additional information: 1
MMAN (ospid: 9025): terminating the instance due to error 27103
Tue Dec 10 14:49:56 2013
ORA-1092 : opitsk aborting process
Tue Dec 10 14:49:56 2013
ORA-1092 : opitsk aborting process
Tue Dec 10 14:49:56 2013
ORA-1092 : opitsk aborting process
Instance terminated by MMAN, pid = 9025
Tue Dec 10 14:50:24 2013
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.?
IMODE=BR
ILAT =121
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 11.1.0.7.0.
Using parameter settings in server-side pfile /oracle/app/product/11g/db/dbs/initora11g.ora
System parameters with non-default values:
? processes ? ? ? ? ? ? ? ?= 1000
? spfile ? ? ? ? ? ? ? ? ? = "/dev/vx/rdsk/vgora/lvspfile"
? memory_target ? ? ? ? ? ?= 12032M
? control_files ? ? ? ? ? ?= "/dev/vx/rdsk/vgora/lvctl1"
? control_files ? ? ? ? ? ?= "/dev/vx/rdsk/vgora/lvctl2"
? control_files ? ? ? ? ? ?= "/dev/vx/rdsk/vgora/lvctl3"
? db_block_size ? ? ? ? ? ?= 8192
? compatible ? ? ? ? ? ? ? = "11.1.0.0.0"
? log_archive_dest_1 ? ? ? = "location=/home/oracle/archive"
? db_recovery_file_dest ? ?= ""
? db_recovery_file_dest_size= 2G
? undo_tablespace ? ? ? ? ?= "UNDOTBS1"
? sec_case_sensitive_logon = FALSE
? remote_login_passwordfile= "EXCLUSIVE"
? db_domain ? ? ? ? ? ? ? ?= ""
? dispatchers ? ? ? ? ? ? ?= "(PROTOCOL=TCP) (SERVICE=ora11gXDB)"
? local_listener ? ? ? ? ? = ""
? remote_listener ? ? ? ? ?= ""
? audit_file_dest ? ? ? ? ?= "/oracle/app/admin/ora11g/adump"
? audit_trail ? ? ? ? ? ? ?= "NONE"
? db_name ? ? ? ? ? ? ? ? ?= "ora11g"
? open_cursors ? ? ? ? ? ? = 300
? diagnostic_dest ? ? ? ? ?= "/oracle/app"
Tue Dec 10 14:50:26 2013
PMON started with pid=2, OS id=13015?
Tue Dec 10 14:50:26 2013
VKTM started with pid=3, OS id=13017?
VKTM running at (100ms) precision?
Tue Dec 10 14:50:26 2013
DIAG started with pid=4, OS id=13024?
Tue Dec 10 14:50:26 2013
DBRM started with pid=5, OS id=13026?
Tue Dec 10 14:50:26 2013
PSP0 started with pid=6, OS id=13028?
Tue Dec 10 14:50:26 2013
DIA0 started with pid=7, OS id=13030?
Tue Dec 10 14:50:26 2013
MMAN started with pid=8, OS id=13032?
Tue Dec 10 14:50:26 2013
DBW0 started with pid=9, OS id=13034?
Tue Dec 10 14:50:26 2013
LGWR started with pid=10, OS id=13036?
Tue Dec 10 14:50:26 2013
CKPT started with pid=11, OS id=13038?
Tue Dec 10 14:50:26 2013
SMON started with pid=12, OS id=13040?
Tue Dec 10 14:50:26 2013
RECO started with pid=13, OS id=13042?
Tue Dec 10 14:50:26 2013
MMON started with pid=14, OS id=13044?
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
Tue Dec 10 14:50:26 2013
MMNL started with pid=15, OS id=13046?
starting up 1 shared server(s) ...
ORACLE_BASE from environment = /oracle/app
Tue Dec 10 14:50:26 2013
ALTER DATABASE ? MOUNT
Errors in file /oracle/app/diag/rdbms/ora11g/ora11g/trace/ora11g_mman_13032.trc:
ORA-27103: internal error
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: -1
Additional information: 1
MMAN (ospid: 13032): terminating the instance due to error 27103
Instance terminated by MMAN, pid = 13032
?
trc文件如下：
oracle@PGM_DB1:/oracle/app/diag/rdbms/ora11g/ora11g/trace> more /oracle/app/diag/rdbms/ora11g/ora11g/trace/ora11g_mman_13032.trc
Trace file /oracle/app/diag/rdbms/ora11g/ora11g/trace/ora11g_mman_13032.trc
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP, Data Mining,
Oracle Database Vault and Real Application Testing options
ORACLE_HOME = /oracle/app/product/11g/db
System name: ? ?Linux
Node name: ? ? ?PGM_DB1
Release: ? ? ? ?2.6.32.45-0.3.2.4445.1.PTF-default
Version: ? ? ? ?#1 SMP 2011-08-22 10:12:58 +0200
Machine: ? ? ? ?x86_64
Instance name: ora11g
Redo thread mounted by this instance: 0 <none>
Oracle process number: 8
Unix process pid: 13032, image: oracle@PGM_DB1 (MMAN)
?
*** 2013-12-10 14:50:29.525
*** SESSION ID:(1098.1) 2013-12-10 14:50:29.525
*** CLIENT ID:() 2013-12-10 14:50:29.525
*** SERVICE NAME:() 2013-12-10 14:50:29.525
*** MODULE NAME:() 2013-12-10 14:50:29.525
*** ACTION NAME:() 2013-12-10 14:50:29.525
?
error 27103 detected in background process
ORA-27103: internal error
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: -1
Additional information: 1
?
*** 2013-12-10 14:50:29.525
MMAN (ospid: 13032): terminating the instance due to error 27103
?
?
结合alert和trace文件查询MOS，发现ORA-27103 when Memory target parameter is set to more than 3 GB [ID 743012.1]描述相符，是由于Bug:7272646引起.
?
打补丁：
?
oracle@PGM_DB1:~/psu> cd 7272646/
oracle@PGM_DB1:~/psu/7272646>?
oracle@PGM_DB1:~/psu/7272646> $ORACLE_HOME/OPatch/opatch apply
Oracle Interim Patch Installer version 11.1.0.9.10
Copyright (c) 2012, Oracle Corporation. ?All rights reserved.
?
?
Oracle Home ? ? ? : /oracle/app/product/11g/db
Central Inventory : /oracle/app/oraInventory
? ?from ? ? ? ? ? : /oracle/app/product/11g/db/oraInst.loc
OPatch version ? ?: 11.1.0.9.10
OUI version ? ? ? : 11.1.0.7.0
Log file location : /oracle/app/product/11g/db/cfgtoollogs/opatch/7272646_Dec_10_2013_14_59_02/apply2013-12-10_14-59-01PM_1.log
?
Applying interim patch '7272646' to OH '/oracle/app/product/11g/db'
Verifying environment and performing prerequisite checks...
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:?
?
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: ?
Email address/User Name:?
?
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: ?
Email address/User Name:?
?
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: ?y
?
?
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/oracle/app/product/11g/db')
?
?
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
?
Patching component oracle.rdbms, 11.1.0.7.0...
?
Verifying the update...
Patch 7272646 successfully applied
Log file location: /oracle/app/product/11g/db/cfgtoollogs/opatch/7272646_Dec_10_2013_14_59_02/apply2013-12-10_14-59-01PM_1.log
?
OPatch succeeded.
?
?
?
重启数据库成功。
?
至此数据库升级完成。但应用侧反馈使用JDBC无法连接数据库。
检查监听状态，和使用tnsname连接数据库后，确认数据库，监听无问题。
?
应用侧的报错如下：
errdesc = libclntsh.so : cannot open shared object file :No such file or directory
?
检查libclntsh.so 是$ORACLE_HOME/lib，和$ORACLE_HOME/lib32里面的一个连接文件
链接到libclntsh.so.11.1。升级前此文件的权限是755，此时的权限是700.修改权限后应用正常。
?
应用的研发说这个和oracle的umask有关。
umaks是用来给新建文件的赋予初始化权限的。
如，本次升级的oracle的umask是0077，用777-077后，新建文件的权限就是700了。
?
?
