oracle 11.2.0.2 RAC for HPUX.pdf

Version 변경일자 변경자(작성자) 주요내용 1 2011-01-30 조 훈 문서 최초 작성 Author 조 훈 Creation Date 2011-01-30 Last Updated Version 1.0 Copyright(C) 2004 Goodus Inc. All Rights Reserved ORACLE 11gR2 (11.2.0.2) RAC install for HP-UX ORACLE 10g RAC install patch failover ON DELL - 2 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. Contents 1. 들어가며 ............................................................................................................................... 3 2. System Overview ................................................................................................................ 3 2.1. H/W and OS Information .........................................................................................3 2.1.1. Hardware ..................................................................................................................... 3 2.1.2. OS ............................................................................................................................... 3 2.1.3. Shared Storage ........................................................................................................... 3 2.2. IP 구성정보 ...............................................................................................................4 2.3. OS USER 정보 ...........................................................................................................4 2.4. 패키지 .......................................................................................................................4 3. Post Installation Steps ....................................................................................................... 6 3.1. Preparing the system ...............................................................................................6 3.1.1. Host file Configuration ................................................................................................. 6 3.1.2. Kernel Configuration ................................................................................................... 6 3.1.3. Check pach ................................................................................................................. 7 3.1.4. RSH Configuration ...................................................................................................... 7 3.1.5. Oracle User Limits Value Configuration ...................................................................... 7 3.1.6. User OS enviropment Configuration ........................................................................... 7 3.2. Network Configure Check ........................................................................................8 3.3. Raw Device for OCR, VOTEDISK ..............................................................................9 3.3.1. Raw device Creation ................................................................................................... 9 4. Install steps ...................................................................................................................... 10 4.1. Clusterware Installation ....................................................................................... 10 4.1.1. Clusterware Installation ............................................................................................. 10 4.1.2. OCR 변경 ................................................................................................................. 24 4.1.3. votedisk 변경 ............................................................................................................ 24 4.2. Database Installation ........................................................................................... 26 4.2.1. Change Permission ................................................................................................... 26 4.2.2. Database engine install ............................................................................................. 26 4.2.3. DBCA ........................................................................................................................ 34 4.2.4. ASM resource 제거................................................................................................... 46 4.3. Check Configuration ............................................................................................. 46 5. 설치 후 발견된 이슈 ........................................................................................................... 47 5.1. Error while reading ADR file after upgraded to 11.2.0.2 ...................................... 47 5.2. HP-UX Time Mismatch Between Ocssd.Log And Sysdate ..................................... 47 ORACLE 10g RAC install patch failover ON DELL - 3 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 1. 들어가며 이 내용은 HP-UX Itanium 에 Oracle 11.2.0.2 신규 설치에 대한 내용으로 ASM 이나 CFS 가 아닌 raw device 를 사용하여 RAC 구성을 목적으로 합니다. 2. System Overview 2.1. H/W and OS Information 2.1.1. Hardware 항 목 NODE1 ( goodus1 ) NODE2 ( goodus2 ) MODEL HP BL870c HP BL870c CPU 8CPU Quad Core 8CPU Quad Core MEMORY 32GB 32GB Swap 64GB 64GB 2.1.2. OS 항 목 NODE1 ( goodus1 ) NODE2 ( goodus2 ) OS HP-UX 11i v3 HP-UX 11i v3 Hostname goodus1 goodus2 Kernel B.11.31 U ia64 B.11.31 U ia64 2.1.3. Shared Storage 항 목 위치 SIZE OCR /dev/dbs20/rocr01, /dev/dbs20/rocr02 300M VOTE /dev/dbs20/rvot01 , /dev/dbs20/rvot02 , /dev/dbs21/rvote03 300M ASM /dev/rdisk/disk378 1,024M CONTROL /dev/dbs22/rcontrol1 , /dev/dbs22/rcontrol2 , /dev/dbs22/rcontrol3 300M SYSTEM /dev/dbs22/rsystem01 4,096M REDO (thread 1) /dev/dbs22/rredo111 , /dev/dbs22/rredo121 , /dev/dbs22/rredo131 , /dev/dbs22/rredo112 , /dev/dbs22/rredo122 , /dev/dbs22/rredo132 1,024M REDO (thread 2) /dev/dbs22/rredo211 , /dev/dbs22/rredo221 , /dev/dbs22/rredo231 , /dev/dbs22/rredo212 , /dev/dbs22/rredo222 , /dev/dbs22/rredo232 1,024M UNDOTBS1, 2 /dev/dbs23/rundotbs1 , /dev/dbs23/rundotbs2 10,240M TEMP /dev/dbs23/rtemp 1,024M USERS /dev/dbs22/ruser 2,048M ORACLE 10g RAC install patch failover ON DELL - 4 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. SYSAUX /dev/dbs22/rsysaux 10,240M SPFILE /dev/dbs22/rspfile 300M 2.2. IP 구성정보 항 목 NODE1 (goodus1) NODE2 (goodus2) 비 고 lan900 ***.***.***.58 ***.***.***.59 Public IP lan902 192.168.200.1 192.168.200.2 Interconnect IP lan900:1 ***.***.***.158 ***.***.***.159 Service IP lan900:2 ***.***.***.160 SCAN IP 2.3. OS USER 정보 USERNAME GROUP SHELL Home Direcoty PROFILE grid oinstall(700), dba(701), asmdba(702) /bin/ksh /oracle/app/grid .profile oracle oinstall(700), dba(701) /bin/ksh /oracle/app/oracle .profile 2.4. 패키지  Oracle 에서 권고하는 package 목록과 설치된 package 목록 HP-UX Operating System Itanium: » HP-UX 11iV3 patch Bundle Sep/ 2008 (B.11.31.0809.326a)  All installations for HP-UX 11i V3 (11.31) requires the following patches (or newer versions – the current successor status March 2010): » PHCO_40381 11.31 Disk Owner Patch » PHKL_38038 VM patch - hot patching/Core file creation directory → replaced by PHKL_40942 » PHKL_38938 11.31 SCSI cumulative I/O patch → replaced by PHKL_39646 » PHKL_39351 Scheduler patch : post wait hang → replaced by PHKL_40207 » PHSS_36354 11.31 assembler patch → replaced by PHSS_40546 » PHSS_37042 11.31 hppac (packed decimal) » PHSS_37959 Libcl patch for alternate stack issue fix (QXCR1000818011) → replaced by PHSS_40804 » PHSS_39094 11.31 linker + fdp cumulative patch ORACLE 10g RAC install patch failover ON DELL - 5 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. → replaced by PHSS_40538 » PHSS_39100 11.31 Math Library Cumulative Patch → replaced by PHSS_40540 » PHSS_39102 11.31 Integrity Unwind Library → replaced by PHSS_40542 » PHSS_38141 11.31 aC++ Runtime → replaced by PHSS_40544  Patch to run 11gR2 RAC on SG/SGeRAC » PHSS_40886 11.31 Serviceguard for RAC A.11.19.00  Pro*C/C++, Oracle Call Interface, Oracle C++ Call Interface, Oracle XML Developer’s Kit (XDK) Patch for HP-UX 11i V3 (11.31) on HP-UX Itanium: » PHSS_39824 - 11.31 HP C/aC++ Compiler (A.06.23) patch → Replaced by PHSS_40631  VERITAS File System » PHKL_39773: 11.31 VRTS 5.0 GARP6 VRTSvxfs Kernel Patch → Replaced by PHKL_40650 ORACLE 10g RAC install patch failover ON DELL - 6 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 3. Post Installation Steps 3.1. Preparing the system 3.1.1. Host file Configuration /etc/hosts 파일에 각노드의 Pulic IP 주소 및 VIP, Interconnect 주소를 등록합니다. (root 사용자로 양쪽 노드 모두 수행합니다.) [root@goodus1 ~]# cat /etc/hosts 127.0.0.1 localhost.localdomain localhost ***.***.***.58 goodus1 ***.***.***.59 goodus2 ***.***.***.158 goodus1-vip ***.***.***.159 goodus2-vip 192.168.200.1 goodus1-priv 192.168.200.2 goodus2-priv ***.***.***.160 goodus-scan 3.1.2. Kernel Configuration Oracle 을 기동하기 위한 OS kernel 값 설정합니다. Parameter Minimum Required Value ksi_alloc_max 32768 executable_stack 0 maxfiles 1024 maxfiles_lim 63488 max_thread_proc 1024 maxdsiz 1073741824 (1 GB) maxdsiz_64bit 2147483648 (2 GB) maxssiz 134217728 (128 MB) maxssiz_64bit 1073741824 (1 GB) maxuprc 3686 msgmap 4096 msgmni 4096 msgtql 4096 ncsize 35840 nflocks 4096 ninode 34816 nkthread 7184 ORACLE 10g RAC install patch failover ON DELL - 7 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. nproc 4096 semmni 4096 semmns 8192 semmnu 4096 semvmx 32767 shmmax The size of memory or 1073741824 (0X40000000), whichever is greater. shmmni 4092 shmseg 512 3.1.3. Check pach OS 패치 설치 유무를 확인 합니다. (root 사용자로 양쪽 노드 모두 수행합니다.) # /usr/sbin/swlist -l patch 3.1.4. RSH Configuration RAC 설치를 위한 rsh 설정을 합니다. RAC 는 설치중 노드갂 Binary File 을 Copy 합니다. 이를 위해 rsh 나 ssh 의 설치가 필요합니다. rsh, rcp, rlogin 을 패스워드 없이 작동하기 위한 설정 양쪽 host 갂 oracle 유져에 대한 remote 접근을 password 없이 가능하도록 설정합니다. (root 사용자로 양쪽 노드 모두 수행합니다.) [root@goodus1 ~]# su - oracle /u01/app/oracle> cat .rhosts + /u01/app/oracle> 3.1.5. Oracle User Limits Value Configuration Oracle 사용자가 생성하는 최대 process 개수와 open 할 수 있는 최대 file 개수를 설정합니다. /etc/security/limits.conf 파일에 아래 함목을 추가합니다. (root 사용자로 양쪽 노드 모두 수행합니다.) oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 3.1.6. User OS enviropment Configuration 사용자의 shell 홖경을 설정 합니다. .profile 을 아래와 같이 생성합니다. grid user ORACLE 10g RAC install patch failover ON DELL - 8 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. set -o vi umask 022 export PS1=[`hostname`:\$PWD]$ export LANG=C export EDITOR=vi export GRID_HOME=/oracle/app/11.2.0/grid export ORACLE_BASE=/oracle/app/grid export ORACLE_HOME=$GRID_HOME export ORACLE_SID=+ASM1 export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:$ORACLE_HOME/rdbms/lib export SHLIB_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/rdbms/lib export PATH=$PATH:$ORACLE_HOME/bin:/usr/local/bin:$ORACLE_HOME/OPatch: export CLASSPATH=$ORACLE_HOME/jre:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib:$OR ACLE_HOME/OPatch oracle user export ORACLE_BASE=/oracle/app/oracle export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db export GRID_HOME=/oracle/app/11.2.0/grid export ORACLE_SID=goodus1 export ORACLE_TERM=xterm export PATH=$ORACLE_HOME/bin:$GRID_HOME/bin:$PATH export EDITOR=vi 3.2. Network Configure Check 11.2.0.2 에서 Oracle Grid Infrastructure 는 cluster interconnec 에 대한 오라클 공급 중복 허용 인터커넥트 이중화라 는 새로운 기능을 소개합니다. 이 새로운 기능으로, 개인 상호 연결 네트워크에서 멀티 캐스트 네트워크 통신은 클 러스터의 피어 노드와 통신을 설정하는 부트 스트랩에 홗용, 일단 커뮤니케이션이 설립 네트워크 통신은 다음 유니 캐스트로 젂홖됩니다. 이 multicast 통신은 private interconnect network 의 230.0.1.0 address(port 42424)를 사용합니 다. 그러므로 private interconnect network 에 multicast 가 홗성화 되어 있어야만 multicast 주소 230.0.1.0 에 대한 모 든 cluster node 에서 재대로 작동합니다. 오라클 문서 ID 1212703.1 참고 metalink 에서 mcasttest.pl perl 프로그램을 다운받아 확인해 볼 수 있습니다. $perl mcasttest.pl –n goodus1, goodus2 -i lan902 ########### Setup for node goodus1 ########## Checking node access 'goodus1' Checking node login 'goodus1' Checking/Creating Directory /tmp/mcasttest for binary on node 'goodus1' Distributing mcast2 binary to node 'goodus1' ########### Setup for node goodus2 ########## Checking node access 'goodus2' Checking node login 'goodus2' ORACLE 10g RAC install patch failover ON DELL - 9 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. Checking/Creating Directory /tmp/mcasttest for binary on node 'goodus2' Distributing mcast2 binary to node 'goodus2' ########### testing Multicast on all nodes ########## Test for Multicast address 230.0.1.0 Jan 17 17:18:21 | Multicast Succeeded for lan902 using address 230.0.1.0:42000 Test for Multicast address 224.0.0.251 Jan 17 17:18:22 | Multicast Succeeded for lan902 using address 224.0.0.251:42001 3.3. Raw Device for OCR, VOTEDISK 3.3.1. Raw device Creation Clusterware 에 필요한 OCR 과 VOTEDISK 를 위한 raw device 를 생성하고 renaming 합니다. (root 사용자로 양쪽 노드 모두 수행합니다.) chown root:dba /dev/dbs20/rocr01 /dev/dbs20/rocr02 chown oracle:dba /dev/dbs20/rvot01 /dev/dbs20/rvot02 /dev/dbs21/rvote03 /dev/rdisk/disk378 chmod 660 /dev/dbs20/* /dev/dbs21/* /dev/rdisk/disk378 ORACLE 10g RAC install patch failover ON DELL - 10 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 4. Install steps 4.1. Clusterware Installation 4.1.1. Clusterware Installation Clusterware 는 각 노드갂의 동기화 및 클러스터링을 제공합니다. Clusterware 를 설치하기 위해 아래의 젃차대로 수행합니다. (oracle 사용자로 goodus1 에서 수행합니다.). $ export ORACLE_HOME=/oracle/app/11.2.0/grid $ ./runInstaller & 1. 2 ORACLE 10g RAC install patch failover ON DELL - 11 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 3. 4. ORACLE 10g RAC install patch failover ON DELL - 12 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 5 6 ORACLE 10g RAC install patch failover ON DELL - 13 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 7. 8. ORACLE 10g RAC install patch failover ON DELL - 14 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 9. 10. ORACLE 10g RAC install patch failover ON DELL - 15 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 11. 12. ORACLE 10g RAC install patch failover ON DELL - 16 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 13. 14. ORACLE 10g RAC install patch failover ON DELL - 17 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 15. 16. ORACLE 10g RAC install patch failover ON DELL - 18 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 17. 각 노드에서 root 유저로 orainstRoot.sh 와 root.sh 를 실행합니다. ORACLE 10g RAC install patch failover ON DELL - 19 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. goodus1: /oracle/app/oraInventory/orainstRoot.sh #/oracle/app/oraInventory/orainstRoot.sh Changing permissions of /oracle/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /oracle/app/oraInventory to oinstall. The execution of the script is complete. goodus2: /oracle/app/oraInventory/orainstRoot.sh #/oracle/app/oraInventory/orainstRoot.sh Changing permissions of /oracle/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /oracle/app/oraInventory to oinstall. The execution of the script is complete. goodus1: /oracle/app/11.2.0/grid/root.sh #/oracle/app/11.2.0/grid/root.sh Running Oracle 11g root script... The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /oracle/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: The contents of "dbhome" have not changed. No need to overwrite. The contents of "oraenv" have not changed. No need to overwrite. The contents of "coraenv" have not changed. No need to overwrite. Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /oracle/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory LOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'sys'.. Operation successful. OLR initialization - successful root wallet ORACLE 10g RAC install patch failover ON DELL - 20 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. root wallet cert root cert export peer wallet profile reader wallet pa wallet peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert peer root cert TP profile reader root cert TP pa root cert TP peer pa cert TP pa peer cert TP profile reader pa cert TP profile reader peer cert TP peer user cert pa user cert Adding daemon to inittab CRS-2672: Attempting to start 'ora.mdnsd' on 'goodus1' CRS-2676: Start of 'ora.mdnsd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'goodus1' CRS-2676: Start of 'ora.gpnpd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'goodus1' CRS-2672: Attempting to start 'ora.gipcd' on 'goodus1' CRS-2676: Start of 'ora.cssdmonitor' on 'goodus1' succeeded CRS-2676: Start of 'ora.gipcd' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'goodus1' CRS-2672: Attempting to start 'ora.diskmon' on 'goodus1' CRS-2676: Start of 'ora.diskmon' on 'goodus1' succeeded CRS-2676: Start of 'ora.cssd' on 'goodus1' succeeded ASM created and started successfully. Disk Group ASM created successfully. clscfg: -install mode specified Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'sys'.. Operation successful. CRS-4256: Updating the profile Successful addition of voting disk 8fd08aa2236e4f42bf6d75b33ae020d7. ORACLE 10g RAC install patch failover ON DELL - 21 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. Successfully replaced voting disk group with +ASM. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 8fd08aa2236e4f42bf6d75b33ae020d7 (/dev/rdisk/disk378) [ASM] Located 1 voting disk(s). CRS-2672: Attempting to start 'ora.asm' on 'goodus1' CRS-2676: Start of 'ora.asm' on 'goodus1' succeeded CRS-2672: Attempting to start 'ora.ASM.dg' on 'goodus1' CRS-2676: Start of 'ora.ASM.dg' on 'goodus1' succeeded Configure Oracle Grid Infrastructure for a Cluster ... succeeded goodus2: /oracle/app/11.2.0/grid/root.sh #/oracle/app/11.2.0/grid/root.sh Running Oracle 11g root script... The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /oracle/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: The contents of "dbhome" have not changed. No need to overwrite. The contents of "oraenv" have not changed. No need to overwrite. The contents of "coraenv" have not changed. No need to overwrite. Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /oracle/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory LOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'sys'.. Operation successful. OLR initialization - successful Adding daemon to inittab CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node unknown, number unknown, and is terminating An active cluster was found during exclusive startup, restarting to join the cluster Configure Oracle Grid Infrastructure for a Cluster ... succeeded ORACLE 10g RAC install patch failover ON DELL - 22 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. node2 에서 root.sh 돌릴때, CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node unknown, number unknown, and is terminating 에러가 발생합니다. 이 에러는 HP 10 Gigabit Ethernet NIC driver 사용시 10GigEthr-02 (iexgbe) B.11.31.1009.01 or later 필요합니다. 오라클 문서 ID 1276346.1 참고 #ioscan -fnC lan Class I H/W Path Driver S/W State H/W Type Description ===================================================================== lan 0 0/0/0/3/0/0/0 iexgbe CLAIMED INTERFACE HP PCIe 2-p 10GbE Built-in lan 1 0/0/0/3/0/0/1 iexgbe CLAIMED INTERFACE HP PCIe 2-p 10GbE Built-in lan 2 0/0/0/4/0/0/0 iexgbe CLAIMED INTERFACE HP PCIe 2-p 10GbE Built-in lan 3 0/0/0/4/0/0/1 iexgbe CLAIMED INTERFACE HP PCIe 2-p 10GbE Built-in lan 4 1/0/0/3/0/0/0 iexgbe CLAIMED INTERFACE HP PCIe 2-p 10GbE Built-in lan 5 1/0/0/3/0/0/1 iexgbe CLAIMED INTERFACE HP PCIe 2-p 10GbE Built-in lan 6 1/0/0/4/0/0/0 iexgbe CLAIMED INTERFACE HP PCIe 2-p 10GbE Built-in lan 7 1/0/0/4/0/0/1 iexgbe CLAIMED INTERFACE HP PCIe 2-p 10GbE Built-in #swlist -l product|grep -i iex IEXGBE-DRV B.11.31.1011 HP PCIe 10GigEthr Driver 18. 19. ORACLE 10g RAC install patch failover ON DELL - 23 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. CRS 설치가 완료 되었는지 확인합니다. (root 사용자로 양쪽 노드 모두 수행합니다.). $crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.ASM.dg ora....up.type ONLINE ONLINE goodus1 ora....ER.lsnr ora....er.type ONLINE ONLINE goodus1 ora....N1.lsnr ora....er.type ONLINE ONLINE goodus1 ora.asm ora.asm.type ONLINE ONLINE goodus1 ora.cvu ora.cvu.type ONLINE ONLINE goodus1 ora.gsd ora.gsd.type OFFLINE OFFLINE ora....network ora....rk.type ONLINE ONLINE goodus1 ora.oc4j ora.oc4j.type ONLINE ONLINE goodus1 ora.ons ora.ons.type ONLINE ONLINE goodus1 ora....SM1.asm application ONLINE ONLINE goodus1 ora....P1.lsnr application ONLINE ONLINE goodus1 ora....mp1.gsd application OFFLINE OFFLINE ora....mp1.ons application ONLINE ONLINE goodus1 ora....mp1.vip ora....t1.type ONLINE ONLINE goodus1 ora....SM2.asm application ONLINE ONLINE goodus2 ora....P2.lsnr application ONLINE ONLINE goodus2 ora....mp2.gsd application OFFLINE OFFLINE ora....mp2.ons application ONLINE ONLINE goodus2 ORACLE 10g RAC install patch failover ON DELL - 24 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. ora....mp2.vip ora....t1.type ONLINE ONLINE goodus2 ora.scan1.vip ora....ip.type ONLINE ONLINE goodus1 [goodus1:/work/patchset/grid]$ 4.1.2. OCR 변경 해당 설치문서는 ASM 이 아닌 raw 를 사용하므로 ASM 에 설치된 OCR 을 raw device 로 변경합니다. #[/oracle/app/11.2.0/grid/bin]./ocrconfig -add /dev/ dbs20/rocr01 #[/oracle/app/11.2.0/grid/bin]./ocrconfig -add /dev/ dbs20/rocr02 #[/oracle/app/11.2.0/grid/bin]./ocrconfig -delete +ASM #[/oracle/app/11.2.0/grid/bin]./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2356 Available space (kbytes) : 259764 ID : 642797593 Device/File Name : /dev/ dbs20/rocr01 Device/File integrity check succeeded Device/File Name : /dev/ dbs20/rocr02 Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded 4.1.3. votedisk 변경 해당 설치문서는 ASM 이 아닌 raw 를 사용하므로 ASM 에 설치된 votedisk 를 raw device 로 변경합니다. #[/oracle/app/11.2.0/grid/bin]./crsctl replace votedisk /dev/ dbs20/rvot01 Now formatting voting disk: /dev/ dbs20/rvot01. CRS-4256: Updating the profile Successful addition of voting disk ae4387b930394f76bf17923430bae8e3. ORACLE 10g RAC install patch failover ON DELL - 25 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. Successful deletion of voting disk 8fd08aa2236e4f42bf6d75b33ae020d7. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced #[/oracle/app/11.2.0/grid/bin]./crsctl add css votedisk /dev/ dbs20/rvot02 Now formatting voting disk: /dev/ dbs20/rvot02. CRS-4603: Successful addition of voting disk /dev/ dbs20/rvot02. #[/oracle/app/11.2.0/grid/bin]./crsctl add css votedisk /dev/ dbs21/rvot03 Now formatting voting disk: /dev/ dbs21/rvot03. CRS-4603: Successful addition of voting disk /dev/ dbs21/rvot03. #[/oracle/app/11.2.0/grid/bin] #[/oracle/app/11.2.0/grid/bin]./crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE ae4387b930394f76bf17923430bae8e3 (/dev/ dbs20/rvot01) [] 2. ONLINE 02818a19cc1d4ffebff79b96aa2ddea2 (/dev/ dbs20/rvot02) [] 3. ONLINE 63210c7dc5fe4f5cbf682e3dbd9cdd8f (/dev/ dbs21/rvot03) [] Located 3 voting disk(s). ORACLE 10g RAC install patch failover ON DELL - 26 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 4.2. Database Installation Database 를 raw device 로 생성합니다. 4.2.1. Change Permission Database 를 생성하기위해 raw device 의 소유권 및 권한을 변경합니다. (양쪽 노드에서 모두 적용) $chown oracle:dba /dev/dbs22/rcontrol1 /dev/dbs22/rcontrol2 /dev/dbs22/rcontrol3 $chown oracle:dba /dev/dbs22/rsystem01 $chown oracle:dba /dev/dbs22/rredo111 /dev/dbs22/rredo121 /dev/dbs22/rredo131 /dev/dbs22/rredo112 /dev/dbs22/rredo122 /dev/dbs22/rredo132 $chown oracle:dba /dev/dbs22/rredo211 /dev/dbs22/rredo221 /dev/dbs22/rredo231 /dev/dbs22/rredo212 /dev/dbs22/rredo222 /dev/dbs22/rredo232 $chown oracle:dba /dev/dbs23/rundotbs1 /dev/dbs23/rundotbs2 $chown oracle:dba /dev/dbs23/rtemp $chown oracle:dba /dev/dbs22/ruser $chown oracle:dba /dev/dbs22/rsysaux $chown oracle:dba /dev/dbs22/rspfile $chmod 660 /dev/dbs22/rcontrol1 /dev/dbs22/rcontrol2 /dev/dbs22/rcontrol3 $chmod 660 /dev/dbs22/rsystem01 $chmod 660 /dev/dbs22/rredo111 /dev/dbs22/rredo121 /dev/dbs22/rredo131 /dev/dbs22/rredo112 /dev/dbs22/rredo122 /dev/dbs22/rredo132 $chmod 660 /dev/dbs22/rredo211 /dev/dbs22/rredo221 /dev/dbs22/rredo231 /dev/dbs22/rredo212 /dev/dbs22/rredo222 /dev/dbs22/rredo232 $chmod 660 /dev/dbs23/rundotbs1 /dev/dbs23/rundotbs2 $chmod 660 /dev/dbs23/rtemp $chmod 660 /dev/dbs22/ruser $chmod 660 /dev/dbs22/rsysaux $chmod 660 /dev/dbs22/rspfile HP Itanium 에서 11.2.0.2 이상 설치 혹은 업그레이드시 raw device 로 생성한 controlfile, datafile, redo log file 들은 마운트 단계에서 block device 오픈시 에러를 발생하므로 반드시 아래 패치를 적용해줍니다. HPUX patch PHCO_41479 - 오라클 문서 ID 1269346.1 참고 4.2.2. Database engine install (oracle 사용자로 goodus1 에서 수행합니다.). $ export ORACLE_HOME=/oracle/app/oracle/product/11.2.0/db $ ./runInstaller & 1. ORACLE 10g RAC install patch failover ON DELL - 27 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 2. 3. ORACLE 10g RAC install patch failover ON DELL - 28 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 4. 5. ORACLE 10g RAC install patch failover ON DELL - 29 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 6. 7. ORACLE 10g RAC install patch failover ON DELL - 30 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 8. 9. ORACLE 10g RAC install patch failover ON DELL - 31 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 10. 11. ORACLE 10g RAC install patch failover ON DELL - 32 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 12. 13. ORACLE 10g RAC install patch failover ON DELL - 33 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 14. ORACLE 10g RAC install patch failover ON DELL - 34 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 4.2.3. DBCA raw device 를 사용시 OUI 를 사용할 수 없으므로 DBCA 를 실행하여 생성스크립트를 추출하여 editing 합니다. 1. 2. ORACLE 10g RAC install patch failover ON DELL - 35 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 3. 4. 5. ORACLE 10g RAC install patch failover ON DELL - 36 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 6. 7. ORACLE 10g RAC install patch failover ON DELL - 37 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 8. 9. ORACLE 10g RAC install patch failover ON DELL - 38 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 10. 11. ORACLE 10g RAC install patch failover ON DELL - 39 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 12. 13. ORACLE 10g RAC install patch failover ON DELL - 40 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 14. 15. ORACLE 10g RAC install patch failover ON DELL - 41 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 16. 17. ORACLE 10g RAC install patch failover ON DELL - 42 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 18. 19. Create Database 체크를 해제하고 스크립트 생성을 체크해서 스크립트만 생성하도록 합니다. ORACLE 10g RAC install patch failover ON DELL - 43 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 생성된 스크립트 폴더에 들어가서 다음 파일들을 수정합니다. init.ora - controlfile 을 추가하고, ASM 과련 설정을 주석처리 합니다. control_files='/dev/dbs22/rcontrol1','/dev/dbs23/rcontrol2','/dev/dbs24/rcontrol3' #db_create_file_dest="+ASM" goodus1.sql - ASM 관련 부분을 수정합니다. set verify off ACCEPT sysPassword CHAR PROMPT 'Enter new password for SYS: ' HIDE ACCEPT systemPassword CHAR PROMPT 'Enter new password for SYSTEM: ' HIDE ACCEPT sysmanPassword CHAR PROMPT 'Enter new password for SYSMAN: ' HIDE ACCEPT dbsnmpPassword CHAR PROMPT 'Enter new password for DBSNMP: ' HIDE ACCEPT asmSysPassword CHAR PROMPT 'Enter ASM SYS user password: ' HIDE host /oracle/app/oracle/product/11.2.0/db/bin/orapwd file=/oracle/app/oracle/product/11.2.0/db/dbs/orapwGOODUS1 force=y host /oracle/app/11.2.0/grid/bin/setasmgidwrap o=/oracle/app/oracle/product/11.2.0/db/bin/oracle host /oracle/app/oracle/product/11.2.0/db/bin/srvctl add database -d GOODUSDB -o /oracle/app/oracle/product/11.2.0/db -p /dev/dbs22/rspfile -n GOODUSDB host /oracle/app/oracle/product/11.2.0/db/bin/srvctl add instance -d GOODUSDB -i GOODUS1 -n goodus1 host /oracle/app/oracle/product/11.2.0/db/bin/srvctl add instance -d GOODUSDB -i GOODUS2 -n goodus2 host /oracle/app/oracle/product/11.2.0/db/bin/srvctl disable database -d GOODUSDB @/oracle/app/oracle/admin/GOODUSDB/scripts/CreateDB.sql @/oracle/app/oracle/admin/GOODUSDB/scripts/CreateDBFiles.sql ORACLE 10g RAC install patch failover ON DELL - 44 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. @/oracle/app/oracle/admin/GOODUSDB/scripts/CreateDBCatalog.sql @/oracle/app/oracle/admin/GOODUSDB/scripts/context.sql @/oracle/app/oracle/admin/GOODUSDB/scripts/xdb_protocol.sql @/oracle/app/oracle/admin/GOODUSDB/scripts/emRepository.sql @/oracle/app/oracle/admin/GOODUSDB/scripts/apex.sql @/oracle/app/oracle/admin/GOODUSDB/scripts/CreateClustDBViews.sql host echo "SPFILE='/dev/dbs22/rspfile " > /oracle/app/oracle/product/11.2.0/db/dbs/initGOODUS1.ora @/oracle/app/oracle/admin/GOODUSDB/scripts/lockAccount.sql @/oracle/app/oracle/admin/GOODUSDB/scripts/postDBCreation.sql CreateDB.sql - datafile 경로를 수정합니다. SET VERIFY OFF connect "SYS"/"&&sysPassword" as SYSDBA set echo on spool /oracle/app/oracle/admin/GOODUSDB/scripts/CreateDB.log append startup nomount pfile="/oracle/app/oracle/admin/GOODUSDB/scripts/init.ora"; CREATE DATABASE "GOODUSDB" MAXINSTANCES 32 MAXLOGHISTORY 1 MAXLOGFILES 192 MAXLOGMEMBERS 3 MAXDATAFILES 2048 DATAFILE '/dev/dbs22/rsystem01' SIZE 4000M EXTENT MANAGEMENT LOCAL SYSAUX DATAFILE '/dev/dbs22/rsysaux' SIZE 10200M SMALLFILE DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE '/dev/dbs23/rtemp01' SIZE 1000M SMALLFILE UNDO TABLESPACE "UNDOTBS1" DATAFILE '/dev/dbs23/rundotbs1' SIZE 10200M CHARACTER SET AL32UTF8 NATIONAL CHARACTER SET AL16UTF16 LOGFILE GROUP 1 ('/dev/dbs22/rredo111','/dev/dbs23/rredo112') SIZE 1000M, GROUP 2 ('/dev/dbs22/rredo121','/dev/dbs23/rredo122') SIZE 1000M, GROUP 3 ('/dev/dbs22/rredo131','/dev/dbs23/rredo132') SIZE 1000M USER SYS IDENTIFIED BY "&&sysPassword" USER SYSTEM IDENTIFIED BY "&&systemPassword"; set linesize 2048; column ctl_files NEW_VALUE ctl_files; select concat('control_files=''', concat(replace(value, ', ', ''','''), '''')) ctl_files from v$parameter where name ='control_files'; host echo &ctl_files >>/oracle/app/oracle/admin/GOODUSDB/scripts/init.ora; spool off CreateDBFiles.sql – datafile 경로를 수정합니다. SET VERIFY OFF connect "SYS"/"&&sysPassword" as SYSDBA set echo on ORACLE 10g RAC install patch failover ON DELL - 45 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. spool /oracle/app/oracle/admin/GOODUSDB/scripts/CreateDBFiles.log append CREATE UNDO TABLESPACE "UNDOTBS2" DATAFILE '/dev/dbs23/rundotbs2' SIZE 10200M; CREATE TABLESPACE "USERS" LOGGING DATAFILE '/dev/dbs22/ruser' SIZE 2000M EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO; ALTER DATABASE DEFAULT TABLESPACE "USERS"; spool off postDBCreation.sql – datafile 경로 수정 및 ASM 관련 설정을 제거합니다. SET VERIFY OFF connect "SYS"/"&&sysPassword" as SYSDBA set echo on spool /oracle/app/oracle/admin/GOODUSDB/scripts/postDBCreation.log append @/oracle/app/oracle/product/11.2.0/db/rdbms/admin/catbundle.sql psu apply; select 'utl_recomp_begin: ' || to_char(sysdate, 'HH:MI:SS') from dual; execute utl_recomp.recomp_serial(); select 'utl_recomp_end: ' || to_char(sysdate, 'HH:MI:SS') from dual; select group# from v$log where group# =3; select group# from v$log where group# =4; ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 4 ('/dev/dbs22/rredo211','/dev/dbs23/rredo212') SIZE 1000M, GROUP 5 ('/dev/dbs22/rredo221','/dev/dbs23/rredo222') SIZE 1000M, GROUP 6 ('/dev/dbs22/rredo231','/dev/dbs23/rredo232') SIZE 1000M ALTER DATABASE ENABLE PUBLIC THREAD 2; host echo cluster_database=true >>/oracle/app/oracle/admin/GOODUSDB/scripts/init.ora; host echo remote_listener=goodus-cluster-scan:1521>>/oracle/app/oracle/admin/GOODUSDB/scripts/init.ora; connect "SYS"/"&&sysPassword" as SYSDBA set echo on create spfile='SPFILE='/dev/dbs22/rspfile‘ FROM pfile='/oracle/app/oracle/admin/GOODUSDB/scripts/init.ora'; shutdown immediate; host /oracle/app/oracle/product/11.2.0/db/bin/srvctl enable database -d GOODUSDB; host /oracle/app/oracle/product/11.2.0/db/bin/srvctl start database -d GOODUSDB; connect "SYS"/"&&sysPassword" as SYSDBA host /oracle/app/oracle/product/11.2.0/db/bin/emca -config dbcontrol db -silent -cluster -CLUSTER_NAME goodus-cluster -LOG_FILE /oracle/app/oracle/admin/GOODUSDB/scripts/emConfig.log -SID PIS001 - DB_UNIQUE_NAME GOODUSDB -EM_HOME /oracle/app/oracle/product/11.2.0/db -SERVICE_NAME GOODUSDB -PORT 1521 -LISTENER_OH /oracle/app/11.2.0/grid -LISTENER LISTENER -ORACLE_HOME /oracle/app/oracle/product/11.2.0/db -HOST goodus1; spool off exit; 해당파일들을 수정후 goodus1.sh 를 실행시켜 database 를 생성합니다. ORACLE 10g RAC install patch failover ON DELL - 46 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 4.2.4. ASM resource 제거 필요없어진 ASM resource 를 제거합니다. $srvctl stop diskgroup -g ASM $srvctl stop asm -n goodus1 $srvctl stop asm -n goodus2 $srvctl disable diskgroup -g ASM $srvctl disable asm -n goodus1 $srvctl disable asm -n goodus2 4.3. Check Configuration 설치가 완료되면 현재 OCR 에 등록된 구성 정보를 확인합니다. Oifcfg getif 조회 값과 x$ksxpia 조회값이 반드시 올바른 값으로 일치해야 합니다. (oracle 사용자로 양쪽 노드 모두 수행합니다.) [goodus1:/home/grid]$crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.ASM.dg ora....up.type OFFLINE OFFLINE ora....ER.lsnr ora....er.type ONLINE ONLINE goodus1 ora....N1.lsnr ora....er.type ONLINE ONLINE goodus2 ora.asm ora.asm.type OFFLINE OFFLINE ora.cvu ora.cvu.type ONLINE ONLINE goodus2 ora.gsd ora.gsd.type OFFLINE OFFLINE ora....network ora....rk.type ONLINE ONLINE goodus1 ora.oc4j ora.oc4j.type ONLINE ONLINE goodus2 ora.ons ora.ons.type ONLINE ONLINE goodus1 ora....01db.db ora....se.type ONLINE ONLINE goodus1 ora....SM1.asm application OFFLINE OFFLINE ora....S1.lsnr application ONLINE ONLINE goodus1 ora....us1.gsd application OFFLINE OFFLINE ora....us1.ons application ONLINE ONLINE goodus1 ora....us1.vip ora....t1.type ONLINE ONLINE goodus1 ora....SM2.asm application OFFLINE OFFLINE ora....S2.lsnr application ONLINE ONLINE goodus2 ora....us2.gsd application OFFLINE OFFLINE ora....us2.ons application ONLINE ONLINE goodus2 ora....us2.vip ora....t1.type ONLINE ONLINE goodus2 ora.scan1.vip ora....ip.type ONLINE ONLINE goodus2 ORACLE 10g RAC install patch failover ON DELL - 47 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. 5. 설치 후 발견된 이슈 5.1. Error while reading ADR file after upgraded to 11.2.0.2 Database 를 startup 할 때 alert.log 에서 다음과 같은 에러를 확인할 수 있습니다. … Error with dbgriap_init_adr_pga: 48178 ORA-48178: error encountered while reading an ADR block file during ADR initialization [/oracle/app/orapis/diag/rdbms/pis001/PIS0011/metadata/ADR_INTERNAL.mif] ORA-48122: error with opening the ADR block file [/oracle/app/orapis/diag/rdbms/pis001/PIS0011/metadata/ADR_INTERNAL.mif] [0] ORA-27037: unable to obtain file status HPUX-ia64 Error: 251: Function is not available Additional information: 46 Wed Jan 19 13:00:43 2011 ERROR: The process is unable to create the ADR schema in the diagnostic_dest directory ERROR: because of a disk issue or OS platform issue Wed Jan 19 13:00:43 2011 ERROR: Reverting back to using the user_dump_dest and background_dump_dest ERROR: as the location for the traces and logs Starting ORACLE instance (normal) … 이 에러는 11.2.0.2 에서 'ADR_INTERNAL.mif' 파일을 오픈할 때 HP-UX directio code 를 사용하는데 HP OnLineJFS product 이 설치되지 않아 발생하는 이슈입니다. 해결 방법은 HP OnLineJFS product 을 설치하거나 init.ora 에 _DIAG_ADR_ENABLED 를 FALSE 로 설정합니다. 오라클 문서 ID 1268653.1 참고 5.2. HP-UX Time Mismatch Between Ocssd.Log And Sysdate cssd.log 를 확인하면 다음 메시지를 4 초 마다 볼 수 있다. 2011-01-19 15:52:05.243: [ CSSD][38]clssnmSendingThread: sending status msg to all nodes 2011-01-19 15:52:05.243: [ CSSD][38]clssnmSendingThread: sent 4 status msgs to all nodes 2011-01-19 15:52:09.282: [ CSSD][38]clssnmSendingThread: sending status msg to all nodes 2011-01-19 15:52:09.282: [ CSSD][38]clssnmSendingThread: sent 4 status msgs to all nodes 2011-01-19 15:52:13.323: [ CSSD][38]clssnmSendingThread: sending status msg to all nodes 2011-01-19 15:52:13.323: [ CSSD][38]clssnmSendingThread: sent 4 status msgs to all nodes 해당 메시지는 HP-UX 의 sysdate 와 ocssd.log 사이에 time 이 맞지않아 발생하는 에러로 HP-UX 설치 후에 ORACLE 10g RAC install patch failover ON DELL - 48 – 본 산출물의 내용은 굿어스㈜의 사전승인 없이 외부유출 및 공개를 금지합니다. “/etc/default/tz”에 기본 TZ 설정이 포함되는데 이 값이 실제 시갂과 일치하지 않아 발생합니다. 해결방법은 실제 timezone 에 /etc/default/tz 를 설정합니다. 오라클 문서 ID 738174.1 참고 $date Wed Jan 19 15:57:13 KST 2011 $cat /etc/default/tz KST-9