Differences

This shows you the differences between two versions of the page.

Link to this comparison view

en:cic_portal_failover_-_actions_and_schedule [2016/12/16 10:16] (current)
Line 1: Line 1:
 +Last modified: Jul 04, 2008 by Aïdel\\
 +\\
 +
 +====== CIC portal Failover - actions and schedule ======
 +
 +\\
 +\\
 +
 +====== Chapitre 1 : Failover of the GOC db ======
 +
 +A cache has been integrated into the CIC db thanks to the materialized views.\\
 +The main idea is to duplicate all needed tables into the CIC db so that CIC portal are stand-alone.\\
 +All tables prefixed by GOC_%%**%%%%**%% come from the GOC db.\\
 +\\
 +To use the GOC db cache you should modify at level web cluster[cic.in2p3.fr] the **goc_failover** variable to true.\\
 +This variable is located in the file "/​htdocs/​setvariables_global.php"​.\\
 +Once this modification done, a pop-up will display in the header of the CIC Portal to inform all users the GOC db is unreachable and the CIC Portal works in degraded mode. 
 +====== Chapitre 2 : Failover of all components ======
 +
 +=====  Before the switch ​ =====
 +
 +^What                                                                                                            ^Who     ​^When ​   ^
 +|Definition of a precise schedule of actions to perform on the database ​                                         |Osman/JR|D - 15  |
 +|Definition of a precise schedule for the switch ​                                                                ​|All ​    |D - 15  |
 +|Prepare/​update downtime page to put online during the switch ​                                                   |Gilles ​ |        |
 +|Block external access to CIC@CNAF ​                                                                              ​|Al* ​    |D - 15  |
 +|PERFORM ALL DEFINED TESTS (see Cyril'​s doc)                                                                     ​|All ​    |D - 15/8|
 +|Announce the downtime and the switch to relevant bodies ​                                                        ​|CC ​     |D - 8   |
 +|Verify that server certificate is OK (CNAF and CC side)                                                         ​| ​       |D - 8   |
 +|Verify that CA list is up-to-date (CNAF and CC side)                                                            |        |D - 8   |
 +|Verify that script cnaf_update.php gets the whole list all non-CVS files to copy on CNAF side (AUPs, VOMS, MOUs)|Gilles ​ |D - 5   |
 +|Verify that CNAF portal code is synchro with CVS                                                                |CNAF    |D-5/D-1 |
 +|Configure Lavoisier [see Lavoisier Update ]                                                                     ​|All ​    ​| ​       |
 +
 +Optional actions (if needed):
 +^What                                                                  ^Who        ^When^
 +|Copy data from CC-PROD Database to CC-DEV Database ​                   |Osman/​JR ​  ​| ​   |
 +|Copy data from CC-DEV Database to CNAF Database ​                      ​|Osman/​JR/​Al| ​   |
 +|Verify/​test CC-DEV & CNAF Database synchronization ​                   |Osman/​Al ​  ​| ​   |
 +|Provide feedback on this export procedure, to help for the real export|Osman/​Al ​  ​| ​   |
 +
 +=====  Switch from CC to CNAF  =====
 +
 +for Oracle DB switch procedure, please see doc n° [[:​en:​database_replication_oracle_manual|545]]
 +^What                                                                               ​^Who ​      ​^When ​       ^Duration^
 +|Block external access to CIC@CC ​                                                   |Gilles ​   |H - 3:00    |30s     |
 +|Switch DNS to make cic.gridops.org point to CNAF                                   ​|Alfredo ​  |H - 2:55    |5mn     |
 +|Make a DB export of CC-PROD Database ​                                              ​|Jean-René |H - 2:50    |30mn    |
 +|Export CC-PROD data to CNAF DB                                                     ​|Osman/​Al* |H - 2:15    |45mn    |
 +|Verify/​test CC-PROD & CNAF Database synchronization ​                               |Osman ​    |H - 1:30    |5mn     |
 +|Verify everyting is ok on SAMAP UI machine@CNAF ​                                   |Rafal ​    |H - 1:30    |5mn     |
 +|Get at CNAF all non-CVS files using script https:​%%//​%%cic.in2p3.fr/​cnaf_update.php|Al* ​      |H - 2:00    |5mn     |
 +|Verify write access to the documents above                                         ​|Alessandro|H - 1:25    |2mn     |
 +|Verify/​test CIC@CC & CIC@CNAF synchronization (all components) ​                    ​|Cyril ​    |H - 1:30 / H|1h30 ​   |
 +|Open external access to CIC@CNAF ​                                                  ​|Al* ​      ​|H ​          ​|5mn ​    |
 +|Announce CIC@CNAF being online ​                                                    ​|CNAF ​     |H + 00:10   ​| ​       |
 +
 +=====  CIC@CNAF running time  =====
 +
 +^What^Who^When^
 +
 +=====  Switch from CNAF to CC  =====
 +
 +for Oracle DB switch procedure, please see doc n° [[:​en:​database_replication_oracle_manual|545]]
 +^What                                                        ^Who         ​^When ​   ^Duration^
 +|Announce the downtime and the switch to relevant bodies ​    ​|CC ​         |D - 8   ​| ​       |
 +|Block external access to CIC@CNAF ​                          ​|Al* ​        |H - 2:​30|5mn ​    |
 +|Block external access to CIC@CC ​                            ​|Gilles ​     |H - 2:​30|5mn ​    |
 +|Peform DNS switch to make cic.gridops.org point to CIC@CC ​  ​|Alfredo ​    |H - 2:​15|5mn ​    |
 +|Verify everyting is ok on SAMAP UI machine@CC ​              ​|Rafal ​      |H - 2:​00|5mn ​    |
 +|Make a DB export of CNAF Database ​                          ​|Alessandro ​ |H - 2:​00|45mn ​   |
 +|Import data from CNAF DB to CC-PROD DB                      |Osman/​JR/​Al*|H - 1:​15|45mn ​   |
 +|Get from CNAF all non-CVS files using script cnaf_update.php|Gilles ​     |H - 0:​30|5mn ​    |
 +|Verify/​test CC-PROD & CNAF Database synchronization ​        ​|Osman ​      |H - 0:​30|30mn ​   |
 +|Open external access to CIC@CC ​                             |Gilles ​     |H       ​|5mn ​    |
 +|Announce CIC@CC being online ​                               |CC          |H + 0:10|        |
 +
 +=====  After the switch ​ =====
 +
 +^What^Who^When^
 +
 +
 +====== Chapitre 3 : Switching Oracle DB only ======
 +
 +for Oracle DB switch procedure, please see doc n° [[:​en:​database_replication_oracle_manual|545]]
 +=====  Before the switch ​ =====
 +
 +^What                                                                  ^Who     ^When ^
 +|Definition of a precise schedule of actions to perform on the database|Osman/​JR|D - 8|
 +|Definition of a precise schedule for the switch ​                      ​|All ​    |D - 8|
 +|Prepare/​update downtime page to put online during the switch ​         |Gilles ​ |     |
 +|Announce the downtime and the switch to relevant bodies ​              ​|CC ​     |D - 8|
 +
 +
 +====  Lavoisier Update ​ ====
 +
 +
 +===  Copy of lavoisier configuration file  ===
 +
 +<​code>​
 +ws-core-4.0.1]$ cp etc/​in2p3_lavoisier/​lavoisier-config.xml etc/​in2p3_lavoisier/​lavoisier-config.xml.cnaf
 +</​code>​
 +
 +===  Replace database connection in the lavoisier-config.xml.cnaf file  ===
 +
 +<​code>​
 +Replace the string 2 by the string 1
 +
 +String 1 :
 +jdbc:​oracle:​thin:​@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=egee004.cnaf.infn.it)(PORT=1521))(CONNECT_DATA=(SID=cicfailover)))
 +
 +
 +String 2 :
 +jdbc:​oracle:​thin:​@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=ccdbcl01.in2p3.fr)(PORT=1521))(ADDRESS = (PROTOCOL = TCP)(HOST = ccdbcl02.in2p3.fr)(PORT = 1521))(
 +LOAD_BALANCE = yes))(CONNECT_DATA =(SERVICE_NAME = ccora10g.in2p3.fr)))
 +</​code>​
 +
 +===  copy the following files to configRP/​Xquery/​CNAF/ ​ ===
 +
 +VOMSCertificate_move2db.xqy\\
 +alertDataBase.xqy\\
 +listvo.xqy\\
 +\\
 +NB : These files contains a connection to the CIC db\\
 +To find them run this command : grep ccora10g * in the configRP/​Xquery/​ repository ​
 +===  Modify the lavoisier_config.xml.cnaf so that resources using the above files point to the repository configRP/​Xquery/​CNAF/ ​ ===
 +
 +For example :\\
 +\\
 +Lavoisier_config.xml
 +<​code>​
 +...
 +    <​ds:​source id="​VOMSCertificate_move2db"​ adapter="​fr.in2p3.lavoisier.adapter.expl.SQL2XML">​
 +        <​ds:​config>​
 +            <file queryFile="​configRP/​Xquery/​VOMSCertificate_move2db.xqy"/>​
 +        </​ds:​config>​
 +    <​retry>​
 +   <​ds:​whenThrown exception="​java.lang.Exception"​ count="​4"​ period="​PT10M"/>​
 +    </​retry>​
 +        <​ds:​fullCache type="​disk">​
 +           <​ds:​refresh>​
 +               <​ds:​whenRefreshedAndPublished>​
 +                   <​ds:​cacheDependency id="​VOMSCertificate_validcert"/>​
 +              </​ds:​whenRefreshedAndPublished>​
 +            </​ds:​refresh>​
 +        </​ds:​fullCache>​
 +    </​ds:​source>​
 +...
 +</​code>​
 +Lavoisier_config.xml.cnaf
 +<​code>​
 +...
 +    <​ds:​source id="​VOMSCertificate_move2db"​ adapter="​fr.in2p3.lavoisier.adapter.expl.SQL2XML">​
 +        <​ds:​config>​
 +            <file queryFile="​configRP/​Xquery/​CNAF/​VOMSCertificate_move2db.xqy"/>​
 +        </​ds:​config>​
 +    <​retry>​
 +   <​ds:​whenThrown exception="​java.lang.Exception"​ count="​4"​ period="​PT10M"/>​
 +    </​retry>​
 +        <​ds:​fullCache type="​disk">​
 +           <​ds:​refresh>​
 +               <​ds:​whenRefreshedAndPublished>​
 +                   <​ds:​cacheDependency id="​VOMSCertificate_validcert"/>​
 +              </​ds:​whenRefreshedAndPublished>​
 +            </​ds:​refresh>​
 +        </​ds:​fullCache>​
 +    </​ds:​source>​
 +...
 +</​code>​
 +=====  Switch from CC to CNAF  =====
 +
 +What
 +Who
 +When
 +Duration
 +Block external access to CIC@CC
 +Gilles
 +H - 01:30
 +30s
 +Make a DB export of CC-PROD Database
 +Jean-René
 +H - 01:30
 +30mn
 +Modify portal code on CC side to point to CNAF-DB
 +Gilles
 +10mn
 +Modify lavoisier configuration on CC side to point to CNAF-DB
 +Cyril
 +10mn
 +Export CC-PROD data to CNAF DB
 +Osman/Al*
 +H - 01:00
 +45mn
 +Verify/test CC-PROD & CNAF Database synchronization
 +Osman
 +H - 00:15
 +5mn
 +switch the lavoisier configuration file and restart Lavoisier Lavoisier using the CNAF database :
 +<​code>​
 +cp etc/​in2p3_lavoisier/​lavoisier_config.xml etc/​in2p3_lavoisier/​lavoisier_config.xml.in2p3
 +
 +cp etc/​in2p3_lavoisier/​lavoisier_config.xml.cnaf etc/​in2p3_lavoisier/​lavoisier_config.xml
 +
 +bin/​stop-lavoisier.sh
 +bin/​start-lavoisier.sh
 +
 +</​code>​
 +All
 +Open external access to CIC@CC
 +Gilles
 +H
 +5mn
 +Announce CIC@CC being online with DB running at CNAF
 +Gilles
 +H + 00:10
 +=====  CIC@CNAF running time  =====
 +
 +^What                                               ​^Who ​  ​^When ​             ^
 +|Verify insert/​update on the DB are working properly|Gilles|on a regular basis|
 +|Verify RC report generation ​                       |Osman |every day         |
 +|Verify ROC/RC report generation ​                   |Osman |monday ​           |
 +
 +=====  Switch from CNAF to CC  =====
 +
 +What
 +Who
 +When
 +Duration
 +Announce the downtime and the switch to relevant bodies
 +CC
 +D - 8
 +Block external access to CIC@CC
 +Gilles
 +H - 2:00
 +5mn
 +Make a DB export of CNAF Database
 +Alessandro
 +H - 2:00
 +45mn
 +Modify lavoisier configuration on CC side to point back to CC-DB
 +Cyril
 +10mn
 +Import data from CNAF DB to CC-PROD DB
 +Osman/​JR/​Al*
 +H - 1:00
 +45mn
 +Verify/test CC-PROD & CNAF Database synchronization
 +Osman
 +H - 0:15
 +5mn
 +switch the lavoisier configuration file and restart Lavoisier
 +<​code>​
 +cp etc/​in2p3_lavoisier/​lavoisier_config.xml.in2p3 etc/​in2p3_lavoisier/​lavoisier_config.xml
 +
 +bin/​stop-lavoisier.sh
 +bin/​start-lavoisier.sh
 +</​code>​
 +Open external access to CIC@CC
 +Gilles
 +H
 +5mn
 +Announce CIC@CC being online with DB at CC
 +CC
 +H + 0:10
 +=====  After the switch ​ =====
 +
 +^What^Who^When^
 +
 +
 +====== Chapitre 4 : Switching portal only ======
 +
 +
 +====== Chapitre 5 : Switching lavoisier only ======
 +
 +
 +====== Chapitre 6 : Switching Samadmin'​s page only ======
 +
 +
  
  • en/cic_portal_failover_-_actions_and_schedule.txt
  • Last modified: 2016/12/16 10:16
  • (external edit)