EBS 12.2.0 NLS did not go through successfully

[ErmanArslansOracleBlog]

1) Do you have /opt/vis11/fs1/EBSapps/appl/APPSVIS11_db3h-dev.env file?

2)
/opt/vis11/fs1
/opt/vis11/fs2 , these directories are present right?

3)
Is this a new VIISON environment?
I mean; did you install VISION again (a new install) (because you were getting abnormal errors last time) and then upgrade it to 12.2.4 right?

[ErmanArslansOracleBlog]

IF this is a new install, and if your directories are different than the previous problematic install of yours, then your oraInst.loc may be pointing to the old directories.
If this a new and proper VISION install please check oraInst.loc file..
If that 's the case; correct the oraInst.loc and make it point to the correct oraInventory.

[Max]

1. No, I haven't

ls -l /opt/vis11/fs1/EBSapps/appl/APPSVIS11_db3h-dev.env
ls: cannot access /opt/vis11/fs1/EBSapps/appl/APPSVIS11_db3h-dev.env: No such file or directory

Because

[oracle@db3h-dev soft]$ ls -l /opt/vis11/fs1/EBSapps/
total 4
drwxr-xr-x. 3 oracle oinstall 4096 Мар 24 13:58 comn

2. Yes.

ls -l /opt/vis11/fs1/
total 12
drwxr-xr-x. 3 oracle oinstall 4096 Мар 24 13:58 EBSapps
drwxr-xr-x. 3 oracle oinstall 4096 Мар 24 13:59 FMW_Home
drwxr-xr-x. 3 oracle oinstall 4096 Мар 23 18:08 inst
[root@db3h-dev nec_tool]# ls -l /opt/vis11/fs2/
total 12
drwxr-xr-x.  5 oracle oinstall 4096 Мар 23 18:11 EBSapps
drwxr-x---. 11 oracle oinstall 4096 Мар 23 23:02 FMW_Home
drwxr-xr-x.  3 oracle oinstall 4096 Мар 23 18:08 inst


3. Yes, this is a new VISION installation.

The problem is that the directories has disappeared. I think main functionality have a problem.

[Max]

I've reviewed and correct all old installations in oraInst.lock

[Max]

I'd re-run fs_clone stage with force flag.

$ adop phase=fs_clone force=yes

and main process in wait status.

Beginning application tier FSCloneStage - Wed Mar 25 11:42:36 2015

/opt/vis11/fs2/EBSapps/comn/util/jdk32/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /opt/vis11/fs2/FMW_Home/webtier/lib/xmlparserv2.jar:/opt/vis11/fs2/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/opt/vis11/fs2/EBSapps/comn/java/classes:/opt/vis11/fs2/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/opt/vis11/fs2/FMW_Home/webtier/oui/jlib/ewt3.jar:/opt/vis11/fs2/FMW_Home/webtier/oui/jlib/share.jar:/opt/vis11/fs2/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/opt/vis11/fs2/FMW_Home/webtier/jlib/ojmisc.jar:/opt/vis11/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/opt/vis11/fs2/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.FSCloneStageAppsTier -e /opt/vis11/fs2/inst/apps/VIS11_db3h-dev/appl/admin/VIS11_db3h-dev.xml -targ /opt/vis11/fs1/inst/apps/VIS11_db3h-dev/appl/admin/VIS11_db3h-dev.xml -stage /opt/vis11/fs2/EBSapps/comn/adopclone_db3h-dev -tmp /tmp -component appsTier -nopromptmsg
Log file located at /opt/vis11/fs2/inst/apps/VIS11_db3h-dev/admin/log/clone/FSCloneStageAppsTier_03251242.log
Completed FSCloneStage...
Wed Mar 25 11:48:20 2015

# ps -ef|grep adop
oracle   32212 26893  0 11:39 pts/1    00:00:00 /bin/sh /opt/vis11/fs_ne/EBSapps/appl/ad/bin/adop phase=fs_clone force=yes

# strace -p 32212
Process 32212 attached - interrupt to quit
wait4(-1, ^C <unfinished ...>
Process 32212 detached

through top I found process which consume 100% cpu

# ps -ef|grep 25238
oracle   25238  2990 97 11:48 pts/1    02:02:29 /opt/vis11/fs2/FMW_Home/Oracle_EBS-app1/oui/bin/runInstaller -removeHome ORACLE_HOME=/opt/vis11/fs1/FMW_Home/Oracle_EBS-app1 -jreLoc /opt/vis11/fs2/FMW_Home/Oracle_EBS-app1/jdk

# strace -p 25238

write(3, "Continue? (y/n) [n] ", 20)    = 20
read(0, "", 4096)                       = 0
write(3, "Continue? (y/n) [n] ", 20)    = 20
read(0, "", 4096)                       = 0
write(3, "Continue? (y/n) [n] ", 20)    = 20
read(0, "", 4096)                       = 0
write(3, "Continue? (y/n) [n] ", 20)    = 20
read(0, "", 4096)                       = 0
write(3, "Continue? (y/n) [n] ", 20)    = 20
read(0, "", 4096)                       = 0
write(3, "Continue? (y/n) [n] ", 20)    = 20
read(0, "", 4096)                       = 0
write(3, "Continue? (y/n) [n] ", 20)    = 20


WTF? This process wait answer. How it can be?

[Max]

I'm increase swap to avoid that situation. But process has failed on the same error and with the same reason, in /opt/vis11/fs1/EBSapps/ doesn't exists necessary directory. But adop removed this directory by his self!

Enabling EBS_LOGON trigger

ERROR while running FSCloneApply...
Wed Mar 25 14:22:55 2015
*******FATAL ERROR*******
PROGRAM : (/opt/vis11/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Wed Mar 25 14:22:55 2015
FUNCTION: main::runFSCloneApply [ Level 1 ]
ERRORMSG: /opt/vis11/fs2/EBSapps/comn/adopclone_db3h-dev/bin/adclone.pl did not go through successfully.

          [UNEXPECTED]Error occurred while executing "perl /opt/vis11/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/opt/vis11/fs2/inst/apps/VIS11_db3h-dev/appl/admin/VIS11_db3h-dev.xml -patchcontextfile=/opt/vis11/fs1/inst/apps/VIS11_db3h-dev/appl/admin/VIS11_db3h-dev.xml -promptmsg=hide -console=off -mode=create -sessionid=3 -timestamp=20150325_140903 -outdir=/opt/vis11/fs_ne/EBSapps/log/adop/3/fs_clone_20150325_140903/VIS11_db3h-dev"
          [UNEXPECTED]Error occurred while CLONE Patch File System from Run File System using command: "perl /opt/vis11/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/opt/vis11/fs2/inst/apps/VIS11_db3h-dev/appl/admin/VIS11_db3h-dev.xml -patchcontextfile=/opt/vis11/fs1/inst/apps/VIS11_db3h-dev/appl/admin/VIS11_db3h-dev.xml -promptmsg=hide -console=off -mode=create -sessionid=3 -timestamp=20150325_140903 -outdir=/opt/vis11/fs_ne/EBSapps/log/adop/3/fs_clone_20150325_140903/VIS11_db3h-dev".
          [EVENT]     Calling task: "Releasing ports on db3h-dev"; script: "/opt/vis11/fs2/inst/apps/VIS11_db3h-dev/admin/install/txkCloneAcquirePort.pl"; args: " -option=close -server=db3h-dev.oebs.yandex.net -ports=5576,7021,7621,7421,7221,7821,6120,6220 -log=/opt/vis11/fs_ne/EBSapps/log/adop/3/fs_clone_20150325_140903/VIS11_db3h-dev/txkCloneAcquirePort.log"
          [EVENT]     [START 2015/03/25 14:22:56] Releasing ports on db3h-dev
            [EVENT]     Calling: /opt/vis11/fs2/inst/apps/VIS11_db3h-dev/admin/install/txkCloneAcquirePort.pl
          [EVENT]     [END   2015/03/25 14:22:56] Releasing ports on db3h-dev
            [ERROR]     Environment file: /opt/vis11/fs1/EBSapps/appl/APPSVIS11_db3h-dev.env does not exists

            Log file: /opt/vis11/fs_ne/EBSapps/log/adop/3/adop_20150325_140903.log


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


adop exiting with status = 2 (Fail)


How I can clone RUN fs if fs_clone doesn't work?

[Max]

Ok. I've decided to prepare RUN FS for apply NLS patch and it failed.

$ adop phase=prepare
...
...
...
SETUP VALIDATION is in progress. This may take few minutes to complete.

Completed execution of all tests. There are warnings.

=========================== START OF VALIDATION REPORT ====================

Validation Results for Node: db3h-dev
---------------------------------------
[WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.

[WARNING]: Either some of the required entries in /etc/hosts file might be missing (e.g. localhost or hostname) OR the file /etc/hosts could not be read.

There are warnings in tests, please check /opt/vis11/fs_ne/EBSapps/log/adop/4/prepare_20150325_150040/VIS11_db3h-dev/ADOPValidations_Wed_Mar_25_15_01_12_2015.log

=========================== END OF VALIDATION REPORT ======================
      [EVENT]     [END   2015/03/25 15:02:27] Executing txkADOPValidation script on db3h-dev
    [EVENT]     [END   2015/03/25 15:02:27] Executing txkADOPValidation script on db3h-dev
    [START 2015/03/25 15:02:30] prepare phase
      [UNEXPECTED]Dictionary Corrupted:
      [START 2015/03/25 15:02:37] Data Dictionary Curruption Details
        OWBSYS          HAB_RT_EXEC_PROC_RUN_METRICS                   OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
        OWBSYS          WB_RTI_OBJECT_CATALOG                          OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
        OWBSYS          UAB_RT_EXEC_PROCESS                            OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
        OWBSYS          UAB_RT_EXEC_PROCESS_RUNS                       OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
        OWBSYS          UAB_RT_EXEC_PROC_RUN_COUNTS                    OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
        OWBSYS          UAB_RT_EXEC_PROC_RUN_METRICS                   OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
        PUBLIC          WB_OLAP_AW_PRECOMPUTE                          OWBSYS          WB_OLAP_AW_PRECOMPUTE                          TS mismatch: 23-SEP-10 05:17:34 26-NOV-12 18:48:25
        OWBSYS          ALL_RT_INSTALLATIONS                           OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
        OWBSYS          RAB_RT_INSTALLATIONS                           OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
        OWBSYS          HAB_RT_EXEC_PROCESS                            OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
        OWBSYS          HAB_RT_EXEC_PROCESS_RUNS                       OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
        OWBSYS          HAB_RT_EXEC_PROC_RUN_COUNTS                    OWBSYS          WB_RT_VERSION                                  TS mismatch: 16-SEP-10 09:13:44 26-NOV-12 18:48:19
      [END   2015/03/25 15:02:37] Data Dictionary Curruption Details
      NOTE: Please contact Oracle Support and request them to open a bug against
            Oracle Application Install(166), Component Online Patching(OP).
          [ERROR]     Prepare phase completed with errors or warnings. Please check logfiles.
          Log file: /opt/vis11/fs_ne/EBSapps/log/adop/4/adop_20150325_150040.log


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


adop exiting with status = 1 (Fail)

[ErmanArslansOracleBlog]

Hi,

1)
Please send me the action plan that you have followed for installin 12.2.0 and upgrading it to 12.2.4 .. Send me the action plan you followed and specify where the issue begins..

2)
Which documents did you follow?

Please send me in the following format:

12.2.0 installation Docid ?
12.2.4 upgrade Docid?
Please specify something like "Problem started in this step in this Docid?"

[Max]

1. I'm downloaded VISION from https://edelivery.oracle.com/EPD/Download/get_form
2. Installed required packages.
3. Execute startCD/Disk1/rapidwiz/bin/buildStage.sh
4. Execute startCD/Disk1/rapidwiz/rapidwiz
5. After installation 12.2.0 I'm downloaded 12.2.4 Update Pack for Linux x86-64
6. Download R12.AD.C.delta.5 patch and Download R12.TXK.C.DELTA.5 patch
7. unzip and move /opt/vis11/fs_ne/EBSapps/patch/
8. Execute grants script
9. adop phase=prepare
10. adop phase=apply patches=18283295
11. adop phase=apply patches=18288881
12. adop phase=apply patches=17919161
13. adop phase=finalize
14. adop phase=cutover
15. adop phase=cleanup
16. adop phase=fs_clone

And on the 16 step problem appeared how I show you above. EBS logon trigger problem. And whole directories under PATCH FS was deleted.

[Max]

I'm used this document for upgrade to 12.2.4: Doc ID 1617458.1

[ErmanArslansOracleBlog]

Hi Max,

Your patch numbers are old. For example : 18283295 is for AD.C.Delta.5 .. But this should not create such problems, if applied properly.. On the other hand, I think your approach is wrong.

Please read the up-to-date documents.. Login to oracle support and open the document 1617461.1.. This way, you will work with the current copy.. You will use AD.C.Delta.6 now.

Just delete your installation.. Delete the directories and everyting.
Then install 12.2.0 again. A Clean install. .Check the directories and see that they are present..
Then to upgrade it to 12.2.4 ; follow 1617461.1 again..
But this time you need to follow the instructions line by line.
Do the following sections;

Section 2: Apply Required Database Patches --> it will redirect you to Document 1594274.1, once you are done with Document 1594274.1, come back to your main document  and procedd with the AD and TXK upgrades
Section 3: Apply Consolidated Seed Table Upgrade Patch (Required)
Section 4: Apply the R12.AD.C.Delta.6 and R12.TXK.C.Delta.6 Release Update Packs --> choose PATH A , it will redirect you to  "Section 3: Application Tier Technology Patches and Bug Numbers' of My Oracle Support Knowledge Document 1594274.1, Oracle E-Business Suite Release 12.2: Consolidated List of Patches and Technology Bug Fixes" Once you are done come back.
Follow the instructions once you are done with the document 1594274.1.
Then after finishing Section 3, proceed wit the
Section 5 is up to you, read it.. It is conditional
Proceed with section 6
Section 6 :  Apply Oracle E-Business Suite 12.2.4 Release Update Pack, choose "Installing a new Oracle E-Business Suite system Path A"

"I m writing the lines in the docs here, they are already present in the docs, please crosscheck"

You need to follow the instructions line by line.
You need go to the other documents if needed.
For example:
IF the doc says "The application tier technology patches you need to apply are listed in 'Section 3: Application Tier Technology Patches and Bug Numbers' of My Oracle Support Knowledge Document 1594274.1, Oracle E-Business Suite Release 12.2: Consolidated List of Patches and Technology Bug Fixes."
Then you need to go to 1594274.1 and apply the required patches for your environment.. Once you are done you come back to your main document, and proceed with the next line.

[Max]

Hi Erman. Of course I'm follow line by line the documents. For example now I follow Doc ID 1617458.1 and stuck on the step 3 (adop phase=apply apply_mode=downtime patches=17919161). Because 10 of 80 workers in the running mode, but nothing happen really. I've investigated the situation and figured out that the workers do nothing because they have got error.

java.sql.SQLRecoverableException: IO Error: Got minus one from a read call

So, m abort adop executions and try to re-run it. But now I have corrupted packages.

$ adop phase=apply apply_mode=downtime patches=17919161 restart=yes

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Validating credentials...

Initializing...
    Run Edition context  : /opt/vis12/fs1/inst/apps/VIS12_db3h-dev/appl/admin/VIS12_db3h-dev.xml
    Patch edition context: /opt/vis12/fs2/inst/apps/VIS12_db3h-dev/appl/admin/VIS12_db3h-dev.xml
Reading driver file (up to 50000000 bytes).
    Patch file system freespace: 6156.11 GB

Validating system setup...
    Node registry is valid.
    [ERROR]     Failed to execute SQL statement :
 select ad_zd_adop.is_icm_alive() from dual
    [ERROR]     Error Message :
    [UNEXPECTED] Error occured while checking application service status


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


adop exiting with status = 2 (Fail)



SQL> select apps.ad_zd_adop.is_icm_alive() from dual;
select apps.ad_zd_adop.is_icm_alive() from dual
       *
ERROR at line 1:
ORA-04063: package body "APPS.FND_CONC" has errors
ORA-06508: PL/SQL: could not find program unit being called: "APPS.FND_CONC"
ORA-06512: at "APPS.AD_ZD_ADOP", line 1671

[Max]

So. Investigation show me that.

SQL> select object_type,status from dba_objects where object_name = 'FND_CONC';

OBJECT_TYPE    STATUS
------------------- -------
PACKAGE    VALID
PACKAGE BODY    INVALID


SQL> alter package FND_CONC compile body;

Warning: Package Body altered with compilation errors.

SQL> show errors
Errors for PACKAGE BODY FND_CONC:

LINE/COL ERROR
-------- -----------------------------------------------------------------
158/11 PLS-00323: subprogram or cursor 'CANCEL_PATCH_REQUESTS' is
         declared in a package specification and must be defined in the
         package body

Obviously I need new installation. But where is the guarantee that all be alright?

P.S. I've followed line by line through documentation.

[ErmanArslansOracleBlog]

Hi Max,

Please follow this document for this problem and then restart

Issues While Re-starting 12.2.4 RUP Patch 17919161 Failed With ORA-04063, ORA-06508, ORA-06512 on FND_CONC Package Invalid (Doc ID 1926427.1)

[ErmanArslansOracleBlog]

Also dont use 80 parallel workers this time.
Use for example 12 workers.. Also Before restarting the patch application restart your db (just in case, to clean the db shadow processess)

If you get the same error; I mean "java.sql.SQLRecoverableException: IO Error: Got minus one from a read call "; then you need the analyze the adop log and db logs as well, db alert log, listener log,  Os log and Application's adop logs
Also these error may be related with the processes and sessions parameters of your db ..
Increase them and restart your db before applying the patch..
I believe, these actions will fix that.

So;
increase processes and sessions parameter.
Increase SGA ,PGA a little bit
Restart the db.
Follow the note, that I have sent in my previous update.
Run with 12 or 16 parallel.

[Max]

Ok, I'll see this document. By the way I'm not use 80 workers by my self, adop has done it. I've followed instructions!

[ErmanArslansOracleBlog]

Increase processes and sessions parameters of the database and restart, as well

[Max]

I've reviewed the document and I've done all this steps couple hours ago and unfortunately this is not helped me.

AutoPatch error:
adpcdlvlGetTeInfoFromDB(): Unexpected Database Error


AutoPatch error:
ORA-04063: package body "APPS.AD_TRACKABLE_ENTITIES_PKG" has errors
ORA-06508: PL/SQL: could not find program unit being called: "APPS.AD_TRACKABLE_ENTITIES_PKG"
ORA-06512: at line 3

[ErmanArslansOracleBlog]

try to compile the package what s the error?
Also, if it is not invalid; check the permission , you may need to run adgrants for correcting the apps grants.

[Max]

SQL> select object_type,status from dba_objects where object_name = 'AD_TRACKABLE_ENTITIES_PKG' and owner = 'APPS';

OBJECT_TYPE         STATUS
------------------- -------
PACKAGE             VALID
PACKAGE BODY        INVALID

SQL> alter package apps.AD_TRACKABLE_ENTITIES_PKG compile body;

Warning: Package Body altered with compilation errors.

SQL> show errors
Errors for PACKAGE BODY APPS.AD_TRACKABLE_ENTITIES_PKG:

LINE/COL ERROR
-------- -----------------------------------------------------------------
38/13    PLS-00323: subprogram or cursor 'RESET_USED_FLAG' is declared in
         a package specification and must be defined in the package body