Fs_clone failed

[sadaqmk.mohammed]

Hi Emran
I am trying to fs_clone on ebs R12.2.5, some strange errors are appearing. Please find the below error
Could you please help.
PROGRAM : (/oracle/DEV/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Tue Jun 16 10:50:32 2020
FUNCTION: TXK::IO::open [ Level 3 ]
MESSAGES:
error = Unable to open file
errorno = No such file or directory
file = </oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_100930/fs_clone/erptestwb01/TXK_SYNC_create/fsclone_stage/NONE

STACK TRACE
 at /oracle/DEV/apps/fs1/EBSapps/appl/au/12.0.0/perl/TXK/Error.pm line 168
        TXK::Error::abort('TXK::Error', 'HASH(0x1428a78)') called at /oracle/DEV/apps/fs1/EBSapps/appl/au/12.0.0/perl/TXK/Common.pm line 299
        TXK::Common::doError('TXK::IO=HASH(0x21bcc60)', 'HASH(0x2be1838)', undef) called at /oracle/DEV/apps/fs1/EBSapps/appl/au/12.0.0/perl/TXK/Common.pm line 314
        TXK::Common::setError('TXK::IO=HASH(0x21bcc60)', 'HASH(0x2be1838)') called at /oracle/DEV/apps/fs1/EBSapps/appl/au/12.0.0/perl/TXK/IO.pm line 291
        TXK::IO::open('TXK::IO=HASH(0x21bcc60)', 'HASH(0x281e948)') called at /oracle/DEV/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl line 5959
        main::searchFileContents('/oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_100930/fs...', 'Status: Completed Successfully') called at /oracle/DEV/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl line 5838
        main::runFSCloneStage() called at /oracle/DEV/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl line 2198
        main::createPatchApplTop() called at /oracle/DEV/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl line 659
    [UNEXPECTED]Error occurred running "perl /oracle/DEV/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/oracle/DEV/apps/fs1/inst/apps/DEV_erptestwb01/appl/admin/DEV_erptestwb01.xml -patchcontextfile=/oracle/DEV/apps/fs2/inst/apps/DEV_erptestwb01/appl/admin/DEV_erptestwb01.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=61 -timestamp=20200616_100930 -outdir=/oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_100930/fs_clone/erptestwb01"
    [UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /oracle/DEV/apps/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/oracle/DEV/apps/fs1/inst/apps/DEV_erptestwb01/appl/admin/DEV_erptestwb01.xml -patchcontextfile=/oracle/DEV/apps/fs2/inst/apps/DEV_erptestwb01/appl/admin/DEV_erptestwb01.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessioni

[ErmanArslansOracleBlog]

fs_clone complains about a missing file.
The file in question is weird -> /oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_100930/fs_clone/erptestwb01/TXK_SYNC_create/fsclone_stage/NONE

It is in fs_ne..

1) What is stored in the following directory ->
 /oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_100930/fs_clone/erptestwb01/TXK_SYNC_create/fsclone_stage

Please send me the output of the following->

cd to the following directory;
/oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_100930/fs_clone/erptestwb01/TXK_SYNC_create/fsclone_stage

execute ls -lrt and send me the output.


2) What is your AD-TXK Delta patch level?

[sadaqmk.mohammed]

[oracle@erptestwb01 ~]$ ls -ltr /oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_235147/fs_clone/erptestwb01/TXK_SYNC_create/fsclone_stage/NONE
ls: cannot access /oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_235147/fs_clone/erptestwb01/TXK_SYNC_create/fsclone_stage/NONE: No such file or directory
[oracle@erptestwb01 ~]$ ls -ltr /oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_235147/fs_clone/erptestwb01/TXK_SYNC_create/fsclone_stage/
total 16
drwxr-xr-x 2 oracle oinstall 4096 Jun 17 00:04 wlsT2PStage
drwxr-xr-x 2 oracle oinstall 4096 Jun 17 00:04 fmwT2PStage
drwxr-xr-x 2 oracle oinstall 4096 Jun 17 00:04 txkSetOHSConfig_06170004
drwxr-xr-x 2 oracle oinstall 4096 Jun 17 00:04 ohsT2PStage


2) What is your AD-TXK Delta patch level?
AD11 and TXK 11

[ErmanArslansOracleBlog]

Okay.. So your directory and its conents are fine..
The problem is the code. It can not get the correct directory name and it tries to open  NONE..

I will check and get back to you..

[sadaqmk.mohammed]

Thanks, I will wait for your update

Thanks in advance

[ErmanArslansOracleBlog]

Okay..
I am analyzing the code/perl..

It seems the clone_log_name variable is set to NONE.

Actually, the code first sets it to NONE.
my $clone_log = "NONE";

Then it checks the related directory and tries to find FSCloneStageAppsTier or FSCloneApplyAppsTier.

If it finds such a file, it generates the clone name using trDirPathToBase.
If it cannot find such a file, it will not do anything and clone_long will stay the same, NONE..

so in your case, the clone_log is set to NONE.

I quickly checked it  but this is a big code..

1)
Please send me the contents of the following directory (this is the outdir) from run fs->
I expect to see FSCloneStageAppsTier or FSCloneApplyAppsTier there..
ls -lrt /oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_100930/fs_clone/erptestwb01

2)
Please send me the value of the context variable named s_config_home from run fs.

3)
Please send me the contents of the following directory;

ls -lrt <the value/path of s_config_home>/admin/log/clone

4)
Please do the following search and send me the findings ->

with root->

cd /<EBS_install_base_directory>
find . -name "FSCloneStageAppsTier"
find . -name " FSCloneApplyAppsTier"

[sadaqmk.mohammed]

Hi
1)
ls -lrt /oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_100930/fs_clone/erptestwb01/
2)
[oracle]$ cat $CONTEXT_FILE |grep s_config_home
         <config_home oa_var="s_config_home">/oracle/DEV/apps/fs1/inst/apps/DEV_erptestwb01</config_home>
3)
[oracle]$ ls -ltr /oracle/DEV/apps/fs1/inst/apps/DEV_erptestwb01/admin/log/clone
total 16
drwxr-xr-x 2 oracle oinstall 4096 Jun 16 23:59 fmwT2PStage
drwxr-xr-x 2 oracle oinstall 4096 Jun 17 00:04 wlsT2PStage
drwxr-xr-x 2 oracle oinstall 4096 Jun 17 01:06 txkSetOHSConfig_06170004
drwxr-xr-x 2 oracle oinstall 4096 Jun 17 01:07 ohsT2PStage

4) cd /<EBS_install_base_directory>
find . -name "FSCloneStageAppsTier"
find . -name " FSCloneApplyAppsTier"

I could not find anything this point

[sadaqmk.mohammed]

Sorry correction Point 1

1)
ls -lrt /oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_100930/fs_clone/erptestwb01/
drwxr-xr-x 3 oracle oinstall 4096 Jun 16 10:50 TXK_SYNC_create
-rw-r--r-- 1 oracle oinstall 1965 Jun 16 10:50 txkCloneAcquirePort.log

[ErmanArslansOracleBlog]

1) you have "/oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_100930/fs_clone/erptestwb01/TXK_SYNC_create".. So what is inside TXK_SYNC_create directory?

*fs_clone certainly checks for a file name with the pattern FSCloneStageAppsTier or FSCloneApplyAppsTier inside
"/oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200616_100930/fs_clone/erptestwb01/TXK_SYNC_create/fsclone_stage/" directory, and it can't find it. That's why you end up with "NONE" as the filename..

use tree command to get the directory structure..

2) Send me the full log of this fs_clone session.

[sadaqmk.mohammed]

├── txkCloneAcquirePort.log
└── TXK_SYNC_create
    ├── fsclone_stage
    │   ├── fmwT2PStage
    │   │   ├── CLONE2020-06-16_10-39-13_1657580368.error
    │   │   └── CLONE2020-06-16_10-39-13_1657580368.log
    │   ├── ohsT2PStage
    │   │   ├── CLONE2020-06-16_10-50-29_1013065732.error
    │   │   ├── CLONE2020-06-16_10-50-29_1013065732.log
    │   │   ├── CLONE2020-06-16_10-50-30_346502354.error
    │   │   └── CLONE2020-06-16_10-50-30_346502354.log
    │   ├── txkSetOHSConfig_06161050
    │   │   ├── preCloneInstance.log
    │   │   └── provision_06161050.log
    │   └── wlsT2PStage
    │       ├── CLONE2020-06-16_10-42-37_1121262474.error
    │       ├── CLONE2020-06-16_10-42-37_1121262474.log
    │       ├── CLONE2020-06-16_10-50-28_1630742223.error
    │       └── CLONE2020-06-16_10-50-28_1630742223.log
    └── txkADOPPreparePhaseSynchronize.log
2) Please find the attachment20200616_100930.rar

[ErmanArslansOracleBlog]

Okay I will review the log..

Please send me the tree output for the directory path -> /oracle/DEV/apps/fs1/inst/apps/DEV_erptestwb01/admin/log/clone

I want to see all the subdirectory and files located in there.

--txkADOPPreparePhaseSynchronize.pl has functions to copy log directories from there to FS_NE...
Let's see whether we have a file name with the pattern FSCloneStageAppsTier or FSCloneApplyAppsTier there or not.

[ErmanArslansOracleBlog]

The logs that you sent approves my findings.. Findings from the perl code..

One thing in the log is interesting..
In txkADOPPreparePhaseSynchronize.log, we see the following ->

Method startCloneProcess() is already done. Skipping it...

So, it says the clone is already started and maybe that's it doesn't create new log files for it, and try to use the one associated with that "already started" clone process..  
However, the log file isn't there.. That s why the txkADOPPreparePhaseSynchronize.pl sets it to NONE and you get the expected and inevitable I/O error...

[ErmanArslansOracleBlog]

1) Let's check the clone status from db ;

SELECT clone_status||',' FROM ad_adop_session_patches WHERE status in ('R','F','N') AND bug_number = 'CLONE' ;

2) Send me the output of "adop -status"

[ErmanArslansOracleBlog]

Correction for 1 ->

SELECT clone_status||',' FROM ad_adop_session_patches WHERE status in ('R','F','N') AND bug_number = 'CLONE' AND node_name = 'erptestwb01' AND adop_session_id = 61


Bytheway, what happens when you start a new fs_clone..
Do you still get the adop_session_id=61?

If so, we need to do something for correcting this adop session information.. We need to clear things..

[sadaqmk.mohammed]

SQL> SELECT clone_status||',' FROM ad_adop_session_patches WHERE status in ('R','F','N') AND bug_number = 'CLONE' AND node_name = 'erptestwb01' AND adop_session_id = 61;

CLONE_STATUS||','
-------------------------------
STARTED,

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@erptestwb01 apps]$ adop -status

Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.11)
Session Id: 61
Command: status
Output: /oracle/DEV/apps/fs_ne/EBSapps/log/adop/61/20200618_104932/adzdshowstatus.out
===============================================================
Node Name Node Type  Phase   Status   Started Finished     Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
erptestwb01 master   FS_CLONE   FAILED   2020/06/10 16:43:11     186:10:32
File System Synchronization Type: Full
adop exiting with status = 0 (Success)

Yeah its using same session,

[ErmanArslansOracleBlog]

Aha! Most probably this is the problem.

Now, what I may recommend you is an update to the adop tables.. (ad_adop_session_patches)

However; I think you may also get rid of this by using force=yes option..

So please run adop phase=fs_clone force=yes

Update me with the outcome..

[sadaqmk.mohammed]

HI Emarn,

I have already tried before with force=yes, and normal same error was receiving.

[ErmanArslansOracleBlog]

Then, I will recommend you  "an update to the adop tables".. (-> ad_adop_session_patches)
What I think is, the script shouldn't see the CLONE_STATUS as STARTED.. When it sees CLONE_STATUS as STARTED, it doesn't create a  new log file and search for the old log file.. However; as the old log file isn't there, you get unable to open fıle ..(FILE name NONE)

Well.. As I mentioned I think a manual update is required.. But, I have no permission to do this kind of a recommendation.
So , please create a SR in Oracle Support and get the approval from them...

Update me with the outcome please.. This is an interesting issue, so I wonder the result.

[ErmanArslansOracleBlog]

One more thing,

Ensure there are no adop/adworker or other  adop related processes running in the systems..
The following note may help you on this ->

Script to Diagnose adop and Other AD-TXK Issues in Oracle E-Business Suite Release 12.2 [ Patch 19045166:R12.AD.C ] (Doc ID 1901242.1)
Details of running adop, adpatch and adworker processes..

However; most probably , you will get approval from SR and perform the manual update I mentioned....

[ErmanArslansOracleBlog]

Lastly, the following is worth trying.

adop phase=abort
adop phase=cleanup cleanup_mode=full
adop phase=fs_clone force=yes

If the issue persist, a manual update is required..