R12.2.4 adcfgclone on taret - unable to delete managed servers, OHS Config errors

[bhaskar]

Hi Erman,

Thank you for writing the blog. It helps other DBAs :)

We are facing an issue in cloning R12.2.4 from production to a new server. Source has 4 node application servers and 2 node RAC.
Target is a one node application and one single instance database on (stand alone ASM).

We are on DELTA7 patchset.While running adcfgclone.pl appsTier (with dualfs or without dualfs) following error is thrown.
I saw your other blog where you advised to run adpreclone before starting post clone steps on TARGET.
However, in our case its failing at the very first step, adcfgclone appsIier

PS:error is occuring while deleting managed servers defined on secondary servers at source.A SEV1 SR is open for last 10 days with no luck

Can you please share your thoughts on error and resolution ?  THank you !

ERROR: Failed to delete the managed server oacore_server2
ERROR: Unable to delete some of the managed servers. Please delete them manually
AdminServer is in RUNNING mode.
Verified that AdminServer is Up and Running
Inside invokeOHSClone
Checking whether instance EBS_web_ERPDEV_OHS1 is registered with domain...
RegisteredInstanceNames MBean attribute value is NULL
Instance EBS_web_ERPDEV_OHS1 is not registered with Domain
/shared-u01/oracle/ERPDEV/fs1/FMW_Home/webtier/instances/EBS_web_ERPDEV_OHS1 directory deleted successfully.

START: Instantiating the OHS Config.
Running /shared-u01/oracle/ERPDEV/fs1/FMW_Home/webtier/perl/bin/perl  -I/shared-u01/oracle/ERPDEV/fs1/FMW_Home/webtier/perl/lib/5.10.0 -I/shared-u01/oracle/ERPDEV/fs1/FMW_Home/webtier/perl/lib/site_perl/5.10.0 -I/shared-u01/oracle/ERPDEV/fs1/EBSapps/appl/au/12.0.0/perl -I/shared-u01/oracle/ERPDEV/fs1/FMW_Home/webtier/ohs/mod_perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /shared-u01/oracle/ERPDEV/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkSetOHSConfig.pl cfgCloneInstance -ctxfile=/shared-u01/oracle/ERPDEV/fs1/inst/apps/ERPDEV_la-erp-app-dev/appl/admin/ERPDEV_la-erp-app-dev.xml -archvloc=/shared-u01/oracle/ERPDEV/fs1/EBSapps/comn/clone/FMW/OHS/ohsarchive.jar -mvpln=/shared-u01/oracle/ERPDEV/fs1/EBSapps/comn/clone/FMW/OHS/moveplan.xml -t2plogloc=/shared-u01/oracle/ERPDEV/fs1/inst/apps/ERPDEV_la-erp-app-dev/admin/log/clone/run/ohsT2PApply -logdir=/shared-u01/oracle/ERPDEV/fs1/inst/apps/ERPDEV_la-erp-app-dev/admin/log/clone/run -console=off -promptmsg=hide
Script Executed in 2703 milliseconds, returning status 1
ERROR: Script failed, exit code 1

Thank you,
Bhaskar

[ErmanArslansOracleBlog]

Hi ,

Thanks for your feedback Bhaskar.

Please send me the logs under the directory: /shared-u01/oracle/ERPDEV/fs1/inst/apps/ERPDEV_la-erp-app-dev/admin/log/clone/run

[Bhaskar]

Thank you for the quick response, Erman. Can you please let me know your email so that I can send logs to your email ?

Thanks,
Bhaskar

[ErmanArslansOracleBlog]

Please write the following to your SR. It will increase the SR's speed.
Also please read the following and try to get why WLST is failing;

The main class that triggers the sub class in your case is; Binary file apps/ad/clone/ApplywlsTechStack.class


  if (!FSCloneApplyAppsTier.isFSClone() && this.srcSecondaryNodeManagedSrvrs != null && this.srcSecondaryNodeManagedSrvrs.size() > 0) {
            if (!WLSDomainCleanupUtil.deleteManagedServers(this.srcSecondaryNodeManagedSrvrs, (String)this.m_sContext, (String)ApplywlsTechStack.getWlsAdminPasswd())) {
                LogFile.getLogFile().severe("Unable to delete some of the managed servers. Please delete them manually");
            } else if (!WLSDomainCleanupUtil.deleteSecondaryMachines((String)this.m_sContext, (String)ApplywlsTechStack.getWlsAdminPasswd())) {
                LogFile.getLogFile().severe("Unable to delete some of the secondary machines. Please delete them manually");
            }
        }
        this.updateCloneTaskXML("APPLY-WLSCONFIG-DONE");
    }


This class execute the class named "oracle.apps.ad.clone.util.WLSDomainCleanupUtil" and "WLSDomainCleanupUtil" class encounter erros while running the WLST commands.
So in the lower level, it is WLST that causes this error.


What this java does is to delete the managed server using WLST.

A command like this one;
 stringBuffer.append("edit()\nstartEdit()\ncd('/Servers/" + string8 + "')\n" + "set('Cluster','standalone')\n" + "save()\n" + "cd('/Servers')\n" + "delete('" + string8 + "','Server')\n" + "activate()\n" + "cancelEdit('y')\n");

So you should check the admin password is valid.
The server is there in the path (use WLST to check it manually" (execute the WLST commands written above and see if there is abnormal situation.)

See the error that WLST is giving..

For using WLST please check my document:

http://ermanarslan.blogspot.com.tr/2015/02/weblogic-wlst-general-information-and.html

[bhaskar]

Thank you Erman ! Appreciate your efforts in assisting us.  I added the java code that you provided to the SR .

The weblogic password we supplied seems to work. I can say this because, we logged onto  weblogic console with the same password after adcfgclone failed with the error.  Admin server was up when this occured, so same password did work.

We can delete managed servers manually, provided the first step "adcfgclone.pl appsTier (with or without dualfs) completes successfully.  (i.e., to run preclone, then delete managed servers and then rerun adcfgclone)


The error messages also indicate that it deleted EBS_web_SID_OHSn directory ? Can you share your thoughts around that.

RegisteredInstanceNames MBean attribute value is NULL
Instance EBS_web_ERPDEV_OHS1 is not registered with Domain
/XXX/oracle/ERPDEV/fs1/FMW_Home/webtier/instances/EBS_web_ERPDEV_OHS1 directory deleted successfully.


Also , is there a way to exclude these manage servers from moveplan.xml or any other config files so that it will not be an issue while adcfgclone runs.


We noticed that , it does delete otehr manged servers defined on other nodes. However, it fails at deleting managed oacore_server2.

I am not sure what timezone you are in, but if you have time to spare, I can set up a short go to meeting.


Thank you !
BHaskar

[ErmanArslansOracleBlog]

Hi ,

In setup guide it says the following;

"so, in the Post Cloning steps for the Target "Run Edition File System", it says ensure that managed servers and machines of the Source secondary nodes are deleted from the Target primary"

Read pages 17-20, it explains the multi node cloning process.

This might be what you have missed?

-->Reference: http://docs.oracle.com/cd/E51111_01/current/acrobat/122ebssu.pdf

 Configure the Target "Run Edition File System" by running:
$ perl adcfgclone.pl appsTier
• Ensure that Port Pool provided for the Target fs1 is different from that of
any of the Source nodes.
• The Weblogic domain of the Target primary node would be configured
with all the Source managed servers and machines.
Perform the following Post Cloning steps for the Target "Run Edition File
System" to ensure that managed servers and machines of the Source
secondary nodes are deleted from the Target primary node:
1. Log in to the Oracle WebLogic Server Administration Console.
2. Under the Domain Structure section, select Environment > Servers.
Click on the Lock & Edit button in the 'Change Center' panel and delete
the managed servers whose machine is configured as
<hostname>_<offset>. For example, managed servers
corresponding to your source secondary nodes. Click on the Activate
Changes button in the 'Change Center' panel to activate the changes
after deletion.
3. Under the Domain Structure section, select Environment > Machines.
Click on the Lock & Edit button in the 'Change Center' panel and delete
the machine whose machine is configured as <hostname>_<offset>.
For example, machines corresponding to your source secondary nodes.
17-20    Oracle E-Business Suite Setup Guide
Click on the Activate Changes button in the 'Change Center' panel to
activate the changes after deletion.
After the deletion, you should have only one machine associated with
Target "Run Edition File System"

[Bhaskar]

Thanks Erman. We followed those steps. However, it failed at the step,  (#4 on page 17-19)
Configure the Target "Run Edition File System" by running:
$ perl adcfgclone.pl appsTier

--If the above step is successful, then we can log on to admin console and then delete the additional managed servers defined on other nodes , run adpreclone and then configure patch file system.

However, since adcfgclone appstier failed, we could not proceed further.

We also tried removing the managed servers, although adcfg failed and ran adpreclone.  adpreclone failed while  Precloning the OHS Config.

You may have noticed the following in original error message :

Checking whether instance EBS_web_ERPDEV_OHS1 is registered with domain...
RegisteredInstanceNames MBean attribute value is NULL
Instance EBS_web_ERPDEV_OHS1 is not registered with Domain
/XXX/oracle/ERPDEV/fs1/FMW_Home/webtier/instances/EBS_web_ERPDEV_OHS1 directory deleted successfully.


Removal of OHS1 directory is causing the issues. If we can address OHS part, managed server part may be handled using the steps mentioned in your earlier post and the setup guide.

THanks,
Bhaskar

[ErmanArslansOracleBlog]

This may be related even with a open port issue.
If I could see the log, I would have the further details.
Send me the logs, to my email : [e-mail]

Also at the same time, try this:

Clean this failed clone also clear the inventory by following Note 295185.1: How to Recreate the Global oraInventory
Re-do the clone, and update me.

[Bhaskar]

Thank you ! We tried the following steps earlier.

1. Renamed inventory.xml to inventory.xml.bkp
2. Removed FMW_Home directory
3. Removed inst directory.
4. From a brand new session(terminal) where nothing has been sourced, navigated to your COMMON_TOP/clone/bin directory
5.Without any env variables, reran adcfgclone.pl appsTier and hit the same error.*****

When we rename fmw_home and inst directories, will it not create them afresh by reading the JAR files already available in clone directory and hit the same issue again ? Will deleting the managed servers have any effect , if we rename the inst,fmwhome and re run the adcfg again ?

Can you provide more details on openport issue that you mentioned ?

Regards,
Bhaskar

[ErmanArslansOracleBlog]

I analzyed the log files;

here is the situation:

The problem is in "Modifying T2P plan"

You have a null pointer in provision_05301843.log

StackTrace:
java.lang.NullPointerException
        at oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.updateModWLSDirectives(UpdateOhsT2PMovePlan.java:812)
        at oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.updateModWLSConfig(UpdateOhsT2PMovePlan.java:721)
        at oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.updateModWLSConfig(UpdateOhsT2PMovePlan.java:698)
        at oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.updateModWLSConfig(UpdateOhsT2PMovePlan.java:698)
        at oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.processMoveplan(UpdateOhsT2PMovePlan.java:631)
        at oracle.apps.fnd.txk.util.UpdateOhsT2PMovePlan.main(UpdateOhsT2PMovePlan.java:1063)

[ErmanArslansOracleBlog]

When this issue started? Any recent changes? After TXK upgrade maybe?

[ErmanArslansOracleBlog]

So it seems , it is failing while updating the moveplan.
Can you send me the moveplan that it is trying to update. Lets see what s in there,  Any abnormal lines or so..

[Lakshmikanth Boinee]

We have sent you moveplan.xml from location:

/shared-u01/oracle/ERPDEV/fs1/EBSapps/comn/clone/FMW/OHS


Thank you

[ErmanArslansOracleBlog]

Okay. I checked the file.
Can you send me your EBS Weblogic config?
I want to see your weblogic managed servers + weblogic clusters + weblogic cluster & managed server relationships.
Send me Weblogic Console screenshots covering the above.

[Bhaskar]

Hi Erman,

Thank you for the response !

The error was resolved after we removed certain portion of ADF code from moveplan.xml in clone directory.  I sent you the moveplan.xml before and after making changes.(changed host names to dummy names)  Modified the moveplan.xml in clone/FMW/OHS directory and executed adcfgclone for the first time.


<value>/ADF</value>       
<itemMetadata>       
<dataType>STRING</dataType>       
<scope>READ_ONLY</scope>       
</itemMetadata>       
<configProperty>       
<name>WebLogicCluster</name>       
<value>ex-amp-le1.planet.com:7209,ex-amp-le2.planet.com:7209,ex-amp-le1.planet.com:7210,ex-amp-le2.planet.com:7210</value>       
<itemMetadata>       
<dataType>STRING</dataType>       
<scope>READ_WRITE</scope>       
</itemMetadata>       
</configProperty>       
</configProperty>       
<configProperty>       
<name>Location</name>

PS: We have an ADF application deployed in EBS source instance.  This ADF application is deployed on  same managed servers as EBS. Perhaps, this was causing an issue ?

adcfgclone completed successfully and we were able to clone patch FS and bring up services. However, login page was throwing 404 error and we noticed that requests are being redirected to source instance(production). Apparently, some config files like mod_wl_ohs had entries related to source. We resolved the issues by manually deleting managed servers and modifying wl_mos_ohs files.

Issue resolution is still in progress.  We are trying to figure the right approach when ADF is deployed at source EBS instance. If you have any suggestions, please let me know !

Thanks, B

[ErmanArslansOracleBlog]

Hi ,

There are 2 facts;

1) ADF applications are not intended to be used on the same Weblogic server home for E-Business Suite.

2) Online patching manages automated movement of applications/configurations between the two Oracle Homes ( patch and run), and "tools are not aware of any additional applications or customizations". Thus ADF is not intended to be installed inside the FMW_home that EBS uses. It should be installed on a separate Weblogic server.

You can check : FAQ for Integration of Oracle E-Business Suite and Oracle Application Development Framework (ADF) Applications (Doc ID 1296491.1)

So, based on these , probably your ADF application is causing the issue.
Besides, the ADF application definition in the moveplan.xml is written before the OA_HTML itself. So the code parse it before the OA_HTML and as it is not hardoced/not expected, it may be encountring the errors.

I reuqested your cluster and managed server conf, because I saw there were 2 definition in your moveplan.xml, which includes the same managed server+port, that is the oacore_server2:port

I didnt see the ADF there, but when you say that there is an ADF, I recognized the situation..

You can try 3 things:
1)You may need to modify the code for that (put an if clause or something like that in "UpdateOhsT2PMovePlan.class" to make it ignore your ADF application)  -- this will be super unsupported :)

2)You can make Oracle to send you a patch or a modified java class for the above(for the Step1)

3)You can try placing your ADF application in to a new managed server (lets say create a managed server named blabla and place it there) --unsupported (creating a custom managed server in EBS 12.2), but can be tried with taking the risk

4)You can try changing the order of the moveplan.xml, and make the ADF portion  to be after OA_HTML portion..

On the other hand, 3-4 may not work, you need to try and see.
The best solution is , placing ADF application outside the EBS.

Well, waiting for your updates on this.

[Bhaskar]

Thank you Erman ! Appreciate your efforts in analyzing the issue.

1)You may need to modify the code for that (put an if clause or something like that in "UpdateOhsT2PMovePlan.class" to make it ignore your ADF application)  -- this will be super unsupported :)

Modifying the above classes is beyond my expertise. I will explore that option with internal development team.

2)You can make Oracle to send you a patch or a modified java class for the above(for the Step1)
--We will ask oracle if they can provide a patch or an other alternative.

3)You can try placing your ADF application in to a new managed server (lets say create a managed server named blabla and place it there) --unsupported (creating a custom managed server in EBS 12.2), but can be tried with taking the risk
--We already hosted ADF application on oacore servers. oacore servers are servicing both EBS and ADF, Did you mean to create exclusive oacore managed servers for ADF ??

4)You can try changing the order of the moveplan.xml, and make the ADF portion  to be after OA_HTML portion..
--We havent tried moving ADF code to a next section after OA, as you advised earlier. We may try that in a test environment. Although we edited the moveplan in FMW/OHS directory, those entries reappeared after the first phase of adcfgclone. Provided details at the end. Asked oracle to validate our changes. Follow up in progress.

On the other hand, 3-4 may not work, you need to try and see.
The best solution is , placing ADF application outside the EBS.
--Yes, this solution is underway. Team is working on having ADF outside EBS.

Following steps helped us in overcoming the issue.

SEQ1: Remove a portion of ADF entries from moveplan.xml in $COMMON_TOP/clone/FMW/OHS
SEQ2: run adcfgclone.pl appsTier (For RUN edition)
*** succeeded*** .
SEQ3: Started Admin server. Logged onto Admin console and removed entries related to other managed servers and machines( Doc ID 1615332.1 )
SEQ4: Checked mod_wl_ohs and apps.conf and noticed "entries related to source(production instance)". **NO ACTION PERFORMED in this iteration**
SEQ5: Checked config.xml in EBS_DOMAIN_HOME/config and noticed NO entries exist related to source(production instance).
SEQ6: Ran adpreclone.pl appsTier. (successfully Completed)
SEQ-6.1: Noticed ADF entry reappeared in moveplan.xml in $COMMON_TOP/clone/FMW/OHS after preclone.**ADF entries are being read from some other files. Perhaps, removing ADF entry in the SEQ1 was not sufficient ?? MAy be there are other files to look at ??
SEQ7: CLEANED UP RUN edition, Removed Inventory, Unset environment ( will start all over using new JARs created by preclone)
SEQ8: adcfgclone appsTier dualfs.  **SUCCESSFUL**
SEQ9: Brought up Admin Server.
SEQ10:Checked apps.conf and noticed entries related to source(production instance) were not present. **GOOD**
SEQ11: Brought up services.
SEQ12 : EBS Login page is throwing 404 error. **It wasnt redirecting to prod as earlier.**
SEQ13:Checked mod_wl_ohs noticed only one entry related to production. Entries related to dev ports were incorrect.Modified port# and values using command and manual editing  **used txkSetAppsConf.pl  to remove MS with incorrect port#**
SEQ14 : Restarted OPMN and Apache using ADOPMNCTL
SEQ15 : Login Page was displayed properly. Able to logon successfully ***SUCCESS***

Although ADF was commented out in moveplan, I could see ADF related application in EM console. We havent tested if its in "working" condition after making above changes.


Please review and let me know if you have any suggestions !

Thank you , Erman  !

Regards,B

[Bhaskar]

Forgot to mention.  Thank you for sharing your views on ADF on EBS. It helped.

[ErmanArslansOracleBlog]

*For 3)
Yes I mean mean create exclusive managed servers(not named oacore***) for ADF

*For "Noticed ADF entry reappeared in moveplan.xml in $COMMON_TOP/clone/FMW/OHS after preclone.**
ADF entries are being read from some other files. Perhaps, removing ADF entry in the SEQ1 was not sufficient ??
MAy be there are other files to look at ?? "

Did you delete the ADF Server before executing the preclone in the clone environment?

*For "Although ADF was commented out in moveplan, I could see ADF related application in EM console. We havent tested if its in "working" condition after making above changes. "

Do you have the ADF application running on  one of your oacore server residing in your new clone environment or not?

[Bhaskar]

Hi Erman,

Sorry for the late update ! Occupied with troubleshooting the issues.

We did not remove ADF server or the deployment ar source before running adpreclone at source. Only modifying moveplan.xml before running adcfgclong on target  resolved the earlier issue(appears so.Not a clean approach, though)

Although modifying moveplan.xml worked , it now throws errors while adding a second application node .

adclonectx addnode command gives the following error.

Error: Could not find or load main class .wlserver_10.3.server.lib.weblogic.jar:.shared-u01.oracle.ERPXYZ.fs1.EBSapps.comn.clone.bin....jlib.wlfullclient.jar:.shared-u01.oracle.ERPXYZ.fs1.EBSapps.comn.clone.bin....jlib.wlclient.jar:.shared-u01.oracle.ERPXYZ.fs1.EBSapps.comn.clone.bin....jlib.wljmxclient.jar:.shared-u01.oracle.ERPXYZ.fs1.EBSapps.comn.clone.bin....jlib.ojmisc.jar:.shared-u01.oracle.ERPXYZ.fs1.EBSapps.comn.clone.bin....jlib.obfuscatepassword.jar:.shared-u01.oracle.ERPXYZ.fs1.EBSapps.comn.clone.bin....jlib.emCfg.jar

ERROR: add node failed!!!
Please check logfile for details : /shared-u01/oracle/ERPXYZ/fs1/EBSapps/comn/clone/bin/../FMW/logs/sfslog


We removed the ADF  deployment from weblogic console on target and activated changes, reran autoconfig and adpreclone from RUN/patch, per add node document. issue still persists. Raised a new SR with oracle for this issue. No luck yet.

Would you know how to modify a moveplan.xml ? If I were to move the ADF section to a later part in moveplan.xml, would that cause any issue ? Please advise !

Thanks,
Bhaskar