1. /opt/application/V12/fs_ne/EBSapps/log/adop/71/20200217_203937/fs_clone/validate/remote_execution_result_level1.xml: -------------------------------------------------------------------------------------------------------------------- [WARNING]: Found invalid cross references in OHS config files. [WARNING]: Found invalid cross references in node manager and domain config files. 2. /opt/application/V12/fs_ne/EBSapps/log/adop/71/20200217_203937/fs_clone/validate/remote_execution_result_level2.xml: -------------------------------------------------------------------------------------------------------------------- [WARNING]: Found invalid cross references in node manager and domain config files. :successful 3. /opt/application/V12/fs_ne/EBSapps/log/adop/71/20200217_203937/fs_clone/udkfinp03/TXK_SYNC_create/fsclone_apply/FSCloneApplyAppsTier_02172051.log: -------------------------------------------------------------------------------------------------------------------------------------------------- Problem invoking WLST - Traceback (innermost last): File "/opt/application/V12/fs2/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkSetAppsLibsTarget.py", line 909, in ? File "/opt/application/V12/fs2/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkSetAppsLibsTarget.py", line 705, in setNodeManagerCredentials java.lang.AssertionError: java.lang.reflect.InvocationTargetException at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:175) at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:192) at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:114) at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:1191) at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getNodeManagerPassword(SecurityConfigurationMBeanImpl.java:1180) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:525) at weblogic.management.jmx.modelmbean.WLSModelMBean.setAttribute(WLSModelMBean.java:630) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.setAttribute(DefaultMBeanServerInterceptor.java:746) at com.sun.jmx.mbeanserver.JmxMBeanServer.setAttribute(JmxMBeanServer.java:739) at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$14.run(WLSMBeanServerInterceptorBase.java:378) at java.security.AccessController.doPrivileged(Native Method) at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.setAttribute(WLSMBeanServerInterceptorBase.java:376) at weblogic.management.mbeanservers.internal.JMXContextInterceptor.setAttribute(JMXContextInterceptor.java:228) at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$14.run(WLSMBeanServerInterceptorBase.java:378) at java.security.AccessController.doPrivileged(Native Method) at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.setAttribute(WLSMBeanServerInterceptorBase.java:376) at weblogic.management.mbeanservers.edit.internal.RecordingInterceptor.setAttribute(RecordingInterceptor.java:130) at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$14.run(WLSMBeanServerInterceptorBase.java:378) at java.security.AccessController.doPrivileged(Native Method) at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.setAttribute(WLSMBeanServerInterceptorBase.java:376) at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$14.run(WLSMBeanServerInterceptorBase.java:378) at java.security.AccessController.doPrivileged(Native Method) at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.setAttribute(WLSMBeanServerInterceptorBase.java:376) at weblogic.management.mbeanservers.edit.internal.EditLockInterceptor.setAttribute(EditLockInterceptor.java:95) at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$14.run(WLSMBeanServerInterceptorBase.java:378) at java.security.AccessController.doPrivileged(Native Method) at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.setAttribute(WLSMBeanServerInterceptorBase.java:376) at weblogic.management.mbeanservers.internal.SecurityInterceptor.setAttribute(SecurityInterceptor.java:359) at weblogic.management.jmx.mbeanserver.WLSMBeanServer.setAttribute(WLSMBeanServer.java:303) at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$9$1.run(JMXConnectorSubjectForwarder.java:551) at java.security.AccessController.doPrivileged(Native Method) at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$9.run(JMXConnectorSubjectForwarder.java:549) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363) at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.setAttribute(JMXConnectorSubjectForwarder.java:544) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1511) at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1329) at java.security.AccessController.doPrivileged(Native Method) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1428) at javax.management.remote.rmi.RMIConnectionImpl.setAttribute(RMIConnectionImpl.java:743) at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source) at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667) at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146) at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518) at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:173) ... 52 more Caused by: weblogic.security.internal.encryption.EncryptionServiceException at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptBytes(JSafeEncryptionServiceImpl.java:139) at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptString(JSafeEncryptionServiceImpl.java:187) at weblogic.security.internal.encryption.ClearOrEncryptedService.decrypt(ClearOrEncryptedService.java:96) ... 57 more java.lang.AssertionError: java.lang.AssertionError: java.lang.reflect.InvocationTargetException .end err out. Script Executed in 11205 milliseconds, returning status 1 ERROR: Failed to execute /opt/application/V12/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/adPostDomainCloneCfg.pl START: Inside exitClone.... Updating status INCOMPLETE for ApplywlsTechStack START: Updating status INCOMPLETE for action ApplywlsTechStack END: Updated status INCOMPLETE for action ApplywlsTechStack 4. /opt/application/V12/fs_ne/EBSapps/log/adop/71/20200217_203937/fs_clone/udkfinp03/TXK_SYNC_create/txkADOPPreparePhaseSynchronize.log: ------------------------------------------------------------------------------------------------------------------------------------- Copying the directory 269 --------------------- 270 SOURCE : /opt/application/V12/fs1/EBSapps/comn/util 271 TARGET : /opt/application/V12/fs2/EBSapps/comn/util 272 273 Execute SYSTEM command : cp -rH /opt/application/V12/fs1/EBSapps/comn/util /opt/application/V12/fs2/EBSapps/comn/util 274 EXIT STATUS: 1 291 SQL Command: SELECT clone_status||',' FROM ad_adop_session_patches WHERE status in ('R','F','N') AND bug_number = 'CLONE' AND node_name = 'udkfinp03' AND adop_session_id = 71 292 clone_status = REMOVE-FMW-HOME 293 clone_status_from_caller = 8 294 clone_status_from_db = 7 295 Executing SYSTEM command: perl /opt/application/V12/fs1/EBSapps/comn/adopclone_udkfinp03/bin/adclone.pl java=/opt/application/V12/fs1/EBSapps/comn/adopclone_udkfinp03/FMW/t2pjdk mode=fscloneapply stage=/opt/application/V12/fs1/EBSapps/comn/adopclone_udkfinp03 component=appsTier appctx=/opt/application/V12/fs1/inst/apps/DBOFINP1_udkfinp03/appl/admin/DBOFINP1_udkfinp03.xml appctxtg=/opt/application/V12/fs2/inst/apps/DBOFINP1_udkfinp03/appl/admin/DBOFINP1_udkfinp03.xml 296 EXIT STATUS: 1 297 298 ====================================== 299 Inside copyCloneLogsToFSNE()... 300 ====================================== 301 302 Creating the directory: /opt/application/V12/fs_ne/EBSapps/log/adop/71/20200217_203937/fs_clone/udkfinp03/TXK_SYNC_create/fsclone_apply 303 304 Copying the directory 305 --------------------- 306 SOURCE : /opt/application/V12/fs1/inst/apps/DBOFINP1_udkfinp03/admin/log/clone 307 TARGET : /opt/application/V12/fs_ne/EBSapps/log/adop/71/20200217_203937/fs_clone/udkfinp03/TXK_SYNC_create/fsclone_apply 308 309 /opt/application/V12/fs1/EBSapps/comn/adopclone_udkfinp03/bin/adclone.pl did not go through successfully. 310 LOG DIRECTORY: /opt/application/V12/fs_ne/EBSapps/log/adop/71/20200217_203937/fs_clone/udkfinp03/TXK_SYNC_create/fsclone_apply. 311 *******FATAL ERROR******* 312 PROGRAM : (/opt/application/V12/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl) 313 TIME : Mon Feb 17 21:05:51 2020 5. /opt/application/V12/fs_ne/EBSapps/log/adop/71/20200217_203937/fs_clone/udkfinp03/adop.log: ------------------------------------------------------------------------------------------- [PROCEDURE] [END 2020/02/17 20:48:50] Checking if node "udkfinp08" was abandoned [STATEMENT] Fetching the node list where fs_clone or config clone did not complete [STATEMENT] Incomplete nodes: "udkfinp03,udkfinp04,udkfinp05,udkfinp06,udkfinp07,udkfinp08" [STATEMENT] Fetching the node list where fs_clone or config clone completed successfully [STATEMENT] Completed nodes: "" [STATEMENT] An incomplete fs_clone session exists: "ADOP Session ID:71" [STATEMENT] fs_clone will now be run on the incomplete node [STATEMENT] CLONE entry found [UNEXPECTED]Error occurred running "perl /opt/application/V12/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/opt/application/V12/fs1/inst/apps/DBOFINP1_udkfinp03/appl/admin/DBOFINP1_udkfinp03.xml -patchcontextfile=/opt/application/V12/fs2/inst/apps/DBOFINP1_udkfinp03/appl/admin/DBOFINP1_udkfinp03.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=71 -timestamp=20200217_203937 -outdir=/opt/application/V12/fs_ne/EBSapps/log/adop/71/20200217_203937/fs_clone/udkfinp03" [UNEXPECTED]occurred during CLONE Patch File System from Run File System, running command: "perl /opt/application/V12/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/opt/application/V12/fs1/inst/apps/DBOFINP1_udkfinp03/appl/admin/DBOFINP1_udkfinp03.xml -patchcontextfile=/opt/application/V12/fs2/inst/apps/DBOFINP1_udkfinp03/appl/admin/DBOFINP1_udkfinp03.xml -promptmsg=hide -console=off -mode=create -sync_mode=copy -sessionid=71 -timestamp=20200217_203937 -outdir=/opt/application/V12/fs_ne/EBSapps/log/adop/71/20200217_203937/fs_clone/udkfinp03". where applied_file_system_base = '/opt/application/V12/fs1' and patch_file_system_base = '/opt/application/V12/fs2' and bug_number='CLONE' and status <> 'Y' and appltop_id=3233 and node_name='udkfinp03'" [PROCEDURE] [END 2020/02/17 21:05:53] Updating status of patches [UNEXPECTED]fs_clone has failed. [PROCEDURE] [START 2020/02/17 21:05:54] Unlocking sessions table [STATEMENT] Unlocking ad_adop_sessions table for udkfinp03 with wait interval of 60 seconds and number of tries 6 [PROCEDURE] [END 2020/02/17 21:05:54] Unlocking sessions table 6. /opt/application/V12/fs_ne/EBSapps/log/adop/71/20200217_203937/fs_clone/remote_execution_result_level1.xml: -----------------------------------------------------------------------------------------------------------