Adop Prepare Has Failed With adclone.pl Error

[Berke]

Hello Erman,

I have recently encountered an error while I was running adop prepare phase.

It stopped with the following error:

---------------------
Lines #(419-421):
TIME    : Mon Apr 27 16:38:12 2015
FUNCTION: main::migrateCloneComponentApply [ Level 1 ]
ERRORMSG: /u01/install/APPS/fs1/EBSapps/comn/adopclone_T1VSTEBSAPP01/bin/adclone.pl did not go through successfully.
------------------------------------

In the adop log it first had its unexpected error while running txkADOPPreparePhaseSynchronize.pl:
----------------
   [UNEXPECTED]Error occurred while executing "perl /u01/install/APPS/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADO
PPreparePhaseSynchronize.pl -contextfile=/u01/install/APPS/fs1/inst/apps/IKYSDEV_T1VSTEBSAPP01/appl/admin/IKYSDEV_T1VSTEBSAPP01.
xml -patchcontextfile=/u01/install/APPS/fs2/inst/apps/IKYSDEV_T1VSTEBSAPP01/appl/admin/IKYSDEV_T1VSTEBSAPP01.xml -promptmsg=hide
 -console=off -mode=migrate -sessionid=39 -timestamp=20150427_162604 -outdir=/u01/install/APPS/fs_ne/EBSapps/log/adop/39/prepare
_20150427_162604/IKYSDEV_T1VSTEBSAPP01"
-------------------------


Then it led me to  TXK_SYNC_migrate_Mon_Apr_27_16_35_11_2015 because the problem has occurred with mode migrate.

Then I looked at txkADOPPreparePhaseSynchronize_Mon_Apr_27_16_35_11_2015.log in the  TXK_SYNC_migrate_Mon_Apr_27_16_35_11_2015 folder. There I saw the following error:

======================================
Inside copyCloneLogsToFSNE()...
======================================

Creating the directory: /u01/install/APPS/fs_ne/EBSapps/log/adop/39/prepare_20150427_162604/IKYSDEV_T1VSTEBSAPP01/TXK_SYNC_migra
te_Mon_Apr_27_16_35_11_2015/ohsConfig_apply

Copying the directory
---------------------
SOURCE : /u01/install/APPS/fs1/inst/apps/IKYSDEV_T1VSTEBSAPP01/admin/log/clone
TARGET : /u01/install/APPS/fs_ne/EBSapps/log/adop/39/prepare_20150427_162604/IKYSDEV_T1VSTEBSAPP01/TXK_SYNC_migrate_Mon_Apr_27_1
6_35_11_2015/ohsConfig_apply

/u01/install/APPS/fs1/EBSapps/comn/adopclone_T1VSTEBSAPP01/bin/adclone.pl did not go through successfully.
LOG DIRECTORY: /u01/install/APPS/fs_ne/EBSapps/log/adop/39/prepare_20150427_162604/IKYSDEV_T1VSTEBSAPP01/TXK_SYNC_migrate_Mon_Ap
r_27_16_35_11_2015/ohsConfig_apply.
*******FATAL ERROR*******
PROGRAM : (/u01/install/APPS/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Mon Apr 27 16:38:12 2015
FUNCTION: main::migrateCloneComponentApply [ Level 1 ]
ERRORMSG: /u01/install/APPS/fs1/EBSapps/comn/adopclone_T1VSTEBSAPP01/bin/adclone.pl did not go through successfully.

-----------------------

Then I read FSCloneApplyAppsTier_04271637.log which is in ohsConfig_apply folder. In it, it said that  it couldnt start weblogic admin server.
----------

START: Inside doClone..
Obfuscated Password log written to /u01/install/APPS/fs1/EBSapps/comn/adopclone_T1VSTEBSAPP01/FMW/obfuscatedPassword_04271637.lo
g

START: Checking if AdminServer is Up and Running
AdminServer is currently not running.
AdminServer is not UP. Starting UP AdminServer....
Running...../u01/install/APPS/fs2/inst/apps/IKYSDEV_T1VSTEBSAPP01/admin/scripts/adadminsrvctl.sh start -nopromptmsg forcepatchfs

You are running adadminsrvctl.sh version 120.10.12020000.9

Starting WLS Admin Server...
Refer /u01/install/APPS/fs2/inst/apps/IKYSDEV_T1VSTEBSAPP01/logs/appl/admin/log/adadminsrvctl.txt for details

adadminsrvctl.sh: exiting with status 1

adadminsrvctl.sh: check the logfile /u01/install/APPS/fs2/inst/apps/IKYSDEV_T1VSTEBSAPP01/logs/appl/admin/log/adadminsrvctl.txt
for more information ...  


.end std out.

.end err out.

Failed to start AdminServer. OHS clone cannot proceed further !!!
-------------

Lastly I looked /u01/install/APPS/fs2/inst/apps/IKYSDEV_T1VSTEBSAPP01/logs/appl/admin/log/adadminsrvctl.txt to see why it couldnt open admin server. There I saw the following error:

================================================================================
 

04/27/15-16:38:00 :: adadminsrvctl.sh version 120.10.12020000.9
Validated the passed arguments for the option ebs-get-serverstatus
AdminServer is currently not running.

Validated the passed arguments for the option ebs-nmstart-adminsrv
Checking if the Admin Server is already up.
The Admin Server is not already up.
Checking if the Node Manager is already up..
Traceback (innermost last):
  File "<string>", line 1, in ?
  File "<iostream>", line 123, in nmConnect
  File "<iostream>", line 653, in raiseWLSTException
WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Unexpected end of stream
Use dumpStack() to view the full stacktrace

Connecting to Node Manager ...

ERROR: Cannot connect to Node Manager. Check if port 5557 is correct
ERROR: Unable to connect to the Node Manager. The Admin Server cannot be started up.
Successfully disconnected from Node Manager.


04/27/15-16:38:11 :: adadminsrvctl.sh: exiting with status 1

================================================================================
 

04/27/15-16:38:14 :: adadminsrvctl.sh version 120.10.12020000.9
Validated the passed arguments for the option ebs-get-serverstatus
AdminServer is currently not running.

04/27/15-16:38:19 :: adadminsrvctl.sh: exiting with status 2

--------------

So what do you think that is causing this problem.

Thanks

[ErmanArslansOracleBlog]

Hi Berke,

1)Is the node manager listening the port?
netstat -an|grep 5557

2)What is written in node manager's log file?

[Berke]

Hello Erman,


Currently it is not listening to port 5557.

[applmgr@T1VSTEBSAPP01 ~]$ netstat -an|grep 5557
[applmgr@T1VSTEBSAPP01 ~]$


The nodemanager log under
/u01/install/APPS/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1 has not been editied recently.

[applmgr@T1VSTEBSAPP01 nmHome1]$ ls -lrt
total 52
-rw-r-----. 1 applmgr oinstall   178 Dec 17 17:01 nodemanager.domains
-rw-r--r--. 1 applmgr oinstall   963 Dec 17 17:41 nodemanager.properties
-rw-r--r--. 1 applmgr oinstall   130 Dec 17 17:44 nm_data.properties
-rw-r--r--. 1 applmgr oinstall     0 Apr 15 19:19 nodemanager.log.lck
-rw-r--r--. 1 applmgr oinstall 37864 Apr 15 19:22 nodemanager.log


Last messages were something like below:

<Apr 15, 2015 7:22:59 PM> <INFO> <EBS_domain_IKYSDEV> <forms-c4ws_server1> <Working director
y is '/u01/install/APPS/fs1/FMW_Home/user_projects/domains/EBS_domain_IKYSDEV'>
<Apr 15, 2015 7:22:59 PM> <INFO> <EBS_domain_IKYSDEV> <forms-c4ws_server1> <Rotated server o
utput log to "/u01/install/APPS/fs1/FMW_Home/user_projects/domains/EBS_domain_IKYSDEV/server
s/forms-c4ws_server1/logs/forms-c4ws_server1.out00043">
<Apr 15, 2015 7:22:59 PM> <INFO> <EBS_domain_IKYSDEV> <forms-c4ws_server1> <Server error log
 also redirected to server log>
<Apr 15, 2015 7:22:59 PM> <INFO> <EBS_domain_IKYSDEV> <forms-c4ws_server1> <Server output lo
g file is '/u01/install/APPS/fs1/FMW_Home/user_projects/domains/EBS_domain_IKYSDEV/servers/f
orms-c4ws_server1/logs/forms-c4ws_server1.out'>
[applmgr@T1VSTEBSAPP01 nmHome1]$  pwd
/u01/install/APPS/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1
------------


The node manager under FS2 has also no file regarding the startyp of AdminServer. Below I have shown that I entered to the nodemanager of FS2 and searched for nodemanager.log but I have seen nothing:

[applmgr@T1VSTEBSAPP01 common]$ cd nodemanager/
[applmgr@T1VSTEBSAPP01 nodemanager]$ ls
nmHome1  nodemanager.domains
[applmgr@T1VSTEBSAPP01 nodemanager]$ cd nmHome1/
[applmgr@T1VSTEBSAPP01 nmHome1]$ ls
nodemanager.domains
[applmgr@T1VSTEBSAPP01 nmHome1]$ ls -lrt
total 4
-rw-r-----. 1 applmgr oinstall 179 Apr 27 16:20 nodemanager.domains
[applmgr@T1VSTEBSAPP01 nmHome1]$ more nodemanager.domains
#Domains and directories created by Configuration Wizard
#Mon Apr 27 16:11:56 EEST 2015
EBS_domain_IKYSDEV=/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain_IKYSDEV
[applmgr@T1VSTEBSAPP01 nmHome1]$ pwd
/u01/install/APPS/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1
[applmgr@T1VSTEBSAPP01 nmHome1]$
-----------------

[ErmanArslansOracleBlog]

What happens when you run this command manually?

./u01/install/APPS/fs2/inst/apps/IKYSDEV_T1VSTEBSAPP01/admin/scripts/adadminsrvctl.sh start -nopromptmsg forcepatchfs

[Berke]

Hello Erman,

When I ran your command, it started like the the log below. However when I ran the adop phase=prepare while the adadminsrvctl was started forcefully in the patch fs, I see the following error.
-----
04/28/15-16:13:08 :: adadminsrvctl.sh version 120.10.12020000.9
04/28/15-16:29:18 :: adadminsrvctl.sh version 120.10.12020000.9
Validated the passed arguments for the option ebs-get-serverstatus
AdminServer is currently not running.

Validated the passed arguments for the option ebs-nmstart-adminsrv
Checking if the Admin Server is already up.
The Admin Server is not already up.
Checking if the Node Manager is already up..
Traceback (innermost last):
  File "<string>", line 1, in ?
  File "<iostream>", line 123, in nmConnect
  File "<iostream>", line 653, in raiseWLSTException
WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Unexpected end of stream
Use dumpStack() to view the full stacktrace

Connecting to Node Manager ...

ERROR: Cannot connect to Node Manager. Check if port 5557 is correct
ERROR: Unable to connect to the Node Manager. The Admin Server cannot be started up.
Successfully disconnected from Node Manager.


04/28/15-16:29:29 :: adadminsrvctl.sh: exiting with status 1
--

Do you think that it might be due to the faulty run of the adop syncronization(patch fs creation step) that is run while adop prepare phase?

Should I run with an option like skipsyncerror?

[Berke]

I should also have to tell you that I have run adop phase=fs_clone which means that we have copied the working state of EBS

[Berke]

Hello again,

An Update on the matter: Because I have run the prior command with nopromptmsg I couldnt enter the credentials correctly. But now  I have ran the adadminsrvctl  and successfully started it.


Connecting to Node Manager ...

Launching NodeManager ...
 Properties: {Arguments=-Djava.security.egd=file:/dev/./urandom,JavaHome=/u01/install/APPS/fs2/EBSapps/comn/util/jdk64,ListenAddress=T1VSTEBSAPP01.softtech.local,ListenPort=5557,LogAppend=false,NodeManagerHome=/u01/install/APPS/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1,SecureListener=false,StartScriptEnabled=false,StartScriptName=startWebLogic.sh,}
 Command: /u01/install/APPS/fs2/EBSapps/comn/util/jdk64/jre/bin/java -classpath /u01/install/APPS/fs2/EBSapps/comn/util/jdk64/jre/lib/rt.jar:/u01/install/APPS/fs2/EBSapps/comn/util/jdk64/jre/lib/i18n.jar:/u01/install/APPS/fs2/FMW_Home/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/install/APPS/fs2/EBSapps/comn/util/jdk64/lib/tools.jar:/u01/install/APPS/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic_sp.jar:/u01/install/APPS/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/u01/install/APPS/fs2/FMW_Home/modules/features/weblogic.server.modules_10.3.6.0.jar:/u01/install/APPS/fs2/FMW_Home/wlserver_10.3/server/lib/webservices.jar:/u01/install/APPS/fs2/FMW_Home/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u01/install/APPS/fs2/FMW_Home/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u01/install/APPS/fs2/FMW_Home/oracle_common/modules/oracle.jdbc_11.1.1/ojdbc6dms.jar:/u01/install/APPS/fs2/FMW_Home/oracle_common/webcenter/modules/oracle.portlet.server_11.1.1/oracle-portlet-api.jar:/u01/install/APPS/fs2/FMW_Home/oracle_common/modules/oracle.jrf_11.1.1/jrf.jar:/u01/install/APPS/fs2/FMW_Home/wlserver_10.3/common/derby/lib/derbyclient.jar:/u01/install/APPS/fs2/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/install/APPS/fs2/EBSapps/comn/java/classes/oracle/apps/fnd/jar/adall.jar weblogic.NodeManager -v
Successfully launched the Node Manager.
The Node Manager process is running independent of the WLST process.
Exiting WLST will not stop the Node Manager process. Please refer
to the Node Manager logs for more information.
The Node Manager logs will be under /u01/install/APPS/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1
Node Manager starting in the background

NodeManager is not fully up....Waiting for 10 sec....

NodeManager is not fully up....Waiting for 10 sec....

NodeManager is not fully up....Waiting for 10 sec....

NodeManager is not fully up....Waiting for 10 sec....

NodeManager is not fully up....Waiting for 10 sec....

NodeManager is not fully up....Waiting for 10 sec....

NodeManager is not fully up....Waiting for 10 sec....

NodeManager is not fully up....Waiting for 10 sec....

NodeManager is not fully up....Waiting for 10 sec....

NodeManager is not fully up....Waiting for 10 sec....

NodeManager is not fully up....Waiting for 10 sec....

NodeManager is started now
Connecting to Node Manager ...
Successfully Connected to Node Manager.

Connected to the Node Manager.

Starting server AdminServer ...
Successfully started server AdminServer ...




04/28/15-16:46:55 :: adadminsrvctl.sh: exiting with status 0

================================================================================
 
stty: standard input: Inappropriate ioctl for device
Enter the APPS user password:
stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
Enter the WebLogic AdminServer password:
stty: standard input: Inappropriate ioctl for device
The log file is /u01/install/APPS/fs2/inst/apps/IKYSDEV_T1VSTEBSAPP01/logs/appl/rgf/Tue_Apr_28_16_46_57_2015/adRegisterWLSListeners.log
--------------


After that I ran adop phase=prepare command however it failed because we have started adadminsrvctl and it occupied some ports therefore validations have failed.

-------------

-------------------ADX Database Utility Finished---------------


Validating port of all Managed Servers

Checking  Manager Server Port 1 :  Port Value = 7402
Manager Server Port 1 available:  Port Value = 7402

Checking  Manager Server Port 2 :  Port Value = 7602
Manager Server Port 2 available:  Port Value = 7602

Checking  Manager Server Port 3 :  Port Value = 7802
Manager Server Port 3 available:  Port Value = 7802

Checking  Manager Server Port 4 :  Port Value = 7202
Manager Server Port 4 available:  Port Value = 7202


-----------------------------
ERROR: The following required ports are in use:
-----------------------------
5557 : Node Manager Port
7002 : WLS Admin Server Port
Corrective Action: Free the listed ports and retry the adop operation.

Completed execution : ADOPValidations.java
-------------------


The complete failure message of the adop is below:

=========================== START OF VALIDATION REPORT ====================

Validation Results for Node: T1VSTEBSAPP01
---------------------------------------
[WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.

[ERROR]: Some of the ports specified for the patch file system are not available.

Some validations failed, please check /u01/install/APPS/fs_ne/EBSapps/log/adop/39/prepare_20150428_164523/IKYSDEV_T1VSTEBSAPP01/ADOPValidations_Tue_Apr_28_16_46_15_2015.log

=========================== END OF VALIDATION REPORT ======================
        [UNEXPECTED]Error occurred while executing "perl /u01/install/APPS/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl  -contextfile=/u01/install/APPS/fs1/inst/apps/IKYSDEV_T1VSTEBSAPP01/appl/admin/IKYSDEV_T1VSTEBSAPP01.xml -patchctxfile=/u01/install/APPS/fs2/inst/apps/IKYSDEV_T1VSTEBSAPP01/appl/admin/IKYSDEV_T1VSTEBSAPP01.xml -phase=prepare -logloc=/u01/install/APPS/fs_ne/EBSapps/log/adop/39/prepare_20150428_164523/IKYSDEV_T1VSTEBSAPP01 -promptmsg=hide"
        [UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on T1VSTEBSAPP01
        Log file: /u01/install/APPS/fs_ne/EBSapps/log/adop/39/adop_20150428_164523.log


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.


adop exiting with status = 1 (Fail)
---

[ErmanArslansOracleBlog]

What is written in the following file;

/u01/install/APPS/fs1/EBSapps/comn/adopclone_T1VSTEBSAPP01/FMW/obfuscatedPassword_04271637.log

[ErmanArslansOracleBlog]

Without using nopromptmsg, you supply the passwords manually, so the nodemanager and adminserver can be started..

With this in mind; I think that the password that supplied by the adop may be corrupted, or lets say there might be a problem with adop while it was supplying the password ..

Lets see the log file I have reqeusted in my previous update.

[ErmanArslansOracleBlog]

--------------------------------------------------------------------
Also, I see the following in "txkADOPPreparePhaseSynchronize.pl";
It says only if node_count is 1, then dont start Admin and node manager..
so, what is the following sql returns in your environment?


SELECT DISTINCT(host) FROM fnd_nodes WHERE host IS NOT NULL AND (support_cp='Y' OR support_forms='Y' OR support_web='Y' OR support_admin='Y');
------------------------------------------------------


       # Start Node Manager and Admin server only in multi-node
        # If node count is greather than 1, then it is a multi-node setup
        #------------------------------------------------------------------
        my $get_node_count = getNodeCount();

        if ($get_node_count < 2)
        {
            logMessage("\n");
            logMessage("====================================================\n");
            logMessage("Single Node set-up, no need to start Admin Server...\n");
            logMessage("====================================================\n");
            logMessage("\n");
            return;
        }
        else
        {
            execServiceControl({ ctrl_script => 's_nodemanagerctrl',
                                 ctrl_code   => 'start forcepatchfs',
                                 srv_status  => 's_adminserverstatus',
                                 srv_name    => 's_adminservername',
                                 file_system => 'PATCH'
                              });
        }
    }
}

[Berke]

The following
/u01/install/APPS/fs1/EBSapps/comn/adopclone_T1VSTEBSAPP01/FMW/obfuscatedPassword_04271637.log contents are below:

[applmgr@T1VSTEBSAPP01 FMW]$ more obfuscatedPassword_04281629.log
Object persisted successfully
[applmgr@T1VSTEBSAPP01 FMW]$

-----------------


The result of the query is :

SQL> SELECT DISTINCT(host) FROM fnd_nodes WHERE host IS NOT NULL AND (support_cp='Y' OR support_forms='Y' OR support_web='Y' OR support_admin='Y');

HOST
--------------------------------------------------------------------------------
T1VSTEBSAPP01

SQL>

---------------------

I am just thinking, can it be due to a corrupted weblogic configuration?
In the following note, it mentions about an nmconnect thing which I couldnt find as a command in the server
https://community.oracle.com/thread/2449386

[Berke]

Hello Erman,

We have finally solved our problem.

---

You are running adadminsrvctl.sh version 120.10.12020000.9

Stopping WLS Admin Server...
Refer /u01/install/APPS/fs2/inst/apps/IKYSDEV_T1VSTEBSAPP01/logs/appl/admin/log/adadminsrvctl.txt for details

adadminsrvctl.sh: exiting with status 0

adadminsrvctl.sh: check the logfile /u01/install/APPS/fs2/inst/apps/IKYSDEV_T1VSTEBSAPP01/logs/appl/admin/log/adadminsrvctl.txt for more information ...  

        [EVENT]     [END   2015/04/29 11:13:45] Check and Stop Patch FS Admin Server
        [EVENT]     [START 2015/04/29 11:13:47] Check and Stop Patch Node Manager

You are running adnodemgrctl.sh version 120.11.12020000.11


The Node Manager is already shutdown

adnodemgrctl.sh: exiting with status 2

adnodemgrctl.sh: check the logfile /u01/install/APPS/fs2/inst/apps/IKYSDEV_T1VSTEBSAPP01/logs/appl/admin/log/adnodemgrctl.txt for more information ...  

        [EVENT]     [END   2015/04/29 11:13:53] Check and Stop Patch Node Manager
    [END   2015/04/29 11:13:55] adzdoptl.pl run
    adop phase=prepare - Completed Successfully

    Log file: /u01/install/APPS/fs_ne/EBSapps/log/adop/39/adop_20150429_104154.log

adop exiting with status = 0 (Success)
-----
[applmgr@T1VSTEBSAPP01 /]$ adop -status

Enter the APPS password:




Current Patching Session ID: 39

Node Name       Node Type       Phase           Status          Started                        Finished                       Elapsed
--------------- --------------- --------------- --------------- ------------------------------ ------------------------------ ------------
T1VSTEBSAPP01   master          PREPARE         COMPLETED       27-APR-15 16:29:56 +03:00      29-APR-15 11:13:19 +03:00      42:43:23
                                APPLY           NOT STARTED
                                FINALIZE        NOT STARTED
                                CUTOVER         NOT STARTED
                                CLEANUP         NOT STARTED




File System Synchronization Used in this Patching Cycle: Full









For more information, run ADOP Status Report by using -detail option


Generating ADOP Status Report at location: /u01/install/APPS/fs_ne/EBSapps/log/status_20150429_111737/adzdshowstatus.out
Please wait...
Done...!

adop exiting with status = 0 (Success)
-------------




What we did was to update the nodemanager user and password details from the weblogic console.(1)

-- Log in to the WL Console.
-- in the Domain Tree, click at teh root level, which is the domain name.
-- Under the security tab, under advanced there is a node manager username / password.
-- Click on "Lock and edit", and change those to match what you are using through WLST.
-- Activate changes.

(Home >Summary of Environment >Summary of Machines >T1VSTEBSAPP01 >Summary of Services >Summary of Security Realms >myrealm >Users and Groups >EBS_domain_IKYSDEV)

After we have updated the weblogic user information, we started the adop phase again and we  were succesful.

You may also encounter the following error.
"shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory ebs"

It might be solved with the following note.

error retrieving current directory: getcwd: cannot access parent directories: No such file or directory (Doc ID 1475219.1)

Thank you for all the help.

References:
1-https://community.oracle.com/thread/1038799

[ErmanArslansOracleBlog]

:) good I was preparing the following update for you..

When the adadminsrvctl script is executed with nompromptmsg, then the passwords should be supplied to it..

if [ "$promptmsg" = "-nopromptmsg" ]; then
    read wlspass
    read appspass

The adop should supply these password..
It says: Running...../u01/install/APPS/fs2/inst/apps/IKYSDEV_T1VSTEBSAPP01/admin/scripts/adadminsrvctl.sh start -nopromptmsg forcepatchfs
But It does not show us the password supplied there.. Password may be wrong..

Anyways, It supplies the WLS admin password  as follows;

 { echo $wlspass; } | perl /u01/install/APPS/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl -contextfile=/u01/install/APPS/fs2/inst/apps/ORATEST_daroratestappsrv1/appl/admin/ORATEST_daroratestappsrv1.xml ebs-nmstart-adminsrv -wlsadminuser=weblogic -wlsadminport=7002 -domain=EBS_domain_ORATEST -domainhome="/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain_ORATEST" -nmhome="/u01/install/APPS/fs2/FMW_Home/wlserver_10.3/common/nodemanager/nmHome1" -nmport=5557 -nmtype=plain -machinelistenaddress="daroratestappsrv1.dardanel.com" -promptmsg=hide -serverstartargs="-XX:PermSize=128m -XX:MaxPermSize=384m -Xms1024m -Xmx1024m -Djava.security.policy=/u01/install/APPS/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.policy -Djava.security.egd=file:/dev/./urandom -Dweblogic.ProductionModeEnabled=true -da -Dplatform.home=/u01/install/APPS/fs2/FMW_Home/wlserver_10.3 -Dwls.home=/u01/install/APPS/fs2/FMW_Home/wlserver_10.3/server -Dweblogic.home=/u01/install/APPS/fs2/FMW_Home/wlserver_10.3/server -Dcommon.components.home=/u01/install/APPS/fs2/FMW_Home/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain_ORATEST -Djrockit.optfile=/u01/install/APPS/fs2/FMW_Home/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.server.config.dir=/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain_ORATEST/config/fmwconfig/servers/AdminServer -Doracle.domain.config.dir=/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain_ORATEST/config/fmwconfig -Digf.arisidbeans.carmlloc=/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain_ORATEST/config/fmwconfig/carml -Digf.arisidstack.home=/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain_ORATEST/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain_ORATEST/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain_ORATEST/servers/AdminServer/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/install/APPS/fs2/FMW_Home/oracle_common/modules/oracle.ossoiap_11.1.1,/u01/install/APPS/fs2/FMW_Home/oracle_common/modules/oracle.oamprovider_11.1.1 -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dweblogic.jdbc.remoteEnabled=false -Dportlet.oracle.home=/u01/install/APPS/fs2/FMW_Home/oracle_common -Dem.oracle.home=/u01/install/APPS/fs2/FMW_Home/oracle_common -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/install/APPS/fs2/FMW_Home/patch_wls1036/profiles/default/sysext_manifest_classpath" -logfile=$LOG

So adProvisionEBS.pl takes the password. It is wrapper to the EBSProvisioner.class.
It run EBSProvisioner.class ..
  my $cmd = "$javaPrg -classpath $ENV{'CLASSPATH'} oracle.apps.ad.tools.configuration.EBSProvisioner $params";

[Berke]

You were right about the password matter that's how we were directed to the nodemanager issue.

We should be very careful about the passwords because I dont know why,it might get corrupted. We shouldn't have had to enter the passwords again. However the things done during the fs_clone, finalize or cutover seems to break things.

[ErmanArslansOracleBlog]

Yes.
We can say that the problem was in nm_password.properties file ..
When you do your update from the weblogic console,  you actually fix this file.

It can be fsclone which caused this to happen. Maybe the file was not there at all..

Anyways, I m glad it got solved.

[hith]

I had same issue but not during adop prepare. I have enabled the online patch and i need to start the admin server to apply latest TXK patches.

while bring up the admin server i am having the issue like

Validated the passed arguments for the option ebs-nmstart-adminsrv
Checking if the Admin Server is already up.
The Admin Server is not already up.
Traceback (innermost last):
  File "<string>", line 1, in ?
  File "<iostream>", line 123, in nmConnect
  File "<iostream>", line 653, in raiseWLSTException
WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Access to domain 'EBS_domain_PROD' for user 'weblogic' denied
Use dumpStack() to view the full stacktrace

But Node Manager is available on 5557
[applprod@ebs16apps scripts]$ netstat -nape | grep 5557
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 ::ffff:192.168.1.41:5557    :::*                        LISTEN      502        10635957   1162/java
tcp        0      0 ::ffff:192.168.1.41:22824   ::ffff:192.168.1.41:5557    TIME_WAIT   0          0          -

[ErmanArslansOracleBlog]

Please open a new issue/thread (not write in to this issue/thread) into this forum.

2016-05-03 12:52 GMT+03:00 hith [via Erman Arslan's Oracle Forum] <[hidden email]>:

I had same issue but not during adop prepare. I have enabled the online patch and i need to start the admin server to apply latest TXK patches.

while bring up the admin server i am having the issue like

Validated the passed arguments for the option ebs-nmstart-adminsrv
Checking if the Admin Server is already up.
The Admin Server is not already up.
Traceback (innermost last):
  File "<string>", line 1, in ?
  File "<iostream>", line 123, in nmConnect
  File "<iostream>", line 653, in raiseWLSTException
WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Access to domain 'EBS_domain_PROD' for user 'weblogic' denied
Use dumpStack() to view the full stacktrace

But Node Manager is available on 5557
[applprod@ebs16apps scripts]$ netstat -nape | grep 5557
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 ::ffff:192.168.1.41:5557    :::*                        LISTEN      502        10635957   1162/java
tcp        0      0 ::ffff:192.168.1.41:22824   ::ffff:192.168.1.41:5557    TIME_WAIT   0          0          -

---

If you reply to this email, your message will be added to the discussion below:

http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/Adop-Prepare-Has-Failed-With-adclone-pl-Error-tp292p954.html

To start a new topic under EBS 12.2, email [hidden email]
To unsubscribe from Erman Arslan's Oracle Forum, click here.
NAML

--

[Roshan]

I am having same issue. I managed to start AdminServer in patch edition;

[appluat@erpuat scripts]$ ./adadminsrvctl.sh AdminServer start -nopromptmsg forcepatchfs

You are running adadminsrvctl.sh version 120.10.12020000.9

Do you think it is a password issue?

[Roshan]

If yes, did you put same password as AdminServer when changing Node Manager password in weblogic console?

[ErmanArslansOracleBlog]

Yes.

See what I wrote there;
 
We can say that the problem was in nm_password.properties file ..
When you do your update from the weblogic console,  you actually fix this file.

It can be fsclone which caused this to happen. Maybe the file was not there at all..

Anyways, I m glad it got solved.