Tuesday, February 2, 2010

Apache Check List

Note: Apache versions 1.3.9 and 1.3.12s have been de-supported

You can check your version by doing:

$IAS_ORACLE_HOME/Apache/Apache/bin/httpd –version

 1. Validate TNS Connectivity
a. Make sure you can TNSPING and sqlplus the database alias used from the APPL_TOP (i.e. after running APPSORA.env). Do:
Tnsping [Sid]
Sqlplus apps/appspassword
Sqlplus apps/appspassword @ [Sid]

Also, validate that APPLSYSPUB/PUB can connect

b. Change to the $IAS_ORACLE_HOME, run the [sid]_[host].env file.
This will set the $ORACLE_HOME to be iAS.
Now check that you can TNSPING and sqlplus the database alias.
2. Determine the DBC file in use and make sure it is valid:
a. (11.5.9+) Run the profile option query provided in Appendix A when prompted for profile_option_name enter: APPS_DATABASE_ID.
If this returns a value, then this is the DBC file name that Apps is expecting to use.
Otherwise do:
select host_name||'_'||instance_name from v$instance;

b. Validate the OS location & permissions of the dbc file. Do:
 ls –al $FND_SECURE/*.dbc (11.5.10+).
If $FND_SECURE is not set, then the dbc filebr> This should return the value from step 1ad.
The permissions on the file should be 644 and it should be owned my the “applmgr” user.

c. Verify the contents of the dbc file with the AdminAppServer utility. Do:
java oracle.apps.fnd.security.AdminAppServer appsun/appspw STATUS DBC=[path to dbc]/[dbc_name].dbc
This should return STATUS: VALID and the current status of AUTHENTICATION: [SECURE/ON/OFF/null] among other values.

d. Validate the autoconfig context file (11.5.8+) uses the correct dbc file. Your context file is located in $APPL_TOP/admin and is typically named [SID]_[host].xml. Review the value for s_dbc_file_name, it should match the dbc just tested in 1c.
If any of this errors you should run autoconfig and/or $COMMON_TOP/admin/install/adgendbc.sh appsus appspw and retest (this will require iAS restart)


3. Is the web server running and able to render static html?
a. Are you able to access the page:
http:// :/aplogon.html
b. If not, then do:
ps –fu [applmgr] | grep http
(or ps –fu [applmgr] | grep http  | wc –l )
This should return at least 5 http processes running … otherwise you webserver may not be started…Make sure you are using the Oracle provided start script in:
$COMMON_TOP/admin/scripts/[SID]_[host]
If these tests fail you need to review your iAS installation and/or log a TAR.


4. (11.5.2-11.5.9) Check that the CGI environment and mod_plsql is functioning
a. Run the following URL:
http://:/pls//FND_WEB.PING
If you are unable to run the procedure below because of an 'internal error' or a 'cannot find host' or a similar error, then it could indicate that there is a problem with the PLSQL Configuration. Failures here may indicate problems with mod_plsql…Follow Note 116715.1 - How to Enable Logging for the PL/SQL Gateway. Redo the test and log a tar with the logs attached.
Or this may be a result of a RDBMS issue (For example - invalid packages, rdbms crashed)…See RDBMS section below.
b. Validate the APPS password used by the plsql gateway.
The password may be stored as clear text as a parameter (password= APPSpw) in
$IAS_ORACLE_HOME/Apache/modplsql/cfg/wdbsvr.app
If the password is encrypted in the file, update the parameter:
administrators = system
to be
administrators = all 
Then navigate to:
http://:/pls//admin_/
Select the Gateway Database Access Descriptor Settings link
Select the Edit icon next to the Database Access Descriptor Name which matches your SID
Make sure the Oracle User Name = APPS
Enter the Apps user password in the Oracle Password field.
Make sure the Oracle Connect String is the SID for the instance.
If you made any changes here you will have to review your autoconfig context file to make sure the changes are permanent.
(11.5.10) After validating the above items, you may still get 'Forbidden - You don't have permission to access /pls//fnd_web.ping' on this server.” This may be by design for enhanced security. Please skip to the next test.
5. Ensure that JDK is installed correctly & is a certified version.
a. Get the value of wrapper.bin in:
$iAS_ORACLE_HOME/Apache/Jserv/etc
This will either point directly to the java executable in use or to $iAS_ORACLE_HOME/Apache/Apache/bin/java.sh. If this points to the java.sh go to step 5b…otherwise skip to step 5c.

b.Open the java.sh File under $iAS_ORACLE_HOME/Apache/Apache/bin. In this file and you will find path to java executable in variable "JSERVJAVA"

c. Use the absolute path to the java executable and do:
/absolute/path/to/java –version
e. Check that the version returned is certified by using the certify website on Metalink. Assuming this returns a supported version of JDK. Use the following notes to validate your JDK installation (i.e. that all required patches, autoconfig templates, etc have been completed):
Note 304099.1 => if using J2SE Version 5.0
Or
Note 246105.1 => if using J2SE 1.4
OR
Note 130091.1 => if using JDK 1.3
6. Use the following programs to verify the installation and check that the servlets are functioning.
a. http(s)://:/servlets/IsItWorking
(11.5.10) This may fail with:
Forbidden You don't have permission to access /servlets/IsItWorking on this server.
This is due to enhanced security delivered with the autoconfig templates.
Please try test 4b instead.
b.http(s)://:/servlets/Hello
If this one fails, this indicates an issue with your Jserv set-up. You then need to follow Note 230688.1 to drill down into this problem.

7. Validate that jsp work.
a. http(s)://:/OA_HTML/jsp/fnd/aoljtest.jsp
If this one fails to render, this indicates an issue with your Jserv set-up. You then need to follow Note 230688.1 to drill down into this problem.Otherwise you enter the values requested, and follow the link at the bottom of the first page to run through this set of diagnostic tests. Report all tests that fail in a TAR.
Note: The initial page of this test may show some "missing" files.
Depending on your configuration the following missing files are acceptable:
apps.zip (is normal to be missing since it has been exploded on $JAVA_TOP (ref : Note 220188.1 ))
iAS/mp/jlib/opreopi-rt.jar (Used for Oracle Personalization. Can be ignored if you are not using MP.)
iAS/mp/jlib/dmtutil.jar (Used for Data Mining)
iAS/dm/jlib/odmapi.jar (Used for Data Mining - If you are using Data Mining, and these are listed as missing, please see Note 281739.1 )
iAS/portal30/jpdk/lib/partnerApp.jar (If you are not using Portal, this can be ignored.)

8. Check the "session.topleveldomain" setting in the
$IAS_ORACLE_HOME/Apache/Jserv/etc/zone.properties
This should match the domain you are using as defined in the SESSION_COOKIE_DOMAIN column in ICX_PARAMETERS table.
From sqlplus do:
select SESSION_COOKIE_DOMAIN from ICX_PARAMETERS;
Notes: It is acceptable to have SESSION_COOKIE_DOMAIN set to null
You MUST have a valid domain that is composed of 2 or more components (see Bug 2510732). I.e: .oracle is an INVALID domain, but .oracle.com IS a valid domain.

Environment Checks
9. Check the profile options

APPS_FRAMEWORK_AGENT (Application Framework Agent), APPS_JSP_AGENT (Applications JSP Agent), APPS_SERVLET_AGENT (Apps Servlet Agent), APPS_WEB_AGENT (Applications Web Agent)…And for ICX_FORMS_LAUNCHER (ICX: Forms Launcher), POR_SERVLET_VIRTUAL_PATH (POR: Servlet Virtual Path), GUEST_USER_PWD (Guest User Password)

Application Framework Agent should be of the format:
http://myHost.myDomain.com:8000

Applications JSP Agen 
http://myHost.myDomain.com:8000

Applications Web Agent should be of the format:
http://myHost.myDomain.com:8000/pls/SID

Apps Servlet Agent should be of the format:
http://myHost.myDomain.com:8000/oa_servlets

POR: Servlet Virtual Path should be of the format:
oa_servlets

ICX: Forms Launcher should be of the foomain.com:8000/dev60er Password should match the GUEST_USER_PWD value in the DBC file (see step 2)

For a basic single node install these should be set at the site level and should all be pointing at the same server. For a load balanced or a DMZ setup, this may vary. See Note 217368.1 and Note 287176.1 for more details on these advanced configurations.

Internet Explorer can give 'Your session is no longer valid' if the domain name is not specified in these profile options or in the login URL. You MUST have a valid domain that is composed of 2 or more components (see Bug 2510732). I.e. apps.oracle is an INVALID host.domain combination, but apps.oracle.com IS a valid host.domain.


select p.profile_option_name SHORT_NAME, n.user_profile_option_name NAME,
decode(v.level_id,

10001, 'Site',

10002, 'Application',

10003, 'Responsibility',

10004, 'User',

10005, 'Server',

10007, 'SERVRESP',
'UnDef') LEVEL_SET,
decode(to_char(v.level_id),

'10001', '',

'10002', app.application_short_name,

'10003', rsp.responsibility_key,

'10005', svr.node_name,

'10006', org.name,

'10004', usr.user_name,

'10007', 'Serv/resp',
'UnDef') "CONTEXT",
v.profile_option_value VALUE
from fnd_profile_options p,
fnd_profile_option_values v,
fnd_profile_options_tl n,
fnd_user usr,
fnd_application app,
fnd_responsibility rsp,
fnd_nodes svr,
hr_operating_units org
where p.profile_option_id = v.profile_option_id (+)
and p.profile_option_name = n.profile_option_name
and upper(n.user_profile_option_name) like upper('%&profile_name%')
and  usr.user_id (+) = v.level_value
and  rsp.application_id (+) = v.level_value_application_id
and  rsp.responsibility_id (+) = v.level_value
and  app.application_id (+) = v.level_value
and  svr.node_id (+) = v.level_value
and  org.organization_id (+) = v.level_value
order by short_name, level_set;


10. Is the system in Maintenance Mode?
Run the following sql to verify:
Select fnd_profile.value ('APPS_MAINTENANCE_MODE') from dual;
If this returns Y, then run adadmin to disable maintenance mode (see Note 291901.1)
11. Check browser settings:
a. Ensure your browser has cookies enabled.
In Netscape this can be checked from Edit -> Preferences -> Advanced.
In Internet Explorer, this can be checked from Tools -> Internet Options -> Security 
Then select the correct zone for your Web Server and click on 'Custom Level'
Scroll down to the 'Cookies' section. Oracle Applications uses 'per-session' cookies.

It is worth verifying the same problem occurs in both IE and Netscape.
Try setting the cookies setting to "prompt" to ensure they are being set correctly.

b. With Internet Explorer, you must add your Web Server to the list of 'trusted sites'. You must enter the fully qualified hostname (hostname, plus domain name) of the Release 11i HTTP server node or nodes in the "Trusted sites" security zone and you must leave the security level of that zone at its default setting of Low. Security levels are set on the Security tab of the Internet Options window.
12. Are there any uncommitted changes in your Autoconfig context file? Run:
$AD_TOP/bin/adchkcfg.sh
This will create output in:
$APPL_TOP/admin/[SID]_[host]/out/[timestamp]
Review the report for any pending changes. Run autoconfig (see Note 165195.1) as required.
RDBMS Checks
1. Check the GUEST user information.
a. Run following SQL:
select user_name, start_date, end_date
from  fnd_user
where  user_name = 'GUEST';
This should return one row, end_date should be NULL or in advance of today's date, and start_date should be before today's date.
b. Validate the GUEST username/password combinations from the DBC file.
Using the GUEST_USER_PWD value in the DBC file (see step 2) run the following sql:
select fnd_web_sec.validate_login('GUEST',' ') from dual ;
This should return Yes
If this returns N, then do:
select fnd_message.get from dual;
This should give a reason why the validation failed, or an error message. If this fails with a database error, confirm the problem is not specific to the GUEST user. Redo the SQL command  with a different userid (like sysadmin)
select fnd_web_sec.validate_login('SYSADMIN','') from dual;
Again, this command should return 'Y' if it is working OK. If not, you should reload the jar files to the database using adadmin.
2. Run the following script to ensure there are no invalid objects:
select owner, object_name, object_type
from  all_objects
where  status != 'VALID'
order by owner, object_type, object_name;
Ideally, this should return no rows, but some invalid objects may be acceptable, depending on what they are.
3. Review your Alert.log.
Does the RDBMS alert log show any errors? What are they? Report them in a TAR.
4. Review any database trace files.
cd to the directory specified by user_dump_dest in the init.ora file.&nbttempt at login may create a .trc file
5. Validate the FND_NODES table:
Run the following sql:
Select NODE_NAME, NODE_ID, SERVER_ID, SERVER_ADDRESS from FND_NODES;

Each SERVER_ID and NODE_ID should be unique.
The Node Name for all servers involved for the instance should appear here. We typically expect this to be e hostname or virtual hostname (alias), not the IP address. If the servers do not appear, use OAM to register them.
The SERVER_ADDRESS is optional, but if present should reflect the IP of the host. If necessary, update the IP using the system administrator responsibility.


For 11.5.2 – 11.5.9 this should return 56 rows.
For 11.5.10 this should return 131 rows.
(11.5.10+) Note: in 11.5.10 the required plsql patckages can be quickly enabled using the script: $FND_TOP/patch/sql/txkDisableModPLSQL.sql Y

6. Validate the FND_ENABLED_PLSQL table Run the script in Appendix B. For 11.5.2 – 11.5.9 this should return 56 rows. For 11.5.10 this should return 131 rows. (11.5.10+) Note: in 11.5.10 the required plsql packages can be quickly enabled using the script: $FND_TOP/patch/sql/txkDisableModPLSQL.sql Y :
7. Validate the 'ICX_SESSIONS_S' synonym
This should be shown as owned by ICX, if not, drop the synonym and run adadmin to regenerate grants and synonyms.
8. Ensure ICX module is showing as an "Installed" module.
Run "adutconf.sql" to check this, particularly important to check if migrating to 11i from previous release of Applications.
9. Validate the ICX_PARAMETERS table.
Run the following SQL:

select count(*) from icx_parameters;
This should only return one row.

10. Check for Bug 3275654
Run the following SQL:

SELECT c.function_id, nvl(b.prompt,c.user_function_name) prompt,c.TYPE,a.menu_id from
fnd_responsibility_vl a,
fnd_menu_entries_vl b,
fnd_form_functions_vl c
where a.responsibility_id=21584
and a.menu_id=b.menu_id
and b.function_id=c.function_id
and c.type in ('WWW','WWK','SERVLET','JSP','FORM','INTEROPJSP');

This should only return one row. If not, apply Patch 3275654

11. Oracle 8i ONLY do the following sql:
select fnd_crypto.SmallRandomNumber from dual;
If this errors with: ORA-28235: algorithm not available
Then make sure you have the file $ORACLE_HOME/lib/naeet.o (7112 bytes) in place.
If not, you need to get the file from a known good source (like another instance or from the CD). Again use the size of the file to validate. Next relink the Oracle Executables:
cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk install
$ORACLE_HOME/bin/genclntsh.sh



OA Framework Checks
These are only applicable to OA Framework 5.7 and above. (11.5.9+)

1. Disable all personalizations on the homepage
Run the following sql:
set serverout on;
exec jdr_utils.listcustomizations(‘/oracle/apps/fnd/framework/navigate/webui/HomePG’);
If this returns any rows then you have personalizations.
Disable your personalizations by logging into forms directly and setting the profile option: Disable Self Service Personal = Yes. Then retest the login. If the homepage works, you need to remove your “bad” personalization. See Note 304670.1

2. You can turn on OA Framework diagnostics logging in several ways Please refer to Note 139863.1, section 4 Profile Options Reference for more information, if required. Set the following profile options at USER level for one Applications user. It is important this is setup for one user only. You can launch Core Forms directly to gain access to the profile options, as you may not be able to login if you are reading this note!
Name  Value
---------------------------------------------
FND: Debug Log Enabled  Yes
FND: Debug Log Filename  /tmp/OAF_Debug.txt
FND: Debug Log Module  %
FND: Debug Log Level  STATEMENT
Login to Applications using the Apps Username you specified above. Any error or problem you experience will still occur, but the file /tmp/OAF_Debug.txt should be created with some diagnostics information. Please upload this file to Oracle Support for further analysis. After you have created this diagnostics file, you should disable logging by setting the USER level profile option listed below:
Name  Value
---------------------------------------------
FND: Debug Log Enabled  No


Detailed checks
Note: It is recommended that you have a current backup your system and have no users on the system while performing these checks/tests

1. Enable detailed logging in IAS

Setting up detailed logging on the web server:

1.Shut the HTTP Server (Apache/iAS) down. - You can grep for the 'httpd' process to verify it is down
2. Rename (or delete) the following files so we get a fresh copy of them:

/Apache/Jserv/logs/jserv.log

/Apache/Jserv/logs/forms.log

/Apache/Jserv/logs/mod_jserv.log

/Apache/Jserv/logs/jvm  (the whole directory if it exists)

/Apache/Apache/logs/access_log*

/Apache/Apache/logs/error_log

/Apache/Apache/logs/error_log_pls (if it exists)

Now we will turn
on debugging in the log files:

3. Modify the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.conf file. Search for the parameter:

ApJServLogLevel notice
Change the 'notice' to 'info'

4. Modify the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties file. Search for the following section:

log=false
Change this to be log=true
and
Change

log.channel=false
To be: log.channel=true

5. Modify the $IAS_ORACLE_HOME/Apache/Jserv/etc/forms.properties file. Search for the following section:

log=false
Change this to be log=true
and  Change:

log.channel=false
To be  log.channel=true

6. Modify the $IAS_ORACLE_HOME/Apache/Apache/conf/httpd.conf file. Search for:

LogLevel
Set the LogLevel to 'info' from its current value.

7. Modify the $IAS_ORACLE_HOME/Apache/Apache/conf/httpd_pls.conf file (if it exists). Search for the following parameter

LogLevel
Set the LogLevel to 'info' from its current value.

8. Modify the:
$IAS_ORACLE_HOME/Apache/modplsql/cfg/wdbsrv.app
add
the line:
debugModules=Info
immediately after the line:
custom_auth  = CUSTOM

9. Start the HTTP Server.

10. Run a test to recreate the error


2. Temporarily enable iAS to use symbolic links.
Replace all occurrences of
'Option -FollowSymLinks'
 for
'Option +FollowSymLinks'
in
$IAS_ORACLE_HOME/Apache/Apache/conf/httpd.conf
and
$IAS_ORACLE_HOME/Apache/Apache/conf/httpd_pls.conf
and bounce apache. If this is the issue, you should then go back and implement the changes via autoconfig. The value from this is controllable from your Apps Context, see s_options_symlinks.

3. Clear the iAS cache.
To clear the jsp & modplsql caches either renames or deletes the sub-directories below following directories and restart apache. This will clear out the compiled JSP classes and cached modplsql modules causing them to be automatically recompiled next time they are accessed.
rm -Rf $OA_HTML/_pages/*
rm -Rf $COMMON_TOP/_pages/*
rm -Rf $IAS_ORACLE_HOME/Apache/modplsql/cache/*
Note: Depending on your configuration and patch level all directories may not exist.


4. Check the sqlnet.ora setup in the database Oracle Home. Edit the
$ORACLE_HOME/network/admin//sqlnet.ora
Is tcp.validnode_checking enabled (i.e. = yes)? If so, make sure the parameter tcp.invited_nodes contains an entry for all the nodes involved in your configuration. This parameter, in conjunction with tcp.validnode_checking determines which clients can connect to the database.
Or you can temporarily disable node checking by removing the tcp.validnode_checking and retest. This is enabled for security reasons.


5. Modify AppsLocalLogin.jsp to trap exceptions.
If you get java error while using the login page like NoClassDefFound or NullPointerException, we need to trap the context of the message.
Backup and edit your:
$OA_HTML/AppsLocalLogin.jsp.
On the line that reads
} catch(Exception e) {}
Change to:
} catch(Exception e) {
} catch(Throwable t) {
System.err.println("OSS: Caught throwable in AppsLocalLogin.jsp : " +
t.toString()); t.printStackTrace();
}
Then bounce Apache and reload the page and more detail should show up in the OACoreGroup.0.stderr file.
< status of the Server Security, which was verified in step 2c, is anything other then OFF, try toggling the current setting to see if the nature of the problem changes. Use the following command to temporarily disable the Security Server



After bouncing the apps services and retesting it is important to re-enable the Security.
Use the command:
java oracle.apps.fnd.security.AdminAppServer appsun/appspw AUTHENTICATION ON DBC=


6. Check file permissions are not causing any problems.
Check file permissions are appropriate. In particular, the liboci806jdbc.so (.sl on HP, .DLL on NT) shared library should have read, write and execute permissions. It may be advisable to temporarily change all permissions in $JAVA_TOP to read, write and execute to see if it resolves the problem. On UNIX, you can use the command 'chmod -R 777 $JAVA_TOP', assuming $JAVA_TOP has been set correctly in your environment.


7. Regenerate JAVA_TOP:
Ensure you have a full backup of your system! To run this process, run adadmin and select Maintain Applications Files then Copy Files to Destinations.


8. Generate database trace and SQL*Net traces.
To set-up SQL*Net Trace on the Web Server set
TRACE_LEVEL_CLIENT = 16
in the
$TNS_ADMIN/SQLNET.ORA
file. This should be in the iAS file system and not in the 8.0.6. Oracle Home. By default the SQL*Net trace file will be called 'sqlnet.trc'.

Important:
a.From "$IAS_TOP/Apache/Apache/logs" check the error_log and the access_log
b.From "$IAS_TOP/Apache/JServ/logs" check the mod_jserv.log and the jserv.log 

Check the error_log, jserv.log or mod_jserv.log files for errors

11i Basic Apache/mod_jserv troubleshooting with Hello.class Note: 230688.1

Troubleshooting Login Problems in Oracle Applications 11i (11.5.2 – 11.5.10+) Note: 342332.1

No comments:

Post a Comment