Wikipedia

Search results

Saturday 24 August 2024

Shell Script for WebSphere Patching

#!/bin/bash # Configuration WAS_HOME="/opt/IBM/WebSphere/AppServer" # WebSphere home directory PROFILE_NAME="AppSrv01" # WebSphere profile name IMCL_PATH="/opt/IBM/InstallationManager/eclipse/tools/imcl" # Installation Manager CLI path REPOSITORY_PATH="/path/to/repository" # Path to the repository containing the patches LOG_DIR="$WAS_HOME/logs/patching" # Directory to store patch logs PATCHES="com.ibm.websphere.ND.v85_8.5.5.15" # Patch ID(s) # Create log directory if it doesn't exist mkdir -p "$LOG_DIR" # Stop the server echo "Stopping WebSphere server..." "$WAS_HOME/profiles/$PROFILE_NAME/bin/stopServer.sh" server1 if [ $? -ne 0 ]; then echo "Failed to stop WebSphere server. Exiting." exit 1 fi # Apply the patch echo "Applying WebSphere patch..." "$IMCL_PATH" install "$PATCHES" -repositories "$REPOSITORY_PATH" -installationDirectory "$WAS_HOME" -log "$LOG_DIR/patching_$(date +'%Y%m%d%H%M%S').log" -acceptLicense if [ $? -ne 0 ]; then echo "Failed to apply WebSphere patch. Exiting." exit 1 fi # Start the server echo "Starting WebSphere server..." "$WAS_HOME/profiles/$PROFILE_NAME/bin/startServer.sh" server1 if [ $? -ne 0 ]; then echo "Failed to start WebSphere server. Exiting." exit 1 fi echo "WebSphere patching completed successfully."

Sunday 26 May 2024

Was Patching

#!/bin/bash # Configuration WAS_HOME="/opt/IBM/WebSphere/AppServer" # WebSphere home directory PROFILE_NAME="AppSrv01" # WebSphere profile name IMCL_PATH="/opt/IBM/InstallationManager/eclipse/tools/imcl" # Installation Manager CLI path REPOSITORY_PATH="/path/to/repository" # Path to the repository containing the patches LOG_DIR="$WAS_HOME/logs/patching" # Directory to store patch logs PATCHES="com.ibm.websphere.ND.v85_8.5.5.15" # Patch ID(s) # Create log directory if it doesn't exist mkdir -p "$LOG_DIR" # Stop the server echo "Stopping WebSphere server..." "$WAS_HOME/profiles/$PROFILE_NAME/bin/stopServer.sh" server1 if [ $? -ne 0 ]; then echo "Failed to stop WebSphere server. Exiting." exit 1 fi # Apply the patch echo "Applying WebSphere patch..." "$IMCL_PATH" install "$PATCHES" -repositories "$REPOSITORY_PATH" -installationDirectory "$WAS_HOME" -log "$LOG_DIR/patching_$(date +'%Y%m%d%H%M%S').log" -acceptLicense if [ $? -ne 0 ]; then echo "Failed to apply WebSphere patch. Exiting." exit 1 fi # Start the server echo "Starting WebSphere server..." "$WAS_HOME/profiles/$PROFILE_NAME/bin/startServer.sh" server1 if [ $? -ne 0 ]; then echo "Failed to start WebSphere server. Exiting." exit 1 fi echo "WebSphere patching completed successfully."

Script for WebSphere Log Housekeeping

#!/bin/bash # Configuration LOG_DIR="/path/to/websphere/logs" # Directory where logs are stored ARCHIVE_DIR="/path/to/websphere/archive" # Directory to move compressed logs DAYS_TO_KEEP=30 # Number of days to keep logs before deletion LOG_EXTENSION="log" # Log file extension (e.g., log, out) # Create archive directory if it doesn't exist mkdir -p "$ARCHIVE_DIR" # Find and compress log files older than $DAYS_TO_KEEP days find "$LOG_DIR" -name "*.$LOG_EXTENSION" -type f -mtime +$DAYS_TO_KEEP -exec gzip {} \; -exec mv {}.gz "$ARCHIVE_DIR" \; # Delete compressed log files older than $DAYS_TO_KEEP days from the archive directory find "$ARCHIVE_DIR" -name "*.$LOG_EXTENSION.gz" -type f -mtime +$DAYS_TO_KEEP -exec rm -f {} \; # Log the housekeeping activity echo "$(date +'%Y-%m-%d %H:%M:%S') - WebSphere log housekeeping completed" >> "$LOG_DIR/housekeeping.log"

Friday 27 March 2020

Some Exceptions in Websphere Application Server

Hi Friends , today we will discuss about some errors and exceptions in WAS.

 Application profiling exceptions :

The following exceptions are thrown in response to various illegal actions related to application profiling. 
com.ibm.ws.exception.RuntimeWarning : This exception is thrown when the application is started, if the application is configured incorrectly. The startup is consequently terminated. Some examples of bad configurations include: v A task configured on two different application profiles. v A method configured with two different task run-as policies . 

com.ibm.websphere.appprofile.IllegalTaskNameException : This exception is raised if an application attempts to programmatically set a task when that task has not been configured as a task name reference.


Error: java.lang.NoClassDefFoundError:

Explaination: This exception is thrown when Java code cannot load the specified class.
Possible Cause: Invalid or non-existent class v Class path problem v Manifest problem
Recommndation: 
 Check to determine if the specified class exists in a Java Archive (JAR) file within your Enterprise  Archive (EAR) file. If it does, make sure the path for the class is correct. 

For example, if you get the exception: java.lang.NoClassDefFoundError: WebSphereSamples.HelloEJB.HelloHome verify that the HelloHome class exists in one of the JAR files in your EAR file. If it exists, verify that the path for the class is WebSphereSamples.HelloEJB

If both the class and path are correct, then it is a class path issue. Most likely, you do not have the failing class JAR file specified in the client JAR file manifest. To verify this situation, perform the following steps:

1. Open your EAR file with an assembly tool, and select the Application Client. 
2. Add the names of the other JAR files in the EAR file to the Classpath field. This exception is generally caused by a missing Enterprise Java Beans (EJB) module name from the Classpath field.

 If you have multiple JAR files to enter in the Classpath field, be sure to separate the JAR names with spaces.

com.ibm.websphere.ce.cm.StaleConnectionException: [IBM][CLI Driver] SQL1013N The database alias name or database name “NULL” could not be found:

com.ibm.websphere.ce.cm.StaleConnectionException: [IBM][CLI Driver] SQL1013N The database alias name or database name “NULL” could not be found. SQLSTATE=42705. This error occurs when a data source is defined but the databaseName attribute and the corresponding value are not added to the custom properties panel.

To add the databaseName property: 
1. Click Resources > Manage JDBC Providers in the administrative console. 
2. Select the JDBC_provider that supports the problem data source. 
3. Select Data Sources and then select the problem data source. 
4. Under Additional properties, click Custom Properties. 
5. Select the databaseName property, or add one if it does not exist, and enter the actual database       
    name as the value. 
6. Click Apply or OK, and then click Save from the action bar.
7. Access the data source again

Transaction recovery failure (for XA data sources) Problem :

When WebSphere Application Server attempts to recover Oracle database transactions, the transaction service issues the following exception:

WTRN0037W: The transaction service encountered an error on an xa_recover operation. The resource was com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl@1114a62. The error code was XAER_RMERR. The exception stack trace follows: javax.transaction.xa.XAException at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:726) at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:954) at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:137) at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:609) at com.ibm.ws.Transaction.JTA.PartnerLogTable.recover(PartnerLogTable.java:511) at com.ibm.ws.Transaction.JTA.RecoveryManager.resync(RecoveryManager.java:1784) at com.ibm.ws.Transaction.JTA.RecoveryManager.run(RecoveryManager.java:2241)


Cause:
Oracle requires services such as the WebSphere Application Server transaction service to have special permissions for performing transaction recoveries.

Solution :

As user SYS, run the following commands on your Oracle server:

grant select on pending_trans$ to public;
grant select on dba_2pc_pending to public; grant select on dba_pending_transactions to public;
grant execute on dbms_system to ;

 User is a user ID in the application server that is authorized to perform transaction recovery for the XA data source. If you have not authorized any user IDs to perform transaction recovery, the application server uses the login alias for the data source as the user ID.


Thursday 13 December 2018

Some important unix commands and resolved deployment issue.

CPU for top 10 process:

ps aux | sort -nrk 3,3 | head -n 10

Certificate Expiry date:

** gsk8capicmd_64 -cert -details -db key.kdb -stashed -label | sed -n -e 1,8p | cut -f1

** echo q | openssl s_client -connect IHSHOSTNAME:443 | openssl x509 -noout -enddate

grep "21:0[5-9]" logfile

**Determines top 5 proceses 

svmon -Pu -t 5 | grep -p Pid | grep '^.*[0-9]'

** Last Logged in user on system:

last | head -n 10 | awk NR==1'{print $1}'

*** Chk memory in GB and MB:

**svmon -G -O unit=GB
**svmon -G -O unit=MB

** svmon -C java -O filtertype=working,segment=category,filtercat=exclusive,unit=MB

** Chk the largest file size in dir:

du -sk * | sort -rn | head -5

====================================================================
Application was not deployed successfully:

Deployment of the 1st ear completed successfully however 2nd ear couldn’t be exploded in the container because of which you are not able to access the application. please find below snippet which shows cache ear is available in the container.

It appears that the deployment script would have run multiple times as can be seen from the dmgr logs where the ear is exploded first but failed in subsequent distributions and lead to synchronization failure.  FYI. Disk space and inode count was under threshold at container level.

Please re-trigger the deployment to fix the issue

@ Application was not deployed cause , /opt/IBM was 82% full only 2.1 GB memory was there free, and application size was 1 GB. So, due to this deployment went in redeployment status.

   So, we freed some space . And deployment was successful.



Thursday 13 September 2018

Java heap memory leak or sizing problem? .

Situation

Occasionally when reviewing a heapdump that was generated from an out of memory event, the largest consumer of heap memory may consume slightly more memory than the other objects. Because of this, WebSphere users and moderators may incorrectly accuse the largest heap consumer as a memory leak.
 
If a Java™ heap is sized too small, an out of memory error will occur regardless if a memory leak exists. The leak suspect shown in IBM HeapAnalyzer or Memory Analyzer (MAT) will show a java object that is expected to exhaust a relatively larger portion of the heap.
 
Resolution

The best course of action will be to increase the heap size and enable verboseGC (if not already enabled). Doing this will produce one of two results: it will alleviate the out of memory problem by letting your heap utilize more memory, or the leaking object will cause another out of memory event. If the test yields the first of the two outcomes, the administrator can then tune the heap appropriately and scale down if need be. If the latter happens, the relative heap memory consumed by the leaking object will be much greater than the other java objects, thus confirming the suspected leaking object as seen by the first heapdump. If the leaking object is an IBM based object then further investigation would be needed, and engaging WebSphere Application Server support using the Service Request (SR) tool is recommended.
 
Related Links:
 

List of firewall ports in WebSphere Application Server V7, V8, and V8.5

List of firewall ports that must be open for communication between the deployment manager, nodeagent, and Application Server
 
It's very important to know the ports that should be opened in the firewall for proper communication between deployment manager, nodeagent, and application servers. Follow the below steps to achieve this task.
 
Find the port numbers in the serverindex.xml file or from the ISC (Integrated Solution Console)
The serverindex.xml file can be found under the profile-root/config/cells/cellName/nodes/nodeName folder.
 
From Integrated Solution Console:
Application Server ports
Click on servers -> server Name -> Expand ports under communication
Nodeagent ports
Click on System administration -> node agents -> nodeagent -> Expand ports under Additional Properties
Deployment Manager ports
Click on System administration -> Deployment manager -> Expand ports under Additional Properties
 
Note: The example endpoints are derived from version 8.5 configuration, please ignore the endpoints/ports if you don't find it in your configuration.
 
DMGR Ports to be opened with security enabled and disabled
Port/Endpoint Name
Security Disabled
Security Enabled
Reason/Comment
CELL_DISCOVERY_ADDRESS
Yes
Yes
Discovery between nodeagent and DMgr will not work
BOOTSTRAP_ADDRESS
Yes
Yes
Naming service or RMI service between DMgr and node might not work
SOAP_CONNECTOR_ADDRESS
Yes
Yes
Synchronization will not work
ORB_LISTENER_ADDRESS
Yes
Yes
Port value can't be zero. Should have a static value. More info
WC_adminhost
Yes
Yes
File transfer application will not work
DCS_UNICAST_ADDRESS
Yes
Yes
HA Manager won't work properly (i.e., WLM, DRS, Transaction log recovery )
IPC_CONNECTOR_ADDRESS
Yes
Yes
Internal communication might fail
WC_adminhost_secure
No
Yes
File Transfer won't work
SAS_SSL_SERVERAUTH_LISTENER_ADDRESS
No
No
This port is used for communication with version 6.0.x servers federated in a 6.1 or later cell. Should open if you have V6.0 mixed node.
CSIV2_SSL_MUTUALAUTH_LISTENER_ADDRESS
No
Yes
Required when security enabled
CSIV2_SSL_SERVERAUTH_LISTENER_ADDRESS
No
Yes
Required when security enabled
DataPowerMgr_inbound_secure
Yes
Yes
Required only when you use DataPower
 
 
Nodeagent Ports to be opened with security enabled and disabled
Port Name/Endpoint Name
Security Disabled
Security Enabled
Reason/Comment
BOOTSTRAP_ADDRESS
Yes
Yes
Naming service or RMI service between dmgr and node might not work
ORB_LISTENER_ADDRESS
Yes
Yes
Port value can't be zero. Should have a static value. More info
DCS_UNICAST_ADDRESS
Yes
Yes
HA Manager won't work(WLM, DRS, Transaction log recovery etc)
NODE_DISCOVERY_ADDRESS
Yes
Yes
Discovery between nodeagent and dmgr will not work
NODE_IPV6_MULTICAST_DISCOVERY_ADDRESS
Yes (if NO to ipv4)
Yes (if NO to ipv4)
Multicast discovery for application servers (during startup) to discover nodeagent. The endpoint can be removed, if you prefer to use IPV4.
NODE_MULTICAST_DISCOVERY_ADDRESS (ipV4)
Yes (if NO to ipv6)
Yes (if NO to ipv6)
Multicast discovery for application servers (during startup) to discover nodeagent. The endpoint can be removed, if you prefer to use IPV6
SOAP_CONNECTOR_ADDRESS
Yes
Yes
Synchronization will not work
IPC_CONNECTOR_ADDRESS
Yes
Yes
Internal WebSphere communication might fail
SAS_SSL_SERVERAUTH_LISTENER_ADDRESS
No
No
This port is used for communication with version 6.0.x servers federated in a 6.1 or later cell. Should open if you have V6.0 mixed node.
CSIV2_SSL_MUTUALAUTH_LISTENER_ADDRESS
No
Yes
Required when security enabled
CSIV2_SSL_SERVERAUTH_LISTENER_ADDRESS
No
Yes
Required when security enabled
 
 
Application Server ports to be opened
Port Name/Endpoint Name
Security Disabled
Security Enabled
Reason/Comment
DCS_UNICAST_ADDRESS
Yes
Yes
HA Manager won't work(WLM, DRS, Transaction log recovery etc). All application server DCS ports should be opened.
 
 
Additional firewall considerations
You might choose to separate the WebSphere application servers from your database and LDAP servers with a firewall. If so, you might have to open the following ports. The following ports are default ports, please consult with your admin to find out the right port numbers:
  • DB2: 50000 and 50001
  • Oracle: 1521
  • SQL Server: 1433
  • LDAP: 389

How to Identify Class Loader Synchronization Performance Bottlenecks in AIX, Linux and Windows

Class loader synchronization bottlenecks are common problems brought to the attention of WebSphere Application Server L2 Support by our clients.  This is a common type of performance problem called a livelock where many threads are contending on a set of locks. The performance overhead is proportional to the lock contention, so the problem might not show up in a test environment with a low number of concurrent requests, or patterns of requests which aren't exercising the lock at the same time.  From the viewpoint of the application end user, this is seen as an application that is moving slowly.  From the viewpoint of the WebSphere Administrator viewing the WebSphere Application Server SystemOut.log, this is typically seen as a series of hung (WSVR0605W) thread messages which "complete" (WSVR0606W) on their own....eventually.

image
To diagnose, manually capture a javacore DURING the time while the thread is still hung, (or automatically capture javacore by configuring the hang detection policy - com.ibm.websphere.threadmonitor.dump.java) and search for the name of the reported hung (WSVR0605W) thread:
image
As you can see, the thread which has been reported "hung" is waiting in line with multiple other threads which are all waiting on the "owner" thread to complete.  Once the owner thread completes, it will release the lock on "com/ibm/ws/classloader/CompoundClassLoader" and the next thread in line will become the owner of the lock and will be allowed to "complete".  Because class loading is synchronized, only one thread may own the lock at a time.  This results in a class loader bottleneck:
image
To identify the offending code, the next step is to investigate the stack of the "owner" thread (the one holding the lock while the rest of the threads wait), to identify where the bottleneck originates.  Search further down in the javacore to locate the stack of the owner thread.  Generally, walk down from the top of the stack, skipping any java/* and com/ibm/ws/* method frames until you reach the application or third party library code that's driving the livelock :
image

In the case of the example above, the development team  using the "org/MYAPP/myclass" code will need to implement coding best practices to avoid class loader bottlenecks in this code:

Related Topics:

DeadLock Identification from a Single Javacore File in WebSphere on Linux .

Sample Javacore Extracts and Locking Explanation for a DeadLock Condition:

Deadlock detected !!!

Thread "WebContainer : 6" (0x0000000009306B00)
is waiting for:
sys_mon_t:0x00002B9F669C9D50 infl_mon_t: 0x00002B9F669C9DC8:
java/lang/Object@0x000000002D7D8810
which is owned by:
Thread "Agent Heartbeat" (0x0000000000808400)
which is waiting for:
sys_mon_t:0x00002B9F681EA980 infl_mon_t: 0x00002B9F681EA9F8:
com/ibm/ws/session/store/memory/MemorySession@0x000000011EF1EF18
which is owned by:
Thread "WebContainer : 29" (0x000000000A4E7500)
which is waiting for:
sys_mon_t:0x00002B9F6AA7A7C0 infl_mon_t: 0x00002B9F6AA7A838:
org/apache/log4j/spi/RootLogger@0x0000000025878DC0
which is owned by:
Thread "WebContainer : 6" (0x0000000009306B00)
 

The above entries from the LOCKS section of Javacore.txt shows a DeadLock condition happened between (3) threads ' WebContainer : 6  ', ' Agent Heartbeat ' and ' WebContainer : 29 '.
Note:  When reviewing Systremout.log from WebSphere the profile/logs directory, DeadLocks will most probably show up as WSRV0605W hung thread warning messages and can only be diagnosed as a "DeadLock" when reviewing the Javacores in either a text editor or using specialized tooling such as IBM Thread and Monitor Dump Analyzer for Java.
 

-->>  WebContainer : 6 thread shows an exception happened at spring framework at org/apache/log4j/Category.callAppenders(Category.java:203(Compiled Code))
which is being logged by org/apache/log4j by acquiring the ' root logger ' lock and then this thread is calling a mail service and then being introspected by Wily.  For introspection, Wily needs the ' StallSweeper ' lock which is being owned by the " Agent Heartbeat " thread.

Thread Name
WebContainer : 6
State
Deadlock/Blocked

Monitor
Owns Monitor Lock on org/apache/log4j/spi/RootLogger@0x0000000025878DC0
Waiting for Monitor Lock on java/lang/Object@0x000000002D7D8810

Java Stack
at com/wily/introscope/agent/blamestackfeature/BlameStackFeatureBlameStack.IBlameStack_addExtraParameter(BlameStackFeatureBlameStack.java:105(Compiled Code))
at com/wily/introscope/agent/blame/DuplicateHandlingBlameStack.IBlameStack_addExtraParameter(DuplicateHandlingBlameStack.java:60(Compiled Code))
at com/wily/introscope/agent/blame/CompoundBlameStack.IBlameStack_addExtraParameter(CompoundBlameStack.java:165(Compiled Code))
at com/wily/introscope/agent/blame/ComponentTracer.addExtraParameter(ComponentTracer.java:328(Compiled Code))
at com/wily/introscope/agent/blame/ComponentTracer.addExtraParameter(ComponentTracer.java:316(Compiled Code))
at com/wily/introscope/agent/trace/io/SocketBackendTracer.annotateBlameStack(SocketBackendTracer.java:153(Compiled Code))
at java/net/ManagedSocketInputStreamHighPerformance.annotateBlameStack(ManagedSocketInputStreamHighPerformance.java:328(Compiled Code))
at java/net/ManagedSocketInputStreamHighPerformance.read(ManagedSocketInputStreamHighPerformance.java:231(Compiled Code))
at com/sun/mail/util/TraceInputStream.read(TraceInputStream.java:106(Compiled Code))
at java/io/BufferedInputStream.fill(BufferedInputStream.java:229(Compiled Code))
at java/io/BufferedInputStream.read(BufferedInputStream.java:248(Compiled Code))
at com/sun/mail/util/LineInputStream.readLine(LineInputStream.java:84(Compiled Code))
at com/sun/mail/smtp/SMTPTransport.readServerResponse(SMTPTransport.java:1742(Compiled Code))
at com/sun/mail/smtp/SMTPTransport.openServer(SMTPTransport.java:1523(Compiled Code))
at com/sun/mail/smtp/SMTPTransport.protocolConnect(SMTPTransport.java:453(Compiled Code))
at javax/mail/Service.connect(Service.java:291(Compiled Code))
at javax/mail/Service.connect(Service.java:172(Compiled Code))
at javax/mail/Service.connect(Service.java:121(Compiled Code))
at javax/mail/Transport.send0(Transport.java:190(Compiled Code))
at javax/mail/Transport.send(Transport.java:120(Compiled Code))
at org/apache/log4j/net/SMTPAppender.sendBuffer(Bytecode PC:214(Compiled Code))
at org/apache/log4j/net/SMTPAppender.append(Bytecode PC:56)
at org/apache/log4j/AppenderSkeleton.doAppend(AppenderSkeleton.java:230(Compiled Code))
at org/apache/log4j/helpers/AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65(Compiled Code))
at org/apache/log4j/Category.callAppenders(Category.java:203(Compiled Code))
at org/apache/log4j/Category.forcedLog(Category.java:388(Compiled Code))
at org/apache/log4j/Category.error(Category.java:302)
at com/applicationX/pocs/ui/web/controllers/ExceptionHandler.resolveException(ExceptionHandler.java:61)
at org/springframework/web/servlet/DispatcherServlet.processHandlerException(DispatcherServlet.java:1120(Compiled Code))
at org/springframework/web/servlet/DispatcherServlet.doDispatch(DispatcherServlet.java:944(Compiled Code))
at org/springframework/web/servlet/DispatcherServlet.doService(DispatcherServlet.java:852(Compiled Code))
at org/springframework/web/servlet/FrameworkServlet.processRequest(FrameworkServlet.java:882(Compiled Code))
at org/springframework/web/servlet/FrameworkServlet.doGet(FrameworkServlet.java:778(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:575(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:668(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1227(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:776(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:458(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:97(Compiled Code))
at com/applicationX/pocs/ui/web/util/SessionExpiryFilter.doFilter(SessionExpiryFilter.java:60(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/applicationX/pocs/ui/web/util/CommonFilter.doFilter(CommonFilter.java:44(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/applicationX/pocs/ui/web/util/GZipServletFilter.doFilter(GZipServletFilter.java:42(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:928(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebApp.handleRequest(WebApp.java:3761(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebGroup.handleRequest(WebGroup.java:304(Compiled Code))
at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:976(Compiled Code))
at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662(Compiled Code))
at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:200(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:526(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:312(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.ready(HttpInboundLink.java:283(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1048(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLConnectionLink.readyInboundPostHandshake(SSLConnectionLink.java:716(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLConnectionLink$MyHandshakeCompletedCallback.complete(SSLConnectionLink.java:412(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLUtils.handleHandshake(SSLUtils.java:1066(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLHandshakeIOCallback.complete(SSLHandshakeIOCallback.java:87(Compiled Code))
at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175(Compiled Code))
at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code))
at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161(Compiled Code))
at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:138(Compiled Code))
at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:204(Compiled Code))
at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:816(Compiled Code))
at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905(Compiled Code))
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1862(Compiled Code))

Waiting Threads:  36
Thread-76
WebContainer : 0
WebContainer : 1
WebContainer : 10
WebContainer : 11
WebContainer : 15
WebContainer : 17
WebContainer : 19
WebContainer : 2
WebContainer : 20
WebContainer : 21
WebContainer : 22
WebContainer : 23
WebContainer : 24
WebContainer : 25
WebContainer : 26
WebContainer : 29
WebContainer : 3
WebContainer : 31
WebContainer : 32
WebContainer : 36
WebContainer : 37
WebContainer : 39
WebContainer : 4
WebContainer : 40
WebContainer : 41
WebContainer : 45
WebContainer : 46
WebContainer : 47
WebContainer : 48
WebContainer : 49
WebContainer : 5
WebContainer : 7
WebContainer : 8
WebContainer : 9
pool-13-thread-1

Blocked by:  1
Agent Heartbeat



-->> Agent Heartbeat thread is on progress with introspection by owning the ' StallSweeper ' lock but needs the ' MemorySession ' lock to complete the process, which is being owned by the " WebContainer : 29 " thread.

Thread Name
Agent Heartbeat
State
Deadlock/Blocked

Monitor
Owns Monitor Lock on java/lang/Object@0x000000002D7D8810
Waiting for Monitor Lock on com/ibm/ws/session/store/memory/MemorySession@0x000000011EF1EF18

Java Stack
at com/ibm/ws/session/store/memory/MemorySession.updateLastAccessTime(MemorySession.java:638(Compiled Code))
at com/ibm/ws/session/store/memory/MemoryStore.getSession(MemoryStore.java:194(Compiled Code))
at com/ibm/ws/session/store/memory/MemoryStore.getSession(MemoryStore.java:712(Compiled Code))
at com/ibm/ws/session/SessionManager.getSessionFromStore(SessionManager.java:497(Compiled Code))
at com/ibm/ws/session/SessionManager.getSession(SessionManager.java:476(Compiled Code))
at com/ibm/ws/session/SessionManager.getSession(SessionManager.java:462(Compiled Code))
at com/ibm/ws/session/SessionManager.getSession(SessionManager.java:693(Compiled Code))
at com/ibm/ws/session/SessionContext.getIHttpSession(SessionContext.java:466(Compiled Code))
at com/ibm/ws/session/SessionContext.getIHttpSession(SessionContext.java:426(Compiled Code))
at com/ibm/ws/webcontainer/srt/SRTRequestContext.getSession(SRTRequestContext.java:104(Compiled Code))
at com/ibm/ws/webcontainer/srt/SRTServletRequest.getSession(SRTServletRequest.java:2152(Compiled Code))
at sun/reflect/GeneratedMethodAccessor406.invoke(Bytecode PC:65(Compiled Code))
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:611(Compiled Code))
at com/wily/introscope/agent/trace/servlet/ObjectWrapper.invokeMethodOnObject(ObjectWrapper.java:102(Compiled Code))
at com/wily/introscope/agent/trace/servlet/RequestWrapper.getSession(RequestWrapper.java:84(Compiled Code))
at com/wily/introscope/agent/trace/servlet/ServletParameterLoader.addSessionID(ServletParameterLoader.java:687(Compiled Code))
at com/wily/introscope/agent/trace/servlet/ServletParameterLoader.addParameters(ServletParameterLoader.java:625(Compiled Code))
at com/wily/introscope/agent/trace/servlet/ServletParameterLoader.doWithWrappers(ServletParameterLoader.java:384(Compiled Code))
at com/wily/introscope/agent/trace/servlet/ServletInvocationDataHelper$SafeGetServletWrappers.execute(ServletInvocationDataHelper.java:89(Compiled Code))
at com/wily/introscope/agent/trace/servlet/ServletInvocationDataHelper$SafeExecuteOnInvocationDataWithThrottling.executeSafe(ServletInvocationDataHelper.java:36(Compiled Code))
at com/wily/introscope/agent/trace/HttpServletTracer.IInvocationDataParameterCallback_addParameters(HttpServletTracer.java:551(Compiled Code))
at com/wily/introscope/agent/trace/InvocationData.IComponentParameterCallback_addParameters(InvocationData.java:580(Compiled Code))
at com/wily/introscope/agent/trace/BlamePointTracer$1.IComponentParameterCallback_addParameters(BlamePointTracer.java:196(Compiled Code))
at com/wily/introscope/agent/blamestackfeature/BlameStackFeatureStackEntry.createComponentEventData(BlameStackFeatureStackEntry.java:67(Compiled Code))
at com/wily/introscope/agent/blamestackfeature/BlameStackFeatureBlameStack.makeAndSendSnapshot(BlameStackFeatureBlameStack.java:281(Compiled Code))
at com/wily/introscope/agent/stalls/StallFeatureStackEntry.checkIfStalled(StallFeatureStackEntry.java:76(Compiled Code))
at com/wily/introscope/agent/stalls/StallFeature.checkIfStalled(StallFeature.java:179(Compiled Code))
at com/wily/introscope/agent/stalls/StallSweeper.sweepStacks(StallSweeper.java:125(Compiled Code))
at com/wily/introscope/agent/stalls/StallSweeper.ITimestampedRunnable_execute(StallSweeper.java:96(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat$BehaviorNode.execute(IntervalHeartbeat.java:944(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime(IntervalHeartbeat.java:489(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat.access$2(IntervalHeartbeat.java:443(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:665(Compiled Code))
at java/lang/Thread.run(Thread.java:773)

Waiting Threads:  1
WebConatiner :  6

Blocked by:  1
WebConatiner :  29


-->>  WebContainer : 29 thread is in progress with session invalidation after owning the ' MemorySession ' lock at com/ibm/ws/session/store/memory/MemorySession.invalidate(MemorySession.java:232(Compiled Code)).
But for logging the invalidation process, this thread needs ' root logger ' lock, which is owned by the " WebContainer : 6 " thread.
 
Thread Name
WebContainer : 29
State
Deadlock/Blocked

Monitor
Owns Monitor Lock on com/ibm/ws/session/store/memory/MemorySession@0x000000011EF1EF18
Waiting for Monitor Lock on org/apache/log4j/spi/RootLogger@0x0000000025878DC0

Java Stack
at org/apache/log4j/Category.callAppenders(Category.java:201(Compiled Code))
at org/apache/log4j/Category.forcedLog(Category.java:388(Compiled Code))
at org/apache/log4j/Category.info(Category.java:663(Compiled Code))
at com/applicationX/pocs/ui/web/util/SessionListener.sessionDestroyed(SessionListener.java:22(Compiled Code))
at com/ibm/ws/session/http/HttpSessionObserver.sessionDestroyed(HttpSessionObserver.java:179(Compiled Code))
at com/ibm/ws/session/SessionEventDispatcher.sessionDestroyed(SessionEventDispatcher.java:160(Compiled Code))
at com/ibm/ws/session/StoreCallback.sessionInvalidated(StoreCallback.java:126(Compiled Code))
at com/ibm/ws/session/store/memory/MemorySession.invalidate(MemorySession.java:232(Compiled Code))
at com/ibm/ws/session/http/HttpSessionImpl.invalidate(HttpSessionImpl.java:303(Compiled Code))
at com/ibm/ws/session/SessionData.invalidate(SessionData.java:247(Compiled Code))
at com/ibm/ws/session/HttpSessionFacade.invalidate(HttpSessionFacade.java:200(Compiled Code))
at com/applicationX/pocs/ui/web/controllers/common/ErrorPageController.display500Error(ErrorPageController.java:29)
at sun/reflect/GeneratedMethodAccessor3149.invoke(Bytecode PC:48)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:611(Compiled Code))
at org/springframework/web/method/support/InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219(Compiled Code))
at org/springframework/web/method/support/InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132(Compiled Code))
at org/springframework/web/servlet/mvc/method/annotation/ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:100(Compiled Code))
at org/springframework/web/servlet/mvc/method/annotation/RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:604(Compiled Code))
at org/springframework/web/servlet/mvc/method/annotation/RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:565(Compiled Code))
at org/springframework/web/servlet/mvc/method/AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80(Compiled Code))
at org/springframework/web/servlet/DispatcherServlet.doDispatch(DispatcherServlet.java:923(Compiled Code))
at org/springframework/web/servlet/DispatcherServlet.doService(DispatcherServlet.java:852(Compiled Code))
at org/springframework/web/servlet/FrameworkServlet.processRequest(FrameworkServlet.java:882(Compiled Code))
at org/springframework/web/servlet/FrameworkServlet.doGet(FrameworkServlet.java:778(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:575(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:668(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1227(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:776(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:458(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:97(Compiled Code))
at com/applicationX/pocs/ui/web/util/GZipServletFilter.doFilter(GZipServletFilter.java:50(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:928(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebAppRequestDispatcher.dispatch(WebAppRequestDispatcher.java:1385(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:194(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebApp.sendError(WebApp.java:3263(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebApp.handleException(WebApp.java:3791)
at com/ibm/ws/webcontainer/webapp/WebApp.handleRequest(WebApp.java:3772(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebGroup.handleRequest(WebGroup.java:304(Compiled Code))
at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:976(Compiled Code))
at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662(Compiled Code))
at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:200(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:526(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:312(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.ready(HttpInboundLink.java:283(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1048(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLConnectionLink.readyInboundPostHandshake(SSLConnectionLink.java:716(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLConnectionLink$MyHandshakeCompletedCallback.complete(SSLConnectionLink.java:412(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLUtils.handleHandshake(SSLUtils.java:1066(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLHandshakeIOCallback.complete(SSLHandshakeIOCallback.java:87(Compiled Code))
at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175(Compiled Code))
at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code))
at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161(Compiled Code))
at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:138(Compiled Code))
at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:204(Compiled Code))
at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:816(Compiled Code))
at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905(Compiled Code))
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1862(Compiled Code))

Waiting Threads:  1
Agent Heartbeat

Blocked Threads:  1
WebContainer :  6


Summary: 
These locks we have just seen above are raised from Wily Instroscope Agent, WebSphere Session Handler and applicationX, so needs further investigation by involving appropriate product support component teams and application developers to review their corresponding code.

References: