WebSphere Application Server
Wikipedia
Search results
Saturday 24 August 2024
Shell Script for WebSphere Patching
#!/bin/bash
# Configuration
WAS_HOME="/opt/IBM/WebSphere/AppServer" # WebSphere home directory
PROFILE_NAME="AppSrv01" # WebSphere profile name
IMCL_PATH="/opt/IBM/InstallationManager/eclipse/tools/imcl" # Installation Manager CLI path
REPOSITORY_PATH="/path/to/repository" # Path to the repository containing the patches
LOG_DIR="$WAS_HOME/logs/patching" # Directory to store patch logs
PATCHES="com.ibm.websphere.ND.v85_8.5.5.15" # Patch ID(s)
# Create log directory if it doesn't exist
mkdir -p "$LOG_DIR"
# Stop the server
echo "Stopping WebSphere server..."
"$WAS_HOME/profiles/$PROFILE_NAME/bin/stopServer.sh" server1
if [ $? -ne 0 ]; then
echo "Failed to stop WebSphere server. Exiting."
exit 1
fi
# Apply the patch
echo "Applying WebSphere patch..."
"$IMCL_PATH" install "$PATCHES" -repositories "$REPOSITORY_PATH" -installationDirectory "$WAS_HOME" -log "$LOG_DIR/patching_$(date +'%Y%m%d%H%M%S').log" -acceptLicense
if [ $? -ne 0 ]; then
echo "Failed to apply WebSphere patch. Exiting."
exit 1
fi
# Start the server
echo "Starting WebSphere server..."
"$WAS_HOME/profiles/$PROFILE_NAME/bin/startServer.sh" server1
if [ $? -ne 0 ]; then
echo "Failed to start WebSphere server. Exiting."
exit 1
fi
echo "WebSphere patching completed successfully."
Sunday 26 May 2024
Was Patching
#!/bin/bash
# Configuration
WAS_HOME="/opt/IBM/WebSphere/AppServer" # WebSphere home directory
PROFILE_NAME="AppSrv01" # WebSphere profile name
IMCL_PATH="/opt/IBM/InstallationManager/eclipse/tools/imcl" # Installation Manager CLI path
REPOSITORY_PATH="/path/to/repository" # Path to the repository containing the patches
LOG_DIR="$WAS_HOME/logs/patching" # Directory to store patch logs
PATCHES="com.ibm.websphere.ND.v85_8.5.5.15" # Patch ID(s)
# Create log directory if it doesn't exist
mkdir -p "$LOG_DIR"
# Stop the server
echo "Stopping WebSphere server..."
"$WAS_HOME/profiles/$PROFILE_NAME/bin/stopServer.sh" server1
if [ $? -ne 0 ]; then
echo "Failed to stop WebSphere server. Exiting."
exit 1
fi
# Apply the patch
echo "Applying WebSphere patch..."
"$IMCL_PATH" install "$PATCHES" -repositories "$REPOSITORY_PATH" -installationDirectory "$WAS_HOME" -log "$LOG_DIR/patching_$(date +'%Y%m%d%H%M%S').log" -acceptLicense
if [ $? -ne 0 ]; then
echo "Failed to apply WebSphere patch. Exiting."
exit 1
fi
# Start the server
echo "Starting WebSphere server..."
"$WAS_HOME/profiles/$PROFILE_NAME/bin/startServer.sh" server1
if [ $? -ne 0 ]; then
echo "Failed to start WebSphere server. Exiting."
exit 1
fi
echo "WebSphere patching completed successfully."
Script for WebSphere Log Housekeeping
#!/bin/bash
# Configuration
LOG_DIR="/path/to/websphere/logs" # Directory where logs are stored
ARCHIVE_DIR="/path/to/websphere/archive" # Directory to move compressed logs
DAYS_TO_KEEP=30 # Number of days to keep logs before deletion
LOG_EXTENSION="log" # Log file extension (e.g., log, out)
# Create archive directory if it doesn't exist
mkdir -p "$ARCHIVE_DIR"
# Find and compress log files older than $DAYS_TO_KEEP days
find "$LOG_DIR" -name "*.$LOG_EXTENSION" -type f -mtime +$DAYS_TO_KEEP -exec gzip {} \; -exec mv {}.gz "$ARCHIVE_DIR" \;
# Delete compressed log files older than $DAYS_TO_KEEP days from the archive directory
find "$ARCHIVE_DIR" -name "*.$LOG_EXTENSION.gz" -type f -mtime +$DAYS_TO_KEEP -exec rm -f {} \;
# Log the housekeeping activity
echo "$(date +'%Y-%m-%d %H:%M:%S') - WebSphere log housekeeping completed" >> "$LOG_DIR/housekeeping.log"
Friday 27 March 2020
Some Exceptions in Websphere Application Server
Hi Friends , today we will discuss about some errors and exceptions in WAS.
Application profiling exceptions :
The following exceptions are thrown in response to various illegal actions related to application profiling.
com.ibm.ws.exception.RuntimeWarning : This exception is thrown when the application is started, if the application is configured incorrectly. The startup is consequently terminated. Some examples of bad configurations include: v A task configured on two different application profiles. v A method configured with two different task run-as policies .
com.ibm.websphere.appprofile.IllegalTaskNameException : This exception is raised if an application attempts to programmatically set a task when that task has not been configured as a task name reference.
Error: java.lang.NoClassDefFoundError:
Explaination: This exception is thrown when Java code cannot load the specified class.
Possible Cause: Invalid or non-existent class v Class path problem v Manifest problem
Recommndation:
Check to determine if the specified class exists in a Java Archive (JAR) file within your Enterprise Archive (EAR) file. If it does, make sure the path for the class is correct.
For example, if you get the exception: java.lang.NoClassDefFoundError: WebSphereSamples.HelloEJB.HelloHome verify that the HelloHome class exists in one of the JAR files in your EAR file. If it exists, verify that the path for the class is WebSphereSamples.HelloEJB
If both the class and path are correct, then it is a class path issue. Most likely, you do not have the failing class JAR file specified in the client JAR file manifest. To verify this situation, perform the following steps:
1. Open your EAR file with an assembly tool, and select the Application Client.
2. Add the names of the other JAR files in the EAR file to the Classpath field. This exception is generally caused by a missing Enterprise Java Beans (EJB) module name from the Classpath field.
If you have multiple JAR files to enter in the Classpath field, be sure to separate the JAR names with spaces.
com.ibm.websphere.ce.cm.StaleConnectionException: [IBM][CLI Driver] SQL1013N The database alias name or database name “NULL” could not be found:
com.ibm.websphere.ce.cm.StaleConnectionException: [IBM][CLI Driver] SQL1013N The database alias name or database name “NULL” could not be found. SQLSTATE=42705. This error occurs when a data source is defined but the databaseName attribute and the corresponding value are not added to the custom properties panel.
To add the databaseName property:
1. Click Resources > Manage JDBC Providers in the administrative console.
2. Select the JDBC_provider that supports the problem data source.
3. Select Data Sources and then select the problem data source.
4. Under Additional properties, click Custom Properties.
5. Select the databaseName property, or add one if it does not exist, and enter the actual database
name as the value.
6. Click Apply or OK, and then click Save from the action bar.
7. Access the data source again
Transaction recovery failure (for XA data sources) Problem :
When WebSphere Application Server attempts to recover Oracle database transactions, the transaction service issues the following exception:
WTRN0037W: The transaction service encountered an error on an xa_recover operation. The resource was com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl@1114a62. The error code was XAER_RMERR. The exception stack trace follows: javax.transaction.xa.XAException at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:726) at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:954) at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:137) at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:609) at com.ibm.ws.Transaction.JTA.PartnerLogTable.recover(PartnerLogTable.java:511) at com.ibm.ws.Transaction.JTA.RecoveryManager.resync(RecoveryManager.java:1784) at com.ibm.ws.Transaction.JTA.RecoveryManager.run(RecoveryManager.java:2241)
Cause:
Oracle requires services such as the WebSphere Application Server transaction service to have special permissions for performing transaction recoveries.
Solution :
As user SYS, run the following commands on your Oracle server:
grant select on pending_trans$ to public;
grant select on dba_2pc_pending to public; grant select on dba_pending_transactions to public;
grant execute on dbms_system to;
User is a user ID in the application server that is authorized to perform transaction recovery for the XA
data source. If you have not authorized any user IDs to perform transaction recovery, the application
server uses the login alias for the data source as the user ID.
Application profiling exceptions :
The following exceptions are thrown in response to various illegal actions related to application profiling.
com.ibm.ws.exception.RuntimeWarning : This exception is thrown when the application is started, if the application is configured incorrectly. The startup is consequently terminated. Some examples of bad configurations include: v A task configured on two different application profiles. v A method configured with two different task run-as policies .
com.ibm.websphere.appprofile.IllegalTaskNameException : This exception is raised if an application attempts to programmatically set a task when that task has not been configured as a task name reference.
Error: java.lang.NoClassDefFoundError:
Explaination: This exception is thrown when Java code cannot load the specified class.
Possible Cause: Invalid or non-existent class v Class path problem v Manifest problem
Recommndation:
Check to determine if the specified class exists in a Java Archive (JAR) file within your Enterprise Archive (EAR) file. If it does, make sure the path for the class is correct.
For example, if you get the exception: java.lang.NoClassDefFoundError: WebSphereSamples.HelloEJB.HelloHome verify that the HelloHome class exists in one of the JAR files in your EAR file. If it exists, verify that the path for the class is WebSphereSamples.HelloEJB
If both the class and path are correct, then it is a class path issue. Most likely, you do not have the failing class JAR file specified in the client JAR file manifest. To verify this situation, perform the following steps:
1. Open your EAR file with an assembly tool, and select the Application Client.
2. Add the names of the other JAR files in the EAR file to the Classpath field. This exception is generally caused by a missing Enterprise Java Beans (EJB) module name from the Classpath field.
If you have multiple JAR files to enter in the Classpath field, be sure to separate the JAR names with spaces.
com.ibm.websphere.ce.cm.StaleConnectionException: [IBM][CLI Driver] SQL1013N The database alias name or database name “NULL” could not be found:
com.ibm.websphere.ce.cm.StaleConnectionException: [IBM][CLI Driver] SQL1013N The database alias name or database name “NULL” could not be found. SQLSTATE=42705. This error occurs when a data source is defined but the databaseName attribute and the corresponding value are not added to the custom properties panel.
To add the databaseName property:
1. Click Resources > Manage JDBC Providers in the administrative console.
2. Select the JDBC_provider that supports the problem data source.
3. Select Data Sources and then select the problem data source.
4. Under Additional properties, click Custom Properties.
5. Select the databaseName property, or add one if it does not exist, and enter the actual database
name as the value.
6. Click Apply or OK, and then click Save from the action bar.
7. Access the data source again
Transaction recovery failure (for XA data sources) Problem :
When WebSphere Application Server attempts to recover Oracle database transactions, the transaction service issues the following exception:
WTRN0037W: The transaction service encountered an error on an xa_recover operation. The resource was com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl@1114a62. The error code was XAER_RMERR. The exception stack trace follows: javax.transaction.xa.XAException at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:726) at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:954) at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:137) at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:609) at com.ibm.ws.Transaction.JTA.PartnerLogTable.recover(PartnerLogTable.java:511) at com.ibm.ws.Transaction.JTA.RecoveryManager.resync(RecoveryManager.java:1784) at com.ibm.ws.Transaction.JTA.RecoveryManager.run(RecoveryManager.java:2241)
Cause:
Oracle requires services such as the WebSphere Application Server transaction service to have special permissions for performing transaction recoveries.
Solution :
As user SYS, run the following commands on your Oracle server:
grant select on pending_trans$ to public;
grant select on dba_2pc_pending to public; grant select on dba_pending_transactions to public;
grant execute on dbms_system to
Thursday 13 December 2018
Some important unix commands and resolved deployment issue.
CPU for top 10 process:
ps aux | sort -nrk 3,3 | head -n 10
Certificate Expiry date:
** gsk8capicmd_64 -cert -details -db key.kdb -stashed -label | sed -n -e 1,8p | cut -f1
** echo q | openssl s_client -connect IHSHOSTNAME:443 | openssl x509 -noout -enddate
grep "21:0[5-9]" logfile
**Determines top 5 proceses
svmon -Pu -t 5 | grep -p Pid | grep '^.*[0-9]'
** Last Logged in user on system:
last | head -n 10 | awk NR==1'{print $1}'
*** Chk memory in GB and MB:
**svmon -G -O unit=GB
**svmon -G -O unit=MB
** svmon -C java -O filtertype=working,segment=category,filtercat=exclusive,unit=MB
** Chk the largest file size in dir:
du -sk * | sort -rn | head -5
====================================================================
Application was not deployed successfully:
Deployment of the 1st ear completed successfully however 2nd ear couldn’t be exploded in the container because of which you are not able to access the application. please find below snippet which shows cache ear is available in the container.
It appears that the deployment script would have run multiple times as can be seen from the dmgr logs where the ear is exploded first but failed in subsequent distributions and lead to synchronization failure. FYI. Disk space and inode count was under threshold at container level.
Please re-trigger the deployment to fix the issue
@ Application was not deployed cause , /opt/IBM was 82% full only 2.1 GB memory was there free, and application size was 1 GB. So, due to this deployment went in redeployment status.
So, we freed some space . And deployment was successful.
Thursday 13 September 2018
Java heap memory leak or sizing problem? .
Situation
Occasionally when reviewing a heapdump that was generated from an out of memory event, the largest consumer of heap memory may consume slightly more memory than the other objects. Because of this, WebSphere users and moderators may incorrectly accuse the largest heap consumer as a memory leak.
Occasionally when reviewing a heapdump that was generated from an out of memory event, the largest consumer of heap memory may consume slightly more memory than the other objects. Because of this, WebSphere users and moderators may incorrectly accuse the largest heap consumer as a memory leak.
If a Java™ heap is sized too small, an out of memory error will occur
regardless if a memory leak exists. The leak suspect shown in IBM
HeapAnalyzer or Memory Analyzer (MAT) will show a java object that is
expected to exhaust a relatively larger portion of the heap.
Resolution
The best course of action will be to increase the heap size and enable verboseGC (if not already enabled). Doing this will produce one of two results: it will alleviate the out of memory problem by letting your heap utilize more memory, or the leaking object will cause another out of memory event. If the test yields the first of the two outcomes, the administrator can then tune the heap appropriately and scale down if need be. If the latter happens, the relative heap memory consumed by the leaking object will be much greater than the other java objects, thus confirming the suspected leaking object as seen by the first heapdump. If the leaking object is an IBM based object then further investigation would be needed, and engaging WebSphere Application Server support using the Service Request (SR) tool is recommended.
The best course of action will be to increase the heap size and enable verboseGC (if not already enabled). Doing this will produce one of two results: it will alleviate the out of memory problem by letting your heap utilize more memory, or the leaking object will cause another out of memory event. If the test yields the first of the two outcomes, the administrator can then tune the heap appropriately and scale down if need be. If the latter happens, the relative heap memory consumed by the leaking object will be much greater than the other java objects, thus confirming the suspected leaking object as seen by the first heapdump. If the leaking object is an IBM based object then further investigation would be needed, and engaging WebSphere Application Server support using the Service Request (SR) tool is recommended.
Related Links:
List of firewall ports in WebSphere Application Server V7, V8, and V8.5
List of firewall ports that must be open for communication between the deployment manager, nodeagent, and Application Server
It's very important to know the ports that should be opened in the
firewall for proper communication between deployment manager, nodeagent,
and application servers. Follow the below steps to achieve this task.
Find the port numbers in the serverindex.xml file or from the ISC (Integrated Solution Console)
The serverindex.xml file can be found under the prof ile- root /con fig/ cell s/ce llNa me/n odes /nod eNam e folder.
From Integrated Solution Console:
Application Server ports
Click on servers -> server Name -> Expand ports under communication
Nodeagent ports
Click on System administration -> node agents -> nodeagent -> Expand ports under Additional Properties
Deployment Manager ports
Click on System administration -> Deployment manager -> Expand ports under Additional Properties
Note: The example endpoints are derived from version 8.5 configuration,
please ignore the endpoints/ports if you don't find it in your
configuration.
DMGR Ports to be opened with security enabled and disabled
Port/Endpoint Name
|
Security Disabled
|
Security Enabled
|
Reason/Comment
|
CELL
|
Yes
|
Yes
|
Discovery between nodeagent and DMgr will not work
|
BOOTSTRAP_ADDRESS
|
Yes
|
Yes
|
Naming service or RMI service between DMgr and node might not work
|
SOAP
|
Yes
|
Yes
|
Synchronization will not work
|
ORB_
|
Yes
|
Yes
|
Port value can't be zero. Should have a static value. More info
|
WC_adminhost
|
Yes
|
Yes
|
File transfer application will not work
|
DCS_UNICAST_ADDRESS
|
Yes
|
Yes
|
HA Manager won't work properly (i.e., WLM, DRS, Transaction log recovery )
|
IPC_
|
Yes
|
Yes
|
Internal communication might fail
|
WC_adminhost_secure
|
No
|
Yes
|
File Transfer won't work
|
SAS_
|
No
|
No
|
This
port is used for communication with version 6.0.x servers federated in a
6.1 or later cell. Should open if you have V6.0 mixed node.
|
CSIV
|
No
|
Yes
|
Required when security enabled
|
CSIV
|
No
|
Yes
|
Required when security enabled
|
Data
|
Yes
|
Yes
|
Required only when you use DataPower
|
Nodeagent Ports to be opened with security enabled and disabled
Port Name/Endpoint Name
|
Security Disabled
|
Security Enabled
|
Reason/Comment
|
BOOTSTRAP_ADDRESS
|
Yes
|
Yes
|
Naming service or RMI service between dmgr and node might not work
|
ORB_
|
Yes
|
Yes
|
Port value can't be zero. Should have a static value. More info
|
DCS_UNICAST_ADDRESS
|
Yes
|
Yes
|
HA Manager won't work(WLM, DRS, Transaction log recovery etc)
|
NODE
|
Yes
|
Yes
|
Discovery between nodeagent and dmgr will not work
|
NODE
|
Yes (if NO to ipv4)
|
Yes (if NO to ipv4)
|
Multicast
discovery for application servers (during startup) to discover
nodeagent. The endpoint can be removed, if you prefer to use IPV4.
|
NODE
|
Yes (if NO to ipv6)
|
Yes (if NO to ipv6)
|
Multicast
discovery for application servers (during startup) to discover
nodeagent. The endpoint can be removed, if you prefer to use IPV6
|
SOAP
|
Yes
|
Yes
|
Synchronization will not work
|
IPC_
|
Yes
|
Yes
|
Internal WebSphere communication might fail
|
SAS_
|
No
|
No
|
This
port is used for communication with version 6.0.x servers federated in a
6.1 or later cell. Should open if you have V6.0 mixed node.
|
CSIV
|
No
|
Yes
|
Required when security enabled
|
CSIV
|
No
|
Yes
|
Required when security enabled
|
Application Server ports to be opened
Port Name/Endpoint Name
|
Security Disabled
|
Security Enabled
|
Reason/Comment
|
DCS_UNICAST_ADDRESS
|
Yes
|
Yes
|
HA Manager won't work(WLM, DRS, Transaction log recovery etc). All application server DCS ports should be opened.
|
Additional firewall considerations
You might choose to separate the WebSphere application servers from
your database and LDAP servers with a firewall. If so, you might have to
open the following ports. The following ports are default ports, please
consult with your admin to find out the right port numbers:
- DB2: 50000 and 50001
- Oracle: 1521
- SQL Server: 1433
- LDAP: 389
How to Identify Class Loader Synchronization Performance Bottlenecks in AIX, Linux and Windows
Class loader synchronization bottlenecks are common
problems brought to the attention of WebSphere Application Server L2
Support by our clients. This is a common type of performance problem
called a livelock
where many threads are contending on a set of locks. The performance
overhead is proportional to the lock contention, so the problem might
not show up in a test environment with a low number of concurrent
requests, or patterns of requests which aren't exercising the lock at
the same time. From the viewpoint of the application end user, this is
seen as an application that is moving slowly. From the viewpoint of the
WebSphere Administrator viewing the WebSphere Application Server SystemOut.log, this is typically seen as a series of hung (WSVR0605W) thread messages which "complete" (WSVR0606W) on their own....eventually.
To diagnose, manually capture a javacore DURING the time while the thread is still hung, (o r au toma tica lly capt ure java core by conf igur ing the hang det ecti on p olic y - com. ibm. webs pher e.th read moni tor. dump .jav a) and search for the name of the reported hung (WSVR0605W) thread:
As you can see, the thread which has been reported "hung"
is waiting in line with multiple other threads which are all waiting on
the "owner" thread to complete. Once the owner thread completes, it
will release the lock on "com /ibm /ws/ clas sloa der/ Comp ound Clas sLoa der" and
the next thread in line will become the owner of the lock and will be
allowed to "complete". Because class loading is synchronized, only one
thread may own the lock at a time. This results in a class loader
bottleneck:
To identify the offending code, the next step is to
investigate the stack of the "owner" thread (the one holding the lock
while the rest of the threads wait), to identify where the bottleneck
originates. Search further down in the javacore to locate the stack of
the owner thread. Generally, walk down from the top of the stack,
skipping any java/* and com/ibm/ws/* method frames until you reach the
application or third party library code that's driving the livelock :
In the case of the example above, the development team using the "org/MYAPP/myclass" code will need to implement coding best practices to avoid class loader bottlenecks in this code:
Related Topics:
DeadLock Identification from a Single Javacore File in WebSphere on Linux .
Sample Javacore Extracts and Locking Explanation for a DeadLock Condition:
Deadlock detected !!!
Thread "WebContainer : 6" (0x0
is waiting for:
sys_
java
which is owned by:
Thread "Agent Heartbeat" (0x0
which is waiting for:
sys_
com/
which is owned by:
Thread "WebContainer : 29" (0x0
which is waiting for:
sys_
org/
which is owned by:
Thread "WebContainer : 6" (0x0
The above entries from the LOCKS section of Javacore.txt shows a DeadLock condition happened between (3) threads ' WebContainer : 6 ', ' Agent Heartbeat ' and ' WebContainer : 29 '.
Note: When reviewing Systremout.log from WebSphere the profile/logs directory, DeadLocks will most probably show up as WSRV0605W hung thread warning messages
and can only be diagnosed as a "DeadLock" when reviewing the Javacores
in either a text editor or using specialized tooling such as IBM Thread
and Monitor Dump Analyzer for Java.
-->> WebContainer : 6 thread shows an exception happened at spring framework at org/ apac he/l og4j /Cat egor y.ca llAp pend ers( Cate gory .jav a:20 3(Co mpil ed Code))
which is being logged by org/apache/log4j by acquiring the ' root logger ' lock and then this thread is calling a mail service and then being introspected by Wily. For introspection, Wily needs the ' StallSweeper ' lock which is being owned by the " Agent Heartbeat " thread.
Thread Name
WebContainer : 6
State
Deadlock/Blocked
Monitor
Owns Monitor Lock on org/ apac he/l og4j /spi /Roo tLog ger@ 0x00 0000 0025 878D C0
Waiting for Monitor Lock on java /lan g/Ob ject @0x0 0000 0002 D7D8 81 0
Java Stack
at com/ wily /int rosc ope/ agen t/bl ames tack feat ure/ Blam eSta ckFe atur eBla meSt ack. IBla meSt ack_ addE xtra Para mete r(Bl ameS tack Feat ureB lame Stac k.ja va:1 05(C ompi led Code))
at com/ wily /int rosc ope/ agen t/bl ame/ Dupl icat eHan dlin gBla meSt ack. IBla meSt ack_ addE xtra Para mete r(Du plic ateH andl ingB lame Stac k.ja va:6 0(Co mpil ed Code))
at com/ wily /int rosc ope/ agen t/bl ame/ Comp ound Blam eSta ck.I Blam eSta ck_a ddEx traP aram eter (Com poun dBla meSt ack. java :165 (Com pile d Code))
at com/ wily /int rosc ope/ agen t/bl ame/ Comp onen tTra cer. addE xtra Para mete r(Co mpon entT race r.ja va:3 28(C ompi led Code))
at com/ wily /int rosc ope/ agen t/bl ame/ Comp onen tTra cer. addE xtra Para mete r(Co mpon entT race r.ja va:3 16(C ompi led Code))
at com/ wily /int rosc ope/ agen t/tr ace/ io/S ocke tBac kend Trac er.a nnot ateB lame Stac k(So cket Back endT race r.ja va:1 53(C ompi led Code))
at java /net /Man aged Sock etIn putS trea mHig hPer form ance .ann otat eBla meSt ack( Mana gedS ocke tInp utSt ream High Perf orma nce. java :328 (Com pile d Code))
at java /net /Man aged Sock etIn putS trea mHig hPer form ance .rea d(Ma nage dSoc ketI nput Stre amHi ghPe rfor manc e.ja va:2 31(C ompi led Code))
at com/ sun/ mail /uti l/Tr aceI nput Stre am.r ead( Trac eInp utSt ream .jav a:10 6(Co mpil ed Code))
at java /io/ Buff ered Inpu tStr eam. fill (Buf fere dInp utSt ream .jav a:22 9(Co mpil ed Code))
at java /io/ Buff ered Inpu tStr eam. read (Buf fere dInp utSt ream .jav a:24 8(Co mpil ed Code))
at com/ sun/ mail /uti l/Li neIn putS trea m.re adLi ne(L ineI nput Stre am.j ava: 84(C ompi led Code))
at com/ sun/ mail /smt p/SM TPTr ansp ort. read Serv erRe spon se(S MTPT rans port .jav a:17 42(C ompi led Code))
at com/ sun/ mail /smt p/SM TPTr ansp ort. open Serv er(S MTPT rans port .jav a:15 23(C ompi led Code))
at com/ sun/ mail /smt p/SM TPTr ansp ort. prot ocol Conn ect( SMTP Tran spor t.ja va:4 53(C ompi led Code))
at java x/ma il/S ervi ce.c onne ct(S ervi ce.j ava: 291( Comp iled Code))
at java x/ma il/S ervi ce.c onne ct(S ervi ce.j ava: 172( Comp iled Code))
at java x/ma il/S ervi ce.c onne ct(S ervi ce.j ava: 121( Comp iled Code))
at java x/ma il/T rans port .sen d0(T rans port .jav a:19 0(Co mpil ed Code))
at java x/ma il/T rans port .sen d(Tr ansp ort. java :120 (Com pile d Code))
at org/ apac he/l og4j /net /SMT PApp ende r.se ndBu ffer (Byt ecod e PC:214(Compiled Code))
at org/ apac he/l og4j /net /SMT PApp ende r.ap pend (Byt ecod e PC:56)
at org/ apac he/l og4j /App ende rSke leto n.do Appe nd(A ppen derS kele ton. java :230 (Com pile d Code))
at org/ apac he/l og4j /hel pers /App ende rAtt acha bleI mpl. appe ndLo opOn Appe nder s(Ap pend erAt tach able Impl .jav a:65 (Com pile d Code))
at org/ apac he/l og4j /Cat egor y.ca llAp pend ers( Cate gory .jav a:20 3(Co mpil ed Code))
at org/ apac he/l og4j /Cat egor y.fo rced Log( Cate gory .jav a:38 8(Co mpil ed Code))
at org/ apac he/l og4j /Cat egor y.er ror( Cate gory .jav a:30 2)
at com/ appl icat ionX /poc s/ui /web /con trol lers /Exc epti onHa ndle r.re solv eExc epti on(E xcep tion Hand ler. java :61)
at org/ spri ngfr amew ork/ web/ serv let/ Disp atch erSe rvle t.pr oces sHan dler Exce ptio n(Di spat cher Serv let. java :112 0(Co mpil ed Code))
at org/ spri ngfr amew ork/ web/ serv let/ Disp atch erSe rvle t.do Disp atch (Dis patc herS ervl et.j ava: 944( Comp iled Code))
at org/ spri ngfr amew ork/ web/ serv let/ Disp atch erSe rvle t.do Serv ice( Disp atch erSe rvle t.ja va:8 52(C ompi led Code))
at org/ spri ngfr amew ork/ web/ serv let/ Fram ewor kSer vlet .pro cess Requ est( Fram ewor kSer vlet .jav a:88 2(Co mpil ed Code))
at org/ spri ngfr amew ork/ web/ serv let/ Fram ewor kSer vlet .doG et(F rame work Serv let. java :778 (Com pile d Code))
at java x/se rvle t/ht tp/H ttpS ervl et.s ervi ce(H ttpS ervl et.j ava: 575( Comp iled Code))
at java x/se rvle t/ht tp/H ttpS ervl et.s ervi ce(H ttpS ervl et.j ava: 668( Comp iled Code))
at com/ ibm/ ws/w ebco ntai ner/ serv let/ Serv letW rapp er.s ervi ce(S ervl etWr appe r.ja va:1 227( Comp iled Code))
at com/ ibm/ ws/w ebco ntai ner/ serv let/ Serv letW rapp er.h andl eReq uest (Ser vlet Wrap per. java :776 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ serv let/ Serv letW rapp er.h andl eReq uest (Ser vlet Wrap per. java :458 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ serv let/ Serv letW rapp erIm pl.h andl eReq uest (Ser vlet Wrap perI mpl. java :178 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/W ebAp pFil terC hain .inv okeT arge t(We bApp Filt erCh ain. java :136 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/W ebAp pFil terC hain .doF ilte r(We bApp Filt erCh ain. java :97( Comp iled Code))
at com/ appl icat ionX /poc s/ui /web /uti l/Se ssio nExp iryF ilte r.do Filt er(S essi onEx piry Filt er.j ava: 60(C ompi led Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/F ilte rIns tanc eWra pper .doF ilte r(Fi lter Inst ance Wrap per. java :195 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/W ebAp pFil terC hain .doF ilte r(We bApp Filt erCh ain. java :91( Comp iled Code))
at com/ appl icat ionX /poc s/ui /web /uti l/Co mmon Filt er.d oFil ter( Comm onFi lter .jav a:44 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/F ilte rIns tanc eWra pper .doF ilte r(Fi lter Inst ance Wrap per. java :195 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/W ebAp pFil terC hain .doF ilte r(We bApp Filt erCh ain. java :91( Comp iled Code))
at com/ appl icat ionX /poc s/ui /web /uti l/GZ ipSe rvle tFil ter. doFi lter (GZi pSer vlet Filt er.j ava: 42(C ompi led Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/F ilte rIns tanc eWra pper .doF ilte r(Fi lter Inst ance Wrap per. java :195 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/W ebAp pFil terC hain .doF ilte r(We bApp Filt erCh ain. java :91( Comp iled Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/W ebAp pFil terM anag er.d oFil ter( WebA ppFi lter Mana ger. java :928 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/W ebAp pFil terM anag er.i nvok eFil ters (Web AppF ilte rMan ager .jav a:10 25(C ompi led Code))
at com/ ibm/ ws/w ebco ntai ner/ weba pp/W ebAp p.ha ndle Requ est( WebA pp.j ava: 3761 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ weba pp/W ebGr oup. hand leRe ques t(We bGro up.j ava: 304( Comp iled Code))
at com/ ibm/ ws/w ebco ntai ner/ WebC onta iner .han dleR eque st(W ebCo ntai ner. java :976 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ WSWe bCon tain er.h andl eReq uest (WSW ebCo ntai ner. java :166 2(Co mpil ed Code))
at com/ ibm/ ws/w ebco ntai ner/ chan nel/ WCCh anne lLin k.re ady( WCCh anne lLin k.ja va:2 00(C ompi led Code))
at com/ ibm/ ws/h ttp/ chan nel/ inbo und/ impl /Htt pInb ound Link .han dleD iscr imin atio n(Ht tpIn boun dLin k.ja va:4 59(C ompi led Code))
at com/ ibm/ ws/h ttp/ chan nel/ inbo und/ impl /Htt pInb ound Link .han dleN ewRe ques t(Ht tpIn boun dLin k.ja va:5 26(C ompi led Code))
at com/ ibm/ ws/h ttp/ chan nel/ inbo und/ impl /Htt pInb ound Link .pro cess Requ est( Http Inbo undL ink. java :312 (Com pile d Code))
at com/ ibm/ ws/h ttp/ chan nel/ inbo und/ impl /Htt pInb ound Link .rea dy(H ttpI nbou ndLi nk.j ava: 283( Comp iled Code))
at com/ ibm/ ws/s sl/c hann el/i mpl/ SSLC onne ctio nLin k.de term ineN extC hann el(S SLCo nnec tion Link .jav a:10 48(C ompi led Code))
at com/ ibm/ ws/s sl/c hann el/i mpl/ SSLC onne ctio nLin k.re adyI nbou ndPo stHa ndsh ake( SSLC onne ctio nLin k.ja va:7 16(C ompi led Code))
at com/ ibm/ ws/s sl/c hann el/i mpl/ SSLC onne ctio nLin k$My Hand shak eCom plet edCa llba ck.c ompl ete( SSLC onne ctio nLin k.ja va:4 12(C ompi led Code))
at com/ ibm/ ws/s sl/c hann el/i mpl/ SSLU tils .han dleH ands hake (SSL Util s.ja va:1 066( Comp iled Code))
at com/ ibm/ ws/s sl/c hann el/i mpl/ SSLH ands hake IOCa llba ck.c ompl ete( SSLH ands hake IOCa llba ck.j ava: 87(C ompi led Code))
at com/ ibm/ ws/t cp/c hann el/i mpl/ AioR eadC ompl etio nLis tene r.fu ture Comp lete d(Ai oRea dCom plet ionL iste ner. java :175 (Com pile d Code))
at com/ ibm/ io/a sync /Abs trac tAsy ncFu ture .inv okeC allb ack( Abst ract Asyn cFut ure. java :217 (Com pile d Code))
at com/ ibm/ io/a sync /Asy ncCh anne lFut ure. fire Comp leti onAc tion s(As yncC hann elFu ture .jav a:16 1(Co mpil ed Code))
at com/ ibm/ io/a sync /Asy ncFu ture .com plet ed(A sync Futu re.j ava: 138( Comp iled Code))
at com/ ibm/ io/a sync /Res ultH andl er.c ompl ete( Resu ltHa ndle r.ja va:2 04(C ompi led Code))
at com/ ibm/ io/a sync /Res ultH andl er.r unEv entP roce ssin gLoo p(Re sult Hand ler. java :816 (Com pile d Code))
at com/ ibm/ io/a sync /Res ultH andl er$2 .run (Res ultH andl er.j ava: 905( Comp iled Code))
at com/ ibm/ ws/u til/ Thre adPo ol$W orke r.ru n(Th read Pool .jav a:18 62(C ompi led Code))
Waiting Threads: 36
Thread-76
WebContainer : 0
WebContainer : 1
WebContainer : 10
WebContainer : 11
WebContainer : 15
WebContainer : 17
WebContainer : 19
WebContainer : 2
WebContainer : 20
WebContainer : 21
WebContainer : 22
WebContainer : 23
WebContainer : 24
WebContainer : 25
WebContainer : 26
WebContainer : 29
WebContainer : 3
WebContainer : 31
WebContainer : 32
WebContainer : 36
WebContainer : 37
WebContainer : 39
WebContainer : 4
WebContainer : 40
WebContainer : 41
WebContainer : 45
WebContainer : 46
WebContainer : 47
WebContainer : 48
WebContainer : 49
WebContainer : 5
WebContainer : 7
WebContainer : 8
WebContainer : 9
pool-13-thread-1
Blocked by: 1
Agent Heartbeat
-->> Agent Heartbeat thread is on progress with introspection by owning the ' StallSweeper ' lock but needs the ' MemorySession ' lock to complete the process, which is being owned by the " WebContainer : 29 " thread.
Thread Name
Agent Heartbeat
State
Deadlock/Blocked
Monitor
Owns Monitor Lock on java /lan g/Ob ject @0x0 0000 0002 D7D8 810
Waiting for Monitor Lock on com/ ibm/ ws/s essi on/s tore /mem ory/ Memo rySe ssio n@0x 0000 0001 1EF1 EF1 8
Java Stack
at com/ ibm/ ws/s essi on/s tore /mem ory/ Memo rySe ssio n.up date Last Acce ssTi me(M emor ySes sion .jav a:63 8(Co mpil ed Code))
at com/ ibm/ ws/s essi on/s tore /mem ory/ Memo rySt ore. getS essi on(M emor ySto re.j ava: 194( Comp iled Code))
at com/ ibm/ ws/s essi on/s tore /mem ory/ Memo rySt ore. getS essi on(M emor ySto re.j ava: 712( Comp iled Code))
at com/ ibm/ ws/s essi on/S essi onMa nage r.ge tSes sion From Stor e(Se ssio nMan ager .jav a:49 7(Co mpil ed Code))
at com/ ibm/ ws/s essi on/S essi onMa nage r.ge tSes sion (Ses sion Mana ger. java :476 (Com pile d Code))
at com/ ibm/ ws/s essi on/S essi onMa nage r.ge tSes sion (Ses sion Mana ger. java :462 (Com pile d Code))
at com/ ibm/ ws/s essi on/S essi onMa nage r.ge tSes sion (Ses sion Mana ger. java :693 (Com pile d Code))
at com/ ibm/ ws/s essi on/S essi onCo ntex t.ge tIHt tpSe ssio n(Se ssio nCon text .jav a:46 6(Co mpil ed Code))
at com/ ibm/ ws/s essi on/S essi onCo ntex t.ge tIHt tpSe ssio n(Se ssio nCon text .jav a:42 6(Co mpil ed Code))
at com/ ibm/ ws/w ebco ntai ner/ srt/ SRTR eque stCo ntex t.ge tSes sion (SRT Requ estC onte xt.j ava: 104( Comp iled Code))
at com/ ibm/ ws/w ebco ntai ner/ srt/ SRTS ervl etRe ques t.ge tSes sion (SRT Serv letR eque st.j ava: 2152 (Com pile d Code))
at sun/ refl ect/ Gene rate dMet hodA cces sor4 06.i nvok e(By teco de PC:65(Compiled Code))
at sun/ refl ect/ Dele gati ngMe thod Acce ssor Impl .inv oke( Dele gati ngMe thod Acce ssor Impl .jav a:37 (Com pile d Code))
at java /lan g/re flec t/Me thod .inv oke( Meth od.j ava: 611( Comp iled Code))
at com/ wily /int rosc ope/ agen t/tr ace/ serv let/ Obje ctWr appe r.in voke Meth odOn Obje ct(O bjec tWra pper .jav a:10 2(Co mpil ed Code))
at com/ wily /int rosc ope/ agen t/tr ace/ serv let/ Requ estW rapp er.g etSe ssio n(Re ques tWra pper .jav a:84 (Com pile d Code))
at com/ wily /int rosc ope/ agen t/tr ace/ serv let/ Serv letP aram eter Load er.a ddSe ssio nID( Serv letP aram eter Load er.j ava: 687( Comp iled Code))
at com/ wily /int rosc ope/ agen t/tr ace/ serv let/ Serv letP aram eter Load er.a ddPa rame ters (Ser vlet Para mete rLoa der. java :625 (Com pile d Code))
at com/ wily /int rosc ope/ agen t/tr ace/ serv let/ Serv letP aram eter Load er.d oWit hWra pper s(Se rvle tPar amet erLo ader .jav a:38 4(Co mpil ed Code))
at com/ wily /int rosc ope/ agen t/tr ace/ serv let/ Serv letI nvoc atio nDat aHel per$ Safe GetS ervl etWr appe rs.e xecu te(S ervl etIn voca tion Data Help er.j ava: 89(C ompi led Code))
at com/ wily /int rosc ope/ agen t/tr ace/ serv let/ Serv letI nvoc atio nDat aHel per$ Safe Exec uteO nInv ocat ionD ataW ithT hrot tlin g.ex ecut eSaf e(Se rvle tInv ocat ionD ataH elpe r.ja va:3 6(Co mpil ed Code))
at com/ wily /int rosc ope/ agen t/tr ace/ Http Serv letT race r.II nvoc atio nDat aPar amet erCa llba ck_a ddPa rame ters (Htt pSer vlet Trac er.j ava: 551( Comp iled Code))
at com/ wily /int rosc ope/ agen t/tr ace/ Invo cati onDa ta.I Comp onen tPar amet erCa llba ck_a ddPa rame ters (Inv ocat ionD ata. java :580 (Com pile d Code))
at com/ wily /int rosc ope/ agen t/tr ace/ Blam ePoi ntTr acer $1.I Comp onen tPar amet erCa llba ck_a ddPa rame ters (Bla mePo intT race r.ja va:1 96(C ompi led Code))
at com/ wily /int rosc ope/ agen t/bl ames tack feat ure/ Blam eSta ckFe atur eSta ckEn try. crea teCo mpon entE vent Data (Bla meSt ackF eatu reSt ackE ntry .jav a:67 (Com pile d Code))
at com/ wily /int rosc ope/ agen t/bl ames tack feat ure/ Blam eSta ckFe atur eBla meSt ack. make AndS endS naps hot( Blam eSta ckFe atur eBla meSt ack. java :281 (Com pile d Code))
at com/ wily /int rosc ope/ agen t/st alls /Sta llFe atur eSta ckEn try. chec kIfS tall ed(S tall Feat ureS tack Entr y.ja va:7 6(Co mpil ed Code))
at com/ wily /int rosc ope/ agen t/st alls /Sta llFe atur e.ch eckI fSta lled (Sta llFe atur e.ja va:1 79(C ompi led Code))
at com/ wily /int rosc ope/ agen t/st alls /Sta llSw eepe r.sw eepS tack s(St allS weep er.j ava: 125( Comp iled Code))
at com/ wily /int rosc ope/ agen t/st alls /Sta llSw eepe r.IT imes tamp edRu nnab le_e xecu te(S tall Swee per. java :96( Comp iled Code))
at com/ wily /uti l/he artb eat/ Inte rval Hear tbea t$Be havi orNo de.e xecu te(I nter valH eart beat .jav a:94 4(Co mpil ed Code))
at com/ wily /uti l/he artb eat/ Inte rval Hear tbea t.ex ecut eNex tBeh avio rAnd Calc ulat eSle epTi me(I nter valH eart beat .jav a:48 9(Co mpil ed Code))
at com/ wily /uti l/he artb eat/ Inte rval Hear tbea t.ac cess $2(I nter valH eart beat .jav a:44 3(Co mpil ed Code))
at com/ wily /uti l/he artb eat/ Inte rval Hear tbea t$He artb eatR unna ble. run( Inte rval Hear tbea t.ja va:6 65(C ompi led Code))
at java /lan g/Th read .run (Thr ead. java :773 )
Waiting Threads: 1
WebConatiner : 6
Blocked by: 1
WebConatiner : 29
-->> WebContainer : 29 thread is in progress with session invalidation after owning the ' MemorySession ' lock at com/ ibm/ ws/s essi on/s tore /mem ory/ Memo rySe ssio n.in vali date (Mem oryS essi on.j ava: 232( Comp iled Code)).
But for logging the invalidation process, this thread needs ' root logger ' lock, which is owned by the " WebContainer : 6 " thread.
Thread Name
WebContainer : 29
State
Deadlock/Blocked
Monitor
Owns Monitor Lock on com/ ibm/ ws/s essi on/s tore /mem ory/ Memo rySe ssio n@0x 0000 0001 1EF1 EF18
Waiting for Monitor Lock on org/ apac he/l og4j /spi /Roo tLog ger@ 0x00 0000 0025 878D C 0
Java Stack
at org/ apac he/l og4j /Cat egor y.ca llAp pend ers( Cate gory .jav a:20 1(Co mpil ed Code))
at org/ apac he/l og4j /Cat egor y.fo rced Log( Cate gory .jav a:38 8(Co mpil ed Code))
at org/ apac he/l og4j /Cat egor y.in fo(C ateg ory. java :663 (Com pile d Code))
at com/ appl icat ionX /poc s/ui /web /uti l/Se ssio nLis tene r.se ssio nDes troy ed(S essi onLi sten er.j ava: 22(C ompi led Code))
at com/ ibm/ ws/s essi on/h ttp/ Http Sess ionO bser ver. sess ionD estr oyed (Htt pSes sion Obse rver .jav a:17 9(Co mpil ed Code))
at com/ ibm/ ws/s essi on/S essi onEv entD ispa tche r.se ssio nDes troy ed(S essi onEv entD ispa tche r.ja va:1 60(C ompi led Code))
at com/ ibm/ ws/s essi on/S tore Call back .ses sion Inva lida ted( Stor eCal lbac k.ja va:1 26(C ompi led Code))
at com/ ibm/ ws/s essi on/s tore /mem ory/ Memo rySe ssio n.in vali date (Mem oryS essi on.j ava: 232( Comp iled Code))
at com/ ibm/ ws/s essi on/h ttp/ Http Sess ionI mpl. inva lida te(H ttpS essi onIm pl.j ava: 303( Comp iled Code))
at com/ ibm/ ws/s essi on/S essi onDa ta.i nval idat e(Se ssio nDat a.ja va:2 47(C ompi led Code))
at com/ ibm/ ws/s essi on/H ttpS essi onFa cade .inv alid ate( Http Sess ionF acad e.ja va:2 00(C ompi led Code))
at com/ appl icat ionX /poc s/ui /web /con trol lers /com mon/ Erro rPag eCon trol ler. disp lay5 00Er ror( Erro rPag eCon trol ler. java :29)
at sun/ refl ect/ Gene rate dMet hodA cces sor3 149. invo ke(B ytec ode PC:48)
at sun/ refl ect/ Dele gati ngMe thod Acce ssor Impl .inv oke( Dele gati ngMe thod Acce ssor Impl .jav a:37 (Com pile d Code))
at java /lan g/re flec t/Me thod .inv oke( Meth od.j ava: 611( Comp iled Code))
at org/ spri ngfr amew ork/ web/ meth od/s uppo rt/I nvoc able Hand lerM etho d.in voke (Inv ocab leHa ndle rMet hod. java :219 (Com pile d Code))
at org/ spri ngfr amew ork/ web/ meth od/s uppo rt/I nvoc able Hand lerM etho d.in voke ForR eque st(I nvoc able Hand lerM etho d.ja va:1 32(C ompi led Code))
at org/ spri ngfr amew ork/ web/ serv let/ mvc/ meth od/a nnot atio n/Se rvle tInv ocab leHa ndle rMet hod. invo keAn dHan dle( Serv letI nvoc able Hand lerM etho d.ja va:1 00(C ompi led Code))
at org/ spri ngfr amew ork/ web/ serv let/ mvc/ meth od/a nnot atio n/Re ques tMap ping Hand lerA dapt er.i nvok eHan dler Meth od(R eque stMa ppin gHan dler Adap ter. java :604 (Com pile d Code))
at org/ spri ngfr amew ork/ web/ serv let/ mvc/ meth od/a nnot atio n/Re ques tMap ping Hand lerA dapt er.h andl eInt erna l(Re ques tMap ping Hand lerA dapt er.j ava: 565( Comp iled Code))
at org/ spri ngfr amew ork/ web/ serv let/ mvc/ meth od/A bstr actH andl erMe thod Adap ter. hand le(A bstr actH andl erMe thod Adap ter. java :80( Comp iled Code))
at org/ spri ngfr amew ork/ web/ serv let/ Disp atch erSe rvle t.do Disp atch (Dis patc herS ervl et.j ava: 923( Comp iled Code))
at org/ spri ngfr amew ork/ web/ serv let/ Disp atch erSe rvle t.do Serv ice( Disp atch erSe rvle t.ja va:8 52(C ompi led Code))
at org/ spri ngfr amew ork/ web/ serv let/ Fram ewor kSer vlet .pro cess Requ est( Fram ewor kSer vlet .jav a:88 2(Co mpil ed Code))
at org/ spri ngfr amew ork/ web/ serv let/ Fram ewor kSer vlet .doG et(F rame work Serv let. java :778 (Com pile d Code))
at java x/se rvle t/ht tp/H ttpS ervl et.s ervi ce(H ttpS ervl et.j ava: 575( Comp iled Code))
at java x/se rvle t/ht tp/H ttpS ervl et.s ervi ce(H ttpS ervl et.j ava: 668( Comp iled Code))
at com/ ibm/ ws/w ebco ntai ner/ serv let/ Serv letW rapp er.s ervi ce(S ervl etWr appe r.ja va:1 227( Comp iled Code))
at com/ ibm/ ws/w ebco ntai ner/ serv let/ Serv letW rapp er.h andl eReq uest (Ser vlet Wrap per. java :776 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ serv let/ Serv letW rapp er.h andl eReq uest (Ser vlet Wrap per. java :458 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ serv let/ Serv letW rapp erIm pl.h andl eReq uest (Ser vlet Wrap perI mpl. java :178 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/W ebAp pFil terC hain .inv okeT arge t(We bApp Filt erCh ain. java :136 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/W ebAp pFil terC hain .doF ilte r(We bApp Filt erCh ain. java :97( Comp iled Code))
at com/ appl icat ionX /poc s/ui /web /uti l/GZ ipSe rvle tFil ter. doFi lter (GZi pSer vlet Filt er.j ava: 50(C ompi led Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/F ilte rIns tanc eWra pper .doF ilte r(Fi lter Inst ance Wrap per. java :195 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/W ebAp pFil terC hain .doF ilte r(We bApp Filt erCh ain. java :91( Comp iled Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/W ebAp pFil terM anag er.d oFil ter( WebA ppFi lter Mana ger. java :928 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ filt er/W ebAp pFil terM anag er.i nvok eFil ters (Web AppF ilte rMan ager .jav a:10 25(C ompi led Code))
at com/ ibm/ ws/w ebco ntai ner/ weba pp/W ebAp pReq uest Disp atch er.d ispa tch( WebA ppRe ques tDis patc her. java :138 5(Co mpil ed Code))
at com/ ibm/ ws/w ebco ntai ner/ weba pp/W ebAp pReq uest Disp atch er.f orwa rd(W ebAp pReq uest Disp atch er.j ava: 194( Comp iled Code))
at com/ ibm/ ws/w ebco ntai ner/ weba pp/W ebAp p.se ndEr ror( WebA pp.j ava: 3263 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ weba pp/W ebAp p.ha ndle Exce ptio n(We bApp .jav a:37 91)
at com/ ibm/ ws/w ebco ntai ner/ weba pp/W ebAp p.ha ndle Requ est( WebA pp.j ava: 3772 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ weba pp/W ebGr oup. hand leRe ques t(We bGro up.j ava: 304( Comp iled Code))
at com/ ibm/ ws/w ebco ntai ner/ WebC onta iner .han dleR eque st(W ebCo ntai ner. java :976 (Com pile d Code))
at com/ ibm/ ws/w ebco ntai ner/ WSWe bCon tain er.h andl eReq uest (WSW ebCo ntai ner. java :166 2(Co mpil ed Code))
at com/ ibm/ ws/w ebco ntai ner/ chan nel/ WCCh anne lLin k.re ady( WCCh anne lLin k.ja va:2 00(C ompi led Code))
at com/ ibm/ ws/h ttp/ chan nel/ inbo und/ impl /Htt pInb ound Link .han dleD iscr imin atio n(Ht tpIn boun dLin k.ja va:4 59(C ompi led Code))
at com/ ibm/ ws/h ttp/ chan nel/ inbo und/ impl /Htt pInb ound Link .han dleN ewRe ques t(Ht tpIn boun dLin k.ja va:5 26(C ompi led Code))
at com/ ibm/ ws/h ttp/ chan nel/ inbo und/ impl /Htt pInb ound Link .pro cess Requ est( Http Inbo undL ink. java :312 (Com pile d Code))
at com/ ibm/ ws/h ttp/ chan nel/ inbo und/ impl /Htt pInb ound Link .rea dy(H ttpI nbou ndLi nk.j ava: 283( Comp iled Code))
at com/ ibm/ ws/s sl/c hann el/i mpl/ SSLC onne ctio nLin k.de term ineN extC hann el(S SLCo nnec tion Link .jav a:10 48(C ompi led Code))
at com/ ibm/ ws/s sl/c hann el/i mpl/ SSLC onne ctio nLin k.re adyI nbou ndPo stHa ndsh ake( SSLC onne ctio nLin k.ja va:7 16(C ompi led Code))
at com/ ibm/ ws/s sl/c hann el/i mpl/ SSLC onne ctio nLin k$My Hand shak eCom plet edCa llba ck.c ompl ete( SSLC onne ctio nLin k.ja va:4 12(C ompi led Code))
at com/ ibm/ ws/s sl/c hann el/i mpl/ SSLU tils .han dleH ands hake (SSL Util s.ja va:1 066( Comp iled Code))
at com/ ibm/ ws/s sl/c hann el/i mpl/ SSLH ands hake IOCa llba ck.c ompl ete( SSLH ands hake IOCa llba ck.j ava: 87(C ompi led Code))
at com/ ibm/ ws/t cp/c hann el/i mpl/ AioR eadC ompl etio nLis tene r.fu ture Comp lete d(Ai oRea dCom plet ionL iste ner. java :175 (Com pile d Code))
at com/ ibm/ io/a sync /Abs trac tAsy ncFu ture .inv okeC allb ack( Abst ract Asyn cFut ure. java :217 (Com pile d Code))
at com/ ibm/ io/a sync /Asy ncCh anne lFut ure. fire Comp leti onAc tion s(As yncC hann elFu ture .jav a:16 1(Co mpil ed Code))
at com/ ibm/ io/a sync /Asy ncFu ture .com plet ed(A sync Futu re.j ava: 138( Comp iled Code))
at com/ ibm/ io/a sync /Res ultH andl er.c ompl ete( Resu ltHa ndle r.ja va:2 04(C ompi led Code))
at com/ ibm/ io/a sync /Res ultH andl er.r unEv entP roce ssin gLoo p(Re sult Hand ler. java :816 (Com pile d Code))
at com/ ibm/ io/a sync /Res ultH andl er$2 .run (Res ultH andl er.j ava: 905( Comp iled Code))
at com/ ibm/ ws/u til/ Thre adPo ol$W orke r.ru n(Th read Pool .jav a:18 62(C ompi led Code))
Waiting Threads: 1
Agent Heartbeat
Blocked Threads: 1
WebContainer : 6
-->> WebContainer : 6 thread shows an exception happened at spring framework at org/
which is being logged by org/apache/log4j by acquiring the ' root logger ' lock and then this thread is calling a mail service and then being introspected by Wily. For introspection, Wily needs the ' StallSweeper ' lock which is being owned by the " Agent Heartbeat " thread.
Thread Name
WebContainer : 6
State
Deadlock/Blocked
Monitor
Owns Monitor Lock on org/
Waiting for Monitor Lock on java
Java Stack
at com/
at com/
at com/
at com/
at com/
at com/
at java
at java
at com/
at java
at java
at com/
at com/
at com/
at com/
at java
at java
at java
at java
at java
at org/
at org/
at org/
at org/
at org/
at org/
at org/
at com/
at org/
at org/
at org/
at org/
at org/
at java
at java
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
Waiting Threads: 36
Thread-76
WebContainer : 0
WebContainer : 1
WebContainer : 10
WebContainer : 11
WebContainer : 15
WebContainer : 17
WebContainer : 19
WebContainer : 2
WebContainer : 20
WebContainer : 21
WebContainer : 22
WebContainer : 23
WebContainer : 24
WebContainer : 25
WebContainer : 26
WebContainer : 29
WebContainer : 3
WebContainer : 31
WebContainer : 32
WebContainer : 36
WebContainer : 37
WebContainer : 39
WebContainer : 4
WebContainer : 40
WebContainer : 41
WebContainer : 45
WebContainer : 46
WebContainer : 47
WebContainer : 48
WebContainer : 49
WebContainer : 5
WebContainer : 7
WebContainer : 8
WebContainer : 9
pool-13-thread-1
Blocked by: 1
Agent Heartbeat
-->> Agent Heartbeat thread is on progress with introspection by owning the ' StallSweeper ' lock but needs the ' MemorySession ' lock to complete the process, which is being owned by the " WebContainer : 29 " thread.
Thread Name
Agent Heartbeat
State
Deadlock/Blocked
Monitor
Owns Monitor Lock on java
Waiting for Monitor Lock on com/
Java Stack
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at sun/
at sun/
at java
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at java
Waiting Threads: 1
WebConatiner : 6
Blocked by: 1
WebConatiner : 29
-->> WebContainer : 29 thread is in progress with session invalidation after owning the ' MemorySession ' lock at com/
But for logging the invalidation process, this thread needs ' root logger ' lock, which is owned by the " WebContainer : 6 " thread.
Thread Name
WebContainer : 29
State
Deadlock/Blocked
Monitor
Owns Monitor Lock on com/
Waiting for Monitor Lock on org/
Java Stack
at org/
at org/
at org/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at sun/
at sun/
at java
at org/
at org/
at org/
at org/
at org/
at org/
at org/
at org/
at org/
at org/
at java
at java
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
at com/
Waiting Threads: 1
Agent Heartbeat
Blocked Threads: 1
WebContainer : 6
Summary:
These locks we have just seen above are raised from Wily Instroscope Agent, WebSphere Session Handler and applicationX, so needs further investigation by involving appropriate product support component teams and application developers to review their corresponding code.
References:
Subscribe to:
Posts (Atom)