DeadLock Identification from a Single Javacore File in WebSphere on Linux .

-
Sample Javacore Extracts and Locking Explanation for a DeadLock Condition:

Deadlock detected !!!

Thread "WebContainer : 6" (0x0000000009306B00)
is waiting for:
sys_mon_t:0x00002B9F669C9D50 infl_mon_t: 0x00002B9F669C9DC8:
java/lang/Object@0x000000002D7D8810
which is owned by:
Thread "Agent Heartbeat" (0x0000000000808400)
which is waiting for:
sys_mon_t:0x00002B9F681EA980 infl_mon_t: 0x00002B9F681EA9F8:
com/ibm/ws/session/store/memory/MemorySession@0x000000011EF1EF18
which is owned by:
Thread "WebContainer : 29" (0x000000000A4E7500)
which is waiting for:
sys_mon_t:0x00002B9F6AA7A7C0 infl_mon_t: 0x00002B9F6AA7A838:
org/apache/log4j/spi/RootLogger@0x0000000025878DC0
which is owned by:
Thread "WebContainer : 6" (0x0000000009306B00)
 

The above entries from the LOCKS section of Javacore.txt shows a DeadLock condition happened between (3) threads ' WebContainer : 6  ', ' Agent Heartbeat ' and ' WebContainer : 29 '.
Note:  When reviewing Systremout.log from WebSphere the profile/logs directory, DeadLocks will most probably show up as WSRV0605W hung thread warning messages and can only be diagnosed as a "DeadLock" when reviewing the Javacores in either a text editor or using specialized tooling such as IBM Thread and Monitor Dump Analyzer for Java.
 

-->>  WebContainer : 6 thread shows an exception happened at spring framework at org/apache/log4j/Category.callAppenders(Category.java:203(Compiled Code))
which is being logged by org/apache/log4j by acquiring the ' root logger ' lock and then this thread is calling a mail service and then being introspected by Wily.  For introspection, Wily needs the ' StallSweeper ' lock which is being owned by the " Agent Heartbeat " thread.

Thread Name
WebContainer : 6
State
Deadlock/Blocked
Monitor
Owns Monitor Lock on org/apache/log4j/spi/RootLogger@0x0000000025878DC0
Waiting for Monitor Lock on java/lang/Object@0x000000002D7D8810
Java Stack
at com/wily/introscope/agent/blamestackfeature/BlameStackFeatureBlameStack.IBlameStack_addExtraParameter(BlameStackFeatureBlameStack.java:105(Compiled Code))
at com/wily/introscope/agent/blame/DuplicateHandlingBlameStack.IBlameStack_addExtraParameter(DuplicateHandlingBlameStack.java:60(Compiled Code))
at com/wily/introscope/agent/blame/CompoundBlameStack.IBlameStack_addExtraParameter(CompoundBlameStack.java:165(Compiled Code))
at com/wily/introscope/agent/blame/ComponentTracer.addExtraParameter(ComponentTracer.java:328(Compiled Code))
at com/wily/introscope/agent/blame/ComponentTracer.addExtraParameter(ComponentTracer.java:316(Compiled Code))
at com/wily/introscope/agent/trace/io/SocketBackendTracer.annotateBlameStack(SocketBackendTracer.java:153(Compiled Code))
at java/net/ManagedSocketInputStreamHighPerformance.annotateBlameStack(ManagedSocketInputStreamHighPerformance.java:328(Compiled Code))
at java/net/ManagedSocketInputStreamHighPerformance.read(ManagedSocketInputStreamHighPerformance.java:231(Compiled Code))
at com/sun/mail/util/TraceInputStream.read(TraceInputStream.java:106(Compiled Code))
at java/io/BufferedInputStream.fill(BufferedInputStream.java:229(Compiled Code))
at java/io/BufferedInputStream.read(BufferedInputStream.java:248(Compiled Code))
at com/sun/mail/util/LineInputStream.readLine(LineInputStream.java:84(Compiled Code))
at com/sun/mail/smtp/SMTPTransport.readServerResponse(SMTPTransport.java:1742(Compiled Code))
at com/sun/mail/smtp/SMTPTransport.openServer(SMTPTransport.java:1523(Compiled Code))
at com/sun/mail/smtp/SMTPTransport.protocolConnect(SMTPTransport.java:453(Compiled Code))
at javax/mail/Service.connect(Service.java:291(Compiled Code))
at javax/mail/Service.connect(Service.java:172(Compiled Code))
at javax/mail/Service.connect(Service.java:121(Compiled Code))
at javax/mail/Transport.send0(Transport.java:190(Compiled Code))
at javax/mail/Transport.send(Transport.java:120(Compiled Code))
at org/apache/log4j/net/SMTPAppender.sendBuffer(Bytecode PC:214(Compiled Code))
at org/apache/log4j/net/SMTPAppender.append(Bytecode PC:56)
at org/apache/log4j/AppenderSkeleton.doAppend(AppenderSkeleton.java:230(Compiled Code))
at org/apache/log4j/helpers/AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65(Compiled Code))
at org/apache/log4j/Category.callAppenders(Category.java:203(Compiled Code))
at org/apache/log4j/Category.forcedLog(Category.java:388(Compiled Code))
at org/apache/log4j/Category.error(Category.java:302)
at com/applicationX/pocs/ui/web/controllers/ExceptionHandler.resolveException(ExceptionHandler.java:61)
at org/springframework/web/servlet/DispatcherServlet.processHandlerException(DispatcherServlet.java:1120(Compiled Code))
at org/springframework/web/servlet/DispatcherServlet.doDispatch(DispatcherServlet.java:944(Compiled Code))
at org/springframework/web/servlet/DispatcherServlet.doService(DispatcherServlet.java:852(Compiled Code))
at org/springframework/web/servlet/FrameworkServlet.processRequest(FrameworkServlet.java:882(Compiled Code))
at org/springframework/web/servlet/FrameworkServlet.doGet(FrameworkServlet.java:778(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:575(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:668(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1227(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:776(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:458(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:97(Compiled Code))
at com/applicationX/pocs/ui/web/util/SessionExpiryFilter.doFilter(SessionExpiryFilter.java:60(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/applicationX/pocs/ui/web/util/CommonFilter.doFilter(CommonFilter.java:44(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/applicationX/pocs/ui/web/util/GZipServletFilter.doFilter(GZipServletFilter.java:42(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:928(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebApp.handleRequest(WebApp.java:3761(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebGroup.handleRequest(WebGroup.java:304(Compiled Code))
at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:976(Compiled Code))
at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662(Compiled Code))
at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:200(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:526(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:312(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.ready(HttpInboundLink.java:283(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1048(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLConnectionLink.readyInboundPostHandshake(SSLConnectionLink.java:716(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLConnectionLink$MyHandshakeCompletedCallback.complete(SSLConnectionLink.java:412(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLUtils.handleHandshake(SSLUtils.java:1066(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLHandshakeIOCallback.complete(SSLHandshakeIOCallback.java:87(Compiled Code))
at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175(Compiled Code))
at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code))
at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161(Compiled Code))
at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:138(Compiled Code))
at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:204(Compiled Code))
at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:816(Compiled Code))
at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905(Compiled Code))
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1862(Compiled Code))

Waiting Threads:  36
Thread-76
WebContainer : 0
WebContainer : 1
WebContainer : 10
WebContainer : 11
WebContainer : 15
WebContainer : 17
WebContainer : 19
WebContainer : 2
WebContainer : 20
WebContainer : 21
WebContainer : 22
WebContainer : 23
WebContainer : 24
WebContainer : 25
WebContainer : 26
WebContainer : 29
WebContainer : 3
WebContainer : 31
WebContainer : 32
WebContainer : 36
WebContainer : 37
WebContainer : 39
WebContainer : 4
WebContainer : 40
WebContainer : 41
WebContainer : 45
WebContainer : 46
WebContainer : 47
WebContainer : 48
WebContainer : 49
WebContainer : 5
WebContainer : 7
WebContainer : 8
WebContainer : 9
pool-13-thread-1

Blocked by:  1
Agent Heartbeat


-->> Agent Heartbeat thread is on progress with introspection by owning the ' StallSweeper ' lock but needs the ' MemorySession ' lock to complete the process, which is being owned by the " WebContainer : 29 " thread.

Thread Name
Agent Heartbeat
State
Deadlock/Blocked
Monitor
Owns Monitor Lock on java/lang/Object@0x000000002D7D8810
Waiting for Monitor Lock on com/ibm/ws/session/store/memory/MemorySession@0x000000011EF1EF18
Java Stack
at com/ibm/ws/session/store/memory/MemorySession.updateLastAccessTime(MemorySession.java:638(Compiled Code))
at com/ibm/ws/session/store/memory/MemoryStore.getSession(MemoryStore.java:194(Compiled Code))
at com/ibm/ws/session/store/memory/MemoryStore.getSession(MemoryStore.java:712(Compiled Code))
at com/ibm/ws/session/SessionManager.getSessionFromStore(SessionManager.java:497(Compiled Code))
at com/ibm/ws/session/SessionManager.getSession(SessionManager.java:476(Compiled Code))
at com/ibm/ws/session/SessionManager.getSession(SessionManager.java:462(Compiled Code))
at com/ibm/ws/session/SessionManager.getSession(SessionManager.java:693(Compiled Code))
at com/ibm/ws/session/SessionContext.getIHttpSession(SessionContext.java:466(Compiled Code))
at com/ibm/ws/session/SessionContext.getIHttpSession(SessionContext.java:426(Compiled Code))
at com/ibm/ws/webcontainer/srt/SRTRequestContext.getSession(SRTRequestContext.java:104(Compiled Code))
at com/ibm/ws/webcontainer/srt/SRTServletRequest.getSession(SRTServletRequest.java:2152(Compiled Code))
at sun/reflect/GeneratedMethodAccessor406.invoke(Bytecode PC:65(Compiled Code))
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:611(Compiled Code))
at com/wily/introscope/agent/trace/servlet/ObjectWrapper.invokeMethodOnObject(ObjectWrapper.java:102(Compiled Code))
at com/wily/introscope/agent/trace/servlet/RequestWrapper.getSession(RequestWrapper.java:84(Compiled Code))
at com/wily/introscope/agent/trace/servlet/ServletParameterLoader.addSessionID(ServletParameterLoader.java:687(Compiled Code))
at com/wily/introscope/agent/trace/servlet/ServletParameterLoader.addParameters(ServletParameterLoader.java:625(Compiled Code))
at com/wily/introscope/agent/trace/servlet/ServletParameterLoader.doWithWrappers(ServletParameterLoader.java:384(Compiled Code))
at com/wily/introscope/agent/trace/servlet/ServletInvocationDataHelper$SafeGetServletWrappers.execute(ServletInvocationDataHelper.java:89(Compiled Code))
at com/wily/introscope/agent/trace/servlet/ServletInvocationDataHelper$SafeExecuteOnInvocationDataWithThrottling.executeSafe(ServletInvocationDataHelper.java:36(Compiled Code))
at com/wily/introscope/agent/trace/HttpServletTracer.IInvocationDataParameterCallback_addParameters(HttpServletTracer.java:551(Compiled Code))
at com/wily/introscope/agent/trace/InvocationData.IComponentParameterCallback_addParameters(InvocationData.java:580(Compiled Code))
at com/wily/introscope/agent/trace/BlamePointTracer$1.IComponentParameterCallback_addParameters(BlamePointTracer.java:196(Compiled Code))
at com/wily/introscope/agent/blamestackfeature/BlameStackFeatureStackEntry.createComponentEventData(BlameStackFeatureStackEntry.java:67(Compiled Code))
at com/wily/introscope/agent/blamestackfeature/BlameStackFeatureBlameStack.makeAndSendSnapshot(BlameStackFeatureBlameStack.java:281(Compiled Code))
at com/wily/introscope/agent/stalls/StallFeatureStackEntry.checkIfStalled(StallFeatureStackEntry.java:76(Compiled Code))
at com/wily/introscope/agent/stalls/StallFeature.checkIfStalled(StallFeature.java:179(Compiled Code))
at com/wily/introscope/agent/stalls/StallSweeper.sweepStacks(StallSweeper.java:125(Compiled Code))
at com/wily/introscope/agent/stalls/StallSweeper.ITimestampedRunnable_execute(StallSweeper.java:96(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat$BehaviorNode.execute(IntervalHeartbeat.java:944(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime(IntervalHeartbeat.java:489(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat.access$2(IntervalHeartbeat.java:443(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:665(Compiled Code))
at java/lang/Thread.run(Thread.java:773)

Waiting Threads:  1
WebConatiner :  6

Blocked by:  1
WebConatiner :  29


-->>  WebContainer : 29 thread is in progress with session invalidation after owning the ' MemorySession ' lock at com/ibm/ws/session/store/memory/MemorySession.invalidate(MemorySession.java:232(Compiled Code)).
But for logging the invalidation process, this thread needs ' root logger ' lock, which is owned by the " WebContainer : 6 " thread.
 
Thread Name
WebContainer : 29
State
Deadlock/Blocked
Monitor
Owns Monitor Lock on com/ibm/ws/session/store/memory/MemorySession@0x000000011EF1EF18
Waiting for Monitor Lock on org/apache/log4j/spi/RootLogger@0x0000000025878DC0
Java Stack
at org/apache/log4j/Category.callAppenders(Category.java:201(Compiled Code))
at org/apache/log4j/Category.forcedLog(Category.java:388(Compiled Code))
at org/apache/log4j/Category.info(Category.java:663(Compiled Code))
at com/applicationX/pocs/ui/web/util/SessionListener.sessionDestroyed(SessionListener.java:22(Compiled Code))
at com/ibm/ws/session/http/HttpSessionObserver.sessionDestroyed(HttpSessionObserver.java:179(Compiled Code))
at com/ibm/ws/session/SessionEventDispatcher.sessionDestroyed(SessionEventDispatcher.java:160(Compiled Code))
at com/ibm/ws/session/StoreCallback.sessionInvalidated(StoreCallback.java:126(Compiled Code))
at com/ibm/ws/session/store/memory/MemorySession.invalidate(MemorySession.java:232(Compiled Code))
at com/ibm/ws/session/http/HttpSessionImpl.invalidate(HttpSessionImpl.java:303(Compiled Code))
at com/ibm/ws/session/SessionData.invalidate(SessionData.java:247(Compiled Code))
at com/ibm/ws/session/HttpSessionFacade.invalidate(HttpSessionFacade.java:200(Compiled Code))
at com/applicationX/pocs/ui/web/controllers/common/ErrorPageController.display500Error(ErrorPageController.java:29)
at sun/reflect/GeneratedMethodAccessor3149.invoke(Bytecode PC:48)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code))
at java/lang/reflect/Method.invoke(Method.java:611(Compiled Code))
at org/springframework/web/method/support/InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219(Compiled Code))
at org/springframework/web/method/support/InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132(Compiled Code))
at org/springframework/web/servlet/mvc/method/annotation/ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:100(Compiled Code))
at org/springframework/web/servlet/mvc/method/annotation/RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:604(Compiled Code))
at org/springframework/web/servlet/mvc/method/annotation/RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:565(Compiled Code))
at org/springframework/web/servlet/mvc/method/AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80(Compiled Code))
at org/springframework/web/servlet/DispatcherServlet.doDispatch(DispatcherServlet.java:923(Compiled Code))
at org/springframework/web/servlet/DispatcherServlet.doService(DispatcherServlet.java:852(Compiled Code))
at org/springframework/web/servlet/FrameworkServlet.processRequest(FrameworkServlet.java:882(Compiled Code))
at org/springframework/web/servlet/FrameworkServlet.doGet(FrameworkServlet.java:778(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:575(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:668(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1227(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:776(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:458(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:97(Compiled Code))
at com/applicationX/pocs/ui/web/util/GZipServletFilter.doFilter(GZipServletFilter.java:50(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:928(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebAppRequestDispatcher.dispatch(WebAppRequestDispatcher.java:1385(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:194(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebApp.sendError(WebApp.java:3263(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebApp.handleException(WebApp.java:3791)
at com/ibm/ws/webcontainer/webapp/WebApp.handleRequest(WebApp.java:3772(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebGroup.handleRequest(WebGroup.java:304(Compiled Code))
at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:976(Compiled Code))
at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662(Compiled Code))
at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:200(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:526(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:312(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.ready(HttpInboundLink.java:283(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1048(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLConnectionLink.readyInboundPostHandshake(SSLConnectionLink.java:716(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLConnectionLink$MyHandshakeCompletedCallback.complete(SSLConnectionLink.java:412(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLUtils.handleHandshake(SSLUtils.java:1066(Compiled Code))
at com/ibm/ws/ssl/channel/impl/SSLHandshakeIOCallback.complete(SSLHandshakeIOCallback.java:87(Compiled Code))
at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175(Compiled Code))
at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code))
at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161(Compiled Code))
at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:138(Compiled Code))
at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:204(Compiled Code))
at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:816(Compiled Code))
at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905(Compiled Code))
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1862(Compiled Code))

Waiting Threads:  1
Agent Heartbeat

Blocked Threads:  1
WebContainer :  6


Summary: 
These locks we have just seen above are raised from Wily Instroscope Agent, WebSphere Session Handler and applicationX, so needs further investigation by involving appropriate product support component teams and application developers to review their corresponding code.

References: 

Comments

Post a Comment

Popular posts from this blog

fix pack installation on websphere using command line (imcl)

-
 Hi friends, Today here is one more important topic in WAS. For installing fix packs , ifixes with command line we use imcl commnd. * During installing Products and Fixes with IBM Installation Manager, Installation Manager searches open repositories where Packages and Fixes exist. But how can you verify the applicable packages, updates, and features are in a target repository from the   command line? Installation Manager provides the command line tool, imcl, to manage installation. The imcl command can be found in <IM_ROOT>/eclipse/tools subdirectory. **Following are the commands and description for it 1] encryptString stringToEncrypt:    Encrypt the entered string. Use the encryptString command with the -passwordKey option to increase encryption security. 2] exportInstallData outputFileName:      Export the installation data to the specified file in a compressed file format where outputFileName is the name of the generated file that ...
Read more

How to diagnose error "SRVE0255E: A WebGroup/Virtual Host to handle {0} has not been defined"

-
Image
Error SRVE0255E means that the webcontainer could not find a web group (web module) or virtual host to handle the request. Here are the steps to diagnose this error: 1. Make sure that the URL entered at the browser is correct. Particularly, make sure that the context root from the URL matches the context root configured for the application. 2. Review the SystemOut.log to make sure that the application and server are started successfully and without any errors. 3. Verify that the application web module is mapped to the correct/intended virtual host. You can do this from the admin console by navigating to the following path: Applications > Websphere enterprise applications > [app_name] > Virtual hosts 4. Under the virtual host that the application is mapped to (#3), make sure that there is a host alias definition for the host name and port number that this request is sent to. You can do this from the admin console by navigating to the following path: E...
Read more

HMGR0152W: CPU Starvation detected messages in SystemOut.log

-
Problem(Abstract) New system is working properly but HMGR warning messages are being logged in the SystemOut.log file. Symptom [10/25/05 16:42:27:635 EDT] 0000047a CoordinatorCo W HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 9 seconds. Cause The HMGR0152W message is an indication that JVM thread scheduling delays are occurring for this process. The WebSphere® Application Server high availability manager component contains thread scheduling delay detection logic, that periodically schedules a thread to run and tracks whether the thread was dispatched and run as scheduled. By default, a delay detection thread is scheduled to run every 30 seconds, and will log a HMGR0152W message if it is not run within 5 seconds of the expected schedule. The message will indicate the delay time or time differential between when the thread was expected to get the CPU, and when the thread actually got CPU cycles. The HMGR0152W message can ...
Read more