WebSphere Application Server

Situation Occasionally when reviewing a heapdump that was generated from an out of memory event, the largest consumer of heap memory may consume slightly more memory than the other objects. Because of this, WebSphere users and moderators may incorrectly accuse the largest heap consumer as a memory leak.   If a Java™ heap is sized too small, an out of memory error will occur regardless if a memory leak exists. The leak suspect shown in IBM HeapAnalyzer or Memory Analyzer (MAT) will show a java object that is expected to exhaust a relatively larger portion of the heap.   Resolution The best course of action will be to increase the heap size and enable verboseGC (if not already enabled). Doing this will produce one of two results: it will alleviate the out of memory problem by letting your heap utilize more memory, or the leaking object will cause another out of memory event. If the test yields the first of the two outcomes, the administrator ca...

List of firewall ports that must be open for communication between the deployment manager, nodeagent, and Application Server   It's very important to know the ports that should be opened in the firewall for proper communication between deployment manager, nodeagent, and application servers. Follow the below steps to achieve this task.   Find the port numbers in the serverindex.xml file or from the ISC (Integrated Solution Console) The serverindex.xml file can be found under the prof ile- root /con fig/ cell s/ce llNa me/n odes /nod eNam e folder.   From Integrated Solution Console: Application Server ports Click on servers -> server Name -> Expand ports under communication Nodeagent ports Click on System administration -> node agents -> nodeagent -> Expand ports under Additional Properties Deployment Manager ports Click on System administration -> Deployment manager -> Expand ports under Additional Propertie...

Class loader synchronization bottlenecks are common problems brought to the attention of WebSphere Application Server L2 Support by our clients.  This is a common type of performance problem called a livelock where many threads are contending on a set of locks. The performance overhead is proportional to the lock contention, so the problem might not show up in a test environment with a low number of concurrent requests, or patterns of requests which aren't exercising the lock at the same time.  From the viewpoint of the application end user, this is seen as an application that is moving slowly.  From the viewpoint of the WebSphere Administrator viewing the WebSphere Application Server SystemOut.log , this is typically seen as a series of hung ( WSVR0605W ) thread messages which "complete" ( WSVR0606W ) on their own....eventually. To diagnose, manually capture a javacore DURING the time while the thread is still hung, (o r  au toma tica lly  c...

Sample Javacore Extracts and Locking Explanation for a DeadLock Condition : Deadlock detected !!! Thread " WebContainer : 6 " (0x0 0000 0000 9306 B00) is waiting for: sys_ mon_ t:0x 0000 2B9F 669C 9D50 infl_mon_t: 0x00002B9F669C9DC8: java /lan g/Ob ject @0x0 0000 0002 D7D8 810 which is owned by: Thread " Agent Heartbeat " (0x0 0000 0000 0808 400) which is waiting for: sys_ mon_ t:0x 0000 2B9F 681E A980 infl_mon_t: 0x00002B9F681EA9F8: com/ ibm/ ws/s essi on/s tore /mem ory/ Memo rySe ssio n@0x 0000 0001 1EF1 EF18 which is owned by: Thread " WebContainer : 29 " (0x0 0000 0000 A4E7 500) which is waiting for: sys_ mon_ t:0x 0000 2B9F 6AA7 A7C0 infl_mon_t: 0x00002B9F6AA7A838: org/ apac he/l og4j /spi /Roo tLog ger@ 0x00 0000 0025 878D C0 which is owned by: Thread "WebContainer : 6" (0x0 0000 0000 9306 B00)   The above entries from the LOCKS section of Javacore.txt shows a DeadLock condition happened between (3) threads ...