Dead men may not tell tales.. But dead JVMs do!

During the last couple of days, I was troubleshooting a critical server crash encountered by an airline, on behalf of a software consulting firm which requested my help. The symptoms were that WebSphere 6.0.2 was crashing with an Out of Memory (OOM) condition, sometimes reaching 100% (or a very high) CPU utilization level in addition. Although you can profile an application with excellent profilers such as JProbe, it becomes much more difficult when you are unable to re-create the issues in any other environment but they keep happening on the live production instances, to which you only have limited access.

Luckily the IBM JVM's generate a Portable Heap Dump (PHD) on an OOM, and has an array of extremely helpful tools to analyze information from heap/thread dumps and and other information offline. Thus it is still possible to detect memory leaks in applications, even when they only occur in live production systems. Sometimes the cause would be heap fragmentation, where even if a considerable percentage of memory is still available, a contiguous chunk of the size required cannot be freed.



The above image shows the IBM Heap Analyzer, detecting a memory leak by the WebSphere DRS / Session Replication, where 923MB of heap has been consumed by 14,011 HashMap#Entry objects held onto by the WebSphere Data Replication Service, used for HTTP session replication.

It is also interesting to look for the use of Xalan 2.6.0 by any application code, as I have at earlier instances found memory leaks that are typically more difficult to trace - but which occurs primarily due to a well known bug in 2.6.0 of Xalan.

A project for Christmas 2008

During the last Christmas holidays, I wanted to use my vacation time to build a corner sofa set for use at home.. however, due to many unexpected obligations, I couldn't finally find any time for it.. So this year, I got a deadline to either build it before the 24th, or to buy it! Since I finally had to get someone else to build the pantry cupboards (which I wanted to do - if I had the time) as well, I decided not to let this opportunity pass by.. after all, I am working as an independent consultant now, and it was holidays around the world anyway!

So I started to look for a simple software to design the structure, the Google Sketchup came in very handy! I also used the measurements I took some time ago from the corner sofa set bought for the WSO2 office in Colombo. I had to spend on the fabric, wood and the foam and it cost around the equivalent of $500.. it took about a week to come into shape, and finally my wife helped with the sewing of the fabric and the cushions. I did stay up all night on the 23rd, to ensure that everything was ready by the morning of the 24th :)

Here are some pics taken during the construction.. if you need the Sketchup files, let me know


Finishing off with padding and foam..




Making the frame..



Cutting the wood.. with the use of support 'templates' to get correct edges with the circular saw..