Wily can be uesd to detect troublesome memory leaks and flows http://www.wilytech.com
SAP actually use it themselves for fixing problems in NetWeaver http://www.wilytech.com/solutions/products/IntroscopeForNetWeaver.html
How to analyse thread usage
In HPUX the glance tool can be quite useful to determine whether the operating system threads per process limits are getting close to being reached.
The Integration Server itself gives important information about current number of threads, memory usage, sessions etc. This is accessible from the main web admin screen.