Problem

I've configured the https port. I've created a self signed certificate using open ssl and when I test the url using the browser I get the trusted certficate pop up but after that the page goes into error.

Solution

How did you create your self sign your certificate? Follow the procedure in the walkthrough below.

Walkthrough

Generate your certificates

Install the Certificate Toolkit by running the Webmethods Installer.
Follow the steps found in the CertificateToolkitUsersGuide.pdf or alternatively, follow this simple guide SSL Certificate Setup using WM CertificateToolKit .
You will need to generate a private key and a certificate signing request (CSR).

Sign your certificate

You can get a CA such as verisign to sign it. Or you can sign it yourself. Here's how:

1. Download OpenSSL. I used http://www.slproweb.com/products/Win32OpenSSL.html
2. Once you have installed it go to the OpenSSL bin directory.
3. You'll need to configure the Root CA, you can do this by first deleting PEM directory and running the following commands:

   openssl md5 * > rand1.dat
   
   openssl genrsa -rand rand1.dat -des3 1024 > ca.key
   
   openssl req -new -key ca.key -out ca.csr
   
   openssl x509 -in ca.csr -out ca.crt -req -signkey ca.key -days 3650
   

4. Now you can self sign the certificate. Note that you need to replace csr.pem with the CSR filename you created previously.
   Note: To specify certificate's expiry date, add option -days <no_days_to_expire> at the end of this command below

   openssl x509 -req -in csr.pem -out cert.crt -CA ca.crt -CAkey ca.key -CAcreateserial
   

Convert the Signed Certificate and CA root certficate to the DER format

Again, read CertificateToolkitUsersGuide.pdf. It will tell you how to convert a CRT to DER format.
You will need to convert ca.crt and cert.crt to DER format. ca.crt is the CA root certificate. cert.crt is the server certificate.

Copy the server private key, server certicate and CA root certifcate to the webmethods server.

If you're copying it to a UNIX server remember to set the transfer mode to binary.
You can copy the certificates anywhere under the IntegrationServer directory.
Config directory is a good place.
For example:

1. CA Root - config/cas/ca.der
2. Server Certificate - config/cert.der
3. Server Private Key - config/privateKey.der (you generated this using the certificate toolkit in the very first step)

Add certificates to the server using the IS admin console

Refer to this document IntegrationServerAdminGuide.pdf
Or you can enter the details like so:

Create a Https Port

Again, refer to this document IntegrationServerAdminGuide.pdf.
In the diagram below, the port is using the certificate setup defined in the "Add certificates ..." step.

Test HTTPS Port

Ensure that your browser settings does not use a proxy.
Then try testing by invoking via URL: https://<Host>:<HTTPS_Port>