Mutual Auth - Nginx and ServiceNow
Note: This post details about mutual authentication configuration between ServiceNow and Nginx using self-signed certificates. I have described the steps needed to be taken on both sides in respective sections. In case if you are not interested in reading either part, I would suggest jumping to the relevant section directly as per the below overview. However, it is always a good idea to read the entire post for a better and wholesome understanding.
Overview
Recently, I was able to establish mutual authentication between ServiceNow and Nginx using self-signed certificates. All in all, it was a great learning experience since I built the 3rd party system myself to be integrated with ServiceNow, and I am kind of a newbie when it comes to dealing with certificates. I would not go into the details of what the 3rd party system does as this is not the point in focus. This post is a guidance/note-for-self/means to help others who are trying to achieve the same. Since this is a long post as I cover both sides of the coin, I have divided the topics in the below sequence:
What is Mutual Authentication? - What are certificates? Who issues them? - How to issue self-signed certificates?
Server setup - Nginx installation - Create root key and certificate - Configure HTTPS - Configure mutual authentication on Nginx
ServiceNow configurations - Creating Keystore and CSR
3.1 Creating Client Certificate / Signing the CSR - Importing root and client certificate in Keystore
3.2 Install Keystore in ServiceNow
4. Testing
What is Mutual Authentication?
RFC8120 explains the workings of mutual authentication a.k.a two-way authentication. In that, it describes how this kind of authentication works and provides much better security compared to password-based authentication. I will not go into the details of the messages and their formats being used in exchange, but at the same time, I shall refer to the relevant concepts described in RFC.
At the crux of it, this type of authentication takes place over HTTP protocol and uses certificates that are mutually agreed upon by both client and server. For any kind of communication to be possible between client and server, client-first requests are resources from the server. The server responds by presenting its own certificate and requests the client to provide its certificate. The client validates the server’s certificate and then sends over its own certificate to the server. The server verifies the client certificate - if valid, the server provides access to the requested resource. If not valid, the server denies access to the client. The below diagram represents the basic exchange.
<Diagram>
Well, above is the desired state, and the Server trust store is the server itself in case of self-signed certificates. In order to reach that state, we need to perform certain steps to establish trust between client and server by creating and exchanging appropriate certificates. In our case, the client is ServiceNow and the server is Nginx web/reverse proxy server. This is because ServiceNow requests for a resource and Nginx is responsible to deliver it on demand.
What are certificates? Who issues them?
For clients, there are 2 ways to trust a given server using certificates.
If the certificates are issued to the server by Certificate Authority (CA)
Explicitly trusting certificates that are signed by the server itself (self-signed)
CAs are organizations whose certificates are trusted by default by clients (most browsers) inherently. In the sense, if you bundle certificates issued to you by CA in your responses, then the browsers or clients would trust your server without you having to worry about it. Let’s say the reason is - they have a list of CAs, whose certificates they trust. CAs usually charge some amount to issue you the root certificate in exchange for the client’s trust and saving you a lot of headaches.
An admin can always issue self-signed certificates from a server by using tools like openssl. openssl can help you create root certificate and key using which you can issue client certificates. However, if you implement this in your server responses, clients will have a hard time trusting your server since you who have signed the certificate don’t belong to their list of trusted CAs. Thus, you need to explicitly/manually deliver the root certificate to the client so that they can install it into their list of trusted certificates/sources.
However, things are not so rough in mutual authentication with respect to the example in consideration. As we will actually go through the process of creating root and client certificates, installing these certificates, configuring Nginx and ServiceNow to use them. The good news is - it works!
As far as CAs are concerned, remember, Let’s Encrypt provides free TLS certificates! However, you need to own a domain name in order to work with them. Since I am just working with a public IP in AWS infra, Let’s Encrypt is not useful here. Besides, I wanted to feel the authority of signing the certificates myself.
How to issue self-signed certificates?
Following are the tasks which need to be performed in a sequence which can also be termed as preliminary certificate creation tasks without which we cannot proceed to actually configure any validation logic. All references to CA in the below steps refer to you.
<Diagram>
[On Server] We start with creating the root key a.k.a CA key. You can call this the main password of the entire mutual authentication activity.
[On Server] Using CA key, we create CA certificates. This is the root certificate, your server’s identity.
[On Client] We repeat the same process of creating a key for the client -
client.key.[On Client] It is time for the client to have their own identity. However, like a server, it doesn’t make sense for the client to create its own certificate. This certificate should be issued by the server - since the server own’s the resource requested by the client. Thus, instead of creating certificates for clients, we create a CSR - Certificate Signing Request -
client.csr. The server uses this CSR to sign a new certificate for the client. Handover/pass/send this CSR file to the server.[On Server] Server signs this CSR using root certificate and root key and creates the client certificate -
client.crt.[On Client] Client certificate, key, and root certificate should be bundled together to be used readily in the mutual authentication handshake. We then create a
.pfxfile for the same. This.pfxfile stores the 3 documents together in a single file.
If you were in search of theory, this section was about it. In the next sections, we take a look at commands to be executed using openssl and java keytool and configurations to be done to achieve mutual authentication.
Server setup
Nginx installation
I worked on an Ubuntu 18.04 machine on which I installed the Nginx web server. All the commands below are with respect to Ubuntu 18.04. If you are using Windows or Mac, please refer to the respective corresponding commands.
Update local package repository
sudo apt update
Install nginx
sudo apt install nginx
Adjust firewall settings. This allows traffic on nginx service.
sudo ufw allow 'Nginx Full'
Start nginx
systemctl start nginx
Check nginx service status
systemctl status nginx
If everything worked well, the output of the last command should look something like below:
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2020-07-29 15:31:15 UTC; 2 days ago
Docs: man:nginx(8)
Process: 4735 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited,
Process: 4751 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 4738 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 4753 (nginx)
Tasks: 2 (limit: 1152)
CGroup: /system.slice/nginx.service
├─4753 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
└─4758 nginx: worker process
```Create root key and certificate
This is where we start our work with certificate creation for servers. Follow-through below commands to create the root key and certificate. We use openssl to do the same.
Create CA key. You will be asked to use a passphrase, remember this passphrase because you will need it again in the future.
openssl genrsa -des3 -out ca.key 4096
File created: ca.key
Use CA key to create CA certificate. Here we tell
opensslto createca.crtusingca.keywhich is valid for365days.
openssl req -new -x509 -days 365 -key ca.key -out ca.crt
File created: ca.crt
During this step, you will be asked multiple questions. Be sure to note these responses as these are very critical and they should be matched with client configurations.
Country Name (2 letter code) []: <Country Code>
State or Province Name (full name) [Some-State]: <State Name>
Locality Name (eg, city) []: <City Name>
Organization Name (eg, company) [Internet Widgits Pty Ltd]: <Company Name>
Organizational Unit Name (eg, section) []: <OU Name>
Common Name (e.g. server FQDN or YOUR name) []: <Common Name>
Email Address []: <Email address>
Configuring HTTPS
Try visiting http://<Public IP> from your browser. You should be able to see the Nginx homepage. This means, your web server - Nginx is successfully installed and is serving requests. However, it currently works on port 80 and is not secure. Mutual authentication with ServiceNow works well on port 443 i.e. HTTPS. Thus, to make sure Nginx uses SSL for its traffic, we do the following changes to the default config file.
Navigate to the correct Nginx directory. You should find the
defaultfile here.
cd /etc/nginx/sites-available/
Back this file up.
cp default default.bak
The current version of the file has a default server block in place. It represents the current configuration. As discussed before, if you are able to access the Nginx homepage via your browser, then it is probably serving this page on port 80 (HTTP).
In order to make Nginx use SSL/TLS, we add a server block to use port 443 and provide it with
ssl_certificateandssl_certificate_key. Additional server block should look like below. Please note: If you opted to create a passphrase for the root certificate, you should store the password in a file and give its path inssl_password_fileparameter, so that Nginx can make use of it automatically. Otherwise, the Nginx process fails to run.
server {
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
ssl_certificate <path to>/ca.crt;
ssl_password_file <path to>/capass;
ssl_certificate_key <path to>/ca.key;
server_name <Public IP>;
access_log /var/log/nginx/mylogs.com;
}
Redirect HTTP to HTTPS. To redirect traffic from port 80 to 443, modify the already existing server block as below. The last line automatically redirects traffic being received at ports 80 to 443.
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name <Public IP>;
return 301 https://$server_name$request_uri;
}
Restart Nginx.
systemctl restart nginx
Configure mutual authentication on Nginx
Now that we have configured the Nginx webserver to serve HTTPS traffic, we can now proceed to qualify or authenticate incoming requests. There are not many changes that are required. Navigate to the same default Nginx config file and add the below lines to the SSL server block:
ssl_client_certificate <path to>/client.crt;
ssl_verify_client optional;
location / {
if($ssl_client_verify != SUCCESS){
return 403;
}
}
Final SSL Server Block should look like below:
server {
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
ssl_certificate <path to>/ca.crt;
ssl_password_file <path to>/capass;
ssl_certificate_key <path to>/ca.key;
ssl_client_certificate <path to>/client.crt;
ssl_verify_client optional;
server_name <Public IP>;
access_log /var/log/nginx/mylogs.com;
location / {
if($ssl_client_verify != SUCCESS){
return 403;
}
}
}
By doing these changes, we basically tell Nginx to verify the client certificate included in all the requests for path / by matching it with client.crt. If ssl_client_verify is successful, then process the request normally (reverse proxying it to relevant application), otherwise return 403 (Forbidden).
Before you ask - we don’t have a client.crt file yet. This file needs to be created after we receive the client CSR (Certificate Signing Request). For now, let us assume we have it in the given path. Creation of client.crt is covered in the next section.
At this moment, if you try to access this public IP address using a browser, you should see 403 Forbidden the error. This is as per our expectations. We have successfully been able to restrict access to this server since we do not possess the right certificates.
ServiceNow configurations
Creating Keystore and CSR
ServiceNow makes use of Java Keystore to store root and client certificates. Compared to the server-side steps, we do not have access to the ServiceNow host system to perform openssl operations. So we use Java Keytool to generate client keys and certificates on our local system. The local system could be any system you are working on. Make sure you install JDK, and you are good to go with keytool.
Please note: we need to create a Keystore file to be uploaded into ServiceNow to make ServiceNow behave like a client. The Keystore file should contain the client certificate and root certificate.
ServiceNow documentation provides us with high steps, but somewhere I felt I need more clarity on how the certificates and Keystore are created.
Generate Java Keystore and key pair. You will be asked to set a password for this keystore. Make sure you remember the same.
keytool -genkey -alias nowclient -keyalg RSA -keystore client.keystore
File created: client.keystore
Generate CSR for ServiceNow. In this step, we create the Certificate Signing Request which we would share with the server to issue our client certificate. While doing this, you will be asked the same set of questions that were asked while creating the root certificate. Please make sure your answers are identical here. It is very important because it simply doesn’t work.
keytool -certreq -alias nowclient -keystore client.keystore -file client.csr
File created: client.crt
Share the client.csr with the server team. In this case, you yourself are the owner of this server. In the real-world scenario, this could be some different team.
Creating Client Certificate / Signing the CSR
Log into your server (Nginx) machine and save the client.csr on this machine. Execute below openssl command on the server machine to generate client.crt.
openssl x509 -req -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out client.crt
File created: client.crt
There you have the client certificate. Do check if the path where client.crt is stored, matches with the path specified in Nginx default config file for ssl_client_certificate.
Send this client.crt and ca.crt to your local system for including it in Keystore.
Importing root and client certificate in Keystore
On your local system, where you have created client.keystore, execute the below command to import root and client certificates.
Root certificate:
keytool -import -trustcacerts -alias root -file ca.crt -keystore client.keystoreClient certificate:
keytool -import -trustcacerts -alias nowclient -file client.crt -keystore client.keystore client.keystore- is now ready to be installed in ServiceNow.
Install Keystore in ServiceNow
Log in to ServiceNow using admin credentials. Navigate to System Definition > Certificates. Create a new record and set the type to be Java Key Store. Let’s call it “mycerts” and attach client.keystore it to this record. Save the record and in the “Related Links” click on Validate Stores/Certificates. Doing this should flash a message saying “Valid key_store”.
Next, you need to upload a root certificate in the form of “Trust Store Cert”. To do this, you would need the root certificate in .pem format. It is easy to identify the format - if you open the certificate file (ca.crt) and you can see normal alphanumeric characters in the certificate then it is .pem the format. If you have followed the above steps so far, then you should by default have the root certificate in .pem format.
In Certificates, create a new record and set the type to Trust Store Cert, and format to be PEM. Give this record a name and paste the content of the file in PEM Certificate the text box provided. Saving this record like this, if the certificate is valid, it should populate the fields like “Issuer”, “Subject”, “Expires in days” etc. Verify that Issuer and Subject have the same values - if not, the authentication will not work.
Next, navigate to System Security > Protocol Profiles and create a new record. This step is required as this is the way you tell ServiceNow which client Keystore to use while making REST API calls to our public endpoint. On the form, fill in your desired protocol name - it could be anything other than http or https. I chose rhttps and I don’t know why. Select “mycerts” in Keystore the field. Set the Default port to be 443. Save the record and that is it. Technically, we have been able to establish mutual authentication between ServiceNow and Nginx server. However, we have not tested it.
Testing
This is what I call the easiest and scariest part. Let us define our test cases based on our expectations. Our expectation is that any request hitting the Nginx server from ServiceNow - should serve the Nginx home page - indicating the server is not rejecting our requests. Any request which does not originate from ServiceNow should get 403.
In fact, the 2nd part has already been tested when we modified the Nginx default config file to enable ssl_verify_client.
Navigate to System Web Services > Outbound > REST Message and create a REST message. I assume you already know how to do it, so I would not go into the details. For non-ServiceNow users - in order to make a REST API call to our public IP, we need to configure a REST Message where we set the endpoint as IP of Nginx server and do configurations to make this web service call using mutual authentication profile which we created in the previous step.
Doing this, a Default GET request gets created automatically. Click on the Default GET method and click on the test related link. This should give the Nginx home page in the response.
This is a huge post with a lot of technical details. I have tried my best to put as much information which is required to make this work. Do get in touch if you have any questions and I shall try my best to answer your questions.