Exploring HTTP and PKI

This blog article is an attempt to present the upper layers of OSI reference model from the perspective of network studies. The three upper layers of OSI reference model – Session, Presentation and Application are also sometimes combined into One as Application Layer in general but some protocols make use of each of this layer in-depth.  Also, these layers are mostly ignored when studying for the network certifications. Understanding of these layers is important as you design your network for the applications running at these layers. In this blog article I will be just writing briefly about the working of HTTP. Also at the end of this article there is a brief description of how public key infrastructure works.

HTTP:
HTTP is a text-based protocol and each line terminates with a Carriage Return – Line Feed (CRLF).

By design HTTP is stateless, but it can use cookies to maintain client state across TCP connections between clients and servers. HTTP 1.1 is the currently widely deployed  and it is backward compatible with the earlier HTTP 1.0

Some Common HTTP Methods

GET:        Is used to Retrieve the requested URI (Both headers and Body)
HEAD:     Is used to Retrieve only the Headers of the requested URI. (Headers Only)
POST:     Is used to send information to the server in html forms.
PUT:         Is used to upload a file to the server
DELETE:     Deletes a URI from a server

URI = Uniform Resource Identifier (URI) is a compact sequence of characters that identifies an abstract or physical resource

Some of the Common HTTP Return Codes from the Server are listed at the end of this article.

HTTP Connection Persistence

When a Client asks information from the server, it opens a TCP connection to the server. The Client and Server agree on the TCP Connection with a TCP 3 Way Handshake and once the connection is established, the Server sends the requested content to the client on the same TCP Session.

So this is like a 3 step process

1. Open a TCP Connection
2. Complete the HTTP Transaction
3. Close the TCP Connection

I. Open TCP Connection:
Client ———- SYN —————> Server
Client <———SYN/ACK———-Server
Client———– SYN—————> Server

II. Complete the HTMl Transaction
Client (Request) ————– HTTP GET 1.1——————————> Server
Client <————————-HTTP 200 OK—————— (Response) Server

III. Close the TCP Connection
Client —————- RST/FIN——————> Server

IV. Start over From Step-1
For each URL in the HTTP File  and repeat the process until all the content is downloaded

HTTP Persistence:
Webpages may have multiple URLs in the html file, each URL pointing to a different embedded object like images, videos etc.  Each different URL for the embedded files will create a different TCP Connection,  thus the number of TCP connections from each client to the server will increase drastically. To avoid these duplicate TCP Connections, HTTP 1.0 introduced the concept of Persistent Connections. By using Persistent Connections the extra overhead of creating multiple TCP connections is reduced.

Now With HTTP  Persistence The new Flow Looks simpler and only one TCP Connection is open between the Client and the Server to complete all HTTP transactions.

HTTP Persistence is achieved with the keep alive feature which is default ON in HTTP 1.1, but has to be negotiated in HTTP 1.0. The client and server agree of the Connection Keepalive to keep the TCP Connection active until the HTTP transactions are complete.

I. Open TCP Connection:
Client ———- SYN —————> Server
Client <———SYN/ACK———-Server
Client———– SYN—————> Server

II. Complete multiple HTMl Transactions using the Keepalive Features
Client (x.Request) ————– HTTP GET 1.1——————————> Server
Client <————————-HTTP 200 OK—————— (Response) Server

Client (y.Request) ————– HTTP GET 1.1——————————> Server
Client <————————-HTTP 200 OK—————— (Response) Server

Client (z.Request) ————– HTTP GET 1.1——————————> Server
Client <————————-HTTP 200 OK—————— (Response) Server

III. Close the TCP Connection
Client —————- RST/FIN——————> Server

HTTP Pipelining
HTTP 1.1 Supports  Pipelining the HTTP requests, but the implementation is optional and is not implemented by default.
With HTTP Pipelining, the Client can send multiple HTTP Requests for different URLs before getting a response from the Server.

1. Open a TCP Connection
2. Request multiple URLS, without waiting for Response from Server
3. Server Response
4. Close TCP Connection.

Maintaining Client Side State with HTTP Cookies
Since HTTP 1.1 is stateless, the use of cookies enables the client and server to maintain the state of the HTTP Session.

HTTP Cookies are always created by the Server. There are 2 types of cookies which the Servers can create
1. Session Cookies
2. Persistent Cookies

Session Cookies:
Web Browsers Store the cookies only for the life of the Session.
If the Session ends or Web Browser is closed – the Session cookie is deleted.
Session Cookies are useful for keeping track of user information as they browse from page to page on a website.

Persistent Cookies:
Web Browsers store the cookie on the hard disk for the amount of time indicated by the server in the cookie value.
The cookie is not removed even if session ends or web browser is closed.
the cookie only expires when the time to live of the cookie as specified by the server ends.

Cookies are Set by the Server using the “Set-Cookie:” in the header.
If the Cookie is set by the server but the expiration time is not given, then its a Session Cookie.
If the Cookie is set by the server with the expiration time then these are Persistent Cookies.

Example (Session Cookie):
Set-cookie: Session-ID=0123456789
When the client sends a request for a URI, the Server sets the Session Cookie  and responds with the “Set-Cookie:” header set to a unique value to identify the session. The client will then use this value in the subsequent HTTP transactions with the server.

Example (Persistent-Cookie):
Set-Cookie: name=somevalue; expires=DATE;
The server sets the expiry date of the cookie and client stores the cookie in the hard disk for the amount of time specified by the server.  The client uses this cookie in HTTP GET when contacting the server every time.

HTTP Authentication
HTTP 1.1 uses the MD5 Authentication, which is far more superior than the Base64 encoding used in HTTP 1.0
Any password encoded with Base64 can be easily decoded with many online tools available today over internet.

How MD5 works with HTTP 1.1
1. The server challenges the browser with a nonce value.
2. The browser uses this nonce, along with the supplied username and password , as input to a hash algorithm.
3. The browser encrypts login credentials by computing a message digest before sending them over the network.

Note:
1. A nonce is a one-time random number that HTTP 1.1 uses to prevent replay attacks.

2. To generate nonce values, servers normally use the current timestamp in conjunction with its private key (if it has a private key) and the “ETag:” HTTP headers in the client’s requests.

3. A hash function is a function that takes input values and performs a mathematical computation, resulting in a single (smaller) value that is impossible to determine the original values from. These functions are also called one-way hash functions because they are considered impossible to reverse. You can use Message Digest 5 (MD5) and Secure Hash Algorithm 1 (SHA-1) hash algorithms for your web applications to compute public keys from private keys. MD5 provides hashes 128 bits in length, and SHA-1 provides 168-bit hashes.

Moving on to the second part of this blog which gives some brief introduction on how the PKI works.

Public Key Infrastructure (PKI)
PKI deals with keeping the data between clients and servers confidential, maintain data integrity and address authentication when traversing over the public insecure networks like internet. PKI addresses data confidentiality, data integrity and authentication issues. For data confidentiality PKI uses cryptography. Cryptography addresses the issues of keeping data confidential when its traversing over the public networks. Cryptography comes in two forms
1. Secret Key Cryptography
2. Public Key Cryptography

Secret Key Cryptography
1. Both Client and Server each uses their own private  secret key, which they agree upon before any encryption happens ( that is they have to exchange it somehow to agree on)
2. Each uses their own secret key (private) to both encrypt and decrypt the data exchanged.
3. They Keys are either same or one of them is originated by applying some mathematical function to the other key, therefore  this is also called as Symmetric key cryptography and some of the common ones are DES, 3DES, where DES stands for Data Encryption Standard.

With Secret key Cryptography, two forms of ciphers can be used to encrypt the data
1. Stream Ciphers, which enable the network devices to encrypt individual bytes of data stream
2. Block Ciphers, enable network devices to encrypt blocks od data containing many bytes – DES/3DES.

The big challenge with Secret Key Cryptography is to exchange the private keys between client and server over public networks

Public Key Cryptography
One of the big advantages of public key cryptography is that you do not have to exchange the private key for secure communication ( a big challenge in secret key cryptography)

So in Public Key Cryptography, the server generates a pair of keys
1. Private key
2. Public key

The Server keeps the private key secure and keeps it secret and it also makes the public key available to everyone. With these pair of keys in place you can decrypt the information using a public key or a private key. that is information encrypted by private key can be decrypted using the public key and information encrypted using a public key can be decrypted using a private key.

These Private and Public keys are asymmetric which allows to derive the public key from the private key using a hash function, but not the other way, that is you cannot derive the private key using a hash function on the public key.  The common algorithms which run Public key Cryptography are – RSA (Rivest Shamir Adleman) and DSA (Digital Signature Algorithm).

Server A generates a private/public key pair
Server A sends the public key to client C
Client C encrypts a message using Server A’s public key and sends it over to Server A.
Now only Server A can decrypt the message because only it has the private secret key.

The public keys needs to be very large to be really secure since they are distributed to everyone over the pubic insecure networks. To process these large keys both client and server resources have a high utilization so it is not always practical to use this structure. therefor many deployments today use a combination of both Secret key and public key cryptography.

This is a sample of of things can work in an hybrid approach utilizing both secret key and public key approach.

1. Server- “S” generates a pair of private and public keys
2. “S” then sends the public key to client “C”
3. “C” then generates a random number  message and encrypts it using the public key of “S”, and it sends this to “S”.
4. “S” decrypts the message using its private key and uses this random number to generate a Symmetric key for data encryption and decryption.
5. Next all two-way communication between “S” and “C” takes place using this secret symmetric key, which is based on the random number generated by “C”.

Using Certificates
Certificates are used on the servers to guard the identity of the clients and prevent clients from being connected to any malicious server which poses  it self as the legitimate server.

A malicious server on the public insecure network can pose itself as a legitimate server for a resource and generate a its own private and public keys and then distribute the public key to the clients. The clients unaware of this malicious server and believing this to be a legitimate server can connect to it and become a target of confidential information theft.
To prevent this identity theft,  certificates are available to verify the authenticity of the public key owners.

The Certificate Authority (CA) generates a certificate to avoid a possibility of a malicious third party server posing as a real server and clients connecting to it.

1. When the Server generates the Private and Public keys pair, it also generates a Certificate Signing Request (CSR) which contains the public key and other information proving the identity of the originator.
2. The CSR is then sent to the CA manually or via web.
3. The CA verifies the CSR with Registration Authority (RA). If RA verifies and approves the request, then the CA will sign the CSR to vouch for the server that generated the CSR.
4. The CA will then issue a certificate for public use which includes an expiration time of the certificate.
5. The Certificate is then installed on the server.
6. When a client accesses the server, the server sends the certificate to the client, so client can verify the public key actually belongs to the server.

Below is the process which client “C” and server “S” use with the CA signed certificates to verify the authenticity of the legitimate server and to exchange the confidential information using symmetric keys

1. “C” downloads the CA-signed certificate from “S”
2. Then “C” runs the same hash function that the CA ran on the certificate contents during the signing process
3. “C” decrypts the CA’s signature using the CA’s public key and compares its hash to that of the CA, if the two hash values have to match.
4. If the two hash values matched, then “C” extracts the public key of “S” from the certificate
5. Now “C” generates a random number and encrypts it with the public key of “S” and sends it over to “S”. “C” also generates the symmetric key at this time.
6. “S” decrypts the random number sent by “C” using its own private key and it also generate a symmetric key.
7. Now that both “C” and “S” have a symmetric key generated, they exchange their confidential information over public insecure networks using the symmetric secret keys.

SSL – Secure Socket Layer

SSL is a PKI protocol that applications like HTTP, SMTP and FTP use for data integrity and data confidentiality. SSL uses Symmetric key encryption to encrypt packets and appends a Message Authentication Code (MAC) to the SSL packet header. MAC helps ensure the data integrity. From the OSI Reference model perspective SSL lives in the Session layer

SSL is further subdivided into two sub-layers
1. SSL handshake layer
2. SSL Record layer

SSL Handshake Layer
SSL Handshake negotiates session information between the client and the Server, the session information contains
1. Shared Secret: Is a random number used to create symmetric keys for encrypting data.
2. Session Identifier: Is used to identify an SSL session
3. Certificates:  Certificates installed by Servers and Clients
4. Cipher Suite:  Uses protocols TLS or SSL. Asymmetric algorithm is used for key exchange and symmetric algorithm is used for data encryption. Also a hash algorithm used for data integrity (MD5 or SHA-1).

SSL Change Cipher Spec Protocol:
This is used by clients and servers during the full handshake to indicate each other to use the negotiated keys for the current session or to resume an idle session.

SSL Record Protocol: Does the data encryption using the shared secret established in handshake process
SSL Alert Protocol: Signals problems with SSL session and terminates the SSL connection.

SSL handshake:
SSL handshake is a four step process to start a new connection and a 3 step process to resume an idle session.

Step-1. Client Sends a Hello to the Server to initiate a Session which contains the SSL version number, random number and supported cipher suites
Step-2. Server Responds with Hello and its server certificate. If the Server certificate does not contain a public key then the Server key Exchange can contain a temporary key for the client to encrypt the master secret.
Step-3. The client now computes the master secret key using its random number and the random number received from the server hello. the client then encrypts the master secret with the Server’s public key located in the certificate. If there is not public key in the certificate then client uses the temporary key from server for the encryption. Client sends this back to Server so both parties now have the master  key which they use to derive symmetric key for further data encryption used by the record protocol
Step-4. Both Client and Server now send to each other a Change cipher Spec message and a Finish message. the record protocol from now on performs data encryption using the symmetric keys.

Below are Some common HTTP Return codes from the Server which indicate the status of the client -Server HTTP transaction, like any code returned with 4xx indicates the client error and any starting with 5xx indicate the error on the server side.

Some Common HTTP Return Codes

1xx :     Informational
100:     Continue
101:    Switching Protocols

2xx:     Successful
200:    OK
201:    Created
202:    Accepted
203:    Non-Authoritative Information
204:    No Content
205:    Reset Content
206:    Partial Content

3xx:    Redirection
300:    Multiple Choices
301:    Moved Permanently
302:    Found
303:    See Other
304:    Not Modified
305:    Use Proxy

4xx:    Client Error
400:    Bad Request
401:    Unauthorized
402:    Payment Required
403:    Forbidden
404:    Not Found
405:    Method Not Allowed
406:    Not Acceptable
407:    Proxy Authentication Required
408:    Request Timeout
409:    Conflict
410:    Gone
411:    Length Required
412:    Precondition Failed
413:    Request Entity Too Large
414:    Request URI Too Long
415:    Unsupported Media Type
416:    Requested Range Not Satisfiable
417:    Expectation Failed

5xx:    Server Error
500:    Internal Server Error
501:    Not Implemented
502:    Bad gateway
503:     Service Unavailable
504:    Gateway Timeout
505:    HTTP Version Not Supported