Beyond TLS: custom security protocols

The purpose of this article is to provide a useful starting point for software engineers who wish to implement secure network communication in their products, but are fed up of the subtle pitfalls and irrelevant implementation details associated with TLS and PKI, and for whom interoperability with other vendors' products is not a major concern. It may also be of interest to those wishing to understand the inner workings of modern cryptosystems.

Protocol description

This hybrid protocol features mutual authentication, perfect forward secrecy, anti-replay and anti-downgrade protection. It should be secure against both classical and quantum attacks.

Prerequisites

Prior to session establishment, the client and server must each generate ed25519 and SPHINCS+-SHAKE-128s-simple keypairs, then exchange public keys.

Client hello

The client first generates two ephemeral keypairs (X25519 and SNTRU-Prime761) and a random 192-bit nonce. The ephemeral public keys are concatenated with the nonce and the result is signed twice (once with ed25519, then again with SPHINCS+-SHAKE-128s-simple). These signatures are appended onto the end of the message.

Server hello

The server verifies the two signatures, then generates its own ephemeral X25519 keypair and a second random 192-bit nonce. The server encapsulates a secret using the client's SNTRU-Prime761 public key, then concatenates the result with its own ephemeral X25519 public key and the second nonce. The server signs this twice (in the same way as the client hello) and appends these signatures to the message.

Session key derivation

The client verifies the server's signatures, then de-encapsulates the shared PQC secret. Both sides perform ECDH to derive the shared classical secret. Both random nonces are concatenated with the two secrets; BLAKE2b is used to derive the final session key.

Session format

Each message is encrypted using the XChaCha20-Poly1305 construction, alongside a sequence number which forms part of the authenticated data segment and is incremented with each message sent. The receiver keeps an internal counter for checking this value. When it is about to overflow, the key is rotated using BLAKE2b and the sequence number is reset.

Algorithms

ed25519: Classical signature scheme based on Curve25519. Generally faster than RSA at equivalent security levels and produces shorter signatures.
X25519: Diffie-Hellman with Curve25519.
SPHINCS+-SHAKE-128s-simple: PQC signature scheme based on SHA-3.
SNTRU-Prime761: Lattice-based KEM with reasonable performance and fewer security issues than its competitors.
BLAKE2b: Hashing algorithm with good performance and resistance to length extension attacks.
XChaCha20: Symmetric encryption algorithm with good performance and resistance to side-channel attacks. The extended nonce length reduces the risk of nonce reuse.
Poly1305: Hashing algorithm for computing MACs. Extends XChaCha20 to produce an AEAD scheme.

Practical considerations

You may wish to consider the use of ergonomic networking libraries like ZeroMQ. Some (most?) applications may benefit from plaintext compression prior to encryption. The gold standard for this is currently zstd, although there are plenty of suitable alternatives.