We throw massive amounts of data at machine learning models every day. Most of this data is highly sensitive. If you run a medical startup, you want to use AI to analyze patient scans. If you run a fintech app, you want to analyze transaction logs to flag fraud.
But sending raw data to a cloud provider means decrypting it at some point. Even if you encrypt the data in transit using TLS and encrypt it at rest using AES-256, the data must be decrypted in the memory of the server doing the inference.
This is the decryption gap. If an attacker gains root access to that server, or if the cloud provider has a rogue employee, your data is exposed. For industries with strict compliance rules, especially under new regulations like California's DROP data deletion requirements, this gap makes cloud-based AI a non-starter.
Homomorphic encryption changes this model. It allows a server to perform mathematical operations on encrypted data without decrypting it first. The server receives encrypted input, runs it through a model, and outputs an encrypted result. The server never sees the raw data, and it never knows what the data represents. Only the client, who holds the private key, can decrypt the final output.
The Math Behind the Magic
In a standard encryption scheme like AES, if you add two ciphertexts together, you get random noise. In homomorphic encryption, the mathematics are designed so that operations on ciphertexts map directly to operations on plaintexts.
Let E(x) be the encryption of x. A homomorphic encryption scheme allows you to calculate E(x + y) directly from E(x) and E(y) without knowing x or y.
We classify these schemes based on the operations they support:
- Partially Homomorphic Encryption (PHE): Supports only one type of operation, either addition or multiplication, an infinite number of times. The Paillier cryptosystem allows you to add encrypted numbers, while the RSA cryptosystem allows multiplication.
- Somewhat Homomorphic Encryption (SHE): Allows both addition and multiplication, but only a limited number of times.
- Fully Homomorphic Encryption (FHE): Allows an infinite number of additions and multiplications.
FHE is what we need for neural networks, but it comes with a catch. Every time you perform a multiplication on ciphertext, you add a small amount of mathematical noise to the result. If the noise grows too large, the ciphertext becomes corrupted, and decryption fails.
To prevent this, FHE uses a process called bootstrapping. Bootstrapping runs the decryption circuit homomorphically using an encrypted version of the private key. This resets the noise level back to a manageable state. The problem is that bootstrapping is computationally expensive. It used to take minutes for a single operation. Today, we have brought that down to milliseconds, but it remains the primary speed bottleneck.
Mapping Neural Networks to Encrypted Space
To run a neural network on encrypted data, we have to translate the network's operations into homomorphic operations. A standard neural network consists of two main types of layers: linear layers and non-linear layers.
Linear layers, such as matrix multiplications, convolutions, and additions, are straightforward. Because FHE natively supports addition and multiplication, we can multiply encrypted inputs by unencrypted model weights quite efficiently.
The real challenge comes with non-linear activation functions. Functions like ReLU, Sigmoid, and GeLU are not simple polynomials. ReLU, for example, is defined as f(x) = max(0, x). FHE cannot easily evaluate a conditional branch like "if x is greater than zero" because the server does not know if the encrypted value is positive or negative.
To solve this, we use polynomial approximation. We replace ReLU or Sigmoid with a polynomial that behaves similarly over a specific range. For instance, we might approximate a sigmoid function using a polynomial like 0.125 * x^2 + 0.5 * x + 0.25.
The higher the degree of the polynomial, the more accurate the approximation. However, higher-degree polynomials require more multiplications. More multiplications mean more noise, which triggers more frequent bootstrapping. Building a practical private AI system is a constant trade-off between model accuracy and inference speed.
Building Encrypted Inference with Concrete-ML
You do not have to write lattice-based cryptography from scratch to build private AI. Modern libraries handle the compilation and encryption details. One popular tool is Concrete-ML, built on top of the TFHE (Torus Fully Homomorphic Encryption) scheme.
The typical developer workflow looks like this:
- Train your model in PyTorch or scikit-learn using normal plaintext data.
- Quantize the model. FHE schemes work best with integers rather than floating-point numbers. We need to convert our 32-bit floats into 8-bit or 16-bit integers.
- Compile the model into an FHE-compatible representation. The compiler analyzes the execution graph, optimizes the polynomial approximations, and determines where bootstrapping is needed.
- Deploy the compiled model to the untrusted server.
- The client encrypts their input data, sends it to the server, receives the encrypted prediction, and decrypts it locally.
Here is a simplified example of how you would compile a scikit-learn model for encrypted inference using Concrete-ML:
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from concrete.ml.sklearn import LogisticRegression
# Generate dummy data
X, y = make_classification(n_samples=1000, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Train a homomorphic logistic regression model
# The model is trained on plaintext data but prepared for quantization
model = LogisticRegression(n_bits=8)
model.fit(X_train, y_train)
# Compile the model to FHE
# This step determines the cryptographic parameters based on the input distribution
model.compile(X_train)
# Run inference on encrypted data
# Under the hood: input is encrypted, processed, and decrypted
encrypted_predictions = model.predict(X_test, fhe="execute")When you call model.predict with fhe="execute", the library simulates the client-server boundary. The input X_test is encrypted using the client's public key. The server runs the logistic regression model on the ciphertext. The server returns the encrypted result, which the client decrypts.
The Performance Bottleneck and Hardware Acceleration
If you run the code above, you will notice that encrypted inference is slower than standard inference. FHE is slow because of the size of the data. A single 32-bit float, once encrypted into a ciphertext, can balloon to tens of kilobytes or even megabytes.
This data expansion creates a massive memory bandwidth bottleneck. CPUs spend most of their time moving large ciphertexts in and out of cache rather than performing calculations.
Hardware acceleration is the key to making FHE practical. Several hardware startups and chip manufacturers are building custom silicon to speed up homomorphic operations, reflecting a broader industry shift toward etching models in silicon for specialized workloads:
- ASICs for FHE: Companies like Niobium and Intel are designing application-specific integrated circuits (ASICs) optimized for polynomial arithmetic and Number Theoretic Transforms (NTT), which are the core mathematical operations in FHE.
- Optical Computing: Startups like Optalysys use light instead of electricity to perform Fourier-like transforms, which can speed up the multiplication of massive polynomials.
- GPU Acceleration: Libraries like cuFHE leverage NVIDIA GPUs to parallelize ciphertext operations, providing a bridge until dedicated FHE chips become widely available.
With hardware acceleration, we can expect a 100x to 1000x speedup. This will bring encrypted inference times for medium-sized neural networks down from seconds to milliseconds.
Security Considerations Beyond Encryption
While homomorphic encryption protects data privacy during processing, it does not solve every security problem in machine learning.
One major risk is model extraction. If an untrusted client can query your encrypted model an infinite number of times, they can analyze the inputs and outputs to reconstruct the model's weights. FHE protects the client's data from the server, but it does not automatically protect the server's model from a malicious client.
There is also the threat of side-channel attacks. If a server takes longer to process certain encrypted inputs than others, an attacker might infer information about the data by measuring execution times. Developers must ensure that homomorphic operations run in constant time to prevent this leak.
While homomorphic encryption protects data privacy during processing, it does not solve every security problem in machine learning. Applications must still implement strategies for hardening AI agent gateways against prompt injection and other input-based attacks.
Hybrid Architectures
Because FHE is computationally heavy, production systems often combine it with other privacy-enhancing technologies (PETs).
Secure Multi-Party Computation (SMPC)
SMPC allows multiple parties to jointly compute a function over their inputs while keeping those inputs private. Instead of one server doing all the work on encrypted data, two or more servers split the computation. SMPC is faster than FHE for certain operations but requires low-latency network connections between the servers.
Trusted Execution Environments (TEEs)
TEEs, like Intel SGX or AWS Nitro Enclaves, provide secure hardware partitions on the CPU. The data is decrypted and processed inside a secure enclave, which prevents the host operating system or hypervisor from reading it. TEEs run at near-native speeds, making them much faster than FHE.
However, TEEs rely on trusting the hardware manufacturer. If a vulnerability is found in the CPU's hardware design, the security of the enclave collapses. FHE relies entirely on mathematical proofs, meaning its security holds even if the underlying hardware is compromised.
A hybrid system might use TEEs for the heavy lifting of neural network inference and use FHE for secure key management or pre-processing steps.
What Can We Deploy Today?
We are not yet at the point where we can run large language models like GPT-4 entirely inside FHE. Running a model with billions of parameters would require petabytes of memory and take hours for a single token.
While developers can achieve client-side execution by running local models in the browser, running a model with billions of parameters homomorphically would require petabytes of memory and take hours for a single token.
However, FHE is ready for smaller, specialized models today:
- Tabular Classifiers: Credit scoring, fraud detection, and risk assessment models can run on encrypted data in milliseconds.
- Medical Diagnostics: Simple convolutional neural networks (CNNs) can analyze chest X-rays or skin scans without exposing patient identities.
- Biometric Matching: Fingerprint or facial recognition models can verify a user's identity on a remote server without storing the raw biometric template in the cloud.
As tooling improves and hardware accelerators hit the market, the performance gap will continue to shrink. Homomorphic encryption is moving from a theoretical branch of cryptography to a practical tool for building secure, private AI systems.
