Direct Neural Network Training on Encrypted Datasets
Encryption of a CT image by sequential interlaced padding, random index shuffle, and random orthogonal matrix transformation.
Tensorlock is patent-protected technology for encrypting data in specific ways that allow for neural network training and inference directly on the encrypted data.
The Problem
The vast majority of the world's data is currently inaccessible for frontier model training.
Today's powerful AI models require massive amounts of data, but privacy requirements (think healthcare or business intelligence) and the need to protect intellectual property keep some of the most useful data locked away.
-
Data Owners want to protect the business value of their data, and they need a way to keep their data private during model training and inference.
-
AI Companies are reaching the limits of publicly accessible data and need to access the high-value private data that is currently hidden in silos.
Tensorlock brings these interests together by encrypting private data in specific ways that allow neural networks to be trained on the encrypted data itself, with no decryption step. Inference is done on encrypted examples, maintaining the business value of the underlying data.
How Tensorlock Works
Tensorlock redefines how datasets can be protected and processed in machine learning by enabling direct neural network training and inference on the encrypted data itself.
Let's walk through some of the Tensorlock encryption steps using a simple visual example from MNIST, the classic 'Hello World!' dataset for identification of handwritten numbers.
Starting with an example of a handwritten number 8, we add padding. This step increases the number of vector dimensions by adding noise in additional dimensions. Each training, test, and inference example receives different padding values in the same coordinate locations.
Padding
Padding can assign new dimensions at any position in the original vectors. Here's an example of interlaced padding, adding dimensions that are interlaced into the original vector dimensions. The positions of the new dimensions are invariant across examples, but every example has different values in these positions.
In the two examples above, we added random values in our new dimensions that were sampled from a uniform distribution. We can also perform sampled padding, drawing from the measured distribution of our original data to make the added noise more closely resemble the source data.
We can also use data-informed structured padding techniques like reference-derived structured padding, signal-dependent structured padding, or hybrid padding approaches. These techniques choose padding values on a per-example and per-coordinate basis, further obscuring the statistical difference between plaintext and padding data.
Perturbation
Next, we can perform perturbation, which adds noise in each vector dimension. With this image example, you can see fine gray noise (in this example, the pixel values are randomly altered within a possible range of -5% to +5%).
Now we apply a random index shuffle. The same randomly generated permutation is applied to the coordinates of our input vectors, causing all of our examples to be shuffled in the same way. In this example, we have 784 original vector dimensions + 280 padding dimensions = 1064 dimensions. The number of possible index shuffles is therefore 1064!, which is more than 10²⁷⁶⁰ possibilities.
Random Index Shuffle
Perturbation can alter the values using any function we choose, and we can apply a different function to each value. Approaches like structured non-linear perturbation allow for non-linear changes within user-specified bounds to dimension values across both plaintext and padding dimensions.
Random Orthogonal Matrix Transformation
Finally, we apply a random orthogonal matrix transformation. This example shows a random 4x4 orthogonal matrix transforming a 4-D vector. For our 1064-D examples above, the matrix is 1064x1064.
This orthogonal matrix transformation rotates all of our padded and perturbed vectors the same way in 1064-D space, shifting to a new randomly-chosen basis. This new basis is selected from an infinite choice of Haar-sampled orthogonal transformations.
For larger input vectors, we can break the vectors up into subvectors, generate random orthogonal matrices for each, and then concatenate the results. The whole process is an orthogonal transformation.
Aggregate Transformation
In this simple example, our aggregate process takes the original (28 x 28) = 784 dimensional plaintext examples to encrypted examples in 1064 dimensions.
Padding, perturbation, shuffling, and orthogonal matrix transformation can be done in any order, and more than once.
Each transformation step can have a number of variations.
Padding variations:
-
padding can be appended externally or in interlaced fashion
-
padding can be sampled from any distribution, including the distribution of values measured from the plaintext vectors
-
padding can be algorithmically chosen on a per-example and per-coordinate basis using the structure of the plaintext data to further obscure the statistical difference between plaintext and padding values
Perturbation variations:
-
perturbation can add noise by adjusting values across a range (e.g., -5% to +5%)
-
random perturbation can apply any function or functions to the values
-
edge overflow can be handled by clipping or reflection of values across the (0 - 1) float range
Here, transformed datasets are compared to the original control datasets, feeding each into the same deep neural network architecture. When each transformation type is applied alone, there is no difference in validation accuracy compared to control.
Training Performance
When we compare fully encrypted datasets (padding, perturbation, shuffling, and orthogonal matrix transformation) with control, the results of trials overlap and there is a very small average decrease in validation accuracy (in this case, a mean absolute decrease of 0.0051).
Across different training datasets, Tensorlock-encrypted data shows very small decreases in validation accuracy compared to plaintext controls.
There is no difference in model training time seen between encrypted datasets and controls.
Wait, how?
How does Tensorlock encryption allow a neural network to train on the encrypted data while also preventing an attacker from uncovering the original data?
This can seem counterintuitive at first.
The encryption steps intentionally leak the specific high-dimensional geometry between vectors that allows a neural network to associate vectors with labels. Vector-label associations present in the original data are thus preserved in the encrypted data. However, there is no information present in the encrypted data about the encryption process that was performed, so an attacker is forced to guess how to reverse the process from across an enormous keyspace.
Orthogonal transformations preserve high-dimensional vector geometry
-
Angles between vectors stay the same
-
Distances between vectors stay the same
-
Only the coordinate system changes
In order to reverse the encryption, an attacker need to determine, among other things:
-
the precise coordinate system change (from an infinite choice of possible changes)
-
exactly which original dimensions stored random padding data prior to the coordinate system change, despite not knowing how many dimensions the data had prior to padding