Closed Form Missing Value Imputation Question

Sorry if this is a trivial question, I haven't had any applicable formal education, but I've been looking into how to do missing value imputation, and I am surprised that most methods seem to use iterative methods that involve the whole data set each iteration. I was expecting the imputation of missing values to have a closed form solution for the linear model methods such as PCA.

I'm also not so much interested in imputing missing values of a fixed data set, instead I keep getting new online samples with some missing values, and would like to fill in the missing values (usually about 1% to 5% missing) while their real values haven't arrived yet. My dataset so far contains about 8 million samples of 1000 values (30GB), and all values have a mean of zero and standard deviation of one, and are highly correlated with at least a handful of other values.

Before I had read anything about imputation I assumed I could first build a covariance matrix estimate with my dataset (I'm OK with assuming the covariances are not changed by any missing data), and that after that I wouldn't need the data anymore, except for maybe a random sample of it that fits in memory to tune some hyper-parameters like the number of principal components to use in the case of PCA, or the noise and regularization for a marginalized denoising autoencoder.

I also expected that the calculation of missing values was generally done using a closed form solution as a function of the known values and the weights of the linear reconstruction model, and possibly a regularization term as well.

I was writing some code for handling the large amount of data to estimate the covariance matrix, and calculating a reconstruction matrix, but then I started reading to find the formula to do the imputation in closed form... but I couldn't find anything on that, just iterative methods.

Am I missing something here? Is this a bad or impossible way to approach the problem?

I would really like to know how to calculate the missing values from the known values and a weight matrix W (W = P*P^T , where P is a matrix of the first few principal components).

I think I would need to find the missing values of A that minimizes something like

(A * W - A)² + lambda * Amissing ²

where lambda is the regularization term, A is a vector of input values, and Amissing are only the originally missing values.

I suspect it's something like the differences between the reconstructed values and the input values, projected back to the missing values, a bit like back-propagation, and then multiplied by the inverse of some matrix. But I can't quite figure out the exact solution.

If anyone could help or suggest a better method, I would greatly appreciate it.

Thank you!