Variational Inference algorithm

Sampling data or approximating probabilistic densities is one of the core problems in modern statistics, especially in Bayesian statistic. Beside of MCMC, Variational Inference is one of the two typical Bayesian approaches solving this problem. It leverages the Variational Inference algorithm to develop many recent powerful methods and models: Stochastic Variational Inference, Variational Autoencoder... This…

Jensen’s inequality and applications: EM Algorithm and ELBO in SVI

Accidentally, I learned the ELBO of SVI and realised some interesting things. SVI transforms its optimization problem from finding parameters bringing the minimal error to iteratively maximizing lower bound values. Moreover, it is a big step that allows SVI to use gradient descent then to integrate with neural network models. Iteratively finding distributions which have…