Auto-encoder with Adversarially Regularized Latent Variables for Semi-Supervised Learning
Abstract
The amount of accessible raw data is ever-increasing in spite of the difficulty in obtaining a variety of labeled information; this makes semi-supervised learning a topic of practical importance. This paper proposes a novel regularization algorithm of an autoencoding deep neural network for semi-supervised learning. Given an input data, the deep neural network outputs the estimated label, and the remaining information called style. On the basis of the framework of a generative adversarial network, the proposed algorithm regularizes the label and the style according to a prior distribution, separating the label explicitly from the style. As a result, the deep neural network is trained to estimate correct labels by using a limitedly labeled dataset. The proposed algorithm achieved accuracy comparable with or superior to that of the existing state-of-the-art semi-supervised algorithms for the benchmark tasks, the MNIST database, and the SVHN dataset.
References
J. Lasserre, C. Bishop, and T. Minka, “Principled Hybrids of Generative and Discrim- inative Models,” in Proc. of 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, no. 6. IEEE, 2006, pp. 87–94.
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.
J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural Networks, vol. 61, pp. 85–117, 2015.
G. E. Hinton and R. R. Salakhutdinov, “Reducing the Dimensionality of Data with Neural Networks,” Science, vol. 313, no. 5786, pp. 504–507, 2006.
M.RanzatoandM.Szummer,“Semi-SupervisedLearningofCompactDocumentRep- resentations with Deep Networks,” Proc. of the 25rd International Conference on Ma- chine Learning, pp. 792–799, 2008.
X.ZhaoandP.a.Robinson,“Generalizedseizuresinaneuralfieldmodelwithbursting dynamics,” Journal of Computational Neuroscience, pp. 197–216, 2015.
A. Rasmus, H. Valpola, M. Honkala, M. Berglund, and T.Raiko, “Semi-Supervised Learning with Ladder Network,” Advances in Neural Information Processing Systems, pp. 3546–3554, 2015.
D. P. Kingma, D. J. Rezende, and M. Welling, “Semi-supervised Learning with Deep Generative Models,” Advances in Neural Information Processing Systems, pp. 3581– 3589, 2014.
L. Maaløe, C. K. Sønderby, S. K. Sønderby, and O. Winther, “Auxiliary Deep Genera- tive Models,” Proc. of the 33rd International Conference on Machine Learning, vol. 48, pp. 1–5, 2015.
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Networks,” Advances in Neural Information Processing Systems, pp. 2672–2680, 2014.
A. Radford, L. Metz, and S. Chintala, “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks,” International Conference on Learning Representations, pp. 1–16, 2016.
J. T. Springenberg, “Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks,” arXiv:1511.06390, 2015.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Im- proved Techniques for Training GANs,” arXiv:1606.03498, 2016.
V. Dumoulin, I. Belghazi, B. Poole, A. Lamb, M. Arjovsky, O. Mastropietro, and A.Courville, “Adversarially Learned Inference,” arXiv:1606.00704, 2016.
S. Rifai, Y. N. Dauphin, P. Vincent, Y. Bengio, and X. Muller, “The Manifold Tangent Classifier,” Advances in Neural Information Processing Systems, pp. 2294–2302, 2011.
T. Miyato, S. Maeda, M. Koyama, K. Nakae, and S. Ishii, “Distributional Smoothing with Virtual Adversarial Training,” International Conference on Learning Representa- tions, pp. 1–18, 2015.
D.P.KingmaandM.Welling,“Auto-EncodingVariationalBayes,”InternationalCon- ference on Learning Representations, pp. 1–14, 2014.
S. Nowozin, B. Cseke, and R. Tomioka, “f-GAN: Training Generative Neural Sam- plers using Variational Divergence Minimization,” arXiv:1606.00709, 2016.
M. Arjovsky and L. Bottou, “Towards Principled Methods for Training Generative Adversarial Networks,” Workshop on Adversarial Training, Advances in Neural Infor- mation Processing Systems, pp. 1–16, 2017.
A.Makhzani,J.Shlens,N.Jaitly,I.Goodfellow,B.Frey,“AdversarialAutoencoders,” arXiv:1511.05644, 2015.
R. Tachibana, T. Matsubara, and K. Uehara, “Semi-Supervised Learning Using Ad- versarial Networks,” in Proc. of the 15th IEEE/ACIS International Conference on Com- puter and Information Science, pp. 939–944, 2016
A. B. L. Larsen, S. K. Sønderby, and O. Winther, “Autoencoding beyond pixels using a learned similarity metric,” International Conference on Machine Learning, 2016.
“THE MNIST DATABASE of handwritten digits.” url: http://yann.lecun.com/ exdb/mnist/
“The Street View House Numbers (SVHN) Dataset.” url: http://ufldl. stanford.edu/housenumbers/
“Deep MNIST for Experts,” in TensorFlow Turorials. url: https://www. tensorflow.org/tutorials/mnist/pros/
D. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” International Conference on Learning Representations, 2015.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, and Zhen Wang, “Least Squares Generative Adversarial Networks,” arXiv:1611.04076, 2017.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein, “Unrolled Generagive Adversarial Networks,” International Conference on Learning Representations, 2017.
