Advances in Extreme Learning Machines

Loading...
Thumbnail Image
Journal Title
Journal ISSN
Volume Title
School of Science | Doctoral thesis (article-based) | Defence date: 2015-04-17
Checking the digitized thesis and permission for publishing
Instructions for the author
Date
2015
Department
Tietojenkäsittelytieteen laitos
Department of Information and Computer Science
Major/Subject
Mcode
Degree programme
Language
en
Pages
108 + app. 84
Series
Aalto University publication series DOCTORAL DISSERTATIONS, 43/2015
Abstract
Nowadays, due to advances in technology, data is generated at an incredible pace, resulting in large data sets of ever-increasing size and dimensionality. Therefore, it is important to have efficient computational methods and machine learning algorithms that can handle such large data sets, such that they may be analyzed in reasonable time. One particular approach that has gained popularity in recent years is the Extreme Learning Machine (ELM), which is the name given to neural networks that employ randomization in their hidden layer, and that can be trained efficiently. This dissertation introduces several machine learning methods based on Extreme Learning Machines (ELMs) aimed at dealing with the challenges that modern data sets pose. The contributions follow three main directions.    Firstly, ensemble approaches based on ELM are developed, which adapt to context and can scale to large data. Due to their stochastic nature, different ELMs tend to make different mistakes when modeling data. This independence of their errors makes them good candidates for combining them in an ensemble model, which averages out these errors and results in a more accurate model. Adaptivity to a changing environment is introduced by adapting the linear combination of the models based on accuracy of the individual models over time. Scalability is achieved by exploiting the modularity of the ensemble model, and evaluating the models in parallel on multiple processor cores and graphics processor units. Secondly, the dissertation develops variable selection approaches based on ELM and Delta Test, that result in more accurate and efficient models. Scalability of variable selection using Delta Test is again achieved by accelerating it on GPU. Furthermore, a new variable selection method based on ELM is introduced, and shown to be a competitive alternative to other variable selection methods. Besides explicit variable selection methods, also a new weight scheme based on binary/ternary weights is developed for ELM. This weight scheme is shown to perform implicit variable selection, and results in increased robustness and accuracy at no increase in computational cost. Finally, the dissertation develops training algorithms for ELM that allow for a flexible trade-off between accuracy and computational time. The Compressive ELM is introduced, which allows for training the ELM in a reduced feature space. By selecting the dimension of the feature space, the practitioner can trade off accuracy for speed as required.    Overall, the resulting collection of proposed methods provides an efficient, accurate and flexible framework for solving large-scale supervised learning problems. The proposed methods are not limited to the particular types of ELMs and contexts in which they have been tested, and can easily be incorporated in new contexts and models.
Description
Supervising professor
Oja, Erkki, Aalto Distinguished Prof., Aalto University, Department of Information and Computer Science, Finland
Thesis advisor
Miche, Yoan, Dr., Aalto University, Department of Information and Computer Science, Finland
Keywords
Extreme Learning Machine (ELM), high-performance computing, ensemble models, variable selection, random projection, machine learning
Other note
Parts
  • [Publication 1]: Mark van Heeswijk, Yoan Miche, Tiina Lindh-Knuutila, Peter A.J. Hilbers, Timo Honkela, Erkki Oja, and Amaury Lendasse. Adaptive Ensemble Models of Extreme Learning Machines for Time Series Prediction. In LNCS 5769 - Artificial Neural Networks, ICANN’09: International Conference on Artificial Neural Networks, pp. 305-314, September 2009. doi:10.1007/978-3-642-04277-5_31.
  • [Publication 2]: Mark van Heeswijk, Yoan Miche, Erkki Oja, and Amaury Lendasse. GPU-accelerated and parallelized ELM ensembles for large-scale regression. Neurocomputing, 74 (16): pp. 2430-2437, September 2011. doi:10.1016/j.neucom.2010.11.034.
  • [Publication 3]: Benoît Frenay, Mark van Heeswijk, Yoan Miche, Michel Verleysen, and Amaury Lendasse. Feature selection for nonlinear models with extreme learning machines. Neurocomputing, 102, pp. 111-124, February 2013. doi:10.1016/j.neucom.2011.12.055.
  • [Publication 4]: Alberto Guillén, Maribel García Arenas, Mark van Heeswijk, Dušan Sovilj, Amaury Lendasse, Luis Herrera, Hector Pomares and Ignacio Rojas. Fast Feature Selection in a GPU Cluster Using the Delta Test. Entropy, 16 (2): pp. 854-869, 2014. doi:10.3390/e16020854.
  • [Publication 5]: Mark van Heeswijk, and Yoan Miche. Binary/Ternary Extreme Learning Machines. Neurocomputing, 149, pp. 187-197, February 2015. doi:10.1016/j.neucom.2014.01.072.
  • [Publication 6]: Mark van Heeswijk, Amaury Lendasse, and Yoan Miche. Compressive ELM: Improved Models Through Exploiting Time-Accuracy Trade-offs. In CCIS 459 - Engineering Applications of Neural Networks, pp. 165-174, 2014. doi:10.1007/978-3-319-11071-4_16.
Citation