A proposal for mixture of experts with entropic regularization

Billy Peralta, Ariel Saavedra, Luis Caro

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In these days, there are a growing interest in pattern recognition for tasks as prediction of weather events, recommendation of the best route, intrusion detection or face detection. Each of these tasks can be modelled as classification problem, where a common alternative is to use an ensemble model of classification. A well-known example is given by Mixture-of-Experts model, which represents a probabilistic artificial neural network consisting of local experts classifiers weighted by a gate network, and whose combination creates an environment of competition among experts seeking to obtain patterns of the data source. We observe that this architecture assume that one gate influence only one data point, consequently the training can be misguided in real datasets where the data is better explained by multiple experts. In this work, we present a variant of regular Mixture-of-Experts model, which consists of maximizing of the entropy of gate network in addition to classification cost minimization. The results show the advantage of our approach in multiple datasets in terms of accuracy metric. As a future work, we plan to apply this idea to the Mixture-of-Experts with embedded feature selection.

Original languageEnglish
Title of host publication2017 43rd Latin American Computer Conference, CLEI 2017
EditorsRodrigo Santos, Hector Monteverde
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1-9
Number of pages9
Volume2017-January
ISBN (Electronic)9781538630570
DOIs
Publication statusPublished - 18 Dec 2017
Event43rd Latin American Computer Conference, CLEI 2017 - Cordoba, Argentina
Duration: 4 Sep 20178 Sep 2017

Conference

Conference43rd Latin American Computer Conference, CLEI 2017
CountryArgentina
CityCordoba
Period4/09/178/09/17

Fingerprint

expert
Intrusion detection
Face recognition
Pattern recognition
Feature extraction
Classifiers
Entropy
Neural networks
pattern recognition
entropy
neural network
Costs
event
costs

Keywords

  • classification
  • mixture-of-experts
  • regularization

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Software
  • Education

Cite this

Peralta, B., Saavedra, A., & Caro, L. (2017). A proposal for mixture of experts with entropic regularization. In R. Santos, & H. Monteverde (Eds.), 2017 43rd Latin American Computer Conference, CLEI 2017 (Vol. 2017-January, pp. 1-9). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CLEI.2017.8226425
Peralta, Billy ; Saavedra, Ariel ; Caro, Luis. / A proposal for mixture of experts with entropic regularization. 2017 43rd Latin American Computer Conference, CLEI 2017. editor / Rodrigo Santos ; Hector Monteverde. Vol. 2017-January Institute of Electrical and Electronics Engineers Inc., 2017. pp. 1-9
@inproceedings{070783b05d4c4befaac1957465afa4f0,
title = "A proposal for mixture of experts with entropic regularization",
abstract = "In these days, there are a growing interest in pattern recognition for tasks as prediction of weather events, recommendation of the best route, intrusion detection or face detection. Each of these tasks can be modelled as classification problem, where a common alternative is to use an ensemble model of classification. A well-known example is given by Mixture-of-Experts model, which represents a probabilistic artificial neural network consisting of local experts classifiers weighted by a gate network, and whose combination creates an environment of competition among experts seeking to obtain patterns of the data source. We observe that this architecture assume that one gate influence only one data point, consequently the training can be misguided in real datasets where the data is better explained by multiple experts. In this work, we present a variant of regular Mixture-of-Experts model, which consists of maximizing of the entropy of gate network in addition to classification cost minimization. The results show the advantage of our approach in multiple datasets in terms of accuracy metric. As a future work, we plan to apply this idea to the Mixture-of-Experts with embedded feature selection.",
keywords = "classification, mixture-of-experts, regularization",
author = "Billy Peralta and Ariel Saavedra and Luis Caro",
year = "2017",
month = "12",
day = "18",
doi = "10.1109/CLEI.2017.8226425",
language = "English",
volume = "2017-January",
pages = "1--9",
editor = "Rodrigo Santos and Hector Monteverde",
booktitle = "2017 43rd Latin American Computer Conference, CLEI 2017",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
address = "United States",

}

Peralta, B, Saavedra, A & Caro, L 2017, A proposal for mixture of experts with entropic regularization. in R Santos & H Monteverde (eds), 2017 43rd Latin American Computer Conference, CLEI 2017. vol. 2017-January, Institute of Electrical and Electronics Engineers Inc., pp. 1-9, 43rd Latin American Computer Conference, CLEI 2017, Cordoba, Argentina, 4/09/17. https://doi.org/10.1109/CLEI.2017.8226425

A proposal for mixture of experts with entropic regularization. / Peralta, Billy; Saavedra, Ariel; Caro, Luis.

2017 43rd Latin American Computer Conference, CLEI 2017. ed. / Rodrigo Santos; Hector Monteverde. Vol. 2017-January Institute of Electrical and Electronics Engineers Inc., 2017. p. 1-9.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - A proposal for mixture of experts with entropic regularization

AU - Peralta, Billy

AU - Saavedra, Ariel

AU - Caro, Luis

PY - 2017/12/18

Y1 - 2017/12/18

N2 - In these days, there are a growing interest in pattern recognition for tasks as prediction of weather events, recommendation of the best route, intrusion detection or face detection. Each of these tasks can be modelled as classification problem, where a common alternative is to use an ensemble model of classification. A well-known example is given by Mixture-of-Experts model, which represents a probabilistic artificial neural network consisting of local experts classifiers weighted by a gate network, and whose combination creates an environment of competition among experts seeking to obtain patterns of the data source. We observe that this architecture assume that one gate influence only one data point, consequently the training can be misguided in real datasets where the data is better explained by multiple experts. In this work, we present a variant of regular Mixture-of-Experts model, which consists of maximizing of the entropy of gate network in addition to classification cost minimization. The results show the advantage of our approach in multiple datasets in terms of accuracy metric. As a future work, we plan to apply this idea to the Mixture-of-Experts with embedded feature selection.

AB - In these days, there are a growing interest in pattern recognition for tasks as prediction of weather events, recommendation of the best route, intrusion detection or face detection. Each of these tasks can be modelled as classification problem, where a common alternative is to use an ensemble model of classification. A well-known example is given by Mixture-of-Experts model, which represents a probabilistic artificial neural network consisting of local experts classifiers weighted by a gate network, and whose combination creates an environment of competition among experts seeking to obtain patterns of the data source. We observe that this architecture assume that one gate influence only one data point, consequently the training can be misguided in real datasets where the data is better explained by multiple experts. In this work, we present a variant of regular Mixture-of-Experts model, which consists of maximizing of the entropy of gate network in addition to classification cost minimization. The results show the advantage of our approach in multiple datasets in terms of accuracy metric. As a future work, we plan to apply this idea to the Mixture-of-Experts with embedded feature selection.

KW - classification

KW - mixture-of-experts

KW - regularization

UR - http://www.scopus.com/inward/record.url?scp=85046435373&partnerID=8YFLogxK

U2 - 10.1109/CLEI.2017.8226425

DO - 10.1109/CLEI.2017.8226425

M3 - Conference contribution

VL - 2017-January

SP - 1

EP - 9

BT - 2017 43rd Latin American Computer Conference, CLEI 2017

A2 - Santos, Rodrigo

A2 - Monteverde, Hector

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Peralta B, Saavedra A, Caro L. A proposal for mixture of experts with entropic regularization. In Santos R, Monteverde H, editors, 2017 43rd Latin American Computer Conference, CLEI 2017. Vol. 2017-January. Institute of Electrical and Electronics Engineers Inc. 2017. p. 1-9 https://doi.org/10.1109/CLEI.2017.8226425