An echo state network with working memories for probabilistic language modeling

Yukinori Homma; Masafumi Hagiwara

doi:10.1007/978-3-642-40728-4_74

An echo state network with working memories for probabilistic language modeling

Yukinori Homma, Masafumi Hagiwara

Department of Information and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

Abstract

In this paper, we propose an ESN having multiple timescale layer and working memories as a probabilistic language model. The reservoir of the proposed model is composed of three neuron groups each with an associated time constant, which enables the model to learn the hierarchical structure of language. We add working memories to enhance the effect of multiple timescale layers. As shown by the experiments, the proposed model can be trained efficiently and accurately to predict the next word from given words. In addition, we found that use of working memories is especially effective in learning grammatical structure.

Original language	English
Title of host publication	Artificial Neural Networks and Machine Learning, ICANN 2013 - 23rd International Conference on Artificial Neural Networks, Proceedings
Pages	595-602
Number of pages	8
DOIs	https://doi.org/10.1007/978-3-642-40728-4_74
Publication status	Published - 2013
Event	23rd International Conference on Artificial Neural Networks, ICANN 2013 - Sofia, Bulgaria Duration: 2013 Sept 10 → 2013 Sept 13

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	8131 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Other

Other	23rd International Conference on Artificial Neural Networks, ICANN 2013
Country/Territory	Bulgaria
City	Sofia
Period	13/9/10 → 13/9/13

Keywords

ESNs
Probabilistic language model
working memory

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-642-40728-4_74

Cite this

Homma, Y., & Hagiwara, M. (2013). An echo state network with working memories for probabilistic language modeling. In Artificial Neural Networks and Machine Learning, ICANN 2013 - 23rd International Conference on Artificial Neural Networks, Proceedings (pp. 595-602). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8131 LNCS). https://doi.org/10.1007/978-3-642-40728-4_74

An echo state network with working memories for probabilistic language modeling. / Homma, Yukinori; Hagiwara, Masafumi.
Artificial Neural Networks and Machine Learning, ICANN 2013 - 23rd International Conference on Artificial Neural Networks, Proceedings. 2013. p. 595-602 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8131 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Homma, Y & Hagiwara, M 2013, An echo state network with working memories for probabilistic language modeling. in Artificial Neural Networks and Machine Learning, ICANN 2013 - 23rd International Conference on Artificial Neural Networks, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8131 LNCS, pp. 595-602, 23rd International Conference on Artificial Neural Networks, ICANN 2013, Sofia, Bulgaria, 13/9/10. https://doi.org/10.1007/978-3-642-40728-4_74

Homma Y, Hagiwara M. An echo state network with working memories for probabilistic language modeling. In Artificial Neural Networks and Machine Learning, ICANN 2013 - 23rd International Conference on Artificial Neural Networks, Proceedings. 2013. p. 595-602. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-642-40728-4_74

Homma, Yukinori ; Hagiwara, Masafumi. / An echo state network with working memories for probabilistic language modeling. Artificial Neural Networks and Machine Learning, ICANN 2013 - 23rd International Conference on Artificial Neural Networks, Proceedings. 2013. pp. 595-602 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{da209b23de8c4947b65a5ce855cf9bee,

title = "An echo state network with working memories for probabilistic language modeling",

abstract = "In this paper, we propose an ESN having multiple timescale layer and working memories as a probabilistic language model. The reservoir of the proposed model is composed of three neuron groups each with an associated time constant, which enables the model to learn the hierarchical structure of language. We add working memories to enhance the effect of multiple timescale layers. As shown by the experiments, the proposed model can be trained efficiently and accurately to predict the next word from given words. In addition, we found that use of working memories is especially effective in learning grammatical structure.",

keywords = "ESNs, Probabilistic language model, working memory",

author = "Yukinori Homma and Masafumi Hagiwara",

year = "2013",

doi = "10.1007/978-3-642-40728-4_74",

language = "English",

isbn = "9783642407277",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

pages = "595--602",

booktitle = "Artificial Neural Networks and Machine Learning, ICANN 2013 - 23rd International Conference on Artificial Neural Networks, Proceedings",

note = "23rd International Conference on Artificial Neural Networks, ICANN 2013 ; Conference date: 10-09-2013 Through 13-09-2013",

}

TY - GEN

T1 - An echo state network with working memories for probabilistic language modeling

AU - Homma, Yukinori

AU - Hagiwara, Masafumi

PY - 2013

Y1 - 2013

N2 - In this paper, we propose an ESN having multiple timescale layer and working memories as a probabilistic language model. The reservoir of the proposed model is composed of three neuron groups each with an associated time constant, which enables the model to learn the hierarchical structure of language. We add working memories to enhance the effect of multiple timescale layers. As shown by the experiments, the proposed model can be trained efficiently and accurately to predict the next word from given words. In addition, we found that use of working memories is especially effective in learning grammatical structure.

AB - In this paper, we propose an ESN having multiple timescale layer and working memories as a probabilistic language model. The reservoir of the proposed model is composed of three neuron groups each with an associated time constant, which enables the model to learn the hierarchical structure of language. We add working memories to enhance the effect of multiple timescale layers. As shown by the experiments, the proposed model can be trained efficiently and accurately to predict the next word from given words. In addition, we found that use of working memories is especially effective in learning grammatical structure.

KW - ESNs

KW - Probabilistic language model

KW - working memory

UR - http://www.scopus.com/inward/record.url?scp=84884938773&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84884938773&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-40728-4_74

DO - 10.1007/978-3-642-40728-4_74

M3 - Conference contribution

AN - SCOPUS:84884938773

SN - 9783642407277

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 595

EP - 602

BT - Artificial Neural Networks and Machine Learning, ICANN 2013 - 23rd International Conference on Artificial Neural Networks, Proceedings

T2 - 23rd International Conference on Artificial Neural Networks, ICANN 2013

Y2 - 10 September 2013 through 13 September 2013

ER -

An echo state network with working memories for probabilistic language modeling

Abstract

Publication series

Other

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this