Multi-modal action segmentation in the kitchen with a feature fusion approach

Shunsuke Kogure; Yoshimitsu Aoki

doi:10.1117/12.2591752

Multi-modal action segmentation in the kitchen with a feature fusion approach

Shunsuke Kogure, Yoshimitsu Aoki

Department of Electronics and Electrical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper, we propose a "Multi-modal Action Segmentation approach"that uses three modalities: (i) video, (ii) audio, (iii) thermal to classify cooking behavior in the kitchen. These 3 modalities are assumed to be features related to cooking. However, there is no public dataset containing these three modalities. Therefore, we built the original dataset and frame-level annotation. We then examined the usefulness of Action Segmentation using multi-modal features. We analyzed the effects of each modality using three evaluation metrics. As a result, the accuracy, edit distance, and F1 value were improved by up to about 1%, 2%, and 8%, respectively, compared to the case when only images were used.

Original language	English
Title of host publication	Fifteenth International Conference on Quality Control by Artificial Vision
Editors	Kenji Terada, Akio Nakamura, Takashi Komuro, Tsuyoshi Shimizu
Publisher	SPIE
ISBN (Electronic)	9781510644267
DOIs	https://doi.org/10.1117/12.2591752
Publication status	Published - 2021
Event	15th International Conference on Quality Control by Artificial Vision - Tokushima, Virtual, Japan Duration: 2021 May 12 → 2021 May 14

Publication series

Name	Proceedings of SPIE - The International Society for Optical Engineering
Volume	11794
ISSN (Print)	0277-786X
ISSN (Electronic)	1996-756X

Conference

Conference	15th International Conference on Quality Control by Artificial Vision
Country/Territory	Japan
City	Tokushima, Virtual
Period	21/5/12 → 21/5/14

Keywords

Action Segmentation
Computer Vision
Dataset Construction
Machine Learning
Multi-modal Learning

ASJC Scopus subject areas

Electronic, Optical and Magnetic Materials
Condensed Matter Physics
Computer Science Applications
Applied Mathematics
Electrical and Electronic Engineering

Access to Document

10.1117/12.2591752

Cite this

Kogure, S., & Aoki, Y. (2021). Multi-modal action segmentation in the kitchen with a feature fusion approach. In K. Terada, A. Nakamura, T. Komuro, & T. Shimizu (Eds.), Fifteenth International Conference on Quality Control by Artificial Vision Article 117940Z (Proceedings of SPIE - The International Society for Optical Engineering; Vol. 11794). SPIE. https://doi.org/10.1117/12.2591752

Multi-modal action segmentation in the kitchen with a feature fusion approach. / Kogure, Shunsuke; Aoki, Yoshimitsu.
Fifteenth International Conference on Quality Control by Artificial Vision. ed. / Kenji Terada; Akio Nakamura; Takashi Komuro; Tsuyoshi Shimizu. SPIE, 2021. 117940Z (Proceedings of SPIE - The International Society for Optical Engineering; Vol. 11794).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Kogure, S & Aoki, Y 2021, Multi-modal action segmentation in the kitchen with a feature fusion approach. in K Terada, A Nakamura, T Komuro & T Shimizu (eds), Fifteenth International Conference on Quality Control by Artificial Vision., 117940Z, Proceedings of SPIE - The International Society for Optical Engineering, vol. 11794, SPIE, 15th International Conference on Quality Control by Artificial Vision, Tokushima, Virtual, Japan, 21/5/12. https://doi.org/10.1117/12.2591752

@inproceedings{d6fc52eb1629481497c0c2a52ea46d48,

title = "Multi-modal action segmentation in the kitchen with a feature fusion approach",

abstract = "In this paper, we propose a {"}Multi-modal Action Segmentation approach{"}that uses three modalities: (i) video, (ii) audio, (iii) thermal to classify cooking behavior in the kitchen. These 3 modalities are assumed to be features related to cooking. However, there is no public dataset containing these three modalities. Therefore, we built the original dataset and frame-level annotation. We then examined the usefulness of Action Segmentation using multi-modal features. We analyzed the effects of each modality using three evaluation metrics. As a result, the accuracy, edit distance, and F1 value were improved by up to about 1%, 2%, and 8%, respectively, compared to the case when only images were used.",

keywords = "Action Segmentation, Computer Vision, Dataset Construction, Machine Learning, Multi-modal Learning",

author = "Shunsuke Kogure and Yoshimitsu Aoki",

note = "Publisher Copyright: {\textcopyright} 2021 SPIE.; 15th International Conference on Quality Control by Artificial Vision ; Conference date: 12-05-2021 Through 14-05-2021",

year = "2021",

doi = "10.1117/12.2591752",

language = "English",

series = "Proceedings of SPIE - The International Society for Optical Engineering",

publisher = "SPIE",

editor = "Kenji Terada and Akio Nakamura and Takashi Komuro and Tsuyoshi Shimizu",

booktitle = "Fifteenth International Conference on Quality Control by Artificial Vision",

}

TY - GEN

T1 - Multi-modal action segmentation in the kitchen with a feature fusion approach

AU - Kogure, Shunsuke

AU - Aoki, Yoshimitsu

PY - 2021

Y1 - 2021

N2 - In this paper, we propose a "Multi-modal Action Segmentation approach"that uses three modalities: (i) video, (ii) audio, (iii) thermal to classify cooking behavior in the kitchen. These 3 modalities are assumed to be features related to cooking. However, there is no public dataset containing these three modalities. Therefore, we built the original dataset and frame-level annotation. We then examined the usefulness of Action Segmentation using multi-modal features. We analyzed the effects of each modality using three evaluation metrics. As a result, the accuracy, edit distance, and F1 value were improved by up to about 1%, 2%, and 8%, respectively, compared to the case when only images were used.

AB - In this paper, we propose a "Multi-modal Action Segmentation approach"that uses three modalities: (i) video, (ii) audio, (iii) thermal to classify cooking behavior in the kitchen. These 3 modalities are assumed to be features related to cooking. However, there is no public dataset containing these three modalities. Therefore, we built the original dataset and frame-level annotation. We then examined the usefulness of Action Segmentation using multi-modal features. We analyzed the effects of each modality using three evaluation metrics. As a result, the accuracy, edit distance, and F1 value were improved by up to about 1%, 2%, and 8%, respectively, compared to the case when only images were used.

KW - Action Segmentation

KW - Computer Vision

KW - Dataset Construction

KW - Machine Learning

KW - Multi-modal Learning

UR - http://www.scopus.com/inward/record.url?scp=85112431796&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85112431796&partnerID=8YFLogxK

U2 - 10.1117/12.2591752

DO - 10.1117/12.2591752

M3 - Conference contribution

AN - SCOPUS:85112431796

T3 - Proceedings of SPIE - The International Society for Optical Engineering

BT - Fifteenth International Conference on Quality Control by Artificial Vision

A2 - Terada, Kenji

A2 - Nakamura, Akio

A2 - Komuro, Takashi

A2 - Shimizu, Tsuyoshi

PB - SPIE

T2 - 15th International Conference on Quality Control by Artificial Vision

Y2 - 12 May 2021 through 14 May 2021

ER -

Multi-modal action segmentation in the kitchen with a feature fusion approach

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this