Multi Modal Emotion Recognition and Sentimental Analysis for Next Generation Home Automation
Keywords:
Home Automation, Intelligent Systems, Sentimental Analysis, Emotion RecognitionAbstract
Intelligent devices are the attraction of next generation systems where it does the work automatically without any supervision. Studies on Intelligent Systems shows various approaches in making systems intelligent and how their decision making capabilities are enhanced. All the existing systems involve some sort of human supervision in making decisions. Traditional Human-Machine interaction may not be possible in emergency situations or in real world busy situations. Hence it’s time to think on Intelligent Systems capable of doing tasks without any supervision. Such systems can completely avoid human interventions or any form of instructions. My proposed research study is focusing on Intelligent Devices in a home environment that can assist in day to day activities of the residents in aware of situations. Such devices are capable of recognizing its environment through resident’s voice, emotions, life style, routine, etc and can take decisions appropriately. Also it can provide personalized assistance to individual resident or guests. Hoping that such a system can reduce the workload of human beings and surely it will be a new direction of research in this field.
References
1. Renata L Rosa, Demsteneso Z. Rodriguez and Graca Bressan, “Music recommendation system based on user’s sentiments extracted from social networks”, in IEEE Transactions on Consumer Electronics, IEEE, 2015; 61(3) 359 – 367.
2. Chi Hwan Choi, Jeong Eun Lee, Gyeong Su Park, Jonghwa Na and Wan Sup Cho, “Voice of Customer Analysis for Internet Shopping Malls”, in International Journal of Smart Home, IJSH, 2013; 7(5): 291-304.
3. David Bell, Theodora Koulouri, Stanislao Lauria, Robert D. Macredie, and James Sutton, “Microblogging as a mechanism for human robot interaction”, in KnowledgeBased Systems 69, Elsevier, 2014: 64–77.
4. Gupta DA, Dwith CYN, Vighnesh Ramakanth BA. “Wireless Home Automation Using Social Networking Websites”, in Advanced Computing and Communications (ADCOM), Annual International Conference, IEEE, 2014. doi:10.1109/ADCOM.2014.7103241.
5. Cambria E, Poria S, Bisio F et al. “The CLSA Model: A Novel Framework for Concept-Level Sentiment Analysil”, in Springer International Publishing Switzerland, Part II, LNCS 9042. 2015: 3–22.
6. Cambria E, Poria S, Bajpai R et al. “SenticNet 4: A Semantic Resource for Sentiment Analysis Based on Conceptual Primitives”, Online] Available: http://sentic. net/senticnet-4.pdf.
7. Bisio F, Meda C, Gastaldo P et al. “SentimentOriented Information Retrieval: Affective Analysis of Documents Based on the SenticNet Framework”, in Sentiment Analysis and Ontology Engineering, Springer International Publishing Switzerland, 2016: 175-197.
8. ElKamchouchi H, ElShafee A. “Design and prototype implementation of SMS based home automation system”, in Electronics Design, Systems and Applications (ICEDSA), International Conference Proceedings, IEEE, 2012.
9. AlShu’eili H, Gupta GS, Mukhopadhyay S. “Voice recognition based wireless home automation system”, in Mechatronics (ICOM), International Conference,
2011. Doi: 10.1109/ICOM.2011.5937116.
10. Keumhee Kang, Chanhee Yoon, and Eun Yi Kim, “Identifying depressive users in Twitter using multimodal analysis”, inBig Data and Smart Computing (BigComp), International Conference, IEEE, 2016, pp.231 - 238, DOI: 10.1109/BIGCOMP.2016.7425918.
11. De Silva LC, Miyasato T, Nakatsu R. “Facial emotion recognition using multimodal information”, in Information, Communications and Signal Processing, ICICS., Proceedings of 1997 International Conference, IEEE. 1997; 1: 397–401.
12. Kaushik L, Sangwan A, Hansen JHL. “Sentiment extraction from natural audio streams”, inAcoustics, Speech and Signal Processing (ICASSP), IEEE International Conference, IEEE, 2013. doi: 10.1109/ ICASSP.2013.6639321.
13. Kaushik L, Sangwan A, Hansen JHL. “Automatic sentiment extraction from YouTube videos” in Automatic Speech Recognition and Understanding (ASRU), IEEE Workshop, IEEE, 2013. Doi: 10.1109/ASRU.2013.6707736.
14. Zehnder M. “Energy saving in smart homes based on consumer behaviour data”, Northwestern Switzerland, 2015.
15. EkmanP, Friesen W, “Facial Action Coding System: A Technique for the Measurement of Facial Movement” Consulting Psychologists Press, Stanford University, Palo Alto, 1977.
16. Prafulla Nath Dawadi, Diane Joyce Cook and Maureen Schmitter-Edgecombe, “Automated Cognitive Health Assessment From Smart Home-Based Behavior Data”, in IEEE Journal of Biomedical and Health Informatics, IEEE, 2016; 20(4) 1188 – 1194.
17. Qin Ni, Belén García Hernando, Iván Pau de la Cruz. “The Elderly’s Independent Living in Smart Homes: A Characterization of Activities and Sensing Infrastructure Survey to Facilitate Services Development”, in Sensors, 2015; 15: 11312-11362. doi:10.3390/s150511312.
18. Soujanya Poria, Iti Chaturvedi, Erik Cambria, Amir Hussain, “Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis”, [Online] Available: http://sentic.net/convolutional-mkl-basedmulimodal-sentiment-analysis.pdf.
19. Stuti Jindal, Sanjay Singh, “Image sentiment analysis using deep Convolutional neural networks with domain specific fine tuning”, in Information Processing (ICIP), 2015 International Conference, IEEE, 2015; 447-451. Doi: 10.1109/INFOP.2015.7489424.
20. Yadav SK, Bhushan M, Gupta S. “Multimodal sentiment analysis: Sentiment analysis using audiovisual format”, in Computing for Sustainable Global Development (INDIACom). 2015 2nd International Conference, IEEE, 2015.
21. Mehrabi T, Fung AS, Raahemifar K. “Optimization of home automation systems based on human motion and behaviour”, in Electrical and Computer Engineering (CCECE), Canadian Conference, Toronto Canada, IEEE, 2014.
22. Rozgic V, Ananthakrishnan S, Saleem S et al. “Speech language & multimedia technology”, in Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), Asia-Pacific, IEEE. 2012: 1–4. 23. Rosas V, Mihalcea R, Morency LP. “Multimodal sentiment analysis of Spanish online videos”, in IEEE Intelligent Systems. 2013; 28(3): 38–45.
24. Werapun W, Kamhang A, Wachiraphan A. “Design of Home Automation Framework With Social Network Integration”, in Journal of Networking Technology 2014; 5(4) 119-124.
25. Zheng W, Fang H. “A Retrieval System Based on Sentiment Analysis”. 2010. [Online]. Available: http:// research.microsoft.com/enus/um/people/ryenw/ hcir2010/docs/papers/Zheng_cr35.pdf
26. Yu-Cheng Fan, Hung-Kuan Liu. “Three-dimensional gesture interactive system design of home automation for physically handicapped people”, in Medical Measurements and Applications (MeMeA), International Symposium, IEEE, 2015. Doi: 10.1109/ MeMeA.2015.7145242.
27. Howard N. “Application of Intention Awareness and Sentic Computing for Sensemaking in Joint-Cognitive Systems”, in Intelligent Agent (IA), IEEE Symposium, IEEE, 2013. Doi:10.1109/IA.2013.6595182.
28. Ching- Hu Lu, Chao-Lin Wu, Mao-Yung Weng et al. “Context-Aware Energy Saving System with Multiple Comfort-Constrained Optimization in M2M Based Home Environment”, in IEEE Transactions on Automation Science and Engineering 2015 99: 1-15.
29. Lundberg J. “Situation Awareness Systems, States and Processes: A Holistic Framework”, in Theoretical Issues in Ergonomics Science. Doi: 10.1080/1463922X.2015 1008601.
30. Howard N, Cambria E. “Intention Awareness: Improving Upon Situation Awareness in Human-Centric Environments”, in Human-Centric Computing and Information Sciences, Springer Open Journal, 2013.
2. Chi Hwan Choi, Jeong Eun Lee, Gyeong Su Park, Jonghwa Na and Wan Sup Cho, “Voice of Customer Analysis for Internet Shopping Malls”, in International Journal of Smart Home, IJSH, 2013; 7(5): 291-304.
3. David Bell, Theodora Koulouri, Stanislao Lauria, Robert D. Macredie, and James Sutton, “Microblogging as a mechanism for human robot interaction”, in KnowledgeBased Systems 69, Elsevier, 2014: 64–77.
4. Gupta DA, Dwith CYN, Vighnesh Ramakanth BA. “Wireless Home Automation Using Social Networking Websites”, in Advanced Computing and Communications (ADCOM), Annual International Conference, IEEE, 2014. doi:10.1109/ADCOM.2014.7103241.
5. Cambria E, Poria S, Bisio F et al. “The CLSA Model: A Novel Framework for Concept-Level Sentiment Analysil”, in Springer International Publishing Switzerland, Part II, LNCS 9042. 2015: 3–22.
6. Cambria E, Poria S, Bajpai R et al. “SenticNet 4: A Semantic Resource for Sentiment Analysis Based on Conceptual Primitives”, Online] Available: http://sentic. net/senticnet-4.pdf.
7. Bisio F, Meda C, Gastaldo P et al. “SentimentOriented Information Retrieval: Affective Analysis of Documents Based on the SenticNet Framework”, in Sentiment Analysis and Ontology Engineering, Springer International Publishing Switzerland, 2016: 175-197.
8. ElKamchouchi H, ElShafee A. “Design and prototype implementation of SMS based home automation system”, in Electronics Design, Systems and Applications (ICEDSA), International Conference Proceedings, IEEE, 2012.
9. AlShu’eili H, Gupta GS, Mukhopadhyay S. “Voice recognition based wireless home automation system”, in Mechatronics (ICOM), International Conference,
2011. Doi: 10.1109/ICOM.2011.5937116.
10. Keumhee Kang, Chanhee Yoon, and Eun Yi Kim, “Identifying depressive users in Twitter using multimodal analysis”, inBig Data and Smart Computing (BigComp), International Conference, IEEE, 2016, pp.231 - 238, DOI: 10.1109/BIGCOMP.2016.7425918.
11. De Silva LC, Miyasato T, Nakatsu R. “Facial emotion recognition using multimodal information”, in Information, Communications and Signal Processing, ICICS., Proceedings of 1997 International Conference, IEEE. 1997; 1: 397–401.
12. Kaushik L, Sangwan A, Hansen JHL. “Sentiment extraction from natural audio streams”, inAcoustics, Speech and Signal Processing (ICASSP), IEEE International Conference, IEEE, 2013. doi: 10.1109/ ICASSP.2013.6639321.
13. Kaushik L, Sangwan A, Hansen JHL. “Automatic sentiment extraction from YouTube videos” in Automatic Speech Recognition and Understanding (ASRU), IEEE Workshop, IEEE, 2013. Doi: 10.1109/ASRU.2013.6707736.
14. Zehnder M. “Energy saving in smart homes based on consumer behaviour data”, Northwestern Switzerland, 2015.
15. EkmanP, Friesen W, “Facial Action Coding System: A Technique for the Measurement of Facial Movement” Consulting Psychologists Press, Stanford University, Palo Alto, 1977.
16. Prafulla Nath Dawadi, Diane Joyce Cook and Maureen Schmitter-Edgecombe, “Automated Cognitive Health Assessment From Smart Home-Based Behavior Data”, in IEEE Journal of Biomedical and Health Informatics, IEEE, 2016; 20(4) 1188 – 1194.
17. Qin Ni, Belén García Hernando, Iván Pau de la Cruz. “The Elderly’s Independent Living in Smart Homes: A Characterization of Activities and Sensing Infrastructure Survey to Facilitate Services Development”, in Sensors, 2015; 15: 11312-11362. doi:10.3390/s150511312.
18. Soujanya Poria, Iti Chaturvedi, Erik Cambria, Amir Hussain, “Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis”, [Online] Available: http://sentic.net/convolutional-mkl-basedmulimodal-sentiment-analysis.pdf.
19. Stuti Jindal, Sanjay Singh, “Image sentiment analysis using deep Convolutional neural networks with domain specific fine tuning”, in Information Processing (ICIP), 2015 International Conference, IEEE, 2015; 447-451. Doi: 10.1109/INFOP.2015.7489424.
20. Yadav SK, Bhushan M, Gupta S. “Multimodal sentiment analysis: Sentiment analysis using audiovisual format”, in Computing for Sustainable Global Development (INDIACom). 2015 2nd International Conference, IEEE, 2015.
21. Mehrabi T, Fung AS, Raahemifar K. “Optimization of home automation systems based on human motion and behaviour”, in Electrical and Computer Engineering (CCECE), Canadian Conference, Toronto Canada, IEEE, 2014.
22. Rozgic V, Ananthakrishnan S, Saleem S et al. “Speech language & multimedia technology”, in Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), Asia-Pacific, IEEE. 2012: 1–4. 23. Rosas V, Mihalcea R, Morency LP. “Multimodal sentiment analysis of Spanish online videos”, in IEEE Intelligent Systems. 2013; 28(3): 38–45.
24. Werapun W, Kamhang A, Wachiraphan A. “Design of Home Automation Framework With Social Network Integration”, in Journal of Networking Technology 2014; 5(4) 119-124.
25. Zheng W, Fang H. “A Retrieval System Based on Sentiment Analysis”. 2010. [Online]. Available: http:// research.microsoft.com/enus/um/people/ryenw/ hcir2010/docs/papers/Zheng_cr35.pdf
26. Yu-Cheng Fan, Hung-Kuan Liu. “Three-dimensional gesture interactive system design of home automation for physically handicapped people”, in Medical Measurements and Applications (MeMeA), International Symposium, IEEE, 2015. Doi: 10.1109/ MeMeA.2015.7145242.
27. Howard N. “Application of Intention Awareness and Sentic Computing for Sensemaking in Joint-Cognitive Systems”, in Intelligent Agent (IA), IEEE Symposium, IEEE, 2013. Doi:10.1109/IA.2013.6595182.
28. Ching- Hu Lu, Chao-Lin Wu, Mao-Yung Weng et al. “Context-Aware Energy Saving System with Multiple Comfort-Constrained Optimization in M2M Based Home Environment”, in IEEE Transactions on Automation Science and Engineering 2015 99: 1-15.
29. Lundberg J. “Situation Awareness Systems, States and Processes: A Holistic Framework”, in Theoretical Issues in Ergonomics Science. Doi: 10.1080/1463922X.2015 1008601.
30. Howard N, Cambria E. “Intention Awareness: Improving Upon Situation Awareness in Human-Centric Environments”, in Human-Centric Computing and Information Sciences, Springer Open Journal, 2013.
Published
2019-06-16
Issue
Section
Articles
