Sentiment analysis for e-commerce product reviews by deep learning model of Bert-BiGRU-Softmax

Yi Liu; Jiahuan Lu; Jie Yang; Feng Mao; Yi Liu; Jiahuan Lu; Jie Yang; Feng Mao

doi:10.3934/mbe.2020398

Mathematical Biosciences and Engineering

2020, Volume 17, Issue 6: 7819-7837. doi: 10.3934/mbe.2020398

Previous Article Next Article

Research article Special Issues

Sentiment analysis for e-commerce product reviews by deep learning model of Bert-BiGRU-Softmax

Management School, Hangzhou Dianzi University, Hangzhou, 310018, China

Received: 03 July 2020 Accepted: 26 October 2020 Published: 09 November 2020

Sentiment analysis of e-commerce reviews is the hot topic in the e-commerce product quality management, from which manufacturers are able to learn the public sentiment about products being sold on e-commerce websites. Meanwhile, customers can know other people's attitudes about the same products. This paper proposes the deep learning model of Bert-BiGRU-Softmax with hybrid masking, review extraction and attention mechanism, which applies sentiment Bert model as the input layer to extract multi-dimensional product feature from e-commerce reviews, Bidirectional GRU model as the hidden layer to obtain semantic codes and calculate sentiment weights of reviews, and Softmax with attention mechanism as the output layer to classify the positive or negative nuance. A series of experiments are conducted on the large-scale dataset involving over 500 thousand product reviews. The results show that the proposed model outperforms the other deep learning models, including RNN, BiGRU, and Bert-BiLSTM, which can reach over 95.5% of accuracy and retain a lower loss for the e-commerce reviews.
- sentiment analysis,
- e-commerce product reviews,
- Bert,
- Bidirectional GRU
Citation: Yi Liu, Jiahuan Lu, Jie Yang, Feng Mao. Sentiment analysis for e-commerce product reviews by deep learning model of Bert-BiGRU-Softmax[J]. Mathematical Biosciences and Engineering, 2020, 17(6): 7819-7837. doi: 10.3934/mbe.2020398

Related Papers:

Abstract

Sentiment analysis of e-commerce reviews is the hot topic in the e-commerce product quality management, from which manufacturers are able to learn the public sentiment about products being sold on e-commerce websites. Meanwhile, customers can know other people's attitudes about the same products. This paper proposes the deep learning model of Bert-BiGRU-Softmax with hybrid masking, review extraction and attention mechanism, which applies sentiment Bert model as the input layer to extract multi-dimensional product feature from e-commerce reviews, Bidirectional GRU model as the hidden layer to obtain semantic codes and calculate sentiment weights of reviews, and Softmax with attention mechanism as the output layer to classify the positive or negative nuance. A series of experiments are conducted on the large-scale dataset involving over 500 thousand product reviews. The results show that the proposed model outperforms the other deep learning models, including RNN, BiGRU, and Bert-BiLSTM, which can reach over 95.5% of accuracy and retain a lower loss for the e-commerce reviews.

References

[1]	P. Sasikala, L. M. I. Sheela, Sentiment analysis of online product reviews using DLMNN and future prediction of online product using IANFIS, J. Big Data, 7 (2020), 33-53. doi: 10.1186/s40537-020-00308-7
[2]	P. A. Pavlou, A. Dimoka, The nature and role of feedback text comments in online marketplaces: Implications for trust building, price premiums, and seller differentiation, Inf. Syst. Res., 17 (2006), 392-414. doi: 10.1287/isre.1060.0106
[3]	A. Abbasi, H. Chen, A. Salem, Sentiment analysis in multiple languages: feature selection for opinion classification in web forums, ACM Trans. Inf. Syst., 26 (2008), 12-21.
[4]	Z. Zhang, Y. E. Qiang, Literature review on sentiment analysis of online product reviews, J. Manage. Sci. China, 13 (2010), 84-96.
[5]	C. Chang, C. J. Lin, LIBSVM: A library for support vector machines, ACM Trans. Intel. Syst. Technol., 2 (2011), 1-27.
[6]	F. Hu, L. Li, Z. L. Zhang, et al, Emphasizing essential words for sentiment classification based on recurrent neural networks, J. Comput. Sci. Technol., 32 (2017), 785-795. doi: 10.1007/s11390-017-1759-2
[7]	Y. Mejova, P. Srinivasan, Exploring feature definition and selection for sentiment classifiers. In Proc. 5^th Int. Conf. weblogs social media, Barcelona, Catalonia, Spain, (2011), 17-21.
[8]	W. Casey, G. Navendu, A. Shlomo, Using appraisal groups for sentiment analysis. In Proc. ACM SIGIR Conf. Inform. Knowl. Manag., (2005), 625-31.
[9]	L.-C. Yu, H.-L. Wu, P.-C. Chang, H.-S. Chu, Using a contextual entropy model to expand emotion words and their intensity for the sentiment classification of stock market news, Knowl. Based Syst., 41 (2013), 89-97. doi: 10.1016/j.knosys.2013.01.001
[10]	I. Jolliffe, N. A. Jolliffe, Principal Component Analysis. Springer Series in Statistics, 2nd edition, Springer-Verlag, New York, 2002.
[11]	D. Adnan, F. Song, Feature selection for sentiment analysis based on content and syntax models, Decis. Support Syst., 53 (2012), 704-11. doi: 10.1016/j.dss.2012.05.023
[12]	L. G. Thomas, S. Mark, M. B. David, B. T. Joshua, Integrating topics and syntax, Adv. Neural Inf. Process. Syst., (2005), 537-544.
[13]	A. Yadav, D. K. Vishwakarma, Sentiment analysis using deep learning architectures: a review, Artif. Intell. Rev., 53 (2020), 4335-4385. doi: 10.1007/s10462-019-09794-5
[14]	S. Kim, E. Hovy, Determining the sentiment of opinions. In Proc. Conf. Comput. Linguis., Geneva, Switzerland, (2004), 1367-1373.
[15]	P. D. Turney, M. L. Littman, Measuring praise and criticism: Inference of semantic orientation from association, ACM Trans. Inf. Syst., 21 (2003), 315-346. doi: 10.1145/944012.944013
[16]	Y. L. Zhu, J. Min, Y. Q. Zhou, Semantic orientation computing based on HowNet, J. Chin. Inf. Process., 20 (2006), 14-20.
[17]	G. Somprasertsri, P. Lalitrojwong, Mining feature-opinion in online customer reviews for opinion summarization, J. Univ. Comput. Sci., 16 (2010), 938-955.
[18]	B. Liu, Sentiment analysis and opinion mining, Synth. Lect. Hum. Lang. Technol., 5 (2012), 1-167.
[19]	B. Pang, L. Lee, S. Vaithyanathan, Thumbs up? sentiment classification using machine learning techniques. In Proc. Conf. Empir. Methods Nat. Lang. Process., Philadelphia, USA, (2002), 79-86.
[20]	A. Go, R. Bhayani, L. Huang, Twitter sentiment classification using distant supervision, CS224N Project Report, Stanford, 1 (2009), 1-12.
[21]	M. Guerini, L. Gatti, M. Turchi, Sentiment analysis: How to derive prior polarities from SentiWord Net, arXiv preprint arXiv: 1309.5843.
[22]	A. Tripathy, A. Agrawal, S. K. Rath, Classification of sentiment reviews using n-gram machine learning approach, Expert Syst. Appl., 57 (2016), 117-126. doi: 10.1016/j.eswa.2016.03.028
[23]	Y. KIM, Convolutional neural networks for sentence classification, arXiv preprint arXiv: 1408.5882.
[24]	K. Cho, B. van Merrienboer, D. Bahdanau, On the properties of neural machine translation: encoder-decoder approaches. In Proce. Eighth Workshop Syntax, Semant. Struct. Stat. Trans., Doha, Qatar, (2014), 103-111.
[25]	Z. Qu, Y. Wang, X. Wang, A Hierarchical attention network sentiment classification algorithm based on transfer learning, J. Comput. Appl., 38 (2018), 3053-3056.
[26]	J. Pennington, R. Socher, C. D. Manning, GloVe: Global vectors for word representation, Proc. Int. Conf. Empir. Methods Nat. Lang. Process., (2014), 147-168.
[27]	F. Abid, M. Alam, M. Yasir, C. Li, Sentiment analysis through recurrent variants latterly on convolutional neural network of twitter, Future Gener. Comput. Syst., 95 (2019), 292-308. doi: 10.1016/j.future.2018.12.018
[28]	J. Trofimovich, Comparison of neural network architectures for sentiment analysis of Russian tweets, In Proc. Int. Conf. Dialogue 2016, RGGU, (2016).
[29]	Q. Qian, M. Huang, J. Lei, X. Zhu, Linguistically regularized LSTMs for sentiment classification. In Proc. 55th Ann. Meet. Assoc. Comput. Ling., 1 (2016), 1679-1689.
[30]	L. Nio, K. Murakami, Japanese sentiment classification using bidirectional long short-term memory recurrent neural network. In Proc. 24th Ann. Meeti. Assoc. Nat. Lang. Process., Okayama, Japan, (2018), 1119-1122.
[31]	M.-L. Zhang, Z.-H. Zhou, A review on multi-label learning algorithms, IEEE Trans. Knowl. Data Eng., 26 (2014), 1819-1837. doi: 10.1109/TKDE.2013.39
[32]	J. Devlin, M. W. Chang, K. Lee, K. Toutanova, BERT: Pre-Training of deep bidirectional transformers for language understanding, In Proc. 2019 Conf. North Am. Chapter Assoc. Comput. Linguist. Human Lang. Tech., (2019), 4171-4186.
[33]	S. Seo, C. Kim, H. Kim, et al, Comparative study of deep learning based sentiment classification, IEEE Access, 8 (2020), 6861-6875. doi: 10.1109/ACCESS.2019.2963426
[34]	B. Dzmitry, C. Kyunghyun, B. Yoshua, Neural machine translation by jointly learning to align and translate, In Proc. Int. Conf. Learn. Representations 2015, San Diego, CA, USA, (2015), 1-15.
[35]	D. Li, F. Wei, C. Tan, Adaptive recursive neural network for target-dependent twitter sentiment classification, Baltimore: Assoc. Comput. Ling.-ACL, (2014), 49-54.
[36]	S. M. Grü.er-Sinopoli, C. N. Thalemann, Bidirectional recurrent neural networks, IEEE Trans. Signal Process., 11 (1997), 2673-2681
[37]	S. Zhang, X. Xu, Y. Pang, J. Han, Multi-layer Attention Based CNN for Target-Dependent Sentiment Classification, Neural Process. Lett., 51 (2020), 2089-2103 doi: 10.1007/s11063-019-10017-9
[38]	K. Saranya, S. Jayanthy, Onto-based sentiment classification using machine learning techniques. In Proc. 2017 Int. Conf. Innovations Inf., Embed. Commun. Syst., (2017), 1-5.
[39]	S. Tan, J. Zhang, An empirical study of sentiment analysis for Chinese documents, Expert Syst. Appl., 34 (2008), 2622-2629. doi: 10.1016/j.eswa.2007.05.028
[40]	W. Li, F. Qi, Sentiment analysis based on multi-channel bidirectional long short term memory network, J. Chin. Inf. Process., 33 (2019), 119-128.
[41]	Y. Yang, H. Yuan, Y. Wang, Sentiment analysis method for comment text, J. Nanjing Univ. Sci. Tech., 43 (2019), 280-285.

Reader Comments

Your name:*

Email:*
© 2020 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)