Computational analysis of user experience and customer satisfaction with mobile food delivery services: Evidence from big data approaches

1.   Introduction

The global diffusion of COVID-19 has had notable effects on customer lifestyles, including food and eating behavior. In light of its rapid diffusion and negative effects on our society, the majority of national and local governments have implemented their own diffusion-prevention policies (i.e., stay-at-home plans) ^[1]. Moreover, in South Korea, social distance rules to reduce personal interactions that can diffuse infectious diseases, have been implemented. Due to social distance rules and stay-at-home plans, several delivery services have been introduced to support the ''untact economy'' in Korea ^[2]. Among these services, mobile food delivery services (MFDSs) are among the greatest beneficiaries by linking consumers and restaurants, allowing a number of restaurants to start providing delivery services through mobile applications ^[3].

In the restaurant industry, consumers avoid physically visiting restaurants, but are more likely to use mobile food delivery applications for their meals. For instance, the total revenue of online food delivery services is estimated to be 22,000 million USD, with an expected compound annual growth rate (from 2022 to 2026) of about 15% in South Korea^*. Along with this growth of MFDSs, the needs of consumers are constantly becoming more diverse and complex ^[4]. Thus, it is necessary to computationally and systematically investigate user experiences with MFDSs to afford the providers of MFDS applications the opportunity to improve their applications based on user feedback. Although many scholars have explored users' perspectives toward MFDSs ^[5,6], the majority of these scholars' approaches are limited due to such factors as small numbers of participants in surveys ^[7,8,9], or a reliance on traditional marketing/consumer-oriented theories ^[10].

^* https://www.statista.com/outlook/dmo/eservices/online-food-delivery/south-korea

Thus, this study aims to computationally address factors of the user experience with MFDSs as determinants of user satisfaction when employing MFDSs. To this end, the following procedures are applied.

$1)$ We collected the datasets of user information, review comments, and ratings on MFDSs.

$2)$ The elements of user experiences with MFDSs were computationally and systematically extracted from the datasets.

$3)$ Two multiple regression analyses were conducted to explore UX factors as key determinants of user satisfaction with MFDSs.

The next section presents the literature review. The data collection and processing procedures are examined next, followed by the data analysis strategy. After examining the empirical results, a general discussion and concluding remarks are presented.

2.   Literature review

2.1.   User experience with mobile food delivery services

When innovative or new services are proposed and diffused ^[11], users' individual perceptions of the services are notable factors in determining their perceived satisfaction and adoption ^[12]. Among these factors, several components related to user experience (UX) are considered potential antecedents of users' perceived satisfaction. Generally, the definition of UX is "an individual assessment and feeling of the quality of a user's interaction with a particular service or product, for completing a certain task in a specific context" ^[13]. Based on this definition, several scholars have examined the elements of UX and their effects on users' perspectives toward a service or product ^[4].

Considering users' post-assessments of specific services from UX perspectives, three elements, usability, usefulness, and affection, have been examined as dominant components in UX. The definition of perceived usability is "service/product attributes, which allow users to easily, effectively and efficiently achieve specific tasks" ^[14]. Perceived usefulness is referred as "the level to which an individual user considers that employing a particular service/product could enhance his/her performance" ^[15]. Finally, a common definition of users' affection is "a user's emotional response to the perceptual details of a particular service/product" ^[16].

UX is one of the core concepts in exploring the successful distribution of mobile applications ^[17]. For instance, Harrison et al. ^[18] introduced systematic approaches for improving users' perceived satisfaction through a new usability-evaluation model based on the characteristics of earlier usability-oriented theories. In addition, Park et al. ^[19] examined the motivating factors determining users' behavioral intention to use mobile geographic information services with an integrated user acceptance model. Examining more than 1000 users, they found that users' perceived usefulness played both direct and indirect roles in their intention to use the services; the research model showed great fit to the data. Gupta and Arora ^[20] investigated consumer behavioral intention to adopt mobile payment systems with an integrated acceptance model. Based on the responses of 267 users, they found that hedonic motivation, which is one of the notable affective constructs, is a predictor of users' intention to use the systems.

Shah et al. ^[6] proposed a research model to investigate effects of technological and psychological cues on user continuous intention to employ mobile food ordering applications with consideration of the uses and gratification theory. Considering 661 and 307 responses, they found several notable cues (convenience, price saving orientation, and compatibility) as direct determinants of user satisfaction and intention. Choi ^[21] also focused on user satisfaction of mobile food delivery applications in Korea with the concept of perceived familiarity, ease of use, usefulness, and intention to reuse. Based on 296 valid responses, the satisfaction is mainly determined by perceived usefulness of the applications, which is one of the core UX dimensions. In addition, Shah et al. ^[8] also examined customers' mobile dining choice applications with consideration of the stimulus-organism-responses framework. Considering 351 responses in Indonesia, customers' purchase intention is mainly determined by perceived value, which is organized by nine factors (e.g., source credibility, food quality, and ease of use) via three key sectors (electronic word-of-mouth, overall restaurant image, and system quality).

Although a huge number of mobile applications and their services have appeared in recent years ^[19], user-oriented approaches to MFDSs have not been well examined. Thus, this study aims to explore users' perspectives on MFDSs with consideration of UX as one of the cornerstones of MFDSs.

2.2.   Review analysis

To quantitatively address post-evaluations of UX, user review comments are a useful source of valuable insights into and practical implications of UX, which can allow researchers to improve mobile applications. In particular, Google Play Store and Apple App Store tend to be the globally dominant mobile application markets for mobile devices. Because both stores have a space for users' comments and ratings on each application, users consider them an online forum for comments on the application.

Several scholars have examined users' review comments in this space as key sources for their overall perspectives toward each application, including perceived satisfaction. For example, Park ^[22] showed that the results of sentiment analysis on fusers' review comments can be employed to estimate their overall satisfaction with online information services. Zhao et al. ^[23] examined the linguistic elements of 127,629 responses. They found that customer satisfaction is significantly affected by review diversity (positive), polarity (positive), subjectivity (negative), readability (negative), and word counts (negative). In addition, customers' perceived technical attributes play a notable role in determining satisfaction. As another example, Oh et al. ^[24] applied the widely employed psychological theory of expectation-confirmation theory to customer satisfaction with hospitality services through deep learning approaches. Applying a fused deep learning model to the responses of 12,363 users, they achieved an average precision of 0.9277 in investigating whether customers are satisfied with hospitality services.

Review comments are also helpful for understanding users' perspectives toward mobile services. Jang and Park ^[25] indicated through a consideration of application domains that users' perceived satisfaction with mobile augmented reality applications is significantly affected by several factors of their user experience, based on the responses of 8627 users. Similarly, Phetrungnapha and Senivongse ^[26] showed that users' comments on mobile applications play a notable role in identifying the troubles and issues of the applications based on the extraction and natural language processing of several features. They achieved a recall of 91.23% in addressing bug report classification tasks. Thus, the current study employs user review comments to explore the determinants of users' perceived satisfaction when using MFDSs.

3.   Study methodology

3.1.   Data collection and preprocessing

Users' comments on MFDSs in Apple App Store and Google Play were collected. Three top-ranked download applications in both stores were selected (Uber Eats, DoorDash, and Grubhub). We collected 140,992 responses, including users' review comments and ratings, from 2022. The ratings, which ranged from 1 to 5, are employed as users' perceived satisfaction. We excluded comments with fewer than five words, emoticons, and non-English characters.

We employed two different natural language processing approaches, -words and sentiment analysis, to examine the effects of UX elements on user satisfaction with the MFDSs. We conducted a sentiment analysis through LIWC software to computationally examine the dimensions of UX. Jang and Yi ^[27] employed hedonic, user burden, confirmation, and pragmatic factors as key UX elements, and matched them with LIWC categories, positive emotion (PE), negative emotion (NE), comparisons (COM), and work/leisure/home (WLH). Moreover, the cost (CO) dimension is considered one of the determinants of user satisfaction ^[4]. Based on the guidance of Jang and Yi ^[27], we computed the level of each UX dimension using LIWC software (Table 1).

The following equation was used to explore the effects of UX elements, as well as cost, from the sentiment analysis of user satisfaction, where $b_0$ is the intercept, $x_{PE}$, $x_{NE}$, $x_{COM}$, $x_{WLH}$, and $x_{CO}$ are the regression coefficient levels for UX dimensions (Eq (3.1)).

For a bag-of-words approach, we collected a set of words pertaining to the three dimensions of UX (usefulness, usability (ease of use), and affection) based on earlier research that presented word lists for the dimensions ^[25]. We retained 88 (usability), 62 (usefulness), and 84 (affection) words. Moreover, we conducted the stemming, lemmatization, pos-tagging and validation procedures on each comment. The ratio of each dimension was then calculated by applying a bag-of-words approach to each review comment (Eq (3.2)). For instance, if 12 words from the usability dimension were included in a specific review comment, which is organized by 50 words, then the usability level of the comment was computed as 24% (12/50).

This yields Eq (3.3), where $b_1$ is the intercept, $x_{usability}$, $x_{usefulness}$, and $x_{affection}$ are the regression coefficient levels for a bag-of-words approach.

4.   Results

4.1.   Descriptive analysis

Table 2 presents the descriptive results for the employed constructs. Moreover, we conducted two multiple regression analyses to investigate the effects of the sentiment dimensions and three UX dimensions of bag-of-word approaches on user satisfaction.

4.2.   Effects of sentiment dimension factors

The results of multiple regression analysis indicate that users' hedonic values (PE; $b_{PE}$ = 14.209, $\beta$ = 0.591, p $<$ 0.001) and confirmation (COM; $b_{COM}$ = 14.951, $\beta$ = 0.477, p $<$ 0.001) have notable positive effects on user satisfaction, with an $R^2$ value of 0.811, while cost values are negatively related to satisfaction (CO; $b_{CO}$ = -8.183, $\beta$ = -0.787, p $<$ 0.001). In addition, two factors, user burden values (NE; $b_{NE}$ = -1.083, $\beta$ = -0.056, p $<$ 0.001) and pragmatic values (WLH; $b_{WLH}$ = 0.570, $\beta$ = 0.052, p $<$ 0.001), are marginally associated with satisfaction.

4.3.   Effects of UX dimension factors from bag-of-word approach

The results of multiple regression analysis indicate that usability ($b_{usability}$ = 6.263, $\beta$ = 0.202, p $<$ 0.001) and usefulness ($b_{usefulness}$ = 7.645, $\beta$ = 0.300, p $<$ 0.001) have notable effects on user satisfaction with an $R^2$ value of 0.460. Moreover, affection has a significant positive association with satisfaction ($b_{affection}$ = 12.579, $\beta$ = 0.513, p $<$ 0.001).

4.4.   Additional correlation analysis of the sentiment and UX dimensions

We conducted additional correlation analysis of the sentiment and UX dimensions using two separate approaches. Table 3 shows the results of the analysis.

5.   Discussion

This study examines the effects of UX dimension factors on user satisfaction through two text analysis approaches on MFDSs. To this end, we compiled a dataset of 140,992 responses on three top-ranked MFDS applications from two widely used mobile application stores. Both sentiment analysis and bag-of-words approaches to user reviews were examined, and multiple regression analyses were conducted to identify which UX factors significantly affect satisfaction.

As shown using Eq (3.1), UX factors projected by sentiment analysis play dominant roles in determining user satisfaction with MFDSs. Of these factors, two positive ones (hedonic values and confirmation) and one negative one (cost values) are examined as key determinants of satisfaction. Moreover, the UX factors contributed 81.1% to the variance in satisfaction. In relation to Eq (3.3), usability, usefulness, and affection as computed by the bag-of-word approach play determining roles in user satisfaction: About 46.0% of the variance in satisfaction is contributed by usability, usefulness, and affection.

The main contributions of the current study are summarized as follows: First, two basic natural language processing approaches can be easily applied to address user satisfaction of MFDSs, one of the widely-used mobile applications. Second, user review comments are one of the valuable and useful resources to improve mobile applications. Third, UX dimensions are comprehensive and essential components in addressing MFDSs.

Considering that the variance of user satisfaction explained by sentiment analysis factors (81.1%) is significantly greater than with a bag-of-word approach (46.0%), we infer that sentiment analysis can be more effective than the bag-of-word approach in addressing user satisfaction with MFDSs. This difference may be due to the fact that sentiment analysis considers more dynamic constructs than the bag-of-word approach. The correlation analysis found several highly correlated factors (e.g., affection-cost values), indicating that the UX factors are complex and composite concepts that need to be analyzed when we seek to improve UX with mobile services. Thus, additional discussions are needed to provide a better understanding of the UX of a huge number of mobile services.

One of the intriguing practical findings of the current study is the marginal effects of user burden values on user satisfaction, which implies that users are more likely to concentrating on the positive aspects of mobile applications, rather than negative points or inconveniences of the applications. Thus, developers and researchers should aim at strengthening/maximizing the advantages and benefits of the applications. This study also considers both the applicability and availability of two natural language processing approaches to the use of user review comments as potential resources for improving user experiences and satisfaction with mobile applications. Mobile application developers can easily employ our approaches to improve their mobile applications.

Despite several significant findings of this study, some limitations remain. First, we employed two basic natural language processing methods. There thus might be better text processing methods for identifying key determinants of user satisfaction with MFDSs. Second, other comprehensive natural language processing approaches can provide the better understanding and examination in investigating user satisfaction (e.g., deep neural network ^[28]). Third, there can be other effective indicators of UX with MFDSs. Among a number of indicators, we employed two widely-employed approaches (Bag-Of-Words and sentimental dimensions) ^[29,30]. Fourth, we only considered user review comments written in English ^[31]. Thus, the results of the current study might be poorly generalizable to comments in low-resource languages. Future research can thus address the limitations of the current study.

Acknowledgments

This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ICAN (ICT Challenge and Advanced Network of HRD) program (IITP-2020-0-01816) supervised by the IITP (Institute of Information & Communications Technology Planning & Evaluation). This work was also supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)(NRF-2020R1C1C1004324).

Conflict of interest

The authors declare there is no conflict of interest.