A deep learning-based medication behavior monitoring system

1.   Introduction

The convergence of the internet of things (IoT) and deep learning is an emerging technology in various fields of research, as illustrated in Figure 1 ^[1]. Intelligent services are being developed to collect human activity data through diverse sensors of IoT devices and quickly process and apply large volumes of data through deep learning to provide convenient user experiences and improve overall system performances. The healthcare sector is most interested in these convergence technologies.

The application of IoT technology to medical systems can improve patient treatment processes and enable the collection of large amounts of medical data and patient's disease data through various sensors. Many studies are underway to develop intelligent healthcare systems, as deep learning can be used to rapidly process large amounts of data to solve various problems such as a remote treatment and a rapid recovery of the patient. These converged healthcare systems based on the IoT and deep learning focus primarily on providing convenient and intelligent IT-based healthcare services to patients ^[2]. In addition, it concentrates on making the patient's quick and comfortable treatment possible.

The main concern of intelligent health IT services is the periodic remote patient monitoring healthcare of elderly people suffering from chronic diseases or patients with mobility difficulties due to aging. These services continuously help patients take medications to maintain health, and to alleviate disease in a comfortable place such as home. Consistent medication is the most basic disease treatment method, and to keep one's medication cycle can have excellent therapeutic effects. However, in the COVID-19 era, senior citizens with little experience with IT services cannot visit hospitals, making it difficult to treat them. In addition, various medical systems available only within the hospital are not actively utilized due to the limited availability of hospitals. Doctors' face-to-face interaction with patients is affected, making treatment difficult. Taking medication without a doctor's supervision can lead to health deterioration or death. In addition, if the medication is not taken in time, it causes the patient's disease improvement rate to slow down. To alleviate this problem, monitorable healthcare IT services are urgently needed in places other than hospitals ^[3]. One of the previous solutions is to use wearable devices to detect medication behaviors of patients. A patient needs to equip the wearable device such as smart watch to monitor diverse behaviors. However, it causes inconvenience while equipping the devices. In addition, various sensors of the wearable devices have their own measurement problems that incur inconsistency due to errors of measured values.

We devise a smart medication behavior monitoring system (MBMS) that leverages the IoT and deep learning to effectively detect patients' medication taking behavior without any inconvenience of patients. Thus, doctors can efficiently perform regular health check-ups in remote environment and provide stable health monitoring ^[4]. The basic operations of MBMS are as follows. First, the doctor provides the accurate medication interval cycle to patients who need to take the medication periodically. The patient registers the time to take the medication in MBMS.

MBMS notifies an alarm when it is time to take medicine. This can take place on the patient's cell phone if desired alarm. When a patient hears an alarm and approaches an MBMS device, its camera records video based on the movement detected around motion sensing sensors and stores the patient's medication behavior. Based on OpenPose, our human activity recognition component in the system analyzes the patient's behaviors of recorded images and identifies the act of raising the arm to drink water. After detecting the action, MBMS drops the right amount of medication to easily provide medication service to the patient. The weight of the dropped medication is measured and checked whether the patient takes the medication or not through change in weight of medication container. If the patient's behavior matches the act of taking water, and the weight of the medication converges to zero, then it is considered that the medication has been taken on time. MBMS sends the weekly results of medication behavior monitoring to the doctor on a set day, which allows the monitoring of the patient's medication behavior and health status. At this time, the MQTT protocol can guarantee reliable transmissions so that the doctor can conduct an accurate examination without omission of the patient's medication status data. It also provides information to help manage a patient's disease, such as food that should or should not be eaten, through a monitor connected to an MBMS device, and to enable periodic consultation with a physician.

The proposed system prevents unnecessary overmedication and side effects, because it provides a fixed amount of medication by checking the state of the user's behavior at a given time. Users can experience excellent medication effects and conveniently manage their health at home through appropriate medication cycles. Doctors can effectively store and check patient data ^[5]. It can reduce patient contact and reduce COVID-19 infections through effective online health consultation.

Our main contributions are summarized as follows.

● We propose a periodic non-face-to-face health management system using remote monitoring without any inconvenience wearable devices.

● We devise an automated medication supply device improving user experiences with intelligent algorithm via various sensors of IoT devices and deep learning.

● We develop the medical information notification system employing a reliable MQTT protocol to accurately transmit medical information between patients and doctors.

● We propose a medical system, suitable for the COVID-19 era, through a monitoring-based non-face-to-face treatment method and improve the efficiency of the current medical system.

The rest of this article is organized as follows. Section 2 explains related work on the importance of taking medication, medication taking management systems, and artificial intelligence convergence in medical systems. We explain the design method and principle of operation of the hardware and software of the MBMS system in section 3. We present the results of experiments in section 4. Section 5 provides our conclusions and suggested future work.

2.   Related works

We explain the importance of correct medication use, the existing medication management and patient monitoring system. In addition, we explore a medical system using artificial intelligence.

2.1.   A study on the importance of taking medication

Steady medication use is important in the treatment of disease. A patient needs to take medication on time to enhance the performance of medication therapy. However, taking medication on time is difficult for busy people and abnormal patterns of visiting hospitals can agammaavate a fast disease recovery. Thus, increasing medication spending each year has led to an increase in medication consumption ^[6], indicating an increase in actual individuals taking medication. Taking medication to temporarily relieve the disease does not guarantee a good therapeutic effect. It means that it is important to prevent abuse or misuse of medication, and to prevent side effects according to doctor's prescription.

In medical studies, the aim of a new system is to address medication misuse problems through the management of patients' medication, and to ensure correct medication intake ^[7,8]. Convergence research in healthcare and engineering allows people to search for correct medication information and manage medication ^[9].

To handle these problems, we design MBMS that supports on-time medication service to enhance overall treatment effects. Modern people who have no enough time to visit hospitals can experience convenient remote service with a positive treatment effect by steadily taking medication on time.

2.2.   A study on medication taking management system

Most developed countries focus on medication-related systems ^[10] and are working to maximize the effectiveness of medication therapy. They use methods to alert patients when they need to take medication or for their doctors or family members to monitor their condition. In addition, several previous studies proposed a medication behavior monitoring system with wearable devices and a medical system based on artificial intelligence to check the medication status.

2.2.1.   Management of medication cycle

Taking medication on time is important to maximize the effectiveness of gradual treatment of the disease. For people in areas where immediate medical service is difficult, there is a way to improve the healthcare system by providing alarms through the application to remotely manage the medication cycle and the patient's recovery process ^[11]. Physicians can consult with patients with recorded content and use it to remotely feedback their status. According to a study on the effect of improving user medication-taking patterns based on how penalties are provided when not taking the medication, penalty and reward approaches can have a better effect than simply providing a notification ^[12]. However, the drawback of this study is that it is not known whether participants record their behaviors only when they actually take the medication. In addition, since the medication management robot manages the use of medications on behalf of the elderly with limited mobility, it can check the side effects and schedules ^[13]. Our proposed system continuously monitors the medication behaviors and schedules. Since MBMS captures motion images based on the patient's motion, it accurately detects whether the patient actually takes medication or not. If the patient has not taken the medication, the information can be immediately sent to the doctor to assist in contacting the patient directly. Thus, patients cannot forget to take the medication and improve their disease recovery.

2.2.2.   Real-time patient monitoring

Steady efforts to take medication are emphasized for quick treatment and recovery. IoT devices transmit them wirelessly to mobile devices to enable doctors to monitor medication use ^[14]. In addition, a wireless sensor network system ^[15] and framework ^[16] are introduced to enable the elderly to continuously monitor for 24 hours for ease of use in their daily lives so that they can be transmitted to their neighbors, such as nurses and relatives, to understand the situation. Because our proposed system does not require additional equipment to be installed, the patient's behavior can be monitored without any inconvenience that patients experience when wearing wearable devices.

2.3.   A study on the artificial intelligence convergence medical system

Research has been conducted to develop systems through the convergence of medical systems with artificial intelligence. Research is also underway to process images through artificial intelligence to efficiently manage medication use. In addition, several researches are being conducted to analyze behavioral patterns based on wearable devices to which various machine learning techniques are applied.

A study applies calculations to various pill images to extract and classify pill imprinted information ^[17] or to determine patients' symptoms by applying logistic regression through natural language processing ^[18]. In previous studies, machine learning techniques such as random forest, gradient-boosted tree and logistic regression were applied. Such techniques have attempted to predict a set of behavioral patterns applied to taking medication by predicting a series of behaviors ^[19,20]. Our proposed scheme analyzes both the medication status of the proposed system and the medication behaviors with logistic regression.

In addition, convolutional neural networks (CNNs) with architecture are used, as illustrated in Figure 2, and deep learning methods such as ResNet and GoogleNet are being studied ^[21,22] to create pill-aware systems ^[23], or to build new hardware to wirelessly communicate information to caregivers. We adopt the image processing method for the patient's motion using the CNN-based OpenPose algorithm.

3.   Design of MBMS

3.1.   MBMS hardware

A model of the proposed MBMS device is illustrated in Figure 3. It is of a size that can contain all prescribed medication, with eight compartments at a tilt of 30° so that medication can fall out normally. The user stores needed medication in a box in MBMS devices.

MBMS device controls the pill box by moving the central and rotary motors at regular intervals. It moves one space at a time according to the pill delivery time. In order to drop the pre-loaded pill down, when the set time is reached, the sub-motor connected to the central motor rotates to open the box containing the pill. When the pill box opens and the pill falls down, the motion sensor detects the movement of the patient approaching to take the medication. If the device repeats this process 7 times, it reaches to the last 8^th pill box. At that time, it speeds up and moves one space larger set it to its initial state.

3.2.   MBMS software

The software structure of the MBMS is illustrated in Figure 4. Mobile devices can be securely connected to the MBMS device. They immediately can receive regular alarms and exchange medication status data. Once the mobile device provides medication information, it binds I/O events that occurred through motion detection sensors connected to the MBMS and starts recording video. The Python-based OpenPose API is applied to captured image data obtained through the camera sensor. It identifies the human activity drinking water and stores the medication status in the database.

3.3.   Principle of MBMS operation

The MQTT protocol is applied, as illustrated in Figure 5, to establish a system that can communicate seamlessly with MBMS devices with low power, so that data can be received from small sensors and devices and transmitted to IoT devices and mobile terminals.

The MQTT protocol controls small sensors and collects their information. MQTT has a Pub/Sub model that publishes topics in Publisher and subscribes to them in Subscribers. The MQTT broker acts as a repeater between Publisher and Subscriber to notify alarms and facilitate subscribed activities. The MQTT protocol stores information obtained through MBMS sensors and enables data communication between mobile handsets and MBMS.

Our proposed MQTT topic is alarms. If a user sets an alarm to be received by a mobile device, the MQTT protocol provides the alarm to the user. Real-time video recording begins when a user who has checked a time to take medication with an MBMS device is notified by the alarm. It checks whether the act of drinking water matches the images recorded by the camera of the MBMS device in real time. Drinking water is one of the most representative acts in medication use. Therefore, we designed the system to detect the behavior of drinking water via OpenPose, which works as illustrated in Figure 6 ^[24].

MBMS provides convenience to patients because it enables them to be monitored without equipping any additional wearable devices. Accurate monitoring is possible through alarms, motion detection, medication behavior matching, and detection of changes in the weight of provided pills. The operation of MBMS is illustrated in Figure 7. First, the mobile device of doctors periodically notifies patients of the appropriate pill cycle. Patients set the time suggested by the doctor on the MBMS device. If the patients need to receive various alarms from the mobile device, they set it separately. Patients receive alarms from MBMS devices or mobile devices at their convenience. When a patient approaches the device and turns off the alarm, the MBMS device detects the movement of a person and starts recording in real time. The alarm repeats until the patient turns off. After turning off the alarm, if the patient does not take medication, the procedure repeats again after 10 minutes. OpenPose is applied to the image of a person in the video to determine whether the patient is drinking water. If the image of drinking water is recognized, MBMS detects whether the weight of the dropped medication has changed.

The patient's medication is stored in the database every week and periodically delivered to the doctor. The doctor can check the received information about the case when the patient does not turn off the alarm for 10 minutes, the drinking behavior is not recognized, or the weight of the provided medication has not changed.

4.   Performance evaluation

We evaluate whether all medication in the box is provided after the alarm goes off, and whether motion is correctly recognized. Experiments were conducted to ensure the correct frequency of medication.

4.1.   MQTT protocol accuracy evaluation

We implement the MQTT protocol for a reliable data transmission among devices. As shown in Table 1, we conducted experiments on the transfer rate, transfer loss, and delay of MQTT. We measure one-hop delay as a function of various data sizes such as 100 bytes, 500 bytes and 1500 bytes considering maximum transmission unit (MTU). MTU is the maximum transmission unit of a computer network and is applied to a communication protocol in the Internet environment. In general, the smaller the MTU value, the lower the maximum transmission delay. We designed the MQTT protocol applied to MBMS to not exceed 1500 bytes in consideration of the MTU to reduce unnecessary overhead. Thus, network delay is reduced and communication errors are minimized.

The results show that the overall loss is less than 1% even though the transfer loss occurs when MBMS communicates with various mobile devices. In case of 1500 bytes data transmission, the one-hop delay is around 336µs and 367µs, respectively. It means that the system can support a reliable and fast notification service between the patients and doctors.

4.2.   MBMS performance evaluation

In order to evaluate the performance of MBMS, we measure diverse factors such as the delivery speed of pills, the accuracy of gesture and human activity recognition. We perform various experiments as a function of the shape of a pill and the distance from the body part. Table 2 shows the experimental results of analyzing the speed according to the shape of the pills when providing them to patients. We show the supply speed and accuracy based on pill shape. We conducted 100 experiments to determine the speed and accuracy of dropping pills stored in each compartment of the MBMS, according to pill shape.

Most pills are supplied within an average of 1 second. Most are normally provided except for flat-shaped pills. A flat pill may not be normally supplied when only one pill is in the device, but in combination with several pills, it is served normally. And then, the patient is considered to have taken the medication only after performing the actions shown in Figure 8. We intensively analyze the face and arm joints so that OpenPose can quickly recognize them. The angle and distance of the camera are important to properly recognize the motion shown in Figure 8.

It is important for MBMS to accurately classify and recognize the patient's status as well as to identify the motion correctly. In order to compare the classification accuracy with other existing approaches, we conducted an experiment to recognize human motions in three categories: inactive state, active state, drinking water state. Figure 9 shows the classification accuracy as a function of these three states.

We propose the combination of motion sensor and OpenPose. In that case, the recognition rate of drinking water is around 94%. In addition, the recognition rates of 93% and 99% are shown when a person is active and inactive, respectively. In the case of using only the motion sensor, the recognition rate in the inactive state is 95%, but it can be seen that the motion of drinking water was not recognized at all. In addition, when using only OpenPose, the active state is recognized as 71%, but the recognition rate for drinking water is 53%, indicating that the performance is degraded. Thus, we proved that the best performance was achieved when our system is applied.

The angle of the camera mounted on the MBMS was tested by dividing the experimental range into face, upper body, and whole body. The appropriate distance to accurately include the target range is classified as shown in Table 3.

Judging the accuracy of OpenPose recognition depending on the body part of the person reflected in the camera, the highest accuracy rate was 98% when the upper body was set to be visible and the distance between the MBMS device and the user was between 0.6 and 1.3 m. If the entire body was set to be visible, other actions interfered with recognition, resulting in 91% accuracy, and if only the face was visible, hand motion was difficult to measure, and accuracy was the least.

4.3.   Frequency of medication behavior on time

We compared three ways to take medication on time: taking medication without a device, using small portions from plastic containers on a weekly basis, and using an MBMS device. The experimental times were set at 7 a.m., 1 p.m., and 7 p.m., i.e., morning, lunch, and evening, and the experiment was conducted based on the act of the user taking medication standing in front of the device. The results of the experiment are shown in Figure 10.

When MBMS devices were used, performance was best, at 99% in the morning, 98% in the day, and 99% in the evening. The results of this experiment show that when administering medication using an MBMS device, the user does not forget to take medication. Periodic administration of medication was found to be difficult with the Medication Sachet, and highly accurate using the Medication Organizer, but less accurate during the busy lunch hour.

We solved the problem of periodically taking medication using MBMS, whose devices can be monitored by doctors with high accuracy when users take medication at a distance of 0.6 to 1.3 m from the device. The user's medication cycle is managed with alarms, making it easier and more convenient to take medication. In addition, we conducted an experiment for one month to analyze the change in the actual medication intake rate of patients in various scenarios. Figure 11 illustrates the actual medication intake rate as a function of four approaches. The typical approach is the basic memory-based method without any devices and algorithms.

In the case of MBMS, it was found that 98% of the patients took the medication without missing the medication taking time. It can be seen that the typical memory-based approach cannot guarantee the correct medication intake. The wearable device-based medication-taking method showed an accuracy of 60-80%, and the mobile app-based method became more familiar as time passed, indicating that the patient delayed taking the medication.

5.   Conclusions

We summarize the MBMS process and suggest future work. We proposed a system to periodically monitor the state of a patient's medication by detecting medication behavior. First, the doctor provides the medication frequency to the patient, and the patient directly sets the alarm time and pills on the device. When it is time to take the medication, an alarm allows the user to recognize the medication time. After that, a patient moves in front of the device to take medication. At this point, the device recognizes the patient's motion and matches the action of drinking water. Finally, if MBMS analyzes the joint of the patient's arm in the image, it identifies the patient's medication behaviors. These operations are repeated for a week and MBMS stores patient's actions. The stored information accumulated over a week is transmitted to the doctor for future feedback on the patient's medication behaviors. The proposed approach utilizes IoT devices and OpenPose libraries to enable monitoring without wearing additional wearable devices. In addition, it gave a penalty effect so that medications can be taken at the correct time. Therefore, it proved the effect of improving medication treatment compared to the system that keeps the medication administration cycle only with the existing alarm. Especially, our messaging protocol improves the reliability of the system mechanism by implementing a reliable data transmission through remote monitoring based on the MQTT protocol so that patients can be monitored without missing patient data. Our experiments show that MBMS typically delivers medications on time, regardless of their shapes, with an accuracy of about 98.3%. It has been shown that when a patient takes a medication using MBMS, the accuracy of taking the medication consistently is more than 10% higher than when a wearable device is used. From the experimental results, we can see that the intelligent monitoring method of the IoT device and deep learning technologies employing human activity recognition is superior to the existing wearable device-based monitoring method. In the future, we plan to build an MBMS that enhances the security of IoT devices and provides care by periodically transmitting a patient's state to relatives. In addition, we will strengthen the deep learning model to make it more accurate.

Acknowledgments

This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. NRF-2018R1C1B5038818).

Conflict of interests

All authors declare no conflicts of interest in this paper.