Information hiding based on Augmented Reality

1.   Introduction

Augmented reality (AR) has drawn lots of attention in both research field and industry field. AR is a technique which integrates the live view of a camera with virtual digital objects, and displays the object in the real-world environment through the camera ^[1,2]. This paper presents a novel concept which combines augmented reality with information hiding. Steganography and watermark are two common ways to realize secret communication ^[3]. Both of these two methods conceal the secret information inside another carrier. The carrier is usually an image with high texture complexity which can minimize the distortion on the original image. Traditional steganographic algorithms typically select secret embedding position using human metrics and often suffer low embedding rate and high detection rate. More recently, there are researchers propose the method of using deep learning to realize image steganography. Deep learning has proven to be very effective among computer vision field. The emergence of the convolutional neural network gives machine the power to extract more useful image features ^[4]. Li et al. introduce the method of using generative adversarial network (GAN)-generated texture image as the cover image and hide the secret information into it ^[3]. By deploying the information hiding neural network on the mobile device and integrating the neural network with augmented reality, the secret information can be just hidden in the real world. The approach proposed in this paper utilizes the camera of the augmented reality device for detecting objects in the real world, and then hides the secret message according to the detected object. The communication parties can only conceal or reveal the secret using the system installed on the augmented reality device. By integrating augmented reality, the secret embedding rate can be improved dramatically in potential. The highlights of this paper are summarized as follows: The application of Augmented Reality in academic fields; the combination of Augmented Reality, information hiding and deep learning techniques; the extension of traditional information hiding matching policy; the practical application of coverless information hiding; a potential approach to improve the secret transmission rate.

This paper focuses on the application of augmented reality in conjunction with information hiding instead of the internal augmented reality concept. Section 2 summarizes some related researches about information hiding, deep learning, and augmented reality and puts forward the proposed augmented reality based information hiding architecture in Section 3. Section 4 shows the experimental results and Section 5 concludes this paper.

2.   Related work

The applications of augmented reality mostly focus on entertainment currently, the combination of augmented reality with academia is still in its infancy. Here we mainly consider some of the works related with steganography using deep learning and some convolutional neural network best practice since this is the principal part of the whole architecture.

Convolutional neural network is a revolutionary technology in machine learning field, it typically consists of one input layer and one output layer and includes multiple hidden layers between the input and output layer ^[5]. The convolutional layer applies a convolution kernel over the input tensor. The convolutional kernel is essentially a cross-correlation mathematical operation which can reduce huge amount of network parameters while achieve the same goal as the traditional fully connect network. Krizhevsky et al. first introduced deep convolutional neural network to classify the ImageNet dataset and outclassed the traditional classification methods ^[5]. A large number of novel applications using CNN springed up after their successful work. CNN is popular used in image classification, image object detection, image object segmentation, image style transfer and so on in computer vision field ^[6,7,8,9].

Rahim et al. introduce the method of using convolutional neural network based encoder-decoder networks for image steganography ^[10]. The encoder takes two images as input, one serves as cover image and the other serves as the secret message. The encoder encodes the secret message into the cover image and generates a stego image which resembles the cover image. The decoder accepts the stego image as input and restores the secret. Baluja also proposes a similar auto-encoder architecture as the one of Rehman et al., he points out that the secret image is scattered among all the image channel ^[11]. Li et al. introduce the method of using GAN-generated texture image as the cover image and hide the secret information into it ^[12]. The input noise to the generative network is mapped to a small patch chosen from the reference image. The synthesized texture image together with the secret message are sent to the information hiding network to generate the final stego image.

Augmented reality extends virtual scenes into real world by a series hardware and software collaboration. It presents the virtual object into the real-world environment through a specific hardware. In order to realize the most realistic scene, the hardware and the augmented reality algorithm has to work seamlessly. Also, the interaction between the virtual objects and user should be just like in the real-world environment. Augmented reality can be utilized in many areas such as gaming, education, commercial marketing, advertisement, scientific research and military operations etc ^{[13,14,15,16,17,18,19]}. In the past few years, AR was mostly achieved through expensive head-mounted sets such as AR goggles, projection-based camera systems, and AR-extended professional devices such as x-ray scanners ^{[13,14,16,18,20,21,22,23]}. Generally, these devices work by transforming the real-world scene to a live video feed and integrating the video feed with the computer-generated objects. Moreover, with the rapid development of modern mobile devices such as iPhone and iPad, it seems that the most promising hardware to achieve this immersive user experience is through these commonly used handheld mobile devices. Gervautz and Schmalstieg investigate the developments in handheld AR and put forward several challenges on implementing augmented reality on mobile devices ^[18]. They address the challenges from three aspects including localization tracking, interaction, and visualization and summarize several available solutions. Carmigniani et al. survey the technology, systems and applications in Augmented Reality and point out the possible future directions for AR applications ^[16]. Four types of applications are summarized which include advertising and commercial, entertainment and education, medical, and mobile application for iPhones. One of the contributions of their survey is that they focused more on the mobile augmented reality applications and addressed the privacy and ethical issues along the way. Kroeker mentioned that current AR applications continue to be much more compact and powerful, and many of which require nothing more than a current-generation smartphone ^[17]. However, the complete augmented reality infrastructure standard is hard to build and the acceptance of the applications could also become a serious problem due to the privacy and security concerns ^[24]. Gattullo et al. propose a method which can support technical writers in the creation of technical documentation suitable for augmented reality interfaces and also point out that AR is one of the most promising technology for technical manuals in the context of Industry 4.0 ^[25]. Kohn and Harborth analyze the application of utilizing AR in the manufacturing industry and conclude that augmented reality will surely change the manufacturing industry but still needs further improvements to achieve the transformation ^[26]. Grubert et al. present a novel Pervasive Augmented Reality concept which expects to improve the user experience based on particular context ^[27]. With the announce of the ARKit by Apple company, augmented reality can be easily implemented and taken to more users. ARKit manages most of the device motion tracking callbacks, camera scene capture and image scene processing. With the even more powerful compute capability on mobile devices, machine learning or even deep learning method is also possible. By combining the modern deep learning computer vision algorithms with augmented reality, the real-world environment can be used as a whole new information carrier.

Zhou et al. first proposed the concept of coverless information hiding and addressed that coverless information hiding methods take advantage of the independent carrier itself to represent the secret message ^[28]. Coverless information hiding typically works by designing a carrier-secret matching policy in order to minimize the secret detection rate from third party tools ^[29]. Augmented reality based information hiding approach follows the general idea of coverless information hiding methods but further extends its flexibility.

3.   Proposed architecture

The key of integrating augmented reality with information hiding lies in the secret mapping policy. The mapping policy follows the idea of representing one object in another form. For example, an image of the Statue of Liberty could actually represent the Mona Lisa portrait, a specific coordinate could actually be a bunch of hidden messages. Moreover, a specific image could combine a specific location information such as GPS and compass information to represent even more complicated things. The matched object here serves as the traditional information hiding key for deciphering the messages in this scheme. This kind of mapping strategy is simple and widely used at the early stage of information hiding. The augmented system detects the shared key objects and restores the message into the real-world environment. However, the biggest problem of this kind of mapping scheme lies in the limitation of the matching library. The matching database needs to be extended every time a new message is added which makes it less convenient and practical. Currently, some augmented reality frameworks do have the feature which allows multiple users to share one unique AR scene. This kind of feature provides a new scheme to transmit the secret messages between users but generally still requires the users to be near to each other.

As mentioned before, multiply secret keys can be combined in order to largely extend the secret matching library. Generally, more combination of keys would generate more mapping branches but also increase the complexity of the mapping library. In order to conquer the conundrum addressed by the simple key-secret mapping scheme, deep neural network based architecture can be a suitable solution. Deep neural network is good at mapping complex object features into different patterns and shows great robustness in various tasks. Deep neural network is usually trained with a specific dataset which contains the representative data of the corresponding task. In term of the secret matching scenario, the traditional matching policy can be implemented through the robust neural network and obtain the outcome in a general manner. Furthermore, various deep learning methods can be applied throughout the architecture in conjunction with the augmented reality. For instance, object detection methods can be applied to images so that the system could detect both the class and the location of certain objects existed in the image. The object detection rate can be used as an additional standard for secret matching such that the carrier is considered to be detected or the secret is allowed to be revealed only when the detection rate is above certain threshold. In order to enlarge the text library, natural language processing technique can be integrated in the secret revealing process for secret text restoration. Other advanced deep learning approaches such as image caption, image segmentation, and even image generation are also good candidates when it comes to image matching situation.

The proposed framework is shown in Figure 1 with the explanation and discussion down below. Two parties desire to transmit secret information between each other. Here A represents the message sender and B represents the secret receiver. A party wishes to hide secret messages inside another carrier and transmits the modified carrier to B side. The secret carrier should not look suspicious to another third party and the secret information should be able to be lossless restored when B extracts the information from the secret carrier. As Figure 1 shows, the secret messages are matched to different target images or different objects beforehand. Then the matched objects are sent to a secret hiding model for further process. Party A is in charge of the secret matching and the information hiding model is shared between two parties. For instance, suppose the secret message is matched to the real-world dog category, when A party tries to send message to B, he just needs to inform B what object he should look for in order to decode the secret message. The matched real-world dog category serves as the traditional information hiding key for deciphering the messages in this context. B can use the AR based information hiding system to find certain dog object and decode the secret message to the real world after receiving the secret key. Figure 2 depicts the mapping strategy in detail.

Figure 1.  Information hiding based on deep learning with augmented reality.

DownLoad: Full-Size Img PowerPoint

As Figure 2 shows, most object detection methods can detect certain object which belongs to specific class. The secret communication parties can transmit a concrete coordinate, a specific direction together with the target objects to enhance the secret communication. The secret variety and secrecy can be greatly enhanced by combining different elements together. The authorized receiver can point the augmented reality system to a specific direction and move to a specific location to decode the message.

Figure 2.  Secret key matching scheme.

DownLoad: Full-Size Img PowerPoint

The proposed mapping architecture can improve the secret embedding rate dramatically in potential since the target object can correspond to any kind of message. The secret communication parties usually share the secret messages via binary, text or image formats. This is mainly constrained by the secret embedding rate and detection rate. In general, by increasing the secret embedding rate, the secret detection rate would increase dramatically simultaneously. However, the augmented reality technique can expand the carrier into other formats while still maintaining high secret embedding rate. For instance, three-dimensional (3D) models and even videos can be transmitted as the original secret messages which would be a huge improvement compared to conventional information hiding methods. Figure 3 demonstrates a feasible framework which utilizes deep learning technique to realize three-dimensional model based information hiding via augmented reality platform. The video frame or picture is first segmented using object detection and object segmentation methods. Then the segmented object is further processed to achieve secret transmission. Three schemes are proposed regarding to the secret embedding process. The first scheme attempts to hide the secret message in the two-dimensional (2D) object beforehand and then transform the stego object into three-dimensional object. The second attempts to hide the secret message during the object transformation process by simultaneously learning an information hiding algorithm. The last scheme tries to embed the secret messages inside the transformed 3D object afterwards by modifying the 3D point cloud information. The transformation process is achieved by a neural network which learns a 3D point cloud generation algorithm that maps the 2D coordinates into 3D space ^{[30,31,32,33,34,35,36,37,38]}.

Figure 3.  Three-dimensional model based information hiding via augmented reality.

DownLoad: Full-Size Img PowerPoint

4.   Experimental results

A system prototype is developed according to the proposed architecture using iOS SDK. The system is designed under the image matching logic which corresponds the secret message to concrete real-world scene objects. It recognizes the specific image or object and then presents the corresponding secret message in the real-world environment. In this prototype system, the image to be recognized serves as the carrier of the secret message and is mapped to the designated secret message. The system can only be allowed to present the secret message when it detects the corresponding mapping image or object inside the real-world scene. Furthermore, the system is also implemented to show the secret message according to the specific object class and the detection rate of specific object class. The object detection method used in the system is implemented using You Only Look Once (YOLO)v3 which is a novel object detection method based on deep convolutional neural network ^[6]. YOLO can achieve a relative high detection accuracy while remaining a high detection speed which is suitable for real-time applications. The YOLO network used in this system is derived from the original network with some modifications in order to run on mobile devices and is trained using COCO2017 dataset which contains 80 categories of different objects. The programming language includes Swift, Python, C and C++, the YOLO model is trained using one GTX 1080 GPU and the system is tested on the Apple iPhone 7 plus.

The execution flow of the prototype system is shown in Figure 4. The secret matching policy is the combination of image recognition, object detection and orientation configuration. The communication parties share the target image, the desired object along with the detection threshold and certain orientation information. Figure 5 shows one scenario where the AR system works by first detecting a laptop object and a target image and is only allowed to reveal the secret message when the detection rate is above 80 percent and the target image is recognized. The secret message can also be matched to other objects and updated correspondingly.

Figure 4.  System execution flow.

DownLoad: Full-Size Img PowerPoint

Figure 5.  System interface.

DownLoad: Full-Size Img PowerPoint

5.   Conclusion

This paper proposes a novel scheme which utilizes the augmented reality to achieve secret communication. The general idea for utilizing augmented reality to do secret communication lies in the secret matching policy which derives from coverless information hiding. The secret message is mapped to secret keys and can only be revealed when the corresponding key object is detected by the AR system. The mapping scheme can either derive from a concrete matching database or base on a certain deep learning model. The former only works when the mapping scheme is inside the database which is less flexible while the latter is more complicated to design but has fewer limitations. Several deep-learning based mapping schemes are mentioned in this paper including object detection, image caption, and natural language processing etc. The deep learning model is trained on the local server and the trained inference model is optimized to be deployed on the mobile device. The experimental AR system automatically detects the carrier and corresponds the secret messages to different objects. The secret message can only be revealed when the carrier object is detected with a certain confidence score. Clearly, information hiding combined with augmented reality is still in its infancy, the design of the secret mapping model is the crucial part of this kind of approach and is worthy of further study.

Acknowledgments

This work is supported by the National Key R & D Program of China under grant 2018YFB1003205; by the National Natural Science Foundation of China under grant U1836208, U1536206, U1836110, 61602253, 61672294; by the Jiangsu Basic Research Programs-Natural Science Foundation under grant numbers BK20181407; by the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD) fund; by the Collaborative Innovation Center of Atmospheric Environment and Equipment Technology (CICAEET) fund, China.

Conflict of interest

The authors declare that there are no actual or potential conflicts of interest in relation to this article.