The Virtual reality electrical substation field trip: Exploring student perceptions and cognitive learning

1.   Introduction

Virtual reality (VR) stems from and is associated with decades of research and innovation into virtual worlds, bringing people together for social, educational and business purposes, immersing users in simulated computer generated 3D environments [1]. These virtual environments can be configured and designed to meet social, pedagogical and corporate requirements, and are limited by the creativity and technical capability of the 3D designer and available technology. Virtual reality evolves from virtual worlds by changing the user's perspective to be fully immersed into the 3D environment by using special apparatus such as a head mounted display (HMD) to provide a life-like experience. VR can be used to simulate real world situations that may otherwise be out of reach of the user, providing a quality learning experience that is cost effective, time efficient, versatile and safe [2]. The benefits of VR have been amplified in 2020 with the world facing challenges associated with remote learning and social distancing caused by preventative measures to stop the spread of COVID19.

The University of Wollongong (UOW) provides students with a range of work integrated learning (WIL) opportunities scattered across their degree. These WIL opportunities go beyond traditional work placements, integrating career based learning into courses with the goal of bringing academia and industry closer together for the benefit of the student learning experience [3]. This is accomplished via a range of methods including guest lectures, online collaboration sessions, field visits and project-based learning. An annual activity for power engineering courses was a field visit to an electrical substation and a gas fired power station. Enabling such activities requires funding, support and cooperation from an industry partner, substantial logistical workload and safety challenges. Moreover, if students missed the opportunity due to illness or other time related conflicts the experience could not be easily replicated. Restrictions resulting from COVID19 meant that this field trip could not go ahead, bringing forward plans to create alternative immersive learning experiences. This paper explores the research question 'Can a VR replica of an electrical substation provide an equal or better cognitive learning and student experience compared to traditional methods?'. The student experience was compared to the traditional field trip and learning was compared to traditional learning via PowerPoint.

2.   Related Literature

The idea of using virtual reality to provide some form of electrical substation training is not new, and examples that illustrate the benefits and possibility of such a system can be found at least two decades ago in work presented in [4]. With technological advances including in computer graphics processing, peripherals such as HMD and joysticks, and software to design 3D environments, the line between virtual and real is increasingly becoming blurred. The work presented in [5] provides evidence of the realism, training and safety benefits capable with VR.

VR has opened new avenues in the way we can relate to the design and delivery of learning, and the feeling of 'presence' plays an important role in acceptance [6]. Educators from a very diverse range of fields, while to a lesser degree in engineering, have been exploring and collecting evidence of the educational benefits of VR [7]. A review of VR literature in [8] discovered benefits including enhanced spatial knowledge representation, greater opportunities for experiential learning, increased motivation and engagement, improved contextualization of learning and richer/more effective collaborative learning. However, when it comes to engineering and more specifically to electrical substations or power systems [4,5,9,10,11] the greatest focus on many VR innovations is on the technology and implementation. This is also the case with VR field trips [12]. Therefore, a major limitation of such studies is the depth of evidence provided in relation to learning, resulting in the focus of this study. The more understanding that we gain about power systems based VR learning and the student experience, the better incremental pedagogical improvements can be had, such as those implemented by Grivokostopoulou and Paraskevas [13].

Some engineering-based studies that have explored learning, such as the work of Fogarty and McCormick [14] in a structural engineering context, used pre- and post- tests to find that VR effectively aids students' understanding of complex spatial arrangements. A similar study within applied sciences by Tarng and Lee [15] also found learning benefits using pre- and post- tests comparing traditional and VR learning environments. These studies had relatively small sample sizes, so the need for further evidence collection is warranted.

The VR immersive experience is only as good as the software that is being used. The software has a very important role when it comes to the performance of the system. Some key factors influence the overall success of a VR simulation and need to be implemented correctly to allow users to have an immersive experience. These key factors include: interaction cycle [16] - the interaction between the user and 3D objects within a VR simulation and includes, but not limited to, picking up objects, flipping switches, and using pointers; object rendering [17] - this affects the overall performance of the VR simulation; and, lighting - adverse lighting can result in strained eyes, simulation sickness and user disorientation. Therefore, as these technologies have improved over time from 1999 [4] to 2020 [18], we need to continuously reevaluate the learning success of VR implementations.

3.   Electrical Substation Simulation Design

Some of the key benefits of taking students on a field trip to an electrical substation include understanding the station layout, understanding the purpose of the main components, and increasing motivation. Using a VR system, several extra learning benefits can be added, those that would more likely be taught via more traditional methods such as through a lecture. This includes the fundamental theories behind the components, the ability to read and understand line diagrams, and to build intuitive cognition on electrical substations. Therefore, the six identified benefits form the learning objectives used to drive the design of the VR electrical substation setting.

Design considerations for the VR substation simulation included selecting the VR headset, the software used to create the VR simulation, and the simulation design itself. The Oculus Rift was chosen as the headset due to its ability to meet standard VR requirements, which includes its 1080x1200 resolution running at 90 Hz [19]. The software selected to create the VR simulation is Unity, a cross-platform game engine developed by Unity Technologies. This multiplatform game engine allowed for VR integration with a high level of flexibility, such as efficient rendering of 2D and 3D scenes and an extensive asset store of 3D components that could be used.

The VR environment was created by implementing 3D models into the arrangement of an electrical substation. The layout was modelled off a 33-kV/11-kV substation line diagram. A screenshot of the simulation environment is shown in Figure 1. The models used in the simulation are from the Unity Asset Store and are accurate representations of real electrical substation components. Key design decisions in terms of layout and programming include guided, interaction and usability. The simulation was guided, showing the user where to go with relevant signage. This ensured that all of the content within the simulation was reviewed by the user, and promoted self-directed learning to allow students to make their own choices, as seen in [20].

Figure 1.  Distance view of simulation environment.

DownLoad: Full-Size Img PowerPoint

Each component within the simulation could be interacted with by the user. This allowed the user to ascertain information on the specific component, how it works and the mechanics of its operation. Knowledge was developed by the user through observation, interaction and reading the information of each component as they traversed at their own pace throughout the simulated substation environment. Reinforcement of knowledge was developed through an interactive activity by providing the user with substation drawings which they had to match to the corresponding component using the provided descriptions. The user needed to grab and place the drawings within the silhouettes provided next to each component model. Users were provided immediate feedback allowing them to learn and address misconceptions as needed. When all the drawings were placed correctly, a message was shown to the user. An example of this is shown in Figure 2.

Figure 2.  Drawing in correct location with correct text.

DownLoad: Full-Size Img PowerPoint

The user could move around at any time with smooth locomotion using the joystick on the Oculus touch controllers. This gave the user the freedom they would have if they were to visit a real electrical substation. Substations deal with high voltages and have potentially hazardous components. The VR substation simulates the experience without the need of supervision, or the safety risks involved. While the design of the simulation environment contains nothing breakthrough and is not as visually stunning as the substation designed by Tanaka and Paludo [5], it is the missing gap of understanding the student learning experience that is the focus of this study.

4.   Experiments

4.1.   Overview

To answer the main research question, two sub questions were devised with one focused on the student experience and the other on student learning. The experiments were undertaken during 2020 and as such the researchers were faced with many COVID related limitations that are addressed below. This study was conducted with human ethics research approval number 2020/364 granted at the University of Wollongong. Written consent was obtained from participants. No research funding was obtained for this study.

4.1.1.   Student Experience: Field Trip vs VR

A field trip is very different to the standard learning environment at university. The goal of a field trip is not to provide students with more PowerPoint presentations, but to engage all their senses with the working environment, the people, the technology, and help them build connections between what they do in the classroom and what is done in industry. Measuring these non-cognitive learning benefits is hard and can be overlooked in engineering and is generally explored via survey based instruments [21]. These survey-based instruments try to capture students' perceptions of their experience. Therefore, to answer the primary research question, a sub question 'Do students perceive that a VR replica of an electrical substation provides an equal or better learning experience compared to the traditional field trip?' was explored. This was accomplished via an experimental study comparing student perceptions between students that had attended the substation field trip previously (the control group) and students that had undertaken the VR experience (experimental group).

A ten question experience questionnaire that built upon the work in [15] was used to evaluate students perceptions of their experience. The questionnaire was distributed to participants using Survey Monkey. The results of the of the questionnaire was collected using a five-point Likert scale, ranging from strongly agree, which had a point value of 5, to strongly disagree, which had a point value of 1. The questions were divided into two categories, learning contents and operational experience. Questions one to five were associated with learning contents as they primarily focused on the teaching capabilities of the activity, and questions six to ten were associated with operational experience as they focused on the usability of the activity.

Students that participated in the control and experimental groups were aligned to the electrical engineering degree. Due to COVID no field trip was run, resulting in control group participants that had attended the field trip in the previous year. A total of 10 students provided consent to participate in the questionnaire. For the experimental group, due to COVID it was difficult to create an environment that encouraged large participation. Again, 10 students provided consent to participate in the VR experience. The participants used the VR simulation for a set amount of time (15 minutes), then were required to complete the same questionnaire as the control group. The results of the questionnaire represent the possible benefits of visiting an electrical substation as well as its limitations. These benefits and limitations were then compared to the VR simulation to determine the difference between the two forms in terms of perceived learning.

Several limitations are associated with this research component. Firstly, is that it focuses on the attitudes of students as opposed to their learning achievement. This was deemed acceptable because traditionally students have never been explicitly assessed on the field trip. Secondly, the questionnaire was the only form of data collection, limiting the scope. Thirdly, COVID resulted in very small participation rates, reducing the likelihood of obtaining statistically significantly results. Another potential limitation is the time spent within each teaching activity. Those who visited the electrical substation would have had more exposure to the activity than those within the VR simulation, impacting the reliability of the results. Those that visited the site would have spent at least an hour, those participating via VR were restricted to 15mins, but may have otherwise stayed longer in a normal setting. Control group participants were relying on the memory of a previous experience, while the experimental participants were providing immediate feedback.

4.1.2.   Student Learning: Traditional vs VR

The first research sub question while providing an insight into student perceptions of their experiences, provided very little data on learning. Therefore, a more comprehensive experiment was undertaken to evaluate the effectiveness of the virtual reality simulation by comparing it with traditional learning materials. The control group learned using traditional learning materials, and the experimental group learned using the virtual reality simulation. This was designed to answer the research sub question 'Does a VR replica of an electrical substation provides an equal or better cognitive learning experience compared to the traditional methods?'. While interactive learning experiences provides students with multi-domain learning experiences [21], the focus of this study was only on cognitive learning. On the field trip students were primarily observing and hence, most probably not gaining substantial psychomotor development from the experience. However, students spend substantial time engaging with the industry members and other students and build confidence, and as such not comparing affective learning is a major limitation in comparing between the two.

Cognitive learning had never been explicitly assessed on the field trip and due to COVID no new data could be collected. The alternative approach was to compare VR learning with PowerPoint material they could receive during a lecture. This would allow for some understanding between the learning differences between 2D traditional lecture styled model representations and 3D representations as would be experienced on a field trip or in VR (with the only data being collected being through VR). While the many limitations of this approach are obvious, the lack of VR learning evidence in this space as outlined earlier provided justification for the authors to collect this data to develop an informed decision on the future use of this technology should COVID restrictions continue.

The experiment commenced with a 15-minute pre-test done to evaluate the background knowledge of the participants. This was then followed by the teaching activity for each group, either being traditional learning materials (PowerPoint slides) or the virtual reality simulation. This section was 25-minutes in duration to ensure adequate time was provided to the participants to learn the materials given the amount of information shown. This was then followed by a 15-minute post-test to evaluate the retention of the allocated learning material. Example test questions included: What is a typical shape of a busbar? What is the purpose of a circuit breaker? What are the major components within a transformer? The following image refers to which component? Electrical engineering students were targeted, and a snowballing recruitment method was used for this experiment. Participants forwarded the invitation of the study to other potential participants, who then decided if they would like to be involved or not. A total of 15 students participated in the control group and 15 in the experimental group. The traditional learning materials consisted of PowerPoint slides comparable to what they would have received in a lecture and what was also delivered via VR in 3D. The control group learned by reading from the PowerPoint slides while the experimental group learned by interacting and identifying the correct substation components in the 3D environment.

Apart from the limitations already identified, the small sample reduces the probability of statistically significant findings. The snowballing method used to find participants (primarily due to the difficulty of recruitment during COVID) for this experiment could also be a possible limitation as the snowballing method can present bias. Sampling bias and margin error is one of the drawbacks of snowball sampling. Moreover, participants using VR for the first time may have taken more time adjusting to the technology rather than completing the learning activity.

5.   Results

5.1.   Student Experience: Field Trip vs VR

The questionnaire results from both the control group (students that had previously attended the electrical substation field trip) and the experimental group (VR participants) was analyzed and tabulated as seen in Table 1. MedCalc software was used to calculate Cronbach's alpha, which in this case was found to be 0.9678 and 0.9638 for the substation visit and the VR simulation respectively. The alpha's obtained are all above the 0.90 threshold, placing the internal consistency of the results into excellent for both questionnaires [22]. The average score for the learning contents for the substation visit was 3.64, whereas the VR simulation was found to be 3.98. For the operational experience the average score for the substation visit was 3.80, whereas for the VR simulation it was 4.06. The VR simulation scored higher in both learning contents and operational experience. However, no statistically significant difference was found between the two groups, all p values were greater than 0.05. Based on the numbers presented, if the sample sizes were larger it may have been possible to find statistically significant differences. As the VR scores are higher any statistical difference would have been higher for VR and this data suggests that that the student experience is at least equally engaging as attending a field trip.

Table 1.  Questionnaire results (substation visit/virtual reality).

Category	Questions	Average score (substation)	Average score (virtual reality)	Difference between traditional and VR	Significance level
Learning Contents	1. I understood most of the learning contents throughout the teaching activity	3.7	3.9	0.2 4%	P = 0.8282
	2. I can identify the major components in an electrical substation and what they look like after the teaching activity	3.6	4.1	0.5 10%	P = 0.6045
	3. I understood the operation of major components within a substation during the teaching activity	3.6	3.8	0.2 4%	P = 0.8525
	4. The learning activity provided useful knowledge on electrical substations	3.6	4.1	0.5 10%	P = 0.6045
	5. The learning activity was helpful at learning components in an electrical substation and their operation	3.7	4.0	0.3 6%	P = 0.7560
Operational Experience	6. It was easy to coordinate through the teaching activity	3.8	3.8	0.0 0%	P = 1.0000
	7. The speed and execution of the teaching activity was easy to keep up with	3.0	4.4	1.4 28%	P = 0.1642
	8. The teaching activity was not disorienting	3.9	3.4	0.5 10%	P = 0.6235
	9. The teaching activity motivated me to learn more about electrical substations	4.0	4.4	0.4 8%	P = 0.6344
	10. I am satisfied by the experience of the learning activity	4.3	4.3	0.0 0%	P = 1.0000
Total		37.2	40.2	3.0 6%	P = 0.7545
*Statistical Significance (P < 0.05)

---

 | Show Table

DownLoad: CSV

5.2.   Student Learning: Traditional vs VR

Table 2 outlines the pre-test and post-test performance for the control and experimental groups. The sample sizes for both test groups was 15. The mean found for the pre-test for the control group and the experimental group was 8.5333 and 8.4000 respectively. From Table 2 the post-test of the control group, being the traditional learning materials, is 11.2667, whereas the post-test result of the experimental group, being the VR simulation, is 13.6667. This shows that the VR simulation group completed the post-test on average 2.4 points higher with a 21.3 percent increase.

Table 2.  Overview of pre-test and post-test results.

Achievement Test	Mean	Standard Error of the mean	Standard Deviation
Control Group
Pre-Test	8.5333	0.1919	0.7432
Post-Test	11.2667	0.3581	1.3870
Experimental Group
Pre-Test	8.4000	0.2545	0.9856
Post-Test	13.6667	0.2702	1.0465

---

 | Show Table

DownLoad: CSV

The Levene test was used to test for the homogeneity of variances. If the Levene test significance level was found to be less than 0.05 the groups were not homogeneous. The significance level found between the control and experimental group was 0.234, which is greater than 0.05, which indicates that there was no significant difference in background knowledge between the control and experimental groups.

To determine if there was a statistically significant learning difference between the pre-test and the post-test a paired samples t-test was undertaken. Using MedCalc software the calculated P-value was found to be less than 0.0001 (P < 0.0001), which is less than 0.05, statistically indicating that the mean differences between the paired observations is significantly different from zero, indicating a difference in learning performance. Therefore, both methods resulted in cognitive learning.

The final test performed was analysis of covariance, also known as ANCOVA. This was used to determine if the difference between the control and the experimental group was statistically significant. The first process in ANCOVA is Levene's test for equality of error variances. This ensured the homogeneity of variance between the pre-test and post-test. The P value found was 0.768, which is greater than 0.05, indicating that the variances in the groups had not achieved a standard of significance and ANCOVA could be continued. Next, the homogeneity of regression slopes was found to be 0.137 (P > 0.05), indicating that the ANCOVA results are reliable. Lastly, the test of between-subjects effects gives a P value of < 0.001 (P < 0.05) for the group source, indicating a significant difference exists between the control and the experimental group. Therefore, cognitive learning was greater for the experimental group.

6.   Discussion

The results have extended knowledge of the benefits of VR in engineering education. For the first research sub question the data suggests that students perceive that a VR replica of an electrical substation provides at least an equal learning experience compared to the traditional field trip. For all, but one question student's perception was higher for the VR experience, but these perceptions were not statistically different which may have been a result of the small sample size. This suggests that an interactive VR experience can to some degree substitute or compliment substation-based field trips, especially if cost, logistics, workload or safety factors come into play. The only question indicating a difference in statistical significance was in relating to disorientation. This is not surprising as it supports the findings of Ma and Jaradat [23] that disorientation can be a symptom of VR, particularly for first time users. With improvements in technology and greater exposure to the technology the disorientation effect should dissipate over time.

While students' perceptions are very useful and have played a big role in shaping the development and measure of the success/failure of virtual environments in education [24], more quantitative data is needed to understand learning. For this reason, even if a holistic capture of student learning cannot be achieved, gaining at least some understanding moves knowledge forward in this area. While not directly measuring the learning differences between a substation field trip and VR this study has confirmed, even with a small sample, that cognitive learning at a statistically significant level is occurring and that it is higher than traditional methods. This builds upon other studies such as [25] that show the benefits of moving to 3D learning experiences. As previous VR substation-based studies such as [4,5,26,27] focused on the technology, this study expands knowledge in this area by confirming the cognitive learning benefits afforded by the technology. Therefore, for the second research sub question the data suggests that a VR replica of an electrical substation provides a better cognitive learning experience compared to traditional methods. This finding compliments other low sample sized, non-substation based studies, of the cognitive learning benefits associated with VR [14,15].

The use of written pre- and post- tests to explore cognitive learning benefits is common in the VR field and the results presented continue the narrative outlined in literature [28]. However, this approach only represents a portion of the learning outcomes available in interactive and engaging experiences like field trips and there may be other forms of learning being impacted by using VR as a substitute [21]. Field trips may provide many psychomotor or affective domain learning benefits that may not be found in a VR experience. Some of the many things missing in the VR simulation include pressing buttons, opening cupboards or pulling levers; observing smells and noise; seeing random behaviors from the machinery, humans or other animals; interactions between staff and students; asking and receiving feedback from questions; impact of weather and so much more. This also has the potential to impact knowledge sharing between education and industry [29]. As a result, more work is still needed in this area to more holistically understand impacts on learning.

VR within a work integrated setting offers an immersive, interactable, safe experience for what otherwise could be a dangerous learning environment [9]. VR can be an ideal solution to teaching engineers new work practices and safety procedures, particularly with the software capability offered by programs such as Unity. Based on the design requirements for the VR environment, students were able to: understand the layout of a substation through the use of smooth locomotion, understand the purpose of main components within a substation, understand the fundamental theories behind the components within a substation, and able to read and interpret diagrams associated with substations. This was done with the design and teaching activity implemented within the VR substation simulation.

The VR substation simulation offered a higher retention of knowledge than the traditional learning materials. The VR simulation provided the user with added interactivity and agency which should be beneficial to learning as they were able to control the pace, thereby engaging in higher generative processing. It is important how this is implemented within a VR simulation as more control and agency can have negative consequences for learning due to the cognitive load. This presents a complicated learning environment for the user which ultimately hinders the learning achievement, as seen in [18]. Within the simulation students were required to pick up and drag diagrams using information given for each component. As there was a significant increase in student knowledge found, the authors suggest that the cognitive load within the simulated environment was not enough to divert students' attention from important electrical substation related material.

7.   Conclusion

With VR electrical substation implementations focused on the technology rather than the educational benefits, this study has contributed to engineering education by exploring two research sub questions focused on student learning experience perceptions and cognitive learning. Together, they have answered the primary research question being that a VR replica of an electrical substation does provide an equal or better cognitive learning and student experience compared to traditional methods. While the sample size was low, statistically significant findings were found, complimenting other low sample sized, non-substation-based studies, of the cognitive learning benefits associated with VR [14,15]. Further research to validate these findings is warranted. In a time of COVID related educational disruptions, this evidence should provide institutions with some confidence to explore and implement VR based alternatives to the field trip. Many research limitations are raised pointing to the need for further work to consider psychomotor and affective impacts, ensuring a holistic understanding is achieved.

Acknowledgements

We would like to thank all students that participated in this study during this disruptive, stressful period caused by COVID19.