Performance and functionality tests
Markers, e.g., QR codes and bar codes, are important elements in AR systems because they represent a valuable tracking method to overlay digital contents on physical and real objects. The capability of the AR device to detect markers and the distance from which they are detectable are aspects that allow us to understand the accessibility of the digital information. For these reasons, the minimum and maximum scanning distances of different sized QR codes using the M400 were evaluated. Figure 2 describes the results obtained from the scanning distance of the V3 (Fig. 2a) and V11 (Fig. 2b) QR codes in relation to their size. As expected, the minimum and maximum scanning distances were directly proportional to the QR code size for both code types. The minimum distance and code dimension ratio for the two types of codes was constant (6.5 on average), as the minimum scanning distance is dictated by the camera framing. A different situation was observed for the maximum scanning distance of the two codes. In fact, the maximum scanning distance was higher for the V3 than the V11 code, with a distance and code dimension ratio of 48.8 and 21.5, respectively. Hence, using the M400 allowed detection of the augmented information from almost 2 m scanning a QR code of 40 × 40 mm size with 29 modules.
These results underline the importance of choosing an appropriate marker type and size to encode the required information. In farms, there are different contexts where specific information are needed to be detected (e.g., form long distance, in a small area, etc.). The scanning distance or the marker size may be influenced by the surrounding context. Furthermore, knowing the maximum scanning distance improves the labor organization. High ambient brightness (full sun) could affect the scanning of markers and thus the augmented information visualization. Although the agricultural context is characterized mainly by outdoor activities where brightness could be a marker-detection-related issue. Previous studies have shown the feasibility to obtain augmented information, by scanning a QR code through the SG camera, during outdoor activities and from the tractor cabin11. Moreover, high brightness would mainly affect AR content visualization on devices with the optical see-through systems rather than video see-through systems30,35.
Table 2 reports the results of the scanning time where significant differences were recorded among the three QR code sizes considered, which ranged on average between 1.90 s (75 mm) and 2.03 s (35 mm). However, these differences are very low (0.13 s) and do not appear to affect farmer performance.
Moreover, the battery life was monitored during the QR code scanning tests and the videocall tests using both applications (App1, App2). The overall average of 3 h and 28 min was recorded for the scanning tests, whereas 3 h and 50 min and 4 h were recorded for the videocall test, respectively. These values could be a limit if considering the amount of worker hours per day (8 h). In any case, this device should be used for specific and punctual activities carried out on the farm.
The audio and video transmission lag time from the M400 to a laptop was measured to test the quality of a videocall and to evaluate the feasibility of video-remote assistance in real time. The comparison of the results obtained by the two applications (Table 3) showed that the audio transmission was generally less than 0.5 s delay, and no significant difference was observed compared with the transmission quality (Tq). This delay seems to not influence the on-field assistance quality, since delays ranging from 0.4 to 0.15 s are considered acceptable values for high-quality real-time audio transmission36.
Considering the lag time for the video transmission, a 0.9 s delay on average was observed for both applications. Moreover, a significant difference of 0.7 s of delay between App1 and App2 was observed within the lower Tq. The results obtained were lower than 3 s on average observed by Muensterer et al.32, thus the video conferencing quality of SG may be considered adequate to connect agricultural operators and technicians. Furthermore, the average delay observed in video transmission does not represent a limiting factor since packet loss is more influential on video quality transmission, which was not observed in our tests37.
In general, the audio–video lag time was affected by the internet connection. From this perspective, it is important for the farm to have an adequate internet connection (20–30 Mbps) to support the data transfer and video streaming from the SG to a central or remote computer and minimize the delay during a remote videocall.
Figure 3 shows the results obtained from the visual acuity test using the standard Snellen chart. This test allows us to verify the level of content detail transmitted by the SG, which is important when farmers share information in remote assistance with an expert, technician, or another farm operator. The results underlined that a 13 mm character transmitted by the M400, from 0.5 m, on the laptop screen can be read by the receiving operator in every test. In the HD resolution, 9 mm characters were always recognized (100% of characters correctly recognized). On the other hand, 94% of the 9 mm characters were correctly detected at a lower resolution. In addition, it is possible to consider the 7 mm character the lowest printed size easily readable with both resolutions (HD and VGA) with 92% and 88% of the character recognized, respectively. With the 4 mm character size, the gap between the two resolutions increases drastically. In fact, 70% of characters were detected with the HD resolution, in contrast to 15% with the lower resolution. Hance, a farmer with the M400 could share his point of view with a high level of detail with an expert elsewhere, considering that the printed character size is recognizable depending on the videocall resolution.
The transmission results obtained in this study, both in terms of lag time and quality of the video sheering, were found to be appropriate to carry out maintenance activities in milking machine equipment. In fact, small details of the components (milking tubes, claws, liners, valves, etc.) were easily distinguishable from the remote technician.
The interactive system of the M400 includes three buttons, a touchpad, and a vocal interaction (voice commands/control). Considering the voice commands, several trials were carried out to evaluate the capability of the M400 system to respond to the voice action commands enabled at different levels of noise. Studying an environment with soft noise (60–68 dB)38 was observed that all the speech commands tested were recognized by the M400 that made the specific action requested (Table 4). A similar situation was observed with a noise level of 70 dB, where for all operators, 89% of the commands were recognized on average. At the 75 dB level of noise, an operator effect was observed. In fact, it was detected that three operators had a mean recognition rate of 79%, while the last operator had a recognition rate of just 5%. Therefore, 70–75 dB represents the border points of speech command detection for the Vuzix M400, considering that noises greater than or equal to 80 dB do not allow the use of the voice control function. This is an important function that, when implemented into the SG, allows the device to be managed completely hands-free. Regardless, in agriculture, the environmental noise can have different levels depending on the activities carried out.
Despite Depczynski et al.39 reported that only a few farm conditions stay below the threshold level of 70–75 dB observed for the M400, the noise levels in specific farm situations were measured. Through the sound level meter, the noise related to some agricultural and livestock activities (e.g., use of tractor or milking machine) was monitored to explore in which cases the voice control function of M400 could be effectively used. The results showed that M400 voice control could operate in a cabin tractor (73 kW power), in the milk room and in the milking parlor (Table 5). In fact, the most frequent level of noise measured in the milk room and milking parlor was between 68.7 and 75.3 dB. Moreover, the levels over the threshold detected in these rooms (maximum levels) regarded different sources of sound, e.g., the entry of the animals in the milking parlor, the opening and closing of the gates, and the animals themselves. In contrast, the speech commands were difficult to use when the operator was inside the engine room because the minimum level of decibel recorded was 93.7, which is over the threshold measured for M400. Another agricultural working situation was monitored, involving tractor driving. Two situations were checked: a tractor with a cabin that was more isolated from external noise and a tractor without a cab. The minimum level of decibel recorded allowed us to interact with the M400 by voice, and was recorded in a no working condition or when the tractor engines were running but idling. When the tractors were set in motion, a level of noise over the acceptable threshold was found, as attested by the mode and maximum values recorded for both situations.
Milking machine assistance scenario
To verify the proper operation of the milking machine, it is periodically necessary to check the system by certified operators. The ISO 6690:2007 standard specifies mechanical tests for the milking machine to verify installation compliance and components. In this work, a specific milking machine checking process was considered and made available in AR. In particular, the measurement process steps of pulsation characteristics were built on the Brochesia® portal in the workflow section (B Step). The workflow was developed as a sequence of step-by-step instructions that the operator must follow while wearing SG.
Pulsation is the cyclic opening and closing of the liner34. The pulsation cycle is composed of 4 phases: liner opening (phase a); liner open (phase b: milking); liner closing (phase c); liner close (phase d: rest). The pulsation characteristics affect milking performance and teat condition40,41. Hence, checking the pulsation characteristics as well as the milking system and components is needed at least once or twice a year. The certified operator to verify the system compliance must use specific measuring instruments and follow several procedures depending on the test to be performed, e.g., connect the sensors (flowmeter, pulsograph), disconnect some components of the milking machine (vacuum controller). Commonly, operators are supported by a paper guide or paper checklist to ensure that all test procedures are performed correctly. In this context, it appears that an augmented reality procedure supported by icons and images (visual information procedures) might be an important element in learning and performing this type of test. In addition, an SG-specific maintenance application for these mechanical tests is even more important, considering that during maintenance procedures, the operators need to make various movements and use their hands frequently.
The test for pulsation characteristics consists of several steps and is related to the previous tests carried out on the milking machine. Three steps are fundamental: the connection of the pulsograph to the machine, the recording of five complete pulsation cycles and the evaluation of the data recorded for each milking unit, i.e., length of the a, b, c, and d phases per pulsation cycle. In Fig. 4, the workflow scheme of the pulsation characteristics process developed on the BStep application portal is reported, following the ISO 6690:2007 standard.
The workflow of the pulsation characteristics test was available from a 40 × 40 mm QR code with 29 modules placed on the milking machine (Fig. 5). The QR code represented the marker that the operator needed to scan with the SG to obtain the augmented information. As observed in the previous tests, this type of QR code can be scanned quickly from a distance of 2 m using the M400. In addition, although different sources of noise were found in the milking parlor (animal noises, pulsators, vacuum regulator valve, etc.), where the maintenance scenario was contextualized, the noise levels most frequently recorded did not affect the use of SG by voice commands (Table 5). Therefore, the operator was able to proceed with the milking machine and components inspection while having his hands completely free.
The first augmented instruction provided to the operator was the question: Is this the first test performed on the milking machine? which implies if other instruments meter and configurations are in place or not. Interacting with the workflow by the touchpad of the M400, the operator can choose two different ways (Fig. 4). Nevertheless, some preliminary steps before checking the pulsation were needed: in one case, e.g., stop the vacuum pump and set the system in simulated milking, mount vacuum regulator, disconnect the flowmeter (if previously used); in another case, turn on the vacuum pump (Fig. 6a), wait fifteen minutes, set the system in simulated milking, etc., as shown in Fig. 6c. Then, attaching the sensors of the pulsograph with the T-piece connection was requested and the right way was explained through the augmented instructions (Fig. 6b). Afterward, recording a minimum of 5 pulsation cycles with the test equipment to obtain reliable data was indicated. The recommended values regarding the length of each phase (a, b, c, d) were visible on M400 to support the operator during the evaluation of the data recorded in all milking units. When anomalous pulsation curves were observed, the operator was able to view tips and suggestions to solve the problem and restore the pulsation system.
Overall, the results obtained confirmed the suitability of the device tested for milking machine maintenance. The aspects characterizing the agricultural environment such as remote location, limited or difficult access routes, and reduced number of specialized maintenance workers per farm, are the main driving factors for using AR and SG as a support tool for specialized remote assistance. Additionally, the agricultural environment is remarkably heterogeneous and different from other workplace (i.e., industry sector) where different operating situations (field crop management, greenhouse, barn animals, grazing animals, etc.) and different machinery (tractors, implements, milking equipment, etc.) can coexist. The milking machine is one of the farm machines that require specific maintenance by skilled technicians. As indicated by ISO 6690:2007, several mechanical components (vacuum pump, milking units, pulsators, etc.) and operative parameters during milking must be checked periodically for an efficient milking operation42. Failures during milking of such equipment can cause economic losses due to a decreasing milk yield. Therefore, prompt and precise maintenance is necessary both in emergency situations and in scheduled checks.