Vision System in the Task of Determining Distances from the Video Camera to the Object

Purpose of research. Development of a computational process in a vision system to determine the distance to objects from a single video camera based on the transformation of RGB data received from a CMOS video camera into three-dimensional coordinates showing the location of the object on the 3d scene.

Methods. The research paper focuses on methods for determining distances from a video camera to objects of significant importance in the field of computer vision and machine learning. The study covers a variety of approaches used for distance estimation using video analytics, traditional image analysis models and machine learning. A method based on the classical marker detection algorithm incorporating a recent approach that realizes the principle of neuro-fuzzy learning in a vision system is considered. Special attention is paid to the visual evaluation of the effectiveness and accuracy of the proposed neuro-fuzzy approach in estimating the movements of actuators of mechatronic complex. This study provides an overview of the current state of the art of methods for determining distances from a video camera to objects and suggestions for further improvement and development of this field.

Results. The methodology for determining distances from the vision system to objects proposed in this study represents a significant step in the development of methods for positioning actuators of mechatronic complexes. The methodology was tested in real conditions and demonstrated a significant improvement in the accuracy of actuator positioning. During computational experiments it was possible to determine in real time the three-dimensional coordinates (center of mass) of the detected objects. This led to a 12% improvement in the positioning of mechatronic drives compared to similar solutions, which is important for achieving the required performance and efficiency of the production system.

Conclusion. The accuracy of the developed methodology was tested on various test datasets, including scenarios with different lighting conditions, changed background and recognition of different types of objects. Experimental results confirmed the effectiveness of the proposed methodology and its applicability in real-world conditions, providing improved positioning accuracy of the mechatronic system actuators.

1. Potapov A. A., Gulyaev Yu. V., Nikitov S. A. Noveishie metody obrabotki izobrazhenii [Newest methods of image processing]. Moscow, Fiziko-matematicheskaya literatura, 2008. 496 p. ISBN 978-5-9221-0841-6.

2. Bobyr M. V., Emel'yanov S. G., Arkhipov A. E., Milostnaya N. A. Prikladnye neironechetkie vychislitel'nye sistemy i ustroistva [Applied neuro-fuzzy computing systems and devices]. Moscow, Infra-M Publ., 2023, 263 p. ISBN 978-5-16-017976-6. DOI 10.12737/1900641.

3. Dvoinishnikov S. V., Bakakin G. V., Pavlov V. A., Meledin V. G. Bystryi algoritm rasshifrovki izobrazheniya ob"ekta v strukturirovannom svete dlya izmereniya trekhmernogo profilya v usloviyakh nelineinosti opticheskogo trakta [Fast algorithm for deciphering an object image in structured light for measuring a three-dimensional profile in conditions of nonlinearity of the optical path]. Izmeritel'naya tekhnika = Measuring technology, 2023, no. 8, pp. 36-41. DOI 10.32446/0368-1025it.2023-8-36-41.

4. You X., Shen L., Zhao Y., Ling Zhao D., Teng J., Lin H., … Zhang M. Quantifying interfacial interactions for improved membrane antifouling: A novel approach using triangulation and surface element integration method. Journal of Colloid and Interface Science. 2023. https://doi.org/10.1016/j.jcis.2023.06.117.

5. Peshkhoev A. A. Effektivnost' primeneniya metoda triangulyatsii s ispol'zovaniem Beacon mayachkov dlya opredeleniya mestopolozheniya vnutri pomeshchenii [Efficiency of using the triangulation method using Beacon beacons for determining location indoors]. Vestnik nauki = Bulletin of Science, 2019, vol. 4, no. 5(14), pp. 527-529.

6. Chen M., Bärwolff G., & Schwandt H. Time-of-flight technology applied in pedestrian movement detection. In Transportation Research Procedia, 2014, vol. 2, pp. 189–194. Elsevier. https://doi.org/10.1016/j.trpro.2014.09.028

7. Klempt W. Review of particle identification by time of flight techniques. Nuclear Instruments and Methods in Physics Research. Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 1999, vol. 433, no. 1-2, pp. 542-553. DOI 10.1016/S0168-9002(99)00323-X.

8. Zaripova A. D., Zaripov D. K., Usachev A. E. Vizualizatsiya defektov vysokovol'tnykh izolyatorov na infrakrasnykh izobrazheniyakh s pomoshch'yu metodov komp'yuternogo zreniya [Visualization of defects in high-voltage insulators on infrared images using computer vision methods]. Nauchnaya vizualizatsiya = Scientific Visualization, 2019, vol. 11, no. 2, pp. 88-98. DOI 10.26583/sv.11.2.07.

9. Geraskin D. P. Kolichestvennaya otsenka effektivnosti infrakrasnoi sistemy tekhnicheskogo zreniya meditsinskogo rabotnika teplovizionnoi diagnostiki [Quantitative assessment of the effectiveness of the infrared technical vision system of a medical worker for thermal imaging diagnostics]. Izvestiya vysshikh uchebnykh zavedenii. Elektromekhanika = Proceedings of Higher Educational institutions. Electromechanics, 2000, no. 2, pp. 93-96.

10. Poruchikova E. P., Mamedov Z. I., Loskutov I. A., Poruchikova O. L. Ob"ektivnoe opredelenie ostroty zreniya u patsientov s porazheniem makulyarnoi oblasti c pomoshch'yu infrakrasnoi videonistagmografii [Objective determination of visual acuity in patients with damage to the macular area using infrared videonystagmography]. Effektivnaya farmakoterapiya = Effective Pharmacotherapy, 2022, vol. 18, no. 45, pp. 6-13. DOI 10.33978/2307-3586-2022-18-45-6-12.

11. Smirnov D. A., Ter-Sahakyan A. S., Delendik A. M. [Algorithm for the initial exhibition of controlled aircraft using an on-board technical vision system and a ground-based laser beacon]. Aviatsionnye sistemy v XXI veke: Tezisy dokladov yubileinoi Vserossiiskoi nauchno-tekhnicheskoi konferentsii [Aviation systems in the XXI century. Reports of scientific and technical conference]. Moscow, State Research Institute of Aviation Systems Publ., 2022, pp. 139-140 (In Russ.).

12. Elistratov V.V. Ob ispol'zovanii lazernykh ustroistv tekhnicheskogo zreniya na avtomobil'noi tekhnike [On the use of laser technical vision devices on automotive vehicles]. Aspirant i soiskatel' = Graduate Student and Applicant, 2007, no. 3(40), 126 p.

13. Borisov E. G. Kombinirovannaya televizionno - ul'trazvukovaya sistema tekhnicheskogo zreniya [Combined television - ultrasonic technical vision system]. Voprosy radioelektroniki. Seriya: Tekhnika televideniya = Issues of Radio Electronics. Series: Television technology, 2011, no. 1, pp. 91-99.

14. Danishevsky O. V., Parfenov A. V., Stankevich V. V. Sovremennye resheniya robototekhnicheskogo zreniya universal'noi kiberneticheskoi platformy [Modern solutions of robotic vision of a universal cybernetic platform]. Voprosy radioelektroniki = Issues of Radio Electronics, 2019, no. 5, pp. 32-38.

15. Avrashkov P. P., Koskin N. A., Konstantinov I. S. Otsenka effektivnosti razlichnykh modelei konvertatsii izobrazhenii v stereoformat s pomoshch'yu kart glubiny [Evaluating the effectiveness of various models for converting images into stereo format using depth maps]. Nauchno-tekhnicheskii vestnik Povolzh'ya = Scientific and Technical Bulletin of the Volga Region, 2020, no. 12, pp. 165-168.

16. Bobyr M., Arkhipov A., Emelyanov S., Milostnaya N. A method for creating a depth map based on a three-level fuzzy model. Engineering Applications of Artificial Intelligence, 2023, 117. https://doi.org/10.1016/j.engappai.2022.105629

17. Bobyr M. V., Arkhipov A. E., Milostnaya N. A. Metod rascheta karty glubin na osnove myagkikh operatorov [Method for calculating depth maps based on soft operators]. Sistemy i sredstva informatiki = Systems and Informatics Tools, 2019, vol. 29, no. 2, pp. 71–84. https://doi.org/10.14357/08696527190207.

18. Bobyr M. V., Arkhipov A. E., Yakushev A. S. Raspoznavanie ottenka tsvetovoi metki na osnove nechetkoi klasterizatsii [Recognition of the shade of a color mark based on fuzzy clustering]. Informatika i avtomatizatsiya = Informatics and Automation, 2021, no 20(2), pp. 407-434.

19. Bobyr M. V., Arkhipov A. E., Gorbachev S. V. [et al.] Nechetko-logicheskie metody v zadache detektirovaniya granits ob"ektov [Fuzzy logical methods in the problem of detecting object boundaries]. Informatika i avtomatizatsiya = Informatics and Automation, 2022, vol. 21, no. 2, pp. 376-404. DOI 10.15622/ia.21.2.6.

20. Rastyazhenie gistogrammy [Histogram stretching]. CoderLessons: [website]. 2022. URL: https://coderlessons.com/tutorials/akademicheskii/tsifrovaia-obrabotka-izobrazhenii/rastiazhenie-gistogrammy.

21. Ekvalizatsiya gistogramm [Histogram equalization]. LightHouse Software: [website]. 2022. URL: https://lhs-blog.info/programming/dlang/ekvalizacziya-gistogramm/.

22. Obrabotka izobrazheniya [Image processing]. NDT-testing: [website]. 2022. URL: https://ndt-testing.ru/porogovaja-obrabotka-izobrazhenij.html.

23. Prosteishaya klasterizatsiya izobrazhenii metodom k-srednikh [The simplest image clustering using the k-means method]. Habr. 2022. URL: https://habr.com/ru/articles/165087/ .

24. Raspoznavanie obrazov s pomoshch'yu iskusstvennogo intellekta [Pattern recognition using artificial intelligence]. Habr. 2023. URL: https://habr.com/ru/articles/709432/ (Access date: 25.01.2023).

25. Kiseleva T.V., Maslova E.V., Bychkov A.G. Mashinnoe obuchenie dlya resheniya zadach raspoznavaniya obrazov [Machine learning for solving pattern recognition problems]. ITNOU: Informatsionnye tekhnologii v nauke, obrazovanii i upravlenii = ITNOU: Information technologies in science, education and management, 2021, no. 2(18), pp. 19-24. doi: 10.47501/itnou.2021.2.19-24

26. Belim S. V., Larionov S. B. Segmentatsiya izobrazhenii na osnove algoritma vydeleniya soobshchestv na grafe [Image segmentation based on an algorithm for identifying communities on a graph]. Matematicheskie struktury i modelirovanie = Mathematical Structures and Modeling, 2016, no. 3(39), pp. 74-85.

27. Sorokin D. V., Krylov A. S. Metod aktivnykh konturov dlya segmentatsii izobrazhenii [Method of active contours for image segmentation]. Moscow, MAX Press Publ., 2022, 16 p. ISBN 978-5-317-06761-8.

28. Vychislenie opticheskogo potoka metodom Lukasa-Kanade [Calculation of optical flow by the Lucas-Kanade method]. Habr. 2023. URL: https://habr.com/ru/articles/169055/.

29. Chigrinsky V.V., Matveev I.A. Issledovanie dvizheniya struktury raduzhnoi obolochki glaza metodom opticheskogo potoka [Study of the movement of the structure of the iris of the eye using the optical flow method]. Matematicheskie metody raspoznavaniya obrazov = Mathematical Methods of Pattern Recognition, 2017, vol. 18, no. 1, pp. 108-109.

30. Belokon A.V., Proskurin A.V., Favorskaya M.N. Klassifikatsiya metodov sinteza tekstur [Classification of texture synthesis methods]. Aktual'nye problemy aviatsii i kosmonavtiki = Current Problems of Aviation and Astronautics, 2011, vol. 1, no. 7, pp. 350-351.

31. Sintez izobrazhenii s pomoshch'yu glubokikh neirosetei [Image synthesis using deep neural networks]. Komp'yuternye nauki = Computer Science. 2023. URL: https://youtu.be/_rfmEzb6YP4.

32. Bobyr M. V., Milostnaya N. A., Bulatnikov V. A. The fuzzy filter based on the method of areas' ratio. Applied Soft Computing, 2022, vol. 117, 108449 p. DOI 10.1016/j.asoc.2022.108449.

33. Bobyr M. V., Emelyanov S. G. A nonlinear method of learning neuro-fuzzy models for dynamic control systems. Applied Soft Computing, 2020, vol. 88, 106030 p. DOI 10.1016/j.asoc.2019.106030.

34. Bobyr M.V., Dorodnykh A.A., Yakushev A.S. Ustroistvo i programmnaya model' upravleniya pnevmaticheskim mekhatronnym kompleksom [Device and software model for controlling a pneumatic mechatronic complex]. Mekhatronika, avtomatizatsiya, upravlenie = Mechatronics, Automation, Control, 2018, no. 19(9), pр. 612-617. https://doi.org/10.17587/mau.19.612-617.

35. Chaki J., Dey N., Moraru L., Shi F. Fragmented plant leaf recognition: Bag-offeatures, fuzzy-color and edge-texture histogram descriptors with multi-layer perceptron.Optik, 2019, no. 181, pp. 639–650. DOI:10.1016/j.ijleo.2018.12.107

36. Boukezzoula R., Coquin D., Nguyen T. L., Perrin S. Multi-sensor information fusion: Combination of fuzzy systems and evidence theory approaches in color recognition for the NAO humanoid robot. Robotics and Autonomous Systems, 2018, no. 100, pp. 302–316. Doi:10.1016/j.robot.2017.12.002

37. Bobyr M. V., Yakushev A. S., Dorodnykh A. A. Fuzzy devices for cooling the cutting tool of the CNC machine implemented on FPGA. Measurement, 2020, vol. 152, pp. 107378. DOI 10.1016/j.measurement.2019.107378.

38.