Automated Processing of Primary Field Data on the Behavior of Natural-Technological Systems under Climate Change and Anthropogenic Impacts in the Far North

Introduction. This work addresses the scientific problem of studying natural-technological systems (NTS) of the Far North under conditions of climate change and anthropogenic impacts. The relevance of ensuring their stability is emphasized, which requires a comprehensive analysis of field data. Problems in automated processing methods of such specific data have been identified. The aim of the study is to develop automated methods for processing field data to reveal patterns. Python libraries for data analysis, processing, and visualization are used as tools.
Materials and Methods. The research object is described — the Main Building of the Yakutsk Thermal Power Plant (TPP) in permafrost conditions. The study materials include field data obtained from engineering-geological boreholes at the Yakutsk TPP, monitoring stations Chabyda and Tuymaada, as well as a section of the Amur-Yakutsk railway (AYR). The data include measurements of soil temperature and moisture, seasonal thaw layer dynamics, snow cover characteristics, and others. A detailed sequence of automated processing of primary data from XLS files using the pandas library is presented, including reading, cleaning, format conversion, filling or replacing missing values, removing duplicates, and saving processed data in CSV, JSON, and XLSX formats.
Results. Specific results of automated processing and systematization of primary field data are presented. Heterogeneous measurements were successfully unified into a single format, ensuring their proper use. A unique data array was formed based on empirical observations under the specific conditions of the Far North. The practical application of Python libraries for executing key stages of preprocessing and data preparation is demonstrated.
Discussion and Conclusion. It is shown that the application of a systematic approach and automated data processing significantly improves the quality and reliability of natural-technological system data analysis. Handling missing data and normalization enhance accuracy, and the final data formats are convenient for further modeling. The universality of Python is highlighted. Prospects for further research include applying machine learning, clustering, and modeling methods aimed at uncovering patterns and forecasting the behavior of natural-technological systems in the Far North under climate and anthropogenic influences

1. Poruchikov M. A. Data analysis. Samara: Samara University Press; 2016. 88 p. (In Russ.)

2. Introduction to Data Analysis. URL: https://mipt-stats.gitlab.io/courses/ad_fivt/titanik.html (accessed: 08.08.2024)

3. Leskovec Yu. Analysis of Large Datasets. Moscow: DMK; 2016. 498 p. (In Russ.)

4. McKinney W. Python and Data Analysis. Moscow: DMK; 2015. 482 p. (In Russ.)

5. Makshanov A.V. Data Mining Technologies. Saint Petersburg: Lan; 2018. 212 p. (In Russ.)

6. Mirkin B.G. Introduction to Data Analysis. Lyubertsy: Yurayt; 2016. 174 p. (In Russ.)

7. Rafalovich V. Data Mining, or Data Analysis for the Busy: Practical Course. Moscow: SmartBook; 2018. 352 p. (In Russ.)

8. Chashkin Yu.R. Mathematical Statistics. Data Analysis and Processing. Rostov-on-Don: Feniks; 2017. 236 p. (In Russ.)

9. Gron A. Machine Learning and Data Analysis with Python. Saint Petersburg: Piter; 2019. 460 p. (In Russ.)

10. Vorontsov K.V. Machine Learning and Data Mining. Moscow: MIPT Publishing; 2018. 328 p. (In Russ.)

11. Algozina E.R. Data Processing and Analysis with Python and Pandas: Beginner’s Guide. Moscow: DMK; 2020. 312 p. (In Russ.)

12. Troyan I.V. Data Analysis and Machine Learning in Python. Saint Petersburg: BHV-Peterburg; 2019. 320 p. (In Russ.)

13. McKinney W. Python Programming and Data Analysis. Saint Petersburg: Piter; 2020. 504 p. (In Russ.)

14. Mueller A.C., Guido S. Introduction to Machine Learning with Python. Saint Petersburg: Piter; 2017. 420 p. (In Russ.)

15. Rozhkov R.S., Molchanova E.S., Fadeev M.K., Pimenov N.A. Managing Natural-Technical Systems through the Concept of Sustainable Development and Acceptable Risk. Bulletin of Eurasian Science. 2024;16(5):1–10. (In Russ.)

16. Dregulo N. Influence of Climatic Factors on the Operation of Natural-Technical Systems for Wastewater Treatment. URL: https://pureportal.spbu.ru/ru/publications/---------%28a9c91d8e-c5e0-43d9-98b5-8aac39cc4a61%29.html (accessed: 13.01.2025) (In Russ.)

17. Impact of Anthropogenic Factors on Thermal Pollution of the Urban Environment. URL: https://www.abok.ru/for_spec/articles.php?nid=7635 (accessed: 13.01.2025) (In Russ.)

18. Impact of Anthropogenic Factors on Agricultural Development. URL: https://biohim.com.ru/articles/page2 (accessed: 13.01.2025) (In Russ.)

19. Levashkin S.P., Ivanov K.N., Kushukov S.V. Data Farm: An Information System for Collecting, Storing, and Processing Unstructured Data from Heterogeneous Sources. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2023;35(2):57–72. https://doi.org/10.15514/ISPRAS-2023-35(2)-5 (In Russ.)