SKA低频成像管线并行优化
Abstract
Data processing of the square kilometer array (SKA) is performed in pipeline mode, and the execution efficiency of pipeline mode is an important factor in SKA data processing. Continuum imaging is a primary observation mode of SKA and a prerequisite for many other scientific works. In this paper, we take the imaging pipeline of the SKA low-frequency precursor Murchison widefield array as an example and optimize the parallel processing pipeline on the China SKA regional centre prototype (CSRC-P). Previous optimization schemes have focused on a few performance hotspots and lacked systematic optimization of the overall pipeline, resulting in a relatively poor overall speedup ratio. In this paper, we propose a global optimization scheme that combines C+ multi-threading, Python multi-processing, and Shell multi-tasking parallelism for pipelines using multiple programming languages and image datasets that can be processed independently and verify the accuracy of the optimization results. Experiments show that the optimized pipeline achieves an overall speedup of 2.7- and 2.4-fold on the x86 and advanced RISC machine (ARM) nodes of CSRC-P, respectively, and the ARM compute nodes show good adaptability to SKA applications. The optimization strategies and methods in this paper also apply to other SKA applications and will be useful for the scientific operation and future operation of the SKA precursor telescope.
