Prepping for Even Bigger Data in the Era of Interferometry

Interferometric arrays collect massive amounts of information, leaving astronomers with a happy problem: too much data! How can we handle mountains of data in an efficient way?

One of many tiles comprising the Murchison Widefield Array (MWA). Radio interferometric arrays like MWA generate vast amounts of data. [Dr. John Goldsmith/Celestial Visions]

Too Much of a Good Thing?

Astronomers have come a long way from the early days of manually cataloging stars and sketching sunspots by hand. Even though today’s data sets are larger and more complex, many astronomers still manually calibrate and process their data.

This hands-on data processing won’t always be feasible, though; interferometry — the process of linking together tens to thousands of telescopes or antennae to produce images with ever-finer angular resolution — generates far more data than humans could hope to handle manually. Just one minute’s worth of data from the Murchison Widefield Array (MWA), a radio interferometer made up of 4,096 antennae, yields roughly 10,000 images!

With the number of interferometers increasing, we’ll need to be smart about how we process all that data to minimize computing hours while maximizing the quality of the output. Among the many detectors requiring novel data-processing techniques is the planned Square Kilometer Array (SKA), which will comprise a million antennae and 2,000 radio telescopes. How can we get a handle on all this data without getting too hands-on?

An illustration of how increasing numbers of detectors are included in the model of the target for self-calibration. The first step includes only the blue detectors near the center, and subsequent steps add the red, teal, black, and yellow detectors to increase the complexity of the model. [Mondal et al. 2019]

Dealing with Data Pileup

To tackle this problem, a team led by Surajit Mondal (Tata Institute of Fundamental Research, India) developed an automated processing pipeline for interferometric data — the Automated Imaging Routine for Compact Arrays for the Radio Sun (AIRCARS). They focused on processing solar radio images, which need to capture a huge dynamic range — from extremely bright active regions to faint, wispy filaments.

One of the challenges in radio interferometry is removing the effects of instrumental artifacts and the plasma in Earth’s atmosphere. Most radio interferometry data are corrected with a self-calibration process that treats the instrumental artifacts and the brightness of the target as free parameters and iteratively minimizes the difference between the observations and a model of the target.

AIRCARS works especially well when applied to a compact array — one with many detectors clustered in the center and fewer near the outskirts. This configuration allows the pipeline to start with relatively little information about the target from just a few central detectors and gradually build a complex model of the target to be used in its self-calibration routine.

An example of the improvement of the dynamic range of MWA images through the self-calibration process. The number of iterations increases from top to bottom and left to right. The dashed circle indicates the location of the Sun’s disk. Click to enlarge. [Mondal et al. 2019]

AIRCARS in Our Future

In their tests on MWA data, the authors find that AIRCARS is capable of capturing a dynamic range up to 100,000:1 — a huge improvement over previous processing methods.

Mondal and collaborators note that AIRCARS can be configured to attain the maximum possible dynamic range without constraints on computing time, or to accept user-imposed time limits to rapidly process large amounts of data, depending on the user’s computational requirements.

Because the pipeline needs no human supervision, astronomers can take a step back from processing the vast amount of incoming data and focus instead on the exciting science we can do with interferometry.

Citation

“Unsupervised Generation of High Dynamic Range Solar Images: A Novel Algorithm for Self-calibration of Interferometry Data,” Surajit Mondal et al 2019 ApJ 875 97. doi:10.3847/1538-4357/ab0a01