Detection of Martian Dust Storms Using Mask-Regional Convolutional Neural Networks

Martian dust plays a crucial role in the meteorology and climate of the Martian atmosphere. It heats the atmosphere, enhances the atmospheric general circulation, and aﬀects spacecraft instruments and operations. Compliant with that, studying dust is also essential for future human exploration. In this work, we present a method for the deep-learning-based detection of the areal extent of dust storms in Mars satellite imagery. We use a mask regional convolutional neural network (R-CNN), consisting of a regional-proposal network (RPN) and a mask network. We apply the detection method to Mars Daily Global Maps (MDGMs) of the Mars Global Surveyor (MGS) Mars Orbiter Camera (MOC). We use center coordinates of dust storms from the eight-year Mars Dust Activity Database (MDAD) as ground-truth to train and validate the method. The performance of the regional network is evaluated by the average precision score with 50% overlap ( mAP 50 ), which is around 62 . 1% .


Introduction 1
The Martian dust cycle is of fundamental importance to the meteorology and cli-2 mate of the Martian atmosphere (e.g., Hab17;Kas16;Mon15).Atmospheric dust may be local and regional dust storms at any time of the Martian year, GDEs occur only during the second half of the Martian year (Ls = 180 • -360 • ).The latter is known as the dust storm season and coincides with the Mars southern hemisphere spring and summer.A yearly repeatable phenomenon is multiple local dust storms at the northern/southern Mars polar edge in the respective hemispheric fall to the spring season, known as polar cap edge storms.By contrast, dust devils are another phenomenon and have a diameter of less than 1 km and a duration of fewer than 10 minutes (For17).
A comprehensive dust climatology was detailed in (Mon15).The basis for that are data on the column dust optical depth from the satellite instruments MCS/MRO (Mars Climate Sounder/MARS Reconnaissance Orbiter), THEMIS/MO (Thermal Emission Imaging System/Mars Odyssey), and TES/MGS (Thermal Emission Spectrometer/Mars Global Surveyor).The latter operate at different wavelength ranges, measurement geometries, and spatial and temporal coverage.This dust climatology is made publicly available via the Mars Climate Database (MCD) [1] , together with many other parameters of the Mars atmosphere and surface.It has a moderate spatial resolution of few degree latitude and longitude [2] and was demonstrated to be suitable to follow the evolution of certain regional dust storms by (Mon15).Various studies identified and explored dust storms based on the visual inspection of Mars Daily Global Maps (MDGMs) from the camera system MOC/MGS (Can07;Hin12).Other studies focused on MDGMs from both MOC/MGS and MARCI/MRO (Bat21; Wan15).In this work, we perform feasibility study on a deep-learning-based approach for dust storm detection from the record of MDGMs by the Mars Orbiter Camera (MOC) (Mal10) aboard the Mars Global Surveyor (MGS), by applying convolutional neural networks (CNNs) (Sim14).
Recently, deep convolutional networks have made significant improvements in the accuracy of object detection.As widely known, object detection is a challenging task because it requires the accurate localization of candidate objects.In this paper, we use mask regional convolutional neural networks (R-CNNs) that jointly learn to classify dust storm candidates and refine their spatial locations.The spatial location of potential dust storm candidates (i.e., the ground-truth objects) may be to a certain degree arbitrary because dust storm boundaries are identified based on the subjective perception of individual observers and are interpolated if intersecting satellite image gaps and/or the polar night.Here we make the simplifying assumption that dust storms have a rectangular shape around their central coordinates.
The main contribution of this work can be summarized as follows: • It is the first work on the deep-learning-based detection of Mars dust storms which is applied to several Martian-year records of MDGMs.Also, it uses the Mars dust storm database of Battalio and Wang (Bat21), which is one of the most recent and comprehensive of its kind, as a ground-truth.
• It uses a new architecture that consists of two networks to enhance the boundaries of dust storm areas, although the ground-truth boundaries include a certain degree of subjectivity and arbitrariness.
[1] http://www-mars.lmd.jussieu.fr/ [2]http://www-mars.lmd.jussieu.fr/mars/dust_climatology/index.html • It uses a dice score as mask loss function to overcome ambiguous cases at the boundary between a dust storm and non-dust-storm categories with a lower level of uncertainty between the two categories.
The outline of this paper is the following.Section 2 describes the previous work related to automated dust storm detection and the latest R-CNN techniques.In Section 3, we explain the observation-based dataset and ground-truth we used.In Section 4, we illustrate the methodology to detect dust storms on the Martian surface.We discuss the performance of our method in Section 5.In Section 6, we summarize the main findings and provide an outlook for the future.

Detection of Martian dust storms
Maeda et al. (Mae15) proposed an automatic method to detect dust storms.Their method is based on selected features using minimal redundancy maximal relevance algorithm and classification using Support Vector Machine (SVM).It successfully detects around 80% of dust storms, but it did not define the locations of dust storms.Gichu and Ogohara (Gic19) suggested a segmentation method to classify Martian images into either dust areas or cloud areas.They used principle component analysis (PCA) and supervised multi-layer perceptron (MLP) neural networks based on subjective ground-truth images.They only focused on the regions (patches) with a high frequency of dust storms revealed by Guzewich et al. (Guz15) and Kulowski et al. (Kul17).In this work, we concentrate on complete Martian images.

Regional convolutional neural networks (R-CNNs)
The R-CNNs are used to predict object's position and objectness scores at each posi-

Method
We use a mask regional convolutional neural network (Mask R-CNN) to estimate the spatial probabilities of each dust storm in the Martian map.The Mask R-CNN is a modified version of fast R-CNN (Gir15), but it has a new branch masker that predicts a segmentation mask on each region of interest (RoI) in pixel-to-pixel manner (He17).This is added to classical region proposal network (RPN), classifier and bounding box regressor that proposes candidates object bounding boxes, determines the category of the region and estimates its four border coordinates, respectively (Gir15; He17).Because of the inaccuracy of candidates' borders (dust boundaries), we use RPN with multiple reference rectangles (anchors).Figure 1 present a flowchart of the current method and we will discuss each part in the following sections.Each anchor is associated with a scale s and aspect ratio r.For a feature map of W × H, there are W × H × k anchors.We generate invariant translation anchors; if an object translates in an image, the reference proposal should be translated and the model should be able to predict the proposal in various locations.Therefore, in the case of k anchors, we have (4 + 2) × k-dimensional convolution output layers.
Figure 2 shows an overview of the RPN.

Mask R-CNN
We use the Feature Pyramid Network (FPN) (Lin17) as a segmentation network and run on the feature maps of the ResNet backbone network to produce a binary We use RoIAlign (He17) as a standard sampling/quantization operation to extract RoI from feature maps for both classification/regression network and segmentation mask network.In the mask network, we us RoIAlign from feature pyramid level {P 2 , P 3 , P 4 , P 5 }, as in (Lin17), in the first phase of the mask network (M asker A ).
While we use RoIAlign from the finest-resolution feature in the second phase of the mask network (M asker B ).In M asker A , the RoI feature is fed into four consecutive 3 × 3 convolutions.While in M asker B , the RoI feature is fed into two consecutive 3 × 3 and fed into M asker A after down-scaling.This process enriches mask features and obtains an accurate mask prediction with shape and edge details.
Both the original and the proposed mask R-CNNs use RoIAlign as a standard sampling operation to derive feature maps for classifiers, box regressors and maskers.
However, we use separate networks for both classifier and regressor after applying full-connected layers.We also use convolution layers with various convolution kernels to extract feature maps for maskers to preserve dust boundaries' pixels and provide better localization.

Learning and optimization
The loss function L is a combination of classification L c , regression L r and segmentation mask L m losses.We use a binary cross-entropy to classify each box and a mean absolute error (MAE) to estimate four coordinates of each box.To alleviate the class-imbalance problem between positive pixels (dust) and negative pixels (non-dust), we use Dice loss (Mil16) to measure overlapping between prediction and ground-truth.
L m (y n , ỹn ) = 2 where N and I are number of anchor in each images and number of pixels of each anchor, respectively.y and ỹ are the ground-truth and the predicted probability of each anchor n.

Performance
In this section, we present the performance of mask R-CNNs on MGS/MOC dataset using various training strategies and compare with state-of-the-art methods.

Evaluation metrics
We assign a binary class to each RoI (anchor).We assign a positive value to RoI with the highest intersection-over-union (IoU) overlap with a ground-truth box.We assign a negative value to RoI with the lowest IoU overlap.The highest IoU ratio is 0.7 and lowest IoU ratio is 0.3.We do not use RoI that is neither positive nor negative in minimizing the objective function and mask loss L m is defined only on positive RoIs.The IoU is defined as the area of the interaction of predicted mask Ỹm = {ỹ m 1 , ..., ỹm N } and ground-truth mask Y m = {y m 1 , ..., y m N } divided by the area of the union of predicted and ground-truth masks: We calculate the precision (P = T P /T P + F P ) and recall (R = T P /T P + F N ) of each image in the testing dataset, where T P , F N and F P are the number of dust pixels corrected classified as dust pixels, the number of dust pixels classified as nondust pixels and the number of non-dust pixels classified as dust pixels, respectively.
We evaluate the performance of the network based on the mean average precision (mAP) score, where AP is an area under precision-recall curve averaging for each class in an image.We calculate mAP at various intersection-over-union thresholds IoU th (25%, 50% and 75%).

Implementation details
• Data: we use daily Martian map as an input to mask R-CNNs.We use reflected data from red, green and blue bands from MGS/MOC maps.We prepare the ground-truth of mask R-CNNs based on center coordinates of each dust storm event in MDAD dataset (Bat21).We estimate approximate area with 120 × 120 around center coordinates to draw ground-truth anchors and train mask R-CNNs.
• Training: we use adam optimization function.We assign the learning rate to 0.0001 which decreases by 10 every 10k iterations, weight decay to 0.001 and momentum of 0.9, step per epoch to 1000 and validation step to 50.We train on 1 GPU with a mini-batch size equal to 32.We train with 1 image per GPU.
• Inference: we assign the maximum number of detection instances to 100 with confidence greater than 90%.After box prediction, we predict a binary mask per RoI.RoI is considered positive if it has IoU with a ground-truth box of at least 0.5.

Experimental results in different seasons
We  and CL=75, respectively.This may be at least partly due to the fact that CL=100 means the dust storms instance still has an error of few pixels, or approximately around 0.5 • , and CL=75 and CL=50 have an error greater than 0.5 • , accordingly.
Also, it fails to distinguish the dust storm instances with CL=100 and CL=50 at the coordinates (147.75 • E, 33.7 • N) and (46.55 • W, 17.9 • S) from the background.A potential explanation for that is increased atmospheric background dustiness during the dust storm season.(fall) and Ls 313.57• (winter).Here, our method successfully identifies most of the dust storm instances.In the case of slightly overlapping ground-truth objects, at least one of both is successfully detected.Also, the center coordinates of some of them are mismatched.This may be because the ground-truth rectangles are set by visual inspection.Thus, they are to a certain extent subjective and the dust storm instances may even extend over a larger area.If so, our method may have identified nearby regions because they have similar spatial and spectral characteristics.
Among others, our method may also have produced some false-negative and falsepositive cases due to the presence of water ice clouds and/or increased atmospheric background dustiness in MDGMs or MDGMs have image gaps, as in Figure 5(d) and Figure 6(b) respectively.In line with that, we may integrate some additional processes in the future (e.g., filling missing data, cloud detection, etc.).(delta-longitude dx and delta-latitude dy) using the first and the second training strategies.We note dx and dy are approximately between -4 and 4 of both strategies, but longitude variations is more compared to latitude variations, as it is obvious in Figures 3, 4, 5 and 6.This is because dust storm events have higher probabilities to include a wider area than subjective ground-truth areas.

Comparison with state-of-the-art methods
We compare the performance of the regional networks based on the first training strategy because it is more significant to predict future dust storms.In Table 1, we compare between fast R-CNN, mask R-CNN, SPPnet and the current R-CNN.We use mAP with IoU thresholds equal to 25%, 50% and 75%.As expected, selecting higher thresholds reduces the effectiveness of all R-CNNs.In addition, inference time which is required for each image of all networks between 300-370 milliseconds (ms).Mask R-CNNs have higher mAP and faster compared to non-mask networks.
However, the current method has a slightly higher score because the mask network has an additional component (M asker B ) that focuses on edges or boundaries to refine the mask with minor improvement.One of the main strengths of this method is its speed and ease of use after training.Potential challenges are due to MDGMs from MGS/MOC partly having gaps (block areas), the R-CNN fails to detect dust storms around these areas.Moreover, it is possible that R-CNN confuses between dust storms and enhanced atmospheric background dustiness in the dust storm season, different dust storms that are near to each other, and/or dust storms and water ice clouds.
The proposed mask R-CNN has been applied to a several-Martian-Year record of satellite images and demonstrated to provide reasonable results at various seasons.
We may refine the current results further and thus obtain more accurate dust storm characteristics (location, size, shape, texture, etc.) as follows.It is widely known that Mars dust storms are bright in the red band and dark in the blue band.By contrast, Martian clouds are bright in the red and blue bands and much brighter than the surface in the blue band (Gic19).In the future, we aim to include surface albedo and/or cloudiness when preparing the ground-truth to avoid confusion between dust storms, clouds, and albedo features.Also, we aim to predict the probability of accurate contours based on polygon areas in the MDAD dataset.In addition, we aim to classify each dust storm based on class (main, continuous, sequential, etc.), type (flushing, turning, GDE, etc.) and K16 class (A, C, GDE, etc.).All in all, our method is basically suitable to create another perspective on the climatology of Martian dust storms by a deep-learning-based method.
We attempted to apply the proposed R-CNN on the Mars Reconnaissance Orbiter tion.Girshick et.al. (Gir13) proposed the first R-CNN by generating CNN features of each object proposal and applying the SVM to classify proposals.The fast R-CNN is a new version of the previous R-CNN.It trains the very deep VGG16 network faster by 9× and 213× at train time and test time, respectively.Ren et.al. (Ren16) introduced a regional proposal network (RPN) that shares full-image convolutional features and merges with the detection fast R-CNN.The PRN works as an attention mechanism telling the fast-RCN where to look.He et al. (He17) integrated fast R-CNN with segmentation to generate a high-quality mask for each object instance in the image (mask R-CNN).Cheng at al. (Che20) presents a boundary-preserving mask R-CNN that explicitly exploits object boundary to improve localization accuracy.In this work, we use mask R-CNN to determine the whole area of dust storms in Martian images.
truth images, we use the Mars Dust Activity Database (MDAD) (Bat21).It is a dust storm database compiled from eight Martian years (MY) of Mars Daily Global Maps (MDGMs), which means from MY 24, Ls 150 • (1999) to MY 32, Ls 171 • (2014).The MDAD comprises 14, 974 dust storm instances, which are, by definition, enclosed dust storm regions on a single sol (Martian day).The dust storm instances are combined into 7, 927 dust storm members.These are subdivided further into a total of 228 dust storm sequences (125 originated in the northern hemisphere and 103 in the southern hemisphere).Dust storm members are multisol dust storm instances that can be tracked from one sol to the next.Dust storm sequences are one or more dust storm members following a coherent trajectory and having a lifetime of at least three sols.Unorganized dust storm instances are equivalent to dust storm members having just one constituent.The latter are found mostly along with the polar ice cap edges in the northern and southern hemispheres around their respective fall and winter seasons (i.e., polar cap edge storms).These unorganized instances do not change much between different Martian years in terms of latitude, timing, and number.By contrast, the most dominant sources of interannual variability are the global dust storm events at around Ls 185 • -210 • in MY 25 and Ls 265 • -290 • in MY 28.The Mars Dust Activity Database can be found at https://doi.org/10.7910/DVN/F8R2JX.It includes the center coordinates (longitude and latitude) and area (in km 2 ) of individual dust storm instances.We use the center coordinates of each such dust storm instance but, as a simplifying assumption, consider rectangle areas around the center coordinates.The MDAD also includes confidence levels (CL) of 100, 75, and 50, which are assigned to each dust storm instance based on visual inspection.They rate the accuracy of dust storm boundaries with the highest confidence level of 100 and the lowest confidence level of 25.CL = 100 means the entire perimeter of the dust storm instance is distinct against the background so that the dust storm edge has an error on the order of a few pixels only (which is equivalent with approximately 0.5 • ).CL = 25 shows rather nebulous boundaries that cannot be exactly discerned from the background within few degrees of latitude/longitude.The CL is also used to determine how distinct a dust storm instance is from the background atmospheric opacity.Only dust storm instances with CL = 100, 75, 50 are listed in the MDAD.In the following, we include all Mars Daily Global Maps (MDGMs) based on MOC/MGS, from MY 24, Ls 150 • (1999) to MY 28, Ls 121 • (2006), as obtained from https://doi.org/10.7910/DVN/WWRT1V.We consider the non-polar versions of these MDGMs, which cover latitudes from 60 • N-60 • S and longitudes from 180 • E-180 • W and have simple cylindrical map projection.The MDGMs have a resolution of 7.5 km per pixel with 0.1 • longitude by 0.1 • latitude.They are available as RGB images.Details on the MDGM production process can be found in (Wan02).Each MDGM is based on 13 wide-angle global map swath images of the Mars Global Surveyor (MGS) Mars Orbiter Camera (MOC).The latter covers the whole sun-lit planet around 2 PM local time each sol.The MDGMs consist of imagery from the two visible bands, red (575-625 nm) which is more sensitive to dust storms, and blue (400-450 nm) which is more sensitive to water ice clouds (Can01).The green component of the MDGMs is synthesized by combining 1/3 red and 2/3 blue and applying linear stretching.

Figure 1 :
Figure 1: A flowchart of the used mask R-CNN.
use images from MY 25, Ls 0 • to MY 27, Ls 180 • as training dataset (1216 MDGMs).We randomly select validation images from MY 25 to MY 27 (614 MDGMs) which are not used in training dataset to validate the performance of convolutional networks during training process to obtain lower error.We use images from MY 27, Ls 180 • to MY 28, Ls 121 • as a testing dataset (659 MDGMs).Each of the datasets includes images from all four seasons: spring (0 • < Ls < 90 • ), summer (90 • < Ls < 180 • ), fall (180 • < Ls < 270 • ) and winter (270 • < Ls < 360 • ).The reason for using MDGMs from middle MY 27 as a testing dataset is that MY 28 only includes MDGMs from spring and summer seasons.Figure3and Figure4show ground-truth (a) and (c) and predicted regions from R-CNN (b) and (d) for selected MDGMs of the testing dataset.In Figure3(a), the ground-truth for MY 28, Ls 83.04 • is given by dust storm instances at the coordinates (89.25 • W, 26.7 • S) and (132.05• E, 30.2 • N).The latter have a CL=75 and CL=50, respectively, which implies that their boundaries are not fully accurate.As follows from Figure3(b), the R-CNN identifies another dust storm instance close to the ground-truth dust storm instance (89.25 • W, 26.7 • S).However, the latter has some overlap with the true positive dust storm instance, and may thus be considered to be not entirely false.

Figure 3
Figure 3(c) and 3(d) present accurate results for MY 28, Ls 110.25 • (summer).The ground-truth is given by dust storm instances at the coordinates (96.75 • W, 26 • S) and (73.05 • W, 34.3 • S) with CL=100 and CL=75 near to southern polar ice cap.The detection accuracy is approximately 0.99 for both, i.e., high overlapping areas with the ground-truth.Figure 4(a) and Figure 4(b) at MY 27, Ls 222.83 • , i.e. during the dust storm season, show that the R-CNN identifies dust storm instances in different regions.However, it mismatches some of the center coordinates and has a certain overlap with surrounding areas.That is the case around the ground-truth objects at (148.85 • W, 39.5 • N), (54.35 • W, 4.7 • N), (14.65 • W, 46.2 • N) with CL=50, CL=75 Figure 4(c) and Figure 4(d) show accurate results at Ls 305.93 • with ground-truth objects at (32.55 • W, 0.90 • N) and (158.55 • W, 36.1 • N) and CL = 75.We also apply the network to images from MY 25, Ls 0 • to MY 28, Ls 121 • , which are randomly divided into a training dataset (1300 MDGMs), a validation dataset (586 MDGMs) and a testing dataset (600 MDGMs) and analyze the performance in all seasons.Figure 5 and Figure 6 show examples from all four seasons at Ls 53.47 • , Ls 105.36 • , Ls 238.51 • and Ls 313.57• in MY 26 and MY 27, respectively.

Figure 7
Figure 7 (a-d) show the distribution of longitude-latitude coordinates of the predicted RoI compared to the subjective coordinates of the ground-truth PRN RoI

Figure 3 :
Figure 3: 1 st strategy: the training and validation images are selected randomly from MY 25 to middle of MY 27.The testing images are selected randomly from middle of MY 27 to MY 28.(a) and (c) are ground-truth images from spring and summer seasons and (b) and (d) are their predicted dust maps.

Figure 4 :
Figure 4: 1 st strategy: the training and validation images are selected randomly from MY 25 to middle of MY 27.The testing images are selected randomly from middle of MY 27 to MY 28.(a) and (c) are ground-truth images from fall and winter seasons and (b) and (d) are their predicted dust maps.

Figure 5 :
Figure 5: 2 nd strategy: the training, validation and testing images are selected randomly from MY 25 to MY 28.(a) and (c) are ground-truth images from spring and summer seasons and (b) and (d) are their predicted dust maps.

Figure 6 :
Figure 6: 2 nd strategy: the training, validation and testing images are selected randomly from MY 25 to MY 28.Left panel shows ground-truth maps and right panel shows predicted maps.(a) and (c) are ground-truth images from fall and winter seasons and (b) and (d) are their predicted dust maps.

Figure 7 :
Figure 7: Histograms of dust proposals: (a-b) dx and dy of the first training strategy and (b) dx and dy of the second training strategy.

Table 1 :
Comparison between R-CNNs based on various mAP .We use mask R-CNN for the automated localization of dust storms in Mars Daily Global Maps (MDGMs) from MGS/MOC.We evaluate the performance of the network by calculating the area under the ROC curves from the dust storm probability images by using various IoU thresholds and obtain the best performance at AP 25 .