The following list identifies theoretical, methodological, and technological aspects of the CDF algorithm that require further investigation and development. These topics represent opportunities for future research and are organized by priority and relevance to algorithm improvement.
Research Agenda for CDF Development
A priori constraint strength for fused products
Develop a principled method to select the optimal a priori profile and variance-covariance matrix for the fused product, ensuring that the constraint strength is neither too weak (leading to poorly constrained solutions) nor too strong (suppressing real atmospheric variability).
Status (2026): still chosen empirically. In the N2O limb/nadir prototype (test days of July 2011) the fusion a priori was selected after comparing 19 variants on single profiles: relative standard deviation 1 % below 8 km, ramp up to 16 km, 30 % with 6 km correlation above for MIPAS and 10 % with 10 km for Odin-SMR. The most oscillating fusions are in the Antarctic vortex, where the a priori profile is far from the truth; an a priori that distinguishes the vortex is proposed. See Future work, Section 1.4.
A priori constraint strength for generic OE products
Extend the understanding of a priori constraint selection to individual optimal estimation products, independent of the fusion context, to improve general product quality.
Reformulation of fusion with total columns and profiles
Develop explicit CDF(2015) and CDF(2022) formulations for scenarios where total-column measurements are fused with vertical profiles, accounting for the mathematical differences between scalar and vector-valued quantities.
Quantification of coincidence errors
Establish rigorous, data-driven methods to quantify errors arising from imperfect spatial and temporal coincidence of measurements. Current approaches use fixed adjustments based on a priori information; more sophisticated methods that account for actual atmospheric variability patterns are needed.
Interpolation error strategies
Systematically test and compare different mathematical strategies for quantifying and including vertical interpolation errors in the fusion covariance matrices. Evaluate trade-offs between simplicity and accuracy across different atmospheric conditions and measurement types.
Extended vertical interpolation algorithm
Thoroughly test and validate the extended vertical interpolation algorithm (which allows fusion on vertical grids that exceed the intersection of individual retrieval grids) before deploying it in operational settings. Evaluate performance with real atmospheric data.
Numerical error analysis
Conduct detailed studies of truncation and round-off errors in the CDF equations, especially for state vectors with large dynamic range. Develop guidelines for detecting and mitigating numerical instabilities and establish best practices for matrix inversion thresholds.
Auto-consistency test criteria
Develop precise, mathematically rigorous criteria to determine whether an input product passes the auto-consistency test. Move beyond current heuristic approaches (e.g., “differences must be within total error”) to objective, quantitative standards.
Status (2026): for products without an a priori covariance the test is replaced by the symmetry of \mathbf{A}_{i}\mathbf{C}, by test A (identity with the constraint of the retrieval) and by test B (fusion with a reasonable a priori), see the Non-OE and incomplete products page. Test B has a calibrated statistic (the reduced \chi^{2} over the retained directions follows the corresponding \chi^{2} distribution and is read through its p-value, verified on a synthetic retrieval with a common a priori different from the a priori of the product), which is a first non-heuristic criterion; the traffic-light thresholds of all the tests remain heuristic.
Mono-type fusion with simulated data
Design and implement systematic mono-type fusion tests based on single measured profiles used as templates to generate sets of realistic simulated products with controlled noise and systematic differences. This enables controlled validation of the fusion algorithm.
Mono-type fusion test criteria
Establish precise criteria for determining mono-type fusion test success, including quantitative thresholds for acceptable oscillations, error reduction, and DOF improvement. Validate these criteria across diverse datasets and measurement types.
Status (2026): the GOME-2/AC SAF mono-type study selects the fusion configuration (coincidence error coefficient and a priori scaling) by maximising a composite score of DOF gain, closeness to the input mean, error reduction and smoothness, under hard constraints (positive DOF gain, error reduction, at most one negative level, total-variation ratio of the fused profile to the input mean at most 2). Pre-fusion diagnostics (profile difference normalised by the combined errors, its total variation, Mahalanobis \chi^{2}/n between the inputs) separate the pairs for which the literature coincidence coefficient fails from the others better as a classification than as a regression. For the N2O products the indicators are the DOF by altitude band, the error ratio \sigma(N)/\sigma(1) and an oscillation index (sum of the increases of the profile with altitude). The criteria remain empirical.
Fusion of state vectors with components on different vertical grids
Extend CDF to handle products whose state vectors contain different atmospheric species (gases), each defined on its own vertical grid. Current CDF formulations assume all components of the state vector are defined on a common grid. The proposed extension requires developing and implementing separate interpolation matrices for each state vector component (species). This approach, inspired by the multi-target retrieval methodology, must:
- Define component-specific interpolation matrices \mathbf{H}^{(k)} for species k, mapping each species grid to the fusion grid
- Generalize the sampling matrices \mathbf{C}^{(i,k)} and \mathbf{C}^{(f,k)} to account for different grids per component
- Reformulate error covariance modifications to handle species-dependent interpolation errors
- Ensure consistency of the fused averaging kernel matrices and covariance matrices across species boundaries
- Validate the extended algorithm with multi-species atmospheric products (e.g., simultaneous temperature, O3, and H2O retrievals)
This extension is essential for modern atmospheric remote sensing, where multi-species products are increasingly common and often have species-specific vertical resolution limitations.
Consistency of characterisation matrix relationships
A well-posed optimal estimation product must satisfy three fundamental relationships among its characterisation matrices (see Prerequisites section, Eq. P1–P3). The current CDF formulations [Eq. A.1–A.5 for Config A, and generalizations for other configurations] derive the output averaging kernels and covariance matrices directly from the input quantities without explicitly enforcing these relationships. This is both inefficient and risky (errors can propagate unchecked). Develop an alternative CDF(2022) formulation that:
- Leaves the fused state vector \mathbf{x}_{f} unchanged
- Modifies the derivation of \mathbf{S}_{nf} and \mathbf{S}_{sf} to guarantee that the three output matrices satisfy P1–P3
- Provides similar corrections for CDF(2015)
This reformulation would increase robustness and eliminate a potential source of internal inconsistencies in the fused product.
Products not retrieved with optimal estimation, and incomplete products
The CDF takes from each product only the information of the measurement, which can be reconstructed from the AKM and the matching error covariance for any retrieval with a symmetric quadratic constraint (Tikhonov, Levenberg–Marquardt at convergence); the a priori covariance of the product is not needed. This is established, together with the checks that replace the auto-consistency test, on the Non-OE and incomplete products page. Open points:
- Levenberg–Marquardt iterations stopped with residual damping: the equivalent constraint is in general not symmetric and no exact CDF formula exists; the asymmetry of \mathbf{A}_{i}\mathbf{C} bounds the departure but does not correct it (MIPAS N2O: 5–8 %, origin to be confirmed with the processor team)
- Reference profile for products regularised after convergence: \mathbf{x}_{r} = 0 is consistent with the regularisation but passes the smoothing to the fusion a priori; the retrieved profile itself, adopted in the prototype, counts the smoothing twice
- Products with diagonal covariances only: sufficient when the product dominates the information, not where it competes with a tight a priori or another product
- Residual excess of test B on TES N2O in ln(VMR) (\chi^{2} \approx 3), origin unknown
- Products from neural-network or regression retrievals, without AKM: not addressed
A priori covariance of the extra state-vector elements in mono-type fusion
In GOME-2 products the state vector contains, besides the ozone profile, extra elements (surface albedo, wavelength shift) retrieved jointly with it. In the mono-type fusion of pairs of pixels, large albedo differences between the two pixels produce oscillations and negative levels at the literature coincidence coefficient, because the coincidence term \mathbf{S}_{coin} = k_{coin}^{2}\,\mathbf{S}_{a} carries no albedo–profile coupling that could absorb the difference. Inflating the diagonal of \mathbf{S}_{a} for the extra elements reduces the weight that the inverse of the summed covariances assigns to the albedo discrepancy, and with it the perturbation that propagates to the profile through the profile–albedo cross terms of the total error covariance of the coincident pixel; an additive standard deviation of 1.0 was adopted. A rigorous treatment would require the albedo–profile cross terms of \mathbf{S}_{coin}, to be estimated from retrieval statistics: not available.
Mono-type fusion with an increasing number of products
On the N2O products of the July 2011 test days, nested groups of N = 1 to 4 neighbours of the same instrument (up to 8 for IASI-TN2OR) were fused with the same grid and a priori. For MIPAS the error decreases as 1/\sqrt{N} above 8 km (ratio 0.66–0.72 for N = 2), that is, it is dominated by the measurement; for the nadir products and for Odin-SMR it decreases less (0.83–0.98), because the common a priori still weighs. On the same reference profiles a second MIPAS adds 0.42 DOF (0.04 at 8–20 km), the nadir partner 1.09 (0.95 at 8–20 km); a second IASI-TN2OR pixel adds 0.39 DOF against 12.7 added by the MIPAS limb profile. The oscillation index grows with N for the nadir products and for Odin-SMR. Open: the neighbour criterion for limb products (500 km and 6 h were used) and the coincidence coefficient between neighbours (0 and 0.1 were compared; with 0.1 the DOF gains decrease by 0.004–0.10).
Implementation Priorities
These research topics are not equally urgent. The following grouping suggests a prioritization:
| Priority | Topics |
|---|---|
| High | 8, 9, 10, 11, 12, 13 — Define rigorous test criteria (8, 10) and implement simulation framework (9). Topic 11 (fusion on non-overlapping grids) and 12 (matrix consistency) improve algorithm robustness and practical applicability directly. Topic 13 (non-OE and incomplete products) decides which products can enter the fusion at all. |
| Medium | 4, 5, 7, 14, 15 — Address coincidence and interpolation errors (4, 5) and understand numerical stability (7). Topics 14 and 15 (extra state-vector elements, number of products in mono-type fusion) condition the production of mono-type averages. Important for real-world datasets. |
| Medium-Low | 1, 2, 3, 6 — Refine constraint selection (1, 2), extend to total columns (3), validate extended interpolation (6). Important for advanced applications. |