Figure of Merit (FoM)

What is the figure of merit?

When we fit a model to some data, the figure of merit is the value of a merit function, which calculates how different the model is from the data. This means we can use it as a quantitative assessment of how ‘good’ a model is; the smaller the figure of merit, the closer our model is to the data.

MDMC uses the chi-squared test as a merit function; this is popular for non-linear optimisation.

Indeed, from the use of chi-squared one can note overlap between hypothesis testing and assessment of merit - in hypothesis testing, our ‘model’ is the result we’d expect under the null hypothesis, and our data is the results of our experiment. The hypothesis test is a quantitative indicator of how different these are.

How does MDMC use the figure of merit?

MDMC uses figure of merit as the basis of refinement. We create a simulation with some parameters, calculate the figure of merit of a dynamical property between our simulation and experimental data, and use Minimizers to adjust the simulation parameters in order to minimize the figure of merit.

The chi-squared figure of merit test

In MDMC, the dynamical property is called an Observable. A number of datasets can be given to an optimisation with weights of how important each one is to the refinement. MDMC calculates the weighted average of the Figure of Merits:

\[FoM_{total} = \sum_{i}\frac{FoM_{i}}{w_{i}}\]

Here the weighted Figure of Merit for the \(i\)-th dataset, \(FoM_{i}\), is given by a sum of the square difference between data points for a single pair of observables, normalised by the errors and the number of data points, i.e. the reduced chi-squared:

\[FoM_{i} = \frac{w_{i}}{\nu_{i}} \sum\limits_{j} \frac{(D_{j}^{\textrm{exp}} - D_{j}^{\textrm{sim}})^2}{(\sigma_{j}^{\textrm{exp}})^2}\]

where the sum is over the \(N_{i}\) data points in the pair of observables corresponding to the \(i\)-th dataset, and the normalisation factor \(\nu_{i}\) is either \(N_{i}\), \(N_{i} - M\) where \(M\) is the number of fitting parameters, or \(1\). \(w_{i}\) is an importance weighting assigned to the \(i\)-th dataset.

The sets \(D_{j}^{\textrm{sim}}\) and \(D_{j}^{\textrm{exp}}\) are the individual data points of the experimental and simulated Observable respectively, and \(\sigma_{j}^{\textrm{exp}}\) corresponds to the error of the \(j\)-th data point. Note that the subtraction and division over the arrays are element-wise.

Rescaling

Note also that if the experimental Observable is not on an absolute scale, an additional rescale_factor can be specified (or automatically determined) in the code to scale the experimental data points and errors by a simple linear scaling.

If automatically determined, this rescaling is done to the value which minimizes the FoM. If we label the rescale factor \(\lambda\) then the minimum of the FoM is obtained as:

\[\begin{split}FoM_{i}(\lambda) &=& w_{i} \sum\limits_{j} \left(\frac{\lambda*D_{j}^{\textrm{exp}} - \\\\ D_{j}^{\textrm{sim}}}{\lambda*\sigma_{j}^{\textrm{exp}}\right)^2 \\\\ \left. \frac{dFoM_{i}}{d\lambda}\right|_{\lambda=\lambda_{\textrm{min}}} &=& 0 \\\\ \lambda_{\textrm{min}} &=& \frac{A}{B} \\\\\end{split}\]

where we have:

\[\begin{split}A &=& \sum\limits_{j}\left(\frac{D_{j}^{\textrm{sim}}}{\sigma_{j}^{\textrm{exp}}}\right)^2 \\\\ B &=& \sum\limits_{j} \frac{D_{j}^{\textrm{exp}}*D_{j}^{\textrm{sim}}}{(\sigma_{j}^{\textrm{exp}})^2}\end{split}\]