Floating Point Error Estimation using CHEF-FP

Introduction

Data-intensive sciences that work with increasing data volumes and often in heterogeneous computing environments require floating point stability. Robust floating-point error detection can help reduce data reprocessing costs and help develop important new lossy compression algorithms.

How does Automatic Differentiation (AD) fit into this?

AD helps evaluate the exact derivative of a function. AD applies the differential calculus chain rule throughout the semantics of the original program. In the context of FP error estimation, the implementation relies on:

  • Reverse-Mode AD (as opposed to Forward-Mode AD), since it provides the derivative of the function with respect to all intermediate and input variables, and

  • Source Transformation (as opposed to Operator Overloading), since it does most of the work at compile time. The Clad Framework that is used in this research also uses source transformation.

Clad is implemented as a plugin for the Clang compiler. It inspects the internal compiler representation of the target function to generate its derivative. Clad requires little or no code modification, supports a growing subset of C++ constructs, statements and data types, it enables efficient gradient computation for large and complex codebases, and is deeply integrated with the compiler, allowing automatic generation of error estimation code.

Where does the CHEF-FP logic reside?

CHEF-FP (the tool created for Floating Point Error Estimation) is a flexible, scalable, and easy-to-use source-code transformation tool based on Automatic Differentiation (AD) for analyzing approximation errors in HPC applications.

The main logic for CHEF-FP resides in the following files:

How does the FPEE Logic work?

While parsing the code using Clad, if it encounters a floating point variable, it needs to be tracked (to accumulate relevant errors against that variable). Next, the Error Estimation Calculation Formula (Error Model) needs to be built (using EstimationModel.h).

EstimationModel.h contains the information needed to calculate the estimate value of the error. It is highly customizable (e.g., you can plug in your own custom formula as well). The default formula multiplies the derivative (dfdx) with the value of the variable (delta_x), for which the error estimate is required, and the machine epsilon (Em).

std::abs(dfdx * delta_x * Em)

For this formula to work, the value of the variable (delta_x) should be saved at the relevant time.

This model will return a formula that is represented using a Clang expression.This Clang expression can, in turn, be written into the derivative code that is generated using Clad.

What else is CHEF-FP capable of?

Sensitivity Analysis

A lot of information can be extracted from the intermediate floating point errors. This includes sensitivity, that is, how sensitive a particular variable is to floating point errors. The lower the sensitivity, the smaller the likelihood that the variable will have a large impact on the total floating point error of the function.

Note: a more complex model may or may not have a direct relationship with sensitivity.

This has important implications on the numerical stability of the algorithms. It also builds the foundation for the set of type-optimization techniques called Mixed Precision Tuning.

Mixed Precision Tuning involves demoting certain types to lower sensitivity, and subsequently, a lower contribution to the function’s final error.

How do I create my own Custom model?

Custom Models may be one of the main reasons that new users may be interested in adapting the CHEF-FP code to their specific use cases.

Top define a custom model using Clad:

1. Implement the clad::FPErrorEstimationModel class, a generic interface that provides the error expressions for clad to generate.

2. Override the AssignError() function. This function is called for all LHS of every assignment expression in the target function.

The function AssignError() represents the mathematical formula of an error model in a form that Clang can understand and convert to code. It provides users with a reference to the variable of interest and its derivative. The user, in turn, must return an expression that will be used to accumulate the error.

Note: Creating these functions requires knowledge of the Clang APIs.

Demo customization examples can be found here:

The CustomModel and PrintModel demos are useful for users who would like to write their own models.

Further Reading

For more technical details, please view:

Appendix - Notable Classes

clad::ErrorEstimator::ErrorEstimationHandler

The ErrorEstimationHandler class is used to estimate errors in a target function. When you use Clad in Error Estimation mode, the ErrorEstimationHandler class is responsible for handling the derivative and error information exchange between Clad and the Error Estimation module (CHEF-FP). This class is responsible for a lot of the housekeeping tasks as well.

clad::ErrorEstimator::EmitFinalErrorStmts

This function adds the final error and the other parameter errors to the forward block.