Fractional Differentiation

This chapter introduces fractional differentiation as a method to solve the "Stationarity vs. Memory Dilemma." This is the core problem where standard data transformations create a conflict:

• Prices (d=0) have memory (predictive power) but are non-stationary.
• Returns (d=1) are stationary but are memory-less.

ML models need stationarity, but they also need memory to have predictive power. The standard practice of using returns (d=1) often "over-differentiates" the data, wiping out valuable memory and reinforcing the efficient market hypothesis.

Fractional differentiation finds the minimum amount of differentiation $d$ (where $d$ is a real number, e.g., $d=0.4$) required to make a series stationary, thereby preserving the maximum possible memory.

The Method

The method generalizes the integer differencing operator $(1-B)^d$ to allow $d$ to be any real number. This is based on the binomial series expansion of the backshift operator $B$:

A fractionally differentiated series $\tilde{X}_t$ is the dot product of the original series $X_t$ and a set of weights $\omega$:

When $d$ is an integer (like 1), the weights $\omega_k$ become $0$ for $k > d$, cutting off all memory. When $d$ is a non-integer, the weights converge to zero but never become exactly zero, thus preserving memory.

The weights $\omega_k$ can be calculated iteratively (with $\omega_0=1$):

Implementation

The chapter compares two implementation methods:

1. Expanding Window (Standard Method): This method uses an increasing number of data points to compute the weights for each subsequent observation. This is flawed as it causes a negative drift in the transformed series.

2. Fixed-Width Window Fracdiff (FFD): This is the author's preferred method.

   • It first determines a fixed number of weights $l^*$ by finding where the weight modulus $|\omega_{l^*}|$ falls below a given tolerance threshold $\tau$.
   • This same fixed set of weights is then applied to all observations.
   • This method avoids the negative drift and produces a stationary series that retains memory.

Finding the Optimal $d$

The primary goal is to find the minimum differentiation $d^*$ that makes a series stationary. This is achieved by:

1. Generating multiple FFD series for $d$ values in the range $[0, 1]$.
2. Running an Augmented Dickey-Fuller (ADF) test on each series.
3. Identifying the minimum $d$ where the ADF statistic falls below the 95% confidence level critical value.

An example on E-mini S&P 500 futures shows that while the ADF critical value is -2.86, the original series is -0.33 (non-stationary) and the returns series (d=1) is -46.91 (hyper-stationary). The series becomes stationary at $d \approx 0.35$, while still retaining a 0.995 correlation with the original series. In contrast, returns (d=1) only have a 0.03 correlation, showing that all memory was destroyed.

The conclusion is that most financial analysis is over-differentiated, and FFD provides a "third way" to get stationary, memory-filled data for ML models.

Implementation: Fractional Differentiation

In RiskLabAI, we implement fractional differentiation in the data.differentiation.differentiation module. This technique allows us to make a time series stationary while preserving memory, which is crucial for financial machine learning models.

We provide two main methods for differentiation:

1. Standard (Expanding Window): This method uses all available history for each data point. It is more memory-intensive but used for finding the optimal 'd'.
2. Fixed-Width Window (FFD): This is the preferred method for feature generation. It uses a fixed window determined by a weight threshold, making it faster and preventing the series from fading to zero. We provide a highly optimized version using np.convolve.

Finding the Optimal 'd'

We also provide utility functions to find the minimum differentiation factor d that results in a stationary series (as determined by the ADF test).

The fractionally_differentiated_log_price function is particularly useful, as it iterates d from 0 upwards by step until the ADF test p-value drops below the p_value_threshold, returning the stationary series.

API reference

RiskLabAI implements these in Python and Julia (signatures auto-generated from the package source):

def calculate_weights_std(degree: float, size: int) -> np.ndarray:

function calculate_weights_std(degree::Real, size::Integer)

def calculate_weights_ffd(degree: float, threshold: float = 1e-5) -> np.ndarray:

function calculate_weights_ffd(degree::Real, threshold::Real = 1e-5)

def fractional_difference_std(
    series: pd.DataFrame, degree: float, threshold: float = 0.01
) -> pd.DataFrame:

function fractional_difference_std(
    series::AbstractVector{<:Real},
    degree::Real;
    threshold::Real = 0.01,
)

def fractional_difference_fixed(
    series: pd.DataFrame, degree: float, threshold: float = 1e-5
) -> pd.DataFrame:

function fractional_difference_fixed(
    series::AbstractVector{<:Real},
    degree::Real;
    threshold::Real = 1e-5,
)

def find_optimal_ffd_simple(
    input_series: pd.DataFrame, p_value_threshold: float = 0.05
) -> pd.DataFrame:

function find_optimal_ffd(
    close_prices::AbstractVector{<:Real};
    p_value_threshold::Real = 0.05,
)

def fractionally_differentiated_log_price(
    input_series: pd.Series,
    threshold: float = 1e-5,
    step: float = 0.01,
    p_value_threshold: float = 0.05,
) -> pd.Series:

function fractionally_differentiated_log_price(
    prices::AbstractVector{<:Real};
    threshold::Real = 1e-5,
    step::Real = 0.01,
    p_value_threshold::Real = 0.05,
)

Full source: Python · Julia