OptimizationLib Reference

OptimizationLib.Optimization — mathematical optimization for Free Pascal.

For dense convex quadratic and second-order-cone models with explicit feasibility and shared termination diagnostics, see the convex optimisation guide. The APIs on this page remain source compatible.

Learning routes

Beginner route

Copy and run the bounded scalar quick start. It prints x = 3.000000 from a plain double-real callback. The scalar convenience call does not expose a destination or workspace.

Common tasks and algorithm choice

Advanced route

Run example 18 for explicit model, feasibility, and termination diagnostics, or example 09 for the compatibility facade. Both use TDoubleArray; options/workspaces and result arrays deepen that same path without undocumented conversions.

---

Quick Start

uses OptimizationLib.Optimization;



function Objective(X: Double): Double;

begin

  Result := Sqr(X - 3.0);

end;



begin

  WriteLn('x = ', TOptimizationKit.GoldenSection(@Objective, 0, 10):0:6);

end.

Expected output:

x = 3.000000

All methods are class static — no Create/Free needed.

---

Function Types

TUnivarFunc   = function(X: Double): Double;

TMultivarFunc = function(const X: TDoubleArray): Double;

TGradFunc     = function(const X: TDoubleArray): TDoubleArray;

TConstraintFunc = function(const X: TDoubleArray): Double;  // g(x) <= 0

---

Result Records

TOptResult = record

  X:         TDoubleArray;  { solution vector }

  FVal:      Double;        { objective value at solution }

  Iters:     Integer;       { iterations used }

  Converged: Boolean;       { True if convergence criterion was met }

  Status: TIterationStatus;

  GradientNorm: Double;

  ConstraintViolation: Double;

  Evaluations: Integer;

  BestX: TDoubleArray;

  BestFVal: Double;

end;



TLPStatus = (lpsOptimal, lpsUnbounded, lpsIterationLimit,

  lpsUnsupportedStart, lpsInfeasible);



TLPResult = record

  X:        TDoubleArray;

  ObjVal:   Double;

  Feasible: Boolean;

  Iters:    Integer;

  Status:   TLPStatus;

end;

---

Solver Selection Guide

---

Detailed solver contract

The newer solvers take TOptimizationOptions.Defaults, with separate absolute, relative, and gradient tolerances; iteration and evaluation limits; line-search/trust controls; history size; optional lower/upper bounds; seed; start count; and a cancellation callback. Empty bound arrays mean unbounded coordinates.

Every returned TOptResult keeps the final X/FVal, detailed Status, gradient norm, maximum constraint violation, evaluations, and the best finite BestX/BestFVal observed. Inspect Status; Converged is only the legacy compatibility view. Cancellation and limits preserve the best finite iterate.

TOptimizationWorkspace is caller-owned and contains a copied WarmStartX, Runs, and TotalEvaluations. BoundedLBFGSWithWorkspace uses the warm start only when its dimension and bounds are valid, then updates the workspace after the run. Clear releases it. Separate workspaces are independent and reentrant; sharing a mutable workspace requires synchronization.

SolveConstrained accepts TSmoothConstraint records. Inequalities use Value(X) <= 0; equalities use Value(X) = 0 within their tolerance. This is a local dense smooth solver, not a proof of global optimality or infeasibility. ExplorePareto is a supplied-weight scalarization baseline and can miss non-convex parts of a Pareto front.

The public collection aliases are TSmoothConstraints, TMultivarFunctions, TOptResults, and TObjectiveMatrix; TConstraintKind selects equality or inequality. TOptimizationProgress is the detailed cancellation callback.

---

Single-Variable Minimization

GoldenSection

xMin := TOptimizationKit.GoldenSection(F, A, B);

xMin := TOptimizationKit.GoldenSection(F, A, B, Tol, MaxIter);

Reduces the interval by the golden ratio each step. Requires f to be unimodal on [A, B] (one valley). Guaranteed O(log(1/Tol)) evaluations.

BrentMinimize

xMin := TOptimizationKit.BrentMinimize(F, A, B);

Combines golden-section with parabolic interpolation. Faster than golden section on smooth functions, same guarantees. Preferred default for 1-D problems.

---

Multi-Variable Minimization

GradientDescent

result := TOptimizationKit.GradientDescent(F, Grad, X0);

result := TOptimizationKit.GradientDescent(F, nil, X0);  // uses numerical gradient

Steepest descent with Armijo backtracking line search. Simple but slow — primarily useful for education or as a baseline.

Parameters: LR (step size, default 0.1), Tol (gradient norm threshold, default 1e-6), MaxIter (default 5000).

Adam

result := TOptimizationKit.Adam(F, Grad, X0);

result := TOptimizationKit.Adam(F, nil, X0, LR := 0.001);

Adaptive moment estimation. The standard optimizer in machine learning. Robust to:

• Non-convex landscapes
• Noisy/stochastic gradients
• Different scales across dimensions

Parameters: LR (0.001), Beta1 (0.9), Beta2 (0.999), Eps (1e-8), Tol (1e-6), MaxIter (10000).

L-BFGS

result := TOptimizationKit.LBFGS(F, Grad, X0);

result := TOptimizationKit.LBFGS(F, nil, X0, M := 10);

Limited-memory quasi-Newton. Uses the last M gradient differences to approximate the inverse Hessian. Much faster than gradient descent on smooth convex problems (super-linear convergence). Memory: O(M × N) instead of O(N²) for full BFGS.

Parameters: M history size (default 10), Tol (1e-6), MaxIter (1000).

NelderMead (Simplex)

result := TOptimizationKit.NelderMead(F, X0);

result := TOptimizationKit.NelderMead(F, X0, Scale, Tol, MaxIter);

Moves a geometric simplex (N+1 points) through the search space. No gradient needed. Works on:

• Non-smooth functions
• Simulation outputs (black-box)
• Functions with noise

Parameters: Scale — initial simplex size (default 1.0, reduce for fine-grained search), Tol (1e-8), MaxIter (10000).

SimulatedAnnealing

result := TOptimizationKit.SimulatedAnnealing(F, X0);

result := TOptimizationKit.SimulatedAnnealing(F, X0, T0, TMin, CoolRate, StepSize, MaxIter, Seed);

Probabilistic global search. Accepts worse solutions with probability exp(-ΔE/T), where T decreases over time (cooling schedule). Can escape local minima — other solvers cannot.

Parameters:

• T0 — initial temperature (default 100; higher = more exploration)
• TMin — stop temperature (default 1e-8)
• CoolRate — cooling factor per step (default 0.995; closer to 1 = slower cooling)
• StepSize — random perturbation size (default 0.1)
• Seed — RNG seed for reproducibility (default 42)

Tuning tips:

• If the solver misses the global minimum: increase T0 or increase CoolRate closer to 1
• If cooling is too slow: decrease CoolRate
• For higher-dimensional problems: increase MaxIter

---

Constrained Optimization

PenaltyMethod

// minimise f(x)  subject to g1(x) <= 0, g2(x) <= 0, ...

result := TOptimizationKit.PenaltyMethod(F, [g1, g2], X0);

Converts the constrained problem to unconstrained by adding a quadratic penalty:

F_pen(x) = F(x) + Mu * Σ max(0, g_i(x))²

Mu is automatically increased across 10 outer iterations to drive constraint violations to zero. Uses NelderMead internally so no gradient is needed.

Example:

function MyObj(const X: TDoubleArray): Double;

begin Result := Sqr(X[0]-5) + Sqr(X[1]-5); end;



function Constraint1(const X: TDoubleArray): Double;

begin Result := X[0] + X[1] - 6; end;  // x1 + x2 <= 6



result := TOptimizationKit.PenaltyMethod(@MyObj,

  [TConstraintFunc(@Constraint1)],

  TDoubleArray.Create(1, 1));

// X ≈ [3, 3],  FVal ≈ 8

---

Linear Programming

SimplexLP

// minimise   c' x

// subject to A x <= b,   x >= 0



lp := TOptimizationKit.SimplexLP(C, A, B);

if lp.Feasible then

  WriteLn('Optimal: ', lp.ObjVal);

Feasible is retained as a compatibility flag and is True only when Status = lpsOptimal. Inspect Status to distinguish an unbounded model, infeasibility, and an iteration limit.

Standard form requirements:

• Variables implicitly >= 0
• Only <= constraints are supported
• A two-phase tableau constructs a feasible basis when a right-hand side is negative and returns lpsInfeasible when Phase I cannot remove artificial feasibility

Example — capacity-constrained production:

// maximise x1+x2 by minimising -x1-x2

// subject to x1+x2 <= 4, x1 <= 3, x2 <= 3, and x >= 0



lp := TOptimizationKit.SimplexLP(

  TDoubleArray.Create(-1, -1),

  [TDoubleArray.Create(1, 1),

   TDoubleArray.Create(1, 0),

   TDoubleArray.Create(0, 1)],

  TDoubleArray.Create(4, 3, 3));

Feasible remains False for every non-optimal status; Status distinguishes infeasibility, unboundedness, and a limit.

---

Utilities

Numerical Gradient

// Central differences: (f(x+h*e_i) - f(x-h*e_i)) / 2h

G := TOptimizationKit.NumGrad(F, X);

G := TOptimizationKit.NumGrad(F, X, H := 1E-5);

Useful for verifying analytical gradients or using gradient-based solvers without implementing a gradient function.

Maximize

// Find the maximum of F by minimising -F internally

result := TOptimizationKit.Maximize(F, X0);

// result.FVal is the maximum value (positive)

---

Passing Nil for Gradient

All gradient-based solvers (GradientDescent, Adam, LBFGS) accept nil for the Grad parameter. They will compute a numerical gradient internally using central differences.

// Both of these work:

result := TOptimizationKit.LBFGS(@MyFunc, @MyGrad, X0);   // analytical

result := TOptimizationKit.LBFGS(@MyFunc, nil, X0);        // numerical

Numerical gradients are slightly slower (~2N function evaluations per gradient) but require no extra implementation.

---

Error Handling

EOptimizationError is raised for:

• GoldenSection / BrentMinimize: B <= A
• Any multi-variable solver: empty X0
• SimplexLP: no constraints or no variables provided
• nil callbacks, non-finite inputs or callback results, and gradient dimension mismatches
• non-positive tolerances or iteration counts and invalid solver hyperparameters
• ragged or mismatched linear-program dimensions
• scalar solvers exhausting their iteration limit

PenaltyMethod and Maximize use unit-level callback state to bridge Free Pascal procedure-variable restrictions. Calls through these adapters are serialized so overlapping threads cannot corrupt the shared callback state.

---

Dependencies

• MathBase.SharedTypes — TDoubleArray

No other external libraries required.