Initialisation

Model Initialisation

To initialise a model for profile likelihood evaluation we use initialise_LikelihoodModel which returns a struct of type LikelihoodModel. This struct contains all the information we require for the PWA workflow and will also contain computed profiles and profile-wise predictions.

LikelihoodBasedProfileWiseAnalysis.initialise_LikelihoodModelFunction
initialise_LikelihoodModel(loglikefunction::Function,
    predictfunction::Union{Function, Missing},
    errorfunction::Union{Function, Missing}
    data::Union{Tuple, NamedTuple},
    θnams::Vector{<:Symbol},
    θinitialguess::AbstractVector{<:Real},
    θlb::AbstractVector{<:Real},
    θub::AbstractVector{<:Real},
    θmagnitudes::AbstractVector{<:Real}=Float64[];
    <keyword arguments>)

Initialises a LikelihoodModel struct, which contains all model information, profiles, samples and predictions. Solves for the maximum likelihood estimate of loglikefunction.

Arguments

  • loglikefunction: a log-likelihood function to maximise which takes two arguments, θ and data, in that order, where θ is a vector containing the values of each parameter in θnames and data is a Tuple or NamedTuple - see data below. Set up to be used in a maximisation objective.
  • predictfunction: a prediction function to generate model predictions from that is paired with the loglikefunction. Requirements for the prediction function can be seen in add_prediction_function!. It can also be missing if no function is provided to initialise_LikelihoodModel, because predictions are not required when evaluating parameter profiles. The function can be added at a later point using add_prediction_function!.
  • errorfunction: an error function used to predict realisations from predictions generated with predictfunction. Requirements for the error function can be seen in add_error_function!. It can also be missing if no function is provided to initialise_LikelihoodModel, because predictions are not required when evaluating parameter profiles. The function can be added at a later point using add_error_function!.
  • data: a Tuple or a NamedTuple containing any additional information required by the log-likelihood function, such as the time points to be evaluated at.
  • θnames: a vector of symbols containing the names of each parameter, e.g. [:λ, :K, :C0].
  • θinitialguess: a vector containing the initial guess for the values of each parameter. Used to find the MLE point.
  • θlb: a vector of lower bounds on parameters.
  • θub: a vector of upper bounds on parameters.
  • θmagnitudes: a vector of the relative magnitude of each parameter. If not provided, it will be estimated using the difference of θlb and θub with LikelihoodBasedProfileWiseAnalysis.calculate_θmagnitudes. Can be updated after initialisation using setmagnitudes!.

Keyword Arguments

  • optimizationsettings: optimization settings used to optimize the log-likelihood function using Optimization.jl. Default is default_OptimizationSettings() (see default_OptimizationSettings).
  • uni_row_prealloaction_size: number of rows of uni_profiles_df to preallocate. Default is missing (a single row).
  • biv_row_preallocation_size: number of rows of biv_profiles_df to preallocate. Default is missing (a single row).
  • dim_row_preallocation_size: number of rows of dim_samples_df to preallocate. Default is missing (a single row).
  • find_zero_atol: a Real number greater than zero for the absolute tolerance of the log-likelihood function value from the target value to be used when searching for confidence intervals/boundaries. Default is 0.001.
  • show_progress: Whether to show the progress of profiling and predictions.
Array initialisation within a log-likelihood function

If you initialise an array within the provided log-likelihood function, e.g. using zeros, then for automatic differentiation methods to work you need to also initialise the type of the array to be based on the type of the input values of θ. Otherwise, zeros will by default create an array with element types Float64 which will likely return errors. For example, use:

my_new_array = zeros(eltype(θ), dimensions)

where eltype passes the element type of the θ vector into the zeros function.

source
initialise_LikelihoodModel(loglikefunction::Function,
    predictfunction::Function,
    data::Union{Tuple, NamedTuple},
    θnames::Vector{<:Symbol},
    θinitialGuess::Vector{<:Float64},
    θlb::Vector{<:Float64},
    θub::Vector{<:Float64},
    θmagnitudes::Vector{<:Real}=zeros(0);
    <keyword arguments>)

Alternate version of initialise_LikelihoodModel that can be called without a error function. The function can be added at a later point using add_error_function!.

source
initialise_LikelihoodModel(loglikefunction::Function,
    data::Union{Tuple, NamedTuple},
    θnames::Vector{<:Symbol},
    θinitialGuess::Vector{<:Float64},
    θlb::Vector{<:Float64},
    θub::Vector{<:Float64},
    θmagnitudes::Vector{<:Real}=zeros(0);
    <keyword arguments>)

Alternate version of initialise_LikelihoodModel that can be called without a prediction and error function. The functions can be added at a later point using add_prediction_function! and add_error_function!.

source

Model Representation

LikelihoodBasedProfileWiseAnalysis.LikelihoodModelType
LikelihoodModel(core::Union{CoreLikelihoodModel, BaseLikelihoodModel}, 
    ellipse_MLE_approx::Union{Missing, EllipseMLEApprox},
    num_uni_profiles::Int, 
    num_biv_profiles::Int, 
    num_dim_samples::Int, 
    uni_profiles_df::DataFrame, 
    biv_profiles_df::DataFrame, 
    dim_samples_df::DataFrame, 
    uni_profile_row_exists::Dict{Tuple{Int, AbstractProfileType}, DefaultDict{Float64, Int}},
    biv_profile_row_exists::Dict{Tuple{Tuple{Int, Int}, AbstractProfileType, AbstractBivariateMethod}, DefaultDict{Float64, Int}},
    dim_samples_row_exists::Dict{Union{AbstractSampleType, Tuple{Vector{Int}, AbstractSampleType}}, DefaultDict{Float64, Int}},
    uni_profiles_dict::Dict{Int, UnivariateConfidenceStruct}, 
    biv_profiles_dict::Dict{Int, BivariateConfidenceStruct}, 
    dim_samples_dict::Dict{Int, SampledConfidenceStruct},
    uni_predictions_dict::Dict{Int, AbstractPredictionStruct}, 
    biv_predictions_dict::Dict{Int, AbstractPredictionStruct}, 
    dim_predictions_dict::Dict{Int, AbstractPredictionStruct},
    show_progress::Bool)

Mutable struct containing all the information required to compute profiles, samples and predictions. Created by initialise_LikelihoodModel.

Fields

  • core: a CoreLikelihoodModel or BaseLikelihoodModel struct.
  • ellipse_MLE_approx: a EllipseMLEApprox struct OR a missing value if the ellipse approximation of the log-likelihood at the MLE point has not been evaluated yet.
  • num_uni_profiles: the number of different univariate profiles that have been evaluated (distinct combinations of different confidence levels, AbstractProfileType structs and single interest parameters). Specifies the number of valid rows in uni_profiles_df.
  • num_biv_profiles: the number of different bivariate profiles that have been evaluated (distinct combinations of different confidence levels, AbstractProfileType structs, AbstractBivariateMethod structs and two interest parameters). Specifies the number of valid rows in biv_profiles_df.
  • num_dim_samples: the number of different dimensional profiles that have been evaluated (distinct combinations of different confidence levels, AbstractProfileType structs, AbstractSampleType structs and sets of interest parameters). Specifies the number of valid rows in dim_samples_df.
  • uni_profiles_df: a DataFrame with each row containing information on each univariate profile evaluated, where the row index is the key for that profile in uni_profiles_dict and uni_predictions_dict.
  • biv_profiles_df: a DataFrame with each row containing information on each bivariate profile evaluated, where the row index is the key for that profile in biv_profiles_dict and biv_predictions_dict.
  • dim_samples_df: a DataFrame with each row containing information on each dimensional sample evaluated, where the row index is the key for that sample in dim_samples_dict and dim_predictions_dict.
  • uni_profile_row_exists: a dictionary containing information on whether a row in uni_profiles_df exists for a given combination of interest parameter, degrees of freedom, AbstractProfileType and confidence level. If it does exist, the value of uni_profile_row_exists[(θi, dof, profile_type)][confidence_level] will be the row index in uni_profiles_df, otherwise it will be 0.
  • biv_profile_row_exists: a dictionary containing information on whether a row in biv_profiles_df exists for a given combination of two interest parameters, AbstractProfileType, AbstractBivariateMethod and confidence level. If it does exist, the value of biv_profile_row_exists[((θi, θj), dof, profile_type, bivariate_method)][confidence_level] will be the row index in biv_profiles_df otherwise it will be 0.
  • dim_samples_row_exists: a dictionary containing information on whether a row in dim_samples_df exists for a given combination of interest parameter, AbstractProfileType, AbstractSampleType and confidence level. If it does exist, it's value will be the row index in dim_samples_df otherwise it will be 0. For a full likelihood sample this can be accessed using dim_samples_row_exists[sample_type][confidence_level]. For a lower dimensional sample this can be accessed using dim_samples_row_exists[(θindices, dof, sample_type)][confidence_level].
  • uni_profiles_dict: a dictionary with keys of type Integer and values of type UnivariateConfidenceStruct containing the profile for each valid row in uni_profiles_df. The row index of uni_profiles_df is the key for the corresponding profile.
  • biv_profiles_dict: a dictionary with keys of type Integer and values of type BivariateConfidenceStruct containing the profile for each valid row in biv_profiles_df. The row index of biv_profiles_df is the key for the corresponding profile.
  • dim_samples_dict: a dictionary with keys of type Integer and values of type SampledConfidenceStruct containing the profile for each valid row in dim_samples_df. The row index of dim_samples_df is the key for the corresponding profile.
  • uni_predictions_dict: a dictionary with keys of type Integer and values of type PredictionStruct containing the predictions from the profiles in uni_profiles_dict for each valid row in uni_profiles_df. The row index of uni_profiles_df is the key for the corresponding prediction, if that prediction has been calculated using generate_predictions_univariate!.
  • biv_predictions_dict: a dictionary with keys of type Integer and values of type PredictionStruct containing the predictions from the profiles in biv_profiles_dict for each valid row in biv_profiles_df. The row index of biv_profiles_df is the key for the corresponding prediction, if that prediction has been calculated using generate_predictions_bivariate!.
  • dim_predictions_dict: a dictionary with keys of type Integer and values of type PredictionStruct containing the predictions from the profiles in dim_samples_dict for each valid row in dim_samples_df. The row index of dim_samples_df is the key for the corresponding prediction, if that prediction has been calculated using generate_predictions_dim_samples!.
  • find_zero_atol: a Real number greater than zero for the absolute tolerance of the log-likelihood function value from the target value to be used when searching for confidence intervals/boundaries.
  • show_progress: a boolean specifying whether to show the progress of profile methods with respect to sets of interest parameter(s).

Supertype Hiearachy

LikelihoodModel <: Any

source
LikelihoodBasedProfileWiseAnalysis.CoreLikelihoodModelType
CoreLikelihoodModel(loglikefunction::Function, 
    predictfunction::Union{Function, Missing}, 
    errorfunction::Union{Function, Missing}
    data::Union{Tuple, NamedTuple}, 
    θnames::AbstractVector
    θname_to_index::Dict{Symbol, Int}, 
    θlb::AbstractVector{<:Real}, 
    θub::AbstractVector{<:Real}, 
    θmagnitudes::AbstractVector{<:Real}, 
    θmle::Vector{<:Float64}, 
    ymle::Array{<:Real}, 
    maximisedmle::Float64, 
    num_pars::Int)

Struct containing the core information required to define a LikelihoodModel. For additional information on parameters (where repeated), see initialise_LikelihoodModel.

Fields

  • loglikefunction: a log-likelihood function which takes two arguments, θ and data, in that order. Set up to be used in a maximisation objective.
  • predictfunction: a prediction function to generate model predictions from that is paired with the loglikefunction.
  • errorfunction: an error function used to predict realisations from predictions generated with predictfunction.
  • optimizationsettings: a OptimizationSettings struct of settings to use for optimisation unless others are specified.
  • data: a Tuple or a NamedTuple containing any additional information required by the log-likelihood function, such as the time points to be evaluated at.
  • θnames: a vector of symbols containing the names of each parameter, e.g. [:λ, :K, :C0].
  • θname_to_index: a dictionary with keys of type Symbol and values of type Int, with the key being an element of θnames and the value being the corresponding index of the key in θnames.
  • θlb: a vector of lower bounds on parameters.
  • θub: a vector of upper bounds on parameters.
  • θmagnitudes: a vector of the relative magnitude of each parameter.
  • θmle: a vector containing the maximum likelihood estimate for each parameter.
  • ymle: an array containing the output of the prediction function at θmle and data.
  • maximisedmle: the value of the log-likelihood function evaluated at θmle.
  • num_pars: the number of model parameters, length(θnames).

Supertype Hiearachy

CoreLikelihoodModel <: Any

source
LikelihoodBasedProfileWiseAnalysis.BaseLikelihoodModelType
BaseLikelihoodModel(data::Union{Tuple, NamedTuple}, 
    θnames::AbstractVector,
    θname_to_index::Dict{Symbol, Int}, 
    θlb::AbstractVector{<:Real}, 
    θub::AbstractVector{<:Real}, 
    θmagnitudes::AbstractVector{<:Real}, 
    θmle::Vector{<:Float64}, 
    ymle::Array{<:Real}, 
    maximisedmle::Float64, 
    num_pars::Int)

Version of CoreLikelihoodModel without functions, allowing loading of the model struct without those functions defined.

Supertype Hiearachy

BaseLikelihoodModel <: Any

source

Ellipse Approximation

In order to evaluate the ellipse approximation of the normalised log-likelihood function we need to evaluate the observed Fisher information matrix (FIM) [2]. The observed FIM is a quadratic approximation of the curvature of the log-likelihood function at the maximum likelihood estimate (MLE) for parameters. This then allows us to define the ellipse approximation of the log-likelihood function, represented as the EllipseApproxAnalytical and EllipseApprox profile types. These have corresponding equations to evaluate their approximation to the log-likelihood function, as given by LikelihoodBasedProfileWiseAnalysis.analytic_ellipse_loglike and LikelihoodBasedProfileWiseAnalysis.ellipse_loglike, respectively.

LikelihoodBasedProfileWiseAnalysis.EllipseMLEApproxType
EllipseMLEApprox(Hmle::Matrix{<:Float64}, Γmle::Matrix{<:Float64})

Struct containing two n*n arrays representing the ellipse approximation of the log-likelihood function around the MLE point. See getMLE_ellipse_approximation!

Fields

  • Hmle: a n*n array, where n is the number of model parameters, containing the negative Hessian of the log-likelihood function evaluated at the MLE point.
  • Γmle: a n*n array, where n is the number of model parameters, containing the inverse of Hmle.

Supertype Hiearachy

EllipseMLEApprox <: Any

source

Modifying Parameter Magnitudes and Bounds

If desired, the defined parameter magnitudes and bounds contained with a LikelihoodModel can be updated using setmagnitudes! and setbounds!.

LikelihoodBasedProfileWiseAnalysis.setbounds!Function
setbounds!(model::LikelihoodModel; 
    lb::AbstractVector{<:Real}=Float64[], 
    ub::AbstractVector{<:Real}=Float64[])

Updates the parameter bounds in model from model.core.θlb to lb if specified and from model.core.θub to ub if specified. lb and ub are keyword arguments.

source

Parameter Transformations

To assist with parameter transformations we provide functions for transforming parameter bounds given a monotonic forward mapping from the original parameter space to the transformed parameter space.

LikelihoodBasedProfileWiseAnalysis.transformboundsFunction
transformbounds(transformfun::Function, 
    lb::AbstractVector{<:Real}, 
    ub::AbstractVector{<:Real}, 
    independentParameterIndexes::Vector{<:Int}=Int[], 
    dependentParameterIndexes::Vector{<:Int}=Int[])

Given a monotonic (increasing or decreasing) function, transformfun, that describes a parameter transformation, return the lower and upper bounds in the transformed space that correspond to the lower and upper bounds in the original space. Uses a heuristic to evaluate the bound transformation. We assume that the ordering of parameters stay the same for the purposes of independentParameterIndexes and dependentParameterIndexes.

Arguments

  • transformfun: a function describing the forward transformation between parameter space θ and Θ. Should take in a single argument, θ, a vector of parameter values in the original space and return Θ, a vector parameter values in the transformed space. These vectors need to be the same length as lb and ub.
  • lb: a vector of lower bounds on parameters.
  • ub: a vector of upper bounds on parameters.
  • independentParameterIndexes: a vector of parameter indexes where the new parameter Θ[i] depends only on transformfun(θ[i]).
  • dependentParameterIndexes: a vector of parameter indexes where the new parameter Θ[i] depends on transformfun(θ[i], θ[j], j!=i).

The heuristic used for dependent parameters may fail if there are multiple local minima for the appropriate bounds to use. In this case transformbounds_NLopt should be used.

Warns if any of the returned bounds are Inf using LikelihoodBasedProfileWiseAnalysis.checkforInf.

source
LikelihoodBasedProfileWiseAnalysis.transformbounds_NLoptFunction
transformbounds_NLopt(transformfun::Function, 
    lb::AbstractVector{<:Real}, 
    ub::AbstractVector{<:Real})

Given a monotonic (increasing or decreasing) function, transformfun, that describes a parameter transformation, return the lower and upper bounds in the transformed space that correspond to the lower and upper bounds in the original space. Uses a naturally binary integer programme if transformfun is monotonic on θ between lb and ub.

Arguments

  • transformfun: a function describing the forward transformation between parameter space θ and Θ. Should take in a single argument, θ, a vector of parameter values in the original space and return Θ, a vector parameter values in the transformed space. These vectors need to be the same length as lb and ub.
  • lb: a vector of lower bounds on parameters.
  • ub: a vector of upper bounds on parameters.

Warns if any of the returned bounds are Inf using LikelihoodBasedProfileWiseAnalysis.checkforInf.

source

Example Usage

For example, if we want to log transform the parameterisation given a defined log-likelihood function and lower and upper bounds we:

Calculate the new, log-transformed bounds using the forward transformation log.(θ). This is easy enough to be done without transformbounds_NLopt, but other more complex transformations may not be.

lb, ub = ..., ...

f(x) = log.(x)
lb_log, ub_log = transformbounds_NLopt(f, lb, ub)

Define a new log-likelihood function which uses the backward transformation exp.(θ) from the transformed parameter space to the original space.

function loglike(θ, data)
    ...
end

function loglike_log(Θ, data)
    return loglike(exp.(Θ), data)
end

Optimization Settings

We can set our default optimisation settings using a OptimizationSettings struct. This will be contained within the CoreLikelihoodModel field of a LikelihoodModel and can be passed as an option to initialise_LikelihoodModel. Unless different ones are passed to functions for computing profiles, they will also be used for that purpose. It may be useful to compute the MLE using conservative settings for accuracy, and then use less conservative settings for the optimisation of nuisance parameters along parameter profiles.

LikelihoodBasedProfileWiseAnalysis.create_OptimizationSettingsFunction
create_OptimizationSettings(model::LikelihoodModel;
    adtype::Union{SciMLBase.AbstractADType, Missing}=missing, 
    solve_alg=missing, 
    solve_kwargs::Union{NamedTuple, Missing}=missing)

Method for creating a OptimizationSettings struct with each field of the struct as a keyword argument. If a keyword argument is not provided, then the setting in model.core.optimizationsettings is used (the currently set optimization settings).

source
create_OptimizationSettings(;
    adtype::Union{SciMLBase.AbstractADType, Missing}=missing, 
    solve_alg=missing, 
    solve_kwargs::Union{NamedTuple, Missing}=missing)

Method for creating a OptimizationSettings struct with each field of the struct as a keyword argument. If a keyword argument is not provided, then the default setting in default_OptimizationSettings is used.

source
LikelihoodBasedProfileWiseAnalysis.OptimizationSettingsType
OptimizationSettings(adtype::SciMLBase.AbstractADType,
    solve_alg,
    solve_kwargs::NamedTuple)

Contains the optimization settings used to solve for nuisance parameters of the log-likelihood function using Optimization.jl. Note: Optimization.jl uses a minimisation objective, while the log-likelihood functions are setup with a maximisation objective in mind. This is handled inside the package. Defaults can be seen in default_OptimizationSettings.

Fields

  • adtype: a method for automatically generating derivative functions of the log-likelihood function being optimised. For available settings see Automatic Differentiation Construction Choice Recommendations. Note: the corresponding package to adtype needs to be loaded with using. e.g. setting adtype = AutoFiniteDiff() requires using FiniteDiff. Derivative-based algorithms in solve_alg will require an adtype to be specified.
    • AutoFiniteDiff(), AutoForwardDiff(), AutoReverseDiff(), AutoZygote() and AutoTracker() have been tested to work for finding the maximum likelihood estimate with regular model parameters.
    • For profiling, AutoFiniteDiff(), AutoForwardDiff(), AutoReverseDiff() and AutoTracker() have been tested to work with regular model parameters.
    • If the variance of the error distribution is included as a parameter to be estimated, AutoFiniteDiff(), AutoForwardDiff() and AutoTracker() have been tested to work for finding the MLE and profiling.
    • AutoFiniteDiff() will always work, regardless of model specification, but may be less optimal than other methods.
  • solve_alg: an algorithm to use to solve for the nuisance parameters of the log-likelihood function defined within Optimization.jl. The package is loaded with the Optimization integration of NLopt, so any of the NLopt algorithms are available without having to load another package (see OptimizationNLopt). Good starting methods may be NLopt.LN_BOBYQA(), NLopt.LN_NELDERMEAD() and NLopt.LD_LBFGS. Other packages can be used as well - see Overview of the Optimizers.
  • solve_kwargs: a NamedTuple of keyword arguments used to set solver options like maxiters and maxtime. For a list of common solver arguments see: Common Solver Options. Other specific package arguments may also be available. For NLopt see Methods.

Supertype Hiearachy

OptimizationSettings <: Any

source
LikelihoodBasedProfileWiseAnalysis.optimiseFunction
optimise(fun, q, options::OptimizationSettings, θ₀, lb, ub)

Optimization.jl optimiser used for calculating the values of nuisance parameters. Default values of options use NLopt.jl algorithms (seedefault_OptimizationSettings).

source
optimise(fun, options::OptimizationSettings, θ₀, lb, ub)

Optimization.jl optimiser used for calculating the bound transformations. Default values of options use NLopt.jl algorithms (seedefault_OptimizationSettings).

source

Index