GitHub

Dimensional Samples

We can generate samples within the log-likelihood boundary at any dimension of model interest parameters. Nuisance parameters will be set to the values that maximise the log-likelihood function, found using an optimisation scheme. Samples are only implemented for the true log-likelihood function (the LogLikelihood profile type).

Sample Types

Three methods are implemented for rejection sampling of parameter confidence sets within the supplied parameter bounds. These will be inefficient if the bounds are not well-specified (are not close to the boundary of the desired parameter confidence set), and as both the model parameter dimension increases and the interest parameter dimension of the profile increases. By default these are the bounds contained with CoreLikelihoodModel, however, seperate bounds can be supplied for the purposes of sampling. Uniform grid sampling, uniform random sampling and sampling from a random Latin Hypercube scheme (the default) are supported.

LikelihoodBasedProfileWiseAnalysis.UniformGridSamples — Type
UniformGridSamples()

Evaluate the interest parameter bounds space on a uniform grid, determining the optimised log-likelihood value at each grid point. Keep samples which are inside the confidence threshold boundary.

Supertype Hiearachy

UniformGridSamples <: AbstractSampleType <: Any

source
LikelihoodBasedProfileWiseAnalysis.UniformRandomSamples — Type
UniformRandomSamples()

Take uniform random samples of interest parameter bounds space, determining the optimised log-likelihood value at each point. Keep samples which are inside the confidence threshold boundary.

Supertype Hiearachy

UniformRandomSamples <: AbstractSampleType <: Any

source

Full Likelihood Sampling

To sample a confidence set for the full parameter vector, a full parameter confidence set, we use full_likelihood_sample!. The sampled confidence set will be contained within a SampledConfidenceStruct that is stored in the LikelihoodModel.

LikelihoodBasedProfileWiseAnalysis.full_likelihood_sample! — Function
full_likelihood_sample!(model::LikelihoodModel,
    num_points_to_sample::Union{Int, Vector{Int}};
    <keyword arguments>)

Samples num_points_to_sample points from full parameter space, evaluating the log-likelihood function at each, saving all points that are inside the confidence_level log-likelihood threshold. Saves this sample by modifying model in place.

Arguments

  • model: a LikelihoodModel containing model information, saved profiles and predictions.
  • num_points_to_sample: integer number of points to sample (for UniformRandomSamples and LatinHypercubeSamples sample types). For the UniformGridSamples sample type, if integer it is the number of points to grid over in each parameter dimension. If it is a vector of integers each index of the vector is the number of points to grid over in the corresponding parameter dimension. For example, [1,2] would mean a single point in dimension 1 and two points in dimension 2.

Keyword Arguments

  • confidence_level: a number ∈ (0.0, 1.0) for the confidence level which . Default is 0.95 (95%).
  • sample_type: the sampling method used to sample parameter space. Available sample types are UniformGridSamples, UniformRandomSamples and LatinHypercubeSamples. Default is LatinHypercubeSamples() (LatinHypercubeSamples).
  • lb: optional vector of lower bounds on parameters. Use to specify parameter lower bounds to sample over that are different than those contained in model.core. Default is Float64[] (use lower bounds from model.core).
  • ub: optional vector of upper bounds on parameters. Use to specify parameter upper bounds to sample over that are different than those contained in model.core. Default is Float64[] (use upper bounds from model.core).
  • existing_profiles: Symbol ∈ [:ignore, :overwrite] specifying what to do if samples already exist for a given confidence_level and sample_type. Default is :overwrite.
  • show_progress: boolean variable specifying whether to display progress bars on the percentage of θcombinations completed and estimated time of completion. Default is model.show_progress.
  • use_distributed: boolean variable specifying whether to use a threaded for loop or distributed for loop to evaluate the log-likelihood at each sampled point. This should be set to true if Julia instances have been started with low numbers of threads or distributed computing is being used. Default is false.
  • use_threads: boolean variable specifying, if use_distributed is false, whether to use a parallelised for loop across Threads.nthreads() threads or a non-parallel for loop to evaluate the log-likelihood at each sampled point. Default is true.

Details

Using full_likelihood_sample this function calls the sample method specified by sample_type (depending on the setting for existing_profiles and confidence_level if a full likelihood sample already exists). Updates model.dim_samples_df if the sample is successful and saves the results as a SampledConfidenceStruct in model.dim_samples_dict, where the keys for the dictionary is the row number in model.dim_samples_df of the corresponding sample.

Extended help

Parallel Computing Implementation

If Distributed.jl is being used and use_distributed is true,the log-likelihood value of sampled points will be computed in parallel across Distributed.nworkers() workers. If use_distributed is false and use_threads is true then the log-likelihood value of sampled points will be computed in parallel across Threads.nthreads() threads.

Iteration Speed Of the Progress Meter

The time/it value is the time it takes for each point chosen under the specified sampling scheme to be evaluated as valid or not. A point is valid if the log-likelihood function value at that point is greater than the confidence log-likelihood threshold.

source

Dimensional Likelihood Sampling

Similarly, to sample a confidence set for an interest subset of the parameter vector, a 'dimensional profile', we use dimensional_likelihood_samples!. The sampled confidence set(s) will be contained within a SampledConfidenceStruct that is stored in the LikelihoodModel.

Note: dimensional likelihood samples can be 'full' likelihood samples as well. These will be computed before any other dimensional samples.

LikelihoodBasedProfileWiseAnalysis.dimensional_likelihood_samples! — Function
dimensional_likelihood_samples!(model::LikelihoodModel,
    θindices::Vector{Vector{Int}},
    num_points_to_sample::Union{Int, Vector{Int}};
    <keyword arguments>)

Samples num_points_to_sample points from interest parameter space, for each interest parameter combination in θindices, determining the values of nuisance parameters that maximise log-likelihood function at each, saving all points that are inside the confidence_level log-likelihood threshold. Saves these samples by modifying model in place.

Arguments

  • model: a LikelihoodModel containing model information, saved profiles and predictions.
  • θindices: a vector of vectors of parameter indexes for the combinations of interest parameters to samples points from.
  • num_points_to_sample: integer number of points to sample (for UniformRandomSamples and LatinHypercubeSamples sample types). For the UniformGridSamples sample type, if integer it is the number of points to grid over in each parameter dimension. If it is a vector of integers each index of the vector is the number of points to grid over in the corresponding parameter dimension. For example, [1,2] would mean a single point in dimension 1 and two points in dimension 2.

Keyword Arguments

  • confidence_level: a number ∈ (0.0, 1.0) for the confidence level to find samples within. Default is 0.95 (95%).
  • sample_type: the sampling method used to sample parameter space. Available sample types are UniformGridSamples, UniformRandomSamples and LatinHypercubeSamples. Default is LatinHypercubeSamples() (LatinHypercubeSamples).
  • lb: optional vector of lower bounds on interest parameters. Use to specify interest parameter lower bounds to sample over that are different than those contained in model.core (must be the same length as original bounds). Default is Float64[] (use lower bounds from model.core).
  • ub: optional vector of upper bounds on interest parameters. Use to specify interest parameter upper bounds to sample over that are different than those contained in model.core (must be the same length as original bounds). Default is Float64[] (use upper bounds from model.core).
  • θlb_nuisance: a vector of lower bounds on nuisance parameters, require θlb_nuisance .≤ model.core.θmle. Default is model.core.θlb.
  • θub_nuisance: a vector of upper bounds on nuisance parameters, require θub_nuisance .≥ model.core.θmle. Default is model.core.θub.
  • existing_profiles: Symbol ∈ [:ignore, :overwrite] specifying what to do if samples already exist for a given confidence_level and sample_type. Default is :overwrite.
  • optimizationsettings: a OptimizationSettings containing the optimisation settings used to find optimal values of nuisance parameters for a given interest parameter values. Default is missing (will use model.core.optimizationsettings).
  • show_progress: boolean variable specifying whether to display progress bars on the percentage of θcombinations completed and estimated time of completion. Default is model.show_progress.
  • use_distributed: boolean variable specifying whether to use a normal for loop or a @distributed for loop across combinations of interest parameters. Set this variable to false if Distributed.jl is not being used. Default is true.
  • use_threads: boolean variable specifying, if use_distributed is false, to use a parallelised for loop across Threads.nthreads() threads to evaluate the log-likelihood at each sampled point. Default is true.

Details

Using dimensional_likelihood_sample this function calls the sample method specified by sample_type for each set of interest parameters in [θindices] (depending on the setting for existing_profiles and confidence_level if these samples already exist). Updates model.dim_samples_df for each successful sample and saves their results as a SampledConfidenceStruct in model.dim_samples_dict, where the keys for the dictionary is the row number in model.dim_samples_df of the corresponding sample. model.dim_samples_df.num_points is the number of points within the confidence boundary from those sampled.

Support for `dof`

Setting the degrees of freedom of a sampled parameter confidence set to a value other than the interest parameter dimensionality is not currently supported (e.g. as supported for univariate and bivariate profiles). Support may be added in the future, with a slight change in the API of this function.

Extended help

Parallel Computing Implementation

If Distributed.jl is being used use_distributed is true, then the dimensional samples of distinct interest parameter combinations will be computed in parallel across Distributed.nworkers() workers. If use_distributed is false and use_threads is true then the dimensional samples of each distinct interest parameter combination will be computed in parallel across Threads.nthreads() threads. It is highly recommended to set use_threads to true in that situation.

Iteration Speed Of the Progress Meter

The time/it value is the time it takes for each point chosen under the specified sampling scheme to be evaluated as valid or not, for each interest parameter combination. A point is valid if the log-likelihood function value at that point is greater than the confidence log-likelihood threshold.

source
dimensional_likelihood_samples!(model::LikelihoodModel,
    θnames::Vector{Vector{Symbol}},
    num_points_to_sample::Union{Int, Vector{Int}};
    <keyword arguments>)

Samples just the provided θnames interest parameter sets, provided as a vector of vectors.

source
dimensional_likelihood_samples!(model::LikelihoodModel,
    sample_dimension::Int,
    sample_m_random_combinations::Int,
    num_points_to_sample::Union{Int, Vector{Int}}
    <keyword arguments>)

Samples m random combinations of sample_dimension model parameters (sampling without replacement), where 0 < m ≤ binomial(model.core.num_pars, sample_dimension).

source
dimensional_likelihood_samples!(model::LikelihoodModel,
    sample_dimension::Int,
    num_points_to_sample::Union{Int, Vector{Int}};
    <keyword arguments>)

Samples all combinations of sample_dimension model parameters.

source

Index