gpml-matlab-v4.0-2016-10-19/000755 000765 000024 00000000000 12777227614 015076 5ustar00carlstaff000000 000000 gpml-matlab-v4.0-2016-10-19/.octaverc000644 000765 000024 00000000010 12701705605 016660 0ustar00carlstaff000000 000000 startup gpml-matlab-v4.0-2016-10-19/priorDistributions.m000644 000765 000024 00000003767 12777227614 021207 0ustar00carlstaff000000 000000 % prior distributions to be used for hyperparameters of Gaussian processes % using infPrior. % There are two different kinds of prior distributions: simple and composite: % % simple prior distributions: % % priorGauss - univariate Gaussian % priorLaplace - univariate Laplace % priorT - univariate Student's t % % priorSmoothBox1 - univariate interval (linear decay in log domain) % priorSmoothBox2 - univariate interval (quadr. decay in log domain) % % priorGamma - univariate Gamma, IR+ % priorWeibull - univariate Weibull, IR+ % priorInvGauss - univariate Inverse Gaussian, IR+ % priorLogNormal - univariate Log-normal, IR+ % % priorClamped or - fix hyperparameter to its current value by setting % priorDelta derivatives to zero, no effect on marginal likelihood % % priorGaussMulti - multivariate Gauss % priorLaplaceMulti - multivariate Laplace % priorTMulti - multivariate Student's t % % priorClampedMulti or - fix hyperparameter to its current value by setting % priorDeltaMulti derivatives to zero, no effect on marginal likelihood % % composite prior distributions (see explanation at the bottom): % % priorMix - nonnegative mixture of priors % priorTransform - prior on g(t) rather than t % % Naming convention: all prior distributions are named "prior/prior*.m". % % % 1) With only a fixed input arguments: % % r = priorNAME(par1,par2,parN) % % The function returns a random sample from the distribution for e.g. % random restarts, simulations or optimisation initialisation. % % 2) With one additional input arguments: % % [lp,dlp] = priorNAME(par1,par2,parN, t) % % The function returns the log density at location t along with its first % derivative. % % See also doc/usagePrior.m, inf/infPrior.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-11. % File automatically generated using noweb. gpml-matlab-v4.0-2016-10-19/gp.m000644 000765 000024 00000022162 12777227613 015664 0ustar00carlstaff000000 000000 function [varargout] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys) % Gaussian Process inference and prediction. The gp function provides a % flexible framework for Bayesian inference and prediction with Gaussian % processes for scalar targets, i.e. both regression and binary % classification. The prior is Gaussian process, defined through specification % of its mean and covariance function. The likelihood function is also % specified. Both the prior and the likelihood may have hyperparameters % associated with them. % % Two modes are possible: training or prediction: if no test cases are % supplied, then the negative log marginal likelihood and its partial % derivatives w.r.t. the hyperparameters is computed; this mode is used to fit % the hyperparameters. If test cases are given, then the test set predictive % probabilities are returned. Usage: % % training: [nlZ dnlZ ] = gp(hyp, inf, mean, cov, lik, x, y); % prediction: [ymu ys2 fmu fs2 ] = gp(hyp, inf, mean, cov, lik, x, y, xs); % or: [ymu ys2 fmu fs2 lp] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys); % % where: % % hyp struct of column vectors of mean/cov/lik hyperparameters % inf function specifying the inference method % mean prior mean function % cov prior covariance function % lik likelihood function % x n by D matrix of training inputs % y column vector of length n of training targets % xs ns by D matrix of test inputs % ys column vector of length nn of test targets % % nlZ returned value of the negative log marginal likelihood % dnlZ struct of column vectors of partial derivatives of the negative % log marginal likelihood w.r.t. mean/cov/lik hyperparameters % ymu column vector (of length ns) of predictive output means % ys2 column vector (of length ns) of predictive output variances % fmu column vector (of length ns) of predictive latent means % fs2 column vector (of length ns) of predictive latent variances % lp column vector (of length ns) of log predictive probabilities % % post struct representation of the (approximate) posterior % 3rd output in training mode or 6th output in prediction mode % can be reused in prediction mode gp(.., cov, lik, x, post, xs,..) % % See also infMethods.m, meanFunctions.m, covFunctions.m, likFunctions.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-11. % File automatically generated using noweb. if nargin<7 || nargin>9 disp('Usage: [nlZ dnlZ ] = gp(hyp, inf, mean, cov, lik, x, y);') disp(' or: [ymu ys2 fmu fs2 ] = gp(hyp, inf, mean, cov, lik, x, y, xs);') disp(' or: [ymu ys2 fmu fs2 lp] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys);') return end if isempty(mean), mean = {@meanZero}; end % set default mean if ischar(mean) || isa(mean, 'function_handle'), mean = {mean}; end % make cell if isempty(cov), error('Covariance function cannot be empty'); end % no default if ischar(cov) || isa(cov,'function_handle'), cov = {cov}; end % make cell cstr = cov{1}; if isa(cstr,'function_handle'), cstr = func2str(cstr); end if (strcmp(cstr,'covFITC') || strcmp(cstr,'apxSparse')) && isfield(hyp,'xu') cov{3} = hyp.xu; %use hyp.xu end if isempty(inf), inf = {@infGaussLik}; end % set default inference method if ischar(inf), inf = str2func(inf); end % convert into function handle if ischar(inf) || isa(inf,'function_handle'), inf = {inf}; end % make cell istr = inf{1}; if isa(istr,'function_handle'), istr = func2str(istr); end if strcmp(istr,'infPrior') istr = inf{2}; if isa(istr,'function_handle'), istr = func2str(istr); end end if isempty(lik), lik = {@likGauss}; end % set default lik if ischar(lik) || isa(lik,'function_handle'), lik = {lik}; end % make cell lstr = lik{1}; if isa(lstr,'function_handle'), lstr = func2str(lstr); end D = size(x,2); if strncmp(cstr,'covGrid',7) || strcmp(cstr,'apxGrid') % only some inf* possible D = 0; xg = cov{3}; p = numel(xg); for i=1:p, D = D+size(xg{i},2); end % dims end if ~isfield(hyp,'mean'), hyp.mean = []; end % check the hyp specification if eval(feval(mean{:})) ~= numel(hyp.mean) error('Number of mean function hyperparameters disagree with mean function') end if ~isfield(hyp,'cov'), hyp.cov = []; end if eval(feval(cov{:})) ~= numel(hyp.cov) error('Number of cov function hyperparameters disagree with cov function') end if ~isfield(hyp,'lik'), hyp.lik = []; end if eval(feval(lik{:})) ~= numel(hyp.lik) error('Number of lik function hyperparameters disagree with lik function') end try % call the inference method % issue a warning if a classification likelihood is used in conjunction with % labels different from +1 and -1 if strcmp(lstr,'likErf') || strcmp(lstr,'likLogistic') if ~isstruct(y) uy = unique(y); if any( uy~=+1 & uy~=-1 ) warning('You try classification with labels different from {+1,-1}') end end end if nargin>7 % compute marginal likelihood and its derivatives only if needed if isstruct(y) post = y; % reuse a previously computed posterior approximation else post = feval(inf{:}, hyp, mean, cov, lik, x, y); end else if nargout<=1 [post nlZ] = feval(inf{:}, hyp, mean, cov, lik, x, y); dnlZ = {}; else [post nlZ dnlZ] = feval(inf{:}, hyp, mean, cov, lik, x, y); end end catch msgstr = lasterr; if nargin>7, error('Inference method failed [%s]', msgstr); else warning('Inference method failed [%s] .. attempting to continue',msgstr) varargout = {NaN, vec2any(hyp,zeros(numel(any2vec(hyp)),1))}; return % go on end end if nargin==7 % if no test cases are provided varargout = {nlZ, dnlZ, post}; % report -log marg lik, derivatives and post else alpha = post.alpha; L = post.L; sW = post.sW; if issparse(alpha) % handle things for sparse representations nz = alpha ~= 0; % determine nonzero indices if issparse(L), L = full(L(nz,nz)); end % convert L and sW if necessary if issparse(sW), sW = full(sW(nz)); end else nz = true(size(alpha,1),1); end % non-sparse representation if isempty(L) % in case L is not provided, we compute it K = feval(cov{:}, hyp.cov, x(nz,:)); L = chol(eye(sum(nz))+sW*sW'.*K); end %verify whether L contains valid Cholesky decomposition or something different Lchol = isnumeric(L) && all(all(tril(L,-1)==0)&diag(L)'>0&isreal(diag(L))'); ns = size(xs,1); % number of data points if strncmp(cstr,'apxGrid',7), xs = apxGrid('idx2dat',cov{3},xs); end % expand nperbatch = 1000; % number of data points per mini batch nact = 0; % number of already processed test data points ymu = zeros(ns,1); ys2 = ymu; fmu = ymu; fs2 = ymu; lp = ymu; % allocate mem while nact use Cholesky parameters (alpha,sW,L) V = L'\(repmat(sW,1,length(id)).*Ks); fs2(id) = kss - sum(V.*V,1)'; % predictive variances else % L is not triangular => use alternative parametrisation if isnumeric(L), LKs = L*Ks; else LKs = L(Ks); end % matrix or callback fs2(id) = kss + sum(Ks.*LKs,1)'; % predictive variances end fs2(id) = max(fs2(id),0); % remove numerical noise i.e. negative variances Fs2 = repmat(fs2(id),1,N); % we have multiple values in case of sampling if nargin<9 [Lp, Ymu, Ys2] = feval(lik{:},hyp.lik,[], Fmu(:),Fs2(:)); else Ys = repmat(ys(id),1,N); [Lp, Ymu, Ys2] = feval(lik{:},hyp.lik,Ys(:),Fmu(:),Fs2(:)); end lp(id) = sum(reshape(Lp, [],N),2)/N; % log probability; sample averaging ymu(id) = sum(reshape(Ymu,[],N),2)/N; % predictive mean ys|y and .. ys2(id) = sum(reshape(Ys2,[],N),2)/N; % .. variance nact = id(end); % set counter to index of last processed data point end if nargin<9 varargout = {ymu, ys2, fmu, fs2, [], post}; % assign output arguments else varargout = {ymu, ys2, fmu, fs2, lp, post}; end end gpml-matlab-v4.0-2016-10-19/mean/000755 000765 000024 00000000000 12777227614 016016 5ustar00carlstaff000000 000000 gpml-matlab-v4.0-2016-10-19/util/000755 000765 000024 00000000000 12777227614 016053 5ustar00carlstaff000000 000000 gpml-matlab-v4.0-2016-10-19/likFunctions.m000644 000765 000024 00000010312 12777227613 017720 0ustar00carlstaff000000 000000 % likelihood functions are provided to be used by the gp.m function: % % likErf (Error function, classification, probit regression) % likLogistic (Logistic, classification, logit regression) % likUni (Uniform likelihood, classification) % % likGauss (Gaussian, regression) % likGaussWarp (Warped Gaussian, regression) % likGumbel (Gumbel likelihood for extremal values) % likLaplace (Laplacian or double exponential, regression) % likSech2 (Sech-square, regression) % likT (Student's t, regression) % % likPoisson (Poisson regression, count data) % likGamma (Nonnegative regression, positive data) % likExp (Nonnegative regression, positive data) % likInvGauss (Nonnegative regression, positive data) % likBeta (Beta regression, interval data) % % likMix (Mixture of individual likelihood functions) % % The likelihood functions have three possible modes, the mode being selected % as follows (where "lik" stands for any likelihood function in "lik/lik*.m".): % % 1) With one or no input arguments: [REPORT NUMBER OF HYPERPARAMETERS] % % s = lik OR s = lik(hyp) % % The likelihood function returns a string telling how many hyperparameters it % expects, using the convention that "D" is the dimension of the input space. % For example, calling "likLogistic" returns the string '0'. % % % 2) With three or four input arguments: [PREDICTION MODE] % % lp = lik(hyp, y, mu) OR [lp, ymu, ys2] = lik(hyp, y, mu, s2) % % This allows to evaluate the predictive distribution. Let p(y_*|f_*) be the % likelihood of a test point and N(f_*|mu,s2) an approximation to the posterior % marginal p(f_*|x_*,x,y) as returned by an inference method. The predictive % distribution p(y_*|x_*,x,y) is approximated by. % q(y_*) = \int N(f_*|mu,s2) p(y_*|f_*) df_* % % lp = log( q(y) ) for a particular value of y, if s2 is [] or 0, this % corresponds to log( p(y|mu) ) % ymu and ys2 the mean and variance of the predictive marginal q(y) % note that these two numbers do not depend on a particular % value of y % All vectors have the same size. % % % 3) With five or six input arguments, the fifth being a string [INFERENCE MODE] % % [varargout] = lik(hyp, y, mu, s2, inf) OR % [varargout] = lik(hyp, y, mu, s2, inf, i) % % There are three cases for inf, namely a) infLaplace, b) infEP and c) infVB. % The last input i, refers to derivatives w.r.t. the ith hyperparameter. % % a1) [lp,dlp,d2lp,d3lp] = lik(hyp, y, f, [], 'infLaplace') % lp, dlp, d2lp and d3lp correspond to derivatives of the log likelihood % log(p(y|f)) w.r.t. to the latent location f. % lp = log( p(y|f) ) % dlp = d log( p(y|f) ) / df % d2lp = d^2 log( p(y|f) ) / df^2 % d3lp = d^3 log( p(y|f) ) / df^3 % % a2) [lp_dhyp,dlp_dhyp,d2lp_dhyp] = lik(hyp, y, f, [], 'infLaplace', i) % returns derivatives w.r.t. to the ith hyperparameter % lp_dhyp = d log( p(y|f) ) / ( dhyp_i) % dlp_dhyp = d^2 log( p(y|f) ) / (df dhyp_i) % d2lp_dhyp = d^3 log( p(y|f) ) / (df^2 dhyp_i) % % % b1) [lZ,dlZ,d2lZ] = lik(hyp, y, mu, s2, 'infEP') % let Z = \int p(y|f) N(f|mu,s2) df then % lZ = log(Z) % dlZ = d log(Z) / dmu % d2lZ = d^2 log(Z) / dmu^2 % % b2) [dlZhyp] = lik(hyp, y, mu, s2, 'infEP', i) % returns derivatives w.r.t. to the ith hyperparameter % dlZhyp = d log(Z) / dhyp_i % % % c1) [b,z] = lik(hyp, y, [], ga, 'infVB') % ga is the variance of a Gaussian lower bound to the likelihood p(y|f). % p(y|f) \ge exp( b*(f+z) - (f+z).^2/(2*ga) - h(ga)/2 ) \propto N(f|b*ga-z,ga) % The function returns the linear part b and z. % % Cumulative likelihoods are designed for binary classification. Therefore, they % only look at the sign of the targets y; zero values are treated as +1. % % Some examples for valid likelihood functions: % lik = @likLogistic; % lik = {'likMix',{'likUni',@likErf}} % lik = {@likPoisson,'logistic'}; % % See the help for the individual likelihood for the computations specific to % each likelihood function. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-11. % File automatically generated using noweb. gpml-matlab-v4.0-2016-10-19/covFunctions.m000644 000765 000024 00000016453 12777227614 017745 0ustar00carlstaff000000 000000 % Covariance functions to be use by Gaussian process functions. There are two % different kinds of covariance functions: simple and composite: % % 1) Elementary and standalone covariance functions: % covZero - zero covariance function % covEye - unit covariance function % covOne - unit constant covariance function % covDiscrete - precomputed covariance for discrete data % % 2) Composite covariance functions: % covScale - scaled version of a covariance function % covSum - sums of covariance functions % covProd - products of covariance functions % covMask - mask some dimensions of the data % covPref - difference covariance for preference learning % covPER - make stationary covariance periodic % covADD - additive covariance function % % 3) Mahalanobis distance based covariances and their modes % covMaha - generic "mother" covariance % covGE - Gamma exponential covariance % covMatern - Matern covariance function with nu=1/2, 3/2 or 5/2 % covPP - piecewise polynomial covariance function (compact support) % covRQ - rational quadratic covariance function % covSE - squared exponential covariance function % * eye - unit length scale % * iso - isotropic length scale % * ard - automatic relevance determination % * pro - (low-rank) projection in input space % * fac - factor analysis covariance % * vle - spatially varying length scale % % 4) Dot product based covariances and their modes % covDot - generic "mother" covariance % covLIN - linear covariance function % covPoly - polynomial covariance function % * eye - unit length scale % * iso - isotropic length scale % * ard - automatic relevance determination % % 5) Standalone covariances % covNNone - neural network covariance function % covLINone - linear covariance function with bias % covPeriodic - smooth periodic covariance function (1d) % covPeriodicNoDC - as above but with zero DC component and properly scaled % covCos - sine periodic covariance function (1d) with unit period % covGabor - Gabor covariance function % % 6) Shortcut covariances assembled from library % covConst - covariance for constant functions % covNoise - independent covariance function (i.e. white noise) % covPERiso - make isotropic stationary covariance periodic % covPERard - make ARD stationary covariance periodic % covMaterniso - Matern covariance function with nu=1/2, 3/2 or 5/2 % covMaternard - Matern covariance function with nu=1/2, 3/2 or 5/2 with ARD % covPPiso - piecewise polynomial covariance function (compact support) % covPPard - piecewise polynomial covariance function (compact support) % covRQiso - isotropic rational quadratic covariance function % covRQard - rational quadratic covariance function with ARD % covSEiso - isotropic squared exponential covariance function % covSEisoU - same as above but without latent scale % covSEard - squared exponential covariance function with ARD % covSEvlen - spatially varying lengthscale squared exponential % covSEproj - projection squared exponential covariance function % covLINiso - linear covariance function % covLINard - linear covariance function with ARD % covGaborard - Gabor covariance function with ARD % covGaborsio - isotropic Gabor covariance function % covSM - spectral mixture covariance function % % 7) Special purpose (approximation) covariance functions % apxSparse - sparse approximation: to be used for large scale inference % problems with inducing points aka FITC % apxGrid - grid interpolation: to be used for large scale inference % problems with Kronecker/Toeplitz/BTTB covariance matrix % % The covariance functions are written according to a special convention where % the exact behaviour depends on the number of input and output arguments % passed to the function. If you want to add new covariance functions, you % should follow this convention if you want them to work with the function gp. % There are four different ways of calling the covariance functions: % % 1) With no (or one) input argument(s): % % s = cov % % The covariance function returns a string s telling how many hyperparameters it % expects, using the convention that "D" is the dimension of the input space. % For example, calling "covRQard" returns the string '(D+2)'. % % 2) With two input arguments and one output argument: % % K = cov(hyp, x) equivalent to K = cov(hyp, x, []) % % The function computes and returns the covariance matrix where hyp are % the hyperparameters and x is an n by D matrix of cases, where % D is the dimension of the input space. The returned covariance matrix is of % size n by n. % % 3) With three input arguments and one output argument: % % Kz = cov(hyp, x, z) % kx = cov(hyp, x, 'diag') % % The function computes test set covariances; kx is a vector of self covariances % for the test cases in x (of length n) and Kz is an (n by nz) matrix of cross % covariances between training cases x and test cases z. % % 4) With two output arguments: % % [K,dK] = cov(hyp, x) equivalent to [K,dK] = cov(hyp, x, []) % [K,dK] = cov(hyp, x, z) % [K,dK] = cov(hyp, x, 'diag') % % The function computes and returns the covariances K as in 3) above. % In addition to that, the (linear) directional derivative function dK is % returned. The two possible calls dhyp = dK(Q) and [dhyp,dx] = dK(Q) for a % direction Q of the same size as K are possible. The return arguments dhyp % and dx are the directional derivatives dhyp = d trace(Q'*K) / d hyp and % dx = d trace(Q'*K) / d x are of the same size as the hyperparameter % vector hyp and the input data x, respectively. The components of dhyp and % dx are defined as follows: dhyp(i) = trace(Q'*( d K / d hyp(i) )) % and dx(i,j) = trace(Q'*( d K / d x(i,j) )). % % Covariance functions can be specified in two ways: either as a string % containing the name of the covariance function or using a cell array. For % example: % % cov = 'covRQard'; % cov = {'covRQard'}; % cov = {@covRQard}; % % are supported. Only the second and third form using the cell array can be used % for specifying composite covariance functions, made up of several % contributions. For example: % % cov = {'covScale', {'covRQiso'}}; % cov = {'covSum', {'covRQiso','covSEard','covNoise'}}; % cov = {'covProd',{'covRQiso','covSEard','covNoise'}}; % cov = {'covMask',{mask,'covSEiso'}} % q=1; cov = {'covPPiso',q}; % d=3; cov = {'covPoly',d}; % cov = {'covADD',{[1,2],'covSEiso'}}; % cov = {@covFITC, {@covSEiso}, u}; where u are the inducing inputs % % specifies a covariance function which is the sum of three contributions. To % find out how many hyperparameters this covariance function requires, we do: % % feval(cov{:}) % % which returns the string '3+(D+1)+1' (i.e. the 'covRQiso' contribution uses % 3 parameters, the 'covSEard' uses D+1 and 'covNoise' a single parameter). % % See also doc/usageCov.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-11. % File automatically generated using noweb. gpml-matlab-v4.0-2016-10-19/README000644 000765 000024 00000045521 12777227614 015765 0ustar00carlstaff000000 000000 GAUSSIAN PROCESS REGRESSION AND CLASSIFICATION Toolbox version 4.0 for GNU Octave 3.2.x and Matlab 7.x Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-19. 0) HOW TO READ ============== If you want to get started without further delay, then read section 1) below and jump right to the examples in doc/index.html. 1) ABOUT THESE PROGRAMS ======================= This collection of matlab programs implements and demonstrates some of the algorithms described in a) the book by Rasmussen and Williams: "Gaussian Processes for Machine Learning", the MIT Press 2006, in b) the article by Nickisch and Rasmussen: "Approximations for Binary Gaussian Process Classification", JMLR 2008, in c) the article by Candela and Rasmussen: "A Unifying View of Sparse Approximate Gaussian Process Regression", JMLR 2005, in d) the paper by Murray, Adams and Mackay: "Elliptical slice sampling", AISTATS 2010, in e) the report by Neal: "Annealed Importance Sampling", UToronto 1998, in f) the paper by Naish-Guzman and Holden: "The Generalized FITC Approximation", NIPS, 2007, in g) the paper by Duvenaud, Nickisch and Rasmussen: "Additive Gaussian Processes", NIPS, 2011, in h) the paper by Wilson and Adams: "Gaussian Process Kernels for Pattern Discovery and Extrapolation", ICML, 2013, and in i) the paper Snelson, Rasmussen and Ghahramani: "Warped Gaussian Processes", NIPS, 2003. There are 7 code subdirectories: cov, doc, inf, lik, mean, prior and util. cov/ contains covariance functions cov*.m => see covFunctions.m doc/ contains an index.html file providing documentation. This information is also available from http://www.GaussianProcess.org/gpml/code. Usage of mean, cov, classification and regression is demonstrated in usage*.m. Further details can be found in the developer documentation manual.pdf. inf/ contains the inference algorithms inf*.m => see infMethods.m lik/ contains the likelihood functions lik*.m => see likFunctions.m mean/ contains the mean functions mean*.m => see meanFunctions.m prior/ contains the hyperparameter prior distributions prior*.m => see priorDistributions.m util/ contains optimisation routines, backward compatibility programs and small auxiliary scripts Before running the demos or any other part of the code, you should execute startup.m to add all necessary directories to your path. In Octave, you can also use the .octaverc file. 2) ABOUT MEX FILES ================== Some of the programs make use of the MEX facility in matlab for more efficient implementation. However, if you don't know about how to compile MEX files, you do not need to worry about this -- the code runs anyway. If you do compile the MEX files, this is automatically detected, and the program will run more efficiently. If you do not have a compiler installed on your system, you might want to consult [3]. Two components of the toolbox can be accelerated by MEX. First, there is an auxiliary functions solve_chol.m having a MEX equivalent. This can be compiled by either executing util/make.m from the Matlab/Octave command line. In addition to that, we provide a Makefile for Matlab in util/ that you can run by simply typing make in your shell after having supplied your Matlab path and your operating system. We recommend the make.m script because it works both under Matlab and Octave. Second, in order to use the L-BFGS minimiser for hyperparameter optimisation as an equivalent to minimize.m, you have to compile Peter Carbonetto's "Matlab interface for L-BFGS-B" [1] which was extended to also work for Octave by José Vallet. The challenge here is the Fortran 77 code. We provide a Makefile suitable for Linux 32/64 bit and Mac whenever you have 1) g77 or 2) gfortran properly set up. The latter is prefered on current Ubuntu systems. Under Ubuntu, you can achieve this by installing the packages fort77 and gfortran, respectively. Note that the code is GNU licensed. Compilation is done by editing util/lbfgsb/Makefile. In any case, you need to provide $MATLAB_HOME which can be found by the commands 'locate matlab' or 'find / -name "matlab"'. You can choose between two compilation modes: a) without mex utility by Matlab [default] provide $MEX_SUFFIX and $MATLAB_LIB, then type 'make nomex' b) using the mex utility by Matlab provide $MEX, then type 'make mex' c) compilation for Octave provide Octave variables and type 'make oct' In Ubuntu 10.04 LTS, the libg2c library needed for both 1)+a) and 1)+b) is not included per default. If 'ls /usr/lib/libg2c.*' does not list anything this is the case on your machine. You then whant to install the packages gcc-3.4-base and libg2c0 e.g. from http://packages.ubuntu.com/hardy/gcc-3.4-base and http://packages.ubuntu.com/hardy/libg2c0. After installation, you have to create a symbolic link by 'cd /usr/lib' and 'ln -s libg2c.so.0 libg2c.so'. There were reports of problems running the mex files on recent 64bit Ubuntu since the libgfortran.3.0.0 that comes with matlab ($(MATLAB_HOME)/sys/os/glnxa64/libgfortran.so.3.0.0) is not the same as the libgfortran.3.0.0 that comes with the ubuntu package libgfortran3 and that is used when linking. So running the mex file results in a version mismatch error. The issue can be solved by changing the name of to the matlab file and linking to the ubuntu file instead: in $(MATLAB_HOME)/sys/os/glnxa64 run 'sudo mv libgfortran.3.0.0 libgfortran.3.0.0_matlab' 'sudo ln -s /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 libgfortran.3.0.0' Thanks to Giampiero Salvi for posting the workaround. [1] https://github.com/pcarbo/lbfgsb-matlab [2] http://www.mathworks.com/support/compilers/R2010a 3) CURRENT VERSION ================== The current version of the programs is 4.0. Previous versions of the code are available at http://www.gaussianprocess.org/gpml/code/oldcode.html. 4) DIFFERENCES TO PREVIOUS VERSIONS =================================== NEW in version 4.0, 2016-09-29 ------------------------------ A major code restructuring effort did take place in the current release unifying certain inference functions and allowing more flexibility in covariance function composition. We also redesigned the whole derivative computation pipeline to strongly improve the overall runtime. We finally include grid-based covariance approximations natively. More generic sparse approximation using Power EP - unified treatment of FITC approximation, variational approaches VFE and hybrids - inducing input optimisation for all (compositions of) covariance functions dropping the previous limitation to a few standard examples - infFITC is now covered by the more generic infGaussLik function Approximate covariance object unifying sparse approximations, grid-based approximations and exact covariance computations - implementation in cov/apx, cov/apxGrid, cov/apxSparse - generic infGaussLik unifies infExact, infFITC and infGrid - generic infLaplace unifies infLaplace, infFITC_Laplace and infGrid_Laplace Hiearchical structure of covariance functions - clear hierachical compositional implementation - no more code duplication as present in covSEiso and covSEard pairs - two mother covariance functions covDot for dot-product-based covariances and covMaha for Mahalanobis-distance-based covariances - a variety of modifiers: eye, iso, ard, proj, fact, vlen - more flexibility as more variants are available and possible - all covariance functions offer derivatives w.r.t. inputs Faster derivative computations for mean and cov functions - switched from partial derivatives to directional derivatives - simpler and more concise interface of mean and cov functions - much faster marginal likelihood derivative computations - simpler and more compact code New mean functions - new mean/meanWSPC (Weighted Sum of Projected Cosines or Random Kitchen Sink features) following a suggestion by William Herlands - new mean/meanWarp for constructing a new mean from an existing one by means of a warping function adapted from William Herlands New optimizer - added a new minimize_minfunc, contributed by Truong X. Nghiem New GLM link function - added the logistic link function util/glm_invlink_logistic2 Smaller fixes - two-fold speedup of util/elsympol used by covADD by Truong X. Nghiem - bugfix in util/logphi as reported by John Darby NEW in version 3.6, 2015-07-07 ------------------------------ - bugfix in likGaussWarp as reported by Obaid Malik - added a new inference function infGrid_Laplace allowing to use non-Gaussian likelihoods for large grids - fixed a bug due to Octave evaluating norm([]) to a tiny nonzero value, modified all lik/lik*.m functions reported by Philipp Richter - small bugfixes in covGrid and infGrid - bugfix in predictive variance of likNegBinom due to Seth Flaxman - bugfix in infFITC_Laplace as suggested by Wu Lin - bugfix in covPP{iso,ard} NEW in version 3.5, 2014-12-08 ------------------------------ - mechanism for specifying hyperparameter priors (together with Roman Garnett and José Vallet) - new inference method inf/infGrid allowing efficient inference for data defined on a Cartesian grid (together with Andrew Wilson) - new mean/cov functions for preference learning: meanPref/covPref - new mean/cov functions for non-vectorial data: meanDiscrete/covDiscrete - new piecewise constant nearest neighbor mean function: meanNN - new mean functions being predictions from GPs: meanGP and meanGPexact - new covariance function for standard additive noise: covEye - new covariance function for factor analysis: covSEfact - new covariance function with varying length scale : covSEvlen - make covScale more general to scaling with a function instead of a scalar - bugfix in covGabor* and covSM (due to Andrew Gordon Wilson) - bugfix in lik/likBeta.m (suggested by Dali Wei) - bugfix in solve_chol.c (due to Todd Small) - bugfix in FITC inference mode (due to Joris Mooij) where the wrong mode for post.L was chosen when using infFITC and post.L being a diagonal matrix - bugfix in infVB marginal likelihood for likLogistic with nonzero mean function (reported by James Lloyd) - removed the combination likErf/infVB as it yields a bad posterior approximation and lacks theoretical justification - Matlab and Octave compilation for L-BFGS-B v2.4 and the more recent L-BFGS-B v3.0 (contributed by José Vallet) - smaller bugfixes in gp.m (due to Joris Mooij and Ernst Kloppenburg) - bugfix in lik/likBeta.m (due to Dali Wei) - updated use of logphi in lik/likErf - bugfix in util/solve_chol.c where a typing issue occured on OS X (due to Todd Small) - bugfix due to Bjørn Sand Jensen noticing that cov_deriv_sq_dist.m was missing in the distribution - bugfix in infFITC_EP for ttau->inf (suggested by Ryan Turner) NEW in version 3.4, 2013-11-11 ------------------------------ - new likelihood function for warped GPs: likGaussWarp - new exponential likelihood function: likExp - new Weibull likelihood function: likWeibull - new Gumbel likelihood function for extremal value regression: likGumbel - new polynomial mean function: meanPoly - included derivatives w.r.t. inducing points xu in infFITC, infFITC_Laplace, infFITC_EP - allow for zero noise variance for infExact NEW in version 3.3, 2013-10-19 ------------------------------ - new variational inference algorithm based on direct KL minimisation: infKL - improved inf/infVB double loop scheme so that only very few likelihood properties are required; infVB is now internally a sequence of infLaplace runs - improved inf/infLaplace to be more generic so that optimisers other than scaled Newton can be used - improved inf/infEP so that the internal variables (mu,Sigma) now represent the current posterior approximation - added three new generalised linear model likelihoods: gamma, beta, inverse Gaussian - new covariance functions: spectral mixture covSM, covGaboriso and covGaborard - new meta covariance function to turn a stationary covariance into a periodic covariance function covPERard, covPERiso - new periodic covariance function with zero DC component and correct scaling covPeriodicNoDC New likelihood functions in lik/: - likGamma.m, likBeta.m, likInvGauss.m New inference functions in inf/: - infKL.m New covariance functions in cov/: - covSM.m, covGaboriso.m, covGaborard.m (contributed by Andrew Gordon Wilson) - covPPard.m, covMaternard.m, covLINiso.m - covCos.m, covPeriodicNoDC.m (contributed by James Robert Lloyd) - covPERiso.m, covPERard.m NEW in version 3.2, 2013-01-15 ------------------------------ - added inf/infFITC_Laplace.m and inf/infFITC_EP.m for large scale approximate inference - added inf/infMCMC.m to allow for Markov Chain Monte Carlo sampling and Annealed Importance Sampling; we currently support two samplers: Elliptical Slice Sampling (ESS) and Hybrid Monte Carlo (HMC) - improved accuracy of the digamma function lik/likT.m/dloggamma from three to ten digits precision (thanks to Edward Challis) - added gfortran support to util/lbfgsb/Makefile (thanks to Ernst Kloppenburg) - fixed a stupid bug in cov/covFITC.m (thanks to Krzysztof Chalupka) - fixed a severe stability bug in the derivatives of inf/infFITC.m (thanks to Andrew McHutchon) - changed scalars in util/solve_chol.c to long to provide 64bit compatibility (thanks to Ernst Kloppenburg) - added infLOO to allow pseudo-likelihood optimisation instead of the marginal likelihood - fixed a performance issue in util/sq_dist.m (thanks to Krzysztof Chalupka) - fixed a bug causing numerical problems in infFITC and covFITC whenever Kuu is close to singular (thanks to Joris M. Mooij) - fixed an issue in covNoise (thanks to Daniel Marthaler) - fixed implicit derivative w.r.t. likelihood in infLaplace* - added two more likelihood functions likMix.m for mixtures and likUni for classification noise - slightly changed the interface for the likelihood functions - added likPoisson for Poisson regression of count data NEW in version 3.1, 2010-10-10 ------------------------------ - following a suggestion by Ed Snelson we now support FITC regression - cov/covFITC.m and inf/infFITC.m have been added with Ed Snelson's help - the covariance interface was slightly changed to make that possible - changed parametrisation in cov/covPeriodic.m (thanks to Philipp Hennig) NEW in version 3.0, 2010-07-23 ------------------------------ A major code reorganisation effort did take place in the current release. First, classification and regression are now done by a single file gp.m which is completely generic in the likelihood. The previous regression program gpr.m corresponds to gp.m with Gaussian likelihood. Several other likelihoods for robust regresssion were added. Further, the code now supports mean functions with a similar specification mechanism as already used by the covariance functions. Previous implementations correspond to using meanZero.m. We merged the covariance functions covMatern3iso.m and covMatern5.iso into a single covMaterniso.m and added a covariance function for additive functions as well as the possibility to use only certain components of the data. Finally, we included covPPiso.m a piecewise polynomial covariance function with compact support. New likelihood functions in lik/: likGauss.m, likLaplace.m, likLogistic.m and likT.m New mean functions in mean/: meanConst.m, meanLinear.m, meanPow.m, meanProd.m, meanSum.m and meanZero.m New covariance functions in cov/: covADD.m, covMask.m, covPPiso.m The gprSRPP.m function which previously provided "Subset of Regressors" and the "Projected Process" approximation has now been removed. NEW in version 2.1, 2007-07-25 ------------------------------ covConst.m: fixed a bug which caused an error in the derivative of the log marginal likelihood for certain combinations of covariance functions and approximation methods. (Thanks to Antonio Eleuteri for reporting the problem) gauher.m: added the function "gauher.m" which was mistakenly missing from the previous release. This caused an error for certain combinations of approximation method and likelihood function. logistic.m: modified the approximation of moments calculation to use a mixture of cumulative Gaussian, rather than Gauss-Hermite quadrature, as the former turns out to be more accurate. NEW in version 2.0, 2007-06-25 ------------------------------ Some code restructuring has taken place for the classification code to make it more modular, to facilitate addition of new likelihood functions and approximations methods. Now, all classification is done using the binaryGP function, which (among other things) takes an approximation method and a likelihood function as an arguments. Thus, binaryGP replaces both binaryEPGP and binaryLapaceGP, although wrapper functions are still provided for backward compatibility. This gives added flexibility: now EP can also be used wth the logistic likelihood function (implemented using Gauss-Hermite quadrature). approxEP.m: New file, containing the Expectation Propagation approximation method, which was previously contained in binaryEPGP.m approxLA.m: New file, containing Laplaces approximation method, which was previously contained in binaryLaplace.m approximations.m: New file, help for the approximation methods. binaryEPGP.m: This file has been replaced by a wrapper (for backward compatibility) which calls the more general binaryGP function. binaryGP.m: New general function to do binary classification. binaryLaplaceGP.m: This file has been replaced by a wrapper (for backward compatibility) which calls the more general binaryGP function. covMatern3iso.m, covMatern5iso.m, covNNone.m, covRQard.m, covRQiso.m, cosSEard, covSEiso: now check more carefully, that persistent variables have the correct sizes, and some variable names have been modified. cumGauss.m: New file, containing code for the cumulative Gaussian likelihood function likelihoods.m: New file, help for likelihood functions logistic.m: New file, logistic likelihood NEW in version 1.3, 2006-09-08 ------------------------------ covRQard.m: bugfix: replaced x with x' and z with z' in line 36 covRQiso.m: bugfix: replaced x with x' and z with z' in line 28 minimize.m: correction: replaced "error()" with "error('')", and made a few cosmetic changes binaryEPGP.m: added the line "lml = -n*log(2);" in line 77. This change should be largely inconsequential, but occationally may save things when the covariance matrix is exceptionally badly conditioned. NEW in version 1.2, 2006-05-10 ------------------------------ added the "erfint" function to "binaryLaplaceGP.m". The erfint function was missing by mistake, preventing the use of the "logistic" likelihood. NEW in version 1.1, 2006-04-12 ------------------------------ added files: "covProd.m" and "covPeriodic.m" changes: "covSEiso.m" was changed slightly to avoid the use of persistent variables NEW in version 1.0, 2006-03-29 ------------------------------ initial version shipped with the book gpml-matlab-v4.0-2016-10-19/inf/000755 000765 000024 00000000000 12777227614 015652 5ustar00carlstaff000000 000000 gpml-matlab-v4.0-2016-10-19/prior/000755 000765 000024 00000000000 12777227614 016231 5ustar00carlstaff000000 000000 gpml-matlab-v4.0-2016-10-19/infMethods.m000644 000765 000024 00000005024 12777227613 017354 0ustar00carlstaff000000 000000 % Inference methods: Compute the (approximate) posterior for a Gaussian process. % Methods currently implemented include: % % infGaussLik Exact inference (only possible with Gaussian likelihood) % infLaplace Laplace's Approximation % infEP Expectation Propagation % infVB Variational Bayes Approximation % infKL Kullback-Leibler optimal Approximation % % infMCMC Markov Chain Monte Carlo and Annealed Importance Sampling % We offer two samplers. % - hmc: Hybrid Monte Carlo % - ess: Elliptical Slice Sampling % No derivatives w.r.t. to hyperparameters are provided. % % infLOO Leave-One-Out predictive probability and Least-Squares Approxim. % infPrior Perform inference with hyperparameter prior. % % The interface to the approximation methods is the following: % % function [post nlZ dnlZ] = inf..(hyp, cov, lik, x, y) % % where: % % hyp is a struct of hyperparameters % cov is the name of the covariance function (see covFunctions.m) % lik is the name of the likelihood function (see likFunctions.m) % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of targets % % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood w.r.t. each hyperparameter % post struct representation of the (approximate) posterior containing % alpha is a (sparse or full column vector) containing inv(K)*(mu-m), % where K is the prior covariance matrix, m the prior mean, % and mu the approx posterior mean % sW is a (sparse or full column) vector containing diagonal of sqrt(W) % the approximate posterior covariance matrix is inv(inv(K)+W) % L is a (sparse or full) triangular matrix, L = chol(sW*K*sW+eye(n)), % or a full matrix, L = -inv(K+inv(W)), % or a function L(A) of a matrix A such that -(K+inv(W))*L(A) = A % % Usually, the approximate posterior to be returned admits the form % N(mu=m+K*alpha, V=inv(inv(K)+W)), where alpha is a vector and W is diagonal. % % For more information on the individual approximation methods and their % implementations, see the separate inf??.m files. See also gp.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-11. % File automatically generated using noweb. gpml-matlab-v4.0-2016-10-19/meanFunctions.m000644 000765 000024 00000004642 12777227613 020072 0ustar00carlstaff000000 000000 % Mean functions to be use by Gaussian process functions. There are two % different kinds of mean functions: simple and composite: % % Simple mean functions: % % meanZero - zero mean function % meanOne - one mean function % meanConst - constant mean function % meanLinear - linear mean function % meanPoly - polynomial mean function % meanDiscrete - precomputed mean for discrete data % meanGP - predictive mean of another GP % meanGPexact - predictive mean of a regression GP % meanNN - nearest neighbor mean function % meanWSPC - weighted sum of projected cosines % % Composite mean functions (see explanation at the bottom): % % meanScale - scaled version of a mean function % meanSum - sum of mean functions % meanProd - product of mean functions % meanPow - power of a mean function % meanMask - mask some dimensions of the data % meanPref - difference mean for preference learning % meanWarp - warped mean function % % Naming convention: all mean functions are named "mean/mean*.m". % % % 1) With no or only a single input argument: % % s = meanNAME or s = meanNAME(hyp) % % The mean function returns a string s telling how many hyperparameters hyp it % expects, using the convention that "D" is the dimension of the input space. % For example, calling "meanLinear" returns the string 'D'. % % 2) With two input arguments and one output argument: % % m = meanNAME(hyp, x) % % The function computes and returns the mean vector m with components % m(i) = m(x(i,:)) where hyp are the hyperparameters and x is an n by D matrix % of data, where D is the dimension of the input space. The returned mean % vector m is of size n by 1. % % 3) With two input arguments and two output arguments: % % [m,dm] = meanNAME(hyp, x) % % The function computes and returns the mean vector m as in 2) above. % In addition to that, the (linear) directional derivative function dm is % returned. The call dhyp = dm(q) for a direction vector q of size n by 1 % returns a vector of directional derivatives dhyp = d (q'*m(x)) / d hyp of % the same size as the hyperparameter vector hyp. The components of dhyp are % defined as follows: dhyp(i) = q'*( d m(x) / d hyp(i) ). % % See also doc/usageMean.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-11. % File automatically generated using noweb. gpml-matlab-v4.0-2016-10-19/lik/000755 000765 000024 00000000000 12777227614 015655 5ustar00carlstaff000000 000000 gpml-matlab-v4.0-2016-10-19/doc/000755 000765 000024 00000000000 12777227614 015643 5ustar00carlstaff000000 000000 gpml-matlab-v4.0-2016-10-19/Copyright000644 000765 000024 00000003455 12777177045 017002 0ustar00carlstaff000000 000000 GAUSSIAN PROCESS REGRESSION AND CLASSIFICATION Toolbox version 4.0 for GNU Octave 3.2.x and Matlab 7.x The code is released under the FreeBSD License. Copyright (c) 2005-2016 Carl Edward Rasmussen & Hannes Nickisch. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY CARL EDWARD RASMUSSEN & HANNES NICKISCH ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CARL EDWARD RASMUSSEN & HANNES NICKISCH OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of Carl Edward Rasmussen & Hannes Nickisch. The code and associated documentation is available from http://gaussianprocess.org/gpml/code.gpml-matlab-v4.0-2016-10-19/cov/000755 000765 000024 00000000000 12777227614 015665 5ustar00carlstaff000000 000000 gpml-matlab-v4.0-2016-10-19/startup.m000644 000765 000024 00000001440 12761006053 016736 0ustar00carlstaff000000 000000 % startup script to make Octave/Matlab aware of the GPML package % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2016-08-29. disp(['executing gpml startup script...']); OCT = exist('OCTAVE_VERSION') ~= 0; % check if we run Matlab or Octave me = mfilename; % what is my filename mydir = which(me); mydir = mydir(1:end-2-numel(me)); % where am I located if OCT && numel(mydir)==2 if strcmp(mydir,'./'), mydir = [pwd,mydir(2:end)]; end end % OCTAVE 3.0.x relative, MATLAB and newer have absolute path addpath(mydir(1:end-1)) addpath([mydir,'cov']) addpath([mydir,'doc']) addpath([mydir,'inf']) addpath([mydir,'lik']) addpath([mydir,'mean']) addpath([mydir,'prior']) addpath([mydir,'util']) clear me mydirgpml-matlab-v4.0-2016-10-19/cov/covPERiso.m000644 000765 000024 00000001215 12777177046 017716 0ustar00carlstaff000000 000000 function varargout = covPERiso(varargin) % Wrapper for PERiodic covariance function covPER.m. % % Periodic covariance function from an arbitrary covariance function k0 via % embedding IR^D into IC^D. % The covariance function is parameterized as: % % k(x,z) = k0(u(x),u(z)), u(x) = [sin(pi*x/p); cos(pi*x/p)] % % where the period p belongs to covPERiso and hyp0 belongs to k0: % % hyp = [ log(p) % hyp0 ] % % Note that for k0 = covSEiso and D = 1, a faster alternative is covPeriodic. % % Copyright (c) by Hannes Nickisch, 2016-04-25. % % See also COVFUNCTIONS.M. varargout = cell(max(1,nargout),1); [varargout{:}] = covPER('iso',varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covPref.m000644 000765 000024 00000004025 12757566731 017454 0ustar00carlstaff000000 000000 function [K,dK] = covPref(cov, hyp, x, z) % covPref - covariance function for preference learning. The covariance % function corresponds to a prior on f(x1) - f(x2). % % k(x,z) = k_0(x1,z1) + k_0(x2,z2) - k_0(x1,z2) - k_0(x2,z1). % % The hyperparameters are: % % hyp = [ hyp_k0 ] % % For more help on design of covariance functions, try "help covFunctions". % % See Collaborative Gaussian Processes for Preference Learning, NIPS 2014. % % Copyright (c) by Hannes Nickisch and Roman Garnett, 2016-04-17. % % See also COVFUNCTIONS.M. if nargin<3, K = strrep(feval(cov{:}),'D','D/2'); return; end % no of params if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode x1 = x(:,1:end/2); x2 = x(:,1+end/2:end); if dg || xeqz z1 = x1; z2 = x2; else z1 = z(:,1:end/2); z2 = z(:,1+end/2:end); end if xeqz [K11,dK11] = feval(cov{:},hyp,x1); [K22,dK22] = feval(cov{:},hyp,x2); else if dg [K11,dK11] = feval(cov{:},hyp,x1,'diag'); [K22,dK22] = feval(cov{:},hyp,x2,'diag'); else [K11,dK11] = feval(cov{:},hyp,x1,z1); [K22,dK22] = feval(cov{:},hyp,x2,z2); end end [K12,dK12] = feval(cov{:},hyp,x1,z2); [K21,dK21] = feval(cov{:},hyp,x2,z1); if dg, K12 = diag(K12); K21 = diag(K21); end K = K11 + K22 - K12 - K21; dK = @(Q) dirder(Q,dK11,dK22,dK12,dK21,dg,xeqz); function [dhyp,dx] = dirder(Q,dK11,dK22,dK12,dK21,dg,xeqz) if nargout > 1 [dhyp11,dx11] = dK11(Q); [dhyp22,dx22] = dK22(Q); if dg [dhyp12,dx12] = dK12(diag(Q)); [dhyp21,dx21] = dK21(diag(Q)); else [dhyp12,dx12] = dK12(Q); [dhyp21,dx21] = dK21(Q); end if xeqz [junk,dx21t] = dK12(Q'); [junk,dx12t] = dK21(Q'); dx = [dx11-dx12-dx21t, dx22-dx21-dx12t]; else dx = [dx11-dx12, dx22-dx21]; end else dhyp11 = dK11(Q); dhyp22 = dK22(Q); if dg dhyp12 = dK12(diag(Q)); dhyp21 = dK21(diag(Q)); else dhyp12 = dK12(Q); dhyp21 = dK21(Q); end end dhyp = dhyp11 + dhyp22 - dhyp12 - dhyp21;gpml-matlab-v4.0-2016-10-19/cov/covLIN.m000644 000765 000024 00000001535 12757566731 017205 0ustar00carlstaff000000 000000 function [K,dK] = covLIN(mode,hyp,x,z) % Linear covariance function. % The covariance function is parameterized as: % % k(x,z) = dot(x,z) % % where dot(x,z) is a dot product. The hyperparameters are: % % hyp = [ hyp_maha ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-26. % % See also covDot.m. if nargin<1, mode = 'eye'; end, narg = nargin; % default mode if ~ischar(mode) % compatible to old version if nargin>2, z = x; end if nargin>1, x = hyp; end if nargin>0, hyp = mode; end mode = 'eye'; narg = narg+1; end if narg<3, K = covDot(mode); return, end if narg<4, z = []; end % make sure, z exists k = @(s) s; dk = @(s) ones(size(s)); if nargout > 1 [K,dK] = covDot(mode,k,dk,hyp,x,z); else K = covDot(mode,k,dk,hyp,x,z); endgpml-matlab-v4.0-2016-10-19/cov/covLINard.m000644 000765 000024 00000001313 12777177047 017666 0ustar00carlstaff000000 000000 function varargout = covLINard(varargin) % Wrapper for Linear covariance function covLin.m. % % Linear covariance function with Automatic Relevance Determination (ARD). The % covariance function is parameterized as: % % k(x,z) = x'*inv(P)*z % % where the P matrix is diagonal with ARD parameters ell_1^2,...,ell_D^2, where % D is the dimension of the input space. The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) ] % % Note that there is no bias term; use covConst to add a bias. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-26. % % See also covLin.m. varargout = cell(max(1,nargout),1); [varargout{:}] = covLIN('ard',varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covPoly.m000644 000765 000024 00000003244 12757566731 017505 0ustar00carlstaff000000 000000 function [K,dK] = covPoly(mode, d, hyp, x, z) % Polynomial covariance function. The covariance function is parameterized as: % % k(x,z) = sf^2 * ( c + s )^d , where s = x*inv(P)*z is the dot product % % The hyperparameters are: % % hyp = [ hyp_dot % log(c) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-23. % % See also covDot.m. if nargin<1, mode = 'eye'; end % default mode if ~ischar(mode) % compatible to old version if nargin>3, z = x; end if nargin>2, x = hyp; end if nargin>1, hyp = d; end if nargin>0, d = mode; end mode = 'eye'; narg = nargin+1; else narg = nargin; end if narg<4, K = [covDot(mode),'+2']; return, end % report number of parameters if narg<5, z = []; end % make sure, z exists [n,D] = size(x); % dimensions ne = eval(covDot(mode)); c = exp(hyp(ne+1)); % inhomogeneous offset sf2 = exp(2*hyp(ne+2)); % signal variance if d~=max(1,fix(d)), error('only nonzero integers allowed for d'), end % degree k = @(s) (c+s).^d; dk = @(s) d*(c+s).^(d-1); if nargout > 1 [K,dK0] = covScale({'covDot',mode,k,dk},hyp([1:ne,ne+2]),x,z); S = covDot(mode,@(s)s,[],hyp(1:ne),x,z); dK = @(Q) dirder(Q,S,dK0,c,d,sf2,ne); else K = covScale({'covDot',mode,k,dk},hyp([1:ne,ne+2]),x,z); end function [dhyp,dx] = dirder(Q,S,dK0,c,d,sf2,ne) if nargout > 1, [dhyp,dx] = dK0(Q); else dhyp = dK0(Q); end dhyp = [dhyp(1:ne); c*d*sf2*(Q(:)'*(c+S(:)).^(d-1)); dhyp(ne+1)]; % insert cgpml-matlab-v4.0-2016-10-19/cov/covLINone.m000644 000765 000024 00000002706 12757566731 017710 0ustar00carlstaff000000 000000 function [K,dK] = covLINone(hyp, x, z) % Linear covariance function with a single hyperparameter. The covariance % function is parameterized as: % % k(x,z) = (x'*z + 1)/t^2; % % where the P matrix is t2 times the unit matrix. The second term plays the % role of the bias. The hyperparameter is: % % hyp = [ log(t) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-23. % % See also COVFUNCTIONS.M. if nargin<2, K = '1'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode it2 = exp(-2*hyp); % t2 inverse % precompute inner products if dg % vector kxx K = sum(x.*x,2); else if xeqz % symmetric matrix Kxx K = x*x'; else % cross covariances Kxz K = x*z'; end end K = it2*(K+1); % covariances if nargout > 1, dK = @(Q) dirder(Q,K,x,z,it2,xeqz,dg); end % dir derivative function [dhyp,dx] = dirder(Q,K,x,z,it2,xeqz,dg) dhyp = -2*Q(:)'*K(:); if nargout>1 if dg dx = zeros(size(x)); else if xeqz dx = it2*(Q*x+Q'*x); else dx = it2*Q*z; end end endgpml-matlab-v4.0-2016-10-19/cov/covLINiso.m000644 000765 000024 00000001115 12777177047 017712 0ustar00carlstaff000000 000000 function varargout = covLINiso(varargin) % Wrapper for Linear covariance function covLin.m. % % Linear covariance function with Automatic Relevance Determination (ARD). The % covariance function is parameterized as: % % k(x,z) = x'*inv(P)*z % % where the P matrix is ell^2 times the unit matrix. The hyperparameters are: % % hyp = [ log(ell) ] % % Note that there is no bias term; use covConst to add a bias. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-26. % % See also covLin.m. varargout = cell(max(1,nargout),1); [varargout{:}] = covLIN('iso',varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covPERard.m000644 000765 000024 00000001303 12777177047 017671 0ustar00carlstaff000000 000000 function varargout = covPERard(varargin) % Wrapper for PERiodic covariance function covPER.m. % % Periodic covariance function from an arbitrary covariance function k0 via % embedding IR^D into IC^D. % The covariance function is parameterized as: % % k(x,z) = k0(u(x),u(z)), u(x) = [sin(pi*x./p); cos(pi*x./p)] % % where the period p belongs to covPERiso and hyp0 belong to k0: % % hyp = [ log(p_1) % log(p_2) % . % log(p_D) % hyp0 ] % % Note that for k0 = covSEard and D = 1, a faster alternative is covPeriodic. % % Copyright (c) by Hannes Nickisch, 2016-04-25. % % See also COVFUNCTIONS.M. varargout = cell(max(1,nargout),1); [varargout{:}] = covPER('ard',varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covOne.m000644 000765 000024 00000002130 12757566731 017274 0ustar00carlstaff000000 000000 function [K,dK] = covOne(hyp, x, z) % Constant (degenerate) covariance function, with unit variance. % The covariance function is specified as: % % k(x,z) = 1 % % hyp = [ ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Hannes Nickisch, 2016-04-17. % % See also COVFUNCTIONS.M. if nargin<2, K = '0'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists dg = strcmp(z,'diag'); % determine mode n = size(x,1); if dg % vector kxx K = ones(n,1); else if isempty(z) % symmetric matrix Kxx K = ones(n,n); else % cross covariances Kxz K = ones(n,size(z,1)); end end if nargout > 1 dK = @(Q) dirder(Q,K,x); % directional hyper derivative end function [dhyp,dx] = dirder(Q,K,x) dhyp = zeros(0,1); if nargout > 1, dx = zeros(size(x)); end gpml-matlab-v4.0-2016-10-19/cov/covSEisoU.m000644 000765 000024 00000001255 12777177047 017731 0ustar00carlstaff000000 000000 function varargout = covSEisoU(varargin) % Wrapper for Squared Exponential covariance function covSE.m. % % Squared Exponential covariance function with isotropic distance measure with % unit magnitude. The covariance function is parameterized as: % % k(x,z) = exp(-(x-z)'*inv(P)*(x-z)/2) % % where the P matrix is ell^2 times the unit matrix and sf2 is the signal % variance. The hyperparameters are: % % hyp = [ log(ell) ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-27. % % See also covSE.m. varargout = cell(max(1,nargout),1); [varargout{:}] = covSE('iso',[],varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covMaha.m000644 000765 000024 00000020045 12757566731 017426 0ustar00carlstaff000000 000000 function [K,dK,D2] = covMaha(mode, par, k, dk, hyp, x, z) % Mahalanobis distance-based covariance function. The covariance function is % parameterized as: % % k(x,z) = k(r^2), r^2 = maha(x,P,z) = (x-z)'*inv(P)*(x-z), % % where the matrix P is the metric. % % Parameters: % 1) mode,par: % We offer different modes (mode) with their respective parameters (par): % mode = par = inv(P) = hyp = % 'eye' [] eye(D) [] % 'iso' [] ell^2*eye(D) [log(ell)] % 'ard' [] diag(ell.^2) [log(ell_1); ..; log(ell_D)] % 'proj' d L'*L [L_11; L_21; ..; L_dD] % 'fact' d L'*L + diag(f) [L_11; L_21; ..; L_dD; f_1; ..; f_D] % 'vlen' llen l(x,z)^2*eye(D) [hyp_llen] % In the last mode, the covariance function is turned into a nonstationary % covariance by a variable lengthscale l(x,z) = sqrt((len(x)^2+len(z)^2)/2), % where len(x) = exp(llen(x)) and llen is provided as additional parameter % in the form of a GPML mean function cell array. The final expression for the % 'vlen' covariance is: % k(x,z) = ( len(x)*len(z)/l(x,z)^2 )^(D/2) * k( (x-z)'*(x-z)/l(x,z)^2 ). % % 2) k,dk: % The functional form of the covariance is governed by two functions: % k: r^2 -> k(x,z), r^2 = maha(x,P,z) = (x-z)'*inv(P)*(x-z) % dk: r^2,k(x,z) -> d k(x,z) / d r2 % For example, the squared exponential covariance uses % k = @(r2) exp(-r2/2); dk = @(r2,k) (-1/2)*k; % Note that not all functions k,dk give rise to a valid i.e. positive % semidefinite covariance function k(x,z). % % 3) hyp,x,z: % These input parameters follow the usual covariance function interface. For the % composition of hyp, see 1). % % 4) K,dK: % See the usual covariance function interface. % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-05-04. % % See also COVFUNCTIONS.M. if nargin<1, mode = 'eye'; end, if nargin <2, par = []; end % default values mode_list = '''eye'', ''iso'', ''ard'', ''proj'', ''fact'', or ''vlen'''; switch mode case 'eye', ne = '0'; case 'iso', ne = '1'; case 'ard', ne = 'D'; case 'proj', ne = [num2str(par),'*D']; case 'fact', ne = [num2str(par),'*D+D']; case 'vlen', ne = feval(par{:}); otherwise, error('Parameter mode is either %s.',mode_list) end if nargin<6, K = ne; return; end % report number of parameters if nargin<7, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % sort out different modes [n,D] = size(x); ne = eval(ne); % dimensions hyp = hyp(:); % make sure hyp is a column vector if numel(hyp)~=ne, error('Wrong number of hyperparameters'), end switch mode % mvm with metric A=inv(P) case {'eye','vlen'} A = @(x) x; dAdhyp = @(dAdiag,dAmvm) zeros(0,1); case 'iso', A = @(x) x*exp(-2*hyp); dAdhyp = @(dAdiag,dAmvm) -2*sum(A(dAdiag')); case 'ard', A = @(x) bsxfun(@times,x,exp(-2*hyp')); dAdhyp = @(dAdiag,dAmvm) -2*A(dAdiag')'; case 'proj', d = par; L = reshape(hyp,d,D); A = @(x) (x*L')*L; dAdhyp = @(dAdiag,dAmvm) 2*reshape(dAmvm(L')',d*D,1); case 'fact', d = par; L = reshape(hyp(1:d*D),d,D); f = exp(hyp(d*D+1:end)); A = @(x) (x*L')*L + bsxfun(@times,x,f'); dAdhyp = @(dAdiag,dAmvm)[2*reshape(dAmvm(L')',d*D,1); f.*dAdiag]; end [D2,dmaha] = maha(x,A,z); T = 1; L2 = 1; % compute Mahalanobis distance if isequal(mode,'vlen') % evaluate variable lengthscales lx = exp(feval(par{:},hyp,x)); % L2 = (lx^2+lz^2)/2, P = lx*lz if dg L2 = lx.*lx; P = L2; else if xeqz, lz = lx; else lz = exp(feval(par{:},hyp,z)); end L2 = bsxfun(@plus,(lx.*lx)/2,(lz.*lz)'/2); P = lx*lz'; end D2 = D2./L2; T = (P./L2).^(D/2); % non-stationary covariance end K = k(D2); % evaluate covariance if nargout > 1 dK = @(Q) dirder(Q,K,dk,T,D2,L2,dmaha,dAdhyp,mode,par,hyp,x,z,dg,xeqz); % dir hyp deriv end if isequal(mode,'vlen'), K = K.*T; end function [dhyp,dx] = dirder(Q,K,dk,T,D2,L2,dmaha,dAdhyp, mode,par,hyp,x,z,dg,xeqz) R = T.*dk(D2,K).*Q; switch mode case 'eye', dx = dmaha(R); dhyp = zeros(0,1); % fast shortcut case 'iso', dx = dmaha(R); dhyp = -2*R(:)'*D2(:); % fast shortcut case 'vlen', dx = dmaha(R); lx2 = exp(-2*feval(par{:},hyp,x)); dx = bsxfun(@times,dx,lx2); D = size(x,2); % only correct for lx = const. if dg dhyp = zeros(size(hyp)); else [llx,dllx] = feval(par{:},hyp,x); lx = exp(llx); if xeqz, lz = lx; dllz = dllx; else [llz,dllz] = feval(par{:},hyp,z); lz = exp(llz); end A=(D/2)*Q.*K.*((lx*lz')./L2).^(D/2-1)./L2; B=(D2.*R+A.*(lx*lz'))./L2; dhyp = dllx(lx.*(A*lz-sum(B,2).*lx)) + dllz(lz.*(A'*lx-sum(B,1)'.*lz)); end otherwise, [dx,dAdiag,dAmvm] = dmaha(R); dhyp = dAdhyp(dAdiag,dAmvm); end % Mahalanobis squared distance function for A spd % D2 = maha(x,A,z) = (x-z)'*A*(x-z) % dx(Q) = d tr(Q'*D2) / d x % dA(Q) = d tr(Q'*D2) / d A function [D2,dmaha] = maha(x,A,z) if nargin<2, A = @(x) x; end % make sure the metric exists if nargin<3, z = []; end % make sure z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % sort out different modes n = size(x,1); m = size(z,1); % dimensions if dg % vector d2xx D2 = zeros(n,1); else % Computation of a^2 - 2*a*b + b^2 is less stable than (a-b)^2 because % numerical precision can be lost when both a and b have very large absolute % value and the same sign. For that reason, we subtract the mean from the % data beforehand to stabilise the computations. This is OK because the % squared error is independent of the mean. if xeqz mu = mean(x,1); else mu = (m/(n+m))*mean(z,1) + (n/(n+m))*mean(x,1); z = bsxfun(@minus,z,mu); end x = bsxfun(@minus,x,mu); Ax = A(x); sax = sum(x.*Ax,2); if xeqz % symmetric matrix D2xx Az = Ax; saz = sax; else % cross terms D2xz Az = A(z); saz = sum(z.*Az,2); end % remove numerical noise at the end and ensure that D2>=0 D2 = max(bsxfun(@plus,sax,bsxfun(@minus,saz',2*x*Az')),0); % computation end if nargout>1 dmaha = @(Q) maha_dirder(Q,x,A,z); end function [dx,dAdiag,dAmvm] = maha_dirder(Q,x,A,z) % directional derivative if nargin<3, z = []; end xeqz = isempty(z); dg = strcmp(z,'diag'); % sort out different modes q2 = sum(Q,2); q1 = sum(Q,1)'; sym = @(X) (X+X')/2; dAdense = size(x,2)<5; % estimated break-even between O(D*D*n) and O(4*D*n) if dg dx = zeros(size(x)); if nargout > 1 dAdiag = zeros(size(x,2),1); dAmvm = @(r) zeros(size(r)); end else if xeqz y = bsxfun(@times,q1+q2,x) - (Q+Q')*x; if nargout > 1 if dAdense % construct a dense matrix dA of size DxD in O(D*D*n) dA = sym(x'*y); dAdiag = diag(dA); dAmvm = @(r) dA*r; else % just perform an MVM avoiding a DxD matrix dA in O(4*D*n) dAdiag = sum(x.*y,1)'; dAmvm = @(r) (x'*(y*r) + y'*(x*r))/2; end end else Qz = Q*z; y = bsxfun(@times,q2,x) - Qz; if nargout > 1 yz = y-Qz; qz = bsxfun(@times,q1,z); if dAdense % construct a dense matrix dA of size DxD in O(D*D*n) dA = sym(x'*yz + z'*qz); dAdiag = diag(dA); dAmvm = @(r) dA*r; else % just perform an MVM avoiding a DxD matrix in O(8*D*n) dAdiag = sum(x.*yz,1)'+sum(z.*qz,1)'; dAmvm = @(r) (x'*(yz*r)+yz'*(x*r) + z'*(qz*r)+qz'*(z*r))/2; end end end dx = 2*A(y); endgpml-matlab-v4.0-2016-10-19/cov/covConst.m000644 000765 000024 00000001025 12777177047 017643 0ustar00carlstaff000000 000000 function varargout = covConst(varargin) % Wrapper for unit constant covariance function covOne.m. % % Covariance function for a constant function. The covariance function is % parameterized as: % % k(x,z) = sf^2 % % The scalar hyperparameter is: % % hyp = [ log(sf) ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-19. % % See also covOne.m. varargout = cell(max(nargout,1),1); [varargout{:}] = covScale({'covOne'},varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covADD.m000644 000765 000024 00000010055 12757566731 017150 0ustar00carlstaff000000 000000 function [K,dK] = covADD(cov, hyp, x, z) % Additive covariance function using 1d base covariance functions % cov_d(x_d,z_d;hyp_d) with individual hyperparameters hyp_d, d=1..D. % % k (x,z) = \sum_{r \in R} sf^2_r k_r(x,z), where 1<=r<=D and % k_r(x,z) = \sum_{|I|=r} \prod_{i \in I} cov_i(x_i,z_i;hyp_i) % % hyp = [ hyp_1 % hyp_2 % ... % hyp_D % log(sf_R(1)) % ... % log(sf_R(end)) ] % % where hyp_d are the parameters of the 1d covariance function which are shared % over the different values of r=R(1),..,R(end) where 1<=r<=D. % % Usage: covADD({[1,3,4],cov}, ...) or % covADD({[1,3,4],cov_1,..,cov_D}, ...). % % Please see the paper "Additive Gaussian Processes" by Duvenaud, Nickisch and % Rasmussen, NIPS, 2011 for details. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-26. % multiple covariance support contributed by Truong X. Nghiem % % See also COVFUNCTIONS.M. R = fix(cov{1}); % only positive integers are allowed if min(R)<1, error('only positive R up to D allowed'), end nr = numel(R); % number of different degrees of interaction nc = numel(cov)-1; % number of provided covariances for j=1:nc, if ~iscell(cov{j+1}), cov{j+1} = {cov{j+1}}; end, end if nc==1, nh = eval(feval(cov{2}{:})); % no of hypers per individual covariance else nh = zeros(nc,1); for j=1:nc, nh(j) = eval(feval(cov{j+1}{:})); end, end if nargin<3 % report number of hyper parameters if nc==1, K = ['D*', int2str(nh), '+', int2str(nr)]; else K = ['(',int2str(nh(1))]; for ii=2:nc, K = [K,'+',int2str(nh(ii))]; end K = [K, ')+', int2str(nr)]; end return end if nargin<4, z = []; end % make sure, z exists [n,D] = size(x); % dimensionality if nc==1, nh = ones(D,1)*nh; cov = [cov(1),repmat(cov(2),1,D)]; end nch = sum(nh); % total number of cov hypers sf2 = exp( 2*hyp(nch+(1:nr)) ); % signal variances of individual degrees [Kd,dKd] = Kdim(cov(2:end),nh,hyp(1:nch),x,z); % eval dimensionwise covariances EE = elsympol(Kd,max(R)); % Rth elementary symmetric polynomials K = 0; for ii=1:nr, K = K + sf2(ii)*EE(:,:,R(ii)+1); end % sf2 weighted sum if nargout > 1 dK = @(Q) dirder(Q,Kd,dKd,EE,R,sf2); end function [dhyp,dx] = dirder(Q,Kd,dKd,EE,R,sf2) D = numel(dKd); n = size(Q,1); nr = numel(R); dhyp = zeros(0,1); if nargout > 1, dx = zeros(n,D); end % allocate if required for j=1:D % the final derivative is a sum of multilinear terms, so if only one term % depends on the hyperparameter under consideration, we can factorise it % out and compute the sum with one degree less, the j-th elementary % covariance depends on the hyperparameter batch hyp(j) and inputs x(:,j) E = elsympol(Kd(:,:,[1:j-1,j+1:D]),max(R)-1); % R-1th elementary sym polyn Qj = 0; for ii=1:nr, Qj = Qj + sf2(ii)*E(:,:,R(ii)); end % sf2 weighted sum if nargout > 1, [dhypj,dxj] = dKd{j}(Qj.*Q); dx(:,j) = dxj; else dhypj = dKd{j}(Qj.*Q); end, dhyp = [dhyp; dhypj]; end dhyp = [dhyp; 2*squeeze(sum(sum(bsxfun(@times,EE(:,:,R+1),Q),1),2)).*sf2]; % evaluate dimensionwise covariances K function [K,dK] = Kdim(cov,nh,hyp,x,z) [n,D] = size(x); % dimensionality xeqz = numel(z)==0; dg = strcmp(z,'diag') && numel(z)>0; % determine mode if dg % allocate memory K = zeros(n,1,D); else if xeqz, K = zeros(n,n,D); else K = zeros(n,size(z,1),D); end end dK = cell(D,1); for d=1:D hyp_d = hyp(sum(nh(1:d-1))+(1:nh(d))); % hyperparamter of dimension d if dg [K(:,:,d),dK{d}] = feval(cov{d}{:},hyp_d,x(:,d),'diag'); else if xeqz [K(:,:,d),dK{d}] = feval(cov{d}{:},hyp_d,x(:,d)); else [K(:,:,d),dK{d}] = feval(cov{d}{:},hyp_d,x(:,d),z(:,d)); end end endgpml-matlab-v4.0-2016-10-19/cov/covGaboriso.m000644 000765 000024 00000001353 12777177052 020322 0ustar00carlstaff000000 000000 function varargout = covGaboriso(varargin) % Wrapper for Gabor covariance function covGabor.m. % % Gabor covariance function with length scale ell and period p. The % covariance function is parameterized as: % % k(x,z) = h(x-z) with h(t) = exp(-t'*t/(2*ell^2))*cos(2*pi*sum(t)/p). % % The hyperparameters are: % % hyp = [ log(ell) % log(p) ] % % Note that covSM implements a weighted sum of Gabor covariance functions, but % using an alternative (spectral) parameterization. % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Hannes Nickisch, 2016-04-25. % % See also COVFUNCTIONS.M, COVGABOR.M, COVSM.M. varargout = cell(max(1,nargout),1); [varargout{:}] = covGabor('iso',varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covSEproj.m000644 000765 000024 00000001242 12777177047 017760 0ustar00carlstaff000000 000000 function varargout = covSEproj(d,varargin) % Wrapper for Squared Exponential covariance function covSE.m. % % Projected squared exponential covariance. The covariance function is % parameterized as: % % k(x,z) = sf^2 * exp(-(x-z)'*inv(P)*(x-z)/2) % % where the inv(P) matrix is L'*L with L of size (d,D) and sf^2 is the signal % variance. % % The hyperparameters are: % hyp = [ L_11; % L_21; % .. % L_d1; % .. % L_dD; % log(sf) ], % % Copyright (c) by Roman Garnett & Hannes Nickisch, 2016-04-27. % % See also COVFUNCTIONS.M. varargout = cell(max(1,nargout),1); [varargout{:}] = covScale({'covSE','proj',d},varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covGaborard.m000644 000765 000024 00000001536 12777177047 020305 0ustar00carlstaff000000 000000 function varargout = covGaborard(varargin) % Wrapper for Gabor covariance function covGabor.m. % % Gabor covariance function with length scales ell and periods p. The covariance % function is parameterized as: % % k(x,z) = h(x-z), h(t) = exp(-sum(t.^2./(2*ell.^2)))*cos(2*pi*sum(t./p)). % % The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) % log(p_1) % log(p_2) % .. % log(p_D) ] % % For more help on design of covariance functions, try "help covFunctions". % % Note that covSM implements a weighted sum of Gabor covariance functions, but % using an alternative (spectral) parameterization. % % Copyright (c) by Hannes Nickisch, 2016-04-25. % % See also COVFUNCTIONS.M, COVGABOR.M, COVSM.M. varargout = cell(max(1,nargout),1); [varargout{:}] = covGabor('ard',varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covSM.m000644 000765 000024 00000015466 12757566731 017112 0ustar00carlstaff000000 000000 function [K,dK] = covSM(Q, hyp, x, z) % Gaussian Spectral Mixture covariance function. The covariance function % parametrization depends on the sign of Q. % % Let t(Dx1) be an offset vector in dataspace e.g. t = x-z. Then w(DxP) % are the weights and m(Dx|Q|) = 1/p, v(Dx|Q|) = (2*pi*ell)^-2 are spectral % means (frequencies) and variances, where p is the period and ell the length % scale of the Gabor function h(t2v,tm) given by the expression % h(t2v,tm) = exp(-2*pi^2*t2v).*cos(2*pi*tm) % % Then, the two covariances are obtained as follows: % % SM, spectral mixture: Q>0 => P = 1 % k(x,z) = w'*h((t.*t)'*v,t'*m), t = x-z % % SMP, spectral mixture product: Q<0 => P = D % k(x,z) = prod(w'*h(T*T*v,T*m)), T = diag(t), t = x-z % % Note that for D=1, the two modes +Q and -Q are exactly the same. % % The hyperparameters are: % % hyp = [ log(w(:)) % log(m(:)) % log(sqrt(v(:))) ] % % For more help on design of covariance functions, try "help covFunctions". % % Note that the spectral density H(s) = F[ h(t) ] of covGaboriso is given by % H(s) = N(s|m,v)/2 + N(s|-m,v)/2 where m=1/p is the mean and v=(2*pi*ell)^-2 % is the variance of a symmetric Gaussian mixture. Hence the covGaboriso % covariance forms a basis for the class of stationary covariances since a % weighted sum of covGaboriso covariances corresponds to an isotropic % location-scale mixture of a symmetric Gaussian mixture in the spectral domain. % % For more details, see % [1] SM: Gaussian Process Kernels for Pattern Discovery and Extrapolation, % ICML, 2013, by Andrew Gordon Wilson and Ryan Prescott Adams, % [2] SMP: GPatt: Fast Multidimensional Pattern Extrapolation with GPs, % arXiv 1310.5288, 2013, by Andrew Gordon Wilson, Elad Gilboa, % Arye Nehorai and John P. Cunningham, and % [3] Covariance kernels for fast automatic pattern discovery and extrapolation % with Gaussian processes, Andrew Gordon Wilson, PhD Thesis, January 2014. % http://www.cs.cmu.edu/~andrewgw/andrewgwthesis.pdf % [4] http://www.cs.cmu.edu/~andrewgw/pattern/. % % For Q>0, covSM corresponds to Eq. 4.7 in Ref [3]. % For Q<0, covSM corresponds to Eq. 14 in Ref [2] and Eq. 5.3 in Ref [3] but our % w here corresponds to w^2 in Eq. 14. % % Copyright (c) by Andrew Gordon Wilson and Hannes Nickisch, 2016-05-06. % % See also COVFUNCTIONS.M, COVGABORISO.M, COVGABORARD.M. if nargin<1, error('You need to provide Q.'), end smp = Q<0; Q = abs(Q); % switch between covSM and covSMP mode if nargin<3 % report no of parameters if smp, K = '3*D*'; else K = '(1+2*D)*'; end, K = [K,sprintf('%d',Q)]; return end if nargin<4, z = []; end % make sure, z exists % Note that we have two implementations covSMgabor and covSMfast. The former % constructs a weighted sum of products of covGabor covariances using covMask, % covProd, covScale and covSum while the latter is a standalone direct % implementation. The latter tends to be faster. if nargout > 1 if smp [K,dK] = covSMgabor(Q,hyp,x,z,smp); else [K,dK] = covSMfast(Q,hyp,x,z,smp); % faster direct alternative end else K = covSMfast(Q,hyp,x,z,smp); % faster direct alternative end function [K,dK] = covSMgabor(Q,hyp,x,z,smp) [n,D] = size(x); P = smp*D+(1-smp); % dimensionality, P=D or P=1 lw = reshape(hyp( 1:P*Q) ,P,Q); % log mixture weights lm = reshape(hyp(P*Q+ (1:D*Q)),D,Q); % log spectral means ls = reshape(hyp(P*Q+D*Q+(1:D*Q)),D,Q); % log spectral standard deviations if smp % 1) the product of weighted sums of 1d covGabor functions or fac = cell(1,D); for d=1:D add = cell(1,Q); % a) addends for weighted sum of 1d Gabor functions for q=1:Q, add{q} = {'covScale',{'covMask',{d,{'covGaboriso'}}}}; end fac{d} = {'covSum',add}; % b) combine addends into sum end fac{D+1} = 0; % disable cache to avoid memory problems cov = {'covProd',fac}; % c) combine factors into product else % 2) the weighted sum of multivariate covGaborard covariance functions. % weighted sum of multivariate Gabor functions add = cell(1,Q); for q=1:Q, add{q} = {'covScale',{'covGaborard'}}; end cov = {'covSum',add}; % combine into sum end if smp % assemble hyp; covGabor is parametrised using -ls-log(2*pi) and -lm hypgb = [-log(2*pi)-ls(:)'; -lm(:)'; lw(:)'/2]; else hypgb = [-log(2*pi)-ls; -lm; lw/2 ]; end if nargout>1 [K,dK] = feval(cov{:},hypgb(:),x,z); dK = @(R) dirder_gabor(R,dK,Q,x,smp); else K = feval(cov{:},hypgb(:),x,z); end function [dhyp,dx] = dirder_gabor(R,dK,Q,x,smp) if nargout > 1 [dhyp,dx] = dK(R); else dhyp = dK(R); end D = size(x,2); if smp dhyp = reshape(dhyp,3,D*Q); dls = -dhyp(1,:); dlm = -dhyp(2,:); dlw = 0.5*dhyp(3,:); else dhyp = reshape(dhyp,2*D+1,Q); dls = -dhyp(1:D,:); dlm = -dhyp(D+1:2*D,:); dlw = 0.5*dhyp(2*D+1,:); end dhyp = [dlw(:); dlm(:); dls(:)]; function [K,dK] = covSMfast(Q,hyp,x,z,smp) xeqz = isempty(z); dg = strcmp(z,'diag'); % sort out different types [n,D] = size(x); P = smp*D+(1-smp); % dimensionality, P=D or P=1 w = exp(reshape( hyp( 1:P*Q) ,P,Q)); % mixture weights m = exp(reshape( hyp(P*Q+ (1:D*Q)),D,Q)); % spectral means v = exp(reshape(2*hyp(P*Q+D*Q+(1:D*Q)),D,Q)); % spectral variances if dg T = zeros(n,1,D); else if xeqz T = 2*pi*bsxfun(@minus,reshape(x,n,1,D),reshape(x,1,n,D)); else T = 2*pi*bsxfun(@minus,reshape(x,n,1,D),reshape(z,1,[],D)); end end, T = reshape(T,[],D); if smp h = @(t2v,tm) exp(-0.5*t2v).*cos(tm); % Gabor function K = 1; w = reshape(w,Q,P)'; m = reshape(m,Q,D)'; v = reshape(v,Q,D)'; for d=1:D K = K .* ( h( (T(:,d).*T(:,d))*v(d,:), T(:,d)*m(d,:) )*w(d,:)' ); end K = reshape(K.*ones(size(T,1),1),n,[]); else E = exp(-0.5*(T.*T)*v); H = E.*cos(T*m); K = reshape(H*w',n,[]); if nargout>1 vec = @(x) x(:); dKdhyp = @(R) [ (H'*R(:)).*w'; -vec( ((E.*sin(T*m).*(R(:)*w))'* T )'.*m ); -vec( ((H.* (R(:)*w))'*(T.*T))'.*v ) ]; dK = @(R) dirder_fast(R,E,T,H,dKdhyp,x,z, m,v,w); end end function [dhyp,dx] = dirder_fast(R, E,T,H,dKdhyp,x,z, m,v,w) dhyp = dKdhyp(R); if nargout>1 xeqz = isempty(z); dg = strcmp(z,'diag'); [n,D] = size(x); if dg dx = zeros(size(x)); else A = reshape( (R(:)*w).*E.*sin(T*m)*m' + (((R(:)*w).*H)*v').*T, n, [], D); dx = -2*pi*squeeze(sum(A,2)); if xeqz, dx = dx + 2*pi*squeeze(sum(A,1)); end end endgpml-matlab-v4.0-2016-10-19/cov/covGE.m000644 000765 000024 00000002242 12757566731 017052 0ustar00carlstaff000000 000000 function [K,dK] = covGE(mode, par, hyp, varargin) % Gamma Exponential covariance function. % The covariance function is parameterized as: % % k(x,z) = exp(-r^gamma), r = maha(x,z) % % where maha(x,z) is a Mahalanobis distance and gamma is the shape parameter % for the GE covariance. The hyperparameters are: % % hyp = [ hyp_maha % log(gamma/(2-gamma)) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-27. % % See also COVFUNCTIONS.M. if nargin<1, mode = 'eye'; end, if nargin <2, par = []; end % default values if nargin<4, K = [covMaha(mode,par),'+1']; return, end gamma = 2/(1+exp(-hyp(end))); k = @(d2) exp(-d2.^(gamma/2)); dk = @(d2,k) -gamma/2*set_zero(k.*d2.^(gamma/2-1),d2==0); if nargout==2 [K,dKmaha,D2] = covMaha(mode,par,k,dk,hyp(1:end-1),varargin{:}); dK = @(Q) dirder(Q,K,D2,dKmaha,gamma); else K = covMaha(mode,par,k,dk,hyp(1:end-1),varargin{:}); end function [dhyp,dx] = dirder(Q,K,D2,dKmaha,gamma) if nargout==1 dhyp = dKmaha(Q); else [dhyp,dx] = dKmaha(Q); end Q = Q.*K; B = (gamma-2)*gamma/4 * D2.^(gamma/2) .* set_zero(log(D2),D2==0); dhyp = [dhyp; Q(:)'*B(:)]; function A = set_zero(A,I), A(I) = 0;gpml-matlab-v4.0-2016-10-19/cov/covRQ.m000644 000765 000024 00000002235 12757566731 017103 0ustar00carlstaff000000 000000 function [K,dK] = covRQ(mode, par, hyp, varargin) % Rational Quadratic covariance function. % The covariance function is parameterized as: % % k(x,z) = [1 + maha(x,z)/(2*alpha)]^(-alpha) % % where maha(x,z) is a Mahalanobis distance and alpha is the shape parameter % for the RQ covariance. The hyperparameters are: % % hyp = [ hyp_maha % log(alpha) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-05-23. % % See also COVFUNCTIONS.M. if nargin < 1, error('Mode cannot be empty.'); end % no default if nargin < 2, par = []; end % default if nargin<4, K = [covMaha(mode,par),'+1']; return, end alpha = exp(hyp(end)); k = @(d2) (1+0.5*d2/alpha).^(-alpha); dk = @(d2,k) -k./(2+d2/alpha); if nargout==2 [K,dKmaha,D2] = covMaha(mode,par,k,dk,hyp(1:end-1),varargin{:}); dK = @(Q) dirder(Q,K,D2,dKmaha,alpha); else K = covMaha(mode,par,k,dk,hyp(1:end-1),varargin{:}); end function [dhyp,dx] = dirder(Q,K,D2,dKmaha,alpha) if nargout==1 dhyp = dKmaha(Q); else [dhyp,dx] = dKmaha(Q); end Q = Q.*K; B = 1+0.5*D2/alpha; dhyp = [dhyp; sum(sum( Q.*(0.5*D2./B-alpha*log(B)) ))];gpml-matlab-v4.0-2016-10-19/cov/covPP.m000644 000765 000024 00000003600 12757566731 017075 0ustar00carlstaff000000 000000 function varargout = covPP(mode, par, v, hyp, x, varargin) % Piecewise Polynomial covariance function with compact support, v = 0,1,2,3. % The covariance functions are 2v times contin. diff'ble and the corresponding % processes are hence v times mean-square diffble. The covariance function is: % % k(x,z) = max(1-r,0)^(j+v) * f(r,j) with j = floor(D/2)+v+1 % % where r is the Mahalanobis distance sqrt(maha(x,z)). The hyperparameters are: % % hyp = [ hyp_maha ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-05-23. % % See also covMaha.m. if nargin < 1, error('Mode cannot be empty.'); end % no default if nargin < 2, par = []; end % default varargout = cell(max(1, nargout), 1); % allocate mem for output if nargin<5, varargout{1} = covMaha(mode,par); return, end [n,D] = size(x); if all(v~=[0,1,2,3]), error('only 0,1,2 and 3 allowed for v'), end % degree j = floor(D/2)+v+1; % exponent switch v case 0, f = @(r,j) 1; df = @(r,j) 0; case 1, f = @(r,j) 1 + (j+1)*r; df = @(r,j) (j+1); case 2, f = @(r,j) 1 + (j+2)*r + ( j^2+ 4*j+ 3)/ 3*r.^2; df = @(r,j) (j+2) + 2*( j^2+ 4*j+ 3)/ 3*r; case 3, f = @(r,j) 1 + (j+3)*r + (6*j^2+36*j+45)/15*r.^2 ... + (j^3+9*j^2+23*j+15)/15*r.^3; df = @(r,j) (j+3) + 2*(6*j^2+36*j+45)/15*r ... + (j^3+9*j^2+23*j+15)/ 5*r.^2; end cs = @(r,e) (r<1).*max(1-r,0).^e; pp = @(r,j,v,f) cs(r,j+v ).* f(r,j); dpp = @(r,j,v,f) r.*cs(r,j+v-1).*( f(r,j)*(j+v) - max(1-r,0).*df(r,j) ); k = @(d2) pp( sqrt(d2), j, v, f ); dk = @(d2,k) set_zero( -(1/2)*dpp( sqrt(d2), j, v, f )./d2 , d2==0); [varargout{:}] = covMaha(mode, par, k, dk, hyp, x, varargin{:}); function A = set_zero(A,I), A(I) = 0;gpml-matlab-v4.0-2016-10-19/cov/covZero.m000644 000765 000024 00000002134 12757566731 017476 0ustar00carlstaff000000 000000 function [K,dK] = covZero(hyp, x, z) % Constant (degenerate) covariance function, with zero variance. % The covariance function is specified as: % % k(x,z) = 0 % % hyp = [ ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Hannes Nickisch, 2016-04-17. % % See also COVFUNCTIONS.M. if nargin<2, K = '0'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists dg = strcmp(z,'diag'); % determine mode n = size(x,1); if dg % vector kxx K = zeros(n,1); else if isempty(z) % symmetric matrix Kxx K = zeros(n,n); else % cross covariances Kxz K = zeros(n,size(z,1)); end end if nargout > 1 dK = @(Q) dirder(Q,K,x); % directional hyper derivative end function [dhyp,dx] = dirder(Q,K,x) dhyp = zeros(0,1); if nargout > 1, dx = zeros(size(x)); end gpml-matlab-v4.0-2016-10-19/cov/apx.m000644 000765 000024 00000051470 12777177052 016642 0ustar00carlstaff000000 000000 function K = apx(hyp,cov,x,opt) % (Approximate) linear algebra operations involving the covariance matrix K. % % A) Exact covariance computations. % There are no parameters in this mode. % Depending on the sign of W, we switch between % - the symmetric Cholesky mode [1], where B = I + sqrt(W)*K*sqrt(W), and % - the asymmetric LU mode [2], where B = I + K*W. % Note that (1) is faster but it is only applicable if all W are positive. % % B) Sparse covariance approximations aka FITC [4], VFE [5] and SPEP [3]. % We have a parameter opt.s from [0,1], the power in sparse power EP [3] % interpolating between the Fully Independent Training Conditionals (FITC) % approach [4] and a Variational Free Energy (VFE) approximation [5]. % In particular: % opt.s, default is 1.0 for FITC, opt.s = 0.0 corresponds to VFE. % Please see cov/apxSparse.m for details. % % C) Grid-based covariance approximations aka KISS-GP [6]. % Please see cov/apxGrid.m for further details and more parameters. % opt.cg_tol, default is 1e-6 as in Matlab's pcg function % opt.cg_maxit, default is min(n,20) as in Matlab's pcg function % The conjugate gradient-based linear system solver has two adjustable % parameters, the relative residual threshold for convergence opt.cg_tol and % the maximum number of MVMs opt.cg_maxit until the process stops. % opt.deg, default is 3 degree of interpolation polynomial % For equispaced axes, opt.deg sets the degree of the interpolation % polynomial along each of the p axes. Here 0 means nearest neighbor, % 1 means linear interpolation, and 3 uses a cubic. % For non-equispaced axes, only linear interpolation with inverse distance % weighting is offered and opt.deg is ignored. % opt.ldB2_cheby = true employs Monte-Carlo trace estimation aka the Hutchinson % method and Chebyshev polynomials to approximate the term log(det(B))/2 % stochastically, see [7]. The following four parameters configure different % aspects of the estimator and are only valid if opt.ldB2_cheby equals true. % opt.ldB2_cheby_hutch, default is 10, number of samples for the trace estim % opt.ldB2_cheby_degree, default is 15, degree of Chebyshev approx polynomial % opt.ldB2_cheby_maxit, default is 50, max # of MVMs to estimate max(eig(B)) % opt.ldB2_cheby_seed, default is [], random seed for the stoch trace estim % opt.stat = true returns a little bit of output summarising the exploited % structure of the covariance of the grid. % The log-determinant approximation employs Fiedler's 1971 inequality and a % rescaled version of the eigenspectrum of the covariance evaluated on the % complete grid. % % The call K = apx(hyp,cov,x,opt) yields a structure K with a variety of % fields. % 1) Matrix-vector multiplication with covariance matrix % K.mvm(x) = K*x % 2) Projection and its transpose (unity except for mode B) Sparse approx.) % post.alpha = K.P(solveKiW(f-m)) % post.L = L = @(r) -K.P(solveKiW(K.Pt(r))) % 3) Linear algebra functions depending on W % [ldB2,solveKiW,dW,dldB2,L] = K.fun(W) % a) Log-determinant (approximation), called f in the sequel % ldB2 = log(det(B))/2 % b) Solution of linear systems % solveKiW(r) = (K+inv(W)) \ r % c) Log-determinant (approximation) derivative w.r.t. W % dW = d f / d W, where f = ldB2(W), exact value dW = diag(inv(B)*K)/2 % d) Log-determinant (approximation) derivative w.r.t. hyperparameters % dhyp = dldB2(alpha,a,b) % Q = d f / d K, exact value would be Q = inv(K+inv(W)) % R = alpha*alpha' + 2*a*b' % Here dhyp(i) = tr( (Q-R)'*dKi )/2, where dKi = d K / d hyp(i). % e) Matrix (operator) to compute the predictive variance % L = -K.P(solveKiW(K.Pt(r))) either as a dense matrix or function. % See gp.m for details on post.L. % [1] Seeger, GPs for Machine Learning, sect. 4, TR, 2004. % [2] Jylanki, Vanhatalo & Vehtari, Robust GPR with a Student's-t % Likelihood, JMLR, 2011. % [3] Bui, Yan & Turner, A Unifying Framework for Sparse GP Approximation % using Power EP, 2016, https://arxiv.org/abs/1605.07066. % [4] Snelson & Ghahramani, Sparse GPs using pseudo-inputs, NIPS, 2006. % [5] Titsias, Var. Learning of Inducing Variables in Sparse GPs, AISTATS, 2009 % [6] Wilson & Nickisch, Kernel Interp. for Scalable Structured GPs, ICML, 2015 % [7] Han, Malioutov & Shin, Large-scale Log-det Computation through Stochastic % Chebyshev Expansions, ICML, 2015. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2016-10-05. % % See also apxSparse.m, apxGrid.m, gp.m. if nargin<4, opt = []; end % make sure variable exists if isnumeric(cov), c1 = 'numeric'; else c1 = cov{1}; end % detect matrix if isa(c1, 'function_handle'), c1 = func2str(c1); end % turn into string sparse = strcmp(c1,'apxSparse') || strcmp(c1,'covFITC'); grid = strcmp(c1,'apxGrid') || strcmp(c1,'covGrid'); exact = ~grid && ~sparse; if exact % A) Exact computations using dense matrix operations if strcmp(c1,'numeric'), K = cov; dK = []; % catch dense matrix case else [K,dK] = feval(cov{:},hyp.cov,x); % covariance matrix and dir derivative end K = struct('mvm',@(x)K*x, 'fun',@(W)ldB2_exact(W,K,dK), ... % mvm and fun 'P',@(x)x, 'Pt',@(x)x); % projection operations elseif sparse % B) Sparse approximations if isfield(opt,'s'), s = opt.s; else s = 1.0; end % default is FITC if isfield(hyp,'xu'), cov{3} = hyp.xu; end % hyp.xu provided, replace cov{3} xu = cov{3}; nu = size(xu,1); % extract inducing points [Kuu, dKuu] = feval(cov{2}{:}, hyp.cov, xu); % get the building blocks [Ku, dKu] = feval(cov{2}{:}, hyp.cov, xu, x); [diagK, ddiagK] = feval(cov{2}{:}, hyp.cov, x, 'diag'); snu2 = 1e-6*(trace(Kuu)/nu); % stabilise by 0.1% of signal std Luu = chol(Kuu+snu2*eye(nu)); % Kuu + snu2*I = Luu'*Luu V = Luu'\Ku; % V = inv(Luu')*Ku => V'*V = Q g = max(diagK-sum(V.*V,1)',0); % g = diag(K) - diag(Q) K.mvm = @(x) V'*(V*x) + bsxfun(@times,s*g,x); % efficient matrix-vector mult K.P = @(x) Luu\(V*x); K.Pt = @(x) V'*(Luu'\x); % projection operations xud = isfield(hyp,'xu'); % flag deciding whether to compute hyp.xu derivs K.fun = @(W) ldB2_sparse(W,V,g,Luu,dKuu,dKu,ddiagK,s,xud); elseif grid % C) Grid approximations n = size(x,1); if isfield(opt,'cg_tol'), cgtol = opt.cg_tol; % stop conjugate gradients else cgtol = 1e-6; end % same as in pcg if isfield(opt,'cg_maxit'), cgmit = opt.cg_maxit; % number of cg iterations else cgmit = min(n,20); end % same as in pcg if isfield(opt,'deg'), deg = opt.deg; else deg = 3; end % interpol. degree if isfield(opt,'stat'), stat = opt.stat; else stat = false; end % show stat cgpar = {cgtol,cgmit}; xg = cov{3}; p = numel(xg); % conjugate gradient, grid if isfield(opt,'ldB2_cheby'),cheby=opt.ldB2_cheby; else cheby=false; end if isfield(opt,'ldB2_cheby_hutch'),m=opt.ldB2_cheby_hutch; else m=10; end if isfield(opt,'ldB2_cheby_degree'),d=opt.ldB2_cheby_degree; else d=15; end if isfield(opt,'ldB2_cheby_maxit'),mit=opt.ldB2_cheby_maxit; else mit=50; end if isfield(opt,'ldB2_cheby_seed'),sd=opt.ldB2_cheby_seed; else sd=[]; end ldpar = {cheby,m,d,mit,sd}; % logdet parameters [Kg,Mx] = feval(cov{:},hyp.cov,x,[],deg); % grid cov structure, interp matrix if stat % show some information about the nature of the p Kronecker factors fprintf(apxGrid('info',Kg,Mx,xg,deg)); end K.mvm = @(x) Mx*Kg.mvm(Mx'*x); % mvm with covariance matrix K.P = @(x)x; K.Pt = @(x)x; % projection operations K.fun = @(W) ldB2_grid(W,K,Kg,xg,Mx,cgpar,ldpar); end %% A) Exact computations using dense matrix operations ========================= function [ldB2,solveKiW,dW,dldB2,L] = ldB2_exact(W,K,dK) isWneg = any(W<0); n = numel(W); if isWneg % switch between Cholesky and LU decomposition mode A = eye(n) + bsxfun(@times,K,W'); % asymmetric A = I+K*W [L,U,P] = lu(A); u = diag(U); % compute LU decomposition, A = P'*L*U signU = prod(sign(u)); % sign of U detP = 1; % compute sign (and det) of the permutation matrix P p = P*(1:n)'; for i=1:n % swap entries if i~=p(i), detP = -detP; j = find(p==i); p([i,j]) = p([j,i]); end end if signU~=detP % log becomes complex for negative values, encoded by inf ldB2 = Inf; else % det(L) = 1 and U triangular => det(A) = det(P)*prod(diag(U)) ldB2 = sum(log(abs(u)))/2; end % compute inverse if required if nargout>1, Q = U\(L\P); solveKiW = @(r) bsxfun(@times,W,Q*r); end if nargout>4, L = -diag(W)*Q; end % overwrite L else % symmetric B = I+sW*K*sW sW = sqrt(W); L = chol(eye(n)+sW*sW'.*K); % Cholesky factor of B ldB2 = sum(log(diag(L))); % log(det(B))/2 solveKiW = @(r) bsxfun(@times,solve_chol(L,bsxfun(@times,r,sW)),sW); if nargout>2, Q = bsxfun(@times,1./sW,solve_chol(L,diag(sW))); end end if nargout>2 dW = sum(Q.*K,2)/2; % d log(det(B))/2 / d W = diag(inv(inv(K)+W)) dldB2 = @(varargin) ldB2_deriv_exact(W,dK,Q, varargin{:}); % derivatives end function dhyp = ldB2_deriv_exact(W,dK,Q, alpha,a,b) if nargin>3, R = alpha*alpha'; else R = 0; end if nargin>5, R = R + 2*a*b'; end dhyp.cov = dK( bsxfun(@times,Q,W) - R )/2; %% B) Sparse approximations ==================================================== function [ldB2,solveKiW,dW,dldB2,L]=ldB2_sparse(W,V,g,Luu,dKuu,dKu,ddiagK,s,xud) z = s*g.*W; t = 1/s*log(z+1); i = z<1e-4; % s=0: t = g*W, s=1: t = log(g*W+1) t(i) = g(i).*W(i).*(1-z(i)/2+z(i).^2/3); % 2nd order Taylor for tiny z dt = 1./(z+1); d = W.*dt; % evaluate derivatives nu = size(Luu,1); Vd = bsxfun(@times,V,d'); Lu = chol(eye(nu) + V*Vd'); LuV = Lu'\V; % Lu'*Lu=I+V*diag(d)*V' ldB2 = sum(log(diag(Lu))) + sum(t)/2; % s=1 => t=log(g.*W+1), s=0 => t=g.*W md = @(r) bsxfun(@times,d,r); solveKiW = @(r) md(r) - md(LuV'*(LuV*md(r))); if nargout>2 % dW = d log(det(B))/2 / d W = diag(inv(inv(K)+W)) dW = sum(LuV.*((LuV*Vd')*V),1)' + s*g.*d.*sum(LuV.*LuV,1)'; dW = dt.*(g+sum(V.*V,1)'-dW)/2; % add trace "correction" term dldB2 = @(varargin) ldB2_deriv_sparse(V,Luu,d,LuV,dKuu,dKu,ddiagK,s,xud,... varargin{:}); if nargout>4 L = solve_chol(Lu*Luu,eye(nu))-solve_chol(Luu,eye(nu)); % Sigma-inv(Kuu) end end function dhyp = ldB2_deriv_sparse(V,Luu,d,LuV,dKuu,dKu,ddiagK,s,xud,alpha,a,b) % K + 1./W = V'*V + inv(D), D = diag(d) % Q = inv(K+inv(W)) = inv(V'*V + diag(1./d)) = diag(d) - LuVd'*LuVd; LuVd = bsxfun(@times,LuV,d'); diagQ = d - sum(LuVd.*LuVd,1)'; F = Luu\V; Qu = bsxfun(@times,F,d') - (F*LuVd')*LuVd; if nargin>9, diagQ = diagQ-alpha.*alpha; Qu = Qu-(F*alpha)*alpha'; end Quu = Qu*F'; if nargin>11 diagQ = diagQ-2*a.*b; Qu = Qu-(F*a)*b'-(F*b)*a'; Quu = Quu-2*(F*a)*(F*b)'; end diagQ = s*diagQ + (1-s)*d; % take care of s parameter Qu = Qu - bsxfun(@times,F,diagQ'); Quu = Quu - bsxfun(@times,F,diagQ')*F'; nu = size(Quu,1); Quu = Quu + 1e-6*trace(Quu)/nu*eye(nu); if xud dhyp.cov = ddiagK(diagQ)/2; dhyp.xu = 0; [dc,dx] = dKu(Qu); dhyp.cov = dhyp.cov + dc; dhyp.xu = dhyp.xu + dx; [dc,dx] = dKuu(Quu); dhyp.cov = dhyp.cov - dc/2; dhyp.xu = dhyp.xu - dx/2; else dhyp.cov = ddiagK(diagQ)/2 + dKu(Qu) - dKuu(Quu)/2; end %% C) Grid approximations ===================================================== function [ldB2,solveKiW,dW,dldB2,L] = ldB2_grid(W,K,Kg,xg,Mx,cgpar,ldpar) if all(W>=0) % well-conditioned symmetric case sW = sqrt(W); msW = @(x) bsxfun(@times,sW,x); mvmB = @(x) msW(K.mvm(msW(x)))+x; solveKiW = @(r) msW(linsolve(msW(r),mvmB,cgpar{:})); else % less well-conditioned symmetric case if some negative W mvmKiW = @(x) K.mvm(x)+bsxfun(@times,1./W,x); solveKiW = @(r) linsolve(r,mvmKiW,cgpar{:}); end % K*v = Mx*Kg.mvm(Mx'*v) dhyp.cov = []; % init if ldpar{1} % stochastic estimation of logdet cheby/hutchinson dK = @(a,b) apxGrid('dirder',Kg,xg,Mx,a,b); if nargout<3 % save some computation depending on required output ldB2 = logdet_sample(W,K.mvm,dK, ldpar{2:end}); else [ldB2,emax,dhyp.cov,dW] = logdet_sample(W,K.mvm,dK, ldpar{2:end}); end else s = 3; % Whittle embedding overlap factor [V,ee,e] = apxGrid('eigkron',Kg,xg,s); % perform eigen-decomposition [ldB2,de,dW] = logdet_fiedler(e,W); % Fiedler's upper bound, derivatives de = de.*double(e>0); % chain rule of max(e,0) in eigkron, Q = V*diag(de)*V' if nargout>3, dhyp.cov = ldB2_deriv_grid_fiedler(Kg,xg,V,ee,de,s); end end dldB2 = @(varargin) ldB2_deriv_grid(dhyp, Kg,xg,Mx, varargin{:}); if ~isreal(ldB2), error('Too many negative W detected.'), end L = @(r) -K.P(solveKiW(K.Pt(r))); function dhyp = ldB2_deriv_grid_fiedler(Kg,xg,V,ee,de,s) p = numel(Kg.kron); % number of Kronecker factors ng = [apxGrid('size',xg)',1]; % grid dimension dhyp = []; % dhyp(i) = trace( V*diag(de)*V'*dKi ) for i=1:p z = reshape(de,ng); Vi = V{i}; for j=1:p, if i~=j, z = apxGrid('tmul',ee{j}(:)',z,j); end, end if isnumeric(Vi) Q = bsxfun(@times,Vi,z(:)')*Vi'; dhci = Kg.kron(i).dfactor(Q); else kii = Kg.kron(i).factor.kii; [junk,ni] = apxGrid('expand',xg{i}); di = numel(ni); xs = cell(di,1); % generic (Strang) circular embedding grid for j=1:di, n2 = floor(ni(j)-1/2)+1; xs{j} = [1:n2,n2-2*ni(j)+2:0]'; end Fz = real(fftn(reshape(z,[ni(:)',1]))); % imaginary part of deriv is zero rep = 2*s*ones(1,di); if di==1, rep = [rep,1]; end % replication Fzw = repmat(Fz,rep); % replicate i.e. perform transpose of circ operation [junk,xw] = apxGrid('circ',kii,ni,s); % get Whittle circ embedding grid [junk,dkwi] = kii(apxGrid('expand',xw)); % evaluate derivative dhci = dkwi(Fzw(:)); end dhyp = [dhyp; dhci]; end function dhyp = ldB2_deriv_grid(dhyp, Kg,xg,Mx, alpha,a,b) if nargin>4, dhyp.cov = dhyp.cov-apxGrid('dirder',Kg,xg,Mx,alpha,alpha)/2; end if nargin>6, dhyp.cov = dhyp.cov-apxGrid('dirder',Kg,xg,Mx,a,b); end function q = linsolve(p,mvm,varargin) % solve q = mvm(p) via conjugate gradients [q,flag,relres,iter] = conjgrad(mvm,p,varargin{:}); % like pcg if ~flag,error('Not converged after %d iterations, r=%1.2e\n',iter,relres),end % Solve x=A*b with symmetric A(n,n), b(n,m), x(n,m) using conjugate gradients. % The method is along the lines of PCG but suited for matrix inputs b. function [x,flag,relres,iter,r] = conjgrad(A,b,tol,maxit) if nargin<3, tol = 1e-10; end if nargin<4, maxit = min(size(b,1),20); end x0 = zeros(size(b)); x = x0; if isnumeric(A), r = b-A*x; else r = b-A(x); end, r2 = sum(r.*r,1); r2new = r2; nb = sqrt(sum(b.*b,1)); flag = 0; iter = 1; relres = sqrt(r2)./nb; todo = relres>=tol; if ~any(todo), flag = 1; return, end on = ones(size(b,1),1); r = r(:,todo); d = r; for iter = 2:maxit if isnumeric(A), z = A*d; else z = A(d); end a = r2(todo)./sum(d.*z,1); a = on*a; x(:,todo) = x(:,todo) + a.*d; r = r - a.*z; r2new(todo) = sum(r.*r,1); relres = sqrt(r2new)./nb; cnv = relres(todo)=tol; d = d(:,~cnv); r = r(:,~cnv); % get rid of converged if ~any(todo), flag = 1; return, end b = r2new./r2; % Fletcher-Reeves d = r + (on*b(todo)).*d; r2 = r2new; end % Upper determinant bound on log |K*diag(W)+I| using Fiedler's 1971 inequality. % K = kron( kron(...,K{2}), K{1} ), W = diag(w) both symmetric psd. % The bound is exact for W = w*ones(N,1). Here, E = eig(K) are the % eigenvalues of the matrix K. % See also Prob.III.6.14 in Matrix Analysis, Bhatia 1997. % % Given nxn spd matrices C and D with ordered nx1 eigenvalues c, d % then det(C+D) <= prod(c+flipud(d))=exp(ub). function [ub,dE,dW,ie,iw] = logdet_fiedler(E,W) [E,ie] = sort(E,'descend'); [W,iw] = sort(W,'descend'); % sort vectors N = numel(E); n = numel(W); k = n/N*E; % dimensions, approximate spectrum of K if n>N, k = [k;zeros(n-N,1)]; else k = k(1:n); end % extend/shrink to match W kw1 = k.*W+1; ub = sum(log(kw1))/2; % evaluate upper bound dW = zeros(n,1); dW(iw) = k./(2*kw1); m = min(n,N); % derivative w.r.t. W dE = zeros(N,1); dE(ie(1:m)) = W(1:m)./(N*2/n*kw1(1:m)); % deriative w.r.t. E % Approximate l = log(det(B))/2, where B = I + sqrt(W)*K*sqrt(W) and compute % the derivatives d l / d hyp w.r.t. covariance hyperparameters and % d l / d W the gradient w.r.t. the precision matrix W. % % Large-scale Log-det Computation through Stochastic Chebyshev Expansions % Insu Han, Dmitry Malioutov, Jinwoo Shin, ICML, 2015. % % Chebyshev polynomials T[0],..,T[d], where T[0](x)=1, T[1](x)=x, and % T[i+1](x)=2*x*T[i](x)-T[i-1](x). % dT[0](x)=0, dT[1](x)=1, dT[i+1](x)=2*T[i](x)+2*x*dT[i](x)-dT[i-1](x) % % W is the precision matrix % K(z) yields mvm K*z % dK(y,z) yields d y'*K*z / d theta % % Copyright (c) by Insu Han and Hannes Nickisch 2016-09-27. function [ldB2,emax,dhyp,dW] = logdet_sample(W,K,dK, m,d, maxit,emax,seed) sW = sqrt(W); n = numel(W); emin = 1; % size and min eigenvalue bound B = @(x) x + bsxfun(@times,sW, K(bsxfun(@times,sW,x) ));%B=I+sqrt(W)*K*sqrt(W) if nargin<6, maxit = 50; end if nargin<7 || numel(emax)~=1 % evaluate upper eigenvalue bound if n==1, emax = B(1); else opt.maxit = maxit; % number of iterations to estimate the largest eigval opt.issym = 1; opt.isreal = 1; % K is real symmetric and - of course - psd cstr = 'eigs(B,n,1,''lm'',opt)'; % compute largest eigenvalue if exist('evalc'), [txt,emax] = evalc(cstr); else emax = eval(cstr); end end end if nargin<5, d = 15; end if nargin<4, m = 10; end d = round(abs(d)); if d==0, error('We require d>0.'), end a = emin+emax; del = 1-emax/a; ldB2 = n*log(a)/2;% scale eig(B) to [del,1-del] if emax>1e5 fprintf('B has large condition number %1.2e\n',emax) fprintf('log(det(B))/2 is most likely overestimated\n') end sf = 2/a/(1-2*del); B = @(x) sf*B(x) - 1/(1-2*del)*x;% apply scaling transform xk = cos(pi*((0:d)'+0.5)/(d+1)); % zeros of Tn(x) fk = log(((1-2*del)/2).*xk+0.5); % target function, [-1,1]->[del,1-del] Tk = ones(d+1,d+1); Tk(:,2) = xk; % init recursion for i=2:d, Tk(:,i+1) = 2*xk.*Tk(:,i) - Tk(:,i-1); end % evaluate polynomials c = 2/(d+1)*(Tk'*fk); c(1) = c(1)/2; % compute Chebyshev coefficients if nargin>7 && numel(seed)>0, randn('seed',seed), end % use seed V = sign(randn(n,m)); dhyp = 0; dW = 0; % bulk draw Rademacher variables, init p1 = [1; zeros(d,1)]; p2 = [0;1;zeros(d-1,1)]; % Chebyshev->usual coefficients p = c(1)*p1 + c(2)*p2; for i=2:d, p3 = [0;2*p2(1:d)]-p1; p = p + c(i+1)*p3; p1 = p2; p2 = p3; end if nargout<3 % use bulk mvms with B, one for each j=1..d U = p(1)*V; BjV = V; for j=1:d, BjV = B(BjV); U = U + p(j+1)*BjV; end ldB2 = ldB2 + sum(sum(V.*U))/(2*m); else % deal with one Rademacher vector at a time Bv = zeros(n,d+1); % all powers Bv(:,j) = (B^j)*v for i=1:m v = V(:,i); Bv(:,1) = v; for j=1:d, Bv(:,j+1) = B(Bv(:,j)); end ldB2 = ldB2 + (v'*Bv*p)/(2*m); sWBv = bsxfun(@times,sW,Bv); for j=1:d % p(1)*I + p(2)*B + p(3)*B^2 + .. + p(d+1)*B^d for k=1:ceil(j/2) akj = sf*p(j+1)/m; if k==ceil(j/2) && mod(j,2)==1, akj = akj/2; end dhyp = dhyp + akj * dK(sWBv(:,j-k+1),sWBv(:,k)); end end if nargout>3, u = bsxfun(@times,1./sW,Bv); w = K(sWBv); % precompute for j=1:d dWj = sum( u(:,j:-1:1).*w(:,1:j) + w(:,j:-1:1).*u(:,1:j), 2 ); dW = dW + sf*p(j+1)/(4*m) * dWj; end end end endgpml-matlab-v4.0-2016-10-19/cov/covPER.m000644 000765 000024 00000005270 12757566731 017211 0ustar00carlstaff000000 000000 function [K,dK] = covPER(mode, cov, hyp, x, z) % Periodic covariance function from an arbitrary covariance function k0 via % embedding IR^D into IC^D. % The covariance function is parameterized as: % % k(x,z) = k0(u(x),u(z)), u(x) = [sin(pi*x/p); cos(pi*x/p)] % % where the period p belongs to covPER and hyp0 belongs to k0: % % hyp = [ hyp_p % hyp0 ] % % We offer three different modes: % 'eye': p = ones(D,1); hyp_p = []; % 'iso': p = exp(hyp_p)*ones(D,1); hyp_p = [log(p)]; % 'ard': p = exp(hyp_p); hyp_p = [log(p_1); ..; log(p_D)]; % % Note that for k0 = covSEiso and D = 1, a faster alternative is covPeriodic. % % Copyright (c) by Hannes Nickisch, 2016-04-25. % % See also COVFUNCTIONS.M. if nargin<2, error('We require a mode and a base covariance k0.'), end if isequal(mode,'ard'), ne = 'D'; elseif isequal(mode,'iso'), ne = '1'; elseif isequal(mode,'eye'), ne = '0'; else error('Parameter mode is either ''eye'', ''iso'' or ''ard''.'), end if nargin<4 % report number of parameters K = ['(',ne,'+',strrep(feval(cov{:}),'D','2*D'),')']; return end if nargin<5, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); ne = eval(ne); p = exp(hyp(1:ne)); if numel(p)==0, p = 1; end, p = p.*ones(D,1); % period par ux = u(x,p); uz = u(z,p); % apply the embedding u:IR^D->IR^2*D [K,dK0] = feval(cov{:},hyp(ne+1:end),ux,uz); % covariance function if nargout > 1 % directional derivative dK = @(Q) dirder(Q,dK0,p,mode,dg,xeqz,x,ux,z,uz,cov,hyp(ne+1:end)); end function [dhyp,dx] = dirder(Q,dK0,p,mode,dg,xeqz,x,ux,z,uz,cov,hypc) if dg % dx not required for dg so we assume zero dhyp = dK0(Q); dux = zeros(size(ux)); % only correct for stationary else [dhyp,dux] = dK0(Q); end if isequal(mode,'eye') dp = zeros(0,1); if nargout > 1, dx = duxdx(dux,ux,p); end else dx = duxdx(dux,ux,p); dp = -sum(dx.*x,1); if ~xeqz && ~dg [junk,dK0t] = feval(cov{:},hypc,uz,ux); [junk,duz] = dK0t(Q'); dp = dp - sum(duxdx(duz,uz,p).*z,1); end if isequal(mode,'iso'), dp = sum(dp); end end dhyp = [dp(:); dhyp]; function ux = u(x,p) % apply the embedding u:IR^D->IR^2*D if numel(x)==0 || ischar(x) ux = x; else ux = 2*pi*bsxfun(@times,x,1./p(:)'); ux = [sin(ux), cos(ux)]; end function dx = duxdx(du,ux,p) D = size(ux,2)/2; dx = bsxfun(@times, du(:,1:D).*ux(:,D+1:2*D), 2*pi./p(:)') ... -bsxfun(@times, du(:,D+1:2*D).*ux(:,1:D), 2*pi./p(:)');gpml-matlab-v4.0-2016-10-19/cov/covPeriodic.m000644 000765 000024 00000003452 12757566731 020321 0ustar00carlstaff000000 000000 function [K,dK] = covPeriodic(hyp, x, z) % Stationary covariance function for a smooth periodic function, with period p % in 1d (see covPERiso and covPERard for multivariate data): % % k(x,z) = sf^2 * exp( -2*sin^2( pi*(x-z)/p )/ell^2 ) % % where the hyperparameters are: % % hyp = [ log(ell) % log(p) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-24. % % See also COVFUNCTIONS.M. if nargin<2, K = '3'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); if D>1, error('Covariance is defined for 1d data only.'), end ell = exp(hyp(1)); p = exp(hyp(2)); sf2 = exp(2*hyp(3)); % extract hyperparams % precompute deviations and exploit symmetry of sin^2 if dg % vector txx T = zeros(size(x,1),1); else if xeqz % symmetric matrix Txx T = pi/p*bsxfun(@plus,x,-x'); else % cross covariances Txz T = pi/p*bsxfun(@plus,x,-z'); end end S2 = (sin(T)/ell).^2; K = sf2*exp( -2*S2 ); % covariances if nargout>1 dK = @(Q) dirder(Q,K,S2,T,ell,xeqz,x,z,p); % directional hyper derivative end function [dhyp,dx] = dirder(Q,K,S2,T,ell,xeqz,x,z,p) Q = K.*Q; P = sin(2*T).*Q; dhyp = [4*(S2(:)'*Q(:)); 2/ell^2*(P(:)'*T(:)); 2*sum(Q(:))]; if nargout > 1 R = P./T; R(T==0) = 0; q2 = sum(R,2); q1 = sum(R,1)'; if xeqz y = bsxfun(@times,q1+q2,x) - (R+R')*x; else Rz = R*z; y = bsxfun(@times,q2,x) - Rz; end dx = -2*pi^2/(ell*p)^2 * y; endgpml-matlab-v4.0-2016-10-19/cov/covNNone.m000644 000765 000024 00000004205 12757566731 017575 0ustar00carlstaff000000 000000 function [K,dK] = covNNone(hyp, x, z) % Neural network covariance function with a single parameter for the distance % measure. The covariance function is parameterized as: % % k(x,z) = sf2 * asin(x'*P*z / sqrt[(1+x'*P*x)*(1+z'*P*z)]) % % where the x and z vectors on the right hand side have an added extra bias % entry with unit value. P is ell^-2 times the unit matrix and sf2 controls the % signal variance. The hyperparameters are: % % hyp = [ log(ell) % log(sqrt(sf2) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-23. % % See also COVFUNCTIONS.M. if nargin<2, K = '2'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode ell2 = exp(2*hyp(1)); sf2 = exp(2*hyp(2)); sx = 1 + sum(x.*x,2); if dg % vector kxx A = sx./(sx+ell2); sz = sx; else if xeqz % symmetric matrix Kxx S = 1 + x*x'; sz = sx; A = S./(sqrt(ell2+sx)*sqrt(ell2+sx)'); else % cross covariances Kxz S = 1 + x*z'; sz = 1 + sum(z.*z,2); A = S./(sqrt(ell2+sx)*sqrt(ell2+sz)'); end end K = sf2*asin(A); % covariances if nargout > 1 dK = @(Q) dirder(Q,K,A,sx,sz,ell2,sf2,x,z,dg,xeqz); % dir hyper derivative end function [dhyp,dx] = dirder(Q,K,A,sx,sz,ell2,sf2,x,z,dg,xeqz) n = size(x,1); if dg V = A; else vx = sx./(ell2+sx); if xeqz V = repmat(vx/2,1,n) + repmat(vx'/2,n,1); else vz = sz./(ell2+sz); nz = size(z,1); V = repmat(vx/2,1,nz) + repmat(vz'/2,n,1); end end P = Q./sqrt(1-A.*A); dhyp = [-2*sf2*sum(sum((A-A.*V).*P)); 2*Q(:)'*K(:)]; if nargout > 1 if dg dx = zeros(size(x)); else W = P./(sqrt(ell2+sx)*sqrt(ell2+sz)'); ssx = sqrt(ell2+sx); if xeqz, W = W+W'; z = x; ssz = ssx; else ssz = sqrt(ell2+sz); end dx = sf2*(W*z - bsxfun(@times,x,((W.*A)*ssz)./ssx)); end endgpml-matlab-v4.0-2016-10-19/cov/covEye.m000644 000765 000024 00000002742 12757566731 017306 0ustar00carlstaff000000 000000 function [K,dK] = covEye(hyp, x, z) % Independent covariance function, i.e. "white noise", with unit variance. % The covariance function is specified as: % % k(x^p,x^q) = \delta(p,q) % % \delta(p,q) is a Kronecker delta function which is 1 iff p=q and zero % otherwise in mode 1). % In cross covariance mode 2) two data points x_p and z_q are considered equal % if their difference norm |x_p-z_q| is less than eps, the machine precision. % The hyperparameters are: % % hyp = [ ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Hannes Nickisch, 2016-04-18. % % See also COVFUNCTIONS.M. tol = eps; % threshold on the norm when two vectors are considered to be equal if nargin<2, K = '0'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists dg = strcmp(z,'diag'); % determine mode n = size(x,1); if dg % vector kxx K = ones(n,1); else if isempty(z) % symmetric matrix Kxx K = eye(n); else % cross covariances Kxz K = double(sq_dist(x',z') 1 dK = @(Q) dirder(Q,x); % directional hyper derivative end function [dhyp,dx] = dirder(Q,x) dhyp = zeros(0,1); if nargout > 1, dx = zeros(size(x)); endgpml-matlab-v4.0-2016-10-19/cov/covRQard.m000644 000765 000024 00000002447 12777177052 017573 0ustar00carlstaff000000 000000 function varargout = covRQard(varargin) % Wrapper for Rational Quadratic covariance function covRQ.m. % % Rational Quadratic covariance function with Automatic Relevance Determination % (ARD) distance measure. The covariance function is parameterized as: % % k(x,z) = sf^2 * [1 + (x-z)'*inv(P)*(x-z)/(2*alpha)]^(-alpha) % % where the P matrix is diagonal with ARD parameters ell_1^2,...,ell_D^2, where % D is the dimension of the input space, sf2 is the signal variance and alpha % is the shape parameter for the RQ covariance. The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) % log(sf) % log(alpha) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-01. % % See also covRQ.M. varargout = cell(max(1,nargout),1); if nargin>0 % restore old hyper parameter order hyp = varargin{1}; if numel(hyp)>2, varargin{1} = hyp([1:end-2,end,end-1]); end end [varargout{:}] = covScale({'covRQ','ard',[]},varargin{:}); if nargout>1 % restore old hyper parameter order o2 = varargout{2}; varargout{2} = @(Q) dirder(Q,o2); end function [dKdhyp,dKdx] = dirder(Q,dK) if nargout>1, [dKdhyp,dKdx] = dK(Q); else dKdhyp = dK(Q); end dKdhyp = dKdhyp([1:end-2,end,end-1]);gpml-matlab-v4.0-2016-10-19/cov/covSum.m000644 000765 000024 00000005073 12757566731 017330 0ustar00carlstaff000000 000000 function [K,dK] = covSum(cov, hyp, x, z) % covSum - compose a covariance function as the sum of other covariance % functions. This function doesn't actually compute very much on its own, it % merely does some bookkeeping, and calls other covariance functions to do the % actual work. % % Note that cov = {cov1, cov2, .., false} turns of covariance matrix caching. % This option slows down the computations but can help out if you sums of % many huge matrices lead to working memory shortage. % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2016-04-18. % % See also COVFUNCTIONS.M. if isempty(cov), error('We require at least one summand.'), end if isnumeric(cov{end}) || islogical(cov{end}) % detect whether to cache cache = ~isequal(0,cov{end}) && ~isequal(false,cov{end}); cov = cov(1:end-1); % chop off last element from cov else cache = true; end nc = numel(cov); % number of terms in covariance function for ii = 1:nc % iterate over covariance functions f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary j(ii) = cellstr(feval(f{:})); % collect number hypers end if nargin<3 % report number of parameters K = char(j(1)); for ii=2:nc, K = [K, '+', char(j(ii))]; end, return end if nargin<4, z = []; end % make sure, z exists [n,D] = size(x); v = []; % v vector indicates to which covariance parameters belong for ii = 1:nc, v = [v repmat(ii, 1, eval(char(j(ii))))]; end K = 0; dKi = cell(nc,1); % init K for ii = 1:nc % iteration over summand functions f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary if nargout>1 && cache [Kii,dKi{ii}] = feval(f{:}, hyp(v==ii), x, z); % keep track else Kii = feval(f{:}, hyp(v==ii), x, z); end K = K + Kii; % accumulate covariances end dK = @(Q) dirder(Q,dKi,v,nc,cov,hyp,x,z,cache); % directional derivative function [dhyp,dx] = dirder(Q,dKi,v,nc,cov,hyp,x,z,cache) dhyp = zeros(size(v,2),1); dx = 0; for ii = 1:nc if cache dKii = dKi{ii}; else f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell if necessary [junk,dKii] = feval(f{:}, hyp(v==ii), x, z); end if nargout > 1 [dhyp(v==ii,1),dxi] = dKii(Q); dx = dx+dxi; else dhyp(v==ii,1) = dKii(Q); end endgpml-matlab-v4.0-2016-10-19/cov/apxGrid.m000644 000765 000024 00000105032 12777177047 017446 0ustar00carlstaff000000 000000 function [K,Mx,xe] = apxGrid(cov, xg, hyp, x, z, b) % apxGrid - Covariance function approximation based on an inducing point grid. % % A grid covariance function k(x,z) is composed as a product % k(x,z) = k1(x(i1),z(i1)) * .. * kp(x(ip),z(ip)) of p covariance functions % operating on mutually disjoint components of the data x,z. % The resulting covariance matrix is given by the Kronecker product % K = kron(kron(kron(...,K3),K2),K1) = K_p x .. x K_2 x K_1. % % The function is designed to be used with infGaussLik and infLaplace. % % The covariance function grid contains p grid factors xg = {x1,..,xp} % so that for each of the factors xi a covariance function ki can be defined. % A factor xi is of size (ni,di) so that the Kronecker grid g has size % (n1,n2,..,np,D), where D=d1+d2+..+dp. Hence, the grid g contains N=n1*n2*..*np % data points overall. Note that the factors xi do neither need to % be sorted nor do they need to be 1d a priori. If a factor xi contains unevenly % spaced values, we require di=1. % % A factor xi can contain: % a) a univariate axis xi of size (ni,1), % b) a multivariate axis xi of size (ni,di), % where the values need to be equispaced, or % c) a stationary equispaced subgrid composed of di univariate equispaced axes % {xi_1,xi_2,xi_di}, where each axis xij is of size (nij,1) so that % ni=ni_1*..*ni_di intended to be used with stationary covariance functions % When a stationary equispaced subgrid is specified, we silently assume % the covariance function to be stationary. % % For fast computations, we exploit two kinds of structure: % 1) The Kronecker structure of the covariance matrix induced BY the p factors. % Hence, for p=1, there is nothing to gain here. % 2) The Toeplitz or BTTB (Block-Toeplitz with Toeplitz Blocks) WITHIN a factor % if a grid factor xi is equispaced (or multivariate and equispaced). % Note that empirically, only for matrices of sizes above 500x500, the % FFT-based MVMs are faster than dense matrix operations. % % Some examples with sizes and domain: % - a single factor with a univariate axis, case a) % xg = { 5*rand(150,1) } => N = 150, D = 1, dom = [0,5] % xg = { linspace(0,3,400)' } => N = 400, D = 1, dom = [0,3] % - a single factor with a multivariate axis (equispaced is mandatory), case b) % xg = { [linspace(0,3,100)',... % linspace(1,4,100)'] } => N = 100, D = 2, dom = [0,3]x[1,4] % - a single factor with a univariate equispaced&stationary subgrid, case c) % where we assume a stationary covariance and exploit Toeplitz structure % xg = { {linspace(0,10,175)'} } => N = 175, D = 1, dom = [0,10] % - a single factor with a bivariate equispaced&stationary subgrid, case c) % where we assume a stationary covariance and exploit BTTB structure % xg = { {linspace(0,3,20)',... % linspace(1,7,30)'} } => N = 600, D = 2, dom = [0,3]x[1,7] % - a two-factor grid of 2 univariate axes, case a) % xg = { 2*rand(40,1), 5*rand(20,1) } => N = 800, D = 2, dom = [0,2]x[0,5] % - a two-factor grid of 2 univariate equispaced and stationary axes, case c) % where we assume two stationary covariances and exploit Toeplitz structure % in both factors % xg = { {linspace(0,2,25)'}, ... % {linspace(1,3,25)'} } => N = 625, D = 2, dom = [0,2]x[1,3] % - a four-factor grid with a Toeplitz factor, two ordinary Kronecker % factors and a 2d BTTB factor % xg = { {linspace(0,1,50)'}, ... => N = 4e6, D = 5, dom = [0,1]^5 % rand(20,1), ... % linspace(0,1,40)', ... % {linspace(0,1,10)',linspace(0,1,10)'} } % % The apxGrid function can be used to expand the (nested) cell array xg into a % multivariate grid xe of size (N,D) via: % [xe,nx,Dx] = apxGrid('expand',xg); => mode 1) % The operation can be reverted (if no subgrids are used) by: % xg = apxGrid('factor',{xe,ng,Dg}); => mode 2) % % Given scattered data x of size (n,D), we can create a grid xg covering % the support of x using: % xg = apxGrid('create',x,eq,k); => mode 3) % The flag eq (default value 1) can be used to enforce an equispaced % grid. The integer k scalar or vector, indicates the number of grid points per % dimension. If k is a real number from (0,1], then the number of grid points % equals k*numel(unique(x(:,1))). % We require at least two different components per dimension. % % The variables v={x,z} can either be a) grid indices or b) data points. % a) The variable v has size (nv,1) and contains integers from [1,N]. Then % the datapoints are obtained as g2 = reshape(g,N,D); v = g2(v,:). % b) The variable v has size (nv,D) and directly represents the data points. % The mechanism works for x and z separately. % % Given a grid xg, the grid size can be obtained by: % [ng,Dg] = apxGrid('size',xg); => mode 4) % % An arbitrary data point x -- be it an index vector of size (n,1) or a data % point of size (n,D) -- is converted into a regular data point xx of % size (n,D) by: % [xx,ng,Dg] = apxGrid('idx2dat',xg,x); => mode 5) % If x is already of size (n,D), xx will simply equal x. % % Given a grid xg and given arbitrary data points x, the interpolation % matrix Mx can directly be computed without computing cross-covariances via: % [Mx,dMx] = apxGrid('interp',xg,x,deg); => mode 6) % For equispaced grids, deg can be used to set the degree of the interpolation % polynomial in all p axes. Here deg=0 means nearest neighbor, deg=1 means % linear interpolation, and deg=3 uses a cubic. % The cell array dMx contains the derivatives d Mx / d xi with respect to % the i=1..p grid components. % % Given a nested grid xg, we can compute a flattened grid xf by: % xf = apxGrid('flatten',xg); => mode 7) % without nesting and containing p axis elements. % % Given a covariance K, and a grid xg, and an integer embedding factor s, % we can compute the Kronecker eigen decomposition such that % K = V*diag(e)*V', where e = kron(ee,..). % [V,ee,e] = apxGrid('eigkron',K,xg,s); => mode 8) % Note that this holds only approximately for Toeplitz/BTTB due to the % circulant embedding. For details about s, see mode 9). % % Given a covariance function k where the call k(1:n) returns a vector of % length, an interger length n and an integer embedding factor s, we % construct a circulant embedding c (nx1). % [c,xg] = apxGrid('circ',k,n,s); => mode 9) % Here s is allowed to have the following values: % s=0 No embedding c = k. % s>0 Whittle embedding [1] as described by Guinness & Fuentes [2] in % equation (5) with N = |s|. % [1] Whittle, On stationary processes in the plane, Biometrika, 1954. % [2] Guinness & Fuentes, Circulant embedding of approximate covariances for % inference from Gaussian data on large lattices, 2014. % % Given a grid covariance K, a grid xg, an interpolation matrix Mx and % two vectors a and b, we can compute the directional derivative % d a'*Mx*K*Mx'*b / d hyp: % dhyp = apxGrid('dirder',K,xg,Mx,a,b); => mode 10) % % Multiplication with a matrix/operator A along dimension dim of the tensor B. % Note that tmul(A,B,1) = A*B if B is a matrix. % C = apxGrid('tmul',A,B,dim); => mode 11) % % Test grid for being equispaced along an axis i % eq = apxGrid('equi',xg,i); => mode 12) % % Return a descriptive string about the nature of Kg and Mx. % s = apxGrid('info',Kg,Mx,xg,deg); => mode 13) % % The hyperparameters are: % hyp = [ hyp_1 % hyp_2 % .. % hyp_p ], % % Copyright (c) by Hannes Nickisch and Andrew Wilson 2016-10-06. % % See also COVFUNCTIONS.M, APX.M, INFLAPLACE.M, INFGAUSSLIK.M. if nargin<2, error('Not enough parameters provided.'), end dense_max = 0; % 500 if larger grid we use FFT-algebra rather than dense algebra % mode 1) expand axes xg representation into full grid x if strcmp(cov,'expand') % call: [xe,nx,Dx] = apxGrid('expand',xg); [K,Mx,xe] = expandgrid(xg); return % mode 2) factor full x grid into axes representation xg elseif strcmp(cov,'factor') % call: xg = apxGrid('factor',{xe,ng,Dg}); K = factorgrid(xg{:}); return % mode 3) create axes representation xg from scattered data elseif strcmp(cov,'create') % call: xg = apxGrid('create',x,eq,k); if nargin<3, eq = 1; else eq = hyp; end % set default values if nargin<4, k = 1; else k = x; end, x = xg; % set input params K = creategrid(x,eq,k); return % mode 4) convert possible index vector into data space elseif strcmp(cov,'size') % call: [ng,Dg] = apxGrid('size',xg); [ng,Dg] = sizegrid(xg); K = ng; Mx = Dg; return % mode 5) convert possible index vector into data space elseif strcmp(cov,'idx2dat') % call: [xx,ng,Dg] = apxGrid('idx2dat',xg,x); [ng,Dg] = sizegrid(xg); N = prod(ng); if isidx(hyp,N), xe = expandgrid(xg); K = xe(hyp,:); else K = hyp; end Mx = ng; xe = Dg; return % mode 6) compute interpolation matrix elseif strcmp(cov,'interp') % call [Mx,dMx] = apxGrid('interp',xg,x,deg); if nargout>1, [K,Mx]=interpgrid(xg,hyp,x); else K=interpgrid(xg,hyp,x); end return % mode 7) provide flattened interpolation grid without nesting elseif strcmp(cov,'flatten') % call xf = apxGrid('flatten',xg); K = flattengrid(xg); return % mode 8) compute eigen-decomposition of Kronecker matrix elseif strcmp(cov,'eigkron') % call [V,ee,e] = apxGrid('eigkron',K,xg,s); [K,Mx,xe] = eigkron(xg,hyp,x); return % mode 9) compute a circulant embedding elseif strcmp(cov,'circ') % call [c,xg] = apxGrid('circ',k,n,s); [K,Mx] = circ(xg,hyp,x); return % mode 10) compute directional derivatives elseif strcmp(cov,'dirder') % call dhyp = apxGrid('dirder',K,xg,Mx,a,b); K = dirder(xg,hyp,x,z,b); return % mode 11) matrix multiplication along a tensor elseif strcmp(cov,'tmul') % call C = apxGrid('tmul',A,B,dim); K = tmul(xg,hyp,x); return % mode 12) test grid for being equispaced elseif strcmp(cov,'equi') % call eq = apxGrid('equi',xg,i); K = equi(xg,hyp); return % mode 13) report a descriptive status elseif strcmp(cov,'info') % call s = apxGrid('info',Kg,Mx,xg,deg); K = info(xg,hyp,x,z); return end % mode 0) regular covariance function computations p = numel(xg); [ng,Dg] = sizegrid(xg); % number of Kronecker factors if numel(cov)~=p, error('We require p factors.'), end for ii = 1:p % iterate over covariance functions f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary D = Dg(ii); j(ii) = cellstr(num2str(eval(feval(f{:})))); % collect nbr hypers end if nargin<4 % report number of parameters K = char(j(1)); for ii=2:length(cov), K = [K, '+', char(j(ii))]; end, return end if nargin<5, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode v = []; % v vector indicates to which covariance parameters belong for ii = 1:p, v = [v repmat(ii, 1, eval(char(j(ii))))]; end N = prod(ng); n = size(x,1); D = sum(Dg); % expanded grid and data dimension ix = isidx(x,N); % determine whether x is an index or a data array if ~ix && size(x,2)~=D, error('Grid and data dimension are different.'), end if nargout>1 || ~(dg||xeqz||ix) % off-grid interpolation if nargin>5, Mx = interpgrid(xg,x,b); else Mx = interpgrid(xg,x); end end if dg % evaluate as full dense vector for diagonal covariance case K = 1; % xg is not assumed to form a grid for z = 'diag' for ii = 1:length(cov) % iteration over factor functions f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary d = sum(Dg(1:ii-1))+(1:Dg(ii)); % dimensions of interest if ix, xii = xg{ii}; else xii = x(:,d); end % switch Kronecker/plain prod Kj = feval(f{:}, hyp(v==ii), xii, z); % plain Kronecker factor if ix, K = kron(K,Kj); else K = K.*Kj; end % switch Kronecker/plain prod end if ix, K = K(x); end, return end if isidx(z,N), iz = z; z = apxGrid('expand',xg); z = z(iz,:); end % expand z K = cell(p,1); dK = cell(p,1); sz = [1,1]; % cov Kronecker factors total size for ii = 1:p % iteration over factor functions f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary d = sum(Dg(1:ii-1))+(1:Dg(ii)); % dimensions of interest if isnumeric(z) && ~isempty(z) % cross terms zd = z(:,d); elseif xeqz && iscell(xg{ii}) % Toeplitz/BTTB zd = []; else % symmetric matrix zd = z; end xii = xg{ii}; % grid factor if xeqz && iscell(xg{ii}) % Toeplitz/BTTB di = numel(xii); ni = sizegrid(xii); wi = zeros(di,1); for j=1:di, wi(j) = xii{j}(end)-xii{j}(1); end % width eqstat = true; for j=1:di, eqstat = eqstat & equi(xii,j); end if ~eqstat, error('Subgrid not equispaced.'), end % stop if problem if prod(ni)>dense_max % empirical thresh: FFT-based MVM with 1 rhs faster kii = @(n) feval(f{:},hyp(v==ii),(n-1)*diag(wi./(ni-1)), zeros(1,di)); xc = cell(di,1); % generic (Strang) circular embedding grid, kii = k(int) for j=1:di, n2 = floor(ni(j)-1/2)+1; xc{j} = [1:n2,n2-2*ni(j)+2:0]'; end ci = kii(apxGrid('expand',xc)); ci = reshape(ci,[2*ni(:)'-1,1]); dki = []; fi = real(fftn(ci)); % precompute FFT for circular filter mvmKi = @(x) bttbmvmsymfft(fi,x); % MVM with Toeplitz/BTTB matrix if di==1, s = 'toep'; else s = ['bttb',num2str(di)]; end ki = struct('descr',s, 'mvm',mvmKi, 'kii',kii, 'size',prod(ni)*[1,1]); else % simply evaluate covariance matrix if prod(ni) too small [ki,dki] = feval(f{:},hyp(v==ii),expandgrid(xii)); end K{ii} = ki; dK{ii} = dki; sz = sz.*K{ii}.size; else if iscell(xii), xii = expandgrid(xii); end [K{ii},dK{ii}] = feval(f{:}, hyp(v==ii), xii, zd); % plain Kronecker factor sz = sz.*size(K{ii}); end end if xeqz % create mvm and rest K = struct('mvm',@(a)kronmvm(K,a),'size',sz,'kronmvm',@kronmvm,... 'kron',struct('factor',K,'dfactor',dK)); else % expand cross terms Ks = K; K = Ks{1}; for ii = 2:p, K = kron1(Ks{ii},K); end if ix, if numel(x)~=N || max(abs(x-(1:N)'))>0, K = K(x,:); end else K = Mx*K; end end if nargout>2, xe = apxGrid('expand',xg); end % Perform a matrix vector multiplication b = A*x with a matrix A being a % Kronecker product given by A = kron( kron(...,As{2}), As{1} ). function b = kronmvm(As,x,transp) if nargin>2 && ~isempty(transp) && transp % transposition by transposing parts for i=1:numel(As) if isnumeric(As{i}) As{i} = As{i}'; else As{i}.mvm = As{i}.mvmt; As{i}.size = [As{i}.size(2),As{i}.size(1)]; end end end m = zeros(numel(As),1); n = zeros(numel(As),1); % extract sizes for i=1:numel(n) if isnumeric(As{i}) [m(i),n(i)] = size(As{i}); else m(i) = As{i}.size(1); n(i) = As{i}.size(2); end end d = size(x,2); b = x; for i=1:numel(n) % apply As{i} to the 2nd dimension sa = [prod(m(1:i-1)), n(i), prod(n(i+1:end))*d]; % size a = reshape(permute(reshape(full(b),sa),[2,1,3]),n(i),[]); if isnumeric(As{i}), b = As{i}*a; else b = As{i}.mvm(a); end % do batch MVM b = permute(reshape(b,m(i),sa(1),sa(3)),[2,1,3]); end b = reshape(b,prod(m),d); % bring result in correct shape % Perform MVM b = T*a with a of size (n,m) with a BTTB (Block-Toeplitz with % Toeplitz-blocks) matrix T of size (n,n) by pointwise multiplication with the % Fourier-transformed filter f. All variables are assumed real valued. % Needs O(3*m*n*log(n)) time and O(n*m) space. function b = bttbmvmsymfft(f,a) ng = (size(f)+1)/2; p = numel(ng); Ng = prod(ng); % extract sizes if p==2 && ng(2)==1, p = 1; ng = ng(1); end % detect 1d and reduce p m = numel(a)/Ng; b = reshape(a,[ng,m]); for i=1:p, b = fft(b,2*ng(i)-1,i); end % emulate fftn with new shape b = bsxfun(@times,f,b); % pointwise multiplication for i=1:p, b = ifft(b,[],i); end % emulate ifft for i=1:p % only keep the relevant part of the result b = reshape(b,prod(ng(1:i-1)),2*ng(i)-1,prod(2*ng(i+1:p)-1)*m); b = b(:,1:ng(i),:); end b = real(reshape(b,[],m)); % perform kron along first dimension only % the code is equivalent to the following loop % z = zeros(size(x,1)*size(y,1),size(x,2)); % for i=1:size(z,2), z(:,i) = kron(x(:,i),y(:,i)); end function z = kron1(x,y) nx = size(x,1); ny = size(y,1); z = repmat(reshape(x,1,nx,[]),[ny,1,1]).*repmat(reshape(y,ny,1,[]),[1,nx,1]); z = reshape(z,nx*ny,[]); function r = isidx(i,N) % check whether i represents an integer index vector r = false; if numel(i)>0 && ~strcmp(i,'diag') && size(i,2)==1 && ndims(i)==2 if max(abs(i-floor(i)))<1e-13 if 00, k = ones(p,1).*k(:); end % enforce vector-valued k for j=1:p % iterate over dimensions u = sort(unique(x(:,j))); if numel(u)<2, error('Two few unique points.'),end if isempty(k) % determine number of grid points if eq ngj = ceil( (u(end)-u(1))/min(abs(diff(u))) ); % use minimum spacing else ngj = numel(u); end elseif 0<=k(j) && k(j)<=1 ngj = ceil(k(j)*numel(u)); else ngj = k(j); end du = (u(end)-u(1))/ngj; bu = [u(1)-5*du, u(end)+5*du]; if eq % equispaced grid xg{j} = linspace(bu(1),bu(2),max(ngj,5))'; % at least 5 grid points else % non-equispaced grid [idx,xgj] = kmeans(u,min(numel(u),ngj-2)); xgj = sort(xgj(:))'; % cluster nb = ngj-numel(xgj); nb1 = floor(nb/2); nb2 = nb - nb1; % size of boundary xg1 = linspace(bu(1),xgj(1),nb1+1); xg2 = linspace(xgj(end),bu(2),nb2+1); xg{j} = [xg1(1:nb1), xgj, xg2(1+(1:nb2))]'; end end % mode 4 (size) function [ng,Dg] = sizegrid(xg) % report the size of the p grid factors if ~iscell(xg), [ng,Dg] = size(xg); return, end % catch trivial case p = numel(xg); ng = zeros(p,1); Dg = zeros(p,1); % number of grid factors for i=1:p % iterate over grid factors x = xg{i}; if iscell(x) % stationary and equispace grid for j=1:numel(x) if j==1 [ng(i),Dg(i)] = size(x{j}); else ng(i) = ng(i)*size(x{j},1); Dg(i) = Dg(i)+size(x{j},2); end end else % arbitrary grid [ng(i),Dg(i)] = size(x); end end % mode 6 (interp) % deg, degree of equispaced interpolation polynomial, 0:nn, 1:lin, 3:cub function [Mx,dMx] = interpgrid(xg,x,deg) xg = flattengrid(xg); % remove nestedness p = numel(xg); Dg = zeros(p,1); ng = zeros(p,1); n = size(x,1); % dims .. for i=1:p, [ng(i),Dg(i)] = size(xg{i}); end, N = prod(ng); %.. and sizes ix = isidx(x,N); % determine whether x is an index or a data array if ix Mx = sparse(1:n,x,1,n,N); if nargout>1, dMx = repmat({sparse(n,N)},p,1); end else if nargin<3, deg = 3; end, deg = deg(:).*ones(p,1); % cubic is default s = 1; % initial stride for i=1:p % iterate over Toeplitz components d = sum(Dg(1:i-1))+(1:Dg(i)); % dimensions of interest xt = xg{i}; it = find(abs(xt(2,:)-xt(1,:))); % grid nonzero inc idx if equi(xg,i) % compute interpolation coefficients if nargout>1 % equispaced grid pts [Ji,Ci,dCi] = eqinterp(xt(:,it),x(:,d(it)),deg(i)); else [Ji,Ci] = eqinterp(xt(:,it),x(:,d(it)),deg(i)); end else if nargout>1 % non-equispaced grid pts, lin interp, inv dist weighting [Ji,Ci,dCi] = neqinterp(xt(:,it),x(:,d(it))); else [Ji,Ci] = neqinterp(xt(:,it),x(:,d(it))); end end nc = size(Ci,2); % number of interpolation coefficients along dimension if i==1 C = Ci; J = ones(n,1); if nargout>1, dC = repmat({Ci},p,1); dC{1} = dCi; end else C = repmat(C,[1,1,nc]) .* repmat(reshape(Ci,n,1,nc),[1,size(C,2),1]); C = reshape(C,n,[]); if nargout>1 for j=1:p if i==j, dCij = dCi; else dCij = Ci; end dC{j} = repmat(dC{j},[1,1,nc]) .* ... repmat(reshape(dCij,n,1,nc),[1,size(dC{j},2),1]); dC{j} = reshape(dC{j},n,[]); end end end J = repmat(J(:),[1,nc]) + s*repmat(Ji-1,[size(C,2)/nc,1]); % blow 2nd idx s = s*ng(i); % update stride end I = repmat((1:n)',[1,size(C,2)]); id = 01 dMx = cell(p,1); for i=1:p,dMx{i} = sparse(I(id),J(id),dC{i}(id),n,N); end end end % Compute interpolation coefficients C (nt,nc) and interpolation coefficient % indices J (nt,nc) from a source grid s (ns,1) to a target array t (nt,1). % The coefficient matrix C has rows summing up to 1. function [J,C,dC] = eqinterp(s,t,d) gp = false; switch d case 0, k = @(x) -0.5%d.',nc-1), end j = floor((t-s(1))/ds)+1; % index of closest smaller grid point w = (t-s(1))/ds-j+1; % relative distance to closest smaller grid point [0,1] j = j-it(nc); C = zeros(nt,nc); dC = zeros(nt,nc); for i=1:nc C(:,i) = k(w+it(nc+1-i)); if nargout>2, dC(:,i)=dk(w+it(nc+1-i))*(ns-1); end end if gp, kn = k(sqrt(sq_dist(1:d))); C = C/kn; dC = dC/kn; end v = 1; id = find(jv); C(id,:) = 0; dC(id,:) = 0; % fix upper boundary D = abs(repmat(s(ns-nc+1:ns)',numel(id),1)-repmat(t(id),[1,nc])); [junk,jid] = min(D,[],2); % index of closest index in boundary region for i=1:numel(id), C(id(i),jid(i)) = 1; dC(id(i),jid(i)) = 0; end, j(id) = v; J = zeros(nt,nc); for i=1:nc, J(:,i) = j+i-1; end % construct index array J % Robert G. Keys, Cubic Convolution Interpolation for Digital Image Processing, % IEEE ASSP, 29:6, December 1981, p. 1153-1160. function y = kcub(x) y = zeros(size(x)); x = abs(x); q = x<=1; % Coefficients: 1.5, -2.5, 0, 1 y(q) = (( 1.5 * x(q) - 2.5) .* x(q) ) .* x(q) + 1; q = 12) = 0; % Perform piecewise linear interpolation using inverse distance weighting. % s (ns,1) source nodes, need neither be sorted nor equispaced % t (nt,1) target nodes % M (nt,ns) interpolation matrix, M = sparse((1:N)'*[1,1],J,C,nt,ns); % % z = M*y where y (ns,1) are source values and z (nt,1) are target values function [J,C,dC] = neqinterp(s,t) ns = size(s,1); nc = 2; % get dimensions if size(s,2)*size(t,2)~=1, error('Interpolation only possible for d==1.'), end if ns=nc.'), end [s,ord] = sort(s); ds = diff(s); if min(ds)<1e-10, error('Some source points are equal.'), end [junk,ii] = histc(t(:),[-inf;s(2:end-1);inf]); d0 = t(:)-s(ii); d1 = s(ii+1)-t(:); d0n = d0<0; d1n = d1<0; d0(d0n) = 0; d1(d1n) = 0; % boundary conditions J = [ord(ii),ord(ii+1)]; C = [d1./(d1+d0),d0./(d1+d0)]; nz = 1-(d1n|d0n); if nargout>2, dC = [-nz./(d1+d0),nz./(d1+d0)]; end % mode 7 (flatten) function xf = flattengrid(xg) % convert nested grid into flat grid if ~iscell(xg), xf = xg; return, end % catch trivial case xf = cell(1,0); for i=1:numel(xg) x = xg{i}; if iscell(x), xf = [xf,x]; else xf = [xf,{x}]; end end % mode 8 (eigkron) % Eigendecomposition of a Kronecker matrix K with dense, Toeplitz or BTTB % factors so that K = V*diag(e)*V', where e = kron(ee,..). Note that this holds % only approximately for Toeplitz/BTTB due to the circulant embedding. function [V,ee,e] = eigkron(K,xg,s) isbttb = @(Ki) isstruct(Ki) && (strcmp (Ki.descr,'toep') ... || strncmp(Ki.descr,'bttb',4)); % BTTB covariance p = numel(K.kron); V = cell(p,1); ee = cell(p,1); % sizes and allocate memory for j=1:p % compute eigenvalue diagonal matrix if isbttb(K.kron(j).factor) [xj,nj] = apxGrid('expand',xg{j}); % extract subgrid size ej = fftn(circ(K.kron(j).factor.kii,nj,s)); % circ embedded cov mat V{j}.mvm = @(v) Fmvm( v,nj); % V{j}' is Fourier matrix V{j}.mvmt = @(v) Fmvmt(v,nj); V{j}.size = numel(ej)*[1,1]; else Kj = K.kron(j).factor; Kj = (Kj+Kj')/2; % enforce symmetry [V{j},ej] = eig(Kj); % compute eigenvalues of non-Toeplitz matrix ej = diag(ej); V{j} = real(V{j}); % cosmetics end ee{j} = max(real(ej),0); % thresholding to ensure pd approximation end if nargout>2, e = 1; for j=1:p, e = kron(ee{j}(:),e); end, end function a = Fmvmt(b,nj) % fast Fourier transform transpose for multiple rhs Nj = prod(nj); sNj = sqrt(Nj); % scaling factor to make FFTN orthonormal nr = numel(b)/Nj; % number of right-hand-side arguments b = reshape(b,[nj(:)',nr]); for i=1:numel(nj), b = fft(b,[],i); end % emulate fftn a = reshape(b,Nj,[])/sNj; % perform rescaling function b = Fmvm(a,nj) % fast Fourier transform for multiple rhs Nj = prod(nj); sNj = sqrt(Nj); % scaling factor to make FFTN orthonormal nr = numel(a)/Nj; % number of right-hand-side arguments b = a; b = reshape(b,[nj(:)',nr]); % accumarray and target shape for i=1:numel(nj), b = ifft(b,[],i); end % emulate ifftn b = reshape(b,Nj,nr)*sNj; % perform rescaling % mode 9 (circ) % Construct a circular embedding c(nx1) from a covariance function k. % - k is a function and the call k(1:n) returns a vector of length n % - s is the setting for the embedding with values s={0,1,2,..}. % % s=0 No embedding c = k. % s>0 Whittle embedding [1] as described by Guinness & Fuentes [2] in % equation (5) with N = |s|. % % [1] Whittle, On stationary processes in the plane, Biometrika, 1954, 41(3/4). % [2] Guinness & Fuentes, Circulant embedding of approximate covariances for % inference from Gaussian data on large lattices, 2014, preprint, % http://www4.stat.ncsu.edu/~guinness/circembed.html. function [c,xg] = circ(k,n,s) p = numel(n); n = n(:)'; % dimensions and row vector if nargin<3, s = 2; end % default value if s==0 % no embedding at all xg = cell(p,1); for i=1:p, xg{i} = (1:n(i))'; end % standard grid c = reshape(k(apxGrid('expand',xg)),[n,1]); elseif s>0 % Whittle/Guinness aliasing xg = cell(p,1); for i=1:p, xg{i} = (1-s*n(i):s*n(i))'; end sz = [n; 2*s*ones(1,p)]; c = reshape(k(apxGrid('expand',xg)), sz(:)'); for i=1:p, c = sum(c,2*i); end, c = squeeze(c); end % mode 10 (dirder) % d a'*Mx*Kg*Mx'*b / d hyp function dhyp = dirder(Kg,xg,Mx,a,b) p = numel(Kg.kron); % number of Kronecker factors ng = [apxGrid('size',xg)',1]; % grid dimension Mta = Mx'*a; Mtb = Mx'*b; dhyp = []; % dhyp(i) = a'*dKi*b for i=1:p sz = [prod(ng(1:i-1)),ng(i),prod(ng(i+1:p))]; % bring arrays in vector shape shp = @(x) reshape(permute(reshape(x,sz),[2,1,3]),ng(i),[]); v = reshape(Mta,ng); for j=1:p, if i~=j, v = tmul(Kg.kron(j).factor,v,j); end, end if isnumeric(Kg.kron(i).factor) dhci = Kg.kron(i).dfactor( shp(v)*shp(Mtb)' ); else kii = Kg.kron(i).factor.kii; [junk,ni] = apxGrid('expand',xg{i}); di = numel(ni); xs = cell(di,1); % generic (Strang) circular embedding grid for j=1:di, n2 = floor(ni(j)-1/2)+1; xs{j} = [1:n2,n2-2*ni(j)+2:0]'; end [junk,dksi] = kii(apxGrid('expand',xs)); Fvb = fftn(sum(fftn2(shp(v),ni',1).*fftn2(shp(Mtb), ni'),di+1)); dhci = real(dksi(Fvb(:))); end dhyp = [dhyp; dhci(:)]; end function y = fftn2(x,n,t) % fftn on trailing dimensions with padding if nargin<3, t = 0; end % set a default value nx = numel(x)/prod(n); y = reshape(x,[n(:)',nx]); % #instances if t for i=1:numel(n), y = fft(y,2*n(i)-1,i); end % fftn on relevant dimensions else for i=1:numel(n), y = ifft(y,2*n(i)-1,i); end % ifftn on relevant dimensions end % mode 11 (tmul) % Multiplication with matrix/operator A along dimension dim of the tensor B. % Note that tmul(A,B,1) = A*B if B is a matrix. function C = tmul(A,B,dim) if isnumeric(A), sa = size(A); else sa = A.size; end sb = size(B); nb = ndims(B); assert(dim>0 && floor(dim)==dim && dim<=nb) assert(numel(sa)==2 && sb(dim)==sa(2)) if isnumeric(A), mvmA = @(x) A*x; else mvmA = A.mvm; end if dim==1 % along first dimension (save on permute) C = reshape(mvmA(reshape(B,sa(2),[])),[sa(1),sb(2:nb)]); elseif dim==nb % along last dimension (save on permute) C = reshape(mvmA(reshape(B,[],sa(2))')',[sb(1:nb-1),sa(1)]); else sb3 = [prod(sb(1:dim-1)),sa(2),prod(sb(dim+1:end))]; C = permute(reshape(B,sb3),[2,3,1]); C = reshape(mvmA(reshape(C,sa(2),[])),[sa(1),sb3(3),sb3(1)]); C = reshape(permute(C,[3,1,2]),[sb(1:dim-1),sa(1),sb(dim+1:end)]); end % mode 12 (equi) function eq = equi(xg,i) % grid along dim i is equispaced xi = xg{i}; if iscell(xi) eq = true; for j=1:numel(xi), eq = eq && equi(xi,j); end else ni = size(xi,1); if ni>1 % diagnose if data is linearly increasing dev = abs(diff(xi)-ones(ni-1,1)*(xi(2,:)-xi(1,:))); eq = max(dev(:))<1e-9; else eq = true; end end % mode 13 (info) function s = info(Kg,Mx,xg,deg) sk = 'K ='; Ks = Kg.kron; sk = [sk,' kron[ ']; p = numel(xg); for i=1:p if isnumeric(Ks(i).factor) si = sprintf('mat(%d)',size(Ks(i).factor,1)); else sz = num2str(size(xg{i}{1},1)); for j=2:numel(xg{i}), sz = [sz,'x',num2str(size(xg{i}{j},1))]; end si = sprintf('%s(%s)',Ks(i).factor.descr(1:4),sz); end if i0 else dk = @(d2,k) -dm(sqrt(d*d2),f)*d/2; end [varargout{:}] = covMaha(mode, par, k, dk, varargin{:}); function A = set_zero(A,I), A(I) = 0;gpml-matlab-v4.0-2016-10-19/cov/covMask.m000644 000765 000024 00000003614 12757566731 017456 0ustar00carlstaff000000 000000 function [K,dK] = covMask(cov, hyp, x, z) % Apply a covariance function to a subset of the dimensions only. The subset can % either be specified by a 0/1 mask by a boolean mask or by an index set. % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls another covariance function to do the actual work. % % The function was suggested by Iain Murray, 2010-02-18 and is based on an % earlier implementation of his dating back to 2009-06-16. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-17. % % See also COVFUNCTIONS.M. mask = fix(cov{1}(:)); % either a binary mask or an index set cov = cov(2); % covariance function to be masked if iscell(cov{:}), cov = cov{:}; end % properly unwrap nested cell arrays nh_string = feval(cov{:}); % number of hyperparameters of the full covariance if max(mask)<2 && length(mask)>1, mask = find(mask); end % convert 1/0->index D = length(mask); % masked dimension if nargin<3, K = num2str(eval(nh_string)); return, end % number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode if eval(nh_string)~=length(hyp) % check hyperparameters error('number of hyperparameters does not match size of masked data') end xm = x(:,mask); if ~dg && ~xeqz, zm = z(:,mask); else zm = z; end if nargout>1 [K,dK] = feval(cov{:}, hyp, xm, zm); dK = @(Q) dirder(Q,dK,x,mask); else K = feval(cov{:}, hyp, xm, zm); end function [dhyp,dx] = dirder(Q,dK,x,mask) if nargout>1 [dhyp,dxm] = dK(Q); n = size(x,1); subs = [repmat((1:n)',length(mask),1), reshape(repmat(mask(:)',n,1),[],1)]; dx = accumarray(subs,dxm(:),size(x)); else dhyp = dK(Q); endgpml-matlab-v4.0-2016-10-19/cov/covPPard.m000644 000765 000024 00000001654 12777177046 017572 0ustar00carlstaff000000 000000 function varargout = covPPard(v,varargin) % Wrapper for Piecewise Polyonomial covariance function covPP.m. % % Piecewise Polynomial covariance function with compact support, v = 0,1,2,3. % The covariance functions are 2v times contin. diff'ble and the corresponding % processes are hence v times mean-square diffble. The covariance function is: % % k(x,z) = s2f * max(1-r,0)^(j+v) * f(r,j) with j = floor(D/2)+v+1 % % where r is the distance sqrt((x-z)'*inv(P)*(x-z)), and the P matrix % is diagonal with ARD parameters ell_1^2,...,ell_D^2, where D is the dimension % of the input space and sf2 is the signal variance. The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-27. % % See also covPP.M. varargout = cell(max(1,nargout),1); [varargout{:}] = covScale({'covPP','ard',[],v},varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covScale.m000644 000765 000024 00000006220 12757566731 017606 0ustar00carlstaff000000 000000 function [K,dK] = covScale(cov, lsf, hyp, x, z) % covScale - compose a covariance function as a scaled version of another % one to model functions of the form f(x) = sf(x) f0(x), where sf(x) is a % scaling function determining the function's standard deviation given f0(x) % is normalised. % % The covariance function is parameterized as: % k(x,z) = sf(x) * k_0(x,z) * sf(z) % with an important special case being % k(x,z) = sf^2 * k_0(x,z). % % You can either use K = covScale(cov, lsf, hyp, x, z) where the log scaling % function lsf is a GPML mean function with hyperparameters hyp_sf yielding % hyp = [ hyp_cov % hyp_lsf ] % as hyperparameters % or you can use covScale(cov, hyp, x, z) to perform % rescaling by a scalar value sf specified as an additional variable yielding % hyp = [ hyp_cov % log(sf) ] % as hyperparameters. % % Copyright (c) by Carl Edward Rasmussen, Hannes Nickisch & Roman Garnett % 2016-04-26. % % See also COVFUNCTIONS.M. if nargin==0, error('cov function must be specified'), end if nargin<=1, lsf = []; end, narg = nargin; % set a default value if isnumeric(lsf)&&~isempty(lsf) % shift parameters if sf contains actually hyp if nargin>3, z = x; end if nargin>2, x = hyp; end if nargin>1, hyp = lsf; end narg = nargin+1; lsf = []; end % below we us narg instead of nargin to be independent of the parameter shift if isempty(lsf), ssf = '1'; else ssf = feval(lsf{:}); end % number of hypers if narg<4, K = ['(',feval(cov{:}),'+',ssf,')']; return, end if narg<5, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); % dimension of input data ncov = eval(feval(cov{:})); hyp_cov = hyp(1:ncov); % number of params, split hyp nsf = eval(ssf); hyp_lsf = hyp(ncov+(1:nsf)); scalar = isempty(lsf); if scalar, sf = exp(hyp_lsf); end if ncov+nsf~=numel(hyp), error('Wrong number of hyper parameters.'), end if nargout > 1 [K0,dK0] = feval(cov{:},hyp_cov,x,z); else K0 = feval(cov{:},hyp_cov,x,z); end if scalar sfx = sf; dsfx = @(q) sf*sum(q); else [lsfx,dlsfx] = feval(lsf{:},hyp_lsf,x); sfx = exp(lsfx); dsfx = @(q) dlsfx(q.*sfx); end if dg S = sfx.*sfx; sfz = sfx; dsfz = dsfx; else if xeqz sfz = sfx; dsfz = dsfx; else if scalar sfz = sf; dsfz = @(q) sf*sum(q); else [lsfz,dlsfz] = feval(lsf{:},hyp_lsf,z); sfz = exp(lsfz); dsfz = @(q) dlsfz(q.*sfz); end end S = sfx*sfz'; end K = S.*K0; % covariance if nargout > 1 % directional hyper derivative dK = @(Q) dirder(Q,S,K0,dK0,sfx,dsfx,sfz,dsfz,dg); end function [dhyp,dx] = dirder(Q,S,K0,dK0,sfx,dsfx,sfz,dsfz,dg) if nargout>1 [dhyp0,dx0] = dK0(Q.*S); dx = dx0; % sx contributions are missing else dhyp0 = dK0(Q.*S); end Q = Q.*K0; if dg qz = Q.*sfx; qx = Q.*sfz; else qz = sum(Q'*sfx,2); qx = sum(Q*sfz,2); end dhyp = [dhyp0; dsfx(qx)+dsfz(qz)];gpml-matlab-v4.0-2016-10-19/cov/covRQiso.m000644 000765 000024 00000002215 12777177047 017614 0ustar00carlstaff000000 000000 function varargout = covRQiso(varargin) % Wrapper for Rational Quadratic covariance function covRQ.m. % % Rational Quadratic covariance function with isotropic distance measure. The % covariance function is parameterized as: % % k(x,z) = sf^2 * [1 + (x-z)'*inv(P)*(x-z)/(2*alpha)]^(-alpha) % % where the P matrix is ell^2 times the unit matrix, sf2 is the signal % variance and alpha is the shape parameter for the RQ covariance. The % hyperparameters are: % % hyp = [ log(ell) % log(sf) % log(alpha) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-01. % % See also covRQ.m. varargout = cell(max(1,nargout),1); if nargin>0 % restore old hyper parameter order hyp = varargin{1}; if numel(hyp)>2, varargin{1} = hyp([1:end-2,end,end-1]); end end [varargout{:}] = covScale({'covRQ','iso',[]},varargin{:}); if nargout>1 % restore old hyper parameter order o2 = varargout{2}; varargout{2} = @(Q) dirder(Q,o2); end function [dKdhyp,dKdx] = dirder(Q,dK) if nargout>1, [dKdhyp,dKdx] = dK(Q); else dKdhyp = dK(Q); end dKdhyp = dKdhyp([1:end-2,end,end-1]);gpml-matlab-v4.0-2016-10-19/cov/covNoise.m000644 000765 000024 00000001450 12777177047 017634 0ustar00carlstaff000000 000000 function varargout = covNoise(varargin) % Wrapper for unit "white noise" covariance function covEye.m. % % Independent covariance function, i.e. "white noise". % The covariance function is specified as: % % k(x^p,x^q) = sf^2 * \delta(p,q) % % \delta(p,q) is a Kronecker delta function which is 1 iff p=q and zero % otherwise in mode 1). % In cross covariance mode 2) two data points x_p and z_q are considered equal % if their difference norm |x_p-z_q| is less than eps, the machine precision. % The hyperparameters are: % % hyp = [ log(sf) ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-19. % % See also covEye.m. varargout = cell(max(nargout,1),1); [varargout{:}] = covScale({'covEye'},varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covDot.m000644 000765 000024 00000004740 12757566731 017312 0ustar00carlstaff000000 000000 function [K,dK,S] = covDot(mode, k, dk, hyp, x, z) % Mahalanobis distance-based covariance function. The covariance function is % parameterized as: % % k(x,z) = k(s), s = dot(x,z) = x'*inv(P)*z % % where the P matrix is the metric. The hyperparameters are: % % hyp = [ log(ell) ] % % We offer three different modes: % 'eye': inv(P) = eye(D); hyp = []; % 'iso': inv(P) = ell^2*eye(D); hyp = [log(ell)]; % 'ard': inv(P) = diag(ell.^2); hyp = [log(ell_1); ..; log(ell_D)]; % % For more help on design of covariance functions, try "help covFunctions". % % Note: dk(s) = d k / d s % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-27. % % See also COVFUNCTIONS.M. if nargin<1, mode = 'eye'; end % default value if isequal(mode,'ard'), ne = 'D'; elseif isequal(mode,'iso'), ne = '1'; elseif isequal(mode,'eye'), ne = '0'; else error('Parameter mode is either ''eye'', ''iso'' or ''ard''.'), end if nargin<5, K = ne; return; end % report number of parameters if nargin<6, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % sort out different types [n,D] = size(x); ne = eval(ne); % determine mode hyp = hyp(:); % make sure hyp is a column vector ell = exp(hyp(1:ne)); % characteristic length scale if numel(ell)==0, ell = 1; end % catch eye A = @(x) bsxfun(@times,x,1./ell(:)'.^2); % mvm with metric A=inv(P) % compute dot product if dg % vector sxx Az = A(x); S = sum(x.*Az,2); else if xeqz % symmetric matrix Sxx Az = A(x); else % cross terms Sxz Az = A(z); end S = x*Az'; end K = k(S); % covariance if nargout > 1 dK = @(Q) dirder(Q,S,dk,x,Az,dg,xeqz,mode); % dir hyper derivative end function [dhyp,dx] = dirder(Q,S,dk,x,Az,dg,xeqz,mode) R = dk(S).*Q; if isequal(mode,'ard') if dg dhyp = -2*sum(x.*bsxfun(@times,R,Az),1)'; else dhyp = -2*sum(x.*(R*Az),1)'; end elseif isequal(mode,'iso') dhyp = -2*R(:)'*S(:); else dhyp = zeros(0,1); end if nargout > 1 if xeqz, dx = R*Az+R'*Az; else dx = R*Az; end endgpml-matlab-v4.0-2016-10-19/cov/covCos.m000644 000765 000024 00000003151 12757566731 017303 0ustar00carlstaff000000 000000 function [K,dK] = covCos(hyp, x, z) % Stationary covariance function for a sinusoid with period p in 1d: % % k(x,z) = sf^2*cos(2*pi*(x-z)/p) % % where the hyperparameters are: % % hyp = [ log(p) % log(sf) ] % % Note that covPeriodicNoDC converges to covCos as ell goes to infinity. % % Copyright (c) by James Robert Lloyd and Hannes Nickisch, 2016-04-24. % % See also COVFUNCTIONS.M, COVPERIODICNODC.M. if nargin<2, K = '2'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); if D>1, error('Covariance is defined for 1d data only.'), end p = exp(hyp(1)); sf2 = exp(2*hyp(2)); % precompute deviations and exploit symmetry of cos if dg % vector txx T = zeros(size(x,1),1); else if xeqz % symmetric matrix Txx T = 2*pi/p*bsxfun(@plus,x,-x'); else % cross covariances Txz T = 2*pi/p*bsxfun(@plus,x,-z'); end end K = sf2*cos(T); % covariances if nargout > 1 dK = @(Q) dirder(Q,K,T,x,p,sf2,dg,xeqz); end function [dhyp,dx] = dirder(Q,K,T,x,p,sf2,dg,xeqz) dhyp = [sf2*(sin(T(:)).*T(:))'*Q(:); 2*Q(:)'*K(:)]; if nargout > 1 R = -sf2*pi/p * Q .* sin(T); if dg dx = zeros(size(x)); else if xeqz dx = 2*(sum(R,2)-sum(R,1)'); else dx = 2*sum(R,2); end end endgpml-matlab-v4.0-2016-10-19/cov/covMaternard.m000644 000765 000024 00000001740 12777177047 020476 0ustar00carlstaff000000 000000 function varargout = covMaternard(d,varargin) % Wrapper for Matern covariance function covMatern.m. % % Matern covariance function with nu = d/2 and with Automatic Relevance % Determination (ARD) distance measure. For d=1 the function is also known as % the exponential covariance function or the Ornstein-Uhlenbeck covariance % in 1d. The covariance function is: % % k(x,z) = f( sqrt(d)*r ) * exp(-sqrt(d)*r) % % with f(t)=1 for d=1, f(t)=1+t for d=3 and f(t)=1+t+t^2/3 for d=5. % Here r is the distance sqrt((x-z)'*inv(P)*(x-z)), where the P matrix % is diagonal with ARD parameters ell_1^2,...,ell_D^2, where D is the dimension % of the input space and sf2 is the signal variance. The hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % .. % log(ell_D) % log(sf) ] % % Copyright (c) by Hannes Nickisch, 2016-04-17. % % See also COVFUNCTIONS.M. varargout = cell(max(1,nargout),1); [varargout{:}] = covScale({'covMatern','ard',[],d},varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covGabor.m000644 000765 000024 00000005606 12757566731 017620 0ustar00carlstaff000000 000000 function [K,dK] = covGabor(mode, hyp, x, z) % Gabor covariance function with length scale ell and period p. The % covariance function is parameterized as: % % k(x,z) = h(x-z), h(t) = exp(-sum(t.^2./(2*ell.^2)))*cos(2*pi*sum(t./p)). % % The hyperparameters are: % % hyp = [ hyp_ell % hyp_p ] % % We offer three different modes: % 'eye': ell = ones(D,1); p = ones(D,1); % 'iso': ell = exp(hyp_ell)*ones(D,1); p = exp(hyp_p)*ones(D,1); % 'ard': ell = exp(hyp_ell) ; p = exp(hyp_p) ; % % Note that covSM implements a weighted sum of Gabor covariance functions, but % using an alternative (spectral) parameterization. % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Hannes Nickisch, 2016-05-03. % % See also COVFUNCTIONS.M, COVGABORARD.M, COVSM.M. if nargin<1, error('We require a mode.'), end if isequal(mode,'ard'), np = 'D'; elseif isequal(mode,'iso'), np = '1'; elseif isequal(mode,'eye'), np = '0'; else error('Parameter mode is either ''eye'', ''iso'' or ''ard''.'), end if nargin<3, K = ['(2*',np,')']; return, end % report number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode [n,D] = size(x); np = eval(np); % dimensionality p = exp(hyp(np+1:2*np)); if numel(p)==0, p = 1; end, p = p.*ones(D,1); % period [Kse,dKse] = covSE(mode,[],hyp(1:np),x,z); % squared exponential base covariance Dp = zeros(size(Kse)); % init sum(t)/p computation if ~dg if xeqz % symmetric matrix Kxx for d=1:D, Dp = Dp + bsxfun(@minus,x(:,d),x(:,d)')/p(d); end else % cross covariances Kxz for d=1:D, Dp = Dp + bsxfun(@minus,x(:,d),z(:,d)')/p(d); end end end C = cos(2*pi*Dp); K = Kse .* C; % covariance if nargout > 1 % directional derivative dK = @(Q) dirder(Q,C,Dp,Kse,dKse,np,dg,xeqz,x,z,D,p,mode); end function [dhyp,dx] = dirder(Q,C,Dp,Kse,dKse,np,dg,xeqz,x,z,D,p,mode) dhyp = zeros(2*np,1); % allocate memory if nargout > 1 [dhyp(1:np),dx] = dKse(Q.*C); else dhyp(1:np) = dKse(Q.*C); end if ~dg Q = Q .* Kse .* -sin(2*pi*Dp)*2*pi; if isequal(mode,'ard') for d=1:D if xeqz Dd = bsxfun(@minus,x(:,d),x(:,d)'); else Dd = bsxfun(@minus,x(:,d),z(:,d)'); end dhyp(np+d) = -(Q(:)'*Dd(:))/p(d); end elseif isequal(mode,'iso') dhyp(np+1) = -(Q(:)'*Dp(:)); end if nargout > 1 if xeqz dx = dx + (sum(Q,2)-sum(Q,1)')*(1./p)'; else dx = dx + sum(Q,2)*(1./p)'; end end endgpml-matlab-v4.0-2016-10-19/cov/covDiscrete.m000644 000765 000024 00000004614 12757566731 020326 0ustar00carlstaff000000 000000 function [K,dK] = covDiscrete(s, hyp, x, z) % Covariance function for discrete inputs. Given a function defined on the % integers 1,2,3,..,s, the covariance function is parameterized as: % % k(x,z) = K_{xz}, % % where K is a matrix of size (s x s). % % This implementation assumes that the inputs x and z are given as integers % between 1 and s, which simply index the matrix K. % % The hyperparameters specify the upper-triangular part of the Cholesky factor % of K, where the (positive) diagonal elements are specified by their logarithm. % If L = chol(K), so K = L'*L, then the hyperparameters are: % % hyp = [ log(L_11) % L_21 % log(L_22) % L_31 % L_32 % .. % log(L_ss) ] % % The hyperparameters hyp can be generated from K using: % L = chol(K); L(1:(s+1):end) = log(diag(L)); % hyp = L(triu(true(s))); % % The covariance matrix K is obtained from the hyperparameters hyp by: % L = zeros(s); L(triu(true(s))) = hyp(:); % L(1:(s+1):end) = exp(diag(L)); K = L'*L; % % This parametrization allows unconstrained optimization of K. % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Roman Garnett, 2016-04-18. % % See also MEANDISCRETE.M, COVFUNCTIONS.M. if nargin==0, error('s must be specified.'), end % check for dimension if nargin<3, K = num2str(s*(s+1)/2); return; end % report number of parameters if nargin<4, z = []; end % make sure, z exists xeqz = isempty(z); dg = strcmp(z,'diag'); % determine mode if xeqz, z = x; end % make sure we have a valid z L = zeros(s); L(triu(true(s))) = hyp(:); % build Cholesky factor L(1:(s+1):end) = exp(diag(L)); A = L'*L; % A is a placeholder for K to avoid a name clash with the return arg if dg K = A(sub2ind(size(A),fix(x),fix(x))); else K = A(fix(x),fix(z)); end if nargout > 1 dK = @(Q) dirder(Q,s,L,x,z,dg); % directional hyper derivative end function [dhyp,dx] = dirder(Q,s,L,x,z,dg) nx = numel(x); Ix = sparse(fix(x),1:nx,ones(nx,1),s,nx); % K==Ix'*L'*L*Iz if dg B = Ix*diag(Q)*Ix'; else nz = numel(z); Iz = sparse(fix(z),1:nz,ones(nz,1),s,nz); B = Iz*Q'*Ix'; end B = L*(B+B'); B(1:(s+1):end) = diag(B).*diag(L); % fix exp on diagonal dhyp = B(triu(true(s))); if nargout > 1, dx = zeros(size(x)); endgpml-matlab-v4.0-2016-10-19/cov/covSEiso.m000644 000765 000024 00000001300 12777177052 017567 0ustar00carlstaff000000 000000 function varargout = covSEiso(varargin) % Wrapper for Squared Exponential covariance function covSE.m. % % Squared Exponential covariance function with isotropic distance measure. The % covariance function is parameterized as: % % k(x,z) = sf^2 * exp(-(x-z)'*inv(P)*(x-z)/2) % % where the P matrix is ell^2 times the unit matrix and sf^2 is the signal % variance. The hyperparameters are: % % hyp = [ log(ell) % log(sf) ] % % For more help on design of covariance functions, try "help covFunctions". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-27. % % See also covSE.M. varargout = cell(max(1,nargout),1); [varargout{:}] = covScale({'covSE','iso',[]},varargin{:});gpml-matlab-v4.0-2016-10-19/cov/apxSparse.m000644 000765 000024 00000005034 12777177046 020016 0ustar00carlstaff000000 000000 function K = apxSparse(cov, xu, hyp, x, z) % apxSparse - Covariance function for sparse approximations. % % This is a special covariance function which handles some non-standard % requirements of sparse covariances % 1) it holds the inducing inputs in xu, and % 2) it returns covariances for prediction purposes. Note, that sparse % inference methods don't actually call this function. % % Any sparse approximation to the posterior Gaussian process is equivalent to % using the approximate covariance function: % Kt = Q + s*diag(g); g = diag(K-Q); Q = Ku'*inv(Kuu+snu2*eye(nu))*Ku; % where Ku and Kuu are covariances w.r.t. to inducing inputs xu and % snu2 is the noise variance of the inducing inputs. We fix the standard % deviation of the inducing inputs snu = sfu/1e3 to be a one per mil of the % signal standard deviation sfu^2 = trace(Kuu)/nu of the inducing inputs. % The parameter s from [0,1] (see [1]) interpolates between the two limiting % cases s=1: FITC [2] and s=0 VFE [3]. % % The function is designed to be used with infGaussLik and infLaplace. % % The implementation exploits the Woodbury matrix identity % inv(Kt+inv(W)) = D - D*Ku'*inv(Kuu+snu2*eye(nu)+Ku*D*Ku')*Ku*D, where % D = diag(d), and d = W./(s*g.*W+1) % and the Cholesky decomposition in order to be applicable to large datasets. % The computational complexity is O(n nu^2) where n is the number of data % points x and nu the number of inducing inputs in xu. % % The inducing points can be specified through % 1) the 2nd apxSparse parameter or by % 2) providing a hyp.xu hyperparameters to the inference function. % Note that 2) has priority over 1). % In case 2) is provided and derivatives dnlZ are requested, there will also be % a dnlZ.xu field allowing to optimise w.r.t. to the inducing points xu. % % [1] Bui, Yan & Turner, A Unifying Framework for Sparse GP Approximation % using Power EP, 2016, https://arxiv.org/abs/1605.07066. % [2] Snelson & Ghahramani, Sparse GPs using pseudo-inputs, NIPS, 2006. % [3] Titsias, Var. Learning of Inducing Variables in Sparse GPs, AISTATS, 2009 % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-09-09. % % See also COVFUNCTIONS.M, APX.M, INFLAPLACE.M, INFGAUSSLIK.M. if nargin<4, K = feval(cov{:}); return, end if size(xu,2) ~= size(x,2) error('Dimensionality of inducing inputs must match training inputs'); end if strcmp(z, 'diag') K = feval(cov{:}, hyp, x, z); else K = feval(cov{:}, hyp, xu, z); % substitue inducing inputs for x endgpml-matlab-v4.0-2016-10-19/cov/covProd.m000644 000765 000024 00000006100 12757566731 017460 0ustar00carlstaff000000 000000 function [K,dK] = covProd(cov, hyp, x, z) % covProd - compose a covariance function as the product of other covariance % functions. This function doesn't actually compute very much on its own, it % merely does some bookkeeping, and calls other covariance functions to do the % actual work. % % Note that cov = {cov1, cov2, .., false} turns of covariance matrix caching. % This option slows down the computations but can help out if you products of % many huge matrices lead to working memory shortage. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-18. % % See also COVFUNCTIONS.M. if isempty(cov), error('We require at least one factor.'), end if isnumeric(cov{end}) || islogical(cov{end}) % detect whether to cache cache = ~isequal(0,cov{end}) && ~isequal(false,cov{end}); cov = cov(1:end-1); % chop off last element from cov else cache = true; end nc = numel(cov); % number of terms in covariance function for ii = 1:nc % iterate over covariance functions f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary j(ii) = cellstr(feval(f{:})); % collect number hypers end if nargin<3 % report number of parameters K = char(j(1)); for ii=2:nc, K = [K, '+', char(j(ii))]; end, return end if nargin<4, z = []; end % make sure, z exists [n,D] = size(x); nh = numel(hyp); % number of hyperparameters v = []; % v vector indicates to which covariance parameters belong for ii = 1:nc, v = [v repmat(ii, 1, eval(char(j(ii))))]; end K = 1; Ki = cell(nc,1); dKi = cell(nc,1); % init K for ii = 1:nc % iteration over factor functions f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary if cache if nargin>1 [Ki{ii},dKi{ii}] = feval(f{:}, hyp(v==ii), x, z); Kii = Ki{ii}; % track else Ki{ii} = feval(f{:}, hyp(v==ii), x, z); Kii = Ki{ii}; end else Kii = feval(f{:}, hyp(v==ii), x, z); end K = K .* Kii; % accumulate covariances end dK = @(Q) dirder(Q,Ki,dKi,v,nc,nh,cov,hyp,x,z,cache); % directional derivative function [dhyp,dx] = dirder(Q,Ki,dKi,v,nc,nh,cov,hyp,x,z,cache) dhyp = zeros(nh,1); dx = 0; for ii = 1:nc Qi = Q; for jj=1:nc % accumulate if ii~=jj if cache Qi = Qi .* Ki{jj}; else f = cov(jj); if iscell(f{:}), f = f{:}; end % expand cell if necessary Qi = Qi .* feval(f{:}, hyp(v==jj), x, z); end end end if cache dKii = dKi{ii}; else f = cov(ii); if iscell(f{:}), f = f{:}; end % expand cell if necessary [junk,dKii] = feval(f{:}, hyp(v==ii), x, z); end if nargout==1 dhyp(v==ii,1) = dKii(Qi); else [dhyp(v==ii,1),dxi] = dKii(Qi); dx = dx+dxi; end endgpml-matlab-v4.0-2016-10-19/cov/covSEard.m000644 000765 000024 00000001441 12777177052 017551 0ustar00carlstaff000000 000000 function varargout = covSEard(varargin) % Wrapper for Squared Exponential covariance function covSE.m. % % Squared Exponential covariance function with Automatic Relevance Detemination % (ARD) distance measure. The covariance function is parameterized as: % % k(x,z) = sf^2 * exp(-(x-z)'*inv(P)*(x-z)/2) % % where the P matrix is diagonal with ARD parameters ell_1^2,...,ell_D^2, where % D is the dimension of the input space and sf2 is the signal variance. The % hyperparameters are: % % hyp = [ log(ell_1) % log(ell_2) % . % log(ell_D) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-27. % % See also covSE.m. varargout = cell(max(1,nargout),1); [varargout{:}] = covScale({'covSE','ard',[]},varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covSE.m000644 000765 000024 00000002037 12757566731 017070 0ustar00carlstaff000000 000000 function varargout = covSE(mode, par, varargin) % Squared Exponential covariance function with unit amplitude. The covariance % function is: % % k(x,z) = exp(-maha(x,z)/2) % % where maha(x,z) is a squared Mahalanobis distance. The function takes a "mode" % parameter, which specifies precisely the Mahalanobis distance used, see % covMaha. The function returns either the number of hyperparameters (with less % than 3 input argments) or it returns a covariance matrix and (optionally) a % derivative function. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-05-23. % % See also covMaha.m. if nargin < 1, error('Mode cannot be empty.'); end % no default if nargin < 2, par = []; end % default varargout = cell(max(1, nargout), 1); % allocate mem for output if nargin < 4, varargout{1} = covMaha(mode,par); return, end k = @(d2) exp(-d2/2); dk = @(d2,k) (-1/2)*k; % covariance and derivative [varargout{:}] = covMaha(mode, par, k, dk, varargin{:});gpml-matlab-v4.0-2016-10-19/cov/covPeriodicNoDC.m000644 000765 000024 00000010031 12777177047 021014 0ustar00carlstaff000000 000000 function [K,dK] = covPeriodicNoDC(hyp, x, z) % Stationary covariance function for a smooth periodic function, with period p: % % k(x,z) = sf^2 * [k0(pi*(x-z)/p) - f(ell)] / [1 - f(ell)] % with k0(t) = exp( -2*sin^2(t)/ell^2 ) and f(ell) = \int 0..pi k0(t) dt. % % The constant (DC component) has been removed and marginal variance is sf^2. % The hyperparameters are: % % hyp = [ log(ell) % log(p) % log(sf) ] % % Note that covPeriodicNoDC converges to covCos as ell goes to infinity. % % Copyright (c) by James Robert Lloyd and Hannes Nickisch 2016-04-24. % % See also COVFUNCTIONS.M, COVCOS.M. if nargin<2, K = '3'; return; end % report number of parameters if nargin<3, z = []; end % make sure, z exists xeqz = numel(z)==0; dg = strcmp(z,'diag') && numel(z)>0; % determine mode [n,D] = size(x); if D>1, error('Covariance is defined for 1d data only.'), end ell = exp(hyp(1)); p = exp(hyp(2)); sf2 = exp(2*hyp(3)); % extract hyperparams % precompute deviations and exploit symmetry of sin^2 if dg % vector txx T = zeros(size(x,1),1); else if xeqz % symmetric matrix Txx T = pi/p*bsxfun(@plus,x,-x'); else % cross covariances Txz T = pi/p*bsxfun(@plus,x,-z'); end end K = covD(2*T,ell); K = sf2*K; % covariance if nargout>1 ebi0 = embi0(1/ell^2); ebi1 = embi1(1/ell^2); dK = @(Q) dirder(Q,K,T,ell,p,sf2,xeqz,x,z,ebi0,ebi1); % directional hyp deriv end function [dhyp,dx] = dirder(Q,K,T,ell,p,sf2,xeqz,x,z,ebi0,ebi1) S2 = (sin(T)/ell).^2; if ell>1e4 % limit for ell->infty Z = zeros(size(T)); % no further progress in ell possible elseif 1/ell^2<3.75 cK = cos(2*T); ecK = exp(cK/ell^2); b0 = besseli(0,1/ell^2); b1 = besseli(1,1/ell^2); Z = 2*(exp(1/ell^2)-ecK )*b1 ... - 2*(exp(1/ell^2)-ecK.*cK)*b0 ... + 4*exp(2*(cos(T)/ell).^2).*sin(T).^2; Z = sf2/(ell*(exp(1/ell^2)-b0))^2 * Z; else cK = cos(2*T); ecK = exp((cK-1)/ell^2); b0 = ebi0; b1 = ebi1; Z = 2*(1-ecK)*b1 - 2*(1-ecK.*cK)*b0 ... + 4*exp(2*(cos(T).^2-1)/ell^2).*sin(T).^2; Z = sf2/(ell*(1-b0))^2 * Z; end if ell>1e4 % limit for ell->infty Y = 2*sf2* sin(2*T).*T; a = 1; elseif 1/ell^2<3.75 c = 1/(exp(1/ell^2)-b0)/ell^2; Y = 2*c*sf2*exp( cos(2*T) /ell^2).*sin(2*T).*T; a = 1/(1-b0*exp(-1/ell^2))/ell^2; else c = 1/(1-b0)/ell^2; Y = 2*c*sf2*exp((cos(2*T)-1)/ell^2).*sin(2*T).*T; a = c; end dhyp = [Z(:)'*Q(:); Y(:)'*Q(:); 2*(Q(:)'*K(:))]; if nargout > 1 Kdc = sf2*exp( -2*S2 ); % inkl. DC component R = Kdc.*sin(2*T).*Q./T; R(T==0) = 0; r2 = sum(R,2); r1 = sum(R,1)'; if xeqz y = bsxfun(@times,r1+r2,x) - (R+R')*x; else Rz = R*z; y = bsxfun(@times,r2,x) - Rz; end dx = -2*a*pi^2/p^2 * y; end function K = covD(D,ell) % evaluate covariances from distances if ell>1e4 % limit for ell->infty K = cos(D); elseif 1/ell^2<3.75 K = exp(cos(D)/ell^2); b0 = besseli(0,1/ell^2); K = (K-b0)/(exp(1/ell^2)-b0); else K = exp((cos(D)-1)/ell^2); b0 = embi0(1/ell^2); K = (K-b0)/(1-b0); end function y = embi0(x) % = exp(-x)*besseli(0,x) => 9.8.2 Abramowitz & Stegun y = 3.75/x; y = 0.39894228 + 0.01328592*y + 0.00225319*y^2 - 0.00157565*y^3 ... + 0.00916281*y^4 - 0.02057706*y^5 + 0.02635537*y^6 - 0.01647633*y^7 ... + 0.00392377*y^8; y = y/sqrt(x); function y = embi1(x) % = exp(-x)*besseli(1,x) => 9.8.4 Abramowitz & Stegun y = 3.75/x; y = 0.39894228 - 0.03988024*y - 0.00362018*y^2 + 0.00163801*y^3 ... - 0.01031555*y^4 + 0.02282967*y^5 - 0.02895312*y^6 + 0.01787654*y^7 ... - 0.00420059*y^8; y = y/sqrt(x);gpml-matlab-v4.0-2016-10-19/cov/covMaterniso.m000644 000765 000024 00000001513 12777177047 020520 0ustar00carlstaff000000 000000 function varargout = covMaterniso(d,varargin) % Wrapper for Matern covariance function covMatern.m. % % Matern covariance function with nu = d/2 and isotropic distance measure. For % d=1 the function is also known as the exponential covariance function or the % Ornstein-Uhlenbeck covariance in 1d. The covariance function is: % % k(x,z) = sf^2 * f( sqrt(d)*r ) * exp(-sqrt(d)*r) % % with f(t)=1 for d=1, f(t)=1+t for d=3 and f(t)=1+t+t^2/3 for d=5. % Here r is the distance sqrt((x-z)'*inv(P)*(x-z)), P is ell times % the unit matrix and sf2 is the signal variance. The hyperparameters are: % % hyp = [ log(ell) % log(sf) ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-27. % % See also COVFUNCTIONS.M. varargout = cell(max(1,nargout),1); [varargout{:}] = covScale({'covMatern','iso',[],d},varargin{:});gpml-matlab-v4.0-2016-10-19/doc/f8.gif000644 000765 000024 00000034552 12701705600 016637 0ustar00carlstaff000000 000000 GIF89aA/?O_/o?O_o/?O_oo_O?/!Created with GIMP,A H*\ȰÇ#JHEI\VƎ CQɓHjDɲ%&+]ʜfL8s*YRO2 [5 FgJf5׏a;S%ŸeF(ʷkXy( 蛃X(p艇"bpad "8h$:fbIF(@FP9G[W;WoWUQpz /,m%.Ye7E/ϼי87ʼ׋7]wX̠_O\޲V$E(F/)8 g!oH%{Ԙ`,ЁHEnCD߀QP` :.D4HC-‰"!:"yA)1`xD#!cg&fqP a;bQ 7)FAE/;5ưBq,d%!1I^ҍP9IKO/޻_1)b%V&m)4)a^2Ō"1'JA[ēIrdfxN5:c)LMlSKۣ/*~SCKu92xS Y8cx=<?r~$AaPF"Ԙ$C(IBE @JGwjOQ"wzS K9Nt3"l&D< [8uDJֲZRaJP@­)TiѩFª ׾|j@aDC "cGe+!4I7)+ hW(X"v_ _,U1VI4Me0JYIVL_fuF%yzpJӶ=;[ZJ4lVK nZA﷉pNj3ѸD%Z5Z]A\wͭ׶+ ih-y!v^/3,b|)@bWSmuƭaݔ`dE(NW#=3ar!TAh7{E>ib>.XJ1'0Iwq!| k #SKNOʘʛ&L^v` B'\$lE[9O?t-eI G>82FAYG?:Œ4})MK?!+"tWv^׼^FQjT:7l\I@[Z-;<8v'LmeNӰij٪iM֠;km޾v.Vqۦc=t[ӕk_;QY9o\4جxWc3 #+d`<&.7Ϸ\oIݼK؆_B9F<@H/:m~B2sjC 1@Q`peN/q OX`V uEԽlk=Y)~vZp;wgc{;YTc?{8|)v.|!$a}~MXzYwʦ@)kE_r&=#[XϾ{yԿ~p#KQyM4&qv\]wY6~E Gys$Wgzٗ2 bD5!#h%apw"x w !@XeP GA#&4סlTI.80Ȅg'p^wU:eЃsvA8`DZ&VńX&%"gNn8l[~ס7!eqA$gQPjX2*R tXshMx\gUnA4H|tUhc_'88/1H)Xhb`R(}Ds|7{֖8 AP4,&B]ƷzzE苘Sx֘g?b菼Z+1x$Iɋ8P1/4ny @\u^؅'u<ba0R ɋ\PXR,6=QHsk!Y&֎)U&ov}YA`b9dA15PlYPK9uyw~/k)y^Ypؤ _\I{ey0d=H\K؄n(YDu)bdF5^(*ɍD9ٚcɘ03/E(O樊T7wVYPyWx@};!&隮 ؆H 'SxAz9j9swyv}Ie~&pi pr'&5V737p婎 8yYQb U"꟔.y=p`x^2[iŢZ҉":I0Dvzi_7ؐ~C`jF=&ڋ&e L N wGE"E5 A@ea:cZ'.hI٠Siv金7yn5H< YککB~.J 'a+LIyR 5juG $txJ79 ZZ5 ǶI/o]Z[|ꫥz@U)ZiZڇneE$0*JXI : 8P[&*r^H{ ե: v S~zsZu|e &IY4[6{8[j;1Mep)z)I5_`09;TJ2>vXV_ zI)cu8KzO2T7k%-^iGzQWPEHsvEVsPTp۸* /O|0'BC|U=[=gҴjů<붎S+`rd6UT.u5gJ՗7!0,P{țƻg0{ [:(}%B1wv[5[HPq J{˼(ZzQ;_rTEY{}Yq)s[ȫ%*?h+V]s«qָP:7>g*~Fa"о[ 1)"UOLv*Eu_;;[q̪ \ѫ-0P5:E~5~Uuܖ=\^`B0Dfr6AÈ`s)s8Hr Q'a~<;,(@YY47Xi,A>RqUElGxG8™7ʌ6) ۶ʼL\L gY9V  }Yܼ<V!u#'O[2)[Hhm rC]MDӦ#*.Bs:,=]?-fdMRR:c7".,g:6D0!hZ] `zlUf$ݏ4 IHK3,;85YdY ԯ \Y1ɲ=u`:X Вݣ XUD8RAقp MBqt^b3]E;yjg__IE{H;88vQmJ-o]ߍԷTg$w7/C<5esR0n]SU#}CZ٩]+2[ST&dnBq\70$Tt^ f@?R皉(㮴0>څOB?FB/}D}J} @z|zQz\ Znި$ge_Q/a6А[@L=~Mud~x6Vn&'JIsr @.FnC%`e5遌'|ND!JP)'P=됌[P|enTDF{tc&^Hsz>N?Mʉۃ6סMmE+C>7!ol>>xqȝpݢn۸\\Y[@0> <#(+$vx@B?D~UΘ(Fl'';IG{imwiޕ18 Ei0#"-%Me-5atcq.^ݣ?..U Or8\nQ5_a p9/߼g?@Iΰ~EWsb UjB6r\;ah죌L?Ugq?~-σ3}Plu׮~ c'>_ۀ@(hH`8Ȩ)9IYiyYYyZ"êC)"++Hi(2jH*l,`L-h .>N޽Xx~ؾ/?H?=m/<,H+zA%… 4W Ä,2rG<mV.eJܱt$ȥL4k^bI6~7)4 P"[45)U'Vhg$%qqHy:ᴄRZsfBȥ͹t RV֝5QZBmBBT {oI|3i}KlK>軬[o[=rM~ JѬJkʵL2V^,QAjn^mE ]{>;u5zPy-)|j"nEa+? XIfIuބ]^1HyR{!&nDCqaV\&1W }! CXH8YMFBQ{HdN~o, `W${GY<]:Ј c 9a$EbW`UJ5$ 5C3 !s_UTDd,(&YdZC^if&(4iܶaѤ@)ڢgj)N8j٢ziri0[ =Ŋڏe!jLFoTZ2D`/]HVۚ)K/:/%ΘH*rw*oF:PBzkj느r2mRZoKr=J+( r7\x 0ȷ i@ }` jϣ.~l\rTKz|F6!5e+\5Yy%4OCzw~}uu<CocTWҙ3IO]uor8P5ߦNYuˌ綏${ ;eLJzn B!Go>AƓ5ZN}߬"3LІ43P8=n+0_ G7wCz=9(ǣƴGovSQ[q[g a.R9o{oL*"F7Q 8K*J/gŇBt*JhSYBHJGϱ(JqTYB`F @q]T ;%;PBT( j'I@b1f vEaB. *-А2 nK;M܁уtl%IH- Jdcx6S;R|Q89C(c#)1!Lt7a˓,%SRh B6ժgI p2$?lpFX֙LQ^Jo\PDdL;yKba`2bja/@""#ٟh&+7]MG&D-QYAъ%8I90PnxO`#2)% *=[A]͔MАfA;" ET9uFMR"luSgV~SS.j3!LzҌ3OK^ fð)eL،byt}]WE_JnSO*MJWJ*`%T$DS}D,)PE~}(XGx͛ E&ቛۅRGbA&69%M1.c2: ђԄGu F#Y>kwс8 @:f,99&;9\qx/ f$) UM0 (KyP6w1 L%88a ٰa9kW{y!{BZc9vЭh\1&wubvoB@ ʯr~/苻"YO)Q1$XI+)Z޹J8iiCM?qn WC/B%SEs=v㞲37Ԑ}`|.̀nY`$,5Ю6* o|ٵ2`Alm̥l&Aa&0_1hJmj_)۵EI)*%x| 5c lҔASDB\#^Ay 5ɗ䒄4o v8mk6 ecd|oc2Cy^*rlMۥ= rvh룴;۔y7E1ocbo~6H|,, uq31e3//~'tD"F6]jwûe7EGg:;߷ϲoOZu}'})ͅt5-~앚}1k'}~QR P_çW.br Lw7x '#Y4NKX HTF!y -O@$ FCzdTu7 UpH#W&(bFH2*,hwR8}*?c~vwK,w6azuzϒkXK({/Ph W1x'w&c%|;1{0Q`A|I膲kxQ5n$F%iRS814aHZ̧?h20(H(xV60!08%0xisA;NQ9$r7Ê|/ȍ݈h]x t8>悺*8WyC"8ONC(38uэIߨp~5s苍Qh gA{X I~X"A .6-W4aJ]A)9ybI@HRtxi2W9l(9h"Ψ QUzv$ɒW  Y'C;C !B}Ϩ9!>xh8"o q)s OYh Uv6*8yyI ['(tm%u6Wb(dtiiRTقxce;:-!ls(Mq_8&95%2ry Tɗ2yyyv#6q22bgyJ)s i™\[3'𘚈xh)xP|HޙIv Imb ˀI|A6b3%z"Y2بy1@%j'%*!`/ ."`mwwgơ^9#I+J}-ʋZ*X5jbRjC"1e.<"`<8z)LTJ'ʤ&.ڌ06 4cCoY(`yFblֱmo:P Fi;cy$zU 9#bCh"j0ꪯ ʨt=pJs*WFF<>_0|8$z?sdE"),&myJ:S > M 81+4>Uv%kSvIPת:c:6o7*"@$R~J1j^J4MF{ !d!TM trjtxSl'jeQFKzf`RdYПy!+5%[)Ȳ _pLaS %4<+R]H롎PP{HukvTnQ':Q*V49VB1fKնXEw Ke1Rۭ:QY6_yrx ]x껣CJy[ ^{˷}p4vFc oK?D \uYݓFa˾kzg ,:FzcqTBԻH6q )ڻ[3(P[MY˶ bKi,{ f[Ppyєm%羧e)TʋD-`GIlijԴ+[v"s} Y80EY G %=Ӷ(*sLG,$@Q\0VW YLԥѥKc#~0ՑYy'Tۏ<Ձ}yr'L5Gܷ Ԩ9Ke s,-KLkا~+lł,ƶyQ]vm=0a7 - ;[Hfԩ]jvKؓڒ]~#]Ew,M=mvEhڧ0njQ ݧUMbܣh-}ޢm-f=]IM- b]MěUٞ})${Lԉ ؍Mݕt^RBw߼ݙ>4]== lxvW`T߱ܽ$='[][.̼2[܇1&57 H_y想z+->g(7<*Xny.]2Eh1~[}nVPl)MPoD #`>yTZl ,\棘x %.~r~^30R6@t;,^ G쾵3m>;R`B=*vﭻjRkEo2l.[V.wA:/1#3촮UR=4>(H#_a/_d.Ν Ό~<K* I\c{_ +/ nT_D꙰}_g }3nFfD[o"?B@:}ތdV/OA:/?ϝN-X_ٯݯ_4 ȿ!? ]hj$HO#~A/8:A"A Aʢ !㽉A8ƋAX jDZgÔ[T_A.wG5)ݨ}!tnP@B#$澚Kd; !d$2eJhFsLdF}(ԫX-ӇRE-UBIRF*ڛR@WbRgBTӫV}LÄg]LҤ/ s,YSy̶ZFr*楮~6ٌsOʵs^_^u<}FH:HB}zS}s0n9fC_NphF"%)JB9}DW`owZ ڧ`(I`8Oi8al݈߅gn eeL FTdcup)O2ig9Ne!,ZV3m e\1YgqUǨ4]nGi饋ީVejȞYg/fi7V w(FbYff]КlRsqzHt*N"#V'=" VTHFV}FB^H*l:!r[;S#͞HlXMk(Mcj%0w շ" 7 \]G,/K(,*Κ.Vg$!5,SZ%N-,2,Ll4Rتf4 8/xg7Gy $GO,Z!\VӍvWu`֮ŭ]J(L! +GqLpV gHp̠b>HpФgu!GfFԣY t 0L_zP *!0ѓ**MF5ʻ5)pЬ`qOCU#zF٩G*L) 8*ՑP5+Q"R pզnj91IU;+;5ն"Ս\*쮀^W8౐d' ٷ4}AX]!4X6u$ʚ() N&WC,h5i?:,JGu f8^ O]iJַx*\rZC%5'/ԲgFs[ޖb]RuٖwM Gj(ͯ~߆N Y+Ki/+ߖH8ɻdy 7WG$LbR1MT !n07۸eFH\ٶ\9d66rckӔU!yV^$R,.*/r̶#y,߷NY%PsYr΀žL1G|ν9o3dZɨM'ͻ6fz=hVS#gMZZޜ)dîn}mw$[ظFqo|գ:~cAbڳsUc變 Ӕl?S> n{^lКaZFҿev?_`7'98N~;y@gL ri(OyO)K[㑨n蒻mOC|s̫T xpL6G|Nu،h2(XxؘڸHdk;Gt#Th'GզHu7t߈^(> (莕ӊW,%`Qh6x[Qxhfl&gȏhׄdbɐɐV 9HjxX .i!Y9wh'}=9?:)E5הH<0HW9HY=TVdX9敐 *n8:`[_f]GX\9Xky؍2ɒhli3[uVXșG{kIٗɘpAZIyٛ;gpml-matlab-v4.0-2016-10-19/doc/demoGrid1d.m000644 000765 000024 00000005122 12777177044 020001 0ustar00carlstaff000000 000000 disp('See http://www.gaussianprocess.org/gpml/code/matlab/doc/ for details.') clear all, close all, write_fig = 0; N = 30; sd = 3; rand('seed',sd), randn('seed',sd) % set a seed for reproducability liktyp = input('Which likelihood?\n (g)aussian, (l)ogistic: ','s'); a = 0.3; b = 1.2; f = @(x) a*x + b + sin(x); % underlying function n = 30; sn = 0.5; % number of training points, noise standard deviation x = 2*rand(n,1)-1; x = 1+4*x+sign(x); y = f(x)+sn*randn(n,1); % sample data cov = {@covSEiso}; sf = 2; ell = 1.0; hyp.cov = log([ell;sf]); mean = {@meanSum,{@meanLinear,@meanConst}}; hyp.mean = [a;b]; if isequal(liktyp,'g') lik = {@likGauss}; hyp.lik = log(sn); inf = @infGaussLik; else lik = {@likLogistic}; hyp.lik = []; inf = @infLaplace; y = sign(y); end fprintf('Optimise hyperparameters.\n') hyp = minimize(hyp,@gp,-N,inf,mean,cov,lik,x,y); % optimise hyperparameters xs = linspace(-8,10,2e3)'; ys = f(xs); % exact function values [ymu,ys2] = gp(hyp,inf,mean,cov,lik,x,y,xs); % dense prediction [nlZ,dnlZ] = gp(hyp,inf,mean,cov,lik,x,y); % marginal likelihood and derivatives nu = 10; xu = linspace(-6,8,nu)'; covf = {@apxSparse,cov,xu}; % FITC prediction [ymuf,ys2f] = gp(hyp,inf,mean,covf,lik,x,y,xs); ng = 40; xg = linspace(-6,8,ng)'; covg = {@apxGrid,{cov},{xg}};% grid prediction opt.cg_maxit = 500; opt.cg_tol = 1e-5; opt.pred_var = 100; % parameters inf = @(varargin) infGrid(varargin{:},opt); [ymug,ys2g] = gp(hyp,inf,mean,covg,lik,x,y,xs); [postg,nlZg,dnlZg] = infGrid(hyp,mean,covg,lik,x,y,opt); % fast grid prediction [fmugf,fs2gf,ymugf,ys2gf] = postg.predict(xs); fprintf('Sampling estimators for nlZ and dnlZ\n') opt.ndcovs = 25; % ask for (additional) sampling-based (exact) derivatives opt.ldB2_cheby = true; opt.ldB2_cheby_hutch = 30; [posts,nlZs,dnlZs] = infGrid(hyp,mean,covg,lik,x,y,opt); [dnlZ.cov,dnlZg.cov,dnlZs.covs,dnlZs.cov] [nlZ,nlZg,nlZs] subplot(211) plot(xs,ymu,'k','LineWidth',2), hold on plot(xs,ymuf,'g-.','LineWidth',2) plot(xs,ymug,'m:','LineWidth',2) plot(xs,ymugf,'c--','LineWidth',2) legend('exact','FITC','grid','fast-grid'), title('Predictive mean') plot(x,y,'r+'), plot(xs,ys,'r') plot(xs,ymu+2*sqrt(ys2),'k'), plot(xs,ymu-2*sqrt(ys2),'k') xlim([-8,10]), ylim([-3,6]) subplot(212) plot(xs,sqrt(ys2),'k','LineWidth',2), hold on plot(xs,sqrt(ys2f),'g-.','LineWidth',2) plot(xs,sqrt(ys2g),'m:','LineWidth',2) plot(xs,sqrt(ys2gf),'c--','LineWidth',2) legend('exact','FITC','grid','fast-grid'), title('Predictive standard dev') xlim([-8,10]), if write_fig, print -depsc f10.eps; endgpml-matlab-v4.0-2016-10-19/doc/index.html000644 000765 000024 00000164612 12777177044 017653 0ustar00carlstaff000000 000000 Documentation for GPML Matlab Code

Documentation for GPML Matlab Code version 4.0

1) What?

The code provided here originally demonstrated the main algorithms from Rasmussen and Williams: Gaussian Processes for Machine Learning. It has since grown to allow more likelihood functions, further inference methods and a flexible framework for specifying GPs. Other GP packages can be found here.

The code is written by Carl Edward Rasmussen and Hannes Nickisch; it runs on both Octave 3.2.x and Matlab® 7.x and later. The code is based on previous versions written by Carl Edward Rasmussen and Chris Williams.

2) Download, Install and Documentation

All the code including demonstrations and html documentation can be downloaded in a tar or zip archive file.

Minor changes and incremental bugfixes to the current version are documented in the changelog, changes from previous versions are documented in README.

Please read the copyright notice.

After unpacking the tar or zip file you will find 7 subdirectories: cov, doc, inf, lik, mean, prior and util. It is not necessary to install anything to get started, just run the startup script to set your path.

Details about the directory contents and on how to compile mex files can be found in the README. The getting started guide is the remainder of the html file you are currently reading (also available at http://gaussianprocess.org/gpml/code/matlab/doc). A Developer's Guide containing technical documentation is found in manual.pdf, but for the casual user, the guide is below.

3) Getting Started

Gaussian Processes (GPs) can conveniently be used for Bayesian supervised learning, such as regression and classification. In its simplest form, GP inference can be implemented in a few lines of code. However, in practice, things typically get a little more complicated: you might want to use complicated covariance functions and mean functions, learn good values for hyperparameters, use non-Gaussian likelihood functions (rendering exact inference intractable), use approximate inference algorithms, or combinations of many or all of the above. This is what the GPML software package does.

The remainder of section 3 first presents some essential components of the software in the next subsection. This enables understand the usage for the gp function described in section 3b). A practical example follows in 3c), and the section concludes with a more detailed overview in 3d).

3a) Some Essential Components

Before going straight to the examples, just a brief note about the organization of the package. There are four essential types of objects which you need to know about:

Gaussian Process
A Gaussian Process is fully specified by a mean function and a covariance function. These functions are specified separately, and consist of a specification of a functional form as well as a set of parameters called hyperparameters, see below.
Mean functions
Several mean functions are available, an overview is provided by the meanFunctions help function (type help meanFunctions to get help), and an example is the meanLinear function.
Covariance functions
Several covariance functions are available, an overview is provided by the covFunctions help function (type help covFunctions to get help), and an example is the covSEard "Squared Exponential with Automatic Relevance Determination" covariance function.
For a more comprehensive overview of mean and covariance functions, see section 3d) below.
Hyperparameters
GPs are specified using mean and covariance functions which typically have free parameters called hyperparameters. Also likelihood functions may have such parameters. Hyperparameters are represented in a struct with the fields mean, cov and lik (some of which may be empty). It is important that the number of elements in each of the fields in the hyperparameter struct matches the specification of the mean, covariance and likelihood functions.
Likelihood Functions
The likelihood function specifies the probability of the observations given the GP (and the hyperparameters). Several likelihood functions are available, an overview is provided by the likFunctions help function (type help likFunctions to get help). Example likelihood functions include likGauss the Gaussian likelihood for regression and likLogistic the logistic likelihood for classification.
Inference Methods
The inference methods specify how to compute with the model, i.e. how to infer the (approximate) posterior process, how to find hyperparameters, evaluate the log marginal likelihood and how to make predictions. Several inference methods are avaiable, and overview is provided by the infMethods help file (type help infMethods to get help). An example inference method is infGaussLik for exact inference (regression with Gaussian likelihood).

3b) The gp Function

Using the GPML package is simple, there is only one single function to call: gp, it does posterior inference, learns hyperparameters, computes the marginal likelihood and makes predictions. Generally, the gp function takes the following arguments: a hyperparameter struct, an inference method, a mean function, a covariance function, a likelihood function, training inputs, training targets, and possibly test cases. The exact computations done by the function is controlled by the number of input and output arguments in the call. Here is part of the help message for the gp function (follow the link to see the whole thing):

  function [varargout] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys)

  [ ... snip ... ]

  Two modes are possible: training or prediction: if no test cases are
  supplied, then the negative log marginal likelihood and its partial
  derivatives wrt the hyperparameters is computed; this mode is used to fit the
  hyperparameters. If test cases are given, then the test set predictive
  probabilities are returned. Usage:

    training: [nlZ dnlZ          ] = gp(hyp, inf, mean, cov, lik, x, y);
  prediction: [ymu ys2 fmu fs2   ] = gp(hyp, inf, mean, cov, lik, x, y, xs);
          or: [ymu ys2 fmu fs2 lp] = gp(hyp, inf, mean, cov, lik, x, y, xs, ys);

  [ ... snip ... ]

Here x and y are training inputs and outputs, and xs and ys are test set inputs and outputs, nlZ is the negative log marginal likelihood and dnlZ its partial derivatives wrt the hyperparameters (which are used for training the hyperparameters). The prediction outputs are ymu and ys2 for test output mean and covariance, and fmu and fs2 are the equivalent quenteties for the corresponding latent variables. Finally, lp are the test output log probabilities.

3c) A Practical Example

To get started, let's consider the simple example of one-dimensional non-linear regression on data corrupted by Gaussian noise. Our data set is 
  x = randn(20, 1);                 % 20 training inputs
  y = sin(5*x) + 0.1*randn(20, 1);  % 20 noisy training targets
  xs = linspace(-3, 3, 41);         % 41 test inputs
We need specify the mean, covariance and likelihood functions 
  meanfunc = [];                    % empty: don't use a mean function
  covfunc = @covSEiso;              % Squared Exponental covariance function
  likfunc = @likGauss;              % Gaussian likelihood
Finally we initialize the hyperparameter struct 
  hyp = struct('mean', [], 'cov', [0 0], 'lik', -1);

Note, that the hyperparameter struct must have the three fields mean, cov and lik. Each field must have the number of elements which corresponds to the functions specified. In our case, the mean function is empty, so takes no parameters. The covariance function is covSEiso, the squared exponential with isotropic distance measure, which takes two parameters (see help covSEiso). As explained in the help for the function, the meaning of the hyperparameters is "log of the length-scale" and the "log of the signal std dev". Initializing both of these to zero, corresponds to length-scale and signal std dev to be initialized to one. The Gaussian likelihood function has a single parameter, which is the log of the noise standard deviation, setting the log to zero corresponds to a standard deviation of exp(-1)=0.37.

A common situation with modeling with GPs is that approprate settings of the hyperparameters are not known a priori. The situation is reflected in the above initialization of the hyperparameters, where the values values are specified without careful justification, perhaps based on some vague notions of the magnitudes likely to be involved. Thus, a common task is to set hyperparameters by optimizing the (log) marginal likelihood. This is done as follows 

  hyp2 = minimize(hyp, -100, @gp, @infExact, meanfunc, covfunc, likfunc, x, y);
The minimize function minimizes the negative log marginal likelihood, which is returned by the gp function, together with the partial derivatives wrt the hyperparameters. The inference method is specified to be infExact exact inference. The minimize function is allowed a computational budget of 100 function evaluations. The hyperparameters found are 
  hyp2 =
To make predictions using these hyperparameters 
  [mu s2] = gp(hyp2, @infExact, meanfunc, covfunc, likfunc, x, y, xs);
To plot the predictive mean at the test points together with the predictive 95% confidence bounds and the training data 
  f = [mu+2*sqrt(s2); flipdim(mu-2*sqrt(s2),1)];
  fill([xs; flipdim(xs,1)], f, [7 7 7]/8)
  hold on; plot(xs, m); plot(x, y, '+')

3d) A More Detailed Overview

The previous section shows a minimalist example, using the central concepts of GPML. This section provides a less simplistic overview, mainly through a number of useful comments and pointers to more complete treatments.

In order to be able to find things, the toolbox is organized into the following directories mean for mean functions, cov for covariance functions, lik for likelihood functions, inf for inference methods prior for priors and mcmc for Markov Chain monte Carlo tools, doc for documentation and util for general utilities.

In addition to this structure, the naming of functions within some of these directories also start with the letters mean, cov, lik and inf as a further mnemonic aid.

The following paragraphs contain useful further details about some of the concepts we have already used.

Mean functions and covariance functions. As detailed in meanFunctions and covFunctions there are actually two types of these, simple and composite. Composite functions are used to compose simple functions into more expressive structures. For example 

  meanfunc = {@meanSum, {@meanLinear, @meanConst}};
specifies a mean function which is a sum of a linear and a constant part. Note how composite functions are specified using cell arrays. Note also, that the corresponding mean hyperparameter will consist of a vector containing the concatenation of the different parts of the mean function. If you call a mean or covariance function without arguments, they will return a string indicating the number of hyperparameters expected; this also works for composite covariance functions; the letter D in the string designates the dimension of the inputs.

Likelihood functions. As detailed in likFunctions there are also simple and composite likelihood functions; the only composite likelihood function is likMix, which implements a mixture of multiple likelihoods.

Inference methods.

Whereas all mean functions and covariance functions may be used in any context, there are some restrictions on which likelihood functions may be used with which inference method. An exhaustive compatibility matrix between likelihoods (rows) and inference methods (columns) is given in the table below:
Plain Regression Likelihoods
Inference GPML name Gaussian noise
infGaussLik		 EP
infEP		 Laplace
infLaplace		 Variational Bayes
infVB		 Kullback Leibler
infKL		 Sampling
infMCMC		 LOO
infLOO		 type, output domain
 		 alternative names
 
Likelihood
Gaussian likGauss $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR  
Warped Gaussian likGaussWarp   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR  
Gumbel likGumbel     $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ regression, IR extremal value regression
Sech squared likSech2   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR logistic distribution
Laplacian likLaplace   $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR double exponential
Student's t likT     $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ regression, IR  
Composite Likelihoods
Inference GPML name Gaussian noise
infGaussLik		 EP
infEP		 Laplace
infLaplace		 Variational Bayes
infVB		 Kullback Leibler
infKL		 Sampling
infMCMC		 LOO
infLOO		 type, output domain
 		 alternative names
 
Likelihood
Mixture likMix   $\checkmark$ $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ classification, ±1 and regression, IR mixing meta likelihood
Classification Likelihoods
Inference GPML name Gaussian noise
infGaussLik		 EP
infEP		 Laplace
infLaplace		 Variational Bayes
infVB		 Kullback Leibler
infKL		 Sampling
infMCMC		 LOO
infLOO		 type, output domain
 		 alternative names
 
Likelihood
Uniform likUni   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ classification, ±1 label noise
Error function likErf   $\checkmark$ $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ classification, ±1 probit regression
Logistic function likLogistic   $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ $\checkmark$ classification, ±1 logistic regression
logit regression
Generalized Linear Model Likelihoods
Inference GPML name Gaussian noise
infGaussLik		 EP
infEP		 Laplace
infLaplace		 Variational Bayes
infVB		 Kullback Leibler
infKL		 Sampling
infMCMC		 LOO
infLOO		 type, output domain
 		 alternative names
 
Likelihood
Weibull likWeibull     $\checkmark$     $\checkmark$ $\checkmark$ positive data, IR+\{0} nonnegative regression
Gamma likGamma     $\checkmark$     $\checkmark$ $\checkmark$ positive data, IR+\{0} nonnegative regression
Exponential likExp     $\checkmark$     $\checkmark$ $\checkmark$ positive data, IR+\{0} nonnegative regression
Inverse Gaussian likInvGauss     $\checkmark$     $\checkmark$ $\checkmark$ positive data, IR+\{0} nonnegative regression
Poisson likPoisson   ($\checkmark$)* $\checkmark$   $\checkmark$ $\checkmark$ $\checkmark$ count data, IN Poisson regression
Beta likBeta     $\checkmark$     $\checkmark$ $\checkmark$ interval data, [0,1] range regression

* EP might not converge in some cases since quadrature is used.

All of the objects described above are written in a modular way, so you can add functionality if you feel constrained despite the considerable flexibility provided. Details about how to do this are provided in the developer documentation.

Inference by MCMC sampling is the only inference method that cannot be used as a black box. Also gradient-based marginal likelihood optimisation is not possible with MCMC. Please see usageSampling for a toy example illustrating the usage of the implemented samplers.

4) Practice

Instead of exhaustively explaining all the possibilities, we will give two illustrative examples to give you the idea; one for regression and one for classification. You can either follow the example here on this page, or using the two scripts demoRegression and demoClassification (using the scripts, you still need to follow the explanation on this page).

4a) Simple Regression

You can either follow the example here on this page, or use the script demoRegression.

This is a simple example, where we first generate n=20 data points from a GP, where the inputs are scalar (so that it is easy to plot what is going on). We then use various other GPs to make inferences about the underlying function.

First, generate some data from a Gaussian process (it is not essential to understand the details of this):

  clear all, close all
 
  meanfunc = {@meanSum, {@meanLinear, @meanConst}}; hyp.mean = [0.5; 1];
  covfunc = {@covMaterniso, 3}; ell = 1/4; sf = 1; hyp.cov = log([ell; sf]);
  likfunc = @likGauss; sn = 0.1; hyp.lik = log(sn);
 
  n = 20;
  x = gpml_randn(0.3, n, 1);
  K = feval(covfunc{:}, hyp.cov, x);
  mu = feval(meanfunc{:}, hyp.mean, x);
  y = chol(K)'*gpml_randn(0.15, n, 1) + mu + exp(hyp.lik)*gpml_randn(0.2, n, 1);

  plot(x, y, '+')

Above, we first specify the mean function meanfunc, covariance function covfunc of a GP and a likelihood function, likfunc. The corresponding hyperparameters are specified in the hyp structure:

The mean function is composite, adding (using meanSum function) a linear (meanLinear) and a constant (meanConst) to get an affine function. Note, how the different components are composed using cell arrays. The hyperparameters for the mean are given in hyp.mean and consists of a single (because the input will one dimensional, i.e. D=1) slope (set to 0.5) and an off-set (set to 1). The number and the order of these hyperparameters conform to the mean function specification. You can find out how many hyperparameters a mean (or covariance or likelihood function) expects by calling it without arguments, such as feval(meanfunc{:}). For more information on mean functions see meanFunctions and the directory mean/.

The covariance function is of the Matérn form with isotropic distance measure covMaterniso. This covariance function is also composite, as it takes a constant (related to the smoothness of the GP), which in this case is set to 3. The covariance function takes two hyperparameters, a characteristic length-scale ell and the standard deviation of the signal sf. Note, that these positive parameters are represented in hyp.cov using their logarithms. For more information on covariance functions see covFunctions and cov/.

Finally, the likelihood function is specified to be Gaussian. The standard deviation of the noise sn is set to 0.1. Again, the representation in the hyp.lik is given in terms of its logarithm. For more information about likelihood functions, see likFunctions and lik/.

Then, we generate a dataset with n=20 examples. The inputs x are drawn from a unit Gaussian (using the gpml_randn utility, which generates unit Gaussian pseudo random numbers with a specified seed). We then evaluate the covariance matrix K and the mean vector m by calling the corresponding functions with the hyperparameters and the input locations x. Finally, the targets y are computed by drawing randomly from a Gaussian with the desired covariance and mean and adding Gaussian noise with standard deviation exp(hyp.lik). The above code is a bit special because we explicitly call the mean and covariance functions (in order to generate samples from a GP); ordinarily, we would only directly call the gp function.

f1.gif

Let's ask the model to compute the (joint) negative log probability (density) nlml (also called marginal likelihood or evidence) and to generalize from the training data to other (test) inputs z:

  nlml = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y)

  z = linspace(-1.9, 1.9, 101)';
  [m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z);

  f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)]; 
  fill([z; flipdim(z,1)], f, [7 7 7]/8)
  hold on; plot(z, m); plot(x, y, '+')

The gp function is called with a struct of hyperparameters hyp, and inference method, in this case @infExact for exact inference and the mean, covariance and likelihood functions, as well as the inputs and outputs of the training data. With no test inputs, gp returns the negative log probability of the training data, in this example nlml=11.97.

To compute the predictions at test locations we add the test inputs z as a final argument, and gp returns the mean m variance s2 at the test location. The program is using algorithm 2.1 from the GPML book. Plotting the mean function plus/minus two standard deviations (corresponding to a 95% confidence interval):

f2.gif

Typically, we would not a priori know the values of the hyperparameters hyp, let alone the form of the mean, covariance or likelihood functions. So, let's pretend we didn't know any of this. We assume a particular structure and learn suitable hyperparameters:

  covfunc = @covSEiso; hyp2.cov = [0; 0]; hyp2.lik = log(0.1);

  hyp2 = minimize(hyp2, @gp, -100, @infExact, [], covfunc, likfunc, x, y);
  exp(hyp2.lik)
  nlml2 = gp(hyp2, @infExact, [], covfunc, likfunc, x, y)

  [m s2] = gp(hyp2, @infExact, [], covfunc, likfunc, x, y, z);
  f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)];
  fill([z; flipdim(z,1)], f, [7 7 7]/8)
  hold on; plot(z, m); plot(x, y, '+')

First, we guess that a squared exponential covariance function covSEiso may be suitable. This covariance function takes two hyperparameters: a characteristic length-scale and a signal standard deviation (magnitude). These hyperparameters are non-negative and represented by their logarithms; thus, initializing hyp2.cov to zero, correspond to unit characteristic length-scale and unit signal standard deviation. The likelihood hyperparameter in hyp2.lik is also initialized. We assume that the mean function is zero, so we simply ignore it (and when in the following we call gp, we give an empty argument for the mean function).

In the following line, we optimize over the hyperparameters, by minimizing the negative log marginal likelihood w.r.t. the hyperparameters. The third parameter in the call to minimize limits the number of function evaluations to a maximum of 100. The inferred noise standard deviation is exp(hyp2.lik)=0.15, somewhat larger than the one used to generate the data (0.1). The final negative log marginal likelihood is nlml2=14.13, showing that the joint probability (density) of the training data is about exp(14.13-11.97)=8.7 times smaller than for the setup actually generating the data. Finally, we plot the predictive distribution.

f3.gif

This plot shows clearly, that the model is indeed quite different from the generating process. This is due to the different specifications of both the mean and covariance functions. Below we'll try to do a better job, by allowing more flexibility in the specification.

Note that the confidence interval in this plot is the confidence for the distribution of the (noisy) data. If instead you want the confidence region for the underlying function, you should use the 3rd and 4th output arguments from gp as these refer to the latent process, rather than the data points.

  hyp.cov = [0; 0]; hyp.mean = [0; 0]; hyp.lik = log(0.1);
  hyp = minimize(hyp, @gp, -100, @infExact, meanfunc, covfunc, likfunc, x, y);
  [m s2] = gp(hyp, @infExact, meanfunc, covfunc, likfunc, x, y, z);
 
  f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)];
  fill([z; flipdim(z,1)], f, [7 7 7]/8)
  hold on; plot(z, m); plot(x, y, '+');

Here, we have changed the specification by adding the affine mean function. All the hyperparameters are learnt by optimizing the marginal likelihood.

f4.gif

This shows that a much better fit is achieved when allowing a mean function (although the covariance function is still different from that of the generating process).

Guiding marginal likelihood optimisation with a hyperprior

It can be usefull to put a prior distribution on (a part of) the hyperparameters. Sometimes, one may want to exclude some hyperparameters from the optimisation i.e. fix their values beforehand and treat them as constants.

In these cases, a hyperprior comes to bear. A hyperprior is specified by augmenting the inf parameter of gp.m In the regression before, we had inf = @infExact;. To put a Gaussian prior on the first mean hyperparameter hyp.mean(1) and a Laplacian prior on the second mean hyperparameter hyp.mean(2) and wished to fix the noise variance hyperparameter hyp.lik, we simple need to set up the corresponding prior structure as detailed below. 

  mu = 1.0; s2 = 0.01^2;
  prior.mean = {{@priorGauss,mu,s2}; {'priorLaplace',mu,s2}};
  prior.lik = {{@priorDelta}};
  inf = {@infPrior,@infExact,prior};
  hyp = minimize(hyp, @gp, -100, inf, meanfunc, covfunc, likfunc, x, y);

Further examples are provided in usagePrior.

4b) Sparse Approximate Regression

In case the number of training inputs x exceeds a few thousands, exact inference takes too long. We offer the FITC approximation based on a low-rank plus diagonal approximation to the exact covariance to deal with these cases. The general idea is to use inducing points u and to base the computations on cross-covariances between training, test and inducing points only.

Using the FITC approximation is very simple, we just have to wrap the covariance function covfunc into apxSparse.m and call gp.m with the inference method infGaussLik.m as demonstrated by the following lines of code. 

  nu = fix(n/2); u = linspace(-1.3,1.3,nu)';
  covfuncF = {@apxSparse, {covfunc}, u};
  [mF s2F] = gp(hyp, @infGaussLik, meanfunc, covfuncF, likfunc, x, y, z);

We define equispaced inducing points u that are shown in the figure as black circles. Note that the predictive variance is overestimated outside the support of the inducing inputs. In a multivariate example where densely sampled inducing inputs are infeasible, one can simply use a random subset of the training points. 

  nu = fix(n/2); iu = randperm(n); iu = iu(1:nu); u = x(iu,:);
f5.gif

4c) Large scale regression exploiting the grid structure

A covariance function factorising over coordinates axes leads to a covariance matrix with Kronecker structure. This can be exploited to scale GPs beyond the O(n³) limit. If there are observations for all points of the grid, all computations are exact. The example below contains the most relevant code from the script demoGrid2d, where we extrapolate a pixel image beyond its boundaries. An instructive example in 1d can be found in demoGrid1d.

For a comprehensive set of examples and more resources, see a website by Andrew Wilson.

We start off by setting up the training data and the GP on a [-2,2]x[-3,3] lattice with 14400 pixels -- a size where a usual dens GP would be computationally infeasible. Then we wrap the covariance functions for the axes into apxGrid and specify the initial values for the hyperparameters. 

  xg = {linspace(-2,2,120)',linspace(-3,3,120)'}; xe = apxGrid('expand',xg);
  x = (1:size(xe,1))'; y = sin(xe(:,2)) + xe(:,1) + 0.01*gpml_randn(1,size(x));
  cov = {{@covSEiso},{@covSEiso}}; covg = {@apxGrid,cov,xg};
  hyp.cov = zeros(4,1); hyp.mean = []; hyp.lik = log(0.1);
Note that one can create a grid of size 120x130 from scratch covering a given data set x using the command: 
  xg = apxGrid('create',x,true,[120,130]);

As a next step, we perform hyperparameter optimisation as with a usual GP model. The inference method infGaussLik only supports likGauss, hence we can leave the field empty. 

  hyp = minimize(hyp,@gp,-50,@infGaussLik,@meanZero,covg,[],x,y);

Finally, we predict on a [-4,4]x[-6,6] lattice to see the result of the extrapolation. The figure shows on the left the training data and on the right the prediction. Note that, we also support partial observations. 

  xs = apxGrid('expand',{linspace(-4,4,100)',linspace(-6,6,110)'});
  opt.cg_maxit = 200; opt.cg_tol = 5e-3;
  post = apxGrid(hyp,{@meanZero},covg,'likGauss',x,y,opt); post.L = @(a) a;
  ym = gp(hyp, @apxGrid,[],covg,[],x,post,xs);
f9.png

4d) Exercises for the reader

Inference Methods
Try using Expectation Propagation instead of exact inference in the above, by exchanging @infExact with @infEP. You get exactly identical results, why?
Mean or Covariance
Try training a GP where the affine part of the function is captured by the covariance function instead of the mean function. That is, use a GP with no explicit mean function, but further additive contributions to the covariance. How would you expect the marginal likelihood to compare to the previous case?

4e) Classification

You can either follow the example here on this page, or use the script demoClassification.

The difference between regression and classification isn't of fundamental nature. We can use a Gaussian process latent function in essentially the same way, it is just that the Gaussian likelihood function often used for regression is inappropriate for classification. And since exact inference is only possible for Gaussian likelihood, we also need an alternative, approximate, inference method.

Here, we will demonstrate binary classification, using two partially overlapping Gaussian sources of data in two dimensions. First we generate the data:

  clear all, close all
 
  n1 = 80; n2 = 40;                   % number of data points from each class
  S1 = eye(2); S2 = [1 0.95; 0.95 1];           % the two covariance matrices
  m1 = [0.75; 0]; m2 = [-0.75; 0];                            % the two means
 
  x1 = bsxfun(@plus, chol(S1)'*gpml_randn(0.2, 2, n1), m1);
  x2 = bsxfun(@plus, chol(S2)'*gpml_randn(0.3, 2, n2), m2);
 
  x = [x1 x2]'; y = [-ones(1,n1) ones(1,n2)]';
  plot(x1(1,:), x1(2,:), 'b+'); hold on;
  plot(x2(1,:), x2(2,:), 'r+');

120 data points are generated from two Gaussians with different means and covariances. One Gaussian is isotropic and contains 2/3 of the data (blue), the other is highly correlated and contains 1/3 of the points (red). Note, that the labels for the targets are ±1 (and not 0/1).

In the plot, we superimpose the data points with the posterior equi-probability contour lines for the probability of class two given complete information about the generating mechanism

  [t1 t2] = meshgrid(-4:0.1:4,-4:0.1:4);
  t = [t1(:) t2(:)]; n = length(t);                 % these are the test inputs
  tmm = bsxfun(@minus, t, m1');
  p1 = n1*exp(-sum(tmm*inv(S1).*tmm/2,2))/sqrt(det(S1));
  tmm = bsxfun(@minus, t, m2');
  p2 = n2*exp(-sum(tmm*inv(S2).*tmm/2,2))/sqrt(det(S2));
  contour(t1, t2, reshape(p2./(p1+p2), size(t1)), [0.1:0.1:0.9]);
f6.gif

We specify a Gaussian process model as follows: a constant mean function, with initial parameter set to 0, a squared exponential with automatic relevance determination (ARD) covariance function covSEard. This covariance function has one characteristic length-scale parameter for each dimension of the input space, and a signal magnitude parameter, for a total of 3 parameters (as the input dimension is D=2). ARD with separate length-scales for each input dimension is a very powerful tool to learn which inputs are important for predictions: if length-scales are short, inputs are very important, and when they grow very long (compared to the spread of the data), the corresponding inputs will be largely ignored. Both length-scales and the signal magnitude are initialized to 1 (and represented in the log space). Finally, the likelihood function likErf has the shape of the error-function (or cumulative Gaussian), which doesn't take any hyperparameters (so hyp.lik does not exist).

  meanfunc = @meanConst; hyp.mean = 0;
  covfunc = @covSEard; ell = 1.0; sf = 1.0; hyp.cov = log([ell ell sf]);
  likfunc = @likErf;

  hyp = minimize(hyp, @gp, -40, @infEP, meanfunc, covfunc, likfunc, x, y);
  [a b c d lp] = gp(hyp, @infEP, meanfunc, covfunc, likfunc, x, y, t, ones(n, 1));

  plot(x1(1,:), x1(2,:), 'b+'); hold on; plot(x2(1,:), x2(2,:), 'r+')
  contour(t1, t2, reshape(exp(lp), size(t1)), [0.1:0.1:0.9]);

We train the hyperparameters using minimize, to minimize the negative log marginal likelihood. We allow for 40 function evaluations, and specify that inference should be done with the Expectation Propagation (EP) inference method @infEP, and pass the usual parameters. Training is done using algorithm 3.5 and 5.2 from the gpml book. When computing test probabilities, we call gp with additional test inputs, and as the last argument a vector of targets for which the log probabilities lp should be computed. The fist four output arguments of the function are mean and variance for the targets and corresponding latent variables respectively. The test set predictions are computed using algorithm 3.6 from the GPML book. The contour plot for the predictive distribution is shown below. Note, that the predictive probability is fairly close to the probabilities of the generating process in regions of high data density. Note also, that as you move away from the data, the probability approaches 1/3, the overall class probability.

f7.gif

Examining the two ARD characteristic length-scale parameters after learning, you will find that they are fairly similar, reflecting the fact that for this data set, both inputs important.

Large scale classification using the FITC approximation

In case the number of training inputs x exceeds a few hundreds, approximate inference using infLaplace.m, infEP.m and infVB.m takes too long. As in regression, we offer the FITC approximation based on a low-rank plus diagonal approximation to the exact covariance to deal with these cases. The general idea is to use inducing points u and to base the computations on cross-covariances between training, test and inducing points only.

Using the FITC approximation is very simple, we just have to wrap the covariance function covfunc into apxSparse.m and call gp.m with the inference methods infLaplace.m as demonstrated by the following lines of code. 

  [u1,u2] = meshgrid(linspace(-2,2,5)); u = [u1(:),u2(:)]; clear u1; clear u2
  nu = size(u,1);
  covfuncF = {@apxSparse, {covfunc}, u};
  inffunc = @infLaplace;
  hyp = minimize(hyp, @gp, -40, inffunc, meanfunc, covfuncF, likfunc, x, y);
  [a b c d lp] = gp(hyp, inffunc, meanfunc, covfuncF, likfunc, x, y, t, ones(n,1));

We define equispaced inducing points u that are shown in the figure as black circles. Alternatively, a random subset of the training points can be used as inducing points.

f8.gif

Large scale classification exploiting the grid structure

Similar to regression using infGaussLik, we can perform approximate inference using infLaplace to scale GPs beyond the O(n³) limit. Please visit the website by Seth Flaxman for an extended example and related datasets.

Exercise for the reader

Inference Methods
Use the Laplace Approximation for inference @infLaplace, and compare the approximate marginal likelihood for the two methods. Which approximation is best?
Covariance Function
Try using the squared exponential with isotropic distance measure covSEiso instead of ARD distance measure covSEard. Which is best?

5) Acknowledgements

Innumerable colleagues have helped to improve this software. Some of these are: John Cunningham, Máté Lengyel, Joris Mooij, Iain Murray, David Duvenaud, Andrew McHutchon, Rowan McAllister, Daniel Marthaler, Giampiero Salvi, Mark van der Wilk, Marco Fraccaro, Dali Wei, Ernst Kloppenburg, Ryan Turner, Seth Flaxman and Chris Williams. Especially Ed Snelson helped to improve the code and to include sparse approximations and Roman Garnett and José Vallet helped to include hyperparameter priors. The spectral mixture covariance function and the grid-based inference were contributed by Andrew Gordon Wilson and periodic covariances were added by James Robert Lloyd.

Last modified: October 19th 2016 gpml-matlab-v4.0-2016-10-19/doc/changelog000644 000765 000024 00000000401 12777177044 017511 0ustar00carlstaff000000 000000 GAUSSIAN PROCESS REGRESSION AND CLASSIFICATION Toolbox version 4.0 for GNU Octave 3.2.x and Matlab 7.x Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-19. Changes and small incremental fixes applied since the last major release. gpml-matlab-v4.0-2016-10-19/doc/usageRegression.m000644 000765 000024 00000005270 12761060376 021163 0ustar00carlstaff000000 000000 % demonstrate usage of regression % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2016-08-29. clear all, close all %% SAY WHICH CODE WE WISH TO EXERCISE id = [1,1]; % use Gauss/Exact id = [1,2; 3,2; 4,2]; % compare Laplace id = [1,3; 2,3; 3,3]; % study EP id = [1,5; 2,5]; % look into KL (takes quite a while) id = [1,4; 2,4; 3,4; 4,4]; % deal with VB seed = 197; randn('seed',seed), rand('seed',seed) ntr = 50; nte = 1e4; % number of training and test points xtr = 10*sort(rand(ntr,1)); % sample dataset f = @(x) sin(x)+sqrt(x); % "true" underlying function sn = 0.2; ytr = f(xtr) + randn(ntr,1)*sn; % add Gaussian noise i = randperm(ntr); nout = 3; % add outliers ytr(i(1:nout)) = 5; xte = linspace(0,10,1e4)'; % support, we test our function on cov = {@covSEiso}; sf = 1; ell = 0.4; % setup the GP hyp0.cov = log([ell;sf]); mean = {@meanSum,{@meanLinear,@meanConst}}; a = 1/5; b = 1; % m(x) = a*x+b hyp0.mean = [a;b]; lik_list = {'likGauss','likLaplace','likSech2','likT'}; % possible likelihoods inf_list = {'infGaussLik','infLaplace','infEP','infVB','infKL'};% inference algs Ncg = 50; % number of conjugate gradient steps sdscale = 0.5; % how many sd wide should the error bars become? col = {'k',[.8,0,0],[0,.5,0],'b',[0,.75,.75],[.7,0,.5]}; % colors ymu{1} = f(xte); ys2{1} = sn^2; nlZ(1) = -Inf; for i=1:size(id,1) lik = lik_list{id(i,1)}; % setup the likelihood if strcmp(lik,'likT') nu = 4; hyp0.lik = log([nu-1;sqrt((nu-2)/nu)*sn]); else hyp0.lik = log(sn); end inf = inf_list{id(i,2)}; fprintf('OPT: %s/%s\n',lik_list{id(i,1)},inf_list{id(i,2)}) if Ncg==0 hyp = hyp0; else hyp = minimize(hyp0,'gp', -Ncg, inf, mean, cov, lik, xtr, ytr); % opt hypers end [ymu{i+1}, ys2{i+1}] = gp(hyp, inf, mean, cov, lik, xtr, ytr, xte); % predict [nlZ(i+1)] = gp(hyp, inf, mean, cov, lik, xtr, ytr); end figure, hold on for i=1:size(id,1)+1 plot(xte,ymu{i},'Color',col{i},'LineWidth',2) if i==1 leg = {'function'}; else leg{end+1} = sprintf('%s/%s -lZ=%1.2f',... lik_list{id(i-1,1)},inf_list{id(i-1,2)},nlZ(i)); end end for i=1:size(id,1)+1 ysd = sdscale*sqrt(ys2{i}); fill([xte;flipud(xte)],[ymu{i}+ysd;flipud(ymu{i}-ysd)],... col{i},'EdgeColor',col{i},'FaceAlpha',0.1,'EdgeAlpha',0.3); end for i=1:size(id,1)+1, plot(xte,ymu{i},'Color',col{i},'LineWidth',2), end plot(xtr,ytr,'k+'), plot(xtr,ytr,'ko'), legend(leg)gpml-matlab-v4.0-2016-10-19/doc/gpml_randn.m000644 000765 000024 00000002125 12701705600 020121 0ustar00carlstaff000000 000000 function x = gpml_randn(seed,varargin) % Generate pseudo-random numbers in a quick and dirty way. % The function makes sure, we obtain the same random numbers using Octave and % Matlab for the demo scripts. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2010-07-07. if nargin==2 sz = varargin{1}; else sz = zeros(1,nargin-1); for i=1:nargin-1 sz(i) = varargin{i}; end end n = prod(sz); N = ceil(n/2)*2; % minimal uniform random number generator for uniform deviates from [0,1] % by Park and Miller a = 7^5; m = 2^31-1; % using Schrage's algorithm q = fix(m/a); r = mod(m,a); % m = a*q+r u = zeros(N+1,1); u(1) = fix(seed*2^31); for i=2:N+1 % Schrage's algorithm for mod(a*u(i),m) u(i) = a*mod(u(i-1),q) - r*fix(u(i-1)/q); if u(i)<0, u(i) = u(i)+m; end end u = u(2:N+1)/2^31; % Box-Muller transform: Numerical Recipies, 2nd Edition, $7.2.8 % http://en.wikipedia.org/wiki/Box-Muller_transform w = sqrt(- 2*log(u(1:N/2))); % split into two groups x = [w.*cos(2*pi*u(N/2+1:N)); w.*sin(2*pi*u(N/2+1:N))]; x = reshape(x(1:n),sz); gpml-matlab-v4.0-2016-10-19/doc/usageCov.m000644 000765 000024 00000006762 12777227613 017607 0ustar00carlstaff000000 000000 % demonstrate usage of covariance functions % % See also covFunctions.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-11. % File automatically generated using noweb. clear all, close all n = 5; D = 3; x = randn(n,D); xs = randn(3,D); % create a data set % set up simple covariance functions cn = {'covNoise'}; sn = .1; hypn = log(sn); % one hyperparameter cc = {@covConst}; sf = 2; hypc = log(sf); % function handles OK ce = {@covEye}; hype = []; % identity cl = {@covLIN}; hypl = []; % linear is parameter-free cla = {'covLINard'}; L = rand(D,1); hypla = log(L); % linear (ARD) cli = {'covLINiso'}; l = rand(1); hypli = log(l); % linear iso clo = {@covLINone}; ell = .9; hyplo = log(ell); % linear with bias cp = {@covPoly,3}; c = 2; hypp = log([c;sf]); % third order poly cga = {@covSEard}; hypga = log([L;sf]); % Gaussian with ARD cgi = {'covSEiso'}; hypgi = log([ell;sf]); % isotropic Gaussian cgu = {'covSEisoU'}; hypgu = log(ell); % isotropic Gauss no scale cra = {'covRQard'}; al = 2; hypra = log([L;sf;al]); % ration. quad. cri = {@covRQiso}; hypri = log([ell;sf;al]); % isotropic cma = {@covMaternard,5}; hypma = log([ell;sf]); % Matern class d=5 cmi = {'covMaterniso',3}; hypmi = log([ell;sf]); % Matern class d=3 cnn = {'covNNone'}; hypnn = log([L;sf]); % neural network cpe = {'covPeriodic'}; p = 2; hyppe = log([ell;p;sf]); % periodic cpn = {'covPeriodicNoDC'}; p = 2; hyppe = log([ell;p;sf]); % w/o DC cpc = {'covCos'}; p = 2; hypcpc = log([p;sf]); % cosine cov cca = {'covPPard',3}; hypcc = hypgu;% compact support poly degree 3 cci = {'covPPiso',2}; hypcc = hypgi;% compact support poly degree 2 cgb = {@covGaboriso}; ell = 1; p = 1.2; hypgb=log([ell;p]); % Gabor Q = 2; w = ones(Q,1)/Q; m = rand(D,Q); v = rand(D,Q); csm = {@covSM,Q}; hypsm = log([w;m(:);v(:)]); % Spectral Mixture cvl = {@covSEvlen,{@meanLinear}}; hypvl = [1;2;1; 0]; % var lenscal s = 12; cds = {@covDiscrete,s}; % discrete covariance function L = randn(s); L = chol(L'*L); L(1:(s+1):end) = log(diag(L)); hypds = L(triu(true(s))); xd = randi([1,s],[n,1]); xsd = [1;3;6]; cfa = {@covSEfact,2}; hypfa = randn(D*2,1); % factor analysis % set up composite i.e. meta covariance functions csc = {'covScale',{cgu}}; hypsc = [log(3); hypgu]; % scale by 9 csu = {'covSum',{cn,cc,cl}}; hypsu = [hypn; hypc; hypl]; % sum cpr = {@covProd,{cc,cci}}; hyppr = [hypc; hypcc]; % product mask = [0,1,0]; % binary mask excluding all but the 2nd component cma = {'covMask',{mask,cgi{:}}}; hypma = hypgi; % isotropic periodic rational quadratic cpi = {'covPERiso',{@covRQiso}}; % periodic Matern with ARD cpa = {'covPERard',{@covMaternard,3}}; % additive based on SEiso using unary and pairwise interactions cad = {'covADD',{[1,2],'covSEiso'}}; % preference covariance with squared exponential base covariance cpr = {'covPref',{'covSEiso'}}; hyppr = [0;0]; xp = randn(n,2*D); xsp = randn(3,2*D); % 0) specify a covariance function % cov = cma; hyp = hypma; % cov = cci; hyp = hypcc; % cov = csm; hyp = hypsm; cov = cds; hyp = hypds; x = xd; xs = xsd; % cov = cfa; hyp = hypfa; % cov = cvl; hyp = hypvl; % cov = cpr; hyp = hyppr; x = xp; xs = xsp; % 1) query the number of parameters feval(cov{:}) % 2) evaluate the function on x [K,dK] = feval(cov{:},hyp,x) % 3) evaluate the function on x and xs to get cross-terms [kss,dkss] = feval(cov{:},hyp,xs,'diag') [Ks, dKs ] = feval(cov{:},hyp,x,xs) gpml-matlab-v4.0-2016-10-19/doc/usagePrior.m000644 000765 000024 00000006620 12777227614 020145 0ustar00carlstaff000000 000000 % demonstrate usage of prior distributions % % See also priorDistributions.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-11. % File automatically generated using noweb. clear all, close all % 1) specify some priors % a) univariate priors mu = 1.0; s2 = 0.01^2; nu = 3; pg = {@priorGauss,mu,s2}; % Gaussian prior pl = {'priorLaplace',mu,s2}; % Laplace prior pt = {@priorT,mu,s2,nu}; % Student's t prior p1 = {@priorSmoothBox1,0,3,15}; % smooth box constraints lin decay p2 = {@priorSmoothBox2,0,2,15}; % smooth box constraints qua decay pd = {'priorDelta'}; % fix value of prior exclude from optimisation pc = {@priorClamped}; % equivalent to above lam = 1.05; k = 2.5; pw = {@priorWeibull,lam,k}; % Weibull prior % b) meta priors pmx = {@priorMix,[0.5,0.5],{pg,pl}}; % mixture of two priors g = @exp; dg = @exp; ig = @log; ptr = {@priorTransform,g,dg,ig,pg}; % Gaussian in the exp domain % c) multivariate priors m = [1;2]; V = [2,1;1,2]; pG = {@priorGaussMulti,m,V}; % 2d Gaussian prior pD = {'priorDeltaMulti'}; % fix value of prior exclude from optim pC = {@priorClampedMulti}; % equivalent to above % 2) evaluation % pri = pt; hp = randn(1,3); % pri = pmx; hp = randn(1,3); % pri = ptr; hp = randn(1,3); pri = pG; hp = randn(2,3); % a) draw a sample from the prior feval(pri{:}) % b) evaluate prior and derivative if requires [lp,dlp] = feval(pri{:},hp) % 3) comprehensive example x = (0:0.1:10)'; y = 2*x+randn(size(x)); % generate training data mean = {@meanSum,{@meanConst,@meanLinear}}; % specify mean function cov = {@covSEiso}; lik = {@likGauss}; % specify covariance and lik hyp.cov = [log(1);log(1.2)]; hyp.lik = log(0.9); hyp.mean = [2;3]; par = {mean,cov,lik,x,y}; mfun = @minimize; % input for GP function % a) plain marginal likelihood optimisation (maximum likelihood) im = @infExact; % inference method hyp_plain = feval(mfun, hyp, @gp, -10, im, par{:}); % optimise % b) regularised optimisation (maximum a posteriori) with 1d priors prior.mean = {pg;pc}; % Gaussian prior for first, clamp second par prior.cov = {p1;[]}; % box prior for first, nothing for second par im = {@infPrior,@infExact,prior}; % inference method hyp_p1 = feval(mfun, hyp, @gp, -10, im, par{:}); % optimise % c) regularised optimisation (maximum a posteriori) with Nd priors prior = []; % clear the structure % multivariate Student's t prior on the first and second mean hyper prior.multi{1} = {@priorTMulti,[mu;mu],diag([s2,s2]),nu,... struct('mean',[1,2])}; % use hyper struct % Equivalent shortcut (same mu and s2 for all dimensions) prior.multi{1} = {@priorTMulti,mu,s2,nu,struct('mean',[1,2])}; % multivariate Gaussian prior jointly on 1st and 3rd hyper prior.multi{2} = {@priorGaussMulti,[mu;mu],diag([s2,s2]),... [1,3]}; % use unwrapped hyper vector % Equivalent shortcut (same mu and s2 for all dimensions) prior.multi{2} = {@priorGaussMulti,mu,s2,[1,3]}; im = {@infPrior,@infExact,prior}; % inference method hyp_pN = feval(mfun, hyp, @gp, -10, im, par{:}); % optimise [any2vec(hyp), any2vec(hyp_plain), any2vec(hyp_p1), any2vec(hyp_pN)] gpml-matlab-v4.0-2016-10-19/doc/README000644 000765 000024 00000045521 12777177043 016532 0ustar00carlstaff000000 000000 GAUSSIAN PROCESS REGRESSION AND CLASSIFICATION Toolbox version 4.0 for GNU Octave 3.2.x and Matlab 7.x Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-19. 0) HOW TO READ ============== If you want to get started without further delay, then read section 1) below and jump right to the examples in doc/index.html. 1) ABOUT THESE PROGRAMS ======================= This collection of matlab programs implements and demonstrates some of the algorithms described in a) the book by Rasmussen and Williams: "Gaussian Processes for Machine Learning", the MIT Press 2006, in b) the article by Nickisch and Rasmussen: "Approximations for Binary Gaussian Process Classification", JMLR 2008, in c) the article by Candela and Rasmussen: "A Unifying View of Sparse Approximate Gaussian Process Regression", JMLR 2005, in d) the paper by Murray, Adams and Mackay: "Elliptical slice sampling", AISTATS 2010, in e) the report by Neal: "Annealed Importance Sampling", UToronto 1998, in f) the paper by Naish-Guzman and Holden: "The Generalized FITC Approximation", NIPS, 2007, in g) the paper by Duvenaud, Nickisch and Rasmussen: "Additive Gaussian Processes", NIPS, 2011, in h) the paper by Wilson and Adams: "Gaussian Process Kernels for Pattern Discovery and Extrapolation", ICML, 2013, and in i) the paper Snelson, Rasmussen and Ghahramani: "Warped Gaussian Processes", NIPS, 2003. There are 7 code subdirectories: cov, doc, inf, lik, mean, prior and util. cov/ contains covariance functions cov*.m => see covFunctions.m doc/ contains an index.html file providing documentation. This information is also available from http://www.GaussianProcess.org/gpml/code. Usage of mean, cov, classification and regression is demonstrated in usage*.m. Further details can be found in the developer documentation manual.pdf. inf/ contains the inference algorithms inf*.m => see infMethods.m lik/ contains the likelihood functions lik*.m => see likFunctions.m mean/ contains the mean functions mean*.m => see meanFunctions.m prior/ contains the hyperparameter prior distributions prior*.m => see priorDistributions.m util/ contains optimisation routines, backward compatibility programs and small auxiliary scripts Before running the demos or any other part of the code, you should execute startup.m to add all necessary directories to your path. In Octave, you can also use the .octaverc file. 2) ABOUT MEX FILES ================== Some of the programs make use of the MEX facility in matlab for more efficient implementation. However, if you don't know about how to compile MEX files, you do not need to worry about this -- the code runs anyway. If you do compile the MEX files, this is automatically detected, and the program will run more efficiently. If you do not have a compiler installed on your system, you might want to consult [3]. Two components of the toolbox can be accelerated by MEX. First, there is an auxiliary functions solve_chol.m having a MEX equivalent. This can be compiled by either executing util/make.m from the Matlab/Octave command line. In addition to that, we provide a Makefile for Matlab in util/ that you can run by simply typing make in your shell after having supplied your Matlab path and your operating system. We recommend the make.m script because it works both under Matlab and Octave. Second, in order to use the L-BFGS minimiser for hyperparameter optimisation as an equivalent to minimize.m, you have to compile Peter Carbonetto's "Matlab interface for L-BFGS-B" [1] which was extended to also work for Octave by José Vallet. The challenge here is the Fortran 77 code. We provide a Makefile suitable for Linux 32/64 bit and Mac whenever you have 1) g77 or 2) gfortran properly set up. The latter is prefered on current Ubuntu systems. Under Ubuntu, you can achieve this by installing the packages fort77 and gfortran, respectively. Note that the code is GNU licensed. Compilation is done by editing util/lbfgsb/Makefile. In any case, you need to provide $MATLAB_HOME which can be found by the commands 'locate matlab' or 'find / -name "matlab"'. You can choose between two compilation modes: a) without mex utility by Matlab [default] provide $MEX_SUFFIX and $MATLAB_LIB, then type 'make nomex' b) using the mex utility by Matlab provide $MEX, then type 'make mex' c) compilation for Octave provide Octave variables and type 'make oct' In Ubuntu 10.04 LTS, the libg2c library needed for both 1)+a) and 1)+b) is not included per default. If 'ls /usr/lib/libg2c.*' does not list anything this is the case on your machine. You then whant to install the packages gcc-3.4-base and libg2c0 e.g. from http://packages.ubuntu.com/hardy/gcc-3.4-base and http://packages.ubuntu.com/hardy/libg2c0. After installation, you have to create a symbolic link by 'cd /usr/lib' and 'ln -s libg2c.so.0 libg2c.so'. There were reports of problems running the mex files on recent 64bit Ubuntu since the libgfortran.3.0.0 that comes with matlab ($(MATLAB_HOME)/sys/os/glnxa64/libgfortran.so.3.0.0) is not the same as the libgfortran.3.0.0 that comes with the ubuntu package libgfortran3 and that is used when linking. So running the mex file results in a version mismatch error. The issue can be solved by changing the name of to the matlab file and linking to the ubuntu file instead: in $(MATLAB_HOME)/sys/os/glnxa64 run 'sudo mv libgfortran.3.0.0 libgfortran.3.0.0_matlab' 'sudo ln -s /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0 libgfortran.3.0.0' Thanks to Giampiero Salvi for posting the workaround. [1] https://github.com/pcarbo/lbfgsb-matlab [2] http://www.mathworks.com/support/compilers/R2010a 3) CURRENT VERSION ================== The current version of the programs is 4.0. Previous versions of the code are available at http://www.gaussianprocess.org/gpml/code/oldcode.html. 4) DIFFERENCES TO PREVIOUS VERSIONS =================================== NEW in version 4.0, 2016-09-29 ------------------------------ A major code restructuring effort did take place in the current release unifying certain inference functions and allowing more flexibility in covariance function composition. We also redesigned the whole derivative computation pipeline to strongly improve the overall runtime. We finally include grid-based covariance approximations natively. More generic sparse approximation using Power EP - unified treatment of FITC approximation, variational approaches VFE and hybrids - inducing input optimisation for all (compositions of) covariance functions dropping the previous limitation to a few standard examples - infFITC is now covered by the more generic infGaussLik function Approximate covariance object unifying sparse approximations, grid-based approximations and exact covariance computations - implementation in cov/apx, cov/apxGrid, cov/apxSparse - generic infGaussLik unifies infExact, infFITC and infGrid - generic infLaplace unifies infLaplace, infFITC_Laplace and infGrid_Laplace Hiearchical structure of covariance functions - clear hierachical compositional implementation - no more code duplication as present in covSEiso and covSEard pairs - two mother covariance functions covDot for dot-product-based covariances and covMaha for Mahalanobis-distance-based covariances - a variety of modifiers: eye, iso, ard, proj, fact, vlen - more flexibility as more variants are available and possible - all covariance functions offer derivatives w.r.t. inputs Faster derivative computations for mean and cov functions - switched from partial derivatives to directional derivatives - simpler and more concise interface of mean and cov functions - much faster marginal likelihood derivative computations - simpler and more compact code New mean functions - new mean/meanWSPC (Weighted Sum of Projected Cosines or Random Kitchen Sink features) following a suggestion by William Herlands - new mean/meanWarp for constructing a new mean from an existing one by means of a warping function adapted from William Herlands New optimizer - added a new minimize_minfunc, contributed by Truong X. Nghiem New GLM link function - added the logistic link function util/glm_invlink_logistic2 Smaller fixes - two-fold speedup of util/elsympol used by covADD by Truong X. Nghiem - bugfix in util/logphi as reported by John Darby NEW in version 3.6, 2015-07-07 ------------------------------ - bugfix in likGaussWarp as reported by Obaid Malik - added a new inference function infGrid_Laplace allowing to use non-Gaussian likelihoods for large grids - fixed a bug due to Octave evaluating norm([]) to a tiny nonzero value, modified all lik/lik*.m functions reported by Philipp Richter - small bugfixes in covGrid and infGrid - bugfix in predictive variance of likNegBinom due to Seth Flaxman - bugfix in infFITC_Laplace as suggested by Wu Lin - bugfix in covPP{iso,ard} NEW in version 3.5, 2014-12-08 ------------------------------ - mechanism for specifying hyperparameter priors (together with Roman Garnett and José Vallet) - new inference method inf/infGrid allowing efficient inference for data defined on a Cartesian grid (together with Andrew Wilson) - new mean/cov functions for preference learning: meanPref/covPref - new mean/cov functions for non-vectorial data: meanDiscrete/covDiscrete - new piecewise constant nearest neighbor mean function: meanNN - new mean functions being predictions from GPs: meanGP and meanGPexact - new covariance function for standard additive noise: covEye - new covariance function for factor analysis: covSEfact - new covariance function with varying length scale : covSEvlen - make covScale more general to scaling with a function instead of a scalar - bugfix in covGabor* and covSM (due to Andrew Gordon Wilson) - bugfix in lik/likBeta.m (suggested by Dali Wei) - bugfix in solve_chol.c (due to Todd Small) - bugfix in FITC inference mode (due to Joris Mooij) where the wrong mode for post.L was chosen when using infFITC and post.L being a diagonal matrix - bugfix in infVB marginal likelihood for likLogistic with nonzero mean function (reported by James Lloyd) - removed the combination likErf/infVB as it yields a bad posterior approximation and lacks theoretical justification - Matlab and Octave compilation for L-BFGS-B v2.4 and the more recent L-BFGS-B v3.0 (contributed by José Vallet) - smaller bugfixes in gp.m (due to Joris Mooij and Ernst Kloppenburg) - bugfix in lik/likBeta.m (due to Dali Wei) - updated use of logphi in lik/likErf - bugfix in util/solve_chol.c where a typing issue occured on OS X (due to Todd Small) - bugfix due to Bjørn Sand Jensen noticing that cov_deriv_sq_dist.m was missing in the distribution - bugfix in infFITC_EP for ttau->inf (suggested by Ryan Turner) NEW in version 3.4, 2013-11-11 ------------------------------ - new likelihood function for warped GPs: likGaussWarp - new exponential likelihood function: likExp - new Weibull likelihood function: likWeibull - new Gumbel likelihood function for extremal value regression: likGumbel - new polynomial mean function: meanPoly - included derivatives w.r.t. inducing points xu in infFITC, infFITC_Laplace, infFITC_EP - allow for zero noise variance for infExact NEW in version 3.3, 2013-10-19 ------------------------------ - new variational inference algorithm based on direct KL minimisation: infKL - improved inf/infVB double loop scheme so that only very few likelihood properties are required; infVB is now internally a sequence of infLaplace runs - improved inf/infLaplace to be more generic so that optimisers other than scaled Newton can be used - improved inf/infEP so that the internal variables (mu,Sigma) now represent the current posterior approximation - added three new generalised linear model likelihoods: gamma, beta, inverse Gaussian - new covariance functions: spectral mixture covSM, covGaboriso and covGaborard - new meta covariance function to turn a stationary covariance into a periodic covariance function covPERard, covPERiso - new periodic covariance function with zero DC component and correct scaling covPeriodicNoDC New likelihood functions in lik/: - likGamma.m, likBeta.m, likInvGauss.m New inference functions in inf/: - infKL.m New covariance functions in cov/: - covSM.m, covGaboriso.m, covGaborard.m (contributed by Andrew Gordon Wilson) - covPPard.m, covMaternard.m, covLINiso.m - covCos.m, covPeriodicNoDC.m (contributed by James Robert Lloyd) - covPERiso.m, covPERard.m NEW in version 3.2, 2013-01-15 ------------------------------ - added inf/infFITC_Laplace.m and inf/infFITC_EP.m for large scale approximate inference - added inf/infMCMC.m to allow for Markov Chain Monte Carlo sampling and Annealed Importance Sampling; we currently support two samplers: Elliptical Slice Sampling (ESS) and Hybrid Monte Carlo (HMC) - improved accuracy of the digamma function lik/likT.m/dloggamma from three to ten digits precision (thanks to Edward Challis) - added gfortran support to util/lbfgsb/Makefile (thanks to Ernst Kloppenburg) - fixed a stupid bug in cov/covFITC.m (thanks to Krzysztof Chalupka) - fixed a severe stability bug in the derivatives of inf/infFITC.m (thanks to Andrew McHutchon) - changed scalars in util/solve_chol.c to long to provide 64bit compatibility (thanks to Ernst Kloppenburg) - added infLOO to allow pseudo-likelihood optimisation instead of the marginal likelihood - fixed a performance issue in util/sq_dist.m (thanks to Krzysztof Chalupka) - fixed a bug causing numerical problems in infFITC and covFITC whenever Kuu is close to singular (thanks to Joris M. Mooij) - fixed an issue in covNoise (thanks to Daniel Marthaler) - fixed implicit derivative w.r.t. likelihood in infLaplace* - added two more likelihood functions likMix.m for mixtures and likUni for classification noise - slightly changed the interface for the likelihood functions - added likPoisson for Poisson regression of count data NEW in version 3.1, 2010-10-10 ------------------------------ - following a suggestion by Ed Snelson we now support FITC regression - cov/covFITC.m and inf/infFITC.m have been added with Ed Snelson's help - the covariance interface was slightly changed to make that possible - changed parametrisation in cov/covPeriodic.m (thanks to Philipp Hennig) NEW in version 3.0, 2010-07-23 ------------------------------ A major code reorganisation effort did take place in the current release. First, classification and regression are now done by a single file gp.m which is completely generic in the likelihood. The previous regression program gpr.m corresponds to gp.m with Gaussian likelihood. Several other likelihoods for robust regresssion were added. Further, the code now supports mean functions with a similar specification mechanism as already used by the covariance functions. Previous implementations correspond to using meanZero.m. We merged the covariance functions covMatern3iso.m and covMatern5.iso into a single covMaterniso.m and added a covariance function for additive functions as well as the possibility to use only certain components of the data. Finally, we included covPPiso.m a piecewise polynomial covariance function with compact support. New likelihood functions in lik/: likGauss.m, likLaplace.m, likLogistic.m and likT.m New mean functions in mean/: meanConst.m, meanLinear.m, meanPow.m, meanProd.m, meanSum.m and meanZero.m New covariance functions in cov/: covADD.m, covMask.m, covPPiso.m The gprSRPP.m function which previously provided "Subset of Regressors" and the "Projected Process" approximation has now been removed. NEW in version 2.1, 2007-07-25 ------------------------------ covConst.m: fixed a bug which caused an error in the derivative of the log marginal likelihood for certain combinations of covariance functions and approximation methods. (Thanks to Antonio Eleuteri for reporting the problem) gauher.m: added the function "gauher.m" which was mistakenly missing from the previous release. This caused an error for certain combinations of approximation method and likelihood function. logistic.m: modified the approximation of moments calculation to use a mixture of cumulative Gaussian, rather than Gauss-Hermite quadrature, as the former turns out to be more accurate. NEW in version 2.0, 2007-06-25 ------------------------------ Some code restructuring has taken place for the classification code to make it more modular, to facilitate addition of new likelihood functions and approximations methods. Now, all classification is done using the binaryGP function, which (among other things) takes an approximation method and a likelihood function as an arguments. Thus, binaryGP replaces both binaryEPGP and binaryLapaceGP, although wrapper functions are still provided for backward compatibility. This gives added flexibility: now EP can also be used wth the logistic likelihood function (implemented using Gauss-Hermite quadrature). approxEP.m: New file, containing the Expectation Propagation approximation method, which was previously contained in binaryEPGP.m approxLA.m: New file, containing Laplaces approximation method, which was previously contained in binaryLaplace.m approximations.m: New file, help for the approximation methods. binaryEPGP.m: This file has been replaced by a wrapper (for backward compatibility) which calls the more general binaryGP function. binaryGP.m: New general function to do binary classification. binaryLaplaceGP.m: This file has been replaced by a wrapper (for backward compatibility) which calls the more general binaryGP function. covMatern3iso.m, covMatern5iso.m, covNNone.m, covRQard.m, covRQiso.m, cosSEard, covSEiso: now check more carefully, that persistent variables have the correct sizes, and some variable names have been modified. cumGauss.m: New file, containing code for the cumulative Gaussian likelihood function likelihoods.m: New file, help for likelihood functions logistic.m: New file, logistic likelihood NEW in version 1.3, 2006-09-08 ------------------------------ covRQard.m: bugfix: replaced x with x' and z with z' in line 36 covRQiso.m: bugfix: replaced x with x' and z with z' in line 28 minimize.m: correction: replaced "error()" with "error('')", and made a few cosmetic changes binaryEPGP.m: added the line "lml = -n*log(2);" in line 77. This change should be largely inconsequential, but occationally may save things when the covariance matrix is exceptionally badly conditioned. NEW in version 1.2, 2006-05-10 ------------------------------ added the "erfint" function to "binaryLaplaceGP.m". The erfint function was missing by mistake, preventing the use of the "logistic" likelihood. NEW in version 1.1, 2006-04-12 ------------------------------ added files: "covProd.m" and "covPeriodic.m" changes: "covSEiso.m" was changed slightly to avoid the use of persistent variables NEW in version 1.0, 2006-03-29 ------------------------------ initial version shipped with the book gpml-matlab-v4.0-2016-10-19/doc/checkmark.png000644 000765 000024 00000000306 12701705600 020257 0ustar00carlstaff000000 000000 PNG  IHDRa*PLTExxxlll```TTTHHH<<<$$$ tRNS@fJIDATc````d6 N`8,~e RS!Ke0O P`ʳ&] NIENDB`gpml-matlab-v4.0-2016-10-19/doc/demoRegression.m000644 000765 000024 00000012060 12777177044 021006 0ustar00carlstaff000000 000000 disp('See http://www.gaussianprocess.org/gpml/code/matlab/doc/ for details.') disp('Hit any key to continue...'); pause disp(' '); disp('clear all, close all') clear all, close all write_fig = 0; disp(' ') disp('meanfunc = {@meanSum, {@meanLinear, @meanConst}}; hyp.mean = [0.5; 1];') meanfunc = {@meanSum, {@meanLinear, @meanConst}}; hyp.mean = [0.5; 1]; disp('covfunc = {@covMaterniso, 3}; ell = 1/4; sf = 1; hyp.cov = log([ell; sf]);') covfunc = {@covMaterniso, 3}; ell = 1/4; sf = 1; hyp.cov = log([ell; sf]); disp('likfunc = @likGauss; sn = 0.1; hyp.lik = log(sn);') likfunc = @likGauss; sn = 0.1; hyp.lik = log(sn); disp(' ') disp('n = 20;') n = 20; disp('x = gpml_randn(0.3, n, 1);') x = gpml_randn(0.3, n, 1); disp('K = feval(covfunc{:}, hyp.cov, x);') K = feval(covfunc{:}, hyp.cov, x); disp('mu = feval(meanfunc{:}, hyp.mean, x);') mu = feval(meanfunc{:}, hyp.mean, x); disp('y = chol(K)''*gpml_randn(0.15, n, 1) + mu + exp(hyp.lik)*gpml_randn(0.2, n, 1);') y = chol(K)'*gpml_randn(0.15, n, 1) + mu + exp(hyp.lik)*gpml_randn(0.2, n, 1); figure(1) set(gca, 'FontSize', 24) disp(' '); disp('plot(x, y, ''+'')') plot(x, y, '+', 'MarkerSize', 12) axis([-1.9 1.9 -0.9 3.9]) grid on xlabel('input, x') ylabel('output, y') if write_fig, print -depsc f1.eps; end disp(' '); disp('Hit any key to continue...'); pause disp(' ') disp('nlml = gp(hyp, @infGaussLik, meanfunc, covfunc, likfunc, x, y)') nlml = gp(hyp, @infGaussLik, meanfunc, covfunc, likfunc, x, y) disp(' ') disp('z = linspace(-1.9, 1.9, 101)'';') z = linspace(-1.9, 1.9, 101)'; disp('[m s2] = gp(hyp, @infGaussLik, meanfunc, covfunc, likfunc, x, y, z);') [m s2] = gp(hyp, @infGaussLik, meanfunc, covfunc, likfunc, x, y, z); figure(2) set(gca, 'FontSize', 24) disp(' ') disp('f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)];') f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)]; disp('fill([z; flipdim(z,1)], f, [7 7 7]/8);') fill([z; flipdim(z,1)], f, [7 7 7]/8); disp('hold on; plot(z, m); plot(x, y, ''+'')') hold on; plot(z, m, 'LineWidth', 2); plot(x, y, '+', 'MarkerSize', 12) axis([-1.9 1.9 -0.9 3.9]) grid on xlabel('input, x') ylabel('output, y') if write_fig, print -depsc f2.eps; end disp(' '); disp('Hit any key to continue...'); pause disp(' ') disp('covfunc = @covSEiso; hyp2.cov = [0; 0]; hyp2.lik = log(0.1);') covfunc = @covSEiso; hyp2.cov = [0; 0]; hyp2.lik = log(0.1); disp('hyp2 = minimize(hyp2, @gp, -100, @infGaussLik, [], covfunc, likfunc, x, y)') hyp2 = minimize(hyp2, @gp, -100, @infGaussLik, [], covfunc, likfunc, x, y); disp(' ') disp('exp(hyp2.lik)') exp(hyp2.lik) disp('nlml2 = gp(hyp2, @infGaussLik, [], covfunc, likfunc, x, y)') nlml2 = gp(hyp2, @infGaussLik, [], covfunc, likfunc, x, y) disp('[m s2] = gp(hyp2, @infGaussLik, [], covfunc, likfunc, x, y, z);') [m s2] = gp(hyp2, @infGaussLik, [], covfunc, likfunc, x, y, z); disp(' ') figure(3) set(gca, 'FontSize', 24) f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)]; disp('fill([z; flipdim(z,1)], f, [7 7 7]/8)'); fill([z; flipdim(z,1)], f, [7 7 7]/8) disp('hold on; plot(z, m); plot(x, y, ''+'')'); hold on; plot(z, m, 'LineWidth', 2); plot(x, y, '+', 'MarkerSize', 12) grid on xlabel('input, x') ylabel('output, y') axis([-1.9 1.9 -0.9 3.9]) if write_fig, print -depsc f3.eps; end disp(' '); disp('Hit any key to continue...'); pause disp(' ') disp('hyp.cov = [0; 0]; hyp.mean = [0; 0]; hyp.lik = log(0.1);') hyp.cov = [0; 0]; hyp.mean = [0; 0]; hyp.lik = log(0.1); disp('hyp = minimize(hyp, @gp, -100, @infGaussLik, meanfunc, covfunc, likfunc, x, y);') hyp = minimize(hyp, @gp, -100, @infGaussLik, meanfunc, covfunc, likfunc, x, y); disp('[m s2] = gp(hyp, @infGaussLik, meanfunc, covfunc, likfunc, x, y, z);') [m s2] = gp(hyp, @infGaussLik, meanfunc, covfunc, likfunc, x, y, z); figure(4) set(gca, 'FontSize', 24) disp(' ') disp('f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)];') f = [m+2*sqrt(s2); flipdim(m-2*sqrt(s2),1)]; disp('fill([z; flipdim(z,1)], f, [7 7 7]/8)') fill([z; flipdim(z,1)], f, [7 7 7]/8) disp('hold on; plot(z, m); plot(x, y, ''+'');') hold on; plot(z, m, 'LineWidth', 2); plot(x, y, '+', 'MarkerSize', 12) grid on xlabel('input, x') ylabel('output, y') axis([-1.9 1.9 -0.9 3.9]) if write_fig, print -depsc f4.eps; end disp(' '); disp('Hit any key to continue...'); pause disp('large scale regression using the FITC approximation') disp('nu = fix(n/2); u = linspace(-1.3,1.3,nu)'';') nu = fix(n/2); u = linspace(-1.3,1.3,nu)'; disp('covfuncF = {@apxSparse, {covfunc}, u};') covfuncF = {@apxSparse, {covfunc}, u}; disp('[mF s2F] = gp(hyp, @infGaussLik, meanfunc, covfuncF, likfunc, x, y, z);') [mF s2F] = gp(hyp, @infGaussLik, meanfunc, covfuncF, likfunc, x, y, z); figure(5) set(gca, 'FontSize', 24) disp(' ') disp('f = [mF+2*sqrt(s2F); flipdim(mF-2*sqrt(s2F),1)];') f = [mF+2*sqrt(s2F); flipdim(mF-2*sqrt(s2F),1)]; disp('fill([z; flipdim(z,1)], f, [7 7 7]/8)') fill([z; flipdim(z,1)], f, [7 7 7]/8) disp('hold on; plot(z, mF); plot(x, y, ''+'');') hold on; plot(z, mF, 'LineWidth', 2); plot(x, y, '+', 'MarkerSize', 12) disp('plot(u,1,''o'')') plot(u,1,'ko', 'MarkerSize', 12) grid on xlabel('input, x') ylabel('output, y') axis([-1.9 1.9 -0.9 3.9]) if write_fig, print -depsc f5.eps; end disp(' ')gpml-matlab-v4.0-2016-10-19/doc/usageClassification.m000644 000765 000024 00000005144 12701705600 021764 0ustar00carlstaff000000 000000 % demonstrate usage of classification % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2013-10-16. clear all, close all %% SAY WHICH CODE WE WISH TO EXERCISE id = [1,1]; % use Gauss/Exact id = [1,2; 2,2; 3,2]; % compare Laplace id = [1,3; 2,3; 3,3]; % study EP id = [1,5; 2,5]; % look into KL (takes quite a while) id = [1,4; 2,4; 3,4]; % deal with VB seed = 943; randn('seed',seed), rand('seed',seed) ntr = 50; nte = 1e4; % number of training and test points xtr = 10*sort(rand(ntr,1)); % sample dataset p = @(x) 1./(1+exp(-5*sin(x))); % "true" underlying probability ytr = 2*(p(xtr)>rand(ntr,1))-1; % draw labels +1/-1 i = randperm(ntr); nout = 3; % add outliers ytr(i(1:nout)) = -ytr(i(1:nout)); xte = linspace(0,10,1e4)'; % support, we test our function on cov = {@covSEiso}; sf = 1; ell = 0.7; % setup the GP hyp0.cov = log([ell;sf]); mean = {@meanZero}; % m(x) = 0 hyp0.mean = []; lik_list = {'likGauss','likErf','likLogistic'}; % allowable likelihoods inf_list = {'infExact','infLaplace','infEP','infVB','infKL'}; % inference algs Ncg = 50; % number of conjugate gradient steps sdscale = 0.5; % how many sd wide should the error bars become? col = {'k',[.8,0,0],[0,.5,0],'b',[0,.75,.75],[.7,0,.5]}; % colors ymu{1} = 2*p(xte)-1; ys2{1} = 0; for i=1:size(id,1) lik = lik_list(id(i,1)); % setup the likelihood if strcmp(lik,'likGauss') sn = .2; hyp0.lik = log(sn); else hyp0.lik = []; end inf = inf_list{id(i,2)}; fprintf('OPT: %s/%s\n',lik_list{id(i,1)},inf_list{id(i,2)}) hyp = minimize(hyp0,'gp', -Ncg, inf, mean, cov, lik, xtr, ytr); % opt hypers [ymu{i+1}, ys2{i+1}] = gp(hyp, inf, mean, cov, lik, xtr, ytr, xte); % predict [nlZ(i+1)] = gp(hyp, inf, mean, cov, lik, xtr, ytr); end figure, hold on for i=1:size(id,1)+1 plot(xte,ymu{i},'Color',col{i},'LineWidth',2) if i==1 leg = {'function'}; else leg{end+1} = sprintf('%s/%s -lZ=%1.2f',... lik_list{id(i-1,1)},inf_list{id(i-1,2)},nlZ(i)); end end for i=1:size(id,1)+1 ysd = sdscale*sqrt(ys2{i}); fill([xte;flipud(xte)],[ymu{i}+ysd;flipud(ymu{i}-ysd)],... col{i},'EdgeColor',col{i},'FaceAlpha',0.1,'EdgeAlpha',0.3); end for i=1:size(id,1)+1, plot(xte,ymu{i},'Color',col{i},'LineWidth',2), end plot(xtr,ytr,'k+'), plot(xtr,ytr,'ko'), legend(leg) gpml-matlab-v4.0-2016-10-19/doc/f9.png000644 000765 000024 00000053646 12701705600 016664 0ustar00carlstaff000000 000000 PNG  IHDR~͕ pHYs B(xtIME 5()"tEXtCreation Time06-Dec-2014 18:53:40ƴkb$tEXtSoftwareMATLAB, The Mathworks, Inc.R IDATxm8s\+\,\A  p+\A Wܮ @"9^t/(Q]f`9{&)X`a`I &)#Ƙ ;=lẁN )0I5Rp8g-]#\muAd~f[N*0d{w}%'>uЧt${!dApan=FGKNWQ?$Ft,@ ΀4Ѻ9I KRgX=V9}Lo@5kUHDdNh::Op4PI^\?)pOI>]gD7:b7{){$7GU_::0)̸XOVuX!"맕x,x)PI[IH `$ngASaxsC&iI` !?`)v+It 0˧q!LO$ ;n#s`do\>%/up}SdA!CtI#`e-(o M 0z,,M,m^7ы`IZTҏܿu`M$nuGUu7~. TҊ&Ws$ǒ6~3׽\ꕡ_]I2ͯhl9; |5t7l/(OAɯo> "_5O/xC }}N ș*\/Fܜ5YQ~W/q5W׹E-Xd6z!"=Jqߗ$}qhNgb{hIZgזۯJK e:Myt}KzVΧqDmޒ m,Kx/gcCԍ(ez-VIN$0:dۈ08|{yF0pJ~I)],*yow{lEBjGQWa?EJg\f\i(ΔH_Vg :)83D: ?0Y+M$Ε&Jr3xL~7}Y VM(BלmLD0$Z.1{p b͚,ol,wvL5!kV߯wpj_:1~6M8JmX9[ZgGvoZX'"PiL0l$3gl}~3_EPjW'ZP%s)O(?O=BժPb9-\O/L<{`΁D> ^B.Խ3g.β>Z$j/PGX-;ʢ:>A-iP| s2Z*yσOy3K,]nfԃ9I W&Gq?iÝf]-ߔ_3chIWCiz{F cީ[i:`.D"p}P%(E,e9R9 ]%K,^KCn,vov _/[nMs֠(oz0Ivq4GE~~gU( ?p%EЎrR+=uee[9>0K6)"f-tzvE:u[&_T~ \-u;C($+&9 l;|ȥUT\Pjċ_ImjX?BKQNm#eRnQ0L|جz%68/ܝU 7"Psq`d&Uscz}W~V:B2G?9US٧=& 9 0 _;))Xy| X$x/j|W)z R`$8x/x)*^d-+7ؾ75h}1rPK =ힽ:5ƌWY~e":ZF3cL)SocE}Şֶ`r_˓^B?  tk1(?Kv7nǣ:Fٓ>h_?Aw||F OI\Rܬٓʠ/޻) ܯμ=*xYNјW!-*:67Wng`)vů!^y2h柭?0XK0p?WPLg3X{Fn Q"ԄԖ̓.63)-*i>*dn2]{;4#xv\>/lyLIz9j)j e|z=Ò2\&"v0 /޻)lv|dQ KWd!63^{d&԰*i\Z}H5߳=Dl-cNޔC%ftkj7ȥr7ĤsZHi࣡ 7uK9t(EKڽJMcc*}86q0ujhjm6faړ|L(Xv!}w+EG%d[L9vl$].u$?x+7>>˹o({Ik'K'*QS!mV˥>7~PsEmJfֆk,= ĩKTK(Ba4+Ѿ?%<8(j7Ѣ˯;Tأs%h!k=bw~0 4s3ӛ.ؼ=~/YgﾴSc 3jd%4ltPv-͝H%]v9ƖL-N0Pepsfޔ:3Ǐ̢'ŭyi|j=7qI٦y0Eޤ/*m\x=d_wSPIqwRv1 e *Լ/OgP}ͯ..NHcLEZJfV' 袃jT.o.IƜ6f%R!aVgESNu}RN R+zJ,vJt[m/X1o^}z]~.sa?H.~6+W2;E|}y SOrPdP? X雹K,ǽOŝ!ho=Z9J6> Aio#t;)CSd[ -z`@%m%/c C(|.K.7w0_]bT%_q_U ^WJ.xE *|56IFnNv}i{٣ӛ,u--C[C;=E)擩!4φS3|&v}Oor;Lo*9S:N[$=mh~5ӫ*&w nn1ٝ{7ZRG˫3F+)1Fb?M2]vjKMg~3p'%ϑ+# n~ޛ_ "2 أ乼UG\frd3qe@\wah Mג?;tӛ { ;ЩE[lѮu|q6M"y*$Uю#]FB*Ӣ'C=k@Q"IN^"{ Ṽot-Lտ `YN¼7+wfj!9!+)1V*T JAOpKF^(} ;&${e~.94ӛxzgaSEf"GP-ܬk_zȫkj.qcKJZ-þN1\8WT6C ;5+ ֒?;$ݜ}+ g/%{Ϲ c\M2K @j1~/]yw1ٕ^ +<0D|tE&MYS0ULyQJɣC 96ˍߟUP:΋ I=ZTf9Rmӛ:B{LRL= MKbY0gA|\2'a Ҧ,]Fkbd߂#. ͭ?EܝԿtrk~WC T!Iiﻃx)?ᾙM3"Pw`Ӏ`aEUvCKt8*kߵP;,R՜r_7< 5(t$JZ)./!K38NǤ;4]^7=Nm+d: ]M`i0~6h`##ԺU|f-97"mDPdo ޘ>BN]K# ew1جjѬm{`aPC*W@(@~EyJVR3޴[-b>491L|Nȵ"Mdy5KW{Ա{v2 :y{ ;F oҘm6:NK:ݱg^AO_u}~ ?_/4?AKMG)Ot֎gͯgm|o+ֿ3hO{%;0޽1馒$cLRILT}M!w $Vj7o"Rŷu-}]o'9{H. wd \\q%*46+.~lPjm]Vo] "ߧj1}jXrTx,jrS8@|GTD{8qGM{tSI{D+)2ma_TpW2E]hS7= mJۮ6b5nM4LU\5(Fj$'K ټ8g^UX,v(gTI5eJsR>+o/Hؤi29E.G;֪DJ=b&95bz ~:\DzI/bښ{riPINkt\C/jh&=61DnN^R- [e&-Ôۍ*Stz;3jAq>3'nE.]M=Qͱ~<#wScF_Yb߄1 s&d G5&NΝ{Mp ϹhVlDXIFx܈# \Ɗ.nIJ"aOnj߮^[l̮!؛g*=^}of=4f*T-1о'kwG 3Fsȋ\O V-}3B3 1x9Y0(.[ 4J<̍|4B {s/y(xc`KwCVx27a/~L 2d|27\ A~.F':~$U¹dwù}NMl/_kPK_I5{((ۨ:Zn}Hy-SjIk׼7h"M.RI"8n;{!UyȓO_|-FtyZict|5O;1X\.>xݥ#DW'1nO5|5dHeO_n0Wd^h< ?O/lao u3{_U&m{U2¶[rݜנ6hCh_̶ JןqMv8UrF?"tSI`A0I)N s0IxI:)r7EN~:|Y~~_3T#,Zͥ?(Hj%}0w ߒRA}rʵ]͠k8&x|I'\4;8>q- 2o3SLohV|c~ K sxkGѬ. Xb `Ƽ]>qBgJPjӛ4iEsO.ZYTwէ͞42eudQXIDFG9G[=8 IDATɤ:ṠTqΊJIhLR=ޛ5Fqef(8rtSLojetu@GX`n)9)4ЎJ t&*Ǘ P. KL?5RXY%՟o[a_|&`s=>W 48M@9K̇J28;Y rp(w~.>;.m{suTQ놑meN*Rtk[p?+cI:9=u-s>TTSH2إLΣuG6iRks'e).[,p%F4s1^"?X cQŊ+?N沎bɊ&t9bq KL7p7bYLs>ZStr;%f\KRFty*~h_#NQrR9, ր< *QdY{TEʰΗ4G}IPKzeca_k.+ǗTCE# ͚g%pшYq(r$A3'_#f=Ke5Oa: KMޢr j87}ٺy6t-Q %l`\祫VQPѫ!PIg(N4 f8H8Y-T0Mo93rjzSv]]n@j!qʨ^WʜrF,$Y";ݔ*)h X1(ȱ},wa*l LJ *4TVZVSLU*:--3sK$*GKFA:2hO7)>)*%Ʃ\*BܳK?3MMqfmiD8`STx27KW4spc2{F@U-ϙ[t6Q T7 %!ZjMONŶL9..-9˯᳦et5v|/Xq? :gK1eXW .ɥ^^F%>|kS#YiRV;5?PIA3W= 3Fy j.gsI[4`plPrpGs+T2s̅ 1x/ȍ61"P#K;-MriEߑ-6~ dIܡ1? o~Q b ɜMvT[:qbA_lvDhn_R FCp{sn S4Ecם#Ӎ9w_Xpb bP@ǯY6h}ϛKZE"'0p?9 8;x6O8 ٺ=zc¦=jJ:9vvTq&y$ESQ mn)UyOl]֓9Σ&9ӛӛZgh`na'Mo1FDMA$S4Pn SN*&TR 73޴1KW v@%8dNRWT^ODzMVL6R>(9 "m*U8ii_AMglz 3h\ɥ+%ʧ%g /wJM6fێ;E,Z*fPdM ؑ=B4þ獼cŮ*lG:~#E)+ rklvG}弚˿ :8q/o þS}г hKZ*vܨŗ Tv`\xo;~/]pNlkP,TCBY̅?ҥ|tjm3+E͊(ٗ4[5vtCJqT%}4v6nD."S5/i)PIA+Y{6}+Bj|~-]pN4V]bZ}0vO&Z6B7MYK(*Ϩ$k _4&OS% (7ML؎^cuh6#$;EuYe>좃ʅȣ;mCM̀%Iڊ݃`Gʳ(|Iq߱Y /i2E[`_^-e>l/<2ڢNW2Pl~.]y7 gt-Q %{s`t|4~?x㓹Ś=k?\/O.󁎯?\ f+I+^?5W~jpcͭoo``I`;g5:ޘ'?y32^g?ʼWo`d{a͏SdɚZc S7O`5w#\žq=uo$pAw _`%ju\?!.attSgڠ_x265 EݾmtV!O &qA|bS=A}ݳa.\@Wx9 AZ _)R=NoRIǵޔn=]{H,+Nſ%Gy MyxV5(B̍]0/G-0I X?̜B7e˖Y׷8g2XTEU1/i4.u+AQш&鬔GM<څb?hQ]\Nhf(7g`tg?ٟ!?`MEI Z8kSTVڄa_uC:v%RKu6_R5{ >kf\|IRqpX`NtIi2O nQq뻘 oS0Oϕ 0/iV.-D% 嘸X7!=hOh_ n *`{tfY`. dQ`'XbK:+]\DDGiþ_K ϐw|DG2{?ʨK=6;{>G""cue^KqcGt7K~0w.@Gs>2|S-ͽ_r ¿˿0o悉n!ȥz+P+ cֿOFv6zLs ܺƒ׾\}-q0nA;g4Q';ͳbNqg[rEEp( E<5i6{c7!fR=0q7qmoˎ3j9Fhq-5@5L8&z(.1OoQaA.ʼX-*~+ޚYrgx:K7}v5"p` 5&IڣPIgE>[F$d둝>Y* #eG4N6R(LZF,S|W̭Văɍדk/A`)mO iq+R%feiz +3tgwfqtrP(L0&΀n/z\/]m{tr],JGOѱ-'BGqdT;Y6Xpz~EnޗZi>c5lvapHO1Uw%͟Z4%87=Fd28 }t)O #qʐv[8ʼ`ca   SdأrӛNLGItþܗ 7b9sርOјͰox=?p ZӅ50hֳޔ(:P kd!U漼EjNr=?U}ߒuVl~VQ LAG nӛp9ٌW7dMY7)[Z"$0[󘜚,m2vI:Ai>V *͠zz7̶ElunAsmyUKiSڈ5(ӛ/Z Ƃ`ۿ *<gJJqPQk41ZHs|Qff?f 0''X*g0p zLokҼ.]Kk|m:ہ.34 ^>6Q]s;>vԠqj|fTyvぎ+r{2xcϛ[hAr%_g̋_0m4!s݋# n $Q%}=*fq%>NJA]fSٛI{\0.huɠ9Yd:d.\0I'~#?a;ޙRͽø]{ȸ`j~"w~*wsld?Hm|ʼ?WAgw!?ٴ%Qj?*THk=u]u ?5ɉG(;^W`Q"5|5|=bbavWjF÷QPC *9V +/6@B~"gJ*54lz eMmܾr7AWn i4~o)|I{v@=Svt-Qn12T:ŵtΣs#? ɽҼQM>|IyģOL Dsa dݑnr@_qolq74|e;Zq4W ʩV&޻CWt1\}Q9:Z^:[̩+d5hz+ۇLx:1vɕQ E3ǭ&)h̀E;-Ax TҘ-A:>sE%%ԋ l6˥r\Z94h%0I[qTLRڔ.,QdVTEw֞bΨ9;E<9.kUC6%V1KWcأ+d(E|o0iZܼu._}H53x|V.9ki!=5ܕ7uEۏ-;KEӆgB(?5%\$alpzѽI7p؂mKK1k帴s4*"֒Jnj*)h k`*_i]&෨ɏ.K"O,MSvk5:h%')h Q,$}džSsgdR]ʼ$]6Ayi#gh}NIN=4ȗ4P0`n} TɁKʹ7;y%eS3V`ZN7խ-:^l`JcM65Ea n{HSe}RøuC +(vI/6˥×|6e!Sr)~T08GA9PIA+~*zX{p>`錾`pl.Cݛ4Mi]ׯ^&nhWIuّBꄮ wR7QZfNQ'5Kg`NN2Ouuc9I×)i0+ʜaH#[2˅ {6h`(}l}]9 {T=>1ߪsþٞ'?3&9>KEqUF ~_|aq4nc! Ll~<-]m{trzuסfL1R_qz-=MEԉ%cԔ5}8{? -xՇ9ޑb$+)5;4&}0{OEXFM x>j\ap,߫U 0IA+ww.޴x=^1&s=rL47j^灓h|_>(~\q%oGcpV?ף%  !  (UZ "2G31&&GNuJ:h Ja6\Q%F3|gU c<\(aszVBlcJGDU*iVx 06Fč{Pgr*5"HWՎb|uIIDATTŴ0ZF'GRc-*{wS% V?kï=zWR O~;Jy)9.n&-u)ui;%=DXd6pkO$^cω44t%6DR&l~%8kMKOorZ)B i|)21PQazM:YĕAN)Ol#XGa:S;&N\cVd$Sعaa˾uY LkFR;@@WR;! *ŭ*!VZ<$q҇=w6Vvu]iS[fVkbO?߮;_F ϼRbDP8*gмN/`΀G8^Chі\Pa킦)=Y,iR;zWIIU7&4l~.֠X[eT~FLQ*9u5.F0NUk T>M/L2B6$]jY|]RNhlGgؽkNԥ2?JnNCڱB Ķض'"\-wvo i=Gp]!G5G5X~Zmw<)Σr)!uJ\Kۏ{䣺Z):M ?ON '&Zdq(ykAۇhi#Ef(1k7{[m'TƓ1n{e0{0p?9@DoGK-dҋA1,*u+lNZqTsL VW14^&i675?Ãs;>G#ʢRR]Mad}幦P%{KK϶>5(w@F LV.^pR+%fb KW#ۣgi.Tͱ }?{cW l{ld3soJ >N*ן"[UnjͲ1`437vk}R^V+<&i+88tΣeأu>>lhDSzL?-Fc8Ujnh1s ڣd`nƼ/$taFAҼ +"ͳ1OgH[!s oJb%ȗ-|2`a8oH1]'tJrA.aGf\u+yQ%}Wg?lA7;4}S'#q/_F ,3'=7nPWIwcG? dƕ FE||m!(f|lPq?E٣Ƙ >#6RRxVԍyQq~kiO:tTy+e@yl2|,:7;}hqT2ƼK#"8#!@hQ/䔫qdywc"dKv^Ov( mT5@F<"]Lb:(3bGlh""'Bʮm.11)JQ1fuI5悙"SeI<*d3Du*UIi)ASazr ^jele"unRR/5bL w;+gcxvT<wSPI;`<Ymy^Z_:V4䧛b?u룕[ѢDyh{ƎOC*D3g_T :!:]M~*r[ؒ{)^O8/AUwO|.wd\@5(J ZDsض%E3A#XmK:-/QElpeQKRSEФd'gKb-7W{w0Ƽ+kx׷0ޡrϦ ^f+vCp",SiȪ\x#=ZwMa&8y9 m+ێ2>F8|ISϚ/Z~l~7`a]]-nb'<}kC.]ki@Ik ]PmGʢw];2+£ wS%Չ$P-f4{92r8(f5\K"ϼRɥVx +BfѼ콛J:9`+O \-eH"`o/3yf} Ma713V;':%)z|2: 67 [^A&)h`&;;p}e-°/7e]O׎3b>eO.)A`qHJg6 QzYL'OlӓrR3ߕC2b9dlL:%Ue 3]!{֑eרӛ} lSw{E ̶تOuFgK>*o,Mt* rRѴ=% =qUgs7Ƽoi8Q☊RYm4/4fO(n8F3BVxB%`zX3z+=F0K7͋6:♯4v<GvYqǔ.bj{>C*إ^t\^=>+a0^d7Y}tר*`;sSc0$}I26}yhTD?"A4hxQCѴB-#5.ŀGe37DQ L|mf`"ͻrc$s!2 DV !|Cw(_h8h\AtꬔVKF*Nf s!K8Z-+kDti^A*:en/;j<"4WȮKU(C"gg::F{I17KDMm{4."뚆=1#:bjUK礏>j))K$ՏMgQ lgqp7{_O3KD{x}g80A.]R(ZP ha\-G7b#(.C] w+hEn*l|`=<݌@3Ͼ]cvTi1Ƽ4 OQjLDS8E-u zgkt(ywi*bPI1;fTlw'Ujt z݅R.UU-Ј lssmJsIūt-6xŮݫ޴Jʽ9bjQu<h75-zр v&%35ۣ{\;Ւ /,]-y{$`zX3ؽ/G8jQħsɲکG){4*Tm\Nee|ut. dU>zʬzcԞN*4E ]Ibdő ;@{\|[Sxg9o r[`:zKc;s8w=ݛg>{⑉'&&-P!q܇ Wblk?hL|n]f̪GD_A-Pt.g"Iڎ߹iJTw mj8JcfF~HfFrNK,/=gWA%](םt~rRW7s n`n$&i_U'VMKU4.ZiJ,A'+Hm90' /d^ɼ"/yE&Lf`;/Qv܀f`^KACDo#mTy.#7ǨA'uށ+k$奾% bB )x.+|ޚ;1x/ǒSC!r^Ġ|'V4>Np=O^`+%&A0(98oT 7X>ÿ,jSd_!+ Agk_kޥ칯7`%칛"Lo?^n `YMףS@_LR0qͻn7u _c/y/Oz_8fnEQzW&Td7_;6 xv~_9/p>lg]zf> ,MmbQLҩ{ 6o_`/=> stream xڅn8ޯЩ"v=l@KMT NiKx8O͛tm$}?}/y6 EuҤVmOکſ?>~MgiʲKNS0z^jxXhb:X%*Z9 }cGJ7[:f4Fy{ .^:_4;NT&,)j@_j$i?A$@W9_fu! tUy8ӌ,~Z;ƾ_NXYRpVz'B˓<ِHbm!"Z{vrMeS{[i]6l<ߨQ"è.˼fE ٢Zn %e ~&yrЪΰŲiz;M$0jBK`ߴN d ^źܛߨPKr@$(,I0^6;+)XxuRa^,IDk)oۊ@֍\$]Q} GHڸ|Nhas}:@5@$>^':_҂C/"_: bC=vuNR}?OoR1 )M&̇T\l à~kwFh{sCsi,h nZwڞ@TQ7Htr4pPVf YL4̮gβy}·;!)Nj]7#NS Wmy [e]G<;U TvBw6#jG:"UH`x_ӻa@w^DKJg b8䡓.tngT[D`lQ3qKw#a aq?͋}dDQ!n[K"橨&z +*!MvZE> 0*=]YFp-U8(il0kF>yg=]gwT66RiwziT/wtbJtZ!`a៛sf]ΚQ9M:em;Œal#Uͮ׍H炂:nDBaXeTOAiEmCp?*}{= {ʨ`eQc1/xs/2U8 }FwՏ?ć9G2*wBRuZ5g y1Oԓo ;b`j=N9V'55$iX&ܬ)ᢲ endstream endobj 2 0 obj << /Type /ObjStm /N 100 /First 811 /Length 1442 /Filter /FlateDecode >> stream xڭXnF}W[ ˽p#ԉQbik2k$Hʗ֭}v9{fR#Ai#2dcr$RLcP$,K22$:"|-HIʐ4[Iґ,c*X1'aF $@`?,v1Y;d -%JbGFPbU`$93 Ff8$@Z"ˆ[6$%T2H$F@ct>^ |Fx qSO! p DD k0mjcf3!86apH"&D I#\Ƅ#D`9^qP !## #Q((yem$a@1\,db"9!!V`HN@RIB'5 TLpD@ZR Z8DmZ 4 8IS sdN&^NZ2@:(,+r3{^l/j|u e78-'~4=Hvӱ#\q1-؛l5tG%=#+82/x;q "sV>E3JuIOws+ODnf# R)|<&H6.Yd@:/GVdu]'.=\4@A6Kv&ĵ49?ݬ} gY~ cߑ+.]U<]}nK0݁aֻvNsRŰt5?^soPL? wv݂5ǦB s髲ʊjvxkn׺0ס\/Sx5׌qpH=,𻯋Yvlң≮",!:ymO4;PWmrЎ`BkGE5U-ïix o+Aߠ$!Q D3 v!ߥACx&0|?ϻ)?ḻ5yG1(q8.=IQAd5Tѩ =P)GngVиl]gEI</&t}hoCl rmII":r<,F~9eǟa AߚtGZ;AI`OGUt[y/#=80O(Oͼ endstream endobj 248 0 obj << /Length 1210 /Filter /FlateDecode >> stream x[s8+x[[q4N_a l Ɍ77d 2>d?OO~BOgz>LW4MJ_-C-r<>i'^EPQ2Oy:EaL'!qH%*Y;Q2ɹ*~G1V]>bH3 4DDfvu +utZay7=A8Γkd&$.eyd).\DkܧSYԯa~JWS卹jխMj'Cp~A竇 ]۷aMBQG3Dk,jBIka䬴^͢9F2 ƿWq|n b/;/Ak*,a.^nla$nPxKuYH_=1G n~΢jA@(+0 °<|I|Pֺ`yK o"]_dLvrugjzW<뚊Mu: 06Y?OC/X.u&Cз4 6z=ACXG4]fڜ]X 7)m,V`/e<7k20kCҋu~ݦ6IS'b 8 WSh-w} Voظ;ngC/eV; alJƻ^nYߥ")G\G M\Bƈ#׊k8*tR:h%3KYOd}4>lNQUo7?6Oz\|}x)f,vHrQ"]wZkiS=A{GϺr&+P=B]cUށ<0LYҼ˹v.K(ڮ ma" |6Gh0=~( endstream endobj 256 0 obj << /Length 178 /Filter /FlateDecode >> stream x= 0὿dH4 ~q*i-!Z 8pw 4)e,kA$$vxyYrA ٮM{:sQ> p醑`,\Xğ!cO~P q1PZZCo?IYs?zmkZ #̧gǫX<Pq endstream endobj 267 0 obj << /Length 3310 /Filter /FlateDecode >> stream xڽZKW.mEArp\g|mNj9"΂V`zN~}TF=sB/RW|so׻/6+7BTrswh*e\e_ ymujoRO}7@R&}Uu{O-U{̡ޏu;J^ lSR^Ч E8J AS%s&ԏuoi*}׎݀5T8e0=6v(Jƹ,fefHzݒ Fnϒ[WzLoSo2Pf+R^N;;Hr>whW/nW ~?PԠ3Zcn{r"we|lgi pu;`J\GӒ=~z* . :h:b #5L]P$sg m7RaĠ֋V93;/WnB+=B gkNf|ß",s_ٔ@IVOVURVc[s;/-;Ilg:}}FUAv<ЋeAa: ĎWchɤT ֚7m(@s'V}XRQHigӃ iD$ 7/lc42;sC?Nf9DcOr,rtǪ( dž㪘 4B$qƊt bZZ_'>+>hL͓=.͂A7_T5c}U 0ËJz|v[7@=Ի9y?9J6n眢C;nSh}k3SxJ&a}  d|ĸ&]kA; s]Dm`@Xݚd6.Ou~ @IXDts&2v@S&1{aпX Q$6{V4$zT7m[+bP0 @;B=JXc>-pI-7MLP21$RT Ri+Fʠ'.B lY'Fʧzkjvkۺs"aw3[%ӭ0:nbu{o混w_S[ k .%s,q'Yu %YTJ(3Xʩ[5CGw.JE Sco@]+_wX0t'O!өs;0ZTAn-Bƶ^oChjb?hXB Vs`]f( LKNIXBӂ IA@Ǐ6{C`:3UlƝ{ CkA g2*1?jp}~y|RL,Id)Kbe{Վ@{ѐцءd)%*;GjzA2*GX M~EX̳f:b@RH"7qy㸮?Qљk)22r|h0C4QLٗtOu`n C3؎I/-ɓZ N=v<.b\N+ s6+[JAC@ñJLݙ;B<B:iErڅcEkLiC?PRtT p(Քyt }_=Sr5>zN8= ܎4,ȖD87 BŒYk,,RjlxRLxqS\rY# Voy!|YKdxͣ*+lպ{Aibso:q(?L1T2GK3rXe¼3a q˫8+AV4J4CLc}Zs BluӋCX/%f^{5uD=>w"!c@GmӁ\pxдv5=#ۢs:ٛ/@Ao''P⿌Bz" n1=bEt ]fPImO=U;!`!wdʉSI%~x0%.VK+e5/e5эQ Y;'F $ht]1JtmQ0t2åitj0ÛkZ}_iL2!!VS@RCGT 6n&-mB<+.+Q/HjL`ՊxC2\N֠hQϝ{_qd=,_{$XWTpDe1&zW5/V]I$Ku;~U}/ AfJl9&JG ,7_J>^!r%8SlՐ{6 n}wTܢb-㖘k&@87/ ~[.ωI^rBDSJz J FRdflϧ㧨$iR8/J"] Ϩ:P,yRyXkρgvAIyP'rj>KgW^hMNZ/a h VF^-z;qvC]w 1|0lm.`KX_R2Ə!bgeU CF P_!ɦm[[p XZL/é%)y2\A7boy`x\h߹.8Pc\Ƨ+~i`bAX\Ϳ6§J3_3]Y&+X'\ *@6N7+Xxh xx={@NC] P}uE$Q׷!ۤJ_ƳWֻ8K jI$DxL؟N\3LNCV,ޓ=ƧI,|;T?ܤ:2j="r;{uihDI}DrR0>ae }~qxG7i ̰* eŠ}_Ζs endstream endobj 278 0 obj << /Length 408 /Filter /FlateDecode >> stream xڝ=0.(hQF(Lc-98w~=:YvWxwf~ rvGC=2)J}ߺrP Eℼa&_ }%%XWӿ[[*ufEb2 ] tH48{¶Hp!W6IRG\T)F} K)S!;r-77n>,P58N"Le Pe'/מp.jBjJ⟇k n!<[ݪޚf,XW9si{?CkXbP"V1Xmv?g- endstream endobj 291 0 obj << /Length 3827 /Filter /FlateDecode >> stream x]I6W2q `&\F%Jhyx߇z'UM.+1 Yŝ[{qWW>N{ %RDTޯ}n8|?$8bQ̔- 2?.]fDv>dh 4NYqHi8q/T3r89Q^ц:ϴyw[fa6H,{Uzrϋm>vq6EQѰj_hR븎^5t}JӲOO.f8DjlH5eŜA_U}ꕩt?¾3ʯ$7}s0& Q4Tfn ^u\7p'?~hE K"P^MU~*0Јjlh/~K@;av}^<2i"*`~gP]tBMi)MF{ vŏ{%.pݷî]_j\_~+7 [c?\uD ds>MŞXtM۵j&QwM%eצd>d+K;m.q X1+*R- E,D혔8]W-3MommST /tUq )NI M{]uCR@y(Z@5E0^'4ʂX긪Y$cHO|WŞhl+'L+~ޣT\_ : 㒮zǩh]?Z$IHі^ v٬rX >T!\Wp%*1MAUDT} S?|j}}ma3{PpƜo!CPCeӅ QoGr'm((d{s'vedqj2F/q,Nl*$5ã բhJêR36Tە!6*[ЙZs(-+=)_Uj?7 [NVj8lrS;t?vx'>I;x\V|=g;9`"R2-yd;^O1foE/>CwJ(L!, 1@`FTG{-ԕCO\"*C(a0ƈ{R!sr##bg'rv zثJD졗:ҮD,\)S/e5V`kC R6dz]fMzenmv@S]*nXB*bw%"6ꍐ ~;!%py8a!1F)jqT## =/HlsX*.|i*tY!ԙ!thO4 ['JU:M1VxKEE#WE%h!ޝn?uJ? n^"Ds"B䓒#ߓz`RR&:!JD]!j+$}ET1=b5%S2^vUnze7"VSE\%Zh Ǟf_.?+z~3j >S>qB +cK< ęV#X*UP# ` 0ןwAK1Ɋ"WjgNJILڤosc\,߀̀R)wrB -%,Եͯ$!"<"e"N,߈XA%@,ɪ?Xr*wצϦ8}w,. I$HUr7[ReRנDqi]q*M&O siF"[P:$w&CwoBEwz#VwwBԮۘκ`(xguu I8GIc}ڳoqh*:XQr3C$NP`bT4[:?ʖ;.Dw2tvz R}l'H_0L#sENg("gż#l ;_R'c}8[54qA%I8]Z# "p慕DK]wz9 ]z1N,tlg#8Z=gvT<3ײɥzMe=5ˆi# )ܠ cIW/vq0BӺ oLQ}Cs*@ЁX|ǦBK \k(0 c܇YS.[#h21 ]0,-59iWz{;~k"*Z~.' FǢ3cڳ_a&:oyi`ly26ؿ!M8nl8suƺgo-:z# AxU4z;[ϑJ i[ uM|4w?Gi ]~< YzOdSf$D3D &/M5ŃCy"8)enH z =G8]x9Oe ~G8~qz)!0?vBaQ';JNP( 7Fq0G-M82 ޚ=SB7Ӝwtݒ9yJ}'#;*$z;OwG;] ~j endstream endobj 252 0 obj << /Type /ObjStm /N 100 /First 912 /Length 2436 /Filter /FlateDecode >> stream xZ]T}_aݮ*WٖJ $ aaGL=率/mxh[US֖J"@ɇ4ԃ{GNKӓTqtFw۩$14Hn&1*h7j6W;i|3*$kIV:baj2k/& dd00NK4vz:R5;ʪN`[!6R͓sZ$7Z{l$o]w'}prUR)qaSke$LNGOF ASLLE&e e>L{jUH͹Cr68zms4zr;?40U7.\ t:'^KNAM)>ap40Qt`i6tZhZTz_Ս'XR7d,4T , ֏  Zeg+eа>w9{=ݸ=a3Bj\+i7vGI'{F}?8yyųq,??<{BDؚbwgxeͶgmNy2ɔ'SLy2ɔ'SLy:锧SNy:锧SNy:gSMy6ٔgSMy6ٔgSMyuʫS^W/=}U.?||~Oͳ?XT&OwzL <=n 4Ҥѹ7jn-{]q#xi?&@E O iŲ t{ѳ+Kkʌs(l;S; rn(^=}dK\Bv%,:KvP3Ȕ-WC9D2~5WxP͑N OvNba$=z(֢0 veһwc*CsjH!¸#޷J̳ÝaQS:n bp1ψNMzC8̞v Pw(uhv){; e{[XQa(ۻP1#Eִ@+Qqk3*t#wuPW[ r LMe- ? 7i֦k!Ar$'Ljt5(C! úٵy dqE@{$y*]{m9PϣT9b9VCD`  Zݶkd1ޘT`00f-, >11r9c"IUHzEPERGBzA8 N^?I3KƏx矟|l0rosvz`㰖1b` <-߾N^[x^VꬮmYj!4G&+tj@NhNbK[K?W>f>f1f1f1f1fiլ"birU98RTŪvγAʓTX 2Kt=ɿP* |*svMInZ} Y|v/6/X?r\/pCfUPiÿT,<$:+F7uY.">&y]hqGX;RWQ?6ރ_V. k:oaz>9ui9TkBoO6x{3s)OJR>)ϋ}}^bG[RBaQ]ӱd$ ~ןr4_\=Mo}4뚼A:om(گE9>y ]nq4_~y2RZI:b]+b)HI^ݼMA `Q':T,rB陽pKPGtXlrj c ?:}%&KeZYe{ҀcOɝa$RmP|Lb31UK< > <|_|_ױxqftP$/Up"-rˡ@ly%oeUc+T\5 u\f=cnU"PY[52/b{r^':[iaR.b\Xr+ɼr endstream endobj 380 0 obj << /Length 4244 /Filter /FlateDecode >> stream x][s~_$rfM6dth8( Iy\{)dزπ zpw.(}+5w{w>}8tn6,h"QtSv*1gK k@π5P1q kLĨnOj?+:V̢!ITO~0)~~iTz)N$LK5TvD:ZVgRx)LWeyLgamx熖7պ9l,h553@o'[7畳rC'[HMK#[Z|9@,X)"3FU!:U3sw,?}HƐ6N $HXz̯zE4#Iy*KZD79@ٯJ.&"l2<D'n/,Qn()t(B,!$NlW~+D]InHL_uFPk/.0@qO1υK<.~atך!=3#쾂LY7ۭPJ}I!R)/*q45\~t7~cJf>mEQk#OoRc #UW SC:xRnT7 >c…"HeBƗƓpJOux!>?/D%ܷԡ"2ݹaވbDQk(~[n-$ ~*0޷6O:w N d}32)K{H8޹T5"z: CSCq"j~m0La;fC0)&Gܫ߫1%LP Cp(ۑijj ;y&:G؎ǚasǹ5D@C0z0Ē:ܾ3 <<`cmS!q,@> HD'<⤼Wb!B9b mws"X~m!qx; G8j=vIA<=:&=oMPH7U6,i6K!B6]lJcq-a\wY }bQ})LVmVֶ [u10h}T[ݺ ;r6^l&F㢂EVh˜xwD (jڄ8zNEMV֘%A"[lʓlhezxߺdAZ]p}FQDhmZ73BI@9>_N @,9DBbAvnKzDPAAt i ] 3mĺ1DkZ>&r? ~ mt*'6|gYG$xkGL1FNJAoPשW3W`l ]I+jbF7h] JWОs| 5Кc[PL66 X!1^ؓbc;v/bLlՃ!n}WJÍ0պ1h' ?|!΄cv q6nC8' fESjݯU}r5_4iʉ81{4 l!s4e~8X@8`/KpÏ!02Tеρf6@*kGNGO6WJ3F7]_XOسof~G,k "3Pg9dK%gh@ݣ0ıj0ʬ,bh: w>zқFE'$iwyb%!Ab[W^ ._ endstream endobj 350 0 obj << /Type /ObjStm /N 100 /First 873 /Length 1807 /Filter /FlateDecode >> stream xY]o\7}_Q @mq4:)6[#-dgZi=߉ӬXz#H ,`+ƞ-q-ÕR5U|W W9\(!r\K!דAsIfcJ^ ].cN!ǒYSwK=tn1J/@(X@+xD?,M*43HyX^ÃTp@B-+pj)@J T.tMU jj VO%[Bйic!l ěA@3¶0cՌJi`$czK-$`v؞ak  ˒qkjT1޲ fӞK/f}KcȄe;W t]O]8^pO&?/д|rytuqZ>[}LH<xj<"Yb"jzc: PsqLS]7:ۖ-%QQѦѧM¢Z'VpP" uaᰭpYq7kʟHD:Nr xǩ-wOt'JL+T~_rP~;t؈{6ׯ_J $zؑbхs':5%Eո_>qBez/=||ݬ@)oX{e(981:<{ͽ'?|O/(G_rytYI =y|>9z},i9YI?_3 uSqH \|h}88zw2oq@(X$rC=GP+:ѻ!%kR;AJzr8P6FG1(E\u}~4k@Q>4iF$We,oelN {e-$H룋ys'A`U)*^rAB{ ZTR ]qm=8͟ TǓX ۑ({Eq >DBY vvۇ ;VB2Hǎ"^Mnw}K1YaD2nagjCin 0VEL~lt6}r'#pkUVqB6?ecApкb'ɭccjyK`[ ڠuk<&6d[fj[f^a~ВًIr;$[Vny,TV F nĘFA܈1?3O1' endstream endobj 452 0 obj << /Length 3779 /Filter /FlateDecode >> stream x]~_h"~SIm"mQEsɃ֖wյ-W}$^rHє,۲K4FCr83&I2ՋOdb8MR4y5< eWS5YrST˹ŻgOZgƯz(X|o~fi0 ӫ(JcQi5Ӈ߯i[BHonbpq9GRĒ]Ef'?<ԬnLE# ĘZV"!ܾ;Aŷ F01nMXT }_hw3'9 (8Usٝzә(TepUE[NgIL DS_/]+kֵ0L/k-_I1MLB10We˭Yl=[Znt%?ZT.j}P.RL7j ,6B rxb#Z*_M5XQ_j^VK&L4[,սi :j>@%Bc wy`L%UE ibY\0'l:׍y .RdLw̳b$ۢT ZF@4zU?M.3Ģppf#L"7oÛta,$f:7B/Ԧ6˼*n2{2;ې5gԦ1 "dlT,~RdH3߬MQzxW4o'l,AxVrKu̫rQ"/vuZh߲q. UQ,6evQ*P4.ʅ(rw ECz?RBZCZRq2Ej~D Όh1&ڈ14+{r 2%$TgvS! z8 %J N0Bt9v:DaXÍ 1a ٵ#oҙs0ڕP`#ie5dkw/GdI 6gj^0#@( I?8Xr*ˬVfÔY(0Α]ұJʍb4J\|.NF5w;j,Yٍ̚|SiCI-? Ma׽ &0V4v t -=/%3CM"ȅfda rY{H[A$@@*GQcG (Cw>ڄ:hSiyPwx9{VFka0vFI&kcswD} ǭ$n-Գӹۅw*? ʷ`pi: %̬͡oc{p$pU4p!lKLL"ICKb-ĮKC # XsbO(S?zJn89G%qpNeE~~~r  ϐ0;m.`g& 3R1[󶰉{G-l-LdJ lO)s̏0^]ת9ͱ+2ir<9Aܓ'@-3<9LWQy^ ,W<媗j!gQ]]pQ>Uzq \Z#jdQbs2wByY~9J-ݑm?D`?D;Ժj ״n"g734ꪠѫreGKt0p*W9Q.s?dV/m;G!z9\ڄ4I_tq  +0}'6Qm"~&*ua,&:/uJt,+qUK gL\?052f tT+Gto 8hTP*Ltfb" s &b{֬ͨLwiGcϿQTiu & v?{^0$Si\_y 1ʩ rn? zq?c"ߥ<#A!iXppq)i9Eg3:-MG>kmn-{Ӽ̪bxCYڗ}+yU74kHN_7|ل؋U{1șui:ee+/ߩo/L *}F_ۼ& ҫjRpi9cvcdg_.UC^ė7גlgQk̪z-z-Dq3̗ۖed몼[Ky^Vk;,BT^ &8T8C_z̄ endstream endobj 508 0 obj << /Length 4670 /Filter /FlateDecode >> stream x]YsF~Kڱ=HP'$M4#'/O=Q%)'/'&b'{pI&%/nhhIr"_SJF$TLM >"x͉٥Tڥ4!I&ʒʪ[ǧOL&_BkK&ϊ$59f$OU69f,1ŵ,+N3>}$p;Ju,$mܿ8V8 Al sT,y>Z)a}d.*S`vy(`_65O")bIJ &`br i/Vnp_/:)ff5iHpv)v.lO*6vFi u4w 41 ℜnqצW$#G"A,Ŷ %vg5whl=nZ{deC7v39SX߃X]&Rn=0 39h\[{aq^? PT|}@f7+Cn@>5 ĀAvc1aV!n}wqbҘ* @*!9g qZ̑_Aqh'mCg~=[8ٛb_F8Xr5p8N̉c>uܭFYlV0Η55C*"DYf.EAwXҳJ`r,X%\ M{ xwx/B~@Kةwp8R AP!ݾJ|f8/bZ'C 0hY ǟ.]]<~ljaduhܷuMs81V9HF&C l3 2W\*Ѵf&@i,.(B)n|H?'FÜ>nyɣ/k>.-pK@v{Gˤ ! M9CԥNŲ,%z0  &}$ĶÿBB*PYAXf_hŕ&!`ت.8b1vU5}@w,k\"kxF6G]wQcu1 #]׹#=v$wxޔ1-O@#Y'r^ ug4+U8.uDP]8=w|UHl)v#ʘ^é[N:I_!({.j=U#jK2)$tSPp][=`qrN >w%[.0}uݫzvߺC2/ j8_ohWW{%a]~鯧eZ6BWF0Z6bk՟4KhP*ktx IIe:?Dvo跆soo%Mæ(64T4BzA=MZ )NP )Kpvm K{wv ܭTCX e,"t.:vl<5GMOj(#ǴVQbgudS-cwд"kkQ|LNjEtlDzZAhM؎Fɍ"XT:TYWO^ /w DF*HW`uCh WVi WJ3lXa J/C*( J!1یP>ș E-{<-Z4ys乞`JZϪe'xۈ]B|,Y# W!mH|(SEW#pάQߺۼAe=a6YWŠ֜SCY 4[~nҐ,&O:rՠꆝJ~X bV5@nXU,kfą{?]iV|\l,;K#&%9 PXq#=5yJR q+?R+AVE辴 ^"[]+d."3J~?$˖&*3-$; j#9d22%PȖ&}tQ362ݎe\iipxJ: O~3F^YCW;!i . Jͭx9xwmI[$N)>OP)ic,A׉slpq,(GfR<]ûhtJҭp -{͛qɨO)t]4ֿX8޻Iߞ!)h*#1܇mM!JVWc<]B+Z(c%ڥm"߻U ^Q٤+r["H*]@'ȸJWuӬj4VTƑLtIlH6eoOƳ:f/1Z$8sYoڭ:>d9_'5-􅄈Xy-4:AEeP= @CyE㏷rep"fwX?Tcq: )^3_,V%Qނ`욽9ܭDۍNCYVM2xIH_NU=B)5ٚ0ҬI G=>@=_A1#$7 *d?_Mm4-H9)obN(n,14B$i¸2uSp?_>i endstream endobj 443 0 obj << /Type /ObjStm /N 100 /First 881 /Length 1564 /Filter /FlateDecode >> stream xY]oT7}_㙱T-"J!B]5=݆b߽3㙛F#*Rhj"h9)ږ#qX:Rc :&@mJGDA,u1"xck F iPC#O/>x:؛w<t `T &ش*1J`Lht3nj`VK}_X3 x :05z1 | ]M5u01&`5h``􆎯~e"j:My:z&f6*fz}%fq=zp6Y]РG u܁3)BX 񭪠R(*: but|n1FnkRN0(|mPCg'aMZ~s`܂F\3\@9+.u=p1v0pzҝ~ ] Q_?l||/H/[Z^{w<{VlyBX\jWF.mBXLprưDJw\G[c5 _-N{d_p+Ccއ}|gWtpqy:N\1by~l|q8}8[;ߜ.ӑp!Ü] h\h]V֦O>.^YwWk t<2qȧ\]+̖Ak3czz XvoM}Ƀj˽Ha {+"ՒIQ#(B4HQ;,.v>KIطKoyȽ7*7 Ҋ/a5tBMV 82҆Akע6[ڋ(զ౎RT%nrG`H8njJȠnŚ[]Ϟ(9U8@'"Pk8`ʹpF^zA5[%Y [7dge k#{A1ӿ2^qn6h7`VWcc܏kk- A>Cb)ñ}c<$q[ o /p<y‡%E"[T ZA(VAڦ9+#hAlgpד5ħg /X!E,X!)$=`ܣH/z^^zEJ^_[LUKS[M|vnaTB)/;M{7uQP&dٿ*#t/6Fu״- _9 DāExQSS,#$Z6'QڈXb刢ECFJYIm*L۱^IUDW?9UR h!dwn !RlŢ[*XAvGAކ0A^ / %ȋ\`e k1{^Y| ېWggx/!lA'16?6 endstream endobj 568 0 obj << /Length 592 /Filter /FlateDecode >> stream xV]@}ϯafvf>2/.eֺҪP{dZn7Hf2s?ιsy0'/F+DTѓeJ['FS1˫#~u0^Y ޕU䲬0n8eҢ)+C:٭2BYJ[Z(?H|{^΀kGO9w$ߖac)gɌ9$W&- ߱msz:Hbl"ߞat|ۥx۸a^\'OMZ~A8mP&fN:6Fڍ DU5 Cpp ΆJA~G3+`>X4|OUcenpH d%JY۞}˘P\7e˸,gVKl V.i9h> stream x]YsF~ϯBN\&UxV7@,A <$ &E.W!c{e',q}tXȔw:dRĉTί8;gIDߏEU{9b~5~/SCaIιHbCy4r?4[Ob6gaaL*Mo2s~^-QIw:&w;/̟̾1EUCX&,:5}n籒ڿٹ7qxs3cLg[G;/o*K[]}\^j29cQSYn}_6>Kq6UgiI\Bf2欙CM1˔Ù:cE<>;WI4u?-*Ӈӧistgpc)[2umKc͚i2:BxnfTa:^Gqo^N/KΩ,r;Kz_tǶ{M8˚߄o@orۭnaҥl?dkZʘ%MMmOO|qRjCR$Co25n D/QfVr PQ7λuӦ0%b¿t"Cd'"zSU> pЀcGMxv{l*\Zr45}0 C|~UӬiZNSY4nSNJgMAfb S훞+)[ҿaGtA'Rhav?-!~IBw|䜳.q33Σwq+u{ } 1GcaL8%io!\i8#S*ak]^kޟqȬ(tf"~~pAo#Re?wKRG: #E~T{o@xp@'Z6DF$~cӮӀ1" kHΘkd=sRngCT 4$\ aYJ3[Eͨu8w4 U/!z(C^یCC}΃?H4\!m*@ kTBODCAFCySzf%̹Eh>CWs"e\9B(pp~1v`zP.!#"خS*bϾl7'J STVHxL*T6 ؘVvf7CI)6krVh59uښܱ<$qes eqKlD'R,@Y[m-4RqāBE98#v>9ҷxErHb՚2Em𯶂>σYHI+Yp'9mrl#00JǨs'ݬTK^REii*6- d˔mD_P#8Xo?fą(F[Ccu2[aJ{11ҋ>34PƶS*|u+iDKy"ite_CzY?sqCxs"q;|: LCvEh?XD>:.LG܃$ kVQUJudǏHFp+=A=?_g_ɡ& r"g!]Mz)7Y'&RX&ѝ1)? LRIsy!ZQcH;_ɈE.m-3Mgi! Ybo5٬úGs__^k|Hx> stream xk{~P ωhM]׎c4F] PZ❘+K,%œ(iVKryp(=( Jw=j>GdCxr'rHFR^MGo/~|׺hQFߞ~C94>nu741cVv*juq"#Fq5̴|8fNIj_y,f[z+DhaeNQְ:Z@UNYH3\桖b4":W_c ϩR,/2_Nr\jJ{wZmvnYyqϋj5˛d[ZA^!=gmxߙELJjzY0U͂Iv%oJ%%U+zsrA7٘9s@`VhɤX`wH2 & e@KO+ff+@Ѐc3lcbQ`T |on`ϗXαJ`f`R":,3,o8s, ,/ G* P 雉J52FE{TӃ3H̪B MdF80o;nYkAm98AA$ zzddfG" ^+D |& ϱ,>]l ,`Ŵݥp ґ@~;oQ<3?pcč9mNe<]z2!f4,ƓZfXj~AS^`ƒD;5Sqϴ!C˭$/[x1_ H|CHR9dOp#Ra@+pmS#8><{Z"Gz)z$>&J@yӅ τrǛN_sU S$X>F돰|wo2l1T@J pKܦe8|z+(>fco&փ3,3 N5r@ AN袇nYoem_v&qk|liw %]@x/-䯆0ȱr}/ZĪKOhj S$RV"Jl׺'QݎyIz5{M3 ݱv(@҄ o|壁e|ˉDL[2m~ښ}/KQh8s2~e1Yh> e툑OձQ޸տ!c?kF+wܸ+ѽqCaQjSLZs=s9WbW^ZQ_6CwxE)9%Ҏqt76ADdLQ: lzj1B_9U@aZ'TeFETfB;CKzany=XfE/}q#+Fz1I:rK^`exL[;~dZw}='DăMhJ +L߈&HSQ#T^S޲X Xn,x2dv qG(Gr*?~Tņ˸g X#,F #9Əmë~vM +6UM&30 R.Íu4=2׾X7QFI+(s&Vbܗ6i6\SىEEr𣔹MBY5+,y ]y?$3Gmo\bj>̊ͭDwaB=ok|.Ko*AШ@ÆіuzƼu̱{ &˶F׭ U zjΐU;?U6McL.mW{ryHOJ9jkϪLE(]'ت1alHw@wx#-8tnpW|RCI@@bK ]B^b>Vv; 'E؋X:99kqML:1{Awd|MǪ=e{ىXtcY:B^FDܟJx= L:@ZT.q\ZdgziN/vzb}CF"e1IKbW.0đyԣ5z"w-GFP_}eDp4o@Fxʯ%G2ё&GQ)[OlSƒߣ灚:O/P` KNOI'\}ök+@ |$Q"LXOO:VvA3_]6l=&'%/rXdn`Aa%&%y{lzD(sG͚{mYLI6_\ zM`r0C1]b5 mdl:C䦼Ͷ7eޗtbt҆u|: Yc)> stream xڽ[o7W=Ƕ!qPVê@U*3BӰEXٟx<"!VJ1hZ%jȽ GJj ZCiժ SN ^P`5x#:C`> $03VlC :ir.t:`M!LGe l:pLxUl)ntӇ2kj:9Q $WAw+bBJDYd!A檡i|jg3;Az60Re=aii0,X阾LtLWVM@AO¡kBJudF`nс[Qsp}EcedJgg50sXCMI,t0&Yu* +fYn 10̵l NUX τ:9{-:8X-wQc)</> Ćnjݛ7ǫ7;-MnVj\,.9N WZm8/U#-5X S4`EaIs3\X؛a[FΓn78($aλgI|K2K|ؒ{6(ɛz$Z2|4U|=گ, n!>_>~X!XBH(GۗOg(,YjJ[Rxv. ~u}8J XSY)siO,+ט`F)e3JUe_J{D'.6-TGDd<+Kpɻg'׫tD>^/;GtlBd7eFK|+ȍ AE;[;< ˽m@m_ğ~y nMK"io,J5ƣ IDrU&3xA ɴ1/>$B>_ySs_<7ZVzL>lW''o*1W/}Ke݇%>)Ł8ġN^|l%YeZ;"N6:bR˥R}X)i5Uі9I[n9iO[E6Mbڗusqʈdf<8yS>^2X^OF"6Y)%lUדbW(Ɯf.Ɋ1wʓjszܩƜ^/k>e{Epz,TcNr"tzAwN-&M`IfH`/gy`#ڮ&Z7&e+֮=G{w> stream x\oVƆs-RҢMȶ.I3Io8%83\ݳpB02ؐ4Q4Q!En8A %,#]"Mc0$fYJpD#!H^?x[5Ž AƯZX*#d=+ f(Kd`@8':3!:Υ)0ģjverءqߪCҰDQ*ݚ}*̢#99D*jR"_) } ,]}ѳW}}fY HbH }8;PbęLK9v?~r,i kD]83$? QF(Xp)U#9X#awBАLvcL!Idzd#O!VzpPtR:rLM^oI0Sqrk)  iL)Al2Ӱ}Ծ _;'-4_:H .9,=r?yXY{F(Nk&1F:Q\ٶb|׹UmLZ;^u\68)"(z7zwE,l:C\g :K%w7Ɨl{Q?n 6MVeӾ*XR*BƒRSʡcDj}ΏH]}6mD(gGFv sdX[<4 dFDNd`>L_3biMf,i/ѨEk.1i6%` E4TF{t/\ėFZV*+`Pr!f-pѤ^N##30U#SG}Z"d/"I,&;!!6bm, å6[>Xo{3)TlU)(+hŦX.r.dMUW6_5UZߊy-뫛ܣ-^e5^/VSQeQw6> ĝeftc\q4tFJJbY9$"0Z4D ẙ/X.'1Q,֯=n.A(se%ufYا!`=$QĻ(i3Cچ!K 86Z'j!r m徝}רC& ?F;T9^$?PuKaeEo W2"eJ1)M+xBKOILvLֻ3-G̺%:V95{moٰ/'y<$jgAq e#^noCv8HPd7cc#H$x5豴!9ܑ(Kꦘl1() Dv%6glȖ{K*(mu6D g0*gٽ'(l5wg6|Nh- ^ɇ}pr4ۥ^{L}t\ׯsX|r/q>(PV9<^,'}h M4&B ғ@hI@pceKD_"/#e" F7@AD4:-9##!R=~<wqgOQn@*i|mԽ~BytA>\_TRma W i3ƨR$p+R>RP*,w+T$~ψ/Ucs */ô~Qz7Ֆqjm}<QmMmTRiawQ_Q-D~.]MfnQ=-9$ POypxS?}u^e"f`rz *m|k8{@ P8&,g4q6c`s[QfkÙ9ޛa+*2P@,o)c""}7JD2M׆;:=II'j ' LU.X`  q( ʆ:&ppe:&*Z[/OYdb"Iϕyb`.lO6s7lsq$`vy< ;t61c'8.gusi",Ŷ a.O2W /İB}I%u[ظ[șmTpB ~8[OC*(X ([ +̭FW"8 ;Ҩzp{HnB0nY[t1BWW!?-ϯNXیc݌U;8t?`1j߈*kIu9ߏIUbNMUĜP=''{H ;9G\wG(`5af)IG:۷'l.ڲɪfZ(8*C{=)6uq~ٟj wì'tý"HUS\;+l1^!-qH4.l{:bݦ\랚K%#4q%OgHmmx}<0XFDw`XrʥAϱu(qNF}:t:/Ӑ@Sh`_]-O9B >B Hm_*<=j=%zi1vR\甅Cq֭cwL@Kz~svn0{X٬E)Q+NS^k b6P"]< =IA9/ȶn=:K$%҇|0:Yb$nO=0dpϺc,X2dɪƘ|9w%BFG/܆܍uY-M]ZTay(ŦeHV7)#SG#/{*?k4XNax)L.S:FO ?hډvND:b]16}='xpnl^s|ȲMke[XI/}??uRFۏsᢃGϗup|  Amj㈊,.7&V1uvsfhqG޺eΫֶuS@*#H#A2H!!dUqCeWM%KԺJ]Tn%6j$~3HhA !޷+ }=ќhU۟UiۻL yA=3C;QVEJ|%OO=q=QرG )Am0Hxf> stream x7Hd<@xG8L2qYcn4wd݅PR3C<4Pgҋ󥾴$9,"0}ӳ' R9˜( ;[WЇ&.r,A30{|5;LQ)Mq9h bs۲8WGr`f$gK T@~)b8OOZp$Iۈý*Hz(H?*u#TI]WGTEJUS>ȋxQ^/jwZp?A(&$,Vac&fψɣGLX=L8iS79DNnyw.;k3"qĕf ISA]M_(3e[ כ5]#0&imBDI V?׾Ic8g"0{IT'"ɤ*a9ohpG?~^9o>eV 5UptTȇX1BaP4{"C{ @%t/)nWAJXL7z^hB(4#g6J84V{pfU0MhnHՇ(:g jI+TIaW|&f' Es妜@8 +wŽ8C4k!jVNZXXbƣ|Xcx!۾Ĭ;v.%[9NhEq oWފ{uG%;}nTg0y@Ewi%tp:}a|lﱒ?Mu*% :ncuDvbl,z:XIg2M]&:N𱫴km*9\,RI~,15s9w\:F=umDߔdY )¤1J#TŏܽdJjD2f[u8 q "kUg ٽƱ%'!dN`r] 2o3D; 2RoS#_gf{uPGq|X˟CRK\Vݙ{a@ <,ؑx8( ^2Ao\^EUf}çN$¸V>/Y0DhT QӿjF8iKf'c['~T!A^ I~tG)$X <W)\y.O2uެXckFf}H%0?/TM߅V 7 S["ME,tC7s["]Q޵ŝ6bͼi{:x,gmh,C& 7epp>USnV.^ TI(8S v7E?=QX$[+T"142Ί/a2b),9rϋp'smskڨiI0"2zQbmynn5P1pbiY 5Q7e `i:lVihC'߻L` <FlTmNm]f\8YC6ǒd4tV'MJh m2`Ϸb:2ג0N!ךdkBN' rF[V"+p]̵owfAE%S[Ѵź :ס;7 ,QRj#7"`uDvnDk1d^} PVYd*z1iW1'$$> stream xk&}sQ$F'qmEk(%t%R;_~}g$wyKIw>#۝:fك>yLPT3AIR6{ڋ=iE^4f`]_;8ղXfYo2Zye_~*R,҄R jNDe7]"Y%f&!h7s]~xoxBvrɜ^8aTYTlrZ$M/S.c1PFݭX\Т% ,&EO&_$b,2iSۋbW|W n l M2ĝ%8Y7I*)/fpg:%Ȏn/a.0qZ&, ȶPgp${63tLoAO>ia'9 &ˉ !*8Cgdk7݁xëҁ 9߿~B!D&|%l OWzՃm?B[Kq0|+DN#8ܝDnN`QW'3tmYWCZh`.4PNR۠P"8^RPhzUayVPk0eR{)^yƠ/[XƔ}<(E]/"Aת* &Kx Vx38TB~ R)R='!t@G>E?C>r;pP@H`GP݂b\==ŮDCJ~S`PqEXq y@5SK_lNь3HŒ(PiF<^]^l/uceF-Uܯ9_欺M56^ 71dLA?# j*Г۩Z. #") QpDX-*7eմW+D*]ɻ*QWv}crvW_s`PgUw+eW@.حm)zm 9hYz~UnrAf)NT5P ŃpˢXgpWwTj ZO[:xДODORXX(X!PX^, EjLEwmetYwJ}J]Scƈ1qxQq:=Fg:TNf:uqm{dip/>NAmcx?VsMt ϣ/F92ف/.q w@p9#K(WO`+K@f:y pӎG4=w|%b;`_hp,ꃥn %<ȱN=}SE*8wcFc74*:鵝>307L7N5lAjbBz!4wj+)v p@X'4Я߯0ՔɈСc1-iATÓs#/_W,SA(ٝ!ty58 ڜ.OnLU1*A} X Q›rKDpJ` ԃ2K̛)}.䠿@\[h  !ڋ/Xừ?jU]pI71 j(49O&f2,_v[U@,Yo*>xPΌmXEбĆe/]w[ )JA)6qv* ҟp_6x|՟拮;@߿⪦lZw"5"o_)Lع2J6I g&wWUg-oՓ)j]SC]k..K&nYTte LӇ_.jWa닮kJNw|a0L0nVodU{a*˜PZ/ʬoFUyӋE|7w`el@!n g ݯ!h4S(еuSxE^HKF^KӇșfѱ3`p%|4!0 4F%95juc#k&/68^EXX4ǟn> sU!IYaElL߱K endstream endobj 722 0 obj << /Length 5170 /Filter /FlateDecode >> stream x('W wYrB#9 gRnnV͋_ "Yj2CVWgivKMX凲ڗkIRvׇ-$w/n뗿~TFN]΍CDxJD7MYx@2#44:FRJY7t7Tlæ, {-NZ&,ܸfe)[~r H?2Ŧ)T TSQ[HKthl-j~,Hb-\sCp[z`s^3ϒ}swiͭO%Po$N8Gi=tLg`j!j#2j X x7,25/v#3BC>ÝAڱ7l?f7O]0=N] IkZ6uϢ'-4ᦪ.;셝 Ys"^;mlDGr.pͼ{-mg@摑ckKguZbujIA>3˶pﮛnup- `L&m\ jhPg|L֌,Gt$}1+4.OŮ+||46t}m@& )`.'Y7Nb2>c MJ2ϲFa'I9jXR__7ا=4އ7~[s]7Q" zlލaaCաhKCAjX*6E͓{t%n{68oreCC^4EՓ.w߅;>_u-j q!r")yr4hP„aEŸa늦"\/sd*>!)_pi.^TՃ7TۺF&atp8F +>ayʂR,S|SKœ}!RBdGYpR6$<0;+̀Ypa2q& fIƦ%X5`}^~ OR'I&Zn} /?3Wf@r#XiFd?0߉s㰄kc{LDh& gbZBI$= Tt˚.onJiy?kADЍ: PCHxYJɐ%AgPdbb1:.cz`]-YGik=ē}޶tZ/= 8!Vu 4g[ܠt "ԤR&ow)$}L8ݽkqs8vdۘJU o@$&))"$sYctO: *@nb`%pG@޺ =nn=ڒnaS|n&=h`|om,Mu *'q@I=B;Lu. c*9vP#g]}d nsSh2GHaPA4M!T Q!KP+˲MSW5#x$pA$_WhXѝ)}xq8b~ݤf7ukP_Cj ʷ9M>z〇xB-ȌwGx>qN qŶL$7xrӺHy6G, ~/¿r&\MgM*4wk$*r1iY`<!BuAuSw[G+O="3튃I Pʍ/u2Sqbm1-O2S3U $]YLj%R?oE:wO -=WVXĽ Drd0&w iJ@ fLef\-&C~Ę/> ѧ3 o#CTCK~i cI9/ &-(COnnAv^.}κ*؈徝#8SAA;֜< e5\#/3UcM 0>+-6'nʹB!Ɔ - [$vYB{VDSFԲ~PdV{v> ::>UW.ΣA>Jg-!n>3"hu>C<#/9HFC t/6$(ֶ5m!<[n]#苦RTxC`-67ǽu8߱hؕ0µW%|+gjUQ-")-3^bz(#`%,ȱwn/GP.cˍi h"t_th?jI::Ւ-*(1 ֡` L٠|LN}M$-B!ɝf\g'DaZN,JNKS_Bq)CF܉Lկ,X.5\%BJ4ybҌ]H2riƆPpǒg>,ŵp%[wl@r7ղq֕+ٱEk:[[7e2foGLE)!S4A1Mov/M@#Ē$yƶ&gS&f3i̲op3OB àF4]X.ŕwN սĨTSG%8Ⱦ3 <I2Q$W]Β$Rf̠kF\֕(w w*=pMƆcxM1:*Jcb ~!mTCeH3rZΉ4ہ'3ध] |}kg\e9δB92y6ctkc) gdc O1~yp1|//=t5(RӦqD`]i O+LfnW5k}q euᢹ+ޔv\9݁)JpnĶFu9le{F sz*<Yw-9\Uȏ{)O_ǁRs<'(8#~KaauJUqvO|I6.1a<ÝgNOwWMbdh"KSU]ou>aVS4 UF=CvxUո(}%QvPV(%CYxmO.*o͟g>l]EK {I|};-3'<3 EUsJl;o Ӈwo][%Ʋ;#kO@}<~~{NĠMJ?]k=9SnnczHiǃC1Qէ"scTU<9&)Wq7vqO?`X7,lH%v~\4#z{c2h^e9Fl8:e(&u٩wHqfl10&'yJnNLٴ2 yڟ#x#zw `?+8N2'yquzv, ,<#1?J,JV3Wz7ߨ7q(:ilLm[eoێ+DJL`zm.l76WfgIV;B!35j\}?$qQze0ʝeY8romjۂ绛ˍ!Hйoca]A|Q')> stream xڕTM6WVDKƋMw-"(hH+RͿP?q 3ߛyPngi\_g7KIqgu>&^vhWxݛ3+6q}$2}a d(Rs$ endstream endobj 735 0 obj << /Length 3573 /Filter /FlateDecode >> stream x\Ys~_GMyHjJjIMRZ-&2O)PekOI4  t^~~zA8b\ \,o͊`?Mv67oVTEٺLx_ i/V ##ǟެ8&] ].\ { X!Iwm ¶DŊ ɔkf۽I]t # CA]wQ?.(bU-nBp$"ZJHh #$l5Ÿrq{O%er v֏R7T E")e;BZ1jgp*޹$+ҲZ#M"?2)E|j5{e<<&c

`<B#k~o;Ch6ɒC}z]1w3P~ hK*$$NP/Z-2٣ (]+ B3pYl:/is<uF>oӝ?$tmX"H A+ѫG;|oVe=a$j!S"/^8e߃,DFBS]Tbd^l26)]8:drkƜ 8m|0nEfgڍp\H\ʈukٚ= xj+Ò0Ï߸_Mi0wOF1^%q0QV5cRviGkJIg#f@DB!a{2?Ň462}J"`F h4|ODx_u@$>_wN&~7 8g>yZxiDu(g+ᰎr$ 5·>@~rnj `rPb wLf׹aBlB+5HioXFiX_ q xR=)npkMqĸ#QO:#\1leU$, R“3#+#x{ Qc]sɃGrR5qG'j p h :ua6TLk!e2i_?td>dsgfƇC00Mo-P>4ۇA.N`B"FoFzHm  ~#.\ߤٸ^Wk/(I2Iu/\Vm,$pAPmY78͊bgǻ3p.gFVGYpLX[h7ĉX}a,ܮ.!~VP8 f-Wj$ 8%_Afp-HI˴xI3C]}lh`(lBEsYvB]x|$9>:J0>`<5蜹K0CE꫈&׫O  M*ûq|UՎCn&& ̟o6i7Elr^lE>kT- 1z-%nOE@_Qslo`B=v!7Q"* T̳2GLn,9u 8ZGb fRD"Ze˷A!+w`:QƟyg$L=H>Z1'Ѻ:-UWLgf=Y_ 9Qt8u)Dz>QhyfGIܽ@-PeaʋnIuviT#t Z xlrao{ /$$DH;HIGSj ATIE{tu3ҵ R:j@6nZ_` X[sUvi^ʯy;(X9l\I  s0o*z&xCwK"p"ie,X<%\ @-6 !dƲ'N QHP" ԜY j4y6M/8H HL:1C9n2 %h@R5(` 1cPNND9(׏5 -{‘iq΋%ЦPJ~yد?곋'Gl .1}RڃTXQg0H8XjnLJr[ffhrSU*oam~d|k_Wcl}mܾ!|_( )LEQ~tiXv!Q%3r$&yfKA%.Ağ|R)U玠,O$:{f_a$: kL0itf>; Rj^w:l޺T}w&,{ ]&{qy:G3?^T[[NLeXxX55V"e^]t8^)e8Z_;#uCJɴuy&}H3ܧn3F3ol̏;̯n;g|Y NĹY.8 e}~eaSXbhB-M`(AhánH[skHhͧ*d`L)zkI~,zaq4zvy]&m o'efX,\ˡ D8! (;QON?_CA endstream endobj 674 0 obj << /Type /ObjStm /N 100 /First 870 /Length 2441 /Filter /FlateDecode >> stream xZmo_A_>Ep-d Q7Zf}FVČcجDi83`5lbk'1G  00gC ;4 Ps'z@BA6ar,hNqFGcoMhXq߳.OA"*"? s!G`bR~01a_aܘC$z#| >2A߷(fbP.mfPX/gPC3ǥ:'h!>@#MFvk 7Hx_EV|7s_X|ahz!}/~a0 X850AZ F&lF]zs^Qog%UD2'19[N&gO OI̒F*@ԳI`{^'S]*̌3sTEU_˫jP)UE]z9UMXG.Wr䐘sm+V Fe^̶0-ahNjbUglfX|EҔ=Lñ%c5e %AYJ &7&Fr;-XP`;O9Ӯs =v6>6⁇f["f#.uD9}9vVsIjwcGREcRX ɵcV СpA ٽ>Z9:j(޴nU9]/,;d.Uq[VRO*f܌U,7ODns nbOJp%'[fW<r27DA6Z>TפD`{nlnRb׎];um]ץ% 6$l`2 b8FƷq9TM)|<>e& Ml.qfWs}ۮ#^{7>s[ff_b&&&wISȽ.A+u{RUv;o rhє{|[m{rD>?YR.؄B==A\'_b;<ݵkWP:$0me=WX<'PoF+"'](d"6Ѻ^+,&n`Ge+Y$ʩ,GryӲíH ™eTsp/p,i5k orC켼'eKw kOvޣe9KY],<Ϋ3M~׋ANz!Rnu .O~DZmzg辰klL%9 >g*wGu7a endstream endobj 746 0 obj << /Length 3569 /Filter /FlateDecode >> stream x\[~_K*trejg:lm%JUv`wb#IF3mhAd_[Utܛxݫ1Cq MzF%QLYnOR,`2[ƪ'yҕy(tY|@(-3*/MO:/ 32γf@(}Hʋy]L~Q?XyYzI!1FzD"F+;vu< xVU}V$bB͊eDގ=%"6"JZ;F}DQQ,Q(>1Yx/`33L0a'aiۄރ?mUؑh)u6lB6X٤l"{޸ږrԱ+z O0|SͰ&nvH$J2F<^Ygvv~6Ҍ nq+ԢC& ݦ$V蠈!iC;bݳwsL2bɹs>ƘxAaI6a(xM6,hBʉV9V f*2q qo ";(I(=}cn. #n-8c֘+B+_4OeE.Sn_Ma_ڶPܗ-"|dX"*hԀy[ PCkO۷c[h9~q}ru*Oyr_N>SQyf c ndW*?VRjX9Xvob{ NG>_'\#\=wK?H:H.WD)b dY%\t]JrV|-fmoty$hhie߸Q&S]ܞ&v7!Mw 5X<uB#2\U&0*50D?$H! dz80|:tB$L:."xml0()`D\:!D29]đ vY7p$BYkL0".a3V/']@9Wei wՔdK3(o' M9>G'g⋶q.ǿ"6N(‘@3U['[=+y\*Vn@$ D$mfVKM^WHӼ矲y>+rSgr 0Ha*fTŲe>/F+]"\Nì)s.mҠgOۘDm ȯzնSWTvuԀ/>+{ΘCnWs4Ϊ4gԟM]q|43 _Y٩Nw v\jۺ! X+aPsrrˇ'i(1\JAAͰe#nKkLDul9Ζ<+aV}2zt_|9V6etA…Zb}7~UdN95E &bSlȸh U9F-b:E `U l8~ aP"kb_@;rH5"Ծ' y N Kcdi[CG:X2˫'ydL@c381Bvrz*沎;J  ڶ=tBKhGVm'M|ishSh˺e{y䍬~cj:@ځ2=)x `b12{J)=G0ET:sdM=^?zFIP%ܐ<]}tN? E^"@O"@<l}֝`N ڥ8wag$H'x*L;M -ShK͡x#odىb H@\tLN6. 9 ;̔߂N=׽{{~eis0WBk4кkbZ?I3Qz5B@&2]d 0y0A{w9E!oiX 0Wb֨SF$RܱO]ke"EBh:zxG"{@[%C g|Wַ\O鵦N<Î^ Wy5в T70^]Ev._-JWHy6i8S΅k/+M̫E=#LwN޿|(#/GguqGп*>  & ,AY@w6UxE˭~6VPzCyw,B=w*Gm ﱒÿ5@(77^+￩9Uî> stream x]ko8_aX:ظ")RIgf/tش‰Ԙ$N,/uI)tv(ȦR"ˤ}{NzqiZW^ӃɴsU`Si?I^~6V ŽYo^i.\#v Ÿ<{JўҜ̢DESJODAѻXmӘGMhDtnoCWQfHhG>~jE3=ż,̎RпoJtk^W757#sJGS?!=sAM=;굔39= jNDK_F7)<^gFt,U=&|E3H}&:#jӪYҵ,33vق&P?p{;'z̝#'AR ޡ6Q6C9AxK#!v9ހẸ(PV[38o4cNIh _FZFbPc8? F}h+NDϨ#ZNٴ8ɭ;URP+Fr%c-p|!;2|{yaSooG=5}0x{ wǍڇizEQ#xLi \Ҙl͍: rrы Snd./OKԘVy.zQwh3Ɍ(bު_Յ.xhg6`*S{wn BPͬ%PA,>9G?u(b51q rkkDkϋ%eEZ~|^Q>m2Ύ\:ؼ(ե=P,ǖ!9v!t9!a?N\lŚ=ayAnjS$4:5%wb姓ʾ@i#1ڷI FC|]Q5U8[SU`a$13)%gf2oH[|ʩ{Ǽ@V&nx ;A^;E,ܲ?3zk47~\3I:!r%h(itNLbmHL)?f.7˧ЍP!Yi~Z@ n7!ɼ9uP`061Il wCBH{ԏE͵={[o'W2oC4 )-Vp Y8<;ھ,m[ײ2pfz{F%cN!Ѻ ;v)$$б\ݢCTJ-cY[,6DTaZ \_[ 2 ZoPsNr8F7ZzAwFت,hKOQr=rȚ-J湣d.Q2t3;b< i̢٘2v"hr'(qSZ(Y ,O@J[Q@!(tCYy3fiwJ 8 daT&<";E1o[U˼Rf)(s[ ˂2e~P,^؎1G@ȱk&ԴHwvZmըRgDJ0c>ֆ̙Cŗ T%_Š׍p2K#PsRE8,3$T9Mh+<&m$>Ws3p؟6-fXcG[tڧ'0 ƓWJصOO7uAOI_ĊOǖ|pwk7<{\ϑ2(W 6r瘟5m24ynߛ[hs m7eVk4q}agձe_9zrf yƍ Қ*+l+;ZƋ3${അf6ŀ}ypO賆^:W@IrL/JKMsKhy9yIÍ &cڧCkz0Ҥ):t-12Y6h6&֮>Ӓc5:NB:N1W8pmmFt&V IXl!]Q%Mi1.ـA@KTJ)[=b|z7?5p++oF]2:t%j$ZPK jB%ݻY'Qw6[c ,g1J8 o em98te 0Zخ4R'z~Lm6-*0"o;޲--[9Ue%xp U=3#kjyX!̿NWu žTSnSs06PKdN;8޻H[[u̜M@L XU)QW?UrdUN}yu Qbj8ۡ^XPĀmr!7QӸ q,6` EH12U*l(w-^(-'6u]D S>]/|;7%Mu?J\)+ <ׯcT6{P(/VxD8`crUv,>e;ޟ1[j2o£[l> stream x][sGv~ׯ> Y¨lDڸ*IE4ڢ9s4e8ry>=}n߹t]'j{7+?]8oSMe5\XH.w3Wf(6KEIR)= !+[Sq邈}Xt0؄HiV|W9WFiLp. ù8Ӽ44 001_^5~c$ ;,1e Xץ>?$8o2Ϣ%ccoӬ4Dk`bA dgaC`wBHu|΃yHГ1FoQ'T~"-娏8yַOcGtzdD'mD?b#:CX+QGy2>2J>xQ(O,ZH>gi/RLe&_?Y(5~~aY/? ܚGQ8qzB#-LlLJ."$U˘X ^NP|(#ogc~=c&>6V)B8OHE)B a1/J<2&!%5aT׍uYXDa1{J␨m,(%Kb&֦*FQy'9O6po(SyΩK*&ʀ`/]JA|Nˍ앓{ƣ̼Ke6U/&bl/"umNϹsgkH#5/嵽u|ҝ>S dMmF5;٩4 jI"p[.*Ey"I֔c*DbpݤzhR إ`Zv)>Tcv',啠9N, lwCDUd-`{͒5=N|cNn JNN}7Gwz@ CHd?9֋TW{f!;T[VܸI$7sF[I9EEe^bUD"eAA}Z8~}2oOQWG3\|~TKgv9v^R Oo)9 nwJ$Dim"' nAMClΞ뿳ׯ+K{y_5,hhn(/r}~4٘0cwՉө}Sd5Ey\3y&ACKnN]XzB* 8ZSy+Be #[]Ad Gpiɵ6`+S ̱P8ȌpXBDw|HԾMXe%kw<ƌ0SĆHJ$5voFTYF 9 V1#>1qEk{fh˶q4p3S%a8 9bLM lcHni9:

d RTE.׉p&l=iaTzG8(DH&RpAwh pU!dZ!{+Tl1/+ǂVA:%an5\64 ʘ'Bʙ9F"Ԃ {\8EHUViꇀ16šJEB@@d<8؋^Pϙ⁅$`i jw1vLCZIńMb9"^1Y`Nؔ]"} s ,##HD^*!_-vjcBk&C+ S 4R9&}pm䀝cRd 5s`{F ^ -@AKCLyטѸJ|p$C}c0a1 „ Q,7kȦ@dM‹MX A3Rjlm:X2Ķ@X,fv/b@W@Foׁ!}&|OA6@,p,Ƌs?tIaK $yL:]\?t(Ii61wtE9;$uRnRIL|ϳpPϸ󝦘UwҐ1ɞ6z,3 SG9b$BmҨ.^i6߹$k;B''Zfeo4~Υb9|T=~MF8w 3I"Q( (7`fHfu`'A{Xe3Xs hlXPV2Dz>B&2nՊ"сgpܡ"wϞflACR5Y[#Z LA/BH_åIͳ1,֔tUc؊vm(Pzjrr(apo1B&T7a]s@ p̔n(dzv >K*qk`1%/ ͵Pai}p kK!<$Ƀ*h'PC^/ +q b> CʌGeW}{; F fMQ\5[y`U+pP#wf] (Ǔak>7\ #_IBЗ"e'bD 9L,5rflK'ۣ;.` ٭AGibxBL :F%tK|{)κXO(O Gg~;?*q#nGգgֶ0ю!|Z{C,a4j4n$ MM/;JP.e'#/: MqjBJ|J匦zS/ޔ;:^%w}X]S@fH^>9OR퀨dih8%|mo Cّ9ʈbIa8Y̼eFUF4XtLgV8Y> ~`̑wI,&Oho1X4MgbOۉw;y|@i }D"i'{Y56Icno4]v!4om'7vsqSZa܌Yx_bs50ߣfw$Om=GN#T^&7zԻvN4b-Kqڵ?ZYαgYVL%D {.K\ E oU`i=Gl95G#a=`XQi Iwh{j_yB]XQPa3^mǞAo{KT0\tYulrZɼeA*E'\xV4a=LTiTghnޠLSܠ"ذ*Re{Y̡Tg4+9Pi]U|5z%7c/zҋqTN8[l&7ڥwOެY`|oVq1减Ӊad؉N35Ɏ|f$"t-_ ?G9:ʵGŸ j+v>٬/o74G6滁m ѐN4CZpzq܂ ^#TuP+rwI* {J0u B>tx23i|~.)1Op٫> stream xڭX[o7}_gK R/QHXU[3eCB>>&^Z1bCǫQv.O)݊oh6JDh>4Ɠ 5:h3 R|0з5/L㟉xBVw8lk#Lb6Cw5qbrӇeb|^_fZ||zyvz=ۿ~YzC9abWwoހh1px^z[,?9߾?y~XߜZmOG&8uȬaT;ƦT%..@|y)֛V'5?ym~^ܡcː{qlku*vBB]/ukhџz8VnHeS#v+bmBرv ΥjirX{Ma:֫S5| F$JaRI^5x K& n\'~Kv/٬m$y!p$jE:v_ @TL6"9ީUµ4;Ge#'D&yA>i|&DOkդW]l8&WW~zwCqp88 D5>* f4|MGoN6㳯).)dgL endstream endobj 870 0 obj << /Length 3498 /Filter /FlateDecode >> stream x]6=/*MHJW ^S\qMч4(^{WX2,oE.eٖe:M?E49 7݌գ<~ٷ<ѫHґLY|z:J$I\7c*j>]L5xYMmU/'Z֏߼ZJHINRhLEr+T-L4MoKiSM؝I2NESsÜٽevŋ.@2ߔ}Ym,_6ԥNIT,-UUsX/˲l4&,i5ؚ9lG~֫餜=W?sP5avNTjI1?DazdZS3{q}]}܌+M*Sx V#\NKDIrC qHiIcΗrR<vxy7ki(Ǎ Xդ5H %1d@QȌZcߎ^:T1 h+hn]C[7y4 ȣ{5RX n zOG0*ͥQ s>3o2Au%UГ޻sRڍ.[ YClHFҰ4; yןQpI(JE@H BGoRdGV޷uPb2~#9=G֜C'<ȘofkANw\3řt fΝTuysPr"DQ8%̚ǩpl_mm,Q,CZ?'bXب6LY+[?SDYnFZ(*݊iJW  +CHG#ex-9@mK)Z+!J:%[8Ik;PJX) FU_Q%$Gqa69@ʾfz 5P;v, e5hop7Y~6H"aΙb%.V,Ϊ4$e;i(xX-M633Gn1a(X[\{ \i\ =!R"fR$9D6 *Z㸄}l^GJuc9|C?Thn}L'f$bkCca$d;h۬LYZxʢRMP7ao`gi}"sHZ2z+* 9qx7<1eɚNia`㤡TEKON={1!2:HǴ7_K'js!w`5w1P1@,@5 (ge2 TTͮy( 8 ZQD8xle}XĆqg9)?vG`0Ӽ{r"0 SSMڠ l@qt7ЮUB,@UnB O3KZG Wb tZ@f†P$WW-7oo)V/^OUbމmaA?&bÍnћqB֮3SY(Ow}8:\\c-OuֱUcOI+tv$!#Jc%Î~bElͭQjEEiF-,J=njBtTK;.}8l`&:;h_%LJ(JIHDɔ= :2R,h[ʆ 3}0bI1#k{1!&Va[gHsI4gD Yj$ vM0%lA&egcI u1*Ф+jQEk;Wf竽 3ʚt؁?r0@j{2=̙+c@_]fԚ)G7l! lcA\^BCIyݾ(u^۽u3_*_y7CkO( 3Ծ.5l*{RQ_Rm199uxH@0/ɀxX_3mC}G9DC,o{5Y`"KڲpYُI`|v #]z&vr;w+ 5ï+(y9C,?L*dѢA)mתB~`9L.XtJaԮЙM7kh`c" D ,r"7\iL=ڡvO;#wxҦH\Ia~~CUƌ9+AI)%jF8RTӻTc mN=e_bп(NBM"GU)ƱDFCzυc/`39 L)=9#4qw8~WB98ې#!r//q/4WLO;ǂh h%c|7>ٮӂy˽532z34,mգ!@ endstream endobj 919 0 obj << /Length 4472 /Filter /FlateDecode >> stream x[6vS%ʈ! ׵qy^{&y`Knf$R#QsT{. )B8-԰)wgg`ڟx_="*2^2%I2$\^͓?Y"6۩fǯU9W"9K3 vj%bYb^,x?ĕu`NۘQ:,AU-Q/2:(KN-wzydL(\DQc-3C]_/l\0)v췎}Ajo;Աֱ͒co-H`ag?)#)1outXV.xft|И mT[7zy^h#v՚R/wn]\ T>{/kfQbԫA1K\ 6؂o6=:,r6KgMԫhݧ3{iNܧM$IFRQA[0HB[b Mչ*x\?^7d:ivRwzv[+,phEGʯCGJ*"K .桲 SNS(ȟql#Sp;?{zPBma $XO2uWs{V\<I_ 8orh= f6Ѯ#/Kr`")zPꝗ'۫Aigj䁴B`u'W}fJo2O⓬}Yx k^zI yv~fecnn,oc{ǢV,nG}7X3ʈ'lyY$9 ު |W qGZg3O1NQ L xtė,2f,"[}{伟H[yRx CH(@+g2VVvz"/6exUB/ G" "HD-BH< M$^@DEQtEG<_xz(oJG传!Wo6zo mJpWp5pG-}Bv#}(4>MGMOHC-}|Q2L!ϔ"Ã( }"*{2S|X]a 'X~XZQ-QcDr$r!M&R.(G "(/JF9 :u{_ziVoKvp!|%;Ho'k?òWX",CKQ ˠ%eP’D2Ѵ ZX=eO $",{ڃ]}ضMtu>I)R$H)eT4WDΣ;򿻑Fy̜R":o@1h"@M $B7.G" h"(I 5"9~tB{ʉ^Fyv#\<?18R:2D'xˡp: Md]@DaEMh $Fc{qSu' O&TK$z.-V@*DN -*n< u59;*cu :Mv!v&.$Tnh.%~c@^37ˢXz3]do(` =G`8K_8=p#퓊ӼuiXOA#4BQY3{!̽2'fQ\Q<"ID-Ci Mt@AD!EHStDi_s4C};C0ziW6~xLB%n qXM}6#PXyYʾcPuP:("CeQ &BBtP&D(D=A\Le!R=Mܴ: LI9KQ y̠Yi^1Aɼo2/W@2y}7zS@RT(&4$uP_tWVeW^#G>1Q!cӓ\y tdy| iI#﹦m{AWԌc[_qA`2fSOy9 +?˞Fx9p2,/I*oy;F2(jyI˞?:/IN"b.Nڇ-/V2-r᎖S@ZRTi%&2$qGK_SRCK ђs箢儠W$X.D iU}u;1)e > |><3PwOMgHI?`s焠& b1%iPq}^.W uˢGjM|͕'=U&zqq$u^ZbsӷĮ cqL'1 ڍ9Bs@@M!Q=_t. cyf:lNhi|At%T= /ԣƧI{i'8c wRrȥ@H4K!QE;.uUw 8CǂƓM',X&z0 KG8GQm9ڷ=s 7!Ti<&.yQjx~͢~b>ᾯ5dy|,~ &`O=X9g5w>}SDvNʾojW׸qvþZEfbFRyZKcYo2jqoYv[}STwe>IE}kO5 4_v[{*[{f~|9WBrjvu<*uiu]k6b/;5y~fg:vpX|7X(Aij<ϒu}IVK=tvæ]UG }H2w媮/hVz"1S;8+Ruq)^[w:xںvVfE=rQ/Pʭ^wx25PvAMMc:.Po^M>~4M Xow88I)V!5`q/" !HݔRO'?wj$a^_ Y KԆ=xl]]p1Z؛EnTjEhW1ס3+KEnNT6cW/;{vV}㼧&uo߷]5&Udh@L,`3Dk6 _`W~|hvUudb]PO@p2&Z.Խ endstream endobj 864 0 obj << /Type /ObjStm /N 100 /First 869 /Length 1449 /Filter /FlateDecode >> stream xXn7 }߯c(ria$z1ˢx AӿLfutDEq$h$ZMmKjhG[bƛ;60_9iN%isFXB[j긞zqH]g$b5uదUJn 0,A'BKB@kԧ5`zWD o9}TK}Qņ +t߹,Fg tϯ.j9܏D{Q_03.ٰ|"a.ҵV}m7#hh{v,fzxza~ؾyS:qLEҭb瘛&6ӃNDe<ಽ"㫋۳w|^NwӣZ9u^cWpKjuHVhժ{PfHӓ.[9}Ϟ?'#GߺH cPfh_2IƥY3yY2DŽ'^mU'OM0CoF{(L,wr鎵p: #ms +eDu[ /1t-VeOcCXDm y8+5咩LhdÖk1/% \aqÃ'"i{JF>Vbrk}!m5#HưZ|<;*-8% |xg >…R |x >»4ȋwa >»aX0`Ðq+ޅeԻ̩o%$sٗQGg x}s"BI-5EJa\sX5%Za\sX+5EJaRsXĮ+5%ZaRsXĮ+5EJaRs}ꜽj;qo0, endstream endobj 954 0 obj << /Length 3462 /Filter /FlateDecode >> stream xk۶{~MNgұ3i6\qfh'qWI]Ι )pe캲͋<7هUnζ<.6 z30Wm[oou[ԳeC}盋T3"3̋+ ̫n߫|U,ye坩YU.Rpp'sQ7%$WSLm]@ tǃnSNaIvfh?оrTPWDŽ%S&"ͥ8RlEPIBAH?Pq F}OFX=+XuJ }RGTCXa n 6&! TGYAHܰŸ<~ LC n(Hi4 #փaM.tI--rwρuU}ۖ"{KP3Dwv 4Qr P EXPA{ω,FKp/PZ{x2E#Pq́ <n=J^uzb{*R٧GsF2aKQ9_yJ\x5VZAwù)a%|牡Ī_JOAeGM}zM;ϑ#C)eOPb(_:~dJ7'@5T2 6e '$faϽ,L #} TiDh 8OcRV.d ]ng'oqUCA :nBcH<]R9Oc{Rjii;WyʩHFG:7`f<RRʲy扩mUjpF35xL`(9aNh3ҝ4`>*<ɏ=xJQ5cއajO~"+>Ʃ7m`lJpCオ;B%U;Jo ʑ <3'HՎֶ ఢê<)Z-> }f߸KyӪ޸ Yi )`bC (Ґ<iX¤ U`ְ̲/p"jNe\>H,6ܘWx9c!ΝW;ra#D bzDF_t0KD>{gP !3oPx3qԉ*t:g鸺fN*TDž ]yWU(eуyA^fb@&b(Ur+yvcJgh47B%HKTȮh,vIϜmwweX#\If:.Ɣm.w\*b'gd[Yn1iY2z BkPqBP/q~4d [!$0ң?qrd0i;C\ 9ӫ }c]|")7H% pְCNIVLJΐ zhVmpo!`1#ۨ+:d PJk-|PN׍Jt\uj >Ȓa| NpABeU !{;ad6l!7VupH:镒69$H&](bq(O[SMlʞDzos9ަWA(?s)[ca䚌NVΩ=Tjjw"zн9GӦt>p}48(l˃n^=,ҥJ1psމMUޅnD1%=~MIý0@g}inQ4OvGcE`$ 6}h`:`:IOYz7Ђeb3$IR.ʂt]}1?8 ^8&'V@ƝYpuG1oS{ϴ7ɤ;CJ`{s}(_}8~>9 Ͽ ,#c#Vb ,NZZq~L<Ē}U/h x~c+s'ա%; D*)vyN*7 G>z(H숨 ^1ک){Z;{ZǯVo8!Ju_oVk+VMuzQ7>1GlV6YًfUk(F計048zrv<[8!3A@ZAY?k'lEsjMBuc= |ENt(u -hQ8 yOIq|wj<ޅc=|}}YEY#zL+fV)IKޭ@'5j*V]{^ʆ ]I#ok  T*\.u\Լ礆7knJ4beA-vylyi؉;XдsM pLa^hỳWۅlH7mQjUgyVY" S!!McEexQ,;\b[щbz@#k9݌MQ?yb$Bzxr4g endstream endobj 1023 0 obj << /Length 3031 /Filter /FlateDecode >> stream x\ݏ6_ᗢ$VMr9 x(i/VF&qo8$Jmf 0p8f8ގŃ'4'9x3lOѯHUJ/Ɣl.Lg?oU}/}d4f*ZcT%1S'e^|u~_VbW7d!lbޕr97y)rZ>&0-s{dy}s)6bm*l' DXT@Ǔ|kFd@y*Enp\.7Wc@O(舋4F̩?ͼ_,Y[.Y-?ήq]=8Y+% 1W)&:c& +(Vbo("R&;j͕m=/nY4Ze3T=RFUh+ RÛP{7ZˁSVy\ÀMsVۑm3 =& wZ؋1ɛRa&:V%4'X-gH6P*bJ{~6LbJ !ev$``"KJF'hxeN81Sm!嬢eXla1rO4,e-5ۮyib P(y 81WJuOpP7qf>GGeU[\v&D-NߵGoA~!1*^os?hL!#G}GM#z E5#:,mz0@FXFCdGh+˒R RkJŇp4fFƌ1k.PAʮO!}xƁYbF^ <3iU% YCNOH̡4N/1!HcU-Ѕ۝ NOpaHp,ԳwH ζkl~!ݯ!GN-ḢA"/HgA"džD^C Dy0$C]z:@gc::D{\1w|@fp d4GrN;m}[9QKot<]_ =4#ݵYFSM7"ڢV˛> ])3SoiI|4tSw,c|\[=߷z?S=_@M˦J*9`u/EkjJ+~UzN!u=3niV_+Mp[%CwX^5pr<⼘q-f6r]![Sw(BHgul;wj Oj^/|ޒC_gA] .Sƺ=82j/ўt#6@ s˖bAfCP6_n[z4~zTx|PJ [VQ|BDlCDD&5_xG99I lBy=n-4X*Zh;`A )zpW`XT`ŷRCroR~weG3jJ J9=Kc}D#_g) Ϭ= ؖ,Js5=`b>H/Y-&Y*9ꕜBlH`KTuZ}ulLX?RSII$&%YEj?E":šMhҰdE@:=NܓzNz[k0bY^s hS5}Y`m3?ؓP0Wx'XZEahxfd>Fq\ξcC08v;AoJSX$Lvͨ/nѢd;P̘ʽfny(B| v#y~ Ho`m։]UI434H"8o1B/5P7ʣNǭcqъ7phK{7|&ehAZZGPm*> stream xY]o\7}_C4("h&)AŶk!N P7u0RGIij*ɘRՖDlh[RFh?/j#Uוc\ReTIWn M\tL S5ƤAPJ5(K̩Que&Z/ v&@Q 36Oo@֊Q +c)jQ @ÕldjH*ull4 n!+3seIC! WA_W(;_B\l@fĀE| |`!u$@| ȣ 4`CH7Pt H=sZA B9bޜ)XN\SgaPѥD$N\*.iޝ >|j%Τ$֕‚p4H'7RٛBk$!}޼/awf20oA}/Wt>1v1@dsfxqOs0Kwtyp7hpOP[u:IݯtWo?vC]\_frwvw5}cv=߽;`k:)xZ;4o.16ҍW{@\w ?˶[<_et~Bq.'*|V'Qr.Y-O(2<\FL=#2qa"#!zŃjW)[5hAXwܭEFm5ܱ;}DƔG_ endstream endobj 1072 0 obj << /Length 3533 /Filter /FlateDecode >> stream x][s~ׯ`艹bi'4tZL"! 1I%G={ $AZv$X랥OWV !H A'׷E'3\/&?M9R/fcq7yJ*x^2dYnmݘwIm?$.m\Sbb1E\' .ewЈj2 k ,F?C̚qICk740%l.iݤ7[Xs#ϳ²|fAp (\\\bLnTtϗMon;э%6>1ۑBXs2cxVo!&iz]/Q#*-zYpja@w6j`CȉO_}z X"ڃ{p/N= >!%J(kʗ>~]=hВE_p2A{&Z'"w+d$1W1c%<^[d$ ,<0Xa-;QLfףNk6aHK7ʀ#d2F 0P@1 2h])glХ[c f1(zUFQ\F>07|إDS$(~#"%6y&ܰCQ^<)x-M~1D`P'ōJ ŠFP#8))deB&BTMQt""B~ k"u]baQuFdroF0ܨF=,j Ue+t}D=Tg  AFvBtpqAG> hajfL+ :xQT$sE Iw񣿳H`L׶^\/dnVl!O9SOXnkd.yisQdeѰ6at6TMzt}؄$ߖY&GMj$\ l 4d+H -\Uk]Ź) O# ͯ&C -JۮIMBh}/y&$bR"D!HO__Do߇1*cIf% aԓ*ȭ)A,? BL7"4|yFM2Gp9b1YD2]!!nw? -1eic[aV1<1/iH֐+U >CQe%wP󨨯q)zE$VNB10`K2!ۼeoo~xd+ *C'>6"]ؿ0!"݅єIZ j:sZb#ȗvvh!f#f5īdsobv_f] (^;lZ tjs2ҹق<.<@B 3,w_W\_] $DO櫫~Ɠ<a{;te{¨rn {OLWR˥ ؕm{:P 5M Y9w8"֐JK Llwe^5Lֈ Ӛp6ˀȺ)G 8-h GXx-T-kw*FC5cQ[سHS(i ;C>HYb#$D1df lh$IXd(x;Ƭn;OG vTA*wCځxb8'+&Պ RBgq) `tT9?0f.kUŠNLBN$B 93,k߉xZ#}v75*xJs 1pc_P9'j OF(pG!6%sOD"*j`~#+R,|eY>ѥ!Y,N%=_y BV?# ~t,^Yb9-M/E6?6Ko; n6p_ ڷ-P {Ý Bf݆_TM2\T ?d,Ⱥ a̖  (,+& fj4I5KbL5?+7U6jCR|QS"ݣ܄ZQ4$6Xx͕ 8cT@RZ^k IA{$ jW]1p>)\dE.e|deN^c I@U3~ 9!#t$NĂ:ȏo2T4 ^k~w*2;sMf)ݢg{,̝ldt.תz~i$p].tw.,lGA\BxA QvjҠ2ݹP6h)/u\{v4_jRv[ތ*Z?Щ-}5G^'"t-Լ77+]eFDaaGn+P -s^Tq,z|R_6/[Pn-O zmjn)"_i_cZoo308$8~Kf'S>;C_sr ϩ3':\liSY Cۑ>Nmg}ι {`6ZVyXMVE}gFJO7Ndbk˶_xr)Yf})`}XI?X=&vJk_Kٟ@J QJ[Z!KCKXAx7qt ڧڙld:f˗4M`e 1j( եQ(Puш1vF˻ PMgN~a-d* ]S0 BO~Tnʇwh23l-Ӑa*Xgݩ|M .T6ʆ|c" t5㟈Ol,EI+p&lQ#S*z_C}kyj!e|2 X}SOV {bHk (ݗ 9T;[ ւcmgk=?i?Ns}|'@yw*Y5jd#?{6~)!g#s@F wmk$p9D+݉{0# e+f}CFE1>/& Эfd;zG2l gw<7=2f  ٜ;/Y#&A%lvT Av<ś#ǣQݤn>gzz@-W h)/1 endstream endobj 1078 0 obj << /Length 4042 /Filter /FlateDecode >> stream x\[o#~_hI4@m@\d{Ubˎ$勵9q8jdٻ%<]gyճ?Ec)g.gόJ5{u1q.}`77g7vzϯ 3c8꘧Ag T&vYVmzT7妪߼yΗwMuQu~ݽnKfeh()+sDi^X0CU4GM.E;Qa*_ fpaϾ~g D0cƚXg7~.350}h0ldnux˥Qie=z0β@K3K-IJs9=[D#9&[L9B&Ȓh6R534PT%)td4Hgy\?)STo/r *n;nH7d`zfbU{'=&Oem`f\W9Z! D#ؠn~@F$g23_q[z}??V6*/Nf&U@D$25CDte gJk@Sԫַj[}O^l%]j9V~^}ȹ!NGvxBtum\W3_mn߭n}Mߞ5ބS9,2s)(afof'z#U-\E2}UUؤR`ο6g=E3BnUYVV{Qmx 2!Uz q`vSkw^,p f 'z L#e$\HcQTXPJ=XRаM1CU6ĔRz~dna,}68"ikפz Ktv*@2f2@ -W@,=&@^pu|nb>-G25{ii h"{:eLtwn.H3WLdDhݖ\SRnywI$'^DkAsLABnphB"Q3g#*8[xbL(Q{h}i4GxiK\1̤@B<. [ b! ?f9&0"o+_@t?hp`?ǀE"neMl&N4(㋬ĸ#LBFcxHrvǙcҁF \WȐ݁?`w2Cf!Cj _d+ a"ŅQV9䒘"o%{3XIДqhK/8z\"AT\`e @ŧH<| Cw U>b5Oe#N.G<īAiwfKhMP>O IzXNY@;Fty70~W>L?j%gݢ^0tr"d}53b3$([}r)%fePktAkb"D{D@OÞ ДC(s}e><ph{,\.TPA,1U: 7w˫wզM3W]uzkh:jٌMnOoou3dáBZ8r[Nw'՛rW՟Xz0#DX=?szꃽnwy)N(~_UA_1HC#4q|Mg.E9X}h' /ƪ8Ppӱo n~qMk,ίƴz$e ? Ä"kHZtljx.iA(> GǦ r~.EȲs %8:&?ˆ=aA0Mbf j50.PYưc~@N Y,BFU Dan'}d&Q*CĝB2=E_{={͔xa<,Jċcf+O<ព!AEF;C-K?!'s 춛Eє) Vn>a=1tQtZLR@&"i>*&zj;t9g⇶=?ް TE?G4Cڽ IMF,W$7 j&%7&+q*mY,yV,$_-՛_!OTN姄Nejੜ <R>^hf ub TaOu>*q>"v>RhHԮ*by.31C&>KPH,oE8,OYpl2E9Ov+Qջ#E>FTǑCG 8ꦻqcܓK)܇3)uİPVͿ 5Lh.lwU ؒVE{u-\ܷ uo;o-7˛jWm;.Y;,ޟmw]we&%o9t6Mtq":R6Ԍ59(kTEgϸHXQM5)ػ@V Z"?N+laU]_4py}+lh]$fa=JE#LIlc^3F=nI?".F%BG _+8xߎ"pMKs o.ME%摩)+k]&k#.%>@M )V$*#Zl1_`G}Wcʁ+?xgBwTܰӺ[Z.PS?~ӧiAɺ_;k 4v[أUak̺ګ$/s*Fz^˦"9-XyjǣUE֢ *͋Đp-@;!Q|y330ѝQZbG}.UG yF,3-"m+VT] 0S}Jwq =yzT)0@#.3A8Lv:hjz]Y$ XP$'т{mg&?@_ynrUMj %S%PD,oB+pD=f-ߙFH<% d?6& bbN.+nqF%VN1lfOrɅKb~i%hihiDKDIOuZ?mw7n#_] !l RR=Id!'xam7,.V\-ݟVbأ~?p7D0"ߌ Uy!9;MBUjȎT0E*={3`K ?dxQ?`Bz?$CS `5I64?eLl[+c~'szi3>%2ek+[{Ք+ÜThMemwٗ ̂(3%Z a>R}BI~3p+E\jx7LӜr:';JZe}f(%B'WM?<'*ӵ41yf Q}7MVmbc"6ɏծ}+R"5O.2F˞h}$g.@֢h?2a endstream endobj 1084 0 obj << /Length 4938 /Filter /FlateDecode >> stream x]KsWlUL4W9VN$:%ʁ(DN53cwf `_h`۝}}!_>W(rΩT mvO_ǛWf'Jmg_97zWt e릡4u sep"6Ԏv ֤]y4rhz*2=2vo^|!җ:X<)=z/7xjMv16}CKݟ.PI1*'S5'jeYnȱdu[ꯋgAP5}Q- D$…q>]vRX cJ )R*%IHMIj;PGiW?~T(?x]|ݻϻWo|]?\PmTdKNNBiRE3{ [vWM-׷QW~9MB nR $!j'܋n}W(A] Cm| *P4"\kڛɮ '|ÕB)),vjP[:ݢf/ۑvSMTJZ$5i%"7Tt)lPeQzZ26Y%>!(QR wJ/n()EHCSKJDM_6 .(l -P;LЩEeeoQ/rjDг.e;U7yÎBҠQaKj12E*t+*wf-tkGÐԏ20lO)`]] ƭڤmxmD2qqeč ش>{ h:\R%z,Oy!A}puO̙ŀouzn%\"Hbdi7B(ϋ9#1?s*Y(˯"PYK+ 1rHƹ1p:jg<\E;o"˜=TL2-@dZLTOG&#|ę/> MMPOs4uӇQ7ZMrJM eFj_feFj_ S/hhH ќ*7#VT{aF]amvZSehMԽpbS:c1j9 W# w2+(h쯌+ g ~0-DA\ccon1s-Ҧ$iԵ1[~ -IJ} Zqk@AKnӗ$5P3fHHVѩҩD̦f-Q3rՈ O+PxJ Q@9'''FoІqjU%<帇s.V#Pr|'̻Xm%RpkwŨb!i"8|m 8WX[\P4Le27#OݹETGmkdo&ՊZ:͎IR09|7Rx^"!Jߚn\^K;zOZhSVKj6ϜL sR&hfΞ zΛ ＝hʠdXwܦu4Wϕ+7KLMHHe$۵Llzv2nN&lJ0mVoNα}ۉۉdsH=d\3:mQ9Rvsh\&M־YCDtq#VB)h;GieN RN Nk7 N`Y˕nagYfB.;8競'*gTnuQIt0LVf=7>& 7Y _7|]Q#7R:j8ѧWY/:˪!&}zpir Yrauyd-HÂ9_tLڮfBE4ϤLg 5=(44C9}ovJ: kh7GAF }osXbm  Tj Pxvp>ɴK1vuRU9I\FPuƾ#\SXeLtV 1",,pe8\yu"X,EJ=3ъQiTRYnB'R%)m*9eVl *(E YP=!U&N)TQvu^0wZo9LmUUh'~PsevXab3rU<$&5ՔjK7[Qو/3b|W ۾c6Lܜœ"@9&Oz(F E=o򀠸q9" 2xΓ/{BΟqŋ=\5dvӴgIZBY} t|Y9Wf4MP:wy0Gw#qK|e#ǕsS2 Ԙp rd0{3gխX B6@w2v6+ؒBw׹qI)TzK ȷDH:ũRBn=)N"DOtzN=wu;RFu!:Jmj(w.DA<ȕ%}m_|] f|,]?e p<@6V %`O{P%1H߃sAOik1ub7PŶ*>ɭHZʃH*gx@HJBFۨ]קǐIL[ (-H6WXuмڋS;״ciF(XZ/Ēw ׃=ްLmµ-޷Gj +kZcebq&RUpw/sWrK/};૾ ⫥o҄䥟:{M@07$k`" TC LhГib(t̩!U6.d)f⦯fT@-O깸nuWǪyCxvM&z~wϪ4ڃtZI(\hAwY=yJnT{l:զt51H<`U#ga0bKLxօ5{.OnhJagZ>&H(oK!j䛷\{v6k\rGI^i!+BX4k+W~XE=cR6p~PFv3 1qܤ򊑄2|yr&Q^茚+d,#|9*I>1;&)!bBBFN>DC$'W:1Z'^ P(xrF[(ϪLӗw]~I$F `N(!2`՞$>aMmqhd[^F hH.=cM$kSg[lr>^.Řau[PBޢxoZ{ ZgQu_26:e %YWXx&*d׺z;GǔϦP޷~PI7C@PBrۨf4ŠmNM_oѶv#} \kVSB"|;/%U(0ې*QT2RegIՂXNbT Hl눗Q*f$CaJPee<8N&-J%bUoWa]>P)tU}IMj"fTc3Aiݹ6V/?~ Zk5ǼE!(AF)F19KU"LAϯ#Ě X#Шz/#PTVqf4#O著a7ѣ9=r,9l% }r.OþτeLd7-xs<( |Zl6-Ex?l=e"& gD >Hм@`:JdjƁ@h żpON򮴖/vNI cjV4"o #B$)Y\S? endstream endobj 1088 0 obj << /Length 3499 /Filter /FlateDecode >> stream x\MWcha  Ba-mUl$>zg6ծ\ MVzbAy&g_Y?()jxWw/v߯߾yzջϯo/g_y9\(b0Îp~$ErZ:7a؝F-?TDq?yȳ?|30rPAX]Paxӳok n}Ӡ$g/{ZT\w\aD TV'q>'Dڛl2‘hLfᣆPN6AeM8ͥf{%à /BX %9av i ;X3CD~LIx0:}Qge.{AByy&`UBTQT%`Ÿ( E } DQQځ5^U4=ơYHЉk:EhC:,%Ϋɗ,v"EIaxQZ2%*? " ׹"vYU lH ¨aih( ]A :;- AO4>OM`kQ4ue{ȶ\a#Vb ŠErC+=N 5Kgl<^3I څVa"epƋ4%ʛbV(#B5ɂL6ؔCI$|mNB, Թ{*3*~Dby~.ς2nD(7OJpXveibY{*BefBJH$*yp,rr 3niFXI*b_ i$HUUD!C$ӄ>.*=i@'A\ -tl"4pu)q!@^΅ǙPi]*| QsIUnirP!vj%xb<: (鐘e2aH8ª9 ?-KdIkbvFl9cZZg8MUo)˸O9H_JE ^q_'E&c'/txx=؞v_a۾3.}$hr'_(bAZl&6ObYb#g.#GZ}\S̝Eyjd'7 Yh9/HĊr1 vPC%lvZ HDžHdQ(Tڎt%!!{eGV8eGRRM1b+8sl4AN:⨅šSA8+m,pM Fs,"H!MfZMυQxӆS#J ?3/w(9ﭗKxPNɀ3g]{ە̧IN|l~9v)0ZOfe0Pj,uUUS5\?\ k 4!)Coᗲ*u$F0Ŷh%&kR)UIZa,v"Ize$5L=Ɇ~CeOx==-ਵަ D#eO+{ yfOn~씮u߿+C#Co=-]- і0%NSqzmyIW.6-=Y i>SfC%ԑ7#eIw{RcLm<[7y/ezn %[WQNkQ}n0a~UGkn0G;[<4o0ATb+*& 4ﻛ2?j*Vye%s;w˟q}qк櫮^ʈÝ[y Mj6Kh]B…D3RvKR:ĿtYR/C`H9Ɩk N*#gV"PYۀ_V+d( B8U\bfDx죰c~&]ϦxoɆ%.;"&.%㒸L ̵y[^xI@htvvW8cL'}o:ɺEbv:i(e013/8ෑh2[WiTqIO9Hfe)}7͹xv~q~E]}5o#R:Se42%6}3v]]\ve.e6 0=/Y&ޣV*r}b90!zDLSQ/rlUpF B0W]"$*/Y4"wt-|TE4!B65ر*=lЎ_@G `j3r*EEC\ɻ$ !Ļ\=B^ٸS<Ś 'EUzKYXe!(5ªoPZ> =Xԫ+qdC8E7d6' &)̭a^/ľC_5boxCiݻe9[ȯ;&Nq_~: "]ˀ Bf;Y}dIZIM~V=t\+.;_SѴGlYo j+ZZVƻ*JTig%cbvvd3EW\yw{uxQW+=K)՞K"@ǫo]>؋.Kم0w?~.]E F)(YRkL-dz;a;F^4D1cmxT endstream endobj 1093 0 obj << /Length 3493 /Filter /FlateDecode >> stream x\KsW-tD~$nRTm%>D"e}>`fP\! Um1b8XeXe"ToՏW?]w&*P #~6"jp)SBt(Ws/Xec!B Z]]/W DKTX%ǟiďVHa}VeҢH)I qhJDgf';Ch5W0v%L+lq00u]xԆC6=T=dL($M[; IՃ 2Ijɉv0T09ITǎjv ˝q+X}saLVs!<.-_A44h%i6d,VZVf.b8Y,j.2e]m<{y22eTʨ 5՜NQƬ6*bcG9L;I\TF/.2h/8#1 6bK>&c?`ތMSƃ*̌1!{5H9ξ?-W^9>{D1v|n}.sooWB᳜.LJa[ܯ]g?V#Z?7 5rv}W/nWǧ]3z~r_w cMlz[<mo/oޑcyܹq vYtn5@@-{= 9tV3sO~Ȣ/4#xEo)x%藙5G0c0#ȶeK ̔]MYX3;IYK v _帇߹+3h)J!ClvDW \gr{[NaYLa]4ыT }{vʶPNk He%̕pP ֏ZbnתDjUpĪ2+ڿ# Sݏh<7TͩqԹ6"Qi} #cjX0r|$/%DzN%t$3I(¨/(C 3yN< 17u4 ;OQYk;b<眎w:[Py&x0 aLL`x0'u)řL[9>x26%;4󝟆O>}CvClB1QAaZ^[ZZ.FgdSg{G;jxE_  ˝#ɑF%fdsɩXvۈ)_1Kq283(3ܫ1Г azZ%_h=鐁rvij qe>b@Ֆ2C; D vP U'408 cgX%R퀥 N;"k""QVDM!4~ aH&xtnM3丙Ap̵f vPӣWX(с+r7sο 73ij&=>+# x7X#}\AeҌ6!g߼s("p ֌s86zȕ}2L'jJ65 [* ``%5<%.\`͘\"E~lKDI} \  endstream endobj 1098 0 obj << /Length 6013 /Filter /FlateDecode >> stream x]s#7rs_Be9NmUr{suJ>خ;4Hv> 6ő[' )h4@C7 xￕvAIfw Zxwa~²bŴYan.~zGC}ŊLHZt6/~xn6U~|N>eCpWU6(03fGpU6'YF`} ^5Ff۠Z6XL1qLR]lF9%:]v?b=11j_?6{;e_¥@fȳb)p L婼Q /$eJ۴ed]ZJpx`YGAٛqXQ+oϋvCWǯ/Vq /.XۭF,߮7wac~8Eo׏f+>wW~\B :(A*4Jڶ]wsѺar]9ޯ/qpLJu [op֦$VʈZ&zb`Y~Fג -b 4ua#{ (h?3k^\eU A^҄;n ޜt}Vw|[f˛|ͱvׇG^W ۏt)E*F-0YjջۨGF1r1!a*Ws!>|pt>9+R LēDZlt{;$b*?l4`>;k8]ctCM'蚪]~T}yP<]?.ʌէ:Eإ| l|&H OvP8#Cu$\Jf+6X/o33\rXx>\&4 f(}FiI2# JG#b"tJ\: ]mZ'BB7k l%9뒌Ov[T~s>PRW2t9++eS͇ӮX*k+=,\z\B=1OgJS|TOAy K `#tuj!WǬkD]AWFAAn֙:%rs]~.5β oh 2t&xH^en}_[lϸlQF:zfyU]q{`6Rm.vw!_C@{A~>O[dbet yY!y%y+4 M藑g/>NXR1Lă*E B"BAo*ᛰ*8!jQEXڧZ-Aԧ^V_1c4KVUNK3j(>Aa0S5c#'/wwW!~-}*…[HS>H_.osww2}鿻r#_HߨʻTwuXnGO6a?UnIχZw xTL> > K9zYWyWg5[K]޽D"x~Bf0[SgOABZV{(Jb$) qij&5@@T0C1(?8e,oQ #xݬq "MLs9KYL \)OiИyʌK1xA>;*4#L3U@S0.%*Lag+OKX@׭JRB'cLDٌ75`U[iP!N.HM'3jډ Xs'n43TĴϩ꘲}EG  K'l\Z7js{{2S;AS9iaYikI9P[/ Ƅb| b Y]N{3ifZZׄ3fEhF "/ ;Wh5ň)D_փB69։nj:a8'nmaK̰*nw* {2%1> q3摁 JӞPOt{csby߸Z  >Dhn0tE"7 ދΠMA"+rOq*71'C2)7zqR~OLI9Ͼ2g {>-srZmu(PRcVW[ eۂRu}[iZ6Mi H]qY.@/_a W_# cw'ow{Lw./.9;>ޭе{?l]\ikqN u5ԽvV_wQS!gd@*yޅDh7E9iowG'ECanb%Mt掎Ąww9n۩Nu5ˉ+5󰋻0~~Y5j{EbS0^ ~<0fBxҶjΧ y4*zMHL9j q${Y1 )![]]0α0t_w cnf\y^#mcM K>UA6S4{F'0F5 endstream endobj 1106 0 obj << /Length 2232 /Filter /FlateDecode >> stream x\Ks6WOKٙii'ͤoIfȴ̉,$ -'v~ `MIB_OKF%g@yٛi6NS!,;p4_pQ`|hOS* 1@ ;󛲇`-l_|_^\r4Mlry6^Yh濞OWa| L"-G?^?xh)SЌ(Kxj>ox-EL3YL%ԚjEcX,=DJMpiXR py19p6@̡dݬ{PX/(eFGKڲ.d:Dw؈lAuӃ2B1L@N5+W ^)JJ{+u`5LH2LՒ8f˱B5{h/ UO%KY )Ԏt#bqc5Ltˏe:C TQ0j; koq+A;|:hڽ#>޶s7Z2+lٽc?Rާ6$ ^r%}2#ȝ@[tYmxJB5į1eubg{;;cT97]uC/.rE,t=+pE,3~.dabw&<4qݺUYZ%XYZG)$@MԄh}n3Md4EM3tv ^!\-_ M!ψXHTb!6yRчhaM%lZ{ZTWu>JKcU43xtWWUwšu4Sײv iig#/-DX=wvC5J@6elU:\!kOQr#r5OM-/i*]BЀU53TnF~ZYR{Ε_ڮz-V\W]E+n px;GPgqUA` 1sqUJ伦ڶNR@QXjg#jȵ>kLy06dHBXNj3%s]G&dyx?5 ‹:. ~}nf*_,]?=##)fo^䔩`= rÂhP endstream endobj 1115 0 obj << /Length 3739 /Filter /FlateDecode >> stream x]Ys~ׯB㹏TJvr9<ت D.IX h^3=I1zb1{zxv5ó^/o^|03\)bbb2JlM.֫b1lrzoDs-]P~jQ EכATT]^-uHDZujt# )V{[LJ1 (b ĕܟw87oZa@bS1yBBFj&2hp Ӑ'A@fYӋadIi\Ogwm|8->2Me5AQZ >Rr>_SW|}v_nUXv8/5jf5+.oW;ud 3c Щ.!-Y2ڵO|\jJPP!hP< F:a{$SzyYK01C(l $g#-Af \3 *gy7(DsKXB>GS]hP@#P®XeGL뀃3` YȣiZ :: Xg((gSW^g @%"tyP CG-Fsf1cb=-<uq؟P e/Q!q$Sޒb801qcK2|0rFɧ [68F{>U<q*7uJ 8U?n{J"IPRԋ]3q3-4,Fo#cqx7x`4 9{FongwyxjҦPa7~6$_r%J;6ʮD,&Y9ړPAr *ķVhB"eg&X8S}v[E_/z}S \i_׷ˌzr1c S4Wikj&O+,.EqD[shU[^WKLk4N"Fua@iEZ#"H>t>SSkW$A%JM|p #F: Q*ŮG\A?HԳU;+ѓiD4|0!;wGxjI?! lIhm[୴IG/!5d[؍$lVF ΀ylW7z}Q<;vtZūUMhbtݕza X_VmUИ*"Y/պ?L̤"u{=ة*52F|s bgaC2o~)S=e$FI܃f 9>b>SOm5v?NA؞9ؽ4 ܱSsDHn0EXǗ^&\dNrQZQAN܀2Qi/ [Nӆ(oO:eaT I%sӾd>G|vI(ԎKD UVVqNT~&zq~]m%Vϫ}.] <[ﮣ7PCƖi04j]QMV}J, Cy(ZABrd@4r9HTPZW.YWJTu= 𹁥xAPHzQT&KSl{k.`nV&vrXn*:US/]E,3 W~EﻱئN ~p{+= HF$ ]ng|?dv]ii"1w3Qih+I)¯OIuBʘ } >l!cF\oFlabDhQ$'sUKʓw*~DNMBZ :KdcLjp&2aJ.J S^Q=Z 7.0F[&π,M>,wUqxuMVw^yǑ,NބbԒUK|" 9"ge”!j GD ޣI5i" V#)YAOgca敛VkW":Hv) fvwnxM ߦG;/ڽbms'@޾tZk5Q>,ZZz"g YbFp ,&Z0dn?ȈwW<S4{vωUω}$_z2@(f)CdF Kgߘja=x޿i[ eÜd)|ȹW36z`¿3+ /*i+uk: ȺvΫa,h?-cq endstream endobj 1120 0 obj << /Length 3644 /Filter /FlateDecode >> stream x\IsWHAsp35T*Qx|X"ް4ЋIKͩ[ }f?OfA02ؐ囅 \,./yA0˟V*myn_]((D\zm7fuX( NzuX)? ҄c!!Ch=C 1_!-2 ׻DZEAd, sްզf'򮼮X7<6nu_>lwo?כxhMۊ:a~nRT*@'mo^ڻfhN`,X|Q#zTC;qum 1R#jzݯiZBxl\! z0` FfȘY@΅Q@ڐ)NZmi}2H3P4R oc y`$xbCyg_Ɓ"#Ι/U̧"ْbR-14GfguFXf$! \NV+}ߤ2Ҭ ck3nƿ[NJ6#ٴOh6=mIد|.,=#)`B(` 66= a";>ˈ1D@ 0Q&yo\V}L՚Ob͏XX){׋S)R8b<x2.`iۤFկwU@eP 2<}QP16O㱁352$fՈfq#MMi̘,ҳ8BM*c6Whݖ2Tdrա[Wy"UaíVb hco{ JJ<9yjr|GfRZ}ԂR}OME8'/,;( .%b8bk 7EGGC< `1Ciޤ$]uv}u̸doVv׬k{h".Rĝ%겴eu:NEq6m{WI.6+M}lP+ﶛrsXTwi Vw!oft  ]1]1g]N.#uuXk/ꮽz]gSҐFkB,H`ז5^&WJ++^ JK[i p;RHҐ<-%EhiKL|Iv?ىA| ӗ/'F|9tv5TDaB8X3jyqg,FUs n L::,Lf:ęjdUѲ7C-'5+|ĈP57 >_]6mbp|ca uU;-kb@'e'OUL`6LR.= '킏G ̃ MH{@G}Xݨ㔴ʍ1N^uAҦr(w6IMQo 3e>0d'M^_h <jn uj.}#e^nLk|ַnW&?lw-1hn)6y00YGK%RišI6O¨> !͇rh~_6Ƕ3k IgxA\EpDT*A;&U1W!rnFWWM kFv8QѲ+ aLO$j睙8#2ie7 'R0LϏQ-ak?Gh#[ɨaHjEo]>IJ{نGO]+lrNeQC;!W`:bc@b%0,9ל 3RS$͑zUO՜,} *Lw1TN"MM @Uȡr^TrF_U0Z*JVݨ7zė#pY]E_ 'wC=!Mg=ÆdԯfGX ãX66`o)wێx ԃ݅TǺlɱ2:H=$)бAHZǭR!H}jU#!w}`\K7H4A@-` dN! ·RZ0\o;3Zm̮G Em(8{i p-$(UhWEh)djHxA“u#qݨOaTйQ<t>agIޣCz:W:{X2-cs-iHQD :;6UGky h$DuHHё<]Rr7,Iuo1XTѾݵW$GmHer31 'hA+y?U)؁2: 7֤6a[(fxWAJk JVadM@~VZj^vݮ٢HX+sm7ܵ-]okWIe|Q!mGǚF5˫)Lu)ԥϛCP5|JAIx)#5(#{_3^<‹V5Hw*a=zd{r*EUƫWUJ?T*Wm n0PE摾ZF/OѤ\T!#OAfdI:v…!M/qx8BJϓzdۀg w4Zx5X}^nM?if¾S --;-1/@qX7ne?2uJJEN8ADjUtUz!04“(yYF.ifu/sRarrmYn[s5^>sz gymƛcW'nfiwYݺ]g1ԕ$Ӊ$)7L[ah'*Oi*-^ >ײ 9v&1;hj" 5yg[ endstream endobj 1124 0 obj << /Length 1002 /Filter /FlateDecode >> stream xXKS@ W xY{ N3Pn0Iڼ2m'40&ȟ%}>F> Xu ί!cNIEFwd;͆pe~O&]dl3]_|Y-G/5zW2ᔣe PnWļ@xah*k?n1>mvˤ jK8URp(Gu;Ө[@ ّ(Jyj S!gagaƿK ?~ nfrT<۟ٷ)?J&RPTI|"W?vjJS1Ke_ h)*'(wPt,XPOϻ. 7d58`Ec뎲QTUe]-5}IV2^ZQPa\[n ip9?# ͩqh2^ nn^VdWa VQSHӜ||,u6 L;Fu(F۱~n"UDF84V̟7̷6S1r oSD%u*`Wo5'P{+]>K>> stream xks6/37Unb ޝ^5i~P$:EJ#\$H1#//D׾w}D#?qvKa"ceG*g1":FoF1%^Dd7[-O?Ѿ"JbC ^'1#1#ڽ}½lܿU$|W'1̂W obMyζP9N4}KHLUɀ[LIqnłvw:_mj>[Zlhlio+;@,1O屖w1'ڳGS^XZ?O[j"cnK ]=Q9ѓW?5)pZ߭nYLfw\;}kӄ24!X֪謑[]8w0b]ӭ_UGz.=N30.'_ u8')[&Ii&4ly~kHWS۹p.r$lJ.Um1-G/luE#"J0/琱շ ,4e+b{G튨2vien:&7hj1.t5pyNϪvX"1/>-V51j=4T67ۊe*",[ƔRܩ!نlm&3cY%%m4K݂'B\ 5CC%`XmbV[%㤱,[B+4iv㈨.I (0Mr S E-%ʁ`ijхk"ԜBOK4(;2ͽZSv:{KO$6`jA ڢ"iɾ/D"cJE-|ͱrݿpYfHx<: I kc33X|ry`-c(78h'Ș%5 vDa'*i&sxqP*-g~/طDr<8:=RF0YT8sx2+\=ƻ6POOӽH-{Es7\LܵL3+Xnܮ]h@vn\'k&`݀v,QdM2 dމN8Qi |Tȩ&OKEpܘ8 ViIN Vg'0Sb -y;4ћ)qHEOq@ʕ@x#35spﱽm-ʊ\C?rPN>T7X0EUZC['rfPqM4"ꯡ<'}dF:TNk=WW-Z`j-PISDBsq |yެLO;$.Q_ x`;k6t*0Rah(&mCY-#jXATuyk\% p.@M,wTAEô̩ t/ C L6fpsDC-J|6rIފYPC$zX$ .ʳE>C4Mh+OzD'ΌᙖX'=U9.z,}9&dTJIm'%62otOny2{e׭[܋R싶ږO t?7H?x%^/p (< L ,KKޔ77I@Y9n6H~ {P^B}/e3D۴'orNiA"'6殸 o"I<3VqYCl_r] k(ݓ -iDaqZ\xx`.⦘:ka"r< "YBy޺,o-zH5xvg|N};q,Hօݏ[zL\LF~H2U'c(Ё-s/`Z:nT$2g;{:ػ-Q/gݨ?Ró q䐬&Pֵa v{c9֬+Ę S=cJJ'ѱ_~}u"Cew(Oӻ/f?z}]Wt2/a1r2GX>ןocfPp$m v!oZ۾h!ޓ$Oq*&4 h>:Uo_%]+dҁ,?'DڊF0UVh h(ݵ5q.}>d9/-m -E(Т."BΛ>m\HV 6?q=S/"{ MV5Wo _Z2w?cha6J3XUp>u89%!f8*G+c*uc5 endstream endobj 1068 0 obj << /Type /ObjStm /N 100 /First 949 /Length 1746 /Filter /FlateDecode >> stream xڭX[oS9~ϯ#<_^UH-Pa Tmв~q[ӤI<goBGG#P>9L!N)ƻl|s%Nȡ4jsjMO-A꓌MH$4!D A 5[h,ZNLԽ%&jh`Ȥcma,D'1\Lj]CB ST? W-2Z5^jZM- BD$I'FMQK0>HyШ[CcκGR*“ȚfϿ6b/_'O V{N0;<&YȨT^Bh]v 2ی Ӹ\-S0dK/-C|#YýX^-Sd-N8 @M (?|Lx]h)F]AA?Fa;.$3XKZcPY2(Ke m= V(Чd{XnZfMF499=Nϼo%1WcB5Vݐk'#{ro6Am5nq7;۷-} oSnJ˅#6Xa&mS΍96;m;mMRΤ+ޤ(%Iq FRl>Jrj6,}B9Y- V]"ǜd^pR#F%…k>Z|rS+$QdhsrS_ ڬO,}+u'^?Wv<~;|Y,/==U|ŒfNӏoOΧ[Re\֬qL@M94z@%Sdig1lӯ!M."K;gG~sU_<xǹ Ll(2$cue'Lr֒Hbƕ,d2*0UuH8ԃ2> n_P7;fS5mww6ݚo;R~MZpޝ_p,/ WQ t0e|Ư_>_޼| md?ʡ*@ x QTTҮ*]EVG -+_cqU~\jWpeL>7`Pʜr'8%[hIk endstream endobj 1181 0 obj << /Length 4903 /Filter /FlateDecode >> stream x][8v~PR#'0qvLv.5x];5~%[3YR[9E")'5eJ-A| M>LO2jeR'ɳ︜h!cl"Eɛ)O߽sE( >Ax3,%X ؃@7W駧WM ^]*6$.^Q6ݧKiR>NB12oZTLi4%ؒC5r181++]v1Z8NzfOFzRJJOm~rH@Os 'ՙ{5 ,׵52 iP<Xվh%dX eȖPVHK:}ҿ:uB8ojo$89γ?wiv6}ƚumo[`%Ć+J [lׯD"W/< /~$EF4 t̾wAvam-m ġ54ƃVuTxge,*RfPyY|~ 'ZdmEU*}zlaɭS#a*{dg<"keKUxozj^b iNR wc>СE|AZZuޮIgJY7ʤU.s `>I9BUXT%ʡ(~-$6u6 J^k :)}ʥ]NQYL{K(Ήɢ'R*"]DFg4{{ݦjf?OZ}cϑ$e.ƄNRHaU,F]Q9@p2 HPZ1UNK5^+ۙNd~ ˤW4U=Dʂ4U5TLDOԿ03FX+ fi!yN۹q-75.m 7FGhM||ZʏU´DK?@yҖ+EB؅fRh\Q~ՙrDl%b#iö+TP >uR~)L@gԀ3=g J+{fGahyP\W2wst:>]ɹ H1zXႜ6MMp||WCk=5˯Senxe^=8%"挎|z4e++96)7l4to@89ׯa([&:ZY;zu(3Ͼ8@ՀH5mŇ 'ٻ}4+#r7V#&t9 r=q!vr 䊂HZy%# gY6a]b}(NEvﾻ]z_Ɋ33 lrEb!Si S(@8_ݛfYY늡sog7n]ܹc?pB ԈYjkp hVrD"Z12 < cHSUmVn`& ?#Hgڋ&YDBDQU2t$9s͈$ʩAO!ӯK3k(~7mew;:_slfƿp(vk'VIg/?~ 2}*&:´>כ_J.t})K`j9LiEiKo{7`_EC`/;?ռ 3j]ǿGy׉$F ʘ:k:!Lsu˩XR(;Q,F\c̲jppoXO#698jF4 8Gv_oVʛ99c p,_gGrc_t1=97r 윶H܍irC&Qcޓnw;0[SXmo[VeHᓨ`ݟ*-e_mys1n:Eሢ÷}'\̜fa'ՓO"Ls; xk*8F_yMџ'{A oWOa KC! *E;]8 ﯼW}!,&LW5$30lefX1!A}<]tCǸ%d! pT* "qsRp7&6˻~Up%:7)f" w6q$x733j ]Fb>Iu\[`=®x/=p;ޒR-m$L;6ioW)IF:6ԡqWcG\^v@|}8&ج)k2=E*Xz΋ b !(b=<%w,E@8-(9Q2b5/zAYGjBQ g'PYU_cDAwġd J'J$zpPቛ-vBc9S|_ ] .\XT&>ډ,;<8- {Vo4d27J )]7a_F3CyL  kߵf+ o}74>j$*uP'چw,hf$J9rUrfӈK3&mgvšTǹV{A{lXWy偊$g]5;8նjĠ.|2qv`\v3){Aðϰi@Bf)QHsư)!n|æ1m+|1;,>5\6׾0_o7u%3uYlrx=_,8F x{Qv忏, ScEVc~LP9!ó[x󭆾`McKsfbi^|g]q[,>dˮ+{{;mAmd!f4zzRNմ6cY9)OjFhm&\u ce֟XGycǴY\MgZ7R99r[.@ŞؤYOlҾ[ECnmCiia4HJo*fCMK5F-'`Ϩ7'ɱzw^a!:"=f" `&M0j+=\⩣띏VmVu [HdY] `Me\[L4׉aW/F;vU carxvUgb]-c-٢t ٰ;Y#Fkwaa ;zS4e̷ 4@KV+~A%Qzvčyty5Cw-5u1:lf}Hp`4Q"b:Ce Y[n[>"~RB ow}[*ϫ;{B0(-m!nϹ77##]cSxTŬpLvʾ-W߽Íi@Kv4 4;DD߭#7/Pk}c6 }pMTsSʊKP (f;=omˎ]|[P2[agv$o3Lc}f!yyV(?g[xdߴXK4 !%ad73Y[EWKŴPB\lF" #ɠm?lx?b4oy7gӺ֗YD-SV+_mm5iq\Wb[fYcDj1,,rưTa ˬlZFD %+s70q\[ +dHz)};'cKؖC=dcPuz23mA _RcT6ߤi7lQkTZ-v.և#4΋憌dc*Ev)Ii}׶I[Kmvz?vT:Q.f?#A5PZCk˩>I9:rڈĚ6:|pM+Kwcyٜ%;%DyxqDZm!--vz泆eSP'@Ik.ṣ&~jxMSA{> ^ڜ:`j+r,q~dgGMٌkmm qm.P'tz fe [ksC|:k>8,Z̓O*>̞_Q:$Z%;!X,.?x!n6wƽ7Ci{f'Еd :1a~}h}9K@=K{¨kDp$ >@s"v*FU(Rh]ž|v{f+G'zp >s+˿g̫ `vY> stream xio{~6I`n6@֛.(-%1v;-R9nW0<s0g/G|X$ Oe11a8!t~<ӳ kEqKLV]87VC.LzEwwﮑs[RbzgzxB-:L>ۋi"0 cJuOTs)O|cMդuC]ޔy1Ue"yP ]֞^L^M,νiZF([xta,`զ#ftY. YNǼF9s}i+FTVW궙C[{q~7j(b"a8$X0Fb 98I3=g鷟a;}ϕ!;ыp6AN/^8:Y9*bQuߧ-]OMRSC "bA ƪtA~`אMS;?uh᫃ FHZ£ %` pjD4 2&kt@0Pi4 Jaр%,,$lX_ I;ȡkwCH@>a0!A# 'T\\AMzh^$=&0; Hv(vd1wF"3lAkDuHm)V݀c1{lP6oo-զE1' >6tjUĄ ':0%cUFUb*+R7ݏρO<_vh/HB1yHsSw_Q,n=4vQtD(Je8^\✒s}WK;קto 1S,!~Ihb.^0)>Q u .tB@<^0A! Hq}`ǁ#1%EQjƔ R6A, R!tM%FJK*[|c :M< 0zl: o)ԙP DɃ,$qrx,ќYqTDz.zflFuнba,-~ʥ1.:ʛ#ޝ3( ^-@5O:j.?%$0šDD4vWA*Xzۚ!+/O{֡D/aFf\8jVރi 1tDEe s.ծԅZ {B]D]>HP K]i_5"#ѷc>ֽWr8ݒdT6=ܼ@)b;z>sz ֺw^"K(5(LPw#iu8CΦơ3$79^~"VU/f_Ÿ1dϐMj,5+d!b2~E].Rq-nsx-s >}uE0$aѝָ$NqcT2KK hI!1c-o-܌ܕ?_s,PJV^U\SFZ#HkT ?Pb\:oᅋ3Sۻr*i0r`&CN[@~x 7 ٫/s|7sx=aHג~`img|)3s(4vc3 O6d3ÑzЖ ̦ii:bdYa 4tB83*` Xm ~.U''QJYOڦX>,)Q]Re%Ր^Z$3( g봪9U2OZ V qcs&.[<#wJAq@roMM 'ݳ9#ttΚ돓)K X\y k^7!VV2vL!ދ`N~U^ ti|+3f].!jIԖb;)`w0Wx (.lgr yhi7a1~߆p~p]Q#&c3'c?lrS1/diM!H횮(xJqR H=r2kO CRr"|5~T ѕEוMᬡU|'FڒQzEm*;iU Kt,.&Mˊb 3@rVΪX93۝ZB{=_s+¢ XV4ư࣎4i.= W˒ÐDnh4VG6} BgĿx/8:E\ >U/#c8o!ڈA_Ġ`/Qv/^ xa \N ;H_<^Nt&DњrmYJjhX k9vTEXy;7i*l-N+= I|W6ى}4Od0sM;nːRtr1{JSeV-No\RP?WSyXo76vܳ˾Hnw 7/?2 Kbwr%Fe9Feȕ"ٽqx?I '69PaD uj qA3x?uBDݔbDbV ^j謐kU(%סַ-'@KNqy-H.ҝ!cui@2Lvl8ye'՛y'测y{2}7Nx gǖ^:9?^BIWl=yȢS$ɉ,{edmn}ӢZښ;TK {zPWE >-lZE{@8p7NܢNUi|=qzyn8|,6/!''Ӵwh'CCsYͰr u9P*qtꕴή݆9 "UZg%wmnٹ6fq p_a:ћWX2$NTޅ,N/tH!$B!@ē4x}u+ҡɄu@OLc* f AGY~= `3>CэeQu>}t4}2TSױp.AϔÕ,Gߴ];s+V?i$tǝ-&hpv o0<wg  ãq4֏=Q2L> stream xX[o7 }P})RitC%C/P4źC,݈v^l}DRGT+k*V}4yfhxfi5upꖖX"Yas xcF料d+EjJNpl(0EӘ,=MLLy-l&LfӊrSA6BpQ;0Oa mB d݆nu>zq|t1>TAבȩ*-a0F7Z|"5qb|I1 FjImBu"XOK{(n!n1 >1G°!^\:`%W2_xb?cBZPa3'BmnL6B1LԬÂѰ_kM,Fz &r[H݆mm&>~q|Hs|zBG'GnQCm6q*)V5IuC9 :i!_WXtutZ3TĆ>IO?gH"-OD8A0\Z\l#6K oKH keF[ό&QZIGsfK]_<|OWfQ/m^^-kj9Y||mۗ7ҙW}9Ŀ\\lvmRm>:`~|u^[-7o֗N^ϗwˣ3N9JJA}[~{Zyzw%`Q:F8^+}F&"ApǡWXř'-nv[0`j(3mA(sۂ {̟t-A0ʹ(mA+Qfۂ(sY*n z ܠ<:muAYg=,v_e n+H~E`H> stream xks637t"oL{mIsӹ4u+,.@ |ئ -žwS@Ã~q@Eȅb$,ЂDp'CJ&4Y'C/'f\|8%aLb!ɐx0@pb@X(XQ 1T@&0FD`HH0;/"W 1x#âʤ:dS! 2]Jb?^"Cm熼móBH[$͑*VIMcUjPy)W&2GyP;rb1xv"#Yx^[ӵXQ2K2O/Am%7C)bq"_P>Xn 6Gh9Kg<.ZPPZ;A023v kYj9jsl:Fdkx+ )5z' 2QRWq9'5faIӿ9Nf7~aL 7$AH,vǬE܌vptӓKhy3h' hGI26;v@N,k}F--Q}p8h,Gf angX|Q҆=@ծ';:j!ͩ͠B(bzD!^4u h-!yn(GrPV8ur"Nٽc8&eh8 GL(''?)chX1Q?VDޱ]@;vIwl^LKvH VڹD;J ~Q޹\~. #0M=95RRW6[shWe0aM Ekdޓ] 홸P^=)# & HL`} ΍ Y(cOn8AƐ8TL龳+]ɤwOt]ԖlpG^kύˢJP5fJ8oX9m#Jam5Fjݕ]bF\~w/A/ztshRy&Ry[ҐYiȽ/ڢoje AEf®D[ؑ﨣U =r ׈rͫ{{/8LR#ɹ^Y&b##ԉQ1^B8G}Nd_ס2闖cdXx5WYvSWdže+(\G)7R ~yQx6slNLP}{㥔}U1>\{Z%.zvRr+/?52 Ou9Ghvm8HPGƱ-~!"\{I\q:W@<;jtKGfY1B}dG~K]K@ >')7^ X%wRc̦S m m9^zDҖ"3#_G{4R8YycEjJcR{''e{Ԟx}Ӹ4ްa~e_ĸl:kmʾ.x#KHNeKJIBIfn%mDY3;.=@[ NRyR I7OWО@|x'Ƀ(=B_]VGv-uxlj3)E'׽>8.zEp N<3qy2=?xյـ'֯"Wy;W{S-'pǢz"km.YJ굹xn^5ŷX,4zmTEo8\* zX}Jm=ewh_B_A5<ǻ2.NO_fn#g3[~aQۮPW.A;ciciS҄҄R҄҄҄A񚇠&+_xx{iQ>Po1ԽOݸѻxȉwx)JiN1]Q6/x/K8׍Ӿ]Heyc WA) cӜա"rJ-Z= endstream endobj 1314 0 obj << /Length 3826 /Filter /FlateDecode >> stream x]]o}ϯeZ!_D&H}H,)n$YWR;Cˇ2R293sfv6vΫ7J_;՛O<~LvusGcRuO~?gUnԙ]~xzbωtWqz_*<6v>EI#giks~O9;I0`wxHG'}aH_[jX mY8C#i 4`vDlF6&03S^}|}7l.כSD UH DWW/>vl(ZjЄ% j}g@ӧ{sja$*!@lW2.fj)P{FjH%>Zؕ`%:wib}Q7vNjIS 5vS'3Z [ *=tƔYk4%J9eڧG! |v̉sj;BjkA8) MnS/Ӑu,qv,><ܵud1c&0:dց&-چmD:$W题<[Y@͐nNI;L;O1xE=>a#~@8h釈-f [q)b@3ǭS̢Nzw/\~ii$H wI3mDj x/qH&7fk1nNr#8 IỴ7ڝ s5HhSY׀O}#):,ԮY-dpE82`pc֒-$_-R *A &A#eLr6sۧ~5:>tgUz}TV~ K͐*rNK+;=rGX\ Cmhf{j_}@8 ^@nnBfگHnns\sYFC:,̾Y2njHbv%JWS; 8$a1*yp/92?P jRTy"T٢kc3B%E)38\`!pE/ޞ)g@$y$k'X3H/廙畃q9 |1jopOtOOI{]R*gulmkk,^"g}|iS ΂mC[mxYPJ||%5ׯ+C{}[*6 <1c N }F&y/NR!A(O΄(i4Ū7F#O )Oo_Eڛd+yNC .8Ƀ==&AҘ8Om6T+ iMK,h1p;L, +krǝH1f8a KvM %⊍ *K/ء&~Ԫ#9h?g'4,UtB8ӊd(qI/')֠DSv07FIOOVvRZ$=hM10&CK7g_YsfrQrOYPLd^hƔ ٫i`l J- 7-a`Zm`吔,sM@egTX&xSJig9k`2x\Tp,S,|Yt?efkNC?ol]`W7D5 Q%-T|!;RNPL)Ut;wc>I?tܼ C`2 ٠+? )AZv"Ay_6~KؐƉS&=3idzR6,;BRUƀz!$4#wBvϵ ] 09GO>k}&Qꏧ}k*h l$l >/r\t~Ƭuw`9#TcO%:mM݀QM\Zӷ&ԅ:*Lx,23>ȽoI)M"mO;yCWӑP%1Y8_H3o%EVpCqnWgL C 4;*@a@YjP3T) aqre.QE-ߤKS5Ӊ7v Hkz`9뭠`%BК<"xٓI&bKx"saY]~۹>|g˒Qs Agm3] ԚخvFh"> stream xڵX]\5 }_Gxh&;VH-JVЏ(;hsn?XA2{ljc::ki:2[YTHz=.])Yx"e4` +Ȥ"=^D(*̲Z Rt婘` [`΍1ހ7V~|^rՋ^LTf$)zmMquVpglQn~lo_ nm9q_S ,TR oIONf X‹}|um~ݻ1}ODSl`N_bRzvR>2?Z5KOƗ"/Uw:K~ ./ESWYWƕMeS jCpi$mLfI.lYᆀ[97=)}*:}I4eq4#鰬2D92 ͑Q1> stream x][6~ϯˢ&㈤HJ t7n <83fv|̤_ꐒ){϶NTMҹ~¸uފ[??ݿ?dQ%f9 Uѐt<ё+FtEx݄p_i(7tYZfr`5J-D)=ԨœvN3C}02zR2HbYn2z}CJ{@8$/bV)MϪ1翀4Ud@:k[P$-e4bgz9PAE\XiHSO(<\?@:C=P8I0eu&:!]=[!ɗ`ñ`J^*ELM4+'DGN [H~HZּ_1*\<(]4!ڡm6\:AA|eמeO<.pZ}&xh{3ݠ ݐ6ɚ '&wCdDv%68f)Փ)- "s'=3:\&#ANȒvXSԍӢ~0%"B"3<6y! ~B;TP̭y0)c )3shLFRƛdK;z#Gw4}߁|BiTfhDɑ"+"{٣HXS*ر+6l]lMV &Ä9dEGY`vX -90 0]བྷ黦3@d3/uj 镴H-GQ̠S!IoV}x啈9^~t:8PxhaRzcܫw9| cmXC;$|[7=z #$s6Op=O1D Ϟfw^s6xQvWcea=ב"Ġ^![e] :&eL`,m8J(e{׍s=Rwrț˗Z~7hjK] D \ưQ28ma{f"Jׇl ,t;NR =Y3zxt*QqΪVTz+#`Xo&;wq aL1$>q3_D-hc(RkiϘ?+osR[~ adjS} =&vGB{k%48Ayt-4fO]td$PXWőCLP\$GeV8\$4tE-WGۈE7a[;ω+FoQ٦yeS}.yEgD{ w] 'wwȽ!KOgB bF^v.";֞ԶQiWP߃;g N4~L7&:@%%!eIs^KJ^hc urcG2}v?c>кN ʠc>%Q+ɱUQ]x%}q2ʈ'.[BmJ۞\2 Nv,?Cх~FuZ%{C`jx_9uC^TT.}zfno*Rѹ`NwPXQ֏ Թ)A󦱕_WO:s{Al8crI^칀peM+0'^^WlVƏ :_LY2pbhfW_ #zHKʓ:>(9UkApf架O|sڙhxD}{ÃՆ7X'2=J=ws)fKPv TURI y,YEzЍ"CC5a/\jH<90 WktD+c&qE96Aͩ!}d9oօ>; X!`Ip|)2.zAOݡ{﻾H4%K)Y'^RfYqIs2k,>!کVWNh|;_W̍+*ɿv%wT~rݬN_Z,XU&T+2:.TjS endstream endobj 1437 0 obj << /Length 7911 /Filter /FlateDecode >> stream x=ے6 S]j7|NéjVf%o_d$AڙlYA7}C^?lEH AW_])ΒeB?Y4M>7o?YS~~|96?Wp U+&Y3oZSpAii^'}xy_C麞|q{:m6wo^կ7OMu啍1|bRcg)_1=&K$#3H``ȀՐiJy`RCTaдYҷF6XC.].>77է!?o7K~:@1&AoS`%7fZ%"k9P~ʯ<1,&Tܳb?30 gN1?|)?.1?;gcmndZXL?Y̐_޹@7҇cԯdebϯoݡ9fgY|i_ҨD$\̓?rsN_Z_8)0R?- ֐LoVJDh# wqL!0r׆W-KrIxiBhȋT:OK9|47+U:/?(Q9?d#{g,d8<ЂtPjL0{rLˀ|u; a*`vCd nP r–s(Q0ZD2,z )ZX}>믟Yݭ%lhNaS c|.{1T`D }R)0,SA WQ]bHyɀ"AJ+(ѰvUk67w,V-66Vi@_}6̿Xo*>q0<:eX6\wyzX{w]*6nwA[NQ90a|)Mp;Nj>J5U'Lt Id}с)R"l*?H+c $jp1ZHY0C|6f`Th>+ DVՔ%ZdtWS0;')86cmطܜ`Xhx<56 3UGG27ؔ`m6i;D)'" DGaV}F55I|QV0%K: VH0N*o^fP1U|lb%z 3lH"CDfM|mWy*ʱx@[ ^~3D)3es4YeֺmkyH˴BT m2'kcBj>Oѥ62#˶pqEseܲՂ"G݈RdX3 _4[dPa@lKI5a~q~L #6HJƲ{'=*o+3rRFEKRt\80~:Q́e<+/0?KhO)M ]ےt8GhHFkg+h8v7>ߜM-!n,F6o|U΅uWt25:$9>XҁG8hKI/Tijc[?*+c$s`#- +\RS4>1"|$щ[AuZ<(rZXZw"U!>90>Mw` & V>a4N'QVoSTdܝxjw(jۛ;nwWØqQ=`[mn);՞Q6iF'2E|8QFm>H q"X'\h$w]j(8h1;) ;QSE s.Y%]h fgՅ~ F#umRTx2Z['E^CjfT()EjDqgwXnBRȺيb2S B;tFS.sIz1q&Ҷ`$Dl.`l8)3_v'met%)R<*dLAf-C-N}<Ӱgй2e|?ypz+l]ezLW-| uY: ) lU f+7&^Uux[nD"Gۨ h(HێdnV0 ,@&ePl+PvJcHkzx2&iq^.ʭ?^ LLrd"qYZƊpvݜyiN ƞ?.+ҠnFutlXg;X{ 7 s bx2x$;P\ޣHͭ,p mRkM5^s3d1#"R8j.=SRI´ctR\X?nJ5`X%??4 kDax/dN{0tV_y]oD G %U+^h f6̮%5/sݶ+jj?յ۝iq0odюt$ LX=y =R]jF[չ`5AMo@ J~n@BS&|`徶c֨j==#C(H0Yf 2۹m;5p%XoeRӀw^&$]&gpJʚLW#,p_mNr /{yޑ79Slc3҂ :.[u['f E祦qœl!PY!yMuq-3w}@fo`3$U[[Z>:ZTщmmmKm$lJ˄?{uNZI0io`G-Zse1G`FP{Iz,hJ!&i6jt {Z$cحGN%}_ J;r{@*+3m*~3Q{Tj &IB84.*OV:ϼ΂S-d mĨ{!ytX^/<;("_,`#gx 3gZp̾V$%eЇ$Ydit*%c yC0Cڮڗj'PuVʽ&ʷ:A3b B]Rz.{ۓtIxniحH95vMGBlb4s$Wˏi-tZ ]\ rpH_ȳե2uqmlj)%=O*w˯7}xS%!]Zi^{Zse-+\W74KDdrAwgbۈ#ǔ{.Au,eĵՅzRT^RBخѣQlQ6߅T}-yq8KRPZ*s!{ rƒmo h85j=mL 4c)}ٚo&7׸7ph_KMR<{Gz,G~m[Fw?7 ^}(r#%|m|?hg,ռcZ0s]׬<<.x[gTcHl0u͸*P/ԊPX3jf_=#:zH:,cy.@:{6ƙ X_ay\Jp!j h^#qP=B퀽oi!ԧ|n3%?pZT,z؍N+aeDYQ$֓twyx^/ xvhĐv.RNj+@ PO`tw /]e"ۨ[Bmgntꜝy:r}Sp:S Y\.dZ$YW]ӎ?^Xҳ߬O/]&*DbAa_N_Mr]5 ԌnsusOh i4=kY8-,a8,v6~ ֣ap]7yh7l4F-\Wt|+soHƛw59UdӁ_Dћa9y\9I(^j zʮZIy}8{ExVq5jv:3l*5r̆1j `e(_@Nq$X[i.o^oNVQ:j2>ӿfThBtۗ"Ë́b.):90Ƌ1Do;c@B+r bDl Je2 SII;n ڇZ {$kYE^SbޛQ2P:_ÔvmO eFjS*Tۭb)3+kMnJ';]qmz .fRUsk']3%Es|Ɩ_s. 1 W]ǏBbr=2V+z VuW3Ud!MFwMͨ 3NV;Z&)3\#nH4IJe E`ޗ)0}wk҇ }0S13$Ub&8,U&ӵ ~7K>]]+_=oc"6Ɓ|o&'k?"Aq oXߟ;Ī_c:x) h?F#+|AP,a:Q>nt_:z|Nu_Ч/o3G[$sÑW<>+28cGzqщAu7ǯ_Yb]J4K fb ֓0"!tobzah}:Nhky{#poX(⣂NCT їt7pMyӻӶԙbS9AsPb@] OZY-iw yz)8_SHhXR] 7!ovWUbnsF_;qȣC)v4huӣ 12J^>` p"y M|3P% ܝA J=kﱜ6rVפEö/T]:ܔ݄Q7f4VJ|sGY@1KDچ8 |J}p ]m#T~iz*s\PEI$ ,5/!dvw7V)p3I]OqiZSoB{ԳkMdQ9\bUjx AuO9h6U]8g,>geTlrxd- "_\:tH ؿY,LUj4 ,>> )/╫ﵫ3N >N{vҼtF阡]31lC?N;?vlzvc{?1g;>|A]WoP'[bRHŝ#]PωΆZhT #j,"?ŭ?K^gث.\;#jޡS@DZG< WZTiӺyޘ{{@7O k/6?OǷZIȽc۷v*#r?7z'ҧJo^H[a*ehth`{ސ{Dyo1V3@Y#wpw~.C7NYDf"ujP$p_ ڕ0Hҭ5q@3@jjВ|`x5owEBV`4M_"(3zvUAޝk]$ʝNGӮ3؁XŦNRqc=ሢDTQ:0[g|DۡGx#Z=OPF7ܽxzlѫ|#ݠxF[#σZd_Ɣo'dotk)LʨSy'U? endstream endobj 1374 0 obj << /Type /ObjStm /N 100 /First 972 /Length 1282 /Filter /FlateDecode >> stream xڝXn\7 Wh.+J 0'"Hax;"H0S8}9hRv=s.IQuQ@8p7$ !5C$d6\ AbhpgQcpCY 5u5R-C㦆V )GCj%siCzbbV" #'a'a>*D `1LCӅKW!FIcnQ$]N<|ѥ"FʂҦh * H05x곽Qt4^CQKSν>B: MbrƲpZՑPVQt CV CfEc Z{1c4R@I ˰a3lX1a a:$KY8us@?FU-( ֘E-Ĩۈb?E-hy #F%Iy 1z3 1&hXў7Lg1 ]c)W,GXZ`j%X6At CN=T86\F|p8$4axukǛ7sΗ!^j>X !A顷z-4TJ h,/PoWÏmzձoDwKߥit?ڣ9fd8ə)'qql47n835'48eP.ZTq^Ϲs$뙙Jvz.c#F597By=L0P' 캵@}F^ y)g@}F^ y)g@}F^ w)g@q}F^ w)g@}B^ y)g@}F^ y)g@}B^ y)g@}F^ y)g@gv]O_0Ofv:tlXb9_C%ǻOZl endstream endobj 1446 0 obj << /Length 7866 /Filter /FlateDecode >> stream x][۶~Q*Muj9qʉ}JRZJt2'} &fDlpJb/LjQYQhu!e5M # CE|@&(t3kA#J,A"|Z8Yd@&@tJFPⓑXDء__}#"cvG Qnˏl޽h6Mڸ]<&t>I:wJ@TAUp 1Z<@!AA@`;!qpsr@ &H҈w"LwSЈ׬f8s9@Z;g8ǡRE=(6rM1sK\y 쐾L|_t!^vo ŜL~cJ(EL "r5ɜb̉q&pZO3%yv<m3f秫 L)C(%P ˶ j _ iH;*RNt{Pni9ӡ 虮@=D[48] NyO}>[Ÿnt>zw;:?ҀĕiR!5J NR`NY7Qs[88 1cyR^ AqnGq"zARJp+͎{<6nzG=ꓲkDXTi^,{J3h&,Ж`.[狜,V}ԑK4"ءA@`؊>3MuBVb}uaLX(Cq{]FJ9M #69;Zlv&nnG-`m-[d(Ȣ"+*4 7fCnwB<!ՅU(lE=}1e5|u6XQ*ޥUܕhh1!DXb٨En2FlH1p>!k4^YQUE 2:M`5 ,K EqU71{î=05aFz٢*Q'0 Vb X֍`=i&VXcU E"!23a@Zaݓ_EZ4M%)m#觃ǤB{K ui}nasB5oq0D3A$=+U;|Ul|\Y ~' bEqE'}W]_C`l fi)rt.82)G|n]g?>M#'5@I޻ N~"PG}Fr؏O 4WWc|6a8u״cCÇ!4:tX0gqe]27E`;vDR.J9n wPB8b_ӟK̛J("64IO 9JH k v%}Gg?*UP x ɦ9"sT!Pf'Bi삚)oq9Op[Jْ.02ݪ_SB*6ND,~Xwom`",8 ,(U]RN'!L ,غ00~ 9 ǂ*-6=2hC!'Pج ;wbR~gݍTd+ ޽ߵA"ЄlP(z&Y:C]1I98`v|u{n\f{eZAeMCPLCp$kHbʹ@G5BRU.uDJ?M_Y>OD\ti[K"hxY F KkR(.3i}[[tE\ؒ ׂtJqT qK!i A(X]tnw"TfPӎsyIZ<c=B/RZq,Mۼsr0s E(E96E[SMlyȘapAߵT(@=A8fysa ԯhKB)0 Wڣ `:{Qa#Cq`zzwu`Lcv08^0ͫI7ցj9ѵ$)[%չUgdF>vka86\sD)2cܕm' Eb< {uS0iC s ElIl(h_,*#:[f6cA99wJis͘]1scHI1H3̖UMaʺ"I_SFu{P{M1JSfϭ>'b4t/0qH1C ǵvP|1띊]-z/&Vݺi aJ$j Jts`ZrzB/܈Ȟ#F6`hkBz5Kyyƚ%&)&߅ȆixI[yMvx#/WEeZꑄ-C-҆v e^$a$}6*Z&y.!.w @pj_0a99L M tX77PYǥ) Q]!`&}hyPRZ?c@2*Xxzl$6 x)4rw")3<0">"(NӶi ADi.(L)khCpA_tO~ es EfSĖ &%־b&)0VckpGO3= Wv'*ATU< 8[0/zPzbiwtүx̮Zn f+w!c)Ơ1ZכF60v˛.ό>4LNj߫|Iv$jh$o-֢?q+?lnUG7}h*5 U8(c( Қ!2LSQ@&+f5r+8c ތ{G2p| m4D'|jT%.Js`T@ QIٴ:mŧX*wh&ؔVR:6G9qαlHG=z4/wP `u2>T- ɗ Sp` }RD]!vU૾ěז9`R]C=F>kZN, gM#N8Zm ` 9ŻpF5Bp|Q#1Ԫ^G' 2 -Pd\(*x/Obk~}:r/HEw|4UJ)D;\ւ~kr00nK/P qx+ڷLc>5gzUqF,{1a]grSgNP4nd ժ2JYSHX.#RYrW+FVΌ*YvFַ^I vNFuƈa?X[ T4\3..Za:iV=[W\1]=@^j7bmtBV1#_LcھƖT6;. I@wC F x`BauaQ#FQ 57aKi҅f>ȔBK4@'}L{{ wduJ L}ݾoa>P10<9 uە0ކ4H+Tvx R/ bR.M|[OprIs%s|Yh'hVYWmhAӯnߪwr]I+c ;6^nR.y`"ZUt܂Q{?T_-wK_.VLg/ HX^Tz %qo:T-(Zi盧"JU!ZNBe[}֛z oyj>$VOit yJb~s ã-'7$x3T׏H~AzԏN/~ʯR IP@M1Uitۼ)^t:W{%ފXj+oV[+ZP.TҸ0'1S&ZX !kO^ҪG/4H BXrrJ_O_NrDPGocRxS#u \]`}0- kZkCݵe Rs- ^H>jV ɖJUk0Ii/| YDz8Mכ\'cq7Nt"*MFgz138GC~Eֿ(mbgx<7<*7OqFiu'oڟo_k_=MKeKl{ɏR}Cjg:|~5i/*@dPqQ1R3GAx\!ip؝o>(3p=C)@5\'rn5vFWUgaUC_<;\~#t6eyJ]Ŵ;!9k*a^В80UiħOms ?sͭh  ]wrNT'|Pq\j~y,tD[oz9Odь&eC>8 qǵM?!zE;dha H@̪ jOE<䒹VMT#Fjٝ/q).L@HĥV\zdfpU8cF<┿V[X|`H~oG-ߗ{_"?nr_z[nၒ̧ky(蟭y ?ݕyΏoW7OJ8Q{9BH O_sЯZi+8줚9XF9.S l~kvҫҽ/J2^wJzSet{)BϭۅTΗfSCƭۥ3mhHuڵ>U3G`2p騦x~y>&So` @_U+>oД4o XzF*KVT-m Vu]`V/?t砸> stream xYoF=B/Enr AN6>eH#ĶTIv;CI}Q`ʌ9b<8ăbfOǣ{oB a&;(>t:Pi2S1x7w ă `>y#0,* 0w-2MaEӃ4$ ]AENkLE4´:>vJ2NGshVEKxKxMϣS223h诫MXC& l ,. S:9L82v3vkEߗm:ߨ[PKcaӶj^YI li& [31`!O]'2hxo8 ĩax,pFC{V]cct2P7A##v tP *fpJ|\~;rX/c*v$/Av(o`6'1w0qu< 9ASMA=h7KdRȤ 'J> #h픓$PˢÉA+N|1KDE SЬYdSO+8u˴bc&Nå[BW[Jy[:ߡ}{ݒ$DL&sҝ.#h4  8֘N]67aWaz@9hI 5.4~3mF5>c#:*yg{=HXϱJ4آV6,ݕ\iI퐴cF :2)s,3#נ4p2tBK~U}+,8v(*_@"nL^6ӌB%gB,=Rz,K4S)BM+[&o) jD}35 VvB e?WP 8-+RYSKMҀC3kH U'`ßJ,e[%l@fЮH#V4j(b_Q´sh}zν6¸tM ֤`e!$8 ԙDynYa"O*v4ו`Qe*285ugt¿zCKL3! qWZFwgNi == 3hkaR5&|cW̢ )<!I楗x7`NY4Az{ yf诗T6}nݠY?޿/K2[y\+.=F(jF#TJO5=R`.e50y.@W6r>v }`%<%W c1C"4a!fPjEEʒ!FȠjȠHσ22\ DT(ՍSg]Qu(훡9OhTƺV[OXp X+A`xNC.-{H_#VXŢ!͂D'WȤ8ip@fT7оx# ewIDtRylꠔhhy~7_Q݌za!w/~Fx Zjlz? LԵtHXJtv4*0 } =/Y,J{b1_y~,柽CIt3S-gL*ϤI?w%L"p(PdË`v `n:| q\6K43sr+ w_lFYQŲH=FG 5`FZ0VfpލLsh9z9nj爐@.`BSQ8e[lx8ўK\)#.U(OX dJ0$`;B֝zˣr (7*\QIMMi jcVmml^ڶT\eq ji 9i>OaY9kr@ b?П⁻Ylä9 c$)s3 +ۼA"f[1M'iܣm]}j6(ЯKWU%J6a:aJ:ft dd[;OZ.2EH0))ɶr[)|ZKg" ^ P<ɚRVU_Z:nf i?_7@d:otV9scoѸ2~[z s81J}sYHgA[)'YLvs)Oywn\[@E>ɽH9z0IB$p [-ٖJNK0]̝S,"+4x4 L4,=,gV@XC(h+ 䵘Wc%Wӭ =\ O'M:7[שu+ch[)Qۙ[_Z!6Y GVZ_g;GZjOR8!\S|^p _`ovj^W@pԫY?9}ȡUiRLdGDAdO&k¨\ʛ"}d>F4vΐkNΐWr?Jތ6XFR3D.>W2CEeSѝJpX!\!(^8bD4PcaF Tt?LSby52|&9[gvWʋԧWR,f."&|z88Omc JxB AԂAil.,l$*WlnuXp!:{b5`CkSf:nd+s$F}DIĀzC)Y7!+0ϴpJV; 򝧷ЯN(_|iv5AfV7vh>薡9{[5=@ʹAA\ ʃ8j5&.Id@(_J%7d:櫐|q#.fLL\m^ ${@;}!c?@s]h;-CsVh]o͋\nKT&/ +\52T*[wigkƚ͈?9GLHt| \ZU*\̔vP.jՋ~2&3y$4!7RK.ϜXR&YH?5_\[r\ѡՀ,XԐab,gsPt endstream endobj 1525 0 obj << /Length 3523 /Filter /FlateDecode >> stream xkoRTFcO y\< >^Vn_>_]w/x4Ep3刌N^?{;jk߽ŧJ52X6:e*B)rt:N2uw1+ rQet՟N2f)g :0OG3nvte+4ʈR'ѱP59Y22hi^D28ISLA/UpXo^b}Ff^\Ccd}d5aD|wt,I $`+Ǫ_9@) L K``0\HlO&cmlcVİ+y0GPto8$aׁs ;̒|lg-0<.=2ٲ3@ JDX ̯Aj[(rB oȌ g SCX1P~@iE~.V;]ݯ&-玾xXX^}[ 8zS #'^&gʈWE&4ǵ.ml̈9xTe=< !?/jQ )C:prb[_U{ɽHN&SM  ;QàH^mJ7#m_Sbd]\mɔd[v_ ܶL3EWqmmYN-amØTry0ryh7u+e0"w6/g.Y~VNkuSg˦\\*\eR-&|>0HJ'yL\0*p,/QkA?'>;͹sjEĆcəp˺fP.|M9-s r *gfgO@rWMdFbF(S)Ȃ=oXKǰvیmLn_|r6h񝎊՞!) HnAjMbRL Ը<0R\e5rUL(Ŭ "/cv0,lT|cLOJʦvUݸ?EU^|RЊs2qhf}swocfZΧ/^!1<jB4y]v2cq2ŏ.ӛ*sDS6n P'k {t VSx\zv(cv';vh9C9t3?@~ԕ`(Y̊ʓЃ%m fu0S[70^57BK2e*`:B5.WA?K0:v^-+7:>j<n)*W}O.*9"uSscE{fph N^SgX_vO5;@t+x\`9q%GbYۻyӠg}·"m5oE,t@cCGP,%]S҅* ;a5>m[(q|Ǟް%2:`O3_Ac.Vꢧ?`ELA$c|M=x|&"ȯ@Œsϓe<ӋRҬE T0l5{҈.ǿ5JjzE'mJ^Ʊ E'\K( %BKbΉJU3 2ŏES"hhX5?"D\20![aG8zaGoG#BGs4q^regSbpHR4`$?넕u ږt])+@?@?Ϊ(cѽ+LXȃtѻQQ{:dsro]DfkjQ@B%tWT`:@0 i%?pr+>OfD;h4mvM$n؃$0h-G3>@t/,㯝K 6U!.*s5ulD].P6Jގr\zm*[cGbb 3K#X29B|2?iCl# zgV̬}de+#cw8;?dw|O4Hql!ެiWZ&ћB9s9Kb}|UGa_"ciM(»6< E/].æO9t'r z8 P%_ė)E,#;$JdzCer,04ic`ut{+c^nSӣ;_ k~nSIp\z?OSStGi4aD6Ur6L'l2\t\B—S4 6(2#]¬-Ly"Sg> %4(q,0W=.EJ,o"&HVX}R2_^D*-8SW@Y쪂{RI딌mu}X 2e;u& <_Rwņ~1'O"9~?thxd4YL^|,"(.-HƺLEN΋jrO'bK&_u-1e~>hIpjȱt-0G~d +SxwErH11Fcߏ]-11ǘ8Hǘk90l-:6{{޸D,NmD_[1fmj k wA$ٞUُ_vtv6Gnt3]W{rJtko <aғe<6Itv]"R]$oFUxўh=0,t"+"nѶiwU*T-oN_zB[ endstream endobj 1439 0 obj << /Type /ObjStm /N 100 /First 987 /Length 1641 /Filter /FlateDecode >> stream xX[oSG~Gx`^)HBIҢ<ĥ)OB}4Nm2Mx3gw;; .q"8,Dsf!A}Pػx'xI>HQ$Gy>&PIyP pYKSc౫*vJ QP^WAu48D[ bA 5bALQ.ʃE `AyQvAGS 5]]תAGk8D%PEyq +t9a.Ǯ昒:&dʙ,,vѡ#:l@P.AVCGeOX0.MVΕu6V :zTtR+*RduPMxE$:`d{w,F=de g"KbDo9<Tð8g bq0܏w%p gﮆ˓jjEgއwFpđT|A(R˾"x$obJS} }Rv%stg'l8}|r>Nkf8h?lnJ]*M޿˦MO X:=!ff..wQm&آZ'prn*2N-RqU*ܥSLFIXiV6> stream x[o6~_!B{؀f`Y8vJ`)ߗH)ta}Fm(paHx}TXXY2-.u6]dRHQ0<;qàe m:=*u! |+i:_} W+28MZvˮ*\B [''C`΂l!T4)U|Al5*@}O5sQ&O8/=<Oְػ5`gg5dx2~}UK^*"oۗX%6oMοYDS,qpj9 Uf-eGiKLV ZZyˣ kB4TאK8G P¨ a-FPE}g VA8 x<iϣ 7vBX ɂ<ְnzA 930 [C^gNvEs'sԹͿFl,Ɠ ?3wFE5t_x:Ì#J8xd)V}@7!5=U2Qd{oTP8`TPZGY׏H7-˾xY]^uȑR(PYYw袬 yCu<0ܸ"D/-><|jP%Ц lgFlt?!QTf]vjyG2*`V5-#Cmr ,NJ.JmTL>K|N endstream endobj 1596 0 obj << /Length 7502 /Filter /FlateDecode >> stream x]YǑ~>iZu^Q`3!!a IE̪>86:22"W 3Rg_ J +%[|XhЂDot/)!7zwZ?_2mn˛a?埡%Rl,LBRo[xM?ٛ¶zu(ݭ]pٽ7i}ީ {g  KW0?|]<;O&-p 5xhIDa4_=>Y6|t?U,?/HYs$c0AԒL $92N44W( o9fSi6oAW"_"U>}z-{9׫]I0I-C]97u] M,u~!%ՂVXSsa3V0'c}+Cg@v?鯜 ΜJpn-M._=NdAu,wT)l6sU >x "/֓^'_&--[(Oyf:v?<Oj^1DjT\:!d<ևDBV>y@vg&FHpD Kʍ($<rCUC\U-}KZ@r P,ָYTꒂ#'Z( V0@є3`98AR}A m mJSӃ~ e$gOs@)FʚFmA+^&7FP6RsbT3VEΦ)aQbhf  a6<1vT;'1BCPUWYrEBgN{t-PIͩifh#'˖l-d0>; %u$AN1. UƦL=iڪ%1=&J`"Ej}aA?Wl28YT%734G~23MtDYEYB8#.r6lRQjY<ÌI{tgϵ) z:x]nir$ *躥106' 8FFѼ VwOm@78d֊ ƎMl9eV4ṆS #7h.HMYܷ*}8HXXxLrq.:zeD% Atwˤ _;Jn8ӇC9m,j 9E̹nC<\`ؖ2&ǃ`i50#DD>H8p5Fй(TM{L"#t+Bը}J}Ήh@'4FP 0qgH\dKj i dV ݂/[uFvR{ڲi =\Lk[H@&:o9h6$wyw_y7Ccw13ynv<+'"N_CQd])BWZw41:2^76;pV,RT$hH+,]02PR&#oe@_<%~ M&{j`mW)4iP̅V;t?{"9+>:vRP.+w}7oOjgl*;:|lA"^w*fxohT}1g CcTL*f5o- $]^ cM1TаX..*(9Rd?@dY(j)jbƒګ8xOm'O &%~oJFĨnꑀҺf٠vڑgd2TJqr2;W'nP%z2_X 4e2p5 m7Qb Z6zrWn۬"qnH~>oƒH(tcz Ly_k'F~xyҞۓL̄Qy@ |(W S4i1=\S@O&STC@VR/WIqU&h>fm88MGCu]+(8^%exAVƝ&u.;PdWwe[ߤT/|ik۩hyQ@|W(A Y%=E<}>!%<͞F%H,K/V,=X7LڔPk٪hВ:Vǔ$m'S6S2 sI=oV1*-ETs)x.xWR-M ݏ0 җ ٴ ?ORk\ KS6mcTh/Fp?EGf&3qZ;G _B '4aͯ*(,pQaAcqk w Q1e,AAᱡ"S, R8 A@3@(kBiRiD6vM;AwQ )F_4q5U`ƻl T$y߹BIC"(JG I:34wBR?u i"S9Vr^е]=t刀%624Uxb/ S5_%r*3[]u=f4 W\MPLRj&bC27>(qW @lx}^z!?Pt]c 80ѡ@ƢKk*6D_&S:ɲht;\&s*L5_~Jm 7MQCB+h\1]}Е=yh؋EuGk6U6#5alaD)ܞ}۔eoڲ9 3>P1pA6F c#]SθbZ!ۦ8 ]a4LĢ I$$#h%}[3nbrj#kFGju? Z"X`[qˌ F0 U-)2jodVP)֙akXcƧ4{ Ƙ_5>Y*I6'O='2\dx .akMRowuXcg@#x;jnVVYZ3^<\A3]a`S~_(q3zTM09sL*&Wcl +01aOulLn<6zh # ~ aC?pgj?>?}L2e"Ǖ×MN(aCފ69)Z0+΅(loZߒU@E?R:1 #nV_VWN}%ݭ&܄{7#CS tYIhçԬ8jO碡,bo)"4nsIiZĪڤp  1ҊbƥVBMky,uY&0:zSw4}bW{jZs5&*/qK S.L{zٮ40].&& uaߏi x]zO f|t}p G cy(zetZ aizh:-8H̛ddS(了D-_wJI4AuN?e-hrG8:O \A|})8;PLvHRzFg ID ǶWtK׀eu"H}n/.rJܦ(m}X?nrB|li75Fj0] 3=*7軎Zt vTjJ$Qb>yj-r Ѱd;dp^2- A#&l_^˅w,hҠ(AUWEEa凅Ef,I4,cjvhw`MhTeՂWAZ  "X2%ꔂZj4ԺxӳwE%7$dTm$#,&`1R⤲*knVmO趥./l8f@stFN؍t/ .9 åh}S/d"էUj{.?"pܝC.:ͱӍ)p\CS ^FQ·oS$ʼN3^?EU;nz|38هcA.LU(|GzMAM byN )BU:A*oR^{O*8Cuz"+=xdIbjKۧ(r0C)Nsrt~6j_FK ]r~%}'/NzuDj~geޏù={8z@iw-F}d !?&f&mنtM=arf)f̸bab< M@wI\FK(yQ̿Yuۙ]ÑccdhF8*-d":(@$ "`/ΛZuls9%U(2RպcTC]ؔ# R@+0y9+/f U˜8vLJ1QNs$r·iDH${R1p;֫:J=4HI4Hh\f{ь SͤIw(6_AWV1xbL0R6߼Y]fTK|dgڬw5K'X _S/̙Σ# {T>l0ѭFrs~*+|Ӝm7g.#Ƹ^^u4 ҳ*#|/)S P }:zFm3StO;xhR7r3n〉ݿܽߞV :)usܓq;mv*}x`:m쁈Q %KkBEm#Fޜ׫?:֫~]<_re,z7yhg$A&vCG簿]n/Yȹ$)bniYV63sWN"­.] O}ɤD).WּCk;^*J~/SҗQLI=ٕRեNP`RD(᱈~{*wo-xWh+VXfkpnsl_rߟNfWX`cq> stream xێ6=_E5"")&P4mt[iزG%jLc8,; rD#wN& p N_Ƴ|kG7_tȦۑp 5p倸Rz)相8ӝV@tHl_ qƙ}aZ6Ou9ln,$'˝ l[qʄş؝2ũYe(޺Jz'n Ol{4l_T8 "H(.h%0F!::jQvT ~P N@  P4\E37a,u\[ `} t5i 1/NUptF-^AMW9D5V1 &jSrIB`ze4Yj&FAI?E(tYbQF`F`9(q['2) 6O{)jYslB9Mx4i14hOPnuښ_VPtO9l2*=({CDsD)exe&(? 1N±3*Tҧa2 #C1u(di\zƼdTN{a c/MUt(UJU._3af[YPJUGWgno GYgSɭJae8܂EDf$eH"@~!Ʀ^xg) cjGӐf(ٴ>@F3if/OW-Pn1ٓdÔƔh\֠(O! 44 7s;ddARnͅGWF"q0e0 -`UB:6 ShbWe ~ GxY;RO;gJ>b1(`{>mY+ -QEh˄wE a4_]F.<~m'nPN +ct"t@ ^1(B: G01F>IgK,EYGTM%J;lFa 6_/W-WĵK ՞T&w?#`eC 41Laj>ل=5AF˽PM=m(&cf(wFr"h%AX%<]L@! ATvBNcah-!42̶g.CʐcNwO#qBM3z=V1uv(5L"0֪ A %(ǓBjHA#_YK V_Ha3/D V*#m(܀J'ؕ;k{q쾫Y|h} Iv2/Iz}0M@.ʱ+ZiI q#//<ݯ!I'm KӴ0w0;M1aցBV$i=;g<%Y4չvk ;/yb;(]V< 9[$3hZ9h>sF.GR> stream xX[T7 ~_ĉDZ!qB%"@e\FTˢ~0K+eώ>)RgʩH˩/i3R]QSi5X ji&T0R˼0SEPTpᚚ]ӒvH5]3ҐfiLhN9\suu{X[7lm扺|}xgkbHXlHs5iBB` g PqcHp̮Hw!ԵN!z]M\m3 `C&lh7`cƄn,\sZ<̩fa~ͩvfaiؤ:m[5 $&H%t;*Z> ])jjAR ,fFF m-Xi5nj{ ]]]66$:j@~Ju_F#T$T B!a' ]&FmurZ3yÁ~9Jb;ټ}|ud(FvbT8\32AaĨ4!0F5LEy{&\ +Z7̵֣c`f2Lv(C2P9uX vLF1̬49\eΕ2ja< eOB=Ğn7$-Mq){{@:+{BձQ%+ɭԾɺ!+sg///YZ=M˓t'OY;XzsƚZl^<_5X}~tfnZa!clvK̟:`<~ҟܼZ-/N^Η7˝3A>7_ږ5amlx+*4Q|k[Qrod_mw?K~zճ(_['`Cꌣ:z:"s h=dpNeİ:lj{ գX&U B[UU̕0# 3PxtRn |eJ="Q:i(3wAp3ufi 30u~A[ԏl>__9[,ԧ 6-=Zm|?} -0Qo#T}Ceo+Dqc}s__G|y}A}YEAX6+HF}>oTNIZdNA_/|R ?XpF1r+1>5E|`9@T3dyEawUJU[ef 2T121?F%n,B-(s4J9#%umL۫.(enLv%k\vTnb`GQfG( `v<`(#bhYdF &hUdEVE[UOn< endstream endobj 1659 0 obj << /Length 2129 /Filter /FlateDecode >> stream x\n7}W} @P 5"AM5G-7(%uƮF4W̜C*eUɜ>R׽~YXuMGJ2RtJZ'n/uCeZ?Cߎa#8w_@ 5hn[8nАşeaUV~[IF9j$3hZ# Q37. l*tPtY)(_f{ kIݥJc jXR6S+1>w2쑯W4eY#D;ZvUCf},mnB܇*لB=H n/c/DIlU{r^!q6*^r9&qm`= 6e9ξOLޱūEj 5)f1k#ԛjBgI5̺ (sDM&mVŋ< &Cnpϧƒ'"R,y(O7ewp+nUeK͂Ar\;n֒yۭ= SnR?9pf*a)4|y3y`us^3Ӫ; Pg[U_eВ<@Oq zY$=ʦ:pɜ&_N:A3*HNhґƍ8(I܀hVi9%d'ު/ b>-b% 7G8|D` _rr 99+8 e]gM5o'T;>a^1Ty=+;EO^@%\$Hؑ%C E9xN绍 gp󂌶֏ۧ%~. R#`YA}c23Ooz)0I0^}isQI'IYnrmjpp9Crd2Mhkq^\r6VvliVLQvy޿刵 endstream endobj 1711 0 obj << /Length1 2242 /Length2 9464 /Length3 0 /Length 10653 /Filter /FlateDecode >> stream xڍTi$Kb莡Sb[iiTPK$yx߫[g}ϏVSSf VA@.Q!ËȨ ;gdA`P_PP($#U XԸO`P&,bk_,Vi'+ v`@ -,rrス>fxBvml Y2@4.LFO{\{# uqZ]tU`Ϊ:p w&@YYAPo`q4T^pjy {? Z}f qqA36CeaNN`( >9+~߽:\('gzP;XY/{?6[0 #+$(^Vv?v?5:Ü6e!6L_7wu ~+8l b ]!^x~~gva0?1JN^A_ $ =&_60ȟz7=j&{.u},t),N"wG?8˟ '_50@jsvwSe8~o$Mք?z?gk ?o'_~o%V0럃+ 1I Hk !60W̟g p[נ \b YoWZ&;M<?7eoaw/k_+W_Js& e< ކx_oA_?5 /E֯7"" 7J-ٯܮ 럷ѯ@_޿orx#k sI~^+w{?.{=Vӓ0+0aҏ<9% Y9}]/qSX2B]ϤS>ͭȳJUG'i5\]Yk:L=#]EAɩ+w\1]W gWeׇ/kZe*XF9cbLߏ3ZfNѡ9z=?=#y7/x7bgX׭ܘ f)߂q6w&gp f lM Wg]O}؃CE=w˪YS"G0{8}4 +,bt凿$erW5Yxjg! I 271,;$lB:qbOG2MF]˕Hw--WRQLhfze4sė[BDUDO)] [ |##YAyZ۞)2~$^iγ4kbG[bɸt݂,]*$eiڱ6O`+0DZ4!R,43b_?c`$e*Nuq^_QƵXٱpV|SB5F]oE#}C#Rh>ژ`y:~)3Ȝ竎pRd[z&H. *o:*9Me1ҍ}/L$_^ ϲvqOY+_ެGVدh$ 2i5n2層 R}q3%)YyU~KMZH1*MW{2d~ve>ݩmFD1.]ű\kB}rZ =JP2JT|ƒmxb囻%& 2l)-\Ąr*H:?R FLT.2\k ȕx 16ҽ%IH -@/1ȳO'8s/CˎqepXX"½U_&_wh:΋bl7}>6xی"EMMEmFOAc۪Jh$Z@h51a,ŗACLۜ,r<<6TD&٥s^1^mJ no=#/bcVj5o> D6vɘB;=@~M'Yh2wq:Z6[Jag2ObZT6]M|؋39HN- gG&Z6Zl'DK$qnvsFM!HC%gϛDmH vqsCXb:7O ?X(YX `F]]Cr#ctáyKqVE:Qp2B70>,Lꣅck5:ͯ0[(G2wQ_w*CSP6*I.ss"/=~+5)RhOt rjq!,P֔ѵYQE-x{88L_ãLpWȉoh=V@ٖq/hV`ش8&?o`%}؉R ҋFE2X:}-+!+,WJ վtD^Šh8{];Ky*'4[]Y†HqTY_^CP2 !I"hǑϯT4ϩ'W9W5<*ۤ CDfwS#Lj^o}>#=26Z5KY}oGb@Tvp|G,l\)W]Blz:/?`Ff0i>՝;4ވ_eUJ*Ū.#=VNwTlzW&N?`s%\":rxo*S=B+.MĬQ͐q +-W8hO }AJ.HdrrV]gṯ1dÐ?LwRԺS-N-+djCE9AJ)jP77$[?K]Jj&_& :pb,D>Z%)I Y&~C\a07hوM`M.'L#If*g޶صeoz銶?4ʖwh6;X,mDog,ph\ZsHހ6ӚㅛL Tdi fzkΛɧ46 5Kr!tzibk&ZGck"_)#.<[S6 ߄(M73B!oeҮS?x6J>3|htQ`|N=T͙0"2BݔLM]\}GUأTdo+>@:+]acڂ$CL].`Z\mNS.QHӐZ'1S^l2:ɮmL3 Bc&5gzk0eH>k0qrXI,FOzK/6qi;,vz)ɝ8WөƩ蔄($3CRy115-Y/+˹p:>W|L$9OKg13շݹi-]J`HAz;d-//- 5\'?F Zz{A}WcT䓥aEə7G .LFk;L.T}rhH%`ՈvcZ\v7h\6s_Ww\ݙF]" h6Ki9")dF=I>=Э;y<2i4 4MaVTcv\R/.Cf>1W< 89岙ai)5V*TϤұ ry xֶГ̗sYPyȔ.u2zPN\tP]jxtfz8eV4\ i W3k<#:"itZ/Ļ}~.;D5t5I iZ7P9>tNqu)rMcèl̑rWoh.m+T0+L"#pK<9i5,$7~AfQ}_9٪M.ZO{$4Ih|*kt[TEm(mmQ,یbp;#"FI3={wR cmmN|Qφ-FYo>=.*0Y@..-Tx̶*dI4&bVrxh!r 'Aף!ʯJa Tsϟ6^I9QRNOjʓ7h3"Eϥ& *bgue*1Hz}Pgْaj&_!Q7~ Q}2"|Alq)쉎IqhDQΨ)D CMՓU$R\g}WT qE5n:5J0w"E)ZMw HzeeKʡyN}i\=^ٿ$دTۂ5|.~'rm s)ӷG0BJǧdD1lM9KZ;dA'+7ȏǗ!dNyNW}#\ xFy#x'=<ю/C.橺hK|^O߄*,:h\(yen jDY/ $kw]^r:hl㖚'`Vؤ<3염-aÍŎfNH!kH-Ef 7r,&DpEL@2K5ήON9Jdf>ch'q @ JC<,ZTnRI'υ̩[7ȃVʱ]xqwO1p1)]pPRFv+ ~XiG<~1ЄfK9vH&I2Wƭk un W)s7qb}PsBswRg{:/Ĝ({$۶ۡ ?X |_+Y`p dJ^v>VlOO.jeH[~ ZQFvxi Hg˒XW2UPތsjލ1Ce'U7TY]֐6N(QHɪ}?xsKg"THҏK9du/#@? 8%۬p5_6 [ z?]Z|nҗ=:$D(XO.Wz%nd[>yȨݏE&o9>4W4}\oTU;H["=f SR3ev̳<-7UKI,t3gzЁA>p%z}tl64 FV&׽VsG2[3+Xbu%Y2j]kΝ}XAYdͷ^&ΰ 'S3@*za ->I2Uי\M hE1 ӭȇ72?@kfO)8fFJԃg 85Ωw0lMhc) ^oԱ'w5CYLHPVZxZfa B v[ 2,%_u(vLIjؔp1<ĩ`b(o ;]$U+[axkP;ʏ5о23q㗘MqS c׸Sۦ7(q(D6{cO;I>~51J%)$;A\U{KM&%A!!kZ"lc_pU=T?*p|NLa ɐq5aU7&Ojuv XE56>R #f}~tqd+KЯ _bᩘt儤OE1O71:< Qz3A!I(M%U a*cv'2JiT-wP1cly ɡI\H=D&igM0B;sMfIR[9m67u˳PwӃ<3=og)3wI+'tu .;dnmmt:/ݛhg$+Ngln)6N'ՑGIɇϧx|lvk$ؓ}>\ߪHAHS+0Tf3R36 blޟt|E1 L3fnQ&j#ƟRiz9{I3j.LB/ئ E)nGmD`|ĸ %b 9#pu$G]/ܩ+E[Mvk^unCϔ#ivc7BiMc @fKwģ*bfh6)OV /BYoNWo Ycw"LL0c>k@`Or\".d7~%b(Lʣ0`=cЮțfD~E~WLݸhE6=46=SмcX)ɗyʃv[-xD:QL9}{ Q%!,6{Ą/fN3d;9Ӆ/ElXb;M$yw<Dzq(ٗJFKm,I9)0ɦǩCE7Ov/`hoes*#+(?=[ ̓z9^ݟՀ$.m(ej➜Ad }f;6EFmbЍ6+'&pUaI908t!%,AMdr0/Sy~0aGCA/إ/fE2R|i&f4/6cӴu$  Z-s:Hn?J-*l*Oȝ U6ɫ7}C".ݍ"Hl\#i=V dlVP6&/YʆKJT\sXYr[~]J~bj'}W4`/mSk^-PVtQ[WB+X@Va?CW&n A%8?iG&hjt{mTqf|U p@Kͧ YF%s¾kF\VM{]ܱ AmF$ia~M: ԦPP&pD/jU;ޑeF?V>zvGkGg(G)[gz{m[X#&#!o\L㤸z$yI7oOVAnf ww0ޯ?_04q[\D)=?!CK p6%&Hé'0([gQ1UHj]bXmz_| )ywQu$Djya\Bs+j`@EZwt߅Qw+4z7궜!{CT"Q;Su뺨 4z&jihk *Uc; baԾ!ˎc$xV8{SM 2~s whpJ/j[uIaJ-2;mh[}(啘#"afE z-Ksc萼ݵP(GD$U spU0|U #m̝s(y@REknG.yVI$Tgϝ / @Əf#*[ ;ia!!a(B* )fiДYrKOķ+CQdN( m5a=_\Ʀh\{Mܴ߾h * ?]^$J REngurw5FI28]eC=Tn'Mf\UAٍrvtMMu.[ endstream endobj 1713 0 obj << /Length1 1894 /Length2 9481 /Length3 0 /Length 10703 /Filter /FlateDecode >> stream xڍxTo6 RH@#l4H7 ݍ(H#!]M9wv{~팑NS穴5 prdtx|ܼXp0/Q BD%! #09K8BP &{X+=dQW{O 7xܿ^?": {#\rҪ7)#<<~գii Dp\vXVuC /7v}忴?zY_=O DSE]5dᖈi":)?'7=Ldi5z liBa6 7CtDl5S ׮<5lKWWK/,D'b*A W6PW_Ae r:KON9;B=A.n [A >?jK ow @0o'Bp{gD^9!pIJA`Np/_:%A_"b8M1JDN08osI_^!Bl2rF TF#`z';C]W x~eT #0  sw"nOs\]{@ (>ZՎx8;ULi J4ͅҰ9_JW/SrH2GOOmA4P u-j< T$Pw[wmc=.J:c%ڈa>b?$MWYo%>R-k$6c˗4|*0"rFɉ7d.dKS ˜X-& ۪>x9V Y>2ҽ!kJNcaFI?j;f%F(b_mM]YL5M ~}+# q1t,@DH@am ~rC$NPHO;5t`a{^nBG}6yAx<Ӝ5.'n_;7UCQXJda>N/O`^Wu)]H4T:`{)OcoăS_.M!i׬?~hozŋaalgQo߰#c4x$4+Sߜ\W7gҵiucT])۞J?.PFEC q-_#L,UΚ0\{lήyL.7vo\dx "|jL">r8„8AsN$; '=(GT_Zjz/)߸k}e!8% roR.H-ˆ4 (nv,PSHwݜQes)T)s.sgo+⇁:-v|IAFs Sz:V䅿*; ?p0)p#fn3Aw@[w -G4׮2-s h_R1<K(msmαA7)'epbGzo}|ϵ,6JR:N=Q+( J%=kbt'ȶ<և֦-'J&ӊ%g׈~X/.xA_~ݐI:}][6e5[KwFʜ#gd4ݑ䕕 x[ZJ 6c1ZyXۑxyO53\k2^FQU[׾{w܄I^M*F^RpXeMoGf9$j$WBqIŗ4y`~xIRfG|)6-a /f ֐: niFPEoNOmO}>{&2;k`1k,{굼Vm>n杔)F8B!Lq!Nz\jNXE*L1 W': ;3/Af L1'FX3SHH7Wښk&f_%)jUv<[73$H,i Aq|wR:FeF\D}e))ڇ~>SR3Tw@qgzY2 C;FuٷRzIfNLӳYl+x8)_0^#½ǼMXH%fwmnE0/vR,|&v 2W:[|=>^DRU*آ!5%}~ h07 \m,T-:7>%YDE)\dZK!1*KQphߥAQׁ³oK+,f'xQoBZb}3CZ-ĕ!y^|>/ȥK~| Qm`odV )[]6v>wzuf$[4ȦS8͕FI= nꅟuާ^{V]leGNZ2EnF.s_$=,R N*GAT.<Z819>\[TϽ?U˩j? l| 3?4j_A= axf~\M1-`@ۖn2-iCǧ7 ?TBOU'}E|k;F{* 箋XttX"apXu.yTrǢcPBIQҺD6Yh}c!{y"uIdIғ`'Kk]5f!BGlD٪( E;%2E>,R۠|xE+d NC:I:ѶZm7PeI{tL+&ģYݶdތ%{'KLDa^n;~7 ) ڵ~Ȟ͔^Ԯf}L&Lf/,S丘&oc3 Y:[סW@v8H(|;0>m ߟB %p(w~*\;{% mGAމ1|y8Nf-5hWzڟxM[ K#xqhSI@kUx| mg:k_dldK8ţ]ZG /Nӆ#f)ǜNzC7& _-bܼ"=Tdϥ )ܝk4XByCCF2&3!Hۆmk/R[Ejq1:.իΤ씖孄!ݐS xDjLkpO?/sZ#lg^Wn8軅%E(p¾0{&GiepVR+<{EH=E+4Y6X[-Sﳕ)j2d ne@ó^NgzE;P>o0g)>LE/`ZG93x-(w ~n~Bje"A(̶1ċ upރ'|eZNOgG>x7#p|5Xuێ}`ʿWEziQ*j*u2zz"͙U&Ik+#j nK”0YiNG|'X&og9hN'&Bc$u{oH\ t!'g1'\N96gU)kW^OX֫,?Qӵidsq3V +)_E*,,#U/EƷ1Y뷫]ɫy;*PwFCaE5-άh?i3 YkfYd?RSk qpgj]SGjP5zj$Z&4sٟ,ӋNQ8׻ҤoU c/o6 Ir\`[SRf։:t)?[;.f1K%=$ZͪORZҠOG𠶙T&9.Jr''^Si͏5[zd./U4j_י4*Fbx/O`nc>IM DG|+=gn[`᧍EI:M)kݺ3%le>Wj'עb iC= MČ,6$\U̥-p>j?3x(>e]@]WTk\K*m=JAK3vx?kMᪿ(B0j ^pfr-btX+ZuRR{Gne_Jv>nJTFC9c5MN?2S>%x | }n:_F7q+/:'7/@p6ZZ^APXQuK-U |7W:ൽ9җ#+pjև/0[W2bƂpN.N^ )8x}s<9>27B3 XHpymA:Ud 'sv_O%n-Q 59P\Eh:q?Jm8jiNAO6TLnE2`q@BKk\XvR$)S]\q _]{X( ,rfج)1 =TV'/ģipxeL kI:p,/mG5b~$^y%aUl?\; %q0ŚX]^H} #:Fا$Ʃ5W8cXBk wh#04ԋPgot}$d}x"k'YݶI2Qc5Ǟɘii&]w< v>`h\Qݽ5@EеbTXpS#=8;]]Lo4[n.ﳩpjy?K5 тf_P4sYO+p ܏;*oHؤ6' -yeaEu[grMgTrQbkYB+?0+%@=U< r)o/#: G(B҄fa-3$hw(h7N87N~ݴ1486]:$ \:9X -QZTiպ4ʡ@t_gukE0!)Irx+ڛxL>z,?nb%u^MgCBkJ0%u2JgtPSOns6y$[,]61-usnc]ꛣ:5Lw> aAk80?{dqw=oOv$X +> ϴl$?|hh/f2]PseXN5xL(%rك[{' Z'L>8I?|± nUx'dst/a7K!~M HL t81dld;5# j]]mZwQw:mZ@i7H2LW.OH}%xIB%1C-Ӓ&JK~ݫ ٖמ޷R+!n,ѷ^UzXq?n1:-fp pu  #x>ÚD_b<bd.9M` %s\%~:PdTw4z;3C[tfTdUNh˭><0/#Ge?LX,qC ЬI__|u<|Ԝ^fy’)CUΦos.BC#`&ն~m wŅa_$^޶{}suА#Siv};q];]80짬IH[Cfzlǡ9ѻґQx=j8.2 _Pi2OKyrzgc>\yteUH% * LU8=M.l0sH?.fk1YʼlB{دdaLA̝IP0R8C GcT_43 :0R/OB6q&7Qp^\6';p'X.E|R):J|K$*Y捸YADqmT+OfNKljmn^'Yj@}wX f_>0CAѬYG-BZ@% Qp-NYjEsE'@wرCDDlV*-%µK^*hm(SFw,GѮae}"杸#k}vl%jO궡O=C`9U ~6u"& {Ign~Al_[yW\3zFXEe ߖ=u#$ߪ\kǺ ~x~x4픵 ({L/^GؤSz7 \?8ж>v~z}RxP3VɀfV.ݟ"@zpDlSϋxIѡ7$IBEƔxC*&鳹v ʩ%Iv- Ar@ܺ?͖VzV͇^I+LXx4ւnNJ/1> G[9R9pF7ksOr sowpIpƌ7梍fG!?~c<H"ڎS\c,s=qP\y:殟8 l Y+n_CבGߩsCu+;A N[`Pa@aѬ1: 6E,$'*y`^,ϒB]2YMgEc{\@* G `1g_ vhJceKUPe7C[I3}cW_>ҧكҫ mSt0]F̶<97I"V;K GqMt Xf͎j572CCwg-KOY!h=Q.Y'n b.5~d>dh gWĕ4J|,%nHyv5"\jD%ڗ/|J|q壧Cӛ-g4W+o3螕O<*8!HTw\8ђj[|<P =A &zޜe]E.j,?+!|]5*Sa55gu;-lriݛp\, endstream endobj 1715 0 obj << /Length1 1597 /Length2 7313 /Length3 0 /Length 8401 /Filter /FlateDecode >> stream xڍtTk.(1(%C#H7҂0303t7R4 )!HJw7 *ouΚyz8tV8 '/ P7 qp@Q`"C $"B1% 4\A1IAqII j ha$"C+| (!!~w8@ A0eqB9`(Rv($rB2ܡ(; AA<9AtO0"p;G(C#\a}8@_] qpl+W"(w0 ;9`P-hh QA!@4: _e$T*v߹Ѭ!e Rqutmm? '_hҺЂeoW#jANmUGB٢'(/ "Uk( l2'AtH读% _6o- íLHT B @Dh: тxf2!t8ZD@3qqEm a$zPz;vam/6 KcbB# (VQ'WG]" bE:0]}kAoh+ sE e tF@ѫ; @7~@'Mq`‚@< `I8a}EH÷2yFwAq&(Ωo;M41g%ttF'}y yf.$e0;W^^BX ⯞2~fcƕh4#TMxF&䦩. PpAƖ̈jŊ2ac_FR R\ ԟ Ċ~Uh\5M훟|%|T$K-?:>C zXwSSń {lɾ_о1\U0: #/CzV Ip= 6V/N% L3#۸\n=־\lMs=e 4j:?YA+6lwaHNT&fं%>ӮQ:n)ٲ啚IDX/rj9kԪZy?Edj{/$ZLIȉ #CfPI&Bc_I/{a]2f>U-?E<0߽1BuTnvl:^қ<RkBL^P&R eyDc| ŪL͵Sr~#[ȽIA_[8~ dw Dv~A G+^9kǻLYb"4V ߜ߾[Ę"ۂp56"T.n/ʏ!\KA|gVUILY ч~c7O {e:bIhg|m:o#V!"yԱh]mI {ȐZ[ߑ7H{A93W^/da xgP\(z2n"3f#וp@#9>y/=qIZ| };kwv fivre_z>y0 D.|x^C!c|QaywALKx_o`&l7fJou"j=Zt9 k~y1Hpr*hsSif;H@ٟdJ'TNSiWI+Vk >f!WQݕ"R*DE:fEx)1e37~̩g9?jpgvo?W8#`ǻG8Y|Ta%1Ť!m&Y?%+hq*w`*(ɻ}au- $DyuF ]EʢK+iVc9>PYml%kMiXe'VnU~TwAr; F{ۈfTY!sdf"2[|$Q\3GfgszcK}2h/n?ϾE}eY ~;G<6i,!%͒Dy VlPߝFMxjoxy!аWu;.c̛k|n}DOţAJhx@LYr9[|2gI/R2bJFdn{Xu> 6 #|ߛ{gMt}7ƕ:gPAq_.QG$5`,V^^ѵ4/%R>'0- J: 0,AfH]2 AwWdSqID<̓lB"1SSzk[3FL5zYk#H2Siz(~a"zϋ 1?Z#Ӛ&8er(K~'*X`-MMy=xmCK/o@% :B}tϒ'j{O y3dUq<:Mɯ;5M=6st4fnȿ~o+F0FzdHFlmטta>Um# &6a Ά_8ˍҚGn"Ny\HD59|SE ?ٱUq1Y,vM'Uzƹ~mHyrfBQEzz#Oc/,=({IfMgw:#8_JgPyz匿$igڂ;\8@PRӹ+X%U-:Ёb;/%m^zEV4<ZՍڛ.=QoRUDpw?vZ3X 5Caz))M|g"b9Lj9TgiqE8}#v\Th`l+|>, ^QONe51OxL^QW@/I: rR1#hVX#MCC@tn4 BYP۶3%pnH&&5n\rl=4As7?G,d[|gUXw%\lor& 8aFאָ&ewh`Y>#k.<;0'#`Zt&t+M w:!9fz$+ryaB1eXㆩ|dKӖiOsʆLz[~^U{+^L-U/:VZR O׳Ay"^͖ ]1ǏZ狋╵1cV/D&Rôz%8h}L}唼b[T l4zluwZqXr-UVvN{#4LakZ;l ӳn?sqE8[aptzc&DVZqS{YraKQ^{s[|jde? n"O$Uh=e>s/I01s;4*^EZ.̀~LB$'[ZKޱtL!ş7E 5L _g}j+6%t-y#=z b<+0usOd0Ъ^ifL>=q`崤`maBf3qhᣑfQӓёasm~~To} )$1ϟ^-CS><4Qq^)_SRSڕ@pC}\)<Ƕ"kWƚlwIvU=guU,?vދ᭗zQc LFT0"c= \|^?7!>۞B{۫*[Eη1(viB5Tnn1L1Aր'2H(I-f V3){_M&&\dǎ314Iw)fX:8{̰r0&|WAI+ HnWKpU{ԺdϾ̞4cR 2~'cʩuBM4<iv= /P2o+%x&hCw UnJb&;n̽yQ`Zv"wq4֓ѥnXQ7PR,(qe #.r[ucz ({1G5L)6Ϲ|jޡ<#S'|Mw(W<&qhau#:"4L8?_Ӵ w 0.p~Y᠆rz JsPj+ m= Ev\*MFWWS&?70#2Ъ ;=*g&Ucm5Ya=>90hJIz򲐕kz;ΡީO;1߼$K$m?+(}YKK/gZklHANsYFP9hnzAݧlV [&5|b4^l2Iz,t/}/ /L3qMw5%coE${fY.FɽG/Joz/~]TVx8*Ur߫yw {A7V7s>\z2"ENR&-Dߎ>`EK^" %pl̟PPY8͸>#U*JIWQm{61[+ wgr[S6]8jay쒹*tFx|Wzݛ|=JbfS-ƸՌ48bU:&ϼƔz$hWl+NLN|n(d7I:ҵ;w$)I,`4+oafGüd1*s#Hc`rF<<s :x֒L[>+9 \:xv bk&Jf˥RɜCwF@"ٴ>GCP ڕxM[tߧX1)me|<ąÉǼbH=pݱܕ0gΝY. q T]wk77$R-T=hNf\ R+\k+9f5zu1%p̺z,vB+G,Ns6:.1\?rΌ<ُ{~wuƸh*א8jLWO|} qZ>GԢ~6Niz!8@P GFDEZC3WD[j7z([FsJSwBrJnuD ~%R Q(ߪ%\שcou!o"ع`5;32ù14T##)p4k1uqKAOt?t΃gQŨ\ gAB,$ w5jKV:r$0\V6 ۟U~i$- |J%Kw둻HM֫fLrsM9Iߎ]c5yŜnHOj RΦpa, %Ȕuo Մ>+g}XjdFFP먓.@)H:Kq ª𕰫+}m 86™Kև8 w4D9b",21d3*92d^MA2?Ucѫr|'<{q\DI*v}q"P3O(];y.} tIKnIH#752K,gb4ulIסqavGOlߕ b뺯5U%7=̰HS$*NB9w=Efͻ&/7(6Ű{TcI,qC4PpˎQO1 D`O))e}{꬟q0m$( n = nڻ̵绐NRQSt~bzY4;:Wifwvgd Jb˼\^},3H]6w[ S%1dQjf |]||u#ۓhk!W_۝kѸFd/ۈ:>~I6%_3RefC%%L]h]"C3]OS7 endstream endobj 1717 0 obj << /Length1 1648 /Length2 3429 /Length3 0 /Length 4396 /Filter /FlateDecode >> stream xڕT 8mNv%ed{fϞ%3L̘d z)eDdɒHY#dVַ;c뼞TPx#T337BU_.99g  sɹD[ D1ShVS`P(0؏DP8, @h.5;Z MZk_W ja>(` qĵ:3| ʃy.HYx.X.={x.RU|]u|iFk}Jw}EY g!ltussB:ue+"CeX0{q}^}'hHSg!hZ[<cC;|z'X, EʴMJqL(5ӟxTI/7I79!w{-MﺐWKJcxY w5-Kf1/=Vvo;S>yÝW 6^=uCQ:vb)pO02'>$u=fSrhu>~SD +k݋Wbt;5] lKK\>Q)Q/}I(EM(@ {"n5_sv$g1VbN+INɡe緣`t(mat]dwf֭x$7Ÿ^Z!P`ϸ#8G]GR:bUKA{k4% !j}yW{y/ {p7nڛZ_H_vUZ4lȏڙˋkΫ$kߡ NI\^5NNML#hڽ.P}&[)sZz469d X ۧӛjjbv~geF?T +Vģ)طrJ1mG[:h1oӼ}butU zoKHA݈M'U}Vuj";\ktSEJ YΛ]I8A>a C{4RowōC(+VG+9s1NI/ t%g,YU<'Խ:{j]д<~. QjǬ%vef @tVO,sQ:tYbQrϘ| Fv3Y3ާlxg|<@BHs)lB@G?0WHbJAQv9_R?,F'(id_Dw圝v# K%td]]۠[qem]ZYS!@,Wa hOL=0r&|f]3Duq"zFlYυǫfDsբ*oUvIǖl,zSӽ-"S1̩/?=~ԫܒIyŗ k:9Ubj{jJQ}HLK b<2 0Z$Uy%~" AݶG{L&{L/%CspM Mk۱\S&.Ci"cfj>W'tHrw鹇b}xh8Y@4;B/0-;H-r7 a'-kr/ eGrGө,Wy$ 'R R 9I{\<_i*4]0YxGx6H.0f~*(y^%sзW+ye8|9zB[q:bδ!7aoĵyexxP;\V&:'~jE6cNwmj5Re"վ^^L7](%@%z?2i|2-cF-_@t@£]z--]}?kAe.ՖN%Q<\"FVX=j1T_1< ;O]xcqq?J~(;[qK%G̭1"mB!Ym7N婄3lia}~0-ɴ_4Y,6V]s+PՆ:Д}ihQX}4Ҹ=&G♦ɢ).u7n$I 07Dڦhܵ4B6Ӏ0,oglR#{&ǎ_%-/||x-c ?=#RC9vp嵗!: Co9uqH픾i_e?^WYT$wyB j_%w:DBSuqDo!ܧcm~Kf Oٰpl<`"ܳOaik~O`uj2뎳:;v ~^WN{<5rHMl\N gQ[A+ﰓO |_ s>P(8o'a \J>:֘U.&h?dO~=?/p\Oʧ vA[!R> X5[G!t&' /4 ]^Rwi7g\m5gDʽ! &$ΗNyGemI4U}#;&G/fr˞Ptsfr/dM~ Q q)1ݺM$1]EkzCl'j I[j Qv[]eZC.tñ~FfC? z<|I]hC3Μ|=D ;|pIruZbt@6w+-8҅&GZq<_2_ZٙcBX[^Q\?33"~{ FW? yڣ{ie7]6ūDM}{!D+Dm6s"M5a]t]ْb+'=tΖ\2qr(W!UfoϊV_&r%^;]N{RUruIff;%b;Kc1B endstream endobj 1719 0 obj << /Length1 1391 /Length2 1741 /Length3 0 /Length 2602 /Filter /FlateDecode >> stream xڕT 8T&1+j"qk5rir4sp93g %עt5T&eɵD\Z=źJR"Mvyr{}=9fL5X'/g*P(tK 0ƅfqq $(bba8p "PJPQY"*Wׄ@`X8mD08"]Dv2M4M  S@02bQHpsd1`!I"+DqV( vY~3`'d `>&4ɓa;!ǃL \-*L\ (fc`#d $8GğTe03"p dW K1qԛ2/g;e7&Ń3 |I``,\={$W2 BgX qa qikr40BxrC1B†;B`6c"C0|1^Td7{窙2:!l39R4Հ% DP@b&./"A * LmA;_ؤq§ŝ5d6KEV!D `D$2_,KL3Ww3ޟ#3ILbsKř /=RU;winԲ_;qpChmC}{;#r{2mWCM72RYޱ]2aѫ[ kd V/yQԣ->?Xz0͋<F ~>A#HiYfD̑d*}zAlSQ+N,I_M ҷ*i]^kӃ?F9*)z)7QUI%p“zxPYQvN_RzIt¸YNY6{#bֶ[CaTSҤG w)UӨ6.z)\n16+IoMK_M CX{6Hy.CR)!}/y )]uu?O} zڥnŽЊѠR(Q*ALu֭oT樼/٦yvT- fvA"zKϵ!Frod1֫_ ]9]X,Gji\jփ|߅ѕyO/$,ŲulC|=[gi=)ϗ/)vT$ę~k {5NH7x6 z-F-:41I9Y{W'G7~]j .ԋ\ߔwQ_#,gsWlefW ̓1V3Zl3"{44 3@Z4{g+.px&/jYk9g])9Lq8+xwNIƌf7/ަů'׏_#^З 0yVΫl:}d'5F[һc+py&DSOwnM6ԗRr@eԁQLDUeB7˽Q!q/卉.tsu΁g#.k "Z-wmU;O(Isx^G w]^ܿ)uJؼ5 txԠ>+|m*.=PkN=U=#lB/P/ !> endstream endobj 1721 0 obj << /Length1 1317 /Length2 1506 /Length3 0 /Length 2330 /Filter /FlateDecode >> stream xڕS 8TNR M˄03g4DԺָǴRN39v4 3D6- *QPTQS*b+QU۞A}^>fv<Yfa ;[ÜF~)T*%8L~HpTY A`a之0"d 5A~A"5" c&#!PDb #0Ҕư2/  BlM`!NIW0oG$(<`!26.B>> " (p2C _|nS`cd1)_hЧrB(6 s"( PöXJ۴j~]uдK?'j(V:a(8sܣcќuUh>pgpN*AuVF^NWGZ(;Ce_vcﱶn RRu/2lycVzG5t_-8U:oӣS }rf&.MF*ġϪ)˲sNnM,7 rͺ XV}.k63!iZwƞg$W?XT>0!.izCEI3o/4-3늻׆;͙v~zgŖDݫ_ubT' 5']}UM CB#Z %1ޥDWCc4{1u֖wfVf|FxuA o4&3{1s7:YS՟v$Gk{CU#w\2MT_ly¨Is[?̪5zn0!`3Fkr}a'f}C/{եq7F%3ZzI~v ֊6M˗-Kȍ^/N~I9#fT=8~~V[v~fR?@54|A}:UWmsOl6CB8,ײǮgl.ɭe{NY52#~lү>'[V7\i<t /ȏ=?yw-Y'A\Nq{߮is#,)w;]+q)b7Oِͼvy<8d=eՅ%wFmfbsBu,)Z5U£1%Cpipn[.E7N~߼ǰd% [W?jV^*j8H}z׷w䯎PȊWt:8Cԁ(Ԓ +z_?kv[g}Fv6=t0H7ڤLt9/bjhu#8iYxz/}d `([{%ibj oZJOҧw=.q8/6Gg]מUbD=']qj>(Q%Cv7s[/ϊS&1ᇗUy̎@~GY}alernpNwRIۦ+{ѿ{nuHLvDu1ԇqxU[cS17G o3θN(w! Bv5\ϑl.!2楽9/I`M:FiwrT^;rӫ}:٨̱Q}2 Obrn\>RQRl}Jm07kj_AzDyٵ- w5! ;,gWY endstream endobj 1723 0 obj << /Length1 1425 /Length2 3010 /Length3 0 /Length 3897 /Filter /FlateDecode >> stream xڕS 8T JdVtP/ cɾDjw ("KP JKBED("BߡyΜ9rNj8ޒBjHu`bD:STt#C_z>E znpg&CiT GJo b:= f>ZM*pT\ MY]z@'9G*|Gl.8<} /~Hv%&VyCEՉp {[x9x>ݶ;E.Ļ 5a9J@=7)skǠBQ8Nj2o3DM<9}~8Z;f䑝߂>H Ѫ3B;7BEN,&fu2ME#eȧݣuGV;VsWYUdJqբYrONxy<Ӿ+͠챼lN{JqdWhx/Q[%ўٶ}βwCQd~=_v-;-c_+Q8V5I%ɵ/妭։ӫCL?9:j& H/ OыAJQ~Vdm6zsghbg{>ԔEkx0,.&1}UEN+oȦKlSz#ȕp)# 4C@nOs]vve|t2<&'G1z,{i a|`_Oדy KVEGK 5K3M#8_Ν׈7V^f}6IB,@am4rj}bZ8Êuc3 8QRAՎdQ1cQDs|P^ ڄVLB[Vxd6}vΨ̎l]uKNRt#6x2W8yQ۳B!/[iba,h\S1={PƧw=0\8ҳ>JdF}\S~E5;wgN*8^3\vlxx^wZyS}X/SpLTX SL_9iah|d(zF4@)}`}GFxw(qMn?/Җ0Ns=1ER|>L> nͭFgkdJFk)8gX(?7޷11&kiAL*}CG-daٽU UhoJg"icXrf%o+d릨^mvHSb>aIV;uWUnqOS-r٢Nrqr %\ud6cяs ,ݒ.fQЅ"O*)OwA 6_]-jkf{e.d 2"p_c}Ro~9HmKs,gZh68ɿR5֖^I1Gt25 ߚ.^❛G 'ˆ-葈W3$/EkYp<-~tWuh8!|vF@cѳfƣKdO,:<82mT,F‡fk}>ᇶʁF;_T9 1wC-yO{mm\=)860cKJV8tY0[ƱYzJumyۺE-,n-[1/p|1.\.=uMYWn7⹳kr<gSFo}D[| f|"r{gEݸ@$;15?Q0d波p+[ׄ\ }@Xt$U6Ƀ Ib;\]02{T9or}ؿ\O&l+K%lsyYk@oAG%ӵ^{z.nkpWrqw?CN1⸳/@p zY+tx!ޔ5oUSy0=ɣA*/c^M/+{No9jʿp$HW[Ree2A[V`xXq4nX}+RyWB 9)`Az32vw씙%zBGI"/%κ["yG>W8h!yFn@ d%C:|Hh|ir*bnl AujNku2Q::G¨PqNG󁻇F .}gWX@YiW+7Җ؄ zAZPY-Y6i{Ff.4a$ Sh ˌ):hIޗtf';.SEȗ*]tjQӫe endstream endobj 1725 0 obj << /Length1 1315 /Length2 2221 /Length3 0 /Length 3056 /Filter /FlateDecode >> stream xڕT 8T{0,'eI!e#9b&D%-T}X*D$)!-r;ֽ}y>ϙ3}}XD3! aq8cjz#?+e@ 7́_l䢾- #,DO0 ?0gBl {Kh6d Cp%Ά\c#EFFuj8#) 6bCCCA&Gai0@р- ZWa΂pCA6LX4ǢAl}9@!kL^?}Ϟ+A*ap p$a\]d `3@09̝x*r90cnD\tV,%dB,.y0% o0AJ\ Y6 /i?c!ߧ.m喆 ` ɟGEo|?7Wk[$1֚OdŒb (w@t!a R@6' ̂ )@0 lφ29＃@Lt9?mZMG~ %Pc80css6 6x8Tox˓?, v=ºo'lP 0zPMbe^R:}ygk)D 攑͊ʀ:~B3?[ ﾺV:߯/l /e*VU5&wזA{qBY)%!T0O'|^Tp]q}qEe7evSx-(3[ىwv7%d*5cgŃtöӗ6\{wca O{v6nBWSVz%F:w۪J,T3go=Y3X̜0*۸d'V?qozL2!AQzo`n'D*,vK3@TqX UXW測<\dVc /oE򩅷Yxqdx5Ye#+{qR[6LuCNqki?1?q\W 9$1#cujq44,-Q - :$gS)Zu$;TnE|'khX+x,0&NK<Ͼlx 9-=T0L鿧߰}띷Ui6MʷLPzW0 ;,vG)8s8Yw2f-[PGۓ2z?2V#ǣ45V8ߚ/0as1f]`Xݦ=v,"8]A/S2AGƎP@=fͶe {*?_رJ"<vZ9Vujss:q8_/B*씗X~doZ"۝I>|=qOC=hNdW^7ⷧEvDϜCc]:` #&bn;=pnA{Je8AGW0ų m6[QӶ5ы\o]#m6|Jn=5@V;yix0p VyI0ϥEwEwu+!o{bl2n{LIemUܹ kVmT{N5 F|tP;?L>K?QF CJrFOR ly\ll .7ca9. YoV? _.3*~;c81;Etf&Ykͅ ډ3^I"mM܋1 _V=s,O4+_>эl"|[gH^ΐe;,'w5f>]j2qUH2NvH[nsݤ}W%~8F` 8gNO(1Lv~7U2+fkzT !w-8͟NHa+ (y.w8/}U=ƀ8g G|2*^eR/ $vlrrЗ+i[5vj\Ue_FmM-~,*]]!Y9vH^$tW(Cp@FB'lYѳLUdν1&ZDZXZP2UԖWMvL2>v3Hn+2L{Jdovj;װBtdp[5{λP Rx/0A -϶H,m[ĝ$AgHYA]{{ݮ0vTy93)YXTRY^%'3WIqa *2:F7j[UQa{KT9,0M uqo=?(YyAe.rE$/ˎmM-{ #N*F7Hh|J]05_m8PǜQ o^Ө>h$Zv89b|w"y46%L_d=هr/b&I /x)'bvJ漐Yu夌[X ž,-@9c_f|vUi|kAtǨ>].3+ _%fҟjw4x~ѝ~ϩeϿ>gR]/׶; < 2׸]]pGՊr E;v3?-{M aP:/$ ֊I1w]U9-*qMD}cQX$̼M9Ei=z}:>kF3W8w/`xPaI@KSloIׂ..>:98R[/*X8m\8h JuԵ5La Sb{죒3#q/ endstream endobj 1727 0 obj << /Length1 2151 /Length2 10279 /Length3 0 /Length 11504 /Filter /FlateDecode >> stream xڕxP .Nq/R$@qww(^!@ Zw-Rw(Nqmqs~gܙYg7@M$ffg8114TX?Tjju Ԛ G  *:A  9g }Y΁ f r!E%m ٙANv` K'~_t6>>Fh4V>ƿbtrgaquue:29X1\NU#d@ h _fTj%o+ l #br@(A6`px  45B 9PV`vrsb!f 6vP l4U: -vO`{'GfGͯY~r/1AHShLظ~}B5:L@`*˯>@fs+`2GeQ=@}V.VSrߦ+s]_zڿ /=l 8X@'( vL-9kZ 0b 0#Wm+? U1ey#..)#O/Jsq@wT贡 =3-bu;;yP "K7HF+Ehk W-7Fа 4o`QAU*A">hAwiЍc[A}M~١Zo5t,f!'l0gc&PCLY0 5u:w4/_-hogEm~@˷ښQגK 4߷0PlmZh?Pk@3o81Xvh!ǯ w@h$C{pC Ev(k`?'%l`_?KRAsBy'uw/үfB5ԒoQsx2A9ebTvV45uvp>Uz>@a u~T *9-ũ0wHRoK} YyWLH'ˇ>mmY㟞u> C&ϨbעK2qMMi>)MvnK81 ŠEsiﭷM6\iQ'iA.1av/9ts+&:9ݗbn |M }TڦlO=U%t`"MGw,έo1>Ǣ׉4J]vy~4IQ"ݒ8ڪ14+oÔP`!g/4 WIa0~[ ū;,ŋ<#E>M*G{q(K$pe`+#2\?Bji:)pߺUIH >x,kWMr5 @53yǝl7ƧrWR$zv&?(>-}'o8\uA삠΅ыS~vq]25S9$#x 9%Kf \üyolX^AM9"+wV"lz'\BQn)1f(z6nPA= ?(z)δZE$'g鏢*n>gQ6ǥL%;%p)7taL&|ga7 Ն˭ ^4k&bjT߸/.yrR =F<;%kvhKY޸ȵmGCԙ|i&=`]2T}C\xV{=roƒB-CI~E#$ 2)mGdl"]*̘ 3΅HbFGFaBL1 \)IMkעֱ۫xr=3rAVzZZשP>K 3݈/ȭΜw9&ӧPc$uAL&"1˻+V#JTkKxzȕ!QL#W(EE7IcɳSzFDlȳU8uSrL$nMU q@3^KY:r(N՘ M|>C|FGLq#lS3:Ȼ8$ڄtEnBfky ЇKӈY>|c,gykU૝A +gE(x̦4|ۯVF6amXT NL|x\!! ɴi5^V"~{;ı$| swܥƪxjGH[o㺂>e Q5=UXj:J~5S7.# ND}Y=E/zI\J\S;'^s/댇NmHp[q gpLl)Q-tsSJiW<:nvA<<Sot^=?F(<"-Zkz XT O 0/p1ZsG/,x%b&LYܖ؟vtDQVFjs!.w $)N1G.G(U8!6sZ̍"^dC5 ]߶ jE,[BSf G6zENHbEkR4Qy)G*]cQ2yƦ[{;MXoQGy?h;*:s#>7}` {p"E+2{G2t1sE4LaN~\Icxrhv)OL1hDg`Bk}6os}Jp:r,MbR|_@db'JK^7|y=C!W[]3*XТ`G%E 83徏D8Yc Q4O1ƃ`30c^EI t={k){sβY 59nwֹX{/j6<&!Iz#"}"u2 15Jz- 'ؾآMov_),Y$h*7Z>)|j"G/S<::,3K VTM~\5 9Cj J]('[l0R="ʜ# Q?,KrZL̺@َz^㈺],`Zots;N6#Խz#5n8ݹIy$b mb*<rOPM0U6lUXK ԋx ҈a=9;ѼGWjUu%ȷLX'G i>4GFIV=V1 AZUQoŭ!lCn k'BH5Q_x=IdyWE|8 ,,/IO9&6~a'ۍ\0Eז@T`OkmAWlYqöguR(ozYrBswWSHL^u꼙֩]٧5@u\Vº9GH;BqxkC"5 $Y8tS7!c"QrVI\GgQr)/̩rxziD(ve6EޜddWsF/zIOEy0i>5vmƐ*ʏֵe fR//Ug^"ʙB( L l0zW\fI:)&-Gv{ [RJYk[[qؾ-CnSXaYXMiVI%y:7߬V> /͞K&@drBB}P<>sNxvJ2u ЮyP1D_8ir=0iqb]-9A೼_l~9L[j ߸tLvai1֬1" v`7YL!*4=rЊfY^,`P~]F{VLc+ `oL/5q *:Ɏ9ăNj׉=MǚXP$" кq..-dD%܏ɖB ࠉB`wh ̼ZOh" +["Ir#((J/K,ǵY8{SwZ\'הe0>41o(7%K7tBΕ n{U^H)Dj cJ.kt>{N ב%ýaN3k%;dXνzٟ_xKbF4 aW!-wjuuYD$һG&WN&}E3BepOSC$q!<9Yգiz- OO /&B ):αƟDikս㗮~}4w䪂ΎN_.}Xe7xW;fOO bi *޽zC:$OUfVBZ`ԉt1 {KN{*-)m~{FfQ~AOV24C4knDm?'&r>Zw (kc_<ՕM~Ղ|.Ϧ6$CaFP;P,1A@:<CyE˅FJo c7"g61|5#vic~&w3notrq?sgVw+Dg׉}q9DZMG: ӊNAcOF z&űg-KMN0?5OsRtY6Ѯ{=[ ]5󋴾Wث~BTV;o~/tbDscaH+%{SIT3ۋdS1L7p]|#K EF|`+cMUmOYGTS5ϳ|Ug;Jx5=p(b{$`ڤ|!<5'ƦWM=jT5J⏿>llVI0.,ws)h*Fޘ<*)JLna:0"sD׉GRln`%2wC#T<OJ{>xqFiĠnk/u Iۇ.w!Rq2 5;S.9@)6ZD0]4AҫP:sVk8"#_"Q?>+W4E {'è _/nuQd݁^ZRqEI =ARs2}FT+B^M72vH_AgTQ %).f2WE=!aGxUK4_L^.GyhAze*kwǬpW58jA*v|#r2h:6GCR}3|)m*4 { T8g p ٸjTEfyYxx+hRGǬT-@CA4M[| (i⃶sM 8o uM=|%;ɪ-Ʋŕ'.$MvZ:k$ |^^)6+bQ#2G}c[SX73tqOit.ɍǖ g:Ǡ PwAqRHja Q[ˇ9tMɔZ 02F<7Z`yHH,c?g`a·l*BԁI!}Ҫʳkt䌸Kz? :¢%w9Ft hn5 ^q9h&_v鷆?9&ؠrFqKjLMjFF0qMQ>Cpmj3 +%LO|4a6InNUg2'QLQAp68La-B)o/`T|p KZ`sbnj69zF7NLeCm Wi߿RP;aSֽ|[ *Kɉ;ia#la뤭Rk9/rjN#[Q%`Ji7{mF&'j< 1:ˣG>/PL8vg<i] }^G^u~\u1Jlg&3cFnRѮXT0=Rv4@oU3+%=Ǎ_m~o_ONGnbSӑU%.(E6} Nfp|+nfIϹ*VHW옂5aU,~AcPy7$Jj0Ū Cb Fܯ>E߂ڙ9'{:==1hvk~#"Dڕ!z~mJt+zT<خ$@bP_bQa3h[ة: |3͕Xo[IL2u}=wΫ"GfKErdF8b8({ԧ.걉-ʏ _u ]ex0f!kιF+/G!.5natt/Z>_t6'lp% ݄eˣo `=]gżr;*x/[K4`M\8-XkNmA2_4OS/)efjrl i!Ɔ5|OS#gLGOQ.aIɈ&lX>NTb~SL 㳔MNb|r$I |ҷ`A] BڴDXIOg6j(&D_Ix^Xj֘t;bOdPiLwsJߠcj4)_yHSa#saa4؉ͮQXT815X9\+lI[Ap&NT(;a?^@k-ٽI/`V%%9y \E1"e&vs >Z%8>(~ۓQcnGqMd9#cXt1Q=W g0,ʫQREfM[•died(d׽.XONTHriFMI. _1H@ ΩD2}tC:Hn t-MX?@Y`wgPh̲: R U;W=;~O QWxddkIc9jMLؕbMp,CU1]k-IE$*)W ol2e@ U}m2Aw|_9PBX?ZkݬBqN$/ەtar3*B܏ J"425>EI 2и,< \fx]#'wp+ , ݴ|tq&#w׈YƇh,p<w>2Vm+ZZ^`wj)o ]zh#<fgM)AFËTL2:XT|a=jxIN|6 VHH'RI턭 4^"[ގuYd7'g+(dĻ/e1QQE/dKPKO {c]PAht3 d|OXM,nS .Ww]SZLoiE{r{B_?|5BqJP4m; BlwglLGFfYs9h7'v"Cʺ+7{X|'LlR- J[~|V̊|D>$M Yzݚi+Jn#0jxvZ{ #1pvjmLLP ՟L\U2ӄ(4u"k "FHʉNnFU׃ػVolsVU("[6Tӏ~5J:%rQjeZD$*qŘ/~O~9,1$b~`c G0 -OZx.3OE1):k)$Ia -wn"gz?:Cg oK滗c^~Vbii+|5tHدwM㖭A5dE׈zCsȴnp-$._Hh2y>>NpZM'jE5DEKy[#cQ 1$HαqqU K?۾)隬> stream xڕuTk6Hww -% 0P ]4tHtt7HZךx]~iu9em! %''/@POS32=A@P0Uy&Sz4!5/W/?O!C ]@O  Ї؀A~7?G +WTTG㯘X$<=ĸ}||.P.+AA [*.r3 п;O8m@P-;vmxx @`׿667`v4<}=9@W_@g(02du@Am6 1_r(}{)@lC@\!0gBn_[7pF"n[W V`4~#ֿt!/[79DaǘF|~)no(,o= Yd\{99/ S']'  OK/k}ö0 /}nb#&+ó$Dl,Po\Tɮ71a~S,#֨ V?*~ ][[̏G4wK&6l{Ge%Tywt z-ĀҤ7 ;yOQRd#jx#BdGkv/Sȅj{%y-fi^2vW-0l %=?JgҺH0 *"fސC ] 7*}^aو"7Kp$V(/^lv}v+ O.&%/_kW=υ vLeCTE1Lќ *W!*?xrQRi(ɀ4@@!o@ݕ0JX+A!Q=S)*M`Pބ@i3{v.1t-j#q0JEFvUQwwk-9g2u/2 3Qx'^AYȆiډ&H !P#[ƐVO/ڠNls,E:?|0@[d^ @Ϟ|+ k'%f&YuvkocxKKk&+N'33ژM|g2n:1 Eua*pHrNL}]<"G箩O3i"#-5 :5J_B|7pL mxEwQuE][ 0k|VyorQY)C Z'jHFරq~ =)A($ h7jQ$`>ޱXfo\fx'0 L)J8"c1e4-bnL4v4GJio 㢝lЖ7,45_–{^bpt㤠'kdi$h@s\G{YrL$wL`9[Kԩ)9;f{Qg;zǮLe7ҝ:ĶQ')/93.y"RBJnA|\[uT:b Oҥ>`g|`ZLA::F4jCK7 jGGh5߰f} YpБO1>k0RyoKC9 9ow;ˍO3}D>s,'d_ŹBQ cQľkUo[Tr4͇&fHa.@n9M>ϢNsh{;+1I(yu+c"61.PFR6(7>@1o9XR mSQ-ߛ5ZHy2 zXާh2 ec;_n!rꦸc*<]-a7}}nA]XA M P%L#%Rrz r#Š/;9e܋&Zax"Eq >2g"BQC4AcKDY9uwnrW99cFTdܗх`b5hCzO:aTxc`U ׾BkBmWPJv˸&ޠ\5c/,igg_y㆜nNY;T$ zW9Fuv?)(fzMq#,F~@-1u155X];\8+Š1%nd;.1WiN8eݫφUM(0~o-ab W@<[uj]PQ? l|_gr峋U5SDm+JG\vmxR>\o54y2 V 0IάGN<$)} MqgX Ãg6+OKV"'[0;9} lK x4|?Kh= 7t Of TTg=\F7fD=vK K_A7JD. % )A&(I;S.VhsJߒ 4ENc]z-/N&)A,Gjٔ<_O.%,##il"uS xQbCdX#xY"[Je)qBޭ8ByrZu\9>|ekl !i a/u-1r6m'W{'a.A8^Fp $VuTJ> >^l|j@7xDK(E~t @-HN~B+65\{D> a⧵հOXbҢr  i$XN'\4c(\O.0=̈B`ktRw{r}R kcY9Ȕ8:?|`]mY /e-y<=FyaY9\ᓈ/:;H͵-L;t3miA:/ E6-3ΦpZ%hwq[\)#Ι-dCD?qd@=v;j; ~xRBZ5бw4!C#+rߐl9AMcy r/EZ) pe5 ?пC 6O0rcE8 ; QbK fppʏvT6L$kJ74UDvtΆt5| ZAܕ'|-gZnY8٠ uj6 cA:T'NR9Gf"7 ~!Ly*K*Fݺ|o_4;0sw,Xv*RFJ.M'ڙ+/&ycg|w7t D<Mп719l8YO~3!c*iO]*U~/ 2O 1 }cZH'YqXe0YqF4!yEհVTX?͉eYM{Snki1FZ+3sjaȟ^x)^fݪ|˯c2bl9pXP][3gR\io$x|JhFu^5y ~3!{Jfzrî@!T=Ku&_{6z*zOX’89b~`V9zk;EǴ0y^b\B_LUD֙~3F;JLO{I]gMr ^>LTý !ol[ q8Oi/ УRHr~ 4}5(T"a{>UV$RObV{*oFi%ãFHV3W(T\?DXtiPĵF'`P|ALD6nW@SyRXŞTxuikB,o XI/:hbZ۸{D]]44{5G CJeSIƝݰL2<9 v/ 0ixZCN:o bjϬ1ERGŦ6AX;ػ4[Z.\ 4+$iUݶm7ZE^sAǒ21a15),D, G֔k\P t%[Zu mAMEgV g~258B>Nsu:Mm~$QCNJ[SOgRVQ鼟Lf5cv ΜqxtV; ;6]m{KG^bɫ[G{ &E fL|hVF4V+9-*Qx:10j]E`ƫIDVxyI>~숛e|#l;! }{ pB>5Lja|j̞xKty y+ܑ(ޚor\Yk&Jk#^an.evoRT#$1ҷmZ`~7i6Wt qU$Hƹ ݅ڡ${rK \(O#1HЩ[b6Ԅ ?t 4;7E-KD!^}՛AmsxIyݠ CsQULV#s_<דz 4 _ߦY,cd^Sֵ->eƋ/PrEr,C& UF/`*<1`=HF%5LvwpEsHkWzOfK43ѣ+4j 1ډ}Mq| =SP8d=:;ǙUg(E{~[BsIAՙZW] СcV+FiZp%+!&# 7\"aF4?}' j0Y]`|˨}Lv>T\e Bgrtl?{ =Wmay'%-vǾWr&HaLDooaK.!Mz~\~8@jyF`̗ǎ*a*b*OM} ҹ ۪6KyUdQ0x*fzOH+A*h*Rd9X*r&gExd_꓁-Ⱥ)b42&LGFe2h^JYtLb?a<[͉qq^W h3)b@u7;ɠ2wqhN dRFƀ+k}3).9{gvY.'̗G_,~DNFT= ye7mWvQii^t >w]A DbhaA_?sMqɘ[~'@7(쇼i6?1>zu'jG8]L0;v؉Wƚ-μ]/q>A^*[@oZқ)v$+Ag:Mr(@3mhc|__L$N_p'y ?^"u8̈CbkͶE;hU+vdRz ɾ\gӆq)P/[~9s?|.k'LY5m,C,$5qTz3H Fpж wX2B$5~tΘk[mҚc'P8916JjgA`դاD;F=#?Lc{hp~LVfHEN^(XNwx'cOo ;Į~tsz}"y}K"Fd~SusGǺwYO._.#et&T|;-z Minvd5U9}i!N9@u\.!֗goN>\‰{Eō CJ8Ѧܺz! ;۪WXyO돌)$xۜ&F5{F(EXS)"Lr݋fI UZ2$1"cbzC^v{g.#u-۰DX䃖:J}$%rJ)*fg<7xtآQ4U|K9 !O?p2^(*X'b]"W21YP.zU?AΈc_'K_eƮ n^UțΙa9_O_a_ja!qQ[т?FS%bA8O2,g ~k~o>*1Ӈz48t/<xnZ6>k"oi]Y5Z0+vet!\͵QMz 1({'3r ՆԱ$պAq 1tӱy4GCx5VS\R Rbc(ȯV"jf\qTh$iG/jGoӴ \▛|iF47yo)w㛹Ŷֻmx }XH%.-/Ӈ(_үBN*h&ZrA3eL28r55R[v_ml2& femJ8fjӊGGi 3Ēdi߭=q^T =+y߅*ţ*KH`M+=eLk.bJ;;F-W\i!G&LWNc+>,s#wAcUw#mgz? p䞠=0d@ecX HXJ)4ܫ?bIˡH"lHm?+dTh)v‰=$=`d^ج@tDEqd{d~x7yOl3>Q,ȫÖ~ߤVF32oan3\giXJ2J&*=q' F.H G6 Ch>π? Aorp!Ȗ\Rvi[Nt29PSg-:AO3@\{+?0{U?eVKD>1֐TѥÈL""l".lk6v7g1I!퀈 ]ꯔ$!-kRi9}/ ;n?Lw"T!v{2^zZ>C_L:`qfR R1C|A8mb_A?\0`w㲐"Qm^}- He͕YEm3eun_K!*n:%R\],Rzwe+(qfι1y(ڲyAۋ=xDDZgC#o_; O#a$! A aeld]1#B{jF$@+,T}D?_Zi!N7q^c`AGcA}#yZtKBuɅ|G;-/O0z43El%āLl94` % ?ie<FDYW;*1 ZɆ'YCǤT hR1VK4G[$ |'MΊXծP!k 1,hj85M1zu:S) f}*Y\*X9f-9zU-0ѐ=~ ĄZёCv&TUIѨo|nu< {o 'ݲJ7d#٤.k۴ӌkNfg_o.Ɵڎ!ۺPiIMv+N; ,q1Ձ[?=OA$Im#&-^ւwF)Ld̅&+C*~n`Tǒ:n O8)ixy#9|etƪ-pXf#\Eg_uePx4**$Q5׏QN@Tz㔵M_Cwx̍ڕ+'K-LJï'ؒ !!b?96 SuVI]timHZ *vYOCJlؿąkOQR1r>nE{!ĪFN#㟇Hl՞'[6M&~zz:ൖݎZ,կ㬠Xk7 ʡV6xr*շ K޺ٌ#Z83wcX0eHGǝZ) /X:zԵ7h $z#\̒L$fQ:aYuJ];Rm4ܔ!:SWkO4;7wƽWHoݗbp1Y$ى'e-+-x>!`)4Ũ;沖۴231_Gh0; xUw]RJ'gYy:e ,P˴oߤzd3]Va@ /Dq󵉴r+Ub aCgt^ST|qn{EdDdӒH7* `e>cNBv;>>/ tDlĮ=~8+J/Uh= 0U!pwe8k].6\ pggDg(NiF:x7R%TcnQB6TR_11U.|cxp 9 `pK +\"VEZtn9%,mxD_}_w8+I!A'meUY炄Rr#;[T)vt&Cؐcֵr~-^%= v3>kxb{MFj]1k Mb6BA8# 9,}Kbгh˝EMԁ{Lޙu/FcjotC(ptwUlmf]jrXw5юCRןDf2ggwVm*={q&t 0w7T|V'DQӸO|;cRy,*OyJ^!vSFwgp6=!O#803vB8@WTbpX vt`yp\ GOvmJ>(M]V=޼|@s©!iRtoS/AiN;/ޱF}֫S.v?&%F8POeL$ H*J78oUqO8Sξ*^Fj&zPJZJýO$Uc VoÛ ^,q I'P?qOW]Q$g?P endstream endobj 1731 0 obj << /Length1 1415 /Length2 2754 /Length3 0 /Length 3636 /Filter /FlateDecode >> stream xڕS 8WT~҂2Sm, 13fƾR=DQ)KE Y*o%*~o>}|IQKk9m< P 9{h"ۓ[VJg`^P1k7o{ۓ.;tDp9‰ӭWRaht7he_^\)[5ۄ)QgQRMilȫFHNkpG9WŘ]}>Gf92R|޹^ry.05cʳ>#[T+ ];+ '|.'S'ݱ^pwa`)1x-73ʱw~OhfpkI4:uAkF Ԕ!SKbY6 F2o%Dmݤ{À̼UbHayd{Zx~!vI@WH^ 6%N&\WU(1j<~Oyϕ^[_\!1>ѽ@'EMZ)Y [gusp2^gC7>Yb-2{ym_orKQinxlIs_|y Kn^utw}3d=mv6b6 w@ݣ 3,CbXIfGsۂc=2 '+OHfBL* ,tZAd }hHmd0]^^rmcR v;kӚ;C/B5!1QfDžGuXq1Ni,:%88vdՃl2=ekJ?ˢYX+^v괔D[Ɲ9RCj*mn`Lx4 Tܨż9\6 0"[GtR]u~~vPz29wiUr90[\ ĺdk:_ʁ' ڇ<<68e1Z#Ǧy⾟C|啜B`PkQ05mQ;xk($k9>hEHhԥX= Lg6[#J 95 剃MFR^<%k5p";sw WCRd}ݚSd_C{KEHq=-R5aC?@ ԿUyL8cDv{Q k?U:x,bڏ/=@d4/o_[T_.9FxzQL}[[!qDzT[.c"t1 CVӌtH:Otg*8&7W8UsU-3UsanZK ڽv|` 3l=)(` ߜLSE+*>>w̋r9xؒzY<@yolCNUD4qύξR}}tfuKCSͿ\!( endstream endobj 1733 0 obj << /Length1 1264 /Length2 1538 /Length3 0 /Length 2337 /Filter /FlateDecode >> stream xڕS 8T'&)ILyxL3H63IQ!JOJp)"'IJwJ.9};߷_kZ{M|,8|,b(nAhVM21Ep<&2C8Ja'|NN<0,@ lFbR6a)ƒPBXLd xGL1Iq6 xd@gEQ q\¦R,! B `i$-OH kI2BD6B(H !Bx0*#2"P>,ρ;X!FoFT xX*++HP$\οD 8:'E$R#Re;|GL,Q\FX+="d Nn|B`9\PѩFe@| E>Jzb0054x!Ŀ#OY}[H6S' Q( BK|/ A(,jb8$x,Ť$%Ӷ IRYLfiIUv!(- e*M-E`5Hڟ͌0Y @Yi0H^TJ(\UbU_mqa9#=z%?x8/~RVV\ /V%^hf@_6?6ShK nٕ.Ee5^R= δw_YVY,]ꉎ@ف%Z -yuv3=˧u)?H=f* d*x$ϴŊ%-f kT]7!rދM:S , &5Ȇl5knp,nN@Fиw{zJiިÈGꇸ ] 0ܣ|͛oj7Ϫ쩬[.~}!,lٝ-Skڼ]ZzLmNwt mr߶.SNxkvga:=zR-eaIcgWzh;KeHi{fBeylp1*نjߺ4Iyxɬ5/Ϲ[vpn nWLSX߶f,-X)#i?ZQ \G}9Zb7PX99ZՙigajDB~0Yw[;l@MRBAFlsЇ}^q2iX詔B,Z9p['>Į2(9f^HvJEAVz]txC150q{~Z0Co󅻦4/}}~cV}%kt?hK&֩/9:($R?[̷k=xE#^b)9|‹ z+|ͬ;ם>wAdiZ?B&**}jbVw %fIۗtg:h w%UOv5pdLKgUu̶JoӪ OO'hl|[iF疧WDG|Mg29J.V-XB2~ Gݭu例Q[BtW][>Ք& :NV:Z/8f.b?t7Y47v}ْ} Zo: nL'fQ}yOԻ7 m󤹵Jd?>TDV,dǺS hkjQ455첩ʸY!BfB3P-*}:4<%-ξ"VwVՈ}w:즐+uPvuy[~vziUjidVaM)-u\ l/i5DC+fߦ{pC Xi7 ~jߑɈye6F^)/ɻd$׭Ac\=~v8{LN!SKI;7CyeafحKLF H|@D.ǿ9Ì-[}Nwi O1EX[~Vgv u?j9[3ZIM3'M{`bKcQT+68}> stream xڍtT6! 0 --C %tH -"?sk}ߚyk_{;1ʃ8 +hk @(HfE9A CH(& EŔ@_D8 J IA Po"! PC->:A)]P;{$_Now3Z =FnCr9#;YNeЇ !wd64>6=n! [ j!o]``p{;@ME ;`^Phj(;#7'vr:yPvs ` k_ `[P9}P օlRor+*l_["(" ! /ED>xV!Gݺ\P~[8H6_oDTo+AWۻ@." w l=ۤ>^ebC8 q 8f6~볈t$MZCɧYT8}tsڀޖ~*Qv;$ۉo3>v}يޭΖ&N[@qѯYly4ln[oZTGoQE` [YZ=<.cOҙӳi܉&DnX"uY CAd+=Sw> {i4>E+o<[D ʽtw(Hoqnd0:,bLYi\m,zSpIn9G(^WQ=f.Xh}{֙D\фNO{,(.(pbhx%sCݞ|n'rb5yKAg_di9Λv$Ā$zWʫ́Qߢ'X'TYi%^cyRZ;Z-?0y¥~.I;6wOϯ ۿ|TA7I?|C;tFOm8}5: .06SMٴ}H ŏAIO{h#t[0hR$ UtuV, c4he `0 g0 Ӻ;'ӻE7ߩ;h]VY}n ,/Y TC>@DjEmɌEP s=LN㥙gDwSQu0XEU['Z2wRD%L1&Zsy<55,`}=05Vq" Fp&|8{L]Fq?Ք^)QKߴ;Da|xah@.ۆ)? upO<1Kk]Gj;bÎ7"q\Jq| o/VLTaߦ]7>)|j83W5#mTz;n";@Ƽ^4=ns/dv#6`tX%t/oFMjP.X@f]Ec;-ɳtw|>@[ўW}2"ANFxkqE#󚋔; x Jhf>f௓zhNuep_™sPת"i&i56JYߧUcK`s)+MIAIw(h΢lȽ%}L򲴱{.|~UlS3ozae?bi0Z1%2D"$G\՞|]P>kbQnI}ˁB|!UoST*h K؟tL).Q^+ȯ'ެ.YFG;Zܘ?~ 5{[=jig czơb<=C)?!3=Nf4>Hcl ̉s.KeO҆{2tV2B+UEcZh_e1rpW7--m$+^^׳. g21\>ƺp/5KoQyy՝K;ml[Ei&=n_H{^pr|I Sδ&ñ7q6Zw>f KOXϕ>z/ME"h>xB.kt^omx0"1K&6}ggϲnpb'<=:3QZ!jH+CyZqh5-K+Ptud&Ǩ)@|pu豤SX9Sn! L;6<{%C #kZZ]*Nߨq\g}݋zf41P,wG< Lo1 ]bILkKlhplW9 D;t^:MGpLE"D%P-vgIC<=%WKkm1d_}oN);K:iCpس+] +)X<(G9v63HNlʆl|:{2YtwV*RT~Thd.n]p_m}x4rDǂoTh1 G'+wVr03U:2} kML:uĘum|!j$Y(| yfb# =j3@k~#}OPߢ-I?[=̋֙G27%,`_Q*n1%e?Zs+s^>)]%HAL|AŖsz@#q–+xJ1l[cꧏ4grQO i› -'^΃µO?AYgҧ s a5%# HuÙ By{ 1VYNiɸȝ%;_ NBY݇5 UZ6$r5lՃ\ F"3J^kUNm3dn\+t$1'Ej;I!~rK5Nwu(h"'n4`%|.#ie Vca37ZN7d¼E8>7ZG-޾ *H ו͐T(b3xVE9/fP (,? *hVU"-ar(ZF|N_AÆ]+ q $"nA[JƉ^4L\ҟzЫml-fgWB±BW#IGUߕ@C*8O =¬/^a}Pg:4TLf|;B d#O]b!jnz2UitgK/U]e,ΉH}h$ +jEwB{bW]Mɘ3VS1\b>{S7;T걉!'llRMIVUhz1n097o8_ jywm'es5͖ |k{r1?)]c!R~B憈wk#)'XMBd6M"~ϚEYμ`k! ď(Gb\ݯK0 yՏƳn^+[KƳ}'$,dN؝+g$I-qdҕȯ8E8a=)͏zQ:wFٔpͅ8_lKLEv˖[0~Lv%] k:s<[S_ fHɳ!Q_0s,U3^τQsr(/V^B.E8%7~եό;_*Wp$bf ]',iGHG5 Y<'Y^Z V@u,vAڭ.vH5%)IMOfL[?^ð*~DYKIvT++e:Wb;ZrJwd+FY`uC yTPR]p*00Mp5,pn,OdVvIB3"3 ~g ǑMȕQI)UJd#))3 '6ԉhMֲijOйNк&;veo~TP%x܋>/<99z5=>Т} {+P#KcD3|)Wڪ&tJxNt߮2K9*;΅ !`EOx q3ڋ]Vvwp Ÿ Q)%6#Z)οz]m$l2ViNܷ~ky,НTfHG5Tj8MnOÁrms}j- Qiȍ<:*7Iwwe ̙Y-;{B]hߍ"S c1*SVC!Z}S袑>rkǐ!2lFx2)eF${{@KV%({xG-%տ=%2՟) 8T`kNOqNԓ >?H3My^`חl6:GA\~ >1?tqgcߵP׾-@PL+5i߳6Wui5W (c}]Q@Qjbv"ihHi:T檹v8GǨ?\Y;4dQ%>yU-1>.AZQ/-qU0y_#5S.g06]~Lvn DIh tO4BzK9lm}0tcHO"(:#^2Ʉuov|ȕFL0jیO){GU$ڙ'ZȳA-m0Z)J7DjwnzUQ'եӜxO<ԉV:uޝDZ}CލeoP\z @Jʔii|RqTA|`4!(߻cDzy]&ڐ+8>F2LyV֍]aovAG޳d[6L/xu WbW8^WΒl;L`u:Q\nm&yYq&/s8F LYa_dh,.lϔšHOG"1!>:w51}&M!(ͱoGNtzTx{|-W }t),6 y"` ct{i<;ma$x`8ڥeH+bF`צ8zwfLj@eu'4Q1F)scnQlΛ|0NGu[\ncMvpw$ߥG\*޹6|6 \c|KW36O:: ),vM5:Ytrji 2\˪5BJүcöSYfLVq M~Z|@cK1!v{fܨ]/ ׶nVv>vBKiy [b*WqGh"+h[6X 5BYo +γW+}z zE8uC#8ڦ["1Nyab'>6Z?få7VsC+*0D93ZlkdIJ򛵣$9'W){'ϥ00' >eDo,>tޘx7=N&H}5%"T@C3) wl-~i-%K'ƭ]knh/^̘zOıa!#" zmY#o/2i=5{2qK܊Ip;%޳4ӌ(IJoca MmX*?_9khc]$K +ajM%u8:"-^8tҶ=8s%S*G?\s|{ߎ0fMBr/i0fFZyV)ȻN{$=,*hR sŖ^KtoGd*"ʄd|$Eh4C@ SLb{a;[GrJ^qK1p(>kg#>kܔr'+Mئʍb̓r?@t endstream endobj 1737 0 obj << /Length1 1390 /Length2 3482 /Length3 0 /Length 4391 /Filter /FlateDecode >> stream xڍS 8חWJ^k̂=Nc([1 ØcldIeɞ$Vd-DY [w]wu=sss92@]2&•sMep DDĂ@#!"g@'LRC bht6M$C/"JJp8 +C$Sm7@C2 h)~T M uıBQN8R X 0\@wX 0'c *.4E b=dChz8`el ,\?d<CH$Oz R퀹1p~Pįt?H;,N$gO )]c(͗&`Hm"Ic1"ƉN)jz?R 'ԓ@>fNh TK㺑>$@yQ`I/@'n94 E(*b]`XQ'bL!Bxz` oQ;D4 t& ? C߯]a820}#K3S˩IH@ZNSY9Y@~3 CYǿb Hx2pG9SO \ \drA@‘p,}'SvS]A~;EM, #x׀ək|O]/ChX2ٱtC’qk"0T*kA`>@0(L/Z'S!O"tM;F R0o$п,3ؙGX/*q;jYoA2V9Ե,n!.9GˑotzϚh|Y5?eL}7+>QJ|u,x)=ܪ=df>q%>}RKioTDK>,U}r0vEGPP` H/,o0sYk>NH'2G#Rɇ޽SKk0rD4ؼ.Z,\Uo\˰F_ǔp!kֆŦ¦|j}nbrCl/.GcxȡPZNɩE.y1[cRmٓQHvT)TWE穀kDuP ŏ1sc߃ Œ/z nA~+5íKt> IΜ yґF>ظ~6/oGnv:l-!ӣ9{P8)C #j Yhsx-lz|EVУȩw>JM+$uמ͠B6F1$_IM*BFֻ$|+K *IyjQVWG9ђfh+:Sbg9`a3[|S|Խ3LHk*/qqX)i@2[]Bݞ37Qg,x70SXJ^5<{^e6, BݻQq6ذXr%]%(4T2y܆*o5"+(l`^fA_2˕z+ɢU쎠+ǫl֙޾ Ss̯ū--c>)oBJ*ƥh|Ȕw ?;tGvmx})iLG.u[9 I2kv* >p (:do*y|(@J'|Hɢ_'ftKP, e6XgK}"'!e~ `AhiO]QGjeٜLv3&)i͍l^+{ǒړ$Q̻[05swfƦqFx/Ɉ= ĚeξMar=}^< v 瓛Ⳑ69ov-|=^PR;ړ\=Lg`tk/bNQ7Ta h'ݻ}C+袞JԨk⛥]*E d;On\~5|}DϮh4W> S*QߥJ^nf! u}z1KZX;;Hra8Fa^5,?^l &WRW;)$In-JFVy}N}&Uٕ݋nR -Wu4'gϽ m}>r_13Vg!c}(GʤTE{XSDFg$Qk*8vY{ƾV:#Ь7G=~6I~7Qi]ܬOy`#r=;ڤy{j'wmO-f+}a4*w<{(\}:~(x]Zv-~|/H'_M&UWS٤$V {梕b$EkP4x̅zMs(RúUzbEfz}ҥL,UBQPq磋3᫮.1׎v9O|EF2/:@YU2MK>Fzsq3M}74qu L  *a9>'FU_h#(lܧ h%oQQ0+*籁U25΃âZ+N  p\O+n, ధO&PI;9c/`a/F2zdYfUJq`̜‚Pˆ5 Q{% 䰞JD\(jn(7HFV7ޖf(gVxT^7QIdvE4و۱g^5=+vǧJ|psEe>5L0vjx!cvO'd&^)t(0ƙTtMbm_C-"TvX=1+i)G1RT;h8'aATGڌ-4neǙmUL {Ú$6tK>M?!& Xva߼[B3:㸅_I *GƗ8Hr)'¢bS^e6Iz:slH"wʃ3 -JRTboW]:Ck‰S'_S#|H?&Er_5#bfR:e3ơfN-ך6B N1<ˉ|iYox~yՉoYX̊&[$ki3}#Sz7:ՐȲ#:}FӤycW] ֑oOI]H kKx4A u^s6ԙui봏 uZV}̀apkÝV)#},EbvOImS=qsJ:enY뚽|JUF4S[3+bSV@ى+5Kva%fw|-qvХP8 %34*jWK)gKD?=2e4wwrW} ͿgapM) ůO2= iBiUW]nzRq 9-Edv/Ob:?0CD+Z?Lg%,DX*46L[ƨ) [K~.pԪVđy/``.sZy}Hkh:&[nCK/^xp) CVxYfFu`5 g'j\泱Ŵ8Coc> stream xڬvP]%,xp݃!C {~SOnխs:{mYGJL+hlgh"fgLHP Y>, Ž&@g ;[ @ t029X9j.N@fNf + ; ,)@`laf lF&pc h0s;YؚM`x؛LF&Nv;JaDW8=Ml?>NhJ&N&&t}h #A[r3d$fnb`ld0Z;ͮjkl,|-hNFV#SZWbJʚ/?-2:VZ:齿=- J_#$dE ebPbl>iQz[31q71]Y3@y (PLm[k i(oL`gjM.uVCPy7,{疲 ѤSYFV1aÍ}Iʏ՝62`&}+`ؑ`j*WHoE=yW8Sm(H>tfSwb*v_V ƆJٍ!_c c6F=Ϧ~ 9q,,#L7^P>#`w;;1Q rkѕ}g:I `0^0z.+3$:/E&3f>#9OQݹI2V\RrK]l[P'<7V75=Gp,=|Ǝ+Wp$#tEj2J~GBZsv12ucPUW_Y/)l-M1#*] )Jͫ w, joH^Bu5<'DLZ㪼.Bz;M7GaQQhncNDNz @[xIt7Õ= "bـ(6֧lZOpGTly[c;Iٖ枥픻/S- g{H(UfWsC6l6Nx6s# ?Nd]lou\jҺﾢEl%p{:2;tMR1589_mPNi+AE&(;,eYW81A(yf aSM-B=1jxZ+[b%#>Ggǯjt2YWS&SEdŽfR I}Qwn _yܺΖs׊v 2Ӧ4iB<YFom'tjdL#]Y}.j_}=ÒTL8d* ]#s5 T3()W.BJdᣝ] AH>YlYr6~4FT\8J<>dWyɒޒlxׁq5+7uP8*?'d'(Kzѕ OH )(1R +: ˿xJUygaN1a*˅{!qT, F5+2HFJ#U-i" &MbnM}'OH#7_ԿN7~pQ˭+eX-[JP UW ?)(y| #W]~ Lx[g\3NK0D760ߵ!6[:{f@] Exrֿ\mRobIB (}sD hzEϮ"4}^ Kv7FSA"jT`UR?-,w6 ν:6-] AЮO[Jc T3^ء7HOţWFt~#ՓMHCSjIlCDU\WYſLf%+{i3f[ М[!oIs*Vn`$o!8Xsv+ǧ𰈒ߝpGHF~zw^LyUӺ%"Ŵ\kQ`Н! @$V~3YSFd+sIvD8tn tv_,y|[sb@<ʔʋ)pGKO*FaPfKXDq!j8!XN_qU{tY-6/l:]|ߺdBͿ*KqάJRK"3k;?'k3^kc][Fʄi ^;cs i`[l[-&<Xaiԡ/g~nY7cu6m㨜ZnjА`CE]nR-ֺ=i'B$a"lqmKoa>qmBAV^IAwMi o/IJYYTË_ q&U_0zp[o֫M4_#XKєٛ ] 72Cd}*4uUÁ9/u? Rq!RL*&iEQW8Y_sp8j7loBeX8/%jX0nlDLFPɊpꠠ4?-`' - 7!PZ>$͊QKሔP7<ȋWkA^@06t󦽠 v3 䏄Sm2l+ 'àCbHs|ؐmT_c&q |7)_vT:^!tJ*Q̜`fI> 1.+ w ٱ#89 Vt/˽͆qHUs ɗQz긇K=Z& GLꄪb $_~HgqK]hŌ" /I hP=BI=ŽZĹ:q?{m ڪy\*6AQa ȗ-]5k:!P21$LI/ن5D U)zea M!TCjP8,8 *Slt}qqJ BK賋&vnZWޟCOp n[ÿ zXc23Ǹӳ:ٟ`QrDcLagI3KԟfN}f˛Q"FC,EEu<7Zoje,}=$Z $voCSSSQ0lgF|ø O;= CҖ'Alf#h7)zᫀV0Vzԑ %{0{ = o8BhUO'6gsE\OMN>ii>AH_7BMrmRzȈ,SXN-(ד 4vv{EK45ѯUwjzx]hNuTcwAO&6dcSc3ry_4kϹuSa*ݗ !O:RzD(H"D@tdQV'F𛪴X#0$" ɾ7ӂ2?'} ;{5 kr M* ȃUɲ2?,f $#?~:q=^Itg䰯Ù Կ(}èn^Bށ~rj9-mk \Uʜy,%[W ynM./|(7p3V3,Ok0MjҶ6Hx-/zyw8Ʃ`E-غ05gKs DicFe49azL`GgXT4Xùc@s 0yg~v1l gizGLur.? ߥ.2gӯȬn>!dU 2"8Ź2%IFN7dw=é%4Q3rKVճ] g01 "x\\KU\'{)uZ_=O\Ԟu4HQ۾k=K`؟^w/[OQ 'xRݬ&U5wáQ !Qvّ3xƗKu\:X O ZYVU‰B^5T>.F-(VTmAcm'lW~%2{L u3\`9߾Ovͬz4TPF0i8aUYTDuC7gmjqw0WOr HnJ>Y = # 2BK 4v έ}Tp?#h9BkF+O-(c-gg_ @~:#L'!zCgͮ:9gfb_Ҿ#eq~3A_8 2y10SPYM?[EKNX3@ T:B"NoP9 f]iK2/cj]pw7N$X;UYYZ-Ƶa||a%cly-g DӷoxWesNb~Z5H̕y I<2K?_;62q[`v?\r/WeܔEZԦ8ŖD0Tvs~*(3~i\fkv.htM컏3:g8a# XL2?KImZr7o#~͵FjYR .2qwh?Gs&8Fmkiѷ{t1GX![`".0dpӸ&UX\.|z(A*&\Ci>W19C&M6$g*BxiVսAJ7-WV*=?\:( _dlkne.4,2*\a=b;_&$RCӱ+$p$/$E'd! s"ORٲxT@+XefC,ZZe*]ik lGxC\WG_A/O ,6Zs2mL~CeS9!9+;#~_S\WzJrN JS-.c}_Hd{[c#%"K} 'MS Os,{}Ai}o7XI[FM"za٪9? kW\Inc@V Zoh:-Fv8FL(KY{)`>M`}3 Gۆv\y0q,?" s] rZAh72^#)z/;yLpBM`(N@lo^L)y_nmݮ'qRtq4#bxly0it| o%EFrHe;0VMiBtxq ? ⿧kX?:?<„-Vm$,ddTh6љJՠh,Յ7P+(7vgoT!kQ K̲ҧs+B},G sKM "ƓXLl(sEybdKO#bYrɲΌXW Yi B.WGxoąѣ]-G}\ad3@_68J䃩%Ph(XNdTW//w#VX}TެX\+ ?`[ Qz6 P90_U" ǔ+d,:}F{/!#%!yو~+-~1#V~=6狽K־cfG0g)6_+m:eI),|gN+ jW3E75Sp fT#;ٹ=lfW\$71Ib A,Lo# l[";` K~f0V $ &x).MoGtv/]+kH!nupÔ.:ĝ$Yz;锑X[e--b_397N~/x`5tG RY[ȔthYNUK.>ZuaLzXH }F{;:$Gg0kmd|ޗ9pT=8SK#TfJ4Qcw=$-3~]f҇0|Apb%iڦOĊl$n%4Lw`PW36`Kbf5O@IWNyT{u Ga40PD^Scd(uxWDdkNTEb xb AV3,jT__v4~qiEgMig$~ׁacQ9 7Ǜ .!~*dNڭt6Km8 ! U7'ew+1 \VɆ~{N.G*D;n+6Qqä7e ]o4[)x]~:5?+5&9Vi:UQ(tam1* ROr:Ƭ]34z\ʑ}T]tYI bd pķ6d6iYtF &&L_֖ܘJS>*+HЙ=?gc](B麲 u\=/mc߷yy)\³8ORM_]yʔ y mMVz8NBݓ! 6lHTOhd9F=XN'a¹|%q5^ߌK/_ؾiLK\uv3x)ƾk!k.<_ _{̭E[ s?hdI^ztmB3,?N@TZi ;cw[Cc+S2>asɅ1q JvYt\!0Rk6)5v=[?^o[%&5ie/`Xv*~}*5h0:( \BYq!ݧ9:mƿ ֽGۥť9")1?&1T64G/{XAcA?;` $!k7]"2yl WP)^Fv܄4\Qw}qd[*c>wjz}̽2}O'\wԄi''v5"xs#pAZY$sj:ǝ[#@ߑnnt9֖y!͐ aDNY)ۉXT9|$Zŀs^AMh$9`Fp>O`3+[n˹;$q8v5+nлˈ<#r3{JWz$*! /o225sEsf@LQi؍-0qlq^Vx6](/}|ku?dݓFEQԈ| ߮x#F,7$,Gf/fF :vʔ )K8>]T ۶ n1HCT?C3|tvHRUNvם>`na]b[6rm]RIrtoIN!&w͵nlr@MrXI` bV,[W8XuT fu_sFR2iw3$sJ&.~?4 A&>w<Ɍ6fW+R'*ߔ&Dul!5L HcD-)R] M{J 6U·LF;('zG )KܸfDE.\.H?Z4gn40JTJO\"LkϪgS&ʯ&qam^ѹhB,0Y>f+p*U'k.zu ]bd/wp7"X3lL]J7H(/@ꂹ.;a'0؀AlH ӗ= YWHr=ˑo1p3k::'!"!\~?{ӐViUhFp?0K*i SBja2e#SM%=Yc<$3x֧AL#~XE:y5Ly|I*TJ}paYu,um 'EU5Jfc^b:(7 akLzDB8*axوw̫4,MlRMzZ(5 (l~9:Ѭ2K80YB;.l%+qsm(R`2h{&@uv=v&z !"G_jmɅPZq>]V m UPf.S :4ɐǮλwHxy{^4.! #chP)L2ZDY]ǽ9\֞#a^]/&dl9 K%(j(cԌ k9PífGwDq7ymJm6 o 'i/}VɠU+>'͈ϰ3K'0"4VNs+Su|"GJ _qɪչo٫t!J|ӪzBb٤ @MX>Qs8k?4-uo.!F0{7nY\Z}MhF]   xYtRޱ[!_Y\J[yRݿb*W]ȋ` ]6wwW[iُ΄@Ӿx0bZ 8J-o9c,+hP{V92W}¢IC9&t e`ԶlݑBwU&cQaM00PVdd3 ?vp<&^OZ*.gj B$ݧg˦I{6<{hQs0h#'A3]'ӕvҳ`MH(  Q7S+d$ qTcRWe= YM754:FH#lOŒc&!K75!&+t)+ H xPrьqEN14 ~P3E!OJ,Jz1gt8 Oؔut3 C%|%g͞n5k#1a[3^`bel_iI &S)Tmi@_{ R*v4Pq/sǖ͞ x9_d|7!4HiVKtZf]P=k{ @+VJ]3rK'j̦)7mJky׮Vpe)^@C߁}x$0MycSWd Ǫe2"CJ] %ώ;Z *Dc!- AO l06! %|WLݓ3 )La~[EE eA!/%\:KV午QSNW/q:1+C9eUى Tu||7@hܔVh0`Y9Nut<p~\FxʒMsP.$T.V# i"H$".J>ik@6JV4ׇyZ{Xoe 'SPx/A=eaWn(C>EK5vxEm;)0q(q7K!$u;]&Y+bh8d p&y Iqۏ_ހRVauh˕kwCks 52Dȉ5绦N pس'ѻ2{yZ7 K_Y=ϿF<vmFj")ŴEJm S<6|s<hΈ\`:ǜM('9HNװ+#Fi6xfQ\nEyYmZȩp[ָ?W fy_hӜIhe.[Ғ06RR$ g ̌1?>JsFg`,Mڏ'j%4ɻU^$UYB PCSo*}1]z2Ay,tzSÉN$F# U4sin't'ʩ+ΡAd0< N !$VVJ^QI3H&—tԺM8c|N V6`480[{=¨ Ѻ$3EƮЗC۽~~VҢH)pmY'`tĽH,ꪙQ<`F;hj_;[iSBH%g-)My4x ƍQI[_K}E(O'M7{VZ|NiAƜ0u$2CFwYھϴ"䨮(jr>f1Y3ޛm&=y2ZҐoeH ]Ww0N⟣T JIE:'ZqḔMy 5i5QRCLyC5`8SMi'16sN4i ;'UQe,>`i'>TRLFs@KSB.O) =ұP[}Ϯ҆ "Lx f'Fm^HD* 1,Π|ۀGr_?q/'sc\4^KʛnIpslo \.?ӄ%Uh ?!ᇾS_r8 umkˌ :q{`.}Aٳ;<|%BZV@6CQ&HMdU {~lwyMd#XqfB鵚UE:Un9YYE474y{DQLeƭ[Mx&; Uh1|}.9@q /WځDDffTDyP ХpC^(8``'g2G7G#OQ#f6rTj@bSkh[CYҒ+ɼĚj 4LSm?:zǭb:|?FXqTKct7,$\x _?עtbEUYMo؇>"=-~=2qRDD?x-\J@\3Prsk\ʡʲs͗|H9$)S;- eKcFqEF0islO(p$.A Wj"v5Zo| ωU&/,ڽ{08oE*Ž0rvV/v'-P# UE`#勎q^'?AZ 4d8|lu$[cT)$^.{2.}Eg]\G\[mܔ[-!>s2!1FRV12AkNgc.ϣQ@HC/݇dOܠcS1 9CTZ`P;jfk~Q[\ÑL5I=U0|Q buVGWv=uIQ}̔2/aˮr;ш_nd5ŝ( u` >$ػw[S ;:FO6 }O0*Q?S}jpHT{9\QNak (ǘ_N8VtN4@uT^Lƒ Ooz7_| Yjr@wYwHc:\M8KBS3g埇ԀMٿN|Q9%5c"E_Q]VMѕUaZSo~vH_D6,0t:`v^6gR@eI(90f#9?;砥IQ 5Kv`207Hg OzRna%ОRL%wsdq[ + b2brLE|b\9䈸뮂,rF>D rޚa?܊"EMJiNmmu6`9D|\#8sKBPO֤u48WgSk\Z3g:ERÞm(4x1[Pu vv=h1_;:#m:rHj L ݫ 7oB?*rj^ ^!&EVfqH#\50cqvv<>EtW:k~XڣV 1o[A@\OQuA'9Dl ``quOd mƢ8sɺ]f = 4_+J/Iܣ+ &&=-z_(>S;ѷW%T _(p"kD")&?v3B;^^Dά]z6fF2o jdVjng,ISQNz6|W}vR p֘]j%^"?Yz#I"E(4p `Uf(UMGJԭVCxb˜[%M|a0R>:|r_nf̪0c|&s g>9 Z<*K^Fb̄S$RݸnCvZ*\ħH|1iZ5QBPo"#^ކ=0H䈚XvK=nx4 yK^af?)Y,pˡ!,Ng (k# ^Ӽ dX<(AOt+Efq4@:Ƨµ99Q= TdFRhы.cRĎuJ+ ظ)HqoY՝IמWq(u%-zkٴkwnŶSeYN%քVi$]"ɁO*Q-8`HcG+(_r4 6a+1 VwN8YB_NM`y 1E>9+_)DM54cBr-f+q#ڧm>]"~.kT>Kjq#@k9?=DFsM ֳw_p艘kmX~(t_ О X8iߣ=[ DYWLL,qCw8lwe7Y$T8A'#p@gljb1%3@K7t f7Cz[~h$ @z`ʱH(OJ=WU =S٪@p\] EZQ`K--A_T_OFrA/j!)-ZJPR\5ڶWAL82 v&`.V=HSz5f9tXht.ndDS'Hw'"Unp|ehA"COA dȟ7yR4t qɈ(س.qMR.zecv1r)ySPG;gzަ= !rB\igTPӌ8xK 2J?TƟ{91(VkXD -,^; INu[,qcrI=YSW O<'xK~~&+jӊJA͞[YId.9@c50,%g4>[;`d _la4i'u™`F;XSEO]dMاؒ(hl*~1|2k1.L69e},ߤ#"JJ|;7!en'Q`fnKzh9\\t6Ge۳N𜌸t1tVL.MK4T3ȟIQ^ZxSPFEIxMQoњN[uPVc[9WE?ck؟IBv~m~lӅw}A?ƪ,Dޏc8"UQPGN b f\5r20nr)wXy˛@|?fng}NP$gpwlNCjkc!geP$㇗N T䎴9 q<%_{?<^L㵟ez8#8P`Z}))BjCe|A0W{gz7<yͭ}yx^'4b#T\ia Ǿӭ;5L,aϢUL'LcKWudfr(jINqVVCɄ[9<Q]10.tTޡt]wiJ}ݞX*:)(3d {,\pf3kDlEa)̻7U9UKxӊ0x#7 9T羷`)hS6_ɠJ$ڝz QՎfkm#2ZfNq^)~BzUfXT3ظ\kuF)ܹMͯa\|GA>UP kTFnq] 8kYdKSQjH5(Zȕe\y:bisDs|A40#I֭\+r܄^rSDJS})OG0P}0-,saXW-@k -^1RGoF]M-N -n9;|Xqr%aj={> I wRekRvX@W~8D Rza╵"j8t9> a0#(UZ"o^BJ D%Hî3"q:֮"7ŷJ~{`{0$7ul#۟uQvo=@ [ \SŴOOFDc漳_^}N66G!`Wn{ Q5~$&qjҔ__(bꩳ?|E񜤦`5٘z?JJneEjC&X B^~CV>b T~?z ~e_MDkaL@T,w0a`6: "`jUdo( Ȝ8 gIZ-u~oZ5\Y/p0(+j`Yh!DM jdԮ!W=)򯏿T*ƒZ%8I}u XcaZ'ΞϷEv2Ud،a.VKpSW?繺G;:q)qTS/Ĺ C̙ɺ@2&;\61f +mqlbzwxO$*l7qF jv9 *-8u,Px<M!fNߧxOV4}@ŤaMR#NۥJhgp.5s'XYU#V*Ic / [fy [S&?Whb;cخw^LK՞3{8™eAz xA'":8Ҵ \^KG8/Wk"c}a! 1U|PN9+&)6Y}Vwqc4/n98颯 R8$HׅVy!5{b#Zw捱dgx _ ҬwK삔Spww -طW8޴ըܘO^ 3ˡzגECࡵzp,w憌Ie_>ɍ&ǯpɋ!aNxYr\N1壒enb"s]Q`=Ff@Ek:Á7ޓu:>oz@*֏׺l"fjҘQأ5#?QzLBKWS'Sl(:u@⳺xo 9|!]^`SZ~J6w,Im LEfĨV(Ө)ղ}Lͪǹ7fO_Y4Ǣ%*A=*jTFrϵWۡxg XsNF^80PgtȔ}t,I,fu?A^r)_<2$[YL4@MlG |!Bb;6Uɗ؁+`6 ?::4i:EM*I#-}Ic }u󌪆{DCP!P<;e-@/i+kΧ '? * )2$Yj%Gd QxL!ղ巖<4L`h9U#RS~Dc3 "@ ujw{p(kwZ4~`(zte抺YIrVFd*zKF,E?א;c @PPxRy> stream xڬtݶ&\]mTlTlsܱm۶mI%XcL<{m2by%:Ac[C#=#7 #㧉L0p41pp͉d DLDLܬlDL\\2N>,l,D,l\LdDf"csSsG++7"1@ nl`h r5qt1LlDv@[c'#sS"G3_nv#і։܁ hnji4;)98qDS8c>??p;эa$? :b:eFdcHd099ml`geף'̧/^6& sbhfw`W; h!2Dw6@Ĵ`lhKʙ~|HP<0|o'Cscs9J; ''{S 3gޟn_ >35쩿<`m9gF45ek%mLlɿNv :|.g;5& %'O'vUB!#g?MD>go%?k0Uzpf<9YYX,?d10oVnxouPcM&071,8?LbcysG#3"+?*6_!o`}_>+ediW>fˠ -*%,BJ%Rr̚ o8~_ZhJ)L{ucbg#cfOf\^iY}ǿ%׿M?`Wm%P_|ܬ'eg(zڬh@5qV"45V:{U-c_3TZ" $~>H2|dϵCBiL`4PxаNNOLQcs u՟J'eUK!x{ {Ű0=? ϒ+u}܅ T)y<<^dRo=q Ӹߤ[T4x6íIڧFwDz^+a~rO e aUݘAX3d ;IW^[dN>;837ȗϐ"CDo6\}08G]Lw{[Wh8BjΑ pZ`_ cʹ,ޝ"cw\$KD XXH$!h;"%캫ZTAS>.W܆DIiG~mj#J,N AP$ZGICP>9s ϱKXsP򽊒Sg Ϯ bܑՅ&{=7%ϼj,ߖx]rmrJY=8 %`&K|kB[Z~?ٽaZܮKz !ҟC{r}.#f1<_q@j(U|Zqqh=tPR!,: -v: 3'[xM)X9a548#ւ}sW@x铘E'rb uesgE06k͙; \iф0– $ ƒ,~b(0#Jŝ; 5}wb MMwbSk<3AhY_ .9إ(n^^yW9&ؙX[Jy~J!cb]Bla=!.CޙlugAT~_ҘUFp=N81]8u]yWd`%j4TS w+Z3PV] m%;㏀coGcՎ5g[Hg !< Dt^pipk!%"ht.N'?][J-fi\\wi0l}d'bk$6DR$ f*[0G&C`30xfd:M6?A toh)|Zď873g)YJ{a Ne%q]$zZnU%l{Ka#՗;1_g65{ֿQC?VuAc%iO \,R.GܿhgxE+~ IF?w_OGtߞ-K0Er^ CZ$@": ,"o!wT6 B|:sKZ,#uwOV>h҈(r=EDr6_W|ŤA|QiB|w ܲXeV=l+WiD56 ꋛ$*ie_UXpYi3B jn:-f+f!,[kB'rH9h bghJF.@tDeO^%Bh(a]e ,5zUGj-N0B(05Ut֒U=@+KL36!!0cM-ٻǁUdB0޾Eu"cUfֳ|]ӋwsHh^uϓe{ Aг&ldo+xTN=-H3#a']؂vuZ{>Cwpo|i8 񒸐yt*7aDi7+"jӰC<[9 ;h:rs ҝ${ssoBo=:OxWNQj*|"q;r2lIwGhqR.{SKü Ҹ)?/#fDx &q!]ߘșp.x.#9R&u8WKKADGUH沽J~,JW eyOKO)a[n31&B"fĝZtCfa}չ>6= "SM2O6UYt='I4~{Q{wS=oՄhK_Z 4_\OTxH0U(돌C)5hUɷ^\ 30 n\KȹP4]b_#ǝP9'^HF,-qD:2k3Ԏ m$3vak%9bxg4=Q:Z> XS_}wa+hoNPe"vv6ii%@({TpgD⚼w`hՎP0tlBX//Sw ;' yThNͽgR=S^EPmn9oϛ`y)=& KK;|gToB;ERRtg 2,_Cckiܾ (N5)VLS1J/iiNi\#H_" QScm+My# Ip=knƣ דr[f :BaN]aXl^.l,Bf o/4X Wr;K Y񋙜KqMGoqjhBqxğ)ұmQ?3O_x% D{cW'LZXW+!9^YIsls0'Ǖ02%aX*!NH5™W#=H@7RWی[5"Yyϋ[nCRrea!⋵{yvdH/G#95 #@w%_bqxE 2^KpAF2*=a;,`9iҰ홬`w 3qos'{75k?S _v&eD.aO}Y4%a4z9T顅*ld!Vᔤ{ޮ5K?\ӇSKJPbSA&vگ!Q) v<}/l Ƌ09?s2(#\T[ Qb5N;xxVNM NX KU1Q#ڶBiN__ &mJ?fB6 xCU YL'^r&> e_ u9R %Fz*ݮY),J8,݈95[Ygsdr*SؔIJ%K=ۏ/_ cxU5+הUjQ )ab;IX7^PԠ0+٧cIb/5)ujdϘ9cAt~f?>rs -|s$Ѥ{- L`ކ;y_;liћ77s5pcڗ U#oUckAv"cD"zK!#5HL!s&S J[% Nb^ ӱWdyKh, ,ׁJ{e45wRym}.T;-t~ `$<w#j&ĥ_ PphߠnؒO'|7uI+NC 4"lN Le+mAl٣˹ s&-pʗ֚bAԂRyw7Vʶ8M ?O_|ݗ'.oa+γq^qj0:K@G^dc._ K:8ٯM!2 MeԸ*x 369RCl0}w<=+e˹@Gp$+~gџ?jN'Gb[%&MR~t=NP|.>J&tռ0O#9_2$f^oS1un^pk+؛Ku73oăRe*`hTĖj rnGyiD`!!pRi{ExEV$IJ )O ƍ+Apo!$IH~DTǧ5'5S[1 M:da_;{_9SÁ8 LWal<;!OiUZ%#A,kx#]P wNXh5 1orλ?;k%$X|DU3Avn]t&utzkvNzR(5h~n(P<@Y-4~tsx . w e\!#03Aɹv헢jT$F3:9YWsnR: ;WX܋է|^vAP!UaB\!,\Gbo& ro5ar@.Lg$)煉aOzyi6+ \w~/ Z)v:R%|l y2yryFMP*=? Cbl]71?b1|~nz"!@i'WOd5xMRRW/:b')A:\ @ƌ:o(JŹM [MůL3 gɬ_a왯\ud^"g>!>.+S3A+|JÑAE=H1^0`@g&=&3lTėSBӣߦ:|D5KRTjk ^6KtFB/kmfXu$;\~0U;5CG{c ƳF?zfAT,,*ggcrL8bwž/װnV᡼noَ5|ɊB'b cFBſ^^JA,bԖ1q=QaE d^YY/)p`]99)Dx]  [>ʧ2"`CPVݥ;_@%֮_ЛAr;EձA/#J 7q*t##qC]m,!`H{ET{-o|BPx&"پߧICqufӳot[bx@ x {==UMnQ,a5ߜf}I!혤tjxHQJ`+imlq^Rr*1a:>y}jO2);2;Ӫx"BEc&U+z%l=ayR. 1Dy*)/.t;XT/*G[ޑ,:,'QY.V1[E{љ*:7n4~^u%4"ʽ+=Y􄂮t3<$lgi AM@*38b@@hia'=}<A…PhYTzMsHJe;-xއgt#x2n -4u/$Stm=~kg?Go?wc;EokEtmƮ.>_cb?= c{%p\0x0ur4uJ'\:5Deh׾/z#XUS0WGR;,&t(Zg3 G#}Ii~$Gc:S5֥~KGPjxAr-z'Br=rD$@uvH)ᖙ|jCqcRݏ>\!\O6ՃLU'^a͋I62=2imٳW/d#Oe lf3hB.nBfGoA}|#Bf \2?Z*眫 bcq?XB!QePk#ht~nB rn=j4`#ڧNU- y p][QnQ3KRjͩZ<.5n\ε SG}ϻ`śW>^y O)I.-[ɟg0ب<9R zJ~GF?loɘvjP3]c9i ~n02; >ê SSQ4P|M H#Y}"Ttb1.hʿ]K:~g5vq϶<.I7-H%?>ب"ǔ\>H}ŋ'YzXG2(ZmE 5=iyN33D^}J5&?B_R<]5lt)Xu1\f:)d>Qs5Xzj$f7)rz3`gg!j VIMcK,d456` (uJbV??;P͗/u2LR[v5w2-aٲ%̸Ɲlȟ2x`_aJ׺)<@#ݿWA)Y8z&#b;b!F(i֝UE u }1b(/)dI_5𞟫ƐWrƫ+Wd Ҡaq.h7Mѯ͕"PENOvԏJ0MuO&  aui.a#wXՑn]csԮ6\vݭO'u*67}S} VpTrc*Ug&0b~)ģQ ֊ϨQc>KKK/#?pV|6ӥW>3v4t.3;e36c{\D/q;gDDZ]ҿ |ٵgjSU MO-ByHF5S+BS\LJa+4zsv'IgQMgrL#&tm%&лAL;xہ=yx]%,l.0Juah]P gⲃ;xوEP9r;7 R)c\/9ݥs ̋dEl t&+eeGBfAT;'=Jc;,$R-ݙZYHx|F0c[%bޛ(}Az?`yv]Od.)Onpq]aM#O4FW-5 >ླ0\KLc 2QXZ3;DfJ=-1a@} '7RpXeW -#^߼ZȢ;{u5M&xq(ײSz`p?qurSgV)f>+O(ƽ@Cc5;{\ Z] M_`X =W7SlFYICj/W ?G2-Jt`vi`',c a2:O/֗a.VژÎ{s^ާ`및ԮdlT' )F'$"hv]CN M~{}?CBn"dy_{ѐE45)0T2i'V!)m(zohЩFޣ Ywz‡p-JޤVv8nl/z~:{Ez~o~Ҭa*byyIB:5@OdZ:B{;!Þ ?Ig07u%"5nƓ8rE%eCYdϧF-wnKe_oIXl+KK͗Kouͧ~ QlBzT}= -v6b%gΩEofTKCiȖ 9Y2Ngw:.bbqpeo$$o{WE$Sv 'Z+SEt׈7`.):rA.~H> Yƶ" U̓7 esA`8ʥ,/Qt#_&\Qy~"qwѩi/Wɵk1ޚjJZ{;X sihL XC9iZ3rT; נ/V[܇Iw%mA'~ߌcSdn Qʍqrbt,Z"Զ+ &WoWgyG)^|³ -g82,<L_XKFC:nerac"MB5TCpT}gw7xٝA=t)0US#7=8wOW=6 >D ЧҊ+QǪf_6_Gz;h{z6@^8u4dNHn< L%h l4gs l%CT:ӑdJcrW&XmZ2'# pD0p/EQu0˩YzAGy~iO&6sU (L{r-Q~!I`Jk1FU)SB]~;<{b ?I L$ܪTzZǰCUay{ m1T81IT{+~ Fp|Sj?bs1fi!ZiB5Ϡ<㵘< W_e0;x)P=qv17US$&"f-=f/̡0i9#nvym9wSNFt ~_,UBmzZŒ4oWGN∠%Jp']nH54 17[v`$fkQ}QSux;ji}{NVhGtQo+*3!kQ\|\X$Į׏N 2CK <ٿ> U(_3tSe51ɮ{8l;,xGF!Yfyn;w G'0sac2n+.fd 6I߁qewz}hMHzLNbƞ\jCP3 f}I17kCm0ƥBm\ڻ9J2NЄ. mW{BnZj`cnŊ_Q"&sfZ=趐Yg(on;EnJݟR)'N =PC~|OxGꃫ|(ROTpWUv93t:59湚PoOV T:/.Zj.2opwᝨo5N':SB>'Y ).nxm +Lbzx +bqaMs ;4jw**K|m֚IȊo1Pр9 ڒ +|vA1Яp).T/``Uwdl4q/+%<(-g T3'h9*P%O-YDDnAt)~)苓Xxh6 Ʃm&:ov0ar3~+%}+bOx8]ЍY-`dI-]"4[2s I?"T8dC+Z[:ǙŬE%)?:Ȏf*}ۈ=v!uk jn-+MO~dĆ}xj>8R7K@nH/0w/-I|b i\iC*:/ňtCoy^m C2HE@#5#%6CRVb[MdxkLQ͸$VY%CB7 ܪ(0HKP#S%6" h-X:XB6Qq5+zx|vbk!Nux0\Sk`7N/W faA+z4Ϗ 􈝼Řf&4ٸ4Ѹ3 ѝ"ĸ5K|bjSgM0/"CdIxS?6ꛏnNvaXs =$jشV:25Xn0Kf6o}KTIA@<->X =OGq٨QgW8}}Z ɧE32#PU@8xt|Rp]#vcCX`޴E fٿ U!ͬFs@8=70Bf䦚_$ߍ85FWpQȺ #Ⴂisj rPt'8dʇSr2LcM'}. KxczӝC< ;-dD#*}'X*푋@kS#ɷo Tradgd~gx#У_8n[Y!A37\/Auz~q>qw ڡ7nJ9Ά0s)oa7͂{[!뚧(};pݸrM8R|:il qѐ~Ib:sbDu !N& wUmɝP~d:h'62>5jfCb^ tu5lET:_, j7ìYnPe%gaeǬ٤W6h/ϣ7c+mfq;ݮ[6l KZ%A fKvL7:Z-&(6\ބnuҝ$:Oqe6NJgsKd%+Q-D!@ʇ5Q3눀%(+,|;Gf%krY)E,UKM7܃ =Ъz?q)wH7c(瑣3LvFҢІE} `Upq6y>L/#kvz eÙзMہuf\g"#{CwtfrA+-[^)}rWis6 f0 v$Gd'3w_\t%{ቆA˺w9:*T Z#5 '^N .2_{ o<5ZXnZYN(zȦd6n>-B}OvJ~#-[:h /poJks] Q;'Y19xSMވƢ-zɧU|޲1VlЫm00ɖo$Iq >O+V3^nW F&㊦ybE@\<"aڊ,Nu̫qQ;]I=؏m ̎z[C:/ 7zFL2rf#RaP`d0Ծ㰡iRMVsScX뚎c&z4i2 ,^t٬H˛q*4@j6ɱ{Hr[OZѢ\¨}1+.@| pB ,c5FGX[r0(6*F[OPGc,(>n,ijЗVEIp+ e/ eC\yJ=g'@ TAZleBB֝@m`I0,j4D"3:eW_ {6dɥ}8Yߠ%ްū4Vk_2zb7xKK6 k 'P @d=(]X>^{;:?PVѤfvBx +xGبpc#A<GZS|ǚM%#1Eu8u\ssoK=Kn&˝wg ʃ SӤgzQ=<Վ멋}T<,apU Si"oFtrg UT/3jfl rR:KgsSYT |0血D'?.o4s 0wYѽk!a;ũ\Q-8;]"GSj\K^] lliAz\je55) `bu {Cdcmdn "w\( v8Lr邡S@%&πI{ | M& igs^ H(Y&*|Γtm#zO6  M'Za907ț[#Wb*A ޙn,)DK 87\c~-+4 NTfYiiEq쬎U2~=EJ6]@&MT<} =c)w7wy[W.#%EJo!7w #|603 3dKmsגR}mi׆ ò¼uw.h8$0oVk wP/݌Y)yW"WWěC 7$3Ca6cӯ\k<o-&#D˺i߂ ²^R$j5Lci8_`2%O%"e|`"^ZXjv镉 a0c }0+Y#qŠDtMw~$zV!12au#+CД#du |a/ȏ"xU.?hD!c=мx %τ̯BS]\|[$lx>[.)Aɕk=0hXV4=PVvpǖY]ߑԑcH8YLu֝y'8rmv4m_=\jx? GAiA4 k稱pNR!e} zA1{ɚ'B6*s?`~ԦIgP.anAyH˭Y)Eiۇi_ 5|=Dk 8 #GnUDpTZ#GͺӲA8t[_A熉2x?oR츚0bIӚECKxjJ#FE-έWdRwMx/ksmZE霷PPvcw˜9K8C˳(k!K4҂1)~49[ ke %< GEOX,.k%'Vm?@ u$$|s6h UI1CIi`u!ӕ׍NZz@n&T}S_~}2=3/;dCy~xN-`ko6H*7 D|RM^49^OȊyz> p'+,lC<.LD2 H\ "}v]UBkbz©"OTS_ӥF=!J4{\T Vl&-TuFEOϘEpQpJGڑ#\FʣFi5֨.-E\I@CТI(iaz2Eȅ2rEAbqt#U[&V&þyY̼E4;\I"| ]v&$pdw1PwGe]nB- J\Ƹa \}hfpS;45`*A$tie&S'D8bߠJmRt9oiuʁdߛ Ey@=L7κk @J% dTO^Io`#hɌEԾ? Hd,FA>;O; k(DKwg eo_:b%p[f MNaoHeK2ԧf&2 4&\:Xkwši0}j0? H 2X*44 qtΌbՔߞV\:uK͢TW i׵"ppG=# {ڛ| Ʌް\št|".q ٕdS!lU"h:c[:g<oCgd{V?iSAeiatݺRJݭ,[Pie -RԵU!N.N>a8ٽY˓$nruϏ Uw$D[.~;jtQZ{.)=%aQr r8^Ѿ!xF~T[z"텀r~îʏ/>G&ѪșOp_"Pò9iSDjZb) Sh/m&V7Q} 4|x HڭBG1OVS93:<$aY,HGߍp(a-h\`4/W`v13JtjxT"O )"bPb,Ea5q@@Ж)Haܡ( iϫDm MYuaqA\Nݹ ő¸}9 85q⚪ *eVsEo[as}ije/d{B[!sW<Qrls)MﾨMynm:Jk]|WNvӶw3 LY]n:ĄtcvED &Œ j+rU8TJ)|;Oh4n7H(#fYjhuWgΑ2>싗t %܉_a"Z^ BeUzn)AY-gԋoX[ޯ0Vm`#k֩P~n@ׁKW@WZڸA,'?luo^2{gl@Oe:O/i[Ubr2Lq31o h,CfPPf\{B-R@&e:V%LImȞU/6L_Eg1 `!pjv .n|JC ᑺL(D$V Y,|j|`).ɨ<܉PeoL//( @ƸZ/(=rL6]-z'4{P|L+_Bwض:qWL6\FG% ?Ԧ 4Yf3]wޔ/3~HCB9gK Hmנ9Zh81@8]EvbQ.WيٱvFIW6q^J13 S_1"2:JfC2=Zē0p4RcҎI~LYvIP ˹|5-8'Ϣ넄-depo˜?^6~Y}/jdyuM2aJŗqPMW=335ñLgLBAg_eɁƉ._:0" 4I+<*9K-QTެ,[j@%{U-PzмVa`*vJ2dѲq*F .'& %hwO}hYM8x:kXb:-!E~ 4DY_C/X\g 0LՃwi!/s;zIVq/~Ků\ B]vMa񆉨\ kQQd Y(Mbjp)_섔)bl#$-I ߏӹ۾V8ICKlW/߬/H2@(wǡLq\SbM\7sVA6NJ:cq9 W%&T6' #{F[w#E"d(~fJKPkF8 e &%O#;5VQr@˿ݤSTeaL˓t9]=T: Ͽ}nXV[ \WShb~#\Iuj-}$X] Sr<@W]0VN[Q\6ص>.=.*&pZ6$JJԘ3 -v zS[^уMMJO<*4[Yk~;ĺv32 NBsCĮUD+ ' VnIU!J12ΝljN+¸´HiFR"0![ S`Pe}/+?6dY_|HSzL^{P7:<ϵ1ͤVœC3AOg,w-ɟ"t9G8i%T(LJQ B`f\sk~*}Y3Ч]1J_s&>?. sԱ9xLYH;CѢ#:/!7= -V]]B\ Bk)g^e}O0I&Bzwg`!Em]]O $`+>7 Fw܇ނq);10CǁZz؀~ M;'t "wQQ\SXT c ߙ`Ea(X|5SŽȷar?^Ñ[õAr@ i1JU|d.@ }zx3.'` 2 ;.ۯgbl8́{d玲0DlZ2sqa26͹+T#ʫ{@%S\s!0Na4;l48wޏD{{&; HjEI{O" 2#/l{ACY. n\H>rvVX& "*[#U9I,R24t[k8;'fsfd#@>q 7~<0ѪYЪd:ӣB3L ^nœwnwr־]7%r0Yiʰb9]~Wʳp`=,6w2jTzIʣr,6l=D;:8뻭[~67dP)<<D?*_;eI1Da;:7R +L6!|WݣXL:N;W")UZǣ{os|<7!]Ц@ Pn+^d 1JwqC8opvC9RHݣ 6էpsP4\$voi3οV 1N6YLO{ЩTB WBTy{e<@VQ8mm!iU,D:Z0?yT*_Z\Ax,|XLh$ Q|1Es@ '/YG3NijZ3gFR!M`ر]*"h?ok(^6OWAǔS/,)3fr~pQE (?-1+X&}LD4F(\-f@cbh 9z}L1@.gW9x{Pqۊ 5FLCFI_ط-Hs(gnYK}&rsX肎^} xÕNKKE=;( 2EhX *-.!\(; jD) tɴ ?[ep 8xxCݣM+S|0*K`>)e]3{ xGpcR[+jQƓf`jYBoKbѣBR``ʇ1P Y>FC; 빐MEKO"ѯqv2c@2!}0k Lq<QJ< 3sN1ݫ~fnp,K?RPh9:#X+>Me xankDGY$NkƩmx\ȱC/BŗqBq;)Le\⹀.R2~%]^^nt4'ĚJhzM ;$iV[٤9D]r}^{Cpzk\+^Aı#FP_Ўk|}ȑ=/XD孧Skٰ+r(qoUN;}mwihish" ϗ@kh1he?':IVpqۓ|RLY''Ÿ i'nn){UiG#btens_kt??Zƺ-YRWw}ع'9"CS Jy<^OfiFY=/SJ0a5r; զʘO76 S\&!3~d Btx:Xx}7q҈^C$} Y.^ԕPcHVRJ8B1 FЦ7όґ0A폽A& YQ=4" qq+4i[B =J&u ' CV2Yf~0,~VYF@do/)i͌ y$|CE.kU+ղ3Yyfi=Qsw[^˂/ur  s'Z}΄!4쀗gݠ R܍9ຒi^RKY.6Add)m㿉.qXկ|eԿĔ['v{%#85P2T Rm)xI w*wa-X8J\DIK)įOMB?# GIJ~1~1Xf,>mM^JA^aAIg/Y`hpd*nuŅ*z ^0nF: MrT/^{p%uZ ATMшstګ5Ec7 񝈪@o&}@j/zbI8!שS[j4  I7y;S9\_zy?V5_$w]#%2%X–Ż"٫-S_52vG#WZ2Nܖ:a GeBA':Z[v I?[5uo; K*9ޅ650yK)ηᘒ.F2(:0wpo:C-O%LkR}n:Za0& UD RO}Psum%܎:L2xq8dpSi`N^jd{EӚ1\i{FOkKY\i^AG/ ID\ɘ׼<3OgoW1E(qIN5+ zڟ=&bMzsVoH:S_.zAk{>=zIE{Vyj} ԡ~Q6شMI},i\KWݙo䯃>k`䳬ϕQȯ?b_{&QT  dgȓ]GyOl99%k+:%[3P҅C+DY$$2{sߪH83UA״ebԧ3<(?H׮N835V_J?*8 %fsMK@/kE7ƺa9Hv(8!/P%og[:_:d0.X4U ?9PAZo-;cM'K,Ʉȵ(ړ-D]ar*+Dp<_v,!4&]}b~L:RQ zu,İFN nU,un<03۴=?]>^ZFJ&!He{s?Yǥs_'4\Bʛail'WYڨ>-m.XAFeuAx] SKd U}Urw5_BjfsPdaaza"ck/@6ƛ)L.|EO%)lޓ`5G~uLۃ Oe 5t9?Wg\b8G5GA=Mɱxa[_MjɠST9규@{6`6XޞU"mUϜQ,ˏ ϲJEG;X&csoF߸9X\_MX9]wZ{L 6 t`׌mTpȭ .U ;lOPt=8u-q`@slFTX/ZJ`%ivY1"}qC]4(];EB!%FQh"UW{;b]-.PZ4⟥չ |~5xwb nO3QpgҎև|)ZSkIpsNj=7Bvk Zc$` GX=#Y I|jjtcq[N[sR[6tu"*I_8?? Cf[7HpB~3{$l9G,jԺj CTɕ%&Ƙ yyZ` ЃD,A:^,FB%ϓ旍X87zEJ/$*3Dut`ړք  pat#&4:%sGRcݞNR/x@?$`cQWa6/!;߉^!)ҟ xLg<[`Z_,?]'i'?S"X`"r@N%_a+-Sa ئs 4A+.tkCP{7"_0 ]36WRG\P7/IfН,Θ2 :9|m? PmmŜzq[ǼKgC>}2X \)̒Ӄ_bjs2ӧZ(UӼ ?.vv)^@JofJ7TxUq1jOjZn_a(`)@:ZS')!dGۺKRwAuaU!J5')Oc!%ws%QӭxwJ}Ct׾%(/[I8dzuG0k*^'/?.4|#Gs= I5%z c>b`S;ˮS*YHwg\yCMvZBYW!λڰ+T*GNNQ gB}Ke[0XѢ%./QrW{#ڏh׮A ΨfzS# NA'*$FԌb4SBvQUh_}:-׫0@SWk>'uV[/}pa(.?"Ez{S$RD_oAT׏3mw.;ݯW{|:,؀,&]YI$9QRX}boe[ànM2JLeb endstream endobj 1743 0 obj << /Length1 2426 /Length2 25801 /Length3 0 /Length 26821 /Filter /FlateDecode >> stream xڬPe˖&;Ppwwwppwww ww+ܽ(p~}=tOq"Nd\oMAJ/lbgugf`l饝&#lag+t$-@G3!4afq"PA 3\kd[؂&@#k 9L\-l栿0wv:?nv.rw99-A+ e/ Kșr^KPbdabtS*5$@s?c/֙Y|_|i:Z93yifaWKۚՓM\mѻWGiL? 'O'Ų7*A+M"AF4&uI! ?Va)/y bmYf0 01Gl ?@?o%%io򯇕G䀎&.{Y haW 's9:;Z2}#FzE""v^lzeppG:M#~ ;auΘF :ؿ37,o۵k/i~dB*5rE#s~۪+N!f;T&$A.Czǩ k;=_BX #G:wL nK^6͔i 0p۰-Q9IFk;rq)K.d9Pm3 !ee5*y㪕cd8ϼ^DHh̳j;d.}F / !e5s@Z4R4jbx U=2i %3! tѷU"r ~v ˖{a$c3S((DʵxD乚pZLkr\B5*>;?pDGJ]ƒ]3T.,yS~HW>- ;o6CiPt2A?nWg$l~ur*BB|5^09>X#1a)C_38K3n:dbA[Xţ*%l21A у|ߕƌ|Orb'>S~nR?{( *M0¼mz6b5֓1#>[6ؽ8KZTu[5_m]= . Su*>Zjp^ҫJ6&E^.ӅFEXĎ0" OL@ YCBpCo4uL,gC:. wLQMo0[Dg ?`ӆEṙBv~`z\!1C; νM$#u%ESvj@8eF]q\E&'|SERҽ0\[;z~P&xu P#b!}1,a% )yc-iRM6 :p?ɮhOHNffȋqX@ĝZ`[kyt{q~l_ 9AJ'^yg>|˪V;:n^}fI\[`7SYM\V,RŲCі3Vie-5P?=eg!y x T픀uT]Uk #:JYq)Mz&Jndo.ϊV칲(4D%X8<ԮX{c~#b3d8ΙT|,V}[Aq?{FlKP#!Cs#Yy!c*4Q꣟Jvu0ӿC~%vptՁ+a:ާzIs7 -jF2GDM#-{m}fˤiЇf 9=)Zjc̃6K=F{rx{,^NzI#taҵ>cȣtmrXAMiLP04ƞ'ۯ(`?BZH`zb#f0 ,Em\uNB(|aɊºXtTiM2?SS 3)p<eVڧkUd 1m"$7=S|%8\v'd`-wY!ɷ^ L'THߜlkK4bɱZW"# x t~x|\A&\.j3xDu6?'i_0 ; MnG71ʰP[ɧ6^"k Oײm0h%8Z|7b/lQ.&~=`&wBAJ`z8tYc-ȴLٟvqzC@m9p`{BLSR*h5tN `p-ԍf:ޠZ ТFȴ t R(]0,~,B%@0lgzqYPrPXBZisv(<6`o,'a>a`~Ӌj*({a_ڐʻwvJً; &Z=2q[kKL~$}R$qMDqe,TŤ}i[ݼAY gг*J7]yml .]M/q׶F&)?b$3 f~uƒ9-t8dyzz-$1$c:MDǫeʹxyek >N\`[}bNքۡ(iu/(o:JZ8XGM"i:QUF1=̆myZé8>xgsj_ѻ7&q,I4,"D\_WdK%GlDPZ3x`  OH+l": l5 %ذ)86^?! &ME bG}~)>U Q'cBXbE\\-Df :W, nw['T۩NFUchYȗ5}>>Le^=Yq.`$L}/?qHP`bL W*PE"!ywS(ѩ9"mcU~$i\ 'pcc6oM nZm!ie¼qG]*{E>%2A&0Љ(׿1#źkdU.mrd93<\:9{Hu[=`|>eәUzFJ%r|<-x7b|̀o ^LWEjC*N)xNQ+vrT:DW8*g7H8{'tIc v7jhY$l\!źd%R5Ϸk ]kh5θ|ہ. o>/"gL_!6H9Ql&{+͵ױr B"U,[3CV,9G %N5=%k];t@% .z/YwN>M:+țX+Zv<* ϤRX'BjY#yIHC3ak]Kq-q .{f]8_^ Ql'iQ 3iVy|kw.\so } 69~AD#-+!^PжȌ̓p7wq3/۟fV*Sҽ6KX"^ҍ~%_[ =QKXjzACb5,g\vY/-2 ʒ9IoG/ߘGֵmvKM`t@RgCWCH;V-.̞qHO8>wcc_BlFb ͗ V]v-&[A?[gjYBܡ@txEH_v^s&"|T]X FŖْ<`yqlhAR5tCʦ٬51uЊşSl&PƭL]֨PUʚ w b]3܂$nWP.O˂CwcUeVyK$QfMAm&M9cCȓ.@QVYM~ä|y4g;;S3"D? Dw\vAh'mF yt0\z)u,aE {\̼VBjTNb$7 ro5 , |r%~Q35LLhHlzV%cicZ1x+Cw1[ (~};_O>3ƻpD}U_lZY1 &6hP RIiW$qXD.p`@};S;jT--VV㼿1;r'o~k@Zhg{Hkuo2`y5;SϜ?<+&pPUr?.2gU5G ([qlHto7"tWczlW6( 3%=&8~B[cG w,[Ce Ȅ!_VL?%o!>e/Ci`l9RKB7-ӶֻYb/)e|SضҐ S%DŽZ|E078DyʟB3HQ|\0^հ햾U[6uN({ ЍS^8ZTw2y9WE+ ^\Y%B`I)4c[Xá>'c0NΰDgu]N oxy"clcԆ,gf˗%2rpgE#|̛]e6ɚІ)4^0ݒkc=+٧Z3 `'sq֮N%޻ŀ'n0(YzT曨$ؠa; R6g:? _8eh5AGQq}5o)jCK-oI !ռ|n\_3Tg?30T.D,X*ذF;ڨ[ -NP$]J|G0+|Dη%!q^kp_IHDF>+zSuc)hISiuEFB@lҷ)S#v,s0|-J`%{C4`rҴAhfr dY@|KVFdW%r=J y =ц'!,? HAL]B(d֦1irm,GѴw:zMcFmtO X IJMk_s,pɃF$ߡ#:>_v[-_XATG㝍Bnlpxbw=Gu{D,v첤RKFFLM`w6ISh_-{ҌC^6ܰY{AO;TVW ܖjwxZw)V {)6H(fּqqΆ*8o /^&A{ N~~8pY! gwՓ F]#ѳP2ew\#6ffVה2Gb{~+2kz0'Uó_G5OuvQFA"䖻 _vPt U \Yb5qi"H>')^ѩ@ޫN> `[,3-)jxwQq2nȏ3SgK-"Be8QY>&dm}BHB/W$OkOoj?>%nij.s 8T6Ǚ({%`iU0· VFO#O?zD5ޙ9a}\iuW޲ƓǑWSzV=dIeԺKe  :0]z3}X)Q0pQki|=^Uf;]! Kϯ ņUS@{ErhEYQH|O2}{@e^6&5Ij)(@i^p')&LQYkj'j#T떵vcPZ{5tBG9}7P0ccND-%e9nYx1;׋- D\Ε,nUJj$v=(OK@/*e*0ԑK#FQ I,_ö[|EKvN ~(ʖj IҏA }d_sw|>h3̭3W>Nj!:X߳@Jamiy&7fջ!Q`/d,7Ls=ӱێ mZ\Ɏ![tiI(h_B!ɼU.褌_(<=̼=?crWoөU5WRCfH8n, 2զjK)OTn1^nmOW*W%+F 92͖{*EڹT`nrv#Zm[{ ;4įPyr[d%>ǝp\WNN&h8m|EWwFx۞9~ %;"+cSx k):JrKRqo;d~$˲qK5SVMtQf}c6:_6hn9lYj^TԹKXJ[^]S#ʔ Sf7AJ]h bYGv(T?W4ٲȏʋ 'GjU_EI7\^ؙQ}b?dAYb3$r(SboV:fgڼzX3 ڰNçs͆PLlkB*l\mEf@"H]Р@uF]VFp׃MS`auby!k4.'Xs!@euM` &}Ӭ$#/ϫ;jP xY+ s :brM,\°VaJQ4JG6oF@0U$='T ry<f \F7"fx,^aBmk0efKwVP@Pw/OOlpll XGwь]+F+!&kSA+-\. iʩu:Kf9%K^ ^A#FޟF1(\zQ8 =̵0u1.*hY%ȧ2"$0q&"3&)6&b !uPm1ګPhVFDC]e7 gy3W 5`{n Fa:KA 'Zϒ2{4ƫCHH-O:AnSƧ47\0p7 &E-r^GKImu lgd Vo&s_F~'&Ky|5"U_PizhB~D؇g4.}҄:q \5>2hkgMɑDxDG՘Pko*NrgUTYp@Y ]`_ 1tńԯH?>t7|}5[ZPMHŊv4qJ"5.ISal5nV~]%S} ]%{U3;RݐUL2{m[1'HWVi̇_6Bmp#قj<:/ӊWD 6z0נ]7 AYEtFJNQ$v~ז|hK~E;$[d: ~V00rhN7؀$R~Fݿ)Bpes($ȊMM.LKRyn{ "~]1lJByRĎ/f}ckjE4)= gYhior̓4C nr>֎sG854޸?K`*EzyKs[2e9{d1H/c4:85S,*yMmגNO>CBb /:zr +8L\X^!qƇ Ѿ:H뙸h>rDfwi1D)/3A(LQf_X(_WaY_GB`Ʒ~d4 }SBޗ]B+vsF[E6iSpQ iW*ߪ:Ći-Y .wfV E^Z??8 Ӆ"lx5"sp$%P@5U}˒Q eT.N\v=?OP'pd撷ytB'@7cP޾_2t"n.t72XZ$;k\ۉ2%UBnd.wƴ#+h1/ҶPv?"QPx/8zy.unr&j 31Y;!b~&Yugx)][i~sXToIdUw7{V8ղH-Wd%HC=0tߍդ rO:;UfONÎ1UH+LS`šI-1rƗүS7aT`(쨟HFmz!(, I:C;] ;. j^]ӎ{NLlڃ2bּ4TF^Ꮩ 08B!r?WhpbB);S>!*  (ˋ"Fzr b;Gbmh̾$yNUρ5q@iuP,ӂ3Nle $K1sh =FEmQq93nY"j 'c3# ]zds0<# +3;Y*O3y}=PJs vGZe Љf"2\jkfwب[\AwQYz"{'|)Pw -rX]y03~ >fQn C Xߝx ;fSo hHXt%wѳ|j;N);8&l=|H0){OOƶB ~HH)/_(5zGjrsc3<4h `j~<1caRN6?}%%WNKzDRk :Y6Y+*}wܪiP۬A.YT4;º=0OGo4R+uveR|f 36A5i4UChzMV xpUaA=C@?ڪ(bh;nHfVhVz:&_,ߚ~z/_D7mGÎ,{ʚhd5k^|r$OWEl9gA- T\ޡ}pinw(k[f5H9V+z$P +-a$D"(N'LfQ rY 9&]/H#Z(x~@M[)0a9@2vX<.+\/!IHJ㶒)rX)$P|X&R1b&C[tH&={#'B70z,4Pdd<  bqڌ&c =_!ė #u(Wexl8YHKi[bM"3ȋ_0^c۹?q{=f΄; ^+J8VoV*iÚA)b˂4>})-*X-;򊆏b\ [r mlM_ WV+[,7Gf:FnTduE_1||; U$>dzp8=u#1uSP%ctĹ[lIBY(_0lNڰ+ *sߡ?e,$MG= ,lf=~n$<0<} *:**ufG@SRuGcyTƉ1~]^ajbFdg֫~~˩\TN*jBA(kkkq{=0S А:9tH,5k4x#w{ 3^x]OR =OȆ:"UZbb\"1bGU]q̿1CP:[M|Ff`Ơ3UX8 zSKdݓ(Fg`:0)?_" rΞ'>|V9n;v4R&ز oo5@S̤G|"DNT΂)u/I}i" ~X75Ss/6 xwV0aV1Tޙ`ܠ7w280GjZuQ\ljnNAje=LbT&o 7axJqP"8nƸZ38gM lףu+:v{WXBCla~.6EUo$&E_;ai @^C^|>kM|x'o=#_(i4E6VۍT#J`ĕ06eMP@dT) =O2DNs)3\y[%k`-{iA%eJr)@P+hϏ!pU9'2ɄE!&ރ;=-bL90m?7ܭdVoԫ"34DkD4)jW2[bhH#B[!q7N:"hWKlv6 lC4lƕ-&W6ӏeツ65Û!=.zũ'\';r:XK*Tժ8רnO!ʻ^W, \_W=ELG":-dW$o""rjhk1APf vua= $Io$/w?S $S!W`+{x;hW7oK}Uzd|@G~R2$9oI /]4GD?^cE2dzZM>@;J ϽaIyD[/%59r]!ʷ㟶U3zd[q8TP+5xU#*H<Ё3[-u3/ /_`ox1}bi#Nwu 'yVgzP"RwގzcI7ӊс#,H0޼ #=sJaZj[)ܭ{&)|2sݢAa yeݎe`<~g(Hc^ڎ{;aI7~\OC_u$@ XsX^g!RĢlpUF?7em|ݕkR$3ή砆6a ;gW2c5`rJ5̱,ɂ}F[Sȯ019``.Zv]Z2W D[-[|z7 oqZjiDX]6b}x̴1p Y҂(hnfP#r~pUqz<_9GOqNӯ.EdPb:ޫ.C1c "?cBsZ7žwRvH MMr5e4C^ݺh/HP.h8Ty):\z|Jf=i(D5kP1((M'?}06rARI9@LSr&NR\TD5~ 相NaɟWZ (U4Ý{r3@; h$yá!|" Ɯ e? n>\-ONoa ݘ0჻ϻ,1A^wKҘV[j6/_ 8]KZ.s0x(B-W^:ًϴ+)Iαv4 揯2J$1P#O[Jr}7Az]GG wv<b4rs L5̊ݴf0InXm34x2Pq@՘Pךz;$G W{֙BvQ'[V C˩'~AjCv?dM@ k|XW-nVeNZUGL "2h㢸i+tx[~4mզLNؔ I Ey0>ڡc&?U5<*˞qna_\-]*ey<Nu%"|7̵SG}㘛!#qK#D8J^ ([*;$oM{զ:CyXm`9?Pe: &^!4hBzAn%@;iI8du>fyR(Ġ=#kXeY̱޿"G%@G0LjIN/@; )(f׍E4h\03;9aMxLGqsTXpۻ~/@5 MJAnNPntN C;U(ʂ)W^FTK178eP-[+t!QG[}A}QQUr'}␍,&Uyķv6Pm[k=.=pSJL—rǗ}J~=v̥*W6 $7@P~vEf/N.,!<D eOlڜX#9 )WV!VҩC,v´^áǥi #Ҁ~|JU[O \3{iԞ*_x7;kZl|(#\qAg`\|Zn~ UT-z0l+vHx`|T/[!AIV4gG\|5cx2}&׎R7hs bva [wYl7H'M/9 "~Nf{ў<[ 8ע>n gO/4!-T[USAoqBH(m !Zxr#2 * 2XpJf"5sfafL@HiY}x򧛥m\x ad:H!my5@{D0,JlhmC#tTOl|XM=Sf9>6|sXgŇ` _;*8m)w#m8`|v7۔ i>j->S!D0RAPZPŋB)VR7#(+ dtAxC[TkIMl?eH$U։ yA>z׳,u*'f#g A*d(x llj@>~(̓?tA|D"u쑻0Z!)ҋ|uϕϜ#NoL$ťӮVq/6mBD3G0# `|sNC+B1 r0It>m|ds{P9I&4,lT@ ofNgb'ڲ"TnXU4[&Skpy^v2x8eC4./b>O9|[( o12+z/ TbVj^w@'QEy eT=~'N@r٩#~bM.B1Q^X[!O@kBN+%O=}:Bu*yM X{q+y^UĈNk%_6b&&I /)TװWi %Ҿl?4$B;C{pCUx3wH}$ܟ8nTATtj&hC1U_qdlMli"tNZjA-^G"g`+ceDIaK)J@26y_M _E9Ph*B[Pxpx:j&ǯ aM X>9$\y) >1gbGUi[EN=4\qq@JVXQL;^]p[%{77d.IPX;N?`sDag5/#9AGbZ`*% ʀkROG'yTlY ,yBʹh|j+xT Y,@`6qJ s-9fX'jYm̈FCH[ b 3e,Ei$?o0H@IR6"Mvt$/U1WR_2V  }^ ]hԴR |c=Bg{O "GKt";.sa&l7ŵ8%`i12'WT% jtf aΊ2t6K>Ӭ^£!BLMez.ͻ-.ft*75mAxvrL&geu(&a.GNͯB v\/H A}ّn٦KH=?3<\lMl"nqqpN;ρ*8HdzLSse^XiG3GH5%*w 1WsbaX,.v6.fT3 Eқ^SX[%N-azWQlR u6go\0i1WWs`/s) w%)Vǥq(^ʿF.~fWD0VApoyqk? E[K3p~& ҃'OM47~) Ff,-%tu eMG{4[0ɱ+F?̓TqUN1 gݬ5t;|zF)o%gv[@ KΣ@X mgYw؞E0an|\2\㈉\@1ӣ nЗbfх3ZsZ.uN; n`YB˘BD]]|p5|uI>ѰlB!{C4uP AWA@O`2`܁H(m]3;Y~ȹ̸:i״k#PD#PKI^ j!-?rŸ$ ]dgX$U g^t>^q@r0a("^X򏔭VD*r'wxEvpJIDOO[PrMj5$L#'tgz+6{PوYVln:=oPq;58ú_aj#4ijg]#ʱp}wRuM ڇxDBsECV|˚a7u2>!׏xO\~@0gx\{9>8<9DDcCܷG;/+lB^MAsJNΒp` nØHD]̄ΊR/ɟC0 :!Pg+KW1 X/ˊ9([5bI@c=Nq"UC%Z%RrdҮɒWڜǴo@rmu`-# ki+ǯ@4*+eÀFZ%*ga, -9RQ|0j,y}ڙdǓv[܁(p? j"<{ Hެ0B2c[$`WKWPr#b>71~`Dᅏ0/u9FG̔7}ZswƧüq'U=鈚ŒZf`VY7b~aw䕯ik:E9Fwҥ^2PD^?ю@WzepGJj [׹$u~#Jh@a,[<Ӣ2bnP6 :y ׅ܇\6{؍о&9'RMm4ջm_-BߛLTnbVn|.zsJj[9 -\3d0Mm$ČOU fR.`(8.OZ4>ug/*EcqJktx'5xI5}?UF #^mG/Z~#\ |ĩOD\q/Rwp ?ʔ}HZ7Eom^5B]2(;"[{Sjsm$`gAh S#.XJ!ψ}8eϧax\Cl#$paܥ˪,^ 9 =pϢ>{ͱQp!ɝeR&.BTVz36S~70fH$P?f-q7h )42̻QI KOfE+O"qcDn?< 51RZ#2W&?8{|SA$0e@.bS'}xMeOs#:؏H!p͹yQU;aMxlڒl./5GlIgN75a> b~R`spɬB]BM_ sDL佹lܟMq ٝ2 TH+!<3Dג_ Zv$EJ=#:-}Ƹc#Ju?׼kdg ^:LȚqH ^;#h(iK׏_ZjoCzv7WGQ~1HG)bp!|[g'IFqSD!Z@}PH}F?F?h|X}0nHLp0\BZr31dWNJ,³8( SJaĀ0Q Bet)638J{RLsZ.VTp>OEPA@Lx.Rt]!SEidМqMf[^?nփT17vY« yc.\u._dz׺b"󅼩hyCɣ>H},g ٍ/ܠ%@üE3.W6)(_ҜjկfnJC bi4͖(a_{{#c.7vmhg=jlNiW[_;徵$iWWG6It`Is̼gaY0bv! +"j7`Iagf|Yf# %z i,Y궪 ˊM_Sxwͪ3{ ch$_? f9X:$WaClT% dVK ؒ?ׄ0*.ʟ.Sj&(h8FIȆ3JL͋t1 o]o۬/2i,,XgܙFJEzH̟hvFeVE9vM#]9G'lBn"d o_{a4!UF^ C lPxvvIA7P3@Շ-dTe09nd,ҎT'^S}1b &2% ZѥAgUInWM'8ɚC`QÓ'wJ]2! "5;Κ/T -Q6ew`I$Pqd0<(BFԀ֗INlϬK``rqc.I x*'&z5*O[SPKUҕQ^_Qh_}c:{51obܬ$ꥭRtѓY^uUX1 /?۷>G F~ kf2t)1w`xjxT*E,KNKz"svDf~oŝg֘@OvK`€o2|FDrcYj/rBՠiʨȳ}BH9^ $[FCS%}=ȣHLYX )&bd@ ^m .-v)> stream xlcnͲ-ܶ~ڶmm6WXm[m۶m{ύs'U#GeԬH2"q{;OS&:&zFn ###LNŔnjP6u01`"N.Jcj6&N5{{7Kc ++??MM.3KS/9 *@jdci 46s6;lL,əL\fdo SS0 L2,sq'_\G1:yGabXL-`/;3{&u|8jc#ghk upu1uڛ:Ž?Zx_.!dg}1Y:[z(X÷YSKW,7vdcRR&[N&c[?_,(e ],=ڌj}to73#' d_p]L\M?Ww3zì-Xi]'V8[N3;mJ2}l9haxs?hķ{뾉0xb2(\w絘ܯf]1.\L X&%}}-Hz53FʱY)ڠi;~uNJ%Cti"Huo $,/x [ߢ<0L+JD[SaS3W7[WfDڦS%Oag`I$[wh}Н*J"2>]Qx_Od3[ލ:S^1$&c_hgk/NJdd$틀ic:u t+yYCuP>β\-BaTˑݰVc~W'`ji6?6SsyziԬT9'lLG '$wj#IB`7qP(Ѷ{~ժfyJxWU0YuRR|mʡit1>U`xqj4_E Bz T$t5q!v)ܪℎӒL\]uZAy:E5K+^iWmXQ5 ($nTEӂ/nךgaϱh &C^Xm ð7]G@yцi%yRI +P?Nn C) P*hk"ΆEfh^[ xQMn?Pd17]ٝP5A/IΪZEHkM&gcSdM'MD)e I3"A;DIy-U޴=MY:w ;^܉ΐc]:Kjz}z2jW6'_D:z_#t7*4s{qDt߯mC.^P]n3J\V5Vt~ـɠմ ͏.N+x1"ɯ!yۋAm0U.T=r{^MDIW>$[KKċ)kO~bZR_5_|gJWoٔrNc(5 3'S5,+Fal§[f,OPF }= "odyQ} ]]z|1.[Q NBXHXQAVEɺ R^a{Q$%cl&6}fT -TۓXd3DzA7{M2^n nneh*.U4E;%.MT-I؂Ö%ר\gME A[ gY7s);6ng52w=DsXL=xnJݯ 갠9Ъ!NI<zƯObO5Cmuij>\1G1}5ؾ!/@ɥU)a`ja2j˒/8yU1|t$4I'NUL/Yc@iqrݣVakrº*8q_ۿ C,2 IUgJg~a([%f%:y׸,|dI  UGFnXs7o 7_pSaL\v>\KQÏ#[yJ`Lx'q:7 LaRHq)֖VӅL;κ&1O+,(0`t5Q\1EZڢ9fs shj+UePHO܊%:]ʐL:tTZ3*'g"`(W\.K <34v8,sy(G&g?5+;"D1@R2H{ف`k/JKjPRXkJ-N(PHƠϸ7L~۟KJiu_ݽ]epnmShxZ :CT\NoRɑW;`aN4eDmM IߪTΫWӢ;rÔ[a\ƱO~bTgr~=y@#M1q66ĘI;&oeɈi`(ͱ&Z26W[ۧQ^׺A)2-lc`{́HGȉDEWs .2K Z T-qf+"uq5TFV_d̒}Cx`pe綼Xy \eW8 ӓ;lg,"RH:-? %Co=L&A<+[92.0 +Y]:Jƛ!f(Lv*a"$ϢHQQo{L/%z;QdsC'sUۙa PPtJallMk?]4 bӴD@Ϻ&@` [MI3+2jɝm Hpy{'xFXH3qľwI1ЙҊ. k|7 9+W$bzR9jĚh Ll po\b=`_cMg݋yG E>h5{$civOqu0AzWQ8t֢X%TQ\(T=BB‘"3G zy"n# ]@ O"Yz_+j!\>"[G%[1Nl $3Ɲ `KG;T5rF3* ;<WB%I)>rnѷTncjkb҂߅~XNO^3Ff;|~+6Gg }`k\~U+;x6=Y=>E[>t,Y(ƭK!DV#4g`Vy32,L7~)l^歰@Zo2 /0^^ \]SPATIs%^h~ NAJWy5d> l +2fmfSn(V@Q+f!yQ,~cǭ2uBcs$+!FRKu|&(37/y=K:,0@ɑݯ3'j>^*{j+RDVv̑,#. Ā`sj&N6_*f!Ēr] G*'>)܇G۞ y\]:zFPX2XT:Z?# Դ !OJ`(Zi!GJ-o[I`p2N//*G|varwU]XC*m;[GkyqM;.V!*~饯2fڠQK"8}᪔Lڅyq3w% }>i]s:_PFH`UқKkm'7pc%U ҹq<{r8GIG|0.!%b{_mҺwZ*4ښĕ%|bc3 Kh98\3k;ClK֖˨X(٠6I2A$FIUiG'Z2D D*eAgg2pQ Of]% x:WHvŬ1,Vi)ywBܻ15r N3J^rG>djR i W ֆCNzS˒Ñ`p"QW8X r".=#_F1 pP^]ky\Vege:ZgfjܝDՋ熖h~d|~I }{bqҪ OҨ *C͠HŻ*dwO7A@u٧>i9 PbA7b12ߛ6YC6Y3ĩHW$0Hz#0{gt)Q>sCB1;]u="\w}\ b06;C^gy*D6Yaj}ܚJ/2fCXմaz<#0xI |Er;6WI7͑1~f i ?Xei ~>KZU )ϧ!z~w@9{ELQ[EG,{ٌ<9e])22؂ZIszK{* AnY?kP6gj\3`ar>tk,0v.qSgwVzX{Fj5}T2^Qщʏ@4@Y`-݊_^OXJNlU DYC}Ck& b# 1 6!oPZHox!$x7Um6gi59:H@ZV?SmA[Z,R?%V\R^-~5pBJ !aD=6$\~Pdo 61ʙUa8ҩsaj+iY 3@l晄]Xg>uzR2pB%!Tr*XYhqg`!Gj~q^[ЮFbŇCOYxAmdw.ҦT%^R2'e7YW$7Z ~_ 5=[ kH1kfE섆:LN,bTJ2v[ |) 1;,y ۵rPqڵ~H7±P΅l< I'ޭufBݬ04ߨ+>Q¢:!Q<<:%fB!Ex㢛BxC!hljo8obW~ @H>6onH:OL-]=/k&C"ي677SF<] #_1 gIXD4{M#  K D o!.,#9ai7'Znވʭ't:%fuLL0H!A95*0nrjY:Ecav}|520^ GÐ_=<_~ݦ+ZK Vyɂt:k;(Ƞm784(0&NS/6 sDG h֔zNBRz>jјwHD*jք Mנ+E),^IZ{` @ߪMGrb%uw?wWA2]wԼ | gߖЦ~|#$jjOvk9i4&N ۆa9{\0NA}#aZ(\jj¿5ޥ U mM4v_ĕ¹V)o|Is"IQCdRA{$Uy >EHd^p~GVQd7@ GznuSo5zG"f򰦅A?=(bo^I?EyYIz/wv$ 2?p'!2 ^|HHS}0wa|2fG-*6?l3J#<§4$^9!&t->\/J|c57{OTtBG#" k*ec4&na.LJ=(Ӫ[lfoM,jO=<7־r|ڷŪ4/`o&ҒC[o8ZBU5ԟς`S! w3M\ꈛʜ­5Xk6[vLs\.f/޽2ĔO?*2ɱװz#y}t 0gAcvWj6 A[a!e)fW4"~^+AzEVrvTeޥ+$$Er,AZu|܇bDݖ>5Λ^FNҊaWɠJRAo]s5i?i,H o9 ޗ{PO (m G8_@S*\sZ*E *P-wa0Eq.JQ˸4 ||DФ{y47/ Qi={4-(.B9X_s4mwWIqzpTyrO] Hk4uvJBXx9r%L9+raF*xbsZt?rV xa= | zFK 6j? ߵ^Up5kaδy𑍴ړxu$ѱ=nKP!_k? ;R舻Hir1$2\`$/Vй5}6@.5^|+!}d]*ouT皘="2e}(sAE4c0V<9bYͯ"]lkkaU ٪գ6/T}%hHulkEPj Y'00"HWrNdX=R_7#Eq5`#7srm-io{sCLkŰjS]S't7eIj(uk0Xh `iy$aӸ׺DM>>L~&4l="HK26˚=Ek5 s2jcޕ\r ,tm>f)^1j6/%UUJr65pXѧOvĭGfn 0'm G:|Ȃt8kjJ `힨#$̉nJPODžc9L\xcSbs#)!xǓ,bR%+I~>6n>fN煲%8-GDk33 YO Bc0#\eeشc|ɩjAu=Y'{QȟUZQOAߵ:sY,N9 !'t  -x&lxS[&$DFpx\YsueF4]a\PO5fW];˴B&8eͻU&r&ÁRᯜ;g09yV$~j2r_ENBE[H1̀h.(Q^KDJvY?Eә z22h0;(+ndȜD|-W =p>r88ce mZY8Ei*ɞHRǙ%7g7爱?w7-,?<#0WIc 2bzʟ*q̰diyhhsU#v1wow:!-d Qa~5 ^T[ M3v +Z̨e|g^ QҞb0ma)w^ڃu~&ŀ]fos7'7k|b>1c[c (Vv:dѰm?$?7wwz7L 6țyl-i d&z~>HD TUJcl5PEkUVv7vB(TH pBE*_bֲK?+ o p 6'lpJ=ʶI-j ,OuV[~A׍ o-;UMwIn/ht9}rB#n*ʑCv) y[ɵru<_[О=7-q"O ߖTe1wKUi$y-҈*jMD9)H$WfW_dL=$?Ң{ iv`VĬnPsĄ*Jlݪ[JKpnWW43#ʀ~t EtN ;doMȐhʂI?q pv6`1 $ ,o$õ/sR~ }TCs{!߈\[H"¬f Xn(}Cpw(9/.ZfIWZU}+_y<3(Q9\&)bZ$ E.|ltQ:g}Աkxc0\ PT$:qpW`Rq 1`9QypWw(FҢQF􌘵g>&|~"KT1F /637n>m "q̤z ]TaEZ'—E'S++TpfWΩ|?1>bf#ow\Teh|Y@%1QN`R^ش N)9 HUQN}/1F;tٛJG=t[7v /R~<֞<)E]u)d) PxAdW6cm}R,t0pH2D駃OjIJ%tpGNh7߼W p) )&G͒کݹNy5/Bɯ T** jcn]ٖؓ/_\VB;-uEqehRd֘NqͅO$1P(Y׌))낪ݹRyt]ci[h1Ձ6kaVubQHo<(D,E0٨eviڑz9Ƕ_=_Zw Dr8ytwEAݪf@@3RSH}fKsq)3t9+h?p/>5ʺ؎vрߎ j9c<2KW=3&\U U3X߯ᗒ[ @T 6;Mct-S@jHƳ|'b fj|KfJAZps&cKeįC5xy]?82! ̽216L)I7\{:YX*a[srp>,YbV!)˿uR-|oAf0 St|5iq?T(;| eĮ'쫆īِ_+p}i؝" Kh%t07<ڨ"I3㿬{cjӇth %<.3`x$՟L0\Z+`oC%RA]# iĐ4Xc}^1mT<.Q(e,DƟ`Z'w諼wKI1aa^19!ID[ON^`]-M^'@>BkoX~! YjUFٛ+E XQh?s[ +7O>wrӖ[/bo2!r}%L^# A~gqn3 r[jiФiM(c".`HFtNd]J+Sޤ/Xsʶ$0Iʶgу+[7@0gM-KwWw4*ܿ/_r!jgPdQQ"ȏF~#G%hb'jQxJ|Ξ`Rv3>/RznK} ڬ|§yi TkX@mUw})݌E6CJf&k"p>\Ao- h=sflfÿzjlVfehZD`ORɣucbwbXȴ|pl\^vd\s:ȠH:>C\N'AhΠ-]pwLdy8Eo>e&brz}kOzQa9 QzZpYs,q{ym ™) 3#Ɓ\s9pg^'gHYܥaqIJʿyQ;=e;ϛE: fu,OGCA1f1~@ٛJ͔rjt_%CkQ]{'qo-pBj'%qqYaE/wi,Jȓ^.jjQхQ25k;X%1rZׅ 9[I{w{ٟ mvBCcw"P'zh#^l綜Xշ \P= +e#x_n@mdThġ ׌y 8}Kh] U= Jv ֭8y JQΐuxz&;jf&91,e"^:D|dxcDyĿIc%,@?ƵXDUTmwye#&EZȡPyY~Ik sM۞{yp(%߉rRwV69bɍ>1q3? !~򟿆 &&T)jHλM` JK8Zht*ݭL2|І69#4aoϙ$4[ G#tfbT3S+z4 ȫ-S6K8R0 Fc$:x5%ƣBP%#BIޅ%L!5a:á048p{:~h fS(j޻Ÿe;bitI]7`hpd|⥔;sFĮWLlұ2X{p_Q{/C g. R HѵoDjFcNX3rةwm(poENYL<1qp!vIIrJKץ${R%ΐvXЎ_OlnzC̭eNg^Љ RFI%Ch`:)׷N~A6!\Dʌ~ v!zT=߾uh_"&dE6i5t񇺒:jgVi~Ln~iuF,_蝞ѸJIS=OU?\Wh(`CDLY7 _٨̜fr-%Ao(U}Wi(y9<@$ NM<ʄΝ /DRS| 67]<*JtJ|ߖmq31tIe,e, /Iic_*yB;]1č"٢N߶~[q):2INvk(%TT؜Ϣ(09s\R,9&dFMgg6^/St4}T%Rf )U> Lo}Y=``4\-UGe zyɱ  7'(5!,U|v@ܮW&g42Ў)j=f v,؃UW9,F@WzS[Kׂ |-ruQ/5f䛺VXirߏTSd/Y|Ll$VټN'PƬĵ%>q:)U¿9(!"Uj}oz32µfl X MFp>{tK oC0eacV eIBopi~HTHjZiqǐЛG } a\NE5t ͦ4G% C<&>fr#l&Ƅ9 ͙͜^% 6( AYvp鶌(sQCY2ʼSWLQ1aB⼛}"6s9e{U6ئԤ=Yg|5FLT?WyרI-49wQu",B #>Ku鞷PVLطGud]ܕUL6ГG%-"tZ ש@w!dS~Ξ];*U& R8$Z:=4 i2,MRpp7g8(7?@F-Y8AG }CcW7H(@\ymd%%xC).V[wtQ t1H@6kP1zYfEwk`YhC6M>NQjWm~^Npm+Ѝ `H = _ 3+Yr *r'^V [ |+֬8U2lSR]W#')H+|*EJՎ*ƚKQO90r|ra!8'٦7Tnh/ojgֲxw[}{P"AYǛ87&@ VƯxU...YïA/$h3,C{CKcG`Մax?nn#B5d U0,:2v *eB˳-eݣ3G~"#E |XJ)W/2ulwsS`JnD.xę@J0Nm(%{WO>P%$H LHTzB ip"7^>Sd4I'>[bܠ<'`L*ǔ^.uwlRfܒ:~=2FmB:= һimr`~$rl(q|5zg9&+1g?Ёͻv 8fd9!QP YHsՖs sF#eJ8 2r9|m7̊b憔畻mC ]E:jY(_0e, Kbx,I/ii~Iw>lM,q?B:Ҽ "67gdò|ֶrM:B1WȼҌsوn@أJ}nzGԊј~x>@5]C s3:tɘ*MS!<7~QY'\(Rև&5a$3أbsUA؜{ĉ,69-__&~Aln F曉#Qma^A. ,; 7j茏lm)zɈ^W'So.5fg;~9 J ^^M qqbe0( 0S< Ѓvay<'=D|⠛S~|p^~fuol+{]Ja8sP><ֿNR' wbdF7.EC|cQք:-S/گP6\@ ~){+a40u]_|0I{ hW]ۥy!,܍{f#(A97Hwz'& x`'}g\LێXP;UBIII)UxjJ"bGp`iZx&>B*):d1 ʨr/"ǛAthm ԸҏIfqYdlat˔aV[~7Ԩ$Q0{ƫ(m7Vƺ"l?!&'phm硤q蠨 O;'D1 xqWK\fWa;ɡ<>()5!|*ULG%OnnzL 6ZuUH-BDψŇ_GWZM. "?B:^S`W{=1t^T7<A7eQx Y!1)YBzg#wdȨ)Qp& t9b#s ͧ TC'#bvLФmLTuesZ -rW6&0`crm^o LzjhNI۳[|m&8:.mpŸ˙7EDa}b8\(=I*o=vUCx5 \}Gb-i}C8'Vu(s>Bki3*T ԱDK3>-0*d: Y3UmWBha*?jm=J&|t&lW]u2)5+8SQC\>HK4? Tc|4S/mrkZ^ZR!M]܁ aޮ uI6>[Bޖ;oe?g,-lf:@JT_\mü15G& g~g&+pQ'n\4 Jb*m1>DqnLaT۽W5|o,KB3G7=Ѭve3,7rf?TgԬ:O9!ې#G^A 5';4.:3b{Q)Q:/xUl8w7#o9~Ι!S=֥e x9}/ByέRkF_x 5 ?|E(ߙS{kv*n5cF3"١16hf=moW,KO>ioM=[ Pfkp4`xo0ؽNq+a*;8y8:q9x?OMpFt$z&"۸vziɇ0jo_ " `J%[L͙@n} ;A.U~jz9^Ih GFŲ/{El=/@H 6@%~KM1_ [z[;#O+J9#*|;[ӰڥU*~!U8Lc#,9d{yuwdoAo7 QcL'i.2^ADn5){4lEH@XbuH*ILw_pX(~0~c9$y:й"[$w@```Tj+M7kJcBlR`V<෾8Y5e@I2~bѣNGr`Q/oT.f\m0>e죌 RӈŚА옛xHӑz<)yP^=˰f^AM UɢO&YH3Q$`sa5AVf_{tA[ SPҘ-1S1L2nPk=L\i5. |R!aDFv(},TCV4Cΐ=5},:>'xM^G5|QP/dg.^wW?6{&@Y6*}D.1d^9x 1 `s2,͓ ɋsTaUtx;Jvh xK_Q#%U ~ 6z{2Wǫ8q8i1L:L,ymăvFܠAPb1 %c۴y F|kd'3VqނlHk+Ib<s[)"PH:ʼn!~ `!kMiuEW!E{'(k)J$#c9b~k+d_|(?֏ޤ" 7`f&W=Bb 8E) ͺ`QDPם47Z%+riTY4r##WC`KzD3BViZCP-a v,Dۘ [ƴ zd>5{ۤoEg0Ml)1=&h*H] A5hI)Wzi\AO~KtUdez׼*YҕA_%- d,f6>Dytg~瞃pO褸ãB/P\󃗏El҆m N_6`(j'5㛣"6teh03.#!{=+mR~/ϕ`hW`z;CX-5-3kʞŖNb԰6 ke0Zbj\(HԃipHZ4o:ensqrt4(Z쀝PVe O9W=&XW'1JiCz9L1ٷ2ӎ_jItʕGՌI/zg^`IΨe BFdUgB`daP,8~ E &.uP9FN _RPc; "5N"l|͌uH0-B10ж.t3)[*,z$L.Q3;ޞ]l?2(mPV}kkȵ緿2]?6b|{ (u E+lU sd?ua,#uE$/7IX::=> Y͎=|C7 ׎}-DxCgd S}֦yF,|1:2M`%˅f递$*/l|0)iMR'T 7yZ˅r;PlmQZUheEi=H$ s:Z^^@qWh%M: 2 _ӰhPz*0M3]%etq&5]o(;J|8*r[V/EVއY%z'Q^- waeZCPHygH`վd>d9'׉y ٰаE\$vj^bOse}HkJ[b7~PNPGY1D1Xcfhu(ן̈;*5bKRF]} _qas {g9DӠc: *)m!CJy&]*% H>SbL]Ԗ<\RM[Rlyn6teGLau$ 9on H'e9j%L^,JdZy)";ޟ+9.4D *DE\+J_vF{8$n?>_g~i9S%Eh|nF0l1|CݸJj_PEgG%mn]Gڀ3~,!Yo6D*2NZ c lGtm2#RvypB˜kuIoȂd5y^,ZX dU7?i>H _{[&*l ^v}7Cr3WLd8O?OBΤ4̈́|VC Gii {r eJt&o'$GK3.xu:u˶;",[zR0&$ôG@D45'ub0/˷W?(GTFoVI4}jGASÎ))F]]TVBii[ͮKHmҬ`~9;ryb)u]㱄Z b^@x0EHB#@&{b\%"#A%A,2I9dm.LhQmoax+Q~T=RJIV2Eehf G_Pq Wv/>&N:%M^qRh T&qrTlg;HvOmߴYԦyQ#jQ|0% @yzڈt$tYlwk]J2˖E!P gmUN{qeBN~t|3=!pZ)đ>ȱxl53fkKFAX==kf Q(-Kc*X:ׇ;1=~V IZO Tֈ.hX GUgr.lȋ Xr^G$(Tm]:+9ڲZ>b= y)e~wE=2yer&FAtcd?ܳ\01oY 3&rT Ԣ͢%+S0zWެ9҄&~N8`7$MHڢ?1OXGKEmѴ@bW}eҝv\t.3jhO8Wm1pd(7!u*΃{hB]4t4N+^H .{2o9cߠL~9w!Gq}1c*?Ma݅; #"q$7Vb5SD敩=u bE)uѨ3ܡ@@~g EK@C;C2@ul:,`Em1GZ?"8  >W|?bp \>eU)h>B'! kNBnxT u0 T2UZt=U5VyFvrV_LB@yqDO_gKd=Ji?uan_sWI :%$1RAY/(*eD?)yCi*$:^ggI'ӶLgor󗼂LjP~$³PDr ;HP<44 Ze]pu>'Y_Qm!Ȼ#W9PсZbuځjӟ 6 \PZLQ&H5%c?\pW˽d"zx1@!XA2@4GxcVvXJs\]OrcGdj 1tDULI3gkG ,q>Se/[ڤ\[{YY_au*~⸒7BA3]f-) fl*0X w%׽p*;L#oPLx?1p 06J>tnCLTU OI^G{mG dlLE>y A:_|.:K|[g͘,LGy*L ;`}DX#ߊyM%R߬Q:|&ʰbÒm'ZW+e' ?g$.oB4<}?S \~ԸM0gV#ʝ':,s 'Mg2$냬qvdFLH4Y ~Q2"Xm~).:KH j9\Afܟ?ʯ1bk1dC$$⮿w۷Bp)肜߭]-ԯEb 9›pr~a DP@,9Vq®9f̝]* ξ JvM4~\ֽxZmAɷqsF~?Uŧ_4[<ξsH)o}{k+ qpG%io.!bĤbp;oX%RAK/jI0ۺSoS4[:o뱎Pw7{~C8k ^C0ޛ͂J?H'7Lta*.~E/KY6L^X\0ӅM6Oz4PB-1< \k<+Xl3~Dk5 8dmPMRIk T[xZ$.rY-UaeJ`<_< HiaN$cͼ0E&VsVh4_ޜS6\ Yv U3.Sp `2#dyw"a|.dJFqLCx<@z ,NXC؎VJ!AX?t䤑hs97lT7AGa9&$kk7[y;6>ʙ )֊6y;?LW Y<B;<@셣:silmL"d#Q'&UMFΒKS9:~EN GY[NE?!jV.z7 aG|=$KRYPQ'{YzGCY.Ș`q(QWo_GCV/1at=y+yA1 ̵'Y1vgnEAKdf|UcBcY]L؇ \:垤՗h .Wz!U!b|LA*8ϔy/S3b`e1WlyAp!gnz:Tmt:;Kcò9LNSH.ke}N pvq՝ $ [a|z7ĻlvxBAҾoѓlYYnE cIyd|:6}I.6{3vߨҸX>w|@?-򒧫!!&S,C{xS);gx.<)Z'^'# Йi☤8zsIRAƫ+_HZdBj*"e.h.65S:IJesCaG.amx6/mHz:U>,rR'S)UM-_ fZb ?Ά!pi&@=0X\4 /땑ao?MQ)6(U+h]C4 CjB.8Wlf!ökprį6ׇ vLbQh۹rə=Yæ^;¦"yIĉHP(.n : lr.XӨZeuWӧ Smи1'BKcN$lzYXٕkn ȵ<#=X_vI: n5 ᣊ]1&aТzݙ 0h=#XQOeL j~fOa -β»0ӣ N[?~N/p(w* }( P駥xXSNY𿫈2u\ ͸l-D&Heb50e/dywByH pS:@:ė`_+Aˑ0,.;AXhd5 p-fd)^.S;x4FnX ,E&qESc8iX"+3e*Ԃ4&φ[aPr8Q:;{t*w)t6DSMUWS2CӏlܮэkW̎S/ɮ%dRϲ~0dxXg{O`B\9[ேugu[u)KQE֭XL$#5.`VG鑻+爟_GgU V%})rZWCyN&(8yPx WUHT?UbxiN2MTsJ/ٖ:_?ʍJ)wh,l bEzTcWZ뻍Z9EPyz7ClUVOT( 5⹄ʅkc/z6:4 ;wԅ SQ Ҽ6l߈ndY H8@^E{sgStg?ngdnF49+wVO{Nݩ;r&a7US f$<%Ta:? B͑!+-/r1\pB\Y!vm]s UR1>qL SV:xSY_҈iES!6B/T+ՓI U!5Ӈ**KP6)75ȫGXv 󚵜%Z(FגN2L'iغ1rkDa;vkn@_5\Oך?SМiRp7;,Q,eFñmtjt*!"fʺח7VХ&]ߚrt%U"2 '77vj3;=Wt6IƛO~PG"l)fG3cy&Eb $"sl:08w/0֤ !mzf&~\ < ({FEt?]d;wKyg,JB TH2#*/<.2YòD]/t*T3]""[5 uYD4`(Y+‹ZYy_5׆ܔ*D&OCUE۸e'f);yjqj.a5fߓ-{%!pO }9@.ۄkjYqn|e|LkuȊW63z;C-EbU^r!tٟx[2u/bjd~ !şUU4YMNe<`vb}dNٸtEz,_4w;?Fv|-\cB+)T2ފ# CamA=@t ,4;Fs ׸JA==VI^:0Y@#kͥn~Vj2$3Z8QJF_NCa•P* +[:{7ll" #5=}^TfU14]A AAOLbh-z2&>:"$j:RCKjVoAQ.CI0X,d^p6ӗWo3zC/$lpS*3\}!&SlmHtpU㌙GSFd-JSᶘk'퉴{˧zf5&`7.9VmuMrFa_M|fazz{Ex.2"o؍ӆ?[UHtr86#8irIt{< H?+=<2Rh76S}\bz M9+뢃rmeh2;ۭuh6S0 Z]GAêҗ3Yx޳b FŋK 9%a =ZtZE uUG]w ]D? k ˬW hL,~iW,39[ wnbuLzM]vQ~} /mr $ ;~ DY[5;4$ҧTn~#I ݺ-S kF{x}e>RYUzNd5ޙEܹ|fụ 8Q?yU~NL}{OM%l!y}IGP3z!u[ fFCP5#6>xbd B,u|"[djZt!Y_S C9y8z(^h(1kh 5ʩ@3^a] rBkV?{ nRA&X" %^0a/T e2P%YδPܕj…"WЫZ"ֺ@!+!d}3_ `̗uXtb:C$?,1x PB7,>`~s2{08S'?"?qMJBtVɣaUf5KAOX&9Ե]TW'5Ri!n)6QOf\{4-fA}mNB6 @s6FjW6Wɺ+ ݥR_QØ}h $[r1 a2>E#& h0E,TǮOǎNP5EΜ3%AI؞\4gXHFo> (r]m"JJ1zX"FdQhh e> F#ڑ_UIʈO}^4Mf%A-߆#CP. ] &#L?\FOn iHLs@܆V@CNjK~%HټyӽG>RKAp7 tW.hv6S_HdכfL8'CܕwWM):AGRB4"}Bl~izǹlS'?(%m,Z6EojI!zJ) {yPy'-+ P1 Jnar~c۷a >{L0]l@[B*DCQ0~ךfɫ8Ȝht6bxsKC-O`|ϺfRoۯ6Dwgk(w>$=6۰HWw&;j&^6xlE"qK*_6>v&ƙ{y'b"CS}ܒ  kg}H)a&N`Oqx ñblJ*^3t=-ʛu̖Y=Xṯ0KzrVl)eXP"B2DzCu0xy|Bo՟C,c;T=&4sQo\ ~vLZ۞%l mgBI%82W5I} f3履FFlJ!^) @wQGn?H_t聥4-Z]E%iriLSU ۼgV@F|/Zlٹ+$kq,]N^*&Vi֍SHI3^P1bx`)LYi@ǢeĤZD`)nRR^윸ZqL@(7Ի*Pa" ;_ Tۿ*ۚiV@Yp8+OvHd]RCI$*ehy28NM4":!aBW#Yڠ d<.uBXٻ '[ؒ>JԐQ/32_ʐśAX75ZaG' iSmT+n£K2{KZXާ(nק!ZbݵdtElD]>uҞ졸3>!u'_L {=:O 8\_,>*SWt) rfԽY-ELD4\qeGWv*I Kmb9VH~B Eb3 endstream endobj 1747 0 obj << /Length1 725 /Length2 36093 /Length3 0 /Length 36672 /Filter /FlateDecode >> stream xlspf6{b9my7mcmN͉mvսukZO?V)I%A.,Yyj,<fF6JJ1'sc=HŜenP3wXY(bN@K+)AдwZ܄ݜ]\IR37X,1%eE)@ddl Pv5䁦 gsZ? = /NΌ&r3wr@AB]DRIQ !Ƥ.0I8mn.lg?# h01 ΅9@󏆴3s]mm4bv.N{3s'_ڻ;Ajl ?2.!G>@@gI2]\kl̀vI7GNdvǤ(#J&[djoY\t,w`13E2QY8Y,l,n6f]SW''s˿EL+|!?INW@OOР_d_u23ڿUsG" 4>ߒm&37G:!0X.&xU0ij3UqX] R`9u,@bQu3ay$Ek@dK_:mZpi"tEgdUP_a-S1a1U=T-G(>1r`_ioGכcxAB*zQ"6f}T7y"L r=JJ/ntL^-hwxH9Fn |O G~-7jkrg?@GOa޷~Abީ'>jIxOٴxL 7mb-lpfG0&θժV|գċ֤QėS?T21I&ߔFBw9XzhKVyu3yOɲ?200"'tN4Pߡ9"0̿d4n%d+f[L1;}#AFx,D`e.9v(f>>&"pK Gj|=LFz[zUs"#3άQM} rg؞6}cO"̻ zͣ5pI6Xr*L'`{#ݟbpɼCۂϲ2XH;K,Z 38pWhe-~(1PQ|̋3$h|cs,oMq+;-od"Ԟ=)KJTIWr{@ot{gCbH2BXme>s GPo(7z(2fPGEg[]%WqF}yH0EVrh P,{R3w|S|はx{)Ȣ!D bRcHX/,B?x$ \dYP<`_b|\ s-b/nՙVRN*gu,^{XߝYR(xS_g ZW4i Ǘ]%t_'lL00mx=S11iӣr3P3ى>"%2,\ftC$'S'O#]Cs¡(4fCUA3j3 z6v߉CۊaaCX]B"[?cUϞLB?ĶgMJQɡfT:4/ֽ~i"hRأf~ V~ C+[p02#N71(LHe6;gD+bkڨF^v=bH!]#=;ɯ Cyͽ&!^0,*"k ԧs"V{= pѭT@:~CG9hiؖ„ڔK@F*rv-MIeb 6xMkgd힡*|-c3wWx1H<^^|]Ù6$ q7㞘(%T!AO *ֆlh*KQ-[vt_l⣀ ?̙+O s02_ k^ע}Y4LfD{)mݖI6=OC"m : g-֝IqHu/6 @68 "yZH(6|5(fk1(EUzPr\ 86fmoTW5zQ*S͋Pt{Guc={,P EJ(&oB ޭ{+fh (Qz)kD?ǔV`>7򂍙PȎt= áU FQfTMB&Qeӭ+1 ߋO3jr厡X߇ oԔ[NGQf Lܶ42[ڶ. ,.v?w E*x#5kTC3S4RPN1lz@Z'˯sRS w;ՃQYÊҡ WtHQsڕ( * 3`4vdifI$<3/]a'z~>Wç~Mi6=W#޾%|}#=i8/uO RڔqXX/ koͳM;Gھf~&sRҲl1Yue®oiBֆz'Ъ[]$zJo0'T|k0X3'hmCDO=)qoo!Jl5s7q3/vta:+'G ^*C.}=mtf=`i |7\#{o]ӮRjDH#{lCAloxS=c0i{)ZHNa %98rF0Dž`cX/O >}+z.DpS$3̜RsȑP: lͥx|O ;Y"NX Q"z%+F;.fȼW(&0‚/N/Co^&g}Ӗy_q@tX&/G?F%fRfW8Mz0:Qi*׶zQRO)mpS l[q:׳R gMcdPJM{L]}QG@2Cx΍c_h#E¶Fx"vN\󿽁g=śqy_I!8кϧSo1B"]Nϴ8G]7 e{ݏ6U >K\pNǯoaj.as)RԔq =v`'>%|˪ZԌ55d&4Y 3P$I£ddd+==K! Vd9xRȳ4@#ѥ^0p8&֮!wI;ںp&2Bkmry3z*B3q\RXDhU}ۄ,HKЩak@`% ?+# I. Lk mȢ]kD O4&Wj`1q`fwH%̉u`iu88oT6VqQ~53ZقaSo:]&X!mExN-؜sl-F_TG% }V&r~q`_WCQL{ kd,Or;=B:jjGMa\st& əq4 }#nU+; N0@Sޭ,⦬$cgdJ5E4칣W&xk8 R?S@ZFwE(?%(i.0c ЃB6  9@LC"HNx~cG,؋C b"Ñy] v׻ !0̈͊c BV2Tbj)K \BV+$P,bޜ8mٿasiԛKݳ!37d*G}io?}W;dݰ#rk#~p2dwNbVHpĂeq:su qLC-zm!V} [z>qIhB͍F9W,<~5[E8ַ\$ѸD҉cIKae}pي<%GO3ap H%)-n׸J)EbG0_w;{/dWrEv"%c>Ad>V:"5vUPڥ7Q{q?Ox3C\5.v7QE5{(}&xk?U:*b(k -HH^3J YDXP(x0v$ ~d&) 'k[aOK8[M!M;ڳaCD!Y-rtS"uWjsI9I'kwp티~/8n4w[վMD~ْќyF&~?=$mUh ʗ߲cu;Zm, v @kd ⥉<q*PQ?39BKe%)G1Lk+h(\GE™/{ԋVxwsq\h2颺té "%PICAϔG b<,}7L3!1]%{}j+SVetS:c0PAB*|9ݯdtgQraUsYR9وd@" ̅iY&E$p ēKg5Ee\N1eC:-AsJsޭ_1:jwg9XrLJAǭcR;AU| ԫĒɪy3*.՘jE'|<x +R9VrI;2OBD9{Z '`?oj`l4/?Al;#٥ _ƐV[ z=wX?{X] 6oo, 1C2bͩ A 9l1^dpO}`o6Tu'Eo>EkhJMP(j{JzƐ)A 8d4-Mg_,6ڠ.TwՁߨ3p:Wc l7T5\u@d 1 tD\LF8D3E\hz 0[3]!=D BZ*Vp ,U2P([Ө{dJ(9&)칯~$yRg wuCo9[k'^^ݛ$XmDǒOE7h>z*(:\3ꬓK|%+j0nV;ucsX>ByHN/pL9Iag{corMu)F9Sxqo:0Ac X=l7m?.7iWI#mju"bB;6 $NeJ(3FAFߝq=]3S! 8wYaŁ}=dyr؅˴MPj{v)w@n<[`qVn0J3k֍ұ5O@S|?Ta:b"ش1']^/5ƺ$Y-xrMin l\<^jWIkӫ"7&C͜|3u(gukwktjf. gb.*\ * 3Q^2)a8l,48(Tʉ?(3L?˺jp'*QZRm͕XYP*|0jƮR?>zCPn2h<ޒnV; Mil8S;jtzF&=E.@ɜ7ah-B} '3B_{xș^.)T̝ +`"JHјf 2UtiCU+%]m~QEO/? `G|D-u-N$86G4>Ċ+=d˯^Cqti `S_53u8E;4.UGvQR^ g,/!2X }}B Jd $;'B~QQt:C8"yOC! e#,ur [H-Gx &^n>MbGٌ H>Z}B$0_x ܸRꉪV[tY(&r#IL*lqYЧ6o8g9|FV=cCQIՈ!7)gz~,Ay넑vd͎Dz--Al/f6JV!D !r:V+qsLdRv{wjVߛ2T&- T~:|KM(-;'x? OA; ⦽E%*(t;Ӟ,M9*/[V_ę'>- 1m3)Dյb|F 7t2cP[+6?[=/7՗V!ă^<ۄt[vu>4(dwD Di~*K~ne6cd|73G1I఼8)wH3)c+ T^|W`/,`W `BQCJ:g%tnA<ϡ{O j*:>{$oP)=ИHApЕQ5-#ƛ:*^9pQ!H144VK Pk07^cwό:*ᣠ=|Z9CfPCV n4) Ds I^D/8VHv #up'>m~ڑ8wsHC}p όܭWvXʞxD:]ݖu/"P5%{5W($W#+;_"vG ͬI,i7MuRP ^>nOTItt(z…Q')v86RyUtP-.*-tm1LG G#Ea+֌ku{Q Jr ]o&^#l>ܥ>-Rndk- Lzx.41 ||[REyeK{{DW-ПG[+s-UJSd;I@/riU|EeeS-gYtrmnr="cɜ1SN Azq p-dkďַ/-mV+=کFHHhb]Bw^..p,lx*>1_nN=kѕ8b}7O 7<$bd؁=O2¹hv.g(MFk-(Ѫ1^馸;FNfG41^w9j|Xfa}Qy #*E ]*ޫCJܥ;[ʶU$b3.nCO= ]0vJm(4 _ݖH.q}޴\pLL=R}X0~.DH /5;-ram&+nsTY$z]mx.["ѝo1nWtw=xFDm<Z@OηV |0s1s@ɗY7-F 2(t"8n Q>`p\?{\qTRxD\GdϢ/}׺!\S6V`)^)-ֳd5~ ,h5uﺎv %xF?B+`Bw!/Y$B$&.FAt65wgva8ouPEu\kn$Z}eS@_l#&OR^'o@'oYI kM ddߗϺ{dBB|S/4w$7 7+F}j]Š-9vW3䓌u{v}+A31Y_f67+1M>Oe!W#+_<8ݤ/'KޘZho tffGc}$ p}bAR5Xb ̾RL|N[zW0j7&bI@d9jHk5fe^`QbҜv ? RqsK#L;/!j?UQh˝]A |wu&k Smu{&?`w0V^(a}< !|c}/Zi^mņigšIHg鹽))TBh\37-9`z@=В ݁6#UN^`D3uh.af'G3 ,GD&ͼGs?a|edNVWuCŝyll:J|^W%NCZUT$c0rla3nPKdwXdH+E"=or㛆~0okE) |xÖa25A`E =jYx,QCti%cr ݊F]Ύ Z5@ң 5zH\M _iwK6Vώ{1J.Yp1SVӼ[A! J0gi[9DRQ^-zmS˛C(eMkՏ & Y@-O-CJ>f"[Lk$gń٢M٘+]ΰM!LD`ȫ-,SyʟGұ-ȇν98!jHeYgbHC+ƦpHdbrËX_ikbZz|}*X̷3!ͺNykbB|?&/B^ u\G%fﾱ0EeYMRiN% b8)Yrk<Pm5jUNΧUյ ! m>FKi!@JcY:ސNSʔ}L:ʋvؓvL~XX="D\J]Gp2Կ~SLaȵZ]Ҳߞ:ر {MWt]#t]^ſ8weĠ/x7#HG`:"G)6e@|ASxP~ժ̜S,ۈF0J,D" RO! 5S F{Cj0 Nyt+سt۷z]4k GY#џ ZH7Z? |`aSssӊyL;^eΘ@ke찱?s;'y"GM4NB)⹷N nAGϥJq! j91XN.:TDrW7\2~[*g[ʶ 3qϬ6WcQv9\/{J,r"tb⥷lK F\'R0lqECڧś#,HMhG;Kb: 0]"/:%X s$ד" nf2~s?.ہ{Ѣ'YDp>R5sߢi'p:j ߂UOtU"ULrD-g e ^hhC'JRq?y7xpRwQ*H汕;qN~!> ,v:w)5kRQ_s o먆I+׽V,r-0_^EPMs熼EY:sZXϨ?;)d1.P%rj)>u)FnVg>\Us-Z75C֔f6-ɖ+ 4nI<nxqD9o򔥃xrxd0< ,U[xvQ:㾼|Kq%fdz * *62kߤD^@aG70:E;y2Swx VBnؙ#5Qh91WJ(+"o1" A䮼F+:n*6 ŭx'5DK\cQOR8#L35ܼ?|Y@6Pw EP$L/$>z퍟+WbmdD,38Xᐘ@|~ ΃e|Z3p %WH FipؼmkxB^=wJȰw:VB^\]T6dPP߳Ev/&\rBpH~GCޖ|eTfs@tPk@QfP9&2fD]`"/:{Ѵw Z̬axcYz&<[SSͫІg#fk-]h»lj7XmHpP4J.gtL|9;dXr}-/k͓Oq!ϑYG,hY=+Laq-s5Hk `Yx/yd[WC+y;<޶Hz]j5LE4}+wv3JHi rZ= G^=gWj'qo hsG(?x*Cdt~cE AbXYr FYH4_r7U)ܒ3 lZu/z6؞~^_Vg1^>$t{rZn))ib}Td1ȅ`-t?tYh5ᆞ٤WYzy s5 nz_=J^at_ƚ2; O/KW /Q{kEhޟ6bQ?* 3Q^J\پ# wid_K)|"<E (^]mBo@ݧ|VqG &_6 -9Om ѩ\/5%a:C}ĸB}||j˄ $g5Ek`y= )\;uIh-#߫0iy~ީ=iГ7]v[e˹sn*=hY& 2%Hh2}owę EDunH|];\! b?2֍OZp:ٳcjj^ XMhM# {fu+$A 9^ළ#5bvp6ym)3>rJ!229< 62S`c;qY52ٟwp:7җ?ӂR -u0I(w#XʆQ ,fT_g,.JTSҎ?cgevbţj?\5kKiFo-9V>0}rHRx~KEEhzHDDYIϺ7VV`@5TsZɑnW].MyhQ%P7h)cZѻ༪e ⾵JᨯZ^J@[T&/hLNIr~ǽ2=Q5|QTd#w* a };cE\daar!Ɔ;.++j璆9hyRt"vLsq׍czהCɻA\rK81/ooE|3G'!vxSsj9["ahɀ;ԑw{='\ϣv4q/ta7]M,=U.ABP}=&lfHZɍ5*oZEl۶mi۶m۶m۶mvʪHj?Tv|uBa|\!p&.¥{0X 42ʤqS` B9g` n0x*%xGsװI?|;8$`D썠ǑBjSދth9Ѿq/a<`~& ╘EC@]%Ѐw&K.X5rs^~O(=q,@kcħwQĊӑjsi }Qr \bpGz>[pw`j.  $"B¦]VP!ԍU9iW*rnU'-c˲iwqps|s VpC8bc,y=\"3ŸJk,Sj" ޕАA+V ~zNf(|.fǦa 76t[ȔaEnqL%W7fH/]Lyi %*j0U֏ցgk }7+z;6J.k9vLA5'~]}0h9.o;L` 8v 2_U0ؤJО̉/|"y1P5(X6D^h?!vC44Ȕ!um`N$p"ԻX~{!"j|DET@G"Cڈʻح\PCU6<yT4?I3d@FG ˻,B@M'yD\*#19TXRO7dU< 4 ?ihoeHvsvL|yAMyjBF)UtɮEEI8.VБI<@ SB98;yASq 0a< Dmw=_+7k0QQ6P[de^7S0<ӻRъ3N%J/"=^,R>_7 1e\nLԐ ؜CiD(rȳ$ˍ› bQFY}X]"zJk0Wݟp(B>9r M׋\uٮy0j;cB@z:Bt#+Db@*aJTFB9[VGs?7]q(#Ixi9#͸]aaFrŧ̥ >tVg F$ c^뙧KRшUQMvbTR6bT=5OUD9Ff9xaLZF8LC^>&eSnf &u yLPyOϯ8i d2س\=~OᵀM_F-AOr2- r->fWJ=YM% : .=]c`"#8Q2{wA2d%3MV;x^GB<.3ʐ+#X:_LcHFMƂ ?UMH-_S{Y^a{KδéMn([[0S;>QlHD2>9e5;eo^ӷF击7|) Y>,&v+HAS:7IaJ6'S?!A E"1#dPF!64USh'ǵ:|%`픵}bذ)?](NoJAQ6IiIHlNywT1@T`xGъ\31spѺYW?-w&*gZkN\9f4侽-֎HZ6)XȮS~qd2a%HKM@e<Ɇ a;UGz%/l7,&9N+>U1D/]-[<ou>eT|D7(ír0]pMD ם!HfgKEw :fu9N~`4%lOoSE>X) pYI*hl_T_%[V87Rףr1@{f!'d^ƚ/l9!DgeՉgd/F6z,U]V1" #L7`IfE:  U +X18PMH v2 Gv ,WC(ɐum԰ سhɿj_mG1>qHYo}'v1k& 24F^Hc3{02RU9JoAr5gūԠ {n Xjetaeflbfő_O4uՀ/ /O_ӢqE(*掓(;PUM.ᮽ> v&,RXA\xRI ca^ʢKR(H!U4/[% IaNj Qɣ-X>y*oT`?w \Oh-=St2o]-iCOaM,lcw䒖@̷#};ZXrnmnRCߒ e)c,}^ÿ)G GƝyctaqYFs/wMw152lvG y"}H}zHBɄ(U&2#j\[BFU;d%wpb"3~ "s dcPȂ1̈!>fO7[CE80_ ZR=TZ F}U/ׂ͇(J%qX@K[ÇEH''mֶk Bdb|"[Ҕ!:,,Ů@$K^+#\ )׻;KԎb% IPTӴٛ~$6%sF8$*ZD7N:Spsqk -?eGr^ٰU,NLkkU}~Gh}T85%HK-fg6 %MOL$#%/NI*/@L OeI\FETZp,O GE4OReG[FLs{>u ?$̄Ö gum*4Q kIITz_ W]hKBvnȨ?C7KX~r#KbIAsYƪIr8ձu YJy$)ReJ| [I ξi3Zb^mz/(DF4J {ʋJ0ؒ@濤0׼2P>$q-h3 vE1L!+pDfALN8?fZZR?4X֔l]'mhVP-ꥺ p^6rFx@ZRyG/'> N ՆTRW0{V2x7‰kS*8k#Բ]5ިA'M_텭PZ%XK՘$ô~>Ԋ.:MqRR僔L_?щo)4VMB嵏䚵+7N[(tdd%7wT~v`C7? wnpnە #$ { GrEkM48%R2}6^D s&DӸwH=S5fFcvґT\,i7|\ 6^Cx\9BOp*%f07Y 8'*<`j[`Q%װiX{J8w;mD~ʼn.Y͑NP jVmhuF8dWsG*C綶Lv@~ŅBZW,)h,pM.m=.ai nvn!|5P1C*>'Ern;xzlݲh!{S!n'xdydt> CBoܙD:cS4"Ϋ+~8_9%ŃTcv[6Zzޏxs!Wo ag1ybZr o-~ә܀|ER'ISp=ܐYL9!## m7/Bۨ |#x >w{nEUem|RXQ q_-}`FUqaAS*")cP8-.śȪdkf!V3!r1IY3Hxk|.@D"@a*WP?n!-GcL1p5 Zr1^\=KhFkb]^f#H)d-]ax\zxs.q ,~mm/,F>)[ivJMA),rA$lv^f͔ǏxÇ)Խϭ\O Wٓ+ܞ2y}RS\&g%'@ec XĭQ=I(NE$]̺cb\[%'U='s-ɨXk(V.{m q8%PFg{-@\QV( *E hI'ߟmi}}P^\ԴT;Ѳ`]Fb_ 6M eݚP1ZW6Z 1sZrVoQwq;h(DcSuyp=:k6pDP$sssʰRz4\9P0k*56ݔ# "[ݠg<R!Ɨ0ytg_p[}WH/MoUa -\Iǣ¶'&k% sN؂ C$N|2HծD C-w/inZS"c3t"W IS'?@eTO]I1'*s;jX3XLL$mX 9PyaW^%J%.ZNUSbLLW\](Ig,hAF]Q<B`). ]šgsf y0CAhcIȐk%dLЁ'w oѧ#lՌ8h YɩLې9f5ߗP&E˜x$P Qp \>?#ze8'C,Vw>Dp?=@J[\8}؋ۼtJ7TreDBt-1Tz]cAw 1's?g$;J͹"=e1f:2+NuؑY0"["BާۺITS[)5˭K#N7 [Λɦ'xP.>ƣ"x<Ǧ1l#s2tщ ݢ? 2F)A,|lʁp9Nh6Ëسoo֠(=7W^<:r#P'k딣7OOZUeO<4Đ-[(GXu]j񝬈OJ" NВ =42YΊ-_'{UrÈW\)E-ETX+\RnTSE28첤nV 8Ȩp&bBTk*sW W[nmP/Fq:_6-cs`l_SFg'T(Z'4Xl+a<kheQ|IX+P&}@)8auܴl[^+C֬rUP(E`te7fޝ%&•|mX5 k< nnה?H T¡4Hˠ,GKŠ@d8]+&D1jaT\- GmB￬9 @ghp&Q#^f].TxI:'gA7)4WpN2c>EWp%vŹLqB ?ҙAKT%NLx%Yc7ov]4`%}])=BpR 0Gq{ڷ`ޔ9ϡə _vX}:#ٖB!cH5Ă>|5\tL'$(L4(^96&N]nnQ9"ԯQ-̊ )Lʴ)TH˔@  TG۲x"3q} \GD݂տ`QY: xy夺G$2YceauM%bOq43iͪH}m._$` 2 Mbp8%>R ;(tA/%"Ű6'e2wQ!79O 84_P̿ B6U xXQ?z&B֭wc/I& @LroStu_~_Gdv14)G#!p;=?akW 06w͊X@H am56 i~6TNZ*E=FK &y: 0^t a$j7&G,?(nN2K&ON 2K?((YRE^1h7p9onQAR@J=_B슋o0aBa'G^Q^~0_I9Qljm|>m H.jmSoז 7R)w(tTn+H hajFxZlA& Xdt}0E{#w!a{]j{\>kPȊ%V)Fݖ{pr3Xk$~`JD,@Ξyfm7MmHb4uw"*x P* ; 51*ht/840 Zp.EZLT[ƮH_ knПfbl'w)};~|p' To+5aq0@:Zr)}D?*uaK͎TP*kOr5@k[_YrQ^=TRUu>D+ul!0A!PglKmkH'Ƿ5+`5`QBsFD}Ύkcf\6֞qVC2ԡjKSaFZN-} 2=^^!{"tzW+&t "A O=apu<]3.ծxq* {O/NE`aj5& h&7jQG<i`OJ5r\Lq;5;NccݗBRġ*LKR<(J-~u] 9|(-(BnY.JLDiɀXIjNF|u5H;3s_; o+q ̓t68Xg;}PKvh4swhp" #1c(%[|m+{jb9|;BeռuHj3+,:-[4Y|^}=bYrOxUѷcDiUh۝ Z^2,z6s2*9FpI`c@pӊS |wsMp(V71j̼}v;k>mMgC޳4!K#LM7aR'򓁽lZS'ֲW@J@3h/y2% j26L iZR1 ʓJΞ.[ f//W+4vMQ]1=#ruσm$|;>Alp87^(\}##pezVwM}!hH QzxQ`t寱!\1/^!wM![5Ҡ75lŃAsϹ,n GU&s$X7Qf_0T` k-~Vn57!+#<3uIU_Bl%pyep/ lɆZﾛI4uN 7ξY&t|eD*f@ʯϸwWoaj7ba{Pfݾ,s>}obWڍ94="m\z+F0ٮ(Cr-H~6|AI4 tߊk-|IaF,S$ xBB %Py'm)7IЇWDz3v9alwdc;nǸ٠A jSՒ )6aV9YZ̟Qt'c#1 V"F3R? TS0MظaKGw9a q4K 97y&t3Y HxOKt\N<S Blh#!蚙/? "w@()ߙJJeHnE4DK[aVP F _NR}*gUp^Uޮ>uMV?EpBeɺ)S Ҽ8E<ϊmJjM)Qu)cjpJà&w`dZ5ϓ^E7CwNrNwl.JUPY2e(yۚ/fWrSrdl c}͢$ xWagO%խ]حuY)p +JU^Hעqξ%[#,'Wꍷ^{肎U=#r A ü f !})W({`Mn❬GdiŽ,+sĆ^!}6}eq|Jvhz &cyY'!)Dmhhsy/[&z f3MV:쮙:K3R>irV7b} Hۮ 4(i%gM溓-⊀͑AL.ŝ ,o{1ZB9A]J~mѲ;,HYߓˤ|j\y#xTO?H{SHNe0a'ӟwPEݨ1T LK!>NqĢPb7 .HȈp;Spȼ JM0*< Q׍ufF 0kzALl4n^&A$tPL ͚[#fTZ9KvtSmǦ* 1R+Iz1b"^`)@6e6EV}OmpQ8N㨳XE\nE)̩v瀿iv ?-@iЗA/tƒFI`;JLn.t2~6>G]٦$ %):wKo@]Bkòlu9bFW̳5Yv|6bۻ5@z9gԅW]iNϯf&O4_6 u2k.p^UT@d 朧OB*_)gf] % ^%~(bH[ ~>H57pd8WK] G̥ d wci\Q\IVKШ^6U][vQ>2VylJ#مtWyq,T!v3w LG&"R?uwr2cGƮ.9 #FG.ԡ9Ҟ|6>}S:Eё6$a?5kby y;LQ\P5rNaAXy΀_`GW%U8 {Q]C(t8yX_Bf2FE-l/S늋ZsتHML DC U'%%YIZ3tIe0(tm6^dUv^O;qXebì1Q (X[|JBT]dcxSݳHj$'w=[WK-S* qmٟխ"3VěD96`ì׋<&.cT5 2P 4Y$hBS\Me肤_7e@ R?uht[+;|4. /ԲO@Orz6}k,ԁ0OZ#15*zJ:6\6Qw\'߾?X ٲ9LTI! e:go*ЄGK=686yⶢ" `0Vc0ZՒݞ=#2m"I_Xf^3Yf|^chݗޓ Wn 堢ΫH<Y ak3\F]BsCB~[{'W_wE%M ޽=UOf#iOFrV&I\{p@I?G?F?q技l|ڔ4J6Dsn ;4${w D>{](:љOi 1%*UQRB]3ϳ#`-V!9+f>RN4LvBGؘԕ@Rc<R'!$@v 7\Vfܮ>%/WI]O[{O$:Dit7go~ϡժTMKdUp`V#oo~䈿Lfh2;C.k;q(S=8r= slNj5  \tzREij1_5I%"wBVPMnq3:pe)wb2wnÁ}-2#2\n:BTJҍ|6SQIZZ{9/fSVvxR] b6h+b5I³cAd`4vk]~/31#3|c`O;mBvyCAZrO%m=K2W y6]']2=wM-39(,pW,k2|ITR vqJI$J,Cd S~WUbi(q=lAp@>6{Wt?Zxfh< +T[[ i9^*JOFM;S[۫HyN&4(!m7=x\zŅ 8@" ̔J ٨R}&7V?J%EbHԎ A:R_2ST]zD8к%z_?;-J8rj#A{p %vyAaj,KdO6rq| ZWDmƛVb;/@)E lBdQGeZȆ),Bz;EBir//OC0~/4ɛ-lY@GqR~Qmr|Ȑ'(\9?;fƪeA R_8.eEVEҔяr b|D'kjNɹY`:XO/-4/'mn*%eIw.%=8?#Bv('@.RX;3)5eZ@ɄpSTr^r"x1Xq7 ?M R(L$_El`YP?9k^"+76wgƮ VȔ֋4BIj 9=VXUc<'Ql?P9Wj6Y,)*/#P?p\sJ?7hCJaU\;LwKS0RYEamL+fj8fٳTӣ)(+,qL.?dG475s 7eN)v Ȕě+ShFRfZE{G)efsȰox3Sk%˭~9>3~Q,*n݊(ne0qE67:β9Ҟ"0{KiHߒcv amlz6:3o{N*)Q8Dkf=L@H+:Ǡ+i_B:djεL i2a`V4Gz+2?ik4x&؛y#9>dy̻#K&KB;mژ`5L|x윈A- lƍ85SQypBJAYWcB<(C__ ޺7'bpmƒnc͹|!K&g! 6[~]^bBCAekЃ}J_8Kv43rU2% ȵzC2Xص:`OgS^)-j*$ѝf}_SVGd{ˌwF񄕨%"1nxϒvp'4sǽVˣޖ;}ًAYgHuK FJ dśX9 6&gxAe/FL~_ mj\hDepJN'9@vq_NaBB bGN'{y_~+Q7<F-4VA|k-!+լ5,p?17&mG+rNgt"`p KevF)+_ԇrjy1-}73WR6k{HZr=S5{?>ۅW"(s=F ̹Hjd\'>A2q ū3V|Ȥ-)˦3 9:ah=`Mu!#5e6GgN0ҢZ$ x~o gR=ɜvNjj) uKw}?1 fNwXDK+FGSV=¡' ,D8R\j+|W1NR L;/\mL׮p{H""ҨL2>s,L,]`(xߏ|ym1R8 z䪹ڪ1n! kWon-uNhdছ;4PP@;ptyeNDO6EdqE^-'Gtl:Or_#=wb-R7p$2Ai]c?2l#]լ2>yf4 CG!2(̙uW~rcm(p}d}eh1/ )LSh# RX,`o5 e[gyV͘2fbJ7}hKO yPLDFE!`HYlX$܏tO} >p Ŏ LMq'124\quSxDYw0vۃB諛n(S Zl(t8`bYں YDL̜>Q`LGE\WMmӴeRBK7jXL68xYu7ED9!p*~Lۑ!@=ai/Z`>Z ΄aޝB $qYN0*@K& ⒦F(4k 7RxF/8Ok|;_^Ol/!}# l%;QfZ}\x6Ҳp0~xמp|"3ibcy"4u dZ'A) k4ݳNV)wz7ߔ9wUxV8zDYb fճ^VQE(=H!'҇S"GGWLv'W^|  LTapʷUd 'M@é5SnZL+8Y9$tVsbGIKCW-ȭ[et(QŐtO+F/M(f#eLI=4ʍI pTw$ݣCr[_a͛E IKL\oؓ*(-㻟9倣_|*y/8H^(`6%1KCed^|Y"41ޘ.OUTƥ.%-NA Hbfs8KT~Y"׀%^ں&׃qz \h-kA%`]k~ɖm ]Xs^-xVlZ#&jzՆ줯A}=Ƴ E$.=y5szVD*Р]oHOMyj(bi =93XqCNil']v&c*ϳ?rN䭿vfrO2}Fןk"rͺԁ ("̰lꘙ%BՆ'259ﰢa&0[Q]8q0o"gQĘ,H|H1W8,9h2%)x&0FMhzhЧC(Iadp͊sf[V)pwqԢ1ywBf*%kUk"_uMxw#0@D-`*~b:b#_~:ZcBFf'K)d.uBg̯ڗU}n03f%g3Un{F8Z߼RlgU\ah|}2fa3NG؝"?tߡ9Frs<gv{<1?aAg#vcVEŐ?s3 endstream endobj 1749 0 obj << /Length1 725 /Length2 15995 /Length3 0 /Length 16538 /Filter /FlateDecode >> stream xmstf-tl?aǶm~b6:ձm۶͎{Ϲ|Y5W{MF$fgadec(00aȄv"@.: 02avf c5kC  G9UɅхE@ 04%rq-bdma 0:)v;c;[D/@G:dEUT*C[?m;;Ou4Xֆc'o`ba 0Ynv&.?hH 0bm-ghPػ8v&@G[hh򿩆67C1mߐ;D]Ձ>YͿS5;SʿWQUO+,jklgbakPvGOCG +ZvΎmiYBBv^Zf6#38:m%?f݁ưkv͡eԳˌ+S-MpTp~^H5OLF.Vo)օ]CaAuN__ Td9r(Eg6(mDڎ[]xO]4kRpm/yAYaVa;I% sKK69d ,eYIAEI[& R5F>7ٵ&@,>0f5~s7oL'QMa%= n`^d+.Y+-sh-"ʬſ~LxēDX^93;k)+x\oUT4K&2dfj|.>Fٍ["fݏ.Vج{btà˪N$:_zc/euoFD`ϫEV!e3Uy^x5^2YLuDS[v^' l"_e_r2"XQ-nvG.6KB8#{䠃)Eo)Y"9&C[3qVL*,lΰLJ4.hO:He;7c,:0LbmiNe% qҳWQYaT+f >TsU0f~z}w/E*VDVҴvE/], 3;/,SHL}ӣ;(4b֓0JJY^AOxZ%:Re7֢_IPH t?ߦ2.zsb3E3FB طj#.\+U@D%T> vl6-"PySlJ6{&*8`-/?7A_:~ >TJb+!.!*_8SA-W2n)^uש…"@liwW,y ' s, R\+㱵'yb]*wqQD)㉚m/Iof{o0 GDHTΘChd х.{T8*sordR@V:QR["|Yp`kIܒ,#;@`)H<'3 ŗVN[FtQ>9U$od ]tdm2*fGg4Wuޜ5dizNA  qdsXS*t3QfrǨfi{A<N@ q3$\,Z0b@X`Y^@&usR(zBO3O.$6aĘm!uB^T(O\'6g[:Ah>hgy~g=,k H DIH(qJf$f yu؛t٬D1Ѳ_}vkzZe R0H%hL?>Ifo! _\vޘN%mNπjDMݲP6P-9_~=\q"^l.ʧOLX&!?N| /r~`:;DC4n6B| kXߨfyǫ L*x Sh䅂s'ɭ~KFdh0bgVyq*lؐՋDCETZ<0S|_=5NYY]s9"jSr&W=8>Gk"CQsIyxr * 2M55j~rc,7bҚZ t#q/Bc <,}˦LY$itnA(2+$)!Ccu o TQn!lo5cK f #}d?b_p)H` TH ꃯq72-t/5G:b:L hTs&,x@xtR^n}*r)tIEAQC30T:}p.R6o6{F~>L"ړFT ,;'Jө1"3ı[UfPΫ`+6fVG_^Q-ƧoQJGg dѪ*Q_k1Ϭݜ(޵ivfA,74Kn9 vNaYVZKUoD_kh]ķ}## ;[cq+!,lH+)D%縙eSp&1 p T]ensu~;J1&,STM?'sY'-MũKL=(t.+O?4zZ(5_6o=H 1+?(D_~v(POxo;ۺlm N~jpcIzeI۩0#Mn5r49=W[mV"q6_txuWaBk_0kfWN|/:9Ǵ 0vZ3_8Y#$du=Z[bM^C"id-1M~3Z`0,*7֜7w&]܉YpVr4.ב X-Y_3LR/Š|Be)V/b(QD+ldv2oh, %F0ޠl|X/{[)hz&>حؐI>{xՐ%Bմ'׽ZC"C N#% Ԗ='&ŸQ,OUS@RfxM[e&֝Eξ!@&wb|Oxk @*'8k"Q@i'QdK%1,ן*LDhvgd hc+6܅&>Gsm?\z3qCN(4)Q^ w](Xƙ6 H Cbz&Dqsު#kЭGS\b%B"aڠNUZe2'KQb=T K /3]~1:nt.eHH:f)%}9,?M!->}*6MOB{(O7ԐZ%9П1[yI{r,XT=&5MvJ~#Ƃ8@3n0:F4hudbB}Sk aa}H1[&mcN_`*Yut5?hac5E ۚ,L _Gzd>u5j~. ǸA˜!U䔵 .)Dq馳QJX~dzJqMݸ|1?+XtcOTHݏL ifʤnhIthTLb[!BTApf`!PIW5P/dlV4Zhty)&uRO>LVڷJMA,@$Jk.?N9ݤv,*OS;UPu.%?ɕE5yh!3br5!-g S8)KO#֚zK#?QၨBs:uu0-έNq΄`ctɻ ^FZ4Tt14P.HVU=K0u&"2+uzY#9PrhC /q;ean^j+yK lVV5G {Ea-HU Q" OL!f߮Jb\78X@t}б(:OubsroF @>2Va8)䢖 MnS Sބm.幜} ro,5Q㹣~:eګn zuu2|ǒn```Fl"exasJf(6)8 4.i,U0۫/Y~ѸZf}UX4Yj[saut% a3v96OݒYIywfW.|gZ^#2]tQ`q6+w9巀o%$)tbXV!Mԡ_k`oz3Oɼe-aym@*@f9lmqWН@PödjgߴޟVNxt6_gGgvqkOeGOMφAp< ,Q,a{;{̠NxAV;>#ӜBLy/e8r}3MU+#U;;d/*ޞI =JR>ē_L.mָ%uM ֐LbĽ'wɳjH@A3>ZbJFsi!7GQn?1?͵>_[g>9اdөt.)kDNB6.dxVEkt(<@5dYDO鬲h`[٩SQ!}Vql(|n/6FP!H1Rчd$HcG>r04h[^ ,%lXJar;" mRD 0 [CWAiKidD8s2#okJ[Zg'\?-\=Hr&\/~Kbj;j9c™7cn]r^l}"tj"_HT'OM!*;T7էAԽ=Ǜy;0u'<2c]א'n|rܚ׺Ia}HTǡ'?urC-?3e_@hX9QNc?#Igp rVJwlG5JߘBiBMbqXe sj5Rtoxm #c+Z+wy8,:]~m 31,X?|hPr, ԬK|雵j|ߖF'?n/5aQldWPkY&̜):G'PTIIM'ŬcW۱Zn ]*@Ԫeҥ(bsqʛO*ݯGBc+3h~P 1`mLmA "J\".l;dG |^WpړHBpuTHIS*ob Ci-c&sO1l>^t] +~rdώScq]bH$Z.V 7z'>1[;ؚZz&%vXXF O^a8^=L8Wm 8T/~zB'MwH67(~ u\bqK_ !ڻeTUŧŁE׉WW@Ĉ:h](A!V y7u'3b_[vE{bO~X*O;[9POkû'2r~_5M+K=`0$DfV,wG xRJGA2} B0J=젎s+0窑/hR.: [ kň1Nƙ79Tï@(HαJeNJVea@߿>":c6si#\d!luPU~ydcDƽJaLorRb8dN3ث?{޽5cH.кJRc>Qe DHf۔"g4c-mhkasӃ{&Kd.V&;Xg۵j/6hqXN e'>`L%WA1 '?ŢmrF-4Wd xuv -DžsvZ<,g3jb׉hb:2m-WY̾͘EY~:(sj14w9q*(su摔ۅoxNVw"9Na)HpP‡UREu8u?$zNЇh7ߐ$Q&m4x*T$0 ӨzlEx]t@u?> eD-\99bbxݦM>@F%dfЙ/!s3"V587x4pD4¿ AH>g2lOU̖6}J3 bPd}63 PvD/>H-:,=g+D8iM5]py{}b_nk=K.\\' T%z6Gz~[1hl5mo)m1ӧZBas6>zZPbOBTfXx7*|/"}$'F,*k{WК5_vb"b:n(Q8x܍c4i8,Ðzu ?og2meYAaƯu ~]h7 n/F4FG/=AT7֖&oB-Ζz9p I!=rw_*ݍCM.L|`zD5NdV&sq0s%;;,4o$tRgVFiQCl#֮~T>^ES`*N̎I~lNT^=Jb|T|aTPUz83ĥQz!_טjIqzd*:-r:zO|JX Lb)z+yV;:G1WnJ,WH]C~N Ӣ?.#f Vn"Q"v^}\v/}$OOZ|-ro&l)_^ /ŜOV yKfj=p{˙=o{o!dk1S}* ԁAvVU_-\XGЀY-@cB~9B1ԛ4 L?D2v͍˚P;DMpN]B:Vr ?יÌ s z+[CKRSjJ`0JCsO5j&8 a0z~;O`;v|`FeJffMw,[Y~:gߎ  !>G/JVo2({v>(5:x*}.^ ۿYc+S?t߯ǝ"%jg@=l)8cdϼɲug`$~jjI^Ϥs9)ù=1f2/w"pƘ(G4PXG_BX pLB< )BmoUʗ-z<0泞yӉiARϸݚvY0%SXmw{L-L8a0m*3 9eYhol^ޏ/"*MX3½A̟ԓTKHݫ#jJe"Gqo],ON!O畦ñ ɡp-DExXm P8 FFځDiJQ ;Qfοx Ey͓0)r]ؿsmlxYe#F .``Z Ϟ|22rߏڙ1G'pF(71Ip%3Sxmw[G!jX "=烙噟v%PV22\d-gA[/AWmq_ڽ-azf]k>4%p1=UQ/aY񐤌zݛ1Mb(7^@ny1J`]N) 'q:#l)< ,R|n6| U&(Oaŭ2"nŗ}ޑk{eJ5 FcQ]c>chv(Nwt:5hquA_K+?Dj˦`[,fW&\w+Ԡx*2t.֦8R u.QQ|a%BcHF9֖?5emʨk4zWj'\RJ41d[T1 e;8|\ѳ͓#GS B~$g*{QOZ^i.QܬoUqBMGU ,_4u[c;%xBf)(|zn=jV ⴝ\n4sU"rܽ)Gº*}u~73;Q$EXIl\S=dS@**(?4wfH@fVRUNxra$6=VAA0>Ep-?ꗞ,,,0z$rX`}$.zX{r GCV&M6R CI|b.Rx5GSJi;aI?H̷~͠K"JRS5l&d<3YR4Jӛc|}læu]rdwRJԢsر=~co'2LleWbEoMxF(0!/(uM ̌Nљ58h2.b@hUHm&BmɷCm.щs FA >|aVvOF9—! ^[7%mՂb3nYj@}*9 iF?UUUTA< |t@z8>uSDt$涋109 DY_Eݜ@&"1@4 kd2Ӓ 0~/K,W৆#ۮ]*|˘= h۶79vd|ch.4>):G綵| -$H8KIRrt M0e]]Ϣ Y+Z! e4M&CU"ScWy 5'h4T xE? "ާ{(+C kZl#ߌG}5,wJ$vf{_,oDBCAP-\{.3)O룽E. `[HP)>9CbA IPyRZ-LϷmK-E;gK_kje}cH{j؆3?f>mzz ;4 5Lۏ)침]R iOܗ9OZM}b*k|Roˤ0x963PQ`|PXEXbˆNORIOsW#zNV5Y[xoS%S0,d;xn*Bå:1WOl4Iԣk# Y.2PxU[ﺓ(e67"It} {zfRd kO ;v=ᬸo UcF6ld ;anԹ:rdΤO5=zl)V~"i,UXp??=b/ aǸP0ؖmZhe a~\7 #` uH=ZFlkQO/)<>Z2I[s#y' ][I\(}Ha!$o-LIb =W g6e ^)dH` p`%9y}eN+IiZj+D+Wc8~>,CUnZ9Jͮ` hÿz%gM1)nȕN.σNWv5NX}atTWx[Ny3#Wy(g7Yl_+X}^5$BE+{^4K^B #̶֜.K=S;/4J7 OyUCe@VcWCC Ff3 v L:OҎ`"[iR ?N4r:w [KFbYz126m'{P $4Gha mLZUĕKғrfbUӪ2YFjiZ,{o04Ǯ^'NlvȎDPyZ+Q,) H+OBW*m&I; қ5ǓTVØja㦛!3=@+H}$9\ʭY[z#MW❀VFtR" ?r;zf4rd{'ښyP}:FrVshȎ &H" b҈ͮS\HNPFwi"vt>1MQT|iXYe qkDj@~iPj5jU^N&H46Y0ʈ}'.}AK>)FSfVHK{tӁPEpw!RX8<tn.uAJ:]w#ȵ&Pʝ%0M> stream xmreTj5!0t -)0 080tH4 H -- ])-r}}׷ַ?:p{z M AP`nj D99UP0FBeP'  |NW @xp{=`WWA  ]g P50h4P0u ]8{" &xAaOwcSeu}ScUaSU y`C!H_ EpUD?>pAG _/ߴ<=N^<l~%OzPt5m~Cז<_O:Svx}#$j޵&GU)lꈩWV^eަV ?߂:Xwe~E7yST?@1 "ӛ34rG hB:+ډDA˯919= 'nl (qG'^#4IzTdze:AE76z~rZgq%UW]@Uf *Љ[&4%ݓ3]j g)qkQ?>rGXME9R>[R(3NtF~WPHaEk9c/v]J *2C[, aYWP-n.j$mZʺbBqmSwτM=,us?c䵶mTi@HcYr;2:Ơ69Aj%|+:ܲPݘÓEeH:OP$u4L6}NqhOiWkw2aH)X<^?pSjat,h~Li2oUV hA#JVu| E'F"^/p< x`v.":pɏhH+7$?FY4F*֤fO.(UfuWrMNw)NO ƒÃJΤk3bNMNu%$vzp?)ʋ-Uɔ(F m޻x] iagopDW,cψ?e L0uT}q/p}nRs뭸OI&] v7,ܲaK暗/LU0iD[RNdSpr\\LQļ I9by:K4Ӹ eLP-zX}*@ J %2mx0gxrW(l+QZ3ﲲf74+Ԧ-~#=Z{ukv$WbnbX[{Œ9|?QsnWd>J-.`RZ 0{BsU iVO\Z Dopկ)NvagϖnfAr l-׆PRQ;7)=Ev&BX5I2:~%`\7ǮޏJ륓wO $IʩFҋgKR$+<9AެCP1d+2 -_?tTO/aX?A^E+oߛg5X.?Cgf{"b>-{5s4_i.>CgV7UϮ |P T:佰؈@?mu]&_(r o%O.Q}[ƪBӌ"04֟j'`UI_(yAYLMqyf%LeYux[f*#|n `d$N4ymjn#N#_) D*hFLqEsZ tlx\9TGjٷe^&Fb(dLe*Ľ ш_+Q}D=7kFJʧ1Yxsbe>?&:z7(![>1rRA[º+xMԧEv7ٴT.{= IPW> ,5jG_NO_p$il_yJ~6VPc W<<%qc ;zqu Dƚ?qz!I| >Ɍ{gA0?9@e2C5?WyK.4vMeqwM"+ t8e 8,Nj#Np4YmȄסۙ~9n>^O޼?`:3O8o&9zhg<08vD=”*vOCZcpW*~ хN~C-W5! Y-7M4ǷK F/ f5EknezWNmQ`/ڟZYNuJfݻTmd,8?O' G s렐z^/(Ʌ 'Keݠ0zW]pNᑘ0G^= s> \[ajš\H/|joXcG&g|PA.{<9@ZZH| L^"=ܖtOۘD PV]=9v^|WQ2jYNʵظ6i]SrzpoIϬcZv̕83^t)t@k;E# +T,cJᗾΰIKĘlN̠wWD̕55e  [w%`x_^Nr>=5B9?q5]cJ<<4.VlM՟qK5UK8 2Z%]ۓ{O;<[s;!nj=66:A .*=K}[;hC/j6C˜3J,n-2oԿa8a}7.Exߑ6]:4Iɷt*= ?.65q_)T&IԼ_gb`eb}w$i8%WJS#/j#e07Mg`>.O|]#j%tF.,%:~PNyvHFƥ42 ntXI%jƘk 9ISʒ64J'Y|4"RWHY|?QqɌ$:G86mDhWJ-MQw,B| ?܋sN @;j0/ﮥpuP()&LZuhȊ|.JhILͺg#jk#5a[h92njE%=*W^yxQYS/2iaTqs ޻YA=mNZY{RK+^lݰ5=Fh;M=ql6XB8ruaqeoz| endstream endobj 1656 0 obj << /Type /ObjStm /N 100 /First 999 /Length 5366 /Filter /FlateDecode >> stream x\[~WnL*ݥMp{]t{q_f/U4e%_~VWEUE e Bs)[H5Pk|,ׄBW枰Pch܀6FFs*]X1u4DC/-DF*RRyLN F.BZci Tr.TD](Dvrۑ* Y9ҏ st=ʩVaP g4s7s8NviUŲ+FС'<ԩK!$*d, HguStइ,[k-=g֞@u\}Q<;/XOueVӃu}|@֓l̃+_׋j>y=Op0E?ޠ0߸n֓ەPްADY*tlKBЭ1!KnmSztkL{-AK ݱ =e:ʂbLO#K2ud!zFuل3^ul][cgz5Bs*|udzȎuR;8UR]GW# x3@)"QzG!%Ǥ[qʇ&>~-ٽF;?r8 N'Zn gBwHmOsJ%G1ش:;V8&1d!Oo #V]9H9uJ[2>͑2MOشXr]G@:q:|n+&cuN G8CV\YmĨOsƒbyGcχe gk3,9!Z`J1}\W T{~ci .X1_a郳})!HZH> p`؎b`@BG%UJzk162 WWGH+,O܀Cŗ:Ui }Bc0|I5ٸ:ΐύ``F!u!e1M`-^t2# ?eUC8v DNdATaɑZhρՉx*ywR[)|h-)(wO(mi4"5=DIʂZzuMK1|5nɘ~H &:]tMG Q<&t!zʈ`{:ܲ6؊O"_VP $//Kx%4S[ؐ6PF8tF ́)Gd@0/耵޼*Nт8we|Q4s<CM]9xk_{,1o ĩsR>i3#xehWݯoPdhZPmR}x-,25< ]JI&hdv̐ ^:٪^`3H2ඝmoJ1Ie\"AAi{C6i^c-PuDUυP^Z 4 o2WK@uDRnSi6Ө:-NY|k8g6/J_հþW׶_ݤׇUYGtMKO>{wO?|+*z0_/ [תK]ܗ9|d14uyП=G7˂ ۊJOhp2xxoP~|zh%h_Zj|ttUlFvyyi9kJs!)+F;孊e4t:oVͅ3|Ko^JvWGM]iRίw*ūVU\fffWp41&cܿz2NWW#Gȿ5><{A>cT_3-Z20/A'" fU\ëty r5Y9z8]j_ʚڲ-,jדA]&ztRg崼MVx(cL>,M9"DSSҺg9ғ*yq\^0jBQ哇ޱ2~] C7eSe8\Y hOʺIokhP/onĹ%;sxvO[υMHYdN7$ofvMj5k7YͲ-3h)&G zGL|yhe5x$w~=|xqAJ:~PIdNtdh! }8*g3U9sS92"N1=xKq`f\/8e1`G((?9K|Q.X[.7H+*.?_n(}ws̙ ؖ6hrm*ɖ> icGã5];ľ:(^= _4LJ񁐤wW; )D1~1QlCq:;w:<Djg+G1˓?=89Dҡ+ŗ}|jL`:gMD N̓Wlm~˯Ӄ>>ϔJ+ahp䎭w|]^o6usOt<<|ILqG,'[)mN}T({$ 1F 168+GXh:0W!l@n<ŋ'O_'8sNpa|ѝPbmazБA5goQQ1g<~&~Kz]ζ6tT4ś' 9mH7ԿzCbޔNhrM9#8JȠP>c 9FMkb0n \z |5Vt.k̄W'E)wo gRCB:utF"TzB5wVΆWOq{(rUSk?C\@:|+Xrp[ez^ߨΦa~e 2Ւ$k>/jE[&|4_,aаgfj/K5G;Ek~oĝߍMWwVti7KOxqVoWggU~gU*JUUjG[;#]or9|* n}л }HA!䎪tN,8ynd"SÝ[o7/,Q endstream endobj 1753 0 obj << /Type /ObjStm /N 100 /First 914 /Length 2911 /Filter /FlateDecode >> stream xڭYn9}ﯨ4EQ0vf"6"{c6ykc5b$X,'ZfaGqX{ˏ/WWζ߾(1]:^m>olʔ}\o_ka-QOfu}89075M&|iiOҔIT/띜OZ=䊗@yR {7'b1R"冎 }g8 X#[7WawNB* L*fl)"XR;[Wr'fe>m==N,]aKEj}zh}z k&wE>iAWJ.mL~lzS2[T fAo+GXa'*FAs76I!Xh`Fӎ-Jkn]5فjk:4OMiGv$TKdClGG٦JUg6+%=ZT吏w-W ',Y%',Y=Taɪ,9aɪ,9aɪI-:X*zs%%$gETdUPXu3 Tf*K(,BX%@e %P7J![*zs%(vTAxaɩʉUxr Gq\ZNfa(A.iwj};5I^גԐ{&K5^!w\.Ր;.ij}K5Iڥr$R eRR K5^!A5qɫ$I*IenPH.PZfR K5^4\RI Tr$1I%)wLR GSxߒQA(7̠rfP9P c::gStB ?#JU;W\>:3U -$2+#@ #j|o6ܩH(m>Ϡm>Ϡm>Ϡm>@:IDi!.UfUTz AItBSP >X*j5 endstream endobj 1789 0 obj << /Type /ObjStm /N 100 /First 1048 /Length 4150 /Filter /FlateDecode >> stream xڅ,z͟=u oVq3t 3a5w*WY.3E"&.ṕjvU6xUJe?Uǭ+,kÕ_^=q{gFѭ~qEqe^aݾĀ0f-WZqD.ծ+:4WzsnWWj@)}UXC-}=:r}1zlIGPZ;Uw^%:/W}ќEւii60fya]1|}x= 5]SchSޓSpWx><=:aBUpԳai mo= kzZ/u@z+*aL 6c&xpdsdY߳ }/̱n+hӒ+!AX:?29֒UL5wNPPR@I9%W3_!y>@Xy'4uA) r Y%WeN?/wgORJJJ))S))ʙʙ,%N,z,)ZI ]eUfy}hJB,Z,,r1ZDI (UI pH}hJJJԲJJ*ŀ2k;˗;CSU%Aɴ@P2-L D 0%Br)Mє,P29AU 20AKKHI%$:t Me\x Zt-id4]2B.a2vaVǜ n*;ktSٻ !,TOBEajaڣF3gbv a*RF7ղz;DM)T f*CltC<;9{I /SB)! M fHy3_зԔhdD/4%#zK^(볾};7xd`K%VP*ʻ/ú|9X2oG|)cX1gX0xN)s2IOʞ"c*c,w<O2KJn+*꘬WH^,±nu;Hd]5wYs\e\S{Y祈Gy<|/B{+3\Wܳ"=[A$r<N!yu<7yu<7yu<7uO;Q߻6j?]m算s׫g^?G IAm8AN9H[Cu{2hyo|8A1].Oo6 _.Fϲ, o7}MAA㮠q5u*qWT+*hw(J1Z񬫿YϺ+,t4:r 5qK(,8,PXpꡰ@WUpSIσg=)O >.e#^jb >/-YVƪbUjQh[vX".xyywXȘ3d,cC:cO={IS>Χ X2VUqY UX0Xt|O<4!Xf$0#0acO#'dX X\"Ί8/xPn%.9u K`N`].gGy̝xI`CJ*6$ !Hg*RB`S.M9A~+>dzfăL*q9M9M9\RΝƊ=l/wI)y5JɫWJ^Rj䕲|vvWy0;qE.iQFHXTw1Em|&o7wĺUXc,e T3^1*c&cGSP֧OT_^1ՙm}LNJnc}xPIcD`.K`>)^_G|O<$0h}%0V K`>rA|ăJZ =\Z BKǢ=˼Yz>u[ *TIc!XH`>rX+>ִk5 ǚFcM.ZkYԣ>VZk5 ǚFc].$U5,:龜 cg e: 6xfgq7y[޽GO5 $١/GEӋEˇf-xqԣ5?_hh~t|Q4?ZzZдE'R'}Q5?_T͏/G_l]4oܘ-ROr陾-j~4Q5?:0͏0L0;b#b4>0>0L ꡗ{n={I,`G7kztqVIm+/zK|ѨN—m~aKh7+6NO{A>ЖfzLtZh?O-٧ڍS<;N;܇C!BV` Sw~l0H`]n_w/ںp"qs;8-T]˵u5ڲpǓؿI~W]w essss]]nT7`O-\ܦ޿{Py&zzzz?76c~c~M۷>r|+׻|'0:-~OX'0: %N`!HgлG+mq);,$A`!  lXqGoqB y5kW+^cD޽qYNj$&qMj$&qMjrLy7yh&y5kW&yukW:ɊC9&/&/ K`I`]KXXɊͻˣKGI`CKSyjj]>U9g[I~~ vƪB1KC>U9ߍk ]IǭoϴO2y7s$K61$06.ߍkM]:V`dxIMܧ)ڔYmJ`6'#qy:ugɋmo+V$/H^[[UPX=[A$V$.vH\z[ض"qmmE.XdHk endstream endobj 1890 0 obj << /Type /ObjStm /N 100 /First 1048 /Length 4000 /Filter /FlateDecode >> stream xڕ\[Ǎ}ׯǝtXE ȃ퍕XV;A]`KZrקxj8>SzPj<෿}_|fW?|GczT,^/BAL!KLfrD+Բ7O^|;WǃnR^FȄd2d dJex\0L ,rwAXVFSAXSPQ<@YAXWAX׀e/ʏBE+N10YBe+!as2$lV*C–iG0%PH [(a#;ht"+12MPQ<@R.B P"+d2D\flW< „A'K#@P4w"^v"(ܭ ]V@X @XAk4G0AXea^aAXP(̀(a5JXaAXexq&0L I”V@R SJXaJ?.Ԓrګ@D3u1S<G 꾬BE3a2 }*)2ʾhe_\/l.ZFٗR@B6_{# (dFeШ |E3권e;()a()a((a((a(emIi3Q< % XiP%QP%QP%$@ݖP(DٗD CٗL CٗL CٗL C>ln|M2<+ʕDٗL CٗL CٗL CٗLu[QAv#PPPPPPPP/d9@KtP("$Y@p@)e<+_B3{["4%B3{["+@#-\V{[RYV)aےJ ޖTJޖTmý2R°%-02`oK%-O${[ҸK Q*k %F?5L ~w8+#akIw"dDyhd"F!d Li 79ii^W"H&DM `VVgHL!ٴ`n`ձ(Չ`Dl]r٢, "Y;9F&&ILLDMR i4e+)ɑb)6ϑe#+Ɋi4Ge\ƕQUT`LQe<|0 6iPߎP?&2ȤQ`roU2>tKd%2`.+X"SdhR;` D 4 $#vhTQ:f4Fܡ!SX,0L:%Ƹhר+F2Zl%+6_24ldMS`1Ze؞MVF`devKVْq^k"\Y-[YZ*Y|+80 aݓP|*ƒN4Idd$$9ngKz4[RCb_i!P® (vA1jC}XSòa][P|%1Rl$aw# qI T$KwqmK:f$Ɗ}UmT`Wu>{`Ia֧'í9Me%/)^GO_'_>~/^~C[n|KHn{<1{ҦhmdC:X gy6b^4_HNWY˦?\yK~ DK/??Տh }j|wqさ/J;C`cGƓ^.cI/Iſut ǪDֱ>NWsog%֛.v_~Yo>Ozx4^ǪxbGokjqĺVBwưuH\Gg^;#Uol_9;Nqpz58 :8 8Ce :qWw*u\wQ? Qp&SN$/#xdsõ#Ɲae_c<xq^gI7)^=N.ݸ{ݕxaj5A[[ܷ)ד7Ͽy/Mٯ{/[W},#yq7O7]{ǻcIۺD}1*!$۾t8ɸy6t-,7z427 .V{u]wscZsLyncG%;+=yk[uʞ6eԁ$#wP6fXk,乭00yHX xUCߘ2(jsq[0'"@9D0~YHbi qwvZ^UUm<#\XiRp^]xyʮGX0+j1p mZ {ho^STg/=̪78׼o~ގm^yw9[coTo7®]޾ٴ +CVGuq >_fO35M:ꀪnM齃=VG9栚j~Mzg6:vf!UF=zq䶮sld;_ qqf5o87ܯN|$@n28ࠂ:/0{ mnrsS'C)z{5=E K'@-FGQtDݫ4͞Qo9 p!%R@'w+M/@zs1阒cJ){X?[o֬n3;젲~6;ng.&TqPAuquOpuo8⨊ڼ6^,5 aob[ܼ7q`^O&r`uw͜g帉vFwmܼ$갼6m KGJ(ܼ&樼6M@5yMnQyUm҃kQܼ&7 {Um^U8c^d=*977bKgEY(yɹѼ3^keUjK7\/d ˪S[L zAxU6\Rbx9hWTHjܰ8> endobj 1991 0 obj << /Type /ObjStm /N 7 /First 61 /Length 302 /Filter /FlateDecode >> stream xڅ9k0w}76C JT28_o_Қ <410C63AEE0E6F32290E0BF0C19DB69E2>] /Length 4488 /Filter /FlateDecode >> stream x%I$]|Y]=LwޓV6 2@C07l/qB[Xe 2a!! d_$\ŅL}svVWF/^E_pgН\VK0Tvp&n#3[+Gj `V;e!,U{vp,W{vp5Ԏīj GjccXqpW6gkUaMY!-u8=vplW[vpT۠pVۦpUۮ0^د6?Q݆qBNů9vZmaL1[jqj1'nm7 zBj[ C9j1% j6F.}aRmU6[Cu_oxf/a 6Q[.j {0{ bG'jGX-yx:;S=jVOSx +xEk8esWUowu.iwr7 nk^Awc܃떂[ut.s)Ug;ۙڎN:)ħN:IDN;AfW?Lл&e57XnGM;wr'ٝ{jIqg},Zt,„1g mw> ֭fĶ]n`v`6eLΥ4ؐ堳dR҉cw{=랂u!pu WT:[goCĨ> X `5` kaN]74.N& [`+lv. {`/p!8 p18'$rB1N: g,p.% W*\pnAd܅܇cxO< | /3x Aj9堖Zj9堖Zj9c7~z?žTZ-pfUl]g|,cS/M/]ZjeZjSW/HEHTKS-M4TKS-M4TKS-MzSO->SO->SO->SO->SO->SO->SG|䥖Z^y奖ZJj){ATꛦ=cvxɩZhjZhjZhjZhjZhj~ `O#)8xNpzx^?O} 7 Vư4(Ini?& [`+쀝lOv. `?p4p 8 4p2\p ܂ppx1< >ݥ(.c^Qcdd伌O ـ2>p^x缌g~x缌{KKKKKKKK6%%%ӥ0=Db!ܕx!2LDQ<%_r^]][L~yQG__r^r^r^r^\N-"PJ)%ė"P_[Z\R[2][Ay# )nRTJ))aUa(R;?Cc0314'f2q1odF"HD#D4HD#D4HD#M|ΓF"h$F"h$fs7zSZ.J6hdփF @# 4RX yi,D4F,KA#l4X KA#*F"hĢF,9h,4RX ah䠑F~KACrCr#FJiFT/ c_jL[$L̅`X"X KB|l?\~ ܂)]W:WW[`+lv. {`/p!8 p18'$p98"\p57')9{tpx1< >Y\Lڪo}[دrX+j J|E|E|______________________________`1,CX +`%հ>r/J=g6ڔ+ja#l]%o݈W)} پjhhJ'ͳxO'^FY48utvM? m/vG*)?}4\xcx pW zắū+x oܫ999$coooooooooos~s~s~s~eΙΙΙΙ?1sj oo~ͷ4|Iӑjt~HI4ȏIvj\%99m\ֹ4x_s~s~ha^h\RkW淁.+C䏁\g@|?ʼnωωωo_K|K|K|K|K|K|K|K|K|K|K|K|K|K|K|K|kJZ[[[[iw񾜷ѭ]/ "rrrrrrrr꨷޺[[[[[[[[[[[[[[[[[[4[j--mg----M 縄i_4-------------------[o>CIm7tF~#]Ho7tF~#]Ho7tF~#]`s~ķ+ؚèmvn{apa#pq8'3py\+pu[p]͈{` Bo7Fz}#^Ho7FEL4MҟQA|yyy[GDˡX?J 9(䠐B 9(䠐B 9(c(rPAV"lc^/ޓ$"P@!E8wk  }}}ɱKcMc|X }Xa p>Ohc;V= <ÍUפΉ-=:XB>7f[Xglzvж،JA8z2c[#ib0atfbWy2M?p.% W>\plmCM#x O vA#mqt?c-1ψψbC,clQlI81L2D L2VjXr _:33333Lg#ٯlfbyτ!,;@ iԑi R7r޿ѕ4_i'[Z4G7>73qt+MG;i׾GwWi{i48F_n=H_g=Ls8z!'IWgK|qjė[2>//Iķk=S}'سhӎEڷpPܻ֭xR[ B+ N{;<xR)Q7 A!@ˇ %hY Q >Ȝe{` '@ ztԣ[^]zۿw]<Q ??;x׿{D0t }P{,@vހWt u@\x&PA Px{3!޵X @ PGY@48FPBPg%NQHExT FQ;~Qm U-":@6<<@i%A=1!SPQ%0‹FzE rHX6TkSA\lDֲj?򄏘"9O "FddQP `P[%șEIX%X[EOcN#YGǐ8!8$ Ɠ'T k _r{  \WTE$> BrD>1aD|+YLԁ3J$ST s,^&_Lגz!3! ɇTH &@88Z E%ȅKZH+eB c!<\Jdُ&kI=b~eF" SOHH"$59QD4p$D(GR$T.)'FҎ=r&P\JQo3P[ (Ђ.3&D /0Nm %т =8&#I4X@ *?t(LX{@5aiB^r!BKd*_{XK*tG 9IВ"|B͇p!ԩDfD#`n6⇒ I8bԑ"ՠZ\H:veS?޹^{ئC F<+;G4pGF|]<2&k5H JR.Sr ]Cr֏Vťy .ڐ{\5!=g8v];e\D)Gnِj q[І&)mM* _H5'=HREҞ#ьB$Bn ihLΦ07"~$nMxّ3C'.HU]c a"0B^484oBڪ\D)iȕ(Nxk.xV+E4C"1\V&N~x+iQn`A!~#lH ߻Ӫ5P(~|\~dǓ"3b !Ë }&`H+]XΕ&frL鄧kyi@t!nA1惸#A0MY5x7!j15@ָoEbiLdi"+>gbd,ȵumk]#Tp"UW!3F3zMnHiSs/}YCZ-<3 N1|CAq{P$p+66#HNLB !'dZ{$ G4dE(~ A(fX YO*߹7OޑgI#r&4%搲A{$x:BQxF}w G#Kt[!0sr<됧59B+HSːrt 3A0?}tW 6Dz" YF]L˟=i|?컁Ǘ9c%QZC=/H8F+]gW AE#H~πk[ ڕr烴 +pRWeh E  | wp7'n@~V|^Gv%61<5B^pdMu}At`tЀ[‚'7 Xk#P _M pP{@tC]Ig~Qȧ|ߦM1Bp(te1z 1~i jC!| Hx灀{YAT!5WĀq D @t>h{DH!|@挩QHS%k9D1s7MQIWhwlz @u֓VWCx|mU 0@!)#qcggsHK)ZP^n=BiHP{ H@D(1I D [T "YJArnVWaFЅzVc]&π~ A?x 2 ߰C |{ GB)Y) Cwg~RepMф1٢)Z2,!w2J&̩P0aAhXt FgiI- I/!bOq HD6wǖ C  QZ@@Htxƶ[ʥYU\.rf΀hϐH1yI@|*H@ )! Χ앩V!1v:*@ ;;p0*:˰H}l!PAa-J󊘓vqMKR` X޲TD*I|N 8:`  < HPA E(p4.z:IAXyZIGFC n9;`;A`[ ڲ0˲25[OVqNWA n0pu}@17p  6` G{G KЏaGRQ31k{%7Z^iVMu Kۺ {DPFa6WpS~ik/ >R?#TU18aI0@F  6 A6Pq;ꉮ?i^D`{۹p6p˼Tp 9cA@ѽV;6:jWP˿-ۿ2,pv0 /P`N~yOx:q([+ 8`?;v0U(Kwh<[<D ^ `;1N{l[<'«h9TA a]py I ̺>P : 6`J,Fă3d{dZŬ,Q cL])D4v'[Г}z;~  Px<Fw[M+I]L\c ΐА~ʴ &`))&P ( =EwKW~-Ȱ MEHX\1 fe?C 0Л@% q>0 qQT| @ w /a{P;WqkOCK'`AQ Q)P !mi9^b aD J] P(V=}RehSGi 4ˣGY<0a njpP +h)~`qanAR$FNnjȺ˽`;- v}L )`IڋrWʿ=({Sp`,S$f_7It 0;!l%  <0=C q Ѩ /7 п?A~Rq`Q!+)}~ o<Pp׀ Qzz042yP=8N= E}{fM!lQ3@Z7`m& A@<3)^mN)a@C.p qJ:p]QNRXY}) +ڐt wۻ  i]qlO սk9=p G/.@tz A1p\ Dиj  |P Q &X!! "/ A3c!^ J% AAmܚg`Im,EݿԣR9KAa^t !;`^@+U> '! !?B>&LcP aL~K2/4n6`[ܝP  |=CN?v6|J>at_*Q[ {Cͻ ZQ!TR1B}|yw _A~p$B >x >vD߶ (B6]f@$J-{̆ fSΝ@0P'p)TR01UjC)2Cð ~H`! lSbp7C!j #g"{nbM ~(ϜB+@P|YRRV/A~z j`=W]d)h KU?{A|HB94i&:nK ƟfwC C3 Ak?m>| T)AJ`"Wb Ew*=ZkI.C㎧! ky3R c G!Rq]l#F?3Ck F0r .~*3Q^PA1ެ;/ pG0p;(;@0=}~s|cg8,~ }seᄕt@HPmZX (zEbc`pҪGtuZUA>vs(e j!R?{ {C<l@ D@3 y<*tqɋ45B̫aDO*")WÅ= {P€sB @`c5X {'L2ԄUB$9(}P6>Ms{~0#7(^|CDhĩ`•hh }CphKhӁF"n-9Ar <M;Fums$Dt` vwFl YDtKċe %C4$:XHH:`" wpG#y~dž0HjBwG_@f8} j% OIсy$U K(zA:yl|0PH _ɹ}Ka bx`{d X\p;@B.tLПʢP}@qWhfpN:]`H[J]KK )('~(2hg`[~X? }p?"0 mA!K3'F,\ׄM | MLpIP~ AT ?HPPPPXLh@{PH\6`}O$[$֔MMɀB̹(lc>XX} D`D%#"(ӁE2#0| Wg~ @{0]hr`Ѩ =8Ed؄|ːO@J-M4}SL(Ŗ/h8!8pz*e@,=1S6mESE8U9STE\bxHpnyLpRS !} }Fgρi08S;E?TT-U0URU]U:}Օ)<ʆ6x)ȔI`ĖUUW`FT@ǁkeHCҁ5lwQ=Us%W4Utuݕլ|0zᴐs5ZC9Bf `q@܏v؃M22$ nI-WqYrZ =:u 1XD\0҆p|hШ ځ#  PV؇|H>6H|/(P[MI [R-WR%[7R0A'dە`B[X#zeG8408E0ˁ!I04oX/ Y%ٻLѵ}EߔDՀǮ$퇆ۡ@a M" Tpn VAC؃O=X^p`vvЍ V}Ke:!@^@;mGp6Mp$Q' bh6p0"~a͛pf%n`{J>sF^qK@r0]pkZ@iceEEpJx||6Ńn\pEf=fbr04;=]g܇3PȇC8͇FY`HXR5#HƇ؇nubޏ~>5/P_"1xˆvond`h`諞d07ɂ UhF^tnBE8>sh7pROr+nzPᾎdR*чehF>8iF/PHgMڨz表jjIdS 71`KvF=np)ƹ\R y;҇EP݊2`u`Zp 𺇧rͥ^Yp#fEh-WfPcƷVb m 툠}Cp\-0dz'K`mW~n jV  Zp{Y<@/_ZEAzּ+~̙mOh?8q`^@mi"nl p0w@vb؃\JE]x )J:nxz[q47zl_p?4-O $fU l&jj8dŸf u([kg6(xul$>c'cordef %o82(_Xqi2togPb|Ou>GM&gON֫Nh'h́y0wHkw (~G,nr|L?=dzWu8ADE pPUpY]b''|M}yyOJ#8 0hͨlQ$a%du~phq`@#8]yj}@C-L'&ҘQ0D'x'˘~x hnѸıY*|n"vio󹢕H{\n;v`uY(lC „mjicˆRL"0`7rG\@VL@ Kh1KK8 9ԏBJ0(R$mT(-(:B3a;@!<(j[EqIQ#*T(6#Ȩ=Hίm1HܓY ǒ$H07sy߃w!@@dESǴ5R^o,&Eb}u}H}<(;*XH EQ B~!ׇXB^dT Ĝ շʱ֑cOX2I͝o%>MBLO%Fj R -R{]w,ArW@bOBі8H!&+ (,QV5b;xP2{v:fяaE%I|!laLf~ p`?M@-T 儛9N =i!{puP3\+@C+4IpPd(BPhSA@!ŕN2ϔHO8U@~\%5GR-OYQ9d:$;pG6wJգI*P<>%3tDzAĝyzfBoE◙]KT'O+GOBiJ?@26@\+ˊΥDeY@ #X PNц sNADRYzY -%ҝjB Fk{w #? HeԊp; 'BAfPQHsBeMװI;l-MIdqt$2uxϵ>*b +ހDe0eIF=N:{ {İ=9=oGOMJ=g+y`fIIXm0 AH<ճ l vЁ v#w'KCL1 |xAGf8p!Gw=}4~WԤRD 01# "8Ez8Ȅ M$,p}H,"Cg!p_B3)&_h هf=yi7UFL.n+EQ9 p` zPdVn!x92G ^8Br%:j(1CC`r-,/-i2t&I͑yT &Ik16x9G(#L aRRiuVe;H9˃h#aSYG L Վb`q(F1:E:^Ew#J>E1gA wV=@:bB!s_EP ~pppd-:3"C˃TJWOB\t5EɫD +܄Y5;0"A&ȜALavn==tl@o |Uf#JI9uBV?2s',毲# p\FQ{V抭BefٍpЃDtAz{RcU$#0 j*F[?t@ H3q z&-lHG6d-\4pTZ!&ӏX a[v@OhWxA Rƪ`q3miI׻"Š ycg9g~<;D}|cqYy*Є Lh"%`CQE*~ R0gĈ[D2!-IPb֔#^h _ZD)*iΛ7|6!*XV C(ՊH0L@Vr Yv?nH g?~%uƝr?XU7k wo򶴽.sJ9˫q7)~>xb?dW JO>@.ɨh|HdA,D!#}wd@ksyQ E8iW76m$ȩS*PRv.cWF{ G> qݜ Hڝņc?g1<5-L_Atg1O-nQ}d^^), W 8X)}Y1^ !AC0dB,;4AAՐuVy I zIƣ9_~ !aOT5 `~ VR8iuhQ&@aC @HD $?(H#DÒd\` ̂\-G@b@ !AL=Tm|S7A8"@& 0%5\&rbF%\,>ʵA3bSb㯠 f/СHA+^S-8->Ə$yW? ؄E@^C:<Aă `SE[&C'@@u HB_[G!X=8|>n)Ov5-0dR;_GTA0#0E*@ A\C p#E@@JRJ,&,&H4h&2L`R%QEQRJ2JQ@%/Dxx1>UD-ģG8#UD!-DP$5AxC\&';=D&B穌G~SM[5H,)A}"D҉edn}Sg{zh|(}[CAV-9B@#D"-mZjJ7XG[5(B@z Ąer_&eo50A\%yĊ(yF`}.h!1!?DDD-D/Ι;BEFr5&&llJ('fB6NCqBUB (%n&ESWq/IP4 vhARwIݠ8 耮m)"gGo?l]CI B:S?h?+BHV.kf{8qP-H:04xq<@|񫌱^1#Ex^#\ A )DjGx TY\4C+/AC D-Aeya{4d.B(2,rP2I/B-_ϐAX @,%ĕF\t0 `&ܺB Ax9,eSe2>خm.jѤ)urdqPNATd]] ,Up(C Ct% 6*_$@Gש@ 鉀 L4r:H%IV&gE'MMi5kL+G3P?Pr)w%p%@tSUmP5C@¿!B OA 'AuIu%qrzqbi>uphJ$dN:x-kAhv u?ɫP<@zHM4jKA/duC@l։>6X *s0Hs{'9 -, ''C,Æ>̱!GLIbGDame;3I @QA u7|GÀ TciPĀ:BxLS'EsFP;A:W;ā1ACI-.ñS9WaACaA||ȃ<{|ϋ<}?X=<%HcF0X zQ@QXg(R3zSzBS'Bk+)Dk m!>wC8H &*(~whīS؛ƭoDV( 0, XGo*oؤrq"$MbB^Ë' 3p? l/_G|,#EOSʾ˾>T9@]J&TaN  1fԸcG|̖kӯ 4JS M $C觃TH $FB'oz4L#0h\plVg~%m ;n]{iAXZh EFY$v 1b'Ѝ>FW*7N1LId&)~~Ə~ M6ph1FW`Ӳ+W[flWܜk֚Gv #}ĴHL'PydvȧulLȟ'i''*(IլF5؃:ZZmU.[ɔs|\M#:(FSi3[yC РQ5 }y|4I'S5SBu,%Z'7u0lluW5Yj$V Yقu:HG'yBa1ڶtÅL$Z4+y`S\z2"}@֞g~rE!)p }[!3X`Vf% Wɦ׭m*I?IPdX |•P!2h7Å@~Hh<E:$~~ACZư[R^jYub,a%]!u̙bFd(GL֫p<p2)n}"jkt#Lm.Ch$C[PV_תtէ:#' 3})bt8[6ƿwCW&KԚ: :J<z֓#Mm,Wֶm{D1(|4hmbVBU=0= ¸fn%D40SC+4WFJ+_7!㗲>T-S-a[+]Ko{րJVcd9_nc=Aj[9GMUB06F1ڠL ;eߞ* ]CtLL#`p0(O2m#aL $b(2Яf6B.l!*σ* e]܁A!OLTG!N#T!0Y$b@ka #Ta0Laol2/2 4tP̳k<`f#"CI㈔۞00 ]os#a$%<#J b8D#$՘ d-H! %γ#\q@%cnA d  % )QN/! $]C!(@ 3b!#(oO F 0A4KpTV('VHDDD!w1"!Ax>bT #=0JNNh# 0d=#a6%ʢOt{KQJXhG>Mb*_#[ @?X@7R.4.J" @A'C@#POL#PC N#EQpME1JJcF?ZnVi@bДT! m87"r.8rJY"t2&T+?bvh4t!rR#ڑ +DʯM\(R#V!2A<lTt4- ##9N+2`U2@YT!IuZR2B"O#BB91Bx|:U@S#sU[H6 5a `XX{bYIEA+f&7"l#v6BA9O;1 T  S V_ie` a;A85jbC U5! . !c6. @ -MڂAS=>BL ,#%582#9DMM'/!" *&h4>A (ru !Pm_zpmi f-]Q5"Z2lUludc,`Z226ʄ6eΕTR`g"-SpA2N P"*RWbiGx6~a%C{{` kXWc{8<mb4B 0DԄ-3 %j6z4Mt7BHtz≌#sug%,BbIlQ8vl<$`,>%)0t#Fo!#HA4N#;O8E&OHab,Q/Ƙ{Em=}ɶVl،AiW#XJ6 V!D_'ށF`HF#HD2~`ohK Z/!?);W#gom65L؞S͸]C%?v_AYj6~.#3]SBk5r`0/5csG¸JX2bsji#'b[;*&SYu'+Xb7R#X*ox*+[L;@7#;S!6eqv)9Sֈ5b.aaR'^ ԡH1NY+ֵ\Ӌt=}# wMQEJ8;V2Əxboutv٠*Zl Lt*]yk v!$A]2uԗ"3b[!T1"y2"VX#(27:(Z(7e v<u!g~ x>2dt37Da#|F#I{!:!VUU[!E&dx|*S`smhAmFn}U zJ gMB\MG,uBbBu2EEA"OIrABqdfP.xBG~hIqB ( Q$F+cDG^FsEIEB'C^A}WAqZAߐ*RdP.`2(Ej ;BE=!-MQmJ5%IAd ^!d>3ݨYAx9U*EL*ŲRBFgTVJ9) ^ϕ@I(>xYQ7Ty sQ'Mx{бM)e(§p@=:i&R2 MGSAk9=. ;/Z;!OA˛ZբuCcAǭ:4|e b^h'>RXr?;RD# R7MqQU42!V#@BGt0`eFahAuD3VᲺ#F^ڗA7_?2\"ggTd||KG6)QDLLr`PDVt$ G%U)HJDF~GvXsv:qAnEM 5E!D~Y} FրLf*ԙv$*YȄ ABBHEN,h,er'ɯUA3* @ga)Rq9@4PB͡h?PkbD< J6F@B AHH: ]$ 8?rӶ 3"O'(5QkhҌJ.dFW)U,$;5)8z?2~ 6#ZCD'xͼ80i>v*x.K,P@mf"] JHuwEBy^O AC,., Zn{׊՜bFzG._A;@y"Dzp"vG}]7ǢYgh)Q4M߫Yhna|3|XP Q W1x@)Ta$PC7R96+R}Plq fm'A KT_p|2:(s'a 1ENu54gaevx.Ni~0xXF$3Ȇ87h @s=ȃ@xp` `q%pn&GNR4eTxPL0Ximȉga-!998txs O@ A @a}( FC-#PKg+?@XdE;o<3Z8@ 0 p D^!=Fbش}a('Ǒr~ vE2/&cpV70 c HfwU0*(@/9-2-R `2i\@Y;YFf?! %UDB7FEYC@a! ?V9 g ILpɕNٔJtyv9zyrٗxo ɗFIY~i)u uy9` ' Psx|qЙ<q1 ȍG@Dˠ)D_4!6)0JnFW>9 nXr.haGifi_ɝ "Yy\쨚Ɋp@ (y7yO[~Fa% hFy^)l+#r?cB|hfxՃI!ڝwXx9` py1*7gZ(D%Gz%` v19buAU:`٩Iɑ胿|0$FXA>6MB!MqWܠTkZbZuf: %*)oƏG}*F ?sb $5/zU :e MJ=B}CBN1 :PFu~픛[@ Q;lډly_YJA3:ɑAڭ! 6kSilqoњV!8f ʝ ;aAFa$`NC@YZPe2Hiʲzj ӧМgbAA "h` [D0NU@: QۉRf:zpG?ob{QAA:J 8Ѩt YY |бUK;ٯ;zCиA 1 t;0*tQ q Ԯr9G!0-g#"z@_ hޠ8P#GnRK Ѵq  Ae 4;+ k9BA@˭J` 7`:H[H5ozgT+[w]F1?B DkU6P;ř%|; /rNf3xХ5u=|@B\ͰzAk[, A]us34Dphu/r < I X.K-FQA`&gJ+(!k%Dn Dž E%!O!T=[z(IlzJҨFAÐliuYͿ ,gA׷3Lˆ JL(h0aOli`<ο|\`@N|`J"t ʲ; cV ) mpr+\"Gţ>p 6r6e{h\^O[Q+0-s}C-0HP D@EK`G~* ?h@!҉X -]QF1 1!D}D р֐li22,~{PM_%F@F gmjr+o_:,a$ lG9-(L H".^UB/QA ?~(zap!@QAo3,uj{΍Ywݟ^\; hA.UGQ@>r3' -~3؍! 1b# ϾRPQHJ-rnb@DnN000^ ? NN5{RA,k'.^y.A pg1p+ @ ~f͜W}^h1B_ n kFw#}P@nBq!kE_ ۀ 1 %f ;>\;ñ☆ ka㭅_앏. .WB ^Wa 5 P=,+? `.o0_ 0DohYg/%/rEpg /OVAoކZ>OWA >kDPCÅ֪+]ѣg>tvhdcNF$J1eΤY'lRW`BhD-:gOԠ;En=H a3'8'Η\re6ZhΥifAPEe)ݕQCĕߣG|g#H+Z LY(6^y'gP|y&P'\akJG a gle-#h0[,V)atfZg9@_^vB /{D{f, 0:_=(w&Hy9oQ ged˽˸24&Nt^kpjJ(g*@ilsK0cfyƾgqG@t(0&UʂqJb ;,t2Y2q. J<9=iü̚)HKD"ljC}J4 #"ԋL.*p"y`=Qb}TzFeO1)&A2G(h)Qg!حPGR1TvYfJ 2Px!VEYe.Tv)׹ֻ( ^AàI=:Y~ ,82bGwm !G꣝eu w)w.Z)<uY͂I*4}jvHid%m$b)6BÖz̡BIh宻k#"Gahy&ܥ'~ܚ>rh 'xb )hŗ)ͦ&Ӿm~ߖ.n)dNhF`]fFD Fֶ&ù_#'f<0"|G 7 Ԇh5+~EwTIƦ7ez+WCrs#^|OFH H1eZ6.n1<(Dq^wn)4j&z/h5|B*l'nL>Prs?j.&0CB$P{ !C#dˌ;!/׸}*H5b0DRx&>b'E=?yJW)/yÓ|1Q&d (A= 4&C1GFH+?1jt t35lCR>`RdPEd&JN,fM)MawJUE^J@2uMyDZvSEB8lU'G2&Sz%PҜ&iNMDt{kc3Ls~UfײFӝnIk 4R0WjZCG PeNz,in beNnMζccV.E+50V!duV6t.8>;|:VQj#r׃ղ~]e/{vlg:|XWJ_XAA ^vo|3_oOЁXGp[Ađ epX^:0M;=̋c#sAv _15[U1] Lt}- gd9˶:Dc\ j&U~цHrQL@stR;LFbdaVQgÿUMfڽLh2 h4L\BȲFSG4;j'jjü#aCӍQ;U50"64cb֩4Wlh?$Åټdw!ua2ho73Q39mO#xlyr&dv |w j-8PDaX&F^odz\i @7[5Q3q}CJ3ޏ\h(Yo<;(UM1G ͛nL+M0@uip0@^9cBX9cw֛Kgs7׆$];%(!žw6xRg~J0^7ۙ{_Wbdx@A@oDŤgxJ| {M#NHSp)@@`oW?ٟ_G?_?? +P?C@?@K@C9d9CKC=ÑK;>,DCL3>DDDWWWSMMfffkilwwvXccZYY55666Vvr-EEUUSSvvffSNEEUUMMvvww}}ffff-b|q6?)DJQNVEfw~qmPqxzpsaDJTZSfvypcZ..DDUUJJffde66737Vv9JwwDPIUVYCKVafphrwwqxchWGV3èe?pvV0w|lCKSYIfkS)mvwtQ[MbroWfʓڳƾƙɎٺͭ蚙Ӊͫڋ㫱щDZ!, H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵׯ`ÊNhӪmNkʝ3YtݫrKo LX#?.̸1WjP#HUŎCݻ?XHW^ ϒp\zx y:֫ONһ[;޷[OA߷' u xӱg{7`vQ7z|P~ȟxww j`yX!7`d68(`PT@TP @z5Gvax]F. A`^Vg@P=OXBPs &Wu`Ep8Y9kHL)H t@OAdR&!hJi6'&w*DZꪬw<8` joX\"AvH?HJG҄??(`SPo@s%|F[@K` ;.{>N$ a6LO Jڈ@?d&m7Щ@I?d_US1,P8+#6NxO #J B<{ne:xӋH.=Y#PC4< E+@t[A%|R<$@t?7P0a 1@D3 ~EaіOG #xʣt@.xbP'%,?7KRč?D鷿zDgfK/|"4PRxZ=<$0nTptuCaHq#Q(FHB3w$E'DEF8=t A2AC%h@CȌ ^vЍz4F@)A,#lx@z 5*P@t+K`3h ?,"tG,$G&0phaL*>XC 2:ۻ}|E }ŷXar),&x# \bt:H 'm^a|vf|͗W|uftޔ GVP&dE-`{og' K0 9 @Zh 1qaQɗV %hpp % 0B vAvo" 0pvp Zs  m0[bQ ͷ|PPQ`ZS rV"0{[:? M0 2tͰX?8fWWqj$x8?b@ P@) D$3]={J4  i |.>x|؄0J73'xh+dhp ZW(@@ꈎχB(`&iPNk6jlDD9`p?G fӰL^ l |K71i Z0?&"dF0 $0Kb @qA(N GPE }px[Ѵxi;1!}WyIc:&CqrgPP O hF? >:x@m k`bÇv uC9J|!5}9C38 H@^@ dpU! ' 0 v|pSH嘎xyGIz!)G``& A RIEȴ Йq2!CsgFu ( CFעh~[uw7p :CPPg xj^JtAa戠< rEc`刐"Yl4pWwgD e1 Ѡp t>? 0_[a Q2hn i \] OQ x Rg ppڣȦnN[Cjz |*(cx=dX(p16`I@H4X W ҶXpP 8ѠGZQ%x\f7R  i@f*aFlnbԠ4VY"(`<BV+Kc0TJ7"  Tx$mƪ hbvt[H<*g;P PV ӕ@(9K@ :Y@bW Z@+Ѿ 3Hs&I˵+?(`<O zdoZ# @doc) DO1X}%G4E? o 0]Ѫ,/^\!3!͈?+GN&A a{L\b14 (u[Fi=ю;Q} lA `rt 7I0=)j񫻗m}Jp@g\plԡ]aVa`pPS'Lֺ۽mY˄K L-*VJT15ol10, \r&_2۾=]2IpY P@ ~-)ȹ}(xhkqJ5!M]tjX k-"U9x:/I0o P! k@m6qPm[ ׍J+P@K ~e܍Z|4:q>fng~iΦj=Q!q(!h^h\d ;̥VE?Mi 6\h梺kN-=Ѱ npd*=Σb0v]r Cb=8\ `3͛柾n>@T~s=}(qg(CsߴCQT(Yl^&hm.;q-GmM}]&;ⵞ1Pf}8Mh0 ^~߬ >N81:ƩiK +pJ/hƉ B@9iQه@i_.BN} rS|]S*o{c]<\7e1f_q߱ z=~OPkp1X=(^>)_bQt+Aܳw:`3A ԰vd*pg{3Z_D ! bĝ QL2%Z멳:SsaDf[{lz/.-^ĘQF-"h%Mܰ!]!L5męSN=5r"69:ۊAU -:VU\4bBSH /3AT\E]Y*OkR@^2 FXb ئQ9b״򸊆M9vlꦱ)U*~0khD>lUr.\99iV%+ENgW*i-ϟ*$Hp')Ad1d×̙H;iNZp9  Тw*İ %n >RǛ%|''!QEa a$}ya8.2 ALjAL #2J)+., Pӟ37$`R@PBEP׎WfY'.pYI⧔+E={Hwa9R8")j[=R0]H^WUk7fYC~LA! od1"oYŸ٣Gr k(djt֎k'-'5 q#t̜Zig~P ?I$ŷ(\ uݫ!n:m' q*G.LBG"Y"Z(pm}"FbǍ74\quoH iC]_O< s Ĉ2l"MMPOO4}GH0*2U\TC֥BqD\ e!ql2 G B@X )446[|&D$D[UD;5 " Qυ-\ Cxiьn8>"DPa L4`E&$b6X_fFΥB22 EbxFN2R7< A@z@DBg49X7G$B`р#4$VÜEј_L$ٓwp, wu#EbYvy:FJZ9 *25B>hDrs !'/.\&O&\"2 Mfӡ2>Ԩ;B.$J]&ÔL}̩Kp8Z {C]8ц^ "sL1R̜0oU&c,<ÒiQai2HNSLUHVpu| W)hj^0;֋Դ8M#Yj$t@jT(3Qپ>!X1ra3v" %~Ԓav?0r<%ݩhF>x2m$@%#`ؐ[@GVfQs Ll@YH>AeƇ6 XCO9} `b>yy`';kQ1i],c1i|c:\?2 yHϐaS?#Ї^lx Ѐ- q9{kwL H^j3sk.:ǃ# +Vu<~m9A o6IΊ2 Vdx xbÚ,*jAl:>gD"vo܊ ՙȖQ e@3Q]lo y@ykJ `Q5n͑k\hL͐p\T?)TuW@ ]3 fsׅCQ S4݈mԈVh'! I`2wA_ u,JJ׵1=G^ʅ&ܨ H!nm3^bΛTS~zݓExpVؚ XV*)DŦCѻ?Y?[p#wXRԩ}(D'5A`T:B]Tfqgoz] l,R&HG _zEĊU Mɛ;x"`36R0ar50qЫGS[? ~C=NjAX1!fC_K6r89h3@P T?##۫qY*)#T/ '8"4"8[Ox-,ۋB)D@C-ix/k9*(`C*⪫($`JXL0)/d;+B@ C5P BA5錋@Cy,XAdQY?ZdWXĐ{%Ar(ZDPG#Р8􇰪eC! n<3 HL?dYɹp$񇵨+!*&ċPL%!f7A9=0h5ȅ30j4k3H3ĄԝwȂ`?Zl:bGlW8C{Hp~t*P.4$>4țT ʖq|08n6= )0D4pLKxB"kH YA¿[#XkFLHÈb,>* 39\+ ȰD(CQ/++!b@Ƞ]*MxĪHwɊrz8x LMOL Lq9=o*8aWNN2{L>+ LLV4Ono8<~1G飌)(BT z܇?HiK ݝt5O}PD QF ?ɊJ咖hNjȆX(!H٠(eJ((̈́8Ȅ 1(CB j,OuPF'j4B}06(Y8AȌohgp%#t ҝS-.sQ%SR5X#>]bZ`ˈM@J7|gpX |5`"5Q|lռa[3 dE97yPO蓬0Qh"0lLgwTeŋ?v  f%"YHG#%OV)NPh:%4hI)}%VWyxאW{; 򣉅dI>a1 Ą*T~@>e<= էbYاX̦&d| } q,ʪԛa Y9θ: YsX؋Z- ˨X XZڐXWwiD 5ZrXÍȆ!!R@`[mѨ})܏(Vey%ĭԽ]a G5hV <ڈUx|9"LIͳ,@EeE^feWKS>H9Bːȹl;zXouA c*| ސ~W݉9ͦLU 4^(+3?<8~? [uҳXNxhx']A;Mx}aE`^ h61 7Pˢ` ,8G#s^'87<6h66Ή V%,cAЀ-BY/@u.נd=Sc.`Ze%H:;%H7ȓ#xXΧTB|;XxAz{,h:Mfca`cZkbd 17C?0D8U,u!˲Y9%-8f3(`՟m` !ʘg51J\ J,м,e˳@G--d[- 6ב6a 9&U~< q#AIRM=FSE48,N#>be6iޑVi`i5i >^n'$1#=B Ji 09_^֝Kjk{E|d0-|h)*5Ýkepk%@֨ kkjl&&Etd~ #yzVh3hC!tHm8mи6pmnn}iV)Q@8ސpCU~@ )+ʝn%6Eo(o8o輅ledT=p*ߌ]E?ba.^HFpn8bg;iVz%ۦJ-W qK cxRR+R)}ȱ5PE~^gqj9pﲸTÎ0NV0CHLk*Y1C dad#ĈwMljzIm( 'up4p h(Ng)Qo؅%ˌ =ψq@pa@^h3b'b uX(i0Po2sQF@ݱ]ʘz;2V2dtl0SPde]FV=^gaMl6/!xoO˺xQx׈`xBj=}H-,v.o%.6ΈHp/^8b6ktN gGw8KxzWB«geq/myGBp+Pʦ0 sv5`* 1z :ԚNb` 3@̤f؏ T'!n @ ,)4ȇ\ܺ8{ppi NG ؉3.iN 8H3nFk@,a@ ;pͳF1:B?l4Oab,DK@岽wQGr㓮!#I3lS8٠  ?~D@avz|@{`Lڶy ցC'14YPdضWFnR\d0bːw.EyDTb ߔ˨!+I*'D*=)8E.ʀB>1F50 z0^Hjp̥*S8>̇rIyǬB[Q1+G$anb=4<*r KIY:΁(Q 0x]ﹽ촧*J@&!vTGMȺԄ&Әn@!Os#! S8ւ\ / 6ex N"$J=1SBhD؄#ֱZHq@| l 0D1L DV լ%A1\r3$&lwqQI0E82 $:V$'<zyӒD4~0?eTdRƥ% ,AQ; il#PE?ZZ: N.'c|Ё$CD$"A %04U;.P:{$Pj",|& j\@zF=Xe nP|ӰS!FgAbSI&Mf?n"Z2&sX3&S\ӢYY`AׁNo8=G\!NC@BiL(OF[oP==7'EQ"нvin]8! "";fu#pM@`MP@`JH=h !DJ rT'}' zo7@8fI> I2Gn ? rye/}@opKӠ@vm_{3y~ ={dVnJ 9C 0ڛĈ,L@ y [xB6- Z~?%x@_J^ tE܌E^=G٠ZdC@{5Ë`ݰD_Yh t h&h&Bux@xł'^ 6E E1ćO}9DP2` X39胮i<`X_D`xyqxBy-O A)C'!p]KDFE2/,D ?̝!x-_=(^HB$4=( &@7L& hN*L2"@.hh =s2&<$DFDFd AcDEВ!@/X,#DCZD ,ZA4]1p礄XD̯+/ SJ)DC8 V,<;\L$D~ȁ@Uъ`#$Y(i7| D<=E AB(meB+! (Jl"S1֥Aҽ:l?)(P.C DC8"D L;J@Do ^?d?lV/bvoRd*C 2 >@C$; ̳N2jlԭ%sKT+&\+d+A9dAMhI>(%uc\((Z9+OCX##(r[2S!~KuئAEܴ)D*PƭEp%n< 'Dh4F f]ٙ5Cd8MŰtAq?-22HPtKFqE4c˳ ?h(.54𰛅 83ؙ,G-@6 *4J D78CC ,rqˁ|fA"b;#B^ b d8k8* _yz{7jόqw>H,HPjB:ne64AP@ y6HVX)/5Q8t]-9?TCv86JTCAD4 Ŀ݃/=( (E @[ YFD.ޔKt o4.A ͑rD$:9[ۊr@CI ظ1?B' d}7Y!AD9-CH$蠂f0y?~K|}z0qVå; pz;_Ya_)'E3B#>P/tk{= a5-0R]cD't6;e~YӃAzHt L?BM$0 #oph[W~c~=~YN :#&^_jkCDȧEe 9h2 J3PP)๳<2D@'@5D@-@40l,g;/rrcG}FdI'Ezr8b`QJ(qF0>hQG*8ZOS;Myӧf5!V_˥E5|-nTF Do9ttN%DdV$NPtcl\;+Z$QQ#GL))ɏb:+뚵HB]w&:TV`r`mɅQk6C#x7Nw2^$@ N`FZP‰v<&.G-LÉz1 #7 3Cր3#8Râ[EtBh#hXFGR"Nab!Z@ <1iLG(YA0e ~DABjѧ?:6*C5Q{q$,8M9(ĢFģ%A UϒjQ9 A dfR`z>@9A:mP$lHϓ:ZngzHAbN(aG?'K'r'0|Fz a=H,D =4rKQx`0j|ɱT%hy`lPPL0(a0dՓT8a=@82&A  ~@GYfߑ7-I<j[wL"5*uMQwş}; YC4`(e (FS5?@hokJH8LZej) Hg PwNW˓wirVACv9~rZYaZ#~ ʣZfIè0cd2`c7а0̩ 1Xi%5԰Ɲ}(.oP I7G?+]18|$10AkP7z޵@=%)HrW#G1z>BpfH0?xc k NFWԗKPb0?A`^7,=4TJD2e`Oy0%;rdC1)cb|KiA0o,p!4$6ŘV\8#5S$DCD&$F|C4INuȓhDp|P=vHB)Єzi>1LL[d; b*+pEmI<щ50Zy!1C75Cݔ%˕Ilz 9 tbK~4ZkDhّ峖HD@]#i@gj292pE~7 GE@^)=e (D@؁y聂l#$lώhCmz ֫cL*F6 H({G 0,!EMbS(&H)E=cG6hG$T# 0/OFTV#؏TL+A ؘɆ1PLaJ@1( bE\9tV:"[%hZԨ"Q`9v8S|lJ~ȣa"G f)]M \\y$E5*6P Mf |mPFpyl3>'FmD$ ^\>ӑ1Quv@,bxeЄ8AR;"VmD^4%o X Tm ZB8S4cV|_.xq/D5`7I9a#Xr %B XFHR<=}; -GUüKA!Rd-p BdD 2>2 덏%'l&g#e=I¡򋾌`d_P.BPK $L*)y\g(xAE E$#q+LiwjAz~"1aߙ< `drEDVi{/ڀlE91=8X\(IԢ~0]"\%EbGB1C}JY(x6IBO A؄?jYeeB"햕'BObk8=\]'3mn7|j^aV#R!8wĎ°42M6G:А,oyO`a{D> 䫛>˵ |ݣ h :aS!  `VYWHCF"&@D4|#ZXMn%&(T~톯~@ݒo+8G'R&~ G`# K>N?rl4#r'ހ;"LĄIP 'l$+@"ī|@d`AP+m1&)$ $<ѭN$"0LPsR #z @iNg~$ !I~+,䁧ЏUF@ #j d.dLa" #taɪ(14$# 'M2%0f>̠p/XA#UOC $!@2Wv@PDd%;ɶQqF#$+ r ?$ER8ؠpF!.๾a.(\7l:Z0 jA`r>.2Md =32 5 2$1B-U3%Y3: ,1`RaF rr(lPL,Ni"R40JHa*3 5'xPgJB%&vj#PNR->AE>V6Ev$/`/iax^#ܣr^:=!l?`Ќ`*E䶠A A((lFMj>$UR-Y$##@ۊOl A@ةT <TZe0_wp,s=A3!!j%tt8+X#GMS$5-*H#" $TBOe !/kDRw!9y s1OFJ)g Ѐ\fOsU/EFu963KU5qPE)H5HUH`@`>D]dA@fG ~A-b;qbJ"xX fQM.B Y=zn+3R4zjZ?BZͪZ= UJR@8¸ABU@6azRLb(B_?=:`0:_LA*Ϯ.EhOHftab5#xG?Z_#L9PV+Xh/aJ`ZZ~ ,x(I*tJDA!& =Y4gH$$7ycԓ(5$k-=lk!A'NPokCb_#TmM`в # j{OYâ#c3j(:s8.C`tp~$8`$ #^7o4(L=B|a N!.aA(SuE\y=qt+zK5k<t%Dtb,8BҀ`ܬ[`b jco< qRSA*%Yp#Cqz+6M %$)8':W8%5OV;G #  @ؗpITL,^oӦwRp3WW3*"Yz9&v"%67PD8k~+r . TbT`'~a9dX$a \yȕ31JdA[7<x?djّkc8'S\;gy `|"`av[#P(MY1ȕbYmpäVoAPYNUO)q#,8Y%ي5X YC#%tS]İdUC_Q!BoEat2X`aiـYiF7 #8$c7<#XQ_θVa(TvU9waIzbao@v5sȚjD::{cW( aAh1.@ tmNBdK4#7!/.!Ȉ!ғO*w];yïqդzs" jꡔ1v$ l# *ᇃزɴUɊa3e6(iRRmQ bAb0wfK1EQa\a Ltsg.6[ő[:¯8YCtj*TB鐔Wnw2C6 LI `ͤ{ck_)e! >,") `v =(MUjz"2?2 il%1!ZmK @ݠ i4!\K;B.@C.H(#Hs$#>h#AV"ٍ݆9wf G]%OV無x]!O vj R̐'no*W|\!s*ig@!fG fRRB0Xaa~! 0"Rj O(RjgxT~MΙË+a}oATal9o;Bp7@a A~QB]DGwb1b!R-]XDu$z!@3ߑv~Ocis^~$|ҕ!v7UC'r0A ^!.TAA$m4Ÿ <0… :|Dz*ZxF=@1B+?B$ 4}J`& @$'HvG,WSI)nxE֭\ q0d˚= RC3r6[hkK0ehx $N#Eʱ Qn f,PEU?-1D#vO\ʬzPlٴkTЀo-ʵH׶iɥnMh)"sJ/S R&[-!=G=&y]mr;fa;AHB Lo%hQ \EЅ:(A+:lM (O'" ?qNBD $ t=KR#:!SB#Vj4)9 #%(iL` ^`C?|1RlC6*hdP> e^m\ꑇ_yR0 2NQa |rBN L@.!O (A,$@roA)A 2E:*r*_NA҄̃h5RLвdH<00 M8Bh AvAY "K\v ɝ #YIÕ!\y {<]ʈ;b!SdҐQKEK\$/I 3dpq<8@R `![92DPgB B@Q8٢y A:ِ|Scw!߁E "4FE@=I,`C-Dt:A|`Fq@:dyGJ `\};UQXBVh=ܚVU5N`#Q{*W+bXp,e/+YfֱMd7kYbֳ%-g?kЦv%"U6d/P2Ԗ|o B 8 yy.DH zLp B(JYYx]sR,oy%[68P@j­};i@~ BfE~ (>.ln+[w~zJ5F,Ko,X_#Օ@amH:̇Q2jE {ƭ:Cx. '.ܐ aP/x3``t :$xމtxf0iҕ"/}@4nOp(AeH8Oꬳ/ Ib^}G':`8PXB I#h9l XR!op :׼&x ?8Pz !o:a50ʰ$P= &C9]'=mlr*O5ЊAC?3LX%b]{N[ir2O Y#jV/ D5 H`@[f@#pPnpF 0 4ako50b-8QJ0O,T/MyJ4&'u`X0F$D>Qtn;g)Ḝu !8) D1`-挷~=g ir;7Ļ5ا//y:|e.Hdk:X L~|,*0iwsL~f=bK8 |hlתꇏwKYQ[T $߂o0]B~"yJFpLf3g4!nt" oS%h18[$]H&h ,Ep0`pFuq7GsYB(B'ͧG*w 0pw崅$w !|×Ih%X;>D' :}I ;h7eMAdPh6t0p# rAV#Tx)eh؇a(ń "+ 1 ; lo4(khGGc+ zT ͆] aHfB}CxhH|g: 7xR kXPh똂Xjp@}=& hUs&h툁Y yٰ Ph Ph@y)AoA4'`x's7#MQJhj,FBCu<* YdC=2̀ !"e+ [`qQ|>(t[̯,W`˹-1tQP0XkY4"T" |Yrδ Wyc@8YJ[JHH:,Ξg[R:`Ljo <*/ j, 9 JW }`' -1m3 4*nLa( D]YH]K;<%,=kp\HlEnEh <,M+˵>\(hSv@0K~\,\CW'9 jj@ yAlW,=ڢ Ӹ<6V; JP}ٝ aڦڹ]ڧ Б`BHp!?q C\Mڠ?͇#P=}k MGmEmE-p]VڏnzEK@դ۷=׽wpzjWJҬ;@>)*`}X#?m0pmE<B>BrۢFѬfk(@ mpH@xIPZ@W~7m[i4?nQv p怜B~],LZA7쇠jܵ鱳鱓g p)N%iDebe> :]?"fo&`îclPnbA-ru5nNj%bLT~jfYg \]P. !b-UgSiz~+pŐ)OTh OTv$@ p8<=0&U0@@@9>WUoZoYRS`_Obdfjihs/uwo@xv_~|7_Y5oM42_>p}/J+ROG ou__*fl/>Ԡfn.6F>fS3\/4oVׯ͕/FD>+vODU \D ]OB2tm ? .dC%B\oG!E`I)ZpQK)'RKK1uHp?3 xY襋9S,]B:)U$&[~BeBӅH &}{_0.,qݴ5;;(\w9w\ i*jĦfA@ʴaΔ[XS{QS-{re>E,|;yBB}޻zMz5TNz'K PpA8:>jp: QүÔ0!H Q$.!:ҺE)F:hFIDoI\2( Uh# P!!=2G'ݔ. T0AQфƶM#(>E!R ;gpml-matlab-v4.0-2016-10-19/doc/f4.gif000644 000765 000024 00000031525 12701705600 016630 0ustar00carlstaff000000 000000 GIF89a """***---...///000111333555888:::;;;<<<>>>???@@@AAADDDHHHIIIJJJMMMNNNOOOPPPQQQRRRSSSUUU[[[\\\]]]^^^```aaabbbcccdddfffggnhhhiiijjjkkklllmmmnnnpppssstttuuuvvvwwwxxxzzz~~~tteell --!!%% ""((//22114477==>>ccBBFFIIKKRRUU\\jjaacc``jjkkppppzz񃃃파읝롡뫫鹹!,A H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵW/HKٳhӪ]˶[ʝK] D˷߿fŚOÈ+^̸ǐ#7ec5_g}S^"E,/LSho? b.N8cNJi CW{:Cw~Ν5"Q81wn=u_\ ov#ki!`twg`|vE^F e<ߎWoK܊rM,krwqϾLx03ng,U#"l' k wHXO{E(\'Uṃ@V'p`qpMٜ<]p5<%1V7Y< R14q\ HA%͹dps@qwn{Hc 5 DϑX& uJX=t+ϘEJn}cg/D_?*0 CgN K#(II2@ %bF~Ͳcq.A$D&06q-<gV3g(1OyBK3九I"Ĉ'z/T:JtT+we`tz[ʄI#[%V6% UN+4p B-ښo&"W5H%rnJ|Mpcܭ)jJ٪ȩT:K d"cmh"i&=*$`5PQVy:{- ^@(lemMo0lU|[FvR _nC&5_ȎAc8K,zLSBZM\7tzD. \`EItH9D6ҦuM(qQE1 ȭt11&#Pڜ]ƫ* 4O !!$Fn6KTs;$=)cp@H*Mx  %(ۜv~;$yd&qF,ܑlwdM4t:pI&~JX3ԺǦ31/PдNq!1JvE Y=ۊ:QJsfo)vP'z4U֝Tq741I l>?>c71{wmc[! ͑kG&9iANBwBnm|o737}!|u95U 9D!w6qo:q5l'tǂ$bzqSO6qp" 1hģvۑ.!m13k8|*xAӄTH#Eb+!r`傋&Va?L ˜a &;0o/==LU9Wq^1!MbG))qrnAk"w lCaj`Q.fZU Uhk'+t !A3wU? V&bXʣ"!Lá烤GevQsX_Q jlӑ.qNzPSzea7# *'s0a!zs D#!/ fDէ wvQdfQH媿 Qqw73,~izb=H=U{&ecɚ*G)TFb; YէdgkI/ [Ikv\ =%Y0sPf2+A.vЬ팀jC.q?=n|QiM=uщܐ9k>A 4W"y]jR֑bfgχqKq7*<3nw{=4{E)R;]/ɺyyBՇ-ՊpǐaJ1 ?Q۷\0f<ЊŽ Լ:ι#3Oq9{6}f3Cń$3$lL5߽/<8F A2YR|#.V +؅mՓWEpQsǍ Aw}*ͽ0rѼU ƌ؇1'y߹Kaw (pܐ!0,~`Gaw atnE?Л ',e6M7]'*Dt~$7RoNmd_qCM; ?x2n,eo<ڝȊ 8p1a&QQ﷍szm݊!AP/sOPtmRab`!K!t%\7Q0bs4O>~/q$SꎁaK?>{\@ZmAUQLп 6a<mqEualv8o`Ѵ(_~ Pv7`#!^|}k)lh)}|qN9#C&ƕtar'&0S(}Vi!zț!`'POQLwf1Z%yN=ﶯoB O\"ZQL'CEݹDVpI@ DPB I (pL/#S"5~r MDRJXjA'P,:}TPE4pS"f"è4>2RG!QIX2M@sZ ]\uV^}QQ(R^\*re-̐)foSoOΥj6h j>BbjFcƄ/Cm =Grj??x+EEM;G!`ZqVޑ ]qտi&BJ?>r66$P tj N(<Њ4L!r/B 7롁+*).aj@B;ˌ&lB Cdǡ295b+8&1VF^M8CB,D. " Ŀ*B74̏"lʃҮOA=$J;X-<|;:Ǻ-JOl+CNGN84ЏZ,N$4Kˊ1R;]IuWa[ D(`mPjֱ(OU 6۠PVh".!< V1h6(S^kA$p> t_6X "r'Z`m)01w\ ߃L2Aa]@"XEرNbPҌ5ďL^c*ԡH:N=GD1@ޝ?a[*J~=zǶ`ZN.) :[JQ% x*AXB rĄB;@q6 ޹;o.2#)b`k}c:"enjέu"0*K ȏ(t!BQ >.oʓOP_.߾GOR [,?oN$ Ч!ܽNHu0@F*#L<0}栓 /!H[It 0> .!7^3 UsƐER~d;M`rLĕӐ'A$;Q %|y=,!D38p _JqE|` I(q sF6YnCn B<1+z0ɹpq*"t'sl"CDrC΁\ d3Y>$/ <#iI&B͉2$HX9hO\?$ !&كz”&(Ak4Љ9p~3 <שNy g;y3d;)za|4}CiІ&ed/c2?)ёZtH!zRT —J!L\S>U}=4$cB %S)ط` r$ I'Zq3LFy۩ W$+ *BX1P*H^Xd*^BYOd !J d QJ(e ~HՖFA} >bvwq4*B׆կA. oXW gNk(ҙ!؇ed .4SRի&$[-r\0Iu˩+3!~u=p+)n';F7! PY,ɄeEN'cMUlBїB>8I34<%SEV0L6=nƁLb>ֳW:%rRA2 ́VoʵF !2@r%?zX+9yÆiX432kM  A,ALPY6H&Zd1fu'zpllH;. Ѡf$2iI_X ʏLr\z6v,LNFF#ȋ ̨'Aٺ m :V.oHtJv[)& ?\2Qsݐ1#9'.@z ʲWOX=?F\!B-/3^ȩ#skZ Gm"Dk1$GZ-Ϝ]Lb*=#hw7V`XPtg ?-6 u4gO*74*}Y&^uW9'`*H.< JdE2{Ud:94ͳ̑\:9C u[d\ ~^#> g]_@l({9>CĊ;:? TZs*(5  4k 81#IA!2` 082<<d %.O.ɻ S K&\!N;d=Xr@%830r58F`sr$<2=48RC˘c#?̘qcrD c5  uBM,PGPL0 2|XDŹP ON{6E9DxJ׹ƂE9Ө 5[ 8|*P/ a /kY M{n@x%*kFЙB{|m<t8T9YD+ dGT.` '[H|T} M0g 4P9 Qŕ۱e%[Ʊh|#gGFC"CHEtXQJػ$/0ā0G xID="H (Nh4h+ʐ GܓĐ>N ZlP$HTEGS:U:)!96(Ӛ{8֒1V[IJ?ǰ<-\% @ 9`ԆϽpW(K&h-7:`{NVN5eK;$֩RaPk)8t]gkAWZN3<Čݞ]mU̘ EٝSA_}X5e޴OtGY٭NkO="@YXe%x,zIN8%Pԁ\<8}9W^KϮX)4Xۛ yN=x/Yΐ֣}ٍݙ̝Q d J+[hY}MM {u[ m;gE6@H40S\U՝ `MK ٚJXH!,3pE dYYW\t͈GH4O/˲(45¿Tq֭ԀA_ތqݺh%^PWΩ88 d{y7-VĥhAeP`ȁ#8vݚ`a_]ˠ7j:p0Y#>;ߧ3=^[){ik;8`a$6_[,K%T:(Ҳ<؁MS[kiS( Z=]9)1֋|Es֕^ЉdLx _[`pUt<бж=8. J dPMR P}Jϩ?l%YAIZ ,N>)1\)ϲhf/>TM qOP @X\~ЃЃHI(g}m[v>:;( ;reVaT44`+.в!n `g"><#F^!a\IP&.´U+Ѻ&Z.~xV  G |GrjCC0jBA#dh)+^M7"0hkR)huϨF)fZ)Rƞl6)~lͦNhmֈ@6D MDxEmG/Xk+ظbg` kgLf(Eo ,1 =G4gnQmAIpB汱ffP(^dЮ`F%ֻ"ｰeee2m!1&hA(ӭ $3{Kv( ?oYniF7 sЙᆸ6F`#W _ 7;0s?^˨ >c&^ gL__-qO6V ~#L0;gs`_ ]'@9Pi}qu.2ӱeF۝-8 :G7oߵDvP ?Iu H|!vwpǭ&2g[/A xPsX5f" pضtawa%vS泖j#V7'O.߈ҾIzP0X[7}ߋ" ةqHߘ{z wgt׈GFcxzx7(㋦'|7|S1ȇɇuHHg୮7/z`Gr&?$ ];oΠ{)H}(10πS.~}ťC]/ ?~d4=IߗϖgY){0~T@,h „ P'8"ƌ7r#Ȑ"G,iB (K3Q"^^Ҭi#=](d3B-j(ҤGU*ؒ&"E ݬZP=8qG9̤lQhעT-Fp >ȉ"hC܄dDЂ31䋏#˸D4Ep僙(.m4-~R#<Vm6ܺOSf"u7ʗ 3͛aQw%lk?˽Jo]G$ uo=IlƼj_ZOE7}UA *n$XU &aHrȐgGpMhS&8$7ه"G/ˆPZЅt't$3#(R)M~|$M:WdP!6$Y>9R0tY4wZ.d $IZ]BHuڹ`HI䝁 ^GקDs (jo}Vj@>ZTaNbr2)j)QV G*du(5z+*H+N8%hq.edN!-zmĮDkfhB&m.2 oޢnzл[ hFa$\ ;0K+lqC\O$k1(g1-31:L'r) sLt1\PKM4KA_œ 5a=6e}6i6m6q=7u}7y  b{ۉ@wV"80fz¸iVR/נ hyY9끮A_i v>{A;X"&Y |MRg}?rd}0o+BϞ~Լ5"z R}xu1\#3 q*g&3G8Á-!s>!(!F@=?<;.>@=@B?15DFCFHE;;IKH0AOM0KMKNNFNOMDA;GPROQPSURLHUVTCLWXVINQMZ\YKR]_\YS\ac_a^SXac`ceb^YefdfheX^il^ikhd__dlnkjnqnpmiekfdjrtqtvsolvxuipxzwsq{|y|~{vtnw~}~xxn{섅|~톈뉋쏑ꔖ헙陛뜞蜡럡磥槩骬歯豰嬰泵䷹滶⹻ν!Created with GIMP, H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիX,ׯ`ÊKٳhўzu*۷p}jݻxU+} LÈ+^̘08o5gE˘3k̹Ϟ[ xӨSNHװcc>12* Aͻ NqߙEs=أ pԲkνO{I˧/tbV%[>={F$@H2~ 6}4C @!DoLsC:A8(bxTx4]bC"C $E'ՌLS.(ӊ3i$F!N,A'TV Q@ : %R"?i eㅖ\QhG5u ?  ω64#9}29H㝐1/0Rs*p'59$nx>4+::j70B('jH6^`HV+hw҈ .[QJ^ݫ4&ނU^BO:EE,Q`߼KM5tp3p rh5)܈-]#O48ZXdƤ41Ś 0Ì-H1KBm#4,9-PÇ BSݦ4أ\v\5|*׉F3hـjF)swq4w{ gBQ ^aw Ҥ8҈w{P7:ӘI(CxsdzV ,:;`s5/BpxZ99Ǔ~M46c^'>[5.g4pi$ ~$~+ȾǺ5ֿ>@U ; hL9\#.U"P6,xA!(QTFQ =p%P s N!UfTPԈ^6H#JFD<1+8T6A%cjUαj&]Blڌꆦ1H0a sU08/FE f8DidKU`$5DZ#kh0 X)CZ%xL /IL;H2$ t#AL* %%6#F?VJ8,0e.FeMqΐn \^z;?ΩC<?!t>=ϩ0PxΌP uDX0F-# '@y5riH@,G}#!iKHqژ7GҍT>$Gi|@/̱Azss3B#c,S X"wX\H$TT#? / PGf$r_*t̘_1ئP #8P:.7^L٥"xct}P^Mh9-uXVdZ&2aY7I `6!.p" ($qQyLppsUجVI `#21i`c=^Փἒ0!`p0G1ġt}F4*Z5{-y+&}ApvuVKS밙2tڿ D<③b ^mںҡ4t@fq QDn_"Cra'DU! 9PHhʨ(M #GLލL@5ƒ1W9P13 D!*D dbh>H+!8^i b ixFAB1lb V:B!1!@x=_ D@b;i@0;{>bm; dv@h{i/"niIrBUpHLJY 1c G`$|i?.r0 8B؁px=rr$q| @Ks\=Hc.)r 2t=Hat|Ugd+T4fڄaM <8o%eƆ34q@P ޷u<>ą[yx @vKR :p`.$aZH?d509wYX=e[Ld*ٞzDX4L (c͞/Cl8CP%;UԘ}򕏒8_^*@ 疮QT5KM%Ґ b:(A>Q}7IP\ h6}d 9WHĀ=s @ (x P@]H#ҰP=!w$(w#*HXExV' q A&I0 J'$0&6"2W6(& 0o G K(]` = :a D! ߐŔ |)~K#4p IpCqŰ#Ҁx8@-؇.|'pAq2c 2 w*t1T=#`QP,H:Ap uhvOn?a*`! aBgȋi=q1ڠ\%`w8' x ~t҅5!P"" -_ U6De1 6$ Xx pn)HMm:F -*_y&yj1 :/P yG]ׄ @qGu2g sy;K1a R9 Rst(gU Q0 r " *?)[&] )ИՐb9=f:7blYB)(@>P9YiU1 l"5Ydt ‰ ȗ G2R 2հG a?3q0Cv::2 *t >R 3'P 1 " " fym S(B 8%G䞄>, h0 t ܒؠXzq ܑ ͢+ r 4p!=ubb&ę4ܹ=]FE0(Y 4z(`" X`i3q>u( **й#؜4? ^H Wv$Be  o8 2I _W3r hB s L`T'r-{@5H x'a*pF*y) @2zT0 H^ƫ$!@ P ,4 P2QVK( Ƣ _y K`ڰUzt p*@(0uL&O1 PLQ Xq,~pc y.MCjtNddQ* v }g*hF2 q<`FLЬ Q;+*E*OPAa o אh+g0{Je) T6ި0@j/ũ + 2!l_=@foT C^J<0)H8 ΀*P R{!&ް hEא."p q !Q ~Ѝ>@ɑln x@2[DEz;+ Tٔ[E+jm) ݡY*${h i4 ހ ͒,2Ut*P{ h&{ --Y';"P@ph&"C{&s@ $rV bVL1 Ā*>9 a'V/PL @1 py|DIpW@4YaPkl@ ( 3` \` J~k)Lc0AݰF*")' p ,G/B W~*pX)'1_8;`p~ Q͕Ta0 Icexj`R! \  R X0L*߸/:P@0@PP*@QAĐK'Ēہ0b *TFe)4q(MJ*(M/2gyي 9LJI9'*I rK,] U'@DQh= -0@n%&-O`w a"&pj l%$M@Y1 ;@a "?PpP )P߀;G@ q@9;ͧHkp^9Pj!<֪*\$7ف `ִ$X#gE2圀P p uu 1 gu2X3cB3 esMu*@kv{'@8\d 9݀ 000Ӏ=~U+hw^ Τp^=ه^* ˣ㑉1 2JP!]g <TxX 74bA&{9^y0`bB>" 5t )E3 @!ax@@'—!]φ<5~^hd! '8}&0*:"|0`፨N j `d`i: XV^el.<'"y}$2`b@!R̀90"])TN - Pvm\2Ur +Ґ)P@?p>ѐPn1l JӠ# n'䀕 DX/zfR ǸǸF)' &ǘmþ ]I VG?U0,hDQ o0/oP0q H?/U ܓT * 1Sґ /'P7p Y Tޑo?P7u1ӯP7Y2{ȀJ DP 6tD V8Qŋ9vLH1F"?fD5ԤU05 cYMԦ-Pˆ>x$yr EŚUVPQEAؤ'I"Vm-jpZuś^}")1A '%P_NsQӚI4\4Ehҥ{NOfвNVùm߮nu=xNន<kjRČƜy-lc)6LuZ8O60  ءW;|]V^kVsp䚞h$+BAh8%C ,H%" C+ƭ>D_> k z:k lpǦaGXȮh <4h)`MϭZL13MkʚTk)%$`qF.&z2Ѭ)34РRJ@L5trSO9:~y. lzd"F*-+9J|bZJE= r8#MH{-gm/O=}gZk Ub@PԉJZbRӪ ovD%6$ tYޠ!`&BkFlj%iZkZx`fWRx" D _ba{ @Yx 3N:ѕjt ahBR/4؀CL`NV&sC\@:KH@GKgʑv/L?$u5ceħVtwI8K#M Ad% R.#V66 P&`{~>yBN#ԎEF!N#~h [:ֵ~@\YG7?]6pDT' BHi܃VKp4׹%K_KpL.c&a: 0Nŗ2bsqW LyEҵ.Y._d-F#b dH:(ᒻ7bI[5ZW&.ſA€y2 <89D*nU s#& !ڗ֖)Zҍ"T3ķC:W* d>܊>dX;ńx7YgX",$ލld!&Xh1hX8F$=,*֘w[BFP/iC28qckK8XWUȎE8Ec?D9oаU4kxY 4>7"(TX :_0(9"˔h j꺤=-?crܸr(FzǔxY&Lru4t=Ag |zlHyqM0}v1sI"%s< t0QFr* 4z?#y:*k0 ~>ssYRkځ^}MҠk@#9Nsh9KH/h߸sh%Rp Usl׹G`0膡[: 06;+{"ʑ5"C[a#aӡZOoSlH  h=t+56؁U )&QN H 2?&L?C{P]Hø#– HEpijȂw+GhP@Hp1';Hl+5H(ƀ J@>3xizOp ](g\(F]j {{Rcbl(+hcZ,-v@Z-ظxJƙH{(B*qQ 8rrzV8CwCȀȋȌ,CGGh7a R8szs mk#):Ǚl[(ᮚ hs+PW) q`JqNhJieKkQOh"C0Ml p`# Ik ˺ i =#sʴ'L@jPKBf:(qp!-x>ɨu(rOl/FR/-E^&#b+`ҁ+Ԃk6,6G06J}EjL 0qI9ĠS@#VgͽJ|Hi X*l)ΐ N! p0]4i@rM&,#H(&H5s{TP?cqQ42O8Hia ؃:vG[- ZS^0 i83)"Cᬕ1rr0|!T0: B Ȃ@jQ:jقQ,X4'ɸN iclKPZh( eL3xSpK\b'e-1GomMHn1SpKjP|cPi];sRPj( U[u.7 it Vt>5EڸW}XK HgMzXΪ" vؾD &7<2^%+PWFkSEm`OT|-b9זh`MKlsӇ`DCGCGC7`@e[XO%iP׿7xsFj sK ]k 9}7|= vb?PI^Y\X\:܄Xi7^yDF)d'ވ77/8NpTe*Ȁx(\PUhvhENH870hӨr E_Uxo޴D~{9j]wڽ8FhN4X-HZr fXMzv (M f$J0̓6 }8֞`X)܊eҋx~ۋ{10ℹؑ uD@Ip^J %״zɄd~)L3h%84PR“0u\ͽ8Y`t$ #+:ЃFJȇD 2.0je-8@-$0XOqSH8䴥+pT gE\ ,C6\x&Ǹ+o19m*#ɨ p9[PS ]7dVp<.fSX`ُU }xR5^1lȁ$vXNhx=p J @zƋ{(LЂ`<``V$h0hix% v.Df"DDhGD0jP r(..h̀a,Vpvl l?(-ekh 8،kl=dXjႸ1m*bXЃl`nXw>dn~ЋeKboYwA.Յ9.K4h`w uNŦF&5]b8@ƾfPl9Ї$F\E 8PjHOOqZ " o H~~e/)n,r`VsU•l xn0jF `ljsƋ=O{ 2\Xa` 7h#X8FBqj8IN>6ph3n / E9<@Pl" י\^$>gp)DB,ƵW p })%Ӎzà 'AKgCp%8>0`ݻrvw8 (@0bx *WA}&305jAj7$v$ʅPtJhxh njN]5a'қЮI7Kiu) G+X$ Ӂ( Y gj)ӴzROCvf*`< +g5(df&@~H\ %0a+,@3t؍J&M4v(w|& $G{abϧY(m܅ta5i"OuUV7^g"ndĎ~jGJ;78z?5#2$)Z љJ]_zӜQM)H[1nTBc!:ar4Ȑ䄲Q$X|?[^_f2, XsOLi"liJ{:*@(N[-Y+ )}8(DȲ!5U8&^WRF^yϜNS5X2_ޅ'2[{u~ז[8cuq !Ǒ҄-KR,MbxJ@Ӑ=?">*8\sn47 $#W5*Hx@Mn4PLpfM<<+;HKO^sD< TYлRArULqtD&B+\f7C28S'?Jя!^D#oa8 30b3`t O؋8Dp e)d8N!xz%\@ Pg]`>B*Kn@k @&P" pİ4M7Сn&*:1-&Z%S[/ d~DZ +%vH0J=P!e :,bì`jc#T;+āo4RɐAv "jD|A LihcDWAR_$Xd{f`PNR#-5ЏTI݋4^)BxEF2K\jb}%ũJ̵s`;,%i\łz Xb``$1 4 (5Pf8i!V/&_/m ~)s_p A~DCRe!`qt o@:%l4q-lT#@q^q>r.>@O"8"KKј!eMvaTO 6" s`A>F``'cN).H:pD۪4XPVzi# 2;R^3&!!K0m|!zڴ68O@Ҙgk,({f8zd^DuzgePPf m!ym# =& bAM?Ptbd:c/!O:<7{ cpĜ 0d2agM#N4 )퓇ޛU0, yh ChQxe T>׭  5I>_(ݞt _q2MxdU: H8y߀ $  5F"ְL%f\L6pN NC7pM@I7 |堡AŞRG%^"-v4|B]9'^?< "J: zB L@ OCxyE&$,Ң%J;0\9c9:LipM )`4cP]#d6-/Ԛ-,p[BjM89 6<`) =f>!$68iB)$[fFPC52͞}B.d=&ma5EBcedj9\LA7h 9|&hJhiEU'La>9-[dN&5D7p.>,>T'Aojo2Fpuyyޥa<UE4$efO`ǽSE8pa/EWcXr]xx_5|&(&'LQN@>h'|B!&|(&4B(&T:4$ HI!2$4xB|=h]$(sT)(|f6(&pH%_g5#,#0#4Nqi%E1 XR ۭ\{28`(\480N 'i@7䘩2> ~4[)14A*TH7|F 8V4i4)ipMH3$~&|X̂C#%X#%g*F~Й9-M^\]ሩB^5d@;a$J)*]bꛕG5jHkH3D$ iqBr_l+C '-5`C.-߾_>;ZY*% >^DX.<@L(! ܃[ťȉ-D>A&5~lib,rX96+WR^^\"CdzQS.4$k@&E6 >m4\gD3xk%1CC@n5(HxA5CFFA#v]%k$6 ՝Fr-H"PО|ø@6x5`5qP-W.,傰ah0hjHxnJ8%d%Fz64dDK7-B9|~^mz@8W(BetBPBoiЀ4(ÿb oRpaf cf8Ec 4`o|\h]Bp`:@ː0]DB6(q\)hX*]%(Rn ?C.v É4P`4ht'@*"E>AT_}M4^ԙo0u$D zNeI~M"|Bʃ<-4_$pt09Ha1|'D$:i.M!^9/ q30_4 F3is=EM/(Q XgDC耋]x@˃ 00Cw{I1K\CHG\&CVk 'ژI VM74躷14k:z]MusZED2,!N0@]ȣPK";+H[j$ká',_5NC{~5 fw 0vzJT:g0^$q4D UN\6?Tĉ'<.h2`.C:h(KC1pBT+hpöLB+$?.?::B@C<H|dB{\*xB^?w_ kW/;@X4j&T!i8bE1fԸcljy9dIϜTeG\ZxZC )0G4i7_f( %dF:jUWfRHLmJ[7l`т$қh:Օn]wUůpD6ͺoNq$-/|oe˗1g7"p:Pi(kMQOٷ-m3fπiIikt,cYqӮ5WtfիĞhB֔Av ~}Yvl5%Ai(f CˠeF!P *o)il!K,T**lKʙUiaܺ$F_-ǁ !^t` O4L ӈ5b$$3 #≻S0DgLi~eh)8x0TYw9+ AVyb5Zh6x}i2vdy}wn-3eǑ뫾 hqioQbLi9m}U Vip4̐ǫJ_F'3턑ZɼX[6z=i,]Wة!Xdmʦl_hDDaOCasːX塽F74s$o`7TO3N #þw}E޲- `8nL W>@ Cu"'s!_4xƒ, ]~W6Tm:d/m[:#.d'Fq?9a‘7A h  =}l,HE.t#CB%1IM6'AJQf4)Qt+aKYΒ-qK]/LaM)N,eBFM NwAڤfjM"H D&t1{BS=i q"If橁^ @($Q z1oR]4'1Rd@BS)IeRaH Ftf4M]S.$EZB0UB"EQ+T-AjF +@K*ԮV`hU/zl~L"M̸V&QM c@=@%Z^&B? M5Z,TV}Q6(4?d@ 'գ =X[ Vmq[G[V "bqV^e@sN /˅ue]nZwr8L h@FЁ"ȀЁ 8O"~(@6;@B-%80G.$=R6qZH8j-(+D19xD D_x$IHH9gWఇ EP"pG8T,vG\,lr`"eű$D><NFDDȡ +D6c0%Q$@_5"liTaQdg)wv" n=F?sI)rh4E<}cp)6SZ"g5j8˹9C6'#4I &%A0G  DܙՌmhAx7_,!H$M P;3X4z |s"D*:3Xa?`c I R5(8ΠWAo?.c *, O]UQԱ^SE2=uA9rs/ˆw& ;gpml-matlab-v4.0-2016-10-19/doc/f1.gif000644 000765 000024 00000011576 12701705600 016631 0ustar00carlstaff000000 000000 GIF89a? """***---111333555888AAADDDFFFKKKSSSUUUYYY___```dddfffhhhwww~~~88HHXXllhhppttwwڊ磣!,pH,Ȥrl:ШtJZجvzxL.zn|N~  uIT..SFDNOtBIKsK.+JrzS"pl<X-ŋ&:a!Ə ilj8@ĸH&%ݜ IKbiO9nA^ػO< w@~\߁ڇ FXZ@xah Gb'fa(>R"'"+^43*"`;;&7VcD$CGNPѤOa;W% :3:_~xΔQTo hOe鄜MI_ʄ&K U[ө6yXA٦})X ì Q*g=jYfpvn0:Ĭ 4(}%j),X m0-ET0DZüBK < SpOw\1{|l ,r*ղ 4P>+*PV-mIn0+yڍ`}pR@uм邰cc )V x'zAJƊ4`iD#nސ{14mGNGTn9N;zU{x5X cyn_$-Ͷ.e !Ժ'b7gz`>_h;QףN/}M>/ SADPF)Q AuUoLGxoB<*ئj D?kaאr(|ab(üaXvcnF 8%n~yh2ZN]q9D71FeBbB4F]d#E9JWcH )N #eb.!e0@ Є")baRj`P l^±KH C U`L Ί/e Ph\ `+_(NnI8m$Y%rh5 -^j *ЇV#lN\fQ@.mJWI?*`A[KPv;;lg Dk*o%B{ WP`#bHq)`%$K[%,oYׯAq&S$V:7|{|)y0>Έ!abƈP{I.Z承a C[WW ߌq<+ T1C @[e ,bsOB*v^LJ_,$-=BrD|8A _!B7 =8{vS сߪ)^!QMupI/B׎fbLfC4#R V jWr|l$ˮP{Y0BܐhCh>k'1gJ/TmѶ¹mI'9lwi$ۺY63o2X?܊u$~784hYl>{~x0g7&'xw`p8m;,Ѱ\ƶ~AGH\NDYGIι4 np]AOur1RF6CuK?[ʰt^1Q?Ws,W k.9~Xjoj`Og{j֫(t-sIGNÿ+\A@orV .3[ׯ=k* M  6t8*_Wxq+J]֯Uku\~/ߺb1PD`q@!7~eu88Oz}FB OQfEkzZyħ3wIV0,mVvRq z[ #1}O@a'GP:ǃFqH+ HχmgG}(f}IFLQ8C| td8fVw!-y%넂2xivx?HE.`|zQh<ЇFĈE`wE8kHJh${XJfZCaqBj`P0z g<68hH(E8@!Do@eʨ#twg@͈8hMD؃Ƙb@騆@n*;k:qpv8@8۸l}~0V j {6Xd=#э~YzDmYB<}'2KGx3-^>0  < dqd%IsJv-[U9Seyg{bm]F1(VY ҭ7PM/}B8\T-}DLm5]s ]ce=϶Q)^mSʢ`=zP&=4%,$'E \T[,*PO@EMF}_sHEچȬȱ =gSۜpͺ} ^bۍͺ9EA;gpml-matlab-v4.0-2016-10-19/doc/demoMinimize.m000644 000765 000024 00000001616 12777177044 020454 0ustar00carlstaff000000 000000 clear all, close all, nit = 50; % tabular rasa, max number of fun/grad evals n = 300; x = 10*sort(rand(n,1)); % sample data sn = 0.2; y = sin(x)+sqrt(x) + randn(n,1)*sn; cov = {@covSEiso}; sf = 1; ell = 0.4; hyp0.cov = log([ell;sf]); % covariance mean = {@meanSum,{@meanLinear,@meanConst}}; hyp0.mean = [0.2;1]; % mean lik = 'likGauss'; hyp0.lik = log(sn); inf = 'infExact'; % likelihood/inference par = {inf, mean, cov, lik, x, y}; [x1 fx1 c1] = minimize( hyp0,@gp,-nit,par{:}); [x2 fx2 c2] = minimize_minfunc(hyp0,@gp,-nit,par{:}); try [x3 fx3 c3] = minimize_lbfgsb(hyp0,@gp,-nit,par{:}); catch x3 = hyp0; fx3 = gp(hyp0,par{:}); c3 = 0; end fprintf('nit=%02d: minimize: %1.2f\n',c1,fx1(end)) fprintf('nit=%02d: minimize_minfunc: %1.2f\n',c2,fx2(end)) fprintf('nit=%02d: minimize_lbfgsb: %1.2f\n',c3,fx3(end))gpml-matlab-v4.0-2016-10-19/doc/f2.gif000644 000765 000024 00000035352 12701705600 016630 0ustar00carlstaff000000 000000 GIF89a !!!"""***---///000111333555888;;;<<<>>>???@@@AAADDDIIIKKKLLLMMMNNNOOOPPPQQQRRRSSSUUUYYYZZZ[[[\\\]]]^^^___```aaabbbcccdddeeefffhhhiiijjjkkklllmmmooohhxpppssstttuuuvvvwwwxxxzzz~~~kkQQUU !!'' ""//2211447744>>??BBFFHHIIMMRRUU\\aacc``jjkkppvvzz}}Á파헗읝뙙뫫鹹!,E H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵW3LKٳhӪ]˶[ "Kݻo2`0V.޿ AnŸǐ#K0$G9cׄA>SLH$CiXLא;!AH3gϦ 4oC>qɞ 2CW{:Cg:ν{ٙmlu_h^9/ ɜ6 =P 1D|%(YfG~3'G *cnR}9_$H҅aH7ϙ(e⁊8r ReB2Q&mk$!TVP&-"H\ڏgd Zr׉lrmv(RY3bf ɗ1B X(Fg"Ӕh$RJnlv)fa#H(Q0A:֩ff 뮿@ ~+oF,FD!dlw $~ǯ&m}jٖkU(spۭ|,zba+ULݹDljj$ܐ{lnV'4 '(>kBǫ/՛yIܑqkR,9w YXF7Kvn{v#~Ý>%XZ4y*ʹm  cT#z#ƙ}/?Y*]5νqr!seO0 c4 R yoBz 5;A d@1`W'⃠kG"8‚q"̷d`V!_J D1| AP|ā(їsB.2Dstx)ð8Ch ` MT 7 8df?"~^Ip<̍ ɠJnCi36$U(G .BV.QٛA!)C$ Ich-Buͤ&dUąa%I/ǝ89d`DžcLL遖| \'RHϕi ́2Cܬ[E931%g{dA`B+ (P (Pt<7 1A1*`+'S hXM(. SSD.2uXfCzd@BV1M%N,.#L$U"LX(# Qhid7#$؁aX$"CM2NƲAJE9Z("$))eWʮĩ0@!6Y(]2{DT{঵ %`f;PH޵`w;^񮕼5(k7]/z^ͽo~_w kdw5p_‡zse@–n zN! kUJ(\N撩}P*9Rw!O&%ȏcDt$Wr2|2I-$ʢ2-PzWb}uCR1g,9 ?fv%ik *fdeK n&zd"k (gD/0 @ fMG!,$▮!uilat2lCAzj mSײ]S281?dA$mudwUqrI*uMr|0tMV;#kAٞag_"A-l`R 9tE:N;q"O ]{M~&Y6rQ\UC7vDDsW\SψAo 'l^[aH@-oy[3:L%0k>GIxkؘM]&g`;>u!Sv^<{DO]o)"-N ,] !sm_~MnؾZ.OAa߷INyOt/d[AQz ޹ЅM7Z9d'.ĉMHYy2 -Glq{v0\>}5D }`RuQ n0HS!YU|A`h 2XwOzZ1pS0Y]z' HHV #'l79 )|#x|f1D !w33}sWFE0xQ%@:I~űBXnlx)rg0s:mqWLYqQ-qaH:}Ew炓>na XIU3ExQֱR/AH'}rC@rjzAsHyZ~wuw1qKaF,HS-NG$vmׇln8JfCz`*qmIօp6Y7Bt9 F'Q1'|88m !+ уĸ/ 4&5B ZD Y o5h'R|Qp@S8}-H#LOdhaa 2>I8OHw4 s!,WtwS`B]A,rZ s` %,!x+qAq~@W~a82:UO9٩QtD80V2KS j0%&)PB&ѳq[4Oǁ ixP L 92?VᵍQ tdBYMK0SwRJ* >ƫKb Њ>(,4'lYIAd`"b [֥ٺj, 0e0hֻb#~6Yj뼮1KM,K vUK}F c"˶fx1kM:(*9¾0K|emyS"Vvfy{a6"$ȢJ?6%, ZҎǡb5fU0ĊSV~&řRŝň!o. \Kr r؋k-T`Cq,Ǚiz;3hZ%CdA1d+#aPk*kSQlK  EX)(CiRU0Zt)l'@L C,9 'qZ%Ubc q #9g(l*YZ:Dh0kk'ṭͫ(#1X8#&a.|KOa m-_M?WhY}ֿʴbӐn5M'~֭asD¢|bW ar=R8 DiYP1"Ϗ-B2"qi߄ٷ948$-+Y ,nm;)"dv}0IOŴ A}#-$֠Ő1aɁB6IG,*|^m\&,LL7 3M9kT R$-Y; ,Ps W W 2 p0Y0eld: EZD~ϡԭ1 "d+OϩmN[]Xq qẠ--ZfQf AZ; Il+=AK?f]cdH 3-aV3#S*Lۋ`i>IOމP:-RSؘ1򞀤 U}}uD0oԁr#nm) ˍAQ=?b7bP"09R1xhxHL3.QNѮ"= =3m L\ A$#|.vM3*;M@-W|S<1]n]j G؄J \2 d)L鐄Aa+- 23 c}-CYFcez N5'90GڒqGD#k4u/A*41SOdIpvz\:eMsT0zH#>MQ&톝6B 4/$&s-C :"  ne6Cf~s4H(-O\6MQ @ DPaCAD=QD'T@F `IqF%M0J-]S̕38 JI4Yd' $1ȡ'>U) JM^'O~8X% Z`StBm!s 7葨`RFZ^xpǕ-_~,x $h)dqb&Ғ]~"쁔 eGF vBL*DG2TkPHPRr62h⌠Ö=1 )$@Ǐl!7+9/I43), 6Q0JsP2;/PԋB)d~j#LH02?$N@LkC{h!H|p؈$,5\ $*|oD$ zRŊ@Բ-=J*j 5Zh) d@6$+)tI(+8>tKG- @+N61[)HD ŃE%&hDSuŵ9RCh1;|$<=ܬT=%]u"YCV%sPlB(Vex:@H\tZjMq A`P nkFÍI;dTi-.0Ҳ VD?{ٽ%?p8,JQ' 譍Pcxc )e͍@9d#$5-4А@+jEnZSK^I#0>us&#z%SAT83dOzdcAVNhJ??N$s ^2_yRb*QC3JpC0 2H87)%ȶZNJO3@ZIP FC)ؠSF."Xӌ娞d{!LX#bBB\hm=D I"H7Dqi~!<pfb s  `$vVƒi\#GGӍ[ZB2Ad'''29,, L",aQC9yC3SUCS+R9LA;ZuzAE$(K"5e~#KrȻ}%A 2=v]W94 $w'I'+q|6ӶpTӒJL&y >ڤI *\ByvDc% ˀ&dStU$0 ZBR L0p l3e. I/3h9$uRRshdI$Ҧ0?,^.V<Ip*ȓP:q$d(HTB6dq$mS.ID9QVӒ`V׃8y ɺ) Y|bSe-x +P`q4Q,^Ϣp/1;؁neRX,"b!/B[|xJX7&%iM[ɢWqD2 qA!hAR H-/} U#c~$]dLGN pA@r5H˕o6-);&p3U;lЦ{2X0IiOaHhkXD@$$NΆ c@R&ky&CPoXHe" գIbM- z ܈"pCqAe g8H[K$@iX$D2M5"Ie`*Si3a8CWR#ǁ4W7U&8:l[HLRv,YS\ML`i0,^{ `=qY  rϗ$цx%I@i!< g`ƛo}JMD˧d &58D( q0'&UqCbݢ8~),*.yBYso2rLZ.:]<'+$sCҦXY$}eR:EiKM&[H V]n CvVY7ʋ"X>)E rLu &M_鳗};O·byZݐ*pO^=m?y<3^I&42sY#tⷳxBӝ&)>2m&܁4C g]*O;[V;p;+1(p[ #s r KJ?Q5- c72 K 68rc7389+Zg:YÏ8 +H3-@*2 6 Lkp* + ;  8d )$XzA/< AA4C;!Zy4-D; *yz"+lc7 P QPB4DyC(A@P\z# lꔫHB83@>DAPMȸ#zʠCN0X>>{]|)EГ N4 CʵG)AH =TF7؋ op|9G  ,ĈO" @`#P4P,|2rNFc `HEU<  =#F3S[Is|{60,A GPA $ Ès:#ȕDK7ěD%~!@bԫ%"II]ˮHLXǀңKĘڵ)$k Nh͔ !l6Nh?pM (M=QJX: \ ̀JƇK; d >H ,@-! t J" "Ő8y+ قHpNk ˕وχh ߑtMeLM}JDIJ5c.4ϐNuMG9Y"АpKˆU~NNG)iH`4]s=jU@ eS2 v^`+{V19iq\nhH$hpxGGf(l(LXBzwzvu@r@Jpt'WsoQ<7Un+pG2g⫐r x1DDnkwm]4FM(#% ,*CJ9[raXppwdZ>g?xb/7[NO"wXf\,J`zGIo&rz`Z?=ǃĕS!|f(wϴJ_M0km>zCuM"ZM =` u q‰뚯XO# 'ʐ4 ,z]7`O9xn(A|׫tϕ#J7՝B5X[  aDoy7L˭\'PN ?Ar|tM2;Wo-@AD,h „ 2la&Rh"ƌ7rh "GB@UhW!DHI(yz,@/M$3ȄWm'#=Y4ᛕh.4!ׄL/aJhj:s\:MOɧ1>44H]QTEBQBL)MHͪQHv4|HPD] *!Rn*1G%Fn' 9|hG(yjs@g1;J#)grخxH { (U1 A7P9l5 YX6+f$UA.A.кsCS!I /t/#]!(AFԠPq-fOrE/cQ"h\K¿ixEbNoYt$ޢdaA垰ěqf9,U$ hI; '"*ں̄6_įїAhcjM̰yp Np+A UFE*_??gH*-?ora|`Ӂ{_iԓ`)`1`w A``[JRMhViM ĝ Z >} .D%Nm'Y!B|[ ։K9ҷ`O^!@IQaXa/E_>1 _",_A N1 " a!!*!&" #b 2!."R%Zb&n&*b$!&&b$j$"""*rb)z"&^*R+"b%")"$,6"'2b,b/0"A#2&2.#363>#4F4N#5V5^#6f6n#7v7~#88cZ9njji;#eB 멆1?#Ņ$!A(UIBAaTB` СAEDN]d!&dFdN&eVe^&f6af&gRfgvhZhi YijNH(D#(B ?TRܤ&qDkV#؁hD%%P@xaz wZ%AZeZX#wN'dOjIw'J穄B sngYV}XDJ@m(u (Jc*@w"h}'~c̀ GZ hUpAxB愒GhqN'h{(I0 (2(>h H HhtO''ezFza'hBǨ $vhth>mJ(At6P AEH2 8JtD!\kNZȢ KjGF*:DjV^Ǫf*Dp֪)i *+RH;gpml-matlab-v4.0-2016-10-19/doc/usageSampling.m000644 000765 000024 00000005114 12701705600 020600 0ustar00carlstaff000000 000000 clear all, close all n1=10; n2=5; % number of data points from each class n1/n2 sc=1; % change the scale of the initial problem % 1) draw data from two Gaussians S1 = eye(2); S2 = [1 0.95; 0.95 1]; m1 = [0.75; 0]; m2 = [-0.75; 0]; randn('seed',17); rand('seed',17) % coordinates x1 = chol(S1)'*randn(2,n1)+repmat(m1,1,n1); x2 = chol(S2)'*randn(2,n2)+repmat(m2,1,n2); x1(:,5) = x1(:,2); x1(:,1) = x1(:,4); x2(:,1) = x2(:,2); % rank deficiency in K xtr = [x1 x2]'; % class labels ytr = [repmat(-1,[1,n1]) repmat(1,[1,n2])]'; tt = (-5:.3:5)*sc; [t1 t2] = meshgrid(tt,tt); x1 = x1*sc; x2=x2*sc; xtr=xtr*sc; m1=m1*sc; m2=m2*sc; t = [t1(:) t2(:)]; z1 = exp(-sum((t-repmat(m1',length(t),1))*inv(S1).*(t-repmat(m1',length(t),1)),2)/2); z2 = exp(-sum((t-repmat(m2',length(t),1))*inv(S2).*(t-repmat(m2',length(t),1)),2)/2); xte = t; clear t xtr(end,:)=[2.9,2.3]; ytr(end)=-1; % modif % 2) set GP parameters cov = {@covSum,{@covSEiso,@covNoise}}; hyp.cov = [0; 2; -Inf]; % ell,sf,sn lik = {'likLogistic'}; hyp.lik = []; % likLogistic or likErf mn = {'meanZero'}; hyp.mean = []; inf = 'infEP'; % 3) predict using approximated inference [nlZ,dnlZ,post] = gp(hyp, inf, mn, cov, lik, xtr, ytr); [ymu,ys2,fmu,fs2,junk,post] = gp(hyp,inf,mn,cov,lik,xtr,post,xte); % 4) prediction using MCMC sampling % set MCMC parameters, see some more details in inf/infMCMC.m % We have two samplers implemented, namely % hmc - Hybrid Monte Carlo, and % ess - Elliptical Slice Sampling. par.sampler = 'hmc'; par.Nsample = 20; % par.sampler = 'ess'; % par.Nais = 5; par.Nsample = 200; par.Nskip = 100; tic [posts,nlZs,dnlZs] = infMCMC(hyp,mn,cov,lik,xtr,ytr,par); toc [ymus,ys2s,fmus,fs2s,junk,posts] = gp(hyp,@infMCMC,mn,cov,lik,xtr,posts,xte); % 5a) echo results fprintf('nlZ-EP=%f, nlZ-AIS=%f\n', nlZ, nlZs) fprintf('acceptance rate (MCMC) = %1.2f%%\n',100*posts.acceptance_rate_MCMC) for r=1:length(posts.acceptance_rate_AIS) fprintf('acceptance rate (AIS) = %1.2f%%\n',100*posts.acceptance_rate_AIS(r)) end % 5b) print results figure subplot(221) plot([-12,12],[-12,12],'r'), hold on, plot(fmus,fmu,'k.'), title('\mu_f') xlabel('MCMC'), ylabel(inf) subplot(222) plot([0,10],[0,10],'r'), hold on, plot(sqrt(fs2s),sqrt(fs2),'k.') title('\sigma_f') xlabel('MCMC'), ylabel(inf) subplot(223) plot([-1,1],[-1,1],'r'), hold on, plot(ymus,ymu,'k.'), title('\mu_y') xlabel('MCMC'), ylabel(inf) subplot(224) plot([0,1],[0,1],'r'), hold on, plot(sqrt(ys2s),sqrt(ys2),'k.') title('\sigma_y') xlabel('MCMC'), ylabel(inf)gpml-matlab-v4.0-2016-10-19/doc/demoClassification.m000644 000765 000024 00000011024 12777177044 021620 0ustar00carlstaff000000 000000 disp('See http://www.gaussianprocess.org/gpml/code/matlab/doc/ for details.') disp('Hit any key to continue...'); pause disp(' '); disp('clear all, close all') clear all, close all write_fig = 0; disp(' ') disp('n1 = 80; n2 = 40; % number of data points from each class') n1 = 80; n2 = 40; disp('S1 = eye(2); S2 = [1 0.95; 0.95 1]; % the two covariance matrices') S1 = eye(2); S2 = [1 0.95; 0.95 1]; disp('m1 = [0.75; 0]; m2 = [-0.75; 0]; % the two means') m1 = [0.75; 0]; m2 = [-0.75; 0]; disp(' ') disp('x1 = bsxfun(@plus, chol(S1)''*gpml_randn(0.2, 2, n1), m1);') x1 = bsxfun(@plus, chol(S1)'*gpml_randn(0.2, 2, n1), m1); disp('x2 = bsxfun(@plus, chol(S2)''*gpml_randn(0.3, 2, n2), m2);') x2 = bsxfun(@plus, chol(S2)'*gpml_randn(0.3, 2, n2), m2); disp(' ') disp('x = [x1 x2]''; y = [-ones(1,n1) ones(1,n2)]'';') x = [x1 x2]'; y = [-ones(1,n1) ones(1,n2)]'; figure(6) disp('plot(x1(1,:), x1(2,:), ''b+''); hold on;'); plot(x1(1,:), x1(2,:), 'b+', 'MarkerSize', 12); hold on disp('plot(x2(1,:), x2(2,:), ''r+'');'); plot(x2(1,:), x2(2,:), 'r+', 'MarkerSize', 12); disp(' ') disp('[t1 t2] = meshgrid(-4:0.1:4,-4:0.1:4);') [t1 t2] = meshgrid(-4:0.1:4,-4:0.1:4); disp('t = [t1(:) t2(:)]; n = length(t); % these are the test inputs') t = [t1(:) t2(:)]; n = length(t); % these are the test inputs disp('tmm = bsxfun(@minus, t, m1'');') tmm = bsxfun(@minus, t, m1'); disp('p1 = n1*exp(-sum(tmm*inv(S1).*tmm/2,2))/sqrt(det(S1));') p1 = n1*exp(-sum(tmm*inv(S1).*tmm/2,2))/sqrt(det(S1)); disp('tmm = bsxfun(@minus, t, m2'');') tmm = bsxfun(@minus, t, m2'); disp('p2 = n2*exp(-sum(tmm*inv(S2).*tmm/2,2))/sqrt(det(S2));') p2 = n2*exp(-sum(tmm*inv(S2).*tmm/2,2))/sqrt(det(S2)); set(gca, 'FontSize', 24) disp('contour(t1, t2, reshape(p2./(p1+p2), size(t1)), [0.1:0.1:0.9])') contour(t1, t2, reshape(p2./(p1+p2), size(t1)), [0.1:0.1:0.9]) [c h] = contour(t1, t2, reshape(p2./(p1+p2), size(t1)), [0.5 0.5]); set(h, 'LineWidth', 2) colorbar grid axis([-4 4 -4 4]) if write_fig, print -depsc f6.eps; end disp(' '); disp('Hit any key to continue...'); pause disp(' ') disp('meanfunc = @meanConst; hyp.mean = 0;') meanfunc = @meanConst; hyp.mean = 0; disp('covfunc = @covSEard; hyp.cov = log([1 1 1]);') covfunc = @covSEard; hyp.cov = log([1 1 1]); disp('likfunc = @likErf;') likfunc = @likErf; disp(' ') disp('hyp = minimize(hyp, @gp, -40, @infEP, meanfunc, covfunc, likfunc, x, y);') hyp = minimize(hyp, @gp, -40, @infEP, meanfunc, covfunc, likfunc, x, y); disp('[a b c d lp] = gp(hyp, @infEP, meanfunc, covfunc, likfunc, x, y, t, ones(n, 1));') [a b c d lp] = gp(hyp, @infEP, meanfunc, covfunc, likfunc, x, y, t, ones(n,1)); disp(' ') figure(7) set(gca, 'FontSize', 24) disp('plot(x1(1,:), x1(2,:), ''b+''); hold on') plot(x1(1,:), x1(2,:), 'b+', 'MarkerSize', 12); hold on disp('plot(x2(1,:), x2(2,:), ''r+'')') plot(x2(1,:), x2(2,:), 'r+', 'MarkerSize', 12) disp('contour(t1, t2, reshape(exp(lp), size(t1)), 0.1:0.1:0.9)') contour(t1, t2, reshape(exp(lp), size(t1)), 0.1:0.1:0.9) [c h] = contour(t1, t2, reshape(exp(lp), size(t1)), [0.5 0.5]); set(h, 'LineWidth', 2) colorbar grid axis([-4 4 -4 4]) if write_fig, print -depsc f7.eps; end disp(' '); disp('Hit any key to continue...'); pause disp('large scale classification using the FITC approximation') disp('[u1,u2] = meshgrid(linspace(-2,2,5)); u = [u1(:),u2(:)];') [u1,u2] = meshgrid(linspace(-2,2,5)); u = [u1(:),u2(:)]; clear u1; clear u2 disp('nu = size(u,1);') nu = size(u,1); disp('covfuncF = {@apxSparse, {covfunc}, u};') covfuncF = {@apxSparse, {covfunc}, u}; disp('inffunc = @infLaplace;') inffunc = @infLaplace; disp('hyp = minimize(hyp, @gp, -40, inffunc, meanfunc, covfuncF, likfunc, x, y);') hyp = minimize(hyp, @gp, -40, inffunc, meanfunc, covfuncF, likfunc, x, y); disp('[a b c d lp] = gp(hyp, inffunc, meanfunc, covfuncF, likfunc, x, y, t, ones(n,1));') [a b c d lp] = gp(hyp, inffunc, meanfunc, covfuncF, likfunc, x, y, t, ones(n,1)); disp(' ') figure(8) set(gca, 'FontSize', 24) disp('plot(x1(1,:), x1(2,:), ''b+''); hold on') plot(x1(1,:), x1(2,:), 'b+', 'MarkerSize', 12); hold on disp('plot(x2(1,:), x2(2,:), ''r+'')') plot(x2(1,:), x2(2,:), 'r+', 'MarkerSize', 12) disp('contour(t1, t2, reshape(exp(lp), size(t1)), 0.1:0.1:0.9)') contour(t1, t2, reshape(exp(lp), size(t1)), 0.1:0.1:0.9) [c h] = contour(t1, t2, reshape(exp(lp), size(t1)), [0.5 0.5]); set(h, 'LineWidth', 2) disp('plot(u(:,1),u(:,2),''ko'', ''MarkerSize'', 12)') plot(u(:,1),u(:,2),'ko', 'MarkerSize', 12) colorbar grid axis([-4 4 -4 4]) if write_fig, print -depsc f8.eps; end disp(' ')gpml-matlab-v4.0-2016-10-19/doc/f3.gif000644 000765 000024 00000033052 12701705600 016624 0ustar00carlstaff000000 000000 GIF89a !!!"""&&&(((***---...///000111222333555888:::<<<>>>???@@@AAADDDKKKMMMNNNOOOPPPQQQRRRSSSUUUYYY[[[\\\]]]^^^___```aaabbbcccdddfffggnhhhiiijjjkkklllmmmnnnooopppqqqrrrssstttuuuwwwxxxzzz}}}~~~^^WW99 >> !!%%'' //2211447766==BBFFHHMMOORRUUVV\\aacc``jjkkppppttvv񀀀Ӂ파읝롡뫫鹹!,k H*\ȰÇ#JHŋ3jȱǏ CIɓ(S\ɲ˗0cʜI͛8sɳϟ@ JѣH*]ʴӧPJJիXjʵW'PKٳhӪ]˶[&Kݻt(߿ˊ-+ԪW+^̸ǐ#Kf< ǫ&|aW>>SDI/L^ͺkȞ qYB@.(dwDJ}?D-ȓ+oȫ6D#oN}!z 3mla蔆47DH>AϿql$a&zrB)bC .Pr\H`!z3 (-8(I(׆H*䡁hipX<Ht6BCLKL6$yg"YKGfib`JsRXuh|cKW&;ۚxJ oJlhbcLaenH}V%HL!)hFebFVY#XZ-JF"lhWvU덷V&E[&VG&ud'Iv.{f;':fsB&DT+ &fãڶ޺a@ʝ嚩X^{$RrB`"']P1͸FTw$&Bv bw "E=AAqbH@raXu0W;mb'P.͜"۪f"֑nacuKRi3h+(z(c46Ke#vAxlSkc+ y'"uMҍ-t{V=8^7v ă 銧Rvcw*~Q,*D{J{ĸAcrEbzRԘT%N*ĊvʊG ή!D@NQZ!UCMZdMFG.pkiRc慞Ft "J3TbɘWRp&bnyMbIt $HJƲ1+L4 |GGItMY}T N2t%D)n"dVI|/1Otc LgZ57~9Lk1{q3Y'9NyΓ9?1:T:"/ayP*NL;*cR#BQg1Iu<"Fe,d&Ǭt83qg H-IibN J$>@p*yiίO̩bӓ"kYChdV}X4Ҩeje_@^xA!QŜB5 `  puHA7xA6dYOdjJۛ֨ UzJ MTfZiblXRDk@ՠ+<ʢ&i4Éxw"lAJPɔE߁wpf\ p[R%aOvp! kp3b "NUa?5m8 bU"g9^F@UercJyF2M{gT*F} pɫA3UQr1ucjI"l cŐ_1K#k[T$n0d"= a~Wù䡹$2&깾s.1Mv2IJX;o ȞRQY@!\5^:1@-e[B/R-4 +N[FuA&˄M)ZN 9Ff]H3 gDǼ۠ MH&$nm?En&]MtLX%ce#ܺ,$Tj'CWdc|yt:e|Dvh[#FI }<#tL4E<:D]&(W%#xR'B4HU$)X_&V|5h!#զ8 C.3CX2zE|@)1;KdT>F~Io^"rdXyݓUu!2<5=Eb19NJ5c'#м+3#{ w-r$}ʗ"Ο+BihW#B g b([`(/!Pvnv"W&HrW|dC{t'* q\$5eT2\n"yxDz4@aq/?X6:w DaD|8gT}}ڷǷ%?4SJ3`%gq,q}g{8pTW}B}\9.Y2gNa@QLY؄QA`*=6A񀿔p[({㑁j]H0>dD'|s&pt0Z$]r&H7IX+m+asFcQq(H!665:ACF4~9wLW{P!{0эq4M;#8Dk.7w2Ň/QoFl!w#498Wx5yLXΈ^ *)^q%7 ;0N 9d 2 2. HzOs uHCExKY _Ï%uomxW*KД ,B/D7Q玉<H"%xx2!F F' M8i G(hC?%zOP>/hjG3ԧw q2 2$Oi1AvHqbAN Uɜ,Y!x{3 e)9,qMcihI!etgI"z9 C<)KI9*< :x~WGhSS*QS2xH*zT,щ6'QoI|\K  (G#ኃ)~Zz?7irafJq,cT5z)]aViNo*p)=*tyV1?,"4 1p'ERxg9T*3^?ĚPGu^Dv a>t1|AB5L.q}CíԅxVwd")jRm,288<Q!! Pe)كjrQMH%!Z6ۄ/j5lljkYFtB~m HޙLf D 5Fpð;Ts:Y#4 VXVmh0CJF^x -])whYM}|~Kwp֖Vݓg?#BuZ"cCx @z}@e,M㓣 >Jr:p^$phNK[,8> *A$DWǻHV͋55%0=BC7b[bb0bb6k_nzv@ _cg'qE<bTqpcB< ALhM$i:~Q!Įik0| 5dH"1/Vۋ}ས Ŧaqu<I&5xgSU"?}ƪ b裱}PɀT k(z  ?>~l0l$><@w?\ܧIjuᴖQ,:$>ߴzxItZN|!ZGsEZ&գ8 A $Khq#i+k$EL  cuB̤w1Mc]`@ FV 5(: A1&HÓa]qargAwQ4!aز7MFFb*ȽJu V͙ͯͩ`M>RG 9][}sNag ^9Ӊq˝d8aK]1lW؆M[;|cx'pBL&ƞ$C ћMƝ;L׎18Y+۝޽QTaD!Ğӯ -MP;*HI3)m-MAh8B$--\W+nF⨴2˚LqPeZYU qՊ CdwyสiԜN֨&Q˪ڕBl8׼&E-M>ϱ3v 6쯑惘}G1=qz _J-wX&JLKT'#q銁!$:0g)ɡn5b~3 1)%-U $펤!N 1k;T^p{XgxJ.2P&/P O눱U4>^JDE <-&&&ލ1=K OL [ i$)/qwGUO!AEpcmgwzzF O2.lncrLL, j8+  d-s6=`1ONy % |v'^nB )qWwYQK@ $HA@$B[zQF=~H%2RJ-k KEt)HڀTPEAhP=8+*,y p*R@iVXe X NO0K,qU^EYmWaa_u:-_lNk9KhFF+R+><2Y^Pn޽MR'PcSp?/Ԇ"Y]QJ{;PI5^%̍HnĠTp7y <^?W֩'O$@x1Rԃ0¬@h^yd"Hc0ĠP03+bA1Oč&{MBw$L+}$:Cmd#S2c +2˖jl2x3L㈸B4z1Te#R/erH44W685/5?.5@-EVIVs1ΒfC 1>uW42k-US>rŴTW}6pVSU6nx_9pԍHy}UNb !&wT1#7u7 XO=BU#'l&,3DS);H0&dUfFϴt9Ervf2}/K=:lJk4nިh'7#hx(i\-:;p䨌T [ǤNLV-Rƻ)!_PQ`ҠیO<ч=9rrՆC81ߛxh ϭjQO\ I# r;o`knA߷:0a'/ГDmbŴr[q*!_HZXG°j  HĢ %J6h:AWx,o?`pr _Q ;ؒqeDk`Z-t#JR+ R8]g~Yіm(%җ*J(ZdD( 2Df.)е' *`tLzm'$VQlHO0!媇2ʹ!Zw;BqB܎Ȓ a w$U&9cA2)pt#r!DRfBw@{I-a""( ]B)i)u[UGSdL"AODyGeG$RL{fJ\`"@X(U̓H,VD1'#(<)Fihf!49g! -u45bЁGuxK3>ONmZI8e0.kD*|P%H@B\;7"ρBH's*Vl \lڥ"h { eH =H m@0;&znlf/ ъx=i7+rM-lW٦u| D[٢vekZXŬN( *ԏ+:>/du-;t,(x$ tFv6V]sP bɾH,hn_< dyA׽Ɛ]A4`-HDi;`F[^<1 (_!IuRX ua D0Mxj)E1 )QE⋟8\ h DҬXBc^AˆʄH0a)at0;9^@C&Rʰ A e``. t&v:h|37#n>)yquݜ@# ĺ)(-C! FFrckԌlBxMܡe+NЇnي nN4pH PX ];nÇ}jLƉwFHDrD _J+"u!B숭.NaY#U[N+Dmn xI du? ^AQZj_Ʃ+PXڞa.8[)X<M%Id`P  %`y\Nʺ*EnN{!bݎzԫŠ@"(\,4Ң }t}[*#'i{dT7p R \@MqFw0Nvzzh ]lJWK(/r$R@Z#4]_s)%ħ7~QND!>^0{H̦cC6hF2{)?3{-s#E}^}&BY,8?h1 p>aU [R<7o@K,@[ - TU|[ Zs = @q/ 89ŠP; eb6$1M;"̕lP >:>.ڑ 6$H0W8i+d>3C" 0q;𪱠N2@{ =C&Y*Rxåı:{rJވ) LD, ; 1A*d?/:9YL 99cġp>3 0 9% Z -z9-̡?L1lO$2 :ؿkCP2GZ %c^ TLR ǮW+ $xbD HmWpy,X8Wl +d 0d1KXcI0\Z䍿tư@Z)A 1C \J n\QD 60X2"6ȏUpωŵ GJy s&z M0sQ̒AP NȏxR̓گɘހ@ΩXA܍PK%au)hH qNGjѹ b@G=UK#)3HR@JU`4 SviƒFRjdl+!KMOx#SQӈGr><T&Yw)ɳB䯌8EtK /}bح-b׎xb&FLXJ64f$[  <D]bCA.gd`eFե 8R0{l~cNf0e AQ;e批d&:HZa~\ؠ(DHN`&nr孻a] \hM \!N@3lgGfL,02,di3GS׌j^ ImUЀ RxlZ AuCkX'%OXa8V­*& :433/8h¯U6\@NX5 iW5M,Vb\ܾXfr1瘨ֈ-1X)`.D쥄`~ {952hm$AލUx6 hދ% XX֌a.֕;ꮶRh(!UpoB䡈 l–FXeo crY!@yHY8b)Bselzc2halq0= > Q$T8 [!s$Z Sx_8U8z4!R(xt!0PgMHgBn]KT5g ;B`d d!H8umGI0ti݈8D qrsH'*H[N.[$xL=[[ kO 82.-;7Gy?yO".RDyS  hN04}2F| 51P>@:o dWl EW('@k7ZȘq+,ZJi؎'r<Nh2R]OnJQR/Sw 0sfWA[93wv|{nXhϗ8*;q ,Jwhs{5K NӑR0s va'&rd2w}7$ W@ԉp@nڗhW,h „ &Ĉ'Rhb- 7rE^Ô 貖>*g|Ec˜:%jU;-j4J2-PG5Jdә F!kSP-Z]tdײeaظr %ջC_ֵ}Aޡu|, Jq:},l9+ Ȓ)ewp*t§=QfڵXƈ4זعwE_v5OΧ>:3nxnnlADJ)V=| (A^է^\`Yn)" rk)tu_h'h&Xr`PGJV)!&GI!=lgv…'kÁr`B'#bËA5ZAEB^B4gg%MLda4($j\ 'D(mYd } 9G#u6ki))% Ș؆UL{vAwNG\yVyQuUJZ6)u:\u Z埝hũWq\{)*Vd_(s6e"rYGۂeH ܳYf|b.!\},|]V!^V0%KRa"^J{a]q%o-Mi)])`& \|zRuR&|3WG#] "lY]bJBt\[O4qm>K\W {=7k(=,'_d TJ}Z=}wBho\@-Ax/❳3%=O  hIuGR)9X~WJ~^z; |V#?=y.PRέ!QR=^Po\k84ʽM4.c?&t~ @ p1޷7-&JP., +6 !`mjt.i!.&0a, sOQ߲r鰈r o:Kg)6I|Es?)f*Fu`EHNt&6G.Yρ4!nH *bQOtq)>@!#9٢gW"ᎊDEGD0.$*CQN*RY4~0F?R%0{C ƌo#K3J@&rJ2,y2Eh)3x '?=)I@v\^bBEG mM RG`P41fN\32l ԥ2N}*T*թRU*VխrMW*ֱfU)"D'\*׹ҵv+^׽~+`+=,bXSHHFM ],Rl`Xd+ ft8 ִjs[D"/ֶ|P}A.׸ԫn%Օ("v{Zyŋ*м !{;Ͻ}|}g_8n".uKSOwkׂ͝s%Yx"cblx IF>2%3N~2,)S,C2;wel9^>q,uyffri[IֻrA[ O-.AO%}45V"yΡU4~_nQngAsz(IDZ/0N,RiRGz(Ep&ʭӶ X,nY`W"@~2|eZ 6![w&PR+@Em^c%X~#B[ "s]l(ބ7 MorNZK"NX9":9~_^{ҏo#e%\/B4AT<KH@;gpml-matlab-v4.0-2016-10-19/lik/likSech2.m000644 000765 000024 00000020502 12701705603 017460 0ustar00carlstaff000000 000000 function [varargout] = likSech2(hyp, y, mu, s2, inf, i) % likSech2 - sech-square likelihood function for regression. Often, the sech- % square distribution is also referred to as the logistic distribution not to be % confused with the logistic function for classification. The expression for the % likelihood is % likSech2(t) = Z / cosh(tau*(y-t))^2 where % tau = pi/(2*sqrt(3)*sn) and Z = tau/2 % and y is the mean and sn^2 is the variance. % % hyp = [ log(sn) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. The moments % \int f^k likSech2(y,f) N(f|mu,var) df are calculated via a Gaussian % scale mixture approximation. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-09-02. % % See also LIKFUNCTIONS.M, LIKLOGISTIC.M. if nargin<3, varargout = {'1'}; return; end % report number of hyperparameters sn = exp(hyp); tau = pi/(2*sqrt(3)*sn); lZ = log(pi) - log(sn) - log(4*sqrt(3)); if nargin<5 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = lZ - 2*logcosh(tau*(y-mu)); s2 = 0; else % prediction lp = likSech2(hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment if nargout>2 ys2 = s2 + sn.^2; % second y moment end end varargout = {lp,ymu,ys2}; else % inference mode switch inf case 'infLaplace' r = y-mu; [g,dg,d2g,d3g] = logcosh(tau.*r); % precompute derivatives if nargin<6 % no derivative mode dlp = {}; d2lp = {}; d3lp = {}; lp = lZ - 2*g; if nargout>1 % first derivatives dlp = 2*tau.*dg; if nargout>2 % 2nd derivative of log likelihood d2lp = -2*tau.^2.*d2g; if nargout>3 % 3rd derivative of log likelihood d3lp = 2*tau.^3.*d3g; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode lp_dhyp = 2*tau.*r.*dg - 1; % derivative w.r.t. sn dlp_dhyp = -2*tau.*(dg+tau.*r.*d2g); d2lp_dhyp = 2*tau.^2.*(2*d2g + tau.*r.*d3g); varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' n = max([length(y),length(mu),length(s2),length(sn)]); on = ones(n,1); y = y.*on; mu = mu.*on; s2 = s2.*on; sn = sn.*on; % vectors only fac = 1e3; % factor between the widths of the two distributions ... % ... from when one considered a delta peak, we use 3 orders of magnitude idlik = fac*sn1) and nu*sn^2/(nu-2) is the variance (for nu>2). % % The hyperparameters are: % % hyp = [ log(nu-1) % log(sn) ] % % Note that the parametrisation guarantees nu>1, thus the mean always exists. % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-01. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'2'}; return; end % report number of hyperparameters if ~exist('psi'), mypsi = @digamma; else mypsi = @psi; end % no psi in Octave numin = 1; % minimum value of nu nu = exp(hyp(1))+numin; sn2 = exp(2*hyp(2)); % extract hyperparameters lZ = gammaln(nu/2+1/2) - gammaln(nu/2) - log(nu*pi*sn2)/2; if nargin<5 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = lZ - (nu+1)*log( 1+(y-mu).^2./(nu.*sn2) )/2; s2 = 0; else % prediction lp = likT(hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment; for nu<=1 this is the mode if nargout>2 if nu<=2 ys2 = Inf(size(mu)); % variance does not always exist else ys2 = (s2 + nu*sn2/(nu-2)).*ones(size(mu)); % second y moment end end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' r = y-mu; r2 = r.*r; if nargin<6 % no derivative mode dlp = {}; d2lp = {}; d3lp = {}; lp = lZ - (nu+1)*log( 1+r2./(nu.*sn2) )/2; if nargout>1 a = r2+nu*sn2; dlp = (nu+1)*r./a; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = (nu+1)*(r2-nu*sn2)./a.^2; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = (nu+1)*2*r.*(r2-3*nu*sn2)./a.^3; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode a = r2+nu*sn2; a2 = a.*a; a3 = a2.*a; if i==1 % derivative w.r.t. nu lp_dhyp = nu*( mypsi(nu/2+1/2)-mypsi(nu/2) )/2 - 1/2 ... -nu*log(1+r2/(nu*sn2))/2 +(nu/2+1/2)*r2./(nu*sn2+r2); lp_dhyp = (1-numin/nu)*lp_dhyp; % correct for lower bound on nu dlp_dhyp = nu*r.*( a - sn2*(nu+1) )./a2; dlp_dhyp = (1-numin/nu)*dlp_dhyp; % correct for lower bound on nu d2lp_dhyp = nu*( r2.*(r2-3*sn2*(1+nu)) + nu*sn2^2 )./a3; d2lp_dhyp = (1-numin/nu)*d2lp_dhyp; % correct for lower bound on nu else % derivative w.r.t. sn lp_dhyp = (nu+1)*r2./a - 1; dlp_dhyp = -(nu+1)*2*nu*sn2*r./a2; d2lp_dhyp = (nu+1)*2*nu*sn2*(a-4*r2)./a3; end varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargout>1 error('infEP not supported since likT is not log-concave') end n = max([length(y),length(mu),length(s2)]); on = ones(n,1); y = y(:).*on; mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only % since we are not aware of an analytical expression of the integral, % we use Gaussian-Hermite quadrature N = 20; [t,w] = gauher(N); oN = ones(1,N); lZ = likT(hyp, y*oN, sig*t'+mu*oN, []); lZ = log_expA_x(lZ,w); % log( exp(lZ)*w ) varargout = {lZ}; case 'infVB' % variational lower site bound % t(s) \propto (1+(s-y)^2/(nu*s2))^(-nu/2+1/2) % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = zeros(n,1); y = y.*ones(n,1); z = y; varargout = {b,z}; end end % computes y = log( exp(A)*x ) in a numerically safe way by subtracting the % maximal value in each row to avoid cancelation after taking the exp function y = log_expA_x(A,x) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns y = log(exp(A-maxA*ones(1,N))*x) + maxA; % exp(A) = exp(A-max(A))*exp(max(A))gpml-matlab-v4.0-2016-10-19/lik/likInvGauss.m000644 000765 000024 00000011241 12777177046 020274 0ustar00carlstaff000000 000000 function [varargout] = likInvGauss(link, hyp, y, mu, s2, inf, i) % likInvGauss - Inverse Gaussian likelihood function for strictly positive data % y. The expression for the likelihood is % likInvGauss(f) = sqrt(lam/(2*pi*y^3))*exp(-lam*(mu-y)^2/(2*mu^2*y)) with % mean=mu and variance=mu^3/lam where mu = g(f) is the Inverse Gaussian % intensity, f is a Gaussian process, y is the strictly positive data. % Hence, we have -- with log(Zy) = -(log(lam)-log(2*pi*y^3))/2 % llik(f) = log(likInvGauss(f)) = -lam*(y-mu)^2/(2*mu^2*y) - log(Zy). % The larger one chooses lam, the stronger the likelihood resembles a Gaussian % since skewness = 3*sqrt(mu/lam) and kurtosis = 15*mu/lam. % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(lam) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2016-10-04. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters lam = exp(hyp); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); elg = exp(lg); lZy = -(log(lam)-log(2*pi*y.^3))/2; % normalisation constant lp = -lam*(y./elg-1).^2 ./(2*y) - lZy; else lp = likInvGauss(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg.^3/lam; % second y moment from inverse Gaussian distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); if nargin<7 % no derivative mode lZy = -(log(lam)-log(2*pi*y.^3))/2; % normalisation constant lp = -lam*(y./elg-1).^2 ./(2*y) - lZy; dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = lam*(y./elg-1).*dlg./elg; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = lam*( (y./elg-1).*(d2lg-dlg.^2) - y.*dlg.^2./elg )./elg; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = lam*( (y./elg-1) .* (4*dlg.^3-6*dlg.*d2lg+d3lg) ... + (3*dlg.^3 - 3*dlg.*d2lg) )./ elg; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode lp_dhyp = 1/2-lam*(y./elg-1).^2 ./(2*y); % deriv. of log lik w.r.t. lam dlp_dhyp = lam*(y./elg-1).*dlg./elg; % first derivative d2lp_dhyp = lam*( (y./elg-1).*(d2lg-dlg.^2) - y.*dlg.^2./elg )./elg; % 2nd varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likInvGauss,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if isequal(link,'exp') [varargout{:}] = glm_invlink_exp(f); elseif isequal(link,'logistic') [varargout{:}] = glm_invlink_logistic(f); else [varargout{:}] = glm_invlink_logistic2(link{2},f); endgpml-matlab-v4.0-2016-10-19/lik/likMix.m000644 000765 000024 00000020074 12701705603 017255 0ustar00carlstaff000000 000000 function [varargout] = likMix(lik, hyp, varargin) % likMix - Mixture of likelihoods for regression/classification. % The expression for the likelihood is % likMix(t) = sum_i=1..m w_i * lik_i(t), % where lik_i are the m individual likelihood functions combined by a weighted % sum with weights wi. % % The hyperparameters are: % % hyp = [ log(w_1) % log(w_2) % ... % log(w_m) % hyp_1 % hyp_2 % ... % hyp_m ] % % Note that the mixture weights are normalised using the softmax function, so % that 1 = sum w_i. Here, hyp_i are the hyperparameters for the individual % likelihoods. % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Hannes Nickisch and Rowan McAllister, 2013-10-23. % % See also LIKFUNCTIONS.M. m = numel(lik); % number of mixture components if m==0, error('We require at least one mixture component.'), end for i = 1:m % iterate over mixture components f = lik(i); if iscell(f{:}), f = f{:}; end % expand cell array if necessary j(i) = cellstr(feval(f{:})); % collect number hypers end s = @(x) logsumexp2(x); % define a shortcut to util/logsumexp2.m if nargin<4 % report number of parameters L = char(j(1)); for i=2:length(lik), L = [L, '+', char(j(i))]; end varargout = {['(',num2str(m),'+',L,')']}; return end mu = varargin{2}; n = length(mu); % number of query points if numel(varargin)>3 inf = varargin{4}; % varargin = {y, mu, s2, inf, i} else inf = ''; end v = []; % v vector indicates to which likelihood parameters belong nhyp = m; % number of hyperparameters; the first m are the mixture weights for i = 1:m ni = eval(char(j(i))); nhyp = nhyp + ni; v = [v, repmat(i,1,ni)]; end if nhyp>length(hyp), error('not enough hyperparameters'), end if strcmp(inf,'infVB'), error('infVB not supported'); end na = nargout; if nargout>1 && strcmp(inf,'infLaplace') && nargin>6, na = max(na,3); end argout = zeros(n,m,na); % collect output arguments from individual lik fun for i=1:m % iteration over mixture components f = lik(i); if iscell(f{:}), f = f{:}; end % expand cell array if necessary id = find(v==i)+m; % hyperparameter indices out = cell(size(argout,3),1); nn = min(nargin,6)-2; % do not ask for derivs [out{:}] = feval(f{:}, hyp(id), varargin{1:nn}); % ask individual likelihoods for j=1:size(argout,3), argout(:,i,j) = out{j}; end end lw = reshape(hyp(1:m),1,[]); lwn = lw-s(lw); % w = exp(lwn), sum(w)=1, weights one_lwn = ones(n,1)*lwn; % the same as above only that the weights are repeated varargout = cell(nargout,1); % allocate memory for output arguments if nargin<6 % prediction mode if inf is not present % lp and ymu outputs are of the weighted sum of integral structure if nargout>0, varargout{1} = s(one_lwn + argout(:,:,1)); end if nargout>1, varargout{2} = sum(exp(one_lwn).*argout(:,:,2), 2); end if nargout>2 % ys2 output computed using the parallel axis theorem argout3corr = argout(:,:,3) + (argout(:,:,2)-varargout{2}*ones(1,m)).^2; varargout{3} = sum(exp(one_lwn).*argout3corr, 2); end else if nargin>6, i = varargin{5}; end % make arguments available switch inf case {'infEP','infLaplace'} % the two cases are structurally similar if nargin<7 % no derivative mode if nargout>0 % lf={lZ,lp} for inf={'infEP','infLaplace'} lfi = argout(:,:,1); % partition functions of individual likelihoods lf = s(lfi + one_lwn); % weighted sum in the exp domain varargout{1} = lf; if nargout>1 % dlf dlfi = argout(:,:,2); % derivatives of individual terms % Using f*dlf=df, f=exp(lf), f=sum_i wi*fi, df=sum_i wi*dfi we obtain % dlf = sum_i exp(lfi-lf+lwi)*dlfi = sum_i ai*dlfi. a = exp(lfi - lf*ones(1,m) + one_lwn); dlf = sum(a.*dlfi,2); varargout{2} = dlf; if nargout>2 % d2lf d2lfi = argout(:,:,3); % 2nd derivatives of individual terms % Using d2lf=d(df/f)=(d2f*f-df^2)/f^2 <=> d2f=f*(d2lf+dlf^2) and % d2f = sum_i wi*d2fi, we get d2lf = sum_i ai*(d2lfi+dlfi^2)-dlf^2. d2lf = sum(a.*(d2lfi+dlfi.*dlfi),2) - dlf.*dlf; varargout{3} = d2lf; if nargout>3 && strcmp(inf,'infLaplace') % d3lf % Using d3f = sum_i wi*d3fi, d3lf=d((d2f*f-df^2)/f^2), we obtain % d3f = f*(d3lf+dlf*(3*d2lf+dlf^2)) <=> % d3lf = d3f/f - dlf*(3*d2lf+dlf^2). d3lfi = argout(:,:,4); % 3rd derivatives of individual terms d3lf = sum(a.*(d3lfi+dlfi.*(3*d2lfi+dlfi.*dlfi)),2); d3lf = d3lf - dlf.*(3*d2lf+dlf.*dlf); varargout{4} = d3lf; end end end end else % d(lf)/dhyp and d(d2lf)/dhyp if nargout>0 lfi = argout(:,:,1); % partition functions of individual likelihoods lf = s(lfi + one_lwn); % weighted sum in the exp domain a = exp(lfi - lf*ones(1,m) + one_lwn); if i<=m % the first m hyperparameters are mixture weights lf_dhyp = a(:,i) - exp(lwn(i)); varargout{1} = lf_dhyp; if nargout>1 && strcmp(inf,'infLaplace') % d(dlf)/dhyp dlfi = argout(:,:,2); dlwn = zeros(size(lw)); dlwn(i) = 1; dlwn = dlwn - exp(lw(i)-s(lw)); dla = ones(n,1)*dlwn - lf_dhyp*ones(1,m); % d log(v) / dhyp da = a.*dla; dlf_dhyp = sum(da.*dlfi,2); varargout{2} = dlf_dhyp; if nargout>2 % d(d2lf)/dhyp d2lfi = argout(:,:,3); dlf = sum(a.*dlfi,2); d2lf_dhyp = sum(da.*(d2lfi+dlfi.*dlfi),2) - 2*dlf.*dlf_dhyp; varargout{3} = d2lf_dhyp; end end else j = v(i-m); % index of mixture component under consideration id = find(v==j)+m; % hyper parameter indices i = i-min(id)+1; % index of likelihood parameter of mixture j f = lik(j); if iscell(f{:}), f = f{:}; end % expand cell if necessary if nargout==1 lfi_dhyp = feval(f{:}, hyp(id), varargin{1:end-1}, i); elseif nargout==2 [lfi_dhyp,dlfi_dhyp] = feval(f{:}, hyp(id), varargin{1:end-1}, i); else [lfi_dhyp,dlfi_dhyp,d2lfi_dhyp] = ... feval(f{:}, hyp(id), varargin{1:end-1}, i); end lf_dhyp = exp(lwn(j)+lfi(:,j)-lf) .* lfi_dhyp(:); % d(lf)/dhyp varargout{1} = lf_dhyp; if nargout>1 % d(dlf)/dhyp dlfi = argout(:,:,2); d2lfi = argout(:,:,3); dlf = sum(a.*dlfi,2); dla = -lf_dhyp*ones(1,m); dla(:,j) = dla(:,j)+lfi_dhyp(:); da = a.*dla; dlf_dhyp = sum(da.*dlfi,2) + a(:,j).*dlfi_dhyp(:); varargout{2} = dlf_dhyp; if nargout>2 % d(d2lf)/dhyp p = d2lfi+dlfi.*dlfi; dpj = 2*dlfi(:,j).*dlfi_dhyp(:) + d2lfi_dhyp(:); d2lf_dhyp = a(:,j).*dpj + sum(da.*p,2) - 2*dlf.*dlf_dhyp; varargout{3} = d2lf_dhyp; end end end end end case 'infVB' error('infVB not supported') end endgpml-matlab-v4.0-2016-10-19/lik/likWeibull.m000644 000765 000024 00000011036 12777177052 020137 0ustar00carlstaff000000 000000 function [varargout] = likWeibull(link, hyp, y, mu, s2, inf, i) % likWeibull - Weibull likelihood function for strictly positive data y. The % expression for the likelihood is % likWeibull(f) = g1*ka/mu * (g1*y/mu)^(ka-1) * exp(-(g1*y/mu)^ka) with % gj = gamma(1+j/ka), mean=mu and variance=mu^2*(g2/g1^2-1) where mu = g(f) is % the Weibull intensity, f is a Gaussian process, y is the positive data. % Hence, we have llik(f) = log(likWeibull(f)) = % log(g1*ka/mu) + (ka-1)*log(g1*y/mu) - (g1*y/mu)^ka. % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(ka) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2016-10-04. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters ka = exp(hyp); lg1 = gammaln(1+1/ka); g1 = exp(lg1); dlg1 = -psi(1+1/ka)/ka; if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); lp = lg1 + log(ka) + (ka-1)*(lg1+log(y)) - ka*lg - exp(ka*(lg1+log(y)-lg)); else lp = likWeibull(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); g2 = gamma(1+2/ka); yv = elg.^2*(g2/g1^2-1); % second y moment from Weibull distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(-ka*lg); if nargin<7 % no derivative mode lp = lg1 + log(ka) + (ka-1)*(lg1+log(y)) -ka*lg - exp(ka*(lg1+log(y)-lg)); dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = -ka*dlg + ka*(g1*y).^ka .* elg.*dlg; % dlp, deriv of log lik if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = -ka*d2lg + ka*(g1*y).^ka .* ( -ka*elg.*dlg.^2 + elg.*d2lg ); if nargout>3 % d3lp, 3rd derivative of log likelihood a = ka^2*dlg.^3 -3*ka*dlg.*d2lg + d3lg; d3lp = - ka*d3lg + ka*(g1*y).^ka .* a .*elg; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode v = ka*(lg1+log(y)-lg); ev = exp(v); % derivative of log lik w.r.t. ka w = v+ka*dlg1; dw = -ka*dlg; d2w = -ka*d2lg; lp_dhyp = 1 + w - ev.*w; dlp_dhyp = dw.*(1-ev.*(1+w)); % first derivative d2lp_dhyp = d2w.*(1-ev.*(1+w)) - dw.^2.*(ev.*(2+w)); % and also second varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likWeibull,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if isequal(link,'exp') [varargout{:}] = glm_invlink_exp(f); elseif isequal(link,'logistic') [varargout{:}] = glm_invlink_logistic(f); else [varargout{:}] = glm_invlink_logistic2(link{2},f); endgpml-matlab-v4.0-2016-10-19/lik/likBeta.m000644 000765 000024 00000011335 12777177045 017413 0ustar00carlstaff000000 000000 function [varargout] = likBeta(link, hyp, y, mu, s2, inf, i) % likBeta - Beta likelihood function for interval data y from [0,1]. % The expression for the likelihood is % likBeta(f) = 1/Z * y^(mu*phi-1) * (1-y)^((1-mu)*phi-1) with % mean=mu and variance=mu*(1-mu)/(1+phi) where mu = g(f) is the Beta intensity, % f is a Gaussian process, y is the interval data and % Z = Gamma(phi)/Gamma(phi*mu)/Gamma(phi*(1-mu)). % Hence, we have % llik(f) = log(likBeta(f)) = -lam*(y-mu)^2/(2*mu^2*y) - log(Zy). % % We provide two inverse link functions 'logit' and 'expexp': % g(f) = 1/(1+exp(-f)) and g(f) = exp(-exp(-f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(phi) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2014-03-04. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters phi = exp(hyp); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); elg = exp(lg); v = phi*elg; w = phi-v; a0 = gammaln(w)-gammaln(phi); lp = (v-1).*log(y) + (w-1).*log(1-y) - gammaln(v) - a0; else lp = likBeta(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg.*(1-elg)/(1+phi); % second y moment from Beta distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); v = phi*elg; w = phi-v; if nargin<7 % no derivative mode a0 = gammaln(phi-v)-gammaln(phi); lp = (v-1).*log(y) + (w-1).*log(1-y) - gammaln(v) - a0; dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 % dlp, derivative of log likelihood a1 = v.*(log(y)-log(1-y) + psi(0,w)-psi(0,v)); dlp = dlg.*a1; if nargout>2 % d2lp, 2nd derivative of log likelihood a2 = v.^2.*(psi(1,w)+psi(1,v)); z = dlg.^2+d2lg; d2lp = z.*a1 - dlg.^2.*a2; if nargout>3 % d3lp, 3rd derivative of log likelihood a3 = v.^3.*(psi(2,w)-psi(2,v)); d3lp = (dlg.*z+2*dlg.*d2lg+d3lg).*a1 - 3*dlg.*z.*a2 + dlg.^3.*a3; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode % deriv. of log lik w.r.t. phi lp_dhyp = v.*log(y)+w.*log(1-y)-v.*psi(0,v)-w.*psi(0,w)+phi*psi(0,phi); a1 = v.*(log(y)-log(1-y) + psi(0,w)-psi(0,v)); da1 = a1 + v.*(w.*psi(1,w)-v.*psi(1,v)); dlp_dhyp = dlg.*da1; % first derivative a2 = v.^2.*(psi(1,w)+psi(1,v)); z = dlg.^2+d2lg; da2 = v.^2.*(w.*psi(2,w)+v.*psi(2,v)) + 2*a2; d2lp_dhyp = z.*da1 - dlg.^2.*da2; % second derivative varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likBeta,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(link,'expexp') [varargout{:}] = glm_invlink_expexp(f); else [varargout{:}] = glm_invlink_logit(f); endgpml-matlab-v4.0-2016-10-19/lik/likUni.m000644 000765 000024 00000005066 12701705603 017257 0ustar00carlstaff000000 000000 function [varargout] = likUni(hyp, y, mu, s2, inf, i) % likUni - Uniform likelihood function for classification. The expression for % the likelihood is % likUni(t) = 1/2. % % There are no hyperparameters: % % hyp = [ ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Hannes Nickisch, 2013-09-02. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'0'}; return; end % report number of hyperparameters if nargin<5 % prediction mode if inf is not present lp = -log(2)*ones(size(mu)); ymu = {}; ys2 = {}; if nargout>1 p = exp(lp); ymu = 2*p-1; % first y moment if nargout>2 ys2 = 4*p.*(1-p); % second y moment end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' if nargin<6 % no derivative mode lp = -log(2)*ones(size(mu)); dlp = {}; d2lp = {}; d3lp = {}; if nargout>1 dlp = zeros(size(mu)); % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = zeros(size(mu)); if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = zeros(size(mu)); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode varargout = {[],[],[]}; % derivative w.r.t. hypers end case 'infEP' if nargin<6 % no derivative mode lZ = -log(2)*ones(size(mu)); % log part function dlZ = {}; d2lZ = {}; if nargout>1 dlZ = zeros(size(mu)); % 1st derivative w.r.t. mean if nargout>2 d2lZ = zeros(size(mu)); % 2nd derivative w.r.t. mean end end varargout = {lZ,dlZ,d2lZ}; else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' % variational lower site bound % t(s) = 1/2 % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = zeros(n,1); y = y.*ones(n,1); z = y; varargout = {b,z}; end endgpml-matlab-v4.0-2016-10-19/lik/likPoisson.m000644 000765 000024 00000010254 12777177046 020172 0ustar00carlstaff000000 000000 function [varargout] = likPoisson(link, hyp, y, mu, s2, inf, i) % likPoisson - Poisson likelihood function for count data y. The expression for % the likelihood is % likPoisson(f) = mu^y * exp(-mu) / y! with mean=variance=mu % where mu = g(f) is the Poisson intensity, f is a % Gaussian process, y is the non-negative integer count data and % y! = gamma(y+1) its factorial. Hence, we have -- with Zy = gamma(y+1) = y! -- % llik(f) = log(likPoisson(f)) = log(g(f))*y - g(f) - log(Zy). % The larger the intensity mu, the stronger the likelihood resembles a Gaussian % since skewness = 1/sqrt(mu) and kurtosis = 1/mu. % % We provide two inverse link functions 'exp' and 'logistic': % For g(f) = exp(f), we have lik(f) = exp(f*y-exp(f)) / Zy. % For g(f) = log(1+exp(f))), we have lik(f) = log^y(1+exp(f)))(1+exp(f)) / Zy. % The link functions are located at util/glm_invlink_*.m. % % Note that for both intensities g(f) the likelihood lik(f) is log concave. % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-04. if nargin<4, varargout = {'0'}; return; end % report number of hyperparameters if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); lp = lg.*y - exp(lg) - gammaln(y+1); else lp = likPoisson(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg; % second y moment from Poisson distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' if nargin<7 % no derivative mode [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); lp = lg.*y - elg - gammaln(y+1); dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = dlg.*(y-elg); % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = d2lg.*(y-elg) - dlg.*dlg.*elg; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = d3lg.*(y-elg) - dlg.*(dlg.*dlg+3*d2lg).*elg; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode varargout = {[],[],[]}; % derivative w.r.t. hypers end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % hence we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likPoisson,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if isequal(link,'exp') [varargout{:}] = glm_invlink_exp(f); elseif isequal(link,'logistic') [varargout{:}] = glm_invlink_logistic(f); else [varargout{:}] = glm_invlink_logistic2(link{2},f); endgpml-matlab-v4.0-2016-10-19/lik/likLaplace.m000644 000765 000024 00000015412 12701705603 020061 0ustar00carlstaff000000 000000 function [varargout] = likLaplace(hyp, y, mu, s2, inf, i) % likLaplace - Laplacian likelihood function for regression. % The expression for the likelihood is % likLaplace(t) = exp(-|t-y|/b)/(2*b) with b = sn/sqrt(2), % where y is the mean and sn^2 is the variance. % % The hyperparameters are: % % hyp = [ log(sn) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-10-16. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'1'}; return; end % report number of hyperparameters sn = exp(hyp); b = sn/sqrt(2); if nargin<5 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = -abs(y-mu)./b -log(2*b); s2 = 0; else % prediction lp = likLaplace(hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment if nargout>2 ys2 = s2 + sn.^2; % second y moment end end varargout = {lp,ymu,ys2}; else % inference mode switch inf case 'infLaplace' if nargin<6 % no derivative mode if numel(y)==0, y=0; end ymmu = y-mu; dlp = {}; d2lp = {}; d3lp = {}; lp = -abs(ymmu)/b -log(2*b); if nargout>1 dlp = sign(ymmu)/b; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = zeros(size(ymmu)); if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = zeros(size(ymmu)); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode lp_dhyp = abs(y-mu)/b - 1; % derivative of log likelihood w.r.t. hypers dlp_dhyp = sign(mu-y)/b; % first derivative, d2lp_dhyp = zeros(size(mu)); % and also of the second mu derivative varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' n = max([numel(y),numel(mu),numel(s2),numel(sn)]); on = ones(n,1); y = y(:).*on; mu = mu(:).*on; s2 = s2(:).*on; sn = sn(:).*on; % vectors only fac = 1e3; % factor between the widths of the two distributions ... % ... from when one considered a delta peak, we use 3 orders of magnitude idlik = fac*sn1 lqp = -zp.^2/2 - log(2*pi)/2 - logphi(-zp); % log( N(z)/Phi(z) ) lqm = -zm.^2/2 - log(2*pi)/2 - logphi( zm); dap = -exp(lqp-log(s2(id))/2) + sqrt(2)./sn(id); dam = exp(lqm-log(s2(id))/2) - sqrt(2)./sn(id); % ( exp(ap).*dap + exp(am).*dam )./( exp(ap) + exp(am) ) dlZ(id) = expABz_expAx([ap,am],[1;1],[dap,dam],[1;1]); if nargout>2 a = sqrt(8)./sn(id)./sqrt(s2(id)); bp = 2./sn(id).^2 - (a - zp./s2(id)).*exp(lqp); bm = 2./sn(id).^2 - (a + zm./s2(id)).*exp(lqm); % d2lZ(id) = ( exp(ap).*bp + exp(am).*bm )./( exp(ap) + exp(am) )... % - dlZ(id).^2; d2lZ(id) = expABz_expAx([ap,am],[1;1],[bp,bm],[1;1]) - dlZ(id).^2; end end end varargout = {lZ,dlZ,d2lZ}; else % derivative mode dlZhyp = zeros(n,1); if any(idlik) dlZhyp(idlik) = 0; end if any(idgau) dlZhyp(idgau) = ... likLaplace(log(sn(idgau)), mu(idgau), y(idgau), 'infLaplace', 1); end if any(id) % substitution to obtain unit variance, zero mean Laplacian tmu = (mu(id)-y(id))./sn(id); tvar = s2(id)./sn(id).^2; zp = (tvar+tmu/sqrt(2))./sqrt(tvar); vp = tvar+sqrt(2)*tmu; zm = (tvar-tmu/sqrt(2))./sqrt(tvar); vm = tvar-sqrt(2)*tmu; dzp = (-s2(id)./sn(id)+tmu.*sn(id)/sqrt(2)) ./ sqrt(s2(id)); dvp = -2*tvar - sqrt(2)*tmu; dzm = (-s2(id)./sn(id)-tmu.*sn(id)/sqrt(2)) ./ sqrt(s2(id)); dvm = -2*tvar + sqrt(2)*tmu; lezp = logerfc(zp); % ap = exp(vp).*ezp lezm = logerfc(zm); % am = exp(vm).*ezm vmax = max([vp+lezp,vm+lezm],[],2); % subtract max to avoid numerical pb ep = exp(vp+lezp-vmax); em = exp(vm+lezm-vmax); dap = ep.*(dvp - 2/sqrt(pi)*exp(-zp.^2-lezp).*dzp); dam = em.*(dvm - 2/sqrt(pi)*exp(-zm.^2-lezm).*dzm); dlZhyp(id) = (dap+dam)./(ep+em) - 1; end varargout = {dlZhyp}; % deriv. wrt hypers end case 'infVB' % variational lower site bound % t(s) = exp(-sqrt(2)|y-s|/sn) / sqrt(2*sn²) % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = zeros(n,1); y = y.*ones(n,1); z = y; varargout = {b,z}; end end % logerfc(z) = log(1-erf(z)) function lc = logerfc(z) lc = logphi(-z*sqrt(2)) + log(2); function y = expABz_expAx(A,x,B,z) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns A = A-maxA*ones(1,N); % subtract maximum value y = ( (exp(A).*B)*z ) ./ ( exp(A)*x );gpml-matlab-v4.0-2016-10-19/lik/likGaussWarp.m000644 000765 000024 00000021625 12701705603 020437 0ustar00carlstaff000000 000000 function [varargout] = likGaussWarp(warp, hyp, y, mu, varargin) % likGaussWarp - Warped Gaussian likelihood for regression. % The expression for the likelihood is % likGaussWarp( y | t ) = likGauss( g(y) | t ) * g'(y), % where likGauss is the Gaussian likelihood and g is the warping function. % % The hyperparameters are: % % hyp = [ theta_1 % theta_2 % .. % theta_ng % log(sn) ] % % Here, sn is the standard deviation of the underlying Gaussian and theta_i for % i=1..ng are the ng hyperparameters of the warping function g. % % At the moment, likGaussWarp offers 3 different warping functions: % id yields g(y) = y => likGaussWarp = likGauss % poly e.g. 'poly1' yields g(y) = y => likGaussWarp = likGauss % 'poly3' yields g(y) = y + c1*sy*ay^2 + c2*sy*ay^3 % where sy = sign(y), ay = abs(y), cj = exp(theta_j) % tanh e.g. 'tanh0' yields g(y) = y => likGaussWarp = likGauss % 'tanh2' yields g(y) = y + a1*tanh(b1*(y+c1)) + a2*tanh(b2*(y+c2)) % where aj = exp(theta_j), bj = exp(theta_j+m), bj = theta_j+2*m % % The code is based on the exposition in the paper Warped Gaussian Processes, % NIPS, 2003 by Edward Snelson, Carl Rasmussen and Zoubin Ghahramani. % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Hannes Nickisch, 2013-10-24. % % See also LIKFUNCTIONS.M. lik = {@likGauss}; % in principle any likelihood function can be warped but only % for homoscedastic likelihoods, in particular Gaussian has feasible integrals if numel(warp)==0, warp = 'id'; end % set default warping function ng = g(warp); % number of hyperparameters for the warping function nhyp = ['(',num2str(ng),'+',feval(lik{:}),')']; % number of hyperparameters if nargin<4, varargout = {nhyp}; return, end % report number of parameters nhyp = eval(nhyp); if nhyp>length(hyp), error('not enough hyperparameters'), end [gy,lgpy] = g(warp,y,hyp(1:ng)); % evaluate warping function i = 0; if nargin>6, i = varargin{3}; varargin{3} = varargin{3}-ng; end varargout = cell(nargout,1); % allocate memory for output arguments if i==0 || ng4, s2 = varargin{1}; end % s2==0 ? if nargin>4&&numel(s2)>0&&norm(s2)>eps>0, s2zero = 0; end if s2zero % log probability lp = likGaussWarp(warp, hyp, y, mu, [], 'infLaplace'); s2 = 0*mu; else lp = likGaussWarp(warp, hyp, y, mu, s2, 'infEP'); % prediction end if nargout>0, varargout{1} = lp; end % works for any lik % the predictive moments are very hard to compute for lik not being likGauss if nargout>1 ymu = mu; % first g(y) moment sn2 = exp(2*hyp(ng+1)); % Gaussian noise variance ys2 = s2 + sn2; % second g(y) moment % ymuM = ig(warp,ymu,hyp(1:ng)); % median % yupp = ig(warp,ymu+2*sqrt(ys2),hyp(1:ng)); % 95% confidence interval % ylow = ig(warp,ymu-2*sqrt(ys2),hyp(1:ng)); % ys2C = (yupp-ylow).^2/16; N = 20; [t,w] = gauher(N); oN = ones(1,N); % Gaussian-Hermite quadrature Z = sqrt(ys2(:))*t'+ymu(:)*oN; Y = ig(warp,Z,hyp(1:ng)); ymu = Y*w; ys2 = (Y-ymu*oN).^2*w; % first and second y moment varargout{2} = reshape(ymu,size(mu)); if nargout>2 varargout{3} = ys2; end end else inf = varargin{2}; % obtain input variables switch inf case {'infLaplace','infEP'} % they have the same structure if nargin<7 % no derivative mode if nargout>0, varargout{1} = varargout{1} + lgpy; end else % derivative mode if i<=ng % derivatives w.r.t. warping function parameters n = max([numel(y),numel(mu)]); for j=2:nargout, varargout{j} = zeros(n,1); end [dgy,dlgpy] = g(warp,y,hyp(1:ng),i); % warping function derivative out = cell(nargout+1,1); % allocate memory [out{:}] = likGaussWarp(warp, hyp, y, mu, varargin{1:2}); % query lik % works only for homoscedastic likelihoods where y and mu can be swapped if nargout>0, varargout{1} = dlgpy - out{2}.*dgy; end % apply chain rule if nargout>1, varargout{2} = - out{3}.*dgy; end if nargout>2, varargout{3} = - out{4}.*dgy; end end end case 'infVB' % output does not depend on mu and following parameters end end % marshalling of parameters and available warping functions function varargout = g(warp,varargin) varargout = cell(nargout, 1); % allocate the right number of output arguments if strcmp(warp,'id') % indentity warping likGaussWarp = likGauss if nargin<2 if nargout>0, varargout{1} = 0; end elseif nargin<4 if nargout>0, varargout{1} = varargin{1}; end if nargout>1, varargout{2} = 0*varargin{1}; end end elseif numel(strfind(warp,'poly'))>0 m = str2double(warp(5:end)); if nargin<2 && nargout>0, varargout{1} = m-1; return, end [varargout{:}] = g_poly(varargin{:}); elseif numel(strfind(warp,'tanh'))>0 m = str2double(warp(5:end)); if nargin<2 && nargout>0, varargout{1} = 3*m; return, end [varargout{:}] = g_tanh(varargin{:}); end % invert g(y) = z <=> ig(z) = y via bisection search + Newton iterations function [y,n,d] = ig(warp,z,hyp) y = z; gy = g(warp,z,hyp)-z; dz = max(abs(z(:))); % lower bound search ylow while any(0gy(:)), y(0>gy) = y(0>gy)+dz; gy = g(warp,y,hyp)-z; end, yupp = y; for n=1:12 % bisection search ylow<=y<=yupp d = max(abs(gy(:))); if d0) = y(gy>0); end for n=1:12 % Newton iterations [gy,lgpy] = g(warp,y,hyp); gpy = exp(lgpy); y = y - (gy-z)./gpy; y(yyupp) = yupp(y>yupp); % keep brackets d = max( abs(gy(:)-z(:)) ); if dsqrt(eps), fprintf('Not converged: res=%1.4e\n',d), end % poly warping function g(y) and log of the derivative log(g'(y))>0 % or derivatives of the latter w.r.t. ith hyperparameter function [gy,lgpy] = g_poly(y,hyp,i) m = numel(hyp)+1; c = exp(hyp); if nargin==2 % function values gy = y; gpy = 1; ay = abs(y); for j=2:m gy = gy + c(j-1)*ay.^j; gpy = gpy + c(j-1)*j*ay.^(j-1); end gy = sign(y).*gy; lgpy = log(gpy); else % derivatives gpy = 1; ay = abs(y); for j=2:m gpy = gpy + c(j-1)*j*ay.^(j-1); end gy = c(i)*ay.^j; lgpy = c(i)*j*ay.^(j-1)./gpy; end % tanh warping function g(y) and log of the derivative log(g'(y))>0 % or derivatives of the latter w.r.t. ith hyperparameter function [gy,lgpy] = g_tanh(y,hyp,i) m = numel(hyp)/3; a = exp(hyp(1:m)); b = exp(hyp(m+(1:m))); c = hyp(2*m+(1:m)); if nargin==2 % function values gy = y; gpy = 1; for j=1:m ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gy = gy + ai *ti; gpy = gpy + ai*bi*dti; end lgpy = log(gpy); else % derivatives gpy = 1; for j=1:m ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gpy = gpy + ai*bi*dti; end if i<=m j = i; ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gy = ai*ti; lgpy = ai*bi*dti./gpy; elseif i<=2*m j = i-m; ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gy = ai*bi*dti.*(y+ci); d2ti = -2*ti.*dti; lgpy = ai*bi*(dti+bi*d2ti.*(y+ci))./gpy; else j = i-2*m; ai = a(j); bi = b(j); ci = c(j); ti = tanh(bi*(y+ci)); dti = 1-ti.^2; gy = ai*bi*dti; d2ti = -2*ti.*dti; lgpy = ai*bi^2*d2ti./gpy; end endgpml-matlab-v4.0-2016-10-19/lik/likGumbel.m000644 000765 000024 00000007610 12701705603 017734 0ustar00carlstaff000000 000000 function [varargout] = likGumbel(sign, hyp, y, mu, s2, inf, i) % likGumbel - Gumbel likelihood function for extremal value regression. % The expression for the likelihood is % likGumbel(t) = exp(-z-exp(-z))/be, z = ga+s*(y-t)/be, be = sn*sqrt(6)/pi % where s={+1,-1} is a sign switching between left and right skewed, ga is the % Euler-Mascheroni constant, y is the mean, sn^2 is the variance. % The skewness and kurtosis of likGumbel are 1.14*s and 2.4, respectively. % % The hyperparameters are: % % hyp = [ log(sn) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Hannes Nickisch, 2013-11-01. % % See also LIKFUNCTIONS.M. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters if sign=='-', s = -1; else s = 1; end % extract sign of skewness sn2 = exp(2*hyp); % extract hyperparameters ga = 0.5772156649; % Euler-Mascheroni constant be = sqrt(6*sn2)/pi; lZ = -log(be); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = likGumbel(sign, hyp, y, mu, [], 'infLaplace'); s2 = 0; else % prediction lp = likGumbel(sign, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment if nargout>2 ys2 = s2 + sn2; % second y moment end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' z = ga+s*(y-mu)/be; emz = exp(-z); if nargin<7 % no derivative mode dlp = {}; d2lp = {}; d3lp = {}; lp = lZ -z -emz; if nargout>1 dz = -s/be; % dz/dmu dlp = dz*(emz-1); % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = -dz^2*emz; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = dz^3*emz; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative w.r.t. log(sn) dz = -s/be; % dz/dmu dzs = -s*(y-mu)/be; % dz/dlog(sn) lp_dhyp = dzs.*(emz-1) -1; dlp_dhyp = dz*(1-emz.*(1+dzs)); d2lp_dhyp = dz^2*emz.*(2+dzs); varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargout>1 error('infEP not supported since likT is not log-concave') end n = max([length(y),length(mu),length(s2)]); on = ones(n,1); y = y(:).*on; mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only % since we are not aware of an analytical expression of the integral, % we use Gaussian-Hermite quadrature N = 20; [t,w] = gauher(N); oN = ones(1,N); lZ = likGumbel(sign, hyp, y*oN, sig*t'+mu*oN, []); lZ = log_expA_x(lZ,w); % log( exp(lZ)*w ) varargout = {lZ}; case 'infVB' error('infVB not supported') end end % computes y = log( exp(A)*x ) in a numerically safe way by subtracting the % maximal value in each row to avoid cancelation after taking the exp function y = log_expA_x(A,x) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns y = log(exp(A-maxA*ones(1,N))*x) + maxA; % exp(A) = exp(A-max(A))*exp(max(A))gpml-matlab-v4.0-2016-10-19/lik/likNegBinom.m000644 000765 000024 00000011424 12777177045 020235 0ustar00carlstaff000000 000000 function [varargout] = likNegBinom(link, hyp, y, mu, s2, inf, i) % likNegBinom - Negative binomial likelihood function for count data y. % The expression for the likelihood is % likNegBinom(f) = 1/Z * mu^y / (r+mu)^(r+y), Z = r^r*G(y+r)/(G(y+1)*G(r)) % with G(t)=gamma(t)=(t-1)!, mean=mu and variance=mu*(mu+r)/r, where r is the % number of failures parameters, mu = g(f) is the negative binomial intensity, % f is a Gaussian process and y is the non-negative integer count data. % Hence, we have -- with log(Z)=r*log(r)+L(y+r)-L(y+1)-L(r), L(t)=gammaln(t) -- % llik(f) = log(likNegBinom(f)) = y*log(mu)-(r+y)*log(r+mu)-log(Z). % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(r) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2016-10-04. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters if ~exist('psi'), mypsi = @digamma; else mypsi = @psi; end % no psi in Octave lr = hyp; r = exp(lr); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lZ = gammaln(y+1) + gammaln(r) - gammaln(y+r) - r*lr; lg = g(mu,link); mx = max(lg,lr); lgr = log(exp(lg-mx)+exp(lr-mx))+mx; % log(exp(lg)+r) lp = y.*lg - (y+r).*lgr - lZ; else lp = likNegBinom(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg.*(elg/r+1); % second y moment from negative binomial distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); lZ = gammaln(y+1) + gammaln(r) - gammaln(y+r) - r*lr; mx = max(lg,lr); lgr = log(exp(lg-mx)+exp(lr-mx))+mx; % log(exp(lg)+r) a = 1./(1+r./elg); da = a.*(1-a).*dlg; % auxiliary quantities if nargin<7 % no derivative mode lp = y.*lg - (y+r).*lgr - lZ; dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = y.*dlg - (y+r).*a.*dlg; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = y.*d2lg - (y+r).*(a.*d2lg + da.*dlg); if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = y.*d3lg - (y+r).*(a.*d3lg + da.*(3*d2lg +(1-2*a).*dlg.*dlg)); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode b = (y+r)./(elg+r); lp_dhyp = r*(1+log(r)-lgr-b-mypsi(r)+mypsi(y+r)); dlp_dhyp = r*dlg.*a.*(b-1); % first derivative d2lp_dhyp = r*((d2lg.*a+dlg.*da).*(b-1)-(dlg.*a).^2.*b); % and also second varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % hence we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likNegBinom,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if isequal(link,'exp') [varargout{:}] = glm_invlink_exp(f); elseif isequal(link,'logistic') [varargout{:}] = glm_invlink_logistic(f); else [varargout{:}] = glm_invlink_logistic2(link{2},f); endgpml-matlab-v4.0-2016-10-19/lik/likGauss.m000644 000765 000024 00000006631 12777227613 017622 0ustar00carlstaff000000 000000 function [varargout] = likGauss(hyp, y, mu, s2, inf, i) % likGauss - Gaussian likelihood function for regression. The expression for the % likelihood is % likGauss(t) = exp(-(t-y)^2/2*sn^2) / sqrt(2*pi*sn^2), % where y is the mean and sn is the standard deviation. % % The hyperparameters are: % % hyp = [ log(sn) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-11. % File automatically generated using noweb. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'1'}; return; end % report number of hyperparameters sn2 = exp(2*hyp); if nargin<5 % prediction mode if inf is not present if isempty(y), y = zeros(size(mu)); end s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lp = -(y-mu).^2./sn2/2-log(2*pi*sn2)/2; s2 = 0; else lp = likGauss(hyp, y, mu, s2, 'infEP'); % prediction end ymu = {}; ys2 = {}; if nargout>1 ymu = mu; % first y moment if nargout>2 ys2 = s2 + sn2; % second y moment end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' if nargin<6 % no derivative mode if isempty(y), y=0; end ymmu = y-mu; dlp = {}; d2lp = {}; d3lp = {}; lp = -ymmu.^2/(2*sn2) - log(2*pi*sn2)/2; if nargout>1 dlp = ymmu/sn2; % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = -ones(size(ymmu))/sn2; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = zeros(size(ymmu)); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode lp_dhyp = (y-mu).^2/sn2 - 1; % derivative of log likelihood w.r.t. hypers dlp_dhyp = 2*(mu-y)/sn2; % first derivative, d2lp_dhyp = 2*ones(size(mu))/sn2; % and also of the second mu derivative varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<6 % no derivative mode lZ = -(y-mu).^2./(sn2+s2)/2 - log(2*pi*(sn2+s2))/2; % log part function dlZ = {}; d2lZ = {}; if nargout>1 dlZ = (y-mu)./(sn2+s2); % 1st derivative w.r.t. mean if nargout>2 d2lZ = -1./(sn2+s2); % 2nd derivative w.r.t. mean end end varargout = {lZ,dlZ,d2lZ}; else % derivative mode dlZhyp = ((y-mu).^2./(sn2+s2)-1) ./ (1+s2./sn2); % deriv. w.r.t. hyp.lik varargout = {dlZhyp}; end case 'infVB' % variational lower site bound % t(s) = exp(-(y-s)^2/2sn2)/sqrt(2*pi*sn2) % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = zeros(n,1); y = y.*ones(n,1); z = y; varargout = {b,z}; end end gpml-matlab-v4.0-2016-10-19/lik/likGamma.m000644 000765 000024 00000011067 12777177046 017565 0ustar00carlstaff000000 000000 function [varargout] = likGamma(link, hyp, y, mu, s2, inf, i) % likGamma - Gamma likelihood function for strictly positive data y. The % expression for the likelihood is % likGamma(f) = al^al*y^(al-1)/gamma(al) * exp(-y*al/mu) / mu^al with % mean=mu and variance=mu^2/al where mu = g(f) is the Gamma intensity, f is a % Gaussian process, y is the strictly positive data. Hence, we have -- with % log(Zy) = log(gamma(al)) - al*log(al) + (1-al)*log(y) % llik(f) = log(likGamma(f)) = -al*( log(g(f)) + y/g(f) ) - log(Zy). % The larger one chooses al, the stronger the likelihood resembles a Gaussian % since skewness = 2/sqrt(al) and kurtosis = 6/al. % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % % Note that for neither link function the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ log(al) ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M. % % Copyright (c) by Hannes Nickisch, 2016-10-04. if nargin<4, varargout = {'1'}; return; end % report number of hyperparameters al = exp(hyp); if nargin<6 % prediction mode if inf is not present if numel(y)==0, y = zeros(size(mu)); end s2zero = 1; if nargin>4&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability lg = g(mu,link); lZy = gammaln(al) - al*log(al) + (1-al)*log(y); % normalisation constant lp = -al*(lg+y./exp(lg)) - lZy; else lp = likGamma(link, hyp, y, mu, s2, 'infEP'); end ymu = {}; ys2 = {}; if nargout>1 % compute y moments by quadrature n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only lg = g(sig*t'+mu*oN,link); ymu = exp(logsumexp2(lg+lw)); % first moment using Gaussian-Hermite quad if nargout>2 elg = exp(lg); yv = elg.^2/al; % second y moment from Gamma distribution ys2 = (yv+(elg-ymu*oN).^2)*w; end end varargout = {lp,ymu,ys2}; else switch inf case 'infLaplace' [lg,dlg,d2lg,d3lg] = g(mu,link); elg = exp(lg); if nargin<7 % no derivative mode lZy = gammaln(al) - al*log(al) + (1-al)*log(y); % normalisation constant lp = -al*(lg+y./elg) - lZy; dlp = {}; d2lp = {}; d3lp = {}; % return arguments if nargout>1 dlp = -al*dlg.*(1-y./elg); % dlp, derivative of log likelihood if nargout>2 % d2lp, 2nd derivative of log likelihood d2lp = -al*d2lg.*(1-y./elg) - al*dlg.*dlg.*y./elg; if nargout>3 % d3lp, 3rd derivative of log likelihood d3lp = -al*d3lg.*(1-y./elg) + al*dlg.*(dlg.*dlg-3*d2lg).*y./elg; end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode dlZy = al*psi(0,al) - al*(log(al) + 1 + log(y)); lp_dhyp = -al*(lg+y./elg) - dlZy; % derivative of log likelihood w.r.t. al dlp_dhyp = -al*dlg.*(1-y./elg); % first derivative d2lp_dhyp = -al*d2lg.*(1-y./elg) - al*dlg.*dlg.*y./elg; % and also second varargout = {lp_dhyp,dlp_dhyp,d2lp_dhyp}; end case 'infEP' if nargin<7 % no derivative mode % Since we are not aware of an analytical expression of the integral, % we use quadrature. varargout = cell(1,nargout); [varargout{:}] = lik_epquad({@likGamma,link},hyp,y,mu,s2); else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end end % compute the log intensity using the inverse link function function varargout = g(f,link) varargout = cell(nargout, 1); % allocate the right number of output arguments if isequal(link,'exp') [varargout{:}] = glm_invlink_exp(f); elseif isequal(link,'logistic') [varargout{:}] = glm_invlink_logistic(f); else [varargout{:}] = glm_invlink_logistic2(link{2},f); endgpml-matlab-v4.0-2016-10-19/lik/likErf.m000644 000765 000024 00000005637 12701705603 017244 0ustar00carlstaff000000 000000 function [varargout] = likErf(hyp, y, mu, s2, inf, i) % likErf - Error function or cumulative Gaussian likelihood function for binary % classification or probit regression. The expression for the likelihood is % likErf(t) = (1+erf(t/sqrt(2)))/2 = normcdf(t). % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2014-03-19. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'0'}; return; end % report number of hyperparameters if nargin>1, y = sign(y); y(y==0) = 1; else y = 1; end % allow only +/- 1 values if numel(y)==0, y = 1; end if nargin<5 % prediction mode if inf is not present y = y.*ones(size(mu)); % make y a vector s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation lp = logphi(y.*mu); else % prediction lp = likErf(hyp, y, mu, s2, 'infEP'); end p = exp(lp); ymu = {}; ys2 = {}; if nargout>1 ymu = 2*p-1; % first y moment if nargout>2 ys2 = 4*p.*(1-p); % second y moment end end varargout = {lp,ymu,ys2}; else % inference mode switch inf case 'infLaplace' if nargin<6 % no derivative mode f = mu; yf = y.*f; % product latents and labels varargout = cell(nargout,1); [varargout{:}] = logphi(yf); % query logphi if nargout>1 varargout{2} = y.*varargout{2}; if nargout>3, varargout{4} = y.*varargout{4}; end end else % derivative mode varargout = {[],[],[]}; % derivative w.r.t. hypers end case 'infEP' if nargin<6 % no derivative mode z = mu./sqrt(1+s2); dlZ = {}; d2lZ = {}; if numel(y)>0, z = z.*y; end if nargout<=1, lZ = logphi(z); % log part function else [lZ,n_p] = logphi(z); end if nargout>1 if numel(y)==0, y=1; end dlZ = y.*n_p./sqrt(1+s2); % 1st derivative wrt mean if nargout>2, d2lZ = -n_p.*(z+n_p)./(1+s2); end % 2nd derivative end varargout = {lZ,dlZ,d2lZ}; else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' error('infVB not supported') end endgpml-matlab-v4.0-2016-10-19/lik/likLogistic.m000644 000765 000024 00000013771 12701705603 020303 0ustar00carlstaff000000 000000 function [varargout] = likLogistic(hyp, y, mu, s2, inf, i) % likLogistic - logistic function for binary classification or logit regression. % The expression for the likelihood is % likLogistic(t) = 1./(1+exp(-t)). % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. The moments % \int f^k likLogistic(y,f) N(f|mu,var) df are calculated via a cumulative % Gaussian scale mixture approximation. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-09-02. % % See also LIKFUNCTIONS.M. if nargin<3, varargout = {'0'}; return; end % report number of hyperparameters if nargin>1, y = sign(y); y(y==0) = 1; else y = 1; end % allow only +/- 1 values if numel(y)==0, y = 1; end if nargin<5 % prediction mode if inf is not present y = y.*ones(size(mu)); % make y a vector s2zero = 1; if nargin>3&&numel(s2)>0&&norm(s2)>eps, s2zero = 0; end % s2==0 ? if s2zero % log probability evaluation yf = y.*mu; % product latents and labels lp = yf; ok = -351 p = exp(lp); ymu = 2*p-1; % first y moment if nargout>2 ys2 = 4*p.*(1-p); % second y moment end end varargout = {lp,ymu,ys2}; else % inference mode switch inf case 'infLaplace' if nargin<6 % no derivative mode f = mu; yf = y.*f; s = -yf; % product latents and labels dlp = {}; d2lp = {}; d3lp = {}; % return arguments ps = max(0,s); lp = -(ps+log(exp(-ps)+exp(s-ps))); % lp = -(log(1+exp(s))) if nargout>1 % first derivatives s = min(0,f); p = exp(s)./(exp(s)+exp(s-f)); % p = 1./(1+exp(-f)) dlp = (y+1)/2-p; % derivative of log likelihood if nargout>2 % 2nd derivative of log likelihood d2lp = -exp(2*s-f)./(exp(s)+exp(s-f)).^2; if nargout>3 % 3rd derivative of log likelihood d3lp = 2*d2lp.*(0.5-p); end end end varargout = {lp,dlp,d2lp,d3lp}; else % derivative mode varargout = {[],[],[]}; % derivative w.r.t. hypers end case 'infEP' if nargin<6 % no derivative mode y = y.*ones(size(mu)); % make y a vector % likLogistic(t) \approx 1/2 + \sum_{i=1}^5 (c_i/2) erf(lam_i/sqrt(2)t) lam = sqrt(2)*[0.44 0.41 0.40 0.39 0.36]; % approx coeffs lam_i and c_i c = [1.146480988574439e+02; -1.508871030070582e+03; 2.676085036831241e+03; -1.356294962039222e+03; 7.543285642111850e+01 ]; [lZc,dlZc,d2lZc] = likErf([], y*ones(1,5), mu*lam, s2*(lam.^2), inf); lZ = log_expA_x(lZc,c); % A=lZc, B=dlZc, d=c.*lam', lZ=log(exp(A)*c) dlZ = expABz_expAx(lZc, c, dlZc, c.*lam'); % ((exp(A).*B)*d)./(exp(A)*c) % d2lZ = ((exp(A).*Z)*e)./(exp(A)*c) - dlZ.^2 where e = c.*(lam.^2)' d2lZ = expABz_expAx(lZc, c, dlZc.^2+d2lZc, c.*(lam.^2)') - dlZ.^2; % The scale mixture approximation does not capture the correct asymptotic % behavior; we have linear decay instead of quadratic decay as suggested % by the scale mixture approximation. By observing that for large values % of -f*y ln(p(y|f)) for likLogistic is linear in f with slope y, we are % able to analytically integrate the tail region. val = abs(mu)-196/200*s2-4; % empirically determined bound at val==0 lam = 1./(1+exp(-10*val)); % interpolation weights lZtail = min(s2/2-abs(mu),-0.1); % apply the same to p(y|f) = 1 - p(-y|f) dlZtail = -sign(mu); d2lZtail = zeros(size(mu)); id = y.*mu>0; lZtail(id) = log(1-exp(lZtail(id))); % label and mean agree dlZtail(id) = 0; lZ = (1-lam).* lZ + lam.* lZtail; % interpolate between scale .. dlZ = (1-lam).* dlZ + lam.* dlZtail; % .. mixture and .. d2lZ = (1-lam).*d2lZ + lam.*d2lZtail; % .. tail approximation varargout = {lZ,dlZ,d2lZ}; else % derivative mode varargout = {[]}; % deriv. wrt hyp.lik end case 'infVB' % variational lower site bound % using -log(1+exp(-s)) = s/2 -log( 2*cosh(s/2) ); % the bound has the form: (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 n = numel(s2); b = (y/2).*ones(n,1); z = zeros(size(b)); varargout = {b,z}; end end % computes y = log( exp(A)*x ) in a numerically safe way by subtracting the % maximal value in each row to avoid cancelation after taking the exp function y = log_expA_x(A,x) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns y = log(exp(A-maxA*ones(1,N))*x) + maxA; % exp(A) = exp(A-max(A))*exp(max(A)) % computes y = ( (exp(A).*B)*z ) ./ ( exp(A)*x ) in a numerically safe way % The function is not general in the sense that it yields correct values for % all types of inputs. We assume that the values are close together. function y = expABz_expAx(A,x,B,z) N = size(A,2); maxA = max(A,[],2); % number of columns, max over columns A = A-maxA*ones(1,N); % subtract maximum value y = ( (exp(A).*B)*z ) ./ ( exp(A)*x ); gpml-matlab-v4.0-2016-10-19/lik/likExp.m000644 000765 000024 00000002600 12701705603 017247 0ustar00carlstaff000000 000000 function [varargout] = likExp(link, hyp, varargin) % likExp - Exponential likelihood function for strictly positive data y. The % expression for the likelihood is % likExp(f) = exp(-y/mu) / mu with % mean=mu and variance=mu^2 where mu = g(f) is the intensity, f is a % Gaussian process, y is the strictly positive data. Hence, we have % llik(f) = log(likExp(f)) = -y/g(f) - log(g(f)). % % Internally, the function is in fact a wrapper around likGamma with hyper % parameter al=1. % % We provide two inverse link functions 'exp' and 'logistic': % g(f) = exp(f) and g(f) = log(1+exp(f))). % The link functions are located at util/glm_invlink_*.m. % Note that for the 'exp' intensity the likelihood lik(f) is log concave. % % The hyperparameters are: % % hyp = [ ] % % Several modes are provided, for computing likelihoods, derivatives and moments % respectively, see likFunctions.m for the details. In general, care is taken % to avoid numerical issues when the arguments are extreme. % % See also LIKFUNCTIONS.M, LIKGAMMA.M. % % Copyright (c) by Hannes Nickisch, 2013-10-29. if nargin<4, varargout = {'0'}; return; end % report number of hyperparameters varargout = cell(nargout, 1); % allocate the right number of output arguments for j=1:nargout, varargout{j} = []; end % derivative output if nargin<=6, [varargout{:}] = likGamma(link,0,varargin{:}); end % log(al) = 0gpml-matlab-v4.0-2016-10-19/prior/priorLogNormal.m000644 000765 000024 00000001666 12701705626 021355 0ustar00carlstaff000000 000000 function [lp,dlp]=priorLogNormal(mu,s2,x) % Univariate Log-normal hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-(log(x)-mu)^2/(2*s2)) / (sqrt(2*pi*s2)*x) % % where mu(1x1) is the log scale parameter, s2(1x1) is the shape parameter % and x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('mu and s2 parameters need to be provided'), end if ~(isscalar(mu)&&isscalar(s2)) error('mu and s2 parameters need to be scalars'), end if nargin<3, lp = exp(sqrt(s2)*randn+mu); return, end % return a sample lx = log(x); lp = -(lx-mu).^2/(2*s2) - log(2*pi*s2)/2 - lx; dlp = -(lx-mu+s2)./(x*s2); lp(x<0) = -inf; dlp(x<0) = 0;gpml-matlab-v4.0-2016-10-19/prior/priorSmoothBox1.m000644 000765 000024 00000003353 12701705626 021461 0ustar00carlstaff000000 000000 function [lp,dlp] = priorSmoothBox1(a,b,eta,x) % Univariate smoothed box prior distribution with linear decay in the log domain % and infinite support over the whole real axis. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = 1/w*sigmoid(eta*(x-a))*(1-sigmoid(eta*(x-b))), % where w = abs(b-a) and sigmoid(z) = 1/(1+exp(-z)) % % a(1x1) is the lower bound parameter, b(1x1) is the upper bound parameter, % eta(1x1)>0 is the slope parameter and x(1xN) contains query hyperparameters % for prior evaluation. Larger values of eta make the distribution more % box-like. % % The distribution p(x) has mean and variance given by: % mu = (a+b)/2 and s2 = w^2/(1-exp(-2*g))*(1+(pi/g)^2)/12, g = eta*w/2. % % /------------\ % / \ % -------- | | --------> x % a b % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Jose Vallet and Hannes Nickisch, 2015-03-27. % % See also PRIORDISTRIBUTIONS.M. if nargin<3, error('a, b and eta parameters need to be provided'), end if b<=a, error('b must be greater than a.'), end if ~(isscalar(a)&&isscalar(b)&&isscalar(eta)) error('a, b and eta parameters need to be scalar values') end if nargin<4 % inverse sampling u = exp((b-a)*eta*rand()); lp = log((u-1)/(exp(-eta*a)-u*exp(-eta*b)))/eta; return end [lpa,dlpa] = logr(eta*(x-a)); [lpb,dlpb] = logr(-eta*(x-b)); lp = lpa + lpb - log(b-a) + log(1-exp((a-b)*eta)); dlp = eta*(dlpa - dlpb); % r(z) = 1/(1+exp(-z)), log(r(z)) = -log(1+exp(-z)) function [lr,dlr] = logr(z) lr = z; ok = -351 % both mu/s2 scalar => inflate D = size(x,1); mu = mu*ones(D,1); s = s*ones(D,1); lds = D*lds; end if ~(ndims(x)==2 && size(x,1)==D), error('x needs to be (Dxn)'), end oN = ones(1,size(x,2)); if size(s,2)==D xm = x-mu*oN; xs = s\xm; else xm = x-mu*oN; xs = xm./(s*oN); end dlp = -sqrt(2)*sign(xs); lp = -sqrt(2)*sum(abs(xs),1) - D*log(2)/2 - lds; if size(s,2)==D, dlp = s'\dlp; else dlp = dlp./(s*oN); endgpml-matlab-v4.0-2016-10-19/prior/priorLaplace.m000644 000765 000024 00000001476 12701705626 021023 0ustar00carlstaff000000 000000 function [lp,dlp] = priorLaplace(mu,s2,x) % Univariate Laplacian hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-abs(x-mu)/b)/(2*b), where b = sqrt(s2/2), % % mu(1x1) is the mean parameter, s2(1x1) is the variance parameter and % x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-09. % % See also PRIORDISTRIBUTIONS.M. if nargin<3 % return a sample u = rand-1/2; lp = mu - sqrt(s2/2)*sign(u)*log(1-2*abs(u)); return end b = sqrt(s2/2); lp = -abs(x-mu)/b - log(2*b); dlp = -sign(x-mu)/b;gpml-matlab-v4.0-2016-10-19/prior/priorWeibull.m000644 000765 000024 00000001654 12701705626 021063 0ustar00carlstaff000000 000000 function [lp,dlp] = priorWeibull(lam,k,x) % Univariate Weibull hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = (k/lam) * (x/lam)^(k-1) * exp(-(x/lam)^k) % % where lam(1x1) is the scale parameter, k(1x1) is the scale parameter % and x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('lam and k parameters need to be provided'), end if ~(isscalar(lam)&&isscalar(k)) error('lam and k parameters need to be scalars'), end if nargin<3,lp = lam*(-log(rand))^(1/k); return, end % return a sample lp = log(k/lam) + (k-1)*log(x/lam) - (x/lam).^k; dlp = (k-1)./x - (k/lam)*(x/lam).^(k-1); lp(x<0) = -inf; dlp(x<0) = 0;gpml-matlab-v4.0-2016-10-19/prior/priorSmoothBox2.m000644 000765 000024 00000003762 12701705626 021466 0ustar00carlstaff000000 000000 function [lp,dlp] = priorSmoothBox2(a,b,eta,x) % Univariate smoothed box prior distribution with quadratic decay in the log % domain and infinite support over the whole real axis. % The plateau pdf is built by cutting a Gaussian into two parts and % inserting a uniform distribution from a to b. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % / G(x|a,s^2), x<=a, % p(x) = 1/(w*(1/eta+1)) * | 1 , a0 is the slope parameter and x(1xN) contains query hyperparameters % for prior evaluation. Larger values of eta make the distribution more % box-like. % % The distribution p(x) has mean and variance given by: % mu = (a+b)/2 and s2 = (w/2)^2*(eta^3/3+eta^2+4*eta/pi+2/pi)/(eta^3+eta^2). % % /------------\ % / \ % -------- | | --------> x % a b % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Jose Vallet and Hannes Nickisch, 2015-03-27. % % See also PRIORDISTRIBUTIONS.M. if nargin<3, error('a, b and eta parameters need to be provided'), end if b<=a, error('b must be greater than a.'), end if ~(isscalar(a)&&isscalar(b)&&isscalar(eta)) error('a, b and eta parameters need to be scalar values') end w = abs(b-a); sab = w/(eta*sqrt(2*pi)); % box width and boundary slope if nargin<4 % return a sample if randb; lp(i) = lp(i) - (x(i)-b).^2/(2*sab^2); dlp(i) = (b-x(i))/sab^2;gpml-matlab-v4.0-2016-10-19/prior/priorDeltaMulti.m000644 000765 000024 00000001032 12701705626 021512 0ustar00carlstaff000000 000000 function [lp,dlp] = priorDeltaMulti(x) % Dummy hyperparameter prior distribution to fix the value of a hyperparameter. % The function is not intended to be evaluated but exists merely to make % the user aware of the possibility to use it. The function is equivalent % to priorClampedMulti. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-12-06. % % See also PRIORDISTRIBUTIONS.M. error('The function is not intended to be called directly.')gpml-matlab-v4.0-2016-10-19/prior/priorInvGauss.m000644 000765 000024 00000002174 12701705626 021215 0ustar00carlstaff000000 000000 function [lp,dlp] = priorInvGauss(mu,lam,x) % Univariate Inverse Gaussian hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-lam*(x-mu)^2/(2*mu^2*x)) / sqrt(2*pi*x^3/lam) % % where mu(1x1) is the mean parameter, lam(1x1) is the scale parameter % and x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('mu and lam parameters need to be provided'), end if ~(isscalar(mu)&&isscalar(lam)) error('mu and lam parameters need to be scalars'),end if nargin<3 % return a sample n = randn; y = n*n; r = mu + mu*(mu*y-sqrt(4*mu*lam*y+mu^2*y^2))/(2*lam); z = rand; if z<=mu/(mu+r) lp = r; else lp = mu^2/r; end return end lp = -lam*(x-mu).^2./(2*mu^2*x) - log(2*pi*x.^3/lam)/2; q = (x-mu)./x; dlp = -lam*q.*(2-q)/(2*mu^2) - 3./(2*x); lp(x<0) = -inf; dlp(x<0) = 0;gpml-matlab-v4.0-2016-10-19/prior/priorDelta.m000644 000765 000024 00000001020 12701705626 020474 0ustar00carlstaff000000 000000 function [lp,dlp] = priorDelta(x) % Dummy hyperparameter prior distribution to fix the value of a hyperparameter. % The function is not intended to be evaluated but exists merely to make % the user aware of the possibility to use it. The function is equivalent % to priorClamped. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-12-06. % % See also PRIORDISTRIBUTIONS.M. error('The function is not intended to be called directly.')gpml-matlab-v4.0-2016-10-19/prior/priorGaussMulti.m000644 000765 000024 00000003466 12701705626 021560 0ustar00carlstaff000000 000000 function [lp,dlp] = priorGaussMulti(mu,s2,x) % Multivariate Gaussian hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-r2/2) / sqrt(det(2*pi*s2)) where r2(x) = (x-mu)'*inv(s2)*(x-mu), % % mu(Dx1) is the mean parameter, s2(Dx1) or s2(DxD) is the variance parameter % and x(DxN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-12. % % See also PRIORDISTRIBUTIONS.M, PRIORGAUSS.M. if nargin<2, error('mu and s2 parameters need to be provided'), end if ndims(mu)~=2 || size(mu,2)~=1, error('mu needs to be (Dx1)'), end D = size(mu,1); s2_ok = ndims(s2)==2 && all(size(s2)==[D,1] | size(s2)==[D,D]); if ~s2_ok, error('s2 needs to be (DxD) or (Dx1)'), end if size(s2,2)==D % full multivariate case s = chol(s2)'; lds = sum(log(diag(s))); % lds = log(det(s2))/2 else % diagonal covariance s = sqrt(s2); lds = sum(log(s)); end if nargin<3 % return a random sample lp = randn(D,1); % unit sample if size(s,2)==D, lp = s*lp+mu; else lp = s.*lp+mu; end % affine transformation return end if D==1 && size(x,1)>1 % both mu/s2 scalar => inflate D = size(x,1); mu = mu*ones(D,1); s = s*ones(D,1); lds = D*lds; end if ~(ndims(x)==2 && size(x,1)==D), error('x needs to be (Dxn)'), end oN = ones(1,size(x,2)); if size(s,2)==D xs = s\(x-mu*oN); dlp = -s'\xs; else xs = (x-mu*oN)./(s*oN); dlp = -xs./(s*oN); end lp = -sum(xs.^2,1)/2 - D*log(2*pi)/2 - lds;gpml-matlab-v4.0-2016-10-19/prior/priorGamma.m000644 000765 000024 00000002515 12701705626 020477 0ustar00carlstaff000000 000000 function [lp,dlp] = priorGamma(k,t,x) % Univariate Gamma hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(x/t)/gamma(k)*x^(k-1)/t^k % % where k(1x1) is the shape parameter, t(1x1) is the scale parameter % and x(1xN) contains query hyperparameters for prior evaluation. % % Sampling is done using the algorithm from p. 53 of the paper Generating gamma % variates by a modified rejection technique by J.H. Ahrens and U. Dieter, 1982, % ACM, 25, 47–54. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('k and t parameters need to be provided'), end if ~(isscalar(k)&&isscalar(t)) error('k and t parameters need to be scalars'), end if nargin<3 m = 1; d = k-floor(k); % fractional part v0 = exp(1)/(exp(1)+d); while true v = rand(3,1); if v(1)<=v0 r = v(2)^(1/d); s = v(3)*r^(d-1); else r = 1-log(v(2)); s = v(3)*exp(-r); end if s<=r^(d-1)*exp(-r), break, end m = m+1; end u = rand(floor(k),1); lp = t*(r-sum(log(u))); return end lx = log(x); lp = -gammaln(k) - k*log(t) + (k-1)*lx - x/t; dlp = (k-1)./x - 1/t; lp(x<0) = -inf; dlp(x<0) = 0;gpml-matlab-v4.0-2016-10-19/prior/priorGauss.m000644 000765 000024 00000001565 12701705626 020543 0ustar00carlstaff000000 000000 function [lp,dlp] = priorGauss(mu,s2,x) % Univariate Gaussian hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = exp(-(x-mu)^2/(2*s2)) / sqrt(2*pi*s2), where % % mu(1x1) is the mean parameter, s2(1x1) is the variance parameter and % x(1xN) contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-09. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('mu and s2 parameters need to be provided'), end if ~(isscalar(mu)&&isscalar(s2)) error('mu and s2 parameters need to be scalars') end if nargin<3, lp = sqrt(s2)*randn+mu; return, end % return a sample lp = -(x-mu).^2/(2*s2) - log(2*pi*s2)/2; dlp = -(x-mu)/s2;gpml-matlab-v4.0-2016-10-19/prior/priorClamped.m000644 000765 000024 00000001020 12701705626 021010 0ustar00carlstaff000000 000000 function [lp,dlp] = priorClamped(x) % Dummy hyperparameter prior distribution to fix the value of a hyperparameter. % The function is not intended to be evaluated but exists merely to make % the user aware of the possibility to use it. The function is equivalent % to priorDelta. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-12-06. % % See also PRIORDISTRIBUTIONS.M. error('The function is not intended to be called directly.')gpml-matlab-v4.0-2016-10-19/prior/priorMix.m000644 000765 000024 00000002362 12701705626 020212 0ustar00carlstaff000000 000000 function [lp,dlp] = priorMix(w,priors,x) % Mixture of Univariate or Multivariate hyperparameter prior distributions. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = sum_i w_i p_i(x), % % w(bx1) is the mixture weight vector (nonnegative, normalised to add up to 1), % priors(bx1) is a cell array containing the priors to be mixed and x(1xN) % contains query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<2, error('specification of w and priors required'), end n = numel(w); w = w(:); if abs(sum(w)-1)>1e-9, error('w needs to sum up to 1'), end if any(w<0), error('w needs to be nonnegative'), end for i=1:n, if ~iscell(priors{i}), priors{i} = {priors{i}}; end, end % sample if nargin<3, i = 1+nnz(rand>cumsum(w)); lp = feval(priors{i}{:}); return, end lpi = zeros(numel(x),n); dlpi = zeros(numel(x),n); for i=1:n, [lpi(:,i),dlpi(:,i)] = feval(priors{i}{:},x(:)); end mx = max(lpi,[],2); lp = log(exp(lpi-mx*ones(1,n))*w) + mx; pi = exp(lpi); dlp = ((pi.*dlpi)*w)./(pi*w);gpml-matlab-v4.0-2016-10-19/prior/priorTransform.m000644 000765 000024 00000002256 12701705626 021432 0ustar00carlstaff000000 000000 function [lp,dlp] = priorTransform(g,dg,ig,prior,x) % Transformed univariate hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % prior(y), where y = g(x), % % g is the transformation function (monotonically increasing and invertable), % dg is its derivative, ig is its inverse, prior is the distribution to be % transformed and x(1xN) contains query hyperparameters for prior evaluation. % Note that p(x) = prior(g(x)) is not necessarily normalised w.r.t. x. % % For example, to put a nonnegative gamma prior on the lenthscale % parameter ell of covSEiso, one can use % {@priorTransform,@exp,@exp,@log,{@priorGamma,k,t}} % since ell is represented in the log domain. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-09-08. % % See also PRIORDISTRIBUTIONS.M. if nargin<4, error('g, dg, ig and prior parameters need to be provided'), end if nargin<5, lp = ig(feval(prior{:})); return, end % apply inverse sampling if ~iscell(prior), prior = {prior}; end [lp,dlp] = feval(prior{:},g(x)); dlp = dlp.*dg(x);gpml-matlab-v4.0-2016-10-19/prior/priorT.m000644 000765 000024 00000002170 12701705626 017655 0ustar00carlstaff000000 000000 function [lp,dlp] = priorT(mu,s2,nu,x) % Univariate Student's t hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = ( 1 + (x-mu)^2/(s2*(nu-2)) )^(-(nu+1)/2) / Z, where % Z = gamma(nu/2) * sqrt(pi*(nu-2)*s2) / gamma((nu+1)/2), % % mu(1x1) is the mean parameter, s2(1x1) is the variance parameter, % nu(1x1) > 2 is the degrees of freedom parameter and x(1xN) contains query % hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-10-16. % % See also PRIORDISTRIBUTIONS.M. if nargin<3, error('mu, s2 and nu parameters need to be provided'), end if ~(isscalar(mu)&&isscalar(s2)&&isscalar(nu)) error('mu, s2 and nu parameters need to be scalars') end if nargin<4, lp = mu + sqrt(s2*(nu-2)/priorGamma(nu/2,2))*randn; return, end lZ = gammaln((nu+1)/2)-gammaln(nu/2)-log(pi*(nu-2)*s2)/2; lp = -(nu+1)/2*log(1+(x-mu).^2/(s2*(nu-2))) + lZ; dlp = (nu+1)*(mu-x)./(mu^2-2*mu*x+s2*(nu-2)+x.^2);gpml-matlab-v4.0-2016-10-19/prior/priorTMulti.m000644 000765 000024 00000004270 12701705626 020673 0ustar00carlstaff000000 000000 function [lp,dlp] = priorTMulti(mu,s2,nu,x) % Multivariate Laplace hyperparameter prior distribution. % Compute log-likelihood and its derivative or draw a random sample. % The prior distribution is parameterized as: % % p(x) = ( 1 + r2/(nu-2) )^(-(nu+D)/2) / Z, where % Z = gamma(nu/2) * sqrt(det(pi*(nu-2)*s2)) / gamma((nu+D)/2), and % r2(x) = (x-mu)'*inv(s2)*(x-mu), % % further mu(Dx1) is the mean parameter, s2(Dx1) or s2(DxD) is the variance % parameter, nu(1x1) > 2 is the degrees of freedom parameter and x(DxN) contains % query hyperparameters for prior evaluation. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Hannes Nickisch, 2014-10-16. % % See also PRIORDISTRIBUTIONS.M, PRIORLAPLACE.M. if nargin<3, error('mu, s2 and nu parameters need to be provided'), end if ndims(mu)~=2 || size(mu,2)~=1, error('mu needs to be (Dx1)'), end D = size(mu,1); s2_ok = ndims(s2)==2 && all(size(s2)==[D,1] | size(s2)==[D,D]); if ~s2_ok, error('s2 needs to be (DxD) or (Dx1)'), end if size(s2,2)==D % full multivariate case s = chol(s2)'; lds = sum(log(diag(s))); % lds = log(det(s2))/2 else % diagonal covariance s = sqrt(s2); lds = sum(log(s)); end if ~isscalar(nu), error('nu parameter needs to be a scalar'), end if nargin<4 % return a random sample lp = randn(D,1); for d=1:D, lp(d) = sqrt((nu-2)/priorGamma(nu/2,2)) * lp(d); end % unit sample if size(s,2)==D, lp = s*lp+mu; else lp = s.*lp+mu; end % affine transformation return end if D==1 && size(x,1)>1 % both mu/s2 scalar => inflate D = size(x,1); mu = mu*ones(D,1); s = s*ones(D,1); lds = D*lds; end if ~(ndims(x)==2 && size(x,1)==D), error('x needs to be (Dxn)'), end oN = ones(1,size(x,2)); oD = ones(D,1); if size(s,2)==D xm = x-mu*oN; xs = s\xm; else xm = x-mu*oN; xs = xm./(s*oN); end lZ = gammaln((nu+D)/2) - gammaln(nu/2) - D*log(pi*(nu-2))/2 - lds; r2 = sum(xs.^2,1); lp = -(nu+D)/2*log(1+r2/(nu-2)) + lZ; dlp = -(nu+D)/(nu-2)*xs./(oD*(1+r2/(nu-2))); if size(s,2)==D, dlp = s'\dlp; else dlp = dlp./(s*oN); endgpml-matlab-v4.0-2016-10-19/prior/priorClampedMulti.m000644 000765 000024 00000001032 12701705626 022026 0ustar00carlstaff000000 000000 function [lp,dlp] = priorClampedMulti(x) % Dummy hyperparameter prior distribution to fix the value of a hyperparameter. % The function is not intended to be evaluated but exists merely to make % the user aware of the possibility to use it. The function is equivalent % to priorDeltaMulti. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2014-12-06. % % See also PRIORDISTRIBUTIONS.M. error('The function is not intended to be called directly.')gpml-matlab-v4.0-2016-10-19/inf/infEP.m000644 000765 000024 00000013366 12777200551 017031 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infEP(hyp, mean, cov, lik, x, y) % Expectation Propagation approximation to the posterior Gaussian Process. % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m. See also infMethods.m. In the EP algorithm, the sites are % updated in random order, for better performance when cases are ordered % according to the targets. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2016-05-02. % % See also INFMETHODS.M. persistent last_ttau last_tnu % keep tilde parameters between calls tol = 1e-4; max_sweep = 10; min_sweep = 2; % tolerance to stop EP iterations inf = 'infEP'; n = size(x,1); if isnumeric(cov), K = cov; % use provided covariance matrix else [K,dK] = feval(cov{:}, hyp.cov, x); end % covariance matrix and deriv if isnumeric(mean), m = mean; % use provided mean vector else [m,dm] = feval(mean{:}, hyp.mean, x); end % mean vector and deriv % A note on naming: variables are given short but descriptive names in % accordance with Rasmussen & Williams "GPs for Machine Learning" (2006): mu % and s2 are mean and variance, nu and tau are natural parameters. A leading t % means tilde, a subscript _ni means "not i" (for cavity parameters), or _n % for a vector of cavity parameters. N(f|mu,Sigma) is the posterior. % marginal likelihood for ttau = tnu = zeros(n,1); equals n*log(2) for likCum* nlZ0 = -sum(feval(lik{:}, hyp.lik, y, m, diag(K), inf)); if any(size(last_ttau) ~= [n 1]) % find starting point for tilde parameters ttau = zeros(n,1); tnu = zeros(n,1); % init to zero if no better guess Sigma = K; % initialize Sigma and mu, the parameters of .. mu = m; nlZ = nlZ0; % .. the Gaussian posterior approximation else ttau = last_ttau; tnu = last_tnu; % try the tilde values from previous call [Sigma,mu,L,alpha,nlZ] = epComputeParams(K, y, ttau, tnu, lik, hyp, m, inf); if nlZ > nlZ0 % if zero is better .. ttau = zeros(n,1); tnu = zeros(n,1); % .. then init with zero instead Sigma = K; % initialize Sigma and mu, the parameters of .. mu = m; nlZ = nlZ0; % .. the Gaussian posterior approximation end end nlZ_old = Inf; sweep = 0; % converged, max. sweeps or min. sweeps? while (abs(nlZ-nlZ_old) > tol && sweep < max_sweep) || sweep tol error('maximum number of sweeps exceeded in function infEP') end last_ttau = ttau; last_tnu = tnu; % remember for next call post.alpha = alpha; post.sW = sqrt(ttau); post.L = L; % return posterior params if nargout>2 % do we want derivatives? dnlZ = hyp; % allocate space for derivatives tau_n = 1./diag(Sigma)-ttau; % compute the log marginal likelihood nu_n = mu./diag(Sigma)-tnu; % vectors of cavity parameters sW = sqrt(ttau); F = alpha*alpha'-repmat(sW,1,n).*solve_chol(L,diag(sW)); % covariance hypers dnlZ.cov = -dK(F)/2; for i = 1:numel(hyp.lik) % likelihood hypers dlik = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf, i); dnlZ.lik(i) = -sum(dlik); end [junk,dlZ] = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf);% mean hyps dnlZ.mean = -dm(dlZ); end % function to compute the parameters of the Gaussian approximation, Sigma and % mu, and the negative log marginal likelihood, nlZ, from the current site % parameters, ttau and tnu. Also returns L (useful for predictions). function [Sigma,mu,L,alpha,nlZ] = epComputeParams(K,y,ttau,tnu,lik,hyp,m,inf) n = length(y); % number of training cases sW = sqrt(ttau); % compute Sigma and mu L = chol(eye(n)+sW*sW'.*K); % L'*L=B=eye(n)+sW*K*sW V = L'\(repmat(sW,1,n).*K); Sigma = K - V'*V; alpha = tnu-sW.*solve_chol(L,sW.*(K*tnu+m)); mu = K*alpha+m; v = diag(Sigma); tau_n = 1./diag(Sigma)-ttau; % compute the log marginal likelihood nu_n = mu./diag(Sigma)-tnu; % vectors of cavity parameters lZ = feval(lik{:}, hyp.lik, y, nu_n./tau_n, 1./tau_n, inf); p = tnu-m.*ttau; q = nu_n-m.*tau_n; % auxiliary vectors nlZ = sum(log(diag(L))) - sum(lZ) - p'*Sigma*p/2 + (v'*p.^2)/2 ... - q'*((ttau./tau_n.*q-2*p).*v)/2 - sum(log(1+ttau./tau_n))/2; gpml-matlab-v4.0-2016-10-19/inf/infVB.m000644 000765 000024 00000013520 12777200706 017026 0ustar00carlstaff000000 000000 function [post, nlZ, dnlZ] = infVB(hyp, mean, cov, lik, x, y, opt) % Variational approximation to the posterior Gaussian process. % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m. See also infMethods.m. % % Minimisation of an upper bound on the negative marginal likelihood using a % sequence of infLaplace calls where the smoothed likelihood % likVB(f) = lik(..,g,..) * exp(b*(f-g)), g = sign(f-z)*sqrt((f-z)^2+v)+z, where % v .. marginal variance = (positive) smoothing width, and % lik .. lik function such that p(y|f)=lik(..,f,..). % % The problem is convex whenever the likelihood is log-concave. At the end, the % optimal width W is obtained analytically. % % Copyright (c) by Hannes Nickisch 2016-05-02. % % See also INFMETHODS.M. n = size(x,1); if isnumeric(cov), K = cov; % use provided covariance matrix else [K,dK] = feval(cov{:}, hyp.cov, x); end % covariance matrix and deriv if isnumeric(mean), m = mean; % use provided mean vector else [m,dm] = feval(mean{:}, hyp.mean, x); end % mean vector and deriv if iscell(lik), likstr = lik{1}; else likstr = lik; end if ~ischar(likstr), likstr = func2str(likstr); end if nargin<=6, opt = []; end % make opt variable available if ~isfield(opt,'postL'), opt.postL = false; end % not compute L in infLaplace if isfield(opt,'out_nmax'), out_nmax = opt.out_nmax; % maximal no of outer loops else out_nmax = 15; end % default value if isfield(opt,'out_tol'), out_tol = opt.out_tol; % outer loop convergence else out_tol = 1e-5; end % default value sW = ones(n,1); % init with some reasonable value opt.postL = false; % avoid computation of L inside infLaplace for i=1:out_nmax U = chol(eye(n)+sW*sW'.*K)'\(repmat(sW,1,n).*K); v = diag(K) - sum(U.*U,1)'; % v = diag( inv(inv(K)+diag(sW.*sW)) ) post = infLaplace(hyp, m, K, {@likVB,v,lik}, x, y, opt); % post.sW is very different from the optimal sW for non Gaussian likelihoods sW_old = sW; f = m+K*post.alpha; % posterior mean [lp,dlp,d2lp,sW,b,z] = feval(@likVB,v,lik,hyp.lik,y,f); if max(abs(sW-sW_old))2 % do we want derivatives? iKtil = repmat(sW,1,n).*solve_chol(post.L,diag(sW));% sW*B^-1*sW=inv(K+inv(W)) dnlZ = hyp; % allocate space for derivatives w = iKtil*(c.*ga); dnlZ.cov = dK(iKtil-w*w')/2; % covariance hypers if ~strcmp(likstr,'likGauss') % likelihood hypers for j=1:length(hyp.lik) sign_fmz = 2*(f-z>=0)-1; % strict sign mapping; sign(0) = 1 g = sign_fmz.*sqrt((f-z).^2 + v) + z; dhhyp = -2*feval(lik{:},hyp.lik,y,g,[],'infLaplace',j); dnlZ.lik(j) = sum(dhhyp)/2; end else % special treatment for the Gaussian case dnlZ.lik = sum(sum( (post.L'\eye(n)).^2 )) - exp(2*hyp.lik)*(alpha'*alpha); end dnlZ.mean = -dm(alpha); % mean hypers end % Smoothed likelihood function; instead of p(y|f)=lik(..,f,..) compute % likVB(f) = lik(..,g,..)*exp(b*(f-g)), g = sign(f-z)*sqrt((f-z)^2+v)+z, where % v .. marginal variance = (positive) smoothing width, and % lik .. lik function such that feval(lik{:},varargin{:}) yields a result. % The smoothing results from a lower bound on the likelihood: % p(y|f) \ge exp( (b+z/ga)*f - f.^2/(2*ga) - h(ga)/2 ) function [varargout] = likVB(v, lik, varargin) [b,z] = feval(lik{:},varargin{1:2},[],zeros(size(v)),'infVB'); f = varargin{3}; % obtain location of evaluation sign_fmz = 2*(f-z>=0)-1; % strict sign mapping; sign(0) = 1 g = sign_fmz.*sqrt((f-z).^2 + v) + z; varargin{3} = g; id = v==0 | abs(f./sqrt(v+eps))>1e10; % correct asymptotics of f -> +/-Inf varargout = cell(nargout,1); % allocate output, eval lik(..,g,..) [varargout{1:min(nargout,3)}] = feval(lik{:},varargin{1:3},[],'infLaplace'); if nargout>0 lp = varargout{1}; varargout{1} = lp + b.*(f-g); varargout{1}(id) = lp(id); % correct asymptotics of f -> +/-Inf if nargout>1 % first derivative dg_df = (abs(f-z)+eps)./(abs(g-z)+eps); % stabilised dg/df for v=0, f=0 dlp = varargout{2}; varargout{2} = dlp.*dg_df + b.*(1-dg_df); varargout{2}(id) = dlp(id); % correct asymptotics of f -> +/-Inf if nargout>2 % second derivative d2lp = varargout{3}; g_e = g-z + sign_fmz*eps; v_g3 = v./(g_e.*g_e.*g_e); % stabilised v./g.^3 to cover v=0 and f=0 varargout{3} = (dlp-b).*v_g3 + d2lp.*dg_df.*dg_df; varargout{3}(id) = d2lp(id); % correct asymptotics of f -> +/-Inf if nargout>3 W = abs( (b-dlp)./(g-z+sign_fmz/1.5e8) ); % optimal sW value varargout{4} = sqrt(W); if nargout>4 varargout{5} = b; if nargout>5 varargout{6} = z; end end end end end end gpml-matlab-v4.0-2016-10-19/inf/infGaussLik.m000644 000765 000024 00000003251 12777227613 020247 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infGaussLik(hyp, mean, cov, lik, x, y, opt) % Exact inference for a GP with Gaussian likelihood. Compute a parametrization % of the posterior, the negative log marginal likelihood and its derivatives % w.r.t. the hyperparameters. See also "help infMethods". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-10-11. % File automatically generated using noweb. % % See also INFMETHODS.M, APX.M. if nargin<7, opt = []; end % make sure parameter exists if iscell(lik), likstr = lik{1}; else likstr = lik; end if ~ischar(likstr), likstr = func2str(likstr); end if ~strcmp(likstr,'likGauss') % NOTE: no explicit call to likGauss error('Exact inference only possible with Gaussian likelihood'); end [n, D] = size(x); [m,dm] = feval(mean{:}, hyp.mean, x); % evaluate mean vector and deriv sn2 = exp(2*hyp.lik); W = ones(n,1)/sn2; % noise variance of likGauss K = apx(hyp,cov,x,opt); % set up covariance approximation [ldB2,solveKiW,dW,dhyp,post.L] = K.fun(W); % obtain functionality depending on W alpha = solveKiW(y-m); post.alpha = K.P(alpha); % return the posterior parameters post.sW = sqrt(W); % sqrt of noise precision vector if nargout>1 % do we want the marginal likelihood? nlZ = (y-m)'*alpha/2 + ldB2 + n*log(2*pi*sn2)/2; % -log marginal likelihood if nargout>2 % do we want derivatives? dnlZ = dhyp(alpha); dnlZ.mean = -dm(alpha); dnlZ.lik = -sn2*(alpha'*alpha) - 2*sum(dW)/sn2 + n; end end gpml-matlab-v4.0-2016-10-19/inf/infKL.m000644 000765 000024 00000023720 12777200641 017026 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infKL(hyp, mean, cov, lik, x, y) % Approximation to the posterior Gaussian Process by minimization of the % KL-divergence. The function is structurally very similar to infEP; the % only difference being the local divergence measure minimised. % In infEP, one minimises KL(p,q) whereas in infKL one minimises KL(q,p) % locally be iterating over the sites. For log-concave likelihoods, the % latter minimisation constitutes a 2d joint convex problem, so convergence % is guaranteed. % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m. See also infMethods.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2016-06-09. % % See also INFMETHODS.M. out = false; persistent last_ttau last_tnu % keep tilde parameters between calls tol = 1e-3; max_sweep = 15; min_sweep = 2; % tolerance to stop KL iterations n = size(x,1); if isnumeric(cov), K = cov; % use provided covariance matrix else [K,dK] = feval(cov{:}, hyp.cov, x); end % covariance matrix and deriv if isnumeric(mean), m = mean; % use provided mean vector else [m,dm] = feval(mean{:}, hyp.mean, x); end % mean vector and deriv % A note on naming: variables are given short but descriptive names in % accordance with Rasmussen & Williams "GPs for Machine Learning" (2006): mu % and s2 are mean and variance, nu and tau are natural parameters. A leading t % means tilde, a subscript _ni means "not i" (for cavity parameters), or _n % for a vector of cavity parameters. N(f|mu,Sigma) represents the posterior. % marginal likelihood for ttau = tnu = zeros(n,1); equals n*log(2) for likCum* nlZ0 = -sum(likKL(diag(K),lik,hyp.lik,y,m)); if any(size(last_ttau) ~= [n 1]) % find starting point for tilde parameters ttau = zeros(n,1); % initialize to zero if we have no better guess tnu = zeros(n,1); Sigma = K; % initialize Sigma and mu, the parameters of .. mu = m; % .. the Gaussian posterior approximation nlZ = nlZ0; else ttau = last_ttau; % try the tilde values from previous call tnu = last_tnu; [Sigma,mu,L,alpha,nlZ] = klComputeParams(K,y,ttau,tnu,lik,hyp,m); if nlZ > nlZ0 % if zero is better .. ttau = zeros(n,1); % .. then initialize with zero instead tnu = zeros(n,1); Sigma = K; % initialize Sigma and mu, the parameters of .. mu = m; % .. the Gaussian posterior approximation nlZ = nlZ0; end end nlZ_old = Inf; sweep = 0; % converged, max. sweeps or min. sweeps? while (nlZ_old-nlZ > tol && sweep < max_sweep) || sweep tol error('maximum number of sweeps exceeded in function infKL') end last_ttau = ttau; last_tnu = tnu; % remember for next call post.alpha = alpha; post.sW = sqrt(ttau); post.L = L; % return posterior params if nargout>2 % do we want derivatives? v = diag(Sigma); [lp,df,d2f,dv] = likKL(v,lik,hyp.lik,y,mu); dnlZ = hyp; % allocate space for derivatives dnlZ.cov = dK( alpha*(alpha/2-df)'-diag(dv)*A*(eye(n)+A'-A) ); % cov hypers for j=1:length(hyp.lik) % likelihood hypers dnlZ.lik(j) = -sum( likKL(v,lik,hyp.lik,y,K*post.alpha+m,[],[],j) ); end dnlZ.mean = -dm(alpha); % mean hypers end % function to compute the parameters of the Gaussian approximation, Sigma and % mu, from the current site parameters, ttau and tnu. Also returns L and upper % bound on negative marginal likelihood. function [Sigma,mu,L,alpha,nlZ,A] = klComputeParams(K,y,ttau,tnu,lik,hyp,m) n = length(tnu); % number of training cases sW = sqrt(ttau); % compute Sigma and mu L = chol(eye(n)+sW*sW'.*K); % L'*L=B=eye(n)+sW*K*sW V = L'\(repmat(sW,1,n).*K); Sigma = K - V'*V; alpha = tnu-sW.*solve_chol(L,sW.*(K*tnu+m)); mu = K*alpha+m; v = diag(Sigma); A = (eye(n)+K*diag(ttau))\eye(n); % A = Sigma*inv(K) lp = likKL(v,lik,hyp.lik,y,mu); % evaluate likelihood nlZ = -sum(lp) - (logdet(A)-alpha'*(mu-m)-trace(A)+n)/2; % upper bound on -lZ % We compute the Gaussian Q(f)=N(f|m,s^2) minimising the KL divergence % KL(Q||P) where P is the product of the cavity distribution q_n(f) and the % likelihood p(y|f) such that P(f) = 1/Z * q_n(f)*p(y|f). % The cavity distribution q_n(f) is an unnormalised Gaussian with precision % parameter tau_n and location parameter nu_n, hence the cavity can be written % as q_n(f) = exp(nu_n*f-tau_n/2*f^2). % The minimisation is convex iff. the likelihood p(y|f) is log-concave. The % optimisation is performed using Newton descent with backtracking line search. function [m,s,kl] = klmin(lik, hyp, y, nu_n, tau_n) ep = 1e-9; % tiny Hessian ridge for stability gthresh = 1e-8; % gradient convergence threshold lik_str = lik{1}; if ~ischar(lik_str), lik_str = func2str(lik_str); end if strcmp(lik_str,'likGauss') % likGauss can be done analytically sn2 = exp(2*hyp); s = 1/sqrt(1/sn2+tau_n); m = s^2*(nu_n+y/sn2); % init variables to minimum else s = 1/sqrt(tau_n); m = nu_n/tau_n; % init variables to cavity distribution end ll = likKL(s^2,lik,hyp,y,m); % evaluate likelihood kl = (s^2+m^2)*tau_n/2 - log(s) - nu_n*m - ll; % init the KL div up to a const for i=1:20 [ll,dm,d2m,dv,d2v,dmdv] = likKL(s^2,lik,hyp,y,m); % evaluate likelihood klold = kl; mold = m; sold = s; % remember last value kl = (s^2+m^2)*tau_n/2 - log(s) - nu_n*m - ll; % KL-divergence up to const dm = tau_n*m-nu_n-dm; d2m = tau_n-d2m; % compute kl derivatives ds = s*tau_n-1/s-2*s*dv; d2s = tau_n+1/s^2-2*dv-(2*s)^2*d2v; dmds=-2*s*dmdv; detH = ((d2m+ep)*(d2s+ep)-dmds^2); % Hessian determinant m = m-(dm*(d2s+ep)-ds*dmds)/detH; s = s-(ds*(d2m+ep)-dm*dmds)/detH; % Newton for j=1:10 % backtracking line search if klold>kl, break, end % we did descend so no need to step back m = (m+mold)/2; s = (s+sold)/2; kl = (s^2+m^2)*tau_n/2 - log(s) - nu_n*m - likKL(s^2,lik,hyp,y,m); end d = abs(dm)+abs(dv); % overall gradient if j==10, m = mold; s = sold; d = 0; end if d0 using the LU decomposition of A function y = logdet(A) [L,U] = lu(A); u = diag(U); if prod(sign(u))~=det(L), error('det(A)<=0'), end y = sum(log(abs(u))); % Gaussian smoothed likelihood function; instead of p(y|f)=lik(..,f,..) compute % log likKL(f) = int log lik(..,t,..) N(f|t,v) dt, where % v .. marginal variance = (positive) smoothing width, and % lik .. lik function such that feval(lik{:},varargin{:}) yields a result. % All return values are separately integrated using Gaussian-Hermite quadrature. function [ll,df,d2f,dv,d2v,dfdv] = likKL(v, lik, varargin) f = varargin{3}; % obtain location of evaluation sv = sqrt(v); % smoothing width lik_str = lik{1}; if ~ischar(lik_str), lik_str = func2str(lik_str); end if strcmp(lik_str,'likLaplace') % likLaplace can be done analytically b = exp(varargin{1})/sqrt(2); y = varargin{2}; mu = (f-y)/b; z = (f-y)./sv; Nz = exp(-z.^2/2)/sqrt(2*pi); Cz = (1+erf(z/sqrt(2)))/2; ll = (1-2*Cz).*mu - 2/b*sv.*Nz - log(2*b); df = (1-2*Cz)/b; d2f = -2*Nz./(b*(sv+eps)); dv = d2f/2; d2v = (z.*z-1)./(v+eps).*d2f/4; dfdv = -z.*d2f./(2*sv+eps); else N = 20; % number of quadrature points [t,w] = gauher(N); % location and weights for Gaussian-Hermite quadrature ll = 0; df = 0; d2f = 0; dv = 0; d2v = 0; dfdv = 0; % init return arguments for i=1:N % use Gaussian quadrature varargin{3} = f + sv*t(i); % coordinate transform of the quadrature points [lp,dlp,d2lp]=feval(lik{:},varargin{1:3},[],'infLaplace',varargin{6:end}); ll = ll + w(i)*lp; % value of the integral df = df + w(i)*dlp; % derivative wrt. mean d2f = d2f + w(i)*d2lp; % 2nd derivative wrt. mean ai = t(i)./(2*sv+eps); dvi = dlp.*ai; dv = dv+w(i)*dvi; % deriv wrt. var d2v = d2v + w(i)*(d2lp.*(t(i)^2/2)-dvi)./(v+eps)/2; % 2nd deriv wrt. var dfdv = dfdv + w(i)*(ai.*d2lp); % mixed second derivatives end endgpml-matlab-v4.0-2016-10-19/inf/infLaplace.m000644 000765 000024 00000012215 12777200660 020057 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infLaplace(hyp, mean, cov, lik, x, y, opt) % Laplace approximation to the posterior Gaussian process. % The function takes a specified covariance function (see covFunctions.m) and % likelihood function (see likFunctions.m), and is designed to be used with % gp.m. See also infMethods.m. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2016-08-22. % % See also INFMETHODS.M. persistent last_alpha % copy of the last alpha if any(isnan(last_alpha)), last_alpha = zeros(size(last_alpha)); end % prevent if nargin<=6, opt = []; end % make opt variable available inf = 'infLaplace'; n = size(x,1); K = apx(hyp,cov,x,opt); % set up covariance approximation if isnumeric(mean), m = mean; % use provided mean vector else [m,dm] = feval(mean{:}, hyp.mean, x); end % mean vector and deriv likfun = @(f) feval(lik{:},hyp.lik,y,f,[],inf); % log likelihood function if any(size(last_alpha)~=[n,1]) % find a good starting point for alpha and f alpha = zeros(n,1); % start at mean if sizes do not match else alpha = last_alpha; % try last one if Psi(alpha,m,K.mvm,likfun) > -sum(likfun(m)) % default f==m better => use it alpha = zeros(n,1); end end % switch between optimisation methods alpha = irls(alpha, m,K,likfun, opt); % run optimisation f = K.mvm(alpha)+m; % compute latent function values last_alpha = alpha; % remember for next call [lp,dlp,d2lp,d3lp] = likfun(f); W = -d2lp; [ldB2,solveKiW,dW,dhyp] = K.fun(W); % obtain functionality depending on W post.alpha = K.P(alpha); % return the posterior parameters post.sW = sqrt(abs(W)).*sign(W); % preserve sign in case of negative post.L = @(r) -K.P(solveKiW(K.Pt(r))); % diagnose optimality err = @(x,y) norm(x-y)/max([norm(x),norm(y),1]); % we need to have alpha = dlp % dev = err(alpha,dlp); if dev>1e-4, warning('Not at optimum %1.2e.',dev), end nlZ = alpha'*(f-m)/2 - sum(lp) + ldB2; % compute marginal likelihood if nargout>2 % do we want derivatives? dfhat = dW.*d3lp; % deriv. of nlZ wrt. fhat: dfhat=diag(inv(inv(K)+W)).*d3lp/2 dahat = dfhat - solveKiW(K.mvm(dfhat)); dnlZ = dhyp(alpha,dlp,dahat); dnlZ.lik = zeros(size(hyp.lik)); % allocate memory for i=1:length(hyp.lik) % likelihood hypers [lp_dhyp,dlp_dhyp,d2lp_dhyp] = feval(lik{:},hyp.lik,y,f,[],inf,i); dnlZ.lik(i) = -dW'*d2lp_dhyp - sum(lp_dhyp); % explicit part b = K.mvm(dlp_dhyp); % b-K*(Z*b) = inv(eye(n)+K*diag(W))*b dnlZ.lik(i) = dnlZ.lik(i) - dfhat'*( b-K.mvm(solveKiW(b)) ); % implicit end dnlZ.mean = -dm(alpha+dahat); % explicit + implicit part end % Evaluate criterion Psi(alpha) = alpha'*K*alpha + likfun(f), where % f = K*alpha+m, and likfun(f) = feval(lik{:},hyp.lik,y, f, [],inf). function [psi,dpsi,f,alpha,dlp,W] = Psi(alpha,m,mvmK,likfun) f = mvmK(alpha)+m; [lp,dlp,d2lp] = likfun(f); W = -d2lp; psi = alpha'*(f-m)/2 - sum(lp); if nargout>1, dpsi = mvmK(alpha-dlp); end % Run IRLS Newton algorithm to optimise Psi(alpha). function alpha = irls(alpha, m,K,likfun, opt) if isfield(opt,'irls_maxit'), maxit = opt.irls_maxit; % max no of Newton steps else maxit = 20; end % default value if isfield(opt,'irls_Wmin'), Wmin = opt.irls_Wmin; % min likelihood curvature else Wmin = 0.0; end % default value if isfield(opt,'irls_tol'), tol = opt.irls_tol; % stop Newton iterations else tol = 1e-6; end % default value smin_line = 0; smax_line = 2; % min/max line search steps size range nmax_line = 10; % maximum number of line search steps thr_line = 1e-4; % line search threshold Psi_line = @(s,alpha,dalpha) Psi(alpha+s*dalpha, m,K.mvm,likfun);% line search pars_line = {smin_line,smax_line,nmax_line,thr_line}; % line seach parameters search_line = @(alpha,dalpha) brentmin(pars_line{:},Psi_line,5,alpha,dalpha); f = K.mvm(alpha)+m; [lp,dlp,d2lp] = likfun(f); W = -d2lp; n = size(K,1); Psi_new = Psi(alpha,m,K.mvm,likfun); Psi_old = Inf; % make sure while loop starts by the largest old objective val it = 0; % this happens for the Student's t likelihood while Psi_old - Psi_new > tol && it0 if nargout<3 [post nlZ] = inf(hyp, mean, cov, lik, x*hP', y, opt); else [post nlZ dnlZ] = inf(hyp, mean, cov, lik, x*hP', y, opt); if isfield(hyp,'proj') dnlZ.proj=deriv_proj(post.alpha,hP,K,covGrid('flatten',xg),m,mean,hyp,x); if strncmpi(proj,'orth',4), dnlZ.proj=chain_orth(hyp.proj,dnlZ.proj); elseif strncmpi(proj,'norm',4), dnlZ.proj=chain_norm(hyp.proj,dnlZ.proj); end end end else return, end % no of samples for perturb-and-MAP, see George Papandreou and Alan L. Yuille: % "Efficient Variational Inference in Large-Scale Bayesian Compressed Sensing" ns = 0; % do nothing per default, 20 is suggested in paper if isfield(opt,'pred_var'), ns = max(ceil(abs(opt.pred_var)),20); end if ndcovs>0 && nargout>2, ns = max(ns,ndcovs); end % possibly draw more samples Mtal = M'*post.alpha; % blow up alpha vector from n to N kronmvm = K.kronmvm; if ns>0 s = 3; % Whittle embedding overlap factor [V,ee,e] = apxGrid('eigkron',K,xg,s); % perform eigen-decomposition % explained variance on the grid vg=diag(Ku*M'*inv(C)*M*Ku), C=M*Ku*M'+inv(W) % relative accuracy r = std(vg_est)/vg_exact = sqrt(2/ns) A = sample(V,e,M,post.sW,ns,kronmvm); A = post.L(A); % a~N(0,inv(C)) z = K.mvm(M'*A); vg = sum(z.*z,2)/ns; % z ~ N(0,Ku*M'*inv(C)*M*Ku) if ndcovs>0 dnlZ.covs = - apxGrid('dirder',K,xg,M,post.alpha,post.alpha)/2; na = size(A,2); for i=1:na % compute (E[a'*dK*a] - a'*dK*a)/2 dnlZ.covs = dnlZ.covs + apxGrid('dirder',K,xg,M,A(:,i),A(:,i))/(2*na); end if isfield(hyp,'proj') % data projection matrix dPs = zeros(size(hyp.proj)); % allocate memory for result KMtal = K.mvm(Mtal); [M,dM]=covGrid('interp',xg,x*hP'); % precompute for i = 1:size(dPs,1) if equi(xg,i), wi = max(xg{i})-min(xg{i}); else wi = 1; end % scaling for j = 1:size(dPs,2) dMtal = dM{i}'*(x(:,j).*post.alpha/wi); dMtA = dM{i}'*(repmat(x(:,j),1,na).*A/wi); dPs(i,j) = sum(sum(dMtA.*z))/ndcovs - dMtal'*KMtal; end end if strncmpi(proj,'orth',4),dnlZ.projs = chain_orth(hyp.proj,dPs); elseif strncmpi(proj,'norm',4),dnlZ.projs = chain_norm(hyp.proj,dPs); else dnlZ.projs = dPs; end end end else vg = zeros(N,1); % no variance explained end % add fast predictions to post structure, f|y,mu|s2 post.predict = @(xs) predict(xs*hP',xg,K.mvm(Mtal),vg,hyp,mean,cov,lik,deg); % global mem, S = whos(); mem=0; for i=1:numel(S), mem=mem+S(i).bytes/1e6; end % Compute latent and predictive means and variances by grid interpolation. function [fmu,fs2,ymu,ys2] = predict(xs,xg,Kalpha,vg,hyp,mean,cov,lik,deg) Ms = apxGrid('interp',xg,xs,deg); % obtain interpolation matrix xs = apxGrid('idx2dat',xg,xs,deg); % deal with index vector ms = feval(mean{:},hyp.mean,xs); % evaluate prior mean fmu = ms + Ms*Kalpha; % combine and perform grid interpolation if nargout>1 if norm(vg,1)>1e-10, ve = Ms*vg; else ve = 0; end % interp grid var expl. ks = feval(cov{:},hyp.cov,xs,'diag'); % evaluate prior variance fs2 = max(ks-ve,0); % combine, perform grid interpolation, clip if nargout>2, [lp, ymu, ys2] = feval(lik{:},hyp.lik,[],fmu,fs2); end end % sample a~N(0,C), C = M*Ku*M'+inv(W), W=sW^2 function A = sample(V,e,M,sW,ns,kronmvm) [n,N] = size(M); A = randn(N,ns); % a~N(0,I) A = kronmvm(V,repmat(sqrt(e),1,ns).*kronmvm(V,A,1)); % a~N(0,Ku) A = M*A + bsxfun(@times,1./sW,randn(n,ns)); % compute derivative of neg log marginal likelihood w.r.t. projection matrix P function dP = deriv_proj(alpha,P,K,xg,m,mean,hyp,x) xP = x*P'; [M,dM] = covGrid('interp',xg,xP); % grid interp derivative matrices beta = K.mvm(M'*alpha); % dP(i,j) = -alpha'*dMij*beta dP = zeros(size(P)); h = 1e-4; % allocate result, num deriv step for i=1:size(P,1) if equi(xg,i), wi = max(xg{i})-min(xg{i}); else wi = 1; end % scaling factor xP(:,i) = xP(:,i)+h; % perturb ith component of projection matrix dmi = (feval(mean{:},hyp.mean,xP)-m)/h; % numerically estimate dm/dP(:,i) xP(:,i) = xP(:,i)-h; % undo perturbation betai = dmi + dM{i}*beta/wi; for j=1:size(P,2), dP(i,j) = -alpha'*(x(:,j).*betai); end end function eq = equi(xg,i) % grid along dim i is equispaced ni = size(xg{i},1); if ni>1 % diagnose if data is linearly increasing dev = abs(diff(xg{i})-ones(ni-1,1)*(xg{i}(2,:)-xg{i}(1,:))); eq = max(dev(:))<1e-9; else eq = true; end % chain rule for the function Q = sqrtm(P*P')\P; for d sqrtm(X) see the website function dQ = chain_orth(P,dP) % http://math.stackexchange.com/questions/540361 [V,F] = eig(P*P'); sf = sqrt(diag(F)); S = V*diag(sf)*V'; % eig-decomp H = dP'/S; G = H'*(P'/S); o = ones(size(dP,1),1); % chain rule dQ = (H - P'*V*((V'*(G+G')*V)./(sf*o'+o*sf'))*V')'; % chain rule for the function Q = diag(1./sqrt(diag(P*P')))*P; function dQ = chain_norm(P,dP) p = 1./sqrt(diag(P*P')); dQ = diag(p)*dP - diag(diag(dP*P').*p.^3)*P;gpml-matlab-v4.0-2016-10-19/inf/infMCMC.m000644 000765 000024 00000026241 12777200671 017243 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infMCMC(hyp, mean, cov, lik, x, y, par) % Markov Chain Monte Carlo (MCMC) sampling from posterior and % Annealed Importance Sampling (AIS) for marginal likelihood estimation. % % The algorithms are not to be used as a black box, since the acceptance rate % of the samplers need to be carefully monitored. Also, there are no derivatives % of the marginal likelihood available. % % There are additional parameters: % - par.sampler switch between the samplers 'hmc' or 'ess' % - par.Nsample num of samples % - par.Nskip num of steps out of which one sample kept % - par.Nburnin num of burn in samples (corresponds to Nskip*Nburning steps) % - par.Nais num of AIS runs to remove finite temperature bias % - par.st spherical Gaussian width of the posterior KDE % Default values are sampler=hmc, Nsample=200, Nskip=40, Nburnin=10, Nais=3, % st=0.001*sqrt(sum(diag(K))/n). % % The Hybrid Monte Carlo Sampler (HMC) is implemented as described in the % technical report: Probabilistic Inference using MCMC Methods by Radford Neal, % CRG-TR-93-1, 1993. % % Instead of sampling from f ~ 1/Zf * N(f|m,K) P(y|f), we use a % parametrisation in terms of alpha = inv(K)*(f-m) and sample from % alpha ~ P(a) = 1/Za * N(a|0,inv(K)) P(y|K*a+m) to increase numerical % stability since log P(a) = -(a'*K*a)/2 + log P(y|K*a+m) + C and its % gradient can be computed safely. % % The leapfrog stepsize as a time discretisation stepsize comes with a % tradeoff: % - too small: frequently accept, slow exploration, accurate dynamics % - too large: seldomly reject, fast exploration, inaccurate dynamics % In order to balance between the two extremes, we adaptively adjust the % stepsize in order to keep the acceptance rate close to a target acceptance % rate. Taken from http://deeplearning.net/tutorial/hmc.html. % % The code issues a warning in case the overall acceptance rate did deviate % strongly from the target. This can indicate problems with the sampler. % % The Elliptical Slice Sampler (ESS) is a straight implementation that is % inspired by the code shipped with the paper: Elliptical slice sampling by % Iain Murray, Ryan Prescott Adams and David J.C. MacKay, AISTATS 2010. % % Annealed Importance Sampling (AIS) to determine the marginal likelihood is % described in the technical report: AIS, Radford Neal, 1998. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2016-05-09. % % See also INFMETHODS.M. if nargin<7, par = []; end % analyse parameter structure if isfield(par,'sampler'), alg=par.sampler; else alg='hmc'; end if isfield(par,'Nsample'), N =par.Nsample; else N =200; end if isfield(par,'Nskip'), Ns =par.Nskip; else Ns = 40; end if isfield(par,'Nburnin'), Nb =par.Nburnin; else Nb = 10; end if isfield(par,'Nais'), R =par.Nais; else R = 3; end K = feval(cov{:}, hyp.cov, x); % evaluate the covariance matrix m = feval(mean{:}, hyp.mean, x); % evaluate the mean vector n = size(K,1); st_def = 0.001*sqrt(sum(diag(K))/n); % default value if isfield(par,'st'), st =par.st; else st = st_def; end [cK,fail] = chol(K+st^2*eye(n)); % try an ordinary Cholesky decomposition if fail, error('Try increasing the par.st variable.'), end T = (N+Nb)*Ns; % overall number of steps [alpha,Na] = sample(K,cK,m,y,lik,hyp.lik, N,Nb,Ns, alg); % sample w/o annealing post.alpha = alpha; al = sum(alpha,2)/N; post.L = -(cK\(cK'\eye(n)) + al*al' - alpha*alpha'/N); % inv(K) - cov(alpha) post.sW = []; post.acceptance_rate_MCMC = Na/T; % additional output parameter if nargout>1 % annealed importance sampling % discrete time t from 1 to T and temperature tau from tau(1)=0 to tau(T)=1 taus = [zeros(1,Nb),linspace(0,1,N)].^4; % annealing schedule, effort at start % the fourth power idea is taken from the Kuss&Rasmussen paper, 2005 JMLR % Z(t) := \int N(f|m,K) lik(f)^taus(t) df hence Z(1) = 1 and Z(T) = Z % Z = Z(T)/Z(1) = prod_t Z(t)/Z(t-1); % ln Z(t)/Z(t-1) = ( tau(t)-tau(t-1) ) * loglik(f_t) lZ = zeros(R,1); dtaus = diff(taus(Nb+(1:N))); % we have: sum(dtaus)==1 for r=1:R [A,Na] = sample(K,cK,m,y,lik,hyp.lik, N,Nb,Ns, alg, taus); % AIS for t=2:N % evaluate the likelihood sample-wise lp = feval(lik{:},hyp.lik,y,K*A(:,t)+m,[],'infLaplace'); lZ(r) = lZ(r)+dtaus(t-1)*sum(lp); end post.acceptance_rate_AIS(r) = Na/T; % additional output parameters end nlZ = log(R)-logsumexp(lZ); % remove finite temperature bias, softmax average if nargout>2 % marginal likelihood derivatives are not computed dnlZ = struct('cov',0*hyp.cov, 'mean',0*hyp.mean, 'lik',0*hyp.lik); end end %% choose between HMC and ESS depending on the alg string function [alpha,Na] = sample(K,cK,m,y,lik,hyp, N,Nb,Ns, alg, varargin) if strcmp(lik,'likGauss') [alpha,Na] = sample_gauss(K, m,y,hyp, N,Nb,Ns, varargin{:}); else if strcmpi(alg,'hmc') [alpha,Na] = sample_hmc(K, m,y,lik,hyp, N,Nb,Ns, varargin{:}); else [alpha,Na] = sample_ess(cK,m,y,lik,hyp, N,Nb,Ns, varargin{:}); end end %% sample from Gaussian posterior function [alpha,Na] = sample_gauss(K, m,y,hyp, N,Nb,Ns, taus) Na = (N+Nb)*Ns; sn2 = exp(2*hyp); n = size(K,1); if nargin<8 S = K + K*K/sn2; alpha = chol(S)\randn(n,N) + repmat(S\(K*(y-m)/sn2),1,N); else alpha = zeros(n,N); for t=1:N St = K + K*K*(taus(Nb+t)/sn2); alpha(:,t) = chol(St)\randn(n,1) + St\(K*(y-m)*taus(Nb+t)/sn2); end end %% sample using elliptical slices function [alpha,T] = sample_ess(cK,m,y,lik,hyp, N,Nb,Ns, taus) if nargin>=9, tau = taus(1); else tau = 1; end % default is no annealing T = (N+Nb)*Ns; % overall number of steps F = zeros(size(m,1),N); for t=1:T if nargin>=9, tau = taus(1+floor((t-1)/Ns)); end % parameter from schedule if t==1, f=0*m; l=sum(feval(lik{:},hyp,y,f+m)); end % init sample f & lik l r = cK'*randn(size(m)); % random sample from N(0,K) [f,l] = sample_ess_step(f,l,r,m,y,lik,hyp,tau); if mod(t,Ns)==0 % keep one state out of Ns if t/Ns>Nb, F(:,t/Ns-Nb) = f; end % wait for Nb burn-in steps end end alpha = cK\(cK'\F); %% elliptical slice sampling: one step function [f,l] = sample_ess_step(f,l,r,m,y,lik,hyp,tau) if nargin<8, tau=1; end, if tau>1, tau=1; end, if tau<0, tau=0; end h = log(rand) + tau*l; % acceptance threshold a = rand*2*pi; amin = a-2*pi; amax = a; % bracket whole ellipse k = 0; % emergency break while true % slice sampling loop; f for proposed angle diff; check if on slice fp = f*cos(a) + r*sin(a); % move on ellipsis defined by r l = sum(feval(lik{:},hyp,y,fp+m)); if tau*l>h || k>20, break, end % exit if new point is on slice or exhausted if a>0, amax=a; elseif a<0, amin=a; end % shrink slice to rejected point a = rand*(amax-amin) + amin; k = k+1; % propose new angle difference; break end f = fp; % accept %% sample using Hamiltonian dynamics as proposal algorithm function [alpha,Na] = sample_hmc(K,m,y,lik,hyp, N,Nb,Ns, taus) % use adaptive stepsize rule to enforce a specific acceptance rate epmin = 1e-6; % minimum leapfrog stepsize epmax = 9e-1; % maximum leapfrog stepsize ep = 1e-2; % initial leapfrog stepsize acc_t = 0.9; % target acceptance rate acc = 0; % current acceptance rate epinc = 1.02; % increase factor of stepsize if acceptance rate is below target epdec = 0.98; % decrease factor of stepsize if acceptance rate is above target lam = 0.01; % exponential moving average computation of the acceptance rate % 2/(3*lam) steps half height; lam/nstep = 0.02/33, 0.01/66, 0.005/133 l = 20; % number of leapfrog steps to perform for one step n = size(K,1); T = (N+Nb)*Ns; % overall number of steps alpha = zeros(n,N); % sample points al = zeros(n,1); % current position if nargin>=9, tau = taus(1); else tau = 1; end % default is no annealing [gold,eold] = E(al,K,m,y,lik,hyp,tau); % initial energy, gradient Na = 0; % number of accepted points for t=1:T if nargin>=9, tau = taus(1+floor((t-1)/Ns)); end % parameter from schedule p = randn(n,1); % random initial momentum q = al; g = gold; Hold = (p'*p)/2 + eold; % H = Ekin + Epot => Hamiltonian for ll=1:l % leapfrog discretization steps, Euler like p = p - (ep/2)*g; % half step in momentum p q = q + ep*p; % full step in position q g = E(q,K,m,y,lik,hyp,tau); % compute new gradient g p = p - (ep/2)*g; % half step in momentum p end [g,e] = E(q,K,m,y,lik,hyp,tau); H = (p'*p)/2 + e; % recompute Hamiltonian acc = (1-lam)*acc; % decay current acceptance rate if log(rand) < Hold-H % accept with p = min(1,exp(Hold-H)) al = q; % keep new state, gold = g; % gradient eold = e; % and potential energy acc = acc + lam; % increase rate due to acceptance Na = Na+1; % increase number accepted steps end if acc>acc_t % too large acceptance rate => increase leapfrog stepsize ep = epinc*ep; else % too small acceptance rate => decrease leapfrog stepsize ep = epdec*ep; end if epepmax, ep = epmax; end if mod(t,Ns)==0 % keep one state out of Ns if t/Ns>Nb, alpha(:,t/Ns-Nb) = al; end % wait for Nb burn-in steps end end if Na/T dE/dal = K*( al-dloglik(f) ) if nargin<7, tau=1; end, if tau>1, tau=1; end, if tau<0, tau=0; end Kal = K*al; [lp,dlp] = feval(lik{:},hyp,y,Kal+m,[],'infLaplace'); g = Kal-K*(tau*dlp); if nargout>1, e = (al'*Kal)/2 - tau*sum(lp); end %% y = logsumexp(x) = log(sum(exp(x(:))) avoiding overflow function y=logsumexp(x) mx = max(x(:)); y = log(sum(exp(x(:)-mx)))+mx; gpml-matlab-v4.0-2016-10-19/inf/infLOO.m000644 000765 000024 00000007765 12777200651 017165 0ustar00carlstaff000000 000000 function [post nloo dnloo loo] = infLOO(hyp, mean, cov, lik, x, y) % Leave-One-Out - Perform Least-Squares GP predictions in the style of the % Pseudo Likelihood in 5.4.2 p. 117 and the Probabilistic Least-squares % Classifier of 6.5 p. 146 of the GPML book. % For Gaussian likelihood, the prediction is the same as with infExact, for % lik{Erf,Logistic} the result corresponds to probabilistic least-squares % classification. In any case, the objective returned is not the marginal % likelihood but the predictive log probability. % The standard deviation of the noise either comes from the sn parameter of the % likelihood function lik{Gauss,Laplace,Sech2} or from the covariance function % by means of using covSum and covNoise i.e. for lik{Uni,Erf,Logistic}. % We then compute the negative leave-one-out predictive probability and its % derivatives w.r.t. the hyperparameters. See also "help infMethods". % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-05-02 % % See also INFMETHODS.M. [n, D] = size(x); if isnumeric(cov), K = cov; % use provided covariance matrix else [K,dK] = feval(cov{:}, hyp.cov, x); end % covariance matrix and deriv if isnumeric(mean), m = mean; % use provided mean vector else [m,dm] = feval(mean{:}, hyp.mean, x); end % mean vector and deriv if numel(hyp.lik)==0 cov1 = cov{1}; if isa(cov1, 'function_handle'), cov1 = func2str(cov1); end if ~strcmp(cov1,'covSum'); error('Only covSum supported.'), end % check cov cov2 = cov{2}; npar = 0; for i=1:length(cov2) cov2i = cov2{i}; if numel(cov2i)==1 || ischar(cov2i) if isa(cov2i, 'function_handle'), cov2i = func2str(cov2i); end if strcmp(cov2i,'covNoise'), isn = npar+1; sn2 = exp(2*hyp.cov(isn)); end end npar = npar + eval(feval(cov2i)); end if ~exist('sn2','var'), error('We need covNoise in the covSum.'), end else % Gauss, Laplace, Sech2 sn2 = exp(2*hyp.lik(end)); isn = 0; K = K + sn2*eye(n); end L = chol(K)/sqrt(sn2); % Cholesky factor of covariance with noise alpha = solve_chol(L,y-m)/sn2; post.alpha = alpha; % return the posterior parameters post.sW = ones(n,1)/sqrt(sn2); % sqrt of noise precision vector post.L = L; % L = chol(eye(n)+sW*sW'.*K) if nargout>1 % do we want the marginal likelihood? sdeff = 1./post.sW; % effective Gaussian width mneff = K*alpha + m; % alpha = inv(K)*(mneff-m), effective Gaussian center % all loo marginal means and variances at once V = post.L'\diag(1./sdeff); r = sum(V.*V,1)'; % r=diag( inv(K) ) loo.fmu = mneff - alpha./r; % GPML book, eq. 5.12 loo.fs2 = 1./r - sn2; % GPML book, eq. 5.12 [loo.lp, loo.ymu, loo.ys2] = feval(lik{:},hyp.lik,y,loo.fmu,loo.fs2); nloo = -sum(loo.lp); % negative leave-one-out predictive probability if nargout>2 % do we want derivatives? [lZ,dlZmn,d2lZmn] = feval(lik{:},hyp.lik,y,loo.fmu,loo.fs2,'infEP'); dlZva = (d2lZmn+dlZmn.*dlZmn)/2; % derivative w.r.t. variance dnloo = hyp; iK = solve_chol(L,eye(n))/sn2; % based on GPML book, eq. 5.13 Q = iK*diag((dlZmn.*alpha-dlZva)./(r.*r))*iK - (iK*(dlZmn./r))*alpha'; dnloo.cov = dK(Q); if isn, dnloo.cov(isn) = dnloo.cov(isn) + 2*sn2*sum(dlZva); end for j=1:numel(hyp.lik) dnloo.lik(j) = -sum( feval(lik{:},hyp.lik,y,loo.fmu,loo.fs2,'infEP',j) ); if j==numel(hyp.lik) dva = 2*sn2*sum(iK.*iK,2)./(r.*r); % GPML book, eq. 5.13 dmn = 2*solve_chol(L,alpha)./r - alpha.*dva; % GPML book, eq. 5.13 dva = dva - 2*sn2; dnloo.lik(j) = dnloo.lik(j) -dlZmn'*dmn -dlZva'*dva; end end dnloo.mean = -dm(solve_chol(L,dlZmn./(sn2*r))); end end gpml-matlab-v4.0-2016-10-19/inf/infPrior.m000644 000765 000024 00000013235 12777200677 017624 0ustar00carlstaff000000 000000 function [post nlZ dnlZ] = infPrior(inf, prior, hyp, varargin) % Inference with hyperparameter prior. Any inference method can be used. % The prior specification is done by means of the prior parameter that has % the same structure as hyp. Examples: % 1) >> prior.cov{2} = {@priorLaplace,mu,s2}; % put a Laplace prior on the second covariance function hyperparameter % i.e. hyp.cov(2) ~ Laplace(mu,s2) % 2) >> prior.mean{3} = {@priorDelta} or % >> prior.mean{3} = @priorDelta or % >> prior.mean{3} = 'priorClamped' % fix the third mean function hyperparameter to its current value and % exclude it from optimisation % i.e. hyp.mean(3) ~ delta(hyp.mean(3)) % 3) >> prior.xu = repmat({{@priorGauss,0,1}},D*nu,1) % put an individual standard normal prior on every inducing input coordinate % for FITC inference % i.e. hyp.xu(i) ~ N(0,1) for all i = 1..D*nu % 4) >> prior.multi{1} = {@priorGaussMulti,mu,s2, struct('xu',1:D*nu)} % put a joint Gaussian prior with mean vector mu and covariance matrix % s2 on the inducing input coordinates in FITC inference % i.e. hyp.xu ~ N(mu,s2) % 5) >> prior.multi{1} = {@priorTMulti,mu,s2,nu, off+(1:D)} % put a joint Student's t prior with mean vector mu, covariance matrix s2 % and nu degrees of freedom on the weights of the mean function e.g. % hyp.mean ~ T(mu,s2,nu) for meanLinear % Here, we do not have a struct as in 4) but the index in any2vec(hyp). % 6) >> p = {@priorWeibull,lam,k}; g = @exp; dg = @exp; ig = @log; % >> prior.cov{2} = {@priorTransform,g,dg,ig,p} % put a univariate Weibull distribution on the exponentiated second % covariance function hyperparameter % i.e. exp(hyp.cov(2)) ~ Weibull(lam,k) % % Return a parametrization of the posterior, the negative log marginal % likelihood and its derivatives w.r.t. the hyperparameters. % % Note that at most 1 prior is allowed per hyperparameter. We consider the % specification of more than 1 prior for a hyperparameter as an error. % % For more help on design of priors, try "help priorDistributions". % % Copyright (c) by Hannes Nickisch and Roman Garnett, 2016-09-29. % % See also INFMETHODS.M, USAGEPRIOR.M, PRIORDISTRIBUTIONS.M. [post nlZ dnlZ] = inf(hyp, varargin{:}); % perform inference if ~isempty(prior) % add hyperprior contributions to nlZ and dnlZ nam = fieldnames(prior); num = zeros(numel(any2vec(hyp)),1); % number of priors per hyperparameter for i=1:numel(nam) % iterate over kinds of hyperparameters cov/lik/mean/xu ni = nam{i}; % name of the ith field if strcmp(ni,'multi') % first catch multivariate priors p = prior.(ni); for j=1:numel(p) % iterate over individual muti-hyperparameter components pj = p{j}; if ~isempty(pj) % only proceed if a nonempty prior is specified idx = pj{end}; pj = pj(1:end-1); % grab index from cell end pjstr = pj{1}; if ~ischar(pjstr), pjstr = func2str(pjstr); end if numel(pjstr)<5 || ~strcmp(pjstr(end-4:end),'Multi') error('multivariate hyperpriors are called Multi') end if isstruct(idx) % massage idx into a vector idxs = vec2any(hyp,zeros(size(num))); % structured index for nj = fieldnames(idx), idxs.(nj{1})( idx.(nj{1}) ) = 1; end idx = any2vec(idxs)>0; % linearise structured index else idxz = zeros(size(num)); idxz(idx) = 1; idx = idxz>0; % binary index end if sum(idx)<=1, error('multivariate priors need >1 hyperparam'), end num(idx) = num(idx)+1; % inc prior counter hypu = any2vec(hyp); dnlZ = any2vec(dnlZ); if strncmp(pjstr,'priorClamped',12) || ... % clamp derivative to zero strncmp(pjstr,'priorDelta',10), dnlZ(idx) = 0; else [lp,dlp] = feval(pj{:}, hypu(idx)); % evaluate prior distribution nlZ = nlZ-lp; dnlZ(idx) = dnlZ(idx)-dlp(:); % update nlZ and dnlZ end dnlZ = vec2any(hyp,dnlZ); else error('multivariate priors should be non empty') end end continue % jump to univariate priors end if ~isfield(hyp,ni), error(['unknown prior field ',ni,' in hyp']), end p = prior.(ni); if numel(p)~=numel(hyp.(ni)), error(['bad hyp/prior field length ',ni]), end for j=1:numel(p) % iterate over individual hyperparameter components pj = p{j}; if ~iscell(pj) && ~isempty(pj), pj = {pj}; end % enforce cell if ~isempty(pj) % only proceed if a nonempty prior is specified num = vec2any(hyp,num); num.(ni)(j) = num.(ni)(j)+1; % inc prior counter num = any2vec(num); pj1str = pj{1}; if ~ischar(pj1str), pj1str = func2str(pj1str); end if strncmp(pj1str,'priorClamped',12) || strncmp(pj1str,'priorDelta',10) dnlZ.(ni)(j) = 0; % clamp derivative to zero else [lp,dlp] = feval(pj{:}, hyp.(ni)(j)); % evaluate prior distribution nlZ = nlZ-lp; dnlZ.(ni)(j) = dnlZ.(ni)(j)-dlp; % update nlZ and dnlZ end end end end if any(num>1) % check for hypers with more than a single prior num = vec2any(hyp,num); nam = fieldnames(num); s = ''; for i=1:numel(nam) idx = find(num.(nam{i})>1); for j=1:numel(idx) s = [s,sprintf('hyp.%s(%d) ',nam{i},idx(j))]; end end error(['More than 1 prior specified for ',s(1:end-1),'.']) end endgpml-matlab-v4.0-2016-10-19/util/binaryEPGP.m000644 000765 000024 00000005753 12701705601 020163 0ustar00carlstaff000000 000000 function varargout = binaryEPGP(hyper, covfunc, x, y, xstar) % binaryEPGP - The Expectation Propagation approximation for binary Gaussian % process classification. Two modes are possible: training or testing: if no % test cases are supplied, then the approximate negative log marginal % likelihood and its partial derivatives wrt the hyperparameters is computed; % this mode is used to fit the hyperparameters. If test cases are given, then % the test set predictive probabilities are returned. The program is flexible % in allowing a multitude of covariance functions. % % usage: [nlZ, dnlZ ] = binaryEPGP(hyper, covfunc, x, y); % or: [p, mu, s2, nlZ] = binaryEPGP(hyper, covfunc, x, y, xstar); % % where: % % hyper is a (column) vector of hyperparameters % covfunc is the name of the covariance function (see below) % lik is the name of the likelihood function (see below) % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of binary +1/-1 targets % xstar is a nn by D matrix of test inputs % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood wrt each log hyperparameter % p is a (column) vector (of length nn) of predictive probabilities % mu is a (column) vector (of length nn) of predictive latent means % s2 is a (column) vector (of length nn) of predictive latent variances % % The length of the vector of hyperparameters depends on the covariance % function, as specified by the "covfunc" input to the function, specifying the % name of a covariance function. A number of different covariance function are % implemented, and it is not difficult to add new ones. See "help covFunctions" % for the details % % The function can conveniently be used with the "minimize" function to train % a Gaussian process, eg: % % [hyper, fX, i] = minimize(hyper, 'binaryEPGP', length, 'covSEiso', % 'logistic', x, y); % % Copyright (c) 2004, 2005, 2006, 2007 Carl Edward Rasmussen, 2010-07-21. if nargin<4 || nargin>5 disp('Usage: [nlZ, dnlZ ] = binaryEPGP(hyper, covfunc, x, y);') disp(' or: [p, mu, s2, nlZ] = binaryEPGP(hyper, covfunc, x, y, xstar);') return end % Note, this function is just a wrapper provided for backward compatibility, % the functionality is now provided by the more general gp function. lik = @likErf; inf = @infEP; mean = @meanZero; varargout = cell(nargout, 1); % allocate the right number of output arguments hyp.cov = hyper; if nargin==4 [varargout{:}] = gp(hyp,inf,mean,covfunc,lik,x,y); if nargout>1, varargout{2} = varargout{2}.cov; end else [ymu,ys2,fmu,fs2] = gp(hyp,inf,mean,covfunc,lik,x,y,xstar); if nargout>0, varargout{1} = (1+ymu)/2; end if nargout>1, varargout{2} = fmu; end if nargout>2, varargout{3} = fs2; end if nargout>3, varargout{4} = gp(hyp,inf,mean,covfunc,lik,x,y); end % nlZ end gpml-matlab-v4.0-2016-10-19/util/binaryLaplaceGP.m000644 000765 000024 00000007030 12701705601 021206 0ustar00carlstaff000000 000000 function varargout = binaryLaplaceGP(hyper, covfunc, lik, x, y, xstar) % binaryLaplaceGP - Laplace's approximation for binary Gaussian process % classification. Two modes are possible: training or testing: if no test % cases are supplied, then the approximate negative log marginal likelihood % and its partial derivatives wrt the hyperparameters is computed; this mode is % used to fit the hyperparameters. If test cases are given, then the test set % predictive probabilities are returned. The program is flexible in allowing % several different likelihood functions and a multitude of covariance % functions. % % usage: [nlZ, dnlZ ] = binaryLaplaceGP(hyper, covfunc, lik, x, y); % or: [p, mu, s2, nlZ] = binaryLaplaceGP(hyper, covfunc, lik, x, y, xstar); % % where: % % hyper is a (column) vector of hyperparameters % covfunc is the name of the covariance function (see below) % lik is the name of the likelihood function (see below) % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of binary +1/-1 targets % xstar is a nn by D matrix of test inputs % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood wrt each log hyperparameter % p is a (column) vector (of length nn) of predictive probabilities % mu is a (column) vector (of length nn) of predictive latent means % s2 is a (column) vector (of length nn) of predictive latent variances % % The length of the vector of log hyperparameters depends on the covariance % function, as specified by the "covfunc" input to the function, specifying the % name of a covariance function. A number of different covariance function are % implemented, and it is not difficult to add new ones. See "help covFunctions" % for the details. % % The shape of the likelihood function is given by the "lik" input to the % function, specifying the name of the likelihood function. The two implemented % likelihood functions are: % % logistic the logistic function: 1/(1+exp(-x)) % cumGauss the cumulative Gaussian (error function) % % The function can conveniently be used with the "minimize" function to train % a Gaussian process, eg: % % [hyper, fX, i] = minimize(hyper, 'binaryLaplaceGP', length, 'covSEiso', % 'logistic', x, y); % % Copyright (c) 2004, 2005, 2006, 2007 Carl Edward Rasmussen, 2010-07-21. if nargin<5 || nargin>6 disp('Usage: [nlZ, dnlZ ] = binaryLaplaceGP(hyper, covfunc, lik, x, y);') disp(' or: [p, mu, s2, nlZ] = binaryLaplaceGP(hyper, covfunc, lik, x, y, xstar);') return end % Note, this function is just a wrapper provided for backward compatibility, % the functionality is now provided by the more general gp function. if strcmp(lik,'logistic') lik = @likLogistic; elseif strcmpi(lik,'cumgauss') lik = @likErf; else error('Allowable likelihoods: logistic and cumGauss.'); end inf = @infLaplace; mean = @meanZero; varargout = cell(nargout, 1); % allocate the right number of output arguments hyp.cov = hyper; if nargin==5 [varargout{:}] = gp(hyp,inf,mean,covfunc,lik,x,y); if nargout>1, varargout{2} = varargout{2}.cov; end else [ymu,ys2,fmu,fs2] = gp(hyp,inf,mean,covfunc,lik,x,y,xstar); if nargout>0, varargout{1} = (1+ymu)/2; end if nargout>1, varargout{2} = fmu; end if nargout>2, varargout{3} = fs2; end if nargout>3, varargout{4} = gp(hyp,inf,mean,covfunc,lik,x,y); end % nlZ end gpml-matlab-v4.0-2016-10-19/util/logsumexp2.m000644 000765 000024 00000000706 12701705601 020321 0ustar00carlstaff000000 000000 % Compute y = log( sum(exp(x),2) ), the softmax in a numerically safe way by % subtracting the row maximum to avoid cancelation after taking the exp % the sum is done along the rows. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function [y,x] = logsumexp2(logx) N = size(logx,2); max_logx = max(logx,[],2); % we have all values in the log domain, and want to calculate a sum x = exp(logx-max_logx*ones(1,N)); y = log(sum(x,2)) + max_logx;gpml-matlab-v4.0-2016-10-19/util/solve_chol.c000644 000765 000024 00000004371 12701705601 020341 0ustar00carlstaff000000 000000 /* solve_chol - solve a linear system A*X = B using the Cholesky factorization * of A (square, symmetric and positive definite) using LAPACK/DPOTRS. * * Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2014-02-13. * * Modifications from 2014-02-09 as suggested by Todd Small: * - types of q,m,n changed from long to integer, q error check had wrong sign */ #include "mex.h" #include #ifdef MEX_INFORMATION_VERSION /* now we are compiling for Matlab */ #if defined(_WIN64) #define integer long long #else #define integer long #endif #define doublereal double #else /* now we are compiling for Octave */ #ifdef __APPLE__ #include typedef __CLPK_integer integer; typedef __CLPK_doublereal doublereal; #else typedef int integer; typedef double doublereal; #endif #endif #if !defined(_WIN32) || !defined(MEX_INFORMATION_VERSION) /* not Win32/Matlab */ #define dpotrs dpotrs_ #endif extern integer dpotrs_(char *, integer *, integer *, doublereal *, integer *, doublereal *, integer *, integer *); void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) { /* Matlab call: X = solve_chol(A, B) */ doublereal *C; integer n, m, q; if (nrhs != 2 || nlhs > 1) /* check input */ mexErrMsgTxt("Usage: X = solve_chol(A, B)"); n = mxGetM(prhs[0]); /* number of rows in inputs A and B */ if (n != mxGetN(prhs[0])) mexErrMsgTxt("First argument matrix must be square."); if (n != mxGetM(prhs[1])) mexErrMsgTxt("Both argument matrices must have the same number of rows."); m = mxGetN(prhs[1]); /* number of colums in second input B */ plhs[0] = mxCreateDoubleMatrix(n, m, mxREAL); /* space for output X */ C = mxGetPr(plhs[0]); if (n == 0) return; /* if argument was empty matrix, do no more */ memcpy( C, mxGetPr(prhs[1]), m*n*mxGetElementSize(plhs[0]) ); /* copy data */ dpotrs("U", &n, &m, mxGetPr(prhs[0]), &n, C, &n, &q); /* solve system */ if (q < 0) mexErrMsgTxt("Error: illegal input to solve_chol"); } gpml-matlab-v4.0-2016-10-19/util/solve_chol.m000644 000765 000024 00000001742 12701705601 020352 0ustar00carlstaff000000 000000 % solve_chol - solve linear equations from the Cholesky factorization. % Solve A*X = B for X, where A is square, symmetric, positive definite. The % input to the function is R the Cholesky decomposition of A and the matrix B. % Example: X = solve_chol(chol(A),B); % % NOTE: The program code is written in the C language for efficiency and is % contained in the file solve_chol.c, and should be compiled using matlabs mex % facility. However, this file also contains a (less efficient) matlab % implementation, supplied only as a help to people unfamiliar with mex. If % the C code has been properly compiled and is available, it automatically % takes precendence over the matlab code in this file. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2010-09-18. function X = solve_chol(L, B) if nargin ~= 2 || nargout > 1 error('Wrong number of arguments.'); end if size(L,1) ~= size(L,2) || size(L,1) ~= size(B,1) error('Wrong sizes of matrix arguments.'); end X = L\(L'\B);gpml-matlab-v4.0-2016-10-19/util/lbfgsb.m000644 000765 000024 00000005563 12701705601 017461 0ustar00carlstaff000000 000000 % LBFGSB Call the nonlinear bound-constrained solver that uses % limited-memory BFGS quasi-Newton updates. % % The basic function call is % % LBFGSB(x0,lb,ub,objfunc,gradfunc) % % The first input argument x0 is either a matrix or a cell array of % matrices. It declares the starting point for the solver. % % The second and third input arguments lb and ub must be of the exact same % structure as x0. They declare the lower and upper bounds on the % variables, respectively. Set an entry to Inf to declare no bound. % % The next two input arguments must be the names of MATLAB routines % (M-files): % % objfunc Calculates the objective function at the current % point. The routine must accept as many inputs as cell % entry in x0 (or one input if x0 is a matrix). The % output must be a scalar representing the objective % evaluated at the current point. % % gradfunc Computes the gradient of the objective at the current % point. The input is the same as objfunc, but it must % return as many outputs as there are inputs, and each % of the outputs have the same matrix structure as its % corresponding input. % % Optionally, one may choose to pass additional auxiliary data to the % MATLAB callback routines listed above through the function call % LBFGSB(...,auxdata). If auxdata is the empty matrix, no extra % information is passed. It is important to observe that the auxiliary % data MAY NOT CHANGE through the course of the L-BFGS optimization! The % auxiliary data keep the same values as they possessed in the initial % call. If you need variables that change over time, you may want to % consider global variables (type HELP GLOBAL). % % LBFGSB(...,auxdata,iterfunc) specifies an additional callback routine % which is called once per algorithm iteration. The callback routine must % take the form ITERFUNC(T,F,X1,X2,...,AUXDATA). T is the current % iteration of the algorithm. F is the current value of the objective. The % inputs X1, X2,... are exactly the same as the ones passed to objfunc and % gradfunc. Finally, extra information may be passed through the input % AUXDATA. No outputs are expected from iterfunc. If iterfunc is the empty % string, no routine is called. % % These inputs may be followed by parameter/value pairs to modify the % default algorithm options. The algorithm option 'maxiter' sets the % maximum number of iterations. The rest of the options 'm', 'factr' and % 'pgtol' are described in the L-BFGS-B documentation. Note that in this % implementation we divide factr by the machine precision, so that the % convergence condition is satisfied when it is less than factr, not less % than factr times the machine precision. gpml-matlab-v4.0-2016-10-19/util/lbfgsb/000755 000765 000024 00000000000 12777227614 017312 5ustar00carlstaff000000 000000 gpml-matlab-v4.0-2016-10-19/util/infFITC.m000644 000765 000024 00000000364 12757566726 017466 0ustar00carlstaff000000 000000 % Wrapper to infGaussLik to remain backwards compatible. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2016-08-25. function varargout = infFITC(varargin) varargout = cell(nargout, 1); [varargout{:}] = infGaussLik(varargin{:});gpml-matlab-v4.0-2016-10-19/util/minimize_lbfgsb_gradfun.m000644 000765 000024 00000004526 12701705601 023066 0ustar00carlstaff000000 000000 function G = minimize_lbfgsb_gradfun(X,varargin) % extract input arguments varargin = varargin{1}; strctX = varargin{2}; f = varargin{1}; % global variables serve as communication interface between calls global minimize_lbfgsb_iteration_number global minimize_lbfgsb_objective global minimize_lbfgsb_gradient global minimize_lbfgsb_X if norm(unwrap(X)-unwrap(minimize_lbfgsb_X))>1e-10 [y,G] = feval(f,rewrap(strctX,X),varargin{3:end}); else y = minimize_lbfgsb_objective(minimize_lbfgsb_iteration_number); G = minimize_lbfgsb_gradient; end % memorise gradient and position minimize_lbfgsb_gradient = G; minimize_lbfgsb_X = X; G = unwrap(G); % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v4.0-2016-10-19/util/glm_invlink_exp.m000644 000765 000024 00000000722 12777177046 021422 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function g(f) = exp(f). % Output range: 0 <= g(f). % % The function can be used in GLM likelihoods such as likPoisson, likGamma, and % likInvGauss. % % Copyright (c) by Hannes Nickisch, 2016-10-04. function [lg,dlg,d2lg,d3lg] = glm_invlink_exp(f) lg = f; if nargout>1 dlg = ones(size(f)); if nargout>2 d2lg = zeros(size(f)); if nargout>2 d3lg = zeros(size(f)); end end endgpml-matlab-v4.0-2016-10-19/util/minimize_lbfgsb.m000644 000765 000024 00000010574 12701705601 021360 0ustar00carlstaff000000 000000 function [X, fX, i] = minimize_lbfgsb(X, f, length, varargin) % Minimize a differentiable multivariate function using quasi Newton. % % Usage: [X, fX, i] = minimize_lbfgsb(X, f, length, P1, P2, P3, ... ) % % X initial guess; may be of any type, including struct and cell array % f the name or pointer to the function to be minimized. The function % f must return two arguments, the value of the function, and it's % partial derivatives wrt the elements of X. The partial derivative % must have the same type as X. % length length of the run; if it is positive, it gives the maximum number of % line searches, if negative its absolute gives the maximum allowed % number of function evaluations. Optionally, length can have a second % component, which will indicate the reduction in function value to be % expected in the first line-search (defaults to 1.0). % P1, P2 ... parameters are passed to the function f. % % X the returned solution % fX vector of function values indicating progress made % i number of iterations (line searches or function evaluations, % depending on the sign of "length") used at termination. % % The function returns when either its length is up, or if no further progress % can be made (ie, we are at a (local) minimum, or so close that due to % numerical problems, we cannot get any closer). NOTE: If the function % terminates within a few iterations, it could be an indication that the % function values and derivatives are not consistent (ie, there may be a bug in % the implementation of your "f" function). % % Copyright (C) 2010 by Hannes Nickisch, 2010-02-05 % global variables serve as communication interface between calls global minimize_lbfgsb_iteration_number global minimize_lbfgsb_objective global minimize_lbfgsb_gradient global minimize_lbfgsb_X % init global variables minimize_lbfgsb_iteration_number = 0; minimize_lbfgsb_objective = Inf; minimize_lbfgsb_gradient = 0*unwrap(X); minimize_lbfgsb_X = X; X0 = X; lb = -Inf*ones(size(unwrap(X0))); ub = Inf*ones(size(unwrap(X0))); maxiter = abs(length); % max number of iterations % no callback routine used so far % m is the number of saved vectors used to estimate the Hessian % factr is the precision 1e-12 X = lbfgsb( unwrap(X0), lb, ub, 'minimize_lbfgsb_objfun', ... 'minimize_lbfgsb_gradfun', ... {f,X0,varargin{:}}, [], ... 'maxiter',maxiter, 'm',4, 'factr',1e-12, 'pgtol',1e-5); i = minimize_lbfgsb_iteration_number; fX = minimize_lbfgsb_objective; X = rewrap(X0,X); % clear global variables clear minimize_lbfgsb_iteration_number clear minimize_lbfgsb_objective clear minimize_lbfgsb_gradient clear minimize_lbfgsb_X % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v4.0-2016-10-19/util/gpr.m000644 000765 000024 00000004667 12701705601 017016 0ustar00carlstaff000000 000000 function varargout = gpr(hyper, cov, x, y, xs) % gpr - Gaussian process regression, with a named covariance function. Two % modes are possible: training and prediction: if no test data are given, the % function returns minus the log likelihood and its partial derivatives with % respect to the hyperparameters; this mode is used to fit the hyperparameters. % If test data are given, then (marginal) Gaussian predictions are computed, % whose mean and variance are returned. Note that in cases where the covariance % function has noise contributions, the variance returned in s2 is for noisy % test targets; if you want the variance of the noise-free latent function, you % must substract the noise variance. % % usage: [nlZ dnlZ] = gpr(hyp, cov, x, y) % or: [mu s2] = gpr(hyp, cov, x, y, xs) % % where: % % hyp is a (column) vector of log hyperparameters % cov is the covariance function % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of targets % xs is a ns by D matrix of test inputs % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood wrt each log hyperparameter % mu is a (column) vector (of size nn) of prediced means % s2 is a (column) vector (of size nn) of predicted variances % % For more help on covariance functions, see "help covFunctions". % % Copyright (c) 2010 Carl Edward Rasmussen and Hannes Nickisch 2010-06-18. err = 'we need cov = {''covSum'', {cov1, ..,''covNoise'', .., covP} }; to map to gp.m'; if ~strcmp(cov{1},'covSum'), error(err), end id = 0; nhyp = zeros(length(cov{2}),1); for i=1:length(cov{2}) if strcmp(cov{2}{i},'covNoise'), id = i; break, end nhyp(i) = eval(feval(cov{2}{i})); end if id==0, error(err), else nhyp = sum(nhyp(1:id-1)); end cov{2} = cov{2}([1:id-1,id+1:end]); hyp.lik = hyper(nhyp+1); hyp.cov = hyper([1:nhyp,nhyp+2:end]); % Note, this function is just a wrapper provided for backward compatibility, % the functionality is now provided by the more general gp function. lik = @likGauss; inf = @infExact; mean = @meanZero; varargout = cell(nargout, 1); % allocate the right number of output arguments if nargin==4 [varargout{:}] = gp(hyp,inf,mean,cov,lik,x,y); if nargout>1, varargout{2} = [varargout{2}.lik; varargout{2}.cov]; end else [varargout{:}] = gp(hyp,inf,mean,cov,lik,x,y,xs); end gpml-matlab-v4.0-2016-10-19/util/covFITC.m000644 000765 000024 00000000601 12757566726 017473 0ustar00carlstaff000000 000000 % Wrapper to apxSparse to remain backwards compatible. % % Note that covFITC is not a valid covariance function on its own right. % % Copyright (c) by Ed Snelson, Carl Edward Rasmussen % and Hannes Nickisch, 2016-08-25. function varargout = covFITC(varargin) varargout = cell(nargout, 1); [varargout{:}] = apxSparse(varargin{:});gpml-matlab-v4.0-2016-10-19/util/glm_invlink_logit.m000644 000765 000024 00000001503 12777177047 021743 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function g(f) = 1/(1+exp(-f)). % Output range: 0 <= g(f) <= 1. % % The function can be used in GLM likelihoods such as likBeta. % % Copyright (c) by Hannes Nickisch, 2016-10-04. function varargout = glm_invlink_logit(f) varargout = cell(nargout, 1); % allocate the right number of output arguments [varargout{:}] = glm_invlink_logistic(f); if nargout>0 elg = exp(varargout{1}); varargout{1} = min(f-elg,0); % upper bound g by 1 if nargout>1 dlg = varargout{2}; varargout{2} = 1 - elg.*dlg; if nargout>2 d2lg = varargout{3}; varargout{3} = -elg.*(dlg.^2+d2lg); if nargout>3 d3lg = varargout{4}; varargout{4} = -elg.*(dlg.^3+3*d2lg.*dlg+d3lg); end end end endgpml-matlab-v4.0-2016-10-19/util/glm_invlink_expexp.m000644 000765 000024 00000000647 12777177046 022145 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function g(f) = exp(-exp(-f)). % Output range: 0 <= g(f) <= 1. % % The function can be used in GLM likelihoods such as likBeta. % % Copyright (c) by Hannes Nickisch, 2016-10-04. function [lg,dlg,d2lg,d3lg] = glm_invlink_expexp(f) lg = -exp(-f); if nargout>1 dlg = -lg; if nargout>2 d2lg = lg; if nargout>2 d3lg = -lg; end end endgpml-matlab-v4.0-2016-10-19/util/sq_dist.m000644 000765 000024 00000003657 12701705601 017672 0ustar00carlstaff000000 000000 % sq_dist - a function to compute a matrix of all pairwise squared distances % between two sets of vectors, stored in the columns of the two matrices, a % (of size D by n) and b (of size D by m). If only a single argument is given % or the second matrix is empty, the missing matrix is taken to be identical % to the first. % % Usage: C = sq_dist(a, b) % or: C = sq_dist(a) or equiv.: C = sq_dist(a, []) % % Where a is of size Dxn, b is of size Dxm (or empty), C is of size nxm. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-12-13. function C = sq_dist(a, b) if nargin<1 || nargin>3 || nargout>1, error('Wrong number of arguments.'); end bsx = exist('bsxfun','builtin'); % since Matlab R2007a 7.4.0 and Octave 3.0 if ~bsx, bsx = exist('bsxfun'); end % bsxfun is not yet "builtin" in Octave [D, n] = size(a); % Computation of a^2 - 2*a*b + b^2 is less stable than (a-b)^2 because numerical % precision can be lost when both a and b have very large absolute value and the % same sign. For that reason, we subtract the mean from the data beforehand to % stabilise the computations. This is OK because the squared error is % independent of the mean. if nargin==1 % subtract mean mu = mean(a,2); if bsx a = bsxfun(@minus,a,mu); else a = a - repmat(mu,1,size(a,2)); end b = a; m = n; else [d, m] = size(b); if d ~= D, error('Error: column lengths must agree.'); end mu = (m/(n+m))*mean(b,2) + (n/(n+m))*mean(a,2); if bsx a = bsxfun(@minus,a,mu); b = bsxfun(@minus,b,mu); else a = a - repmat(mu,1,n); b = b - repmat(mu,1,m); end end if bsx % compute squared distances C = bsxfun(@plus,sum(a.*a,1)',bsxfun(@minus,sum(b.*b,1),2*a'*b)); else C = repmat(sum(a.*a,1)',1,m) + repmat(sum(b.*b,1),n,1) - 2*a'*b; end C = max(C,0); % numerical noise can cause C to negative i.e. C > -1e-14 gpml-matlab-v4.0-2016-10-19/util/minimize_minfunc.m000644 000765 000024 00000012540 12777177045 021575 0ustar00carlstaff000000 000000 function [X, f, i, exitflag, output] = minimize_minfunc(X, f, options, varargin) % Minimize a differentiable multivariate function using minFunc. % (http://www.cs.ubc.ca/~schmidtm/Software/minFunc.html) % To be used with GPML toolbox. % % Usage: [X, f, i, exitflag, output] = ... % minimize_minfunc(X, f, options/length, P1, P2, ... ) % % X initial guess; may be of any type, including struct and cell array % f the name or pointer to the function to be minimized. The function % f must return two arguments, the value of the function, and it's % partial derivatives wrt the elements of X. The partial derivative % must have the same type as X. % options options to be given to minFunc (see minFunc for details); or % length length of the run; if it is positive, it gives the maximum number of % line searches, if negative its absolute gives the maximum allowed % number of function evaluations. Alternatively, the length of the % run (and other options) can be specified by the option structure. % P1, P2 ... parameters are passed to the function f. % % X the returned solution % f function value at the returned solution % i number of iterations (line searches or function evaluations, % depending on the sign of "length") used at termination. % exitflag returns an exit condition % output returns a structure with other information % Supported Output Options % iterations - number of iterations taken % funcCount - number of function evaluations % algorithm - algorithm used % firstorderopt - first-order optimality % message - exit message % trace.funccount - function evaluations after each iteration % trace.fval - function value after each iteration % % The function returns when either its length is up, or if no further progress % can be made (ie, we are at a (local) minimum, or so close that due to % numerical problems, we cannot get any closer). NOTE: If the function % terminates within a few iterations, it could be an indication that the % function values and derivatives are not consistent (ie, there may be a bug in % the implementation of your "f" function). % % Copyright (C) 2016 by Truong X. Nghiem, 2016-01-22 me = mfilename; % what is my filename? mydir = which(me); mydir = mydir(1:end-2-numel(me)); % where am I located addpath([mydir,'minfunc'],[mydir,'minfunc/compiled']) % add library % Check the option if isempty(options) % Use default options options = struct(); elseif isnumeric(options) && isscalar(options) if options < 0 options = struct('MaxFunEvals', -options); elseif options > 0 options = struct('MaxIter', options); else error('Run length must be non-zero.'); end end assert(isstruct(options), 'Invalid option / length argument.'); % Convert objective function to function_handle if ischar(f) && ~isempty(f), f = str2func(f); end assert(isa(f, 'function_handle'), 'Invalid objective function.'); % Remember that X and the derivative returned by f may not be vectors, so % we must use rewrap and unwrap. X0 = X; % Save the structure / type of X fw = @(cur_x,varargin) wrapped_f(cur_x, f, X0, varargin{:}); [X, f, exitflag, output] = minFunc(fw, unwrap(X0), options,varargin{:}); i = output.funcCount; % X is a vector, we may need to re-wrap it back to the given structure X = rewrap(X0,X); function [fval, dval] = wrapped_f(cur_x, f, X0, varargin) % This function calls unwrap() and rewrap() before and after % calling f % cur_X is a vector; dval must be a vector tmp = rewrap(X0, cur_x); [fval, dstruct] = feval(f,tmp,varargin{:}); % dstruct can be a structure -> convert it to vector dval = unwrap(dstruct); function v = unwrap(s) % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s))); % alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored function [s v] = rewrap(s, v) % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processedgpml-matlab-v4.0-2016-10-19/util/vec2any.m000644 000765 000024 00000001770 12777177047 017611 0ustar00carlstaff000000 000000 % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also any2vec.m. function [s v] = vec2any(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = vec2any(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = vec2any(s{i}, v); end end % other types are not processed gpml-matlab-v4.0-2016-10-19/util/make.m000644 000765 000024 00000004447 12701705601 017137 0ustar00carlstaff000000 000000 % For compilation under Windows/Octave you need to install the package % octavex.x-headers. % Under Linux/Octave, compilation works out of the box. % % There is a big variety of platforms running Matlab/Octave around and the % script is tested for: % % Linux 32bit / Matlab R2008b, 7.7 % Linux 64bit / Matlab R2010b, 7.11 % Windows 7 32bit / Matlab R14, 2004, 7.0 % Windows 7 64bit / Matlab R2011b, 7.13 % Windows XP 32bit / Matlab R2010b, 7.11 % OS X 10.7.5 / Matlab R2013b % OS X 10.9 / Matlab R2013b % % Linux 32bit / Octave 3.2.4 % OS X 10.9.1 / Octave 3.8.0 built with Homebrew % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2014-02-13. OCTAVE = exist('OCTAVE_VERSION') ~= 0; % check if we run Matlab or Octave me = mfilename; % what is my filename mydir = which(me); mydir = mydir(1:end-2-numel(me)); % where am I located cd(mydir) fprintf('Compiling solve_chol.c ...\n') if OCTAVE % Octave if ismac % Accelerate framework needed on OS X fprintf('mkoctfile --mex solve_chol.c "-Wl,-framework,Accelerate"\n') mkoctfile --mex solve_chol.c '-Wl,-framework,Accelerate' else fprintf('mkoctfile --mex solve_chol.c\n') mkoctfile --mex solve_chol.c end delete solve_chol.o else % Matlab if ispc % Windows try % take care of old Matlab versions cc = mex.getCompilerConfigurations; cc = lower(cc.Manufacturer);% compiler catch cc = 'lcc'; end if numel(strfind(computer,'64')) % 64 bit ospath = ['extern/lib/win64/',cc]; else % 32 bit ospath = ['extern/lib/win32/',cc]; end comp_cmd = 'mex -O solve_chol.c -output solve_chol'; fprintf('%s "../%s"\n',comp_cmd,[ospath,'/libmwlapack.lib']) eval([comp_cmd,' ''',matlabroot,'/',ospath,'/libmwlapack.lib','''']) else fprintf('mex -O -lmwlapack solve_chol.c\n') eval('mex -O -lmwlapack solve_chol.c') end end fprintf('Done!\n')gpml-matlab-v4.0-2016-10-19/util/infExact.m000644 000765 000024 00000000365 12757566726 020006 0ustar00carlstaff000000 000000 % Wrapper to infGaussLik to remain backwards compatible. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch 2016-08-25. function varargout = infExact(varargin) varargout = cell(nargout, 1); [varargout{:}] = infGaussLik(varargin{:});gpml-matlab-v4.0-2016-10-19/util/digamma.m000644 000765 000024 00000003665 12777177045 017644 0ustar00carlstaff000000 000000 function y = digamma(x) %DIGAMMA Digamma function. % DIGAMMA(X) returns digamma(x) = d log(gamma(x)) / dx % If X is a matrix, returns the digamma function evaluated at each element. % Reference: % % J Bernardo, % Psi ( Digamma ) Function, % Algorithm AS 103, % Applied Statistics, % Volume 25, Number 3, pages 315-317, 1976. % % From http://www.psc.edu/~burkardt/src/dirichlet/dirichlet.f % Got from https://gist.github.com/renaud/98ee62bb9748904a3432. large = 9.5; d1 = -0.5772156649015328606065121; % digamma(1) d2 = pi^2/6; small = 1e-6; s3 = 1/12; s4 = 1/120; s5 = 1/252; s6 = 1/240; s7 = 1/132; s8 = 691/32760; s9 = 1/12; s10 = 3617/8160; % Initialize y = zeros(size(x)); % illegal arguments i = find(x == -Inf | isnan(x)); if ~isempty(i) x(i) = NaN; y(i) = NaN; end % Negative values i = find(x < 0); if ~isempty(i) % Use the reflection formula (Jeffrey 11.1.6): % digamma(-x) = digamma(x+1) + pi*cot(pi*x) y(i) = digamma(-x(i)+1) + pi*cot(-pi*x(i)); % This is related to the identity % digamma(-x) = digamma(x+1) - digamma(z) + digamma(1-z) % where z is the fractional part of x % For example: % digamma(-3.1) = 1/3.1 + 1/2.1 + 1/1.1 + 1/0.1 + digamma(1-0.1) % = digamma(4.1) - digamma(0.1) + digamma(1-0.1) % Then we use % digamma(1-z) - digamma(z) = pi*cot(pi*z) end i = find(x == 0); if ~isempty(i) y(i) = -Inf; end % Use approximation if argument <= small. i = find(x > 0 & x <= small); if ~isempty(i) y(i) = y(i) + d1 - 1 ./ x(i) + d2*x(i); end % Reduce to digamma(X + N) where (X + N) >= large. while(1) i = find(x > small & x < large); if isempty(i) break end y(i) = y(i) - 1 ./ x(i); x(i) = x(i) + 1; end % Use de Moivre's expansion if argument >= large. % In maple: asympt(Psi(x), x); i = find(x >= large); if ~isempty(i) r = 1 ./ x(i); y(i) = y(i) + log(x(i)) - 0.5 * r; r = r .* r; y(i) = y(i) - r .* ( s3 - r .* ( s4 - r .* (s5 - r .* (s6 - r .* s7)))); end gpml-matlab-v4.0-2016-10-19/util/logphi.m000644 000765 000024 00000004325 12701705601 017477 0ustar00carlstaff000000 000000 % Safe computation of logphi(z) = log(normcdf(z)) and its derivatives % dlogphi(z) = normpdf(x)/normcdf(x). % The function is based on index 5725 in Hart et al. and gsl_sf_log_erfc_e. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2013-11-13. function [lp,dlp,d2lp,d3lp] = logphi(z) z = real(z); % support for real arguments only lp = zeros(size(z)); % allocate memory id1 = z.*z<0.0492; % first case: close to zero lp0 = -z(id1)/sqrt(2*pi); c = [ 0.00048204; -0.00142906; 0.0013200243174; 0.0009461589032; -0.0045563339802; 0.00556964649138; 0.00125993961762116; -0.01621575378835404; 0.02629651521057465; -0.001829764677455021; 2*(1-pi/3); (4-pi)/3; 1; 1]; f = 0; for i=1:14, f = lp0.*(c(i)+f); end, lp(id1) = -2*f-log(2); id2 = z<-11.3137; % second case: very small r = [ 1.2753666447299659525; 5.019049726784267463450; 6.1602098531096305441; 7.409740605964741794425; 2.9788656263939928886 ]; q = [ 2.260528520767326969592; 9.3960340162350541504; 12.048951927855129036034; 17.081440747466004316; 9.608965327192787870698; 3.3690752069827527677 ]; num = 0.5641895835477550741; for i=1:5, num = -z(id2).*num/sqrt(2) + r(i); end den = 1.0; for i=1:6, den = -z(id2).*den/sqrt(2) + q(i); end e = num./den; lp(id2) = log(e/2) - z(id2).^2/2; id3 = ~id2 & ~id1; lp(id3) = log(erfc(-z(id3)/sqrt(2))/2); % third case: rest if nargout>1 % compute first derivative dlp = zeros(size(z)); % allocate memory dlp( id2) = abs(den./num) * sqrt(2/pi); % strictly positive first derivative dlp(~id2) = exp(-z(~id2).*z(~id2)/2-lp(~id2))/sqrt(2*pi); % safe computation if nargout>2 % compute second derivative d2lp = -dlp.*abs(z+dlp); % strictly negative second derivative if nargout>3 % compute third derivative d3lp = -d2lp.*abs(z+2*dlp)-dlp; % strictly positive third derivative end end endgpml-matlab-v4.0-2016-10-19/util/any2vec.m000644 000765 000024 00000001215 12777177047 017603 0ustar00carlstaff000000 000000 % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse vec2any.m. function v = any2vec(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = any2vec(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; any2vec(s{i})]; end end % other types are ignored gpml-matlab-v4.0-2016-10-19/util/minfunc/000755 000765 000024 00000000000 12777227614 017512 5ustar00carlstaff000000 000000 gpml-matlab-v4.0-2016-10-19/util/brentmin.m000644 000765 000024 00000007574 12701705601 020044 0ustar00carlstaff000000 000000 %% BRENTMIN: Brent's minimization method in one dimension function [xmin,fmin,funccount,varargout] = ... brentmin(xlow,xupp,Nitmax,tol,f,nout,varargin) % code taken from % § 10.2 Parabolic Interpolation and Brent's Method in One Dimension % Press, Teukolsky, Vetterling & Flannery % Numerical Recipes in C, Cambridge University Press, 2002 % % [xmin,fmin,funccout,varargout] = BRENTMIN(xlow,xupp,Nit,tol,f,nout,varargin) % Given a function f, and given a search interval this routine isolates % the minimum of fractional precision of about tol using Brent's method. % % INPUT % ----- % xlow,xupp: search interval such that xlow<=xmin<=xupp % Nitmax: maximum number of function evaluations made by the routine % tol: fractional precision % f: [y,varargout{:}] = f(x,varargin{:}) is the function % nout: no. of outputs of f (in varargout) in addition to the y value % % OUTPUT % ------ % fmin: minimal function value % xmin: corresponding abscissa-value % funccount: number of function evaluations made % varargout: additional outputs of f at optimum % % Copyright (c) by Hannes Nickisch 2010-01-10. if nargin<6, nout = 0; end varargout = cell(nout,1); % tolerance is no smaller than machine's floating point precision tol = max(tol,eps); % Evaluate endpoints fa = f(xlow,varargin{:}); fb = f(xupp,varargin{:}); funccount = 2; % number of function evaluations % Compute the start point seps = sqrt(eps); c = 0.5*(3.0 - sqrt(5.0));% golden ratio a = xlow; b = xupp; v = a + c*(b-a); w = v; xf = v; d = 0.0; e = 0.0; x = xf; [fx,varargout{:}] = f(x,varargin{:}); funccount = funccount + 1; fv = fx; fw = fx; xm = 0.5*(a+b); tol1 = seps*abs(xf) + tol/3.0; tol2 = 2.0*tol1; % Main loop while ( abs(xf-xm) > (tol2 - 0.5*(b-a)) ) gs = 1; % Is a parabolic fit possible if abs(e) > tol1 % Yes, so fit parabola gs = 0; r = (xf-w)*(fx-fv); q = (xf-v)*(fx-fw); p = (xf-v)*q-(xf-w)*r; q = 2.0*(q-r); if q > 0.0, p = -p; end q = abs(q); r = e; e = d; % Is the parabola acceptable if ( (abs(p)q*(a-xf)) && (p= xm, e = a-xf; else e = b-xf; end d = c*e; end % The function must not be evaluated too close to xf si = sign(d) + (d == 0); x = xf + si * max( abs(d), tol1 ); [fu,varargout{:}] = f(x,varargin{:}); funccount = funccount + 1; % Update a, b, v, w, x, xm, tol1, tol2 if fu <= fx if x >= xf, a = xf; else b = xf; end v = w; fv = fw; w = xf; fw = fx; xf = x; fx = fu; else % fu > fx if x < xf, a = x; else b = x; end if ( (fu <= fw) || (w == xf) ) v = w; fv = fw; w = x; fw = fu; elseif ( (fu <= fv) || (v == xf) || (v == w) ) v = x; fv = fu; end end xm = 0.5*(a+b); tol1 = seps*abs(xf) + tol/3.0; tol2 = 2.0*tol1; if funccount >= Nitmax % typically we should not get here % warning(sprintf(['Maximum number of iterations (%d) exceeded:', ... % 'precision is not guaranteed'],Nitmax)) % fprintf('[%1.3f,%1.3f,%1.3f]\n',xlow,xf,xupp) break end end % while % check that endpoints are less than the minimum found if ( (fa < fx) && (fa <= fb) ) xf = xlow; fx = fa; elseif fb < fx xf = xupp; fx = fb; end fmin = fx; xmin = xf; gpml-matlab-v4.0-2016-10-19/util/minimize.m000644 000765 000024 00000025666 12777177043 020071 0ustar00carlstaff000000 000000 function [X, fX, i] = minimize(X, f, length, varargin) % Minimize a differentiable multivariate function using conjugate gradients. % % Usage: [X, fX, i] = minimize(X, f, length, P1, P2, P3, ... ) % % X initial guess; may be of any type, including struct and cell array % f the name or pointer to the function to be minimized. The function % f must return two arguments, the value of the function, and it's % partial derivatives wrt the elements of X. The partial derivative % must have the same type as X. % length length of the run; if it is positive, it gives the maximum number of % line searches, if negative its absolute gives the maximum allowed % number of function evaluations. Optionally, length can have a second % component, which will indicate the reduction in function value to be % expected in the first line-search (defaults to 1.0). % P1, P2, ... parameters are passed to the function f. % % X the returned solution % fX vector of function values indicating progress made % i number of iterations (line searches or function evaluations, % depending on the sign of "length") used at termination. % % The function returns when either its length is up, or if no further progress % can be made (ie, we are at a (local) minimum, or so close that due to % numerical problems, we cannot get any closer). NOTE: If the function % terminates within a few iterations, it could be an indication that the % function values and derivatives are not consistent (ie, there may be a bug in % the implementation of your "f" function). % % The Polack-Ribiere flavour of conjugate gradients is used to compute search % directions, and a line search using quadratic and cubic polynomial % approximations and the Wolfe-Powell stopping criteria is used together with % the slope ratio method for guessing initial step sizes. Additionally a bunch % of checks are made to make sure that exploration is taking place and that % extrapolation will not be unboundedly large. % % See also: checkgrad % % Copyright (C) 2001 - 2010 by Carl Edward Rasmussen, 2010-01-03 INT = 0.1; % don't reevaluate within 0.1 of the limit of the current bracket EXT = 3.0; % extrapolate maximum 3 times the current step-size MAX = 20; % max 20 function evaluations per line search RATIO = 10; % maximum allowed slope ratio SIG = 0.1; RHO = SIG/2; % SIG and RHO are the constants controlling the Wolfe- % Powell conditions. SIG is the maximum allowed absolute ratio between % previous and new slopes (derivatives in the search direction), thus setting % SIG to low (positive) values forces higher precision in the line-searches. % RHO is the minimum allowed fraction of the expected (from the slope at the % initial point in the linesearch). Constants must satisfy 0 < RHO < SIG < 1. % Tuning of SIG (depending on the nature of the function to be optimized) may % speed up the minimization; it is probably not worth playing much with RHO. % The code falls naturally into 3 parts, after the initial line search is % started in the direction of steepest descent. 1) we first enter a while loop % which uses point 1 (p1) and (p2) to compute an extrapolation (p3), until we % have extrapolated far enough (Wolfe-Powell conditions). 2) if necessary, we % enter the second loop which takes p2, p3 and p4 chooses the subinterval % containing a (local) minimum, and interpolates it, unil an acceptable point % is found (Wolfe-Powell conditions). Note, that points are always maintained % in order p0 <= p1 <= p2 < p3 < p4. 3) compute a new search direction using % conjugate gradients (Polack-Ribiere flavour), or revert to steepest if there % was a problem in the previous line-search. Return the best value so far, if % two consecutive line-searches fail, or whenever we run out of function % evaluations or line-searches. During extrapolation, the "f" function may fail % either with an error or returning Nan or Inf, and minimize should handle this % gracefully. if max(size(length)) == 2, red=length(2); length=length(1); else red=1; end if length>0, S='Linesearch'; else S='Function evaluation'; end i = 0; % zero the run length counter ls_failed = 0; % no previous line search has failed [f0 df0] = feval(f, X, varargin{:}); % get function value and gradient Z = X; X = unwrap(X); df0 = unwrap(df0); fprintf('%s %6i; Value %4.6e\r', S, i, f0); if exist('fflush','builtin') fflush(stdout); end fX = f0; i = i + (length<0); % count epochs?! s = -df0; d0 = -s'*s; % initial search direction (steepest) and slope x3 = red/(1-d0); % initial step is red/(|s|+1) while i < abs(length) % while not finished i = i + (length>0); % count iterations?! X0 = X; F0 = f0; dF0 = df0; % make a copy of current values if length>0, M = MAX; else M = min(MAX, -length-i); end while 1 % keep extrapolating as long as necessary x2 = 0; f2 = f0; d2 = d0; f3 = f0; df3 = df0; success = 0; while ~success && M > 0 try M = M - 1; i = i + (length<0); % count epochs?! [f3 df3] = feval(f, rewrap(Z,X+x3*s), varargin{:}); df3 = unwrap(df3); if isnan(f3) || isinf(f3) || any(isnan(df3)+isinf(df3)), error(' '),end success = 1; catch % catch any error which occured in f x3 = (x2+x3)/2; % bisect and try again end end if f3 < F0, X0 = X+x3*s; F0 = f3; dF0 = df3; end % keep best values d3 = df3'*s; % new slope if d3 > SIG*d0 || f3 > f0+x3*RHO*d0 || M == 0 % are we done extrapolating? break end x1 = x2; f1 = f2; d1 = d2; % move point 2 to point 1 x2 = x3; f2 = f3; d2 = d3; % move point 3 to point 2 A = 6*(f1-f2)+3*(d2+d1)*(x2-x1); % make cubic extrapolation B = 3*(f2-f1)-(2*d1+d2)*(x2-x1); x3 = x1-d1*(x2-x1)^2/(B+sqrt(B*B-A*d1*(x2-x1))); % num. error possible, ok! if ~isreal(x3) || isnan(x3) || isinf(x3) || x3 < 0 % num prob | wrong sign? x3 = x2*EXT; % extrapolate maximum amount elseif x3 > x2*EXT % new point beyond extrapolation limit? x3 = x2*EXT; % extrapolate maximum amount elseif x3 < x2+INT*(x2-x1) % new point too close to previous point? x3 = x2+INT*(x2-x1); end end % end extrapolation while (abs(d3) > -SIG*d0 || f3 > f0+x3*RHO*d0) && M > 0 % keep interpolating if d3 > 0 || f3 > f0+x3*RHO*d0 % choose subinterval x4 = x3; f4 = f3; d4 = d3; % move point 3 to point 4 else x2 = x3; f2 = f3; d2 = d3; % move point 3 to point 2 end if f4 > f0 x3 = x2-(0.5*d2*(x4-x2)^2)/(f4-f2-d2*(x4-x2)); % quadratic interpolation else A = 6*(f2-f4)/(x4-x2)+3*(d4+d2); % cubic interpolation B = 3*(f4-f2)-(2*d2+d4)*(x4-x2); x3 = x2+(sqrt(B*B-A*d2*(x4-x2)^2)-B)/A; % num. error possible, ok! end if isnan(x3) || isinf(x3) x3 = (x2+x4)/2; % if we had a numerical problem then bisect end x3 = max(min(x3, x4-INT*(x4-x2)),x2+INT*(x4-x2)); % don't accept too close [f3 df3] = feval(f, rewrap(Z,X+x3*s), varargin{:}); df3 = unwrap(df3); if f3 < F0, X0 = X+x3*s; F0 = f3; dF0 = df3; end % keep best values M = M - 1; i = i + (length<0); % count epochs?! d3 = df3'*s; % new slope end % end interpolation if abs(d3) < -SIG*d0 && f3 < f0+x3*RHO*d0 % if line search succeeded X = X+x3*s; f0 = f3; fX = [fX' f0]'; % update variables fprintf('%s %6i; Value %4.6e\r', S, i, f0); if exist('fflush','builtin') fflush(stdout); end s = (df3'*df3-df0'*df3)/(df0'*df0)*s - df3; % Polack-Ribiere CG direction df0 = df3; % swap derivatives d3 = d0; d0 = df0'*s; if d0 > 0 % new slope must be negative s = -df0; d0 = -s'*s; % otherwise use steepest direction end x3 = x3 * min(RATIO, d3/(d0-realmin)); % slope ratio but max RATIO ls_failed = 0; % this line search did not fail else X = X0; f0 = F0; df0 = dF0; % restore best point so far if ls_failed || i > abs(length) % line search failed twice in a row break; % or we ran out of time, so we give up end s = -df0; d0 = -s'*s; % try steepest x3 = 1/(1-d0); ls_failed = 1; % this line search failed end end X = rewrap(Z,X); fprintf('\n'); if exist('fflush','builtin') fflush(stdout); end function v = unwrap(s) % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s))); % alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored function [s v] = rewrap(s, v) % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v4.0-2016-10-19/util/minimize_lbfgsb_objfun.m000644 000765 000024 00000005207 12701705601 022720 0ustar00carlstaff000000 000000 function y = minimize_lbfgsb_objfun(X,varargin) % extract input arguments varargin = varargin{1}; strctX = varargin{2}; f = varargin{1}; % global variables serve as communication interface between calls global minimize_lbfgsb_iteration_number global minimize_lbfgsb_objective global minimize_lbfgsb_gradient global minimize_lbfgsb_X recompute = 0; if minimize_lbfgsb_iteration_number==0 recompute = 1; else if norm(unwrap(X)-unwrap(minimize_lbfgsb_X))>1e-10 recompute = 1; end end if recompute [y,G] = feval(f,rewrap(strctX,X),varargin{3:end}); else y = minimize_lbfgsb_objective(minimize_lbfgsb_iteration_number); G = minimize_lbfgsb_gradient; end % increase global counter, memorise objective function, gradient and position minimize_lbfgsb_iteration_number = minimize_lbfgsb_iteration_number+1; minimize_lbfgsb_objective(minimize_lbfgsb_iteration_number,1) = y; minimize_lbfgsb_gradient = G; minimize_lbfgsb_X = X; % Extract the numerical values from "s" into the column vector "v". The % variable "s" can be of any type, including struct and cell array. % Non-numerical elements are ignored. See also the reverse rewrap.m. function v = unwrap(s) v = []; if isnumeric(s) v = s(:); % numeric values are recast to column vector elseif isstruct(s) v = unwrap(struct2cell(orderfields(s)));% alphabetize, conv to cell, recurse elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially v = [v; unwrap(s{i})]; end end % other types are ignored % Map the numerical elements in the vector "v" onto the variables "s" which can % be of any type. The number of numerical elements must match; on exit "v" % should be empty. Non-numerical entries are just copied. See also unwrap.m. function [s v] = rewrap(s, v) if isnumeric(s) if numel(v) < numel(s) error('The vector for conversion contains too few elements') end s = reshape(v(1:numel(s)), size(s)); % numeric values are reshaped v = v(numel(s)+1:end); % remaining arguments passed on elseif isstruct(s) [s p] = orderfields(s); p(p) = 1:numel(p); % alphabetize, store ordering [t v] = rewrap(struct2cell(s), v); % convert to cell, recurse s = orderfields(cell2struct(t,fieldnames(s),1),p); % conv to struct, reorder elseif iscell(s) for i = 1:numel(s) % cell array elements are handled sequentially [s{i} v] = rewrap(s{i}, v); end end % other types are not processed gpml-matlab-v4.0-2016-10-19/util/covGrid.m000644 000765 000024 00000000470 12757566726 017637 0ustar00carlstaff000000 000000 % Wrapper to apxGrid to remain backwards compatible. % % Note that covGrid is not a valid covariance function on its own right. % % Copyright (c) by Hannes Nickisch and Andrew Wilson 2016-08-25. function varargout = covGrid(varargin) varargout = cell(nargout, 1); [varargout{:}] = apxGrid(varargin{:}); gpml-matlab-v4.0-2016-10-19/util/gauher.m000644 000765 000024 00000004305 12701705601 017466 0ustar00carlstaff000000 000000 % compute abscissas and weight factors for Gaussian-Hermite quadrature % % CALL: [x,w] = gauher(N) % % x = base points (abscissas) % w = weight factors % N = number of base points (abscissas) (integrates an up to (2N-1)th order % polynomial exactly) % % p(x)=exp(-x^2/2)/sqrt(2*pi), a =-Inf, b = Inf % % The Gaussian Quadrature integrates a (2n-1)th order % polynomial exactly and the integral is of the form % b N % Int ( p(x)* F(x) ) dx = Sum ( w_j* F( x_j ) ) % a j=1 % % this procedure uses the coefficients a(j), b(j) of the % recurrence relation % % b p (x) = (x - a ) p (x) - b p (x) % j j j j-1 j-1 j-2 % % for the various classical (normalized) orthogonal polynomials, % and the zero-th moment % % 1 = integral w(x) dx % % of the given polynomial's weight function w(x). Since the % polynomials are orthonormalized, the tridiagonal matrix is % guaranteed to be symmetric. function [x,w]=gauher(N) if N==20 % return precalculated values x=[ -7.619048541679757;-6.510590157013656;-5.578738805893203; -4.734581334046057;-3.943967350657318;-3.18901481655339 ; -2.458663611172367;-1.745247320814127;-1.042945348802751; -0.346964157081356; 0.346964157081356; 1.042945348802751; 1.745247320814127; 2.458663611172367; 3.18901481655339 ; 3.943967350657316; 4.734581334046057; 5.578738805893202; 6.510590157013653; 7.619048541679757]; w=[ 0.000000000000126; 0.000000000248206; 0.000000061274903; 0.00000440212109 ; 0.000128826279962; 0.00183010313108 ; 0.013997837447101; 0.061506372063977; 0.161739333984 ; 0.260793063449555; 0.260793063449555; 0.161739333984 ; 0.061506372063977; 0.013997837447101; 0.00183010313108 ; 0.000128826279962; 0.00000440212109 ; 0.000000061274903; 0.000000000248206; 0.000000000000126 ]; else b = sqrt( (1:N-1)/2 )'; [V,D] = eig( diag(b,1) + diag(b,-1) ); w = V(1,:)'.^2; x = sqrt(2)*diag(D); end gpml-matlab-v4.0-2016-10-19/util/lik_epquad.m000644 000765 000024 00000003126 12701705601 020331 0ustar00carlstaff000000 000000 % Compute infEP part of a likelihood function based on the infLaplace part using % Gaussian-Hermite quadrature. % % The function is used in GLM likelihoods such as likPoisson, likGamma, likBeta % and likInvGauss. % % Copyright (c) by Hannes Nickisch, 2013-10-16. function varargout = lik_epquad(lik,hyp,y,mu,s2) n = max([length(y),length(mu),length(s2)]); on = ones(n,1); N = 20; [t,w] = gauher(N); oN = ones(1,N); lw = ones(n,1)*log(w'); y = y(:).*on; mu = mu(:).*on; sig = sqrt(s2(:)).*on; % vectors only [lpi,dlpi,d2lpi] = feval(lik{:},hyp,y*oN,sig*t'+mu*oN,[],'infLaplace'); lZ = s(lpi+lw); dlZ = {}; d2lZ = {}; if nargout>1 % 1st derivative wrt mean % Using p*dlp=dp, p=exp(lp), Z=sum_i wi*pi, dZ = sum_i wi*dpi we obtain % dlZ = sum_i exp(lpi-lZ+lwi)*dlpi = sum_i ai*dlpi. a = exp(lpi - lZ*oN + lw); dlZ = sum(a.*dlpi,2); if nargout>2 % 2nd derivative wrt mean % Using d2lZ=(d2Z*Z-dZ^2)/Z^2 <=> d2Z=Z*(d2lZ+dlZ^2) and % d2Z = sum_i wi*d2Zi, we get d2lZ = sum_i ai*(d2lpi+dlpi^2)-dlZ^2. d2lZ = sum(a.*(d2lpi+dlpi.*dlpi),2) - dlZ.*dlZ; end end varargout = {lZ,dlZ,d2lZ}; % computes y = log( sum(exp(x),2) ), the softmax in a numerically safe way by % subtracting the row maximum to avoid cancelation after taking the exp % the sum is done along the rows function [y,x] = s(logx) N = size(logx,2); max_logx = max(logx,[],2); % we have all values in the log domain, and want to calculate a sum x = exp(logx-max_logx*ones(1,N)); y = log(sum(x,2)) + max_logx;gpml-matlab-v4.0-2016-10-19/util/glm_invlink_logistic2.m000644 000765 000024 00000001546 12777177047 022533 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function (twice logistic) % g(f) = h(f*(1+a*h(f))), where is the logistic h(f) = log(1+exp(f))). % Output range: 0 <= g(f). % % The function can be used in GLM likelihoods such as likPoisson, likGamma, and % likInvGauss. % % See Seeger et al., Bayesian Intermittent Demand Forecasting for Large % Inventories, NIPS, 2016. % % Copyright (c) by Hannes Nickisch, 2016-10-04. function [lg,dlg,d2lg,d3lg] = glm_invlink_logistic2(a,f) [lh,dlh,d2lh,d3lh] = glm_invlink_logistic(f); h = exp(lh); ft = f + a*f.*h; dft = 1 + a*h.*(1 + f.*dlh); w = a*h.*(dlh.^2 + d2lh); d2ft = 2*a*h.*dlh + f.*w; d3ft = 3*w + a*f.*h.*(dlh.^3+3*d2lh.*dlh+d3lh); [lgt,dlgt,d2lgt,d3lgt] = glm_invlink_logistic(ft); lg = lgt; dlg = dlgt.*dft; d2lg = d2lgt.*dft.^2 + dlgt.*d2ft; d3lg = d3lgt.*dft.^3 + 3*d2lgt.*dft.*d2ft + dlgt.*d3ft;gpml-matlab-v4.0-2016-10-19/util/binaryGP.m000644 000765 000024 00000011554 12760253146 017741 0ustar00carlstaff000000 000000 function varargout = binaryGP(hyper, approx, covfunc, lik, x, y, xstar) % Approximate binary Gaussian Process classification. Two modes are possible: % training or testing: if no test cases are supplied, then the approximate % negative log marginal likelihood and its partial derivatives wrt the % hyperparameters is computed; this mode is used to fit the hyperparameters. If % test cases are given, then the test set predictive probabilities are % returned. Exact inference is intractible, the function uses a specified % approximation method (see approximations.m), flexible covariance functions % (see covFunctions.m) and likelihood functions (see likelihoods.m). % % usage: [nlZ, dnlZ ] = binaryGP(hyper, approx, covfunc, lik, x, y); % or: [p,mu,s2,nlZ] = binaryGP(hyper, approx, covfunc, lik, x, y, xstar); % % where: % % hyper is a column vector of hyperparameters % approx is a function specifying an approximation method for inference % covfunc is the name of the covariance function (see below) % lik is the name of the likelihood function % x is a n by D matrix of training inputs % y is a (column) vector (of size n) of binary +1/-1 targets % xstar is a nn by D matrix of test inputs % nlZ is the returned value of the negative log marginal likelihood % dnlZ is a (column) vector of partial derivatives of the negative % log marginal likelihood wrt each hyperparameter % p is a (column) vector (of length nn) of predictive probabilities % mu is a (column) vector (of length nn) of predictive latent means % s2 is a (column) vector (of length nn) of predictive latent variances % % The length of the vector of hyperparameters depends on the covariance % function, as specified by the "covfunc" input to the function, specifying the % name of a covariance function. A number of different covariance function are % implemented, and it is not difficult to add new ones. See covFunctions.m for % the details. % % The "lik" input argument specifies the name of the likelihood function (see % likelihoods.m). % % The "approx" input argument to the function specifies an approximation method % (see approximations.m). An approximation method returns a representation of % the approximate Gaussian posterior. Usually, the approximate posterior admits % the form N(m=K*alpha, V=inv(inv(K)+W)), where alpha is a vector and W is % diagonal. The approximation method returns: % % alpha is a (sparse or full column vector) containing inv(K)*m, where K % is the prior covariance matrix and m the approx posterior mean % sW is a (sparse or full column) vector containing diagonal of sqrt(W) % the approximate posterior covariance matrix is inv(inv(K)+W) % L is a (sparse or full) matrix, L = chol(sW*K*sW+eye(n)) % % In cases where the approximate posterior variance does not admit the form % V=inv(inv(K)+W) with diagonal W, L contains instead -inv(K+inv(W)), and sW % is unused. % % The alpha parameter may be sparse. In that case sW and L can either be sparse % or full (retaining only the non-zero rows and columns, as indicated by the % sparsity structure of alpha). The L paramter is allowed to be empty, in % which case it will be computed. % % The function can conveniently be used with the "minimize" function to train % a Gaussian Process, eg: % % [hyper, fX, i] = minimize(hyper, 'binaryGP', length, 'approxEP', 'covSEiso', 'logistic', x, y); % % where "length" gives the length of the run: if it is positive, it gives the % maximum number of line searches, if negative its absolute gives the maximum % allowed number of function evaluations. % % Copyright (c) 2007 Carl Edward Rasmussen and Hannes Nickisch, 2010-07-21. if nargin<6 || nargin>7 disp('Usage: [nlZ, dnlZ ] = binaryGP(hyper,approx,covfunc,lik,x,y);') disp(' or: [p,mu,s2,nlZ] = binaryGP(hyper,approx,covfunc,lik,x,y,xstar);') return end % Note, this function is just a wrapper provided for backward compatibility, % the functionality is now provided by the more general gp function. if strcmp(lik,'logistic') lik = 'likLogistic'; elseif strcmpi(lik,'cumgauss') lik = 'likErf'; else error('Allowable likelihoods: logistic and cumGauss.'); end if strcmp(approx,'approxLA') inf = 'infLaplace'; elseif strcmp(approx,'approxEP') inf = 'infEP'; else error('Allowable approximations: approxLA and approxEP.'); end mean = 'meanZero'; varargout = cell(nargout, 1); % allocate the right number of output arguments hyp.cov = hyper; if nargin==6 [varargout{:}] = gp(hyp,inf,mean,covfunc,lik,x,y); if nargout>1, varargout{2} = varargout{2}.cov; end else [ymu,ys2,fmu,fs2] = gp(hyp,inf,mean,covfunc,lik,x,y,xstar); if nargout>0, varargout{1} = (1+ymu)/2; end if nargout>1, varargout{2} = fmu; end if nargout>2, varargout{3} = fs2; end if nargout>3, varargout{4} = gp(hyp,inf,mean,covfunc,lik,x,y); end % nlZ endgpml-matlab-v4.0-2016-10-19/util/elsympol.m000644 000765 000024 00000001740 12701705601 020057 0ustar00carlstaff000000 000000 % Evaluate the order R elementary symmetric polynomials using Newton's identity, % the Newton-Girard formulae: http://en.wikipedia.org/wiki/Newton's_identities % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2010-01-10. % speedup contributed by Truong X. Nghiem, 2016-01-20. function E = elsympol(Z,R) sz = size(Z); % evaluate 'power sums' of the individual terms in Z E = zeros([sz(1:2),R+1]); % E(:,:,r+1) yields polynomial r % fast and efficient version of: for r=1:R, P(:,:,r) = sum(Z.^r,3); end Zr = Z; P(:,:,1) = sum(Zr,3); for r=2:R, Zr = Zr.*Z; P(:,:,r) = sum(Zr,3); end E(:,:,1) = ones(sz(1:2)); if R==0, return, end % init recursion E(:,:,2) = P(:,:,1); if R==1, return, end % init recursion for r=2:R E(:,:,r+1) = P(:,:,1).*E(:,:,r)/r; % i=1 is simpler than the rest for i=2:r E(:,:,r+1) = E(:,:,r+1) + P(:,:,i).*E(:,:,r+1-i)*(-1)^(i-1)/r; end endgpml-matlab-v4.0-2016-10-19/util/glm_invlink_logistic.m000644 000765 000024 00000001305 12777177046 022441 0ustar00carlstaff000000 000000 % Compute the log intensity for the inverse link function g(f) = log(1+exp(f))). % Output range: 0 <= g(f). % % The function can be used in GLM likelihoods such as likPoisson, likGamma, and % likInvGauss. % % Copyright (c) by Hannes Nickisch, 2016-10-04. function [lg,dlg,d2lg,d3lg] = glm_invlink_logistic(f) l1pef = max(0,f) + log(1+exp(-abs(f))); % safely compute log(1+exp(f)) lg = log(l1pef); id = f<-15; lg(id) = f(id); % fix log(log(1+exp(f))) limits if nargout>1 sm = 1./(1+exp(-f)); dlg = sm./l1pef; dlg(f<-15) = 1; if nargout>2 sp = 1./(1+exp(f)); d2lg = dlg.*(sp-dlg); if nargout>2 d3lg = d2lg.*(sp-2*dlg) - dlg.*sp.*sm; end end endgpml-matlab-v4.0-2016-10-19/util/minfunc/taylorModel.m000644 000765 000024 00000001245 12701705601 022145 0ustar00carlstaff000000 000000 function [f,g,H] = taylorModel(d,f,g,H,T) p = length(d); fd3 = 0; gd2 = zeros(p,1); Hd = zeros(p); for t1 = 1:p for t2 = 1:p for t3 = 1:p fd3 = fd3 + T(t1,t2,t3)*d(t1)*d(t2)*d(t3); if nargout > 1 gd2(t3) = gd2(t3) + T(t1,t2,t3)*d(t1)*d(t2); end if nargout > 2 Hd(t2,t3) = Hd(t2,t3) + T(t1,t2,t3)*d(t1); end end end end f = f + g'*d + (1/2)*d'*H*d + (1/6)*fd3; if nargout > 1 g = g + H*d + (1/2)*gd2; end if nargout > 2 H = H + Hd; end if any(abs(d) > 1e5) % We want the optimizer to stop if the solution is unbounded g = zeros(p,1); endgpml-matlab-v4.0-2016-10-19/util/minfunc/License000644 000765 000024 00000003300 12701705601 020773 0ustar00carlstaff000000 000000 The software on this webpage is distributed under the FreeBSD-style license below. Although it is not required, I would also appreciate that any re-distribution of the software contains a link to the original webpage. For example, the webpage for the 'minFunc' software is: http://www.cs.ubc.ca/~schmidtm/Software/minFunc.html ======================================================================================= Copyright 2005-2015 Mark Schmidt. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE AUTHOR ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.gpml-matlab-v4.0-2016-10-19/util/minfunc/lbfgsUpdate.m000644 000765 000024 00000001146 12701705601 022112 0ustar00carlstaff000000 000000 function [old_dirs,old_stps,Hdiag] = lbfgsUpdate(y,s,corrections,debug,old_dirs,old_stps,Hdiag) ys = y'*s; if ys > 1e-10 numCorrections = size(old_dirs,2); if numCorrections < corrections % Full Update old_dirs(:,numCorrections+1) = s; old_stps(:,numCorrections+1) = y; else % Limited-Memory Update old_dirs = [old_dirs(:,2:corrections) s]; old_stps = [old_stps(:,2:corrections) y]; end % Update scale of initial Hessian approximation Hdiag = ys/(y'*y); else if debug fprintf('Skipping Update\n'); end endgpml-matlab-v4.0-2016-10-19/util/minfunc/precondDiag.m000644 000765 000024 00000000052 12701705601 022064 0ustar00carlstaff000000 000000 function [y] = precondDiag(r,D) y = D.*r;gpml-matlab-v4.0-2016-10-19/util/minfunc/isLegal.m000644 000765 000024 00000000153 12701705601 021227 0ustar00carlstaff000000 000000 function [legal] = isLegal(v) legal = sum(any(imag(v(:))))==0 & sum(isnan(v(:)))==0 & sum(isinf(v(:)))==0;gpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/000755 000765 000024 00000000000 12777177045 021310 5ustar00carlstaff000000 000000 gpml-matlab-v4.0-2016-10-19/util/minfunc/conjGrad.m000644 000765 000024 00000003536 12701705601 021406 0ustar00carlstaff000000 000000 function [x,k,res,negCurv] = cg(A,b,optTol,maxIter,verbose,precFunc,precArgs,matrixVectFunc,matrixVectArgs) % [x,k,res,negCurv] = % cg(A,b,optTol,maxIter,verbose,precFunc,precArgs,matrixVectFunc,matrixVect % Args) % Linear Conjugate Gradient, where optionally we use % - preconditioner on vector v with precFunc(v,precArgs{:}) % - matrix multipled by vector with matrixVectFunc(v,matrixVectArgs{:}) if nargin <= 4 verbose = 0; end x = zeros(size(b)); r = -b; % Apply preconditioner (if supplied) if nargin >= 7 && ~isempty(precFunc) y = precFunc(r,precArgs{:}); else y = r; end ry = r'*y; p = -y; k = 0; res = norm(r); done = 0; negCurv = []; while res > optTol & k < maxIter & ~done % Compute Matrix-vector product if nargin >= 9 Ap = matrixVectFunc(p,matrixVectArgs{:}); else Ap = A*p; end pAp = p'*Ap; % Check for negative Curvature if pAp <= 1e-16 if verbose fprintf('Negative Curvature Detected!\n'); end if nargout == 4 if pAp < 0 negCurv = p; return end end if k == 0 if verbose fprintf('First-Iter, Proceeding...\n'); end done = 1; else if verbose fprintf('Stopping\n'); end break; end end % Conjugate Gradient alpha = ry/(pAp); x = x + alpha*p; r = r + alpha*Ap; % If supplied, apply preconditioner if nargin >= 7 && ~isempty(precFunc) y = precFunc(r,precArgs{:}); else y = r; end ry_new = r'*y; beta = ry_new/ry; p = -y + beta*p; k = k + 1; % Update variables ry = ry_new; res = norm(r); end end gpml-matlab-v4.0-2016-10-19/util/minfunc/lbfgsAdd.m000644 000765 000024 00000001247 12701705601 021362 0ustar00carlstaff000000 000000 function [S,Y,YS,lbfgs_start,lbfgs_end,Hdiag,skipped] = lbfgsAdd(y,s,S,Y,YS,lbfgs_start,lbfgs_end,Hdiag,useMex) ys = y'*s; skipped = 0; corrections = size(S,2); if ys > 1e-10 if lbfgs_end < corrections lbfgs_end = lbfgs_end+1; if lbfgs_start ~= 1 if lbfgs_start == corrections lbfgs_start = 1; else lbfgs_start = lbfgs_start+1; end end else lbfgs_start = min(2,corrections); lbfgs_end = 1; end if useMex lbfgsAddC(y,s,Y,S,ys,int32(lbfgs_end)); else S(:,lbfgs_end) = s; Y(:,lbfgs_end) = y; end YS(lbfgs_end) = ys; % Update scale of initial Hessian approximation Hdiag = ys/(y'*y); else skipped = 1; endgpml-matlab-v4.0-2016-10-19/util/minfunc/mchol.m000644 000765 000024 00000002434 12701705601 020755 0ustar00carlstaff000000 000000 function [l,d,perm] = mchol(A,mu) % [l,d,perm] = mchol(A,mu) % Compute the Gill-Murray modified LDL factorization of A, if nargin < 2 mu = 1e-12; end n = size(A,1); l = eye(n); d = zeros(n,1); perm = 1:n; for i = 1:n c(i,i) = A(i,i); end % Compute modification parameters gamma = max(abs(diag(A))); xi = max(max(abs(setdiag(A,0)))); delta = mu*max(gamma+xi,1); if n > 1 beta = sqrt(max([gamma xi/sqrt(n^2-1) mu])); else beta = sqrt(max([gamma mu])); end for j = 1:n % Find q that results in Best Permutation with j [maxVal maxPos] = max(abs(diag(c(j:end,j:end)))); q = maxPos+j-1; % Permute d,c,l,a d([j q]) = d([q j]); perm([j q]) = perm([q j]); c([j q],:) = c([q j],:); c(:,[j q]) = c(:,[q j]); l([j q],:) = l([q j],:); l(:,[j q]) = l(:,[q j]); A([j q],:) = A([q j],:); A(:,[j q]) = A(:,[q j]); for s = 1:j-1 l(j,s) = c(j,s)/d(s); end for i = j+1:n c(i,j) = A(i,j) - sum(l(j,1:j-1).*c(i,1:j-1)); end theta = 0; if j < n theta = max(abs(c(j+1:n,j))); end d(j) = max([abs(c(j,j)) (theta/beta)^2 delta]); if j < n for i = j+1:n c(i,i) = c(i,i) - (c(i,j)^2)/d(j); end end endgpml-matlab-v4.0-2016-10-19/util/minfunc/ArmijoBacktrack.m000644 000765 000024 00000010665 12701705601 022707 0ustar00carlstaff000000 000000 function [t,x_new,f_new,g_new,funEvals,H] = ArmijoBacktrack(... x,t,d,f,fr,g,gtd,c1,LS_interp,LS_multi,progTol,debug,doPlot,saveHessianComp,funObj,varargin) % [t,x_new,f_new,g_new,funEvals,H] = ArmijoBacktrack(... % x,t,d,f,fr,g,gtd,c1,LS_interp,LS_multi,progTol,debug,doPlot,saveHessianComp,funObj,varargin) % % Backtracking linesearch to satisfy Armijo condition % % Inputs: % x: starting location % t: initial step size % d: descent direction % f: function value at starting location % fr: reference function value (usually funObj(x)) % gtd: directional derivative at starting location % c1: sufficient decrease parameter % debug: display debugging information % LS_interp: type of interpolation % progTol: minimum allowable step length % doPlot: do a graphical display of interpolation % funObj: objective function % varargin: parameters of objective function % % Outputs: % t: step length % f_new: function value at x+t*d % g_new: gradient value at x+t*d % funEvals: number function evaluations performed by line search % H: Hessian at initial guess (only computed if requested) % % recet change: LS changed to LS_interp and LS_multi % Evaluate the Objective and Gradient at the Initial Step if nargout == 6 [f_new,g_new,H] = funObj(x + t*d,varargin{:}); else [f_new,g_new] = funObj(x+t*d,varargin{:}); end funEvals = 1; while f_new > fr + c1*t*gtd || ~isLegal(f_new) temp = t; if LS_interp == 0 || ~isLegal(f_new) % Ignore value of new point if debug fprintf('Fixed BT\n'); end t = 0.5*t; elseif LS_interp == 1 || ~isLegal(g_new) % Use function value at new point, but not its derivative if funEvals < 2 || LS_multi == 0 || ~isLegal(f_prev) % Backtracking w/ quadratic interpolation based on two points if debug fprintf('Quad BT\n'); end t = polyinterp([0 f gtd; t f_new sqrt(-1)],doPlot,0,t); else % Backtracking w/ cubic interpolation based on three points if debug fprintf('Cubic BT\n'); end t = polyinterp([0 f gtd; t f_new sqrt(-1); t_prev f_prev sqrt(-1)],doPlot,0,t); end else % Use function value and derivative at new point if funEvals < 2 || LS_multi == 0 || ~isLegal(f_prev) % Backtracking w/ cubic interpolation w/ derivative if debug fprintf('Grad-Cubic BT\n'); end t = polyinterp([0 f gtd; t f_new g_new'*d],doPlot,0,t); elseif ~isLegal(g_prev) % Backtracking w/ quartic interpolation 3 points and derivative % of two if debug fprintf('Grad-Quartic BT\n'); end t = polyinterp([0 f gtd; t f_new g_new'*d; t_prev f_prev sqrt(-1)],doPlot,0,t); else % Backtracking w/ quintic interpolation of 3 points and derivative % of two if debug fprintf('Grad-Quintic BT\n'); end t = polyinterp([0 f gtd; t f_new g_new'*d; t_prev f_prev g_prev'*d],doPlot,0,t); end end % Adjust if change in t is too small/large if t < temp*1e-3 if debug fprintf('Interpolated Value Too Small, Adjusting\n'); end t = temp*1e-3; elseif t > temp*0.6 if debug fprintf('Interpolated Value Too Large, Adjusting\n'); end t = temp*0.6; end % Store old point if doing three-point interpolation if LS_multi f_prev = f_new; t_prev = temp; if LS_interp == 2 g_prev = g_new; end end if ~saveHessianComp && nargout == 6 [f_new,g_new,H] = funObj(x + t*d,varargin{:}); else [f_new,g_new] = funObj(x + t*d,varargin{:}); end funEvals = funEvals+1; % Check whether step size has become too small if max(abs(t*d)) <= progTol if debug fprintf('Backtracking Line Search Failed\n'); end t = 0; f_new = f; g_new = g; break; end end % Evaluate Hessian at new point if nargout == 6 && funEvals > 1 && saveHessianComp [f_new,g_new,H] = funObj(x + t*d,varargin{:}); funEvals = funEvals+1; end x_new = x + t*d; end gpml-matlab-v4.0-2016-10-19/util/minfunc/precondTriuDiag.m000644 000765 000024 00000000074 12701705601 022734 0ustar00carlstaff000000 000000 function [y] = precondUpper(r,U,D) y = U \ (D .* (U' \ r));gpml-matlab-v4.0-2016-10-19/util/minfunc/WolfeLineSearch.m000644 000765 000024 00000024536 12701705601 022674 0ustar00carlstaff000000 000000 function [t,f_new,g_new,funEvals,H] = WolfeLineSearch(... x,t,d,f,g,gtd,c1,c2,LS_interp,LS_multi,maxLS,progTol,debug,doPlot,saveHessianComp,funObj,varargin) % % Bracketing Line Search to Satisfy Wolfe Conditions % % Inputs: % x: starting location % t: initial step size % d: descent direction % f: function value at starting location % g: gradient at starting location % gtd: directional derivative at starting location % c1: sufficient decrease parameter % c2: curvature parameter % debug: display debugging information % LS_interp: type of interpolation % maxLS: maximum number of iterations % progTol: minimum allowable step length % doPlot: do a graphical display of interpolation % funObj: objective function % varargin: parameters of objective function % % Outputs: % t: step length % f_new: function value at x+t*d % g_new: gradient value at x+t*d % funEvals: number function evaluations performed by line search % H: Hessian at initial guess (only computed if requested % Evaluate the Objective and Gradient at the Initial Step if nargout == 5 [f_new,g_new,H] = funObj(x + t*d,varargin{:}); else [f_new,g_new] = funObj(x+t*d,varargin{:}); end funEvals = 1; gtd_new = g_new'*d; % Bracket an Interval containing a point satisfying the % Wolfe criteria LSiter = 0; t_prev = 0; f_prev = f; g_prev = g; gtd_prev = gtd; nrmD = max(abs(d)); done = 0; while LSiter < maxLS %% Bracketing Phase if ~isLegal(f_new) || ~isLegal(g_new) if debug fprintf('Extrapolated into illegal region, switching to Armijo line-search\n'); end t = (t + t_prev)/2; % Do Armijo if nargout == 5 [t,x_new,f_new,g_new,armijoFunEvals,H] = ArmijoBacktrack(... x,t,d,f,f,g,gtd,c1,LS_interp,LS_multi,progTol,debug,doPlot,saveHessianComp,... funObj,varargin{:}); else [t,x_new,f_new,g_new,armijoFunEvals] = ArmijoBacktrack(... x,t,d,f,f,g,gtd,c1,LS_interp,LS_multi,progTol,debug,doPlot,saveHessianComp,... funObj,varargin{:}); end funEvals = funEvals + armijoFunEvals; return; end if f_new > f + c1*t*gtd || (LSiter > 1 && f_new >= f_prev) bracket = [t_prev t]; bracketFval = [f_prev f_new]; bracketGval = [g_prev g_new]; break; elseif abs(gtd_new) <= -c2*gtd bracket = t; bracketFval = f_new; bracketGval = g_new; done = 1; break; elseif gtd_new >= 0 bracket = [t_prev t]; bracketFval = [f_prev f_new]; bracketGval = [g_prev g_new]; break; end temp = t_prev; t_prev = t; minStep = t + 0.01*(t-temp); maxStep = t*10; if LS_interp <= 1 if debug fprintf('Extending Braket\n'); end t = maxStep; elseif LS_interp == 2 if debug fprintf('Cubic Extrapolation\n'); end t = polyinterp([temp f_prev gtd_prev; t f_new gtd_new],doPlot,minStep,maxStep); elseif LS_interp == 3 t = mixedExtrap(temp,f_prev,gtd_prev,t,f_new,gtd_new,minStep,maxStep,debug,doPlot); end f_prev = f_new; g_prev = g_new; gtd_prev = gtd_new; if ~saveHessianComp && nargout == 5 [f_new,g_new,H] = funObj(x + t*d,varargin{:}); else [f_new,g_new] = funObj(x + t*d,varargin{:}); end funEvals = funEvals + 1; gtd_new = g_new'*d; LSiter = LSiter+1; end if LSiter == maxLS bracket = [0 t]; bracketFval = [f f_new]; bracketGval = [g g_new]; end %% Zoom Phase % We now either have a point satisfying the criteria, or a bracket % surrounding a point satisfying the criteria % Refine the bracket until we find a point satisfying the criteria insufProgress = 0; Tpos = 2; LOposRemoved = 0; while ~done && LSiter < maxLS % Find High and Low Points in bracket [f_LO LOpos] = min(bracketFval); HIpos = -LOpos + 3; % Compute new trial value if LS_interp <= 1 || ~isLegal(bracketFval) || ~isLegal(bracketGval) if debug fprintf('Bisecting\n'); end t = mean(bracket); elseif LS_interp == 2 if debug fprintf('Grad-Cubic Interpolation\n'); end t = polyinterp([bracket(1) bracketFval(1) bracketGval(:,1)'*d bracket(2) bracketFval(2) bracketGval(:,2)'*d],doPlot); else % Mixed Case %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% nonTpos = -Tpos+3; if LOposRemoved == 0 oldLOval = bracket(nonTpos); oldLOFval = bracketFval(nonTpos); oldLOGval = bracketGval(:,nonTpos); end t = mixedInterp(bracket,bracketFval,bracketGval,d,Tpos,oldLOval,oldLOFval,oldLOGval,debug,doPlot); end % Test that we are making sufficient progress if min(max(bracket)-t,t-min(bracket))/(max(bracket)-min(bracket)) < 0.1 if debug fprintf('Interpolation close to boundary'); end if insufProgress || t>=max(bracket) || t <= min(bracket) if debug fprintf(', Evaluating at 0.1 away from boundary\n'); end if abs(t-max(bracket)) < abs(t-min(bracket)) t = max(bracket)-0.1*(max(bracket)-min(bracket)); else t = min(bracket)+0.1*(max(bracket)-min(bracket)); end insufProgress = 0; else if debug fprintf('\n'); end insufProgress = 1; end else insufProgress = 0; end % Evaluate new point if ~saveHessianComp && nargout == 5 [f_new,g_new,H] = funObj(x + t*d,varargin{:}); else [f_new,g_new] = funObj(x + t*d,varargin{:}); end funEvals = funEvals + 1; gtd_new = g_new'*d; LSiter = LSiter+1; armijo = f_new < f + c1*t*gtd; if ~armijo || f_new >= f_LO % Armijo condition not satisfied or not lower than lowest % point bracket(HIpos) = t; bracketFval(HIpos) = f_new; bracketGval(:,HIpos) = g_new; Tpos = HIpos; else if abs(gtd_new) <= - c2*gtd % Wolfe conditions satisfied done = 1; elseif gtd_new*(bracket(HIpos)-bracket(LOpos)) >= 0 % Old HI becomes new LO bracket(HIpos) = bracket(LOpos); bracketFval(HIpos) = bracketFval(LOpos); bracketGval(:,HIpos) = bracketGval(:,LOpos); if LS_interp == 3 if debug fprintf('LO Pos is being removed!\n'); end LOposRemoved = 1; oldLOval = bracket(LOpos); oldLOFval = bracketFval(LOpos); oldLOGval = bracketGval(:,LOpos); end end % New point becomes new LO bracket(LOpos) = t; bracketFval(LOpos) = f_new; bracketGval(:,LOpos) = g_new; Tpos = LOpos; end if ~done && abs(bracket(1)-bracket(2))*nrmD < progTol if debug fprintf('Line-search bracket has been reduced below progTol\n'); end break; end end %% if LSiter == maxLS if debug fprintf('Line Search Exceeded Maximum Line Search Iterations\n'); end end [f_LO LOpos] = min(bracketFval); t = bracket(LOpos); f_new = bracketFval(LOpos); g_new = bracketGval(:,LOpos); % Evaluate Hessian at new point if nargout == 5 && funEvals > 1 && saveHessianComp [f_new,g_new,H] = funObj(x + t*d,varargin{:}); funEvals = funEvals + 1; end end %% function [t] = mixedExtrap(x0,f0,g0,x1,f1,g1,minStep,maxStep,debug,doPlot); alpha_c = polyinterp([x0 f0 g0; x1 f1 g1],doPlot,minStep,maxStep); alpha_s = polyinterp([x0 f0 g0; x1 sqrt(-1) g1],doPlot,minStep,maxStep); if alpha_c > minStep && abs(alpha_c - x1) < abs(alpha_s - x1) if debug fprintf('Cubic Extrapolation\n'); end t = alpha_c; else if debug fprintf('Secant Extrapolation\n'); end t = alpha_s; end end %% function [t] = mixedInterp(bracket,bracketFval,bracketGval,d,Tpos,oldLOval,oldLOFval,oldLOGval,debug,doPlot); % Mixed Case %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% nonTpos = -Tpos+3; gtdT = bracketGval(:,Tpos)'*d; gtdNonT = bracketGval(:,nonTpos)'*d; oldLOgtd = oldLOGval'*d; if bracketFval(Tpos) > oldLOFval alpha_c = polyinterp([oldLOval oldLOFval oldLOgtd bracket(Tpos) bracketFval(Tpos) gtdT],doPlot); alpha_q = polyinterp([oldLOval oldLOFval oldLOgtd bracket(Tpos) bracketFval(Tpos) sqrt(-1)],doPlot); if abs(alpha_c - oldLOval) < abs(alpha_q - oldLOval) if debug fprintf('Cubic Interpolation\n'); end t = alpha_c; else if debug fprintf('Mixed Quad/Cubic Interpolation\n'); end t = (alpha_q + alpha_c)/2; end elseif gtdT'*oldLOgtd < 0 alpha_c = polyinterp([oldLOval oldLOFval oldLOgtd bracket(Tpos) bracketFval(Tpos) gtdT],doPlot); alpha_s = polyinterp([oldLOval oldLOFval oldLOgtd bracket(Tpos) sqrt(-1) gtdT],doPlot); if abs(alpha_c - bracket(Tpos)) >= abs(alpha_s - bracket(Tpos)) if debug fprintf('Cubic Interpolation\n'); end t = alpha_c; else if debug fprintf('Quad Interpolation\n'); end t = alpha_s; end elseif abs(gtdT) <= abs(oldLOgtd) alpha_c = polyinterp([oldLOval oldLOFval oldLOgtd bracket(Tpos) bracketFval(Tpos) gtdT],... doPlot,min(bracket),max(bracket)); alpha_s = polyinterp([oldLOval sqrt(-1) oldLOgtd bracket(Tpos) bracketFval(Tpos) gtdT],... doPlot,min(bracket),max(bracket)); if alpha_c > min(bracket) && alpha_c < max(bracket) if abs(alpha_c - bracket(Tpos)) < abs(alpha_s - bracket(Tpos)) if debug fprintf('Bounded Cubic Extrapolation\n'); end t = alpha_c; else if debug fprintf('Bounded Secant Extrapolation\n'); end t = alpha_s; end else if debug fprintf('Bounded Secant Extrapolation\n'); end t = alpha_s; end if bracket(Tpos) > oldLOval t = min(bracket(Tpos) + 0.66*(bracket(nonTpos) - bracket(Tpos)),t); else t = max(bracket(Tpos) + 0.66*(bracket(nonTpos) - bracket(Tpos)),t); end else t = polyinterp([bracket(nonTpos) bracketFval(nonTpos) gtdNonT bracket(Tpos) bracketFval(Tpos) gtdT],doPlot); end endgpml-matlab-v4.0-2016-10-19/util/minfunc/polyinterp.m000644 000765 000024 00000010076 12701705601 022061 0ustar00carlstaff000000 000000 function [minPos,fmin] = polyinterp(points,doPlot,xminBound,xmaxBound) % function [minPos] = polyinterp(points,doPlot,xminBound,xmaxBound) % % Minimum of interpolating polynomial based on function and derivative % values % % It can also be used for extrapolation if {xmin,xmax} are outside % the domain of the points. % % Input: % points(pointNum,[x f g]) % doPlot: set to 1 to plot, default: 0 % xmin: min value that brackets minimum (default: min of points) % xmax: max value that brackets maximum (default: max of points) % % set f or g to sqrt(-1) if they are not known % the order of the polynomial is the number of known f and g values minus 1 if nargin < 2 doPlot = 0; end nPoints = size(points,1); order = sum(sum((imag(points(:,2:3))==0)))-1; xmin = min(points(:,1)); xmax = max(points(:,1)); % Compute Bounds of Interpolation Area if nargin < 3 xminBound = xmin; end if nargin < 4 xmaxBound = xmax; end % Code for most common case: % - cubic interpolation of 2 points % w/ function and derivative values for both if nPoints == 2 && order ==3 && doPlot == 0 % Solution in this case (where x2 is the farthest point): % d1 = g1 + g2 - 3*(f1-f2)/(x1-x2); % d2 = sqrt(d1^2 - g1*g2); % minPos = x2 - (x2 - x1)*((g2 + d2 - d1)/(g2 - g1 + 2*d2)); % t_new = min(max(minPos,x1),x2); [minVal minPos] = min(points(:,1)); notMinPos = -minPos+3; d1 = points(minPos,3) + points(notMinPos,3) - 3*(points(minPos,2)-points(notMinPos,2))/(points(minPos,1)-points(notMinPos,1)); d2 = sqrt(d1^2 - points(minPos,3)*points(notMinPos,3)); if isreal(d2) t = points(notMinPos,1) - (points(notMinPos,1) - points(minPos,1))*((points(notMinPos,3) + d2 - d1)/(points(notMinPos,3) - points(minPos,3) + 2*d2)); minPos = min(max(t,xminBound),xmaxBound); else minPos = (xmaxBound+xminBound)/2; end return; end % Constraints Based on available Function Values A = zeros(0,order+1); b = zeros(0,1); for i = 1:nPoints if imag(points(i,2))==0 constraint = zeros(1,order+1); for j = order:-1:0 constraint(order-j+1) = points(i,1)^j; end A = [A;constraint]; b = [b;points(i,2)]; end end % Constraints based on available Derivatives for i = 1:nPoints if isreal(points(i,3)) constraint = zeros(1,order+1); for j = 1:order constraint(j) = (order-j+1)*points(i,1)^(order-j); end A = [A;constraint]; b = [b;points(i,3)]; end end % Find interpolating polynomial [params,ignore] = linsolve(A,b); % Compute Critical Points dParams = zeros(order,1); for i = 1:length(params)-1 dParams(i) = params(i)*(order-i+1); end if any(isinf(dParams)) cp = [xminBound;xmaxBound;points(:,1)].'; else cp = [xminBound;xmaxBound;points(:,1);roots(dParams)].'; end % Test Critical Points fmin = inf; minPos = (xminBound+xmaxBound)/2; % Default to Bisection if no critical points valid for xCP = cp if imag(xCP)==0 && xCP >= xminBound && xCP <= xmaxBound fCP = polyval(params,xCP); if imag(fCP)==0 && fCP < fmin minPos = real(xCP); fmin = real(fCP); end end end % Plot Situation if doPlot clf; hold on; % Plot Points plot(points(:,1),points(:,2),'b*'); % Plot Derivatives for i = 1:nPoints if isreal(points(i,3)) m = points(i,3); b = points(i,2) - m*points(i,1); plot([points(i,1)-.05 points(i,1)+.05],... [(points(i,1)-.05)*m+b (points(i,1)+.05)*m+b],'c.-'); end end % Plot Function x = min(xmin,xminBound)-.1:(max(xmax,xmaxBound)+.1-min(xmin,xminBound)+.1)/100:max(xmax,xmaxBound)+.1; for i = 1:length(x) f(i) = polyval(params,x(i)); end plot(x,f,'y'); axis([x(1)-.1 x(end)+.1 min(f)-.1 max(f)+.1]); % Plot Minimum plot(minPos,fmin,'g+'); if doPlot == 1 pause(1); end endgpml-matlab-v4.0-2016-10-19/util/minfunc/precondTriu.m000644 000765 000024 00000000063 12701705601 022145 0ustar00carlstaff000000 000000 function [y] = precondUpper(r,U) y = U \ (U' \ r);gpml-matlab-v4.0-2016-10-19/util/minfunc/minFunc_processInputOptions.m000644 000765 000024 00000010007 12701705601 025377 0ustar00carlstaff000000 000000 function [verbose,verboseI,debug,doPlot,maxFunEvals,maxIter,optTol,progTol,method,... corrections,c1,c2,LS_init,cgSolve,qnUpdate,cgUpdate,initialHessType,... HessianModify,Fref,useComplex,numDiff,LS_saveHessianComp,... Damped,HvFunc,bbType,cycle,... HessianIter,outputFcn,useMex,useNegCurv,precFunc,... LS_type,LS_interp,LS_multi,DerivativeCheck] = ... minFunc_processInputOptions(o) % Constants SD = 0; CSD = 1; BB = 2; CG = 3; PCG = 4; LBFGS = 5; QNEWTON = 6; NEWTON0 = 7; NEWTON = 8; TENSOR = 9; verbose = 1; verboseI= 1; debug = 0; doPlot = 0; method = LBFGS; cgSolve = 0; o = toUpper(o); if isfield(o,'DISPLAY') switch(upper(o.DISPLAY)) case 0 verbose = 0; verboseI = 0; case 'FINAL' verboseI = 0; case 'OFF' verbose = 0; verboseI = 0; case 'NONE' verbose = 0; verboseI = 0; case 'FULL' debug = 1; case 'EXCESSIVE' debug = 1; doPlot = 1; end end DerivativeCheck = 0; if isfield(o,'DERIVATIVECHECK') switch(upper(o.DERIVATIVECHECK)) case 1 DerivativeCheck = 1; case 'ON' DerivativeCheck = 1; end end LS_init = 0; LS_type = 1; LS_interp = 2; LS_multi = 0; Fref = 1; Damped = 0; HessianIter = 1; c2 = 0.9; if isfield(o,'METHOD') m = upper(o.METHOD); switch(m) case 'TENSOR' method = TENSOR; case 'NEWTON' method = NEWTON; case 'MNEWTON' method = NEWTON; HessianIter = 5; case 'PNEWTON0' method = NEWTON0; cgSolve = 1; case 'NEWTON0' method = NEWTON0; case 'QNEWTON' method = QNEWTON; Damped = 1; case 'LBFGS' method = LBFGS; case 'BB' method = BB; LS_type = 0; Fref = 20; case 'PCG' method = PCG; c2 = 0.2; LS_init = 2; case 'SCG' method = CG; c2 = 0.2; LS_init = 4; case 'CG' method = CG; c2 = 0.2; LS_init = 2; case 'CSD' method = CSD; c2 = 0.2; Fref = 10; LS_init = 2; case 'SD' method = SD; LS_init = 2; end end maxFunEvals = getOpt(o,'MAXFUNEVALS',1000); maxIter = getOpt(o,'MAXITER',500); optTol = getOpt(o,'OPTTOL',1e-5); progTol = getOpt(o,'PROGTOL',1e-9); corrections = getOpt(o,'CORRECTIONS',100); corrections = getOpt(o,'CORR',corrections); c1 = getOpt(o,'C1',1e-4); c2 = getOpt(o,'C2',c2); LS_init = getOpt(o,'LS_INIT',LS_init); cgSolve = getOpt(o,'CGSOLVE',cgSolve); qnUpdate = getOpt(o,'QNUPDATE',3); cgUpdate = getOpt(o,'CGUPDATE',2); initialHessType = getOpt(o,'INITIALHESSTYPE',1); HessianModify = getOpt(o,'HESSIANMODIFY',0); Fref = getOpt(o,'FREF',Fref); useComplex = getOpt(o,'USECOMPLEX',0); numDiff = getOpt(o,'NUMDIFF',0); LS_saveHessianComp = getOpt(o,'LS_SAVEHESSIANCOMP',1); Damped = getOpt(o,'DAMPED',Damped); HvFunc = getOpt(o,'HVFUNC',[]); bbType = getOpt(o,'BBTYPE',0); cycle = getOpt(o,'CYCLE',3); HessianIter = getOpt(o,'HESSIANITER',HessianIter); outputFcn = getOpt(o,'OUTPUTFCN',[]); useMex = getOpt(o,'USEMEX',1); useNegCurv = getOpt(o,'USENEGCURV',1); precFunc = getOpt(o,'PRECFUNC',[]); LS_type = getOpt(o,'LS_type',LS_type); LS_interp = getOpt(o,'LS_interp',LS_interp); LS_multi = getOpt(o,'LS_multi',LS_multi); end function [v] = getOpt(options,opt,default) if isfield(options,opt) if ~isempty(getfield(options,opt)) v = getfield(options,opt); else v = default; end else v = default; end end function [o] = toUpper(o) if ~isempty(o) fn = fieldnames(o); for i = 1:length(fn) o = setfield(o,upper(fn{i}),getfield(o,fn{i})); end end endgpml-matlab-v4.0-2016-10-19/util/minfunc/mcholinc.m000644 000765 000024 00000001064 12701705601 021445 0ustar00carlstaff000000 000000 function [R,tau] = mcholinc(H,verbose) % Computes Cholesky of H+tau*I, for suitably large tau that matrix is pd p = size(H,1); beta = norm(H,'fro'); if min(diag(H)) > 1e-12 tau = 0; else if verbose fprintf('Small Value on Diagonal, Adjusting Hessian\n'); end tau = max(beta/2,1e-12); end while 1 [R,posDef] = chol(H+tau*eye(p)); if posDef == 0 break; else if verbose fprintf('Cholesky Failed, Adjusting Hessian\n'); end tau = max(2*tau,beta/2); end end gpml-matlab-v4.0-2016-10-19/util/minfunc/lbfgs.m000644 000765 000024 00000001634 12701705601 020751 0ustar00carlstaff000000 000000 function [d] = lbfgs(g,s,y,Hdiag) % BFGS Search Direction % % This function returns the (L-BFGS) approximate inverse Hessian, % multiplied by the gradient % % If you pass in all previous directions/sizes, it will be the same as full BFGS % If you truncate to the k most recent directions/sizes, it will be L-BFGS % % s - previous search directions (p by k) % y - previous step sizes (p by k) % g - gradient (p by 1) % Hdiag - value of initial Hessian diagonal elements (scalar) [p,k] = size(s); for i = 1:k ro(i,1) = 1/(y(:,i)'*s(:,i)); end q = zeros(p,k+1); r = zeros(p,k+1); al =zeros(k,1); be =zeros(k,1); q(:,k+1) = g; for i = k:-1:1 al(i) = ro(i)*s(:,i)'*q(:,i+1); q(:,i) = q(:,i+1)-al(i)*y(:,i); end % Multiply by Initial Hessian r(:,1) = Hdiag*q(:,1); for i = 1:k be(i) = ro(i)*y(:,i)'*r(:,i); r(:,i+1) = r(:,i) + s(:,i)*(al(i)-be(i)); end d=r(:,k+1);gpml-matlab-v4.0-2016-10-19/util/minfunc/dampedUpdate.m000644 000765 000024 00000001743 12701705601 022252 0ustar00carlstaff000000 000000 function [old_dirs,old_stps,Hdiag,Bcompact] = lbfgsUpdate(y,s,corrections,debug,old_dirs,old_stps,Hdiag) %B0 = eye(length(y))/Hdiag; S = old_dirs(:,2:end); Y = old_stps(:,2:end); k = size(Y,2); L = zeros(k); for j = 1:k for i = j+1:k L(i,j) = S(:,i)'*Y(:,j); end end D = diag(diag(S'*Y)); N = [S/Hdiag Y]; M = [S'*S/Hdiag L;L' -D]; ys = y'*s; Bs = s/Hdiag - N*(M\(N'*s)); % Product B*s sBs = s'*Bs; eta = .02; if ys < eta*sBs if debug fprintf('Damped Update\n'); end theta = min(max(0,((1-eta)*sBs)/(sBs - ys)),1); y = theta*y + (1-theta)*Bs; end numCorrections = size(old_dirs,2); if numCorrections < corrections % Full Update old_dirs(:,numCorrections+1) = s; old_stps(:,numCorrections+1) = y; else % Limited-Memory Update old_dirs = [old_dirs(:,2:corrections) s]; old_stps = [old_stps(:,2:corrections) y]; end % Update scale of initial Hessian approximation Hdiag = (y'*s)/(y'*y);gpml-matlab-v4.0-2016-10-19/util/minfunc/mex/000755 000765 000024 00000000000 12777177045 020305 5ustar00carlstaff000000 000000 gpml-matlab-v4.0-2016-10-19/util/minfunc/minFunc.m000644 000765 000024 00000123357 12701705601 021262 0ustar00carlstaff000000 000000 function [x,f,exitflag,output] = minFunc(funObj,x0,options,varargin) % [x,f,exitflag,output] = minFunc(funObj,x0,options,varargin) % % Unconstrained optimizer using a line search strategy % % Uses an interface very similar to fminunc % (it doesn't support all of the optimization toolbox options, % but supports many other options). % % It computes descent directions using one of ('Method'): % - 'sd': Steepest Descent % (no previous information used, not recommended) % - 'csd': Cyclic Steepest Descent % (uses previous step length for a fixed length cycle) % - 'bb': Barzilai and Borwein Gradient % (uses only previous step) % - 'cg': Non-Linear Conjugate Gradient % (uses only previous step and a vector beta) % - 'scg': Scaled Non-Linear Conjugate Gradient % (uses previous step and a vector beta, % and Hessian-vector products to initialize line search) % - 'pcg': Preconditionined Non-Linear Conjugate Gradient % (uses only previous step and a vector beta, preconditioned version) % - 'lbfgs': Quasi-Newton with Limited-Memory BFGS Updating % (default: uses a predetermined nunber of previous steps to form a % low-rank Hessian approximation) % - 'newton0': Hessian-Free Newton % (numerically computes Hessian-Vector products) % - 'pnewton0': Preconditioned Hessian-Free Newton % (numerically computes Hessian-Vector products, preconditioned % version) % - 'qnewton': Quasi-Newton Hessian approximation % (uses dense Hessian approximation) % - 'mnewton': Newton's method with Hessian calculation after every % user-specified number of iterations % (needs user-supplied Hessian matrix) % - 'newton': Newton's method with Hessian calculation every iteration % (needs user-supplied Hessian matrix) % - 'tensor': Tensor % (needs user-supplied Hessian matrix and Tensor of 3rd partial derivatives) % % Several line search strategies are available for finding a step length satisfying % the termination criteria ('LS_type') % - 0 : A backtracking line-search based on the Armijo condition (default for 'bb') % - 1 : A bracekting line-search based on the strong Wolfe conditions (default for all other methods) % - 2 : The line-search from the Matlab Optimization Toolbox (requires Matlab's linesearch.m to be added to the path) % % For the Armijo line-search, several interpolation strategies are available ('LS_interp'): % - 0 : Step size halving % - 1 : Polynomial interpolation using new function values % - 2 : Polynomial interpolation using new function and gradient values (default) % % When (LS_interp = 1), the default setting of (LS_multi = 0) uses quadratic interpolation, % while if (LS_multi = 1) it uses cubic interpolation if more than one point are available. % % When (LS_interp = 2), the default setting of (LS_multi = 0) uses cubic interpolation, % while if (LS_multi = 1) it uses quartic or quintic interpolation if more than one point are available % % To use the non-monotonic Armijo condition, set the 'Fref' value to the number of previous function values to store % % For the Wolfe line-search, these interpolation strategies are available ('LS_interp'): % - 0 : Step Size Doubling and Bisection % - 1 : Cubic interpolation/extrapolation using new function and gradient values (default) % - 2 : Mixed quadratic/cubic interpolation/extrapolation % % Several strategies for choosing the initial step size are avaiable ('LS_init'): % - 0: Always try an initial step length of 1 (default for all except 'sd' and 'cg') % (t = 1) % - 1: Use a step similar to the previous step % (t = t_old*min(2,g'd/g_old'd_old)) % - 2: Quadratic Initialization using previous function value and new % function value/gradient (use this if steps tend to be very long, default for 'sd' and 'cg') % (t = min(1,2*(f-f_old)/g)) % - 3: The minimum between 1 and twice the previous step length % (t = min(1,2*t) % - 4: The scaled conjugate gradient step length (may accelerate % conjugate gradient methods, but requires a Hessian-vector product, default for 'scg') % (t = g'd/d'Hd) % % Inputs: % funObj - is a function handle % x0 - is a starting vector; % options - is a struct containing parameters (defaults are used for non-existent or blank fields) % varargin{:} - all other arguments are passed as additional arguments to funObj % % Outputs: % x is the minimum value found % f is the function value at the minimum found % exitflag returns an exit condition % output returns a structure with other information % % Supported Input Options % Display - Level of display [ off | final | (iter) | full | excessive ] % MaxFunEvals - Maximum number of function evaluations allowed (1000) % MaxIter - Maximum number of iterations allowed (500) % optTol - Termination tolerance on the first-order optimality (1e-5) % progTol - Termination tolerance on progress in terms of function/parameter changes (1e-9) % Method - [ sd | csd | bb | cg | scg | pcg | {lbfgs} | newton0 | pnewton0 | % qnewton | mnewton | newton | tensor ] % c1 - Sufficient Decrease for Armijo condition (1e-4) % c2 - Curvature Decrease for Wolfe conditions (.2 for cg methods, .9 otherwise) % LS_init - Line Search Initialization - see above (2 for cg/sd, 4 for scg, 0 otherwise) % LS - Line Search type - see above (2 for bb, 4 otherwise) % Fref - Setting this to a positive integer greater than 1 % will use non-monotone Armijo objective in the line search. % (20 for bb, 10 for csd, 1 for all others) % numDiff - [ 0 | 1 | 2] compute derivatives using user-supplied function (0), % numerically user forward-differencing (1), or numerically using central-differencing (2) % (default: 0) % (this option has a different effect for 'newton', see below) % useComplex - if 1, use complex differentials if computing numerical derivatives % to get very accurate values (default: 0) % DerivativeCheck - if 'on', computes derivatives numerically at initial % point and compares to user-supplied derivative (default: 'off') % outputFcn - function to run after each iteration (default: []). It % should have the following interface: % outputFcn(x,iterationType,i,funEvals,f,t,gtd,g,d,optCond,varargin{:}); % useMex - where applicable, use mex files to speed things up (default: 1) % % Method-specific input options: % newton: % HessianModify - type of Hessian modification for direct solvers to % use if the Hessian is not positive definite (default: 0) % 0: Minimum Euclidean norm s.t. eigenvalues sufficiently large % (requires eigenvalues on iterations where matrix is not pd) % 1: Start with (1/2)*||A||_F and increment until Cholesky succeeds % (an approximation to method 0, does not require eigenvalues) % 2: Modified LDL factorization % (only 1 generalized Cholesky factorization done and no eigenvalues required) % 3: Modified Spectral Decomposition % (requires eigenvalues) % 4: Modified Symmetric Indefinite Factorization % 5: Uses the eigenvector of the smallest eigenvalue as negative % curvature direction % cgSolve - use conjugate gradient instead of direct solver (default: 0) % 0: Direct Solver % 1: Conjugate Gradient % 2: Conjugate Gradient with Diagonal Preconditioner % 3: Conjugate Gradient with LBFGS Preconditioner % x: Conjugate Graident with Symmetric Successive Over Relaxation % Preconditioner with parameter x % (where x is a real number in the range [0,2]) % x: Conjugate Gradient with Incomplete Cholesky Preconditioner % with drop tolerance -x % (where x is a real negative number) % numDiff - compute Hessian numerically % (default: 0, done with complex differentials if useComplex = 1) % LS_saveHessiancomp - when on, only computes the Hessian at the % first and last iteration of the line search (default: 1) % mnewton: % HessianIter - number of iterations to use same Hessian (default: 5) % qnewton: % initialHessType - scale initial Hessian approximation (default: 1) % qnUpdate - type of quasi-Newton update (default: 3): % 0: BFGS % 1: SR1 (when it is positive-definite, otherwise BFGS) % 2: Hoshino % 3: Self-Scaling BFGS % 4: Oren's Self-Scaling Variable Metric method % 5: McCormick-Huang asymmetric update % Damped - use damped BFGS update (default: 1) % newton0/pnewton0: % HvFunc - user-supplied function that returns Hessian-vector products % (by default, these are computed numerically using autoHv) % HvFunc should have the following interface: HvFunc(v,x,varargin{:}) % useComplex - use a complex perturbation to get high accuracy % Hessian-vector products (default: 0) % (the increased accuracy can make the method much more efficient, % but gradient code must properly support complex inputs) % useNegCurv - a negative curvature direction is used as the descent % direction if one is encountered during the cg iterations % (default: 1) % precFunc (for pnewton0 only) - user-supplied preconditioner % (by default, an L-BFGS preconditioner is used) % precFunc should have the following interfact: % precFunc(v,x,varargin{:}) % lbfgs: % Corr - number of corrections to store in memory (default: 100) % (higher numbers converge faster but use more memory) % Damped - use damped update (default: 0) % cg/scg/pcg: % cgUpdate - type of update (default for cg/scg: 2, default for pcg: 1) % 0: Fletcher Reeves % 1: Polak-Ribiere % 2: Hestenes-Stiefel (not supported for pcg) % 3: Gilbert-Nocedal % HvFunc (for scg only)- user-supplied function that returns Hessian-vector % products % (by default, these are computed numerically using autoHv) % HvFunc should have the following interface: % HvFunc(v,x,varargin{:}) % precFunc (for pcg only) - user-supplied preconditioner % (by default, an L-BFGS preconditioner is used) % precFunc should have the following interface: % precFunc(v,x,varargin{:}) % bb: % bbType - type of bb step (default: 0) % 0: min_alpha ||delta_x - alpha delta_g||_2 % 1: min_alpha ||alpha delta_x - delta_g||_2 % 2: Conic BB % 3: Gradient method with retards % csd: % cycle - length of cycle (default: 3) % % Supported Output Options % iterations - number of iterations taken % funcCount - number of function evaluations % algorithm - algorithm used % firstorderopt - first-order optimality % message - exit message % trace.funccount - function evaluations after each iteration % trace.fval - function value after each iteration % % Author: Mark Schmidt (2005) % Web: http://www.di.ens.fr/~mschmidt/Software/minFunc.html % % Sources (in order of how much the source material contributes): % J. Nocedal and S.J. Wright. 1999. "Numerical Optimization". Springer Verlag. % R. Fletcher. 1987. "Practical Methods of Optimization". Wiley. % J. Demmel. 1997. "Applied Linear Algebra. SIAM. % R. Barret, M. Berry, T. Chan, J. Demmel, J. Dongarra, V. Eijkhout, R. % Pozo, C. Romine, and H. Van der Vost. 1994. "Templates for the Solution of % Linear Systems: Building Blocks for Iterative Methods". SIAM. % J. More and D. Thuente. "Line search algorithms with guaranteed % sufficient decrease". ACM Trans. Math. Softw. vol 20, 286-307, 1994. % M. Raydan. "The Barzilai and Borwein gradient method for the large % scale unconstrained minimization problem". SIAM J. Optim., 7, 26-33, % (1997). % "Mathematical Optimization". The Computational Science Education % Project. 1995. % C. Kelley. 1999. "Iterative Methods for Optimization". Frontiers in % Applied Mathematics. SIAM. if nargin < 3 options = []; end % Get Parameters [verbose,verboseI,debug,doPlot,maxFunEvals,maxIter,optTol,progTol,method,... corrections,c1,c2,LS_init,cgSolve,qnUpdate,cgUpdate,initialHessType,... HessianModify,Fref,useComplex,numDiff,LS_saveHessianComp,... Damped,HvFunc,bbType,cycle,... HessianIter,outputFcn,useMex,useNegCurv,precFunc,... LS_type,LS_interp,LS_multi,checkGrad] = ... minFunc_processInputOptions(options); % Constants SD = 0; CSD = 1; BB = 2; CG = 3; PCG = 4; LBFGS = 5; QNEWTON = 6; NEWTON0 = 7; NEWTON = 8; TENSOR = 9; % Initialize p = length(x0); d = zeros(p,1); x = x0; t = 1; % If necessary, form numerical differentiation functions funEvalMultiplier = 1; if useComplex numDiffType = 3; else numDiffType = numDiff; end if numDiff && method ~= TENSOR varargin(3:end+2) = varargin(1:end); varargin{1} = numDiffType; varargin{2} = funObj; if method ~= NEWTON if debug if useComplex fprintf('Using complex differentials for gradient computation\n'); else fprintf('Using finite differences for gradient computation\n'); end end funObj = @autoGrad; else if debug if useComplex fprintf('Using complex differentials for Hessian computation\n'); else fprintf('Using finite differences for Hessian computation\n'); end end funObj = @autoHess; end if method == NEWTON0 && useComplex == 1 if debug fprintf('Turning off the use of complex differentials for Hessian-vector products\n'); end useComplex = 0; end if useComplex funEvalMultiplier = p; elseif numDiff == 2 funEvalMultiplier = 2*p; else funEvalMultiplier = p+1; end end % Evaluate Initial Point if method < NEWTON [f,g] = funObj(x,varargin{:}); computeHessian = 0; else [f,g,H] = funObj(x,varargin{:}); computeHessian = 1; end funEvals = 1; % Derivative Check if checkGrad if numDiff fprintf('Can not do derivative checking when numDiff is 1\n'); pause end derivativeCheck(funObj,x,1,numDiffType,varargin{:}); % Checks gradient if computeHessian derivativeCheck(funObj,x,2,numDiffType,varargin{:}); end end % Output Log if verboseI fprintf('%10s %10s %15s %15s %15s\n','Iteration','FunEvals','Step Length','Function Val','Opt Cond'); end % Compute optimality of initial point optCond = max(abs(g)); if nargout > 3 % Initialize Trace trace.fval = f; trace.funcCount = funEvals; trace.optCond = optCond; end % Exit if initial point is optimal if optCond <= optTol exitflag=1; msg = 'Optimality Condition below optTol'; if verbose fprintf('%s\n',msg); end if nargout > 3 output = struct('iterations',0,'funcCount',1,... 'algorithm',method,'firstorderopt',max(abs(g)),'message',msg,'trace',trace); end return; end % Output Function if ~isempty(outputFcn) stop = outputFcn(x,'init',0,funEvals,f,[],[],g,[],max(abs(g)),varargin{:}); if stop exitflag=-1; msg = 'Stopped by output function'; if verbose fprintf('%s\n',msg); end if nargout > 3 output = struct('iterations',0,'funcCount',1,... 'algorithm',method,'firstorderopt',max(abs(g)),'message',msg,'trace',trace); end return; end end % Perform up to a maximum of 'maxIter' descent steps: for i = 1:maxIter % ****************** COMPUTE DESCENT DIRECTION ***************** switch method case SD % Steepest Descent d = -g; case CSD % Cyclic Steepest Descent if mod(i,cycle) == 1 % Use Steepest Descent alpha = 1; LS_init = 2; LS_type = 1; % Wolfe line search elseif mod(i,cycle) == mod(1+1,cycle) % Use Previous Step alpha = t; LS_init = 0; LS_type = 0; % Armijo line search end d = -alpha*g; case BB % Steepest Descent with Barzilai and Borwein Step Length if i == 1 d = -g; else y = g-g_old; s = t*d; if bbType == 0 yy = y'*y; alpha = (s'*y)/(yy); if alpha <= 1e-10 || alpha > 1e10 alpha = 1; end elseif bbType == 1 sy = s'*y; alpha = (s'*s)/sy; if alpha <= 1e-10 || alpha > 1e10 alpha = 1; end elseif bbType == 2 % Conic Interpolation ('Modified BB') sy = s'*y; ss = s'*s; alpha = ss/sy; if alpha <= 1e-10 || alpha > 1e10 alpha = 1; end alphaConic = ss/(6*(myF_old - f) + 4*g'*s + 2*g_old'*s); if alphaConic > .001*alpha && alphaConic < 1000*alpha alpha = alphaConic; end elseif bbType == 3 % Gradient Method with retards (bb type 1, random selection of previous step) sy = s'*y; alpha = (s'*s)/sy; if alpha <= 1e-10 || alpha > 1e10 alpha = 1; end v(1+mod(i-2,5)) = alpha; alpha = v(ceil(rand*length(v))); end d = -alpha*g; end g_old = g; myF_old = f; case CG % Non-Linear Conjugate Gradient if i == 1 d = -g; % Initially use steepest descent direction else gotgo = g_old'*g_old; if cgUpdate == 0 % Fletcher-Reeves beta = (g'*g)/(gotgo); elseif cgUpdate == 1 % Polak-Ribiere beta = (g'*(g-g_old)) /(gotgo); elseif cgUpdate == 2 % Hestenes-Stiefel beta = (g'*(g-g_old))/((g-g_old)'*d); else % Gilbert-Nocedal beta_FR = (g'*(g-g_old)) /(gotgo); beta_PR = (g'*g-g'*g_old)/(gotgo); beta = max(-beta_FR,min(beta_PR,beta_FR)); end d = -g + beta*d; % Restart if not a direction of sufficient descent if g'*d > -progTol if debug fprintf('Restarting CG\n'); end beta = 0; d = -g; end % Old restart rule: %if beta < 0 || abs(gtgo)/(gotgo) >= 0.1 || g'*d >= 0 end g_old = g; case PCG % Preconditioned Non-Linear Conjugate Gradient % Apply preconditioner to negative gradient if isempty(precFunc) % Use L-BFGS Preconditioner if i == 1 S = zeros(p,corrections); Y = zeros(p,corrections); YS = zeros(corrections,1); lbfgs_start = 1; lbfgs_end = 0; Hdiag = 1; s = -g; else [S,Y,YS,lbfgs_start,lbfgs_end,Hdiag,skipped] = lbfgsAdd(g-g_old,t*d,S,Y,YS,lbfgs_start,lbfgs_end,Hdiag,useMex); if debug && skipped fprintf('Skipped L-BFGS updated\n'); end if useMex s = lbfgsProdC(g,S,Y,YS,int32(lbfgs_start),int32(lbfgs_end),Hdiag); else s = lbfgsProd(g,S,Y,YS,lbfgs_start,lbfgs_end,Hdiag); end end else % User-supplied preconditioner s = precFunc(-g,x,varargin{:}); end if i == 1 d = s; else if cgUpdate == 0 % Preconditioned Fletcher-Reeves beta = (g'*s)/(g_old'*s_old); elseif cgUpdate < 3 % Preconditioned Polak-Ribiere beta = (g'*(s-s_old))/(g_old'*s_old); else % Preconditioned Gilbert-Nocedal beta_FR = (g'*s)/(g_old'*s_old); beta_PR = (g'*(s-s_old))/(g_old'*s_old); beta = max(-beta_FR,min(beta_PR,beta_FR)); end d = s + beta*d; if g'*d > -progTol if debug fprintf('Restarting CG\n'); end beta = 0; d = s; end end g_old = g; s_old = s; case LBFGS % L-BFGS % Update the direction and step sizes if Damped if i == 1 d = -g; % Initially use steepest descent direction old_dirs = zeros(length(g),0); old_stps = zeros(length(d),0); Hdiag = 1; else [old_dirs,old_stps,Hdiag] = dampedUpdate(g-g_old,t*d,corrections,debug,old_dirs,old_stps,Hdiag); if useMex d = lbfgsC(-g,old_dirs,old_stps,Hdiag); else d = lbfgs(-g,old_dirs,old_stps,Hdiag); end end else if i == 1 d = -g; % Initially use steepest descent direction S = zeros(p,corrections); Y = zeros(p,corrections); YS = zeros(corrections,1); lbfgs_start = 1; lbfgs_end = 0; Hdiag = 1; else [S,Y,YS,lbfgs_start,lbfgs_end,Hdiag,skipped] = lbfgsAdd(g-g_old,t*d,S,Y,YS,lbfgs_start,lbfgs_end,Hdiag,useMex); if debug && skipped fprintf('Skipped L-BFGS updated\n'); end if useMex d = lbfgsProdC(g,S,Y,YS,int32(lbfgs_start),int32(lbfgs_end),Hdiag); else d = lbfgsProd(g,S,Y,YS,lbfgs_start,lbfgs_end,Hdiag); end end end g_old = g; case QNEWTON % Use quasi-Newton Hessian approximation if i == 1 d = -g; else % Compute difference vectors y = g-g_old; s = t*d; if i == 2 % Make initial Hessian approximation if initialHessType == 0 % Identity if qnUpdate <= 1 R = eye(length(g)); else H = eye(length(g)); end else % Scaled Identity if debug fprintf('Scaling Initial Hessian Approximation\n'); end if qnUpdate <= 1 % Use Cholesky of Hessian approximation R = sqrt((y'*y)/(y'*s))*eye(length(g)); else % Use Inverse of Hessian approximation H = eye(length(g))*(y'*s)/(y'*y); end end end if qnUpdate == 0 % Use BFGS updates Bs = R'*(R*s); if Damped eta = .02; if y'*s < eta*s'*Bs if debug fprintf('Damped Update\n'); end theta = min(max(0,((1-eta)*s'*Bs)/(s'*Bs - y'*s)),1); y = theta*y + (1-theta)*Bs; end R = cholupdate(cholupdate(R,y/sqrt(y'*s)),Bs/sqrt(s'*Bs),'-'); else if y'*s > 1e-10 R = cholupdate(cholupdate(R,y/sqrt(y'*s)),Bs/sqrt(s'*Bs),'-'); else if debug fprintf('Skipping Update\n'); end end end elseif qnUpdate == 1 % Perform SR1 Update if it maintains positive-definiteness Bs = R'*(R*s); ymBs = y-Bs; if abs(s'*ymBs) >= norm(s)*norm(ymBs)*1e-8 && (s-((R\(R'\y))))'*y > 1e-10 R = cholupdate(R,-ymBs/sqrt(ymBs'*s),'-'); else if debug fprintf('SR1 not positive-definite, doing BFGS Update\n'); end if Damped eta = .02; if y'*s < eta*s'*Bs if debug fprintf('Damped Update\n'); end theta = min(max(0,((1-eta)*s'*Bs)/(s'*Bs - y'*s)),1); y = theta*y + (1-theta)*Bs; end R = cholupdate(cholupdate(R,y/sqrt(y'*s)),Bs/sqrt(s'*Bs),'-'); else if y'*s > 1e-10 R = cholupdate(cholupdate(R,y/sqrt(y'*s)),Bs/sqrt(s'*Bs),'-'); else if debug fprintf('Skipping Update\n'); end end end end elseif qnUpdate == 2 % Use Hoshino update v = sqrt(y'*H*y)*(s/(s'*y) - (H*y)/(y'*H*y)); phi = 1/(1 + (y'*H*y)/(s'*y)); H = H + (s*s')/(s'*y) - (H*y*y'*H)/(y'*H*y) + phi*v*v'; elseif qnUpdate == 3 % Self-Scaling BFGS update ys = y'*s; Hy = H*y; yHy = y'*Hy; gamma = ys/yHy; v = sqrt(yHy)*(s/ys - Hy/yHy); H = gamma*(H - Hy*Hy'/yHy + v*v') + (s*s')/ys; elseif qnUpdate == 4 % Oren's Self-Scaling Variable Metric update % Oren's method if (s'*y)/(y'*H*y) > 1 phi = 1; % BFGS omega = 0; elseif (s'*(H\s))/(s'*y) < 1 phi = 0; % DFP omega = 1; else phi = (s'*y)*(y'*H*y-s'*y)/((s'*(H\s))*(y'*H*y)-(s'*y)^2); omega = phi; end gamma = (1-omega)*(s'*y)/(y'*H*y) + omega*(s'*(H\s))/(s'*y); v = sqrt(y'*H*y)*(s/(s'*y) - (H*y)/(y'*H*y)); H = gamma*(H - (H*y*y'*H)/(y'*H*y) + phi*v*v') + (s*s')/(s'*y); elseif qnUpdate == 5 % McCormick-Huang asymmetric update theta = 1; phi = 0; psi = 1; omega = 0; t1 = s*(theta*s + phi*H'*y)'; t2 = (theta*s + phi*H'*y)'*y; t3 = H*y*(psi*s + omega*H'*y)'; t4 = (psi*s + omega*H'*y)'*y; H = H + t1/t2 - t3/t4; end if qnUpdate <= 1 d = -R\(R'\g); else d = -H*g; end end g_old = g; case NEWTON0 % Hessian-Free Newton cgMaxIter = min(p,maxFunEvals-funEvals); cgForce = min(0.5,sqrt(norm(g)))*norm(g); % Set-up preconditioner precondFunc = []; precondArgs = []; if cgSolve == 1 if isempty(precFunc) % Apply L-BFGS preconditioner if i == 1 S = zeros(p,corrections); Y = zeros(p,corrections); YS = zeros(corrections,1); lbfgs_start = 1; lbfgs_end = 0; Hdiag = 1; else [S,Y,YS,lbfgs_start,lbfgs_end,Hdiag,skipped] = lbfgsAdd(g-g_old,t*d,S,Y,YS,lbfgs_start,lbfgs_end,Hdiag,useMex); if debug && skipped fprintf('Skipped L-BFGS updated\n'); end if useMex precondFunc = @lbfgsProdC; else precondFunc = @lbfgsProd; end precondArgs = {S,Y,YS,int32(lbfgs_start),int32(lbfgs_end),Hdiag}; end g_old = g; else % Apply user-defined preconditioner precondFunc = precFunc; precondArgs = {x,varargin{:}}; end end % Solve Newton system using cg and hessian-vector products if isempty(HvFunc) % No user-supplied Hessian-vector function, % use automatic differentiation HvFun = @autoHv; HvArgs = {x,g,useComplex,funObj,varargin{:}}; else % Use user-supplid Hessian-vector function HvFun = HvFunc; HvArgs = {x,varargin{:}}; end if useNegCurv [d,cgIter,cgRes,negCurv] = conjGrad([],-g,cgForce,cgMaxIter,debug,precondFunc,precondArgs,HvFun,HvArgs); else [d,cgIter,cgRes] = conjGrad([],-g,cgForce,cgMaxIter,debug,precondFunc,precondArgs,HvFun,HvArgs); end funEvals = funEvals+cgIter; if debug fprintf('newtonCG stopped on iteration %d w/ residual %.5e\n',cgIter,cgRes); end if useNegCurv if ~isempty(negCurv) %if debug fprintf('Using negative curvature direction\n'); %end d = negCurv/norm(negCurv); d = d/sum(abs(g)); end end case NEWTON % Newton search direction if cgSolve == 0 if HessianModify == 0 % Attempt to perform a Cholesky factorization of the Hessian [R,posDef] = chol(H); % If the Cholesky factorization was successful, then the Hessian is % positive definite, solve the system if posDef == 0 d = -R\(R'\g); else % otherwise, adjust the Hessian to be positive definite based on the % minimum eigenvalue, and solve with QR % (expensive, we don't want to do this very much) if debug fprintf('Adjusting Hessian\n'); end H = H + eye(length(g)) * max(0,1e-12 - min(real(eig(H)))); d = -H\g; end elseif HessianModify == 1 % Modified Incomplete Cholesky R = mcholinc(H,debug); d = -R\(R'\g); elseif HessianModify == 2 % Modified Generalized Cholesky if useMex [L D perm] = mcholC(H); else [L D perm] = mchol(H); end d(perm) = -L' \ ((D.^-1).*(L \ g(perm))); elseif HessianModify == 3 % Modified Spectral Decomposition [V,D] = eig((H+H')/2); D = diag(D); D = max(abs(D),max(max(abs(D)),1)*1e-12); d = -V*((V'*g)./D); elseif HessianModify == 4 % Modified Symmetric Indefinite Factorization [L,D,perm] = ldl(H,'vector'); [blockPos junk] = find(triu(D,1)); for diagInd = setdiff(setdiff(1:p,blockPos),blockPos+1) if D(diagInd,diagInd) < 1e-12 D(diagInd,diagInd) = 1e-12; end end for blockInd = blockPos' block = D(blockInd:blockInd+1,blockInd:blockInd+1); block_a = block(1); block_b = block(2); block_d = block(4); lambda = (block_a+block_d)/2 - sqrt(4*block_b^2 + (block_a - block_d)^2)/2; D(blockInd:blockInd+1,blockInd:blockInd+1) = block+eye(2)*(lambda+1e-12); end d(perm) = -L' \ (D \ (L \ g(perm))); else % Take Newton step if Hessian is pd, % otherwise take a step with negative curvature [R,posDef] = chol(H); if posDef == 0 d = -R\(R'\g); else if debug fprintf('Taking Direction of Negative Curvature\n'); end [V,D] = eig(H); u = V(:,1); d = -sign(u'*g)*u; end end else % Solve with Conjugate Gradient cgMaxIter = p; cgForce = min(0.5,sqrt(norm(g)))*norm(g); % Select Preconditioner if cgSolve == 1 % No preconditioner precondFunc = []; precondArgs = []; elseif cgSolve == 2 % Diagonal preconditioner precDiag = diag(H); precDiag(precDiag < 1e-12) = 1e-12 - min(precDiag); precondFunc = @precondDiag; precondArgs = {precDiag.^-1}; elseif cgSolve == 3 % L-BFGS preconditioner if i == 1 old_dirs = zeros(length(g),0); old_stps = zeros(length(g),0); Hdiag = 1; else [old_dirs,old_stps,Hdiag] = lbfgsUpdate(g-g_old,t*d,corrections,debug,old_dirs,old_stps,Hdiag); end g_old = g; if useMex precondFunc = @lbfgsC; else precondFunc = @lbfgs; end precondArgs = {old_dirs,old_stps,Hdiag}; elseif cgSolve > 0 % Symmetric Successive Overelaxation Preconditioner omega = cgSolve; D = diag(H); D(D < 1e-12) = 1e-12 - min(D); precDiag = (omega/(2-omega))*D.^-1; precTriu = diag(D/omega) + triu(H,1); precondFunc = @precondTriuDiag; precondArgs = {precTriu,precDiag.^-1}; else % Incomplete Cholesky Preconditioner opts.droptol = -cgSolve; opts.rdiag = 1; R = cholinc(sparse(H),opts); if min(diag(R)) < 1e-12 R = cholinc(sparse(H + eye*(1e-12 - min(diag(R)))),opts); end precondFunc = @precondTriu; precondArgs = {R}; end % Run cg with the appropriate preconditioner if isempty(HvFunc) % No user-supplied Hessian-vector function [d,cgIter,cgRes] = conjGrad(H,-g,cgForce,cgMaxIter,debug,precondFunc,precondArgs); else % Use user-supplied Hessian-vector function [d,cgIter,cgRes] = conjGrad(H,-g,cgForce,cgMaxIter,debug,precondFunc,precondArgs,HvFunc,{x,varargin{:}}); end if debug fprintf('CG stopped after %d iterations w/ residual %.5e\n',cgIter,cgRes); %funEvals = funEvals + cgIter; end end case TENSOR % Tensor Method if numDiff % Compute 3rd-order Tensor Numerically [junk1 junk2 junk3 T] = autoTensor(x,numDiffType,funObj,varargin{:}); else % Use user-supplied 3rd-derivative Tensor [junk1 junk2 junk3 T] = funObj(x,varargin{:}); end options_sub.Method = 'newton'; options_sub.Display = 'none'; options_sub.progTol = progTol; options_sub.optTol = optTol; d = minFunc(@taylorModel,zeros(p,1),options_sub,f,g,H,T); if any(abs(d) > 1e5) || all(abs(d) < 1e-5) || g'*d > -progTol if debug fprintf('Using 2nd-Order Step\n'); end [V,D] = eig((H+H')/2); D = diag(D); D = max(abs(D),max(max(abs(D)),1)*1e-12); d = -V*((V'*g)./D); else if debug fprintf('Using 3rd-Order Step\n'); end end end if ~isLegal(d) fprintf('Step direction is illegal!\n'); pause; return end % ****************** COMPUTE STEP LENGTH ************************ % Directional Derivative gtd = g'*d; % Check that progress can be made along direction if gtd > -progTol exitflag=2; msg = 'Directional Derivative below progTol'; break; end % Select Initial Guess if i == 1 if method < NEWTON0 t = min(1,1/sum(abs(g))); else t = 1; end else if LS_init == 0 % Newton step t = 1; elseif LS_init == 1 % Close to previous step length t = t*min(2,(gtd_old)/(gtd)); elseif LS_init == 2 % Quadratic Initialization based on {f,g} and previous f t = min(1,2*(f-f_old)/(gtd)); elseif LS_init == 3 % Double previous step length t = min(1,t*2); elseif LS_init == 4 % Scaled step length if possible if isempty(HvFunc) % No user-supplied Hessian-vector function, % use automatic differentiation dHd = d'*autoHv(d,x,g,0,funObj,varargin{:}); else % Use user-supplid Hessian-vector function dHd = d'*HvFunc(d,x,varargin{:}); end funEvals = funEvals + 1; if dHd > 0 t = -gtd/(dHd); else t = min(1,2*(f-f_old)/(gtd)); end end if t <= 0 t = 1; end end f_old = f; gtd_old = gtd; % Compute reference fr if using non-monotone objective if Fref == 1 fr = f; else if i == 1 old_fvals = repmat(-inf,[Fref 1]); end if i <= Fref old_fvals(i) = f; else old_fvals = [old_fvals(2:end);f]; end fr = max(old_fvals); end computeHessian = 0; if method >= NEWTON if HessianIter == 1 computeHessian = 1; elseif i > 1 && mod(i-1,HessianIter) == 0 computeHessian = 1; end end % Line Search f_old = f; if LS_type == 0 % Use Armijo Bactracking % Perform Backtracking line search if computeHessian [t,x,f,g,LSfunEvals,H] = ArmijoBacktrack(x,t,d,f,fr,g,gtd,c1,LS_interp,LS_multi,progTol,debug,doPlot,LS_saveHessianComp,funObj,varargin{:}); else [t,x,f,g,LSfunEvals] = ArmijoBacktrack(x,t,d,f,fr,g,gtd,c1,LS_interp,LS_multi,progTol,debug,doPlot,1,funObj,varargin{:}); end funEvals = funEvals + LSfunEvals; elseif LS_type == 1 % Find Point satisfying Wolfe conditions if computeHessian [t,f,g,LSfunEvals,H] = WolfeLineSearch(x,t,d,f,g,gtd,c1,c2,LS_interp,LS_multi,25,progTol,debug,doPlot,LS_saveHessianComp,funObj,varargin{:}); else [t,f,g,LSfunEvals] = WolfeLineSearch(x,t,d,f,g,gtd,c1,c2,LS_interp,LS_multi,25,progTol,debug,doPlot,1,funObj,varargin{:}); end funEvals = funEvals + LSfunEvals; x = x + t*d; else % Use Matlab optim toolbox line search [t,f_new,fPrime_new,g_new,LSexitFlag,LSiter]=... lineSearch({'fungrad',[],funObj},x,p,1,p,d,f,gtd,t,c1,c2,-inf,maxFunEvals-funEvals,... progTol,[],[],[],varargin{:}); funEvals = funEvals + LSiter; if isempty(t) exitflag = -2; msg = 'Matlab LineSearch failed'; break; end if method >= NEWTON [f_new,g_new,H] = funObj(x + t*d,varargin{:}); funEvals = funEvals + 1; end x = x + t*d; f = f_new; g = g_new; end % Compute Optimality Condition optCond = max(abs(g)); % Output iteration information if verboseI fprintf('%10d %10d %15.5e %15.5e %15.5e\n',i,funEvals*funEvalMultiplier,t,f,optCond); end if nargout > 3 % Update Trace trace.fval(end+1,1) = f; trace.funcCount(end+1,1) = funEvals; trace.optCond(end+1,1) = optCond; end % Output Function if ~isempty(outputFcn) stop = outputFcn(x,'iter',i,funEvals,f,t,gtd,g,d,optCond,varargin{:}); if stop exitflag=-1; msg = 'Stopped by output function'; break; end end % Check Optimality Condition if optCond <= optTol exitflag=1; msg = 'Optimality Condition below optTol'; break; end % ******************* Check for lack of progress ******************* if max(abs(t*d)) <= progTol exitflag=2; msg = 'Step Size below progTol'; break; end if abs(f-f_old) < progTol exitflag=2; msg = 'Function Value changing by less than progTol'; break; end % ******** Check for going over iteration/evaluation limit ******************* if funEvals*funEvalMultiplier >= maxFunEvals exitflag = 0; msg = 'Reached Maximum Number of Function Evaluations'; break; end if i == maxIter exitflag = 0; msg='Reached Maximum Number of Iterations'; break; end end if verbose fprintf('%s\n',msg); end if nargout > 3 output = struct('iterations',i,'funcCount',funEvals*funEvalMultiplier,... 'algorithm',method,'firstorderopt',max(abs(g)),'message',msg,'trace',trace); end % Output Function if ~isempty(outputFcn) outputFcn(x,'done',i,funEvals,f,t,gtd,g,d,max(abs(g)),varargin{:}); end end gpml-matlab-v4.0-2016-10-19/util/minfunc/lbfgsProd.m000644 000765 000024 00000001270 12701705601 021572 0ustar00carlstaff000000 000000 function [d] = lbfgsProd(g,S,Y,YS,lbfgs_start,lbfgs_end,Hdiag) % BFGS Search Direction % % This function returns the (L-BFGS) approximate inverse Hessian, % multiplied by the negative gradient % Set up indexing [nVars,maxCorrections] = size(S); if lbfgs_start == 1 ind = 1:lbfgs_end; nCor = lbfgs_end-lbfgs_start+1; else ind = [lbfgs_start:maxCorrections 1:lbfgs_end]; nCor = maxCorrections; end al = zeros(nCor,1); be = zeros(nCor,1); d = -g; for j = 1:length(ind) i = ind(end-j+1); al(i) = (S(:,i)'*d)/YS(i); d = d-al(i)*Y(:,i); end % Multiply by Initial Hessian d = Hdiag*d; for i = ind be(i) = (Y(:,i)'*d)/YS(i); d = d + S(:,i)*(al(i)-be(i)); end gpml-matlab-v4.0-2016-10-19/util/minfunc/mex/mexAll_octave.m000644 000765 000024 00000001021 12777177045 023240 0ustar00carlstaff000000 000000 % minFunc printf('Compiling minFunc files (octave version)...\n'); ## working around the lack of an -outdir option in octave's mex function mexme(fn) cmd = sprintf("mkoctfile --mex --output ../compiled/%s.mex %s.c", fn, fn) ; [ status output ] = system(cmd) ; if status!=0 error("Executing command %s\n", cmd); else delete(sprintf("%s.o", fn)); printf("%s compiled\n", fn); endif endfunction mexme("mcholC"); mexme("lbfgsC"); mexme("lbfgsAddC"); mexme("lbfgsProdC"); printf("Done.\n")gpml-matlab-v4.0-2016-10-19/util/minfunc/mex/lbfgsAddC.c000644 000765 000024 00000001442 12701705601 022241 0ustar00carlstaff000000 000000 #include #include "mex.h" /* See lbfgsAdd.m for details */ /* This function will not exit gracefully on bad input! */ void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) { /* Variable Declarations */ double *s,*y,*S, *Y, ys; int i,j,nVars,lbfgs_end; /* Get Input Pointers */ y = mxGetPr(prhs[0]); s = mxGetPr(prhs[1]); Y = mxGetPr(prhs[2]); S = mxGetPr(prhs[3]); ys= mxGetScalar(prhs[4]); lbfgs_end = (int)mxGetScalar(prhs[5]); if (!mxIsClass(prhs[5],"int32")) mexErrMsgTxt("lbfgs_end must be int32"); /* Compute number of variables, maximum number of corrections */ nVars = mxGetDimensions(prhs[2])[0]; for(j=0;j #include "mex.h" /* See lbfgsProd.m for details */ /* This function will not exit gracefully on bad input! */ void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) { /* Variable Declarations */ double *S, *Y, *YS, *g, Hdiag, *d, *alpha, *beta; int i,j,nVars,nCor,maxCor,lbfgs_start,lbfgs_end; /* Get Input Pointers */ g = mxGetPr(prhs[0]); S = mxGetPr(prhs[1]); Y = mxGetPr(prhs[2]); YS= mxGetPr(prhs[3]); lbfgs_start = (int)mxGetScalar(prhs[4]); lbfgs_end = (int)mxGetScalar(prhs[5]); Hdiag = mxGetScalar(prhs[6]); if (!mxIsClass(prhs[4],"int32")||!mxIsClass(prhs[5],"int32")) mexErrMsgTxt("lbfgs_start and lbfgs_end must be int32"); /* Compute number of variables, maximum number of corrections */ nVars = mxGetDimensions(prhs[1])[0]; maxCor = mxGetDimensions(prhs[1])[1]; /* Compute number of corrections available */ if (lbfgs_start == 1) nCor = lbfgs_end-lbfgs_start+1; else nCor = maxCor; /* Allocate Memory for Local Variables */ alpha = mxCalloc(nCor,sizeof(double)); beta = mxCalloc(nCor,sizeof(double)); /* Set-up Output Vector */ plhs[0] = mxCreateDoubleMatrix(nVars,1,mxREAL); d = mxGetPr(plhs[0]); for(j=0;j= 0;i--) { alpha[i] = 0; for(j=0;j= lbfgs_start-1;i--) { alpha[i] = 0; for(j=0;j #include "mex.h" double mymax(double x, double y) { if (x > y) return x; else return y; } double absolute(double x) { if (x >= -x) return x; else return -x; } void permuteInt(int *x, int p, int q) { int temp; temp = x[p]; x[p] = x[q]; x[q] = temp; } void permute(double *x, int p, int q) { double temp; temp = x[p]; x[p] = x[q]; x[q] = temp; } void permuteRows(double *x, int p, int q,int n) { int i; double temp; for(i = 0; i < n; i++) { temp = x[p+i*n]; x[p+i*n] = x[q+i*n]; x[q+i*n] = temp; } } void permuteCols(double *x, int p, int q,int n) { int i; double temp; for(i = 0; i < n; i++) { temp = x[i+p*n]; x[i+p*n] = x[i+q*n]; x[i+q*n] = temp; } } void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) { int n,sizL[2],sizD[2],i,j,q,s, *P; double mu,gamma,xi,delta,beta,maxVal,theta, *c, *H, *L, *D, *A; /* Input */ H = mxGetPr(prhs[0]); if (nrhs == 1) { mu = 1e-12; } else { mu = mxGetScalar(prhs[1]); } /* Compute Sizes */ n = mxGetDimensions(prhs[0])[0]; /* Form Output */ sizL[0] = n; sizL[1] = n; plhs[0] = mxCreateNumericArray(2,sizL,mxDOUBLE_CLASS,mxREAL); L = mxGetPr(plhs[0]); sizD[0] = n; sizD[1] = 1; plhs[1] = mxCreateNumericArray(2,sizD,mxDOUBLE_CLASS,mxREAL); D = mxGetPr(plhs[1]); plhs[2] = mxCreateNumericArray(2,sizD,mxINT32_CLASS,mxREAL); P = (int*)mxGetData(plhs[2]); /* Initialize */ c = mxCalloc(n*n,sizeof(double)); A = mxCalloc(n*n,sizeof(double)); for (i = 0; i < n; i++) { P[i] = i; for (j = 0;j < n; j++) { A[i+n*j] = H[i+n*j]; } } gamma = 0; for (i = 0; i < n; i++) { L[i+n*i] = 1; c[i+n*i] = A[i+n*i]; } /* Compute modification parameters */ gamma = -1; xi = -1; for (i = 0; i < n; i++) { gamma = mymax(gamma,absolute(A[i+n*i])); for (j = 0;j < n; j++) { /*printf("A(%d,%d) = %f, %f\n",i,j,A[i+n*j],absolute(A[i+n*j]));*/ if (i != j) xi = mymax(xi,absolute(A[i+n*j])); } } delta = mu*mymax(gamma+xi,1); if (n > 1) { beta = sqrt(mymax(gamma,mymax(mu,xi/sqrt(n*n-1)))); } else { beta = sqrt(mymax(gamma,mu)); } for (j = 0; j < n; j++) { /* Find q that results in Best Permutation with j */ maxVal = -1; q = 0; for(i = j; i < n; i++) { if (absolute(c[i+n*i]) > maxVal) { maxVal = mymax(maxVal,absolute(c[i+n*i])); q = i; } } /* Permute D,c,L,A,P */ permute(D,j,q); permuteInt(P,j,q); permuteRows(c,j,q,n); permuteCols(c,j,q,n); permuteRows(L,j,q,n); permuteCols(L,j,q,n); permuteRows(A,j,q,n); permuteCols(A,j,q,n); for(s = 0; s <= j-1; s++) L[j+n*s] = c[j+n*s]/D[s]; for(i = j+1; i < n; i++) { c[i+j*n] = A[i+j*n]; for(s = 0; s <= j-1; s++) { c[i+j*n] -= L[j+n*s]*c[i+n*s]; } } theta = 0; if (j < n-1) { for(i = j+1;i < n; i++) theta = mymax(theta,absolute(c[i+n*j])); } D[j] = mymax(absolute(c[j+n*j]),mymax(delta,theta*theta/(beta*beta))); if (j < n-1) { for(i = j+1; i < n; i++) { c[i+n*i] = c[i+n*i] - c[i+n*j]*c[i+n*j]/D[j]; } } } for(i = 0; i < n; i++) P[i]++; mxFree(c); mxFree(A); }gpml-matlab-v4.0-2016-10-19/util/minfunc/mex/mexAll_matlab.m000644 000765 000024 00000000265 12701705601 023206 0ustar00carlstaff000000 000000 clear all, close all mex -O -outdir ../compiled/ lbfgsAddC.c mex -O -outdir ../compiled/ lbfgsC.c mex -O -outdir ../compiled/ lbfgsProdC.c mex -O -outdir ../compiled/ mcholC.cgpml-matlab-v4.0-2016-10-19/util/minfunc/mex/lbfgsC.c000644 000765 000024 00000004550 12701705601 021633 0ustar00carlstaff000000 000000 #include #include "mex.h" /* See lbfgs.m for details! */ /* This function may not exit gracefully on bad input! */ void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) { /* Variable Declarations */ double *s, *y, *g, *H, *d, *ro, *alpha, *beta, *q, *r; int nVars,nSteps,lhs_dims[2]; double temp; int i,j; /* Get Input Pointers */ g = mxGetPr(prhs[0]); s = mxGetPr(prhs[1]); y = mxGetPr(prhs[2]); H = mxGetPr(prhs[3]); /* Compute number of variables (p), rank of update (d) */ nVars = mxGetDimensions(prhs[1])[0]; nSteps = mxGetDimensions(prhs[1])[1]; /* Allocated Memory for Function Variables */ ro = mxCalloc(nSteps,sizeof(double)); alpha = mxCalloc(nSteps,sizeof(double)); beta = mxCalloc(nSteps,sizeof(double)); q = mxCalloc(nVars*(nSteps+1),sizeof(double)); r = mxCalloc(nVars*(nSteps+1),sizeof(double)); /* Set-up Output Vector */ lhs_dims[0] = nVars; lhs_dims[1] = 1; plhs[0] = mxCreateNumericArray(2,lhs_dims,mxDOUBLE_CLASS,mxREAL); d = mxGetPr(plhs[0]); /* ro = 1/(y(:,i)'*s(:,i)) */ for(i=0;i=0;i--) { /* alpha(i) = ro(i)*s(:,i)'*q(:,i+1) */ alpha[i] = 0; for(j=0;jHH "uHH !u HH!H9=!t!H !tMĺI!!H\$@Hl$HH|$PLd$XH A_A^A]HHXHpHxATH0ILX!u9/!u 3ۉXtu7HrHtЋ؉D$ LƋI0؉D$ LƋI؉D$ u5u1L3IL3ILMt L3IAӅtu7LƋI#ˋىL$ tHHtLƋIЋ؉D$ 3ۉ\$ ' H\$@Ht$HH|$PH0A\H\$Ht$WH IHukLNjHH\$0Ht$8H _H @SH HH HD$8Hu HB~H HD$8H HD$@HHLD$@HT$8HHL$8xHHL$@fHgSHH [H(GHH(H\$WH HsH=lHHtHH;rH\$0H _H\$WH HKH=DHHtHH;rH\$0H _HMZf9t3HcH5H5V5^54r4d4X4B46444lbfgs_start and lbfgs_end must be int32int32 h4 p`P20  t T 42t d 4RT"P##d42 p2P20T"{ #  4 2p BT"":"#:"B  4 2p3|40340P3h5P03*6065555555v5444444455>5H5V5^54r4d4X4B464444mxFreemxCreateDoubleMatrix_700mxCallocFmxGetDimensions_700mxIsClassmxGetScalarymxGetPrlibmx.dll2mexErrMsgTxtlibmex.dll_malloc_crt_initterm_initterm_ecfree_encoded_null_amsg_exit__C_specific_handler__CppXcptFilter@__clean_type_info_names_internal[_unlockH__dllonexit_lock_onexitMSVCR100.dllEncodePointerDecodePointerSleepDisableThreadLibraryCallsQueryPerformanceCounterGetTickCountGetCurrentThreadIdGetCurrentProcessIdGetSystemTimeAsFileTimeKERNEL32.dll[Wr6h6l6p66lbfgsProdC.mexw64mexFunctionu2-+] f1^1`115 $2D <2 !2 !D!\2D!|!\2"A"h2`""2"7#2P#u#42##42##42##420 HX`Z PAPADDINGXXPADDINGPADDINGXXPADDINGPADDINGXXPADDINGPADDINGXXPADDINGPADDINGXXPAD0 (gpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/mcholC.mexglx000644 000765 000024 00000031366 12701705601 023732 0ustar00carlstaff000000 000000 ELF4p%4 (%"hh..dl..$$PtdLLQtdRtd..88GNUdD)֭!Y      5?r  GSg"?3  __gmon_start____cxa_finalize_Jv_RegisterClassesmexFunctionmxGetPrmxGetScalarmxGetDimensions_700mxCreateNumericArray_700mxGetDatamxCallocsqrtmxFreemexVersionlibmx.solibmex.solibmat.solibm.so.6libstdc++.so.6libpthread.so.0libc.so.6mcholC.mexglxMEXGLIBC_2.1.3GLIBC_2.0v7.0Q si   ii ".(0/// 000 00000 0 $0 US[*t=x X[ hhhhh h($h0(h8p,h@`0hHPUVS *8u]t4$<)p9s A<<9rƃ8[^]US.Ë)tt $҃[]Ë$ÐUEEw]UEr]UVUM 1E0^]UMU E]UWVE}~+4M ȋMȹ9^_]UVMu~,E E9^]UWVSY(u$ǃ}u݃ݝHF$ݝH$@EED$ D$ED$$U $E@MED$ D$ut$$IU B$kED$ D$ t$$M A$(D@,D$4$0ED$4$4E@ݕPݝX54E}E@(}uM9@EE9@Ë@ Mu40E9@EݕPݝXu$\$݅P$ݝPE4<¾9ut"$\$݅X$ݝX}9@˃EEEU9@\$݅P܅X$T܍Hݝ`@,P$d$zt$t&vܽX\$݅H$\$݅P$z t؍v$vݝhF݅H\$݅P$z tؐt&$?vݝh@@UUċMM؋uuEEE@ EEE؃EỦUȋEEuƿDžpDžt$݅pv%$\$݅p$ݝp}̃EuċE9@|$UȉT$M $|$uȉt$D$@T$ |$t$M $@t$ |$EȉD$U$t$ |$MȉL$u4$r@D$ |$UȉT$4$@L$ |$uȉt$E$5@T$ |$t$M $Vx.0 ƋUܸu4  9~EȃE9@4+0E؉EŰ0щUM̉ME؉΃$U$x&u܋M (  9~EEEM9@ủu@9~N9@uݝx$\$݅x$ݝx9@݅hػ \$݅`$R]Ћ0E$UE\$$,EU\M9@A} D70M̋Uȋu}t(Eă9@(9@M $u4$[^_]ݝx݅x݅h\$݅`$iݝ80E$i݅8\$$=EU\M9@OUh ]Ë $ÐUVSJ t&Ћu[^]US[ Y[-q=MATLAB R14 native;Hd5rl0zR| AB 8AB TAB ApAB >AB B9=AB AZAB I4AB   8o ' /P(ooo$ooo.BRbr(0GCC: (SUSE Linux) 4.3.2 [gcc-4_3-branch revision 141291]GCC: (SUSE Linux) 4.3.2 [gcc-4_3-branch revision 141291]GCC: (SUSE Linux) 4.3.2 [gcc-4_3-branch revision 141291]GCC: (SUSE Linux) 4.3.2 [gcc-4_3-branch revision 141291]GCC: (SUSE Linux) 4.3.2 [gcc-4_3-branch revision 141291]GCC: (SUSE Linux) 4.3.2 [gcc-4_3-branch revision 141291]$"$}(y/usr/src/packages/BUILD/glibc-2.9/cc-nptl/csu/crti.S/usr/src/packages/BUILD/glibc-2.9/csuGNU AS 2.19T /tmp/ccN3fq3S.s/usr/src/packages/BUILD/glibc-2.9/csuGNU AS 2.19U%U%l /usr/src/packages/BUILD/glibc-2.9/cc-nptl/csu../sysdeps/genericcrti.Sinitfini.c!/!=Z!gg//!/!=Z!P& /tmpccN3fq3S.s(!!!-!!!(,ospwg.symtab.strtab.shstrtab.note.gnu.build-id.gnu.hash.dynsym.dynstr.gnu.version.gnu.version_d.gnu.version_r.rel.dyn.rel.plt.init.text.fini.rodata.eh_frame_hdr.eh_frame.ctors.dtors.jcr.data.rel.ro.dynamic.got.got.plt.data.bss.comment.debug_aranges.debug_info.debug_abbrev.debug_line.debug_ranges.comment.SUSE.OPTs$288P.o,8 @'HoUo8do$$`s (| P 0,, Ldd...../ /4(0( ,0, , \!P!"$+"7#@E0$$X8+0$E h08$   ,  d.....//(0,0 ! .%.3.@ V,0e00s` .d. .M= > / (0#.0 9,0E \ b m40r,0y .     )8W"sinitfini.ccrtstuff.c__CTOR_LIST____DTOR_LIST____JCR_LIST____do_global_dtors_auxcompleted.5719dtor_idx.5721frame_dummy__CTOR_END____FRAME_END____JCR_END____do_global_ctors_auxmcholC.cmexversion.cversionpermuteColspermuteRows_fini_GLOBAL_OFFSET_TABLE_permute__dso_handle__DTOR_END__absolute__bss_start__i686.get_pc_thunk.cxmymaxpermuteInt_end_edata__i686.get_pc_thunk.bx_DYNAMIC_init__gmon_start___Jv_RegisterClassesmxGetData@@v7.0MEXmxGetScalar@@v7.0sqrt@@GLIBC_2.0mxGetDimensions_700@@v7.0mexVersionmexFunctionmxCalloc@@v7.0mxCreateNumericArray_700@@v7.0__cxa_finalize@@GLIBC_2.1.3mxFree@@v7.0mxGetPr@@v7.0gpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsAddC.mexmaci64000644 000765 000024 00000021140 12701705601 024614 0ustar00carlstaff000000 000000  (__TEXT__text__TEXT __symbol_stub1__TEXT__cstring__TEXT__stub_helper__TEXT|__unwind_info__TEXTPTP__eh_frame__TEXTP `8__DATA__dyld__DATA__nl_symbol_ptr__DATA__la_symbol_ptr__DATA ( H__LINKEDIT  `Iѱ&$"0   h  ! Px!  8@loader_path/libmx.dylib 8@loader_path/libmex.dylib 8@loader_path/libmat.dylib 8/usr/lib/libstdc++.6.dylib 8/usr/lib/libgcc_s.1.dylib 8o/usr/lib/libSystem.B.dylibASLWAS%O%NUHAWAVAUATSHHH9H{IH{IH{IH{ IH{(,H{(H5EtYH{e~;Ẻ1DH HcHHD9IIIITuH[A\A]A^A_H=,%%%%%int32lbfgs_end must be int32H=4tL3AS%#Hh/L.hLhLh>LhQL 888  XzRx 4  __,>! U@dyld_stub_binderQqq @_mexErrMsgTxtq(@_mxGetDimensions_700q0@_mxGetPrq8@_mxGetScalarq@@_mxIsClass_mexFunction (08@ . AO \j @@ __mh_bundle_headerdyld_stub_binding_helper__dyld_func_lookup stub helpers_mexFunction_mexErrMsgTxt_mxGetDimensions_700_mxGetPr_mxGetScalar_mxIsClassdyld_stub_bindergpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsProdC.mexa64000644 000765 000024 00000030322 12701705601 024341 0ustar00carlstaff000000 000000 ELF>P@p!@8@      $$Ptd```QtdRtd  hhGNUdrQL|ގN    @  5 ?  "~jS]G 3 __gmon_start____cxa_finalize_Jv_RegisterClassesmexFunctionmxGetPrmxGetScalarmxIsClassmexErrMsgTxtmxGetDimensions_700mxCallocmxCreateDoubleMatrix_700mxFreelibmx.solibmex.solibmat.solibm.so.6libstdc++.so.6libpthread.so.0libc.so.6lbfgsProdC.mexa64MEXGLIBC_2.2.5T07 Q ui  H H         (  0  8  @  HJ%H5: %< @%: h%2 h%* h%" h% h% h%  h% hp% h`HHu HtHÐU= HATSubH=X t H= :H L% H L)HHH9s DHH AH H9rn [A\fUH= HtH HtH= @ÐUHAWAVAUATSHXHuIH9HEMuI>pHI}dII}XHEM} I?,EIU(HUH:,EI}0EH5uI?MtH5bHUH:6u H=SFI>EI>@E}DmDEMcLIIǾLIHE}HUHHt}~6UH  HufWHH9uDMAUډUDUEfWҋUH<LmDmLuLuMcIK HEDֺLcBYXHH9uC^DֺLc CY \HH9uADUApLmLu}u}DMAUUA9|ۋUډUDUEfWҋUH<LmDmLuLuMcIK HEDֺLcBYXHH9uC^DֺLc CY \HH9uADUD9MpLmLu UH EYHH9u}u}DMAD9M~UUAEfWɃH<L]LmLmLuMcIK H}DֺLcCYXHH9uC^DDֺLcC\BYXHH9uADUD9MlLmLuUUȋUHHUAAUȃH<ADUL}M\ EDLLcCYAXD AD HH9uHU^ AD DLLcAA\D BYXHH9uHDEH;MpL}LH}HX[A\A]A^A_f(C^f(C^f(C^DAD HU^ AD tUHSHH HtH HHHuH[ÐH_Hint32lbfgs_start and lbfgs_end must be int32;0zRx $AC T  o@h   `` ooo@ooo &6FH GCC: (Ubuntu 4.4.3-4ubuntu5) 4.4.3.symtab.strtab.shstrtab.note.gnu.build-id.gnu.hash.dynsym.dynstr.gnu.version.gnu.version_d.gnu.version_r.rela.dyn.rela.plt.init.text.fini.rodata.eh_frame_hdr.eh_frame.ctors.dtors.jcr.dynamic.got.got.plt.data.bss.comment$2P.o@@(8 hhh@HoUo8do@@ s```} PP  @``xxD      `H H P P 0P #s (3 .@h@ `    P `x      H P  P * 8 E p[P jX x     H  P ` P  % +3 B VZ"v call_gmon_startcrtstuff.c__CTOR_LIST____DTOR_LIST____JCR_LIST____do_global_dtors_auxcompleted.7382dtor_idx.7384frame_dummy__CTOR_END____FRAME_END____JCR_END____do_global_ctors_auxlbfgsProdC.c_fini_GLOBAL_OFFSET_TABLE___dso_handle__DTOR_END____bss_start_end_edata_DYNAMIC_initmxGetPr__gmon_start___Jv_RegisterClassesMEX__cxa_finalize@@GLIBC_2.2.5mxCallocmxGetDimensions_700mxIsClassmxFreemexFunctionmexErrMsgTxtmxCreateDoubleMatrix_700mxGetScalargpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsC.mexglx000644 000765 000024 00000017065 12701705601 023725 0ustar00carlstaff000000 000000 ELF44 (!  TX Ptd Qtd   t     HP[|?   d}Y3| " G< __gmon_start____cxa_finalize_Jv_RegisterClassesmexFunctionmxGetPrmxGetDimensions_700mxCallocmxCreateNumericArray_700mxFreemexVersionlibmx.solibmex.solibmat.solibm.so.6libstdc++.so.6libpthread.so.0libc.so.6lbfgsC.mexglxMEXGLIBC_2.1.3v7.0 Q si HL, 048<@DU hhhhh h($h0US[dtX[ÐUS70u8t($i ,ҋ,uƃ0[]Í&US.tt $҃[]Ë$ÐUWVSÓu$EF$xF$|F $|EF$EF$@EtD$$2ED$U$ED$M $EuuD$4$ED$4$E}}ED$ D$ED$$U $E}~h}Ⱦ+ 9EuػpE}9ut+}كtۋ|xكt몃}~%EEUºM9Uu}}EU‰UUډU؃EMMUԋ|lj}UЋUUMщU}Љ}xEEuFF}#UċEFV9MuEEUL^MUEF )9}ummMM}}EEUUċE9EX}~M} NjUƒ9Eu}MMEEu}DD}IEp|‹EE}DT9MuEL\Mpxp}E<plEDEd ll9}uσEUUt9E}~%EEM}׃9UuE$U$M $}<$E$Ĝ[^_]ËEEFUL^$MD}L\HU]Ë $ÐUVS-P@tЋFu[^]US[àTY[?MATLAB R14 native;,zR| AB I   t    80o4ooloooBRbrHGCC: (GNU) 4.1.0 (SUSE Linux)GCC: (GNU) 4.1.0 (SUSE Linux)GCC: (GNU) 4.1.0 (SUSE Linux)GCC: (GNU) 4.1.0 (SUSE Linux)GCC: (GNU) 4.1.0 (SUSE Linux)GCC: (GNU) 4.1.0 (SUSE Linux),t  &$ )|/usr/src/packages/BUILD/glibc-2.4/cc-nptl/csu/crti.S/usr/src/packages/BUILD/glibc-2.4/csuGNU AS 2.16.91.0.5|/usr/src/packages/BUILD/glibc-2.4/cc-nptl/csu/crtn.S/usr/src/packages/BUILD/glibc-2.4/csuGNU AS 2.16.91.0.5%%K /usr/src/packages/BUILD/glibc-2.4/cc-nptl/csucrti.St 3!/!=Z!#!/= !/!=Z!gg//Z!!!sK /usr/src/packages/BUILD/glibc-2.4/cc-nptl/csucrtn.S !!!) !.symtab.strtab.shstrtab.hash.dynsym.dynstr.gnu.version.gnu.version_d.gnu.version_r.rel.dyn.rel.plt.init.text.fini.rodata.eh_frame_hdr.eh_frame.ctors.dtors.jcr.data.rel.ro.dynamic.got.got.plt.data.bss.comment.debug_aranges.debug_info.debug_abbrev.debug_line!  )1o0>o448Moll@\ 0e 8 ni,,tzt t    <         (HH PP P Xhh   A 4l ,  t      HP  A Q\jx PL @ Q  @  <ERZ6  qzH Pt  Pw  T|   #Y0| <"X l< initfini.c/usr/src/packages/BUILD/glibc-2.4/cc-nptl/csu/crti.Scall_gmon_startcrtstuff.c__CTOR_LIST____DTOR_LIST____JCR_LIST__completed.5751p.5749__do_global_dtors_auxframe_dummy__CTOR_END____DTOR_END____FRAME_END____JCR_END____do_global_ctors_aux/usr/src/packages/BUILD/glibc-2.4/cc-nptl/csu/crtn.SlbfgsC.cmexversion.cversion__i686.get_pc_thunk.cx_DYNAMIC__dso_handle_init__bss_start_fini_edata__i686.get_pc_thunk.bx_GLOBAL_OFFSET_TABLE__endmxCalloc@@v7.0MEXmxGetPr@@v7.0mexVersionmxCreateNumericArray_700@@v7.0mxFree@@v7.0mexFunction__cxa_finalize@@GLIBC_2.1.3_Jv_RegisterClassesmxGetDimensions_700@@v7.0__gmon_start__gpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsC.mexw64000644 000765 000024 00000022000 12777177045 023556 0ustar00carlstaff000000 000000 MZ@ !L!This program cannot be run in DOS mode. $\}\}\}3 ^}3 8U}3 9^}UY}\}G}G8]}G ]}G]}Rich\}PEd[W"  x{@p5L82P`Pp 0.textb `.rdata0@@.data`@@.pdataP@@.rsrc` @@.reloc p"@BH\$UVWATAUAVAWHI IHU HKLI HKL= HKH1 HKLHD$8 HKHc8 HcXHˉ$H$HHD$(HHD$@KHD$XHcHHLHD$0E3HT$`|$`EAAIHD$ D$dhHHoH$3fWDLHD$PHDH<I^fffff(LI|kIPHHHI+HL LH HYHY@Xf(LXDYHY@XXuM;}+K HI II+I+HHYAXuHD$(f(IMH^BDL;$-H$LI|iHIT$IL LIHM+M+HMMHLHBH HIDHBIDHBIDHBIDuM;},LIK MIM+MI+HHHJD uD$AIc}L\$(L|$0HIIILlHHHHL$hHL$@HIH\$pH+L+L$HD$Hf(H2LIJ (N NM+M+I@HMMHLAD H HYAXAL YIXAD YAXAL YIXuL$M;}8JN I M+IMI+B HHYAXuAYLINJI@L+H+MHMIL+HILIAD H HY \AD QAD Y\AD IAD Y \AD QD Y\AD uH\$pM;}EJNMHLL+IML+I+ BHHY \BD uH|$hL$HD$HHLAL|$8Ll$0LT$ LI|lIPIIBHI+HL fDDH HAY@LAYHDAY@LAYHuM;}(IIKI+I+HHAY@uH$L|$ H6HT$@LL$XLl$(L+HH$HHII+I+H$NMgHD$HH\$8fff(H1LIqMHIIIH+INDH IY@XLYHXDY@XLYHXuM;}1JMHTIL+I+AHHYBXuL HCYT IMXIIII+IJfffffAH I\YDX@ADA \ YLXHBLA\YDX@ADA \ YLXHBLuI;}9HMIIM+H+AHH\AYDXBADuH\$8HD$HHIMH$+Ll$0H$H|$PIHAALIL+L+HIIL+M+M+IHM+LHLJ8HH4ffHH HJD HBJD HBJDHBJDuI;}'IH HH+L+J;HHIHAuHL$(@HL$@6HL$X,I$IH$HĀA_A^A]A\_^]%x%j%\%N%@@SH HHH&H&HuC#H#H H 3H [HHXHhHxL` AUAVAWH 3ML8 &#D%eH%0HXH;t3H%uA%t_H %LHH %MLHHI;rZH9}tH9EtHMhHHEH r%LH ]%H<L;uL;tLLIQ;H,%H-%=%EH=%3eH%0HXH;t3H$u$t 7>HpH Y$uH7H (a$u HH[$H9=l$t!H c$tMĺII$ $H\$@Hl$HH|$PLd$XH A_A^A]HHXHpHxATH0ILX#u9#u 3ۉXtu7HHtЋ؉D$ LƋI0؉D$ LƋI؉D$ u5u1L3IL3IL%Mt L3IAӅtu7LƋI#ˋىL$ tHHtLƋIЋ؉D$ 3ۉ\$ "H\$@Ht$HH|$PH0A\H\$Ht$WH IHukLNjHH\$0Ht$8H _H i"@SH HH |"VHD$8Hu H~H N"(HD$8H 4"HD$@HHLD$@HT$8HHL$8H"HL$@H!SHH [H(GHH(H\$WH HH=HHtHH;rH\$0H _H\$WH HkH=dHHtHH;rH\$0H _HMZf9t3HcH PAPADDINGXXPADDINGPADDINGXXPADDINGPADDINGXXPADDINGPADDINGXXPADDINGPADDINGXXPAD0 gpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsC.mexmac000644 000765 000024 00000022434 12701705601 023667 0ustar00carlstaff000000 000000  D__TEXT__text__TEXT  __picsymbol_stub__TEXT$__picsymbolstub1__TEXT __literal8__TEXT__eh_frame__TEXTPh __cstring__TEXT__DATA__data__DATA__dyld__DATA__la_symbol_ptr__DATA8__LINKEDIT   4C)../../bin/mac/libmx.dylib 4C../../bin/mac/libmex.dylib 4C../../bin/mac/libmat.dylib 4K\X /usr/lib/libSystem.B.dylib 82"d P++-" |B}|}cx=0}| x=(N |B}h|=kk}iN |ha̐aB!pf|3x|#xH!|sxuH|xxuH |yxu H|txuHуcuH8~óxH8|ux~óxH8|~x~óxHu88|zxxH]8|wxxHM8@8@88|}x8`ADa@H |Ha8||dx@\<~xeX8,<Ɇ`@ iT8|̮|"Į8B`zBK`$8||I9)A/@,}i~cxU8|Gc8cb8BB4A88,T8|.9k|.9k@D;}9i};U8|JU38}3Ȣ8B9)|1|B}+,}K } @@8|{9i}[UB8}"Tb8|̮ɉ9)|+a`<}b8BB4@L8@$iT8t|8C2|KB8|@88,T8|.9k|.9k@,~9iV8|̮}8B|Ԯ;z|ծB}+Ԯ9@}K 9 } ծ@H~{A}9Vg8}'U8|k9J|Ԯ| |"ĮC (|8Bai9)A|}CxAP/@,|#xiW8|Dɂ8Bك8cB~xHUxHMCxHE~xH=xH58!~aN |HB|h|H/SourceCache/Csu/Csu-58.1.1/bundle1.s/SourceCache/Csu/Csu-58.1.1/bundle1.sdyld_stub_binding_helper:F3dyld_lazy_symbol_binding_entry_pointdyld__mh_bundle_header__dyld_func_lookup:F3dyld_func_lookup_pointerEH_frame1_versiongpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/mcholC.mexa64000644 000765 000024 00000030635 12701705601 023530 0ustar00carlstaff000000 000000 ELF>p@p!@8@TT      $$PtdDDQtdRtd  hhGNUYx3n]!     @  5 ?  g"SG3 = F__gmon_start____cxa_finalize_Jv_RegisterClassesmexFunctionmxGetPrmxGetScalarmxGetDimensions_700mxCreateNumericArray_700mxGetDatamxCallocsqrtmxFreelibmx.solibmex.solibmat.solibm.so.6libstdc++.so.6libpthread.so.0libc.so.6mcholC.mexa64MEXGLIBC_2.2.5Q ui ui H H         (  0  8  @  HR H5" %$ @%" h% h% h%  h% h% h% h% hp% h`HHU HtHÐU= HATSubH=8 t H= BH L% H L)HHH9s DHHm AHb H9rN [A\fUH= HtH HtH= @ÐUH_UHf(fWf(fTfUfVf(UHHcH 1HcH0UHHcH H1HcHHHH0UHATS~:A̸A0HcLMHcL IIMADA9u[A\UH~4ѸD0McN MDMcNMMM9uUHAWAVAUATSHIAIH9vHAu}EI|$EI<$EEEHu3IHIŋEEELeLIFHHh LIFHH H8U҉|LcLIľLHE}EEFL A}ALuAEADDHcHH4 I4D9uAIE9uEEEH?LEDMHcHItIID9uEEEALpLPDmHcEHUf(EEEA9t$IcHUf(EEEA9AEEE9LpLPEXE ,gYE@}~g|*Qf.zt Nf(E^f(Ef(E Qf.zt f(P.MEQf.zt ff(P}EHEDžpDž|AEED}Dž4DLIcIYf.Ev If(E1EDeADmE9LXMDdD}UuHhIUuH8DUuLFDUuLDUuL$DUuL]DUuH}DUuH}9DpEx6UHhHcHHcA ^AD DD9~ًEE9E~nD|L]D4IcHI IHEx7DDA4Hc8HcALAY \DD9~σA9}DuD9L~pU9U~h|fWEDXL(MADuHcAD{f(EcEAE9DXML( fWEHcEHhHMYPY^@E|HAME9L~`U9U|M|E}DEIHcIHcAYf(A^ \A9E9E~gAU|pEEEU9U2HEDdLXAMAD}D}H UHTHH9uLH}Hĸ[A\A]A^A_ÐUHSHH HtH HHHuH[ÐHH-q=?;D8`Bq5 zRx  AC </AC \AC | AC JAC G>AC $ FAC S  o@h   x` ooo8ooo .>N^H GCC: (Ubuntu 4.4.3-4ubuntu5) 4.4.3.symtab.strtab.shstrtab.note.gnu.build-id.gnu.hash.dynsym.dynstr.gnu.version.gnu.version_d.gnu.version_r.rela.dyn.rela.plt.init.text.fini.rodata.eh_frame_hdr.eh_frame.ctors.dtors.jcr.dynamic.got.got.plt.data.bss.comment$2P.o@@(8 hhh@ HoUo8do88@sxx`} ppX (DPP      `H H P P 0P #s (9 /@h8 x    pP      H P  p * 8 E [P jX x  P   > J   H   J/)P 5 @ ; yF` KP R [ ai x " = F call_gmon_startcrtstuff.c__CTOR_LIST____DTOR_LIST____JCR_LIST____do_global_dtors_auxcompleted.7382dtor_idx.7384frame_dummy__CTOR_END____FRAME_END____JCR_END____do_global_ctors_auxmcholC.cpermuteColspermuteRows_fini_GLOBAL_OFFSET_TABLE_permute__dso_handle__DTOR_END__absolute__bss_startmymaxpermuteInt_end_edata_DYNAMIC_initmxGetPr__gmon_start___Jv_RegisterClassesMEXmxCreateNumericArray_700__cxa_finalize@@GLIBC_2.2.5mxCallocmxGetDimensions_700mxFreemexFunctionmxGetDatamxGetScalarsqrt@@GLIBC_2.2.5gpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/mcholC.mexw64000644 000765 000024 00000027000 12777177045 023570 0ustar00carlstaff000000 000000 MZ@ !L!This program cannot be run in DOS mode. $\\\3^38U39^UY\BG(8]G( ]G(]Rich\PEd[W"  %f@P6L2P`Pp 0.text `.rdata0@@.data`@&@.pdataP(@@.rsrc`*@@.reloc p,@B@SWHMc3HcHIHl$0Ht$8Ld$@McLl$Lt$KK4MI+LHI+MJIML<$IL4M< IHHI+IIJKAICICKAKC KC IAIAMIuL<$Lt$Ll$Ld$@Ht$8Hl$0I;};IcILN L+HHL+HHBJIIuH_[H\$Ht$H|$Mc3HcHIMcIRIHIL\HI+IHHICADI HIDICACADIDICACALIDICAKADIDACuI;}:IcLL+HIMHL+HHBHIJDBuH\$Ht$H|$HHXUVWATAUAVAWHI )p)xIAHD)@D)HD)xD)h2Lu D HOfD(HE3HT$xHc8EAAI$P|$x|$|HHE3HT$XEAAI|$XHD$@D$\HHFE3HT$XEA AIHD$PyHHFgHHD$`Hc؉$HH=HHD$ +E3EALHHD$8H|$(LMLT$0fD|IIHD$(MIMMIH+IM+L+MML+L+HPHHIMMHHH,ffffffI HH HI IID I IHuLT$0Ll$(HT$8E3I;})HML+IHH IH+J HIHuHT$8Hl$`IHALT$0I;L|$ II?IHt$8H\$@J,LIIHMMMH+HTL+L+H+HH+IIL+LlH+HL$(H+LD.HNt8I?HHH4L|IHDO<IKM<HIM|HD*IIMHuLl$(L|$ E3I?I;}@HT$8LD$@IEHL+H ML+INH+NHJ IHuDF=&MfE(fA(ML|$8J<E3LgIfEf(fWf/sf(fD/wfD(IIKhMMIIIHIIuMAL;t"Bf(fWf/sf(f/wf(IAL;t!f(fWf/sf(f/wf(M;t f(fWf/sf(f/wf(IAL;t!:f(fWf/sf(f/wf(HIIH;LI;}>HIIIL;t f(fWf/sf(f/wf(HHI;|IIM;L|$ E3fD(EXfD/wfD(Ћ$PEYу~&$Hfn ^fD/wfD(fE/wfE(fA( AD$HILd$0f( fWMuY@fA(AMDL;+ƃoL|$ IMD^HIIHI+HT$0HJHIDITHI I+H@ f(fWf/rf(f(f/vf/sf(f/wf(AM f(fWf/rf(f(f/vf/sf(f/wf(A[MP f(fWf/rf(f(f/vf/sf(f/wf(AMP f(fWf/rf(f(f/vf/sf(f/wf(A[MPAIHHAD;H|$0$H$PD;}]IJII  f(fWf/rf(f(f/vf/sf(f/wf(AMAIHD;|HT$PDDJH H$HBHT$` BB IDDËILt$@DDËI_DDËILt$8DDËI:DDËIznEM@ELD$ Ht$@HT$PJ/J<MMMIH+IL+L+H LL+MIL+Ht$(MMDEHHIF4N<fffA H ^B ^J A ^BAL ^JA IIuLl$(H|$0$HD;ULD$@HD$PNJIIL|$ HMI M+DE+AB H^BIIuL|$ D$PD^LgD\$HL$ILd$pE;H\$(HLLIILd$@IJIAMHLIIM+HLD$8H$MA+LcL$L\$hL E3E3CELD$@HLI HD$(KHIJH, H  +uH H u HHH9=t!H tMĺIH\$@Hl$HH|$PLd$XH A_A^A]HHXHpHxATH0ILXAu9Su 3ۉXtu7Hn HtЋ؉D$ LƋI0؉D$ LƋI؉D$ u5u1L3IL3IL Mt L3IAӅtu7LƋI#ˋىL$ tH HtLƋIЋ؉D$ 3ۉ\$ KH\$@Ht$HH|$PH0A\H\$Ht$WH IHuoLNjHH\$0Ht$8H _H @SH HH  HD$8Hu H^ ~H  HD$8H  HD$@H HLD$@HT$8HHL$8 HHL$@ HWHH [H(GHH(H\$WH Ho H=h HHtHH;rH\$0H _H\$WH HG H=@ HHtHH;rH\$0H _HMZf9t3HcH%@H(uH= uH(H\$WH HHd$0H2-+H;t HHvHL$0_H\$0\DI3XDI3THL$8DI3KL\$8L3HL#H3-+L;LDLIL H\$@H _%j%l%n%p@UH HHHM(HM$H ]@UH HH ]@UH HH ]@UH HH3Ɂ8H ]$665555555444455*5<5`5j5x554444v4Z4N4B484!?-q=!1!B B dT1"p0t d4GG ? 7 2 $x h4(  p `P20  t T 422Pt d 4R4($%0)%$%`)d42 p204(W&&z)  4 2p B4('()(B  4 2p34034P003>60$665555555444455*5<5`5j5x554444v4Z4N4B4844mxFreemxCallocCmxGetDatamxCreateNumericArray_700FmxGetDimensions_700mxGetScalarymxGetPrlibmx.dllsqrtMSVCR100.dll_malloc_crt_initterm_initterm_ecfree_encoded_null_amsg_exit__C_specific_handler__CppXcptFilter@__clean_type_info_names_internal[_unlockH__dllonexit_lock_onexitEncodePointerDecodePointerSleepDisableThreadLibraryCallsQueryPerformanceCounterGetTickCountGetCurrentThreadIdGetCurrentProcessIdGetSystemTimeAsFileTimeKERNEL32.dll[W6x6|66P6mcholC.mexw64mexFunctionu2-+] f11J1PP1P!1!:"2<"$2$%42%&p2 &&2&&2& '2 'X'2'!(2@(c(2d()20)U),2`)z),2z)),2)),20 HX`Z PAPADDINGXXPADDINGPADDINGXXPADDINGPADDINGXXPADDINGPADDINGXXPADDINGPADDINGXXPAD0 gpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsProdC.mexglx000755 000765 000024 00000016224 12701705601 024551 0ustar00carlstaff000000 000000 ELF4`4 ( `d$$Ptd ,,QtdRtd((GNU>RAd!b,=tk@w  5?r F"} U , u'i R  __gmon_start___ITM_deregisterTMCloneTable_ITM_registerTMCloneTable__cxa_finalize_Jv_RegisterClassesmexFunctionmxGetPrmxGetScalarmxIsClassmexErrMsgTxtmxGetDimensions_700mxCallocmxCreateDoubleMatrix_700mxFreemexVersionlibmx.solibmex.solibpthread.so.0libc.so.6lbfgsProdC.mexglxMEXGLIBC_2.1.3:7 Q' si +4        $ (  ,  0  S?tz[ hhhhh h($h0(h8p,h@`0hHP$ffffffUSW;8)Ѓw[]Ët$t&US88)u[]ËtD$$却&USW8u$t4$8ƃ8[]Ít&'USwtt $Ѓ[]%UWVSl9$$ŋG$D$,G$D$0G $wD$G$(|$VD$V fD$Tl$T\$@l$VG$|$VD$V fD$Tl$T\$(l$VG$\$H|D$G$t|D$G$xu$G$k0G$^@D$8|$@|$(ED$<$ D$D$<$D$DD$D$4$[$$Z~DЃ9uL$(σ|$|$<ύډT$ L$,щL$4T$0T$$D$WVl$l$4LA\$XD$XQ9uT$l$4Yl$l$$DI,Ѓ9ul$L$ L$|$u؃|$@u L$8i|$@|$ 9|؋T$<ʉ|$ډT$$|$,׉|$<T$0T$4D$|$Wz|$|$< A\$XD$XQ9u|$4Y|$|$4I,Ѓ9ul$|$$|$;l$ }D$H 9u؃|$@u|$(L$@σ|$9|$8~݋T$D ʉL$$ L$4|$0׉|$<T$,T$@D$ |$$W|$ |$< A\$XD$XQ9u|$T$4Y|$ |$@L$L$AL$, Ѓ9uD$D$$|$4|$ |$89|$d|$D|$$<|$4D$ D$|$$|$0|$  \$XD$X9u|$T$4|$,|$ L$L$L$, Ѓ9uD$D$$|$4|$ |$(9|$i؋D$$D$D$[L$T$4_|$4|$_|$L$4ϋ|$$_|$L$4ϋ|$$il[^_] Ë $SÏ[int32lbfgs_start and lbfgs_end must be int32MATLAB R14 native;( DhzR|  F J tx?;*2$"<@AA AACAA AA0   h o8td 7  Ph(@oooooo&6FVfv4 GCC: (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4.symtab.strtab.shstrtab.note.gnu.build-id.gnu.hash.dynsym.dynstr.gnu.version.gnu.version_d.gnu.version_r.rel.dyn.rel.plt.init.text.fini.rodata.eh_frame_hdr.eh_frame.init_array.fini_array.jcr.dynamic.got.got.plt.data.bss.comment$.o88,8 dd@tt7Ho"Uo8do s ((@| hhP #h h 2| | B ,  44 48 808)a@4 (l8dt(h    h  |    4 8   . A0 W8 f   8  h  4 < !8 -68 B X ^d  t} " R    6 J dcrtstuff.c__JCR_LIST__deregister_tm_clonesregister_tm_clones__do_global_dtors_auxcompleted.6591__do_global_dtors_aux_fini_array_entryframe_dummy__frame_dummy_init_array_entrylbfgsProdC.cmexversion.c__FRAME_END____JCR_END____x86.get_pc_thunk.bx_edata_fini__dso_handle_end__bss_start_DYNAMIC__TMC_END___GLOBAL_OFFSET_TABLE__init__x86.get_pc_thunk.cxmxCalloc_ITM_deregisterTMCloneTablemxFreemxIsClassmxGetDimensions_700__cxa_finalize@@GLIBC_2.1.3mxGetScalar__gmon_start__mexVersionmexErrMsgTxtMEXmexFunctionmxCreateDoubleMatrix_700_Jv_RegisterClasses_ITM_registerTMCloneTablemxGetPrgpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsC.mexa64000644 000765 000024 00000017033 12701705601 023520 0ustar00carlstaff000000 000000 ELF>@@8@l l p p p   Ptd   Qtd @  5 ?   d="&[ Gp} 3 Lq__gmon_start____cxa_finalize_Jv_RegisterClassesmexFunctionmxGetPrmxGetDimensions_700mxCallocmxCreateNumericArray_700mxFreelibmx.solibmex.solibmat.solibm.so.6libstdc++.so.6libpthread.so.0libc.so.6lbfgsC.mexa64MEXGLIBC_2.2.5Qui            HH5 % @% h% h% h% h% h% hHH HtHÐU=` HATSubH= t H= zH L% L)HHXH H9v DHH AH H9w [A\fUH=g HtHs HtH=O IAÐUHAWAVAUATSHhIHH9HEH{HxH{HEH{HEH{D(H{@EHcؾHHEIƾHxIǾHhHËEAHHpHEHEHp0HEDmEHuHI$HE}~wE<YXȃHHD9ux^AID;}t,fWE~HcHHMHHxHfWɸE~0HMEAHHuHƾHHHHD9uUHcHM$L]ID߉}DUEEDALI$E6IcLHxLIcHHUH¾YXHHD9uAYHMHHULHELY \HHHD9uAIIDUDMD;u;E~0HMHuYH}ǃHD9uڃ}EAAALJE1IcHHMHHUH¿YXHHD9uHUYHMHHxHCHH}HǿA\YXHHHD9uAIED;U>E~0EAHHUH HUHHHHD9uH}L HH}H}Hh[A\A]A^A_fWAYA$IBHMBYBMUHSH HH HtHHHuH[ÐHH?;<0zRx $qAC M   oX  h` oooooo &6FVfvGCC: (GNU) 4.1.1 20070105 (Red Hat 4.1.1-52)GCC: (GNU) 4.1.2 20070626 (Red Hat 4.1.2-14)GCC: (GNU) 4.1.2 20070626 (Red Hat 4.1.2-14)GCC: (GNU) 4.1.2 20070626 (Red Hat 4.1.2-14)GCC: (GNU) 4.1.1 20070105 (Red Hat 4.1.1-52).symtab.strtab.shstrtab.gnu.hash.dynsym.dynstr.gnu.version.gnu.version_d.gnu.version_r.rela.dyn.rela.plt.init.text.fini.rodata.eh_frame_hdr.eh_frame.ctors.dtors.jcr.data.rel.ro.dynamic.got.got.plt.bss.commentoXX(%  -5oBo8Qo ``jhh topzx     ( ( Dp p       H 1 `Xh       ( p         p * 8 E S b x x h           ! ' / > RV=o"& p  Lqcall_gmon_startcrtstuff.c__CTOR_LIST____DTOR_LIST____JCR_LIST__dtor_idx.6142completed.6140__do_global_dtors_auxframe_dummy__CTOR_END____FRAME_END____JCR_END____do_global_ctors_auxlbfgsC.c_fini_GLOBAL_OFFSET_TABLE___dso_handle__DTOR_END____bss_start_end_edata_DYNAMIC_initmxGetPr__gmon_start___Jv_RegisterClassesMEXmxCreateNumericArray_700__cxa_finalize@@GLIBC_2.2.5mxCallocmxGetDimensions_700mxFreemexFunctiongpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/mcholC.mexmac000644 000765 000024 00000023320 12701705601 023667 0ustar00carlstaff000000 000000  D__TEXT__text__TEXT` `__picsymbol_stub__TEXTtt$__picsymbolstub1__TEXTtt __literal8__TEXTx(x__eh_frame__TEXTLh __cstring__TEXT__DATA__data__DATA__dyld__DATA__la_symbol_ptr__DATA 8__LINKEDIT   4C)../../bin/mac/libmx.dylib 4C../../bin/mac/libmex.dylib 4C../../bin/mac/libmat.dylib 4K\X /usr/lib/libSystem.B.dylib PC# P99;#t |B}|}cx= }| x= N |B}h|=kk t}iN |HQ!PBf|+x|3x|#xH },|zx@<˃ H~H =~;PH 8H888`H~֓LHɐ|H !8@x88|{x8`ATPH|Hx8 8|yx8`H}|HU8|xx~xH%8|}x~xH9|||x@@ x, 9)@ U8W8|:|<|B9A9|@@= ?9@8@|9T8|B}$.8}D.8|D|DB9<| @9 }HJUG8|gP`OYӂA`h@ Hh8|@\U 8|KW88A8Ȃ P `LAA`h@ Hh8|BB9})AT*=Ȩ (@ hH =? ,2@P8w<C0lz@AD=@ '0(H=?$@ H@H @ H H8@!r;^>8w |;x@|G|P|}HUE8}@P@hOYӂAh X@D}hUf8}&HPH`L3A` h@ XHh`}Cx9|BBT8T8}D̮,}j̮T:T:}dͮ}Jͮ},.|H.|L.}(.@0}ISx|#x}W8| ||B}})B9`|9|@4~T8V89k }}9)8BA,@0T8T8ܮW8ܮݮ|BRݮ})RB,@0}fT8Ub8|Iܮ|bܮ|iݮ9)|Bݮ8BB/@0T8T8|W8|||BB|})BB,@0}T8U8|||9)|8BB4A0|"xT8W8}=B8B P$}|JB9|}Cx@l}(*U*8/}j}jAD:U8T8~}"|K}KSxW8 |Bb}})b`<B99J|A|<f @\@T}%|PU*8| |J8BP`LAA`h@ XHh`B~*|DV8|HPhOYӂAh $@@`@\||hx@P}#}e|PUe8||E|J|̮T82|8B9})8$d((|lB||gxOA(9@ U:9.;~y.BxHxH8!H L N PLAA hN T:T:|.}.}.|.N T8T8|"|C|B|#N |3yL |T8T8|$T8||||%|BN |3yL |1||1T8T8|"|E|B8B|%8BN |HB|h|H<?A@=B;><?A@__dyld_func_lookupdyld_stub_binding_helper__mh_bundle_header_absolute_mexFunction_mymax_permute_permuteCols_permuteInt_permuteRowsabsolute.ehmexFunction.ehmymax.ehpermute.ehpermuteCols.ehpermuteInt.ehpermuteRows.eh_mxCalloc_mxCreateNumericArray_mxFree_mxGetData_mxGetDimensions_mxGetPr_mxGetScalar_sqrtrestFPsaveFP_mexVersionmexVersion.eh{standard input}int:t1=r1;-2147483648;2147483647;char:t2=r2;0;127;void:t3=3/SourceCache/Csu/Csu-58.1.1/bundle1.s/SourceCache/Csu/Csu-58.1.1///SourceCache/Csu/Csu-58.1.1/bundle1.s/SourceCache/Csu/Csu-58.1.1/bundle1.sdyld_stub_binding_helper:F3dyld_lazy_symbol_binding_entry_pointdyld__mh_bundle_header__dyld_func_lookup:F3dyld_func_lookup_pointerEH_frame1_versiongpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsC.mexmaci000644 000765 000024 00000030564 12701705601 024043 0ustar00carlstaff000000 000000  \H__TEXT__text__TEXT  __literal8__TEXT__eh_frame__TEXTD `__cstring__TEXT__DATA__data__DATA__dyld__DATA|__IMPORT  __jump_table__IMPORT  8__LINKEDIT00t0 0 P00 4@loader_path/libmx.dylib 4@loader_path/libmex.dylib 4@loader_path/libmat.dylib 4/usr/lib/libstdc++.6.dylib 4/usr/lib/libgcc_s.1.dylib 4o/usr/lib/libSystem.B.dylibX[cXSUWVS|u[$(NjF$EF$ EF $EF$EċF$@D$Eȉ$D$EEȉ$UD$$EuD$FuĉE4$p4$1D$E[MED$ D$M$EED$-U $/EEȅqEF9ut[Eą~EfWɋMU1f@YX9EuE^F9uuEą~)EċUEȍ1B9UuEHu}EEOMFEąEMEċMǍ1AYX9MuEYEą~XEEMUE1EEЋEfDfDGY \9}ĉEuڋ}m'}ą~41MBMYM9Uu܋uȅ)Euԋ}EGMą~vEM1EAYX9MuEY0UąJEԋUċMU9M}GMąEY0Uą~EMMMUEuEEԉu1‰EEUFU\YX9uĉUuԋM9M\Eą~&EċUEȍ1fMB9UuE$U$M $E$U$|[^_fDfDYU勁?zR| ( MATLAB R14 native . AS\ iu __mh_bundle_headerdyld_stub_binding_helper__dyld_func_lookupdyld__mach_header_version_mexFunction_mexVersion_mxCalloc_mxCreateNumericArray_700_mxFree_mxGetDimensions_700_mxGetPrgpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsAddC.mexglx000755 000765 000024 00000016056 12701705601 024340 0ustar00carlstaff000000 000000 ELF4T4 (44TX$$Ptdddd,,QtdRtd((GNU.Y!wd?b   5?r F"} U , ui  __gmon_start___ITM_deregisterTMCloneTable_ITM_registerTMCloneTable__cxa_finalize_Jv_RegisterClassesmexFunctionmxGetPrmxGetScalarmxIsClassmexErrMsgTxtmxGetDimensions_700mexVersionlibmx.solibmex.solibpthread.so.0libc.so.6lbfgsAddC.mexglxMEXGLIBC_2.1.3nL Qsi (       $  S÷tb[ hhhhh h($h0$ffffffUS/,)Ѓw[]Ët$t&US÷,,)u[]ËtD$$却&USWg,u$t($8ƃ,[]Ít&'UStt $Ѓ[]%UWVS,l$LE$ƋE$NjE$D$ E ${D$E$<؋E$/|$D$ fD$l$\$l$JD$E$u8$ E$~0l$L$l$ \9u,[^_] PË $S[lbfgs_end must be int32MATLAB R14 native;( DhzR|  F J tx?;*2$"8@GAA AAC@AA AA| @ $o8Dd   8@opooooRo( GCC: (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4.symtab.strtab.shstrtab.note.gnu.build-id.gnu.hash.dynsym.dynstr.gnu.version.gnu.version_d.gnu.version_r.rel.dyn.rel.plt.init.text.fini.rodata.eh_frame_hdr.eh_frame.init_array.fini_array.jcr.dynamic.got.got.plt.data.bss.comment$.o88,8 dd@DD HoRRUopp8do s @| 8 @@#pp3$$288*dd, (( (, ,0,)U4 B8dDRp @ p  $ 8d ( ,   .@ A W, f 0 , $ ( 0  , ,5, A W@ ] s "      :crtstuff.c__JCR_LIST__deregister_tm_clonesregister_tm_clones__do_global_dtors_auxcompleted.6591__do_global_dtors_aux_fini_array_entryframe_dummy__frame_dummy_init_array_entrylbfgsAddC.cmexversion.c__FRAME_END____JCR_END____x86.get_pc_thunk.bx_edata_fini__dso_handle_end__bss_start_DYNAMIC__TMC_END___GLOBAL_OFFSET_TABLE__init__x86.get_pc_thunk.cx_ITM_deregisterTMCloneTablemxIsClassmxGetDimensions_700__cxa_finalize@@GLIBC_2.1.3mxGetScalar__gmon_start__mexVersionmexErrMsgTxtMEXmexFunction_Jv_RegisterClasses_ITM_registerTMCloneTablemxGetPrgpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/mcholC.mexmaci64000644 000765 000024 00000031600 12701705601 024212 0ustar00carlstaff000000 000000  (__TEXT __text__TEXT6__symbol_stub1__TEXT*__const__TEXT@(@__stub_helper__TEXThh__unwind_info__TEXTh__eh_frame__TEXTxx `8__DATA  __dyld__DATA  __nl_symbol_ptr__DATA  __la_symbol_ptr__DATA 8  H__LINKEDIT00 z9Cx:"000 001p2 P  020 8@loader_path/libmx.dylib 8@loader_path/libmex.dylib 8@loader_path/libmat.dylib 8/usr/lib/libstdc++.6.dylib 8/usr/lib/libgcc_s.1.dylib 8o/usr/lib/libSystem.B.dylibASLAS% % U_HfDf(UfW#Hf(fTfUfVf(DHcHcUH4HH @HcHcUH4HHHHHH UAAHS~;E1E1fA CAEE9HcHH HH1HHH0u[ÐUAH~>AAE1DD@C CAE9HcHH HH1HHH0uDUHAWAVIAUIATSHH9IPI}H]D(Hu1ɺDmDmHI1H޺HxDmE_HIFk1ɺ H޿HE;HIF;HHDAHc؉4HH߾IE1EHEHH]I ILXL`YPAh[4`*Q^PNf(X=QYA]E1HEE]]8EHDEHpDžDf.EA9~DB#HAf.pvf(EpDpA9DH}DDHHDDDDLDDDLDHxDDDHxDDDH}DD~DH}DDUxA1DfDHcH]HcHDA^Hx;u~EgE9DA݉]E9EEfWf(HUHEh^8HRU(AHAF()E9E~WE9~RUDA4Hc։A AYɍ^ HI\ED9uHEEEME9HH1HA9uLH}H[A\A]A^A_AHE9t*D1DHcH]HDA9IHu111HxI?DHHA9LH]HIuHHE1HXH`Dž$$H]E1AHDf(XXE9t*HUHcf(`y`ADE9uAD$E9y` XX6YPAhPXQA_@E9lEfWDEHcAf(f(EEf(9uYDDUDAEE ]fDfDHUIcHM AExA11fDfDAHxDHcHAY\A;u~AA9uvI}1PI%%%% % %%-q=?H=tLAS%HVhyLhjLhOLh>Llh0LRhL8shLa$$<<< $ XzRx ,h  ,LH+ ,|H ,8  ,(O , HK D<h __! W@dyld_stub_binderQqq @_mxCallocq(@_mxCreateNumericArray_700q0@_mxFreeq8@_mxGetDataq@@_mxGetDimensions_700qH@_mxGetPrqP@_mxGetScalar_mexFunction, (08@HP.AHR@^`gth  @@  __mh_bundle_headerdyld_stub_binding_helper__dyld_func_lookup_mymax_absolute_permuteInt_permute_permuteRows_permuteCols stub helpers_mexFunction_mxCalloc_mxCreateNumericArray_700_mxFree_mxGetData_mxGetDimensions_700_mxGetPr_mxGetScalardyld_stub_bindergpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsAddC.mexw64000644 000765 000024 00000016000 12777177045 024172 0ustar00carlstaff000000 000000 MZ@ !L!This program cannot be run in DOS mode. $|///C//u//t//L////u//D//B//Rich/PEd[W"  p@%O\"dP@` .text   `.rdata @@.data`0@.pdata@@@.rsrcP@@.reloc `@BH\$Hl$Ht$WATAUAVAWH I IHMHHMHHMLHM HsHM(jHM(HD,Ou H QHME3Mc*LcIIOLLLM+IHL+L+JT(M+H+IHM+HHI+HL$H,8L4JDH HIDJDHBIDIDIDHBIDIDIDHBHD*JD JD2HBuM;}FIOI+IJ!ITHH+H+I+I+M+HHIHDHDHBuH\$PHl$XHt$`H A_A^A]A\_%$%%%%@SH HH?HPHAHuC#H#H H 3H [HHXHhHxL` AUAVAWH 3ML8#DeH%0HXH;t 3HuA|tcH yS LHH X: MLHHI;rZH9}tD H9EtHM H+ HEH  H H L;uL;tLLI  HH=EH=3eH%0HXH;tu 3HjuSt ;>H H  +uH H u HHH9=t!H tMĺIH\$@Hl$HH|$PLd$XH A_A^A]HHXHpHxATH0ILXAu9Su 3ۉXtu7HV HtЋ؉D$ LƋI0؉D$ LƋI؉D$ u5u1L3IL3IL Mt L3IAӅtu7LƋI#ˋىL$ tH HtLƋIЋ؉D$ 3ۉ\$ KH\$@Ht$HH|$PH0A\H\$Ht$WH IHuoLNjHH\$0Ht$8H _H @SH HH  HD$8Hu HN ~H  HD$8H  HD$@H HLD$@HT$8HHL$8 HHL$@ HWHH [H(GHH(H\$WH HW H=P HHtHH;rH\$0H _H\$WH H/ H=( HHtHH;rH\$0H _HMZf9t3HcH PAPADDINGXXPADDINGPADDINGXXPADDINGPADDINGXXPADDINGPADDINGXXPADDINGPADDINGXXPAD gpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsProdC.mexmaci64000644 000765 000024 00000021440 12701705601 025033 0ustar00carlstaff000000 000000  x__TEXT__text__TEXT0 0 __symbol_stub1__TEXT0__cstring__TEXTP0P__const__TEXT__stub_helper__TEXT__unwind_info__TEXTPTP__eh_frame__TEXTP `8__DATA__dyld__DATA__nl_symbol_ptr__DATA__la_symbol_ptr__DATA @  H__LINKEDIT  @)AqPSY"0   !@" P !  8@loader_path/libmx.dylib 8@loader_path/libmex.dylib 8@loader_path/libmat.dylib 8/usr/lib/libstdc++.6.dylib 8/usr/lib/libgcc_s.1.dylib 8o/usr/lib/libSystem.B.dylibASLAS%%UHAWAVAUATSHLsHHHuH9HEH{H{IH{IHEH{ ,EH{(,uH{0xEH{ H5zieH{(H5bQMI>/I>%@}EDELcLLI1ҾHEHHEH8I~1 11ҐHuHH9fWAu܋EAAADAADfDIcH<I4?H~)L1ɐBHADYH9XuHE^8~;A<LDHcAY \H9uAEE9^}DMDUAAE9EADAIcH<I4?H~)L1ɐBHADYH9XuHE^8~;A<LDHcAY \H9uAEE9^~!L1DEYH9u}tE9EE̅E1E1E1ɐHuNLƅH~(L1ɐA HAYH9XuHEB^~>B< LDCHc\AYDXH9uAAID9]WLH}HH[A\A]A^A_H= AD]ADHuIcH<HH~.L1A HAYH9XuHE^8~>M?A<LDDfDAHc\AYDXH9uAAE9SR%%%%%% % %int32lbfgs_start and lbfgs_end must be int32H=xtLwAS%gH~hLh|LhmLfhRLLhDL2h$LshLahLO 888 P XzRx 4  __0 ! X@dyld_stub_binderQqq @_mexErrMsgTxtq(@_mxCallocq0@_mxCreateDoubleMatrix_700q8@_mxFreeq@@_mxGetDimensions_700qH@_mxGetPrqP@_mxGetScalarqX@_mxIsClass_mexFunction (08@HPX0 .D AOP \jt @@ __mh_bundle_headerdyld_stub_binding_helper__dyld_func_lookup stub helpers_mexFunction_mexErrMsgTxt_mxCalloc_mxCreateDoubleMatrix_700_mxFree_mxGetDimensions_700_mxGetPr_mxGetScalar_mxIsClassdyld_stub_bindergpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsC.mexw32000644 000765 000024 00000016000 12701705601 023532 0ustar00carlstaff000000 000000 MZ@ !L!This program cannot be run in DOS mode. $PELF!  H0p`T@hP|.text<< `.rdataT T @.data<0<@.idatah@h`.reloc|P| .edataT`T @1@L$AtD$T$SVWD$Pjhd5d%D$ Xp t ;t$$t4v L$H |uTd _^[USVWUjjhu8]_^[]USVWU] E@umEEEECs {tb v|t:VUkT]^] t(x1{SkVS vC T{ v41UkjS j ]_^[]USVW} u"} uuu u0_^[ U Wjj]EjjQEjjEEh:0u)0h80u0h80u0 0=0 t jW3= 0 tjWZ_á 0 tPY0 tPY0 tPYU}fE fEm]mEUSVWEGÉ]!<=tE1ٍAG;<uՋ}G<}u;YE0}u1v]T1ٍAG}<=t0uYu5 u:S}4= EU<uU_^[U] ULSVW}4=4Eċ}w#E}wE}w E}w<}}wx}juEjuEjuuEj}ũW[Ej}ũWAE}}Ejj}Wj+u 5} 4=3EEO]E'}uuEЋu>u >]ЃE}9}|ы}uuE}9}|E}uủuUރE}9}|ڋ}̃}}uEB}u4]U5ωȋM׉׋] 5E}9}|}u<=u] =E<}u]ӉщM؃}ߋ} E}9}|m}'E }u]]U E}9}|E}uE9}u܍4]U׉5];] ;5E}9}|}u܍<=u] =EH}u]Uˉȉ uu]u]$E}9}|E}9}"E}u]]̉ڋ]E}9}|uMuBu7u,u!_^[%@%@%@%@%@%@%@%A% A%A%A%A%A% A%$A%(AT%swr@B@@"B@@L hYL 888 p XzRx <* __ 2! U@dyld_stub_binderQqq @_mxCallocq(@_mxCreateNumericArray_700q0@_mxFreeq8@_mxGetDimensions_700q@@_mxGetPr_mexFunction (08@P .d AOp \f @@ __mh_bundle_headerdyld_stub_binding_helper__dyld_func_lookup stub helpers_mexFunction_mxCalloc_mxCreateNumericArray_700_mxFree_mxGetDimensions_700_mxGetPrdyld_stub_bindergpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/lbfgsAddC.mexa64000644 000765 000024 00000020110 12701705601 024117 0ustar00carlstaff000000 000000 ELF>`@X@8@    $$PtdpppQtdRtd hhGNUw4 %WVa쐭    @  5 ?  "jS]G3 ,__gmon_start____cxa_finalize_Jv_RegisterClassesmexFunctionmxGetPrmxGetScalarmxIsClassmexErrMsgTxtmxGetDimensions_700libmx.solibmex.solibmat.solibm.so.6libstdc++.so.6libpthread.so.0libc.so.6lbfgsAddC.mexa64MEXGLIBC_2.2.54dL Qui 0 0          (  H"-H5 % @% h% h% h% h% h% hHHe HtHÐU= HATSubH=H t H= bH L% H L)HHH9s DHHe AHZ H9rF [A\fUH= HtH HtH= @ÐUHAWAVAUATSHIH9IIHIGHEH8IIII I(I?,EH5I?u H=HEH8[~=UăЃH<HcHH4It I4I4HH9uH[A\A]A^A_ÐUHSHHx HtHk HHHuH[ÐH/Hlbfgs_end must be int32;0zRx $AC P~  Ho8`  @` ooooopo .>N0 GCC: (Ubuntu 4.4.3-4ubuntu5) 4.4.3.symtab.strtab.shstrtab.note.gnu.build-id.gnu.hash.dynsym.dynstr.gnu.version.gnu.version_d.gnu.version_r.rela.dyn.rela.plt.init.text.fini.rodata.eh_frame_hdr.eh_frame.ctors.dtors.jcr.dynamic.got.got.plt.data.bss.comment$2D.o88(8 `` @HoppUo8do s`}@@ p``HH2VVppD      H0 08 808#[3 8`p  @   `HVp      0 8  ` * 8 E [8 j@ x    H 0  8 H 8  $ *2 A UY"u ,call_gmon_startcrtstuff.c__CTOR_LIST____DTOR_LIST____JCR_LIST____do_global_dtors_auxcompleted.7382dtor_idx.7384frame_dummy__CTOR_END____FRAME_END____JCR_END____do_global_ctors_auxlbfgsAddC.c_fini_GLOBAL_OFFSET_TABLE___dso_handle__DTOR_END____bss_start_end_edata_DYNAMIC_initmxGetPr__gmon_start___Jv_RegisterClassesMEX__cxa_finalize@@GLIBC_2.2.5mxGetDimensions_700mxIsClassmexFunctionmexErrMsgTxtmxGetScalargpml-matlab-v4.0-2016-10-19/util/minfunc/compiled/mcholC.mexw32000644 000765 000024 00000020000 12701705601 023532 0ustar00carlstaff000000 000000 MZ@ !L!This program cannot be run in DOS mode. $PELK`J!  H0p`T@P.textp p  `.rdataT T @.dataL0L@.idata@`.relocP .edataT`T @1@L$AtD$T$SVWD$Pjhd5d%D$ Xp t ;t$$t4v L$H |uTd _^[USVWUjjhu` ]_^[]USVWU] E@umEEEECs {tb v|t:VUkT]^] t(x1{SkVS vC T{ v41UkjS j  ]_^[]USVW} u"} uuu u0_^[ U Wjj Ejjy Ejjm Eh:0uQ 0h80u? 0h80u- 0 0=0 t jW[ = 0 tjWF Z_á 0 tP Y0 tP Y0 tP YU}fE fEm]mEUSVWEo É]!<=tE1ٍAG;<uՋ}G<}uc YE0}u1v]T1ٍAG}<=t0u% Yu5 u:S}4=@ EU<uU_^[U] UE]{EE]UQPE]E]A{EEUQPSVW} u]}u ]}uEn_^[UQPSVW} u]}u ]}uE_^[USVWEH} uuu]}}u] Uދ}uuuEE}9}|_^[USVWEO}u uu]}u]U ։ыUˋ}uuuEE}9}|_^[USVW}4=E}u D0]}w]E]}4=|<}}}}}jj}Wj0u 5} 4=DE}}Ejj}Wju F} wEjj }Wju F} wEj}VEj}V}EE=}uE}9}|كE}9}|]E1}uu}uu];>E}9}|<0]<0]E}uu$]E$E$_ݝx݅x]E^}9}tR}uuu$;ݝp݅p$E$ݝh݅h]E}9}|E}9}6$EE$]EM]}}WW$$zݝxEܵx$E$\ݝp݅p$E$9ݝh݅h$ݝ`݅`]CE$E$ݝx݅x$ݝp݅p]E<0]E}}}uu$ݝxEܝx{W}uu$jݝp݅p$E؃$+ݝh݅h]؋}}ЃE}9}buuut uuu( uuuuuuuuuuuu[uuuuuuuu3uuuuE'}u]]U2U43E}9}~΋}}p}uuu];>E=}u]Uˉȍ ] ދE  E}9}~E}9}|]ȋ}9}}i}}V}uuu$ݝx݅x$Eȃ$Pݝp݅p]ȃE}9}|}uuރ$0ݝxEݝpEݝh݅p܍p݅h܍h$E$ݝ`݅`$݅x$ݝXu݅X}9}}_}}L}u]׉эˋMݝP݅P܍P}4E}9}|E}9}ZE}u<=E}9}|u,u!_^[%@%@%@%@%@%@%A% A%A%A% A%$A%(A%,A%0A%4A%8A% // Function definitions. // ----------------------------------------------------------------- // Copy a C-style string (a null-terminated character array) to a // non-C-style string (a simple character array). The length of the // destination character array is given by "ndest". If the source is // shorter than the destination, the destination is padded with blank // characters. void copyCStrToCharArray (const char* source, char* dest, int ndest) { // Get the length of the source C string. int nsource = strlen(source); // Only perform the copy if the source can fit into the destination. if (nsource < ndest) { // Copy the string. strcpy(dest,source); // Fill in the rest of the string with blanks. for (int i = nsource; i < ndest; i++) dest[i] = ' '; } } // Return true if the two strings are the same. The second input // argument must be a C-style string (a null-terminated character // array), but the first input argument need not be one. We only // compare the two strings up to the length of "cstr". bool strIsEqualToCStr (const char* str, const char* cstr) { // Get the length of the C string. int n = strlen(cstr); return !strncmp(str,cstr,n); } // Function definitions for class Program. // ----------------------------------------------------------------- Program::Program (int n, double* x, double* lb, double* ub, int* btype, int m, int maxiter, double factr, double pgtol) { this->n = n; this->x = x; this->lb = lb; this->ub = ub; this->btype = btype; this->m = m; this->maxiter = maxiter; this->factr = factr; this->pgtol = pgtol; this->iprint = defaultprintlevel; this->owner = false; initStructures(); } Program::Program (int n, int m, int maxiter, double factr, double pgtol) { this->n = n; this->x = new double[n]; this->lb = new double[n]; this->ub = new double[n]; this->btype = new int[n]; this->m = m; this->maxiter = maxiter; this->factr = factr; this->pgtol = pgtol; this->iprint = defaultprintlevel; this->owner = true; initStructures(); } Program::~Program() { if (owner) { delete[] x; delete[] lb; delete[] ub; delete[] btype; } delete[] g; delete[] wa; delete[] iwa; } SolverExitStatus Program::runSolver() { SolverExitStatus status = success; // The return value. // Initialize the objective function and gradient to zero. f = 0; for (int i = 0; i < n; i++) g[i] = 0; // This initial call sets up the structures for L-BFGS. callLBFGS("START"); // Repeat until we've reached the maximum number of iterations. int t = 0; while (true) { // Do something according to the "task" from the previous call to // L-BFGS. if (strIsEqualToCStr(task,"FG")) { // Evaluate the objective function and the gradient of the // objective at the current point. f = computeObjective(n,x); computeGradient(n,x,g); } else if (strIsEqualToCStr(task,"NEW_X")) { // Go to the next iteration and call the iterative callback // routine. t++; iterCallback(t,x,f); // If we've reached the maximum number of iterations, terminate // the optimization. if (t == maxiter) { callLBFGS("STOP"); break; } } else if (strIsEqualToCStr(task,"CONV")) break; else if (strIsEqualToCStr(task,"ABNO")) { status = abnormalTermination; break; } else if (strIsEqualToCStr(task,"ERROR")) { status = errorOnInput; break; } // Call L-BFGS again. callLBFGS(); } return status; } void Program::callLBFGS (const char* cmd) { if (cmd) copyCStrToCharArray(cmd,task,60); setulb_(&n,&m,x,lb,ub,btype,&f,g,&factr,&pgtol,wa,iwa,task,&iprint, csave,lsave,isave,dsave); } void Program::initStructures() { f = 0; g = new double[n]; wa = new double[(2*m + 5)*n + 12*m*(m + 1)]; iwa = new int[3*n]; } gpml-matlab-v4.0-2016-10-19/util/lbfgsb/matlabstring.cpp000644 000765 000024 00000002032 12701705601 022462 0ustar00carlstaff000000 000000 #include #include "matlabstring.h" #include "matlabexception.h" // Function definitions. // ----------------------------------------------------------------- char* copystring (const char* source) { int n = strlen(source); // The length of the string. char* dest = new char[n+1]; // The return value. strcpy(dest,source); return dest; } // Function definitions for class MatlabString. // ----------------------------------------------------------------- MatlabString::MatlabString (const mxArray* ptr) { s = 0; // Check to make sure the Matlab array is a string. if (!mxIsChar(ptr)) throw MatlabException("Matlab array must be a string (of type CHAR)"); // Get the string passed as a Matlab array. s = mxArrayToString(ptr); if (s == 0) throw MatlabException("Unable to obtain string from Matlab array"); } MatlabString::MatlabString (const MatlabString& source) { s = 0; // Copy the source string. s = copystring(source.s); } MatlabString::~MatlabString() { if (s) mxFree(s); } gpml-matlab-v4.0-2016-10-19/util/lbfgsb/LICENSE000644 000765 000024 00000077330 12701705601 020311 0ustar00carlstaff000000 000000 GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS gpml-matlab-v4.0-2016-10-19/util/lbfgsb/matlabexception.h000644 000765 000024 00000001545 12701705601 022627 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABEXCEPTION #define INCLUDE_MATLABEXCEPTION #include // Class MatlabException // ----------------------------------------------------------------- // This class just makes it easier for me to throw exceptions. Its // functionality really has nothing to do with MATLAB. class MatlabException : public std::exception { public: MatlabException (const char* message) throw(); ~MatlabException() throw() { }; // The copy constructor makes a shallow copy. MatlabException (const MatlabException& source) throw(); // The copy assignment operator makes a shallow copy as well. MatlabException& operator= (const MatlabException& source); // Return the message string. virtual const char* what () const throw() { return message; }; private: const char* message; // The error message. }; #endif gpml-matlab-v4.0-2016-10-19/util/lbfgsb/Makefile000644 000765 000024 00000006734 12701705601 020744 0ustar00carlstaff000000 000000 # for 'make nomex' you need to provide the variables MEX_SUFFIX and MATLAB_LIB # for 'make mex' you need to provide the variable MEX and have mex set up # for 'make oct' you need to provide the variable OCT and OCTAVE_INCLUDE ### 1) Choose Matlab configuration ############################################ ### path to your Matlab installation ### you can find it by the commands 'locate matlab' or 'find / -name "matlab"' MATLAB_HOME=/usr/local/bin/matlab ### For 32 bit Linux, uncomment the following three lines. -------------------- MEX = $(MATLAB_HOME)/bin/mex MEX_SUFFIX = mexglx MATLAB_LIB = -L$(MATLAB_HOME)/bin/glnx86 -lmex ### For 64 bit Linux, uncomment the following three lines. -------------------- #MEX = $(MATLAB_HOME)/bin/mex #MEX_SUFFIX = mexa64 #MATLAB_LIB = -L$(MATLAB_HOME)/bin/glnxa64 -lmex ### For Mac OS X, uncomment the following three lines. ------------------------ #MEX = /Applications/MATLAB7/bin/mex #MEX_SUFFIX = mexmac #MATLAB_LIB = -L$(MATLAB_HOME)/bin/maci -lmex ### 2) Choose Octave configuration ############################################ OCTAVE_INCLUDE = /usr/local/include/octave-3.2.4/octave OCT = mkoctfile ### 3) Choose L-BFGS-B configuration ########################################## ### Uncomment the following 2 lines for LBFGSB 2.4. --------------------------- LBFGSB_OBJS = solver_2_4.o LIBS = ### Uncomment the following 2 lines for LFBGSB 3.0 with blas libs provided. --- #LBFGSB_OBJS = solver_3_0.o linpack.o timer.o blas.o #LIBS = ### Uncomment the following 2 lines for LFBGSB 3.0 own blas libs. ------------- #LBFGSB_OBJS = solver_3_0.o linpack.o timer.o #LIBS = -lblas ### 4) Choose FORTRAN compiler ################################################ FFLAGS = -O3 -fPIC -fexceptions -Wall -g -Wno-uninitialized ### To work with f77, uncomment the following two lines. ---------------------- #F77 = f77 #FORTRAN_LIB = g2c ### To use gfortran, uncomment the following two lines. ----------------------- F77 = gfortran FORTRAN_LIB = gfortran ### 5) Choose C++ compiler #################################################### CXX = g++ CFLAGS = -O3 -fPIC -pthread -Wall -Werror -ansi -ffast-math -fomit-frame-pointer ### 6) If you need to specify extra directories where to look for libraries, ### write the paths of the directories, separated by a white space LIBDIRS = #LIBDIRS = ./ /usr/lib/gcc/x86_64-linux-gnu/4.8/ ############################################################################### ### Do not edit below ######################################################### ############################################################################### TARGET = lbfgsb OBJS = $(LBFGSB_OBJS) matlabexception.o matlabscalar.o matlabstring.o \ matlabmatrix.o arrayofmatrices.o program.o matlabprogram.o \ lbfgsb.o %.o: %.cpp $(CXX) $(CFLAGS) -I$(OCTAVE_INCLUDE) -I$(MATLAB_HOME)/extern/include -o $@ -c $^ %.o: %.f $(F77) $(FFLAGS) -o $@ -c $^ all: nomex mex: $(TARGET)_mex copy tidy nomex: $(TARGET)_nomex copy tidy octave: $(TARGET)_octave copy tidy oct: octave $(TARGET)_mex: $(OBJS) $(MEX) -cxx CXX=$(CXX) CC=$(CXX) FC=$(FCC) LD=$(CXX) -l$(FORTRAN_LIB) -lm \ -O -output $(TARGET) $^ $(TARGET)_nomex: $(OBJS) $(CXX) $^ -shared -o $(TARGET).$(MEX_SUFFIX) $(MATLAB_LIB) $(addprefix -L,$(LIBDIRS)) -l$(FORTRAN_LIB) -lm $(TARGET)_octave: $(OBJS) $(OCT) --mex -o $(TARGET).mex $^ $(LIBS)$ copy: mv $(TARGET).mex* .. tidy: rm -f *.o clean: tidy rm -f ../$(TARGET).mex* gpml-matlab-v4.0-2016-10-19/util/lbfgsb/lbfgsb.cpp000644 000765 000024 00000010024 12701705601 021232 0ustar00carlstaff000000 000000 #include "mex.h" #ifndef __linux__ #include "matrix.h" #endif #include "arrayofmatrices.h" #include "matlabexception.h" #include "matlabscalar.h" #include "matlabstring.h" #include "matlabmatrix.h" #include "arrayofmatrices.h" #include "program.h" #include "matlabprogram.h" #include #include extern void _main(); // Constants. // ----------------------------------------------------------------- const int minNumInputArgs = 5; // Function definitions. // ----------------------------------------------------------------- void mexFunction (int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) try { // Check to see if we have the correct number of input and output // arguments. if (nrhs < minNumInputArgs) throw MatlabException("Incorrect number of input arguments"); // Get the starting point for the variables. This is specified in // the first input argument. The variables must be either a single // matrix or a cell array of matrices. int k = 0; // The index of the current input argument. ArrayOfMatrices x0(prhs[k++]); // Create the output, which stores the solution obtained from // running IPOPT. There should be as many output arguments as cell // entries in X. if (nlhs != x0.length()) throw MatlabException("Incorrect number of output arguments"); ArrayOfMatrices x(plhs,x0); // Load the lower and upper bounds on the variables as // ArrayOfMatrices objects. They should have the same structure as // the ArrayOfMatrices object "x". ArrayOfMatrices lb(prhs[k++]); ArrayOfMatrices ub(prhs[k++]); // Check to make sure the bounds make sense. if (lb != x || ub != x) throw MatlabException("Input arguments LB and UB must have the same \ structure as X"); // Get the Matlab callback functions. MatlabString objFunc(prhs[k++]); MatlabString gradFunc(prhs[k++]); // Get the auxiliary data. const mxArray* auxData; const mxArray* ptr = prhs[k++]; if (nrhs > 5) { if (mxIsEmpty(ptr)) auxData = 0; else auxData = ptr; } else auxData = 0; // Get the intermediate callback function. MatlabString* iterFunc; ptr = prhs[k++]; if (nrhs > 6) { if (mxIsEmpty(ptr)) iterFunc = 0; else iterFunc = new MatlabString(ptr); } else iterFunc = 0; // Set the options for the L-BFGS algorithm to their defaults. int maxiter = defaultmaxiter; int m = defaultm; double factr = defaultfactr; double pgtol = defaultpgtol; // Process the remaining input arguments, which set options for // the IPOPT algorithm. while (k < nrhs) { // Get the option label from the Matlab input argument. MatlabString optionLabel(prhs[k++]); if (k < nrhs) { // Get the option value from the Matlab input argument. MatlabScalar optionValue(prhs[k++]); double value = optionValue; if (!strcmp(optionLabel,"maxiter")) maxiter = (int) value; else if (!strcmp(optionLabel,"m")) m = (int) value; else if (!strcmp(optionLabel,"factr")) factr = value / mxGetEps(); else if (!strcmp(optionLabel,"pgtol")) pgtol = value; else { if (iterFunc) delete iterFunc; throw MatlabException("Nonexistent option"); } } } // Create a new instance of the optimization problem. x = x0; MatlabProgram program(x,lb,ub,&objFunc,&gradFunc,iterFunc, (mxArray*) auxData,m,maxiter,factr,pgtol); // Run the L-BFGS-B solver. SolverExitStatus exitStatus = program.runSolver(); if (exitStatus == abnormalTermination) { if (iterFunc) delete iterFunc; throw MatlabException("Solver unable to satisfy convergence \ criteria due to abnormal termination"); } else if (exitStatus == errorOnInput) { if (iterFunc) delete iterFunc; throw MatlabException("Invalid inputs to L-BFGS routine"); } // Free the dynamically allocated memory. if (iterFunc) delete iterFunc; } catch (std::exception& error) { mexErrMsgTxt(error.what()); } gpml-matlab-v4.0-2016-10-19/util/lbfgsb/matlabstring.h000644 000765 000024 00000002200 12701705601 022124 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABSTRING #define INCLUDE_MATLABSTRING #include "mex.h" #include // Function declarations. // ----------------------------------------------------------------- // Copy a C-style string (i.e. a null-terminated character array). char* copystring (const char* source); // Class MatlabString. // ----------------------------------------------------------------- // This class encapsulates read-only access to a MATLAB character // array. class MatlabString { public: // The constructor accepts as input a pointer to a Matlab array, // which must be a valid string (array of type CHAR). explicit MatlabString (const mxArray* ptr); // The copy constructor makes a full copy of the source string. MatlabString (const MatlabString& source); // The destructor. ~MatlabString(); // Conversion operator for null-terminated string. operator const char* () const { return s; }; protected: char* s; // The null-terminated string. // The copy assignment operator is not proper, thus remains // protected. MatlabString& operator= (const MatlabString& source) { return *this; }; }; #endif gpml-matlab-v4.0-2016-10-19/util/lbfgsb/matlabprogram.h000644 000765 000024 00000005501 12701705601 022274 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABPROGRAM #define INCLUDE_MATLABPROGRAM #include "mex.h" #include "program.h" #include "matlabscalar.h" #include "matlabstring.h" #include "arrayofmatrices.h" // Class MatlabProgram. // ----------------------------------------------------------------- // This is an implementation of the abstract class Program. class MatlabProgram: public Program { public: // On input, "variables" should contain the initial point for the // optimization routine. If no iterative callback function is // specified, "iterFunc" may be set to 0. Also, if no auxiliary data // is needed, it may also be set to 0. MatlabProgram (ArrayOfMatrices& variables, const ArrayOfMatrices& lowerbounds, const ArrayOfMatrices& upperbounds, const MatlabString* objFunc, const MatlabString* gradFunc, const MatlabString* iterFunc, mxArray* auxData, int m = defaultm, int maxiter = defaultmaxiter, double factr = defaultfactr, double pgtol = defaultpgtol); // The destructor. virtual ~MatlabProgram(); // These provide definitions for the pure virtual functions of the // abstract parent class. virtual double computeObjective (int n, double* x); virtual void computeGradient (int n, double* x, double* g); virtual void iterCallback (int t, double* x, double f); // Run the solver. Upon completion, the solution is stored in // "variables". SolverExitStatus runSolver(); protected: ArrayOfMatrices& variables; // Storage for the initial value and // solution. const MatlabString* objFunc; // The objective callback function. const MatlabString* gradFunc; // The gradient callback function. const MatlabString* iterFunc; // The iterative callback function. ArrayOfMatrices* varMatlab; // Inputs to the Matlab callback mxArray** varInputs; // functions representing the // current values of the variables. MatlabScalar* fMatlab; // Input to the Matlab callback // functions representing the // current value of the objective. MatlabScalar* tMatlab; // Input to the Matlab callback // functions representing the // current iteration. int numInputsObjFunc; // The number of inputs passed to the // objective callback function. int numInputsGradFunc; // The number of inputs passed to the // gradient callback function. int numInputsIterFunc; // The number of inputs passed to the // iterative callback function. mxArray** inputsObjFunc; // The inputs passed to the objective // callback function. mxArray** inputsGradFunc; // The inputs passed to the gradient // callback function. mxArray** inputsIterFunc; // The inputs passed to the iterative // callback function. }; #endif gpml-matlab-v4.0-2016-10-19/util/lbfgsb/blas.f000644 000765 000024 00000014162 12701705601 020366 0ustar00carlstaff000000 000000 c c L-BFGS-B is released under the “New BSD License” (aka “Modified BSD c License” or “3-clause license”) c Please read attached file License.txt c double precision function dnrm2(n,x,incx) integer n,incx double precision x(n) c ********** c c Function dnrm2 c c Given a vector x of length n, this function calculates the c Euclidean norm of x with stride incx. c c The function statement is c c double precision function dnrm2(n,x,incx) c c where c c n is a positive integer input variable. c c x is an input array of length n. c c incx is a positive integer variable that specifies the c stride of the vector. c c Subprograms called c c FORTRAN-supplied ... abs, max, sqrt c c MINPACK-2 Project. February 1991. c Argonne National Laboratory. c Brett M. Averick. c c ********** integer i double precision scale dnrm2 = 0.0d0 scale = 0.0d0 do 10 i = 1, n, incx scale = max(scale, abs(x(i))) 10 continue if (scale .eq. 0.0d0) return do 20 i = 1, n, incx dnrm2 = dnrm2 + (x(i)/scale)**2 20 continue dnrm2 = scale*sqrt(dnrm2) return end c====================== The end of dnrm2 =============================== subroutine daxpy(n,da,dx,incx,dy,incy) c c constant times a vector plus a vector. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),da integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if (da .eq. 0.0d0) return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dy(iy) + da*dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,4) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dy(i) + da*dx(i) 30 continue if( n .lt. 4 ) return 40 mp1 = m + 1 do 50 i = mp1,n,4 dy(i) = dy(i) + da*dx(i) dy(i + 1) = dy(i + 1) + da*dx(i + 1) dy(i + 2) = dy(i + 2) + da*dx(i + 2) dy(i + 3) = dy(i + 3) + da*dx(i + 3) 50 continue return end c====================== The end of daxpy =============================== subroutine dcopy(n,dx,incx,dy,incy) c c copies a vector, x, to a vector, y. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*) integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,7) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dx(i) 30 continue if( n .lt. 7 ) return 40 mp1 = m + 1 do 50 i = mp1,n,7 dy(i) = dx(i) dy(i + 1) = dx(i + 1) dy(i + 2) = dx(i + 2) dy(i + 3) = dx(i + 3) dy(i + 4) = dx(i + 4) dy(i + 5) = dx(i + 5) dy(i + 6) = dx(i + 6) 50 continue return end c====================== The end of dcopy =============================== double precision function ddot(n,dx,incx,dy,incy) c c forms the dot product of two vectors. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),dtemp integer i,incx,incy,ix,iy,m,mp1,n c ddot = 0.0d0 dtemp = 0.0d0 if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dtemp = dtemp + dx(ix)*dy(iy) ix = ix + incx iy = iy + incy 10 continue ddot = dtemp return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dtemp = dtemp + dx(i)*dy(i) 30 continue if( n .lt. 5 ) go to 60 40 mp1 = m + 1 do 50 i = mp1,n,5 dtemp = dtemp + dx(i)*dy(i) + dx(i + 1)*dy(i + 1) + * dx(i + 2)*dy(i + 2) + dx(i + 3)*dy(i + 3) + dx(i + 4)*dy(i + 4) 50 continue 60 ddot = dtemp return end c====================== The end of ddot ================================ subroutine dscal(n,da,dx,incx) c c scales a vector by a constant. c uses unrolled loops for increment equal to one. c jack dongarra, linpack, 3/11/78. c modified 3/93 to return if incx .le. 0. c double precision da,dx(*) integer i,incx,m,mp1,n,nincx c if( n.le.0 .or. incx.le.0 )return if(incx.eq.1)go to 20 c c code for increment not equal to 1 c nincx = n*incx do 10 i = 1,nincx,incx dx(i) = da*dx(i) 10 continue return c c code for increment equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dx(i) = da*dx(i) 30 continue if( n .lt. 5 ) return 40 mp1 = m + 1 do 50 i = mp1,n,5 dx(i) = da*dx(i) dx(i + 1) = da*dx(i + 1) dx(i + 2) = da*dx(i + 2) dx(i + 3) = da*dx(i + 3) dx(i + 4) = da*dx(i + 4) 50 continue return end c====================== The end of dscal =============================== gpml-matlab-v4.0-2016-10-19/util/lbfgsb/README000644 000765 000024 00000051032 12701705601 020153 0ustar00carlstaff000000 000000 # A MATLAB interface for L-BFGS-B ###Update As of March 24, 2014, the MATLAB code supports the latest version of the L-BFGS-B solver (version 3.0), and is compatible with [GNU Octave](https://www.gnu.org/software/octave). Thank you to [Jose Vallet](http://github.com/josombio) for providing these updates. ###Overview L-BFGS-B is a collection of Fortran 77 routines for solving large-scale nonlinear optimization problems with bound constraints on the variables. One of the key features of the nonlinear solver is that knowledge of the Hessian is not required; the solver computes search directions by keeping track of a quadratic model of the objective function with a limited-memory BFGS (Broyden-Fletcher-Goldfarb-Shanno) approximation to the Hessian *(see Note #1)*. The algorithm was developed by Ciyou Zhu, Richard Byrd and Jorge Nocedal. For more information, go to the [original distribution site](http://www.ece.northwestern.edu/~nocedal/lbfgsb.html) for the L-BFGS-B software package. I've designed an interface to the L-BFGS-B solver so that it can be called like any other function in MATLAB *(see Note #2)*. See the text below for more information on installing and calling this function in MATLAB. Along the way, I've also developed a C++ class that encapsulates all the "messy" details in executing the L-BFGS-B code. See below for instructions on how to incorporate this C++ class info your own code. This code has been tested in MATLAB and Octave on Linux, and is integrated into [GPML](http://www.gaussianprocess.org/gpml/code). It has also been executed successfully in Windows; see below. ###License Copyright (c) 2013, 2014 Peter Carbonetto The lbfgsb-matlab project repository is free software: you can redistribute it and/or modify it under the terms of the [GNU General Public License](http://www.gnu.org/licenses/gpl.html) as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but **without any warranty**; without even the implied warranty of **merchantability** or **fitness for a particular purpose**. See [LICENSE](LICENSE) for more details. ###Installation Follow these steps to compile and install **lbfgsb-matlab** for MATLAB and Octave. Some of these steps are applicable to MATLAB or Octave only, and you can skip them if you want to install the interface for only one of them. These installation instructions assume you have Linux, although similar steps should also work for other UNIX-based operating systems, such as Mac OS X. These instructions also assume you have [GNU Make](http://www.gnu.org/software/make) installed. (We have successfully compiled this source code with gcc versions 4.6.3 and 4.6.4, under Ubuntu Linux 12.10 and 13.10, and have successfully run the MATLAB scripts in Octave 3.6.2, 3.6.4, 3.8.0 and 3.8.1.) **For Windows users:** If you have a Windows operating system, Guillaume Jacquenot has generously provided [instructions](INSTALL.WINDOWS.txt) for compiling the software on Windows with [gnumex](http://gnumex.sourceforge.net) and Mingw. These instructions are slightly out of date, and suggestions for improving these installation instructions are welcome. We will create a MEX file, which is basically a file that contains a routine that can be called from MATLAB as if it were a built-it function. To learn about MEX files, I refer you to [this document](http://www.mathworks.com/support/tech-notes/1600/1605.html) on the MathWorks website. **Download the source code.** Clone or fork the this repository, or download this repository as a [ZIP file](http://github.com/pcarbo/lbfgsb-matlab/archive/master.zip) and unpack the ZIP file. The [src](src) subdirectory contains some MATLAB files (ending in .m), some C++ source and header files (ending in .h and .cpp), a single Fortran 77 source file, [solver.f](src/solver.f), containing the L-BFGS-B routines, and a [Makefile](src/Makefile). I've included a version of the Fortran routines that is more recent than what is available for download at the [distribution site](http://www.ece.northwestern.edu/~nocedal/lbfgsb.html) at Northwestern University. In the following, we will refer to the directory containing your local copy of the repository as LBFGSB_HOME. Wherever you find LBFGSB_HOME, you should substitute it with your own directory. **Install the C++ and Fortran 77 compilers.** In order to build the MEX file, you will need both a C++ compiler and a Fortran 77 compiler. Unfortunately, you can't just use any compiler. You have to use the precise one supported by MATLAB. For instance, if you are running Mac OS X 7.2 on a Linux operating system, you will need to install [GNU compiler collection](http://www.gnu.org/software/gcc) (GCC) version 3.4.4. Even if you already have a compiler installed on Linux, it may be the wrong version, and if you use the wrong version things could go horribly wrong. It is important that you use the correct version of the compiler, otherwise you will encounter linking errors. Refer to [this document](http://www.mathworks.com/support/tech-notes/1600/1601.html) to find out which compiler is supported by your version of MATLAB. You may also be able to use the **apt-get** program to install gfortran, as in: sudo apt-get install build-essential gfortran **Install Octave.** You will of course need a working installation of Matlab or Octave. In the case of Octave, if you don't have it the easiest way to install it is from the repositories using **apt-get**: sudo apt-get install octave If you want to use a more recent version of Octave, you will have to [install Octave from the source code](http://www.gnu.org/software/octave/doc/interpreter/Installation.html). After that, find the path to the mkoctfile compiler (in the bin directory) and the include directory. You will need those paths for steps below. Additionally you will need the Octave development tools, libs and header files to compile MEX files: sudo apt-get install liboctave-dev **Configure MATLAB.** Next, you need to set up and configure MATLAB to build MEX Files. This is explained quite nicely in [this MathWorks support document](http://www.mathworks.com/support/tech-notes/1600/1605.html). **Simple compilation procedure for MATLAB.** (Thanks to Stefan Harmeling.) You might be able to build and compile the MEX File from source code simply by calling MATLAB's mex program with the C++ and Fortran source files as inputs, like so: mex -output lbfgsb *.cpp solver.f If that doesn't work (e.g. you get linking errors, or it makes MATLAB crash when you try to call it), then you may have to follow the more complicated installation instructions below. **Modify the Makefile.** You are almost ready to build the MEX file. But before you do so, you need to edit the Makefile to coincide with your system setup. Edit the Makefile located inside the LBFGSB_HOME/src directory, following the instructions provided in this file. With this Makefile you may create: - A mex file for Matlab using the mex compiler (mex taget). - A mex file for Matlab using your C compiler (nomex target). - A mex file for Octave using mkoctfile (oct target). Regarding the compilation for Octave, the variable **OCTAVE_INCLUDE** is the directory where the Octave header files required for development are installed. Make sure that it points to the correct directory corresponding to your Octave installation. **OCT** is the command to call the mkoctfile compiler. If mkoctfile is in the path, simply leave it as it is. If not, include in the variable the full path to it. If you have several versions of Octave installed, make sure that OCT points to the mkoctfile program that corresponds to the Octave version you would like to use. For MATLAB, the variable **MEX** is the executable called to build the MEX file. The variables **CXX** and **F77** must be the names of your C++ and Fortran 77 compilers. **MEXSUFFIX** is the MEX file extension specific to your platform. (For instance, the extension for Linux is **mexglx**.) The variable **MATLAB_HOME** must be the base directory of your MATLAB installation. Finally, **CFLAGS** and **FFLAGS** are options passed to the C++ and Fortran compilers, respectively. Often these flags coincide with your MEX options file (see [here](http://www.mathworks.com/support/tech-notes/1600/1605.html) for more information on the options file). But often the settings in the MEX option file are incorrect, and so the options must be set manually. MATLAB requires some special compilation flags for various reasons, one being that it requires position-independent code. These instructions are vague (I apologize), and this step may require a bit of trial and error until before you get it right. **INSTALLDIR** is the installation directory relative to the LBFGSB_HOME/src directory (see Installation below). It may be helpful to look at the GCC documentation in order to understand what these various compiler flags mean. **Build the MEX file.** If you are in the directory containing all the source files, calling **make** at the command prompt will first compile the Fortran and C++ source files into object code (.o files). After that, the make program calls the MEX script, which links all the object files together into a single MEX file. Call **make** from the LBFGSB_HOME/src directory using the appropriate target, one of: make oct make nomex make mex If you didn't get any errors, then you are ready to try out the bound-constrained solver in MATLAB or Octave. Note that even if you didn't get any errors, there's still a possibility that you didn't link the MEX file properly, in which case executing the MEX file will cause MATLAB to crash. But there's only one way to find out: the hard way. **Final installation steps.** Installing the mex file is as simple as placing it wherever MATLAB or Octave can find it; that is, in its "path". There are many ways to do this. Whatever approach you take, note that the mex file does not contain the usage documentation, which is provided in lbfgsb.m. So wherever you place lbfgsb.mex, make sure that you include lbfgsb.m in the same directory. Perhaps the simplest approach to copy lbfgsb.mex and lbfgsb.m to a a specific folder. Alternatively, make install will create the .mex and .m files in the INSTALLDIR directory, and this directory will be automatically created if it does not exist. Once you are in MATLAB or Octave, add the installation directory to the MATLAB or Octave path can be done using the command **addpath**. Note that once you exit MATLAB or Octave the updated path is not preserved, and you will need to use addpath in every new session. You can add the addpath command to ~/.octaverc or ~/matlab/startup. In this way, the path will automatically be updated every time you start a new session. ###A brief tutorial I've written a short script [examplehs038.m](src/examplehs038.m) which demonstrates how to call **lbfgsb** for solving a very small optimization problem, the Hock & Schittkowski test problem #38 **(Note #3)**. The optimization problem has 4 variables. They are bounded from below by -10 and from above by +10. We've set the starting point to (-3, -1, -3, -1). We've written two MATLAB functions for computing the objective function and the gradient of the objective function at a given point. These functions can be found in the files [computeObjectiveHS038.m](src/computeObjectiveHS038.m) and [computeGradientHS038.m](src/computeGradientHS038.m). If you run the script, the solver should very quickly progress toward the point (1, 1, 1, 1). The objective is 0 at this point. The basic MATLAB function call is lbfgsb(x0,lb,ub,objfunc,gradfunc) The first input argument **x0** declares the starting point for the solver. The second and third input arguments define the lower and upper bounds on the variables, respectively. If an entry of **lb** is set to **-Inf**, then this is equivalent to no lower bound for that entry (the same applies to upper bounds, except that it must be **+Inf** instead). The next two input arguments must be the names of MATLAB routines (M-files). The first routine must calculate the objective function at the current point. The output must be a scalar representing the objective evaluated at the current point. The second function must compute the gradient of the objective at the current point. The input is the same as **objfunc**, but it must return as many outputs as there are inputs. For a complete guide on using the MATLAB interface, type **help lbfgsb** in MATLAB. The script [exampleldaimages.m](src/exampleldaimages.m) is a more complicated example. It uses the L-BFGS-B solver to compute some statistics of a posterior distribution for a text analysis model called latent Dirichlet allication (LDA). The model is used to estimate the topics for a collection of documents. The optimization problem is to compute an approximation to the posterior, since it is not possible to compute the posterior exactly. In this case, the optimization problem corresponds to minimizing the distance between the true posterior distribution and an approximate distribution. For more information, see the paper of Blei, Ng and Jordan, 2003 **(Note #4)**. In order to run this example, you will need to install the [lightspeed](http://research.microsoft.com/~minka/software/lightspeed) for MATLAB developed by Tom Minka at Microsoft Research. The script starts by generating a bunch of canonical images that represent topics; if a pixel is white then a word at that position is more likely to appear in that topic. Note that the order of the pixels in the image is unimportant, since LDA is a "bag of words" model. In another figure we show a small sample of the data generated from the topics. Each image represents a document, and the pixels in the image depict the word proportions in that document **(Note #5)**. Next, the script runs the L-BFGS-B solver to find a local minimum to the variational objective. Buried in the M-file [mflda.m](src/mflda.m) is the call to the solver: [xi gamma Phi] = lbfgsb({xi gamma Phi},... {repmat(lb,W,K) repmat(lb,K,D) repmat(lb,K,M)},... {repmat(inf,W,K) repmat(inf,K,D) repmat(inf,K,M)},... 'computeObjectiveMFLDA','computeGradientMFLDA',... {nu eta L w lnZconst},'callbackMFLDA',... 'm',m_lbfgs,'factr',1e-12,'pgtol',... tolerance,'maxiter',maxiter); This example will give us the opportunity to demonstrate some of the more complicated aspects of the our MATLAB interface. The first input to **lbfgsb** sets the initial iterate of the optimization algorithm. Notice that we've passed a cell array with entries that are matrices. Indeed, it is possible to pass either a matrix or a cell array. Whatever structure is passed, the other input arguments (and outputs) must also abide by this structure. The lower and upper bounds on variables are also cell arrays. All the upper bounds are set to infinity, which means that the variables are only bounded from below. It is instructive to examine the callback routines. They look like: f = computeObjectiveMFLDA(xi,gamma,Phi,auxdata) [dxi, dgamma, dPhi] = ... computeGradientMFLDA(xi,gamma,Phi,auxdata) Notice that each entry to the cell array is its own input argument to the callback routines. The same applies for the output arguments of the gradient callback function. The sixth input argument is a cell array that contains auxiliary data. This data will be passed to the MATLAB functions that evaluate the objective and gradient. The seventh input argument specifies a callback routine that is called exactly once for every iteration. This callback routine is most useful for examining the progress of the solver. The remaining input arguments are label/value pairs that override some of the default settings of L-BFGS-B. After converging to a solution, we display topic samples drawn from the variational approximation. A good solution should come close to recovering the true topics from which the data was generated (although they might not be in the same order). Don't be surprised if it takes many iterations before the optimization algorithm converges on a local minimum. (In repeated trials, I found it as many as 2000 iterations to converge to a stationary point of the objective.) This particular problem demonstrates the limitations of limited-memory quasi-Newton approximations to the Hessian; the low storage requirements can come at the cost of slow convergence to the solution. ###The C++ interface One of the nice byproducts of writing a MATLAB interface for L-BFGS-B is that I've ended up with a neat little C++ class that encapsulates the nuts and bolts of executing the solver. A brief description of the Program class can be found in the header file [program.h](src/program.h). "Program" is an abstract class, which means that its impossible to instantiate a Program object. This is because the class has member functions that aren't yet implemented. These are called pure virtual functions. In order to use the Program class, one needs to define another class that inherits the Program class and that implements the pure virtual functions. The class MatlabProgram (in [matlabprogram.h](src/matlabprogram.h)) is an example of such a class. The new child class must declare and implement these two functions: virtual double computeObjective (int n, double* x); virtual void computeGradient (int n, double* x, double* g); See the header file for a detailed description of these functions. The only remaining detail is calling the Program constructor. After that, it is just a matter of declaring a new object of type MyProgram then calling the function runSolver. ###Overview of source files solver.f
Fortran 77 source for the L-BFGS-B solver routines. program.h
program.cpp
 Header and source file for the C++ interface. array.h
arrayofmatrices.h
matlabexception.h
matlabmatrix.h
matlabprogram.h
matlabscalar.h
matlabstring.h
arrayofmatrices.cpp
matlabexception.cpp
matlabmatrix.cpp
matlabprogram.cpp
matlabscalar.cpp
matlabstring.cpp
lbfgsb.cpp
 Header and source files for the MATLAB interface. lbfgsb.m
Usage instructions for the MEX file. examplehs038.m
MATLAB script for solving the H&S test example #38. computeObjectiveHS038.m
computeGradientHS038.m
genericcallback.m
 MATLAB functions used by the script [examplehs038.m](src/examplehs038.m). exampleldaimages.m
MATLAB script that generates synthetic documents and topics, then computes a mean field variational approximation to the posterior distribution of the latent Dirichlet allocation model, then displays the result. mflda.m
This function computes a mean field variational approximation to the posterior distribution of the latent Dirichlet allocation model by minimizing the distance between the variational distribution and the true distribution. It uses the L-BFGS-B to minimize the objective function subject to the bound constraints. The objective function also acts as a lower bound on the logarithm of the denominator that appears from the application of Bayes rule **(Note #4)**. callbackMFLDA.m
computeObjectiveMFLDA.m
computeGradientMFLDA.m
computelnZ.m
computelnZconst.m
computem.m
dirichletrnd.m
gammarnd.m
createsyntheticdata.m
mfldainit.m
 Some functions used by [exampleldaimages.m](src/exampleldaimages.m) and [mflda.m](src/mflda.m). ###Credits MATLAB interface Developed by
[Peter Carbonetto](http://www.cs.ubc.ca/spider/pcarbo)
Dept. of Human Genetics
University of Chicago Support for Octave contributed by
[Jose Vallet](http://github.com/josombio)
School of Electrical Engineering
Aalto University Thank you to Ciyou Zhu, Richard Byrd, Jorge Nocedal and Jose Luis for making their L-BFGS-B Fortran code available, and to Hannes Nickisch for ... ###Notes 1. Ciyou Zhu, Richard H. Byrd, Peihuang Lu and Jorge Nocedal (1997). Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. *ACM Transactions on Mathematical Software* 23(4): 550-560. 2. [Another MATLAB interface](http://www.cs.toronto.edu/~liam/software.shtml) for the L-BFGS-B routines has been developed by Liam Stewart at the University of Toronto. 3. Willi Hock and Klaus Schittkowski (1981). *Test Examples for Nonlinear Programming Codes.* Lecture Notes in Economics and Mathematical Systems, Vol. 187, Springer-Verlag. 4. David M. Blei, Andrew Y. Ng, Michael I. Jordan (2003). Latent Dirichlet allocation. *Journal of Machine Learning Research* Vol3: 993-1022. 5. Tom L. Griffiths and Mark Steyvers (2004). Finding scientific topics. *Proceedings of the National Academy of Sciences* 101: 5228-5235. gpml-matlab-v4.0-2016-10-19/util/lbfgsb/timer.f000644 000765 000024 00000002130 12701705601 020555 0ustar00carlstaff000000 000000 c c L-BFGS-B is released under the “New BSD License” (aka “Modified BSD c License” or “3-clause license”) c Please read attached file License.txt c subroutine timer(ttime) double precision ttime c real temp c c This routine computes cpu time in double precision; it makes use of c the intrinsic f90 cpu_time therefore a conversion type is c needed. c c J.L Morales Departamento de Matematicas, c Instituto Tecnologico Autonomo de Mexico c Mexico D.F. c c J.L Nocedal Department of Electrical Engineering and c Computer Science. c Northwestern University. Evanston, IL. USA c c January 21, 2011 c temp = sngl(ttime) call cpu_time(temp) ttime = dble(temp) return end gpml-matlab-v4.0-2016-10-19/util/lbfgsb/linpack.f000644 000765 000024 00000014274 12701705601 021072 0ustar00carlstaff000000 000000 c c L-BFGS-B is released under the “New BSD License” (aka “Modified BSD c License” or “3-clause license”) c Please read attached file License.txt c subroutine dpofa(a,lda,n,info) integer lda,n,info double precision a(lda,*) c c dpofa factors a double precision symmetric positive definite c matrix. c c dpofa is usually called by dpoco, but it can be called c directly with a saving in time if rcond is not needed. c (time for dpoco) = (1 + 18/n)*(time for dpofa) . c c on entry c c a double precision(lda, n) c the symmetric matrix to be factored. only the c diagonal and upper triangle are used. c c lda integer c the leading dimension of the array a . c c n integer c the order of the matrix a . c c on return c c a an upper triangular matrix r so that a = trans(r)*r c where trans(r) is the transpose. c the strict lower triangle is unaltered. c if info .ne. 0 , the factorization is not complete. c c info integer c = 0 for normal return. c = k signals an error condition. the leading minor c of order k is not positive definite. c c linpack. this version dated 08/14/78 . c cleve moler, university of new mexico, argonne national lab. c c subroutines and functions c c blas ddot c fortran sqrt c c internal variables c double precision ddot,t double precision s integer j,jm1,k c begin block with ...exits to 40 c c do 30 j = 1, n info = j s = 0.0d0 jm1 = j - 1 if (jm1 .lt. 1) go to 20 do 10 k = 1, jm1 t = a(k,j) - ddot(k-1,a(1,k),1,a(1,j),1) t = t/a(k,k) a(k,j) = t s = s + t*t 10 continue 20 continue s = a(j,j) - s c ......exit if (s .le. 0.0d0) go to 40 a(j,j) = sqrt(s) 30 continue info = 0 40 continue return end c====================== The end of dpofa =============================== subroutine dtrsl(t,ldt,n,b,job,info) integer ldt,n,job,info double precision t(ldt,*),b(*) c c c dtrsl solves systems of the form c c t * x = b c or c trans(t) * x = b c c where t is a triangular matrix of order n. here trans(t) c denotes the transpose of the matrix t. c c on entry c c t double precision(ldt,n) c t contains the matrix of the system. the zero c elements of the matrix are not referenced, and c the corresponding elements of the array can be c used to store other information. c c ldt integer c ldt is the leading dimension of the array t. c c n integer c n is the order of the system. c c b double precision(n). c b contains the right hand side of the system. c c job integer c job specifies what kind of system is to be solved. c if job is c c 00 solve t*x=b, t lower triangular, c 01 solve t*x=b, t upper triangular, c 10 solve trans(t)*x=b, t lower triangular, c 11 solve trans(t)*x=b, t upper triangular. c c on return c c b b contains the solution, if info .eq. 0. c otherwise b is unaltered. c c info integer c info contains zero if the system is nonsingular. c otherwise info contains the index of c the first zero diagonal element of t. c c linpack. this version dated 08/14/78 . c g. w. stewart, university of maryland, argonne national lab. c c subroutines and functions c c blas daxpy,ddot c fortran mod c c internal variables c double precision ddot,temp integer case,j,jj c c begin block permitting ...exits to 150 c c check for zero diagonal elements. c do 10 info = 1, n c ......exit if (t(info,info) .eq. 0.0d0) go to 150 10 continue info = 0 c c determine the task and go to it. c case = 1 if (mod(job,10) .ne. 0) case = 2 if (mod(job,100)/10 .ne. 0) case = case + 2 go to (20,50,80,110), case c c solve t*x=b for t lower triangular c 20 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 40 do 30 j = 2, n temp = -b(j-1) call daxpy(n-j+1,temp,t(j,j-1),1,b(j),1) b(j) = b(j)/t(j,j) 30 continue 40 continue go to 140 c c solve t*x=b for t upper triangular. c 50 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 70 do 60 jj = 2, n j = n - jj + 1 temp = -b(j+1) call daxpy(j,temp,t(1,j+1),1,b(1),1) b(j) = b(j)/t(j,j) 60 continue 70 continue go to 140 c c solve trans(t)*x=b for t lower triangular. c 80 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 100 do 90 jj = 2, n j = n - jj + 1 b(j) = b(j) - ddot(jj-1,t(j+1,j),1,b(j+1),1) b(j) = b(j)/t(j,j) 90 continue 100 continue go to 140 c c solve trans(t)*x=b for t upper triangular. c 110 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 130 do 120 j = 2, n b(j) = b(j) - ddot(j-1,t(1,j),1,b(1),1) b(j) = b(j)/t(j,j) 120 continue 130 continue 140 continue 150 continue return end c====================== The end of dtrsl =============================== gpml-matlab-v4.0-2016-10-19/util/lbfgsb/arrayofmatrices.h000644 000765 000024 00000004144 12701705601 022641 0ustar00carlstaff000000 000000 #ifndef INCLUDE_ARRAYOFMATRICES #define INCLUDE_ARRAYOFMATRICES #include "array.h" #include "matlabmatrix.h" #include "mex.h" // Class ArrayOfMatrices. // ----------------------------------------------------------------- class ArrayOfMatrices : public Array { public: // This version of the constructor behaves just like its parent. explicit ArrayOfMatrices (int length) : Array(length) { }; // This constructor creates an array of matrices from a Matlab // array. It accepts either a matrix in double precision, or a cell // array with entries that are matrices. explicit ArrayOfMatrices (const mxArray* ptr); // This constructor creates an array of matrices from a collection // of Matlab arrays. The Matlab arrays must be matrices. ArrayOfMatrices (const mxArray* ptrs[], int numptrs); // This constructor creates an array of matrices and the // associated Matlab structures. The Matlab structures are // matrices. The second input argument acts as a template for the // creation of the matrices, but the data from "model" is not // actually copied into the new ArrayOfMatrices object. It is up // to the user to make sure that the array of mxArray pointers has // enough room for the pointers to the Matlab arrays. ArrayOfMatrices (mxArray* ptrs[], const ArrayOfMatrices& model); // This constructor creates an array of matrices using the second // input argument as a model. The input argument "data" contains // the element data. Note that the information is NOT copied from // the model! ArrayOfMatrices (double* data, const ArrayOfMatrices& model); // The copy constructor makes a shallow copy of the source object. ArrayOfMatrices (const ArrayOfMatrices& source); // The destructor. ~ArrayOfMatrices(); // Copy assignment operator that observes the same behaviour as // the Array copy assignment operator. ArrayOfMatrices& operator= (const ArrayOfMatrices& source); // Return the total number of elements in all the matrices. int numelems() const; protected: static int getnummatlabmatrices (const mxArray* ptr); }; #endif gpml-matlab-v4.0-2016-10-19/util/lbfgsb/array.h000644 000765 000024 00000010714 12701705601 020564 0ustar00carlstaff000000 000000 #ifndef INCLUDE_ARRAY #define INCLUDE_ARRAY #include "matlabexception.h" #include // Function definitions. // ----------------------------------------------------------------- template void copymemory (const Type* source, Type* dest, int length) { memcpy(dest,source,sizeof(Type)*length); } // Class Array // ----------------------------------------------------------------- // An Array object stores a collection of ordered elements. The key // aspect of Array objects is that they do not necessarily achieve // encapsulation; they do not necessarily retain independent // ownership of their data. That is, the data could be modified // externally by another agent. This behaviour is determined by the // data member "owner". If "owner" is false, the array does not take // care of allocation and deallocation of the data in memory. // // Copy assignment behaves different than usual---it copies the // data, but requires that the destination already have the proper // resources allocated in memory. // // The copy constructor performs a shallow copy. template class Array { public: // This constructor allocates memory for an array of elements of // the specificed length. explicit Array (int length); // This constructor is set to point to an already existing // array. As such, it does not gain ownership of the elements. Array (Type* data, int length); // The copy constructor performs a shallow copy, so that both // arrays share the same copy of the data. Array (const Array& source); // The destructor. ~Array(); // Set all the elements of the array to the specified value. void setvalue (const Type& value); // Copy the elements to from the source array. Both the source and // destination array must have the same length. void inject (const Type* source); void inject (const Array& source); Array& operator= (const Type* source); Array& operator= (const Array& source); // Get the number of elements in the array. int length() const { return n; }; // Elements of an array A can be accessed the subscript operator; // e.g. A[i]. For the sake of efficiency, no checks are made to // ensure that the subscripting is legal. It is up to the user to // ensure that i is non-negative and less than A.length(). Note // that subscripts, as per C++ convention, start at 0. Type& operator[] (int i); const Type& operator[] (int i) const; // Returns true if the two arrays have the same dimensions // (i.e. the same lengths). bool operator== (const Array& a) const { return n == a.n; }; bool operator!= (const Array& a) const { return !(*this == a); }; protected: Type* elems; int n; // The number of elements in the array. bool owner; // Whether the object has ownership of the data. }; // Function definitions for class Array. // ----------------------------------------------------------------- template Array::Array (int length) { n = length; owner = true; elems = 0; elems = new Type[length]; } template Array::Array (Type* data, int length) { n = length; owner = false; elems = data; } template Array::Array (const Array& source) { n = source.n; owner = false; elems = source.elems; } template Array::~Array () { if (owner && elems) delete[] elems; } template void Array::setvalue (const Type& value) { for (int i = 0; i < n; i++) elems[i] = value; } template void Array::inject (const Type* source) { // Check for self-assignment. If there is self-assignment, there // is no need to copy because the objects share the same data! if (elems != source) copymemory(source,elems,n); } template void Array::inject (const Array& source) { if (n != source.n) throw MatlabException("Unable to perform copy; arrays have \ different lengths"); inject(source.elems); } template Array& Array::operator= (const Type* source) { inject(source); return *this; } template Array& Array::operator= (const Array& source) { inject(source); return *this; } template Type& Array::operator[] (int i) { return elems[i]; } template const Type& Array::operator[] (int i) const { return elems[i]; } #endif gpml-matlab-v4.0-2016-10-19/util/lbfgsb/solver_3_0.f000644 000765 000024 00000376670 12701705601 021437 0ustar00carlstaff000000 000000 c c L-BFGS-B is released under the “New BSD License” (aka “Modified BSD License” c or “3-clause license”) c Please read attached file License.txt c c=========== L-BFGS-B (version 3.0. April 25, 2011 =================== c c This is a modified version of L-BFGS-B. Minor changes in the updated c code appear preceded by a line comment as follows c c c-jlm-jn c c Major changes are described in the accompanying paper: c c Jorge Nocedal and Jose Luis Morales, Remark on "Algorithm 778: c L-BFGS-B: Fortran Subroutines for Large-Scale Bound Constrained c Optimization" (2011). To appear in ACM Transactions on c Mathematical Software, c c The paper describes an improvement and a correction to Algorithm 778. c It is shown that the performance of the algorithm can be improved c significantly by making a relatively simple modication to the subspace c minimization phase. The correction concerns an error caused by the use c of routine dpmeps to estimate machine precision. c c The total work space **wa** required by the new version is c c 2*m*n + 11m*m + 5*n + 8*m c c the old version required c c 2*m*n + 12m*m + 4*n + 12*m c c c J. Nocedal Department of Electrical Engineering and c Computer Science. c Northwestern University. Evanston, IL. USA c c c J.L Morales Departamento de Matematicas, c Instituto Tecnologico Autonomo de Mexico c Mexico D.F. Mexico. c c March 2011 c c============================================================================= subroutine setulb(n, m, x, l, u, nbd, f, g, factr, pgtol, wa, iwa, + task, iprint, csave, lsave, isave, dsave) character*60 task, csave logical lsave(4) integer n, m, iprint, + nbd(n), iwa(3*n), isave(44) double precision f, factr, pgtol, x(n), l(n), u(n), g(n), c c-jlm-jn + wa(2*m*n + 5*n + 11*m*m + 8*m), dsave(29) c ************ c c Subroutine setulb c c This subroutine partitions the working arrays wa and iwa, and c then uses the limited memory BFGS method to solve the bound c constrained optimization problem by calling mainlb. c (The direct method will be used in the subspace minimization.) c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound on x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound on x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. Typical values for factr: 1.d+12 for c low accuracy; 1.d+7 for moderate accuracy; 1.d+1 for extremely c high accuracy. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c wa is a double precision working array of length c (2mmax + 5)nmax + 12mmax^2 + 12mmax. c c iwa is an integer working array of length 3nmax. c c task is a working string of characters of length 60 indicating c the current job when entering and quitting this subroutine. c c iprint is an integer variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c On exit with 'task' = NEW_X, the following information is c available: c If lsave(1) = .true. then the initial X has been replaced by c its projection in the feasible set; c If lsave(2) = .true. then the problem is constrained; c If lsave(3) = .true. then each variable has upper and lower c bounds; c c isave is an integer working array of dimension 44. c On exit with 'task' = NEW_X, the following information is c available: c isave(22) = the total number of intervals explored in the c search of Cauchy points; c isave(26) = the total number of skipped BFGS updates before c the current iteration; c isave(30) = the number of current iteration; c isave(31) = the total number of BFGS updates prior the current c iteration; c isave(33) = the number of intervals explored in the search of c Cauchy point in the current iteration; c isave(34) = the total number of function and gradient c evaluations; c isave(36) = the number of function value or gradient c evaluations in the current iteration; c if isave(37) = 0 then the subspace argmin is within the box; c if isave(37) = 1 then the subspace argmin is beyond the box; c isave(38) = the number of free variables in the current c iteration; c isave(39) = the number of active constraints in the current c iteration; c n + 1 - isave(40) = the number of variables leaving the set of c active constraints in the current iteration; c isave(41) = the number of variables entering the set of active c constraints in the current iteration. c c dsave is a double precision working array of dimension 29. c On exit with 'task' = NEW_X, the following information is c available: c dsave(1) = current 'theta' in the BFGS matrix; c dsave(2) = f(x) in the previous iteration; c dsave(3) = factr*epsmch; c dsave(4) = 2-norm of the line search direction vector; c dsave(5) = the machine precision epsmch generated by the code; c dsave(7) = the accumulated time spent on searching for c Cauchy points; c dsave(8) = the accumulated time spent on c subspace minimization; c dsave(9) = the accumulated time spent on line search; c dsave(11) = the slope of the line search function at c the current point of line search; c dsave(12) = the maximum relative step length imposed in c line search; c dsave(13) = the infinity norm of the projected gradient; c dsave(14) = the relative step length in the line search; c dsave(15) = the slope of the line search function at c the starting point of the line search; c dsave(16) = the square of the 2-norm of the line search c direction vector. c c Subprograms called: c c L-BFGS-B Library ... mainlb. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to eecs.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ c-jlm-jn integer lws,lr,lz,lt,ld,lxp,lwa, + lwy,lsy,lss,lwt,lwn,lsnd if (task .eq. 'START') then isave(1) = m*n isave(2) = m**2 isave(3) = 4*m**2 isave(4) = 1 ! ws m*n isave(5) = isave(4) + isave(1) ! wy m*n isave(6) = isave(5) + isave(1) ! wsy m**2 isave(7) = isave(6) + isave(2) ! wss m**2 isave(8) = isave(7) + isave(2) ! wt m**2 isave(9) = isave(8) + isave(2) ! wn 4*m**2 isave(10) = isave(9) + isave(3) ! wsnd 4*m**2 isave(11) = isave(10) + isave(3) ! wz n isave(12) = isave(11) + n ! wr n isave(13) = isave(12) + n ! wd n isave(14) = isave(13) + n ! wt n isave(15) = isave(14) + n ! wxp n isave(16) = isave(15) + n ! wa 8*m endif lws = isave(4) lwy = isave(5) lsy = isave(6) lss = isave(7) lwt = isave(8) lwn = isave(9) lsnd = isave(10) lz = isave(11) lr = isave(12) ld = isave(13) lt = isave(14) lxp = isave(15) lwa = isave(16) call mainlb(n,m,x,l,u,nbd,f,g,factr,pgtol, + wa(lws),wa(lwy),wa(lsy),wa(lss), wa(lwt), + wa(lwn),wa(lsnd),wa(lz),wa(lr),wa(ld),wa(lt),wa(lxp), + wa(lwa), + iwa(1),iwa(n+1),iwa(2*n+1),task,iprint, + csave,lsave,isave(22),dsave) return end c======================= The end of setulb ============================= subroutine mainlb(n, m, x, l, u, nbd, f, g, factr, pgtol, ws, wy, + sy, ss, wt, wn, snd, z, r, d, t, xp, wa, + index, iwhere, indx2, task, + iprint, csave, lsave, isave, dsave) implicit none character*60 task, csave logical lsave(4) integer n, m, iprint, nbd(n), index(n), + iwhere(n), indx2(n), isave(23) double precision f, factr, pgtol, + x(n), l(n), u(n), g(n), z(n), r(n), d(n), t(n), c-jlm-jn + xp(n), + wa(8*m), + ws(n, m), wy(n, m), sy(m, m), ss(m, m), + wt(m, m), wn(2*m, 2*m), snd(2*m, 2*m), dsave(29) c ************ c c Subroutine mainlb c c This subroutine solves bound constrained optimization problems by c using the compact formula of the limited memory BFGS updates. c c n is an integer variable. c On entry n is the number of variables. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric c corrections allowed in the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c ws, wy, sy, and wt are double precision working arrays used to c store the following information defining the limited memory c BFGS matrix: c ws, of dimension n x m, stores S, the matrix of s-vectors; c wy, of dimension n x m, stores Y, the matrix of y-vectors; c sy, of dimension m x m, stores S'Y; c ss, of dimension m x m, stores S'S; c yy, of dimension m x m, stores Y'Y; c wt, of dimension m x m, stores the Cholesky factorization c of (theta*S'S+LD^(-1)L'); see eq. c (2.26) in [3]. c c wn is a double precision working array of dimension 2m x 2m c used to store the LEL^T factorization of the indefinite matrix c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c where E = [-I 0] c [ 0 I] c c snd is a double precision working array of dimension 2m x 2m c used to store the lower triangular part of c N = [Y' ZZ'Y L_a'+R_z'] c [L_a +R_z S'AA'S ] c c z(n),r(n),d(n),t(n), xp(n),wa(8*m) are double precision working arrays. c z is used at different times to store the Cauchy point and c the Newton point. c xp is used to safeguard the projected Newton direction c c sg(m),sgo(m),yg(m),ygo(m) are double precision working arrays. c c index is an integer working array of dimension n. c In subroutine freev, index is used to store the free and fixed c variables at the Generalized Cauchy Point (GCP). c c iwhere is an integer working array of dimension n used to record c the status of the vector x for GCP computation. c iwhere(i)=0 or -3 if x(i) is free and has bounds, c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., no bounds on it. c c indx2 is an integer working array of dimension n. c Within subroutine cauchy, indx2 corresponds to the array iorder. c In subroutine freev, a list of variables entering and leaving c the free set is stored in indx2, and it is passed on to c subroutine formk with this information. c c task is a working string of characters of length 60 indicating c the current job when entering and leaving this subroutine. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c c isave is an integer working array of dimension 23. c c dsave is a double precision working array of dimension 29. c c c Subprograms called c c L-BFGS-B Library ... cauchy, subsm, lnsrlb, formk, c c errclb, prn1lb, prn2lb, prn3lb, active, projgr, c c freev, cmprlb, matupd, formt. c c Minpack2 Library ... timer c c Linpack Library ... dcopy, ddot. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c [3] R. Byrd, J. Nocedal and R. Schnabel "Representations of c Quasi-Newton Matrices and their use in Limited Memory Methods'', c Mathematical Programming 63 (1994), no. 4, pp. 129-156. c c (Postscript files of these papers are available via anonymous c ftp to eecs.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical prjctd,cnstnd,boxed,updatd,wrk character*3 word integer i,k,nintol,itfile,iback,nskip, + head,col,iter,itail,iupdat, + nseg,nfgv,info,ifun, + iword,nfree,nact,ileave,nenter double precision theta,fold,ddot,dr,rr,tol, + xstep,sbgnrm,ddum,dnorm,dtd,epsmch, + cpu1,cpu2,cachyt,sbtime,lnscht,time1,time2, + gd,gdold,stp,stpmx,time double precision one,zero parameter (one=1.0d0,zero=0.0d0) if (task .eq. 'START') then epsmch = epsilon(one) call timer(time1) c Initialize counters and scalars when task='START'. c for the limited memory BFGS matrices: col = 0 head = 1 theta = one iupdat = 0 updatd = .false. iback = 0 itail = 0 iword = 0 nact = 0 ileave = 0 nenter = 0 fold = zero dnorm = zero cpu1 = zero gd = zero stpmx = zero sbgnrm = zero stp = zero gdold = zero dtd = zero c for operation counts: iter = 0 nfgv = 0 nseg = 0 nintol = 0 nskip = 0 nfree = n ifun = 0 c for stopping tolerance: tol = factr*epsmch c for measuring running time: cachyt = 0 sbtime = 0 lnscht = 0 c 'word' records the status of subspace solutions. word = '---' c 'info' records the termination information. info = 0 itfile = 8 if (iprint .ge. 1) then c open a summary file 'iterate.dat' open (8, file = 'iterate.dat', status = 'unknown') endif c Check the input arguments for errors. call errclb(n,m,factr,l,u,nbd,task,info,k) if (task(1:5) .eq. 'ERROR') then call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + zero,nseg,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) return endif call prn1lb(n,m,l,u,x,iprint,itfile,epsmch) c Initialize iwhere & project x onto the feasible set. call active(n,l,u,nbd,x,iwhere,iprint,prjctd,cnstnd,boxed) c The end of the initialization. else c restore local variables. prjctd = lsave(1) cnstnd = lsave(2) boxed = lsave(3) updatd = lsave(4) nintol = isave(1) itfile = isave(3) iback = isave(4) nskip = isave(5) head = isave(6) col = isave(7) itail = isave(8) iter = isave(9) iupdat = isave(10) nseg = isave(12) nfgv = isave(13) info = isave(14) ifun = isave(15) iword = isave(16) nfree = isave(17) nact = isave(18) ileave = isave(19) nenter = isave(20) theta = dsave(1) fold = dsave(2) tol = dsave(3) dnorm = dsave(4) epsmch = dsave(5) cpu1 = dsave(6) cachyt = dsave(7) sbtime = dsave(8) lnscht = dsave(9) time1 = dsave(10) gd = dsave(11) stpmx = dsave(12) sbgnrm = dsave(13) stp = dsave(14) gdold = dsave(15) dtd = dsave(16) c After returning from the driver go to the point where execution c is to resume. if (task(1:5) .eq. 'FG_LN') goto 666 if (task(1:5) .eq. 'NEW_X') goto 777 if (task(1:5) .eq. 'FG_ST') goto 111 if (task(1:4) .eq. 'STOP') then if (task(7:9) .eq. 'CPU') then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold endif goto 999 endif endif c Compute f0 and g0. task = 'FG_START' c return to the driver to calculate f and g; reenter at 111. goto 1000 111 continue nfgv = 1 c Compute the infinity norm of the (-) projected gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) if (iprint .ge. 1) then write (6,1002) iter,f,sbgnrm write (itfile,1003) iter,nfgv,sbgnrm,f endif if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL' goto 999 endif c ----------------- the beginning of the loop -------------------------- 222 continue if (iprint .ge. 99) write (6,1001) iter + 1 iword = -1 c if (.not. cnstnd .and. col .gt. 0) then c skip the search for GCP. call dcopy(n,x,1,z,1) wrk = updatd nseg = 0 goto 333 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Compute the Generalized Cauchy Point (GCP). c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) call cauchy(n,x,l,u,nbd,g,indx2,iwhere,t,d,z, + m,wy,ws,sy,wt,theta,col,head, + wa(1),wa(2*m+1),wa(4*m+1),wa(6*m+1),nseg, + iprint, sbgnrm, info, epsmch) if (info .ne. 0) then c singular triangular system detected; refresh the lbfgs memory. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 goto 222 endif call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 nintol = nintol + nseg c Count the entering and leaving variables for iter > 0; c find the index set of free and active variables at the GCP. call freev(n,nfree,index,nenter,ileave,indx2, + iwhere,wrk,updatd,cnstnd,iprint,iter) nact = n - nfree 333 continue c If there are no free variables or B=theta*I, then c skip the subspace minimization. if (nfree .eq. 0 .or. col .eq. 0) goto 555 cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Subspace minimization. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) c Form the LEL^T factorization of the indefinite c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] if (wrk) call formk(n,nfree,index,nenter,ileave,indx2,iupdat, + updatd,wn,snd,m,ws,wy,sy,theta,col,head,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1006) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif c compute r=-Z'B(xcp-xk)-Z'g (using wa(2m+1)=W'(xcp-x) c from 'cauchy'). call cmprlb(n,m,x,g,ws,wy,sy,wt,z,r,wa,index, + theta,col,head,nfree,cnstnd,info) if (info .ne. 0) goto 444 c-jlm-jn call the direct method. call subsm( n, m, nfree, index, l, u, nbd, z, r, xp, ws, wy, + theta, x, g, col, head, iword, wa, wn, iprint, info) 444 continue if (info .ne. 0) then c singular triangular system detected; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 555 continue cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Line search and optimality tests. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Generate the search direction d:=z-x. do 40 i = 1, n d(i) = z(i) - x(i) 40 continue call timer(cpu1) 666 continue call lnsrlb(n,l,u,nbd,x,f,fold,gd,gdold,g,d,r,t,z,stp,dnorm, + dtd,xstep,stpmx,iter,ifun,iback,nfgv,info,task, + boxed,cnstnd,csave,isave(22),dsave(17)) if (info .ne. 0 .or. iback .ge. 20) then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold if (col .eq. 0) then c abnormal termination. if (info .eq. 0) then info = -9 c restore the actual number of f and g evaluations etc. nfgv = nfgv - 1 ifun = ifun - 1 iback = iback - 1 endif task = 'ABNORMAL_TERMINATION_IN_LNSRCH' iter = iter + 1 goto 999 else c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1008) if (info .eq. 0) nfgv = nfgv - 1 info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. task = 'RESTART_FROM_LNSRCH' call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 goto 222 endif else if (task(1:5) .eq. 'FG_LN') then c return to the driver for calculating f and g; reenter at 666. goto 1000 else c calculate and print out the quantities related to the new X. call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 iter = iter + 1 c Compute the infinity norm of the projected (-)gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) c Print iteration information. call prn2lb(n,x,f,g,iprint,itfile,iter,nfgv,nact, + sbgnrm,nseg,word,iword,iback,stp,xstep) goto 1000 endif 777 continue c Test for termination. if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL' goto 999 endif ddum = max(abs(fold), abs(f), one) if ((fold - f) .le. tol*ddum) then c terminate the algorithm. task = 'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH' if (iback .ge. 10) info = -5 c i.e., to issue a warning if iback>10 in the line search. goto 999 endif c Compute d=newx-oldx, r=newg-oldg, rr=y'y and dr=y's. do 42 i = 1, n r(i) = g(i) - r(i) 42 continue rr = ddot(n,r,1,r,1) if (stp .eq. one) then dr = gd - gdold ddum = -gdold else dr = (gd - gdold)*stp call dscal(n,stp,d,1) ddum = -gdold*stp endif if (dr .le. epsmch*ddum) then c skip the L-BFGS update. nskip = nskip + 1 updatd = .false. if (iprint .ge. 1) write (6,1004) dr, ddum goto 888 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Update the L-BFGS matrix. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc updatd = .true. iupdat = iupdat + 1 c Update matrices WS and WY and form the middle matrix in B. call matupd(n,m,ws,wy,sy,ss,d,r,itail, + iupdat,col,head,theta,rr,dr,stp,dtd) c Form the upper half of the pds T = theta*SS + L*D^(-1)*L'; c Store T in the upper triangular of the array wt; c Cholesky factorize T to J*J' with c J' stored in the upper triangular of wt. call formt(m,wt,sy,ss,col,theta,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1007) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. goto 222 endif c Now the inverse of the middle matrix in B is c [ D^(1/2) O ] [ -D^(1/2) D^(-1/2)*L' ] c [ -L*D^(-1/2) J ] [ 0 J' ] 888 continue c -------------------- the end of the loop ----------------------------- goto 222 999 continue call timer(time2) time = time2 - time1 call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + time,nseg,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) 1000 continue c Save local variables. lsave(1) = prjctd lsave(2) = cnstnd lsave(3) = boxed lsave(4) = updatd isave(1) = nintol isave(3) = itfile isave(4) = iback isave(5) = nskip isave(6) = head isave(7) = col isave(8) = itail isave(9) = iter isave(10) = iupdat isave(12) = nseg isave(13) = nfgv isave(14) = info isave(15) = ifun isave(16) = iword isave(17) = nfree isave(18) = nact isave(19) = ileave isave(20) = nenter dsave(1) = theta dsave(2) = fold dsave(3) = tol dsave(4) = dnorm dsave(5) = epsmch dsave(6) = cpu1 dsave(7) = cachyt dsave(8) = sbtime dsave(9) = lnscht dsave(10) = time1 dsave(11) = gd dsave(12) = stpmx dsave(13) = sbgnrm dsave(14) = stp dsave(15) = gdold dsave(16) = dtd 1001 format (//,'ITERATION ',i5) 1002 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 1003 format (2(1x,i4),5x,'-',5x,'-',3x,'-',5x,'-',5x,'-',8x,'-',3x, + 1p,2(1x,d10.3)) 1004 format (' ys=',1p,e10.3,' -gs=',1p,e10.3,' BFGS update SKIPPED') 1005 format (/, +' Singular triangular system detected;',/, +' refresh the lbfgs memory and restart the iteration.') 1006 format (/, +' Nonpositive definiteness in Cholesky factorization in formk;',/, +' refresh the lbfgs memory and restart the iteration.') 1007 format (/, +' Nonpositive definiteness in Cholesky factorization in formt;',/, +' refresh the lbfgs memory and restart the iteration.') 1008 format (/, +' Bad direction in the line search;',/, +' refresh the lbfgs memory and restart the iteration.') return end c======================= The end of mainlb ============================= subroutine active(n, l, u, nbd, x, iwhere, iprint, + prjctd, cnstnd, boxed) logical prjctd, cnstnd, boxed integer n, iprint, nbd(n), iwhere(n) double precision x(n), l(n), u(n) c ************ c c Subroutine active c c This subroutine initializes iwhere and projects the initial x to c the feasible set if necessary. c c iwhere is an integer array of dimension n. c On entry iwhere is unspecified. c On exit iwhere(i)=-1 if x(i) has no bounds c 3 if l(i)=u(i) c 0 otherwise. c In cauchy, iwhere is given finer gradations. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer nbdd,i double precision zero parameter (zero=0.0d0) c Initialize nbdd, prjctd, cnstnd and boxed. nbdd = 0 prjctd = .false. cnstnd = .false. boxed = .true. c Project the initial x to the easible set if necessary. do 10 i = 1, n if (nbd(i) .gt. 0) then if (nbd(i) .le. 2 .and. x(i) .le. l(i)) then if (x(i) .lt. l(i)) then prjctd = .true. x(i) = l(i) endif nbdd = nbdd + 1 else if (nbd(i) .ge. 2 .and. x(i) .ge. u(i)) then if (x(i) .gt. u(i)) then prjctd = .true. x(i) = u(i) endif nbdd = nbdd + 1 endif endif 10 continue c Initialize iwhere and assign values to cnstnd and boxed. do 20 i = 1, n if (nbd(i) .ne. 2) boxed = .false. if (nbd(i) .eq. 0) then c this variable is always free iwhere(i) = -1 c otherwise set x(i)=mid(x(i), u(i), l(i)). else cnstnd = .true. if (nbd(i) .eq. 2 .and. u(i) - l(i) .le. zero) then c this variable is always fixed iwhere(i) = 3 else iwhere(i) = 0 endif endif 20 continue if (iprint .ge. 0) then if (prjctd) write (6,*) + 'The initial X is infeasible. Restart with its projection.' if (.not. cnstnd) + write (6,*) 'This problem is unconstrained.' endif if (iprint .gt. 0) write (6,1001) nbdd 1001 format (/,'At X0 ',i9,' variables are exactly at the bounds') return end c======================= The end of active ============================= subroutine bmv(m, sy, wt, col, v, p, info) integer m, col, info double precision sy(m, m), wt(m, m), v(2*col), p(2*col) c ************ c c Subroutine bmv c c This subroutine computes the product of the 2m x 2m middle matrix c in the compact L-BFGS formula of B and a 2m vector v; c it returns the product in p. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c sy is a double precision array of dimension m x m. c On entry sy specifies the matrix S'Y. c On exit sy is unchanged. c c wt is a double precision array of dimension m x m. c On entry wt specifies the upper triangular matrix J' which is c the Cholesky factor of (thetaS'S+LD^(-1)L'). c On exit wt is unchanged. c c col is an integer variable. c On entry col specifies the number of s-vectors (or y-vectors) c stored in the compact L-BFGS formula. c On exit col is unchanged. c c v is a double precision array of dimension 2col. c On entry v specifies vector v. c On exit v is unchanged. c c p is a double precision array of dimension 2col. c On entry p is unspecified. c On exit p is the product Mv. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return when the system c to be solved by dtrsl is singular. c c Subprograms called: c c Linpack ... dtrsl. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,k,i2 double precision sum if (col .eq. 0) return c PART I: solve [ D^(1/2) O ] [ p1 ] = [ v1 ] c [ -L*D^(-1/2) J ] [ p2 ] [ v2 ]. c solve Jp2=v2+LD^(-1)v1. p(col + 1) = v(col + 1) do 20 i = 2, col i2 = col + i sum = 0.0d0 do 10 k = 1, i - 1 sum = sum + sy(i,k)*v(k)/sy(k,k) 10 continue p(i2) = v(i2) + sum 20 continue c Solve the triangular system call dtrsl(wt,m,col,p(col+1),11,info) if (info .ne. 0) return c solve D^(1/2)p1=v1. do 30 i = 1, col p(i) = v(i)/sqrt(sy(i,i)) 30 continue c PART II: solve [ -D^(1/2) D^(-1/2)*L' ] [ p1 ] = [ p1 ] c [ 0 J' ] [ p2 ] [ p2 ]. c solve J^Tp2=p2. call dtrsl(wt,m,col,p(col+1),01,info) if (info .ne. 0) return c compute p1=-D^(-1/2)(p1-D^(-1/2)L'p2) c =-D^(-1/2)p1+D^(-1)L'p2. do 40 i = 1, col p(i) = -p(i)/sqrt(sy(i,i)) 40 continue do 60 i = 1, col sum = 0.d0 do 50 k = i + 1, col sum = sum + sy(k,i)*p(col+k)/sy(i,i) 50 continue p(i) = p(i) + sum 60 continue return end c======================== The end of bmv =============================== subroutine cauchy(n, x, l, u, nbd, g, iorder, iwhere, t, d, xcp, + m, wy, ws, sy, wt, theta, col, head, p, c, wbp, + v, nseg, iprint, sbgnrm, info, epsmch) implicit none integer n, m, head, col, nseg, iprint, info, + nbd(n), iorder(n), iwhere(n) double precision theta, epsmch, + x(n), l(n), u(n), g(n), t(n), d(n), xcp(n), + wy(n, col), ws(n, col), sy(m, m), + wt(m, m), p(2*m), c(2*m), wbp(2*m), v(2*m) c ************ c c Subroutine cauchy c c For given x, l, u, g (with sbgnrm > 0), and a limited memory c BFGS matrix B defined in terms of matrices WY, WS, WT, and c scalars head, col, and theta, this subroutine computes the c generalized Cauchy point (GCP), defined as the first local c minimizer of the quadratic c c Q(x + s) = g's + 1/2 s'Bs c c along the projected gradient direction P(x-tg,l,u). c The routine returns the GCP in xcp. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c x is a double precision array of dimension n. c On entry x is the starting point for the GCP computation. c On exit x is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c g is a double precision array of dimension n. c On entry g is the gradient of f(x). g must be a nonzero vector. c On exit g is unchanged. c c iorder is an integer working array of dimension n. c iorder will be used to store the breakpoints in the piecewise c linear path and free variables encountered. On exit, c iorder(1),...,iorder(nleft) are indices of breakpoints c which have not been encountered; c iorder(nleft+1),...,iorder(nbreak) are indices of c encountered breakpoints; and c iorder(nfree),...,iorder(n) are indices of variables which c have no bound constraits along the search direction. c c iwhere is an integer array of dimension n. c On entry iwhere indicates only the permanently fixed (iwhere=3) c or free (iwhere= -1) components of x. c On exit iwhere records the status of the current x variables. c iwhere(i)=-3 if x(i) is free and has bounds, but is not moved c 0 if x(i) is free and has bounds, and is moved c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., it has no bounds. c c t is a double precision working array of dimension n. c t will be used to store the break points. c c d is a double precision array of dimension n used to store c the Cauchy direction P(x-tg)-x. c c xcp is a double precision array of dimension n used to return the c GCP on exit. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wt are double precision arrays. c On entry they store information that defines the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wt(m,m) stores the c Cholesky factorization of (theta*S'S+LD^(-1)L'). c On exit these arrays are unchanged. c c theta is a double precision variable. c On entry theta is the scaling factor specifying B_0 = theta I. c On exit theta is unchanged. c c col is an integer variable. c On entry col is the actual number of variable metric c corrections stored so far. c On exit col is unchanged. c c head is an integer variable. c On entry head is the location of the first s-vector (or y-vector) c in S (or Y). c On exit col is unchanged. c c p is a double precision working array of dimension 2m. c p will be used to store the vector p = W^(T)d. c c c is a double precision working array of dimension 2m. c c will be used to store the vector c = W^(T)(xcp-x). c c wbp is a double precision working array of dimension 2m. c wbp will be used to store the row of W corresponding c to a breakpoint. c c v is a double precision working array of dimension 2m. c c nseg is an integer variable. c On exit nseg records the number of quadratic segments explored c in searching for the GCP. c c sg and yg are double precision arrays of dimension m. c On entry sg and yg store S'g and Y'g correspondingly. c On exit they are unchanged. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c sbgnrm is a double precision variable. c On entry sbgnrm is the norm of the projected gradient at x. c On exit sbgnrm is unchanged. c c info is an integer variable. c On entry info is 0. c On exit info = 0 for normal return, c = nonzero for abnormal return when the the system c used in routine bmv is singular. c c Subprograms called: c c L-BFGS-B Library ... hpsolb, bmv. c c Linpack ... dscal dcopy, daxpy. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c (Postscript files of these papers are available via anonymous c ftp to eecs.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical xlower,xupper,bnded integer i,j,col2,nfree,nbreak,pointr, + ibp,nleft,ibkmin,iter double precision f1,f2,dt,dtm,tsum,dibp,zibp,dibp2,bkmin, + tu,tl,wmc,wmp,wmw,ddot,tj,tj0,neggi,sbgnrm, + f2_org double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Check the status of the variables, reset iwhere(i) if necessary; c compute the Cauchy direction d and the breakpoints t; initialize c the derivative f1 and the vector p = W'd (for theta = 1). if (sbgnrm .le. zero) then if (iprint .ge. 0) write (6,*) 'Subgnorm = 0. GCP = X.' call dcopy(n,x,1,xcp,1) return endif bnded = .true. nfree = n + 1 nbreak = 0 ibkmin = 0 bkmin = zero col2 = 2*col f1 = zero if (iprint .ge. 99) write (6,3010) c We set p to zero and build it up as we determine d. do 20 i = 1, col2 p(i) = zero 20 continue c In the following loop we determine for each variable its bound c status and its breakpoint, and update p accordingly. c Smallest breakpoint is identified. do 50 i = 1, n neggi = -g(i) if (iwhere(i) .ne. 3 .and. iwhere(i) .ne. -1) then c if x(i) is not a constant and has bounds, c compute the difference between x(i) and its bounds. if (nbd(i) .le. 2) tl = x(i) - l(i) if (nbd(i) .ge. 2) tu = u(i) - x(i) c If a variable is close enough to a bound c we treat it as at bound. xlower = nbd(i) .le. 2 .and. tl .le. zero xupper = nbd(i) .ge. 2 .and. tu .le. zero c reset iwhere(i). iwhere(i) = 0 if (xlower) then if (neggi .le. zero) iwhere(i) = 1 else if (xupper) then if (neggi .ge. zero) iwhere(i) = 2 else if (abs(neggi) .le. zero) iwhere(i) = -3 endif endif pointr = head if (iwhere(i) .ne. 0 .and. iwhere(i) .ne. -1) then d(i) = zero else d(i) = neggi f1 = f1 - neggi*neggi c calculate p := p - W'e_i* (g_i). do 40 j = 1, col p(j) = p(j) + wy(i,pointr)* neggi p(col + j) = p(col + j) + ws(i,pointr)*neggi pointr = mod(pointr,m) + 1 40 continue if (nbd(i) .le. 2 .and. nbd(i) .ne. 0 + .and. neggi .lt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tl/(-neggi) if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else if (nbd(i) .ge. 2 .and. neggi .gt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tu/neggi if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else c x(i) + d(i) is not bounded. nfree = nfree - 1 iorder(nfree) = i if (abs(neggi) .gt. zero) bnded = .false. endif endif 50 continue c The indices of the nonzero components of d are now stored c in iorder(1),...,iorder(nbreak) and iorder(nfree),...,iorder(n). c The smallest of the nbreak breakpoints is in t(ibkmin)=bkmin. if (theta .ne. one) then c complete the initialization of p for theta not= one. call dscal(col,theta,p(col+1),1) endif c Initialize GCP xcp = x. call dcopy(n,x,1,xcp,1) if (nbreak .eq. 0 .and. nfree .eq. n + 1) then c is a zero vector, return with the initial xcp as GCP. if (iprint .gt. 100) write (6,1010) (xcp(i), i = 1, n) return endif c Initialize c = W'(xcp - x) = 0. do 60 j = 1, col2 c(j) = zero 60 continue c Initialize derivative f2. f2 = -theta*f1 f2_org = f2 if (col .gt. 0) then call bmv(m,sy,wt,col,p,v,info) if (info .ne. 0) return f2 = f2 - ddot(col2,v,1,p,1) endif dtm = -f1/f2 tsum = zero nseg = 1 if (iprint .ge. 99) + write (6,*) 'There are ',nbreak,' breakpoints ' c If there are no breakpoints, locate the GCP and return. if (nbreak .eq. 0) goto 888 nleft = nbreak iter = 1 tj = zero c------------------- the beginning of the loop ------------------------- 777 continue c Find the next smallest breakpoint; c compute dt = t(nleft) - t(nleft + 1). tj0 = tj if (iter .eq. 1) then c Since we already have the smallest breakpoint we need not do c heapsort yet. Often only one breakpoint is used and the c cost of heapsort is avoided. tj = bkmin ibp = iorder(ibkmin) else if (iter .eq. 2) then c Replace the already used smallest breakpoint with the c breakpoint numbered nbreak > nlast, before heapsort call. if (ibkmin .ne. nbreak) then t(ibkmin) = t(nbreak) iorder(ibkmin) = iorder(nbreak) endif c Update heap structure of breakpoints c (if iter=2, initialize heap). endif call hpsolb(nleft,t,iorder,iter-2) tj = t(nleft) ibp = iorder(nleft) endif dt = tj - tj0 if (dt .ne. zero .and. iprint .ge. 100) then write (6,4011) nseg,f1,f2 write (6,5010) dt write (6,6010) dtm endif c If a minimizer is within this interval, locate the GCP and return. if (dtm .lt. dt) goto 888 c Otherwise fix one variable and c reset the corresponding component of d to zero. tsum = tsum + dt nleft = nleft - 1 iter = iter + 1 dibp = d(ibp) d(ibp) = zero if (dibp .gt. zero) then zibp = u(ibp) - x(ibp) xcp(ibp) = u(ibp) iwhere(ibp) = 2 else zibp = l(ibp) - x(ibp) xcp(ibp) = l(ibp) iwhere(ibp) = 1 endif if (iprint .ge. 100) write (6,*) 'Variable ',ibp,' is fixed.' if (nleft .eq. 0 .and. nbreak .eq. n) then c all n variables are fixed, c return with xcp as GCP. dtm = dt goto 999 endif c Update the derivative information. nseg = nseg + 1 dibp2 = dibp**2 c Update f1 and f2. c temporarily set f1 and f2 for col=0. f1 = f1 + dt*f2 + dibp2 - theta*dibp*zibp f2 = f2 - theta*dibp2 if (col .gt. 0) then c update c = c + dt*p. call daxpy(col2,dt,p,1,c,1) c choose wbp, c the row of W corresponding to the breakpoint encountered. pointr = head do 70 j = 1,col wbp(j) = wy(ibp,pointr) wbp(col + j) = theta*ws(ibp,pointr) pointr = mod(pointr,m) + 1 70 continue c compute (wbp)Mc, (wbp)Mp, and (wbp)M(wbp)'. call bmv(m,sy,wt,col,wbp,v,info) if (info .ne. 0) return wmc = ddot(col2,c,1,v,1) wmp = ddot(col2,p,1,v,1) wmw = ddot(col2,wbp,1,v,1) c update p = p - dibp*wbp. call daxpy(col2,-dibp,wbp,1,p,1) c complete updating f1 and f2 while col > 0. f1 = f1 + dibp*wmc f2 = f2 + 2.0d0*dibp*wmp - dibp2*wmw endif f2 = max(epsmch*f2_org,f2) if (nleft .gt. 0) then dtm = -f1/f2 goto 777 c to repeat the loop for unsearched intervals. else if(bnded) then f1 = zero f2 = zero dtm = zero else dtm = -f1/f2 endif c------------------- the end of the loop ------------------------------- 888 continue if (iprint .ge. 99) then write (6,*) write (6,*) 'GCP found in this segment' write (6,4010) nseg,f1,f2 write (6,6010) dtm endif if (dtm .le. zero) dtm = zero tsum = tsum + dtm c Move free variables (i.e., the ones w/o breakpoints) and c the variables whose breakpoints haven't been reached. call daxpy(n,tsum,d,1,xcp,1) 999 continue c Update c = c + dtm*p = W'(x^c - x) c which will be used in computing r = Z'(B(x^c - x) + g). if (col .gt. 0) call daxpy(col2,dtm,p,1,c,1) if (iprint .gt. 100) write (6,1010) (xcp(i),i = 1,n) if (iprint .ge. 99) write (6,2010) 1010 format ('Cauchy X = ',/,(4x,1p,6(1x,d11.4))) 2010 format (/,'---------------- exit CAUCHY----------------------',/) 3010 format (/,'---------------- CAUCHY entered-------------------') 4010 format ('Piece ',i3,' --f1, f2 at start point ',1p,2(1x,d11.4)) 4011 format (/,'Piece ',i3,' --f1, f2 at start point ', + 1p,2(1x,d11.4)) 5010 format ('Distance to the next break point = ',1p,d11.4) 6010 format ('Distance to the stationary point = ',1p,d11.4) return end c====================== The end of cauchy ============================== subroutine cmprlb(n, m, x, g, ws, wy, sy, wt, z, r, wa, index, + theta, col, head, nfree, cnstnd, info) logical cnstnd integer n, m, col, head, nfree, info, index(n) double precision theta, + x(n), g(n), z(n), r(n), wa(4*m), + ws(n, m), wy(n, m), sy(m, m), wt(m, m) c ************ c c Subroutine cmprlb c c This subroutine computes r=-Z'B(xcp-xk)-Z'g by using c wa(2m+1)=W'(xcp-x) from subroutine cauchy. c c Subprograms called: c c L-BFGS-B Library ... bmv. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,pointr double precision a1,a2 if (.not. cnstnd .and. col .gt. 0) then do 26 i = 1, n r(i) = -g(i) 26 continue else do 30 i = 1, nfree k = index(i) r(i) = -theta*(z(k) - x(k)) - g(k) 30 continue call bmv(m,sy,wt,col,wa(2*m+1),wa(1),info) if (info .ne. 0) then info = -8 return endif pointr = head do 34 j = 1, col a1 = wa(j) a2 = theta*wa(col + j) do 32 i = 1, nfree k = index(i) r(i) = r(i) + wy(k,pointr)*a1 + ws(k,pointr)*a2 32 continue pointr = mod(pointr,m) + 1 34 continue endif return end c======================= The end of cmprlb ============================= subroutine errclb(n, m, factr, l, u, nbd, task, info, k) character*60 task integer n, m, info, k, nbd(n) double precision factr, l(n), u(n) c ************ c c Subroutine errclb c c This subroutine checks the validity of the input data. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Check the input arguments for errors. if (n .le. 0) task = 'ERROR: N .LE. 0' if (m .le. 0) task = 'ERROR: M .LE. 0' if (factr .lt. zero) task = 'ERROR: FACTR .LT. 0' c Check the validity of the arrays nbd(i), u(i), and l(i). do 10 i = 1, n if (nbd(i) .lt. 0 .or. nbd(i) .gt. 3) then c return task = 'ERROR: INVALID NBD' info = -6 k = i endif if (nbd(i) .eq. 2) then if (l(i) .gt. u(i)) then c return task = 'ERROR: NO FEASIBLE SOLUTION' info = -7 k = i endif endif 10 continue return end c======================= The end of errclb ============================= subroutine formk(n, nsub, ind, nenter, ileave, indx2, iupdat, + updatd, wn, wn1, m, ws, wy, sy, theta, col, + head, info) integer n, nsub, m, col, head, nenter, ileave, iupdat, + info, ind(n), indx2(n) double precision theta, wn(2*m, 2*m), wn1(2*m, 2*m), + ws(n, m), wy(n, m), sy(m, m) logical updatd c ************ c c Subroutine formk c c This subroutine forms the LEL^T factorization of the indefinite c c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c The matrix K can be shown to be equal to the matrix M^[-1]N c occurring in section 5.1 of [1], as well as to the matrix c Mbar^[-1] Nbar in section 5.3. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c nsub is an integer variable c On entry nsub is the number of subspace variables in free set. c On exit nsub is not changed. c c ind is an integer array of dimension nsub. c On entry ind specifies the indices of subspace variables. c On exit ind is unchanged. c c nenter is an integer variable. c On entry nenter is the number of variables entering the c free set. c On exit nenter is unchanged. c c ileave is an integer variable. c On entry indx2(ileave),...,indx2(n) are the variables leaving c the free set. c On exit ileave is unchanged. c c indx2 is an integer array of dimension n. c On entry indx2(1),...,indx2(nenter) are the variables entering c the free set, while indx2(ileave),...,indx2(n) are the c variables leaving the free set. c On exit indx2 is unchanged. c c iupdat is an integer variable. c On entry iupdat is the total number of BFGS updates made so far. c On exit iupdat is unchanged. c c updatd is a logical variable. c On entry 'updatd' is true if the L-BFGS matrix is updatd. c On exit 'updatd' is unchanged. c c wn is a double precision array of dimension 2m x 2m. c On entry wn is unspecified. c On exit the upper triangle of wn stores the LEL^T factorization c of the 2*col x 2*col indefinite matrix c [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c wn1 is a double precision array of dimension 2m x 2m. c On entry wn1 stores the lower triangular part of c [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c in the previous iteration. c On exit wn1 stores the corresponding updated matrices. c The purpose of wn1 is just to store these inner products c so they can be easily updated and inserted into wn. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wtyy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wtyy(m,m) stores the Cholesky factorization c of (theta*S'S+LD^(-1)L') c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return; c = -1 when the 1st Cholesky factorization failed; c = -2 when the 2st Cholesky factorization failed. c c Subprograms called: c c Linpack ... dcopy, dpofa, dtrsl. c c c References: c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to eecs.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer m2,ipntr,jpntr,iy,is,jy,js,is1,js1,k1,i,k, + col2,pbegin,pend,dbegin,dend,upcl double precision ddot,temp1,temp2,temp3,temp4 double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Form the lower triangular part of c WN1 = [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c where L_a is the strictly lower triangular part of S'AA'Y c R_z is the upper triangular part of S'ZZ'Y. if (updatd) then if (iupdat .gt. m) then c shift old part of WN1. do 10 jy = 1, m - 1 js = m + jy call dcopy(m-jy,wn1(jy+1,jy+1),1,wn1(jy,jy),1) call dcopy(m-jy,wn1(js+1,js+1),1,wn1(js,js),1) call dcopy(m-1,wn1(m+2,jy+1),1,wn1(m+1,jy),1) 10 continue endif c put new rows in blocks (1,1), (2,1) and (2,2). pbegin = 1 pend = nsub dbegin = nsub + 1 dend = n iy = col is = m + col ipntr = head + col - 1 if (ipntr .gt. m) ipntr = ipntr - m jpntr = head do 20 jy = 1, col js = m + jy temp1 = zero temp2 = zero temp3 = zero c compute element jy of row 'col' of Y'ZZ'Y do 15 k = pbegin, pend k1 = ind(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) 15 continue c compute elements jy of row 'col' of L_a and S'AA'S do 16 k = dbegin, dend k1 = ind(k) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 16 continue wn1(iy,jy) = temp1 wn1(is,js) = temp2 wn1(is,jy) = temp3 jpntr = mod(jpntr,m) + 1 20 continue c put new column in block (2,1). jy = col jpntr = head + col - 1 if (jpntr .gt. m) jpntr = jpntr - m ipntr = head do 30 i = 1, col is = m + i temp3 = zero c compute element i of column 'col' of R_z do 25 k = pbegin, pend k1 = ind(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 25 continue ipntr = mod(ipntr,m) + 1 wn1(is,jy) = temp3 30 continue upcl = col - 1 else upcl = col endif c modify the old parts in blocks (1,1) and (2,2) due to changes c in the set of free variables. ipntr = head do 45 iy = 1, upcl is = m + iy jpntr = head do 40 jy = 1, iy js = m + jy temp1 = zero temp2 = zero temp3 = zero temp4 = zero do 35 k = 1, nenter k1 = indx2(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) 35 continue do 36 k = ileave, n k1 = indx2(k) temp3 = temp3 + wy(k1,ipntr)*wy(k1,jpntr) temp4 = temp4 + ws(k1,ipntr)*ws(k1,jpntr) 36 continue wn1(iy,jy) = wn1(iy,jy) + temp1 - temp3 wn1(is,js) = wn1(is,js) - temp2 + temp4 jpntr = mod(jpntr,m) + 1 40 continue ipntr = mod(ipntr,m) + 1 45 continue c modify the old parts in block (2,1). ipntr = head do 60 is = m + 1, m + upcl jpntr = head do 55 jy = 1, upcl temp1 = zero temp3 = zero do 50 k = 1, nenter k1 = indx2(k) temp1 = temp1 + ws(k1,ipntr)*wy(k1,jpntr) 50 continue do 51 k = ileave, n k1 = indx2(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 51 continue if (is .le. jy + m) then wn1(is,jy) = wn1(is,jy) + temp1 - temp3 else wn1(is,jy) = wn1(is,jy) - temp1 + temp3 endif jpntr = mod(jpntr,m) + 1 55 continue ipntr = mod(ipntr,m) + 1 60 continue c Form the upper triangle of WN = [D+Y' ZZ'Y/theta -L_a'+R_z' ] c [-L_a +R_z S'AA'S*theta] m2 = 2*m do 70 iy = 1, col is = col + iy is1 = m + iy do 65 jy = 1, iy js = col + jy js1 = m + jy wn(jy,iy) = wn1(iy,jy)/theta wn(js,is) = wn1(is1,js1)*theta 65 continue do 66 jy = 1, iy - 1 wn(jy,is) = -wn1(is1,jy) 66 continue do 67 jy = iy, col wn(jy,is) = wn1(is1,jy) 67 continue wn(iy,iy) = wn(iy,iy) + sy(iy,iy) 70 continue c Form the upper triangle of WN= [ LL' L^-1(-L_a'+R_z')] c [(-L_a +R_z)L'^-1 S'AA'S*theta ] c first Cholesky factor (1,1) block of wn to get LL' c with L' stored in the upper triangle of wn. call dpofa(wn,m2,col,info) if (info .ne. 0) then info = -1 return endif c then form L^-1(-L_a'+R_z') in the (1,2) block. col2 = 2*col do 71 js = col+1 ,col2 call dtrsl(wn,m2,col,wn(1,js),11,info) 71 continue c Form S'AA'S*theta + (L^-1(-L_a'+R_z'))'L^-1(-L_a'+R_z') in the c upper triangle of (2,2) block of wn. do 72 is = col+1, col2 do 74 js = is, col2 wn(is,js) = wn(is,js) + ddot(col,wn(1,is),1,wn(1,js),1) 74 continue 72 continue c Cholesky factorization of (2,2) block of wn. call dpofa(wn(col+1,col+1),m2,col,info) if (info .ne. 0) then info = -2 return endif return end c======================= The end of formk ============================== subroutine formt(m, wt, sy, ss, col, theta, info) integer m, col, info double precision theta, wt(m, m), sy(m, m), ss(m, m) c ************ c c Subroutine formt c c This subroutine forms the upper half of the pos. def. and symm. c T = theta*SS + L*D^(-1)*L', stores T in the upper triangle c of the array wt, and performs the Cholesky factorization of T c to produce J*J', with J' stored in the upper triangle of wt. c c Subprograms called: c c Linpack ... dpofa. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,k1 double precision ddum double precision zero parameter (zero=0.0d0) c Form the upper half of T = theta*SS + L*D^(-1)*L', c store T in the upper triangle of the array wt. do 52 j = 1, col wt(1,j) = theta*ss(1,j) 52 continue do 55 i = 2, col do 54 j = i, col k1 = min(i,j) - 1 ddum = zero do 53 k = 1, k1 ddum = ddum + sy(i,k)*sy(j,k)/sy(k,k) 53 continue wt(i,j) = ddum + theta*ss(i,j) 54 continue 55 continue c Cholesky factorize T to J*J' with c J' stored in the upper triangle of wt. call dpofa(wt,m,col,info) if (info .ne. 0) then info = -3 endif return end c======================= The end of formt ============================== subroutine freev(n, nfree, index, nenter, ileave, indx2, + iwhere, wrk, updatd, cnstnd, iprint, iter) integer n, nfree, nenter, ileave, iprint, iter, + index(n), indx2(n), iwhere(n) logical wrk, updatd, cnstnd c ************ c c Subroutine freev c c This subroutine counts the entering and leaving variables when c iter > 0, and finds the index set of free and active variables c at the GCP. c c cnstnd is a logical variable indicating whether bounds are present c c index is an integer array of dimension n c for i=1,...,nfree, index(i) are the indices of free variables c for i=nfree+1,...,n, index(i) are the indices of bound variables c On entry after the first iteration, index gives c the free variables at the previous iteration. c On exit it gives the free variables based on the determination c in cauchy using the array iwhere. c c indx2 is an integer array of dimension n c On entry indx2 is unspecified. c On exit with iter>0, indx2 indicates which variables c have changed status since the previous iteration. c For i= 1,...,nenter, indx2(i) have changed from bound to free. c For i= ileave+1,...,n, indx2(i) have changed from free to bound. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer iact,i,k nenter = 0 ileave = n + 1 if (iter .gt. 0 .and. cnstnd) then c count the entering and leaving variables. do 20 i = 1, nfree k = index(i) c write(6,*) ' k = index(i) ', k c write(6,*) ' index = ', i if (iwhere(k) .gt. 0) then ileave = ileave - 1 indx2(ileave) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' leaves the set of free variables' endif 20 continue do 22 i = 1 + nfree, n k = index(i) if (iwhere(k) .le. 0) then nenter = nenter + 1 indx2(nenter) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' enters the set of free variables' endif 22 continue if (iprint .ge. 99) write (6,*) + n+1-ileave,' variables leave; ',nenter,' variables enter' endif wrk = (ileave .lt. n+1) .or. (nenter .gt. 0) .or. updatd c Find the index set of free and active variables at the GCP. nfree = 0 iact = n + 1 do 24 i = 1, n if (iwhere(i) .le. 0) then nfree = nfree + 1 index(nfree) = i else iact = iact - 1 index(iact) = i endif 24 continue if (iprint .ge. 99) write (6,*) + nfree,' variables are free at GCP ',iter + 1 return end c======================= The end of freev ============================== subroutine hpsolb(n, t, iorder, iheap) integer iheap, n, iorder(n) double precision t(n) c ************ c c Subroutine hpsolb c c This subroutine sorts out the least element of t, and puts the c remaining elements of t in a heap. c c n is an integer variable. c On entry n is the dimension of the arrays t and iorder. c On exit n is unchanged. c c t is a double precision array of dimension n. c On entry t stores the elements to be sorted, c On exit t(n) stores the least elements of t, and t(1) to t(n-1) c stores the remaining elements in the form of a heap. c c iorder is an integer array of dimension n. c On entry iorder(i) is the index of t(i). c On exit iorder(i) is still the index of t(i), but iorder may be c permuted in accordance with t. c c iheap is an integer variable specifying the task. c On entry iheap should be set as follows: c iheap .eq. 0 if t(1) to t(n) is not in the form of a heap, c iheap .ne. 0 if otherwise. c On exit iheap is unchanged. c c c References: c Algorithm 232 of CACM (J. W. J. Williams): HEAPSORT. c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c ************ integer i,j,k,indxin,indxou double precision ddum,out if (iheap .eq. 0) then c Rearrange the elements t(1) to t(n) to form a heap. do 20 k = 2, n ddum = t(k) indxin = iorder(k) c Add ddum to the heap. i = k 10 continue if (i.gt.1) then j = i/2 if (ddum .lt. t(j)) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 10 endif endif t(i) = ddum iorder(i) = indxin 20 continue endif c Assign to 'out' the value of t(1), the least member of the heap, c and rearrange the remaining members to form a heap as c elements 1 to n-1 of t. if (n .gt. 1) then i = 1 out = t(1) indxou = iorder(1) ddum = t(n) indxin = iorder(n) c Restore the heap 30 continue j = i+i if (j .le. n-1) then if (t(j+1) .lt. t(j)) j = j+1 if (t(j) .lt. ddum ) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 30 endif endif t(i) = ddum iorder(i) = indxin c Put the least member in t(n). t(n) = out iorder(n) = indxou endif return end c====================== The end of hpsolb ============================== subroutine lnsrlb(n, l, u, nbd, x, f, fold, gd, gdold, g, d, r, t, + z, stp, dnorm, dtd, xstep, stpmx, iter, ifun, + iback, nfgv, info, task, boxed, cnstnd, csave, + isave, dsave) character*60 task, csave logical boxed, cnstnd integer n, iter, ifun, iback, nfgv, info, + nbd(n), isave(2) double precision f, fold, gd, gdold, stp, dnorm, dtd, xstep, + stpmx, x(n), l(n), u(n), g(n), d(n), r(n), t(n), + z(n), dsave(13) c ********** c c Subroutine lnsrlb c c This subroutine calls subroutine dcsrch from the Minpack2 library c to perform the line search. Subroutine dscrch is safeguarded so c that all trial points lie within the feasible region. c c Subprograms called: c c Minpack2 Library ... dcsrch. c c Linpack ... dtrsl, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ********** integer i double precision ddot,a1,a2 double precision one,zero,big parameter (one=1.0d0,zero=0.0d0,big=1.0d+10) double precision ftol,gtol,xtol parameter (ftol=1.0d-3,gtol=0.9d0,xtol=0.1d0) if (task(1:5) .eq. 'FG_LN') goto 556 dtd = ddot(n,d,1,d,1) dnorm = sqrt(dtd) c Determine the maximum step length. stpmx = big if (cnstnd) then if (iter .eq. 0) then stpmx = one else do 43 i = 1, n a1 = d(i) if (nbd(i) .ne. 0) then if (a1 .lt. zero .and. nbd(i) .le. 2) then a2 = l(i) - x(i) if (a2 .ge. zero) then stpmx = zero else if (a1*stpmx .lt. a2) then stpmx = a2/a1 endif else if (a1 .gt. zero .and. nbd(i) .ge. 2) then a2 = u(i) - x(i) if (a2 .le. zero) then stpmx = zero else if (a1*stpmx .gt. a2) then stpmx = a2/a1 endif endif endif 43 continue endif endif if (iter .eq. 0 .and. .not. boxed) then stp = min(one/dnorm, stpmx) else stp = one endif call dcopy(n,x,1,t,1) call dcopy(n,g,1,r,1) fold = f ifun = 0 iback = 0 csave = 'START' 556 continue gd = ddot(n,g,1,d,1) if (ifun .eq. 0) then gdold=gd if (gd .ge. zero) then c the directional derivative >=0. c Line search is impossible. write(6,*)' ascent direction in projection gd = ', gd info = -4 return endif endif call dcsrch(f,gd,stp,ftol,gtol,xtol,zero,stpmx,csave,isave,dsave) xstep = stp*dnorm if (csave(1:4) .ne. 'CONV' .and. csave(1:4) .ne. 'WARN') then task = 'FG_LNSRCH' ifun = ifun + 1 nfgv = nfgv + 1 iback = ifun - 1 if (stp .eq. one) then call dcopy(n,z,1,x,1) else do 41 i = 1, n x(i) = stp*d(i) + t(i) 41 continue endif else task = 'NEW_X' endif return end c======================= The end of lnsrlb ============================= subroutine matupd(n, m, ws, wy, sy, ss, d, r, itail, + iupdat, col, head, theta, rr, dr, stp, dtd) integer n, m, itail, iupdat, col, head double precision theta, rr, dr, stp, dtd, d(n), r(n), + ws(n, m), wy(n, m), sy(m, m), ss(m, m) c ************ c c Subroutine matupd c c This subroutine updates matrices WS and WY, and forms the c middle matrix in B. c c Subprograms called: c c Linpack ... dcopy, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer j,pointr double precision ddot double precision one parameter (one=1.0d0) c Set pointers for matrices WS and WY. if (iupdat .le. m) then col = iupdat itail = mod(head+iupdat-2,m) + 1 else itail = mod(itail,m) + 1 head = mod(head,m) + 1 endif c Update matrices WS and WY. call dcopy(n,d,1,ws(1,itail),1) call dcopy(n,r,1,wy(1,itail),1) c Set theta=yy/ys. theta = rr/dr c Form the middle matrix in B. c update the upper triangle of SS, c and the lower triangle of SY: if (iupdat .gt. m) then c move old information do 50 j = 1, col - 1 call dcopy(j,ss(2,j+1),1,ss(1,j),1) call dcopy(col-j,sy(j+1,j+1),1,sy(j,j),1) 50 continue endif c add new information: the last row of SY c and the last column of SS: pointr = head do 51 j = 1, col - 1 sy(col,j) = ddot(n,d,1,wy(1,pointr),1) ss(j,col) = ddot(n,ws(1,pointr),1,d,1) pointr = mod(pointr,m) + 1 51 continue if (stp .eq. one) then ss(col,col) = dtd else ss(col,col) = stp*stp*dtd endif sy(col,col) = dr return end c======================= The end of matupd ============================= subroutine prn1lb(n, m, l, u, x, iprint, itfile, epsmch) integer n, m, iprint, itfile double precision epsmch, x(n), l(n), u(n) c ************ c c Subroutine prn1lb c c This subroutine prints the input data, initial point, upper and c lower bounds of each variable, machine precision, as well as c the headings of the output. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (iprint .ge. 0) then write (6,7001) epsmch write (6,*) 'N = ',n,' M = ',m if (iprint .ge. 1) then write (itfile,2001) epsmch write (itfile,*)'N = ',n,' M = ',m write (itfile,9001) if (iprint .gt. 100) then write (6,1004) 'L =',(l(i),i = 1,n) write (6,1004) 'X0 =',(x(i),i = 1,n) write (6,1004) 'U =',(u(i),i = 1,n) endif endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format ('RUNNING THE L-BFGS-B CODE',/,/, + 'it = iteration number',/, + 'nf = number of function evaluations',/, + 'nseg = number of segments explored during the Cauchy search',/, + 'nact = number of active bounds at the generalized Cauchy point' + ,/, + 'sub = manner in which the subspace minimization terminated:' + ,/,' con = converged, bnd = a bound was reached',/, + 'itls = number of iterations performed in the line search',/, + 'stepl = step length used',/, + 'tstep = norm of the displacement (total step)',/, + 'projg = norm of the projected gradient',/, + 'f = function value',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 7001 format ('RUNNING THE L-BFGS-B CODE',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 9001 format (/,3x,'it',3x,'nf',2x,'nseg',2x,'nact',2x,'sub',2x,'itls', + 2x,'stepl',4x,'tstep',5x,'projg',8x,'f') return end c======================= The end of prn1lb ============================= subroutine prn2lb(n, x, f, g, iprint, itfile, iter, nfgv, nact, + sbgnrm, nseg, word, iword, iback, stp, xstep) character*3 word integer n, iprint, itfile, iter, nfgv, nact, nseg, + iword, iback double precision f, sbgnrm, stp, xstep, x(n), g(n) c ************ c c Subroutine prn2lb c c This subroutine prints out new information after a successful c line search. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,imod c 'word' records the status of subspace solutions. if (iword .eq. 0) then c the subspace minimization converged. word = 'con' else if (iword .eq. 1) then c the subspace minimization stopped at a bound. word = 'bnd' else if (iword .eq. 5) then c the truncated Newton step has been used. word = 'TNT' else word = '---' endif if (iprint .ge. 99) then write (6,*) 'LINE SEARCH',iback,' times; norm of step = ',xstep write (6,2001) iter,f,sbgnrm if (iprint .gt. 100) then write (6,1004) 'X =',(x(i), i = 1, n) write (6,1004) 'G =',(g(i), i = 1, n) endif else if (iprint .gt. 0) then imod = mod(iter,iprint) if (imod .eq. 0) write (6,2001) iter,f,sbgnrm endif if (iprint .ge. 1) write (itfile,3001) + iter,nfgv,nseg,nact,word,iback,stp,xstep,sbgnrm,f 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 3001 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),1p,2(1x,d10.3)) return end c======================= The end of prn2lb ============================= subroutine prn3lb(n, x, f, task, iprint, info, itfile, + iter, nfgv, nintol, nskip, nact, sbgnrm, + time, nseg, word, iback, stp, xstep, k, + cachyt, sbtime, lnscht) character*60 task character*3 word integer n, iprint, info, itfile, iter, nfgv, nintol, + nskip, nact, nseg, iback, k double precision f, sbgnrm, time, stp, xstep, cachyt, sbtime, + lnscht, x(n) c ************ c c Subroutine prn3lb c c This subroutine prints out information when either a built-in c convergence test is satisfied or when an error message is c generated. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (task(1:5) .eq. 'ERROR') goto 999 if (iprint .ge. 0) then write (6,3003) write (6,3004) write(6,3005) n,iter,nfgv,nintol,nskip,nact,sbgnrm,f if (iprint .ge. 100) then write (6,1004) 'X =',(x(i),i = 1,n) endif if (iprint .ge. 1) write (6,*) ' F =',f endif 999 continue if (iprint .ge. 0) then write (6,3009) task if (info .ne. 0) then if (info .eq. -1) write (6,9011) if (info .eq. -2) write (6,9012) if (info .eq. -3) write (6,9013) if (info .eq. -4) write (6,9014) if (info .eq. -5) write (6,9015) if (info .eq. -6) write (6,*)' Input nbd(',k,') is invalid.' if (info .eq. -7) + write (6,*)' l(',k,') > u(',k,'). No feasible solution.' if (info .eq. -8) write (6,9018) if (info .eq. -9) write (6,9019) endif if (iprint .ge. 1) write (6,3007) cachyt,sbtime,lnscht write (6,3008) time if (iprint .ge. 1) then if (info .eq. -4 .or. info .eq. -9) then write (itfile,3002) + iter,nfgv,nseg,nact,word,iback,stp,xstep endif write (itfile,3009) task if (info .ne. 0) then if (info .eq. -1) write (itfile,9011) if (info .eq. -2) write (itfile,9012) if (info .eq. -3) write (itfile,9013) if (info .eq. -4) write (itfile,9014) if (info .eq. -5) write (itfile,9015) if (info .eq. -8) write (itfile,9018) if (info .eq. -9) write (itfile,9019) endif write (itfile,3008) time endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 3002 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),6x,'-',10x,'-') 3003 format (/, + ' * * *',/,/, + 'Tit = total number of iterations',/, + 'Tnf = total number of function evaluations',/, + 'Tnint = total number of segments explored during', + ' Cauchy searches',/, + 'Skip = number of BFGS updates skipped',/, + 'Nact = number of active bounds at final generalized', + ' Cauchy point',/, + 'Projg = norm of the final projected gradient',/, + 'F = final function value',/,/, + ' * * *') 3004 format (/,3x,'N',4x,'Tit',5x,'Tnf',2x,'Tnint',2x, + 'Skip',2x,'Nact',5x,'Projg',8x,'F') 3005 format (i5,2(1x,i6),(1x,i6),(2x,i4),(1x,i5),1p,2(2x,d10.3)) 3007 format (/,' Cauchy time',1p,e10.3,' seconds.',/ + ' Subspace minimization time',1p,e10.3,' seconds.',/ + ' Line search time',1p,e10.3,' seconds.') 3008 format (/,' Total User time',1p,e10.3,' seconds.',/) 3009 format (/,a60) 9011 format (/, +' Matrix in 1st Cholesky factorization in formk is not Pos. Def.') 9012 format (/, +' Matrix in 2st Cholesky factorization in formk is not Pos. Def.') 9013 format (/, +' Matrix in the Cholesky factorization in formt is not Pos. Def.') 9014 format (/, +' Derivative >= 0, backtracking line search impossible.',/, +' Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding errors dominate computation.') 9015 format (/, +' Warning: more than 10 function and gradient',/, +' evaluations in the last line search. Termination',/, +' may possibly be caused by a bad search direction.') 9018 format (/,' The triangular system is singular.') 9019 format (/, +' Line search cannot locate an adequate point after 20 function',/ +,' and gradient evaluations. Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding error dominate computation.') return end c======================= The end of prn3lb ============================= subroutine projgr(n, l, u, nbd, x, g, sbgnrm) integer n, nbd(n) double precision sbgnrm, x(n), l(n), u(n), g(n) c ************ c c Subroutine projgr c c This subroutine computes the infinity norm of the projected c gradient. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision gi double precision one,zero parameter (one=1.0d0,zero=0.0d0) sbgnrm = zero do 15 i = 1, n gi = g(i) if (nbd(i) .ne. 0) then if (gi .lt. zero) then if (nbd(i) .ge. 2) gi = max((x(i)-u(i)),gi) else if (nbd(i) .le. 2) gi = min((x(i)-l(i)),gi) endif endif sbgnrm = max(sbgnrm,abs(gi)) 15 continue return end c======================= The end of projgr ============================= subroutine subsm ( n, m, nsub, ind, l, u, nbd, x, d, xp, ws, wy, + theta, xx, gg, + col, head, iword, wv, wn, iprint, info ) implicit none integer n, m, nsub, col, head, iword, iprint, info, + ind(nsub), nbd(n) double precision theta, + l(n), u(n), x(n), d(n), xp(n), xx(n), gg(n), + ws(n, m), wy(n, m), + wv(2*m), wn(2*m, 2*m) c ********************************************************************** c c This routine contains the major changes in the updated version. c The changes are described in the accompanying paper c c Jose Luis Morales, Jorge Nocedal c "Remark On Algorithm 788: L-BFGS-B: Fortran Subroutines for Large-Scale c Bound Constrained Optimization". Decemmber 27, 2010. c c J.L. Morales Departamento de Matematicas, c Instituto Tecnologico Autonomo de Mexico c Mexico D.F. c c J, Nocedal Department of Electrical Engineering and c Computer Science. c Northwestern University. Evanston, IL. USA c c January 17, 2011 c c ********************************************************************** c c c Subroutine subsm c c Given xcp, l, u, r, an index set that specifies c the active set at xcp, and an l-BFGS matrix B c (in terms of WY, WS, SY, WT, head, col, and theta), c this subroutine computes an approximate solution c of the subspace problem c c (P) min Q(x) = r'(x-xcp) + 1/2 (x-xcp)' B (x-xcp) c c subject to l<=x<=u c x_i=xcp_i for all i in A(xcp) c c along the subspace unconstrained Newton direction c c d = -(Z'BZ)^(-1) r. c c The formula for the Newton direction, given the L-BFGS matrix c and the Sherman-Morrison formula, is c c d = (1/theta)r + (1/theta*2) Z'WK^(-1)W'Z r. c c where c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c Note that this procedure for computing d differs c from that described in [1]. One can show that the matrix K is c equal to the matrix M^[-1]N in that paper. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c nsub is an integer variable. c On entry nsub is the number of free variables. c On exit nsub is unchanged. c c ind is an integer array of dimension nsub. c On entry ind specifies the coordinate indices of free variables. c On exit ind is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is a integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c x is a double precision array of dimension n. c On entry x specifies the Cauchy point xcp. c On exit x(i) is the minimizer of Q over the subspace of c free variables. c c d is a double precision array of dimension n. c On entry d is the reduced gradient of Q at xcp. c On exit d is the Newton direction of Q. c c xp is a double precision array of dimension n. c used to safeguard the projected Newton direction c c xx is a double precision array of dimension n c On entry it holds the current iterate c On output it is unchanged c gg is a double precision array of dimension n c On entry it holds the gradient at the current iterate c On output it is unchanged c c ws and wy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c iword is an integer variable. c On entry iword is unspecified. c On exit iword specifies the status of the subspace solution. c iword = 0 if the solution is in the box, c 1 if some bound is encountered. c c wv is a double precision working array of dimension 2m. c c wn is a double precision array of dimension 2m x 2m. c On entry the upper triangle of wn stores the LEL^T factorization c of the indefinite matrix c c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c On exit wn is unchanged. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return c when the matrix K is ill-conditioned. c c Subprograms called: c c Linpack dtrsl. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer pointr,m2,col2,ibd,jy,js,i,j,k double precision alpha, xk, dk, temp1, temp2 double precision one,zero parameter (one=1.0d0,zero=0.0d0) c double precision dd_p if (nsub .le. 0) return if (iprint .ge. 99) write (6,1001) c Compute wv = W'Zd. pointr = head do 20 i = 1, col temp1 = zero temp2 = zero do 10 j = 1, nsub k = ind(j) temp1 = temp1 + wy(k,pointr)*d(j) temp2 = temp2 + ws(k,pointr)*d(j) 10 continue wv(i) = temp1 wv(col + i) = theta*temp2 pointr = mod(pointr,m) + 1 20 continue c Compute wv:=K^(-1)wv. m2 = 2*m col2 = 2*col call dtrsl(wn,m2,col2,wv,11,info) if (info .ne. 0) return do 25 i = 1, col wv(i) = -wv(i) 25 continue call dtrsl(wn,m2,col2,wv,01,info) if (info .ne. 0) return c Compute d = (1/theta)d + (1/theta**2)Z'W wv. pointr = head do 40 jy = 1, col js = col + jy do 30 i = 1, nsub k = ind(i) d(i) = d(i) + wy(k,pointr)*wv(jy)/theta + + ws(k,pointr)*wv(js) 30 continue pointr = mod(pointr,m) + 1 40 continue call dscal( nsub, one/theta, d, 1 ) c c----------------------------------------------------------------- c Let us try the projection, d is the Newton direction iword = 0 call dcopy ( n, x, 1, xp, 1 ) c do 50 i=1, nsub k = ind(i) dk = d(i) xk = x(k) if ( nbd(k) .ne. 0 ) then c if ( nbd(k).eq.1 ) then ! lower bounds only x(k) = max( l(k), xk + dk ) if ( x(k).eq.l(k) ) iword = 1 else c if ( nbd(k).eq.2 ) then ! upper and lower bounds xk = max( l(k), xk + dk ) x(k) = min( u(k), xk ) if ( x(k).eq.l(k) .or. x(k).eq.u(k) ) iword = 1 else c if ( nbd(k).eq.3 ) then ! upper bounds only x(k) = min( u(k), xk + dk ) if ( x(k).eq.u(k) ) iword = 1 end if end if end if c else ! free variables x(k) = xk + dk end if 50 continue c if ( iword.eq.0 ) then go to 911 end if c c check sign of the directional derivative c dd_p = zero do 55 i=1, n dd_p = dd_p + (x(i) - xx(i))*gg(i) 55 continue if ( dd_p .gt.zero ) then call dcopy( n, xp, 1, x, 1 ) write(6,*) ' Positive dir derivative in projection ' write(6,*) ' Using the backtracking step ' else go to 911 endif c c----------------------------------------------------------------- c alpha = one temp1 = alpha ibd = 0 do 60 i = 1, nsub k = ind(i) dk = d(i) if (nbd(k) .ne. 0) then if (dk .lt. zero .and. nbd(k) .le. 2) then temp2 = l(k) - x(k) if (temp2 .ge. zero) then temp1 = zero else if (dk*alpha .lt. temp2) then temp1 = temp2/dk endif else if (dk .gt. zero .and. nbd(k) .ge. 2) then temp2 = u(k) - x(k) if (temp2 .le. zero) then temp1 = zero else if (dk*alpha .gt. temp2) then temp1 = temp2/dk endif endif if (temp1 .lt. alpha) then alpha = temp1 ibd = i endif endif 60 continue if (alpha .lt. one) then dk = d(ibd) k = ind(ibd) if (dk .gt. zero) then x(k) = u(k) d(ibd) = zero else if (dk .lt. zero) then x(k) = l(k) d(ibd) = zero endif endif do 70 i = 1, nsub k = ind(i) x(k) = x(k) + alpha*d(i) 70 continue cccccc 911 continue if (iprint .ge. 99) write (6,1004) 1001 format (/,'----------------SUBSM entered-----------------',/) 1004 format (/,'----------------exit SUBSM --------------------',/) return end c====================== The end of subsm =============================== subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, + task,isave,dsave) character*(*) task integer isave(2) double precision f,g,stp,ftol,gtol,xtol,stpmin,stpmax double precision dsave(13) c ********** c c Subroutine dcsrch c c This subroutine finds a step that satisfies a sufficient c decrease condition and a curvature condition. c c Each call of the subroutine updates an interval with c endpoints stx and sty. The interval is initially chosen c so that it contains a minimizer of the modified function c c psi(stp) = f(stp) - f(0) - ftol*stp*f'(0). c c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c interval is chosen so that it contains a minimizer of f. c c The algorithm is designed to find a step that satisfies c the sufficient decrease condition c c f(stp) <= f(0) + ftol*stp*f'(0), c c and the curvature condition c c abs(f'(stp)) <= gtol*abs(f'(0)). c c If ftol is less than gtol and if, for example, the function c is bounded below, then there is always a step which satisfies c both conditions. c c If no step can be found that satisfies both conditions, then c the algorithm stops with a warning. In this case stp only c satisfies the sufficient decrease condition. c c A typical invocation of dcsrch has the following outline: c c task = 'START' c 10 continue c call dcsrch( ... ) c if (task .eq. 'FG') then c Evaluate the function and the gradient at stp c goto 10 c end if c c NOTE: The user must no alter work arrays between calls. c c The subroutine statement is c c subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, c task,isave,dsave) c where c c f is a double precision variable. c On initial entry f is the value of the function at 0. c On subsequent entries f is the value of the c function at stp. c On exit f is the value of the function at stp. c c g is a double precision variable. c On initial entry g is the derivative of the function at 0. c On subsequent entries g is the derivative of the c function at stp. c On exit g is the derivative of the function at stp. c c stp is a double precision variable. c On entry stp is the current estimate of a satisfactory c step. On initial entry, a positive initial estimate c must be provided. c On exit stp is the current estimate of a satisfactory step c if task = 'FG'. If task = 'CONV' then stp satisfies c the sufficient decrease and curvature condition. c c ftol is a double precision variable. c On entry ftol specifies a nonnegative tolerance for the c sufficient decrease condition. c On exit ftol is unchanged. c c gtol is a double precision variable. c On entry gtol specifies a nonnegative tolerance for the c curvature condition. c On exit gtol is unchanged. c c xtol is a double precision variable. c On entry xtol specifies a nonnegative relative tolerance c for an acceptable step. The subroutine exits with a c warning if the relative difference between sty and stx c is less than xtol. c On exit xtol is unchanged. c c stpmin is a double precision variable. c On entry stpmin is a nonnegative lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is a nonnegative upper bound for the step. c On exit stpmax is unchanged. c c task is a character variable of length at least 60. c On initial entry task must be set to 'START'. c On exit task indicates the required action: c c If task(1:2) = 'FG' then evaluate the function and c derivative at stp and call dcsrch again. c c If task(1:4) = 'CONV' then the search is successful. c c If task(1:4) = 'WARN' then the subroutine is not able c to satisfy the convergence conditions. The exit value of c stp contains the best point found during the search. c c If task(1:5) = 'ERROR' then there is an error in the c input arguments. c c On exit with convergence, a warning or an error, the c variable task contains additional information. c c isave is an integer work array of dimension 2. c c dsave is a double precision work array of dimension 13. c c Subprograms called c c MINPACK-2 ... dcstep c c MINPACK-1 Project. June 1983. c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick, Richard G. Carter, and Jorge J. More'. c c ********** double precision zero,p5,p66 parameter(zero=0.0d0,p5=0.5d0,p66=0.66d0) double precision xtrapl,xtrapu parameter(xtrapl=1.1d0,xtrapu=4.0d0) logical brackt integer stage double precision finit,ftest,fm,fx,fxm,fy,fym,ginit,gtest, + gm,gx,gxm,gy,gym,stx,sty,stmin,stmax,width,width1 c Initialization block. if (task(1:5) .eq. 'START') then c Check the input arguments for errors. if (stp .lt. stpmin) task = 'ERROR: STP .LT. STPMIN' if (stp .gt. stpmax) task = 'ERROR: STP .GT. STPMAX' if (g .ge. zero) task = 'ERROR: INITIAL G .GE. ZERO' if (ftol .lt. zero) task = 'ERROR: FTOL .LT. ZERO' if (gtol .lt. zero) task = 'ERROR: GTOL .LT. ZERO' if (xtol .lt. zero) task = 'ERROR: XTOL .LT. ZERO' if (stpmin .lt. zero) task = 'ERROR: STPMIN .LT. ZERO' if (stpmax .lt. stpmin) task = 'ERROR: STPMAX .LT. STPMIN' c Exit if there are errors on input. if (task(1:5) .eq. 'ERROR') return c Initialize local variables. brackt = .false. stage = 1 finit = f ginit = g gtest = ftol*ginit width = stpmax - stpmin width1 = width/p5 c The variables stx, fx, gx contain the values of the step, c function, and derivative at the best step. c The variables sty, fy, gy contain the value of the step, c function, and derivative at sty. c The variables stp, f, g contain the values of the step, c function, and derivative at stp. stx = zero fx = finit gx = ginit sty = zero fy = finit gy = ginit stmin = zero stmax = stp + xtrapu*stp task = 'FG' goto 1000 else c Restore local variables. if (isave(1) .eq. 1) then brackt = .true. else brackt = .false. endif stage = isave(2) ginit = dsave(1) gtest = dsave(2) gx = dsave(3) gy = dsave(4) finit = dsave(5) fx = dsave(6) fy = dsave(7) stx = dsave(8) sty = dsave(9) stmin = dsave(10) stmax = dsave(11) width = dsave(12) width1 = dsave(13) endif c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c algorithm enters the second stage. ftest = finit + stp*gtest if (stage .eq. 1 .and. f .le. ftest .and. g .ge. zero) + stage = 2 c Test for warnings. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax)) + task = 'WARNING: ROUNDING ERRORS PREVENT PROGRESS' if (brackt .and. stmax - stmin .le. xtol*stmax) + task = 'WARNING: XTOL TEST SATISFIED' if (stp .eq. stpmax .and. f .le. ftest .and. g .le. gtest) + task = 'WARNING: STP = STPMAX' if (stp .eq. stpmin .and. (f .gt. ftest .or. g .ge. gtest)) + task = 'WARNING: STP = STPMIN' c Test for convergence. if (f .le. ftest .and. abs(g) .le. gtol*(-ginit)) + task = 'CONVERGENCE' c Test for termination. if (task(1:4) .eq. 'WARN' .or. task(1:4) .eq. 'CONV') goto 1000 c A modified function is used to predict the step during the c first stage if a lower function value has been obtained but c the decrease is not sufficient. if (stage .eq. 1 .and. f .le. fx .and. f .gt. ftest) then c Define the modified function and derivative values. fm = f - stp*gtest fxm = fx - stx*gtest fym = fy - sty*gtest gm = g - gtest gxm = gx - gtest gym = gy - gtest c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fxm,gxm,sty,fym,gym,stp,fm,gm, + brackt,stmin,stmax) c Reset the function and derivative values for f. fx = fxm + stx*gtest fy = fym + sty*gtest gx = gxm + gtest gy = gym + gtest else c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fx,gx,sty,fy,gy,stp,f,g, + brackt,stmin,stmax) endif c Decide if a bisection step is needed. if (brackt) then if (abs(sty-stx) .ge. p66*width1) stp = stx + p5*(sty - stx) width1 = width width = abs(sty-stx) endif c Set the minimum and maximum steps allowed for stp. if (brackt) then stmin = min(stx,sty) stmax = max(stx,sty) else stmin = stp + xtrapl*(stp - stx) stmax = stp + xtrapu*(stp - stx) endif c Force the step to be within the bounds stpmax and stpmin. stp = max(stp,stpmin) stp = min(stp,stpmax) c If further progress is not possible, let stp be the best c point obtained during the search. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax) + .or. (brackt .and. stmax-stmin .le. xtol*stmax)) stp = stx c Obtain another function and derivative. task = 'FG' 1000 continue c Save local variables. if (brackt) then isave(1) = 1 else isave(1) = 0 endif isave(2) = stage dsave(1) = ginit dsave(2) = gtest dsave(3) = gx dsave(4) = gy dsave(5) = finit dsave(6) = fx dsave(7) = fy dsave(8) = stx dsave(9) = sty dsave(10) = stmin dsave(11) = stmax dsave(12) = width dsave(13) = width1 return end c====================== The end of dcsrch ============================== subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, + stpmin,stpmax) logical brackt double precision stx,fx,dx,sty,fy,dy,stp,fp,dp,stpmin,stpmax c ********** c c Subroutine dcstep c c This subroutine computes a safeguarded step for a search c procedure and updates an interval that contains a step that c satisfies a sufficient decrease and a curvature condition. c c The parameter stx contains the step with the least function c value. If brackt is set to .true. then a minimizer has c been bracketed in an interval with endpoints stx and sty. c The parameter stp contains the current step. c The subroutine assumes that if brackt is set to .true. then c c min(stx,sty) < stp < max(stx,sty), c c and that the derivative at stx is negative in the direction c of the step. c c The subroutine statement is c c subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, c stpmin,stpmax) c c where c c stx is a double precision variable. c On entry stx is the best step obtained so far and is an c endpoint of the interval that contains the minimizer. c On exit stx is the updated best step. c c fx is a double precision variable. c On entry fx is the function at stx. c On exit fx is the function at stx. c c dx is a double precision variable. c On entry dx is the derivative of the function at c stx. The derivative must be negative in the direction of c the step, that is, dx and stp - stx must have opposite c signs. c On exit dx is the derivative of the function at stx. c c sty is a double precision variable. c On entry sty is the second endpoint of the interval that c contains the minimizer. c On exit sty is the updated endpoint of the interval that c contains the minimizer. c c fy is a double precision variable. c On entry fy is the function at sty. c On exit fy is the function at sty. c c dy is a double precision variable. c On entry dy is the derivative of the function at sty. c On exit dy is the derivative of the function at the exit sty. c c stp is a double precision variable. c On entry stp is the current step. If brackt is set to .true. c then on input stp must be between stx and sty. c On exit stp is a new trial step. c c fp is a double precision variable. c On entry fp is the function at stp c On exit fp is unchanged. c c dp is a double precision variable. c On entry dp is the the derivative of the function at stp. c On exit dp is unchanged. c c brackt is an logical variable. c On entry brackt specifies if a minimizer has been bracketed. c Initially brackt must be set to .false. c On exit brackt specifies if a minimizer has been bracketed. c When a minimizer is bracketed brackt is set to .true. c c stpmin is a double precision variable. c On entry stpmin is a lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is an upper bound for the step. c On exit stpmax is unchanged. c c MINPACK-1 Project. June 1983 c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick and Jorge J. More'. c c ********** double precision zero,p66,two,three parameter(zero=0.0d0,p66=0.66d0,two=2.0d0,three=3.0d0) double precision gamma,p,q,r,s,sgnd,stpc,stpf,stpq,theta sgnd = dp*(dx/abs(dx)) c First case: A higher function value. The minimum is bracketed. c If the cubic step is closer to stx than the quadratic step, the c cubic step is taken, otherwise the average of the cubic and c quadratic steps is taken. if (fp .gt. fx) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .lt. stx) gamma = -gamma p = (gamma - dx) + theta q = ((gamma - dx) + gamma) + dp r = p/q stpc = stx + r*(stp - stx) stpq = stx + ((dx/((fx - fp)/(stp - stx) + dx))/two)* + (stp - stx) if (abs(stpc-stx) .lt. abs(stpq-stx)) then stpf = stpc else stpf = stpc + (stpq - stpc)/two endif brackt = .true. c Second case: A lower function value and derivatives of opposite c sign. The minimum is bracketed. If the cubic step is farther from c stp than the secant step, the cubic step is taken, otherwise the c secant step is taken. else if (sgnd .lt. zero) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dx r = p/q stpc = stp + r*(stx - stp) stpq = stp + (dp/(dp - dx))*(stx - stp) if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif brackt = .true. c Third case: A lower function value, derivatives of the same sign, c and the magnitude of the derivative decreases. else if (abs(dp) .lt. abs(dx)) then c The cubic step is computed only if the cubic tends to infinity c in the direction of the step or if the minimum of the cubic c is beyond stp. Otherwise the cubic step is defined to be the c secant step. theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) c The case gamma = 0 only arises if the cubic does not tend c to infinity in the direction of the step. gamma = s*sqrt(max(zero,(theta/s)**2-(dx/s)*(dp/s))) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = (gamma + (dx - dp)) + gamma r = p/q if (r .lt. zero .and. gamma .ne. zero) then stpc = stp + r*(stx - stp) else if (stp .gt. stx) then stpc = stpmax else stpc = stpmin endif stpq = stp + (dp/(dp - dx))*(stx - stp) if (brackt) then c A minimizer has been bracketed. If the cubic step is c closer to stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .lt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif if (stp .gt. stx) then stpf = min(stp+p66*(sty-stp),stpf) else stpf = max(stp+p66*(sty-stp),stpf) endif else c A minimizer has not been bracketed. If the cubic step is c farther from stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif stpf = min(stpmax,stpf) stpf = max(stpmin,stpf) endif c Fourth case: A lower function value, derivatives of the same sign, c and the magnitude of the derivative does not decrease. If the c minimum is not bracketed, the step is either stpmin or stpmax, c otherwise the cubic step is taken. else if (brackt) then theta = three*(fp - fy)/(sty - stp) + dy + dp s = max(abs(theta),abs(dy),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dy/s)*(dp/s)) if (stp .gt. sty) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dy r = p/q stpc = stp + r*(sty - stp) stpf = stpc else if (stp .gt. stx) then stpf = stpmax else stpf = stpmin endif endif c Update the interval which contains a minimizer. if (fp .gt. fx) then sty = stp fy = fp dy = dp else if (sgnd .lt. zero) then sty = stx fy = fx dy = dx endif stx = stp fx = fp dx = dp endif c Compute the new step. stp = stpf return end gpml-matlab-v4.0-2016-10-19/util/lbfgsb/solver_2_4.f000644 000765 000024 00000423410 12701705601 021423 0ustar00carlstaff000000 000000 c================ L-BFGS-B (version 2.4) ========================== subroutine setulb(n, m, x, l, u, nbd, f, g, factr, pgtol, wa, iwa, + task, iprint, csave, lsave, isave, dsave) character*60 task, csave logical lsave(4) integer n, m, iprint, + nbd(n), iwa(3*n), isave(44) double precision f, factr, pgtol, x(n), l(n), u(n), g(n), + wa(2*m*n+4*n+11*m*m+8*m), dsave(29) c ************ c c Subroutine setulb c c This subroutine partitions the working arrays wa and iwa, and c then uses the limited memory BFGS method to solve the bound c constrained optimization problem by calling mainlb. c (The direct method will be used in the subspace minimization.) c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound on x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound on x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. Typical values for factr: 1.d+12 for c low accuracy; 1.d+7 for moderate accuracy; 1.d+1 for extremely c high accuracy. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c wa is a double precision working array of length c (2mmax + 4)nmax + 11mmax^2 + 8mmax. c c iwa is an integer working array of length 3nmax. c c task is a working string of characters of length 60 indicating c the current job when entering and quitting this subroutine. c c iprint is an integer variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c On exit with 'task' = NEW_X, the following information is c available: c If lsave(1) = .true. then the initial X has been replaced by c its projection in the feasible set; c If lsave(2) = .true. then the problem is constrained; c If lsave(3) = .true. then each variable has upper and lower c bounds; c c isave is an integer working array of dimension 44. c On exit with 'task' = NEW_X, the following information is c available: c isave(22) = the total number of intervals explored in the c search of Cauchy points; c isave(26) = the total number of skipped BFGS updates before c the current iteration; c isave(30) = the number of current iteration; c isave(31) = the total number of BFGS updates prior the current c iteration; c isave(33) = the number of intervals explored in the search of c Cauchy point in the current iteration; c isave(34) = the total number of function and gradient c evaluations; c isave(36) = the number of function value or gradient c evaluations in the current iteration; c if isave(37) = 0 then the subspace argmin is within the box; c if isave(37) = 1 then the subspace argmin is beyond the box; c isave(38) = the number of free variables in the current c iteration; c isave(39) = the number of active constraints in the current c iteration; c n + 1 - isave(40) = the number of variables leaving the set of c active constraints in the current iteration; c isave(41) = the number of variables entering the set of active c constraints in the current iteration. c c dsave is a double precision working array of dimension 29. c On exit with 'task' = NEW_X, the following information is c available: c dsave(1) = current 'theta' in the BFGS matrix; c dsave(2) = f(x) in the previous iteration; c dsave(3) = factr*epsmch; c dsave(4) = 2-norm of the line search direction vector; c dsave(5) = the machine precision epsmch generated by the code; c dsave(7) = the accumulated time spent on searching for c Cauchy points; c dsave(8) = the accumulated time spent on c subspace minimization; c dsave(9) = the accumulated time spent on line search; c dsave(11) = the slope of the line search function at c the current point of line search; c dsave(12) = the maximum relative step length imposed in c line search; c dsave(13) = the infinity norm of the projected gradient; c dsave(14) = the relative step length in the line search; c dsave(15) = the slope of the line search function at c the starting point of the line search; c dsave(16) = the square of the 2-norm of the line search c direction vector. c c Subprograms called: c c L-BFGS-B Library ... mainlb. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer l1,l2,l3,lws,lr,lz,lt,ld,lwa,lwy,lsy,lss,lwt,lwn,lsnd if (task .eq. 'START') then isave(1) = m*n isave(2) = m**2 isave(3) = 4*m**2 isave(4) = 1 isave(5) = isave(4) + isave(1) isave(6) = isave(5) + isave(1) isave(7) = isave(6) + isave(2) isave(8) = isave(7) + isave(2) isave(9) = isave(8) isave(10) = isave(9) + isave(2) isave(11) = isave(10) + isave(3) isave(12) = isave(11) + isave(3) isave(13) = isave(12) + n isave(14) = isave(13) + n isave(15) = isave(14) + n isave(16) = isave(15) + n endif l1 = isave(1) l2 = isave(2) l3 = isave(3) lws = isave(4) lwy = isave(5) lsy = isave(6) lss = isave(7) lwt = isave(9) lwn = isave(10) lsnd = isave(11) lz = isave(12) lr = isave(13) ld = isave(14) lt = isave(15) lwa = isave(16) call mainlb(n,m,x,l,u,nbd,f,g,factr,pgtol, + wa(lws),wa(lwy),wa(lsy),wa(lss),wa(lwt), + wa(lwn),wa(lsnd),wa(lz),wa(lr),wa(ld),wa(lt), + wa(lwa),iwa(1),iwa(n+1),iwa(2*n+1),task,iprint, + csave,lsave,isave(22),dsave) return end c======================= The end of setulb ============================= subroutine mainlb(n, m, x, l, u, nbd, f, g, factr, pgtol, ws, wy, + sy, ss, wt, wn, snd, z, r, d, t, wa, + index, iwhere, indx2, task, iprint, + csave, lsave, isave, dsave) character*60 task, csave logical lsave(4) integer n, m, iprint, nbd(n), index(n), + iwhere(n), indx2(n), isave(23) double precision f, factr, pgtol, + x(n), l(n), u(n), g(n), z(n), r(n), d(n), t(n), + wa(8*m), ws(n, m), wy(n, m), sy(m, m), ss(m, m), + wt(m, m), wn(2*m, 2*m), snd(2*m, 2*m), dsave(29) c ************ c c Subroutine mainlb c c This subroutine solves bound constrained optimization problems by c using the compact formula of the limited memory BFGS updates. c c n is an integer variable. c On entry n is the number of variables. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric c corrections allowed in the limited memory matrix. c On exit m is unchanged. c c x is a double precision array of dimension n. c On entry x is an approximation to the solution. c On exit x is the current approximation. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c f is a double precision variable. c On first entry f is unspecified. c On final exit f is the value of the function at x. c c g is a double precision array of dimension n. c On first entry g is unspecified. c On final exit g is the value of the gradient at x. c c factr is a double precision variable. c On entry factr >= 0 is specified by the user. The iteration c will stop when c c (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch c c where epsmch is the machine precision, which is automatically c generated by the code. c On exit factr is unchanged. c c pgtol is a double precision variable. c On entry pgtol >= 0 is specified by the user. The iteration c will stop when c c max{|proj g_i | i = 1, ..., n} <= pgtol c c where pg_i is the ith component of the projected gradient. c On exit pgtol is unchanged. c c ws, wy, sy, and wt are double precision working arrays used to c store the following information defining the limited memory c BFGS matrix: c ws, of dimension n x m, stores S, the matrix of s-vectors; c wy, of dimension n x m, stores Y, the matrix of y-vectors; c sy, of dimension m x m, stores S'Y; c ss, of dimension m x m, stores S'S; c wt, of dimension m x m, stores the Cholesky factorization c of (theta*S'S+LD^(-1)L'); see eq. c (2.26) in [3]. c c wn is a double precision working array of dimension 2m x 2m c used to store the LEL^T factorization of the indefinite matrix c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c where E = [-I 0] c [ 0 I] c c snd is a double precision working array of dimension 2m x 2m c used to store the lower triangular part of c N = [Y' ZZ'Y L_a'+R_z'] c [L_a +R_z S'AA'S ] c c z(n),r(n),d(n),t(n),wa(8*m) are double precision working arrays. c z is used at different times to store the Cauchy point and c the Newton point. c c c index is an integer working array of dimension n. c In subroutine freev, index is used to store the free and fixed c variables at the Generalized Cauchy Point (GCP). c c iwhere is an integer working array of dimension n used to record c the status of the vector x for GCP computation. c iwhere(i)=0 or -3 if x(i) is free and has bounds, c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., no bounds on it. c c indx2 is an integer working array of dimension n. c Within subroutine cauchy, indx2 corresponds to the array iorder. c In subroutine freev, a list of variables entering and leaving c the free set is stored in indx2, and it is passed on to c subroutine formk with this information. c c task is a working string of characters of length 60 indicating c the current job when entering and leaving this subroutine. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c csave is a working string of characters of length 60. c c lsave is a logical working array of dimension 4. c c isave is an integer working array of dimension 23. c c dsave is a double precision working array of dimension 29. c c c Subprograms called c c L-BFGS-B Library ... cauchy, subsm, lnsrlb, formk, c c errclb, prn1lb, prn2lb, prn3lb, active, projgr, c c freev, cmprlb, matupd, formt. c c Minpack2 Library ... timer, dpmeps. c c Linpack Library ... dcopy, ddot. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c [3] R. Byrd, J. Nocedal and R. Schnabel "Representations of c Quasi-Newton Matrices and their use in Limited Memory Methods'', c Mathematical Programming 63 (1994), no. 4, pp. 129-156. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical prjctd,cnstnd,boxed,updatd,wrk character*3 word integer i,k,nintol,itfile,iback,nskip, + head,col,iter,itail,iupdat, + nint,nfgv,info,ifun, + iword,nfree,nact,ileave,nenter double precision theta,fold,ddot,dr,rr,tol,dpmeps, + xstep,sbgnrm,ddum,dnorm,dtd,epsmch, + cpu1,cpu2,cachyt,sbtime,lnscht,time1,time2, + gd,gdold,stp,stpmx,time double precision one,zero parameter (one=1.0d0,zero=0.0d0) if (task .eq. 'START') then call timer(time1) c Generate the current machine precision. epsmch = dpmeps() fold = 0.0d0 dnorm = 0.0d0 cpu1 = 0.0d0 gd = 0.0d0 sbgnrm = 0.0d0 stp = 0.0d0 stpmx = 0.0d0 gdold = 0.0d0 dtd = 0.0d0 c Initialize counters and scalars when task='START'. c for the limited memory BFGS matrices: col = 0 head = 1 theta = one iupdat = 0 updatd = .false. iback = 0 itail = 0 ifun = 0 iword = 0 nact = 0 ileave = 0 nenter = 0 c for operation counts: iter = 0 nfgv = 0 nint = 0 nintol = 0 nskip = 0 nfree = n c for stopping tolerance: tol = factr*epsmch c for measuring running time: cachyt = 0 sbtime = 0 lnscht = 0 c 'word' records the status of subspace solutions. word = '---' c 'info' records the termination information. info = 0 itfile = 0 if (iprint .ge. 1) then c open a summary file 'iterate.dat' open (8, file = 'iterate.dat', status = 'unknown') itfile = 8 endif c Check the input arguments for errors. call errclb(n,m,factr,l,u,nbd,task,info,k) if (task(1:5) .eq. 'ERROR') then call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + zero,nint,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) return endif call prn1lb(n,m,l,u,x,iprint,itfile,epsmch) c Initialize iwhere & project x onto the feasible set. call active(n,l,u,nbd,x,iwhere,iprint,prjctd,cnstnd,boxed) c The end of the initialization. else c restore local variables. prjctd = lsave(1) cnstnd = lsave(2) boxed = lsave(3) updatd = lsave(4) nintol = isave(1) itfile = isave(3) iback = isave(4) nskip = isave(5) head = isave(6) col = isave(7) itail = isave(8) iter = isave(9) iupdat = isave(10) nint = isave(12) nfgv = isave(13) info = isave(14) ifun = isave(15) iword = isave(16) nfree = isave(17) nact = isave(18) ileave = isave(19) nenter = isave(20) theta = dsave(1) fold = dsave(2) tol = dsave(3) dnorm = dsave(4) epsmch = dsave(5) cpu1 = dsave(6) cachyt = dsave(7) sbtime = dsave(8) lnscht = dsave(9) time1 = dsave(10) gd = dsave(11) stpmx = dsave(12) sbgnrm = dsave(13) stp = dsave(14) gdold = dsave(15) dtd = dsave(16) c After returning from the driver go to the point where execution c is to resume. if (task(1:5) .eq. 'FG_LN') goto 666 if (task(1:5) .eq. 'NEW_X') goto 777 if (task(1:5) .eq. 'FG_ST') goto 111 if (task(1:4) .eq. 'STOP') then if (task(7:9) .eq. 'CPU') then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold endif goto 999 endif endif c Compute f0 and g0. task = 'FG_START' c return to the driver to calculate f and g; reenter at 111. goto 1000 111 continue nfgv = 1 c Compute the infinity norm of the (-) projected gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) if (iprint .ge. 1) then write (6,1002) iter,f,sbgnrm write (itfile,1003) iter,nfgv,sbgnrm,f endif if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM OF PROJECTED GRADIENT <= PGTOL' goto 999 endif c ----------------- the beginning of the loop -------------------------- 222 continue if (iprint .ge. 99) write (6,1001) iter + 1 iword = -1 c if (.not. cnstnd .and. col .gt. 0) then c skip the search for GCP. call dcopy(n,x,1,z,1) wrk = updatd nint = 0 goto 333 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Compute the Generalized Cauchy Point (GCP). c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) call cauchy(n,x,l,u,nbd,g,indx2,iwhere,t,d,z, + m,wy,ws,sy,wt,theta,col,head, + wa(1),wa(2*m+1),wa(4*m+1),wa(6*m+1),nint, + iprint,sbgnrm,info,epsmch) if (info .ne. 0) then c singular triangular system detected; refresh the lbfgs memory. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 goto 222 endif call timer(cpu2) cachyt = cachyt + cpu2 - cpu1 nintol = nintol + nint c Count the entering and leaving variables for iter > 0; c find the index set of free and active variables at the GCP. call freev(n,nfree,index,nenter,ileave,indx2, + iwhere,wrk,updatd,cnstnd,iprint,iter) nact = n - nfree 333 continue c If there are no free variables or B=theta*I, then c skip the subspace minimization. if (nfree .eq. 0 .or. col .eq. 0) goto 555 cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Subspace minimization. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc call timer(cpu1) c Form the LEL^T factorization of the indefinite c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] if (wrk) call formk(n,nfree,index,nenter,ileave,indx2,iupdat, + updatd,wn,snd,m,ws,wy,sy,theta,col,head,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1006) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif c compute r=-Z'B(xcp-xk)-Z'g (using wa(2m+1)=W'(xcp-x) c from 'cauchy'). call cmprlb(n,m,x,g,ws,wy,sy,wt,z,r,wa,index, + theta,col,head,nfree,cnstnd,info) if (info .ne. 0) goto 444 c call the direct method. call subsm(n,m,nfree,index,l,u,nbd,z,r,ws,wy,theta, + col,head,iword,wa,wn,iprint,info) 444 continue if (info .ne. 0) then c singular triangular system detected; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1005) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 goto 222 endif call timer(cpu2) sbtime = sbtime + cpu2 - cpu1 555 continue cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Line search and optimality tests. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Generate the search direction d:=z-x. do 40 i = 1, n d(i) = z(i) - x(i) 40 continue call timer(cpu1) 666 continue call lnsrlb(n,l,u,nbd,x,f,fold,gd,gdold,g,d,r,t,z,stp,dnorm, + dtd,xstep,stpmx,iter,ifun,iback,nfgv,info,task, + boxed,cnstnd,csave,isave(22),dsave(17)) if (info .ne. 0 .or. iback .ge. 20) then c restore the previous iterate. call dcopy(n,t,1,x,1) call dcopy(n,r,1,g,1) f = fold if (col .eq. 0) then c abnormal termination. if (info .eq. 0) then info = -9 c restore the actual number of f and g evaluations etc. nfgv = nfgv - 1 ifun = ifun - 1 iback = iback - 1 endif task = 'ABNORMAL_TERMINATION_IN_LNSRCH' iter = iter + 1 goto 999 else c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1008) if (info .eq. 0) nfgv = nfgv - 1 info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. task = 'RESTART_FROM_LNSRCH' call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 goto 222 endif else if (task(1:5) .eq. 'FG_LN') then c return to the driver for calculating f and g; reenter at 666. goto 1000 else c calculate and print out the quantities related to the new X. call timer(cpu2) lnscht = lnscht + cpu2 - cpu1 iter = iter + 1 c Compute the infinity norm of the projected (-)gradient. call projgr(n,l,u,nbd,x,g,sbgnrm) c Print iteration information. call prn2lb(n,x,f,g,iprint,itfile,iter,nfgv,nact, + sbgnrm,nint,word,iword,iback,stp,xstep) goto 1000 endif 777 continue c Test for termination. if (sbgnrm .le. pgtol) then c terminate the algorithm. task = 'CONVERGENCE: NORM OF PROJECTED GRADIENT <= PGTOL' goto 999 endif ddum = max(abs(fold), abs(f), one) if ((fold - f) .le. tol*ddum) then c terminate the algorithm. task = 'CONVERGENCE: REL_REDUCTION_OF_F <= FACTR*EPSMCH' if (iback .ge. 10) info = -5 c i.e., to issue a warning if iback>10 in the line search. goto 999 endif c Compute d=newx-oldx, r=newg-oldg, rr=y'y and dr=y's. do 42 i = 1, n r(i) = g(i) - r(i) 42 continue rr = ddot(n,r,1,r,1) if (stp .eq. one) then dr = gd - gdold ddum = -gdold else dr = (gd - gdold)*stp call dscal(n,stp,d,1) ddum = -gdold*stp endif if (dr .le. epsmch*ddum) then c skip the L-BFGS update. nskip = nskip + 1 updatd = .false. if (iprint .ge. 1) write (6,1004) dr, ddum goto 888 endif cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c c Update the L-BFGS matrix. c cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc updatd = .true. iupdat = iupdat + 1 c Update matrices WS and WY and form the middle matrix in B. call matupd(n,m,ws,wy,sy,ss,d,r,itail, + iupdat,col,head,theta,rr,dr,stp,dtd) c Form the upper half of the pds T = theta*SS + L*D^(-1)*L'; c Store T in the upper triangular of the array wt; c Cholesky factorize T to J*J' with c J' stored in the upper triangular of wt. call formt(m,wt,sy,ss,col,theta,info) if (info .ne. 0) then c nonpositive definiteness in Cholesky factorization; c refresh the lbfgs memory and restart the iteration. if(iprint .ge. 1) write (6, 1007) info = 0 col = 0 head = 1 theta = one iupdat = 0 updatd = .false. goto 222 endif c Now the inverse of the middle matrix in B is c [ D^(1/2) O ] [ -D^(1/2) D^(-1/2)*L' ] c [ -L*D^(-1/2) J ] [ 0 J' ] 888 continue c -------------------- the end of the loop ----------------------------- goto 222 999 continue call timer(time2) time = time2 - time1 call prn3lb(n,x,f,task,iprint,info,itfile, + iter,nfgv,nintol,nskip,nact,sbgnrm, + time,nint,word,iback,stp,xstep,k, + cachyt,sbtime,lnscht) 1000 continue c Save local variables. lsave(1) = prjctd lsave(2) = cnstnd lsave(3) = boxed lsave(4) = updatd isave(1) = nintol isave(3) = itfile isave(4) = iback isave(5) = nskip isave(6) = head isave(7) = col isave(8) = itail isave(9) = iter isave(10) = iupdat isave(12) = nint isave(13) = nfgv isave(14) = info isave(15) = ifun isave(16) = iword isave(17) = nfree isave(18) = nact isave(19) = ileave isave(20) = nenter dsave(1) = theta dsave(2) = fold dsave(3) = tol dsave(4) = dnorm dsave(5) = epsmch dsave(6) = cpu1 dsave(7) = cachyt dsave(8) = sbtime dsave(9) = lnscht dsave(10) = time1 dsave(11) = gd dsave(12) = stpmx dsave(13) = sbgnrm dsave(14) = stp dsave(15) = gdold dsave(16) = dtd 1001 format (//,'ITERATION ',i5) 1002 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 1003 format (2(1x,i4),5x,'-',5x,'-',3x,'-',5x,'-',5x,'-',8x,'-',3x, + 1p,2(1x,d10.3)) 1004 format (' ys=',1p,e10.3,' -gs=',1p,e10.3,' BFGS update SKIPPED') 1005 format (/, +' Singular triangular system detected;',/, +' refresh the lbfgs memory and restart the iteration.') 1006 format (/, +' Nonpositive definiteness in Cholesky factorization in formk;',/, +' refresh the lbfgs memory and restart the iteration.') 1007 format (/, +' Nonpositive definiteness in Cholesky factorization in formt;',/, +' refresh the lbfgs memory and restart the iteration.') 1008 format (/, +' Bad direction in the line search;',/, +' refresh the lbfgs memory and restart the iteration.') return end c======================= The end of mainlb ============================= subroutine active(n, l, u, nbd, x, iwhere, iprint, + prjctd, cnstnd, boxed) logical prjctd, cnstnd, boxed integer n, iprint, nbd(n), iwhere(n) double precision x(n), l(n), u(n) c ************ c c Subroutine active c c This subroutine initializes iwhere and projects the initial x to c the feasible set if necessary. c c iwhere is an integer array of dimension n. c On entry iwhere is unspecified. c On exit iwhere(i)=-1 if x(i) has no bounds c 3 if l(i)=u(i) c 0 otherwise. c In cauchy, iwhere is given finer gradations. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer nbdd,i double precision zero parameter (zero=0.0d0) c Initialize nbdd, prjctd, cnstnd and boxed. nbdd = 0 prjctd = .false. cnstnd = .false. boxed = .true. c Project the initial x to the easible set if necessary. do 10 i = 1, n if (nbd(i) .gt. 0) then if (nbd(i) .le. 2 .and. x(i) .le. l(i)) then if (x(i) .lt. l(i)) then prjctd = .true. x(i) = l(i) endif nbdd = nbdd + 1 else if (nbd(i) .ge. 2 .and. x(i) .ge. u(i)) then if (x(i) .gt. u(i)) then prjctd = .true. x(i) = u(i) endif nbdd = nbdd + 1 endif endif 10 continue c Initialize iwhere and assign values to cnstnd and boxed. do 20 i = 1, n if (nbd(i) .ne. 2) boxed = .false. if (nbd(i) .eq. 0) then c this variable is always free iwhere(i) = -1 c otherwise set x(i)=mid(x(i), u(i), l(i)). else cnstnd = .true. if (nbd(i) .eq. 2 .and. u(i) - l(i) .le. zero) then c this variable is always fixed iwhere(i) = 3 else iwhere(i) = 0 endif endif 20 continue if (iprint .ge. 0) then if (prjctd) write (6,*) + 'The initial X is infeasible. Restart with its projection.' if (.not. cnstnd) + write (6,*) 'This problem is unconstrained.' endif if (iprint .gt. 0) write (6,1001) nbdd 1001 format (/,'At X0 ',i9,' variables are exactly at the bounds') return end c======================= The end of active ============================= subroutine bmv(m, sy, wt, col, v, p, info) integer m, col, info double precision sy(m, m), wt(m, m), v(2*col), p(2*col) c ************ c c Subroutine bmv c c This subroutine computes the product of the 2m x 2m middle matrix c in the compact L-BFGS formula of B and a 2m vector v; c it returns the product in p. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c sy is a double precision array of dimension m x m. c On entry sy specifies the matrix S'Y. c On exit sy is unchanged. c c wt is a double precision array of dimension m x m. c On entry wt specifies the upper triangular matrix J' which is c the Cholesky factor of (thetaS'S+LD^(-1)L'). c On exit wt is unchanged. c c col is an integer variable. c On entry col specifies the number of s-vectors (or y-vectors) c stored in the compact L-BFGS formula. c On exit col is unchanged. c c v is a double precision array of dimension 2col. c On entry v specifies vector v. c On exit v is unchanged. c c p is a double precision array of dimension 2col. c On entry p is unspecified. c On exit p is the product Mv. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return when the system c to be solved by dtrsl is singular. c c Subprograms called: c c Linpack ... dtrsl. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,k,i2 double precision sum if (col .eq. 0) return c PART I: solve [ D^(1/2) O ] [ p1 ] = [ v1 ] c [ -L*D^(-1/2) J ] [ p2 ] [ v2 ]. c solve Jp2=v2+LD^(-1)v1. p(col + 1) = v(col + 1) do 20 i = 2, col i2 = col + i sum = 0.0d0 do 10 k = 1, i - 1 sum = sum + sy(i,k)*v(k)/sy(k,k) 10 continue p(i2) = v(i2) + sum 20 continue c Solve the triangular system call dtrsl(wt,m,col,p(col+1),11,info) if (info .ne. 0) return c solve D^(1/2)p1=v1. cc do 30 i = 1, col cc p(i) = v(i)/sqrt(sy(i,i)) cc30 continue c PART II: solve [ -D^(1/2) D^(-1/2)*L' ] [ p1 ] = [ p1 ] c [ 0 J' ] [ p2 ] [ p2 ]. c solve J^Tp2=p2. call dtrsl(wt,m,col,p(col+1),01,info) if (info .ne. 0) return c compute p1=-D^(-1/2)(p1-D^(-1/2)L'p2) c =-D^(-1/2)p1+D^(-1)L'p2. do 40 i = 1, col cc p(i) = -p(i)/sqrt(sy(i,i)) combined with do 30 loop cc into the next line p(i) = -v(i)/sy(i,i) 40 continue do 60 i = 1, col sum = 0.d0 do 50 k = i + 1, col sum = sum + sy(k,i)*p(col+k)/sy(i,i) 50 continue p(i) = p(i) + sum 60 continue return end c======================== The end of bmv =============================== subroutine cauchy(n, x, l, u, nbd, g, iorder, iwhere, t, d, xcp, + m, wy, ws, sy, wt, theta, col, head, p, c, wbp, + v, nint, iprint, sbgnrm, info, epsmch) integer n, m, head, col, nint, iprint, info, + nbd(n), iorder(n), iwhere(n) double precision theta, epsmch, + x(n), l(n), u(n), g(n), t(n), d(n), xcp(n), + wy(n, col), ws(n, col), sy(m, m), + wt(m, m), p(2*m), c(2*m), wbp(2*m), v(2*m) c ************ c c Subroutine cauchy c c For given x, l, u, g (with sbgnrm > 0), and a limited memory c BFGS matrix B defined in terms of matrices WY, WS, WT, and c scalars head, col, and theta, this subroutine computes the c generalized Cauchy point (GCP), defined as the first local c minimizer of the quadratic c c Q(x + s) = g's + 1/2 s'Bs c c along the projected gradient direction P(x-tg,l,u). c The routine returns the GCP in xcp. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c x is a double precision array of dimension n. c On entry x is the starting point for the GCP computation. c On exit x is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is an integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c g is a double precision array of dimension n. c On entry g is the gradient of f(x). g must be a nonzero vector. c On exit g is unchanged. c c iorder is an integer working array of dimension n. c iorder will be used to store the breakpoints in the piecewise c linear path and free variables encountered. On exit, c iorder(1),...,iorder(nleft) are indices of breakpoints c which have not been encountered; c iorder(nleft+1),...,iorder(nbreak) are indices of c encountered breakpoints; and c iorder(nfree),...,iorder(n) are indices of variables which c have no bound constraits along the search direction. c c iwhere is an integer array of dimension n. c On entry iwhere indicates only the permanently fixed (iwhere=3) c or free (iwhere= -1) components of x. c On exit iwhere records the status of the current x variables. c iwhere(i)=-3 if x(i) is free and has bounds, but is not moved c 0 if x(i) is free and has bounds, and is moved c 1 if x(i) is fixed at l(i), and l(i) .ne. u(i) c 2 if x(i) is fixed at u(i), and u(i) .ne. l(i) c 3 if x(i) is always fixed, i.e., u(i)=x(i)=l(i) c -1 if x(i) is always free, i.e., it has no bounds. c c t is a double precision working array of dimension n. c t will be used to store the break points. c c d is a double precision array of dimension n used to store c the Cauchy direction P(x-tg)-x. c c xcp is a double precision array of dimension n used to return the c GCP on exit. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wt are double precision arrays. c On entry they store information that defines the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wt(m,m) stores the c Cholesky factorization of (theta*S'S+LD^(-1)L'). c On exit these arrays are unchanged. c c theta is a double precision variable. c On entry theta is the scaling factor specifying B_0 = theta I. c On exit theta is unchanged. c c col is an integer variable. c On entry col is the actual number of variable metric c corrections stored so far. c On exit col is unchanged. c c head is an integer variable. c On entry head is the location of the first s-vector c (or y-vector) in S (or Y). c On exit col is unchanged. c c p is a double precision working array of dimension 2m. c p will be used to store the vector p = W^(T)d. c c c is a double precision working array of dimension 2m. c c will be used to store the vector c = W^(T)(xcp-x). c c wbp is a double precision working array of dimension 2m. c wbp will be used to store the row of W corresponding c to a breakpoint. c c v is a double precision working array of dimension 2m. c c nint is an integer variable. c On exit nint records the number of quadratic segments explored c in searching for the GCP. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c sbgnrm is a double precision variable. c On entry sbgnrm is the norm of the projected gradient at x. c On exit sbgnrm is unchanged. c c info is an integer variable. c On entry info is 0. c On exit info = 0 for normal return, c = nonzero for abnormal return when the the system c used in routine bmv is singular. c c Subprograms called: c c L-BFGS-B Library ... hpsolb, bmv. c c Linpack ... dscal dcopy, daxpy. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: FORTRAN c Subroutines for Large Scale Bound Constrained Optimization'' c Tech. Report, NAM-11, EECS Department, Northwestern University, c 1994. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical xlower,xupper,bnded integer i,j,col2,nfree,nbreak,pointr, + ibp,nleft,ibkmin,iter double precision f1,f2,dt,dtm,tsum,dibp,zibp,dibp2,bkmin, + tu,tl,wmc,wmp,wmw,ddot,tj,tj0,neggi,sbgnrm, + f2_org double precision one,zero parameter (one=1.0d0,zero=0.0d0) c Check the status of the variables, reset iwhere(i) if necessary; c compute the Cauchy direction d and the breakpoints t; initialize c the derivative f1 and the vector p = W'd (for theta = 1). if (sbgnrm .le. zero) then if (iprint .ge. 0) write (6,*) 'Subgnorm = 0. GCP = X.' call dcopy(n,x,1,xcp,1) return endif bnded = .true. nfree = n + 1 nbreak = 0 ibkmin = 0 bkmin = zero col2 = 2*col f1 = zero if (iprint .ge. 99) write (6,3010) c We set p to zero and build it up as we determine d. do 20 i = 1, col2 p(i) = zero 20 continue c In the following loop we determine for each variable its bound c status and its breakpoint, and update p accordingly. c Smallest breakpoint is identified. do 50 i = 1, n neggi = -g(i) if (iwhere(i) .ne. 3 .and. iwhere(i) .ne. -1) then c if x(i) is not a constant and has bounds, c compute the difference between x(i) and its bounds. if (nbd(i) .le. 2) tl = x(i) - l(i) if (nbd(i) .ge. 2) tu = u(i) - x(i) c If a variable is close enough to a bound c we treat it as at bound. xlower = nbd(i) .le. 2 .and. tl .le. zero xupper = nbd(i) .ge. 2 .and. tu .le. zero c reset iwhere(i). iwhere(i) = 0 if (xlower) then if (neggi .le. zero) iwhere(i) = 1 else if (xupper) then if (neggi .ge. zero) iwhere(i) = 2 else if (abs(neggi) .le. zero) iwhere(i) = -3 endif endif pointr = head if (iwhere(i) .ne. 0 .and. iwhere(i) .ne. -1) then d(i) = zero else d(i) = neggi f1 = f1 - neggi*neggi c calculate p := p - W'e_i* (g_i). do 40 j = 1, col p(j) = p(j) + wy(i,pointr)* neggi p(col + j) = p(col + j) + ws(i,pointr)*neggi pointr = mod(pointr,m) + 1 40 continue if (nbd(i) .le. 2 .and. nbd(i) .ne. 0 + .and. neggi .lt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tl/(-neggi) if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else if (nbd(i) .ge. 2 .and. neggi .gt. zero) then c x(i) + d(i) is bounded; compute t(i). nbreak = nbreak + 1 iorder(nbreak) = i t(nbreak) = tu/neggi if (nbreak .eq. 1 .or. t(nbreak) .lt. bkmin) then bkmin = t(nbreak) ibkmin = nbreak endif else c x(i) + d(i) is not bounded. nfree = nfree - 1 iorder(nfree) = i if (abs(neggi) .gt. zero) bnded = .false. endif endif 50 continue c The indices of the nonzero components of d are now stored c in iorder(1),...,iorder(nbreak) and iorder(nfree),...,iorder(n). c The smallest of the nbreak breakpoints is in t(ibkmin)=bkmin. if (theta .ne. one) then c complete the initialization of p for theta not= one. call dscal(col,theta,p(col+1),1) endif c Initialize GCP xcp = x. call dcopy(n,x,1,xcp,1) if (nbreak .eq. 0 .and. nfree .eq. n + 1) then c is a zero vector, return with the initial xcp as GCP. if (iprint .gt. 100) write (6,1010) (xcp(i), i = 1, n) return endif c Initialize c = W'(xcp - x) = 0. do 60 j = 1, col2 c(j) = zero 60 continue c Initialize derivative f2. f2 = -theta*f1 f2_org = f2 if (col .gt. 0) then call bmv(m,sy,wt,col,p,v,info) if (info .ne. 0) return f2 = f2 - ddot(col2,v,1,p,1) endif dtm = -f1/f2 tsum = zero nint = 1 if (iprint .ge. 99) + write (6,*) 'There are ',nbreak,' breakpoints ' c If there are no breakpoints, locate the GCP and return. if (nbreak .eq. 0) goto 888 nleft = nbreak iter = 1 tj = zero c------------------- the beginning of the loop ------------------------- 777 continue c Find the next smallest breakpoint; c compute dt = t(nleft) - t(nleft + 1). tj0 = tj if (iter .eq. 1) then c Since we already have the smallest breakpoint we need not do c heapsort yet. Often only one breakpoint is used and the c cost of heapsort is avoided. tj = bkmin ibp = iorder(ibkmin) else if (iter .eq. 2) then c Replace the already used smallest breakpoint with the c breakpoint numbered nbreak > nlast, before heapsort call. if (ibkmin .ne. nbreak) then t(ibkmin) = t(nbreak) iorder(ibkmin) = iorder(nbreak) endif c Update heap structure of breakpoints c (if iter=2, initialize heap). endif call hpsolb(nleft,t,iorder,iter-2) tj = t(nleft) ibp = iorder(nleft) endif dt = tj - tj0 if (dt .ne. zero .and. iprint .ge. 100) then write (6,4011) nint,f1,f2 write (6,5010) dt write (6,6010) dtm endif c If a minimizer is within this interval, c locate the GCP and return. if (dtm .lt. dt) goto 888 c Otherwise fix one variable and c reset the corresponding component of d to zero. tsum = tsum + dt nleft = nleft - 1 iter = iter + 1 dibp = d(ibp) d(ibp) = zero if (dibp .gt. zero) then zibp = u(ibp) - x(ibp) xcp(ibp) = u(ibp) iwhere(ibp) = 2 else zibp = l(ibp) - x(ibp) xcp(ibp) = l(ibp) iwhere(ibp) = 1 endif if (iprint .ge. 100) write (6,*) 'Variable ',ibp,' is fixed.' if (nleft .eq. 0 .and. nbreak .eq. n) then c all n variables are fixed, c return with xcp as GCP. dtm = dt goto 999 endif c Update the derivative information. nint = nint + 1 dibp2 = dibp**2 c Update f1 and f2. c temporarily set f1 and f2 for col=0. f1 = f1 + dt*f2 + dibp2 - theta*dibp*zibp f2 = f2 - theta*dibp2 if (col .gt. 0) then c update c = c + dt*p. call daxpy(col2,dt,p,1,c,1) c choose wbp, c the row of W corresponding to the breakpoint encountered. pointr = head do 70 j = 1,col wbp(j) = wy(ibp,pointr) wbp(col + j) = theta*ws(ibp,pointr) pointr = mod(pointr,m) + 1 70 continue c compute (wbp)Mc, (wbp)Mp, and (wbp)M(wbp)'. call bmv(m,sy,wt,col,wbp,v,info) if (info .ne. 0) return wmc = ddot(col2,c,1,v,1) wmp = ddot(col2,p,1,v,1) wmw = ddot(col2,wbp,1,v,1) c update p = p - dibp*wbp. call daxpy(col2,-dibp,wbp,1,p,1) c complete updating f1 and f2 while col > 0. f1 = f1 + dibp*wmc f2 = f2 + 2.0d0*dibp*wmp - dibp2*wmw endif f2 = max(epsmch*f2_org,f2) if (nleft .gt. 0) then dtm = -f1/f2 goto 777 c to repeat the loop for unsearched intervals. else if(bnded) then f1 = zero f2 = zero dtm = zero else dtm = -f1/f2 endif c------------------- the end of the loop ------------------------------- 888 continue if (iprint .ge. 99) then write (6,*) write (6,*) 'GCP found in this segment' write (6,4010) nint,f1,f2 write (6,6010) dtm endif if (dtm .le. zero) dtm = zero tsum = tsum + dtm c Move free variables (i.e., the ones w/o breakpoints) and c the variables whose breakpoints haven't been reached. call daxpy(n,tsum,d,1,xcp,1) 999 continue c Update c = c + dtm*p = W'(x^c - x) c which will be used in computing r = Z'(B(x^c - x) + g). if (col .gt. 0) call daxpy(col2,dtm,p,1,c,1) if (iprint .gt. 100) write (6,1010) (xcp(i),i = 1,n) if (iprint .ge. 99) write (6,2010) 1010 format ('Cauchy X = ',/,(4x,1p,6(1x,d11.4))) 2010 format (/,'---------------- exit CAUCHY----------------------',/) 3010 format (/,'---------------- CAUCHY entered-------------------') 4010 format ('Piece ',i3,' --f1, f2 at start point ',1p,2(1x,d11.4)) 4011 format (/,'Piece ',i3,' --f1, f2 at start point ', + 1p,2(1x,d11.4)) 5010 format ('Distance to the next break point = ',1p,d11.4) 6010 format ('Distance to the stationary point = ',1p,d11.4) return end c====================== The end of cauchy ============================== subroutine cmprlb(n, m, x, g, ws, wy, sy, wt, z, r, wa, index, + theta, col, head, nfree, cnstnd, info) logical cnstnd integer n, m, col, head, nfree, info, index(n) double precision theta, + x(n), g(n), z(n), r(n), wa(4*m), + ws(n, m), wy(n, m), sy(m, m), wt(m, m) c ************ c c Subroutine cmprlb c c This subroutine computes r=-Z'B(xcp-xk)-Z'g by using c wa(2m+1)=W'(xcp-x) from subroutine cauchy. c c Subprograms called: c c L-BFGS-B Library ... bmv. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,pointr double precision a1,a2 if (.not. cnstnd .and. col .gt. 0) then do 26 i = 1, n r(i) = -g(i) 26 continue else do 30 i = 1, nfree k = index(i) r(i) = -theta*(z(k) - x(k)) - g(k) 30 continue call bmv(m,sy,wt,col,wa(2*m+1),wa(1),info) if (info .ne. 0) then info = -8 return endif pointr = head do 34 j = 1, col a1 = wa(j) a2 = theta*wa(col + j) do 32 i = 1, nfree k = index(i) r(i) = r(i) + wy(k,pointr)*a1 + ws(k,pointr)*a2 32 continue pointr = mod(pointr,m) + 1 34 continue endif return end c======================= The end of cmprlb ============================= subroutine errclb(n, m, factr, l, u, nbd, task, info, k) character*60 task integer n, m, info, k, nbd(n) double precision factr, l(n), u(n) c ************ c c Subroutine errclb c c This subroutine checks the validity of the input data. c c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision zero parameter (zero=0.0d0) c Check the input arguments for errors. if (n .le. 0) task = 'ERROR: N .LE. 0' if (m .le. 0) task = 'ERROR: M .LE. 0' if (factr .lt. zero) task = 'ERROR: FACTR .LT. 0' c Check the validity of the arrays nbd(i), u(i), and l(i). do 10 i = 1, n if (nbd(i) .lt. 0 .or. nbd(i) .gt. 3) then c return task = 'ERROR: INVALID NBD' info = -6 k = i endif if (nbd(i) .eq. 2) then if (l(i) .gt. u(i)) then c return task = 'ERROR: NO FEASIBLE SOLUTION' info = -7 k = i endif endif 10 continue return end c======================= The end of errclb ============================= subroutine formk(n, nsub, ind, nenter, ileave, indx2, iupdat, + updatd, wn, wn1, m, ws, wy, sy, theta, col, + head, info) integer n, nsub, m, col, head, nenter, ileave, iupdat, + info, ind(n), indx2(n) double precision theta, wn(2*m, 2*m), wn1(2*m, 2*m), + ws(n, m), wy(n, m), sy(m, m) logical updatd c ************ c c Subroutine formk c c This subroutine forms the LEL^T factorization of the indefinite c c matrix K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c The matrix K can be shown to be equal to the matrix M^[-1]N c occurring in section 5.1 of [1], as well as to the matrix c Mbar^[-1] Nbar in section 5.3. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c nsub is an integer variable c On entry nsub is the number of subspace variables in free set. c On exit nsub is not changed. c c ind is an integer array of dimension nsub. c On entry ind specifies the indices of subspace variables. c On exit ind is unchanged. c c nenter is an integer variable. c On entry nenter is the number of variables entering the c free set. c On exit nenter is unchanged. c c ileave is an integer variable. c On entry indx2(ileave),...,indx2(n) are the variables leaving c the free set. c On exit ileave is unchanged. c c indx2 is an integer array of dimension n. c On entry indx2(1),...,indx2(nenter) are the variables entering c the free set, while indx2(ileave),...,indx2(n) are the c variables leaving the free set. c On exit indx2 is unchanged. c c iupdat is an integer variable. c On entry iupdat is the total number of BFGS updates made so far. c On exit iupdat is unchanged. c c updatd is a logical variable. c On entry 'updatd' is true if the L-BFGS matrix is updatd. c On exit 'updatd' is unchanged. c c wn is a double precision array of dimension 2m x 2m. c On entry wn is unspecified. c On exit the upper triangle of wn stores the LEL^T factorization c of the 2*col x 2*col indefinite matrix c [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c wn1 is a double precision array of dimension 2m x 2m. c On entry wn1 stores the lower triangular part of c [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c in the previous iteration. c On exit wn1 stores the corresponding updated matrices. c The purpose of wn1 is just to store these inner products c so they can be easily updated and inserted into wn. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c ws, wy, sy, and wtyy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c sy(m,m) stores S'Y; c wtyy(m,m) stores the Cholesky factorization c of (theta*S'S+LD^(-1)L') c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return; c = -1 when the 1st Cholesky factorization failed; c = -2 when the 2st Cholesky factorization failed. c c Subprograms called: c c Linpack ... dcopy, dpofa, dtrsl. c c c References: c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c [2] C. Zhu, R.H. Byrd, P. Lu, J. Nocedal, ``L-BFGS-B: a c limited memory FORTRAN code for solving bound constrained c optimization problems'', Tech. Report, NAM-11, EECS Department, c Northwestern University, 1994. c c (Postscript files of these papers are available via anonymous c ftp to ece.nwu.edu in the directory pub/lbfgs/lbfgs_bcm.) c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer m2,ipntr,jpntr,iy,is,jy,js,is1,js1,k1,i,k, + col2,pbegin,pend,dbegin,dend,upcl double precision ddot,temp1,temp2,temp3,temp4 double precision zero parameter (zero=0.0d0) c Form the lower triangular part of c WN1 = [Y' ZZ'Y L_a'+R_z'] c [L_a+R_z S'AA'S ] c where L_a is the strictly lower triangular part of S'AA'Y c R_z is the upper triangular part of S'ZZ'Y. if (updatd) then if (iupdat .gt. m) then c shift old part of WN1. do 10 jy = 1, m - 1 js = m + jy call dcopy(m-jy,wn1(jy+1,jy+1),1,wn1(jy,jy),1) call dcopy(m-jy,wn1(js+1,js+1),1,wn1(js,js),1) call dcopy(m-1,wn1(m+2,jy+1),1,wn1(m+1,jy),1) 10 continue endif c put new rows in blocks (1,1), (2,1) and (2,2). pbegin = 1 pend = nsub dbegin = nsub + 1 dend = n iy = col is = m + col ipntr = head + col - 1 if (ipntr .gt. m) ipntr = ipntr - m jpntr = head do 20 jy = 1, col js = m + jy temp1 = zero temp2 = zero temp3 = zero c compute element jy of row 'col' of Y'ZZ'Y do 15 k = pbegin, pend k1 = ind(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) 15 continue c compute elements jy of row 'col' of L_a and S'AA'S do 16 k = dbegin, dend k1 = ind(k) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 16 continue wn1(iy,jy) = temp1 wn1(is,js) = temp2 wn1(is,jy) = temp3 jpntr = mod(jpntr,m) + 1 20 continue c put new column in block (2,1). jy = col jpntr = head + col - 1 if (jpntr .gt. m) jpntr = jpntr - m ipntr = head do 30 i = 1, col is = m + i temp3 = zero c compute element i of column 'col' of R_z do 25 k = pbegin, pend k1 = ind(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 25 continue ipntr = mod(ipntr,m) + 1 wn1(is,jy) = temp3 30 continue upcl = col - 1 else upcl = col endif c modify the old parts in blocks (1,1) and (2,2) due to changes c in the set of free variables. ipntr = head do 45 iy = 1, upcl is = m + iy jpntr = head do 40 jy = 1, iy js = m + jy temp1 = zero temp2 = zero temp3 = zero temp4 = zero do 35 k = 1, nenter k1 = indx2(k) temp1 = temp1 + wy(k1,ipntr)*wy(k1,jpntr) temp2 = temp2 + ws(k1,ipntr)*ws(k1,jpntr) 35 continue do 36 k = ileave, n k1 = indx2(k) temp3 = temp3 + wy(k1,ipntr)*wy(k1,jpntr) temp4 = temp4 + ws(k1,ipntr)*ws(k1,jpntr) 36 continue wn1(iy,jy) = wn1(iy,jy) + temp1 - temp3 wn1(is,js) = wn1(is,js) - temp2 + temp4 jpntr = mod(jpntr,m) + 1 40 continue ipntr = mod(ipntr,m) + 1 45 continue c modify the old parts in block (2,1). ipntr = head do 60 is = m + 1, m + upcl jpntr = head do 55 jy = 1, upcl temp1 = zero temp3 = zero do 50 k = 1, nenter k1 = indx2(k) temp1 = temp1 + ws(k1,ipntr)*wy(k1,jpntr) 50 continue do 51 k = ileave, n k1 = indx2(k) temp3 = temp3 + ws(k1,ipntr)*wy(k1,jpntr) 51 continue if (is .le. jy + m) then wn1(is,jy) = wn1(is,jy) + temp1 - temp3 else wn1(is,jy) = wn1(is,jy) - temp1 + temp3 endif jpntr = mod(jpntr,m) + 1 55 continue ipntr = mod(ipntr,m) + 1 60 continue c Form the upper triangle of WN = [D+Y' ZZ'Y/theta -L_a'+R_z' ] c [-L_a +R_z S'AA'S*theta] m2 = 2*m do 70 iy = 1, col is = col + iy is1 = m + iy do 65 jy = 1, iy js = col + jy js1 = m + jy wn(jy,iy) = wn1(iy,jy)/theta wn(js,is) = wn1(is1,js1)*theta 65 continue do 66 jy = 1, iy - 1 wn(jy,is) = -wn1(is1,jy) 66 continue do 67 jy = iy, col wn(jy,is) = wn1(is1,jy) 67 continue wn(iy,iy) = wn(iy,iy) + sy(iy,iy) 70 continue c Form the upper triangle of c WN= [ LL' L^-1(-L_a'+R_z')] c [(-L_a +R_z)L'^-1 S'AA'S*theta ] c first Cholesky factor (1,1) block of wn to get LL' c with L' stored in the upper triangle of wn. call dpofa(wn,m2,col,info) if (info .ne. 0) then info = -1 return endif c then form L^-1(-L_a'+R_z') in the (1,2) block. col2 = 2*col do 71 js = col+1 ,col2 call dtrsl(wn,m2,col,wn(1,js),11,info) 71 continue c Form S'AA'S*theta + (L^-1(-L_a'+R_z'))'L^-1(-L_a'+R_z') in the c upper triangle of (2,2) block of wn. do 72 is = col+1, col2 do 74 js = is, col2 wn(is,js) = wn(is,js) + ddot(col,wn(1,is),1,wn(1,js),1) 74 continue 72 continue c Cholesky factorization of (2,2) block of wn. call dpofa(wn(col+1,col+1),m2,col,info) if (info .ne. 0) then info = -2 return endif return end c======================= The end of formk ============================== subroutine formt(m, wt, sy, ss, col, theta, info) integer m, col, info double precision theta, wt(m, m), sy(m, m), ss(m, m) c ************ c c Subroutine formt c c This subroutine forms the upper half of the pos. def. and symm. c T = theta*SS + L*D^(-1)*L', stores T in the upper triangle c of the array wt, and performs the Cholesky factorization of T c to produce J*J', with J' stored in the upper triangle of wt. c c Subprograms called: c c Linpack ... dpofa. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,j,k,k1 double precision ddum double precision zero parameter (zero=0.0d0) c Form the upper half of T = theta*SS + L*D^(-1)*L', c store T in the upper triangle of the array wt. do 52 j = 1, col wt(1,j) = theta*ss(1,j) 52 continue do 55 i = 2, col do 54 j = i, col k1 = min(i,j) - 1 ddum = zero do 53 k = 1, k1 ddum = ddum + sy(i,k)*sy(j,k)/sy(k,k) 53 continue wt(i,j) = ddum + theta*ss(i,j) 54 continue 55 continue c Cholesky factorize T to J*J' with c J' stored in the upper triangle of wt. call dpofa(wt,m,col,info) if (info .ne. 0) then info = -3 endif return end c======================= The end of formt ============================== subroutine freev(n, nfree, index, nenter, ileave, indx2, + iwhere, wrk, updatd, cnstnd, iprint, iter) integer n, nfree, nenter, ileave, iprint, iter, + index(n), indx2(n), iwhere(n) logical wrk, updatd, cnstnd c ************ c c Subroutine freev c c This subroutine counts the entering and leaving variables when c iter > 0, and finds the index set of free and active variables c at the GCP. c c cnstnd is a logical variable indicating whether bounds are present c c index is an integer array of dimension n c for i=1,...,nfree, index(i) are the indices of free variables c for i=nfree+1,...,n, index(i) are the indices of bound variables c On entry after the first iteration, index gives c the free variables at the previous iteration. c On exit it gives the free variables based on the determination c in cauchy using the array iwhere. c c indx2 is an integer array of dimension n c On entry indx2 is unspecified. c On exit with iter>0, indx2 indicates which variables c have changed status since the previous iteration. c For i= 1,...,nenter, indx2(i) have changed from bound to free. c For i= ileave+1,...,n, indx2(i) have changed from free to bound. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer iact,i,k nenter = 0 ileave = n + 1 if (iter .gt. 0 .and. cnstnd) then c count the entering and leaving variables. do 20 i = 1, nfree k = index(i) if (iwhere(k) .gt. 0) then ileave = ileave - 1 indx2(ileave) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' leaves the set of free variables' endif 20 continue do 22 i = 1 + nfree, n k = index(i) if (iwhere(k) .le. 0) then nenter = nenter + 1 indx2(nenter) = k if (iprint .ge. 100) write (6,*) + 'Variable ',k,' enters the set of free variables' endif 22 continue if (iprint .ge. 99) write (6,*) + n+1-ileave,' variables leave; ',nenter,' variables enter' endif wrk = (ileave .lt. n+1) .or. (nenter .gt. 0) .or. updatd c Find the index set of free and active variables at the GCP. nfree = 0 iact = n + 1 do 24 i = 1, n if (iwhere(i) .le. 0) then nfree = nfree + 1 index(nfree) = i else iact = iact - 1 index(iact) = i endif 24 continue if (iprint .ge. 99) write (6,*) + nfree,' variables are free at GCP ',iter + 1 return end c======================= The end of freev ============================== subroutine hpsolb(n, t, iorder, iheap) integer iheap, n, iorder(n) double precision t(n) c ************ c c Subroutine hpsolb c c This subroutine sorts out the least element of t, and puts the c remaining elements of t in a heap. c c n is an integer variable. c On entry n is the dimension of the arrays t and iorder. c On exit n is unchanged. c c t is a double precision array of dimension n. c On entry t stores the elements to be sorted, c On exit t(n) stores the least elements of t, and t(1) to t(n-1) c stores the remaining elements in the form of a heap. c c iorder is an integer array of dimension n. c On entry iorder(i) is the index of t(i). c On exit iorder(i) is still the index of t(i), but iorder may be c permuted in accordance with t. c c iheap is an integer variable specifying the task. c On entry iheap should be set as follows: c iheap .eq. 0 if t(1) to t(n) is not in the form of a heap, c iheap .ne. 0 if otherwise. c On exit iheap is unchanged. c c c References: c Algorithm 232 of CACM (J. W. J. Williams): HEAPSORT. c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c ************ integer i,j,k,indxin,indxou double precision ddum,out if (iheap .eq. 0) then c Rearrange the elements t(1) to t(n) to form a heap. do 20 k = 2, n ddum = t(k) indxin = iorder(k) c Add ddum to the heap. i = k 10 continue if (i.gt.1) then j = i/2 if (ddum .lt. t(j)) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 10 endif endif t(i) = ddum iorder(i) = indxin 20 continue endif c Assign to 'out' the value of t(1), the least member of the heap, c and rearrange the remaining members to form a heap as c elements 1 to n-1 of t. if (n .gt. 1) then i = 1 out = t(1) indxou = iorder(1) ddum = t(n) indxin = iorder(n) c Restore the heap 30 continue j = i+i if (j .le. n-1) then if (t(j+1) .lt. t(j)) j = j+1 if (t(j) .lt. ddum ) then t(i) = t(j) iorder(i) = iorder(j) i = j goto 30 endif endif t(i) = ddum iorder(i) = indxin c Put the least member in t(n). t(n) = out iorder(n) = indxou endif return end c====================== The end of hpsolb ============================== subroutine lnsrlb(n, l, u, nbd, x, f, fold, gd, gdold, g, d, r, t, + z, stp, dnorm, dtd, xstep, stpmx, iter, ifun, + iback, nfgv, info, task, boxed, cnstnd, csave, + isave, dsave) character*60 task, csave logical boxed, cnstnd integer n, iter, ifun, iback, nfgv, info, + nbd(n), isave(2) double precision f, fold, gd, gdold, stp, dnorm, dtd, xstep, + stpmx, x(n), l(n), u(n), g(n), d(n), r(n), t(n), + z(n), dsave(13) c ********** c c Subroutine lnsrlb c c This subroutine calls subroutine dcsrch from the Minpack2 library c to perform the line search. Subroutine dscrch is safeguarded so c that all trial points lie within the feasible region. c c Subprograms called: c c Minpack2 Library ... dcsrch. c c Linpack ... dtrsl, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ********** integer i double precision ddot,a1,a2 double precision one,zero,big parameter (one=1.0d0,zero=0.0d0,big=1.0d+10) double precision ftol,gtol,xtol parameter (ftol=1.0d-3,gtol=0.9d0,xtol=0.1d0) if (task(1:5) .eq. 'FG_LN') goto 556 dtd = ddot(n,d,1,d,1) dnorm = sqrt(dtd) c Determine the maximum step length. stpmx = big if (cnstnd) then if (iter .eq. 0) then stpmx = one else do 43 i = 1, n a1 = d(i) if (nbd(i) .ne. 0) then if (a1 .lt. zero .and. nbd(i) .le. 2) then a2 = l(i) - x(i) if (a2 .ge. zero) then stpmx = zero else if (a1*stpmx .lt. a2) then stpmx = a2/a1 endif else if (a1 .gt. zero .and. nbd(i) .ge. 2) then a2 = u(i) - x(i) if (a2 .le. zero) then stpmx = zero else if (a1*stpmx .gt. a2) then stpmx = a2/a1 endif endif endif 43 continue endif endif if (iter .eq. 0 .and. .not. boxed) then stp = min(one/dnorm, stpmx) else stp = one endif call dcopy(n,x,1,t,1) call dcopy(n,g,1,r,1) fold = f ifun = 0 iback = 0 csave = 'START' 556 continue gd = ddot(n,g,1,d,1) if (ifun .eq. 0) then gdold=gd if (gd .ge. zero) then c the directional derivative >=0. c Line search is impossible. info = -4 return endif endif call dcsrch(f,gd,stp,ftol,gtol,xtol,zero,stpmx,csave,isave,dsave) xstep = stp*dnorm if (csave(1:4) .ne. 'CONV' .and. csave(1:4) .ne. 'WARN') then task = 'FG_LNSRCH' ifun = ifun + 1 nfgv = nfgv + 1 iback = ifun - 1 if (stp .eq. one) then call dcopy(n,z,1,x,1) else do 41 i = 1, n x(i) = stp*d(i) + t(i) 41 continue endif else task = 'NEW_X' endif return end c======================= The end of lnsrlb ============================= subroutine matupd(n, m, ws, wy, sy, ss, d, r, itail, + iupdat, col, head, theta, rr, dr, stp, dtd) integer n, m, itail, iupdat, col, head double precision theta, rr, dr, stp, dtd, d(n), r(n), + ws(n, m), wy(n, m), sy(m, m), ss(m, m) c ************ c c Subroutine matupd c c This subroutine updates matrices WS and WY, and forms the c middle matrix in B. c c Subprograms called: c c Linpack ... dcopy, ddot. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer j,pointr double precision ddot double precision one parameter (one=1.0d0) c Set pointers for matrices WS and WY. if (iupdat .le. m) then col = iupdat itail = mod(head+iupdat-2,m) + 1 else itail = mod(itail,m) + 1 head = mod(head,m) + 1 endif c Update matrices WS and WY. call dcopy(n,d,1,ws(1,itail),1) call dcopy(n,r,1,wy(1,itail),1) c Set theta=yy/ys. theta = rr/dr c Form the middle matrix in B. c update the upper triangle of SS, c and the lower triangle of SY: if (iupdat .gt. m) then c move old information do 50 j = 1, col - 1 call dcopy(j,ss(2,j+1),1,ss(1,j),1) call dcopy(col-j,sy(j+1,j+1),1,sy(j,j),1) 50 continue endif c add new information: the last row of SY c and the last column of SS: pointr = head do 51 j = 1, col - 1 sy(col,j) = ddot(n,d,1,wy(1,pointr),1) ss(j,col) = ddot(n,ws(1,pointr),1,d,1) pointr = mod(pointr,m) + 1 51 continue if (stp .eq. one) then ss(col,col) = dtd else ss(col,col) = stp*stp*dtd endif sy(col,col) = dr return end c======================= The end of matupd ============================= subroutine prn1lb(n, m, l, u, x, iprint, itfile, epsmch) integer n, m, iprint, itfile double precision epsmch, x(n), l(n), u(n) c ************ c c Subroutine prn1lb c c This subroutine prints the input data, initial point, upper and c lower bounds of each variable, machine precision, as well as c the headings of the output. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (iprint .ge. 0) then write (6,7001) epsmch write (6,*) 'N = ',n,' M = ',m if (iprint .ge. 1) then write (itfile,2001) epsmch write (itfile,*)'N = ',n,' M = ',m write (itfile,9001) if (iprint .gt. 100) then write (6,1004) 'L =',(l(i),i = 1,n) write (6,1004) 'X0 =',(x(i),i = 1,n) write (6,1004) 'U =',(u(i),i = 1,n) endif endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format ('RUNNING THE L-BFGS-B CODE',/,/, + 'it = iteration number',/, + 'nf = number of function evaluations',/, + 'nint = number of segments explored during the Cauchy search',/, + 'nact = number of active bounds at the generalized Cauchy point' + ,/, + 'sub = manner in which the subspace minimization terminated:' + ,/,' con = converged, bnd = a bound was reached',/, + 'itls = number of iterations performed in the line search',/, + 'stepl = step length used',/, + 'tstep = norm of the displacement (total step)',/, + 'projg = norm of the projected gradient',/, + 'f = function value',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 7001 format ('RUNNING THE L-BFGS-B CODE',/,/, + ' * * *',/,/, + 'Machine precision =',1p,d10.3) 9001 format (/,3x,'it',3x,'nf',2x,'nint',2x,'nact',2x,'sub',2x,'itls', + 2x,'stepl',4x,'tstep',5x,'projg',8x,'f') return end c======================= The end of prn1lb ============================= subroutine prn2lb(n, x, f, g, iprint, itfile, iter, nfgv, nact, + sbgnrm, nint, word, iword, iback, stp, xstep) character*3 word integer n, iprint, itfile, iter, nfgv, nact, nint, + iword, iback double precision f, sbgnrm, stp, xstep, x(n), g(n) c ************ c c Subroutine prn2lb c c This subroutine prints out new information after a successful c line search. c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i,imod c 'word' records the status of subspace solutions. if (iword .eq. 0) then c the subspace minimization converged. word = 'con' else if (iword .eq. 1) then c the subspace minimization stopped at a bound. word = 'bnd' else if (iword .eq. 5) then c the truncated Newton step has been used. word = 'TNT' else word = '---' endif if (iprint .ge. 99) then write (6,*) 'LINE SEARCH',iback,' times; norm of step = ',xstep write (6,2001) iter,f,sbgnrm if (iprint .gt. 100) then write (6,1004) 'X =',(x(i), i = 1, n) write (6,1004) 'G =',(g(i), i = 1, n) endif else if (iprint .gt. 0) then imod = mod(iter,iprint) if (imod .eq. 0) write (6,2001) iter,f,sbgnrm endif if (iprint .ge. 1) write (itfile,3001) + iter,nfgv,nint,nact,word,iback,stp,xstep,sbgnrm,f 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 2001 format + (/,'At iterate',i5,4x,'f= ',1p,d12.5,4x,'|proj g|= ',1p,d12.5) 3001 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),1p,2(1x,d10.3)) return end c======================= The end of prn2lb ============================= subroutine prn3lb(n, x, f, task, iprint, info, itfile, + iter, nfgv, nintol, nskip, nact, sbgnrm, + time, nint, word, iback, stp, xstep, k, + cachyt, sbtime, lnscht) character*60 task character*3 word integer n, iprint, info, itfile, iter, nfgv, nintol, + nskip, nact, nint, iback, k double precision f, sbgnrm, time, stp, xstep, cachyt, sbtime, + lnscht, x(n) c ************ c c Subroutine prn3lb c c This subroutine prints out information when either a built-in c convergence test is satisfied or when an error message is c generated. c c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i if (task(1:5) .eq. 'ERROR') goto 999 if (iprint .ge. 0) then write (6,3003) write (6,3004) write(6,3005) n,iter,nfgv,nintol,nskip,nact,sbgnrm,f if (iprint .ge. 100) then write (6,1004) 'X =',(x(i),i = 1,n) endif if (iprint .ge. 1) write (6,*) ' F =',f endif 999 continue if (iprint .ge. 0) then write (6,3009) task if (info .ne. 0) then if (info .eq. -1) write (6,9011) if (info .eq. -2) write (6,9012) if (info .eq. -3) write (6,9013) if (info .eq. -4) write (6,9014) if (info .eq. -5) write (6,9015) if (info .eq. -6) write (6,*)' Input nbd(',k,') is invalid.' if (info .eq. -7) + write (6,*)' l(',k,') > u(',k,'). No feasible solution.' if (info .eq. -8) write (6,9018) if (info .eq. -9) write (6,9019) endif if (iprint .ge. 1) write (6,3007) cachyt,sbtime,lnscht write (6,3008) time if (iprint .ge. 1) then if (info .eq. -4 .or. info .eq. -9) then write (itfile,3002) + iter,nfgv,nint,nact,word,iback,stp,xstep endif write (itfile,3009) task if (info .ne. 0) then if (info .eq. -1) write (itfile,9011) if (info .eq. -2) write (itfile,9012) if (info .eq. -3) write (itfile,9013) if (info .eq. -4) write (itfile,9014) if (info .eq. -5) write (itfile,9015) if (info .eq. -8) write (itfile,9018) if (info .eq. -9) write (itfile,9019) endif write (itfile,3008) time endif endif 1004 format (/,a4, 1p, 6(1x,d11.4),/,(4x,1p,6(1x,d11.4))) 3002 format(2(1x,i4),2(1x,i5),2x,a3,1x,i4,1p,2(2x,d7.1),6x,'-',10x,'-') 3003 format (/, + ' * * *',/,/, + 'Tit = total number of iterations',/, + 'Tnf = total number of function evaluations',/, + 'Tnint = total number of segments explored during', + ' Cauchy searches',/, + 'Skip = number of BFGS updates skipped',/, + 'Nact = number of active bounds at final generalized', + ' Cauchy point',/, + 'Projg = norm of the final projected gradient',/, + 'F = final function value',/,/, + ' * * *') 3004 format (/,3x,'N',3x,'Tit',2x,'Tnf',2x,'Tnint',2x, + 'Skip',2x,'Nact',5x,'Projg',8x,'F') 3005 format (i5,2(1x,i4),(1x,i6),(2x,i4),(1x,i5),1p,2(2x,d10.3)) 3007 format (/,' Cauchy time',1p,e10.3,' seconds.',/ + ' Subspace minimization time',1p,e10.3,' seconds.',/ + ' Line search time',1p,e10.3,' seconds.') 3008 format (/,' Total User time',1p,e10.3,' seconds.',/) 3009 format (/,a60) 9011 format (/, +' Matrix in 1st Cholesky factorization in formk is not Pos. Def.') 9012 format (/, +' Matrix in 2st Cholesky factorization in formk is not Pos. Def.') 9013 format (/, +' Matrix in the Cholesky factorization in formt is not Pos. Def.') 9014 format (/, +' Derivative >= 0, backtracking line search impossible.',/, +' Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding errors dominate computation.') 9015 format (/, +' Warning: more than 10 function and gradient',/, +' evaluations in the last line search. Termination',/, +' may possibly be caused by a bad search direction.') 9018 format (/,' The triangular system is singular.') 9019 format (/, +' Line search cannot locate an adequate point after 20 function',/ +,' and gradient evaluations. Previous x, f and g restored.',/, +' Possible causes: 1 error in function or gradient evaluation;',/, +' 2 rounding error dominate computation.') return end c======================= The end of prn3lb ============================= subroutine projgr(n, l, u, nbd, x, g, sbgnrm) integer n, nbd(n) double precision sbgnrm, x(n), l(n), u(n), g(n) c ************ c c Subroutine projgr c c This subroutine computes the infinity norm of the projected c gradient. c c c * * * c c NEOS, November 1994. (Latest revision April 1997.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ integer i double precision gi double precision zero parameter (zero=0.0d0) sbgnrm = zero do 15 i = 1, n gi = g(i) if (nbd(i) .ne. 0) then if (gi .lt. zero) then if (nbd(i) .ge. 2) gi = max((x(i)-u(i)),gi) else if (nbd(i) .le. 2) gi = min((x(i)-l(i)),gi) endif endif sbgnrm = max(sbgnrm,abs(gi)) 15 continue return end c======================= The end of projgr ============================= subroutine subsm(n, m, nsub, ind, l, u, nbd, x, d, ws, wy, theta, + col, head, iword, wv, wn, iprint, info) integer n, m, nsub, col, head, iword, iprint, info, + ind(nsub), nbd(n) double precision theta, + l(n), u(n), x(n), d(n), + ws(n, m), wy(n, m), + wv(2*m), wn(2*m, 2*m) c ************ c c Subroutine subsm c c Given xcp, l, u, r, an index set that specifies c the active set at xcp, and an l-BFGS matrix B c (in terms of WY, WS, SY, WT, head, col, and theta), c this subroutine computes an approximate solution c of the subspace problem c c (P) min Q(x) = r'(x-xcp) + 1/2 (x-xcp)' B (x-xcp) c c subject to l<=x<=u c x_i=xcp_i for all i in A(xcp) c c along the subspace unconstrained Newton direction c c d = -(Z'BZ)^(-1) r. c c The formula for the Newton direction, given the L-BFGS matrix c and the Sherman-Morrison formula, is c c d = (1/theta)r + (1/theta*2) Z'WK^(-1)W'Z r. c c where c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c c Note that this procedure for computing d differs c from that described in [1]. One can show that the matrix K is c equal to the matrix M^[-1]N in that paper. c c n is an integer variable. c On entry n is the dimension of the problem. c On exit n is unchanged. c c m is an integer variable. c On entry m is the maximum number of variable metric corrections c used to define the limited memory matrix. c On exit m is unchanged. c c nsub is an integer variable. c On entry nsub is the number of free variables. c On exit nsub is unchanged. c c ind is an integer array of dimension nsub. c On entry ind specifies the coordinate indices of free variables. c On exit ind is unchanged. c c l is a double precision array of dimension n. c On entry l is the lower bound of x. c On exit l is unchanged. c c u is a double precision array of dimension n. c On entry u is the upper bound of x. c On exit u is unchanged. c c nbd is a integer array of dimension n. c On entry nbd represents the type of bounds imposed on the c variables, and must be specified as follows: c nbd(i)=0 if x(i) is unbounded, c 1 if x(i) has only a lower bound, c 2 if x(i) has both lower and upper bounds, and c 3 if x(i) has only an upper bound. c On exit nbd is unchanged. c c x is a double precision array of dimension n. c On entry x specifies the Cauchy point xcp. c On exit x(i) is the minimizer of Q over the subspace of c free variables. c c d is a double precision array of dimension n. c On entry d is the reduced gradient of Q at xcp. c On exit d is the Newton direction of Q. c c ws and wy are double precision arrays; c theta is a double precision variable; c col is an integer variable; c head is an integer variable. c On entry they store the information defining the c limited memory BFGS matrix: c ws(n,m) stores S, a set of s-vectors; c wy(n,m) stores Y, a set of y-vectors; c theta is the scaling factor specifying B_0 = theta I; c col is the number of variable metric corrections stored; c head is the location of the 1st s- (or y-) vector in S (or Y). c On exit they are unchanged. c c iword is an integer variable. c On entry iword is unspecified. c On exit iword specifies the status of the subspace solution. c iword = 0 if the solution is in the box, c 1 if some bound is encountered. c c wv is a double precision working array of dimension 2m. c c wn is a double precision array of dimension 2m x 2m. c On entry the upper triangle of wn stores the LEL^T factorization c of the indefinite matrix c c K = [-D -Y'ZZ'Y/theta L_a'-R_z' ] c [L_a -R_z theta*S'AA'S ] c where E = [-I 0] c [ 0 I] c On exit wn is unchanged. c c iprint is an INTEGER variable that must be set by the user. c It controls the frequency and type of output generated: c iprint<0 no output is generated; c iprint=0 print only one line at the last iteration; c 0100 print details of every iteration including x and g; c When iprint > 0, the file iterate.dat will be created to c summarize the iteration. c c info is an integer variable. c On entry info is unspecified. c On exit info = 0 for normal return, c = nonzero for abnormal return c when the matrix K is ill-conditioned. c c Subprograms called: c c Linpack dtrsl. c c c References: c c [1] R. H. Byrd, P. Lu, J. Nocedal and C. Zhu, ``A limited c memory algorithm for bound constrained optimization'', c SIAM J. Scientific Computing 16 (1995), no. 5, pp. 1190--1208. c c c c * * * c c NEOS, November 1994. (Latest revision June 1996.) c Optimization Technology Center. c Argonne National Laboratory and Northwestern University. c Written by c Ciyou Zhu c in collaboration with R.H. Byrd, P. Lu-Chen and J. Nocedal. c c c ************ logical temp1_updated integer pointr,m2,col2,ibd,jy,js,i,j,k double precision alpha,dk,temp1,temp2 double precision one,zero parameter (one=1.0d0,zero=0.0d0) if (nsub .le. 0) return if (iprint .ge. 99) write (6,1001) c Compute wv = W'Zd. pointr = head do 20 i = 1, col temp1 = zero temp2 = zero do 10 j = 1, nsub k = ind(j) temp1 = temp1 + wy(k,pointr)*d(j) temp2 = temp2 + ws(k,pointr)*d(j) 10 continue wv(i) = temp1 wv(col + i) = theta*temp2 pointr = mod(pointr,m) + 1 20 continue c Compute wv:=K^(-1)wv. m2 = 2*m col2 = 2*col call dtrsl(wn,m2,col2,wv,11,info) if (info .ne. 0) return do 25 i = 1, col wv(i) = -wv(i) 25 continue call dtrsl(wn,m2,col2,wv,01,info) if (info .ne. 0) return c Compute d = (1/theta)d + (1/theta**2)Z'W wv. pointr = head do 40 jy = 1, col js = col + jy do 30 i = 1, nsub k = ind(i) d(i) = d(i) + wy(k,pointr)*wv(jy)/theta + + ws(k,pointr)*wv(js) 30 continue pointr = mod(pointr,m) + 1 40 continue do 50 i = 1, nsub d(i) = d(i)/theta 50 continue c Backtrack to the feasible region. alpha = one temp1 = alpha do 60 i = 1, nsub k = ind(i) dk = d(i) if (nbd(k) .ne. 0) then temp1_updated = .false. if (dk .lt. zero .and. nbd(k) .le. 2) then temp2 = l(k) - x(k) if (temp2 .ge. zero) then temp1 = zero temp1_updated = .true. else if (dk*alpha .lt. temp2) then temp1 = temp2/dk temp1_updated = .true. endif else if (dk .gt. zero .and. nbd(k) .ge. 2) then temp2 = u(k) - x(k) if (temp2 .le. zero) then temp1 = zero temp1_updated = .true. else if (dk*alpha .gt. temp2) then temp1 = temp2/dk temp1_updated = .true. endif endif cc logical variable temp1_updated added to eliminate unexpected cc trigger of the if statement due to possible difference between cc hardware precision and double precision. if (temp1_updated .and. temp1 .lt. alpha) then alpha = temp1 ibd = i endif endif 60 continue if (alpha .lt. one) then dk = d(ibd) k = ind(ibd) if (dk .gt. zero) then x(k) = u(k) d(ibd) = zero else if (dk .lt. zero) then x(k) = l(k) d(ibd) = zero endif endif do 70 i = 1, nsub k = ind(i) x(k) = x(k) + alpha*d(i) 70 continue if (iprint .ge. 99) then if (alpha .lt. one) then write (6,1002) alpha else write (6,*) 'SM solution inside the box' end if if (iprint .gt.100) write (6,1003) (x(i),i=1,n) endif if (alpha .lt. one) then iword = 1 else iword = 0 endif if (iprint .ge. 99) write (6,1004) 1001 format (/,'----------------SUBSM entered-----------------',/) 1002 format ( 'ALPHA = ',f7.5,' backtrack to the BOX') 1003 format ('Subspace solution X = ',/,(4x,1p,6(1x,d11.4))) 1004 format (/,'----------------exit SUBSM --------------------',/) return end c====================== The end of subsm =============================== subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, + task,isave,dsave) character*(*) task integer isave(2) double precision f,g,stp,ftol,gtol,xtol,stpmin,stpmax double precision dsave(13) c ********** c c Subroutine dcsrch c c This subroutine finds a step that satisfies a sufficient c decrease condition and a curvature condition. c c Each call of the subroutine updates an interval with c endpoints stx and sty. The interval is initially chosen c so that it contains a minimizer of the modified function c c psi(stp) = f(stp) - f(0) - ftol*stp*f'(0). c c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c interval is chosen so that it contains a minimizer of f. c c The algorithm is designed to find a step that satisfies c the sufficient decrease condition c c f(stp) <= f(0) + ftol*stp*f'(0), c c and the curvature condition c c abs(f'(stp)) <= gtol*abs(f'(0)). c c If ftol is less than gtol and if, for example, the function c is bounded below, then there is always a step which satisfies c both conditions. c c If no step can be found that satisfies both conditions, then c the algorithm stops with a warning. In this case stp only c satisfies the sufficient decrease condition. c c A typical invocation of dcsrch has the following outline: c c task = 'START' c 10 continue c call dcsrch( ... ) c if (task .eq. 'FG') then c Evaluate the function and the gradient at stp c goto 10 c end if c c NOTE: The user must no alter work arrays between calls. c c The subroutine statement is c c subroutine dcsrch(f,g,stp,ftol,gtol,xtol,stpmin,stpmax, c task,isave,dsave) c where c c f is a double precision variable. c On initial entry f is the value of the function at 0. c On subsequent entries f is the value of the c function at stp. c On exit f is the value of the function at stp. c c g is a double precision variable. c On initial entry g is the derivative of the function at 0. c On subsequent entries g is the derivative of the c function at stp. c On exit g is the derivative of the function at stp. c c stp is a double precision variable. c On entry stp is the current estimate of a satisfactory c step. On initial entry, a positive initial estimate c must be provided. c On exit stp is the current estimate of a satisfactory step c if task = 'FG'. If task = 'CONV' then stp satisfies c the sufficient decrease and curvature condition. c c ftol is a double precision variable. c On entry ftol specifies a nonnegative tolerance for the c sufficient decrease condition. c On exit ftol is unchanged. c c gtol is a double precision variable. c On entry gtol specifies a nonnegative tolerance for the c curvature condition. c On exit gtol is unchanged. c c xtol is a double precision variable. c On entry xtol specifies a nonnegative relative tolerance c for an acceptable step. The subroutine exits with a c warning if the relative difference between sty and stx c is less than xtol. c On exit xtol is unchanged. c c stpmin is a double precision variable. c On entry stpmin is a nonnegative lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is a nonnegative upper bound for the step. c On exit stpmax is unchanged. c c task is a character variable of length at least 60. c On initial entry task must be set to 'START'. c On exit task indicates the required action: c c If task(1:2) = 'FG' then evaluate the function and c derivative at stp and call dcsrch again. c c If task(1:4) = 'CONV' then the search is successful. c c If task(1:4) = 'WARN' then the subroutine is not able c to satisfy the convergence conditions. The exit value of c stp contains the best point found during the search. c c If task(1:5) = 'ERROR' then there is an error in the c input arguments. c c On exit with convergence, a warning or an error, the c variable task contains additional information. c c isave is an integer work array of dimension 2. c c dsave is a double precision work array of dimension 13. c c Subprograms called c c MINPACK-2 ... dcstep c c MINPACK-1 Project. June 1983. c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick, Richard G. Carter, and Jorge J. More'. c c ********** double precision zero,p5,p66 parameter(zero=0.0d0,p5=0.5d0,p66=0.66d0) double precision xtrapl,xtrapu parameter(xtrapl=1.1d0,xtrapu=4.0d0) logical brackt integer stage double precision finit,ftest,fm,fx,fxm,fy,fym,ginit,gtest, + gm,gx,gxm,gy,gym,stx,sty,stmin,stmax,width,width1 c Initialization block. if (task(1:5) .eq. 'START') then c Check the input arguments for errors. if (stp .lt. stpmin) task = 'ERROR: STP .LT. STPMIN' if (stp .gt. stpmax) task = 'ERROR: STP .GT. STPMAX' if (g .ge. zero) task = 'ERROR: INITIAL G .GE. ZERO' if (ftol .lt. zero) task = 'ERROR: FTOL .LT. ZERO' if (gtol .lt. zero) task = 'ERROR: GTOL .LT. ZERO' if (xtol .lt. zero) task = 'ERROR: XTOL .LT. ZERO' if (stpmin .lt. zero) task = 'ERROR: STPMIN .LT. ZERO' if (stpmax .lt. stpmin) task = 'ERROR: STPMAX .LT. STPMIN' c Exit if there are errors on input. if (task(1:5) .eq. 'ERROR') return c Initialize local variables. brackt = .false. stage = 1 finit = f ginit = g gtest = ftol*ginit width = stpmax - stpmin width1 = width/p5 c The variables stx, fx, gx contain the values of the step, c function, and derivative at the best step. c The variables sty, fy, gy contain the value of the step, c function, and derivative at sty. c The variables stp, f, g contain the values of the step, c function, and derivative at stp. stx = zero fx = finit gx = ginit sty = zero fy = finit gy = ginit stmin = zero stmax = stp + xtrapu*stp task = 'FG' goto 1000 else c Restore local variables. if (isave(1) .eq. 1) then brackt = .true. else brackt = .false. endif stage = isave(2) ginit = dsave(1) gtest = dsave(2) gx = dsave(3) gy = dsave(4) finit = dsave(5) fx = dsave(6) fy = dsave(7) stx = dsave(8) sty = dsave(9) stmin = dsave(10) stmax = dsave(11) width = dsave(12) width1 = dsave(13) endif c If psi(stp) <= 0 and f'(stp) >= 0 for some step, then the c algorithm enters the second stage. ftest = finit + stp*gtest if (stage .eq. 1 .and. f .le. ftest .and. g .ge. zero) + stage = 2 c Test for warnings. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax)) + task = 'WARNING: ROUNDING ERRORS PREVENT PROGRESS' if (brackt .and. stmax - stmin .le. xtol*stmax) + task = 'WARNING: XTOL TEST SATISFIED' if (stp .eq. stpmax .and. f .le. ftest .and. g .le. gtest) + task = 'WARNING: STP = STPMAX' if (stp .eq. stpmin .and. (f .gt. ftest .or. g .ge. gtest)) + task = 'WARNING: STP = STPMIN' cc New warning statement added to eliminate the unexpected case cc of stp=stx due to possible difference between hardware precision cc and double precision. if (stp .eq. stx) + task = 'WARNING: STP = STX' c Test for convergence. if (f .le. ftest .and. abs(g) .le. gtol*(-ginit)) + task = 'CONVERGENCE' c Test for termination. if (task(1:4) .eq. 'WARN' .or. task(1:4) .eq. 'CONV') goto 1000 c A modified function is used to predict the step during the c first stage if a lower function value has been obtained but c the decrease is not sufficient. if (stage .eq. 1 .and. f .le. fx .and. f .gt. ftest) then c Define the modified function and derivative values. fm = f - stp*gtest fxm = fx - stx*gtest fym = fy - sty*gtest gm = g - gtest gxm = gx - gtest gym = gy - gtest c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fxm,gxm,sty,fym,gym,stp,fm,gm, + brackt,stmin,stmax) c Reset the function and derivative values for f. fx = fxm + stx*gtest fy = fym + sty*gtest gx = gxm + gtest gy = gym + gtest else c Call dcstep to update stx, sty, and to compute the new step. call dcstep(stx,fx,gx,sty,fy,gy,stp,f,g, + brackt,stmin,stmax) endif c Decide if a bisection step is needed. if (brackt) then if (abs(sty-stx) .ge. p66*width1) stp = stx + p5*(sty - stx) width1 = width width = abs(sty-stx) endif c Set the minimum and maximum steps allowed for stp. if (brackt) then stmin = min(stx,sty) stmax = max(stx,sty) else stmin = stp + xtrapl*(stp - stx) stmax = stp + xtrapu*(stp - stx) endif c Force the step to be within the bounds stpmax and stpmin. stp = max(stp,stpmin) stp = min(stp,stpmax) c If further progress is not possible, let stp be the best c point obtained during the search. if (brackt .and. (stp .le. stmin .or. stp .ge. stmax) + .or. (brackt .and. stmax-stmin .le. xtol*stmax)) stp = stx c Obtain another function and derivative. task = 'FG' 1000 continue c Save local variables. if (brackt) then isave(1) = 1 else isave(1) = 0 endif isave(2) = stage dsave(1) = ginit dsave(2) = gtest dsave(3) = gx dsave(4) = gy dsave(5) = finit dsave(6) = fx dsave(7) = fy dsave(8) = stx dsave(9) = sty dsave(10) = stmin dsave(11) = stmax dsave(12) = width dsave(13) = width1 end c====================== The end of dcsrch ============================== subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, + stpmin,stpmax) logical brackt double precision stx,fx,dx,sty,fy,dy,stp,fp,dp,stpmin,stpmax c ********** c c Subroutine dcstep c c This subroutine computes a safeguarded step for a search c procedure and updates an interval that contains a step that c satisfies a sufficient decrease and a curvature condition. c c The parameter stx contains the step with the least function c value. If brackt is set to .true. then a minimizer has c been bracketed in an interval with endpoints stx and sty. c The parameter stp contains the current step. c The subroutine assumes that if brackt is set to .true. then c c min(stx,sty) < stp < max(stx,sty), c c and that the derivative at stx is negative in the direction c of the step. c c The subroutine statement is c c subroutine dcstep(stx,fx,dx,sty,fy,dy,stp,fp,dp,brackt, c stpmin,stpmax) c c where c c stx is a double precision variable. c On entry stx is the best step obtained so far and is an c endpoint of the interval that contains the minimizer. c On exit stx is the updated best step. c c fx is a double precision variable. c On entry fx is the function at stx. c On exit fx is the function at stx. c c dx is a double precision variable. c On entry dx is the derivative of the function at c stx. The derivative must be negative in the direction of c the step, that is, dx and stp - stx must have opposite c signs. c On exit dx is the derivative of the function at stx. c c sty is a double precision variable. c On entry sty is the second endpoint of the interval that c contains the minimizer. c On exit sty is the updated endpoint of the interval that c contains the minimizer. c c fy is a double precision variable. c On entry fy is the function at sty. c On exit fy is the function at sty. c c dy is a double precision variable. c On entry dy is the derivative of the function at sty. c On exit dy is the derivative of the function at the exit sty. c c stp is a double precision variable. c On entry stp is the current step. If brackt is set to .true. c then on input stp must be between stx and sty. c On exit stp is a new trial step. c c fp is a double precision variable. c On entry fp is the function at stp c On exit fp is unchanged. c c dp is a double precision variable. c On entry dp is the the derivative of the function at stp. c On exit dp is unchanged. c c brackt is an logical variable. c On entry brackt specifies if a minimizer has been bracketed. c Initially brackt must be set to .false. c On exit brackt specifies if a minimizer has been bracketed. c When a minimizer is bracketed brackt is set to .true. c c stpmin is a double precision variable. c On entry stpmin is a lower bound for the step. c On exit stpmin is unchanged. c c stpmax is a double precision variable. c On entry stpmax is an upper bound for the step. c On exit stpmax is unchanged. c c MINPACK-1 Project. June 1983 c Argonne National Laboratory. c Jorge J. More' and David J. Thuente. c c MINPACK-2 Project. October 1993. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick and Jorge J. More'. c c ********** double precision zero,p66,two,three parameter(zero=0.0d0,p66=0.66d0,two=2.0d0,three=3.0d0) double precision gamma,p,q,r,s,sgnd,stpc,stpf,stpq,theta sgnd = dp*(dx/abs(dx)) c First case: A higher function value. The minimum is bracketed. c If the cubic step is closer to stx than the quadratic step, the c cubic step is taken, otherwise the average of the cubic and c quadratic steps is taken. if (fp .gt. fx) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .lt. stx) gamma = -gamma p = (gamma - dx) + theta q = ((gamma - dx) + gamma) + dp r = p/q stpc = stx + r*(stp - stx) stpq = stx + ((dx/((fx - fp)/(stp - stx) + dx))/two)* + (stp - stx) if (abs(stpc-stx) .lt. abs(stpq-stx)) then stpf = stpc else stpf = stpc + (stpq - stpc)/two endif brackt = .true. c Second case: A lower function value and derivatives of opposite c sign. The minimum is bracketed. If the cubic step is farther from c stp than the secant step, the cubic step is taken, otherwise the c secant step is taken. else if (sgnd .lt. zero) then theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dx/s)*(dp/s)) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dx r = p/q stpc = stp + r*(stx - stp) stpq = stp + (dp/(dp - dx))*(stx - stp) if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif brackt = .true. c Third case: A lower function value, derivatives of the same sign, c and the magnitude of the derivative decreases. else if (abs(dp) .lt. abs(dx)) then c The cubic step is computed only if the cubic tends to infinity c in the direction of the step or if the minimum of the cubic c is beyond stp. Otherwise the cubic step is defined to be the c secant step. theta = three*(fx - fp)/(stp - stx) + dx + dp s = max(abs(theta),abs(dx),abs(dp)) c The case gamma = 0 only arises if the cubic does not tend c to infinity in the direction of the step. gamma = s*sqrt(max(zero,(theta/s)**2-(dx/s)*(dp/s))) if (stp .gt. stx) gamma = -gamma p = (gamma - dp) + theta q = (gamma + (dx - dp)) + gamma r = p/q if (r .lt. zero .and. gamma .ne. zero) then stpc = stp + r*(stx - stp) else if (stp .gt. stx) then stpc = stpmax else stpc = stpmin endif stpq = stp + (dp/(dp - dx))*(stx - stp) if (brackt) then c A minimizer has been bracketed. If the cubic step is c closer to stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .lt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif if (stp .gt. stx) then stpf = min(stp+p66*(sty-stp),stpf) else stpf = max(stp+p66*(sty-stp),stpf) endif else c A minimizer has not been bracketed. If the cubic step is c farther from stp than the secant step, the cubic step is c taken, otherwise the secant step is taken. if (abs(stpc-stp) .gt. abs(stpq-stp)) then stpf = stpc else stpf = stpq endif stpf = min(stpmax,stpf) stpf = max(stpmin,stpf) endif c Fourth case: A lower function value, derivatives of the c same sign, and the magnitude of the derivative does not c decrease. If the minimum is not bracketed, the step is either c stpmin or stpmax, otherwise the cubic step is taken. else if (brackt) then theta = three*(fp - fy)/(sty - stp) + dy + dp s = max(abs(theta),abs(dy),abs(dp)) gamma = s*sqrt((theta/s)**2 - (dy/s)*(dp/s)) if (stp .gt. sty) gamma = -gamma p = (gamma - dp) + theta q = ((gamma - dp) + gamma) + dy r = p/q stpc = stp + r*(sty - stp) stpf = stpc else if (stp .gt. stx) then stpf = stpmax else stpf = stpmin endif endif c Update the interval which contains a minimizer. if (fp .gt. fx) then sty = stp fy = fp dy = dp else if (sgnd .lt. zero) then sty = stx fy = fx dy = dx endif stx = stp fx = fp dx = dp endif c Compute the new step. stp = stpf end c====================== The end of dcstep ============================== subroutine timer(ttime) double precision ttime c ********* c c Subroutine timer c c This subroutine is used to determine user time. In a typical c application, the user time for a code segment requires calls c to subroutine timer to determine the initial and final time. c c The subroutine statement is c c subroutine timer(ttime) c c where c c ttime is an output variable which specifies the user time. c c Argonne National Laboratory and University of Minnesota. c MINPACK-2 Project. c c Modified October 1990 by Brett M. Averick. c c ********** real temp real tarray(2) real etime c The first element of the array tarray specifies user time temp = etime(tarray) ttime = dble(tarray(1)) return end c====================== The end of timer =============================== double precision function dnrm2(n,x,incx) integer n,incx double precision x(n) c ********** c c Function dnrm2 c c Given a vector x of length n, this function calculates the c Euclidean norm of x with stride incx. c c The function statement is c c double precision function dnrm2(n,x,incx) c c where c c n is a positive integer input variable. c c x is an input array of length n. c c incx is a positive integer variable that specifies the c stride of the vector. c c Subprograms called c c FORTRAN-supplied ... abs, max, sqrt c c MINPACK-2 Project. February 1991. c Argonne National Laboratory. c Brett M. Averick. c c ********** integer i double precision scale dnrm2 = 0.0d0 scale = 0.0d0 do 10 i = 1, n, incx scale = max(scale, abs(x(i))) 10 continue if (scale .eq. 0.0d0) return do 20 i = 1, n, incx dnrm2 = dnrm2 + (x(i)/scale)**2 20 continue dnrm2 = scale*sqrt(dnrm2) return end c====================== The end of dnrm2 =============================== double precision function dpmeps() c ********** c c Subroutine dpeps c c This subroutine computes the machine precision parameter c dpmeps as the smallest floating point number such that c 1 + dpmeps differs from 1. c c This subroutine is based on the subroutine machar described in c c W. J. Cody, c MACHAR: A subroutine to dynamically determine machine parameters, c ACM Trans. Math. Soft., 14, 1988, pages 303-311. c c The subroutine statement is: c c subroutine dpeps(dpmeps) c c where c c dpmeps is a double precision variable. c On entry dpmeps need not be specified. c On exit dpmeps is the machine precision. c c MINPACK-2 Project. February 1991. c Argonne National Laboratory and University of Minnesota. c Brett M. Averick. c c ******* integer i,ibeta,irnd,it,itemp,negep double precision a,b,beta,betain,betah,temp,tempa,temp1, + zero,one,two data zero,one,two /0.0d0,1.0d0,2.0d0/ c determine ibeta, beta ala malcolm. a = one b = one 10 continue a = a + a temp = a + one temp1 = temp - a if (temp1 - one .eq. zero) go to 10 20 continue b = b + b temp = a + b itemp = int(temp - a) if (itemp .eq. 0) go to 20 ibeta = itemp beta = dble(ibeta) c determine it, irnd. it = 0 b = one 30 continue it = it + 1 b = b * beta temp = b + one temp1 = temp - b if (temp1 - one .eq. zero) go to 30 irnd = 0 betah = beta/two temp = a + betah if (temp - a .ne. zero) irnd = 1 tempa = a + beta temp = tempa + betah if ((irnd .eq. 0) .and. (temp - tempa .ne. zero)) irnd = 2 c determine dpmeps. negep = it + 3 betain = one/beta a = one do 40 i = 1, negep a = a*betain 40 continue 50 continue temp = one + a if (temp - one .ne. zero) go to 60 a = a*beta go to 50 60 continue dpmeps = a if ((ibeta .eq. 2) .or. (irnd .eq. 0)) go to 70 a = (a*(one + a))/two temp = one + a if (temp - one .ne. zero) dpmeps = a 70 return end c====================== The end of dpmeps ============================== subroutine daxpy(n,da,dx,incx,dy,incy) c c constant times a vector plus a vector. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),da integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if (da .eq. 0.0d0) return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dy(iy) + da*dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,4) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dy(i) + da*dx(i) 30 continue if( n .lt. 4 ) return 40 mp1 = m + 1 do 50 i = mp1,n,4 dy(i) = dy(i) + da*dx(i) dy(i + 1) = dy(i + 1) + da*dx(i + 1) dy(i + 2) = dy(i + 2) + da*dx(i + 2) dy(i + 3) = dy(i + 3) + da*dx(i + 3) 50 continue return end c====================== The end of daxpy =============================== subroutine dcopy(n,dx,incx,dy,incy) c c copies a vector, x, to a vector, y. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*) integer i,incx,incy,ix,iy,m,mp1,n c if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dy(iy) = dx(ix) ix = ix + incx iy = iy + incy 10 continue return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,7) if( m .eq. 0 ) go to 40 do 30 i = 1,m dy(i) = dx(i) 30 continue if( n .lt. 7 ) return 40 mp1 = m + 1 do 50 i = mp1,n,7 dy(i) = dx(i) dy(i + 1) = dx(i + 1) dy(i + 2) = dx(i + 2) dy(i + 3) = dx(i + 3) dy(i + 4) = dx(i + 4) dy(i + 5) = dx(i + 5) dy(i + 6) = dx(i + 6) 50 continue return end c====================== The end of dcopy =============================== double precision function ddot(n,dx,incx,dy,incy) c c forms the dot product of two vectors. c uses unrolled loops for increments equal to one. c jack dongarra, linpack, 3/11/78. c double precision dx(*),dy(*),dtemp integer i,incx,incy,ix,iy,m,mp1,n c ddot = 0.0d0 dtemp = 0.0d0 if(n.le.0)return if(incx.eq.1.and.incy.eq.1)go to 20 c c code for unequal increments or equal increments c not equal to 1 c ix = 1 iy = 1 if(incx.lt.0)ix = (-n+1)*incx + 1 if(incy.lt.0)iy = (-n+1)*incy + 1 do 10 i = 1,n dtemp = dtemp + dx(ix)*dy(iy) ix = ix + incx iy = iy + incy 10 continue ddot = dtemp return c c code for both increments equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dtemp = dtemp + dx(i)*dy(i) 30 continue if( n .lt. 5 ) go to 60 40 mp1 = m + 1 do 50 i = mp1,n,5 dtemp = dtemp + dx(i)*dy(i) + dx(i + 1)*dy(i + 1) + * dx(i + 2)*dy(i + 2) + dx(i + 3)*dy(i + 3) + dx(i + 4)*dy(i + 4) 50 continue 60 ddot = dtemp return end c====================== The end of ddot ================================ subroutine dpofa(a,lda,n,info) integer lda,n,info double precision a(lda,*) c c dpofa factors a double precision symmetric positive definite c matrix. c c dpofa is usually called by dpoco, but it can be called c directly with a saving in time if rcond is not needed. c (time for dpoco) = (1 + 18/n)*(time for dpofa) . c c on entry c c a double precision(lda, n) c the symmetric matrix to be factored. only the c diagonal and upper triangle are used. c c lda integer c the leading dimension of the array a . c c n integer c the order of the matrix a . c c on return c c a an upper triangular matrix r so that a = trans(r)*r c where trans(r) is the transpose. c the strict lower triangle is unaltered. c if info .ne. 0 , the factorization is not complete. c c info integer c = 0 for normal return. c = k signals an error condition. the leading minor c of order k is not positive definite. c c linpack. this version dated 08/14/78 . c cleve moler, university of new mexico, argonne national lab. c c subroutines and functions c c blas ddot c fortran sqrt c c internal variables c double precision ddot,t double precision s integer j,jm1,k c begin block with ...exits to 40 c c do 30 j = 1, n info = j s = 0.0d0 jm1 = j - 1 if (jm1 .lt. 1) go to 20 do 10 k = 1, jm1 t = a(k,j) - ddot(k-1,a(1,k),1,a(1,j),1) t = t/a(k,k) a(k,j) = t s = s + t*t 10 continue 20 continue s = a(j,j) - s c ......exit if (s .le. 0.0d0) go to 40 a(j,j) = sqrt(s) 30 continue info = 0 40 continue return end c====================== The end of dpofa =============================== subroutine dscal(n,da,dx,incx) c c scales a vector by a constant. c uses unrolled loops for increment equal to one. c jack dongarra, linpack, 3/11/78. c modified 3/93 to return if incx .le. 0. c double precision da,dx(*) integer i,incx,m,mp1,n,nincx c if( n.le.0 .or. incx.le.0 )return if(incx.eq.1)go to 20 c c code for increment not equal to 1 c nincx = n*incx do 10 i = 1,nincx,incx dx(i) = da*dx(i) 10 continue return c c code for increment equal to 1 c c c clean-up loop c 20 m = mod(n,5) if( m .eq. 0 ) go to 40 do 30 i = 1,m dx(i) = da*dx(i) 30 continue if( n .lt. 5 ) return 40 mp1 = m + 1 do 50 i = mp1,n,5 dx(i) = da*dx(i) dx(i + 1) = da*dx(i + 1) dx(i + 2) = da*dx(i + 2) dx(i + 3) = da*dx(i + 3) dx(i + 4) = da*dx(i + 4) 50 continue return end c====================== The end of dscal =============================== subroutine dtrsl(t,ldt,n,b,job,info) integer ldt,n,job,info double precision t(ldt,*),b(*) c c c dtrsl solves systems of the form c c t * x = b c or c trans(t) * x = b c c where t is a triangular matrix of order n. here trans(t) c denotes the transpose of the matrix t. c c on entry c c t double precision(ldt,n) c t contains the matrix of the system. the zero c elements of the matrix are not referenced, and c the corresponding elements of the array can be c used to store other information. c c ldt integer c ldt is the leading dimension of the array t. c c n integer c n is the order of the system. c c b double precision(n). c b contains the right hand side of the system. c c job integer c job specifies what kind of system is to be solved. c if job is c c 00 solve t*x=b, t lower triangular, c 01 solve t*x=b, t upper triangular, c 10 solve trans(t)*x=b, t lower triangular, c 11 solve trans(t)*x=b, t upper triangular. c c on return c c b b contains the solution, if info .eq. 0. c otherwise b is unaltered. c c info integer c info contains zero if the system is nonsingular. c otherwise info contains the index of c the first zero diagonal element of t. c c linpack. this version dated 08/14/78 . c g. w. stewart, university of maryland, argonne national lab. c c subroutines and functions c c blas daxpy,ddot c fortran mod c c internal variables c double precision ddot,temp integer case,j,jj c c begin block permitting ...exits to 150 c c check for zero diagonal elements. c do 10 info = 1, n c ......exit if (t(info,info) .eq. 0.0d0) go to 150 10 continue info = 0 c c determine the task and go to it. c case = 1 if (mod(job,10) .ne. 0) case = 2 if (mod(job,100)/10 .ne. 0) case = case + 2 go to (20,50,80,110), case c c solve t*x=b for t lower triangular c 20 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 40 do 30 j = 2, n temp = -b(j-1) call daxpy(n-j+1,temp,t(j,j-1),1,b(j),1) b(j) = b(j)/t(j,j) 30 continue 40 continue go to 140 c c solve t*x=b for t upper triangular. c 50 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 70 do 60 jj = 2, n j = n - jj + 1 temp = -b(j+1) call daxpy(j,temp,t(1,j+1),1,b(1),1) b(j) = b(j)/t(j,j) 60 continue 70 continue go to 140 c c solve trans(t)*x=b for t lower triangular. c 80 continue b(n) = b(n)/t(n,n) if (n .lt. 2) go to 100 do 90 jj = 2, n j = n - jj + 1 b(j) = b(j) - ddot(jj-1,t(j+1,j),1,b(j+1),1) b(j) = b(j)/t(j,j) 90 continue 100 continue go to 140 c c solve trans(t)*x=b for t upper triangular. c 110 continue b(1) = b(1)/t(1,1) if (n .lt. 2) go to 130 do 120 j = 2, n b(j) = b(j) - ddot(j-1,t(1,j),1,b(1),1) b(j) = b(j)/t(j,j) 120 continue 130 continue 140 continue 150 continue return end c====================== The end of dtrsl =============================== gpml-matlab-v4.0-2016-10-19/util/lbfgsb/matlabexception.cpp000644 000765 000024 00000001000 12701705601 023144 0ustar00carlstaff000000 000000 #include "matlabexception.h" // Function definitions for class MatlabException. // ----------------------------------------------------------------- MatlabException::MatlabException (const char* message) throw() : exception() { this->message = message; } MatlabException::MatlabException (const MatlabException& source) throw() : exception() { message = source.message; } MatlabException& MatlabException::operator= (const MatlabException& source) { message = source.message; return *this; } gpml-matlab-v4.0-2016-10-19/util/lbfgsb/arrayofmatrices.cpp000644 000765 000024 00000004741 12701705601 023177 0ustar00carlstaff000000 000000 #ifdef __linux__ #include "mex.h" #else #include "matrix.h" #endif #include "arrayofmatrices.h" // Function definitions for class ArrayOfMatrices. // ----------------------------------------------------------------- ArrayOfMatrices::ArrayOfMatrices (const mxArray* ptr) : Array(getnummatlabmatrices(ptr)) { setvalue(0); if (mxIsCell(ptr)) // Fill in the entries of the array with the entries of the Matlab // cell array. for (int i = 0; i < n; i++) elems[i] = new Matrix(mxGetCell(ptr,i)); else // Fill in the single entry of the array with the Matlab matrix. elems[0] = new Matrix(ptr); } ArrayOfMatrices::ArrayOfMatrices (const mxArray* ptrs[], int numptrs) : Array(numptrs) { setvalue(0); for (int i = 0; i < numptrs; i++) elems[i] = new Matrix(ptrs[i]); } ArrayOfMatrices::ArrayOfMatrices (mxArray* ptrs[], const ArrayOfMatrices& source) : Array(source.length()) { setvalue(0); // Create the new Matlab matrices and copy the data from the // source. for (int i = 0; i < n; i++) { int h = source[i]->height(); int w = source[i]->width(); elems[i] = new Matrix(ptrs[i],h,w); *elems[i] = *source[i]; } } ArrayOfMatrices::ArrayOfMatrices (const ArrayOfMatrices& source) : Array(source) { } ArrayOfMatrices::ArrayOfMatrices (double* data, const ArrayOfMatrices& model) : Array(model.length()) { setvalue(0); // Repeat for each matrix. for (int i = 0; i < n; i++) { const Matrix* A = model[i]; elems[i] = new Matrix(data,A->height(),A->width()); data += A->length(); } } ArrayOfMatrices::~ArrayOfMatrices() { for (int i = 0; i < n; i++) if (elems[i]) delete elems[i]; } int ArrayOfMatrices::numelems() const { int m = 0; // The return value. // Repeat for each matrix. for (int i = 0; i < n; i++) m += elems[i]->length(); return m; } ArrayOfMatrices& ArrayOfMatrices::operator= (const ArrayOfMatrices& source) { // Copy each matrix. for (int i = 0; i < n; i++) *elems[i] = *source.elems[i]; return *this; } int ArrayOfMatrices::getnummatlabmatrices (const mxArray* ptr) { int n; // The return value. if (mxIsCell(ptr)) n = mxGetNumberOfElements(ptr); else if (mxGetNumberOfDimensions(ptr) == 2 && mxIsDouble(ptr)) n = 1; else throw MatlabException("The Matlab array must either be a cell \ array or a double precision matrix"); return n; } gpml-matlab-v4.0-2016-10-19/util/lbfgsb/program.h000644 000765 000024 00000011647 12701705601 021123 0ustar00carlstaff000000 000000 #ifndef INCLUDE_PROGRAM #define INCLUDE_PROGRAM // Type definitions. // ----------------------------------------------------------------- // This defines the possible results of running the L-BFGS-B solver. enum SolverExitStatus { success, // The algorithm has converged to a stationary // point or has reached the maximum number of // iterations. abnormalTermination, // The algorithm has terminated abnormally and // was unable to satisfy the convergence // criteria. errorOnInput // The routine has detected an error in the // input parameters. }; // Constants. // ----------------------------------------------------------------- const int defaultm = 5; const int defaultmaxiter = 100; const double defaultfactr = 1e7; const double defaultpgtol = 1e-5; const int defaultprintlevel = -1; // Function declarations. // ----------------------------------------------------------------- // This is the L-BFGS-B routine implemented in Fortran 77. extern "C" void setulb_ (int* n, int* m, double x[], double l[], double u[], int nbd[], double* f, double g[], double* factr, double* pgtol, double wa[], int iwa[], char task[], int* iprint, char csave[], bool lsave[], int isave[], double dsave[]); // Class Program. // ----------------------------------------------------------------- // This class encapsulates execution of the L-BFGS-B routine for // solving a nonlinear optimization problem with bound constraints // using limited-memory approximations to the Hessian. // // This is an abstract class since some of the class methods have not // been implemented (they are "pure virtual" functions). In order to // use this class, one needs to define a child class and provide // definitions for the pure virtual methods. class Program { public: // The first input argument "n" is the number of variables, and "x" // must be set to the suggested starting point for the optimization // algorithm. See the L-BFGS-B documentation for more information on // the inputs to the constructor. Program (int n, double* x, double* lb, double* ub, int* btype, int m = defaultm, int maxiter = defaultmaxiter, double factr = defaultfactr, double pgtol = defaultpgtol); // This is the same constructor as above, except that the // appropriate amount of memory is dynamically allocated for the // variables, the bounds, and the bound types. The destructor also // makes sure that the memory is properly deallocated, but not so // for the other constructor. Program (int n, int m = defaultm, int maxiter = defaultmaxiter, double factr = defaultfactr, double pgtol = defaultpgtol); // The destructor. virtual ~Program(); // An implementation of this method should return the value of the // objective at the current value of the variables "x", where "n" is // the number of variables. virtual double computeObjective (int n, double* x) = 0; // An implementation of this method should fill in the values of the // gradient "g" and the current point "x". virtual void computeGradient (int n, double* x, double* g) = 0; // The child class may optionally override this method which is // called once per iteration of the L-BFGS-B routine. The input // arguments are the current iteration "t", the current value of the // variables "x" and the current value of the objective "f". virtual void iterCallback (int t, double* x, double f) { }; // Run the solver. Upon completion, the solution is stored in "x". SolverExitStatus runSolver(); protected: // The copy constructor and copy assignment operator are kept // private so that they are not used. Program (const Program& source) { }; Program& operator= (const Program& source) { return *this; }; int n; // The number of variables. double* x; // The current point. double* lb; // The lower bounds. double* ub; // The upper bounds. int* btype; // The bound types. private: // Execute a single step the L-BFGS-B solver routine. void callLBFGS (const char* cmd = 0); // Initialize some of the structures used by the L-BFGS-B routine. void initStructures(); bool owner; // If true, then the blocks of memory pointed to by // x, lb, ub and btype must be deallocated in the // destructor. double f; // The value of the objective. double* g; // The value of the gradient. int iprint; // The print level. int maxiter; // The maximum number of iterations. double pgtol; // Convergence parameter passed to L-BFGS-B. double factr; // Convergence parameter passed to L-BFGS-B. int m; // The number of variable corrections to the // limited-memory approximation to the Hessian. // These are structures used by the L-BFGS-B routine. double* wa; int* iwa; char task[60]; char csave[60]; bool lsave[4]; int isave[44]; double dsave[29]; }; #endif gpml-matlab-v4.0-2016-10-19/util/lbfgsb/matlabscalar.h000644 000765 000024 00000002264 12701705601 022075 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABSCALAR #define INCLUDE_MATLABSCALAR #include "mex.h" // Class MatlabScalar // ----------------------------------------------------------------- // The main appeal of this class is that one can create a scalar // object that accesses a MATLAB array. // // Note that the copy assignment operator is not valid for this class // because we cannot reassign a reference. class MatlabScalar { public: // This constructor accepts as input a pointer to a Matlab array // which must be a scalar in double precision. explicit MatlabScalar (const mxArray* ptr); // This constructor creates a new Matlab array which is a scalar // in double precision. MatlabScalar (mxArray*& ptr, double value); // The copy constructor. MatlabScalar (MatlabScalar& source); // The destructor. ~MatlabScalar() { }; // Access the value of the scalar. operator const double () const { return x; }; // Assign the value of the scalar. MatlabScalar& operator= (double value); protected: double& x; // The copy assignment operator is kept protected because it is // invalid. MatlabScalar& operator= (const MatlabScalar& source) { return *this; }; }; #endif gpml-matlab-v4.0-2016-10-19/util/lbfgsb/matlabmatrix.cpp000644 000765 000024 00000003156 12701705601 022470 0ustar00carlstaff000000 000000 #include "matlabmatrix.h" #include "matlabexception.h" // Function definitions. // ----------------------------------------------------------------- double* getMatlabMatrixDouble (const mxArray* ptr) { if (mxGetNumberOfDimensions(ptr) != 2) throw MatlabException("Matlab array must be a matrix"); if (!mxIsDouble(ptr)) throw MatlabException("Matlab array must be of type double"); return mxGetPr(ptr); } double* createMatlabMatrixDouble (mxArray*& ptr, int height, int width) { ptr = mxCreateDoubleMatrix(height,width,mxREAL); return mxGetPr(ptr); } // Function definitions for class Matrix. // ----------------------------------------------------------------- Matrix::Matrix (int height, int width) : Array(height*width) { h = height; w = width; } Matrix::Matrix (double* data, int height, int width) : Array(data,height*width) { h = height; w = width; } Matrix::Matrix (const mxArray* ptr) : Array(getMatlabMatrixDouble(ptr),mxGetNumberOfElements(ptr)) { h = mxGetM(ptr); w = mxGetN(ptr); } Matrix::Matrix (mxArray*& ptr, int height, int width) : Array(createMatlabMatrixDouble(ptr,height,width), height*width) { h = height; w = width; } Matrix::Matrix (const Matrix& source) : Array(source) { h = source.h; w = source.w; } Matrix& Matrix::operator= (const Matrix& source) { inject(source); return *this; } bool Matrix::operator== (const Matrix& X) const { return (h == X.h) && (w == X.w); } double& Matrix::entry (int r, int c) { return elems[h*c + r]; } double Matrix::entry (int r, int c) const { return elems[h*c + r]; } gpml-matlab-v4.0-2016-10-19/util/lbfgsb/matlabmatrix.h000644 000765 000024 00000004562 12701705601 022137 0ustar00carlstaff000000 000000 #ifndef INCLUDE_MATLABMATRIX #define INCLUDE_MATLABMATRIX #include "array.h" #include "mex.h" // Class Matrix // --------------------------------------------------------------- // A matrix object stores its elements in column-major format, as in // Fortran and Matlab. This means that columns are stored one after // another. For example, the matrix // // 1 2 3 // 4 5 6 // // is stored in memory as // // 1 4 2 5 3 6 // // Like Array objects, Matrix objects are not necessarily // encapsulated, and they exhibit analogous behaviour. class Matrix : public Array { public: // This constructor allocates memory for matrix of the specified // heigth and width. Matrix (int height, int width); // This constructor basically follows the lead of the analagous // constructor for the Array class. Matrix (double* data, int height, int width); // This constructor retrieves a matrix from a Matlab array. This // particular constructor is only defined for Matrix. // Since Matlab handles storage, the object created by this // constructor is not encapsulated. explicit Matrix (const mxArray* ptr); // This constructor creates a new Matlab array as a side effect. // Since Matlab handles storage, the object created by this // constructor is not encapsulated. Matrix (mxArray*& ptr, int height, int width); // The copy constructor makes a shallow copy of the data. Matrix (const Matrix& source); // The destructor. ~Matrix() { }; // Copy assignment operator that observes the same behaviour as // the Array copy assignment operator. Matrix& operator= (const Matrix& source); // Get the height and width of the matrix. int height() const { return h; }; int width () const { return w; }; // Returns true if the two matrices have the same dimensions // (i.e. the same height and width). bool operator== (const Matrix& X) const; bool operator!= (const Matrix& X) const { return !(*this == X); }; // If X is an object of type Matrix, X.entry(r,c) accesses the // entry of the rth row and cth column. double entry (int r, int c) const; double& entry (int r, int c); double operator() (int r, int c) const { return entry(r,c); }; double& operator() (int r, int c) { return entry(r,c); }; protected: int h; // The height of the matrix. int w; // The width of the matrix. }; #endif gpml-matlab-v4.0-2016-10-19/util/lbfgsb/matlabprogram.cpp000644 000765 000024 00000016452 12701705601 022636 0ustar00carlstaff000000 000000 #include "matlabprogram.h" #include "array.h" // Function definitions. // ----------------------------------------------------------------- // Copy the matrix elements to the destination array. It is assumed // that sufficient memory has been allocated for the destination // array. The copying preserves the column-major ordering of the // matrix elements. void copyElemsFromMatrix (const Matrix& source, double* dest) { Matrix destmatrix(dest,source.height(),source.width()); destmatrix = source; } // Copy the elements from the source array into the matrix. void copyElemsToMatrix (const double* source, Matrix& dest) { dest.inject(source); } // Copy the elements contained in all the matrices to the destination // array. It is assumed that sufficient memory has been allocated for // the destination array. The copying preserves the column-major // ordering of the matrix elements. void copyElemsFromArrayOfMatrices (const ArrayOfMatrices& source, double* dest) { // Repeat for each matrix. for (int i = 0; i < source.length(); i++) { copyElemsFromMatrix(*source[i],dest); dest += source[i]->length(); } } // Copy the elements from the source array into the array of matrices. void copyElemsToArrayOfMatrices (const double* source, ArrayOfMatrices& dest) { // Repeat for each matrix. for (int i = 0; i < dest.length(); i++) { copyElemsToMatrix(source,*dest[i]); source += dest[i]->length(); } } int getBoundType (double& lb, double& ub) { bool haslb = !mxIsInf(lb); bool hasub = !mxIsInf(ub); int btype = haslb + 3*hasub - 2*(haslb && hasub); lb = haslb ? lb : 0; ub = hasub ? ub : 0; return btype; } // Function definitions for class MatlabProgram. // ----------------------------------------------------------------- MatlabProgram::MatlabProgram (ArrayOfMatrices& variables, const ArrayOfMatrices& lowerbounds, const ArrayOfMatrices& upperbounds, const MatlabString* objFunc, const MatlabString* gradFunc, const MatlabString* iterFunc, mxArray* auxData, int m, int maxiter, double factr, double pgtol) : Program(variables.numelems(),0,0,0,0,m,maxiter,factr,pgtol), variables(variables) { x = new double[n]; lb = new double[n]; ub = new double[n]; btype = new int[n]; this->objFunc = objFunc; this->gradFunc = gradFunc; this->iterFunc = iterFunc; // Copy some pieces of information from the input arguments to the // respective program structures. copyElemsFromArrayOfMatrices(variables,x); copyElemsFromArrayOfMatrices(lowerbounds,lb); copyElemsFromArrayOfMatrices(upperbounds,ub); // Set the bound types. for (int i = 0; i < n; i++) btype[i] = getBoundType(lb[i],ub[i]); // Create a Matlab array for the variables. int nv = variables.length(); varInputs = new mxArray*[nv]; varMatlab = new ArrayOfMatrices(varInputs,variables); // Set up the inputs to the objective callback function. numInputsObjFunc = nv + (bool) auxData; inputsObjFunc = new mxArray*[numInputsObjFunc]; copymemory(varInputs,inputsObjFunc,nv); if (auxData) inputsObjFunc[nv] = auxData; // Set up the inputs to the gradient callback function. numInputsGradFunc = numInputsObjFunc; inputsGradFunc = inputsObjFunc; // If necessary, set up the inputs to the iterative callback // function. The first two inputs are the iteration and the function // objective. The next of inputs are the values of the variables // then, optionally, the auxiliary data. if (iterFunc) { numInputsIterFunc = 2 + nv + (bool) auxData; inputsIterFunc = new mxArray*[numInputsIterFunc]; mxArray** ptr = inputsIterFunc; tMatlab = new MatlabScalar(*ptr++,0); fMatlab = new MatlabScalar(*ptr++,0); copymemory(varInputs,ptr,nv); ptr += nv; if (auxData) *ptr = auxData; } } MatlabProgram::~MatlabProgram() { delete[] x; delete[] lb; delete[] ub; delete[] btype; // Deallocate the Matlab arrays. int nv = variables.length(); for (int i = 0; i < nv; i++) mxDestroyArray(varInputs[i]); delete varMatlab; delete[] varInputs; // If necessary, deallocate the additional Matlab arrays that act as // inputs to the iterative callback function. if (iterFunc) { delete fMatlab; delete tMatlab; // delete[] inputsIterFunc; } } double MatlabProgram::computeObjective (int n, double* x) { int nlhs = 1; // The number of outputs from Matlab. mxArray* plhs[nlhs]; // The outputs from the Matlab routine. // Copy the current value of the optimization variables. copyElemsToArrayOfMatrices(x,*varMatlab); // Call the designated Matlab routine for evaluating the objective // function. It takes as input the values of the variables, and // returns a single output, the value of the objective function. if (mexCallMATLAB(nlhs,plhs,numInputsObjFunc,inputsObjFunc,*objFunc)) throw MatlabException("Evaluation of objective function failed in \ call to MATLAB routine"); // Get the result passed back from the Matlab routine. MatlabScalar matlabOutput(plhs[0]); double objective = matlabOutput; // Free the dynamically allocated memory. mxDestroyArray(plhs[0]); return objective; } void MatlabProgram::computeGradient (int n, double* x, double* g) { int nlhs = variables.length(); // The number of outputs from Matlab. mxArray* plhs[nlhs]; // The outputs from the Matlab routine. // Copy the current value of the optimization variables. copyElemsToArrayOfMatrices(x,*varMatlab); // Call the designated Matlab routine for computing the gradient // of the objective function. It takes as input the values of the // variables, and returns as many outputs corresponding to the // partial derivatives of the objective function with respect to // the variables at their curent values. if (mexCallMATLAB(nlhs,plhs,numInputsGradFunc,inputsGradFunc,*gradFunc)) throw MatlabException("Evaluation of objective gradient failed in \ call to MATLAB routine"); // Get the result passed back from the Matlab routine. ArrayOfMatrices matlabOutput((const mxArray**) plhs,nlhs); for (int i = 0; i < variables.length(); i++) if (matlabOutput[i]->length() != variables[i]->length()) throw MatlabException("Invalid gradient passed back from MATLAB \ routine"); copyElemsFromArrayOfMatrices(matlabOutput,g); // Free the dynamically allocated Matlab outputs. for (int i = 0; i < variables.length(); i++) mxDestroyArray(plhs[i]); } void MatlabProgram::iterCallback (int t, double* x, double f) { if (iterFunc) { // Copy the current iteration, the current value of the objective // function, and the current value of the optimization variables. *tMatlab = t; *fMatlab = f; copyElemsToArrayOfMatrices(x,*varMatlab); // Call the Matlab iterative callback routine. Since there are no // outputs, there is no memory to deallocate. if (mexCallMATLAB(0,0,numInputsIterFunc,inputsIterFunc,*iterFunc)) throw MatlabException("Call to MATLAB iterative callback function \ failed"); } } SolverExitStatus MatlabProgram::runSolver() { SolverExitStatus status; // The return value. status = Program::runSolver(); // Copy the solution to the class member "variables". copyElemsToArrayOfMatrices(x,variables); return status; } gpml-matlab-v4.0-2016-10-19/mean/meanConst.m000644 000765 000024 00000001111 12757566731 020121 0ustar00carlstaff000000 000000 function [m,dm] = meanConst(hyp, x) % Constant mean function. The mean function is parameterized as: % % m(x) = c % % The hyperparameter is: % % hyp = [ c ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-15. % % See also MEANFUNCTIONS.M. if nargin<2, m = '1'; return; end % report number of hyperparameters if numel(hyp)~=1, error('Exactly one hyperparameter needed.'), end c = hyp; m = c*ones(size(x,1),1); % mean dm = @(q) sum(q); % directional derivative gpml-matlab-v4.0-2016-10-19/mean/meanSum.m000644 000765 000024 00000003000 12757566731 017576 0ustar00carlstaff000000 000000 function [m,dm] = meanSum(mean, hyp, x) % meanSum - compose a mean function as the sum of other mean functions. % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean functions to do the actual work. % % m(x) = \sum_i m_i(x) % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2016-04-15. % % See also MEANFUNCTIONS.M. nm = numel(mean); for ii = 1:nm % iterate over mean functions f = mean(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary j(ii) = cellstr(feval(f{:})); % collect number hypers end if nargin<3 % report number of parameters m = char(j(1)); for ii=2:nm, m = [m, '+', char(j(ii))]; end; return end [n,D] = size(x); v = []; % v vector indicates to which mean parameters belong for ii = 1:nm, v = [v repmat(ii, 1, eval(char(j(ii))))]; end m = zeros(n,1); dmi = cell(nm,1); % allocate space for ii = 1:nm % iteration over summand functions f = mean(ii); if iscell(f{:}), f = f{:}; end % expand cell array if needed [mi,dmi{ii}] = feval(f{:}, hyp(v==ii), x); m = m+mi; % accumulate means end dm = @(q) dirder(q,dmi,v,nm); % directional derivative function dhyp = dirder(q,dmi,v,nm) dhyp = zeros(nm,1); for ii = 1:nm, dhyp(v==ii,1) = dmi{ii}(q); endgpml-matlab-v4.0-2016-10-19/mean/meanWarp.m000644 000765 000024 00000001403 12757566731 017750 0ustar00carlstaff000000 000000 function [m,dm] = meanWarp(mean, g, dg, hyp, x) % meanWarp - compose a mean function by warping another mean function m0. % % m(x) = g( m_0(x) ). % % The hyperparameter is: % % hyp = [ hyp_m0 ] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean function to do the actual work. % % Copyright (c) by William Herlands & Hannes Nickisch, 2016-04-15. % % See also MEANFUNCTIONS.M if nargin<5, m = feval(mean{:}); return, end % report number of parameters [m0,dm0] = feval(mean{:},hyp,x); % evaluate m0 m = g(m0); % mean dm = @(q) dm0(dg(m0).*q); % directional derivativegpml-matlab-v4.0-2016-10-19/mean/meanPoly.m000644 000765 000024 00000002170 12757566731 017764 0ustar00carlstaff000000 000000 function [m,dm] = meanPoly(d, hyp, x) % meanPoly - compose a mean function as a polynomial. % % The degree d has to be a strictly positive integer. % % m(x) = sum_i=1..D sum_j=1..d a_ij * x_i^j % % The hyperparameter is: % % hyp = [ a_11 % a_21 % .. % a_D1 % a_12 % a_22 % .. % a_Dd] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean function to do the actual work. % % Copyright (c) by Hannes Nickisch 2016-04-15. % % See also MEANFUNCTIONS.M. d = max(abs(floor(d)),1); % positive integer degree if nargin<3, m = ['D*',int2str(d)]; return; end % report number of hyperparams [n,D] = size(x); a = reshape(hyp,D,d); m = zeros(n,1); % allocate memory for j=1:d, m = m + (x.^j)*a(:,j); end % evaluate mean dm = @(q) dirder(q,x,a); % directional derivative function dhyp = dirder(q,x,a) [D,d] = size(a); dhyp = zeros(D*d,1); for j=1:d, dhyp((j-1)*D+(1:D)) = (x.^j)'*q(:); endgpml-matlab-v4.0-2016-10-19/mean/meanPref.m000644 000765 000024 00000001427 12757566731 017741 0ustar00carlstaff000000 000000 function [m,dm] = meanPref(mean, hyp, x) % meanPref - mean function for preference learning. % % m(x) = m_0(x1)-m_0(x2), where x1=x(:,1:D), x2=x(:,D+1:2*D), D = size(x,2)/2. % % The hyperparameters are: % % hyp = [ hyp_m0 ] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls another mean function to do the actual work. % % See Collaborative Gaussian Processes for Preference Learning, NIPS 2014. % % Copyright (c) by Hannes Nickisch and Roman Garnett, 2016-04-16. % % See also MEANFUNCTIONS.M and COVPREF.M. if nargin<3, m = strrep(feval(mean{:}),'D','D/2'); return; end % no of params [m1,dm1] = feval(mean{:}, hyp, x(:,1 :end/2)); [m2,dm2] = feval(mean{:}, hyp, x(:,1+end/2:end )); m = m1-m2; dm = @(q) dm1(q)-dm2(q);gpml-matlab-v4.0-2016-10-19/mean/meanWSPC.m000644 000765 000024 00000002203 12777177045 017610 0ustar00carlstaff000000 000000 function [m,dm] = meanWSPC(d, hyp, x) % Weighted Sum of Projected Cosines or Random Kitchen Sink features. % % This function represents the feature function of a zero mean GP with % stationary covariance function. See the paper "Sparse spectrum GP regression" % by Lazaro-Gredilla et al., JMLR, 2010 for details. % % m(x) = sqrt(2/d) sum_j=1..d a_j * cos(w_j'*x + b_j) % % The hyperparameter is: % hyp = [w_1; b_1; a_1; ..; w_d; b_d; a_d] % % Copyright (c) by William Herlands and Hannes Nickisch, 2016-04-15. % % See also MEANFUNCTIONS.M. if nargin<3, m = sprintf('(D+2)*%d',d); return; end % report number of hypers [n,D] = size(x); if any(length(hyp)~=eval(sprintf('(D+2)*%d',d))) error('Incorrect number of hyperparameters for meanRKS.') end hyp = reshape(hyp,D+2,d); w = hyp(1:D,:); b = hyp(D+1,:); a = hyp(D+2,:)'; % separate hyps into w, b, a r = bsxfun(@plus,x*w,b); cr = cos(r); m = sqrt(2/d)*cr*a; % mean dm = @(q) dirder(q,x,a,r,cr,d); % directional derivative function dhyp = dirder(q,x,a,r,cr,d) msr = -sin(r); dhyp = [x'*(msr.*(q*a')); (q'*msr).*a'; q'*cr]; dhyp = sqrt(2/d)*dhyp(:);gpml-matlab-v4.0-2016-10-19/mean/meanZero.m000644 000765 000024 00000000726 12757566731 017765 0ustar00carlstaff000000 000000 function [m,dm] = meanZero(hyp, x) % Zero mean function. The mean function does not have any parameters. % % m(x) = 0 % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-15. % % See also MEANFUNCTIONS.M. if nargin<2, m = '0'; return; end % report number of hyperparameters m = zeros(size(x,1),1); % mean dm = @(q) zeros(0,1); % directional derivative gpml-matlab-v4.0-2016-10-19/mean/meanPow.m000644 000765 000024 00000002167 12757566731 017614 0ustar00carlstaff000000 000000 function [m,dm] = meanPow(d, mean, hyp, x) % meanPow - compose a mean function as the power of another mean function m0. % % m(x) = m_0(x) ^ d % % If the degree d is not a strictly positive integer, we use % m(x) = sign(m_0(x)) * abs(m_0(x)) ^ d % to stay within the reals % % The hyperparameter is: % % hyp = [ hyp_m0 ] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean function to do the actual work. % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2016-05-04. % % See also MEANFUNCTIONS.M. if nargin<4 % report number of parameters m = feval(mean{:}); return end [m0,dm0] = feval(mean{:},hyp,x); % evaluate m0 if d>0 && ceil(d)==d % strictly positive integer s = 1; else % general real number s = sign(m0); m0 = abs(m0); end m = s.*m0.^d; % mean dm = @(q) dm0( (d*m0.^(d-1)).*q ); % directional derivativegpml-matlab-v4.0-2016-10-19/mean/meanMask.m000644 000765 000024 00000001730 12757566731 017735 0ustar00carlstaff000000 000000 function [m,dm] = meanMask(mask, mean, hyp, x) % Apply a mean function to a subset of the dimensions only. The subset can % either be specified by a 0/1 mask by a boolean mask or by an index set. % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls another mean function to do the actual work. % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-16. % % See also MEANFUNCTIONS.M. nh_string = feval(mean{:}); % number of hyperparameters of the full mean if max(mask)<2 && length(mask)>1, mask = find(mask); end % convert 1/0->index D = length(mask); % masked dimension if nargin<4, m = num2str(eval(nh_string)); return; end % number of parameters if eval(nh_string)~=length(hyp) % check hyperparameters error('number of hyperparameters does not match size of masked data') end [m,dm] = feval(mean{:}, hyp, x(:,mask)); gpml-matlab-v4.0-2016-10-19/mean/meanLinear.m000644 000765 000024 00000001221 12757566731 020247 0ustar00carlstaff000000 000000 function [m,dm] = meanLinear(hyp, x) % Linear mean function. The mean function is parameterized as: % % m(x) = sum_i c_i * x_i; % % The hyperparameter is: % % hyp = [ c_1 % c_2 % .. % c_D ] % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-15. % % See also MEANFUNCTIONS.M. if nargin<2, m = 'D'; return; end % report number of hyperparameters [n,D] = size(x); if any(size(hyp)~=[D,1]), error('Exactly D hyperparameters needed.'), end m = x*hyp(:); % evaluate mean dm = @(q) x'*q(:); % directional derivative gpml-matlab-v4.0-2016-10-19/mean/meanOne.m000644 000765 000024 00000000723 12757566731 017564 0ustar00carlstaff000000 000000 function [m,dm] = meanOne(hyp, x) % One mean function. The mean function does not have any parameters. % % m(x) = 1 % % Copyright (c) by Carl Edward Rasmussen and Hannes Nickisch, 2016-04-15. % % See also MEANFUNCTIONS.M. if nargin<2, m = '0'; return; end % report number of hyperparameters m = ones(size(x,1),1); % mean dm = @(q) zeros(0,1); % directional derivative gpml-matlab-v4.0-2016-10-19/mean/meanGPexact.m000644 000765 000024 00000004127 12777177046 020377 0ustar00carlstaff000000 000000 function [m,dm] = meanGPexact(mean,cov,x,y, hypz,z) % Mean function being the predictive mean of a GP model: % % mu(z) = posterior mean of GP at location z as given by % mu(z) = gp(hyp,@infExact,mean,cov,@likGauss,x,y, z) where % hyp.mean = hyp_mean; hyp.lik = log(sn); hyp.cov = hyp.cov; % % The hyperparameters are: % % hypz = [ hyp_cov % log(sn) % hyp_mean ] % % where hyp_cov are the covariance function hyperparameters, sn is the % noise variance of the Gaussian likelihood and hyp_mean are the mean % function hyperparameters. % % Copyright (c) by Hannes Nickisch, 2016-04-16. % % See also MEANFUNCTIONS.M and MEANGP.M. if nargin<4, error('GP must be specified.'), end % check for dimension if isempty(mean), mean = @meanZero; end % set default and make cell if ~iscell(mean), mean = {mean}; end if isempty(cov), cov = @covSEiso; end if ~iscell(cov), cov = {cov}; end nms = feval(mean{:}); ncs = feval(cov{:}); % number of hyperparameter string if nargin<6, m = [ncs,'+1+',nms]; return, end % report number of hyperparameters [nz,D] = size(z); n = size(x,1); nc = eval(ncs); nm = eval(nms); hyp = vec2any(struct('cov',zeros(nc,1),'lik',0,'mean',zeros(nm,1)),hypz); [mu,dmu] = feval(mean{:},hyp.mean,x); [muz,dmuz] = feval(mean{:},hyp.mean,z); [K,dK] = feval(cov{:}, hyp.cov, x); [kz,dkz] = feval(cov{:}, hyp.cov, x,z); sn2 = exp(2*hyp.lik); % noise variance of likGauss if sn2<1e-6 % very tiny sn2 can lead to numerical trouble L = chol(K+sn2*eye(n)); sl = 1; % Cholesky factor of covariance with noise else L = chol(K/sn2+eye(n)); sl = sn2; % Cholesky factor of B end iKs = @(t) solve_chol(L,t)/sl; % iKs(t) = (K+sn2*eye(n))\t alpha = iKs(y-mu); m = muz+kz'*alpha; % eval posterior mean dm = @(q) dirder(q,alpha,dmu,dmuz,kz,dkz,iKs,dK,sn2); % directional derivative function dmdhyp = dirder(q,alpha,dmu,dmuz,kz,dkz,iKs,dK,sn2) v = iKs(kz*q); dmdhyp = [dkz(alpha*q')-dK(alpha*v'); -2*sn2*v'*alpha; dmuz(q)-dmu(v)];gpml-matlab-v4.0-2016-10-19/mean/meanGP.m000644 000765 000024 00000001263 12757566731 017351 0ustar00carlstaff000000 000000 function [m,dm] = meanGP(hyp,inf,mean,cov,lik,x,y, hypz,z) % Mean function being the predictive mean of a GP model: % % m(z) = posterior mean of another GP at location z as given by % m(z) = gp(hyp,inf,mean,cov,lik,x,y, z) % % The hyperparameters are: % % hypz = [ ] % % Copyright (c) by Hannes Nickisch, 2016-04-16. % % See also MEANFUNCTIONS.M and MEANGPEXACT.M. if nargin<7, error('GP must be specified.'), end % check for dimension if nargin<9, m = '0'; return, end % report number of hyperparameters m = gp(hyp,inf,mean,cov,lik,x,y, z); % evaluate posterior mean dm = @(q) zeros(0,1); % directional derivativegpml-matlab-v4.0-2016-10-19/mean/meanDiscrete.m000644 000765 000024 00000001652 12757566731 020607 0ustar00carlstaff000000 000000 function [m,dm] = meanDiscrete(s, hyp, x) % Mean function for discrete inputs x. Given a function defined on the % integers 1,2,3,..,s, the mean function is parametrized as: % % m(x) = mu_x, % % where mu is a fixed vector of length s. % % This implementation assumes that the inputs x are given as integers % between 1 and s, which simply index the provided vector. % % The hyperparameters are: % % hyp = [ mu_1 % mu_2 % .. % mu_s ] % % Copyright (c) by Roman Garnett and Hannes Nickisch, 2016-04-16. % % See also COVDISCRETE.M, MEANFUNCTIONS.M. if nargin==0, error('s must be specified.'), end % check for dimension if nargin<=2, m = num2str(s); return; end % report number of hyperparameters mu = hyp(:); m = mu(x(:)); % evaluate mean dm = @(q) dirder(q,s,x); function dmdhyp = dirder(q,s,x) dmdhyp = zeros(s,1); for i=1:s, dmdhyp(i) = sum(q(x==i)); endgpml-matlab-v4.0-2016-10-19/mean/meanNN.m000644 000765 000024 00000001315 12757566731 017354 0ustar00carlstaff000000 000000 function [m,dm] = meanNN(c,v, hyp, x) % Nearest neighbor mean function. The mean function is parameterized as: % % m(z) = v_j, j = arg min_i d(ci,x) where d is the Euclidean distance and ci is % the ith cluster center. % % The hyperparameters are: % % hyp = [ ] % % Copyright (c) by Hannes Nickisch, 2016-04-16. % % See also MEANFUNCTIONS.M. if nargin<4, m = '0'; return; end % report number of hyperparameters if numel(hyp)~=0, error('No hyperparameters needed for this model.'), end [junk,j] = min(sq_dist(c',x')); m = v(j); m = m(:); % evaluate mean dm = @(q) zeros(0,1); % directional derivative gpml-matlab-v4.0-2016-10-19/mean/meanProd.m000644 000765 000024 00000003145 12757566731 017750 0ustar00carlstaff000000 000000 function [m,dm] = meanProd(mean, hyp, x) % meanProd - compose a mean function as the product of other mean functions. % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean functions to do the actual work. % % m(x) = \prod_i m_i(x) % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2016-04-15. % % See also MEANFUNCTIONS.M. nm = numel(mean); for ii = 1:nm % iterate over mean functions f = mean(ii); if iscell(f{:}), f = f{:}; end % expand cell array if necessary j(ii) = cellstr(feval(f{:})); % collect number hypers end if nargin<3 % report number of parameters m = char(j(1)); for ii=2:nm, m = [m, '+', char(j(ii))]; end; return end [n,D] = size(x); v = []; % v vector indicates to which mean parameters belong for ii = 1:nm, v = [v repmat(ii, 1, eval(char(j(ii))))]; end m = ones(n,1); mi = cell(nm,1); dmi = cell(nm,1); % allocate space for ii = 1:nm % iteration over factor functions f = mean(ii); if iscell(f{:}), f = f{:}; end % expand cell array if needed [mi{ii},dmi{ii}] = feval(f{:}, hyp(v==ii), x); m = m.*mi{ii}; % accumulate means end dm = @(q) dirder(q,mi,dmi,v,nm); % directional derivative function dhyp = dirder(q,mi,dmi,v,nm) dhyp = zeros(nm,1); for ii = 1:nm qi = q; for jj=1:nm, if ii~=jj, qi = qi .* mi{jj}; end, end % accumulate dhyp(v==ii,1) = dmi{ii}(qi); endgpml-matlab-v4.0-2016-10-19/mean/meanScale.m000644 000765 000024 00000001505 12757566731 020071 0ustar00carlstaff000000 000000 function [m,dm] = meanScale(mean, hyp, x) % meanScale - compose a mean function as a scaled version of another one. % % m(x) = a * m_0(x) % % The hyperparameters are: % % hyp = [ a; % hyp_m0 ] % % This function doesn't actually compute very much on its own, it merely does % some bookkeeping, and calls other mean functions to do the actual work. % % Copyright (c) by Carl Edward Rasmussen & Hannes Nickisch 2016-04-15. % % See also MEANFUNCTIONS.M. if nargin<3 % report number of parameters m = [feval(mean{:}),'+1']; return end a = hyp(1); [m0,dm0] = feval(mean{:},hyp(2:end),x); % evaluate m0 m = a*m0; % mean dm = @(q) [m0'*q(:); a*dm0(q)]; % directional derivative